Facial attractiveness has long been a topic of interest for cognitive scientists. Early psychological research has found that averageness, distinctiveness and familiarity of a face can influence facial attractiveness. However, faces also convey rich social information. How various social features are related to facial attractiveness hasn't been systematically studied before. We investigate facial attractiveness in the context of social feature evaluation and find that social attributes like appearing interesting and sociable contribute to facial attractiveness whereas appearing boring, and humble are negatively correlated with attractiveness. We further compare social features of faces with the physical configuration of faces and we are able to use geometric features to predict facial attractiveness. We further study the individual differences on attractiveness perception and find out that. Our study illustrates that social attributes and pixel information can go hand in hand to facilitate attractiveness prediction.