The efficiency of portrait image selection and analysis systems is completely dependent on the quality of the face image, which depends on various factors. Since real-time manual selection of high-quality portrait photos from a sequence of different frames or images is usually impossible, using automatic methods can be useful in selecting photos, especially in large collections. Existing automatic methods may not be able to perform like humans in portrait classification. These methods may consider only special factors like emotional state or gaze direction to select an image. In this work, a large collection of facial images was collected, and under a subjective quality assessment study, they were judged by more than 80 people. A deep classifier network using transfer learning and the fine-tuning approach is proposed, which is learned end-to-end having the subjective labels to select good portrait images objectively. Quantitative and qualitative results show that this model performs better than state-of-the-art image classification networks. In addition, our qualitative evaluation showed that our model can separate good portrait images in the way that humans do. Therefore, this model can be reliably used in mobile phones, digital Cameras, and other imaging systems.