Automatic detection of people in aerial images has potential applications in traffic monitoring, surveillance, human behavior analysis, etc. However, developing an algorithm for detection of human locations in aerial images is challenging because of the small target size, cluttered background, and varying appearance of humans. Deep learning-based object detections frameworks internally use the standard convolutional neural network (CNN) based classifiers for feature extraction and classification. Though these pre-trained classifiers perform image classification tasks with very good accuracy, they are computationally complex and hence require huge computation time. In this work, we custom-designed CNN-based classifiers to perform the human classification in aerial images and compared the performance with the standard VGG-16 based human classifier. Custom-designed classifier with fewer number of layers achieved a reduced computation time while maintaining good accuracy.