Ear biometric is considered as one of the most reliable and invariant biometrics characteristics in line with iris and fingerprint characteristics. In many cases, ear biometrics can be compared with face biometrics regarding many physiological and texture characteristics. In this paper, a robust and efficient ear recognition system is presented, which uses Scale Invariant Feature Transform (SIFT) as feature descriptor for structural representation of ear images. In order to make it more robust to user authentication, only the regions having color probabilities in a certain ranges are considered for invariant SIFT feature extraction, where the K-L divergence is used for keeping color consistency. Ear skin color model is formed by Gaussian mixture model and clustering the ear color pattern using vector quantization. Finally, K-L divergence is applied to the GMM framework for recording the color similarity in the specified ranges by comparing color similarity between a pair of reference model and probe ear images. After segmentation of ear images in some color slice regions, SIFT keypoints are extracted and an augmented vector of extracted SIFT features are created for matching, which is accomplished between a pair of reference model and probe ear images. The proposed technique has been tested on the IITK Ear database and the experimental results show improvements in recognition accuracy while invariant features are extracted from color slice regions to maintain the robustness of the system.