Representation based classification method (RBCM) remains one of the hottest topics in the community of pattern recognition, and the recently proposed non-negative representation based classification (NRC) achieved impressive recognition results in pattern classification. However, there is no regularization term other than the reconstruction error term in the formulation of NRC, which may result in unstable solution leading to misclassification. To overcome this drawback of NRC, in this paper, we propose a class-specific residual constraint non-negative representation (CRNR) for pattern classification. CRNR introduces a class-specific residual constraint into the formulation of NRC, which encourages more homogeneous training samples to participate in the representation of the test sample. Based on the proposed CRNR, we develop a CRNR based classifier (CRNRC) for pattern classification. Experimental results on several benchmark datasets demonstrate the superiority of CRNRC over conventional RBCM as well as the recently proposed NRC. Moreover, CRNRC works better or comparable to some state-of-the-art deep approaches on diverse challenging pattern classification tasks. The source code of our proposed CRNRC is accessible at https://github.com/yinhefeng/CRNRC