Recently, quaternion collaborative representation-based classification (QCRC) and quaternion sparse representation-based classification (QSRC) have been proposed for color face recognition. They can obtain correlation information among different color channels. However, their performance is unstable in different conditions. For example, QSRC performs better than than QCRC on some situations but worse on other situations. To benefit from quaternion-based $e_2$-norm minimization in QCRC and quaternion-based $e_1$-norm minimization in QSRC, we propose the quaternion-based adaptive representation (QAR) that uses a quaternion-based $e_p$-norm minimization ($1 \le p \le 2$) for color face recognition. To obtain the high dimension correlation information among different color channels, we further propose the high-dimension quaternion-based adaptive representation (HD-QAR). The experimental results demonstrate that the proposed QAR and HD-QAR achieve better recognition rates than QCRC, QSRC and several state-of-the-art methods.