Specific emitter identification (SEI) is a potential physical layer authentication technology, which is one of the most critical complements of upper layer authentication. Radio frequency fingerprint (RFF)-based SEI is to distinguish one emitter from each other by immutable RF characteristics from electronic components. Due to the powerful ability of deep learning (DL) to extract hidden features and perform classification, it can extract highly separative features from massive signal samples, thus enabling SEI. Considering the condition of limited training samples, we propose a novel few-shot SEI (FS-SEI) method based on hybrid data augmentation and deep metric learning (HDA-DML) which gets rid of the dependence on auxiliary datasets. Specifically, HDA consisting rotation and CutMix is designed to increase data diversity, and DML is used to extract high discriminative semantic features. The proposed HDA-DML-based FS-SEI method is evaluated on an open source large-scale real-world automatic-dependent surveillance-broadcast (ADS-B) dataset and a real-world WiFi dataset. The simulation results of two datasets show that the proposed method achieves better identification performance and higher feature discriminability than five latest FS-SEI methods.