The ubiquity of unmanned aerial vehicles (UAVs) or drones is posing both security and safety risks to the public as UAVs are now used for cybercrimes. To mitigate these risks, it is important to have a system that can detect or identify the presence of an intruding UAV in a restricted environment. In this work, we propose a radio frequency (RF) based UAV detection and identification system by exploiting signals emanating from both the UAV and its flight controller, respectively. While several RF devices (i.e., Bluetooth and WiFi devices) operate in the same frequency band as UAVs, the proposed framework utilizes a semi-supervised learning approach for the detection of UAV or UAV's control signals in the presence of other wireless signals such as Bluetooth and WiFi. The semi-supervised learning approach uses stacked denoising autoencoder and local outlier factor algorithms. After the detection of UAV or UAV's control signals, the signal is decomposed by using Hilbert-Huang transform and wavelet packet transform to extract features from the time-frequency-energy domain of the signal. The extracted feature sets are used to train a three-level hierarchical classifier for identifying the type of signals (i.e., UAV or UAV control signal), UAV models, and flight mode of UAV. To demonstrate the feasibility of the proposed framework, we carried out an outdoor experiment for data collection using six UAVs, five Bluetooth devices, and two WiFi devices. The acquired data is called Cardinal RF (CardRF) dataset, and it is available for public use to foster UAV detection and identification research.