With the deep integration between the unmanned aerial vehicle (UAV) and wireless communication, UAV-based air-to-ground (AG) propagation channels need more detailed descriptions and accurate models. In this paper, we aim to perform cluster-based characterization and modeling for AG channels. To our best knowledge, this is the first study that concentrates on the clustering and tracking of multipath components (MPCs) for time-varying AG channels. Based on measurement data at 6.5 GHz with 500 MHz of bandwidth, we first estimate potential MPCs utilizing the space-alternating generalized expectation-maximization (SAGE) algorithm. Then, we cluster the extracted MPCs considering their static and dynamic characteristics by employing K-Power-Means (KPM) algorithm under multipath component distance (MCD) measure. For characterizing time-variant clusters, we exploit a clustering-based tracking (CBT) method, which efficiently quantifies the survival lengths of clusters. Ultimately, we establish a cluster-based channel model, and validations illustrate the accuracy of the proposed model. This work not only promotes a better understanding of AG propagation channels but also provides a general cluster-based AG channel model with certain extensibility.