Jittering effects significantly degrade the performance of UAV communications especially in millimeter-wave (mmWave) band. To investigate and mitigate the impacts of UAV jitter to mmWave communications, we firstly model UAV mmWave channel based on the geometric relationship between element antennas of the uniform planar arrays (UPAs) in receiver side and transmitter side, and we incorporate the jittering effects to our channel model through extracting the relationship between UAV attitude & position and angle of arrival (AoA) & angle of departure (AoD) of the UPAs. Then, based on the extracted relationship, we propose to utilize UAV navigation information to obtain a rough estimation of AoA and AoD, and we also analyze the impact of AoA and AoD estimation error to UAV beamforming. Finally, we propose a direction-constrained beam training scheme to refine the AoA/AoD estimation. Particularly, we construct a partially random sensing matrix to measure the channel within a narrow angle range that is centered at the aforementioned rough estimate of AoA/AoD. Numerical results show that our proposed UAV beam training scheme with navigation information is able to fast and accurately estimate the AoA/AoD fluctuation caused by UAV jitter.