Weight reduction and low power consumption are key requirements in the next generation of unmanned aerial vehicle (UAV) networks. Employing modulating retro-reflector (MRR)-based free space optical (FSO) technology is an innovative technique for UAV-to-ground communication in order to reduce the payload weight and power consumption of UAVs which leads to increased maneuverability and flight time of UAV. In this paper, we consider an MRR-based FSO system for UAV-to-ground communication. We will show that the performance of the considered system is very sensitive to tracking errors. Therefore, to assess the benefits of MRR-based UAV deployment for FSO communications, the MRR-based UAV FSO channel is characterized by taking into account tracking system errors along with UAV's orientation fluctuations, link length, UAV's height, optical beam divergence angle, effective area of MRR, atmospheric turbulence and optical channel loss in the double-pass channels. To enable effective performance analysis, tractable and closed-form expressions are derived for probability density function of end-to-end signal to noise ratio, outage probability and bit error rate of the considered system under both weak-to-moderate and moderate-to-strong atmospheric turbulence conditions. The accuracy of the analytical expressions is verified by extensive simulations. Analytical results are then used to study the relationship between the optimal system design and tracking system errors.