Current networking protocols deem inefficient in accommodating the two key challenges of Unmanned Aerial Vehicle (UAV) networks, namely the network connectivity loss and energy limitations. One approach to solve these issues is using learning-based routing protocols to make close-to-optimal local decisions by the network nodes, and Q-routing is a bold example of such protocols. However, the performance of the current implementations of Q-routing algorithms is not yet satisfactory, mainly due to the lack of adaptability to continued topology changes. In this paper, we propose a full-echo Q-routing algorithm with a self-adaptive learning rate that utilizes Simulated Annealing (SA) optimization to control the exploration rate of the algorithm through the temperature decline rate, which in turn is regulated by the experienced variation rate of the Q-values. Our results show that our method adapts to the network dynamicity without the need for manual re-initialization at transition points (abrupt network topology changes). Our method exhibits a reduction in the energy consumption ranging from 7% up to 82%, as well as a 2.6 fold gain in successful packet delivery rate}, compared to the state of the art Q-routing protocols