Unmanned aerial vehicles (UAVs) are the emerging vital components of millimeter wave (mmWave) wireless systems. Accurate beam alignment is essential for efficient beam-based mmWave communications of UAVs with base stations (BSs). Conventional beam sweeping approaches often have large overhead due to the high mobility and autonomous operation of UAVs. Learning-based approaches greatly reduce the overhead by leveraging UAV data, like position to identify optimal beam directions. In this paper, we propose a reinforcement learning (RL)-based framework for UAV-BS beam alignment using deep Q-Network (DQN) in a mmWave setting. We consider uplink communications where the UAV hovers around 5G new radio (NR) BS coverage area, with varying channel conditions. The proposed learning framework uses the location information to maximize data rate through the optimal beam-pairs efficiently, upon every communication request from UAV inside the multi-location environment. We compare our proposed framework against the Multi-Armed Bandit (MAB) learning-based approach and the traditional exhaustive approach, respectively, and also analyse the training performance of DQN-based beam alignment over different coverage area requirements and channel conditions. Our results show that the proposed DQN-based beam alignment converges faster and generic for different environmental conditions. The framework can also learn optimal beam alignment comparable to the exhaustive approach in an online manner under real-time conditions.