Elevator button recognition is an important function to realize the autonomous operation of elevators. However, challenging image conditions and various image distortions make it difficult to accurately recognize buttons. In this work, We propose a novel algorithm that can automatically correct perspective distortions of elevator panel images based on button corner detection results. The algorithm first leverages DeepLabv3+ model and Hough Transform method to obtain button segmentation results and button corner detection results, then utilizes pixel coordinates of standard button corners as reference features to estimate camera motions for correcting perspective distortions. The algorithm is much more robust to outliers and noise on the removal of perspective distortion than traditional geometric approaches as it only performs on a single image autonomously. 15 elevator panel images are captured from different angles of view as the dataset. The experimental results show that our approach significantly outperforms traditional geometric techniques in accuracy and robustness. Rectification results of the proposed algorithm is 77.4% better than the results of traditional geometric algorithm in average.