Gait recognition is an important recognition technology, because it is not easy to camouflage and does not need cooperation to recognize subjects. However, there are still serious challenges in gait recognition, that is, people with similar walking posture are often recognized incorrectly. In this paper, We try to increase the discrimination of extraced gait features of different subjects to increase the recognition efficiency of subjects with similar walking posture. It includes the optimization of network structure and the refinement of extracted gait features. So our method is proposed, it consists of Spatio-temporal Feature Extraction (SFE) and Adaptive Distance Alignment (ADA), which SFE uses Temporal Feature Fusion (TFF) and Fine-grained Feature Extraction (FFE) to effectively extract the spatio-temporal features from raw silhouettes, ADA uses a large number of unlabeled gait data in real life as a benchmark to refine the extracted spatio-temporal features to make them have low inter-class similarity and high intra-class similarity. Extensive experiments on mini-OUMVLP and CASIA-B have proved that we have a good result than some state-of-the-art methods.