Gait recognition has a broad application in social security due to its advantages in long-distance human identification. Despite the high accuracy of gait recognition systems, their adversarial robustness has not been explored. In this paper, we demonstrate that the state-of-the-art gait recognition model is vulnerable to adversarial attacks. A novel temporal sparse adversarial attack under a new defined distortion measurement is proposed. GAN-based architecture is employed to semantically generate adversarial high-quality gait silhouette. By sparsely substituting or inserting a few adversarial gait silhouettes, our proposed method can achieve a high attack success rate. The imperceptibility and the attacking success rate of the adversarial examples are well balanced. Experimental results show even only one-fortieth frames are attacked, the attack success rate still reaches 76.8%.