The transformer has dominated the natural language processing (NLP) field for a long time. Recently, the transformer-based method is adopt into the computer vision (CV) field and shows promising results. As an important branch of the CV field, medical image analysis joins the wave of the transformer-based method rightfully. In this paper, we illustrate the principle of the attention mechanism, and the detailed structures of the transformer, and depict how the transformer is adopted into the CV field. We organize the transformer-based medical image analysis applications in the sequence of different CV tasks, including classification, segmentation, synthesis, registration, localization, detection, captioning, and denoising. For the mainstream classification and segmentation tasks, we further divided the corresponding works based on different medical imaging modalities. We include thirteen modalities and more than twenty objects in our work. We also visualize the proportion that each modality and object occupy to give the readers an intuitive impression. We hope our work can contribute to the development of transformer-based medical image analysis in the future.