Objective: To conduct a systematic review of deep learning methods on Electrocardiogram (ECG) data from the perspective of model architecture and their application task. Methods: First, we extensively searched papers deploying deep learning (deep neural network networks) on ECG data that published between January 1st 2010 and September 30th 2019 from Google Scholar, PubMed and DBLP. Then we analyze them in three aspects including task, model and data. Finally, we conclude unresolved challenges and problems that existing models can not handle well. Results: The total number of papers is 124, among them 97 papers are published after in recent two years. Almost all kinds of common deep learning architectures have been used in ECG analytics tasks like disease detection/classification, annotation/localization, sleep staging, biometric human identification, denoising and so on. Conclusion: The number of works about deep learning on ECG data is growing explosively in recent years. Indeed, these works have achieve a far more better performance in terms of accuracy. However, there are some new challenges and problems like interpretability, scalability, efficiency, which need to be addressed and paid more attention. Moreover, it is also worth to investigate by discovering new interesting applications from both the dataset view and the method view. Significance: This paper summarizes existing deep learning methods on modeling ECG data from multiple views, while also point out existing challenges and problems, while can become potential research direction in the future.