Fast and accurate structural dynamics analysis is important for structural design and damage assessment. Structural dynamics analysis leveraging machine learning techniques has become a popular research focus in recent years. Although the basic neural network provides an alternative approach for structural dynamics analysis, the lack of physics law inside the neural network limits the model accuracy and fidelity. In this paper, a new family of the energy-conservation neural network is introduced, which respects the physical laws. The neural network is explored from a fundamental single-degree-of-freedom system to a complicated multiple-degrees-of-freedom system. The damping force and external forces are also considered step by step. To improve the parallelization of the algorithm, the derivatives of the structural states are parameterized with the novel energy-conservation neural network instead of specifying the discrete sequence of structural states. The proposed model uses the system energy as the last layer of the neural network and leverages the underlying automatic differentiation graph to incorporate the system energy naturally, which ultimately improves the accuracy and long-term stability of structures dynamics response calculation under an earthquake impact. The trade-off between computation accuracy and speed is discussed. As a case study, a 3-story building earthquake simulation is conducted with realistic earthquake records.