With the development of smart grid, the operation and control of power system is realized through power communication network, especially the power production and enterprise management business involve a large amount of sensitive information, and the requirements for data security and real-time transmission are gradually improved. In this paper, a new 9D complex chaotic system with quaternion is proposed for the encryption of smart grid data. Firstly, a new 9D complex chaotic system with quaternion is proposed, and its attractors, bifurcation diagram, complexity, and 0-1 test are analyzed. Secondly, the pseudo-random sequences are generated by the new chaotic system to encrypt power data. Finally, the proposed encryption algorithm is verifed with power data and images in the smart grid, which can ensure the encryption security and real-time. The verifcation results show that the proposed encryption scheme is technically feasible and available for power data and image encryption in smart grid.