Revealing the Phase Diagram of Kitaev Materials by Machine Learning: Cooperation and Competition between Spin Liquids

Kitaev materials are promising materials for hosting quantum spin liquids and investigating the interplay of topological and symmetry-breaking phases. We use an unsupervised and interpretable machine-learning method, the tensorial-kernel support vector machine, to study the classical honeycomb Kitaev-$\Gamma$ model in a magnetic field. Our machine learns the global phase diagram and the associated analytical order parameters, including several distinct spin liquids, two exotic $S_3$ magnets, and two modulated $S_3 \times Z_3$ magnets. We find that the extension of Kitaev spin liquids and a field-induced suppression of magnetic order already occur in the large-$S$ limit, implying that critical parts of the physics of Kitaev materials can be understood at the classical level. Moreover, the two $S_3 \times Z_3$ orders are induced by competition between Kitaev and $\Gamma$ spin liquids and feature a previously unknown type of spin-lattice entangled modulation, which requires a matrix description instead of scalar phase factors. Our work provides the first instance of a machine detecting new phases and paves the way towards the development of automated tools to explore unsolved problems in many-body physics.