Graph learning, which aims to infer the underlying topology behind high dimension data, has attracted intense attention. In this study, we shed a new light on graph learning by considering a pragmatic scenario where data are privacy sensitive and located in separated clients (devices or organizations). The main difficulty in learning graphs in this scenario is that we cannot process all the data in a central server, because the data are not allowed to leave the local clients due to privacy concerns. The problem becomes more challenging when data of different clients are non-IID, since it is unreasonable to learn a global graph for heterogeneous data. To address these issues, we propose a novel framework in which a personalized graph for each client and a consensus graph are jointly learned in a federated fashion. Specifically, we commute model updates instead of raw data to the central server in the proposed federated algorithm. A provable convergence analysis shows that the algorithm enjoys $\mathcal{O}(1/T)$ convergence rate. To further enhance privacy, we design a deferentially privacy algorithm to prevent the information of the raw data from being leaked when transferring model updates. A theoretical guidance is provided on how to ensure that the algorithm satisfies differential privacy. We also analyze the impact of differential privacy on the convergence of our algorithm. Finally, extensive experiments on both synthetic and real world data are carried out to validate the proposed models and algorithms. Experimental results illustrate that our framework is able to learn graphs effectively in the target scenario.