Building the joint probability distribution (JPD) of multiple spatial-correlated wind farms (WFs) is critical for chance-constrained optimal decision-making. The vertical partitioning historical wind power data of WFs is the premise of training the JPD. However, to protect data privacy, WFs with different stakeholders will refuse to share raw data directly or send raw data to a third party as no one knows whether the third party can be trusted. Moreover, the centralized training way is also faced with costly high bandwidth communication, single-point failure and limited scalability. To solve the problems, distributed algorithm is an alternative. But to the best of our knowledge, rarely has literature proposed privacy-preserving distributed (PPD) algorithm to build the JPD of spatial-correlated WFs. Therefore, based on the additive homomorphic encryption and the average consensus algorithm, we first propose a PPD summation algorithm. Meanwhile, based on the binary hash function and the average consensus algorithm, we then present a PPD inner product algorithm. Thereafter, combining the PPD summation and inner product algorithms, a PPD expectation-maximization algorithm for training the Gaussian-mixture-model-based JPD of WFs is eventually developed. The correctness and the robustness to communicate failure of the proposed algorithm is empirically verified using historical data.