White matter fiber clustering (WMFC) parcellates tractography data into anatomically meaningful fiber bundles, usually in an unsupervised manner without the need of labeled ground truth data. While widely used WMFC approaches have shown good performance using classical machine learning techniques, recent advances in deep learning reveal a promising direction towards fast and effective WMFC. In this work, we propose a novel deep learning framework for WMFC, Deep Fiber Clustering (DFC), which solves the unsupervised clustering problem as a self-supervised learning task with a domain-specific pretext task to predict pairwise fiber distances. This accelerates the fiber representation learning to handle a known challenge in WMFC, i.e., the sensitivity of clustering results to the point ordering along fibers. We design a novel network architecture that represents input fibers as point clouds and allows the incorporation of additional sources of input information from gray matter parcellation. Thus DFC makes use of the combined white matter fiber geometry and gray matter anatomical parcellation to improve anatomical coherence of fiber clusters. In addition, DFC conducts outlier removal in a natural way by rejecting fibers with low cluster assignment probabilities. We evaluate DFC on three independently acquired cohorts (including data from 220 subjects) and compare it to several state-of-the-art WMFC algorithms. Experimental results demonstrate superior performance of DFC in terms of cluster compactness, generalization ability, anatomical coherence, and computational efficiency. In addition, DFC parcellates whole brain tractography with 50k fibers in about 1.5 minutes, providing a fast and efficient tool for large data analysis.