A point cloud serves as a representation of the surface of a three-dimensional shape. Deep generative models have been adapted to model their variations typically by a map from a ball-like set of latent variables. However, previous approaches have not paid much attention to the topological structure of a point cloud; a continuous map cannot express the varying number of holes and intersections. Moreover, a point cloud is often composed of multiple subparts, and it is also hardly expressed. In this paper, we propose ChartPointFlow, which is a flow-based generative model with multiple latent labels. By maximizing the mutual information, a map conditioned by a label is assigned to a continuous subset of a given point cloud, like a chart of a manifold. This enables our proposed model to preserve the topological structure with clear boundaries, while previous approaches tend to suffer from blurs and to fail in generating holes. Experimental results demonstrate that ChartPointFlow achieves the state-of-the-art performance in generation and reconstruction among sampling-based point cloud generators.