As the resolution of digital images increase significantly, the processing of images becomes more challenging in terms of accuracy and efficiency. In this paper, we consider image segmentation by solving a partial differentiation equation (PDE) model based on the Mumford-Shah functional. We develop a new algorithm by combining anisotropic mesh adaptation for image representation and finite element method for solving the PDE model. Comparing to traditional algorithms solved by finite difference method, our algorithm provides faster and better results without the need to resizing the images to lower quality. We also extend the algorithm to segment images with multiple regions.