Deep learning has significantly revolutionized the design of numerical algorithms for solving high-dimensional partial differential equations (PDEs). Yet the empirical successes of such approaches remains mysterious in theory. In deep learning-based PDE solvers, solving the original PDE is formulated into an expectation minimization problem with a PDE solution space discretized via deep neural networks. A global minimizer corresponds to a deep neural network that solves the given PDE. Typically, gradient descent-based methods are applied to minimize the expectation. This paper shows that gradient descent can identify a global minimizer of the optimization problem with a well-controlled generalization error in the case of two-layer neural networks in the over-parameterization regime (i.e., the network width is sufficiently large). The generalization error of the gradient descent solution does not suffer from the curse of dimensonality if the solution is in a Barron-type space. The theories developed here could form a theoretical foundation of deep learning-based PDE solvers.