Lorentz transmission electron microscopy is a unique characterization technique that enables the simultaneous imaging of both the microstructure and functional properties of materials at high spatial resolution. The quantitative information such as magnetization and electric potentials is carried by the phase of the electron wave, and is lost during imaging. In order to understand the local interactions and develop structure-property relationships, it is necessary to retrieve the complete wavefunction of the electron wave, which requires solving for the phase shift of the electrons (phase retrieval). Here we have developed a method based on differentiable programming to solve the inverse problem of phase retrieval, using a series of defocused microscope images. We show that our method is robust and can outperform widely used \textit{transport of intensity equation} in terms of spatial resolution and accuracy of the retrieved phase under same electron dose conditions. Furthermore, our method shares the same basic structure as advanced machine learning algorithms, and is easily adaptable to various other forms of phase retrieval in electron microscopy.