This paper develops a Pontryagin differentiable programming (PDP) methodology to establish a unified end-to-end learning framework, which solves a large class of learning and control tasks. The proposed PDP framework distinguishes itself from existing ones by two key techniques: first, by differentiating the Pontryagin's Maximum Principle, the PDP framework allows for an end-to-end learning of any parameterized system, even though differentiation with respect to an unknown objective function is not readily attainable; and second, based on control theory, the PDP framework incorporates both the forward and backward propagations by constructing two separate control systems, which are then efficiently solved using techniques in control domain. Three learning modes of the proposed PDP framework are investigated to address three types of learning problems: inverse optimization, system identification, and control/planing, respectively. Effectiveness of this framework in each learning mode has been validated in the context of pendulum systems.