We propose to learn a hierarchical prior in the context of variational autoencoders to avoid over-regularisation resulting from a standard normal prior distribution. To incentivise an informative latent representation of the data by learning a rich hierarchical prior, we formulate the objective function as the Lagrangian of a constrained-optimisation problem and propose an optimisation algorithm inspired by Taming VAEs. To validate our approach, we train our model on the static and binary MNIST, Fashion-MNIST, OMNIGLOT, CMU Graphics Lab Motion Capture, 3D Faces, and 3D Chairs datasets, obtaining results that are comparable to state-of-the-art. Furthermore, we introduce a graph-based interpolation method to show that the topology of the learned latent representation correspond to the topology of the data manifold.