We detail an approach to develop Stein's method for bounding integral metrics on probability measures defined on a Riemannian manifold $\mathbf{M}$. Our approach exploits the relationship between the generator of a diffusion on $\mathbf{M}$ with target invariant measure and its characterising Stein operator. We consider a pair of such diffusions with different starting points, and investigate properties of solution to the Stein equation based on analysis of the distance process between the pair. Several examples elucidating the role of geometry of $\mathbf{M}$ in these developments are presented.