Passive radar has key advantages over its active counterpart in terms of cost and stealth. In this paper, we address passive radar imaging problem by interferometric inversion using a spectral estimation method with a priori information within a deep learning (DL) framework. Cross-correlating the received signals from different look directions mitigates the influence of shared transmitter related phase components despite lack of a cooperative transmitter, and permits tractable inference via interferometric inversion. We thereon leverage deep architectures for modeling a priori information and for improving sample efficiency of state-of-the-art methods. Our approach comprises of an iterative algorithm based on generalizing the power method, and applies denoisers using plug-and-play (PnP) and regularization by denoising (RED) techniques. We evaluate our approach using simulated data for passive synthetic aperture radar (SAR) by using convolutional neural networks (CNN) as denoisers. The numerical experiments show that our method can achieve faster reconstruction and superior image quality in sample starved regimes than the state-of-the-art passive interferometric imaging algorithms.