We propose a neural network layer structure that combines frequency and image feature representations for robust Fourier image reconstruction. Our work is motivated by the challenges in magnetic resonance imaging (MRI) where the acquired signal is a corrupted Fourier transform of the desired image. The proposed layer structure enables both correction of artifacts native to the frequency-space and manipulation of image-space representations to reconstruct coherent image structures. This is in contrast to the current deep learning approaches for image reconstruction that manipulate data solely in the frequency-space or solely in the image-space. We demonstrate the advantages of the proposed joint learning on three diverse tasks including image reconstruction from undersampled acquisitions, motion correction, and image denoising in brain MRI. Unlike purely image based and purely frequency based architectures, the proposed joint model produces consistently high quality output images. The resulting joint frequency- and image-space feature representations promise to significantly improve modeling and reconstruction of images acquired in the frequency-space. Our code is available at https://github.com/nalinimsingh/interlacer.