Pooled imaging data from multiple sources is subject to variation between the sources. Correcting for these biases has become incredibly important as the size of imaging studies increases and the multi-site case becomes more common. We propose learning an intermediate representation invariant to site/protocol variables, a technique adapted from information theory-based algorithmic fairness; by leveraging the data processing inequality, such a representation can then be used to create an image reconstruction that is uninformative of its original source, yet still faithful to the underlying structure. To implement this, we use a machine learning method based on variational auto-encoders (VAE) to construct scanner invariant encodings of the imaging data. To evaluate our method, we use training data from the 2018 CDMRI Challenge Harmonization dataset. Our proposed method shows improvements on independent test data relative to a recently published baseline method.