Renormalized Mutual Information for Artificial Scientific Discovery

We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction, where the goal is to produce a low-dimensional effective description of a high-dimensional system. Our approach enables the discovery of collective variables in physical systems, thus adding to the toolbox of artificial scientific discovery, while also aiding the analysis of information flow in artificial neural networks.

Renormalized Mutual Information for Extraction of Continuous Features

We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction and for information processing in artificial neural networks. We illustrate in basic examples how the renormalized mutual information can be used not only to compare the usefulness of different ansatz features, but also to automatically extract optimal features of a system in an unsupervised dimensionality reduction scenario.