Alert button

Economical ensembles with hypernetworks

Jul 25, 2020
Figure 1 for Economical ensembles with hypernetworks
Figure 2 for Economical ensembles with hypernetworks
Figure 3 for Economical ensembles with hypernetworks
Figure 4 for Economical ensembles with hypernetworks

Share this with someone who'll enjoy it:

Averaging the predictions of many independently trained neural networks is a simple and effective way of improving generalization in deep learning. However, this strategy rapidly becomes costly, as the number of trainable parameters grows linearly with the size of the ensemble. Here, we propose a new method to learn economical ensembles, where the number of trainable parameters and iterations over the data is comparable to that of a single model. Our neural networks are parameterized by hypernetworks, which learn to embed weights in low-dimensional spaces. In a late training phase, we generate an ensemble by randomly initializing an additional number of weight embeddings in the vicinity of each other. We then exploit the inherent randomness in stochastic gradient descent to induce ensemble diversity. Experiments with wide residual networks on the CIFAR and Fashion-MNIST datasets show that our algorithm yields models that are more accurate and less overconfident on unseen data, while learning as efficiently as a single network.

* 25 pages, 5 figures  

Share this with someone who'll enjoy it: