Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Characteristics of Monte Carlo Dropout in Wide Neural Networks

Jul 10, 2020
Joachim Sicking, Maram Akila, Tim Wirtz, Sebastian Houben, Asja Fischer

Share this with someone who'll enjoy it:

Monte Carlo (MC) dropout is one of the state-of-the-art approaches for uncertainty estimation in neural networks (NNs). It has been interpreted as approximately performing Bayesian inference. Based on previous work on the approximation of Gaussian processes by wide and deep neural networks with random weights, we study the limiting distribution of wide untrained NNs under dropout more rigorously and prove that they as well converge to Gaussian processes for fixed sets of weights and biases. We sketch an argument that this property might also hold for infinitely wide feed-forward networks that are trained with (full-batch) gradient descent. The theory is contrasted by an empirical analysis in which we find correlations and non-Gaussian behaviour for the pre-activations of finite width NNs. We therefore investigate how (strongly) correlated pre-activations can induce non-Gaussian behavior in NNs with strongly correlated weights.

* Accepted at the ICML 2020 workshop for Uncertainty and Robustness in Deep Learning 

   Access Paper Source

Share this with someone who'll enjoy it: