Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Picture for Jeffrey Pennington

Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition


Nov 04, 2020
Ben Adlam, Jeffrey Pennington

* Published as a conference paper in the Proceedings of the Thirty-fourth Conference on Neural Information Processing Systems; 54 pages; 5 figures. arXiv admin note: text overlap with arXiv:2008.06786 

  Access Paper or Ask Questions

Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit


Oct 14, 2020
Ben Adlam, Jaehoon Lee, Lechao Xiao, Jeffrey Pennington, Jasper Snoek

* 23 pages, 11 figures 

  Access Paper or Ask Questions

Temperature check: theory and practice for training models with softmax-cross-entropy losses


Oct 14, 2020
Atish Agarwala, Jeffrey Pennington, Yann Dauphin, Sam Schoenholz


  Access Paper or Ask Questions

Finite Versus Infinite Neural Networks: an Empirical Study


Sep 08, 2020
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein

* 17+11 pages; v2 references added, minor improvements 

  Access Paper or Ask Questions

The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization


Aug 15, 2020
Ben Adlam, Jeffrey Pennington

* Published as a conference paper in the Proceedings of the 37th International Conference on Machine Learning; 31 pages; 4 figures 

  Access Paper or Ask Questions

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks


Jun 25, 2020
Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington


  Access Paper or Ask Questions

Exact posterior distributions of wide Bayesian neural networks


Jun 18, 2020
Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks


Jan 16, 2020
Wei Hu, Lechao Xiao, Jeffrey Pennington

* International Conference on Learning Representations (ICLR) 2020 

  Access Paper or Ask Questions

Disentangling trainability and generalization in deep learning


Dec 30, 2019
Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz

* 22 pages, 3 figures 

  Access Paper or Ask Questions

A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning


Dec 02, 2019
Ben Adlam, Jake Levinson, Jeffrey Pennington


  Access Paper or Ask Questions

A Mean Field Theory of Batch Normalization


Mar 05, 2019
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz

* To appear in ICLR 2019 

  Access Paper or Ask Questions

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent


Feb 18, 2019
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington

* 10+8 pages, 13 figures 

  Access Paper or Ask Questions

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs


Jan 25, 2019
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, Jeffrey Pennington


  Access Paper or Ask Questions

Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes


Oct 11, 2018
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

* 26 pages, 7 figures 

  Access Paper or Ask Questions

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks


Aug 15, 2018
Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz

* ICML 2018 Conference Proceedings 

  Access Paper or Ask Questions

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks


Jul 10, 2018
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington

* ICML 2018 Conference Proceedings 

  Access Paper or Ask Questions

Sensitivity and Generalization in Neural Networks: an Empirical Study


Jun 18, 2018
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

* Published as a conference paper at ICLR 2018 

  Access Paper or Ask Questions

Deep Neural Networks as Gaussian Processes


Mar 03, 2018
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

* Published version in ICLR 2018. 10 pages + appendix 

  Access Paper or Ask Questions

The Emergence of Spectral Universality in Deep Networks


Feb 27, 2018
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

* 17 pages, 4 figures. Appearing at the 21st International Conference on Artificial Intelligence and Statistics (AISTATS) 2018 

  Access Paper or Ask Questions

Estimating the Spectral Density of Large Implicit Matrices


Feb 09, 2018
Ryan P. Adams, Jeffrey Pennington, Matthew J. Johnson, Jamie Smith, Yaniv Ovadia, Brian Patton, James Saunderson


  Access Paper or Ask Questions

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice


Nov 13, 2017
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

* 13 pages, 6 figures. Appearing at the 31st Conference on Neural Information Processing Systems (NIPS 2017) 

  Access Paper or Ask Questions

A Correspondence Between Random Neural Networks and Statistical Field Theory


Oct 18, 2017
Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein


  Access Paper or Ask Questions