Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Picture for Samuel S. Schoenholz

Tilting the playing field: Dynamical loss functions for machine learning


Feb 13, 2021
Miguel Ruiz-Garcia, Ge Zhang, Samuel S. Schoenholz, Andrea J. Liu


  Access Paper or Ask Questions

Finite Versus Infinite Neural Networks: an Empirical Study


Sep 08, 2020
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein

* 17+11 pages; v2 references added, minor improvements 

  Access Paper or Ask Questions

Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible


Aug 25, 2020
Neha S. Wadia, Daniel Duckworth, Samuel S. Schoenholz, Ethan Dyer, Jascha Sohl-Dickstein

* 15+7 pages, 7 figures; added references, edited model descriptions for clarity, results unchanged 

  Access Paper or Ask Questions

On the infinite width limit of neural networks with a standard parameterization


Jan 25, 2020
Jascha Sohl-Dickstein, Roman Novak, Samuel S. Schoenholz, Jaehoon Lee


  Access Paper or Ask Questions

Disentangling trainability and generalization in deep learning


Dec 30, 2019
Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz

* 22 pages, 3 figures 

  Access Paper or Ask Questions

JAX, M.D.: End-to-End Differentiable, Hardware Accelerated, Molecular Dynamics in Pure Python


Dec 09, 2019
Samuel S. Schoenholz, Ekin D. Cubuk


  Access Paper or Ask Questions

Neural Tangents: Fast and Easy Infinite Neural Networks in Python


Dec 05, 2019
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, Samuel S. Schoenholz


  Access Paper or Ask Questions

A Mean Field Theory of Batch Normalization


Mar 05, 2019
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz

* To appear in ICLR 2019 

  Access Paper or Ask Questions

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent


Feb 18, 2019
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington

* 10+8 pages, 13 figures 

  Access Paper or Ask Questions

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs


Jan 25, 2019
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, Jeffrey Pennington


  Access Paper or Ask Questions

Adversarial Spheres


Sep 10, 2018
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, Ian Goodfellow


  Access Paper or Ask Questions

Peptide-Spectra Matching from Weak Supervision


Aug 22, 2018
Samuel S. Schoenholz, Sean Hackett, Laura Deming, Eugene Melamud, Navdeep Jaitly, Fiona McAllister, Jonathon O'Brien, George Dahl, Bryson Bennett, Andrew M. Dai, Daphne Koller


  Access Paper or Ask Questions

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks


Aug 15, 2018
Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz

* ICML 2018 Conference Proceedings 

  Access Paper or Ask Questions

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks


Jul 10, 2018
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington

* ICML 2018 Conference Proceedings 

  Access Paper or Ask Questions

Deep Neural Networks as Gaussian Processes


Mar 03, 2018
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

* Published version in ICLR 2018. 10 pages + appendix 

  Access Paper or Ask Questions

The Emergence of Spectral Universality in Deep Networks


Feb 27, 2018
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

* 17 pages, 4 figures. Appearing at the 21st International Conference on Artificial Intelligence and Statistics (AISTATS) 2018 

  Access Paper or Ask Questions

Mean Field Residual Networks: On the Edge of Chaos


Dec 24, 2017
Greg Yang, Samuel S. Schoenholz

* NIPS 2017 

  Access Paper or Ask Questions

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice


Nov 13, 2017
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

* 13 pages, 6 figures. Appearing at the 31st Conference on Neural Information Processing Systems (NIPS 2017) 

  Access Paper or Ask Questions

Intriguing Properties of Adversarial Examples


Nov 08, 2017
Ekin D. Cubuk, Barret Zoph, Samuel S. Schoenholz, Quoc V. Le

* 17 pages 

  Access Paper or Ask Questions

A Correspondence Between Random Neural Networks and Statistical Field Theory


Oct 18, 2017
Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Combining Machine Learning and Physics to Understand Glassy Systems


Sep 23, 2017
Samuel S. Schoenholz


  Access Paper or Ask Questions

Neural Message Passing for Quantum Chemistry


Jun 12, 2017
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, George E. Dahl

* 14 pages 

  Access Paper or Ask Questions

Deep Information Propagation


Apr 04, 2017
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, Jascha Sohl-Dickstein


  Access Paper or Ask Questions