Alert button
Picture for Samuel S. Schoenholz

Samuel S. Schoenholz

Alert button

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

Feb 18, 2019
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington

Figure 1 for Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Figure 2 for Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Figure 3 for Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Figure 4 for Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Viaarxiv icon

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs

Jan 25, 2019
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, Jeffrey Pennington

Figure 1 for Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Figure 2 for Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Figure 3 for Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Figure 4 for Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Viaarxiv icon

Adversarial Spheres

Sep 10, 2018
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, Ian Goodfellow

Figure 1 for Adversarial Spheres
Figure 2 for Adversarial Spheres
Figure 3 for Adversarial Spheres
Figure 4 for Adversarial Spheres
Viaarxiv icon

Peptide-Spectra Matching from Weak Supervision

Aug 22, 2018
Samuel S. Schoenholz, Sean Hackett, Laura Deming, Eugene Melamud, Navdeep Jaitly, Fiona McAllister, Jonathon O'Brien, George Dahl, Bryson Bennett, Andrew M. Dai, Daphne Koller

Figure 1 for Peptide-Spectra Matching from Weak Supervision
Figure 2 for Peptide-Spectra Matching from Weak Supervision
Figure 3 for Peptide-Spectra Matching from Weak Supervision
Figure 4 for Peptide-Spectra Matching from Weak Supervision
Viaarxiv icon

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

Aug 15, 2018
Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz

Figure 1 for Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Figure 2 for Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Figure 3 for Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Figure 4 for Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Viaarxiv icon

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

Jul 10, 2018
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington

Figure 1 for Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Figure 2 for Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Figure 3 for Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Figure 4 for Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Viaarxiv icon

Deep Neural Networks as Gaussian Processes

Mar 03, 2018
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

Figure 1 for Deep Neural Networks as Gaussian Processes
Figure 2 for Deep Neural Networks as Gaussian Processes
Figure 3 for Deep Neural Networks as Gaussian Processes
Figure 4 for Deep Neural Networks as Gaussian Processes
Viaarxiv icon

The Emergence of Spectral Universality in Deep Networks

Feb 27, 2018
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Figure 1 for The Emergence of Spectral Universality in Deep Networks
Figure 2 for The Emergence of Spectral Universality in Deep Networks
Figure 3 for The Emergence of Spectral Universality in Deep Networks
Figure 4 for The Emergence of Spectral Universality in Deep Networks
Viaarxiv icon

Mean Field Residual Networks: On the Edge of Chaos

Dec 24, 2017
Greg Yang, Samuel S. Schoenholz

Figure 1 for Mean Field Residual Networks: On the Edge of Chaos
Figure 2 for Mean Field Residual Networks: On the Edge of Chaos
Figure 3 for Mean Field Residual Networks: On the Edge of Chaos
Figure 4 for Mean Field Residual Networks: On the Edge of Chaos
Viaarxiv icon

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

Nov 13, 2017
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Figure 1 for Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Figure 2 for Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Figure 3 for Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Figure 4 for Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Viaarxiv icon