Get our free extension to see links to code for papers anywhere online!

 Add to Chrome

 Add to Firefox

CatalyzeX Code Finder - Browser extension linking code for ML papers across the web! | Product Hunt Embed
Towards NNGP-guided Neural Architecture Search

Nov 11, 2020
Daniel S. Park, Jaehoon Lee, Daiyi Peng, Yuan Cao, Jascha Sohl-Dickstein

* 13 + 6 pages, 19 figures; open-source code available at https://github.com/google-research/google-research/tree/master/nngp_nas 

  Access Paper or Ask Questions

Reverse engineering learned optimizers reveals known and novel mechanisms

Nov 04, 2020
Niru Maheswaranathan, David Sussillo, Luke Metz, Ruoxi Sun, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence

Oct 21, 2020
Vinay Rao, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

Sep 23, 2020
Luke Metz, Niru Maheswaranathan, C. Daniel Freeman, Ben Poole, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Finite Versus Infinite Neural Networks: an Empirical Study

Sep 08, 2020
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein

* 17+11 pages; v2 references added, minor improvements 

  Access Paper or Ask Questions

Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible

Aug 25, 2020
Neha S. Wadia, Daniel Duckworth, Samuel S. Schoenholz, Ethan Dyer, Jascha Sohl-Dickstein

* 15+7 pages, 7 figures; added references, edited model descriptions for clarity, results unchanged 

  Access Paper or Ask Questions

A new method for parameter estimation in probabilistic models: Minimum probability flow

Jul 17, 2020
Jascha Sohl-Dickstein, Peter Battaglino, Michael R. DeWeese

* Originally published 2011. Uploaded to arXiv 2020. arXiv admin note: text overlap with arXiv:0906.4779, arXiv:1205.4295 

  Access Paper or Ask Questions

Exact posterior distributions of wide Bayesian neural networks

Jun 18, 2020
Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Infinite attention: NNGP and NTK for deep attention networks

Jun 18, 2020
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, Roman Novak

* ICML 2020 

  Access Paper or Ask Questions

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

Mar 24, 2020
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, Yoshua Bengio


  Access Paper or Ask Questions

Using a thousand optimization tasks to learn hyperparameter search strategies

Mar 11, 2020
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

The large learning rate phase of deep learning: the catapult mechanism

Mar 04, 2020
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari

* 25 pages, 19 figures 

  Access Paper or Ask Questions

On the infinite width limit of neural networks with a standard parameterization

Jan 25, 2020
Jascha Sohl-Dickstein, Roman Novak, Samuel S. Schoenholz, Jaehoon Lee


  Access Paper or Ask Questions

Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Dec 05, 2019
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, Samuel S. Schoenholz


  Access Paper or Ask Questions

Neural reparameterization improves structural optimization

Sep 14, 2019
Stephan Hoyer, Jascha Sohl-Dickstein, Sam Greydanus


  Access Paper or Ask Questions

Using learned optimizers to make models robust to input noise

Jun 08, 2019
Luke Metz, Niru Maheswaranathan, Jonathon Shlens, Jascha Sohl-Dickstein, Ekin D. Cubuk


  Access Paper or Ask Questions

The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study

May 09, 2019
Daniel S. Park, Jascha Sohl-Dickstein, Quoc V. Le, Samuel L. Smith

* 17 pages, 3 tables, 17 figures; accepted to ICML 2019 

  Access Paper or Ask Questions

A RAD approach to deep mixture models

Mar 18, 2019
Laurent Dinh, Jascha Sohl-Dickstein, Razvan Pascanu, Hugo Larochelle

* 9 pages of main content, 4 pages of appendices 

  Access Paper or Ask Questions

A Mean Field Theory of Batch Normalization

Mar 05, 2019
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz

* To appear in ICLR 2019 

  Access Paper or Ask Questions

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

Feb 18, 2019
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington

* 10+8 pages, 13 figures 

  Access Paper or Ask Questions

Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit

Jan 12, 2019
Jascha Sohl-Dickstein, Kenji Kawaguchi


  Access Paper or Ask Questions

Measuring the Effects of Data Parallelism on Neural Network Training

Nov 21, 2018
Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl

* Submitted to JMLR 

  Access Paper or Ask Questions

Learned optimizers that outperform SGD on wall-clock and test loss

Oct 26, 2018
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, C. Daniel Freeman, Jascha Sohl-Dickstein


  Access Paper or Ask Questions

Stochastic natural gradient descent draws posterior samples in function space

Oct 16, 2018
Samuel L. Smith, Daniel Duckworth, Semon Rezchikov, Quoc V. Le, Jascha Sohl-Dickstein

* 11 pages, 6 figures 

  Access Paper or Ask Questions