Get our free extension to see links to code for papers anywhere online!

 Add to Chrome

 Add to Firefox

CatalyzeX Code Finder - Browser extension linking code for ML papers across the web! | Product Hunt Embed
Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates

Oct 01, 2020
Chen Zeno, Itay Golan, Elad Hoffer, Daniel Soudry

* The arXiv paper "Task Agnostic Continual Learning Using Online Variational Bayes" is a preliminary pre-print of this paper. The main differences between the versions are: 1. We develop new algorithmic framework (FOO-VB). 2. We add multivariate Gaussian and matrix variate Gaussian versions of the algorithm. 3. We demonstrate the new algorithm performance in task agnostic scenarios 

  Access Paper or Ask Questions

Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy

Jul 13, 2020
Edward Moroshko, Suriya Gunasekar, Blake Woodworth, Jason D. Lee, Nathan Srebro, Daniel Soudry


  Access Paper or Ask Questions

Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?

Jul 02, 2020
Yaniv Blumenfeld, Dar Gilboa, Daniel Soudry

* ICML 2020 

  Access Paper or Ask Questions

Neural gradients are lognormally distributed: understanding sparse and quantized training

Jun 17, 2020
Brian Chmiel, Liad Ben-Uri, Moran Shkolnik, Elad Hoffer, Ron Banner, Daniel Soudry

* Fix references typos 

  Access Paper or Ask Questions

Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming

Jun 14, 2020
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, Daniel Soudry


  Access Paper or Ask Questions

Kernel and Rich Regimes in Overparametrized Models

Feb 24, 2020
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, Nathan Srebro

* This updates and significantly extends a previous article (arXiv:1906.05827), Sections 6 and 7.1 are the most major additions. 30 pages. arXiv admin note: text overlap with arXiv:1906.05827 

  Access Paper or Ask Questions

MTJ-Based Hardware Synapse Design for Quantized Deep Neural Networks

Dec 29, 2019
Tzofnat Greenberg Toledo, Ben Perach, Daniel Soudry, Shahar Kvatinsky


  Access Paper or Ask Questions

Is Feature Diversity Necessary in Neural Network Initialization?

Dec 12, 2019
Yaniv Blumenfeld, Dar Gilboa, Daniel Soudry

* 4 + 1 pages. Workshop paper 

  Access Paper or Ask Questions

The Knowledge Within: Methods for Data-Free Model Compression

Dec 03, 2019
Matan Haroush, Itay Hubara, Elad Hoffer, Daniel Soudry


  Access Paper or Ask Questions

A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case

Oct 03, 2019
Greg Ongie, Rebecca Willett, Daniel Soudry, Nathan Srebro


  Access Paper or Ask Questions

At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?

Sep 26, 2019
Niv Giladi, Mor Shpigel Nacson, Elad Hoffer, Daniel Soudry


  Access Paper or Ask Questions

Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency

Aug 12, 2019
Elad Hoffer, Berry Weinstein, Itay Hubara, Tal Ben-Nun, Torsten Hoefler, Daniel Soudry


  Access Paper or Ask Questions

Kernel and Deep Regimes in Overparametrized Models

Jun 13, 2019
Blake Woodworth, Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro

* 16 pages 

  Access Paper or Ask Questions

A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

Jun 03, 2019
Yaniv Blumenfeld, Dar Gilboa, Daniel Soudry


  Access Paper or Ask Questions

Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

May 17, 2019
Mor Shpigel Nacson, Suriya Gunasekar, Jason D. Lee, Nathan Srebro, Daniel Soudry

* ICML Camera ready version 

  Access Paper or Ask Questions

How do infinite width bounded norm networks look in function space?

Feb 13, 2019
Pedro Savarese, Itay Evron, Daniel Soudry, Nathan Srebro


  Access Paper or Ask Questions

Augment your batch: better training with larger batches

Jan 27, 2019
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, Daniel Soudry


  Access Paper or Ask Questions

The Global Optimization Geometry of Shallow Linear Neural Networks

Nov 05, 2018
Zhihui Zhu, Daniel Soudry, Yonina C. Eldar, Michael B. Wakin


  Access Paper or Ask Questions

Characterizing Implicit Bias in Terms of Optimization Geometry

Oct 22, 2018
Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro

* (1) A bug in the proof of implicit bias for matrix factorization was fixed. v2 gives a characterization of the asymptotic bias of the factor matrices, while v1 made a stronger claim on the limit direction of the unfactored matrix. (2) v2 also includes new results on implicit bias of mirror descent with realizable affine constraints 

  Access Paper or Ask Questions

ACIQ: Analytical Clipping for Integer Quantization of neural networks

Oct 02, 2018
Ron Banner, Yury Nahshan, Elad Hoffer, Daniel Soudry


  Access Paper or Ask Questions

On the Blindspots of Convolutional Networks

Jul 08, 2018
Elad Hoffer, Shai Fine, Daniel Soudry


  Access Paper or Ask Questions

Scalable Methods for 8-bit Training of Neural Networks

Jun 17, 2018
Ron Banner, Itay Hubara, Elad Hoffer, Daniel Soudry


  Access Paper or Ask Questions

Convergence of Gradient Descent on Separable Data

Jun 12, 2018
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro H. P. Savarese, Nathan Srebro, Daniel Soudry

* Added empirical results of experiments on deep networks (Appendix E). In addition, minor typos and phrasing mistakes were fixed 

  Access Paper or Ask Questions

Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate

Jun 05, 2018
Mor Shpigel Nacson, Nathan Srebro, Daniel Soudry

* 7 pages in main paper, 10 pages of proofs in appendix, 2 figures, 1 table 

  Access Paper or Ask Questions

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Jun 01, 2018
Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro


  Access Paper or Ask Questions

Bayesian Gradient Descent: Online Variational Bayes Learning with Increased Robustness to Catastrophic Forgetting and Weight Pruning

Mar 27, 2018
Chen Zeno, Itay Golan, Elad Hoffer, Daniel Soudry


  Access Paper or Ask Questions

The Implicit Bias of Gradient Descent on Separable Data

Mar 21, 2018
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, Nathan Srebro

* Journal version (previous version appeared as conference paper in ICLR ). Main improvements: We proved measure zero case for main theorem (with implication for the rates), and the multi-class case. Both were not covered in previous version 

  Access Paper or Ask Questions

Fix your classifier: the marginal value of training the last weight layer

Mar 20, 2018
Elad Hoffer, Itay Hubara, Daniel Soudry

* International Conference on Learning Representations 2018 
* https://openreview.net/forum?id=S1Dh8Tg0

  Access Paper or Ask Questions

Norm matters: efficient and accurate normalization schemes in deep networks

Mar 08, 2018
Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry


  Access Paper or Ask Questions