Alert button

ISAAC Newton: Input-based Approximate Curvature for Newton's Method

May 01, 2023
Felix Petersen, Tobias Sutter, Christian Borgelt, Dongsung Huh, Hilde Kuehne, Yuekai Sun, Oliver Deussen

Figure 1 for ISAAC Newton: Input-based Approximate Curvature for Newton's Method
Figure 2 for ISAAC Newton: Input-based Approximate Curvature for Newton's Method
Figure 3 for ISAAC Newton: Input-based Approximate Curvature for Newton's Method
Figure 4 for ISAAC Newton: Input-based Approximate Curvature for Newton's Method

Share this with someone who'll enjoy it:

We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.

* Published at ICLR 2023, Code @ https://github.com/Felix-Petersen/isaac, Video @ https://youtu.be/7RKRX-MdwqM  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it: