Alert button

Stochastic Polyak Stepsize with a Moving Target

Jun 22, 2021
Robert M. Gower, Aaron Defazio, Michael Rabbat

Figure 1 for Stochastic Polyak Stepsize with a Moving Target
Figure 2 for Stochastic Polyak Stepsize with a Moving Target
Figure 3 for Stochastic Polyak Stepsize with a Moving Target
Figure 4 for Stochastic Polyak Stepsize with a Moving Target

Share this with someone who'll enjoy it:

We propose a new stochastic gradient method that uses recorded past loss values to reduce the variance. Our method can be interpreted as a new stochastic variant of the Polyak Stepsize that converges globally without assuming interpolation. Our method introduces auxiliary variables, one for each data point, that track the loss value for each data point. We provide a global convergence theory for our method by showing that it can be interpreted as a special variant of online SGD. The new method only stores a single scalar per data point, opening up new applications for variance reduction where memory is the bottleneck.

* 41 pages, 13 figures, 1 table  
View paper onarxiv icon

Share this with someone who'll enjoy it: