Alert button

Near-Optimal High-Probability Convergence for Non-Convex Stochastic Optimization with Variance Reduction

Feb 13, 2023
Zijian Liu, Perry Dong, Srikanth Jagabathula, Zhengyuan Zhou

Figure 1 for Near-Optimal High-Probability Convergence for Non-Convex Stochastic Optimization with Variance Reduction
Figure 2 for Near-Optimal High-Probability Convergence for Non-Convex Stochastic Optimization with Variance Reduction
Figure 3 for Near-Optimal High-Probability Convergence for Non-Convex Stochastic Optimization with Variance Reduction

Share this with someone who'll enjoy it:

Traditional analyses for non-convex stochastic optimization problems characterize convergence bounds in expectation, which is inadequate as it does not supply a useful performance guarantee on a single run. Motivated by its importance, an emerging line of literature has recently studied the high-probability convergence behavior of several algorithms, including the classic stochastic gradient descent (SGD). However, no high-probability results are established for optimization algorithms with variance reduction, which is known to accelerate the convergence process and has been the de facto algorithmic technique for stochastic optimization at large. To close this important gap, we introduce a new variance-reduced algorithm for non-convex stochastic optimization, which we call Generalized SignSTORM. We show that with probability at least $1-\delta$, our algorithm converges at the rate of $O(\log(dT/\delta)/T^{1/3})$ after $T$ iterations where $d$ is the problem dimension. This convergence guarantee matches the existing lower bound up to a log factor, and to our best knowledge, is the first high-probability minimax (near-)optimal result. Finally, we demonstrate the effectiveness of our algorithm through numerical experiments.

View paper onarxiv icon

Share this with someone who'll enjoy it: