Alert button

K-SAM: Sharpness-Aware Minimization at the Speed of SGD

Oct 23, 2022
Renkun Ni, Ping-yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, Tom Goldstein

Figure 1 for K-SAM: Sharpness-Aware Minimization at the Speed of SGD
Figure 2 for K-SAM: Sharpness-Aware Minimization at the Speed of SGD
Figure 3 for K-SAM: Sharpness-Aware Minimization at the Speed of SGD
Figure 4 for K-SAM: Sharpness-Aware Minimization at the Speed of SGD

Share this with someone who'll enjoy it:

Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks. However, SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. The computational challenge posed by SAM arises because each iteration requires both ascent and descent steps and thus double the gradient computations. To address this challenge, we propose to compute gradients in both stages of SAM on only the top-k samples with highest loss. K-SAM is simple and extremely easy-to-implement while providing significant generalization boosts over vanilla SGD at little to no additional cost.

* 13 pages, 2 figures  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it: