Alert button

Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace

Aug 11, 2022
Figure 1 for Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace
Figure 2 for Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace
Figure 3 for Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace
Figure 4 for Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace

Share this with someone who'll enjoy it:

In this paper we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. Hutchinson method is a classical unbiased estimator for the trace of a matrix, but it is very time-consuming on deep learning models. Hence a dropout scheme is proposed to efficiently implements the Hutchinson method. Then we discuss a connection to linear stability of a nonlinear dynamical system and flat/sharp minima. Experiments demonstrate that our method outperforms existing regularizers and data augmentation methods, such as Jacobian, confidence penalty, and label smoothing, cutout and mixup.

Share this with someone who'll enjoy it: