Alert button

Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data

Sep 30, 2022
Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, Makoto Yamada

Figure 1 for Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data
Figure 2 for Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data
Figure 3 for Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data
Figure 4 for Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data

Share this with someone who'll enjoy it:

SGD with momentum acceleration is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum acceleration is Distributed SGD (DSGD) with momentum acceleration (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several studies have addressed this issue and proposed methods with momentum acceleration that are more robust to data heterogeneity than DSGDm, although their convergence rates remain dependent on data heterogeneity and decrease when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which is a method with momentum acceleration whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the convergence rate of Momentum Tracking in the standard deep learning setting, where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient $\beta\in [0, 1)$. Through image classification tasks, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum acceleration and can consistently outperform these existing methods when the data distributions are heterogeneous.

View paper onarxiv icon

Share this with someone who'll enjoy it: