Alert button

Does compressing activations help model parallel training?

Jan 06, 2023
Song Bian, Dacheng Li, Hongyi Wang, Eric P. Xing, Shivaram Venkataraman

Figure 1 for Does compressing activations help model parallel training?
Figure 2 for Does compressing activations help model parallel training?
Figure 3 for Does compressing activations help model parallel training?
Figure 4 for Does compressing activations help model parallel training?

Share this with someone who'll enjoy it:

Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensive model parallelism. One way to improve training speed is to compress the message size in communication. Previous approaches have primarily focused on compressing gradients in a data parallelism setting, but compression in a model-parallel setting is an understudied area. We have discovered that model parallelism has fundamentally different characteristics than data parallelism. In this work, we present the first empirical study on the effectiveness of compression methods for model parallelism. We implement and evaluate three common classes of compression algorithms - pruning-based, learning-based, and quantization-based - using a popular Transformer training framework. We evaluate these methods across more than 160 settings and 8 popular datasets, taking into account different hyperparameters, hardware, and both fine-tuning and pre-training stages. We also provide analysis when the model is scaled up. Finally, we provide insights for future development of model parallelism compression algorithms.

* 16 pages, 5 figures  
View paper onarxiv icon

Share this with someone who'll enjoy it: