Alert button

Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks

Aug 20, 2022
Figure 1 for Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
Figure 2 for Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
Figure 3 for Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks

Share this with someone who'll enjoy it:

Quantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. Independently, these methods reduce model size and can accelerate inference, but their relative benefit and combinatorial interactions have not been rigorously studied. For each of the eight possible subsets of these techniques, we compare accuracy vs. model size tradeoffs across six BERT architecture sizes and eight GLUE tasks. We find that quantization and distillation consistently provide greater benefit than pruning. Surprisingly, except for the pair of pruning and quantization, using multiple methods together rarely yields diminishing returns. Instead, we observe complementary and super-multiplicative reductions to model size. Our work quantitatively demonstrates that combining compression methods can synergistically reduce model size, and that practitioners should prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to maximize accuracy vs. model size tradeoffs.

* Accepted as short paper at COLING 2022. 5 pages main text, 5 pages appendix  

Share this with someone who'll enjoy it: