Alert button

Compressing Transformer-based self-supervised models for speech processing

Nov 17, 2022
Tzu-Quan Lin, Tsung-Huan Yang, Chun-Yao Chang, Kuang-Ming Chen, Tzu-hsun Feng, Hung-yi Lee, Hao Tang

Figure 1 for Compressing Transformer-based self-supervised models for speech processing
Figure 2 for Compressing Transformer-based self-supervised models for speech processing
Figure 3 for Compressing Transformer-based self-supervised models for speech processing
Figure 4 for Compressing Transformer-based self-supervised models for speech processing

Share this with someone who'll enjoy it:

Despite the success of Transformers in self-supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices. Several isolated attempts have been made to compress Transformers, prior to applying them to downstream tasks. In this work, we aim to provide context for the isolated results, studying several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report wall-clock time, the number of parameters, and the number of multiply-accumulate operations for these techniques, charting the landscape of compressing Transformer-based self-supervised models.

* Submitted to ICASSP 2023  
View paper onarxiv icon

Share this with someone who'll enjoy it: