Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Attention Temperature Matters in Abstractive Summarization Distillation

Jun 07, 2021
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei

Share this with someone who'll enjoy it:

Recent progress of abstractive text summarization largely relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. This paper aims to distill these large models into smaller ones for faster inference and minimal performance loss. Pseudo-labeling based methods are popular in sequence-to-sequence model distillation. In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models. Our experiments on three summarization datasets show our proposed method consistently improves over vanilla pseudo-labeling based methods. We also find that both the pseudo labels and summaries produced by our students are shorter and more abstractive. We will make our code and models publicly available.

* Submitted to NeurIPS 2021 

   Access Paper Source

Share this with someone who'll enjoy it: