Alert button

Perturbation-based Self-supervised Attention for Attention Bias in Text Classification

May 25, 2023
Huawen Feng, Zhenxi Lin, Qianli Ma

Figure 1 for Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
Figure 2 for Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
Figure 3 for Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
Figure 4 for Perturbation-based Self-supervised Attention for Attention Bias in Text Classification

Share this with someone who'll enjoy it:

In text classification, the traditional attention mechanisms usually focus too much on frequent words, and need extensive labeled data in order to learn. This paper proposes a perturbation-based self-supervised attention approach to guide attention learning without any annotation overhead. Specifically, we add as much noise as possible to all the words in the sentence without changing their semantics and predictions. We hypothesize that words that tolerate more noise are less significant, and we can use this information to refine the attention distribution. Experimental results on three text classification tasks show that our approach can significantly improve the performance of current attention-based models, and is more effective than existing self-supervised methods. We also provide a visualization analysis to verify the effectiveness of our approach.

View paper onarxiv icon

Share this with someone who'll enjoy it: