Alert button

Weight Poisoning Attacks on Pre-trained Models

Apr 14, 2020
Keita Kurita, Paul Michel, Graham Neubig

Figure 1 for Weight Poisoning Attacks on Pre-trained Models
Figure 2 for Weight Poisoning Attacks on Pre-trained Models
Figure 3 for Weight Poisoning Attacks on Pre-trained Models
Figure 4 for Weight Poisoning Attacks on Pre-trained Models

Share this with someone who'll enjoy it:

Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.

* Published as a long paper at ACL 2020  
View paper onarxiv icon

Share this with someone who'll enjoy it: