Alert button

TADA: Efficient Task-Agnostic Domain Adaptation for Transformers

May 22, 2023
Chia-Chien Hung, Lukas Lange, Jannik Strötgen

Figure 1 for TADA: Efficient Task-Agnostic Domain Adaptation for Transformers
Figure 2 for TADA: Efficient Task-Agnostic Domain Adaptation for Transformers
Figure 3 for TADA: Efficient Task-Agnostic Domain Adaptation for Transformers
Figure 4 for TADA: Efficient Task-Agnostic Domain Adaptation for Transformers

Share this with someone who'll enjoy it:

Intermediate training of pre-trained transformer-based language models on domain-specific data leads to substantial gains for downstream tasks. To increase efficiency and prevent catastrophic forgetting alleviated from full domain-adaptive pre-training, approaches such as adapters have been developed. However, these require additional parameters for each layer, and are criticized for their limited expressiveness. In this work, we introduce TADA, a novel task-agnostic domain adaptation method which is modular, parameter-efficient, and thus, data-efficient. Within TADA, we retrain the embeddings to learn domain-aware input representations and tokenizers for the transformer encoder, while freezing all other parameters of the model. Then, task-specific fine-tuning is performed. We further conduct experiments with meta-embeddings and newly introduced meta-tokenizers, resulting in one model per task in multi-domain use cases. Our broad evaluation in 4 downstream tasks for 14 domains across single- and multi-domain setups and high- and low-resource scenarios reveals that TADA is an effective and efficient alternative to full domain-adaptive pre-training and adapters for domain adaptation, while not introducing additional parameters or complex training steps.

* ACL-Findings 2023  
View paper onarxiv icon

Share this with someone who'll enjoy it: