Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox


Meta-learning for downstream aware and agnostic pretraining

Jun 06, 2021
Hongyin Luo, Shuyan Dong, Yung-Sung Chuang, Shang-Wen Li


Share this with someone who'll enjoy it:


Neural network pretraining is gaining attention due to its outstanding performance in natural language processing applications. However, pretraining usually leverages predefined task sequences to learn general linguistic clues. The lack of mechanisms in choosing proper tasks during pretraining makes the learning and knowledge encoding inefficient. We thus propose using meta-learning to select tasks that provide the most informative learning signals in each episode of pretraining. With the proposed method, we aim to achieve better efficiency in computation and memory usage for the pretraining process and resulting networks while maintaining the performance. In this preliminary work, we discuss the algorithm of the method and its two variants, downstream-aware and downstream-agnostic pretraining. Our experiment plan is also summarized, while empirical results will be shared in our future works.

* Meta Learning and Its Applications to Natural Language Processing workshop at ACL 2021 
* Extended abstract 


   Access Paper Source



Share this with someone who'll enjoy it: