Alert button

Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training

Oct 13, 2022
Figure 1 for Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training
Figure 2 for Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training
Figure 3 for Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training
Figure 4 for Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training

Share this with someone who'll enjoy it:

Language is constantly changing and evolving, leaving language models to quickly become outdated, both factually and linguistically. Recent research proposes we continuously update our models using new data. Continuous training allows us to teach language models about new events and facts and changing norms. However, continuous training also means continuous costs. We show there is currently limited evidence for the benefits of continuous training, be it for the actual downstream performance or the environmental cost. Our results show continuous training does not significantly improve performance. While it is clear that, sooner or later, our language models need to be updated, it is unclear when this effort is worth the cost. We call for a critical reflection about when and how to use continuous training and for more benchmarks to support this research direction.

Share this with someone who'll enjoy it: