Alert button

CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT

Dec 22, 2022
Dan DeGenaro, Jugal Kalita

Figure 1 for CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT
Figure 2 for CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT
Figure 3 for CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT
Figure 4 for CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT

Share this with someone who'll enjoy it:

Large language models having hundreds of millions, and even billions, of parameters have performed extremely well on a variety of natural language processing (NLP) tasks. Their widespread use and adoption, however, is hindered by the lack of availability and portability of sufficiently large computational resources. This paper proposes a knowledge distillation (KD) technique building on the work of LightMBERT, a student model of multilingual BERT (mBERT). By repeatedly distilling mBERT through increasingly compressed toplayer distilled teacher assistant networks, CAMeMBERT aims to improve upon the time and space complexities of mBERT while keeping loss of accuracy beneath an acceptable threshold. At present, CAMeMBERT has an average accuracy of around 60.1%, which is subject to change after future improvements to the hyperparameters used in fine-tuning.

* 4 pages, 2 figures, 3 tables  
View paper onarxiv icon

Share this with someone who'll enjoy it: