Alert button

Rethinking the Knowledge Distillation From the Perspective of Model Calibration

Oct 31, 2021
Figure 1 for Rethinking the Knowledge Distillation From the Perspective of Model Calibration
Figure 2 for Rethinking the Knowledge Distillation From the Perspective of Model Calibration
Figure 3 for Rethinking the Knowledge Distillation From the Perspective of Model Calibration

Share this with someone who'll enjoy it:

Recent years have witnessed dramatically improvements in the knowledge distillation, which can generate a compact student model for better efficiency while retaining the model effectiveness of the teacher model. Previous studies find that: more accurate teachers do not necessary make for better teachers due to the mismatch of abilities. In this paper, we aim to analysis the phenomenon from the perspective of model calibration. We found that the larger teacher model may be too over-confident, thus the student model cannot effectively imitate. While, after the simple model calibration of the teacher model, the size of the teacher model has a positive correlation with the performance of the student model.

Share this with someone who'll enjoy it: