Get our free extension to see links to code for papers anywhere online!

 Add to Chrome

 Add to Firefox

CatalyzeX Code Finder - Browser extension linking code for ML papers across the web! | Product Hunt Embed

ResKD: Residual-Guided Knowledge Distillation

Jun 08, 2020
Xuewei Li, Songyuan Li, Bourahla Omar, Xi Li



Knowledge distillation has emerge as a promising technique for compressing neural networks. Due to the capacity gap between a heavy teacher and a lightweight student, there exists a significant performance gap between them. In this paper, we see knowledge distillation in a fresh light, using the knowledge gap between a teacher and a student as guidance to train a lighter-weight student called res-student. The combination of a normal student and a res-student becomes a new student. Such a residual-guided process can be repeated. Experimental results show that we achieve competitive results on the CIFAR10/10, Tiny-ImageNet, and ImageNet datasets.



Share this with someone who'll enjoy it:

   Access Paper Source



Share this with someone who'll enjoy it: