Alert button
Picture for Ruoxin Sang

Ruoxin Sang

Alert button

Structured Model Pruning of Convolutional Networks on Tensor Processing Units

Jul 21, 2021
Kongtao Chen, Ken Franko, Ruoxin Sang

Figure 1 for Structured Model Pruning of Convolutional Networks on Tensor Processing Units
Figure 2 for Structured Model Pruning of Convolutional Networks on Tensor Processing Units
Figure 3 for Structured Model Pruning of Convolutional Networks on Tensor Processing Units
Figure 4 for Structured Model Pruning of Convolutional Networks on Tensor Processing Units

The deployment of convolutional neural networks is often hindered by high computational and storage requirements. Structured model pruning is a promising approach to alleviate these requirements. Using the VGG-16 model as an example, we measure the accuracy-efficiency trade-off for various structured model pruning methods and datasets (CIFAR-10 and ImageNet) on Tensor Processing Units (TPUs). To measure the actual performance of models, we develop a structured model pruning library for TensorFlow2 to modify models in place (instead of adding mask layers). We show that structured model pruning can significantly improve model memory usage and speed on TPUs without losing accuracy, especially for small datasets (e.g., CIFAR-10).

* International Conference on Machine Learning 2021 Workshop on Overparameterization: Pitfalls & Opportunities 
Viaarxiv icon

Ranking Neural Checkpoints

Nov 23, 2020
Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, Boqing Gong

Figure 1 for Ranking Neural Checkpoints
Figure 2 for Ranking Neural Checkpoints
Figure 3 for Ranking Neural Checkpoints
Figure 4 for Ranking Neural Checkpoints

This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation cost, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, NLEEP, which gives rise to the best performance in the experiments.

Viaarxiv icon