Low-Rank Matrix Approximation for Neural Network Compression

Deep Neural Networks (DNNs) are often constrained by their large memories and computational restrictions. In this paper, we introduce a novel adaptive-rank Singular Value Decomposition (ARSVD) that dynamically chooses the rank increase of the fully connected layers below a certain threshold in energy expenditure. Unlike conventional SVD compression methods that apply a fixed rank reduction in all layers, our ARSVD method uses energy distribution to adaptively select rank per layer while retaining accuracy. This is done for each layer in an effort to use as much energy as possible while maintaining the lowest accuracy loss. Such accuracy-adaptive approaches outperform traditional static rank reduction methods by providing an improved balance between compression and model performance. We first train a simple Multi-Layer Perceptron (MLP) on the MNIST, CIFAR-10, and CIFAR-100 dataset and evaluate its performance using accuracy and F1-score. After applying ARSVD, our results demonstrate that the technique can achieve substantial model compression without compromising classification accuracy. These results illustrate the usefulness of ARSVD in computing scenarios where both computational and memory resources are scarce.