Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework

Batch Normalization (BN) has become an essential technique in contemporary neural network design, enhancing training stability. Specifically, BN employs centering and scaling operations to standardize features along the batch dimension and uses an affine transformation to recover features. Although standard BN has shown its capability to improve deep neural network training and convergence, it still exhibits inherent limitations in certain cases. Most existing techniques that enhance BN consider a single or a few aspects of BN. In this paper, we first identify problems with BN from a feature perspective and explore that feature condensation exists in the learning when employing BN, which negatively affects testing performance. To tackle this problem, we propose a two-stage unified framework called Unified Batch Normalization (UBN). In the first stage, we utilize a simple feature condensation threshold to alleviate the feature condensation, which hinders inappropriate statistic updates in normalization. In the second stage, we unify various normalization variants to boost each component of BN. Our experimental results reveal that UBN significantly enhances performance across different visual backbones and notably expedites network training convergence, particularly in early training stages. Notably, our method improved about 3% in top-1 accuracy on ImageNet classification with large batch sizes, showing the effectiveness of our approach in real-world scenarios.

Hyperspectral Images Classification Based on Multi-scale Residual Network

Because hyperspectral remote sensing images contain a lot of redundant information and the data structure is highly non-linear, leading to low classification accuracy of traditional machine learning methods. The latest research shows that hyperspectral image classification based on deep convolutional neural network has high accuracy. However, when a small amount of data is used for training, the classification accuracy of deep learning methods is greatly reduced. In order to solve the problem of low classification accuracy of existing algorithms on small samples of hyperspectral images, a multi-scale residual network is proposed. The multi-scale extraction and fusion of spatial and spectral features is realized by adding a branch structure into the residual block and using convolution kernels of different sizes in the branch. The spatial and spectral information contained in hyperspectral images are fully utilized to improve the classification accuracy. In addition, in order to improve the speed and prevent overfitting, the model uses dynamic learning rate, BN and Dropout strategies. The experimental results show that the overall classification accuracy of this method is 99.07% and 99.96% respectively in the data set of Indian Pines and Pavia University, which is better than other algorithms.

Hyperspectral image classification based on multi-scale residual network with attention mechanism

Compared with traditional machine learning methods, deep learning methods such as convolutional neural networks (CNNs) have achieved great success in the hyperspectral image (HSI) classification task. HSI contains abundant spatial and spectral information, but they also contain a lot of invalid information, which may introduce noises and weaken the performance of CNNs. In order to make full use of the useful information in HSI, we propose a multi-scale residual network integrated with the attention mechanism (MSRN-A) for HSI classification in this letter. In our method, we built two different multi-scale feature extraction blocks to extract the joint spatial-spectral features and the advanced spatial features, respectively. Moreover, a spatial-spectral attention module and a spatial attention module were set up to focus on the salient spatial parts and valid spectral information. Experimental results demonstrate that our method achieves high accuracy on the Indian Pines, Pavia University, and Salinas datasets. The source code can be found at https://github.com/XiangdongZ/MSRN-A.