Alert button

Variability of Artificial Neural Networks

May 20, 2021
Yin Zhang, Yueyao Yu

Figure 1 for Variability of Artificial Neural Networks
Figure 2 for Variability of Artificial Neural Networks
Figure 3 for Variability of Artificial Neural Networks
Figure 4 for Variability of Artificial Neural Networks

Share this with someone who'll enjoy it:

What makes an artificial neural network easier to train and more likely to produce desirable solutions than other comparable networks? In this paper, we provide a new angle to study such issues under the setting of a fixed number of model parameters which in general is the most dominant cost factor. We introduce a notion of variability and show that it correlates positively to the activation ratio and negatively to a phenomenon called {Collapse to Constants} (or C2C), which is closely related but not identical to the phenomenon commonly known as vanishing gradient. Experiments on a styled model problem empirically verify that variability is indeed a key performance indicator for fully connected neural networks. The insights gained from this variability study will help the design of new and effective neural network architectures.

View paper onarxiv icon

Share this with someone who'll enjoy it: