Improving Neural Network Learning Through Dual Variable Learning Rates

This paper introduces and evaluates a novel training method for neural networks: Dual Variable Learning Rates (DVLR). Building on techniques and insights from behavioral psychology, the dual learning rates are used to emphasize correct and incorrect responses differently, thereby making the feedback to the network more specific. Further, the learning rates are varied as a function of the network's performance, thereby making it more efficient. DVLR was implemented on both a simple feedforward neural network and a convolutional neural network. Both networks are trained faster and achieve an increased accuracy on the MNIST and CIFAR-10 domains demonstrating that DVLR is a promising, psychologically motivated technique for training neural network models.