Overcoming Catastrophic Forgetting by Soft Parameter Pruning

Catastrophic forgetting is a challenge issue in continual learning when a deep neural network forgets the knowledge acquired from the former task after learning on subsequent tasks. However, existing methods try to find the joint distribution of parameters shared with all tasks. This idea can be questionable because this joint distribution may not present when the number of tasks increase. On the other hand, It also leads to "long-term" memory issue when the network capacity is limited since adding tasks will "eat" the network capacity. In this paper, we proposed a Soft Parameters Pruning (SPP) strategy to reach the trade-off between short-term and long-term profit of a learning model by freeing those parameters less contributing to remember former task domain knowledge to learn future tasks, and preserving memories about previous tasks via those parameters effectively encoding knowledge about tasks at the same time. The SPP also measures the importance of parameters by information entropy in a label free manner. The experiments on several tasks shows SPP model achieved the best performance compared with others state-of-the-art methods. Experiment results also indicate that our method is less sensitive to hyper-parameter and better generalization. Our research suggests that a softer strategy, i.e. approximate optimize or sub-optimal solution, will benefit alleviating the dilemma of memory. The source codes are available at https://github.com/lehaifeng/Learning_by_memory.

Learning to Measure Change: Fully Convolutional Siamese Metric Networks for Scene Change Detection

Following the intuitive idea of detecting changes by directly comparing dissimilarity between a pair of images, we propose a novel Fully Convolutional siamese metric Network (CosimNet) to measure changes by customizing implicit metric. To learn more discriminative metrics, we utilize contrastive loss to reduce the distance between the unchanged feature pair and enlarge the distance between changed feature pair. Specially, to address the issue of large viewpoint difference, we propose Thresholded Contrastive Loss (TCL) with more tolerance strategy to punish this noisy change. We demonstrate the effectiveness of the proposed approach with experiments on three challenging datasets including CDnet, PCD2015, and VL-CMU-CD. Source code is available at https://github.com/gmayday1997/ChangeDet.