Alert button

Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective

Add code
Alert button
Jun 04, 2021
Figure 1 for Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective
Figure 2 for Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective
Figure 3 for Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective
Figure 4 for Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective

Share this with someone who'll enjoy it:

ResNets constrained to be bi-Lipschitz, that is, approximately distance preserving, have been a crucial component of recently proposed techniques for deterministic uncertainty quantification in neural models. We show that theoretical justifications for recent regularisation schemes trying to enforce such a constraint suffer from a crucial flaw -- the theoretical link between the regularisation scheme used and bi-Lipschitzness is only valid under conditions which do not hold in practice, rendering existing theory of limited use, despite the strong empirical performance of these models. We provide a theoretical explanation for the effectiveness of these regularisation schemes using a frequency analysis perspective, showing that under mild conditions these schemes will enforce a lower Lipschitz bound on the low-frequency projection of images. We then provide empirical evidence supporting our theoretical claims, and perform further experiments which demonstrate that our broader conclusions appear to hold when some of the mathematical assumptions of our proof are relaxed, corresponding to the setup used in prior work. In addition, we present a simple constructive algorithm to search for counter examples to the distance preservation condition, and discuss possible implications of our theory for future model design.

* Main paper 10 pages including references, appendix 10 pages. 7 figures and 6 tables including appendix  

Share this with someone who'll enjoy it: