This work investigates fundamental questions related to locating and defining features in convolutional neural networks (CNN). The theoretical investigations guided by the locality principle show that the relevance of locations within a representation decreases with distance from the center. This is aligned with empirical findings across multiple architectures such as VGG, ResNet, Inception, DenseNet and MobileNet. To leverage our insights, we introduce Locality-promoting Regularization (LOCO-REG). It yields accuracy gains across multiple architectures and datasets.