Augmented Equivariant Attention Networks for Electron Microscopy Image Super-Resolution

Taking electron microscopy (EM) images in high-resolution is time-consuming and expensive and could be detrimental to the integrity of the samples under observation. Advances in deep learning enable us to perform super-resolution computationally, so as to obtain high-resolution images from low-resolution ones. When training super-resolution models on pairs of experimentally acquired EM images, prior models suffer from performance loss while using the pooled-training strategy due to their inability to capture inter-image dependencies and common features shared among images. Although there exist methods that take advantage of shared features among input instances in image classification tasks, they in the current form cannot be applied to super-resolution tasks because they fail to preserve an essential property in image-to-image transformation problems, which is the equivariance property to spatial permutations. To address these limitations, we propose the augmented equivariant attention networks (AEANets) with better capability to capture inter-image dependencies and shared features, while preserving the equivariance to spatial permutations. The proposed AEANets captures inter-image dependencies and common features shared among images via two augmentations on the attention mechanism; namely, the shared references and the batch-aware attention during training. We theoretically show the equivariance property of the proposed augmented attention model and experimentally show that AEANets consistently outperforms the baselines in both quantitative and visual results.