Constructing effective image priors is critical to solving ill-posed inverse problems, such as image reconstruction. Recent works proposed to exploit image non-local similarity for inverse problems by grouping similar patches, and demonstrated state-of-the-art results in many applications. However, comparing to classic local methods based on filtering or sparsity, most of the non-local algorithms are time-consuming, mainly due to the highly inefficient and redundant block matching step, where the distance between each pair of overlapping patches needs to be computed. In this work, we propose a novel self-convolution operator to exploit image non-local similarity in a self-supervised way. The proposed self-convolution can generalize the commonly-used block matching step, and produce the equivalent results with much cheaper computation. Furthermore, by applying self-convolution, we propose an effective multi-modality image restoration scheme, which is much more efficient than conventional block matching for non-local modeling. Experimental results demonstrate that (1) self-convolution can significantly speed up most of the popular non-local image restoration algorithms, with two-fold to nine-fold faster block matching; and (2) the proposed multi-modality restoration scheme achieves state-of-the-art denoising results on the RGB-NIR and Stereo image datasets. The code will be released on GitHub.