Non-local attention module has been proven to be crucial for image restoration. Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers. To address this problem, we propose Cross-Layer Attention (CLA) module in this paper. Instead of finding correlated key pixels within the same layer, each query pixel can attend to key pixels at previous layers of the network. In order to further enhance the learning capability and reduce the inference cost of CLA, we further propose Adaptive CLA, or ACLA, as an improved CLA. Two adaptive designs are proposed for ACLA: 1) adaptively selecting the keys for non-local attention at each layer; 2) automatically searching for the insertion locations for ACLA modules. By these two adaptive designs, ACLA dynamically selects the number of keys to be aggregated for non-local attention at layer. In addition, ACLA searches for the optimal insert positions of ACLA modules by a neural architecture search method to render a compact neural network with compelling performance. Extensive experiments on image restoration tasks, including single image super-resolution, image denoising, image demosaicing, and image compression artifacts reduction, validate the effectiveness and efficiency of ACLA.