Accurate diagnosis and prognosis of Alzheimer's disease are crucial for developing new therapies and reducing the associated costs. Recently, with the advances of convolutional neural networks, deep learning methods have been proposed to automate these two tasks using structural MRI. However, these methods often suffer from a lack of interpretability and generalization and have limited prognosis performance. In this paper, we propose a novel deep framework designed to overcome these limitations. Our pipeline consists of two stages. In the first stage, 125 3D U-Nets are used to estimate voxelwise grade scores over the whole brain. The resulting 3D maps are then fused to construct an interpretable 3D grading map indicating the disease severity at the structure level. As a consequence, clinicians can use this map to detect the brain structures affected by the disease. In the second stage, the grading map and subject's age are used to perform classification with a graph convolutional neural network. Experimental results based on 2106 subjects demonstrated competitive performance of our deep framework compared to state-of-the-art methods on different datasets for both AD diagnosis and prognosis. Moreover, we found that using a large number of U-Nets processing different overlapping brain areas improved the generalization capacity of the proposed methods.