Recently, more and more attention has been drawn into the internal mechanism of the convolutional neural network and on what basis does the network make a specific decision. In this paper, we develop a novel post-hoc visual explanation method called Score-CAM based on class activation mapping. Unlike previous class activation mapping based approaches, Score-CAM gets rid of the dependence on gradient by obtaining the weight of each activation map through its forward passing score on target class, the final result is obtained by a linear combination of weights and activation maps. We demonstrate that Score-CAM achieves better visual performance with less noise and has better stability than Grad-CAM and Grad-CAM++. In the experiment, we rethink issues of previous evaluation metrics and propose a representative evaluation approach Energy- Based Pointing Game to measure the quality of the generated saliency maps. Our approach outperforms previous methods on energy-based pointing game and recognition and shows more robustness under adversarial attack.