Joint saliency map (JSM) [1] was developed to assign high joint saliency values to the corresponding saliency structures (called Joint Saliency Structures, JSSs) but zero or low joint saliency values to the outliers (or mismatches) that are introduced by missing correspondence or local large deformations between the reference and moving images to be registered. JSM guides the local structure matching in nonrigid registration by emphasizing these JSSs' sparse deformation vectors in adaptive kernel regression of hierarchical sparse deformation vectors for iterative dense deformation reconstruction. By designing an effective superpixel-based local structure scale estimator to compute the reference structure's structure scale, we further propose to determine the scale (the width) of kernels in the adaptive kernel regression through combining the structure scales to JSM-based scales of mismatch between the local saliency structures. Therefore, we can adaptively select the sample size of sparse deformation vectors to reconstruct the dense deformation vectors for accurately matching the every local structures in the two images. The experimental results demonstrate better accuracy of our method in aligning two images with missing correspondence and local large deformation than the state-of-the-art methods.