We present a novel visual attention tracking technique based on Shared Attention modeling. Our proposed method models the viewer as a participant in the activity occurring in the scene. We go beyond image salience and instead of only computing the power of an image region to pull attention to it, we also consider the strength with which other regions of the image push attention to the region in question. We use the term Attentional Push to refer to the power of image regions to direct and manipulate the attention allocation of the viewer. An attention model is presented that incorporates the Attentional Push cues with standard image salience-based attention modeling algorithms to improve the ability to predict where viewers will fixate. Experimental evaluation validates significant improvements in predicting viewers' fixations using the proposed methodology in both static and dynamic imagery.