Falls are the leading cause of fatal and non-fatal injuries particularly for older persons. Imbalance can result from body internal causes such as illness, or external causes such as active or passive perturbation. Active perturbation is the result of applying an external force to a person, while passive perturbation results from human motion interacting with a static obstacle. This work proposes a metric that allows for the monitoring of the persons torso and its correlation to active and passive perturbations. We show that large change in the torso sway can be strongly correlated to active perturbations. We also show that by conditioning the expected path and torso sway on the past trajectory, torso motion and the surrounding scene, we can reasonably predict the future path and expected change in torso sway. This will have direct future application to fall prevention. The results demonstrated that the torso sway is strongly correlated with perturbations. And our model is able to make use of the visual cues presented in the panorama and condition the prediction accordingly.