Cloud infrastructures are being increasingly utilized in critical infrastructures such as banking/finance, transportation and utility management. Sophistication and resources used in recent security breaches including those on critical infrastructures show that attackers are no longer limited by monetary/computational constraints. In fact, they may be aided by entities with large financial and human resources. Hence there is urgent need to develop predictive approaches for cyber defense to strengthen cloud infrastructures specifically utilized by critical infrastructures. Extensive research has been done in the past on applying techniques such as Game Theory, Machine Learning and Bayesian Networks among others for the predictive defense of critical infrastructures. However a major drawback of these approaches is that they do not incorporate probabilistic human behavior which limits their predictive ability. In this paper, a stochastic approach is proposed to predict less secure states in critical cloud systems which might lead to potential security breaches. These less-secure states are deemed as `risky' states in our approach. Markov Decision Process (MDP) is used to accurately incorporate user behavior(s) as well as operational behavior of the cloud infrastructure through a set of features. The developed reward/cost mechanism is then used to select appropriate `actions' to identify risky states at future time steps by learning an optimal policy. Experimental results show that the proposed framework performs well in identifying future `risky' states. Through this work we demonstrate the effectiveness of using probabilistic modeling (MDP) to predictively secure critical cloud infrastructures.