Wide-area dynamic studies are of paramount importance to ensure the stability and reliability of power grids. The rising deployment synchrophasor and other sensing technologies has made data-driven modeling and analysis possible using the synchronized fast-rate dynamic measurements. This paper presents a general model-free framework of inferring the grid dynamic responses using the ubiquitous ambient data collected during normal grid operations. Building upon the second-order dynamic model, we have established the connection from the cross-correlation of various types of angle, frequency, and line flow data at any two locations, to their corresponding dynamic responses. The theoretical results enabled a fully data-driven framework for estimating the latter using real-time ambient data. Numerical results using the WSCC 9-bus system and a synthetic 2000-bus Texas system have demonstrated the effectiveness of proposed approaches for dynamic modeling of realistic power systems.