Autonomous driving in mixed traffic requires reliable motion prediction of nearby traffic agents such as pedestrians, bicycles, cars, buses, etc.. This prediction problem is extremely challenging because of the diverse dynamics and geometry of traffic agents, complex road conditions, and intensive interactions between them. In this paper, we proposed GAMMA, a general agent motion prediction model for autonomous driving, that can predict the motion of heterogeneous traffic agents with different kinematics, geometry, etc., and generate multiple hypotheses of trajectories by inferring about human agents' inner states. GAMMA formalizes motion prediction as a geometric optimization problem in the velocity space, and integrates physical constraints and human inner states into this unified framework. Our results show that GAMMA outperforms both traditional and deep learning approaches significantly on diverse real-world datasets.