We present the MagNet, a multi-agent interaction network to discover governing dynamics and predict evolution of a complex system from observations. We formulate a multiagent system as a coupled non-linear network with a generic ordinary differential equation (ODE) based state evolution and develop a neural network-based realization of its time-discretized model. MagNet is trained to discover the core dynamics of a multi-agent system from observations and tuned on-line to learn agent-specific parameters of the dynamics to ensure accurate prediction even when physical or relational attributes of agents, or number of agents change. We evaluate MagNet on a point-mass system in two-dimensional space, Kuramoto phase synchronization dynamics and predator-swarm interaction dynamics demonstrating orders of magnitude improvement in prediction accuracy over traditional deep learning models.