Current deep learning models for dynamics forecasting struggle with generalization. They can only forecast in a specific domain and fail when applied to systems with different parameters, external forces, or boundary conditions. We propose a model-based meta-learning method called DyAd which can generalize across heterogeneous domains by partitioning them into separate subdomains, each with a different task. DyAd has two parts: a prediction network which learns the shared dynamics of the entire domain, and an encoder that infers the parameters of the task. The encoder adapts the prediction network during inference time using adaptive instance normalization and a new layer, AdaPad, specifically designed for boundary conditions. The encoder can also use any weak supervision signals that can help distinguish different tasks, allowing the incorporation of additional domain knowledge. Our model outperforms a variety of state-of-the-art approaches on both turbulent flow and real-world ocean data forecasting tasks.