Training machine learning models that can learn complex spatiotemporal dynamics and generalize under distributional shift is a fundamental challenge. The symmetries in a physical system play a unique role in characterizing unchanged features under transformation. We propose a systematic approach to improve generalization in spatiotemporal models by incorporating symmetries into deep neural networks. Our general framework to design equivariant convolutional models employs (1) convolution with equivariant kernels, (2) conjugation by averaging operators in order to force equivariance, (3) and a naturally equivariant generalization of convolution called group correlation. Our framework is both theoretically and experimentally robust to distributional shift by a symmetry group and enjoys favorable sample complexity. We demonstrate the advantage of our approach on a variety of physical dynamics including turbulence and diffusion systems. This is the first time that equivariant CNNs have been used to forecast physical dynamics.