Recurrent neural networks are widely used on time series data, yet such models often ignore the underlying physical structures in such sequences. A new class of physically-based methods related to Koopman theory has been introduced, offering an alternative for processing nonlinear dynamical systems. In this work, we propose a novel Consistent Koopman Autoencoder model which, unlike the majority of existing work, leverages the forward and backward dynamics. Key to our approach is a new analysis that unravels the interplay between consistent dynamics and their associated Koopman operators. Our network is interpretable from a physical viewpoint and its computational requirements are comparable to other baselines. We evaluate our method on a wide range of high-dimensional and short-term dependent problems. The datasets include nonlinear oscillators, sea surface temperature data, and fluid flows on a curved domain. The results show that our model yields accurate estimates for significant prediction horizons, while being robust to noise.