Deep learning is increasingly used for state estimation problems such as tracking, navigation, and pose estimation. The uncertainties associated with these measurements are typically assumed to be a fixed covariance matrix. For many scenarios this assumption is inaccurate, leading to worse subsequent filtered state estimates. We show how to model multivariate uncertainty for regression problems with neural networks, incorporating both aleatoric and epistemic sources of heteroscedastic uncertainty. We train a deep uncertainty covariance matrix model in two ways: directly using a multivariate Gaussian density loss function, and indirectly using end-to-end training through a Kalman filter. We experimentally show in a visual tracking problem the large impact that accurate multivariate uncertainty quantification can have on Kalman filter estimation for both in-domain and out-of-domain evaluation data.