Vision-Conditioned Variational Bayesian Last Layer Dynamics Models

Agile control of robotic systems often requires anticipating how the environment affects system behavior. For example, a driver must perceive the road ahead to anticipate available friction and plan actions accordingly. Achieving such proactive adaptation within autonomous frameworks remains a challenge, particularly under rapidly changing conditions. Traditional modeling approaches often struggle to capture abrupt variations in system behavior, while adaptive methods are inherently reactive and may adapt too late to ensure safety. We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment. The model first learns nominal vehicle dynamics and is then fine-tuned with feature-wise affine transformations of latent features, enabling context-aware dynamics prediction. The resulting model is integrated into an optimal controller for vehicle racing. We validate our method on a Lexus LC500 racing through water puddles. With vision-conditioning, the system completed all 12 attempted laps under varying conditions. In contrast, all baselines without visual context consistently lost control, demonstrating the importance of proactive dynamics adaptation in high-performance applications.

Semantic Property Maps for Driving Applications

We consider the problem of estimating the parameters of a vehicle dynamics model for predictive control in driving applications. Instead of solely using the instantaneous parameters estimated from the vehicle signals, we combine this with cameras and update a probabilistic map with parameter estimates and semantic information using Bayesian moment matching. Key to this approach is the map representation, which is constructed with conjugate priors to the measurement likelihoods and defined in the same path coordinates as the vehicle controller, such that the map can be externalized to provide a local representation of the parameter likelihoods that vary in space. The result is a spatial map of vehicle parameters adapted online to enhance the driving control system. We provide theoretical guarantees on the smoothness of relevant parameter likelihood statistics as a function of space, which is critical for their use in predictive control.