Alert button
Picture for Frank Wood

Frank Wood

Alert button

A Diffusion-Model of Joint Interactive Navigation

Sep 21, 2023
Matthew Niedoba, Jonathan Wilder Lavington, Yunpeng Liu, Vasileios Lioutas, Justice Sefas, Xiaoxuan Liang, Dylan Green, Setareh Dabiri, Berend Zwartsenberg, Adam Scibior, Frank Wood

Figure 1 for A Diffusion-Model of Joint Interactive Navigation
Figure 2 for A Diffusion-Model of Joint Interactive Navigation
Figure 3 for A Diffusion-Model of Joint Interactive Navigation
Figure 4 for A Diffusion-Model of Joint Interactive Navigation

Simulation of autonomous vehicle systems requires that simulated traffic participants exhibit diverse and realistic behaviors. The use of prerecorded real-world traffic scenarios in simulation ensures realism but the rarity of safety critical events makes large scale collection of driving scenarios expensive. In this paper, we present DJINN - a diffusion based method of generating traffic scenarios. Our approach jointly diffuses the trajectories of all agents, conditioned on a flexible set of state observations from the past, present, or future. On popular trajectory forecasting datasets, we report state of the art performance on joint trajectory metrics. In addition, we demonstrate how DJINN flexibly enables direct test-time sampling from a variety of valuable conditional distributions including goal-based sampling, behavior-class sampling, and scenario editing.

* 10 pages, 4 figures 
Viaarxiv icon

Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance

Jul 31, 2023
Saeid Naderiparizi, Xiaoxuan Liang, Berend Zwartsenberg, Frank Wood

Figure 1 for Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance
Figure 2 for Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance
Figure 3 for Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance
Figure 4 for Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance

The maximum likelihood principle advocates parameter estimation via optimization of the data likelihood function. Models estimated in this way can exhibit a variety of generalization characteristics dictated by, e.g. architecture, parameterization, and optimization bias. This work addresses model learning in a setting where there further exists side-information in the form of an oracle that can label samples as being outside the support of the true data generating distribution. Specifically we develop a new denoising diffusion probabilistic modeling (DDPM) methodology, Gen-neG, that leverages this additional side-information. Our approach builds on generative adversarial networks (GANs) and discriminator guidance in diffusion models to guide the generation process towards the positive support region indicated by the oracle. We empirically establish the utility of Gen-neG in applications including collision avoidance in self-driving simulators and safety-guarded human motion generation.

Viaarxiv icon

Realistically distributing object placements in synthetic training data improves the performance of vision-based object detection models

May 24, 2023
Setareh Dabiri, Vasileios Lioutas, Berend Zwartsenberg, Yunpeng Liu, Matthew Niedoba, Xiaoxuan Liang, Dylan Green, Justice Sefas, Jonathan Wilder Lavington, Frank Wood, Adam Scibior

Figure 1 for Realistically distributing object placements in synthetic training data improves the performance of vision-based object detection models
Figure 2 for Realistically distributing object placements in synthetic training data improves the performance of vision-based object detection models
Figure 3 for Realistically distributing object placements in synthetic training data improves the performance of vision-based object detection models
Figure 4 for Realistically distributing object placements in synthetic training data improves the performance of vision-based object detection models

When training object detection models on synthetic data, it is important to make the distribution of synthetic data as close as possible to the distribution of real data. We investigate specifically the impact of object placement distribution, keeping all other aspects of synthetic data fixed. Our experiment, training a 3D vehicle detection model in CARLA and testing on KITTI, demonstrates a substantial improvement resulting from improving the object placement distribution.

Viaarxiv icon

Video Killed the HD-Map: Predicting Driving Behavior Directly From Drone Images

May 19, 2023
Yunpeng Liu, Vasileios Lioutas, Jonathan Wilder Lavington, Matthew Niedoba, Justice Sefas, Setareh Dabiri, Dylan Green, Xiaoxuan Liang, Berend Zwartsenberg, Adam Ścibior, Frank Wood

Figure 1 for Video Killed the HD-Map: Predicting Driving Behavior Directly From Drone Images
Figure 2 for Video Killed the HD-Map: Predicting Driving Behavior Directly From Drone Images
Figure 3 for Video Killed the HD-Map: Predicting Driving Behavior Directly From Drone Images
Figure 4 for Video Killed the HD-Map: Predicting Driving Behavior Directly From Drone Images

The development of algorithms that learn behavioral driving models using human demonstrations has led to increasingly realistic simulations. In general, such models learn to jointly predict trajectories for all controlled agents by exploiting road context information such as drivable lanes obtained from manually annotated high-definition (HD) maps. Recent studies show that these models can greatly benefit from increasing the amount of human data available for training. However, the manual annotation of HD maps which is necessary for every new location puts a bottleneck on efficiently scaling up human traffic datasets. We propose a drone birdview image-based map (DBM) representation that requires minimal annotation and provides rich road context information. We evaluate multi-agent trajectory prediction using the DBM by incorporating it into a differentiable driving simulator as an image-texture-based differentiable rendering module. Our results demonstrate competitive multi-agent trajectory prediction performance when using our DBM representation as compared to models trained with rasterized HD maps.

Viaarxiv icon

Visual Chain-of-Thought Diffusion Models

Mar 28, 2023
William Harvey, Frank Wood

Figure 1 for Visual Chain-of-Thought Diffusion Models
Figure 2 for Visual Chain-of-Thought Diffusion Models
Figure 3 for Visual Chain-of-Thought Diffusion Models
Figure 4 for Visual Chain-of-Thought Diffusion Models

Recent progress with conditional image diffusion models has been stunning, and this holds true whether we are speaking about models conditioned on a text description, a scene layout, or a sketch. Unconditional image diffusion models are also improving but lag behind, as do diffusion models which are conditioned on lower-dimensional features like class labels. We propose to close the gap between conditional and unconditional models using a two-stage sampling procedure. In the first stage we sample an embedding describing the semantic content of the image. In the second stage we sample the image conditioned on this embedding and then discard the embedding. Doing so lets us leverage the power of conditional diffusion models on the unconditional generation task, which we show improves FID by 25-50% compared to standard unconditional generation.

Viaarxiv icon

Vehicle Type Specific Waypoint Generation

Aug 09, 2022
Yunpeng Liu, Jonathan Wilder Lavington, Adam Scibior, Frank Wood

Figure 1 for Vehicle Type Specific Waypoint Generation
Figure 2 for Vehicle Type Specific Waypoint Generation

We develop a generic mechanism for generating vehicle-type specific sequences of waypoints from a probabilistic foundation model of driving behavior. Many foundation behavior models are trained on data that does not include vehicle information, which limits their utility in downstream applications such as planning. Our novel methodology conditionally specializes such a behavior predictive model to a vehicle-type by utilizing byproducts of the reinforcement learning algorithms used to produce vehicle specific controllers. We show how to compose a vehicle specific value function estimate with a generic probabilistic behavior model to generate vehicle-type specific waypoint sequences that are more likely to be physically plausible then their vehicle-agnostic counterparts.

Viaarxiv icon

Conditional Permutation Invariant Flows

Jun 17, 2022
Berend Zwartsenberg, Adam Ścibior, Matthew Niedoba, Vasileios Lioutas, Yunpeng Liu, Justice Sefas, Setareh Dabiri, Jonathan Wilder Lavington, Trevor Campbell, Frank Wood

Figure 1 for Conditional Permutation Invariant Flows
Figure 2 for Conditional Permutation Invariant Flows
Figure 3 for Conditional Permutation Invariant Flows
Figure 4 for Conditional Permutation Invariant Flows

We present a novel, conditional generative probabilistic model of set-valued data with a tractable log density. This model is a continuous normalizing flow governed by permutation equivariant dynamics. These dynamics are driven by a learnable per-set-element term and pairwise interactions, both parametrized by deep neural networks. We illustrate the utility of this model via applications including (1) complex traffic scene generation conditioned on visually specified map information, and (2) object bounding box generation conditioned directly on images. We train our model by maximizing the expected likelihood of labeled conditional data under our flow, with the aid of a penalty that ensures the dynamics are smooth and hence efficiently solvable. Our method significantly outperforms non-permutation invariant baselines in terms of log likelihood and domain-specific metrics (offroad, collision, and combined infractions), yielding realistic samples that are difficult to distinguish from real data.

* 20 pages, 10 figures 
Viaarxiv icon

Critic Sequential Monte Carlo

May 30, 2022
Vasileios Lioutas, Jonathan Wilder Lavington, Justice Sefas, Matthew Niedoba, Yunpeng Liu, Berend Zwartsenberg, Setareh Dabiri, Frank Wood, Adam Scibior

Figure 1 for Critic Sequential Monte Carlo
Figure 2 for Critic Sequential Monte Carlo
Figure 3 for Critic Sequential Monte Carlo
Figure 4 for Critic Sequential Monte Carlo

We introduce CriticSMC, a new algorithm for planning as inference built from a novel composition of sequential Monte Carlo with learned soft-Q function heuristic factors. This algorithm is structured so as to allow using large numbers of putative particles leading to efficient utilization of computational resource and effective discovery of high reward trajectories even in environments with difficult reward surfaces such as those arising from hard constraints. Relative to prior art our approach is notably still compatible with model-free reinforcement learning in the sense that the implicit policy we produce can be used at test time in the absence of a world model. Our experiments on self-driving car collision avoidance in simulation demonstrate improvements against baselines in terms of infraction minimization relative to computational effort while maintaining diversity and realism of found trajectories.

* 20 pages, 3 figures 
Viaarxiv icon

Flexible Diffusion Modeling of Long Videos

May 23, 2022
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, Frank Wood

Figure 1 for Flexible Diffusion Modeling of Long Videos
Figure 2 for Flexible Diffusion Modeling of Long Videos
Figure 3 for Flexible Diffusion Modeling of Long Videos
Figure 4 for Flexible Diffusion Modeling of Long Videos

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA self-driving car simulator.

* 17 pages, 12 figures 
Viaarxiv icon

BayesPCN: A Continually Learnable Predictive Coding Associative Memory

May 20, 2022
Jason Yoo, Frank Wood

Figure 1 for BayesPCN: A Continually Learnable Predictive Coding Associative Memory
Figure 2 for BayesPCN: A Continually Learnable Predictive Coding Associative Memory
Figure 3 for BayesPCN: A Continually Learnable Predictive Coding Associative Memory
Figure 4 for BayesPCN: A Continually Learnable Predictive Coding Associative Memory

Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning. While the machine learning community's interest in associative memories has recently been rekindled, most work has focused on memory recall ($read$) over memory learning ($write$). In this paper, we present BayesPCN, a hierarchical associative memory capable of performing continual one-shot memory writes without meta-learning. Moreover, BayesPCN is able to gradually forget past observations ($forget$) to free its memory. Experiments show that BayesPCN can recall corrupted i.i.d. high-dimensional data observed hundreds of "timesteps" ago without a significant drop in recall ability compared to the state-of-the-art offline-learned associative memory models.

Viaarxiv icon