Alert button
Picture for Edward Schmerling

Edward Schmerling

Alert button

Data Lifecycle Management in Evolving Input Distributions for Learning-based Aerospace Applications

Sep 14, 2022
Somrita Banerjee, Apoorva Sharma, Edward Schmerling, Max Spolaor, Michael Nemerouf, Marco Pavone

Figure 1 for Data Lifecycle Management in Evolving Input Distributions for Learning-based Aerospace Applications
Figure 2 for Data Lifecycle Management in Evolving Input Distributions for Learning-based Aerospace Applications
Figure 3 for Data Lifecycle Management in Evolving Input Distributions for Learning-based Aerospace Applications
Figure 4 for Data Lifecycle Management in Evolving Input Distributions for Learning-based Aerospace Applications

As input distributions evolve over a mission lifetime, maintaining performance of learning-based models becomes challenging. This paper presents a framework to incrementally retrain a model by selecting a subset of test inputs to label, which allows the model to adapt to changing input distributions. Algorithms within this framework are evaluated based on (1) model performance throughout mission lifetime and (2) cumulative costs associated with labeling and model retraining. We provide an open-source benchmark of a satellite pose estimation model trained on images of a satellite in space and deployed in novel scenarios (e.g., different backgrounds or misbehaving pixels), where algorithms are evaluated on their ability to maintain high performance by retraining on a subset of inputs. We also propose a novel algorithm to select a diverse subset of inputs for labeling, by characterizing the information gain from an input using Bayesian uncertainty quantification and choosing a subset that maximizes collective information gain using concepts from batch active learning. We show that our algorithm outperforms others on the benchmark, e.g., achieves comparable performance to an algorithm that labels 100% of inputs, while only labeling 50% of inputs, resulting in low costs and high performance over the mission lifetime.

Viaarxiv icon

Using Spectral Submanifolds for Nonlinear Periodic Control

Sep 14, 2022
Florian Mahlknecht, John Irvin Alora, Shobhit Jain, Edward Schmerling, Riccardo Bonalli, George Haller, Marco Pavone

Figure 1 for Using Spectral Submanifolds for Nonlinear Periodic Control
Figure 2 for Using Spectral Submanifolds for Nonlinear Periodic Control
Figure 3 for Using Spectral Submanifolds for Nonlinear Periodic Control
Figure 4 for Using Spectral Submanifolds for Nonlinear Periodic Control

Very high dimensional nonlinear systems arise in many engineering problems due to semi-discretization of the governing partial differential equations, e.g. through finite element methods. The complexity of these systems present computational challenges for direct application to automatic control. While model reduction has seen ubiquitous applications in control, the use of nonlinear model reduction methods in this setting remains difficult. The problem lies in preserving the structure of the nonlinear dynamics in the reduced order model for high-fidelity control. In this work, we leverage recent advances in Spectral Submanifold (SSM) theory to enable model reduction under well-defined assumptions for the purpose of efficiently synthesizing feedback controllers.

* 8 pages, 6 figures, conference on decision and control 2022 
Viaarxiv icon

Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation

Jun 24, 2022
Sever Topan, Karen Leung, Yuxiao Chen, Pritish Tupekar, Edward Schmerling, Jonas Nilsson, Michael Cox, Marco Pavone

Figure 1 for Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation
Figure 2 for Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation
Figure 3 for Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation
Figure 4 for Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation

To enable safe autonomous vehicle (AV) operations, it is critical that an AV's obstacle detection module can reliably detect obstacles that pose a safety threat (i.e., are safety-critical). It is therefore desirable that the evaluation metric for the perception system captures the safety-criticality of objects. Unfortunately, existing perception evaluation metrics tend to make strong assumptions about the objects and ignore the dynamic interactions between agents, and thus do not accurately capture the safety risks in reality. To address these shortcomings, we introduce an interaction-dynamics-aware obstacle detection evaluation metric by accounting for closed-loop dynamic interactions between an ego vehicle and obstacles in the scene. By borrowing existing theory from optimal control theory, namely Hamilton-Jacobi reachability, we present a computationally tractable method for constructing a ``safety zone'': a region in state space that defines where safety-critical obstacles lie for the purpose of defining safety metrics. Our proposed safety zone is mathematically complete, and can be easily computed to reflect a variety of safety requirements. Using an off-the-shelf detection algorithm from the nuScenes detection challenge leaderboard, we demonstrate that our approach is computationally lightweight, and can better capture safety-critical perception errors than a baseline approach.

* Accepted to Intelligent Vehicles Symposium 2022 
Viaarxiv icon

Robust-RRT: Probabilistically-Complete Motion Planning for Uncertain Nonlinear Systems

May 16, 2022
Albert Wu, Thomas Lew, Kiril Solovey, Edward Schmerling, Marco Pavone

Figure 1 for Robust-RRT: Probabilistically-Complete Motion Planning for Uncertain Nonlinear Systems
Figure 2 for Robust-RRT: Probabilistically-Complete Motion Planning for Uncertain Nonlinear Systems
Figure 3 for Robust-RRT: Probabilistically-Complete Motion Planning for Uncertain Nonlinear Systems
Figure 4 for Robust-RRT: Probabilistically-Complete Motion Planning for Uncertain Nonlinear Systems

Robust motion planning entails computing a global motion plan that is safe under all possible uncertainty realizations, be it in the system dynamics, the robot's initial position, or with respect to external disturbances. Current approaches for robust motion planning either lack theoretical guarantees, or make restrictive assumptions on the system dynamics and uncertainty distributions. In this paper, we address these limitations by proposing the robust rapidly-exploring random-tree (Robust-RRT) algorithm, which integrates forward reachability analysis directly into sampling-based control trajectory synthesis. We prove that Robust-RRT is probabilistically complete (PC) for nonlinear Lipschitz continuous dynamical systems with bounded uncertainty. In other words, Robust-RRT eventually finds a robust motion plan that is feasible under all possible uncertainty realizations assuming such a plan exists. Our analysis applies even to unstable systems that admit only short-horizon feasible plans; this is because we explicitly consider the time evolution of reachable sets along control trajectories. Thanks to the explicit consideration of time dependency in our analysis, PC applies to unstabilizable systems. To the best of our knowledge, this is the most general PC proof for robust sampling-based motion planning, in terms of the types of uncertainties and dynamical systems it can handle. Considering that an exact computation of reachable sets can be computationally expensive for some dynamical systems, we incorporate sampling-based reachability analysis into Robust-RRT and demonstrate our robust planner on nonlinear, underactuated, and hybrid systems.

* 16 pages of main text + 5 pages of appendix, 5 figures, submitted to the 2022 International Symposium on Robotics Research 
Viaarxiv icon

Second-Order Sensitivity Analysis for Bilevel Optimization

May 04, 2022
Robert Dyro, Edward Schmerling, Nikos Arechiga, Marco Pavone

Figure 1 for Second-Order Sensitivity Analysis for Bilevel Optimization
Figure 2 for Second-Order Sensitivity Analysis for Bilevel Optimization
Figure 3 for Second-Order Sensitivity Analysis for Bilevel Optimization
Figure 4 for Second-Order Sensitivity Analysis for Bilevel Optimization

In this work we derive a second-order approach to bilevel optimization, a type of mathematical programming in which the solution to a parameterized optimization problem (the "lower" problem) is itself to be optimized (in the "upper" problem) as a function of the parameters. Many existing approaches to bilevel optimization employ first-order sensitivity analysis, based on the implicit function theorem (IFT), for the lower problem to derive a gradient of the lower problem solution with respect to its parameters; this IFT gradient is then used in a first-order optimization method for the upper problem. This paper extends this sensitivity analysis to provide second-order derivative information of the lower problem (which we call the IFT Hessian), enabling the usage of faster-converging second-order optimization methods at the upper level. Our analysis shows that (i) much of the computation already used to produce the IFT gradient can be reused for the IFT Hessian, (ii) errors bounds derived for the IFT gradient readily apply to the IFT Hessian, (iii) computing IFT Hessians can significantly reduce overall computation by extracting more information from each lower level solve. We corroborate our findings and demonstrate the broad range of applications of our method by applying it to problem instances of least squares hyperparameter auto-tuning, multi-class SVM auto-tuning, and inverse optimal control.

* Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:9166-9181, 2022  
* 16 pages, 6 figures 
Viaarxiv icon

A Unified View of SDP-based Neural Network Verification through Completely Positive Programming

Mar 06, 2022
Robin Brown, Edward Schmerling, Navid Azizan, Marco Pavone

Figure 1 for A Unified View of SDP-based Neural Network Verification through Completely Positive Programming
Figure 2 for A Unified View of SDP-based Neural Network Verification through Completely Positive Programming
Figure 3 for A Unified View of SDP-based Neural Network Verification through Completely Positive Programming
Figure 4 for A Unified View of SDP-based Neural Network Verification through Completely Positive Programming

Verifying that input-output relationships of a neural network conform to prescribed operational specifications is a key enabler towards deploying these networks in safety-critical applications. Semidefinite programming (SDP)-based approaches to Rectified Linear Unit (ReLU) network verification transcribe this problem into an optimization problem, where the accuracy of any such formulation reflects the level of fidelity in how the neural network computation is represented, as well as the relaxations of intractable constraints. While the literature contains much progress on improving the tightness of SDP formulations while maintaining tractability, comparatively little work has been devoted to the other extreme, i.e., how to most accurately capture the original verification problem before SDP relaxation. In this work, we develop an exact, convex formulation of verification as a completely positive program (CPP), and provide analysis showing that our formulation is minimal -- the removal of any constraint fundamentally misrepresents the neural network computation. We leverage our formulation to provide a unifying view of existing approaches, and give insight into the source of large relaxation gaps observed in some cases.

Viaarxiv icon

On the Problem of Reformulating Systems with Uncertain Dynamics as a Stochastic Differential Equation

Nov 11, 2021
Thomas Lew, Apoorva Sharma, James Harrison, Edward Schmerling, Marco Pavone

Figure 1 for On the Problem of Reformulating Systems with Uncertain Dynamics as a Stochastic Differential Equation
Figure 2 for On the Problem of Reformulating Systems with Uncertain Dynamics as a Stochastic Differential Equation

We identify an issue in recent approaches to learning-based control that reformulate systems with uncertain dynamics using a stochastic differential equation. Specifically, we discuss the approximation that replaces a model with fixed but uncertain parameters (a source of epistemic uncertainty) with a model subject to external disturbances modeled as a Brownian motion (corresponding to aleatoric uncertainty).

Viaarxiv icon

Sample-Efficient Safety Assurances using Conformal Prediction

Sep 28, 2021
Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, Marco Pavone

Figure 1 for Sample-Efficient Safety Assurances using Conformal Prediction
Figure 2 for Sample-Efficient Safety Assurances using Conformal Prediction
Figure 3 for Sample-Efficient Safety Assurances using Conformal Prediction

When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; i.e. of the situations that are unsafe, fewer than $\epsilon$ will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an $\epsilon$ false negative rate using as few as $1/\epsilon$ data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate guaranteed false negative rate and low false detection (positive) rate using very little data.

Viaarxiv icon

Towards the Unification and Data-Driven Synthesis of Autonomous Vehicle Safety Concepts

Jul 30, 2021
Andrea Bajcsy, Karen Leung, Edward Schmerling, Marco Pavone

Figure 1 for Towards the Unification and Data-Driven Synthesis of Autonomous Vehicle Safety Concepts
Figure 2 for Towards the Unification and Data-Driven Synthesis of Autonomous Vehicle Safety Concepts

As safety-critical autonomous vehicles (AVs) will soon become pervasive in our society, a number of safety concepts for trusted AV deployment have been recently proposed throughout industry and academia. Yet, agreeing upon an "appropriate" safety concept is still an elusive task. In this paper, we advocate for the use of Hamilton Jacobi (HJ) reachability as a unifying mathematical framework for comparing existing safety concepts, and propose ways to expand its modeling premises in a data-driven fashion. Specifically, we show that (i) existing predominant safety concepts can be embedded in the HJ reachability framework, thereby enabling a common language for comparing and contrasting modeling assumptions, and (ii) HJ reachability can serve as an inductive bias to effectively reason, in a data-driven context, about two critical, yet often overlooked aspects of safety: responsibility and context-dependency.

Viaarxiv icon

On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions

Dec 06, 2020
Karen Leung, Edward Schmerling, Mengxuan Zhang, Mo Chen, John Talbot, J. Christian Gerdes, Marco Pavone

Figure 1 for On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions
Figure 2 for On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions
Figure 3 for On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions
Figure 4 for On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions

Action anticipation, intent prediction, and proactive behavior are all desirable characteristics for autonomous driving policies in interactive scenarios. Paramount, however, is ensuring safety on the road -- a key challenge in doing so is accounting for uncertainty in human driver actions without unduly impacting planner performance. This paper introduces a minimally-interventional safety controller operating within an autonomous vehicle control stack with the role of ensuring collision-free interaction with an externally controlled (e.g., human-driven) counterpart while respecting static obstacles such as a road boundary wall. We leverage reachability analysis to construct a real-time (100Hz) controller that serves the dual role of (i) tracking an input trajectory from a higher-level planning algorithm using model predictive control, and (ii) assuring safety by maintaining the availability of a collision-free escape maneuver as a persistent constraint regardless of whatever future actions the other car takes. A full-scale steer-by-wire platform is used to conduct traffic weaving experiments wherein two cars, initially side-by-side, must swap lanes in a limited amount of time and distance, emulating cars merging onto/off of a highway. We demonstrate that, with our control stack, the autonomous vehicle is able to avoid collision even when the other car defies the planner's expectations and takes dangerous actions, either carelessly or with the intent to collide, and otherwise deviates minimally from the planned trajectory to the extent required to maintain safety.

* International Journal of Robotics Research, vol. 39, no. 10-11, pp. 1326--1345, 2020  
* arXiv admin note: text overlap with arXiv:1812.11315 
Viaarxiv icon