Cerberus: Low-Drift Visual-Inertial-Leg Odometry For Agile Locomotion

We present an open-source Visual-Inertial-Leg Odometry (VILO) state estimation solution, Cerberus, for legged robots that estimates position precisely on various terrains in real time using a set of standard sensors, including stereo cameras, IMU, joint encoders, and contact sensors. In addition to estimating robot states, we also perform online kinematic parameter calibration and contact outlier rejection to substantially reduce position drift. Hardware experiments in various indoor and outdoor environments validate that calibrating kinematic parameters within the Cerberus can reduce estimation drift to lower than 1% during long distance high speed locomotion. Our drift results are better than any other state estimation method using the same set of sensors reported in the literature. Moreover, our state estimator performs well even when the robot is experiencing large impacts and camera occlusion. The implementation of the state estimator, along with the datasets used to compute our results, are available at https://github.com/ShuoYangRobotics/Cerberus.

Differentiable Collision Detection for a Set of Convex Primitives

Collision detection between objects is critical for simulation, control, and learning for robotic systems. However, existing collision detection routines are inherently non-differentiable, limiting their usefulness in optimization-based algorithms. In this work, we propose a fully differentiable collision-detection framework that reasons about distances between a set of composable and highly expressive convex primitive shapes. This is achieved by formulating the collision detection problem as a convex optimization problem that seeks to find the minimum uniform scaling to be applied to each object before there is an intersection. The optimization problem is fully differentiable and is able to return both the collision detection status as well as the contact points on each object.

DiffPills: Differentiable Collision Detection for Capsules and Padded Polygons

Collision detection plays an important role in simulation, control, and learning for robotic systems. However, no existing method is differentiable with respect to the configurations of the objects, greatly limiting the sort of algorithms that can be built on top of collision detection. In this work, we propose a set of differentiable collision detection algorithms between capsules and padded polygons by formulating these problems as differentiable convex quadratic programs. The resulting algorithms are able to return a proximity value indicating if a collision has taken place, as well as the closest points between objects, all of which are differentiable. As a result, they can be used reliably within other gradient-based optimization methods, including trajectory optimization, state estimation, and reinforcement learning methods.

Low-latency Imaging and Inference from LoRa-enabled CubeSats

Recent years have seen the rapid deployment of low-cost CubeSats in low-Earth orbit, primarily for research, education, and Earth observation. The vast majority of these CubeSats experience significant latency (several hours) from the time an image is captured to the time it is available on the ground. This is primarily due to the limited availability of dedicated satellite ground stations that tend to be bulky to deploy and expensive to rent. This paper explores using LoRa radios in the ISM band for low-latency downlink communication from CubeSats, primarily due to the availability of extensive ground LoRa infrastructure and minimal interference to terrestrial communication. However, the limited bandwidth of LoRa precludes rich satellite Earth images to be sent - instead, the CubeSats can at best send short messages (a few hundred bytes). This paper details our experience in communicating with a LoRa-enabled CubeSat launched by our team. We present Vista, a communication system that makes software modifications to LoRa encoding onboard a CubeSat and decoding on commercial LoRa ground stations to allow for satellite imagery to be communicated, as well as wide-ranging machine learning inference on these images. This is achieved through a LoRa-channel-aware image encoding that is informed by the structure of satellite images, the tasks performed on it, as well as the Doppler variation of satellite signals. A detailed evaluation of Vista through trace-driven emulation with traces from the LoRa-CubeSat launch (in 2021) shows 4.56 dB improvement in LoRa image PSNR and 1.38x improvement in land-use classification over those images.

CALIPSO: A Differentiable Solver for Trajectory Optimization with Conic and Complementarity Constraints

We present a new solver for non-convex trajectory optimization problems that is specialized for robotics applications. CALIPSO, or the Conic Augmented Lagrangian Interior-Point SOlver, combines several strategies for constrained numerical optimization to natively handle second-order cones and complementarity constraints. It reliably solves challenging motion-planning problems that include contact-implicit formulations of impacts and Coulomb friction, thrust limits subject to conic constraints, and state-triggered constraints where general-purpose nonlinear programming solvers like SNOPT and Ipopt fail to converge. Additionally, CALIPSO supports efficient differentiation of solutions with respect to problem data, enabling bi-level optimization applications like auto-tuning of feedback policies. Reliable convergence of the solver is demonstrated on a range of problems from manipulation, locomotion, and aerospace domains. An open-source implementation of this solver is available.

Dojo: A Differentiable Simulator for Robotics

We present a differentiable rigid-body-dynamics simulator for robotics that prioritizes physical accuracy and differentiability: Dojo. The simulator utilizes an expressive maximal-coordinates representation, achieves stable simulation at low sample rates, and conserves energy and momentum by employing a variational integrator. A nonlinear complementarity problem, with nonlinear friction cones, models hard contact and is reliably solved using a custom primal-dual interior-point method. The implicit-function theorem enables efficient differentiation of an intermediate relaxed problem and computes smooth gradients from the contact model. We demonstrate the usefulness of the simulator and its gradients through a number of examples including: simulation, trajectory optimization, reinforcement learning, and system identification.

Joint inference and input optimization in equilibrium networks

Many tasks in deep learning involve optimizing over the \emph{inputs} to a network to minimize or maximize some objective; examples include optimization over latent spaces in a generative model to match a target image, or adversarially perturbing an input to worsen classifier performance. Performing such optimization, however, is traditionally quite costly, as it involves a complete forward and backward pass through the network for each gradient step. In a separate line of work, a recent thread of research has developed the deep equilibrium (DEQ) model, a class of models that foregoes traditional network depth and instead computes the output of a network by finding the fixed point of a single nonlinear layer. In this paper, we show that there is a natural synergy between these two settings. Although, naively using DEQs for these optimization problems is expensive (owing to the time needed to compute a fixed point for each gradient step), we can leverage the fact that gradient-based optimization can \emph{itself} be cast as a fixed point iteration to substantially improve the overall speed. That is, we \emph{simultaneously} both solve for the DEQ fixed point \emph{and} optimize over network inputs, all within a single ``augmented'' DEQ model that jointly encodes both the original network and the optimization process. Indeed, the procedure is fast enough that it allows us to efficiently \emph{train} DEQ models for tasks traditionally relying on an ``inner'' optimization loop. We demonstrate this strategy on various tasks such as training generative models while optimizing over latent codes, training models for inverse problems like denoising and inpainting, adversarial training and gradient based meta-learning.

Trajectory Optimization with Optimization-Based Dynamics

We present a framework for bi-level trajectory optimization in which a system's dynamics are encoded as the solution to a constrained optimization problem and smooth gradients of this lower-level problem are passed to an upper-level trajectory optimizer. This optimization-based dynamics representation enables constraint handling, additional variables, and non-smooth forces to be abstracted away from the upper-level optimizer, and allows classical unconstrained optimizers to synthesize trajectories for more complex systems. We provide a path-following method for efficient evaluation of constrained dynamics and utilize the implicit-function theorem to compute smooth gradients of this representation. We demonstrate the framework by modeling systems from locomotion, aerospace, and manipulation domains including: acrobot with joint limits, cart-pole subject to Coulomb friction, Raibert hopper, rocket landing with thrust limits, and planar-push task with optimization-based dynamics and then optimize trajectories using iterative LQR.

Linear Contact-Implicit Model-Predictive Control

We present a general approach for controlling robotic systems that make and break contact with their environments: linear contact-implicit model-predictive control (LCI-MPC). Our use of differentiable contact dynamics provides a natural extension of linear model-predictive control to contact-rich settings. The policy leverages precomputed linearizations about a reference state or trajectory while contact modes, encoded via complementarity constraints, are explicitly retained, resulting in policies that can be efficiently evaluated while maintaining robustness to changes in contact timings. In many cases, the algorithm is even capable of generating entirely new contact sequences. To enable real-time performance, we devise a custom structure-exploiting linear solver for the contact dynamics. We demonstrate that the policy can respond to disturbances by discovering and exploiting new contact modes and is robust to model mismatch and unmodeled environments for a collection of simulated robotic systems, including: pushbot, hopper, quadruped, and biped.

Training Structured Mechanical Models by Minimizing Discrete Euler-Lagrange Residual

Model-based paradigms for decision-making and control are becoming ubiquitous in robotics. They rely on the ability to efficiently learn a model of the system from data. Structured Mechanical Models (SMMs) are a data-efficient black-box parameterization of mechanical systems, typically fit to data by minimizing the error between predicted and observed accelerations or next states. In this work, we propose a methodology for fitting SMMs to data by minimizing the discrete Euler-Lagrange residual. To study our methodology, we fit models to joint-angle time-series from undamped and damped double-pendulums, studying the quality of learned models fit to data with and without observation noise. Experiments show that our methodology learns models that are better in accuracy to those of the conventional schemes for fitting SMMs. We identify use cases in which our method is a more appropriate methodology. Source code for reproducing the experiments is available at https://github.com/sisl/delsmm.