Alert button
Picture for Yi-Ling Qiao

Yi-Ling Qiao

Alert button

Dynamic Mesh-Aware Radiance Fields

Sep 08, 2023
Yi-Ling Qiao, Alexander Gao, Yiran Xu, Yue Feng, Jia-Bin Huang, Ming C. Lin

Figure 1 for Dynamic Mesh-Aware Radiance Fields
Figure 2 for Dynamic Mesh-Aware Radiance Fields
Figure 3 for Dynamic Mesh-Aware Radiance Fields
Figure 4 for Dynamic Mesh-Aware Radiance Fields

Embedding polygonal mesh assets within photorealistic Neural Radience Fields (NeRF) volumes, such that they can be rendered and their dynamics simulated in a physically consistent manner with the NeRF, is under-explored from the system perspective of integrating NeRF into the traditional graphics pipeline. This paper designs a two-way coupling between mesh and NeRF during rendering and simulation. We first review the light transport equations for both mesh and NeRF, then distill them into an efficient algorithm for updating radiance and throughput along a cast ray with an arbitrary number of bounces. To resolve the discrepancy between the linear color space that the path tracer assumes and the sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range (HDR) images. We also present a strategy to estimate light sources and cast shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric formulation can be efficiently integrated with a high-performance physics simulator that supports cloth, rigid and soft bodies. The full rendering and simulation system can be run on a GPU at interactive rates. We show that a hybrid system approach outperforms alternatives in visual realism for mesh insertion, because it allows realistic light transport from volumetric NeRF media onto surfaces, which affects the appearance of reflective/refractive surfaces and illumination of diffuse surfaces informed by the dynamic scene.

* ICCV 2023 
Viaarxiv icon

Towards A Foundation Model for Generalist Robots: Diverse Skill Learning at Scale via Automated Task and Scene Generation

May 17, 2023
Zhou Xian, Theophile Gervet, Zhenjia Xu, Yi-Ling Qiao, Tsun-Hsuan Wang

Figure 1 for Towards A Foundation Model for Generalist Robots: Diverse Skill Learning at Scale via Automated Task and Scene Generation
Figure 2 for Towards A Foundation Model for Generalist Robots: Diverse Skill Learning at Scale via Automated Task and Scene Generation

This document serves as a position paper that outlines the authors' vision for a potential pathway towards generalist robots. The purpose of this document is to share the excitement of the authors with the community and highlight a promising research direction in robotics and AI. The authors believe the proposed paradigm is a feasible path towards accomplishing the long-standing goal of robotics research: deploying robots, or embodied AI agents more broadly, in various non-factory real-world settings to perform diverse tasks. This document presents a specific idea for mining knowledge in the latest large-scale foundation models for robotics research. Instead of directly adapting these models or using them to guide low-level policy learning, it advocates for using them to generate diversified tasks and scenes at scale, thereby scaling up low-level skill learning and ultimately leading to a foundation model for robotics that empowers generalist robots. The authors are actively pursuing this direction, but in the meantime, they recognize that the ambitious goal of building generalist robots with large-scale policy training demands significant resources such as computing power and hardware, and research groups in academia alone may face severe resource constraints in implementing the entire vision. Therefore, the authors believe sharing their thoughts at this early stage could foster discussions, attract interest towards the proposed pathway and related topics from industry groups, and potentially spur significant technical advancements in the field.

Viaarxiv icon

PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

Mar 09, 2023
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, Chuang Gan

Figure 1 for PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Figure 2 for PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Figure 3 for PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Figure 4 for PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

Existing approaches to system identification (estimating the physical parameters of an object) from videos assume known object geometries. This precludes their applicability in a vast majority of scenes where object geometries are complex or unknown. In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology. To this end, we propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. We design PAC-NeRF to only ever produce physically plausible states by enforcing the neural radiance field to follow the conservation laws of continuum mechanics. For this, we design a hybrid Eulerian-Lagrangian representation of the neural radiance field, i.e., we use the Eulerian grid representation for NeRF density and color fields, while advecting the neural radiance fields via Lagrangian particles. This hybrid Eulerian-Lagrangian representation seamlessly blends efficient neural rendering with the material point method (MPM) for robust differentiable physics simulation. We validate the effectiveness of our proposed framework on geometry and physical parameter estimation over a vast range of materials, including elastic bodies, plasticine, sand, Newtonian and non-Newtonian fluids, and demonstrate significant performance gain on most tasks.

* ICLR 2023 Spotlight. Project page: https://sites.google.com/view/PAC-NeRF 
Viaarxiv icon

Differentiable Analog Quantum Computing for Optimization and Control

Oct 28, 2022
Jiaqi Leng, Yuxiang Peng, Yi-Ling Qiao, Ming Lin, Xiaodi Wu

Figure 1 for Differentiable Analog Quantum Computing for Optimization and Control
Figure 2 for Differentiable Analog Quantum Computing for Optimization and Control
Figure 3 for Differentiable Analog Quantum Computing for Optimization and Control
Figure 4 for Differentiable Analog Quantum Computing for Optimization and Control

We formulate the first differentiable analog quantum computing framework with a specific parameterization design at the analog signal (pulse) level to better exploit near-term quantum devices via variational methods. We further propose a scalable approach to estimate the gradients of quantum dynamics using a forward pass with Monte Carlo sampling, which leads to a quantum stochastic gradient descent algorithm for scalable gradient-based training in our framework. Applying our framework to quantum optimization and control, we observe a significant advantage of differentiable analog quantum computing against SOTAs based on parameterized digital quantum circuits by orders of magnitude.

* In the Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022)  
* Code available at https://github.com/YilingQiao/diffquantum 
Viaarxiv icon

Differentiable Hybrid Traffic Simulation

Oct 14, 2022
Sanghyun Son, Yi-Ling Qiao, Jason Sewall, Ming C. Lin

Figure 1 for Differentiable Hybrid Traffic Simulation
Figure 2 for Differentiable Hybrid Traffic Simulation
Figure 3 for Differentiable Hybrid Traffic Simulation
Figure 4 for Differentiable Hybrid Traffic Simulation

We introduce a novel differentiable hybrid traffic simulator, which simulates traffic using a hybrid model of both macroscopic and microscopic models and can be directly integrated into a neural network for traffic control and flow optimization. This is the first differentiable traffic simulator for macroscopic and hybrid models that can compute gradients for traffic states across time steps and inhomogeneous lanes. To compute the gradient flow between two types of traffic models in a hybrid framework, we present a novel intermediate conversion component that bridges the lanes in a differentiable manner as well. We also show that we can use analytical gradients to accelerate the overall process and enhance scalability. Thanks to these gradients, our simulator can provide more efficient and scalable solutions for complex learning and control problems posed in traffic engineering than other existing algorithms. Refer to https://sites.google.com/umd.edu/diff-hybrid-traffic-sim for our project.

* 13 pages, Siggraph Asia 2022 Journal Paper 
Viaarxiv icon

Differentiable Simulation of Soft Multi-body Systems

May 03, 2022
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin

Figure 1 for Differentiable Simulation of Soft Multi-body Systems
Figure 2 for Differentiable Simulation of Soft Multi-body Systems
Figure 3 for Differentiable Simulation of Soft Multi-body Systems
Figure 4 for Differentiable Simulation of Soft Multi-body Systems

We present a method for differentiable simulation of soft articulated bodies. Our work enables the integration of differentiable physical dynamics into gradient-based pipelines. We develop a top-down matrix assembly algorithm within Projective Dynamics and derive a generalized dry friction model for soft continuum using a new matrix splitting strategy. We derive a differentiable control framework for soft articulated bodies driven by muscles, joint torques, or pneumatic tubes. The experiments demonstrate that our designs make soft body simulation more stable and realistic compared to other frameworks. Our method accelerates the solution of system identification problems by more than an order of magnitude, and enables efficient gradient-based learning of motion control with soft robots.

* NeurIPS 2021 
Viaarxiv icon

Efficient Differentiable Simulation of Articulated Bodies

Sep 16, 2021
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin

Figure 1 for Efficient Differentiable Simulation of Articulated Bodies
Figure 2 for Efficient Differentiable Simulation of Articulated Bodies
Figure 3 for Efficient Differentiable Simulation of Articulated Bodies
Figure 4 for Efficient Differentiable Simulation of Articulated Bodies

We present a method for efficient differentiable simulation of articulated bodies. This enables integration of articulated body dynamics into deep learning frameworks, and gradient-based optimization of neural networks that operate on articulated bodies. We derive the gradients of the forward dynamics using spatial algebra and the adjoint method. Our approach is an order of magnitude faster than autodiff tools. By only saving the initial states throughout the simulation process, our method reduces memory requirements by two orders of magnitude. We demonstrate the utility of efficient differentiable dynamics for articulated bodies in a variety of applications. We show that reinforcement learning with articulated systems can be accelerated using gradients provided by our method. In applications to control and inverse problems, gradient-based optimization enabled by our work accelerates convergence by more than an order of magnitude.

* ICML 2021 
Viaarxiv icon

Scalable Differentiable Physics for Learning and Control

Jul 04, 2020
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin

Figure 1 for Scalable Differentiable Physics for Learning and Control
Figure 2 for Scalable Differentiable Physics for Learning and Control
Figure 3 for Scalable Differentiable Physics for Learning and Control
Figure 4 for Scalable Differentiable Physics for Learning and Control

Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments. While notable progress has been made, the capabilities of differentiable physics solvers remain limited. We develop a scalable framework for differentiable physics that can support a large number of objects and their interactions. To accommodate objects with arbitrary geometry and topology, we adopt meshes as our representation and leverage the sparsity of contacts for scalable differentiable collision handling. Collisions are resolved in localized regions to minimize the number of optimization variables even when the number of simulated objects is high. We further accelerate implicit differentiation of optimization with nonlinear constraints. Experiments demonstrate that the presented framework requires up to two orders of magnitude less memory and computation in comparison to recent particle-based methods. We further validate the approach on inverse problems and control scenarios, where it outperforms derivative-free and model-free baselines by at least an order of magnitude.

* Proceedings of the 37th International Conference on Machine Learning, ICML 2020  
Viaarxiv icon

OF-VO: Reliable Navigation among Pedestrians Using Commodity Sensors

Apr 23, 2020
Jing Liang, Yi-Ling Qiao, Dinesh Manocha

Figure 1 for OF-VO: Reliable Navigation among Pedestrians Using Commodity Sensors
Figure 2 for OF-VO: Reliable Navigation among Pedestrians Using Commodity Sensors
Figure 3 for OF-VO: Reliable Navigation among Pedestrians Using Commodity Sensors
Figure 4 for OF-VO: Reliable Navigation among Pedestrians Using Commodity Sensors

We present a novel algorithm for safe navigation of a mobile robot among pedestrians. Our approach uses commodity visual sensors, including RGB-D cameras and a 2D lidar, for explicitly predicting the velocities and positions of surrounding obstacles through optical flow estimation and object detection. Given these partial observations of the environment, we present a modified velocity-obstacle (VO) algorithm to compute collision-free trajectories for the robot. A key aspect of our work is the coupling between the perception (OF: optical flow) and planning (VO) components for reliable navigation. Overall, our OF-VO algorithm is a hybrid combination of learning-based and model-based methods and offers better performance over prior algorithms in terms of navigation time and success rate of collision avoidance. We highlight the realtime performance of OF-VO in simulated and real-world dynamic scenes on a Turtlebot robot navigating among pedestrians with commodity sensors. A demo video is available at \url{https://youtu.be/lbrBIZRAxBs}

* 9 pages 
Viaarxiv icon

Learning-based Real-time Detection of Intrinsic Reflectional Symmetry

Nov 01, 2019
Yi-Ling Qiao, Lin Gao, Shu-Zhi Liu, Ligang Liu, Yu-Kun Lai, Xilin Chen

Figure 1 for Learning-based Real-time Detection of Intrinsic Reflectional Symmetry
Figure 2 for Learning-based Real-time Detection of Intrinsic Reflectional Symmetry
Figure 3 for Learning-based Real-time Detection of Intrinsic Reflectional Symmetry
Figure 4 for Learning-based Real-time Detection of Intrinsic Reflectional Symmetry

Reflectional symmetry is ubiquitous in nature. While extrinsic reflectional symmetry can be easily parametrized and detected, intrinsic symmetry is much harder due to the high solution space. Previous works usually solve this problem by voting or sampling, which suffer from high computational cost and randomness. In this paper, we propose \YL{a} learning-based approach to intrinsic reflectional symmetry detection. Instead of directly finding symmetric point pairs, we parametrize this self-isometry using a functional map matrix, which can be easily computed given the signs of Laplacian eigenfunctions under the symmetric mapping. Therefore, we train a novel deep neural network to predict the sign of each eigenfunction under symmetry, which in addition takes the first few eigenfunctions as intrinsic features to characterize the mesh while avoiding coping with the connectivity explicitly. Our network aims at learning the global property of functions, and consequently converts the problem defined on the manifold to the functional domain. By disentangling the prediction of the matrix into separated basis, our method generalizes well to new shapes and is invariant under perturbation of eigenfunctions. Through extensive experiments, we demonstrate the robustness of our method in challenging cases, including different topology and incomplete shapes with holes. By avoiding random sampling, our learning-based algorithm is over 100 times faster than state-of-the-art methods, and meanwhile, is more robust, achieving higher correspondence accuracy in commonly used metrics.

Viaarxiv icon