A Survey on Reinforcement Learning in Aviation Applications

Compared with model-based control and optimization methods, reinforcement learning (RL) provides a data-driven, learning-based framework to formulate and solve sequential decision-making problems. The RL framework has become promising due to largely improved data availability and computing power in the aviation industry. Many aviation-based applications can be formulated or treated as sequential decision-making problems. Some of them are offline planning problems, while others need to be solved online and are safety-critical. In this survey paper, we first describe standard RL formulations and solutions. Then we survey the landscape of existing RL-based applications in aviation. Finally, we summarize the paper, identify the technical gaps, and suggest future directions of RL research in aviation.

A Verification Framework for Certifying Learning-Based Safety-Critical Aviation Systems

We present a safety verification framework for design-time and run-time assurance of learning-based components in aviation systems. Our proposed framework integrates two novel methodologies. From the design-time assurance perspective, we propose offline mixed-fidelity verification tools that incorporate knowledge from different levels of granularity in simulated environments. From the run-time assurance perspective, we propose reachability- and statistics-based online monitoring and safety guards for a learning-based decision-making model to complement the offline verification methods. This framework is designed to be loosely coupled among modules, allowing the individual modules to be developed using independent methodologies and techniques, under varying circumstances and with different tool access. The proposed framework offers feasible solutions for meeting system safety requirements at different stages throughout the system development and deployment cycle, enabling the continuous learning and assessment of the system product.

Feedback Control of Millimeter Scale Pivot Walkers Using Magnetic Actuation

An external magnetic field can be used to remotely control small-scaled robots, making them promising candidates for diverse biomedical and engineering applications. We showed that our magnetically actuated millirobot is highly agile and can perform a variety of locomotive tasks such as pivot walking and tumbling in a horizontal plane. Here, we focus on controlling the locomotion outcomes of this millirobot in the pivot walking mode. A mathematical model of the system is developed and the kinematic model is derived. The role of the sweep and tilt angles in the robot's motion is also investigated. We propose two controllers to regulate the gait of the pivot walker. The first one is a proportional-geometric controller, which determines the correct pivot point that the millirobot should use. Then, it regulates the angular velocity proportionally based on the error between the center of the millirobot and the reference trajectory. The second controller is based on a gradient descent optimization technique, which expresses the control action as an optimization problem. These control algorithms enable the millirobot to generate a stable gait while tracking the desired trajectory. We conduct a set of different experiments and simulation runs to establish the effectiveness of proposed controllers for different sweep and tilt angles in terms of the tracking error. The two controllers exhibit an appropriate performance, but it is observed that gradient descent based controller yields faster convergence time, smaller tracking error, and fewer number of steps. Finally, we perform an extensive experimentally parametric analysis on the effect of the sweep angle, tilt angle, and step time on the tracking error. As we expect, the optimization-based controller outperforms the geometric based controller.

Swarm Control of Magnetically Actuated Millirobots

Small-size robots offer access to spaces that are inaccessible to larger ones. This type of access is crucial in applications such as drug delivery, environmental detection, and collection of small samples. However, there are some tasks that are not possible to perform using only one robot including assembly and manufacturing at small scales, manipulation of micro- and nano- objects, and robot-based structuring of small-scale materials. The solution to this problem is to use a group of robots as a system. Thus, we focus on tasks that can be achieved using a group of small-scale robots. These robots are typically externally actuated due to their size limitation. Yet, one faces the challenge of controlling a group of robots using a single global input. We propose a control algorithm to position individual members of a swarm in predefined positions. A single control input applies to the system and moves all robots in the same direction. We also add another control modality by using different length robots. An electromagnetic coil system applied external force and steered the millirobots. This millirobot can move in various modes of motion such as pivot walking and tumbling. We propose two new designs of these millirobots. In the first design, the magnets are placed at the center of body to reduce the magnetic attraction force. In the second design, the millirobots are of identical length with two extra legs acting as the pivot points. This way we vary pivot separation in design to take advantage of variable speed in pivot walking mode while keeping the speed constant in tumbling mode. This paper presents a general algorithm for positional control of n millirobots with different lengths to move them from given initial positions to final desired ones. This method is based on choosing a leader that is fully controllable. Simulations and hardware experiments validate these results.