Toward Scalable Visual Servoing Using Deep Reinforcement Learning and Optimal Control

Classical pixel-based Visual Servoing (VS) approaches offer high accuracy but suffer from a limited convergence area due to optimization nonlinearity. Modern deep learning-based VS methods overcome traditional vision issues but lack scalability, requiring training on limited scenes. This paper proposes a hybrid VS strategy utilizing Deep Reinforcement Learning (DRL) and optimal control to enhance both convergence area and scalability. The DRL component of our approach separately handles representation and policy learning to enhance scalability, generalizability, learning efficiency and ease domain adaptation. Moreover, the optimal control part ensures high end-point accuracy. Our method showcases remarkable achievements in terms of high convergence rates and minimal end-positioning errors using a 7-DOF manipulator. Importantly, it exhibits scalability across more than 1000 distinct scenes. Furthermore, we demonstrate its capacity for generalization to previously unseen datasets. Lastly, we illustrate the real-world applicability of our approach, highlighting its adaptability through single-shot domain transfer learning in environments with noise and occlusions. Real-robot experiments can be found at \url{https://sites.google.com/view/vsls}.

A Non-linear MPC Local Planner for Tractor-Trailer Vehicles in Forward and Backward Maneuvering

Designing a local planner to control tractor-trailer vehicles in forward and backward maneuvering is a challenging control problem in the research community of autonomous driving systems. Considering a critical situation in the stability of tractor-trailer systems, a practical and novel approach is presented to design a non-linear MPC(NMPC) local planner for tractor-trailer autonomous vehicles in both forward and backward maneuvering. The tractor velocity and steering angle are considered to be control variables. The proposed NMPC local planner is designed to handle jackknife situations, avoiding multiple static obstacles, and path following in both forward and backward maneuvering. The challenges mentioned above are converted into a constrained problem that can be handled simultaneously by the proposed NMPC local planner. The direct multiple shooting approach is used to convert the optimal control problem(OCP) into a non-linear programming problem(NLP) that IPOPT solvers can solve in CasADi. The controller performance is evaluated through different backup and forward maneuvering scenarios in the Gazebo simulation environment in real-time. It achieves asymptotic stability in avoiding static obstacles and accurate tracking performance while respecting path constraints. Finally, the proposed NMPC local planner is integrated with an open-source autonomous driving software stack called AutowareAi.

Robot to Human Object Handover using Vision and Joint Torque Sensor Modalities

We present a robot-to-human object handover algorithm and implement it on a 7-DOF arm equipped with a 3-finger mechanical hand. The system performs a fully autonomous and robust object handover to a human receiver in real-time. Our algorithm relies on two complementary sensor modalities: joint torque sensors on the arm and an eye-in-hand RGB-D camera for sensor feedback. Our approach is entirely implicit, i.e., there is no explicit communication between the robot and the human receiver. Information obtained via the aforementioned sensor modalities is used as inputs to their related deep neural networks. While the torque sensor network detects the human receiver's "intention" such as: pull, hold, or bump, the vision sensor network detects if the receiver's fingers have wrapped around the object. Networks' outputs are then fused, based on which a decision is made to either release the object or not. Despite substantive challenges in sensor feedback synchronization, object, and human hand detection, our system achieves robust robot-to-human handover with 98\% accuracy in our preliminary real experiments using human receivers.

Least-Restrictive Multi-Agent Collision Avoidance via Deep Meta Reinforcement Learning and Optimal Control

Multi-agent collision-free trajectory planning and control subject to different goal requirements and system dynamics has been extensively studied, and is gaining recent attention in the realm of machine and reinforcement learning. However, in particular when using a large number of agents, constructing a least-restrictive collision avoidance policy is of utmost importance for both classical and learning-based methods. In this paper, we propose a Least-Restrictive Collision Avoidance Module (LR-CAM) that evaluates the safety of multi-agent systems and takes over control only when needed to prevent collisions. The LR-CAM is a single policy that can be wrapped around policies of all agents in a multi-agent system. It allows each agent to pursue any objective as long as it is safe to do so. The benefit of the proposed least-restrictive policy is to only interrupt and overrule the default controller in case of an upcoming inevitable danger. We use a Long Short-Term Memory (LSTM) based Variational Auto-Encoder (VAE) to enable the LR-CAM to account for a varying number of agents in the environment. Moreover, we propose an off-policy meta-reinforcement learning framework with a novel reward function based on a Hamilton-Jacobi value function to train the LR-CAM. The proposed method is fully meta-trained through a ROS based simulation and tested on real multi-agent system. Our results show that LR-CAM outperforms the classical least-restrictive baseline by 30 percent. In addition, we show that even if a subset of agents in a multi-agent system use LR-CAM, the success rate of all agents will increase significantly.

Generalized Grasping for Mechanical Grippers for Unknown Objects with Partial Point Cloud Representations

We present a generalized grasping algorithm that uses point clouds (i.e. a group of points and their respective surface normals) to discover grasp pose solutions for multiple grasp types, executed by a mechanical gripper, in near real-time. The algorithm introduces two ideas: 1) a histogram of finger contact normals is used to represent a grasp 'shape' to guide a gripper orientation search in a histogram of object(s) surface normals, and 2) voxel grid representations of gripper and object(s) are cross-correlated to match finger contact points, i.e. grasp 'size', to discover a grasp pose. Constraints, such as collisions with neighbouring objects, are optionally incorporated in the cross-correlation computation. We show via simulations and experiments that 1) grasp poses for three grasp types can be found in near real-time, 2) grasp pose solutions are consistent with respect to voxel resolution changes for both partial and complete point cloud scans, and 3) a planned grasp is executed with a mechanical gripper.

Identifying Multiple Interaction Events from Tactile Data during Robot-Human Object Transfer

During a robot to human object handover task, several intended or unintended events may occur with the object - it may be pulled, pushed, bumped or simply held - by the human receiver. We show that it is possible to differentiate between these events solely via tactile sensors. Training data from tactile sensors were recorded during interaction of human subjects with the object held by a 3-finger robotic hand. A Bag of Words approach was used to automatically extract effective features from the tactile data. A Support Vector Machine was used to distinguish between the four events with over 95 percent average accuracy.

Path Planning and Controlled Crash Landing of a Quadcopter in case of a Rotor Failure

This paper presents a framework for controlled emergency landing of a quadcopter, experiencing a rotor failure, away from sensitive areas. A complete mathematical model capturing the dynamics of the system is presented that takes the asymmetrical aerodynamic load on the propellers into account. An equilibrium state of the system is calculated around which a linear time-invariant control strategy is developed to stabilize the system. By utilizing the proposed model, a specific configuration for a quadcopter is introduced that leads to the minimum power consumption during a yaw-rate-resolved hovering after a rotor failure. Furthermore, given a 3D representation of the environment, an optimal flight trajectory towards a safe crash landing spot, while avoiding collision with obstacles, is developed using an RRT* approach. The cost function for determining the best landing spot consists of: (i) finding the safest landing spot with the largest clearance from the obstacles; and (ii) finding the most energy-efficient trajectory towards the landing spot. The performance of the proposed framework is tested via simulations.

Optimal-power Configurations for Hover Solutions in Mono-spinners

Rotary-wing flying machines draw attention within the UAV community for their in-place hovering capability, and recently, holonomic motion over fixed-wings. However, they still fall behind in terms of the flight time, power consumption, and safety for centrally-powered and redundant multi-copters have been suggested and studied. In this paper, we investigate about the power-optimality in a mono-spinner, i.e., a class of rotary-wing UAVs with one rotor only, whose main body has a streamlined shape for producing additional lift when counter-spinning the rotor. We provide a detailed dynamic model of our mono-spinner. Two configurations are studied: (1) a symmetric configuration, in which the rotor is aligned with the fuselage's COM, and (2) an asymmetric configuration, in which the rotor is located with an offset from the fuselage's COM. While the former can generate an in-place hovering flight condition, the latter can achieve trajectory tacking in 3D space by resolving the yaw and precession rates. Furthermore, it is shown that by introducing a tilting angle between the rotor and the fuselage, within the asymmetric design, one can further minimize the power consumption without compromising the overall stability. It is shown that an energy optimal solution can be achieved through proper aerodynamic design of the mono-spinner for the first time.

A Unified Approach to Configuration-based Dynamic Analysis of Quadcopters for Optimal Stability

A special type of rotary-wing Unmanned Aerial Vehicles (UAV), called Quadcopter have prevailed to the civilian use for the past decade. They have gained significant amount of attention within the UAV community for their redundancy and ease of control, despite the fact that they fall under an under-actuated system category. They come in a variety of configurations. The "+" and "x" configurations were introduced first. Literature pertinent to these two configurations is vast. However, in this paper, we define 6 additional possible configurations for a Quadcopter that can be built under either "+" or "x" setup. These configurations can be achieved by changing the angle that the axis of rotation for rotors make with the main body, i.e., fuselage. This would also change the location of the COM with respect to the propellers which can add to the overall stability. A comprehensive dynamic model for all these configurations is developed for the first time. The overall stability for these configurations are addressed. In particular, it is shown that one configuration can lead to the most statically-stable platform by adopting damping motion in Roll/Pitch/Yaw, which is described for the first time to the best of our knowledge.