Learning a Sequential Policy of Efficient Actions for Tangled-Prone Parts in Robotic Bin Picking

This paper introduces an autonomous bin picking system for cable harnesses - an extremely challenging object in bin picking task. Currently cable harnesses are unsuitable to be imported to automated production due to their length and elusive structures. Considering the task of robotic bin picking where the harnesses are heavily entangled, it is challenging for a robot to pick harnesses up one by one using conventional bin picking methods. In this paper, we present an efficient approach to overcoming the difficulties when dealing with entangled-prone parts. We develop several motion schemes for the robot to pick up a single harness avoiding any entanglement. Moreover, we proposed a learning-based bin picking policy to select both grasps and designed motion schemes in a reasonable sequence. Our method is unique due to the novelty for sufficiently solving the entanglement problem in picking cluttered cable harnesses. We demonstrate our approach on a set of real-world experiments, during which the proposed method is capable to perform the sequential bin picking task with both effectiveness and accuracy under a variety of cluttered scenarios.

A Topological Solution of Entanglement for Complex-shaped Parts in Robotic Bin-picking

This paper addresses the problem of picking up only one object at a time avoiding any entanglement in bin-picking. To cope with a difficult case where the complex-shaped objects are heavily entangled together, we propose a topology-based method that can generate non-tangle grasp positions on a single depth image. The core technique is entanglement map, which is a feature map to measure the entanglement possibilities obtained from the input image. We use the entanglement map to select probable regions containing graspable objects. The optimum grasping pose is detected from the selected regions considering the collision between robot hand and objects. Experimental results show that our analytic method provides a more comprehensive and intuitive observation of entanglement and exceeds previous learning-based work in success rates. Especially, our topology-based method does not rely on any object models or time-consuming training process, so that it can be easily adapted to more complex bin-picking scenes.

Assembly Planning by Recognizing a Graphical Instruction Manual

This paper proposes a robot assembly planning method by automatically reading the graphical instruction manuals design for humans. Essentially, the method generates an Assembly Task Sequence Graph (ATSG) by recognizing a graphical instruction manual. An ATSG is a graph describing the assembly task procedure by detecting types of parts included in the instruction images, completing the missing information automatically, and correcting the detection errors automatically. To build an ATSG, the proposed method first extracts the information of the parts contained in each image of the graphical instruction manual. Then, by using the extracted part information, it estimates the proper work motions and tools for the assembly task. After that, the method builds an ATSG by considering the relationship between the previous and following images, which makes it possible to estimate the undetected parts caused by occlusion using the information of the entire image series. Finally, by collating the total number of each part with the generated ATSG, the excess or deficiency of parts are investigated, and task procedures are removed or added according to those parts. In the experiment section, we build an ATSG using the proposed method to a graphical instruction manual for a chair and demonstrate the action sequences found in the ATSG can be performed by a dual-arm robot execution. The results show the proposed method is effective and simplifies robot teaching in automatic assembly.

Challenges and Outlook in Robotic Manipulation of Deformable Objects

Deformable object manipulation (DOM) is an emerging research problem in robotics. The ability to manipulate deformable objects endows robots with higher autonomy and promises new applications in the industrial, services, and healthcare sectors. However, compared to rigid object manipulation, the manipulation of deformable objects is considerably more complex and is still an open research problem. Tackling the challenges in DOM demands breakthroughs in almost all aspects of robotics, namely hardware design, sensing, deformation modeling, planning, and control. In this article, we highlight the main challenges that arise by considering deformation and review recent advances in each sub-field. A particular focus of our paper lies in the discussions of these challenges and proposing promising directions of research.

Controlling Pivoting Gait using Graph Model Predictive Control

Pivoting gait is efficient for manipulating a big and heavy object with relatively small manipulating force, in which a robot iteratively tilts the object, rotates it around the vertex, and then puts it down to the floor. However, pivoting gait can easily fail even with a small external disturbance due to its instability in nature. To cope with this problem, we propose a controller to robustly control the object motion during the pivoting gait by introducing two gait modes, i.e., one is the double-support mode, which can manipulate a relatively light object with faster speed, and the other is the quadruple-support mode, which can manipulate a relatively heavy object with lower speed. To control the pivoting gait, a graph model predictive control is applied taking into account of these two gait modes. By adaptively switching the gait mode according to the applied external disturbance, a robot can stably perform the pivoting gait even if the external disturbance is applied to the object.

Error Identification and Recovery in Robotic Snap Assembly

Existing methods for predicting robotic snap joint assembly cannot predict failures before their occurrence. To address this limitation, this paper proposes a method for predicting error states before the occurence of error, thereby enabling timely recovery. Robotic snap joint assembly requires precise positioning; therefore, even a slight offset between parts can lead to assembly failure. To correctly predict error states, we apply functional principal component analysis (fPCA) to 6D force/torque profiles that are terminated before the occurence of an error. The error state is identified by applying a feature vector to a decision tree, wherein the support vector machine (SVM) is employed at each node. If the estimation accuracy is low, we perform additional probing to more correctly identify the error state. Finally, after identifying the error state, a robot performs the error recovery motion based on the identified error state. Through the experimental results of assembling plastic parts with four snap joints, we show that the error states can be correctly estimated and a robot can recover from the identified error state.

Robotic Imitation of Human Assembly Skills Using Hybrid Trajectory and Force Learning

Robotic assembly tasks involve complex and low-clearance insertion trajectories with varying contact forces at different stages. While the nominal motion trajectory can be easily obtained from human demonstrations through kinesthetic teaching, teleoperation, simulation, among other methods, the force profile is harder to obtain especially when a real robot is unavailable. It is difficult to obtain a realistic force profile in simulation even with physics engines. Such simulated force profiles tend to be unsuitable for the actual robotic assembly due to the reality gap and uncertainty in the assembly process. To address this problem, we present a combined learning-based framework to imitate human assembly skills through hybrid trajectory learning and force learning. The main contribution of this work is the development of a framework that combines hierarchical imitation learning, to learn the nominal motion trajectory, with a reinforcement learning-based force control scheme to learn an optimal force control policy, that can satisfy the nominal trajectory while adapting to the force requirement of the assembly task. To further improve the imitation learning part, we develop a hierarchical architecture, following the idea of goal-conditioned imitation learning, to generate the trajectory learning policy on the \textit{skill} level offline. Through experimental validations, we corroborate that the proposed learning-based framework is robust to uncertainty in the assembly task, can generate high-quality trajectories, and can find suitable force control policies, which adapt to the task's force requirements more efficiently.

Planning to Repose Long and Heavy Objects Considering a Combination of Regrasp and Constrained Drooping

This paper presents a hierarchical motion planner for planning the manipulation motion to repose long and heavy objects considering external support surfaces. The planner includes a task level layer and a motion level layer. We formulate the manipulation planning problem at the task level by considering grasp poses as nodes and object poses for edges. We consider regrasping and constrained in-hand slip (drooping) during building graphs and find mixed regrasping and drooping sequences by searching the graph. The generated sequences autonomously divide the object weight between the arm and the support surface and avoid configuration obstacles. Cartesian planning is used at the robot motion level to generate motions between adjacent critical grasp poses of the sequence found by the task level layer. Various experiments are carried out to examine the performance of the proposed planner. The results show improved capability of robot arms to manipulate long and heavy objects using the proposed planner. Our contribution is we initially develop a graph-based planning system that reasons both in-hand and regrasp manipulation motion considering external supports. On one hand, the planner integrates regrasping and drooping to realize in-hand manipulation with external support. On the other hand, it switches states by releasing and regrasping objects when the object is in stably placed. The search graphs' nodes could be retrieved from remote cloud servers that provide a large amount of pre-annotated data to implement cyber intelligence.

A Dual-arm Robot that Autonomously Lifts Up and Tumbles Heavy Plates Using Crane Pulley Blocks

This paper develops a planner that plans the action sequences and motion for a dual-arm robot to lift up and flip heavy plates using crane pulley blocks. The problem is motivated by the low payload of modern collaborative robots. Instead of directly manipulating heavy plates that collaborative robots cannot afford, the paper develops a planner for collaborative robots to operate crane pulley blocks. The planner assumes a target plate is pre-attached to the crane hook. It optimizes dual-arm action sequences and plans the robot's dual-arm motion that pulls the rope of the crane pulley blocks to lift up the plate. The crane pulley blocks reduce the payload that each robotic arm needs to bear. When the plate is lifted up to a satisfying pose, the planner plans a pushing motion for one of the robot arms to tumble over the plate while considering force and moment constraints. The article presents the technical details of the planner and several experiments and analysis carried out using a dual-arm robot made by two Universal Robots UR3 arms. The influence of various parameters and optimization goals are investigated and compared in depth. The results show that the proposed planner is flexible and efficient.

Planning a Sequence of Base Positions for a Mobile Manipulator to Perform Multiple Pick-and-Place Tasks

In this paper, we present a planner that plans a sequence of base positions for a mobile manipulator to efficiently and robustly collect objects stored in distinct trays. We achieve high efficiency by exploring the common areas where a mobile manipulator can grasp objects stored in multiple trays simultaneously and move the mobile manipulator to the common areas to reduce the time needed for moving the mobile base. We ensure robustness by optimizing the base position with the best clearance to positioning uncertainty so that a mobile manipulator can complete the task even if there is a certain deviation from the planned base positions. Besides, considering different styles of object placement in the tray, we analyze feasible schemes for dynamically updating the base positions based on either the remaining objects or the target objects to be picked in one round of the tasks. In the experiment part, we examine our planner on various scenarios, including different object placement: (1) Regularly placed toy objects; (2) Randomly placed industrial parts; and different schemes for online execution: (1) Apply globally static base positions; (2) Dynamically update the base positions. The experiment results demonstrate the efficiency, robustness and feasibility of the proposed method.