Visual Action Planning with Multiple Heterogeneous Agents

Visual planning methods are promising to handle complex settings where extracting the system state is challenging. However, none of the existing works tackles the case of multiple heterogeneous agents which are characterized by different capabilities and/or embodiment. In this work, we propose a method to realize visual action planning in multi-agent settings by exploiting a roadmap built in a low-dimensional structured latent space and used for planning. To enable multi-agent settings, we infer possible parallel actions from a dataset composed of tuples associated with individual actions. Next, we evaluate feasibility and cost of them based on the capabilities of the multi-agent system and endow the roadmap with this information, building a capability latent space roadmap (C-LSR). Additionally, a capability suggestion strategy is designed to inform the human operator about possible missing capabilities when no paths are found. The approach is validated in a simulated burger cooking task and a real-world box packing task.

Low-Cost Teleoperation with Haptic Feedback through Vision-based Tactile Sensors for Rigid and Soft Object Manipulation

Haptic feedback is essential for humans to successfully perform complex and delicate manipulation tasks. A recent rise in tactile sensors has enabled robots to leverage the sense of touch and expand their capability drastically. However, many tasks still need human intervention/guidance. For this reason, we present a teleoperation framework designed to provide haptic feedback to human operators based on the data from camera-based tactile sensors mounted on the robot gripper. Partial autonomy is introduced to prevent slippage of grasped objects during task execution. Notably, we rely exclusively on low-cost off-the-shelf hardware to realize an affordable solution. We demonstrate the versatility of the framework on nine different objects ranging from rigid to soft and fragile ones, using three different operators on real hardware.

Enabling Robot Manipulation of Soft and Rigid Objects with Vision-based Tactile Sensors

Endowing robots with tactile capabilities opens up new possibilities for their interaction with the environment, including the ability to handle fragile and/or soft objects. In this work, we equip the robot gripper with low-cost vision-based tactile sensors and propose a manipulation algorithm that adapts to both rigid and soft objects without requiring any knowledge of their properties. The algorithm relies on a touch and slip detection method, which considers the variation in the tactile images with respect to reference ones. We validate the approach on seven different objects, with different properties in terms of rigidity and fragility, to perform unplugging and lifting tasks. Furthermore, to enhance applicability, we combine the manipulation algorithm with a grasp sampler for the task of finding and picking a grape from a bunch without damaging~it.

Ensemble Latent Space Roadmap for Improved Robustness in Visual Action Planning

Planning in learned latent spaces helps to decrease the dimensionality of raw observations. In this work, we propose to leverage the ensemble paradigm to enhance the robustness of latent planning systems. We rely on our Latent Space Roadmap (LSR) framework, which builds a graph in a learned structured latent space to perform planning. Given multiple LSR framework instances, that differ either on their latent spaces or on the parameters for constructing the graph, we use the action information as well as the embedded nodes of the produced plans to define similarity measures. These are then utilized to select the most promising plans. We validate the performance of our Ensemble LSR (ENS-LSR) on simulated box stacking and grape harvesting tasks as well as on a real-world robotic T-shirt folding experiment.

Distributed Adaptive and Resilient Control of Multi-Robot Systems with Limited Field of View Interactions

In this paper, we consider two coupled problems for distributed multi-robot systems (MRSs) coordinating with limited field of view (FOV) sensors: adaptive tuning of interaction gains and rejection of sensor attacks. First, a typical shortcoming of distributed control frameworks (e.g., potential fields) is that the overall system behavior is highly sensitive to the gain assigned to relative interactions. Second, MRSs with limited FOV sensors can be more susceptible to sensor attacks aimed at their FOVs, and therefore must be resilient to such attacks. Based on these shortcomings, we propose a comprehensive solution that combines efforts in adaptive gain tuning and attack resilience to the problem of topology control for MRSs with limited FOVs. Specifically, we first derive an adaptive gain tuning scheme based on satisfying nominal pairwise interactions, which yields a dynamic balancing of interaction strengths in a robot's neighborhood. We then model additive sensor and actuator attacks (or faults) and derive H infinity control protocols by employing a static output-feedback technique, guaranteeing bounded L2 gains of the error induced by the attack (fault) signals. Finally, simulation results using ROS Gazebo are provided to support our theoretical findings.

Experimental Validation of Stable Coordination for Multi-Robot Systems with Limited Fields of View using a PortableMulti-Robot Testbed

In this paper, we address the problem of stable coordinated motion in multi-robot systems with limited fields of view (FOVs). These problems arise naturally for multi-robot systems that interact based on sensing, such as our case study of multiple unmanned aerial vehicles (UAVs) each equipped with several cameras that are used for detecting neighboring UAVs. In this context, our contributions are: i) first, we derive a framework for studying stable motion and distributed topology control for multi-robot systems with limited FOVs; and ii) Then, we provide experimental results in indoor and challenging outdoor environments (e.g., with wind speeds up to 10 mph) with a team of UAVs to demonstrate the performance of the proposed control framework using a portable multi-robot experimental set-up.

Robotic Wireless Sensor Networks

In this chapter, we present a literature survey of an emerging, cutting-edge, and multi-disciplinary field of research at the intersection of Robotics and Wireless Sensor Networks (WSN) which we refer to as Robotic Wireless Sensor Networks (RWSN). We define a RWSN as an autonomous networked multi-robot system that aims to achieve certain sensing goals while meeting and maintaining certain communication performance requirements, through cooperative control, learning and adaptation. While both of the component areas, i.e., Robotics and WSN, are very well-known and well-explored, there exist a whole set of new opportunities and research directions at the intersection of these two fields which are relatively or even completely unexplored. One such example would be the use of a set of robotic routers to set up a temporary communication path between a sender and a receiver that uses the controlled mobility to the advantage of packet routing. We find that there exist only a limited number of articles to be directly categorized as RWSN related works whereas there exist a range of articles in the robotics and the WSN literature that are also relevant to this new field of research. To connect the dots, we first identify the core problems and research trends related to RWSN such as connectivity, localization, routing, and robust flow of information. Next, we classify the existing research on RWSN as well as the relevant state-of-the-arts from robotics and WSN community according to the problems and trends identified in the first step. Lastly, we analyze what is missing in the existing literature, and identify topics that require more research attention in the future.

Distributed and Proximity-Constrained C-Means for Discrete Coverage Control

In this paper we present a novel distributed coverage control framework for a network of mobile agents, in charge of covering a finite set of points of interest (PoI), such as people in danger, geographically dispersed equipment or environmental landmarks. The proposed algorithm is inspired by C-Means, an unsupervised learning algorithm originally proposed for non-exclusive clustering and for identification of cluster centroids from a set of observations. To cope with the agents' limited sensing range and avoid infeasible coverage solutions, traditional C-Means needs to be enhanced with proximity constraints, ensuring that each agent takes into account only neighboring PoIs. The proposed coverage control framework provides useful information concerning the ranking or importance of the different PoIs to the agents, which can be exploited in further application-dependent data fusion processes, patrolling, or disaster relief applications.

Route Swarm: Wireless Network Optimization through Mobility

In this paper, we demonstrate a novel hybrid architecture for coordinating networked robots in sensing and information routing applications. The proposed INformation and Sensing driven PhysIcally REconfigurable robotic network (INSPIRE), consists of a Physical Control Plane (PCP) which commands agent position, and an Information Control Plane (ICP) which regulates information flow towards communication/sensing objectives. We describe an instantiation where a mobile robotic network is dynamically reconfigured to ensure high quality routes between static wireless nodes, which act as source/destination pairs for information flow. The ICP commands the robots towards evenly distributed inter-flow allocations, with intra-flow configurations that maximize route quality. The PCP then guides the robots via potential-based control to reconfigure according to ICP commands. This formulation, deemed Route Swarm, decouples information flow and physical control, generating a feedback between routing and sensing needs and robotic configuration. We demonstrate our propositions through simulation under a realistic wireless network regime.

Robotic Message Ferrying for Wireless Networks using Coarse-Grained Backpressure Control

We formulate the problem of robots ferrying messages between statically-placed source and sink pairs that they can communicate with wirelessly. We first analyze the capacity region for this problem under both ideal (arbitrarily high velocity, long scheduling periods) and realistic conditions. We indicate how robots could be scheduled optimally to satisfy any arrival rate in the capacity region, given prior knowledge about arrival rates. We find that if the number of robots allocated grows proportionally with the number of source-sink pairs, then the capacity of the network scales as $\Theta(1)$, similar to what was shown previously by Grossglauser and Tse for uncontrolled mobility; however, in contrast to that prior result, we also find that with controlled mobility this constant capacity scaling can be obtained while ensuring finite delay. We then consider the setting where the arrival rates are unknown and present a coarse-grained backpressure message ferrying algorithm (CBMF) for it. In CBMF, the robots are matched to sources and sinks once every epoch to maximize a queue-differential-based weight. The matching controls both motion and transmission for each robot: if a robot is matched to a source, it moves towards that source and collects data from it; and if it is matched to a sink, it moves towards that sink and transmits data to it. We show through analysis and simulations the conditions under which CBMF can stabilize the network. We show that the maximum achievable stable throughput with this policy tends to the ideal capacity as the schedule duration and robot velocity increase.