Fluoroscopy-Constrained Magnetic Robot Control via Zernike-Based Field Modeling and Nonlinear MPC

Magnetic actuation enables surgical robots to navigate complex anatomical pathways while reducing tissue trauma and improving surgical precision. However, clinical deployment is limited by the challenges of controlling such systems under fluoroscopic imaging, which provides low frame rate and noisy pose feedback. This paper presents a control framework that remains accurate and stable under such conditions by combining a nonlinear model predictive control (NMPC) framework that directly outputs coil currents, an analytically differentiable magnetic field model based on Zernike polynomials, and a Kalman filter to estimate the robot state. Experimental validation is conducted with two magnetic robots in a 3D-printed fluid workspace and a spine phantom replicating drug delivery in the epidural space. Results show the proposed control method remains highly accurate when feedback is downsampled to 3 Hz with added Gaussian noise (sigma = 2 mm), mimicking clinical fluoroscopy. In the spine phantom experiments, the proposed method successfully executed a drug delivery trajectory with a root mean square (RMS) position error of 1.18 mm while maintaining safe clearance from critical anatomical boundaries.

Geometric Graph Neural Network Modeling of Human Interactions in Crowded Environments

Modeling human trajectories in crowded environments is challenging due to the complex nature of pedestrian behavior and interactions. This paper proposes a geometric graph neural network (GNN) architecture that integrates domain knowledge from psychological studies to model pedestrian interactions and predict future trajectories. Unlike prior studies using complete graphs, we define interaction neighborhoods using pedestrians' field of view, motion direction, and distance-based kernel functions to construct graph representations of crowds. Evaluations across multiple datasets demonstrate improved prediction accuracy through reduced average and final displacement error metrics. Our findings underscore the importance of integrating domain knowledge with data-driven approaches for effective modeling of human interactions in crowds.

A Low-complexity Brain-computer Interface for High-complexity Robot Swarm Control

A brain-computer interface (BCI) is a system that allows a human operator to use only mental commands in controlling end effectors that interact with the world around them. Such a system consists of a measurement device to record the human user's brain activity, which is then processed into commands that drive a system end effector. BCIs involve either invasive measurements which allow for high-complexity control but are generally infeasible, or noninvasive measurements which offer lower quality signals but are more practical to use. In general, BCI systems have not been developed that efficiently, robustly, and scalably perform high-complexity control while retaining the practicality of noninvasive measurements. Here we leverage recent results from feedback information theory to fill this gap by modeling BCIs as a communications system and deploying a human-implementable interaction algorithm for noninvasive control of a high-complexity robot swarm. We construct a scalable dictionary of robotic behaviors that can be searched simply and efficiently by a BCI user, as we demonstrate through a large-scale user study testing the feasibility of our interaction algorithm, a user test of the full BCI system on (virtual and real) robot swarms, and simulations that verify our results against theoretical models. Our results provide a proof of concept for how a large class of high-complexity effectors (even beyond robotics) can be effectively controlled by a BCI system with low-complexity and noisy inputs.

Overcoming the Force Limitations of Magnetic Robotic Surgery: Impact-based Tetherless Suturing

Magnetic robotics obviate the physical connections between the actuators and end effectors resulting in ultra-minimally invasive surgeries. Even though such a wireless actuation method is highly advantageous in medical applications, the trade-off between the applied force and miniature magnetic end effector dimensions has been one of the main challenges in practical applications in clinically relevant conditions. This trade-off is crucial for applications where in-tissue penetration is required (e.g., needle access, biopsy, and suturing). To increase the forces of such magnetic miniature end effectors to practically useful levels, we propose an impact-force-based suturing needle that is capable of penetrating into in-vitro and ex-vivo samples with 3-DoF planar freedom (planar positioning and in-plane orienting). The proposed optimized design is a custom-built 12 G needle that can generate 1.16 N penetration force which is 56 times stronger than its magnetic counterparts with the same size without such an impact force. By containing the fast-moving permanent magnet within the needle in a confined tubular structure, the movement of the overall needle remains slow and easily controllable. The achieved force is in the range of tissue penetration limits allowing the needle to be able to penetrate through tissues to follow a suturing method in a teleoperated fashion. We demonstrated in-vitro needle penetration into a bacon strip and successful suturing of a gauze mesh onto an agar gel mimicking a hernia repair procedure.

Localization and Control of Magnetic Suture Needles in Cluttered Surgical Site with Blood and Tissue

Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the non-linear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle localization a necessity. Our localization method combines neural network-based segmentation and classical techniques, and we are able to consistently locate our needle with 0.73 mm RMS error in clean environments and 2.72 mm RMS error in challenging environments with blood and occlusion. The average localization RMS error is 2.16 mm for all environments we used in the experiments. We combine this localization method with our closed-loop feedback control system to demonstrate the further applicability of localization to autonomous control. Our needle is able to follow a running suture path in (1) no blood, no tissue; (2) heavy blood, no tissue; (3) no blood, with tissue; and (4) heavy blood, with tissue environments. The tip position tracking error ranges from 2.6 mm to 3.7 mm RMS, opening the door towards autonomous suturing tasks.

Swarm Herding: A Leader-Follower Framework For Multi-Robot Navigation

A leader-follower framework is proposed for multi-robot navigation of large scale teams where the leader agents corral the follower agents. A group of leaders is modeled as a 2D deformable object where discrete masses (i.e., leader robots) are interconnected by springs and dampers. A time-varying domain is defined by the positions of leaders while the external forces induce deformations of the domain from its nominal configuration. The team of followers is performing coverage over the time-varying domain by employing a perspective transformation that maps between the nominal and deformed configurations. A decentralized control strategy is proposed where a leader only takes local sensing information and information about its neighbors (connected by virtual springs and dampers), and a follower only needs partial information about leaders and information about its Delaunay neighbors.

Safe, Remote-Access Swarm Robotics Research on the Robotarium

This paper describes the development of the Robotarium -- a remotely accessible, multi-robot research facility. The impetus behind the Robotarium is that multi-robot testbeds constitute an integral and essential part of the multi-agent research cycle, yet they are expensive, complex, and time-consuming to develop, operate, and maintain. These resource constraints, in turn, limit access for large groups of researchers and students, which is what the Robotarium is remedying by providing users with remote access to a state-of-the-art multi-robot test facility. This paper details the design and operation of the Robotarium as well as connects these to the particular considerations one must take when making complex hardware remotely accessible. In particular, safety must be built in already at the design phase without overly constraining which coordinated control programs the users can upload and execute, which calls for minimally invasive safety routines with provable performance guarantees.

Inter-Robot Interactions in Multi-Robot Systems Using Braids

This paper describes a framework for multi-robot coordination and motion planning with emphasis on inter-agent interactions. We focus on the characterization of inter-agent interactions with sufficient level of abstraction so as to allow for the enforcement of desired interaction patterns in a provably safe (i.e., collision-free) manner, e.g., for achieving rich movement patterns in a shared space, or to exchange sensor information. We propose to specify interaction patterns through elements of the so-called braid group. This allows us to not focus on a particular pattern per se, but rather on the problem of being able to execute a whole class of patterns. The result from such a construction is a hybrid system driven by symbolic inputs that must be mapped onto actual paths that both realize the desired interaction levels and remain safe in the sense that collisions are avoided.