LF-PGVIO: A Visual-Inertial-Odometry Framework for Large Field-of-View Cameras using Points and Geodesic Segments

In this paper, we propose LF-PGVIO, a Visual-Inertial-Odometry (VIO) framework for large Field-of-View (FoV) cameras with a negative plane using points and geodesic segments. Notoriously, when the FoV of a panoramic camera reaches the negative half-plane, the image cannot be unfolded into a single pinhole image. Moreover, if a traditional straight-line detection method is directly applied to the original panoramic image, it cannot be normally used due to the large distortions in the panoramas and remains under-explored in the literature. To address these challenges, we put forward LF-PGVIO, which can provide line constraints for cameras with large FoV, even for cameras with negative-plane FoV, and directly extract omnidirectional curve segments from the raw omnidirectional image. We propose an Omnidirectional Curve Segment Detection (OCSD) method combined with a camera model which is applicable to images with large distortions, such as panoramic annular images, fisheye images, and various panoramic images. Each point on the image is projected onto the sphere, and the detected omnidirectional curve segments in the image named geodesic segments must satisfy the criterion of being a geodesic segment on the unit sphere. The detected geodesic segment is sliced into multiple straight-line segments according to the radian of the geodesic, and descriptors are extracted separately and recombined to obtain new descriptors. Based on descriptor matching, we obtain the constraint relationship of the 3D line segments between multiple frames. In our VIO system, we use sliding window optimization using point feature residuals, line feature residuals, and IMU residuals. Our evaluation of the proposed system on public datasets demonstrates that LF-PGVIO outperforms state-of-the-art methods in terms of accuracy and robustness. Code will be open-sourced at https://github.com/flysoaryun/LF-PGVIO.

Semantic-aware Generation of Multi-view Portrait Drawings

Neural radiance fields (NeRF) based methods have shown amazing performance in synthesizing 3D-consistent photographic images, but fail to generate multi-view portrait drawings. The key is that the basic assumption of these methods -- a surface point is consistent when rendered from different views -- doesn't hold for drawings. In a portrait drawing, the appearance of a facial point may changes when viewed from different angles. Besides, portrait drawings usually present little 3D information and suffer from insufficient training data. To combat this challenge, in this paper, we propose a Semantic-Aware GEnerator (SAGE) for synthesizing multi-view portrait drawings. Our motivation is that facial semantic labels are view-consistent and correlate with drawing techniques. We therefore propose to collaboratively synthesize multi-view semantic maps and the corresponding portrait drawings. To facilitate training, we design a semantic-aware domain translator, which generates portrait drawings based on features of photographic faces. In addition, use data augmentation via synthesis to mitigate collapsed results. We apply SAGE to synthesize multi-view portrait drawings in diverse artistic styles. Experimental results show that SAGE achieves significantly superior or highly competitive performance, compared to existing 3D-aware image synthesis methods. The codes are available at https://github.com/AiArt-HDU/SAGE.

Ring-Rotor: A Novel Retractable Ring-shaped Quadrotor with Aerial Grasping and Transportation Capability

This letter presents a novel and retractable ring-shaped quadrotor called Ring-Rotor that can adjust the vehicle's length and width simultaneously. Unlike other morphing quadrotors with high platform complexity and poor controllability, Ring-Rotor uses only one servo motor for morphing but reduces the largest dimension of the vehicle by approximately 31.4\%. It can guarantee passibility while flying through small spaces in its compact form and energy saving in its standard form. Meanwhile, the vehicle breaks the cross configuration of general quadrotors with four arms connected to the central body and innovates a ring-shaped mechanical structure with spare central space. Based on this, an ingenious whole-body aerial grasping and transportation scheme is designed to carry various shapes of objects without the external manipulator mechanism. Moreover, we exploit a nonlinear model predictive control (NMPC) strategy that uses a time-variant physical parameter model to adapt to the quadrotor morphology. Above mentioned applications are performed in real-world experiments to demonstrate the system's high versatility.

Masked and Adaptive Transformer for Exemplar Based Image Translation

We present a novel framework for exemplar based image translation. Recent advanced methods for this task mainly focus on establishing cross-domain semantic correspondence, which sequentially dominates image generation in the manner of local style control. Unfortunately, cross-domain semantic matching is challenging; and matching errors ultimately degrade the quality of generated images. To overcome this challenge, we improve the accuracy of matching on the one hand, and diminish the role of matching in image generation on the other hand. To achieve the former, we propose a masked and adaptive transformer (MAT) for learning accurate cross-domain correspondence, and executing context-aware feature augmentation. To achieve the latter, we use source features of the input and global style codes of the exemplar, as supplementary information, for decoding an image. Besides, we devise a novel contrastive style learning method, for acquire quality-discriminative style representations, which in turn benefit high-quality image generation. Experimental results show that our method, dubbed MATEBIT, performs considerably better than state-of-the-art methods, in diverse image translation tasks. The codes are available at \url{https://github.com/AiArt-HDU/MATEBIT}.

Sensing with OFDM Waveform at mmWave Band based on Micro-Doppler Analysis

Joint communication and sensing (JCAS) technology has been regarded as one of the innovations in the 6G network. With the channel modeling proposed by the 3rd Generation Partnership Project (3GPP) TR 38.901, this paper investigates the sensing capability using the millimeter-wave (mmWave) band with an orthogonal frequency division multiplexing (OFDM) waveform. Based on micro-Doppler (MD) analysis, we present two case studies, i.e., fan speed detection and human activity recognition, to demonstrate the target modeling with micro-motions, backscattering signal construction, and MD signature extraction using an OFDM waveform at 28 GHz. Simulated signatures demonstrate distinct fan rotation or human motion, and waveform parameters that affect the MD signature extraction are analyzed. Simulation results draw the validity of the proposed modeling and simulation methods, which also aim to facilitate the generation of data sets for various JCAS applications.

Distributed Optimization in Sensor Network for Scalable Multi-Robot Relative State Estimation

This paper is dedicated to achieving scalable relative state estimation using inter-robot Euclidean distance measurements. We consider equipping robots with distance sensors and focus on the optimization problem underlying relative state estimation in this setup. We reveal the commonality between this problem and the coordinates realization problem of a sensor network. Based on this insight, we propose an effective unconstrained optimization model to infer the relative states among robots. To work on this model in a distributed manner, we propose an efficient and scalable optimization algorithm with the classical block coordinate descent method as its backbone. This algorithm exactly solves each block update subproblem with a closed-form solution while ensuring convergence. Our results pave the way for distance measurements-based relative state estimation in large-scale multi-robot systems.

Learning Combinatorial Prompts for Universal Controllable Image Captioning

Controllable Image Captioning (CIC) -- generating natural language descriptions about images under the guidance of given control signals -- is one of the most promising directions towards next-generation captioning systems. Till now, various kinds of control signals for CIC have been proposed, ranging from content-related control to structure-related control. However, due to the format and target gaps of different control signals, all existing CIC works (or architectures) only focus on one certain control signal, and overlook the human-like combinatorial ability. By ``combinatorial", we mean that our humans can easily meet multiple needs (or constraints) simultaneously when generating descriptions. To this end, we propose a novel prompt-based framework for CIC by learning Combinatorial Prompts, dubbed as ComPro. Specifically, we directly utilize a pretrained language model GPT-2 as our language model, which can help to bridge the gap between different signal-specific CIC architectures. Then, we reformulate the CIC as a prompt-guide sentence generation problem, and propose a new lightweight prompt generation network to generate the combinatorial prompts for different kinds of control signals. For different control signals, we further design a new mask attention mechanism to realize the prompt-based CIC. Due to its simplicity, our ComPro can easily be extended to more complex combined control signals by concatenating these prompts. Extensive experiments on two prevalent CIC benchmarks have verified the effectiveness and efficiency of our ComPro on both single and combined control signals.

Tight Collision Probability for UAV Motion Planning in Uncertain Environment

Operating unmanned aerial vehicles (UAVs) in complex environments that feature dynamic obstacles and external disturbances poses significant challenges, primarily due to the inherent uncertainty in such scenarios. Additionally, inaccurate robot localization and modeling errors further exacerbate these challenges. Recent research on UAV motion planning in static environments has been unable to cope with the rapidly changing surroundings, resulting in trajectories that may not be feasible. Moreover, previous approaches that have addressed dynamic obstacles or external disturbances in isolation are insufficient to handle the complexities of such environments. This paper proposes a reliable motion planning framework for UAVs, integrating various uncertainties into a chance constraint that characterizes the uncertainty in a probabilistic manner. The chance constraint provides a probabilistic safety certificate by calculating the collision probability between the robot's Gaussian-distributed forward reachable set and states of obstacles. To reduce the conservatism of the planned trajectory, we propose a tight upper bound of the collision probability and evaluate it both exactly and approximately. The approximated solution is used to generate motion primitives as a reference trajectory, while the exact solution is leveraged to iteratively optimize the trajectory for better results. Our method is thoroughly tested in simulation and real-world experiments, verifying its reliability and effectiveness in uncertain environments.

Roller-Quadrotor: A Novel Hybrid Terrestrial/Aerial Quadrotor with Unicycle-Driven and Rotor-Assisted Turning

Roller-Quadrotor is a novel hybrid terrestrial and aerial quadrotor that combines the elevated maneuverability of the quadrotor with the lengthy endurance of the ground vehicle. This work presents the design, modeling, and experimental validation of Roller-Quadrotor. Flying is achieved through a quadrotor configuration, and four actuators providing thrust. Rolling is supported by unicycle-driven and rotor-assisted turning structure. During terrestrial locomotion, the vehicle needs to overcome rolling and turning resistance, thus saving energy compared to flight mode. This work overcomes the challenging problems of general rotorcraft, reduces energy consumption and allows to through special terrain, such as narrow gaps. It also solves the obstacle avoidance challenge faced by terrestrial robots by flying. We design the models and controllers for the vehicle. The experiment results show that it can switch between aerial and terrestrial locomotion, and be able to safely pass through a narrow gap half the size of its diameter. Besides, it is capable of rolling a distance approximately 3.8 times as much as flying or operating about 42.2 times as lengthy as flying. These results demonstrate the feasibility and effectiveness of the structure and control in rolling through special terrain and energy saving.

Continuous Implicit SDF Based Any-shape Robot Trajectory Optimization

Optimization-based trajectory generation methods are widely used in whole-body planning for robots. However, existing work either oversimplifies the robot's geometry and environment representation, resulting in a conservative trajectory, or suffers from a huge overhead in maintaining additional information such as the Signed Distance Field (SDF). To bridge the gap, we consider the robot as an implicit function, with its surface boundary represented by the zero-level set of its SDF. Based on this, we further employ another implicit function to lazily compute the signed distance to the swept volume generated by the robot and its trajectory. The computation is efficient by exploiting continuity in space-time, and the implicit function guarantees precise and continuous collision evaluation even for nonconvex robots with complex surfaces. Furthermore, we propose a trajectory optimization pipeline applicable to the implicit SDF. Simulation and real-world experiments validate the high performance of our approach for arbitrarily shaped robot trajectory optimization.