Abstract:The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. However, the open-world mobile manipulation (OWMM) task remains a challenge due to the need for generalization to open-ended instructions and environments, as well as the systematic complexity to integrate high-level decision making with low-level robot control based on both global scene understanding and current agent state. To address this complexity, we propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling. A second challenge is the hallucination from domain shift. To enhance the agent performance, we further introduce an agentic data synthesis pipeline for the OWMM task to adapt the VLM model to our task domain with instruction fine-tuning. We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding, robot state tracking, and multi-modal action generation in a unified model. Through experiments, we demonstrate that our model achieves SOTA performance compared to other foundation models including GPT-4o and strong zero-shot generalization in real world. The project page is at https://github.com/HHYHRHY/OWMM-Agent
Abstract:Low-altitude economy (LAE) represents an emerging economic paradigm that redefines commercial and social aerial activities. Large artificial intelligence models (LAIMs) offer transformative potential to further enhance the intelligence of LAE services. However, deploying LAIMs in LAE poses several challenges, including the significant gap between their computational/storage demands and the limited onboard resources of LAE entities, the mismatch between lab-trained LAIMs and dynamic physical environments, and the inefficiencies of traditional decoupled designs for sensing, communication, and computation. To address these issues, we first propose a hierarchical system architecture tailored for LAIM deployment and present representative LAE application scenarios. Next, we explore key enabling techniques that facilitate the mutual co-evolution of LAIMs and low-altitude systems, and introduce a task-oriented execution pipeline for scalable and adaptive service delivery. Then, the proposed framework is validated through real-world case studies. Finally, we outline open challenges to inspire future research.
Abstract:Constant-envelope signals are widely employed in wireless communication systems due to their hardware-friendly design, energy efficiency, and enhanced reliability. However, detecting these signals reliably using simple, power-efficient receivers remains a critical challenge. While coherent detection methods generally offer superior performance, they require complex frequency synchronization, which increases receiver complexity and power consumption. In contrast, noncoherent detection is inherently simpler since it avoids frequency synchronization. However, traditional noncoherent detection approaches still rely on In-phase and Quadrature-phase (IQ) demodulators to extract the noisy received signal magnitudes, and assume the energy detector as the test statistic according to the IQ structure, without rigorous theoretical justification. Motivated by the practical need for robust and low-complexity detection, this paper proposes a comprehensive framework for optimal signal detection using a simple bandpass-filter envelope-detector (BFED) in conjunction with a Bayesian approach and the generalized likelihood ratio test (GLRT) under unknown amplitude conditions. By leveraging approximations of the modified Bessel functions, we derive two distinct regimes of the optimal detector. In the low SNR regime, we rigorously prove that the energy detector emerges as the Bayesian-optimal solution, thereby establishing its theoretical validity for the first time. In the high SNR regime, we derive a novel amplitude-based detector that directly compares the estimated amplitude against the noise standard deviation, leading to a simple yet optimal detection strategy. Numerical simulations validate the theoretical findings, confirming that both the energy and amplitude detectors achieve the minimum error probability in their respective SNR domains.
Abstract:This paper presents a non-cooperative source localization approach based on received signal strength (RSS) and 2D environment map, considering both line-of-sight (LOS) and non-line-of-sight (NLOS) conditions. Conventional localization methods, e.g., weighted centroid localization (WCL), may perform bad. This paper proposes a segmented regression approach using 2D maps to estimate source location and propagation environment jointly. By leveraging topological information from the 2D maps, a support vector-assisted algorithm is developed to solve the segmented regression problem, separate the LOS and NLOS measurements, and estimate the location of source. The proposed method demonstrates a good localization performance with an improvement of over 30% in localization rooted mean squared error (RMSE) compared to the baseline methods.
Abstract:Heterogeneous multi-robot systems (HMRS) have emerged as a powerful approach for tackling complex tasks that single robots cannot manage alone. Current large-language-model-based multi-agent systems (LLM-based MAS) have shown success in areas like software development and operating systems, but applying these systems to robot control presents unique challenges. In particular, the capabilities of each agent in a multi-robot system are inherently tied to the physical composition of the robots, rather than predefined roles. To address this issue, we introduce a novel multi-agent framework designed to enable effective collaboration among heterogeneous robots with varying embodiments and capabilities, along with a new benchmark named Habitat-MAS. One of our key designs is $\textit{Robot Resume}$: Instead of adopting human-designed role play, we propose a self-prompted approach, where agents comprehend robot URDF files and call robot kinematics tools to generate descriptions of their physics capabilities to guide their behavior in task planning and action execution. The Habitat-MAS benchmark is designed to assess how a multi-agent framework handles tasks that require embodiment-aware reasoning, which includes 1) manipulation, 2) perception, 3) navigation, and 4) comprehensive multi-floor object rearrangement. The experimental results indicate that the robot's resume and the hierarchical design of our multi-agent system are essential for the effective operation of the heterogeneous multi-robot system within this intricate problem context.
Abstract:Massive multiple-input multiple-output (MIMO) systems offer significant potential to enhance wireless communication performance, yet efficient and accurate channel state information (CSI) tracking remains a key challenge, particularly in dynamic urban settings. To address this, we propose a radio mapassisted framework for CSI tracking and trajectory discovery, relying on sparse channel observations. The radio map is redefined as a mapping from spatial positions to deterministic channel covariance matrices, which captures the complex and time-varying characteristics of urban wireless environments. Leveraging these covariance maps, we develop a CSI tracking method that enables accurate estimation using only single-dimensional observations collected during user movement. Furthermore, we present an efficient algorithm that constructs and continuously refines the radio map through sequential sparse observations, even when location labels are uncertain. Numerical results based on real city maps and ray-tracing MIMO channel datasets show that the proposed framework significantly outperforms baseline methods in both accuracy and adaptability.
Abstract:Visual navigation tasks are critical for household service robots. As these tasks become increasingly complex, effective communication and collaboration among multiple robots become imperative to ensure successful completion. In recent years, large language models (LLMs) have exhibited remarkable comprehension and planning abilities in the context of embodied agents. However, their application in household scenarios, specifically in the use of multiple agents collaborating to complete complex navigation tasks through communication, remains unexplored. Therefore, this paper proposes a framework for decentralized multi-agent navigation, leveraging LLM-enabled communication and collaboration. By designing the communication-triggered dynamic leadership organization structure, we achieve faster team consensus with fewer communication instances, leading to better navigation effectiveness and collaborative exploration efficiency. With the proposed novel communication scheme, our framework promises to be conflict-free and robust in multi-object navigation tasks, even when there is a surge in team size.
Abstract:This paper explores an energy-modified leverage sampling strategy for matrix completion in radio map construction. The main goal is to address potential identifiability issues in matrix completion with sparse observations by using a probabilistic sampling approach. Although conventional leverage sampling is commonly employed for designing sampling patterns, it often assigns high sampling probability to locations with low received signal strength (RSS) values, leading to a low sampling efficiency. Theoretical analysis demonstrates that the leverage score produces pseudo images of sources, and in the regions around the source locations, the leverage probability is asymptotically consistent with the RSS. Based on this finding, an energy-modified leverage probability-based sampling strategy is investigated for efficient sampling. Numerical demonstrations indicate that the proposed sampling strategy can decrease the normalized mean squared error (NMSE) of radio map construction by more than 10% for both matrix completion and interpolation-assisted matrix completion schemes, compared to conventional methods.
Abstract:Machine learning (ML) facilitates rapid channel modeling for 5G and beyond wireless communication systems. Many existing ML techniques utilize a city map to construct the radio map; however, an updated city map may not always be available. This paper proposes to employ the received signal strength (RSS) data to jointly construct the radio map and the virtual environment by exploiting the geometry structure of the environment. In contrast to many existing ML approaches that lack of an environment model, we develop a virtual obstacle model and characterize the geometry relation between the propagation paths and the virtual obstacles. A multi-screen knife-edge model is adopted to extract the key diffraction features, and these features are fed into a neural network (NN) for diffraction representation. To describe the scattering, as oppose to most existing methods that directly input an entire city map, our model focuses on the geometry structure from the local area surrounding the TX-RX pair and the spatial invariance of such local geometry structure is exploited. Numerical experiments demonstrate that, in addition to reconstructing a 3D virtual environment, the proposed model outperforms the state-of-the-art methods in radio map construction with 10%-18% accuracy improvements. It can also reduce 20% data and 50% training epochs when transferred to a new environment.
Abstract:This paper addresses the challenge of reconstructing a 3D power spectrum map from sparse, scattered, and incomplete spectrum measurements. It proposes an integrated approach combining interpolation and block-term tensor decomposition (BTD). This approach leverages an interpolation model with the BTD structure to exploit the spatial correlation of power spectrum maps. Additionally, nuclear norm regularization is incorporated to effectively capture the low-rank characteristics. To implement this approach, a novel algorithm that combines alternating regression with singular value thresholding is developed. Analytical justification for the enhancement provided by the BTD structure in interpolating power spectrum maps is provided, yielding several important theoretical insights. The analysis explores the impact of the spectrum on the error in the proposed method and compares it to conventional local polynomial interpolation. Extensive numerical results demonstrate that the proposed method outperforms state-of-the-art methods in terms of signal source separation and power spectrum map construction, and remains stable under off-grid measurements and inhomogeneous measurement topologies.