IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text

We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks with motion data. In addition, we show that IMU2CLIP can significantly improve the downstream performance when fine-tuned for each application (e.g. activity recognition), demonstrating the universal usage of IMU2CLIP as a new pre-trained resource. Our code will be made publicly available.

Geometric Learning and Topological Inference with Biobotic Networks: Convergence Analysis

In this study, we present and analyze a framework for geometric and topological estimation for mapping of unknown environments. We consider agents mimicking motion behaviors of cyborg insects, known as biobots, and exploit coordinate-free local interactions among them to infer geometric and topological information about the environment, under minimal sensing and localization constraints. Local interactions are used to create a graphical representation referred to as the encounter graph. A metric is estimated over the encounter graph of the agents in order to construct a geometric point cloud using manifold learning techniques. Topological data analysis (TDA), in particular persistent homology, is used in order to extract topological features of the space and a classification method is proposed to infer robust features of interest (e.g. existence of obstacles). We examine the asymptotic behavior of the proposed metric in terms of the convergence to the geodesic distances in the underlying manifold of the domain, and provide stability analysis results for the topological persistence. The proposed framework and its convergences and stability analysis are demonstrated through numerical simulations and experiments.

Dynamic Topological Mapping with Biobotic Swarms

In this paper, we present an approach for dynamic exploration and mapping of unknown environments using a swarm of biobotic sensing agents, with a stochastic natural motion model and a leading agent (e.g., an unmanned aerial vehicle). The proposed robust mapping technique constructs a topological map of the environment using only encounter information from the swarm. A sliding window strategy is adopted in conjunction with a topological mapping strategy based on local interactions among the swarm in a coordinate-free fashion to obtain local maps of the environment. These maps are then merged into a global topological map which can be visualized using a graphical representation that integrates geometric as well as topological feature of the environment. Localized robust topological features are extracted using tools from topological data analysis. Simulation results have been presented to illustrate and verify the correctness of our dynamic mapping algorithm.

Robust Topological Feature Extraction for Mapping of Environments using Bio-Inspired Sensor Networks

In this paper, we exploit minimal sensing information gathered from biologically inspired sensor networks to perform exploration and mapping in an unknown environment. A probabilistic motion model of mobile sensing nodes, inspired by motion characteristics of cockroaches, is utilized to extract weak encounter information in order to build a topological representation of the environment. Neighbor to neighbor interactions among the nodes are exploited to build point clouds representing spatial features of the manifold characterizing the environment based on the sampled data. To extract dominant features from sampled data, topological data analysis is used to produce persistence intervals for features, to be used for topological mapping. In order to improve robustness characteristics of the sampled data with respect to outliers, density based subsampling algorithms are employed. Moreover, a robust scale-invariant classification algorithm for persistence diagrams is proposed to provide a quantitative representation of desired features in the data. Furthermore, various strategies for defining encounter metrics with different degrees of information regarding agents' motion are suggested to enhance the precision of the estimation and classification performance of the topological method.