SMCNet: Supervised Surface Material Classification Using mmWave Radar IQ Signals and Complex-valued CNNs

Understanding surface material properties is crucial for enhancing indoor robot perception and indoor digital twinning. However, not all sensor modalities typically employed for this task are capable of reliably capturing detailed surface material characteristics. By analyzing the reflected RF signal from a mmWave radar sensor, it is possible to extract information about the reflective material and its composition from a certain surface. We introduce a mmWave MIMO FMCW radar-based surface material classifier SMCNet, employing a complex-valued Convolutional Neural Network (CNN) and complex radar IQ signal input for classifying indoor surface materials. While current radar-based material estimation approaches rely on a fixed sensing distance and constrained setups, our approach incorporates a setup with multiple sensing distances. We trained SMCNet using data from three distinct distances and subsequently tested it on these distances, as well as on two more unseen distances. We reached an overall accuracy of 99.12-99.53 % on our test set. Notably, range FFT pre-processing improved accuracy on unknown distances from 25.25 % to 58.81 % without re-training.

RadarCNN: Learning-based Indoor Object Classification from IQ Imaging Radar Data

Radar sensors operating in the mmWave frequency range face challenges when used as indoor perception and imaging devices, primarily due to noise and multipath signal distortions. These distortions often impair the sensors' ability to accurately perceive and image the indoor environment. Nevertheless, this sensor offers distinct advantages over camera and LiDAR sensors. This encompasses the estimation of object reflectivity, known as radar cross-section (RCS), and the ability to penetrate through objects that are thin or have low reflectivity. This results in a 'through-the-wall' sensing capability. Due to the aforementioned disadvantages, most research in the field of imaging radar tends to exclude indoor areas. We introduce a machine learning-based mmWave MIMO FMCW imaging radar object classifier designed to identify small, hand-sized objects in indoor settings, utilizing only radar IQ samples as input. This system achieves 97-99 % accuracy on our test set and maintains approximately 50 % accuracy even under challenging conditions, such as increased background noise and occlusion of sample objects, without the need for adjusting training or pre-processing. This demonstrates the robustness of our approach and offers insights into what needs to be improved in the future to achieve generalization and very high accuracy even in the presence of significant indoor perturbations.

LEMON: Localized Editing with Mesh Optimization and Neural Shaders

In practical use cases, polygonal mesh editing can be faster than generating new ones, but it can still be challenging and time-consuming for users. Existing solutions for this problem tend to focus on a single task, either geometry or novel view synthesis, which often leads to disjointed results between the mesh and view. In this work, we propose LEMON, a mesh editing pipeline that combines neural deferred shading with localized mesh optimization. Our approach begins by identifying the most important vertices in the mesh for editing, utilizing a segmentation model to focus on these key regions. Given multi-view images of an object, we optimize a neural shader and a polygonal mesh while extracting the normal map and the rendered image from each view. By using these outputs as conditioning data, we edit the input images with a text-to-image diffusion model and iteratively update our dataset while deforming the mesh. This process results in a polygonal mesh that is edited according to the given text instruction, preserving the geometric characteristics of the initial mesh while focusing on the most significant areas. We evaluate our pipeline using the DTU dataset, demonstrating that it generates finely-edited meshes more rapidly than the current state-of-the-art methods. We include our code and additional results in the supplementary material.