Alert button
Picture for Shan Luo

Shan Luo

Alert button

A Message Passing Detection based Affine Frequency Division Multiplexing Communication System

Jul 30, 2023
Lifan Wu, Shan Luo, Dongxiao Song, Fan Yang, Rongping Lin

Figure 1 for A Message Passing Detection based Affine Frequency Division Multiplexing Communication System
Figure 2 for A Message Passing Detection based Affine Frequency Division Multiplexing Communication System
Figure 3 for A Message Passing Detection based Affine Frequency Division Multiplexing Communication System
Figure 4 for A Message Passing Detection based Affine Frequency Division Multiplexing Communication System

The new generation of wireless communication technology is expected to solve the reliability problem of communication in high-speed mobile communication scenarios. An orthogonal time frequency space (OTFS) system has been proposed and can effectively solve this problem. However, the pilot overhead and multiuser multiplexing overhead of the OTFS are relatively high. Therefore, a new modulation technology based on the discrete affine Fourier transform was proposed recently to address the above issues in OTFS, referred to the affine frequency division multiplexing (AFDM). The AFDM attains full diversity due to parameter adjustment according to the delay-Doppler profile of the channel and can achieve performance similar to the OTFS. Due to the limited research on the detection of AFDM currently, we propose a low-complexity yet efficient message passing (MP) algorithm for joint interference cancellation and detection, which takes advantage of the inherent channel sparsity. According to simulation results, the MP detection performs better than the minimum mean square error and maximal ratio combining detection.

* arXiv admin note: text overlap with arXiv:2104.11331 by other authors 
Viaarxiv icon

Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders

Jul 14, 2023
Guanqun Cao, Jiaqi Jiang, Danushka Bollegala, Shan Luo

Figure 1 for Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders
Figure 2 for Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders
Figure 3 for Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders
Figure 4 for Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders

The missing signal caused by the objects being occluded or an unstable sensor is a common challenge during data collection. Such missing signals will adversely affect the results obtained from the data, and this issue is observed more frequently in robotic tactile perception. In tactile perception, due to the limited working space and the dynamic environment, the contact between the tactile sensor and the object is frequently insufficient and unstable, which causes the partial loss of signals, thus leading to incomplete tactile data. The tactile data will therefore contain fewer tactile cues with low information density. In this paper, we propose a tactile representation learning method, named TacMAE, based on Masked Autoencoder to address the problem of incomplete tactile data in tactile perception. In our framework, a portion of the tactile image is masked out to simulate the missing contact region. By reconstructing the missing signals in the tactile image, the trained model can achieve a high-level understanding of surface geometry and tactile properties from limited tactile cues. The experimental results of tactile texture recognition show that our proposed TacMAE can achieve a high recognition accuracy of 71.4% in the zero-shot transfer and 85.8% after fine-tuning, which are 15.2% and 8.2% higher than the results without using masked modeling. The extensive experiments on YCB objects demonstrate the knowledge transferability of our proposed method and the potential to improve efficiency in tactile exploration.

* This paper is accepted at IROS 2023 
Viaarxiv icon

Leveraging Multi-modal Sensing for Robotic Insertion Tasks in R&D Laboratories

Jul 02, 2023
Aaron Butterworth, Gabriella Pizzuto, Leszek Pecyna, Andrew I. Cooper, Shan Luo

Figure 1 for Leveraging Multi-modal Sensing for Robotic Insertion Tasks in R&D Laboratories
Figure 2 for Leveraging Multi-modal Sensing for Robotic Insertion Tasks in R&D Laboratories
Figure 3 for Leveraging Multi-modal Sensing for Robotic Insertion Tasks in R&D Laboratories
Figure 4 for Leveraging Multi-modal Sensing for Robotic Insertion Tasks in R&D Laboratories

Performing a large volume of experiments in Chemistry labs creates repetitive actions costing researchers time, automating these routines is highly desirable. Previous experiments in robotic chemistry have performed high numbers of experiments autonomously, however, these processes rely on automated machines in all stages from solid or liquid addition to analysis of the final product. In these systems every transition between machine requires the robotic chemist to pick and place glass vials, however, this is currently performed using open loop methods which require all equipment being used by the robot to be in well defined known locations. We seek to begin closing the loop in this vial handling process in a way which also fosters human-robot collaboration in the chemistry lab environment. To do this the robot must be able to detect valid placement positions for the vials it is collecting, and reliably insert them into the detected locations. We create a single modality visual method for estimating placement locations to provide a baseline before introducing two additional methods of feedback (force and tactile feedback). Our visual method uses a combination of classic computer vision methods and a CNN discriminator to detect possible insertion points, then a vial is grasped and positioned above an insertion point and the multi-modal methods guide the final insertion movements using an efficient search pattern. Through our experiments we show the baseline insertion rate of 48.78% improves to 89.55% with the addition of our "force and vision" multi-modal feedback method.

* 8 pages, 5 figures, accepted by CASE2023 
Viaarxiv icon

Multimodal Zero-Shot Learning for Tactile Texture Recognition

Jun 22, 2023
Guanqun Cao, Jiaqi Jiang, Danushka Bollegala, Min Li, Shan Luo

Tactile sensing plays an irreplaceable role in robotic material recognition. It enables robots to distinguish material properties such as their local geometry and textures, especially for materials like textiles. However, most tactile recognition methods can only classify known materials that have been touched and trained with tactile data, yet cannot classify unknown materials that are not trained with tactile data. To solve this problem, we propose a tactile zero-shot learning framework to recognise unknown materials when they are touched for the first time without requiring training tactile samples. The visual modality, providing tactile cues from sight, and semantic attributes, giving high-level characteristics, are combined together to bridge the gap between touched classes and untouched classes. A generative model is learnt to synthesise tactile features according to corresponding visual images and semantic embeddings, and then a classifier can be trained using the synthesised tactile features of untouched materials for zero-shot recognition. Extensive experiments demonstrate that our proposed multimodal generative model can achieve a high recognition accuracy of 83.06% in classifying materials that were not touched before. The robotic experiment demo and the dataset are available at https://sites.google.com/view/multimodalzsl.

* Under review at Robotics and Autonomous Systems 
Viaarxiv icon

Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning

May 21, 2023
Daniel Fernandes Gomes, Paolo Paoletti, Shan Luo

Figure 1 for Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning
Figure 2 for Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning
Figure 3 for Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning
Figure 4 for Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning

Recently, several morphologies, each with its advantages, have been proposed for the \textit{GelSight} high-resolution tactile sensors. However, existing simulation methods are limited to flat-surface sensors, which prevents their usage with the newer sensors of non-flat morphologies in Sim2Real experiments. In this paper, we extend a previously proposed GelSight simulation method developed for flat-surface sensors and propose a novel method for curved sensors. In particular, we address the simulation of light rays travelling through a curved tactile membrane in the form of geodesic paths. The method is validated by simulating the finger-shaped GelTip sensor and comparing the generated synthetic tactile images against the corresponding real images. Our extensive experiments show that combining the illumination generated from the geodesic paths, with a background image from the real sensor, produces the best results when compared to the lighting generated by direct linear paths in the same conditions. As the method is parameterised by the sensor mesh, it can be applied in principle to simulate a tactile sensor of any morphology. The proposed method not only unlocks simulating existing optical tactile sensors of complex morphologies but also enables experimenting with sensors of novel morphologies, before the fabrication of the real sensor. Project website: https://danfergo.github.io/geltip-sim

* Robotics: Science and Systems 2023  
Viaarxiv icon

Robotic Perception of Transparent Objects: A Review

Mar 31, 2023
Jiaqi Jiang, Guanqun Cao, Jiankang Deng, Thanh-Toan Do, Shan Luo

Figure 1 for Robotic Perception of Transparent Objects: A Review
Figure 2 for Robotic Perception of Transparent Objects: A Review
Figure 3 for Robotic Perception of Transparent Objects: A Review
Figure 4 for Robotic Perception of Transparent Objects: A Review

Transparent object perception is a rapidly developing research problem in artificial intelligence. The ability to perceive transparent objects enables robots to achieve higher levels of autonomy, unlocking new applications in various industries such as healthcare, services and manufacturing. Despite numerous datasets and perception methods being proposed in recent years, there is still a lack of in-depth understanding of these methods and the challenges in this field. To address this gap, this article provides a comprehensive survey of the platforms and recent advances for robotic perception of transparent objects. We highlight the main challenges and propose future directions of various transparent object perception tasks, i.e., segmentation, reconstruction, and pose estimation. We also discuss the limitations of existing datasets in diversity and complexity, and the benefits of employing multi-modal sensors, such as RGB-D cameras, thermal cameras, and polarised imaging, for transparent object perception. Furthermore, we identify perception challenges in complex and dynamic environments, as well as for objects with changeable geometries. Finally, we provide an interactive online platform to navigate each reference: \url{https://sites.google.com/view/transperception}.

* 21 pages, 10 figures 
Viaarxiv icon

Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors

Jan 19, 2023
Zixi Chen, Shixin Zhang, Shan Luo, Fuchun Sun, Bin Fang

Figure 1 for Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors
Figure 2 for Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors
Figure 3 for Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors
Figure 4 for Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors

Simulation is widely applied in robotics research to save time and resources. There have been several works to simulate optical tactile sensors that leverage either a smoothing method or Finite Element Method (FEM). However, elastomer deformation physics is not considered in the former method, whereas the latter requires a massive amount of computational resources like a computer cluster. In this work, we propose a pluggable and low computational cost simulator using the Taichi programming language for simulating optical tactile sensors, named as Tacchi . It reconstructs elastomer deformation using particles, which allows deformed elastomer surfaces to be rendered into tactile images and reveals contact information without suffering from high computational costs. Tacchi facilitates creating realistic tactile images in simulation, e.g., ones that capture wear-and-tear defects on object surfaces. In addition, the proposed Tacchi can be integrated with robotics simulators for a robot system simulation. Experiment results showed that Tacchi can produce images with better similarity to real images and achieved higher Sim2Real accuracy compared to the existing methods. Moreover, it can be connected with MuJoCo and Gazebo with only the requirement of 1G memory space in GPU compared to a computer cluster applied for FEM. With Tacchi, physical robot simulation with optical tactile sensors becomes possible. All the materials in this paper are available at https://github.com/zixichen007115/Tacchi .

* 8 pages, 6 figures, accepted by IEEE Robotics and Automation Letters 
Viaarxiv icon

Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation

Jan 17, 2023
Guanqun Cao, Jiaqi Jiang, Ningtao Mao, Danushka Bollegala, Min Li, Shan Luo

Figure 1 for Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation
Figure 2 for Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation
Figure 3 for Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation
Figure 4 for Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation

To assist robots in teleoperation tasks, haptic rendering which allows human operators access a virtual touch feeling has been developed in recent years. Most previous haptic rendering methods strongly rely on data collected by tactile sensors. However, tactile data is not widely available for robots due to their limited reachable space and the restrictions of tactile sensors. To eliminate the need for tactile data, in this paper we propose a novel method named as Vis2Hap to generate haptic rendering from visual inputs that can be obtained from a distance without physical interaction. We take the surface texture of objects as key cues to be conveyed to the human operator. To this end, a generative model is designed to simulate the roughness and slipperiness of the object's surface. To embed haptic cues in Vis2Hap, we use height maps from tactile sensors and spectrograms from friction coefficients as the intermediate outputs of the generative model. Once Vis2Hap is trained, it can be used to generate height maps and spectrograms of new surface textures, from which a friction image can be obtained and displayed on a haptic display. The user study demonstrates that our proposed Vis2Hap method enables users to access a realistic haptic feeling similar to that of physical objects. The proposed vision-based haptic rendering has the potential to enhance human operators' perception of the remote environment and facilitate robotic manipulation.

* This paper is accepted at ICRA 2023 
Viaarxiv icon

Inverse Image Frequency for Long-tailed Image Recognition

Sep 11, 2022
Konstantinos Panagiotis Alexandridis, Shan Luo, Anh Nguyen, Jiankang Deng, Stefanos Zafeiriou

Figure 1 for Inverse Image Frequency for Long-tailed Image Recognition
Figure 2 for Inverse Image Frequency for Long-tailed Image Recognition
Figure 3 for Inverse Image Frequency for Long-tailed Image Recognition
Figure 4 for Inverse Image Frequency for Long-tailed Image Recognition

The long-tailed distribution is a common phenomenon in the real world. Extracted large scale image datasets inevitably demonstrate the long-tailed property and models trained with imbalanced data can obtain high performance for the over-represented categories, but struggle for the under-represented categories, leading to biased predictions and performance degradation. To address this challenge, we propose a novel de-biasing method named Inverse Image Frequency (IIF). IIF is a multiplicative margin adjustment transformation of the logits in the classification layer of a convolutional neural network. Our method achieves stronger performance than similar works and it is especially useful for downstream tasks such as long-tailed instance segmentation as it produces fewer false positive detections. Our extensive experiments show that IIF surpasses the state of the art on many long-tailed benchmarks such as ImageNet-LT, CIFAR-LT, Places-LT and LVIS, reaching 55.8% top-1 accuracy with ResNet50 on ImageNet-LT and 26.2% segmentation AP with MaskRCNN on LVIS. Code available at https://github.com/kostas1515/iif

Viaarxiv icon

Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping

Aug 20, 2022
Jiaqi Jiang, Guanqun Cao, Aaron Butterworth, Thanh-Toan Do, Shan Luo

Figure 1 for Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping
Figure 2 for Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping
Figure 3 for Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping
Figure 4 for Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping

Picking up transparent objects is still a challenging task for robots. The visual properties of transparent objects such as reflection and refraction make the current grasping methods that rely on camera sensing fail to detect and localise them. However, humans can handle the transparent object well by first observing its coarse profile and then poking an area of interest to get a fine profile for grasping. Inspired by this, we propose a novel framework of vision-guided tactile poking for transparent objects grasping. In the proposed framework, a segmentation network is first used to predict the horizontal upper regions named as poking regions, where the robot can poke the object to obtain a good tactile reading while leading to minimal disturbance to the object's state. A poke is then performed with a high-resolution GelSight tactile sensor. Given the local profiles improved with the tactile reading, a heuristic grasp is planned for grasping the transparent object. To mitigate the limitations of real-world data collection and labelling for transparent objects, a large-scale realistic synthetic dataset was constructed. Extensive experiments demonstrate that our proposed segmentation network can predict the potential poking region with a high mean Average Precision (mAP) of 0.360, and the vision-guided tactile poking can enhance the grasping success rate significantly from 38.9% to 85.2%. Thanks to its simplicity, our proposed approach could also be adopted by other force or tactile sensors and could be used for grasping of other challenging objects. All the materials used in this paper are available at https://sites.google.com/view/tactilepoking.

* 11 pages, 11 figures, accepted by T-Mech 
Viaarxiv icon