Alert button
Picture for Jingyu Zhang

Jingyu Zhang

Alert button

Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation

Sep 19, 2023
Jingyu Zhang, Huitong Yang, Daijie Wu, Xuesong Li, Xinge Zhu, Yuexin Ma

Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods often face substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5, compared to existing unsupervised and weakly-supervised baselines.

* Chinese Conference on Pattern Recognition and Computer Vision (PRCV) 2023  
* 12 pages,4 figures,accepted 
Viaarxiv icon

Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception

Aug 01, 2023
Kun Yang, Dingkang Yang, Jingyu Zhang, Mingcheng Li, Yang Liu, Jing Liu, Hanqi Wang, Peng Sun, Liang Song

Figure 1 for Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception
Figure 2 for Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception
Figure 3 for Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception
Figure 4 for Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception

Multi-agent collaborative perception as a potential application for vehicle-to-everything communication could significantly improve the perception performance of autonomous vehicles over single-agent perception. However, several challenges remain in achieving pragmatic information sharing in this emerging research. In this paper, we propose SCOPE, a novel collaborative perception framework that aggregates the spatio-temporal awareness characteristics across on-road agents in an end-to-end manner. Specifically, SCOPE has three distinct strengths: i) it considers effective semantic cues of the temporal context to enhance current representations of the target agent; ii) it aggregates perceptually critical spatial information from heterogeneous agents and overcomes localization errors via multi-scale feature interactions; iii) it integrates multi-source representations of the target agent based on their complementary contributions by an adaptive fusion paradigm. To thoroughly evaluate SCOPE, we consider both real-world and simulated scenarios of collaborative 3D object detection tasks on three datasets. Extensive experiments demonstrate the superiority of our approach and the necessity of the proposed components.

* Accepted by ICCV 2023 
Viaarxiv icon

An Efficient Multi-solution Solver for the Inverse Kinematics of 3-Section Constant-Curvature Robots

May 02, 2023
Ke Qiu, Jingyu Zhang, Danying Sun, Rong Xiong, Haojian Lu, Yue Wang

Figure 1 for An Efficient Multi-solution Solver for the Inverse Kinematics of 3-Section Constant-Curvature Robots
Figure 2 for An Efficient Multi-solution Solver for the Inverse Kinematics of 3-Section Constant-Curvature Robots
Figure 3 for An Efficient Multi-solution Solver for the Inverse Kinematics of 3-Section Constant-Curvature Robots
Figure 4 for An Efficient Multi-solution Solver for the Inverse Kinematics of 3-Section Constant-Curvature Robots

Piecewise constant curvature is a popular kinematics framework for continuum robots. Computing the model parameters from the desired end pose, known as the inverse kinematics problem, is fundamental in manipulation, tracking and planning tasks. In this paper, we propose an efficient multi-solution solver to address the inverse kinematics problem of 3-section constant-curvature robots by bridging both the theoretical reduction and numerical correction. We derive analytical conditions to simplify the original problem into a one-dimensional problem. Further, the equivalence of the two problems is formalised. In addition, we introduce an approximation with bounded error so that the one dimension becomes traversable while the remaining parameters analytically solvable. With the theoretical results, the global search and numerical correction are employed to implement the solver. The experiments validate the better efficiency and higher success rate of our solver than the numerical methods when one solution is required, and demonstrate the ability of obtaining multiple solutions with optimal path planning in a space with obstacles.

* Robotics: Science and Systems 2023 
Viaarxiv icon

On the Blind Spots of Model-Based Evaluation Metrics for Text Generation

Dec 20, 2022
Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, Yulia Tsvetkov

Figure 1 for On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
Figure 2 for On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
Figure 3 for On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
Figure 4 for On the Blind Spots of Model-Based Evaluation Metrics for Text Generation

In this work, we explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics: stress tests with synthetic data. Basically, we design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. We examine a range of recently proposed evaluation metrics based on pretrained language models, for the tasks of open-ended generation, translation, and summarization. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics. For example, we find that BERTScore ignores truncation errors in summarization, and MAUVE (built on top of GPT-2) is insensitive to errors at the beginning of generations. Further, we investigate the reasons behind these blind spots and suggest practical workarounds for a more reliable evaluation of text generation.

Viaarxiv icon

PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation

Oct 14, 2022
Jingyu Zhang, James Glass, Tianxing He

Figure 1 for PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation
Figure 2 for PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation
Figure 3 for PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation
Figure 4 for PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation

Existing work on controlled text generation (CTG) assumes a control interface of categorical attributes. In this work, we propose a natural language (NL) interface, where we craft a PCFG to embed the control attributes into natural language commands, and propose variants of existing CTG models that take commands as input. In our experiments, we design tailored setups to test model's generalization abilities. We find our PCFG-based command generation approach is effective for handling unseen commands compared to fix-set templates; our proposed NL models can effectively generalize to unseen attributes, a new ability enabled by the NL interface, as well as unseen attribute combinations. Interestingly, we discover that the simple conditional generation approach, enhanced with our proposed NL interface, is a strong baseline in those challenging settings.

Viaarxiv icon

Learning PAC-Bayes Priors for Probabilistic Neural Networks

Sep 21, 2021
Maria Perez-Ortiz, Omar Rivasplata, Benjamin Guedj, Matthew Gleeson, Jingyu Zhang, John Shawe-Taylor, Miroslaw Bober, Josef Kittler

Figure 1 for Learning PAC-Bayes Priors for Probabilistic Neural Networks
Figure 2 for Learning PAC-Bayes Priors for Probabilistic Neural Networks
Figure 3 for Learning PAC-Bayes Priors for Probabilistic Neural Networks
Figure 4 for Learning PAC-Bayes Priors for Probabilistic Neural Networks

Recent works have investigated deep learning models trained by optimising PAC-Bayes bounds, with priors that are learnt on subsets of the data. This combination has been shown to lead not only to accurate classifiers, but also to remarkably tight risk certificates, bearing promise towards self-certified learning (i.e. use all the data to learn a predictor and certify its quality). In this work, we empirically investigate the role of the prior. We experiment on 6 datasets with different strategies and amounts of data to learn data-dependent PAC-Bayes priors, and we compare them in terms of their effect on test performance of the learnt predictors and tightness of their risk certificate. We ask what is the optimal amount of data which should be allocated for building the prior and show that the optimum may be dataset dependent. We demonstrate that using a small percentage of the prior-building data for validation of the prior leads to promising results. We include a comparison of underparameterised and overparameterised models, along with an empirical study of different training objectives and regularisation strategies to learn the prior distribution.

Viaarxiv icon

Machine learning driven synthesis of few-layered WTe2

Oct 10, 2019
Manzhang Xu, Bijun Tang, Chao Zhu, Yuhao Lu, Chao Zhu, Lu Zheng, Jingyu Zhang, Nannan Han, Yuxi Guo, Jun Di, Pin Song, Yongmin He, Lixing Kang, Zhiyong Zhang, Wu Zhao, Cuntai Guan, Xuewen Wang, Zheng Liu

Figure 1 for Machine learning driven synthesis of few-layered WTe2
Figure 2 for Machine learning driven synthesis of few-layered WTe2
Figure 3 for Machine learning driven synthesis of few-layered WTe2
Figure 4 for Machine learning driven synthesis of few-layered WTe2

Reducing the lateral scale of two-dimensional (2D) materials to one-dimensional (1D) has attracted substantial research interest not only to achieve competitive electronic device applications but also for the exploration of fundamental physical properties. Controllable synthesis of high-quality 1D nanoribbons (NRs) is thus highly desirable and essential for the further study. Traditional exploration of the optimal synthesis conditions of novel materials is based on the trial-and-error approach, which is time consuming, costly and laborious. Recently, machine learning (ML) has demonstrated promising capability in guiding material synthesis through effectively learning from the past data and then making recommendations. Here, we report the implementation of supervised ML for the chemical vapor deposition (CVD) synthesis of high-quality 1D few-layered WTe2 nanoribbons (NRs). The synthesis parameters of the WTe2 NRs are optimized by the trained ML model. On top of that, the growth mechanism of as-synthesized 1T' few-layered WTe2 NRs is further proposed, which may inspire the growth strategies for other 1D nanostructures. Our findings suggest that ML is a powerful and efficient approach to aid the synthesis of 1D nanostructures, opening up new opportunities for intelligent material development.

Viaarxiv icon