Alert button
Picture for Wenjie Li

Wenjie Li

Alert button

KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model

Nov 20, 2023
Lei Geng, Xu Yan, Ziqiang Cao, Juntao Li, Wenjie Li, Sujian Li, Xinjie Zhou, Yang Yang, Jun Zhang

Most biomedical pretrained language models are monolingual and cannot handle the growing cross-lingual requirements. The scarcity of non-English domain corpora, not to mention parallel data, poses a significant hurdle in training multilingual biomedical models. Since knowledge forms the core of domain-specific corpora and can be translated into various languages accurately, we propose a model called KBioXLM, which transforms the multilingual pretrained model XLM-R into the biomedical domain using a knowledge-anchored approach. We achieve a biomedical multilingual corpus by incorporating three granularity knowledge alignments (entity, fact, and passage levels) into monolingual corpora. Then we design three corresponding training tasks (entity masking, relation masking, and passage relation prediction) and continue training on top of the XLM-R model to enhance its domain cross-lingual ability. To validate the effectiveness of our model, we translate the English benchmarks of multiple tasks into Chinese. Experimental results demonstrate that our model significantly outperforms monolingual and multilingual pretrained models in cross-lingual zero-shot and few-shot scenarios, achieving improvements of up to 10+ points. Our code is publicly available at https://github.com/ngwlh-gl/KBioXLM.

Viaarxiv icon

VIBE: Topic-Driven Temporal Adaptation for Twitter Classification

Nov 04, 2023
Yuji Zhang, Jing Li, Wenjie Li

Figure 1 for VIBE: Topic-Driven Temporal Adaptation for Twitter Classification
Figure 2 for VIBE: Topic-Driven Temporal Adaptation for Twitter Classification
Figure 3 for VIBE: Topic-Driven Temporal Adaptation for Twitter Classification
Figure 4 for VIBE: Topic-Driven Temporal Adaptation for Twitter Classification

Language features are evolving in real-world social media, resulting in the deteriorating performance of text classification in dynamics. To address this challenge, we study temporal adaptation, where models trained on past data are tested in the future. Most prior work focused on continued pretraining or knowledge updating, which may compromise their performance on noisy social media data. To tackle this issue, we reflect feature change via modeling latent topic evolution and propose a novel model, VIBE: Variational Information Bottleneck for Evolutions. Concretely, we first employ two Information Bottleneck (IB) regularizers to distinguish past and future topics. Then, the distinguished topics work as adaptive features via multi-task training with timestamp and class label prediction. In adaptive learning, VIBE utilizes retrieved unlabeled data from online streams created posterior to training data time. Substantial Twitter experiments on three classification tasks show that our model, with only 3% of data, significantly outperforms previous state-of-the-art continued-pretraining methods.

* The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)  
* accepted by EMNLP 2023 
Viaarxiv icon

Personalized Federated X -armed Bandit

Oct 25, 2023
Wenjie Li, Qifan Song, Jean Honorio

In this work, we study the personalized federated $\mathcal{X}$-armed bandit problem, where the heterogeneous local objectives of the clients are optimized simultaneously in the federated learning paradigm. We propose the \texttt{PF-PNE} algorithm with a unique double elimination strategy, which safely eliminates the non-optimal regions while encouraging federated collaboration through biased but effective evaluations of the local objectives. The proposed \texttt{PF-PNE} algorithm is able to optimize local objectives with arbitrary levels of heterogeneity, and its limited communications protects the confidentiality of the client-wise reward data. Our theoretical analysis shows the benefit of the proposed algorithm over single-client algorithms. Experimentally, \texttt{PF-PNE} outperforms multiple baselines on both synthetic and real life datasets.

Viaarxiv icon

RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning

Oct 21, 2023
Wenjun Hou, Yi Cheng, Kaishuai Xu, Wenjie Li, Jiang Liu

Figure 1 for RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning
Figure 2 for RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning
Figure 3 for RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning
Figure 4 for RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning

Automating radiology report generation can significantly alleviate radiologists' workloads. Previous research has primarily focused on realizing highly concise observations while neglecting the precise attributes that determine the severity of diseases (e.g., small pleural effusion). Since incorrect attributes will lead to imprecise radiology reports, strengthening the generation process with precise attribute modeling becomes necessary. Additionally, the temporal information contained in the historical records, which is crucial in evaluating a patient's current condition (e.g., heart size is unchanged), has also been largely disregarded. To address these issues, we propose RECAP, which generates precise and accurate radiology reports via dynamic disease progression reasoning. Specifically, RECAP first predicts the observations and progressions (i.e., spatiotemporal information) given two consecutive radiographs. It then combines the historical records, spatiotemporal information, and radiographs for report generation, where a disease progression graph and dynamic progression reasoning mechanism are devised to accurately select the attributes of each observation and progression. Extensive experiments on two publicly available datasets demonstrate the effectiveness of our model.

* Accepted by Findings of EMNLP 2023 
Viaarxiv icon

Self-Detoxifying Language Models via Toxification Reversal

Oct 14, 2023
Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, Wenjie Li

Language model detoxification aims to minimize the risk of generating offensive or harmful content in pretrained language models (PLMs) for safer deployment. Existing methods can be roughly categorized as finetuning-based and decoding-based. However, the former is often resource-intensive, while the latter relies on additional components and potentially compromises the generation fluency. In this paper, we propose a more lightweight approach that enables the PLM itself to achieve "self-detoxification". Our method is built upon the observation that prepending a negative steering prompt can effectively induce PLMs to generate toxic content. At the same time, we are inspired by the recent research in the interpretability field, which formulates the evolving contextualized representations within the PLM as an information stream facilitated by the attention layers. Drawing on this idea, we devise a method to identify the toxification direction from the normal generation process to the one prompted with the negative prefix, and then steer the generation to the reversed direction by manipulating the information movement within the attention layers. Experimental results show that our approach, without any fine-tuning or extra components, can achieve comparable performance with state-of-the-art methods.

* Accepted by EMNLP 2023 main conference 
Viaarxiv icon

Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation

Oct 13, 2023
Jian Wang, Yi Cheng, Dongding Lin, Chak Tou Leong, Wenjie Li

Figure 1 for Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation
Figure 2 for Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation
Figure 3 for Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation
Figure 4 for Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation

Target-oriented dialogue systems, designed to proactively steer conversations toward predefined targets or accomplish specific system-side goals, are an exciting area in conversational AI. In this work, by formulating a <dialogue act, topic> pair as the conversation target, we explore a novel problem of personalized target-oriented dialogue by considering personalization during the target accomplishment process. However, there remains an emergent need for high-quality datasets, and building one from scratch requires tremendous human effort. To address this, we propose an automatic dataset curation framework using a role-playing approach. Based on this framework, we construct a large-scale personalized target-oriented dialogue dataset, TopDial, which comprises about 18K multi-turn dialogues. The experimental results show that this dataset is of high quality and could contribute to exploring personalized target-oriented dialogue.

* Accepted to EMNLP-2023 main conference 
Viaarxiv icon

Aligning Language Models with Human Preferences via a Bayesian Approach

Oct 09, 2023
Jiashuo Wang, Haozhao Wang, Shichao Sun, Wenjie Li

In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as d-PM. Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity "Rule-of-Thumb" generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.

* NeurIPS 2023 
Viaarxiv icon

Survey on Deep Face Restoration: From Non-blind to Blind and Beyond

Sep 27, 2023
Wenjie Li, Mei Wang, Kai Zhang, Juncheng Li, Xiaoming Li, Yuhang Zhang, Guangwei Gao, Weihong Deng, Chia-Wen Lin

Figure 1 for Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
Figure 2 for Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
Figure 3 for Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
Figure 4 for Survey on Deep Face Restoration: From Non-blind to Blind and Beyond

Face restoration (FR) is a specialized field within image restoration that aims to recover low-quality (LQ) face images into high-quality (HQ) face images. Recent advances in deep learning technology have led to significant progress in FR methods. In this paper, we begin by examining the prevalent factors responsible for real-world LQ images and introduce degradation techniques used to synthesize LQ images. We also discuss notable benchmarks commonly utilized in the field. Next, we categorize FR methods based on different tasks and explain their evolution over time. Furthermore, we explore the various facial priors commonly utilized in the restoration process and discuss strategies to enhance their effectiveness. In the experimental section, we thoroughly evaluate the performance of state-of-the-art FR methods across various tasks using a unified benchmark. We analyze their performance from different perspectives. Finally, we discuss the challenges faced in the field of FR and propose potential directions for future advancements. The open-source repository corresponding to this work can be found at https:// github.com/ 24wenjie-li/ Awesome-Face-Restoration.

* Face restoration, Survey, Deep learning, Non-blind/Blind, Joint restoration tasks, Facial priors 
Viaarxiv icon