Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox
Picture for Furu Wei

Furu Wei

Kformer: Knowledge Injection in Transformer Feed-Forward Layers


Jan 15, 2022
Yunzhi Yao, Shaohan Huang, Ningyu Zhang, Li Dong, Furu Wei, Huajun Chen

* Work in progress 
  Access Paper or Ask Questions

PromptBERT: Improving BERT Sentence Embeddings with Prompts


Jan 12, 2022
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, Qi Zhang

Figure 1 for PromptBERT: Improving BERT Sentence Embeddings with Prompts
Figure 2 for PromptBERT: Improving BERT Sentence Embeddings with Prompts
Figure 3 for PromptBERT: Improving BERT Sentence Embeddings with Prompts
Figure 4 for PromptBERT: Improving BERT Sentence Embeddings with Prompts

  Access Paper or Ask Questions

Phrase-level Adversarial Example Generation for Neural Machine Translation


Jan 06, 2022
Juncheng Wan, Jian Yang, Shuming Ma, Dongdong Zhang, Weinan Zhang, Yong Yu, Furu Wei

Figure 1 for Phrase-level Adversarial Example Generation for Neural Machine Translation
Figure 2 for Phrase-level Adversarial Example Generation for Neural Machine Translation
Figure 3 for Phrase-level Adversarial Example Generation for Neural Machine Translation
Figure 4 for Phrase-level Adversarial Example Generation for Neural Machine Translation

  Access Paper or Ask Questions

SMDT: Selective Memory-Augmented Neural Document Translation


Jan 05, 2022
Xu Zhang, Jian Yang, Haoyang Huang, Shuming Ma, Dongdong Zhang, Jinlong Li, Furu Wei

Figure 1 for SMDT: Selective Memory-Augmented Neural Document Translation
Figure 2 for SMDT: Selective Memory-Augmented Neural Document Translation
Figure 3 for SMDT: Selective Memory-Augmented Neural Document Translation
Figure 4 for SMDT: Selective Memory-Augmented Neural Document Translation

  Access Paper or Ask Questions

Distilled Dual-Encoder Model for Vision-Language Understanding


Dec 16, 2021
Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei

Figure 1 for Distilled Dual-Encoder Model for Vision-Language Understanding
Figure 2 for Distilled Dual-Encoder Model for Vision-Language Understanding
Figure 3 for Distilled Dual-Encoder Model for Vision-Language Understanding
Figure 4 for Distilled Dual-Encoder Model for Vision-Language Understanding
* Work in progress 
  Access Paper or Ask Questions

Swin Transformer V2: Scaling Up Capacity and Resolution


Nov 18, 2021
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo

Figure 1 for Swin Transformer V2: Scaling Up Capacity and Resolution
Figure 2 for Swin Transformer V2: Scaling Up Capacity and Resolution
Figure 3 for Swin Transformer V2: Scaling Up Capacity and Resolution
Figure 4 for Swin Transformer V2: Scaling Up Capacity and Resolution

  Access Paper or Ask Questions

Separating Long-Form Speech with Group-Wise Permutation Invariant Training


Nov 17, 2021
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei

Figure 1 for Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Figure 2 for Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Figure 3 for Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Figure 4 for Separating Long-Form Speech with Group-Wise Permutation Invariant Training
* 5 pages, 3 figures, 3 tables, submitted to IEEE ICASSP 2022 
  Access Paper or Ask Questions

Document AI: Benchmarks, Models and Applications


Nov 16, 2021
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei

Figure 1 for Document AI: Benchmarks, Models and Applications
Figure 2 for Document AI: Benchmarks, Models and Applications
Figure 3 for Document AI: Benchmarks, Models and Applications
Figure 4 for Document AI: Benchmarks, Models and Applications

  Access Paper or Ask Questions

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts


Nov 03, 2021
Wenhui Wang, Hangbo Bao, Li Dong, Furu Wei

Figure 1 for VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
Figure 2 for VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
Figure 3 for VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
Figure 4 for VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
* Work in progress 
  Access Paper or Ask Questions

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task


Nov 03, 2021
Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

Figure 1 for Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
Figure 2 for Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
Figure 3 for Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
Figure 4 for Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
* WMT21 
  Access Paper or Ask Questions

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing


Oct 29, 2021
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Furu Wei

Figure 1 for WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Figure 2 for WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Figure 3 for WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Figure 4 for WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

  Access Paper or Ask Questions

s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning


Oct 26, 2021
Hangbo Bao, Li Dong, Wenhui Wang, Nan Yang, Furu Wei

Figure 1 for s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Figure 2 for s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Figure 3 for s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Figure 4 for s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
* Demo paper for the s2s-ft toolkit: https://github.com/microsoft/unilm/tree/master/s2s-ft 
  Access Paper or Ask Questions

Improving Non-autoregressive Generation with Mixup Training


Oct 21, 2021
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, Qi Zhang

Figure 1 for Improving Non-autoregressive Generation with Mixup Training
Figure 2 for Improving Non-autoregressive Generation with Mixup Training
Figure 3 for Improving Non-autoregressive Generation with Mixup Training
Figure 4 for Improving Non-autoregressive Generation with Mixup Training

  Access Paper or Ask Questions

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation


Oct 16, 2021
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

Figure 1 for Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Figure 2 for Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Figure 3 for Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Figure 4 for Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
* Preprint 
  Access Paper or Ask Questions

MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding


Oct 16, 2021
Junlong Li, Yiheng Xu, Lei Cui, Furu Wei

Figure 1 for MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Figure 2 for MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Figure 3 for MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Figure 4 for MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
* Work in Progress 
  Access Paper or Ask Questions

SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing


Oct 14, 2021
Junyi Ao, Rui Wang, Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei

Figure 1 for SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing
Figure 2 for SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing
Figure 3 for SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing
Figure 4 for SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing
* work in process 
  Access Paper or Ask Questions

UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training


Oct 12, 2021
Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu

Figure 1 for UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Figure 2 for UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Figure 3 for UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Figure 4 for UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
* ICASSP 2022 Submission 
  Access Paper or Ask Questions

Jointly Learning to Repair Code and Generate Commit Message


Sep 25, 2021
Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li

Figure 1 for Jointly Learning to Repair Code and Generate Commit Message
Figure 2 for Jointly Learning to Repair Code and Generate Commit Message
Figure 3 for Jointly Learning to Repair Code and Generate Commit Message
Figure 4 for Jointly Learning to Repair Code and Generate Commit Message
* Accepted to the 2021 Conference on Empirical Methods in Natural Language Processing 
  Access Paper or Ask Questions

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models


Sep 25, 2021
Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei

Figure 1 for TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Figure 2 for TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Figure 3 for TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Figure 4 for TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
* Work in Progress 
  Access Paper or Ask Questions

Sequence Level Contrastive Learning for Text Summarization


Sep 24, 2021
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei

Figure 1 for Sequence Level Contrastive Learning for Text Summarization
Figure 2 for Sequence Level Contrastive Learning for Text Summarization
Figure 3 for Sequence Level Contrastive Learning for Text Summarization
Figure 4 for Sequence Level Contrastive Learning for Text Summarization
* 2 figures, 12 tables 
  Access Paper or Ask Questions

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training


Sep 15, 2021
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

Figure 1 for Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Figure 2 for Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Figure 3 for Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Figure 4 for Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
* EMNLP 2021 
  Access Paper or Ask Questions

Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression


Sep 07, 2021
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei

Figure 1 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Figure 2 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Figure 3 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Figure 4 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
* Accepted to EMNLP 2021 (main conference) 
  Access Paper or Ask Questions

LayoutReader: Pre-training of Text and Layout for Reading Order Detection


Aug 27, 2021
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei

Figure 1 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Figure 2 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Figure 3 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Figure 4 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
* EMNLP 2021 
  Access Paper or Ask Questions

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset


Jul 12, 2021
Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Yao Qian, Kenichi Kumatani, Furu Wei

Figure 1 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Figure 2 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Figure 3 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Figure 4 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

  Access Paper or Ask Questions

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA


Jun 30, 2021
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

Figure 1 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Figure 2 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Figure 3 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Figure 4 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

  Access Paper or Ask Questions

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains


Jun 29, 2021
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

Figure 1 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Figure 2 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Figure 3 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Figure 4 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
* accepted as ACL2021 Findings 
  Access Paper or Ask Questions