Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox
Picture for Furu Wei

Furu Wei

Sequence Level Contrastive Learning for Text Summarization


Sep 08, 2021
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei

Figure 1 for Sequence Level Contrastive Learning for Text Summarization
Figure 2 for Sequence Level Contrastive Learning for Text Summarization
Figure 3 for Sequence Level Contrastive Learning for Text Summarization
Figure 4 for Sequence Level Contrastive Learning for Text Summarization
* 2 figures, 12 tables 
  Access Paper or Ask Questions

Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression


Sep 07, 2021
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei

Figure 1 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Figure 2 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Figure 3 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Figure 4 for Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
* Accepted to EMNLP 2021 (main conference) 
  Access Paper or Ask Questions

LayoutReader: Pre-training of Text and Layout for Reading Order Detection


Aug 27, 2021
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei

Figure 1 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Figure 2 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Figure 3 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Figure 4 for LayoutReader: Pre-training of Text and Layout for Reading Order Detection
* EMNLP 2021 
  Access Paper or Ask Questions

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset


Jul 12, 2021
Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Yao Qian, Kenichi Kumatani, Furu Wei

Figure 1 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Figure 2 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Figure 3 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Figure 4 for UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

  Access Paper or Ask Questions

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA


Jun 30, 2021
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

Figure 1 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Figure 2 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Figure 3 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Figure 4 for XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

  Access Paper or Ask Questions

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains


Jun 29, 2021
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

Figure 1 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Figure 2 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Figure 3 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Figure 4 for Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
* accepted as ACL2021 Findings 
  Access Paper or Ask Questions

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders


Jun 25, 2021
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

Figure 1 for DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
Figure 2 for DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
Figure 3 for DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
Figure 4 for DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
* Work in progress 
  Access Paper or Ask Questions

Learning to Sample Replacements for ELECTRA Pre-Training


Jun 25, 2021
Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

Figure 1 for Learning to Sample Replacements for ELECTRA Pre-Training
Figure 2 for Learning to Sample Replacements for ELECTRA Pre-Training
Figure 3 for Learning to Sample Replacements for ELECTRA Pre-Training
Figure 4 for Learning to Sample Replacements for ELECTRA Pre-Training
* Accepted by Findings of ACL 2021 
  Access Paper or Ask Questions

BEiT: BERT Pre-Training of Image Transformers


Jun 15, 2021
Hangbo Bao, Li Dong, Furu Wei

Figure 1 for BEiT: BERT Pre-Training of Image Transformers
Figure 2 for BEiT: BERT Pre-Training of Image Transformers
Figure 3 for BEiT: BERT Pre-Training of Image Transformers
Figure 4 for BEiT: BERT Pre-Training of Image Transformers
* A Path to the BERT Moment of CV. Work in progress 
  Access Paper or Ask Questions

Consistency Regularization for Cross-Lingual Fine-Tuning


Jun 15, 2021
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

Figure 1 for Consistency Regularization for Cross-Lingual Fine-Tuning
Figure 2 for Consistency Regularization for Cross-Lingual Fine-Tuning
Figure 3 for Consistency Regularization for Cross-Lingual Fine-Tuning
Figure 4 for Consistency Regularization for Cross-Lingual Fine-Tuning
* ACL-2021 
  Access Paper or Ask Questions

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment


Jun 11, 2021
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

Figure 1 for Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Figure 2 for Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Figure 3 for Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Figure 4 for Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
* ACL-2021 
  Access Paper or Ask Questions

VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization


Jun 10, 2021
Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei

Figure 1 for VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Figure 2 for VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Figure 3 for VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Figure 4 for VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
* Work in progress 
  Access Paper or Ask Questions

Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding


Jun 09, 2021
Xin Sun, Tao Ge, Furu Wei, Houfeng Wang

Figure 1 for Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Figure 2 for Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Figure 3 for Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Figure 4 for Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
* Accepted by ACL2021 (main conference) 
  Access Paper or Ask Questions

Attention Temperature Matters in Abstractive Summarization Distillation


Jun 08, 2021
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei

Figure 1 for Attention Temperature Matters in Abstractive Summarization Distillation
Figure 2 for Attention Temperature Matters in Abstractive Summarization Distillation
Figure 3 for Attention Temperature Matters in Abstractive Summarization Distillation
Figure 4 for Attention Temperature Matters in Abstractive Summarization Distillation

  Access Paper or Ask Questions

Memory-Efficient Differentiable Transformer Architecture Search


May 31, 2021
Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Weizhu Chen

Figure 1 for Memory-Efficient Differentiable Transformer Architecture Search
Figure 2 for Memory-Efficient Differentiable Transformer Architecture Search
Figure 3 for Memory-Efficient Differentiable Transformer Architecture Search
Figure 4 for Memory-Efficient Differentiable Transformer Architecture Search
* Accepted by Findings of ACL 2021 
  Access Paper or Ask Questions

LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding


Apr 18, 2021
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei

Figure 1 for LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Figure 2 for LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Figure 3 for LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Figure 4 for LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
* Work in progress 
  Access Paper or Ask Questions

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders


Apr 18, 2021
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

Figure 1 for Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Figure 2 for Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Figure 3 for Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Figure 4 for Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
* Preprint 
  Access Paper or Ask Questions

Knowledge Neurons in Pretrained Transformers


Apr 18, 2021
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Furu Wei

Figure 1 for Knowledge Neurons in Pretrained Transformers
Figure 2 for Knowledge Neurons in Pretrained Transformers
Figure 3 for Knowledge Neurons in Pretrained Transformers
Figure 4 for Knowledge Neurons in Pretrained Transformers

  Access Paper or Ask Questions

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs


Apr 18, 2021
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang Xian-Ling Mao, Heyan Huang, Furu Wei

Figure 1 for mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Figure 2 for mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Figure 3 for mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Figure 4 for mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

  Access Paper or Ask Questions

Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge


Apr 06, 2021
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei

Figure 1 for Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Figure 2 for Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Figure 3 for Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Figure 4 for Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
* NAACL 2021 
  Access Paper or Ask Questions

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data


Jan 19, 2021
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang

Figure 1 for UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Figure 2 for UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Figure 3 for UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Figure 4 for UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

  Access Paper or Ask Questions

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting


Jan 02, 2021
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei

Figure 1 for Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Figure 2 for Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Figure 3 for Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Figure 4 for Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

  Access Paper or Ask Questions

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers


Dec 31, 2020
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

Figure 1 for MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Figure 2 for MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Figure 3 for MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Figure 4 for MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

  Access Paper or Ask Questions

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders


Dec 31, 2020
Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei

Figure 1 for XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Figure 2 for XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Figure 3 for XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Figure 4 for XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders

  Access Paper or Ask Questions

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding


Dec 29, 2020
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou

Figure 1 for LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Figure 2 for LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Figure 3 for LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Figure 4 for LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
* Work in progress 
  Access Paper or Ask Questions

Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers


Oct 16, 2020
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei, Ming Zhou

Figure 1 for Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
Figure 2 for Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
Figure 3 for Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
Figure 4 for Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
* 9 pages, 4 figures 
  Access Paper or Ask Questions

Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction


Oct 07, 2020
Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, Ming Zhou

Figure 1 for Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
Figure 2 for Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
Figure 3 for Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
Figure 4 for Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
* Accepted by EMNLP 2020 
  Access Paper or Ask Questions