Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Picture for Furu Wei

Kformer: Knowledge Injection in Transformer Feed-Forward Layers


Jan 15, 2022
Yunzhi Yao, Shaohan Huang, Ningyu Zhang, Li Dong, Furu Wei, Huajun Chen

* Work in progress 

  Access Paper or Ask Questions

PromptBERT: Improving BERT Sentence Embeddings with Prompts


Jan 12, 2022
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, Qi Zhang


  Access Paper or Ask Questions

Phrase-level Adversarial Example Generation for Neural Machine Translation


Jan 06, 2022
Juncheng Wan, Jian Yang, Shuming Ma, Dongdong Zhang, Weinan Zhang, Yong Yu, Furu Wei


  Access Paper or Ask Questions

SMDT: Selective Memory-Augmented Neural Document Translation


Jan 05, 2022
Xu Zhang, Jian Yang, Haoyang Huang, Shuming Ma, Dongdong Zhang, Jinlong Li, Furu Wei


  Access Paper or Ask Questions

Distilled Dual-Encoder Model for Vision-Language Understanding


Dec 16, 2021
Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei

* Work in progress 

  Access Paper or Ask Questions

Swin Transformer V2: Scaling Up Capacity and Resolution


Nov 18, 2021
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo


  Access Paper or Ask Questions

Separating Long-Form Speech with Group-Wise Permutation Invariant Training


Nov 17, 2021
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei

* 5 pages, 3 figures, 3 tables, submitted to IEEE ICASSP 2022 

  Access Paper or Ask Questions

Document AI: Benchmarks, Models and Applications


Nov 16, 2021
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei


  Access Paper or Ask Questions

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts


Nov 03, 2021
Wenhui Wang, Hangbo Bao, Li Dong, Furu Wei

* Work in progress 

  Access Paper or Ask Questions

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task


Nov 03, 2021
Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

* WMT21 

  Access Paper or Ask Questions

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing


Oct 29, 2021
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Furu Wei


  Access Paper or Ask Questions

s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning


Oct 26, 2021
Hangbo Bao, Li Dong, Wenhui Wang, Nan Yang, Furu Wei

* Demo paper for the s2s-ft toolkit: https://github.com/microsoft/unilm/tree/master/s2s-ft 

  Access Paper or Ask Questions

Improving Non-autoregressive Generation with Mixup Training


Oct 21, 2021
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, Qi Zhang


  Access Paper or Ask Questions

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation


Oct 16, 2021
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

* Preprint 

  Access Paper or Ask Questions

MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding


Oct 16, 2021
Junlong Li, Yiheng Xu, Lei Cui, Furu Wei

* Work in Progress 

  Access Paper or Ask Questions

SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing


Oct 14, 2021
Junyi Ao, Rui Wang, Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei

* work in process 

  Access Paper or Ask Questions

UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training


Oct 12, 2021
Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu

* ICASSP 2022 Submission 

  Access Paper or Ask Questions

Jointly Learning to Repair Code and Generate Commit Message


Sep 25, 2021
Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li

* Accepted to the 2021 Conference on Empirical Methods in Natural Language Processing 

  Access Paper or Ask Questions

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models


Sep 25, 2021
Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei

* Work in Progress 

  Access Paper or Ask Questions

Sequence Level Contrastive Learning for Text Summarization


Sep 24, 2021
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei

* 2 figures, 12 tables 

  Access Paper or Ask Questions

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training


Sep 15, 2021
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

* EMNLP 2021 

  Access Paper or Ask Questions

Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression


Sep 07, 2021
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei

* Accepted to EMNLP 2021 (main conference) 

  Access Paper or Ask Questions

LayoutReader: Pre-training of Text and Layout for Reading Order Detection


Aug 27, 2021
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei

* EMNLP 2021 

  Access Paper or Ask Questions

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset


Jul 12, 2021
Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Yao Qian, Kenichi Kumatani, Furu Wei


  Access Paper or Ask Questions

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA


Jun 30, 2021
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei


  Access Paper or Ask Questions

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains


Jun 29, 2021
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

* accepted as ACL2021 Findings 

  Access Paper or Ask Questions