Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

On the Representation Collapse of Sparse Mixture of Experts



Zewen Chi , Li Dong , Shaohan Huang , Damai Dai , Shuming Ma , Barun Patra , Saksham Singhal , Payal Bajaj , Xia Song , Furu Wei


   Access Paper or Ask Questions

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals



Payal Bajaj , Chenyan Xiong , Guolin Ke , Xiaodong Liu , Di He , Saurabh Tiwary , Tie-Yan Liu , Paul Bennett , Xia Song , Jianfeng Gao

* Update details in scaled initialization and add acknowledgement 

   Access Paper or Ask Questions

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators



Yu Meng , Chenyan Xiong , Payal Bajaj , Saurabh Tiwary , Paul Bennett , Jiawei Han , Xia Song

* ICLR 2022. (Code and Models: https://github.com/microsoft/AMOS

   Access Paper or Ask Questions

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA



Zewen Chi , Shaohan Huang , Li Dong , Shuming Ma , Saksham Singhal , Payal Bajaj , Xia Song , Furu Wei


   Access Paper or Ask Questions

Language Scaling for Universal Suggested Replies Model



Qianlan Ying , Payal Bajaj , Budhaditya Deb , Yu Yang , Wei Wang , Bojia Lin , Milad Shokouhi , Xia Song , Yang Yang , Daxin Jiang


   Access Paper or Ask Questions

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining



Yu Meng , Chenyan Xiong , Payal Bajaj , Saurabh Tiwary , Paul Bennett , Jiawei Han , Xia Song


   Access Paper or Ask Questions

MS MARCO: A Human Generated MAchine Reading COmprehension Dataset



Payal Bajaj , Daniel Campos , Nick Craswell , Li Deng , Jianfeng Gao , Xiaodong Liu , Rangan Majumder , Andrew McNamara , Bhaskar Mitra , Tri Nguyen , Mir Rosenberg , Xia Song , Alina Stoica , Saurabh Tiwary , Tong Wang


   Access Paper or Ask Questions

Embedding Logical Queries on Knowledge Graphs



William L. Hamilton , Payal Bajaj , Marinka Zitnik , Dan Jurafsky , Jure Leskovec

* To appear in NIPS 2018 

   Access Paper or Ask Questions

Inferring Generative Model Structure with Static Analysis



Paroma Varma , Bryan He , Payal Bajaj , Imon Banerjee , Nishith Khandwala , Daniel L. Rubin , Christopher Ré

* NIPS 2017 

   Access Paper or Ask Questions