Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals



Payal Bajaj , Chenyan Xiong , Guolin Ke , Xiaodong Liu , Di He , Saurabh Tiwary , Tie-Yan Liu , Paul Bennett , Xia Song , Jianfeng Gao

* Update details in scaled initialization and add acknowledgement 

   Access Paper or Ask Questions

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators



Yu Meng , Chenyan Xiong , Payal Bajaj , Saurabh Tiwary , Paul Bennett , Jiawei Han , Xia Song

* ICLR 2022. (Code and Models: https://github.com/microsoft/AMOS

   Access Paper or Ask Questions

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model



Shaden Smith , Mostofa Patwary , Brandon Norick , Patrick LeGresley , Samyam Rajbhandari , Jared Casper , Zhun Liu , Shrimai Prabhumoye , George Zerveas , Vijay Korthikanti , Elton Zhang , Rewon Child , Reza Yazdani Aminabadi , Julie Bernauer , Xia Song , Mohammad Shoeybi , Yuxiong He , Michael Houston , Saurabh Tiwary , Bryan Catanzaro

* Shaden Smith and Mostofa Patwary contributed equally 

   Access Paper or Ask Questions

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining



Yu Meng , Chenyan Xiong , Payal Bajaj , Saurabh Tiwary , Paul Bennett , Jiawei Han , Xia Song


   Access Paper or Ask Questions

Knowledge-Aware Language Model Pretraining



Corby Rosset , Chenyan Xiong , Minh Phan , Xia Song , Paul Bennett , Saurabh Tiwary

* NeurIPS 2020, somehow the Arxiv latex processing made it over 8 pages -- it wasn't 

   Access Paper or Ask Questions

Generic Intent Representation in Web Search



Hongfei Zhang , Xia Song , Chenyan Xiong , Corby Rosset , Paul N. Bennett , Nick Craswell , Saurabh Tiwary

* SIGIR 2019: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval 

   Access Paper or Ask Questions

MS MARCO: A Human Generated MAchine Reading COmprehension Dataset



Payal Bajaj , Daniel Campos , Nick Craswell , Li Deng , Jianfeng Gao , Xiaodong Liu , Rangan Majumder , Andrew McNamara , Bhaskar Mitra , Tri Nguyen , Mir Rosenberg , Xia Song , Alina Stoica , Saurabh Tiwary , Tong Wang


   Access Paper or Ask Questions

Towards Language Agnostic Universal Representations



Armen Aghajanyan , Xia Song , Saurabh Tiwary


   Access Paper or Ask Questions