Get our free extension to see links to code for papers anywhere online!

Chrome logo  Add to Chrome

Firefox logo Add to Firefox

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models


Jun 10, 2022
Aarohi Srivastava , Abhinav Rastogi , Abhishek Rao , Abu Awal Md Shoeb , Abubakar Abid , Adam Fisch , Adam R. Brown , Adam Santoro , Aditya Gupta , Adrià Garriga-Alonso , Agnieszka Kluska , Aitor Lewkowycz , Akshat Agarwal , Alethea Power , Alex Ray , Alex Warstadt , Alexander W. Kocurek , Ali Safaya , Ali Tazarv , Alice Xiang , Alicia Parrish , Allen Nie , Aman Hussain , Amanda Askell , Amanda Dsouza , Ambrose Slone , Ameet Rahane , Anantharaman S. Iyer , Anders Andreassen , Andrea Madotto , Andrea Santilli , Andreas Stuhlmüller , Andrew Dai , Andrew La , Andrew Lampinen , Andy Zou , Angela Jiang , Angelica Chen , Anh Vuong , Animesh Gupta , Anna Gottardi , Antonio Norelli , Anu Venkatesh , Arash Gholamidavoodi , Arfa Tabassum , Arul Menezes , Arun Kirubarajan , Asher Mullokandov , Ashish Sabharwal , Austin Herrick , Avia Efrat , Aykut Erdem , Ayla Karakaş , B. Ryan Roberts , Bao Sheng Loe , Barret Zoph , Bartłomiej Bojanowski , Batuhan Özyurt , Behnam Hedayatnia , Behnam Neyshabur , Benjamin Inden , Benno Stein , Berk Ekmekci , Bill Yuchen Lin , Blake Howald , Cameron Diao , Cameron Dour , Catherine Stinson , Cedrick Argueta , César Ferri Ramírez , Chandan Singh , Charles Rathkopf , Chenlin Meng , Chitta Baral , Chiyu Wu , Chris Callison-Burch , Chris Waites , Christian Voigt , Christopher D. Manning , Christopher Potts , Cindy Ramirez , Clara E. Rivera , Clemencia Siro , Colin Raffel , Courtney Ashcraft , Cristina Garbacea , Damien Sileo , Dan Garrette , Dan Hendrycks , Dan Kilman , Dan Roth , Daniel Freeman , Daniel Khashabi , Daniel Levy , Daniel Moseguí González , Danielle Perszyk , Danny Hernandez , Danqi Chen , Daphne Ippolito , Dar Gilboa , David Dohan , David Drakard , David Jurgens , Debajyoti Datta , Deep Ganguli , Denis Emelin , Denis Kleyko , Deniz Yuret , Derek Chen , Derek Tam , Dieuwke Hupkes , Diganta Misra , Dilyar Buzan , Dimitri Coelho Mollo , Diyi Yang , Dong-Ho Lee , Ekaterina Shutova , Ekin Dogus Cubuk , Elad Segal , Eleanor Hagerman , Elizabeth Barnes , Elizabeth Donoway , Ellie Pavlick , Emanuele Rodola , Emma Lam , Eric Chu , Eric Tang , Erkut Erdem , Ernie Chang , Ethan A. Chi , Ethan Dyer , Ethan Jerzak , Ethan Kim , Eunice Engefu Manyasi , Evgenii Zheltonozhskii , Fanyue Xia , Fatemeh Siar , Fernando Martínez-Plumed , Francesca Happé , Francois Chollet , Frieda Rong , Gaurav Mishra , Genta Indra Winata , Gerard de Melo , Germán Kruszewski , Giambattista Parascandolo , Giorgio Mariani , Gloria Wang , Gonzalo Jaimovitch-López , Gregor Betz , Guy Gur-Ari , Hana Galijasevic , Hannah Kim , Hannah Rashkin , Hannaneh Hajishirzi , Harsh Mehta , Hayden Bogar , Henry Shevlin , Hinrich Schütze , Hiromu Yakura , Hongming Zhang , Hugh Mee Wong , Ian Ng , Isaac Noble , Jaap Jumelet , Jack Geissinger , Jackson Kernion , Jacob Hilton , Jaehoon Lee , Jaime Fernández Fisac , James B. Simon , James Koppel , James Zheng , James Zou , Jan Kocoń , Jana Thompson , Jared Kaplan , Jarema Radom , Jascha Sohl-Dickstein , Jason Phang , Jason Wei , Jason Yosinski , Jekaterina Novikova , Jelle Bosscher , Jennifer Marsh , Jeremy Kim , Jeroen Taal , Jesse Engel , Jesujoba Alabi , Jiacheng Xu , Jiaming Song , Jillian Tang , Joan Waweru , John Burden , John Miller , John U. Balis , Jonathan Berant , Jörg Frohberg , Jos Rozen , Jose Hernandez-Orallo , Joseph Boudeman , Joseph Jones , Joshua B. Tenenbaum , Joshua S. Rule , Joyce Chua , Kamil Kanclerz , Karen Livescu , Karl Krauth , Karthik Gopalakrishnan , Katerina Ignatyeva , Katja Markert , Kaustubh D. Dhole , Kevin Gimpel , Kevin Omondi , Kory Mathewson , Kristen Chiafullo , Ksenia Shkaruta , Kumar Shridhar , Kyle McDonell , Kyle Richardson , Laria Reynolds , Leo Gao , Li Zhang , Liam Dugan , Lianhui Qin , Lidia Contreras-Ochando , Louis-Philippe Morency , Luca Moschella , Lucas Lam , Lucy Noble , Ludwig Schmidt , Luheng He , Luis Oliveros Colón , Luke Metz , Lütfi Kerem Şenel , Maarten Bosma , Maarten Sap , Maartje ter Hoeve , Maheen Farooqi , Manaal Faruqui , Mantas Mazeika , Marco Baturan , Marco Marelli , Marco Maru , Maria Jose Ramírez Quintana , Marie Tolkiehn , Mario Giulianelli , Martha Lewis , Martin Potthast , Matthew L. Leavitt , Matthias Hagen , Mátyás Schubert , Medina Orduna Baitemirova , Melody Arnaud , Melvin McElrath , Michael A. Yee , Michael Cohen , Michael Gu , Michael Ivanitskiy , Michael Starritt , Michael Strube , Michał Swędrowski , Michele Bevilacqua , Michihiro Yasunaga , Mihir Kale , Mike Cain , Mimee Xu , Mirac Suzgun , Mo Tiwari , Mohit Bansal , Moin Aminnaseri , Mor Geva , Mozhdeh Gheini , Mukund Varma T , Nanyun Peng , Nathan Chi , Nayeon Lee , Neta Gur-Ari Krakover , Nicholas Cameron , Nicholas Roberts , Nick Doiron , Nikita Nangia , Niklas Deckers , Niklas Muennighoff , Nitish Shirish Keskar , Niveditha S. Iyer , Noah Constant , Noah Fiedel , Nuan Wen , Oliver Zhang , Omar Agha , Omar Elbaghdadi , Omer Levy , Owain Evans , Pablo Antonio Moreno Casares , Parth Doshi , Pascale Fung , Paul Pu Liang , Paul Vicol , Pegah Alipoormolabashi , Peiyuan Liao , Percy Liang , Peter Chang , Peter Eckersley , Phu Mon Htut , Pinyu Hwang , Piotr Miłkowski , Piyush Patil , Pouya Pezeshkpour , Priti Oli , Qiaozhu Mei , Qing Lyu , Qinlang Chen , Rabin Banjade , Rachel Etta Rudolph , Raefer Gabriel , Rahel Habacker , Ramón Risco Delgado , Raphaël Millière , Rhythm Garg , Richard Barnes , Rif A. Saurous , Riku Arakawa , Robbe Raymaekers , Robert Frank , Rohan Sikand , Roman Novak , Roman Sitelew , Ronan LeBras , Rosanne Liu , Rowan Jacobs , Rui Zhang , Ruslan Salakhutdinov , Ryan Chi , Ryan Lee , Ryan Stovall , Ryan Teehan , Rylan Yang , Sahib Singh , Saif M. Mohammad , Sajant Anand , Sam Dillavou , Sam Shleifer , Sam Wiseman , Samuel Gruetter , Samuel R. Bowman , Samuel S. Schoenholz , Sanghyun Han , Sanjeev Kwatra , Sarah A. Rous , Sarik Ghazarian , Sayan Ghosh , Sean Casey , Sebastian Bischoff , Sebastian Gehrmann , Sebastian Schuster , Sepideh Sadeghi , Shadi Hamdan , Sharon Zhou , Shashank Srivastava , Sherry Shi , Shikhar Singh , Shima Asaadi , Shixiang Shane Gu , Shubh Pachchigar , Shubham Toshniwal , Shyam Upadhyay , Shyamolima , Debnath , Siamak Shakeri , Simon Thormeyer , Simone Melzi , Siva Reddy , Sneha Priscilla Makini , Soo-Hwan Lee , Spencer Torene , Sriharsha Hatwar , Stanislas Dehaene , Stefan Divic , Stefano Ermon , Stella Biderman , Stephanie Lin , Stephen Prasad , Steven T. Piantadosi , Stuart M. Shieber , Summer Misherghi , Svetlana Kiritchenko , Swaroop Mishra , Tal Linzen , Tal Schuster , Tao Li , Tao Yu , Tariq Ali , Tatsu Hashimoto , Te-Lin Wu , Théo Desbordes , Theodore Rothschild , Thomas Phan , Tianle Wang , Tiberius Nkinyili , Timo Schick , Timofei Kornev , Timothy Telleen-Lawton , Titus Tunduny , Tobias Gerstenberg , Trenton Chang , Trishala Neeraj , Tushar Khot , Tyler Shultz , Uri Shaham , Vedant Misra , Vera Demberg , Victoria Nyamai , Vikas Raunak , Vinay Ramasesh , Vinay Uday Prabhu , Vishakh Padmakumar , Vivek Srikumar , William Fedus , William Saunders , William Zhang , Wout Vossen , Xiang Ren , Xiaoyu Tong , Xinran Zhao , Xinyi Wu , Xudong Shen , Yadollah Yaghoobzadeh , Yair Lakretz , Yangqiu Song , Yasaman Bahri , Yejin Choi , Yichi Yang , Yiding Hao , Yifu Chen , Yonatan Belinkov , Yu Hou , Yufang Hou , Yuntao Bai , Zachary Seid , Zhuoye Zhao , Zijian Wang , Zijie J. Wang , Zirui Wang , Ziyi Wu

* 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-bench 

   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email

OPT: Open Pre-trained Transformer Language Models


May 05, 2022
Susan Zhang , Stephen Roller , Naman Goyal , Mikel Artetxe , Moya Chen , Shuohui Chen , Christopher Dewan , Mona Diab , Xian Li , Xi Victoria Lin , Todor Mihaylov , Myle Ott , Sam Shleifer , Kurt Shuster , Daniel Simig , Punit Singh Koura , Anjali Sridhar , Tianlu Wang , Luke Zettlemoyer


   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email

Efficient Language Modeling with Sparse all-MLP


Mar 16, 2022
Ping Yu , Mikel Artetxe , Myle Ott , Sam Shleifer , Hongyu Gong , Ves Stoyanov , Xian Li


   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email

Efficient Large Scale Language Modeling with Mixtures of Experts


Dec 20, 2021
Mikel Artetxe , Shruti Bhosale , Naman Goyal , Todor Mihaylov , Myle Ott , Sam Shleifer , Xi Victoria Lin , Jingfei Du , Srinivasan Iyer , Ramakanth Pasunuru , Giri Anantharaman , Xian Li , Shuohui Chen , Halil Akin , Mandeep Baines , Louis Martin , Xing Zhou , Punit Singh Koura , Brian O'Horo , Jeff Wang , Luke Zettlemoyer , Mona Diab , Zornitsa Kozareva , Ves Stoyanov


   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email

Few-shot Learning with Multilingual Language Models


Dec 20, 2021
Xi Victoria Lin , Todor Mihaylov , Mikel Artetxe , Tianlu Wang , Shuohui Chen , Daniel Simig , Myle Ott , Naman Goyal , Shruti Bhosale , Jingfei Du , Ramakanth Pasunuru , Sam Shleifer , Punit Singh Koura , Vishrav Chaudhary , Brian O'Horo , Jeff Wang , Luke Zettlemoyer , Zornitsa Kozareva , Mona Diab , Veselin Stoyanov , Xian Li

* 36 pages 

   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email

NormFormer: Improved Transformer Pretraining with Extra Normalization


Nov 01, 2021
Sam Shleifer , Jason Weston , Myle Ott


   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email

8-bit Optimizers via Block-wise Quantization


Oct 06, 2021
Tim Dettmers , Mike Lewis , Sam Shleifer , Luke Zettlemoyer

* ICLR2022 submission with appendix 

   Access Paper or Ask Questions

  • Share via Twitter
  • Share via Facebook
  • Share via LinkedIn
  • Share via Whatsapp
  • Share via Messenger
  • Share via Email
1
2
>>