Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Picture for Joshua Ainslie

RealFormer: Transformer Likes Residual Attention


Dec 23, 2020
Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie

* 13 pages, 8 figures 

  Access Paper or Ask Questions

Informer: Transformer Likes Informed Attention


Dec 21, 2020
Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie

* 13 pages, 8 figures 

  Access Paper or Ask Questions

Big Bird: Transformers for Longer Sequences


Jul 28, 2020
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed


  Access Paper or Ask Questions

ETC: Encoding Long and Structured Data in Transformers


Apr 21, 2020
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, Sumit Sanghai

* Updated ETC 512 results which mistakenly used a stale input format. Paper has not been peer-reviewed. An extended version will be submitted for review in the future 

  Access Paper or Ask Questions