Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

Picture for Jakob Uszkoreit

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers


Jun 18, 2021
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer

* Andreas, Alex, Xiaohua and Lucas contributed equally. We release more than 50'000 ViT models trained under diverse settings on various datasets. We believe this to be a treasure trove for model analysis. Available at https://github.com/google-research/vision_transformer and https://github.com/rwightman/pytorch-image-models 

  Access Paper or Ask Questions

MLP-Mixer: An all-MLP Architecture for Vision


May 17, 2021
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy

* Fixed parameter counts in Table 1 

  Access Paper or Ask Questions

Differentiable Patch Selection for Image Recognition


Apr 07, 2021
Jean-Baptiste Cordonnier, Aravindh Mahendran, Alexey Dosovitskiy, Dirk Weissenborn, Jakob Uszkoreit, Thomas Unterthiner

* Accepted to IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021. Code available at https://github.com/google-research/google-research/tree/master/ptopk_patch_selection/ 

  Access Paper or Ask Questions

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale


Oct 22, 2020
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby

* Fine-tuning code and pre-trained models are available at https://github.com/google-research/vision_transformer 

  Access Paper or Ask Questions

Towards End-to-End In-Image Neural Machine Translation


Oct 20, 2020
Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Firat, Jakob Uszkoreit, Puneet Jain

* Accepted as an oral presentation at EMNLP, NLP Beyond Text workshop, 2020 

  Access Paper or Ask Questions

Object-Centric Learning with Slot Attention


Jun 26, 2020
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf


  Access Paper or Ask Questions

An Empirical Study of Generation Order for Machine Translation


Oct 29, 2019
William Chan, Mitchell Stern, Jamie Kiros, Jakob Uszkoreit


  Access Paper or Ask Questions

Scaling Autoregressive Video Models


Jun 06, 2019
Dirk Weissenborn, Oscar Täckström, Jakob Uszkoreit


  Access Paper or Ask Questions

KERMIT: Generative Insertion-Based Modeling for Sequences


Jun 04, 2019
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, Jakob Uszkoreit

* William Chan, Nikita Kitaev, Kelvin Guu, and Mitchell Stern contributed equally 

  Access Paper or Ask Questions

Insertion Transformer: Flexible Sequence Generation via Insertion Operations


Feb 08, 2019
Mitchell Stern, William Chan, Jamie Kiros, Jakob Uszkoreit


  Access Paper or Ask Questions

Blockwise Parallel Decoding for Deep Autoregressive Models


Nov 07, 2018
Mitchell Stern, Noam Shazeer, Jakob Uszkoreit

* NIPS 2018 

  Access Paper or Ask Questions

Music Transformer


Oct 10, 2018
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck

* Rewrote many sections to clarify the work, and extended relative attention to the local case. Previous title is "An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation" 

  Access Paper or Ask Questions

Universal Transformers


Jul 10, 2018
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser


  Access Paper or Ask Questions

Image Transformer


Jun 15, 2018
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran

* Appears in International Conference on Machine Learning, 2018. Code available at https://github.com/tensorflow/tensor2tensor 

  Access Paper or Ask Questions

Fast Decoding in Sequence Models using Discrete Latent Variables


Jun 07, 2018
Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, Noam Shazeer

* ICML 2018 

  Access Paper or Ask Questions

Self-Attention with Relative Position Representations


Apr 12, 2018
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani

* NAACL 2018 

  Access Paper or Ask Questions

Tensor2Tensor for Neural Machine Translation


Mar 16, 2018
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, Jakob Uszkoreit

* arXiv admin note: text overlap with arXiv:1706.03762 

  Access Paper or Ask Questions

Attention Is All You Need


Dec 06, 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

* 15 pages, 5 figures 

  Access Paper or Ask Questions

One Model To Learn Them All


Jun 16, 2017
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit


  Access Paper or Ask Questions

Hierarchical Question Answering for Long Documents


Feb 08, 2017
Eunsol Choi, Daniel Hewlett, Alexandre Lacoste, Illia Polosukhin, Jakob Uszkoreit, Jonathan Berant


  Access Paper or Ask Questions

A Decomposable Attention Model for Natural Language Inference


Sep 25, 2016
Ankur P. Parikh, Oscar Täckström, Dipanjan Das, Jakob Uszkoreit

* 7 pages, 1 figure, Proceeedings of EMNLP 2016 

  Access Paper or Ask Questions