Get our free extension to see links to code for papers anywhere online!

Add to Chrome

Add to Firefox

Picture for Andreas Steiner

Andreas Steiner

Scaling Vision Transformers to 22 Billion Parameters

Feb 10, 2023
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, Neil Houlsby

Figure 1 for Scaling Vision Transformers to 22 Billion Parameters

Figure 2 for Scaling Vision Transformers to 22 Billion Parameters

Figure 3 for Scaling Vision Transformers to 22 Billion Parameters

Figure 4 for Scaling Vision Transformers to 22 Billion Parameters

Access Paper or Ask Questions

LiT: Zero-Shot Transfer with Locked-image Text Tuning

Nov 15, 2021
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, Lucas Beyer

Figure 1 for LiT: Zero-Shot Transfer with Locked-image Text Tuning

Figure 2 for LiT: Zero-Shot Transfer with Locked-image Text Tuning

Figure 3 for LiT: Zero-Shot Transfer with Locked-image Text Tuning

Figure 4 for LiT: Zero-Shot Transfer with Locked-image Text Tuning

* Xiaohua, Xiao, Basil, Andreas and Lucas contributed equally

Access Paper or Ask Questions

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Jun 18, 2021
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer

Figure 1 for How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Figure 2 for How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Figure 3 for How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Figure 4 for How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

* Andreas, Alex, Xiaohua and Lucas contributed equally. We release more than 50'000 ViT models trained under diverse settings on various datasets. We believe this to be a treasure trove for model analysis. Available at https://github.com/google-research/vision_transformer and https://github.com/rwightman/pytorch-image-models

Access Paper or Ask Questions

MLP-Mixer: An all-MLP Architecture for Vision

May 17, 2021
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy

Figure 1 for MLP-Mixer: An all-MLP Architecture for Vision

Figure 2 for MLP-Mixer: An all-MLP Architecture for Vision

Figure 3 for MLP-Mixer: An all-MLP Architecture for Vision

Figure 4 for MLP-Mixer: An all-MLP Architecture for Vision

* Fixed parameter counts in Table 1

Access Paper or Ask Questions