Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context Mar 08, 2024 Machel Reid , Nikolay Savinov , Denis Teplyashin , Dmitry Lepikhin , Timothy Lillicrap , Jean-baptiste Alayrac , Radu Soricut , Angeliki Lazaridou , Orhan Firat , Julian Schrittwieser , Ioannis Antonoglou , Rohan Anil , Sebastian Borgeaud , Andrew Dai , Katie Millican , Ethan Dyer , Mia Glaese , Thibault Sottiaux , Benjamin Lee , Fabio Viola , Malcolm Reynolds , Yuanzhong Xu , James Molloy , Jilin Chen , Michael Isard , Paul Barham , Tom Hennigan , Ross McIlroy , Melvin Johnson , Johan Schalkwyk , Eli Collins , Eliza Rutherford , Erica Moreira , Kareem Ayoub , Megha Goel , Clemens Meyer , Gregory Thornton , Zhen Yang , Henryk Michalewski , Zaheer Abbas , Nathan Schucher , Ankesh Anand , Richard Ives , James Keeling , Karel Lenc , Salem Haykal , Siamak Shakeri , Pranav Shyam , Aakanksha Chowdhery , Roman Ring , Stephen Spencer , Eren Sezener , Luke Vilnis , Oscar Chang , Nobuyuki Morioka , George Tucker , Ce Zheng , Oliver Woodman , Nithya Attaluri , Tomas Kocisky , Evgenii Eltyshev , Xi Chen , Timothy Chung , Vittorio Selo , Siddhartha Brahma , Petko Georgiev , Ambrose Slone , Zhenkai Zhu , James Lottes , Siyuan Qiao , Ben Caine , Sebastian Riedel , Alex Tomala , Martin Chadwick , Juliette Love , Peter Choy , Sid Mittal , Neil Houlsby , Yunhao Tang , Matthew Lamm , Libin Bai , Qiao Zhang , Luheng He , Yong Cheng , Peter Humphreys , Yujia Li , Sergey Brin , Albin Cassirer , Yingjie Miao , Lukas Zilka , Taylor Tobin , Kelvin Xu , Lev Proleev , Daniel Sohn , Alberto Magni , Lisa Anne Hendricks , Isabel Gao , Santiago Ontañón , Oskar Bunyan , Nathan Byrd , Abhanshu Sharma , Biao Zhang , Mario Pinto , Rishika Sinha , Harsh Mehta , Dawei Jia , Sergi Caelles , Albert Webson , Alex Morris , Becca Roelofs , Yifan Ding , Robin Strudel , Xuehan Xiong , Marvin Ritter , Mostafa Dehghani , Rahma Chaabouni , Abhijit Karmarkar , Guangda Lai , Fabian Mentzer , Bibo Xu , YaGuang Li , Yujing Zhang , Tom Le Paine , Alex Goldin , Behnam Neyshabur , Kate Baumli , Anselm Levskaya , Michael Laskin , Wenhao Jia , Jack W. Rae , Kefan Xiao , Antoine He , Skye Giordano , Lakshman Yagati , Jean-Baptiste Lespiau , Paul Natsev , Sanjay Ganapathy , Fangyu Liu , Danilo Martins , Nanxin Chen , Yunhan Xu , Megan Barnes , Rhys May , Arpi Vezer , Junhyuk Oh , Ken Franko , Sophie Bridgers , Ruizhe Zhao , Boxi Wu , Basil Mustafa , Sean Sechrist , Emilio Parisotto , Thanumalayan Sankaranarayana Pillai , Chris Larkin , Chenjie Gu , Christina Sorokin , Maxim Krikun , Alexey Guseynov , Jessica Landon , Romina Datta , Alexander Pritzel , Phoebe Thacker , Fan Yang , Kevin Hui , Anja Hauth , Chih-Kuan Yeh , David Barker , Justin Mao-Jones , Sophia Austin , Hannah Sheahan , Parker Schuh , James Svensson , Rohan Jain , Vinay Ramasesh , Anton Briukhov , Da-Woon Chung , Tamara von Glehn , Christina Butterfield , Priya Jhakra , Matthew Wiethoff , Justin Frye , Jordan Grimstad , Beer Changpinyo , Charline Le Lan , Anna Bortsova , Yonghui Wu , Paul Voigtlaender , Tara Sainath , Charlotte Smith , Will Hawkins , Kris Cao , James Besley , Srivatsan Srinivasan , Mark Omernick , Colin Gaffney , Gabriela Surita , Ryan Burnell , Bogdan Damoc , Junwhan Ahn , Andrew Brock , Mantas Pajarskas , Anastasia Petrushkina , Seb Noury , Lorenzo Blanco , Kevin Swersky , Arun Ahuja , Thi Avrahami , Vedant Misra , Raoul de Liedekerke , Mariko Iinuma , Alex Polozov , Sarah York , George van den Driessche , Paul Michel , Justin Chiu , Rory Blevins , Zach Gleicher , Adrià Recasens , Alban Rrustemi , Elena Gribovskaya , Aurko Roy , Wiktor Gworek , Séb Arnold , Lisa Lee , James Lee-Thorp , Marcello Maggioni , Enrique Piqueras , Kartikeya Badola , Sharad Vikram , Lucas Gonzalez , Anirudh Baddepudi , Evan Senter , Jacob Devlin , James Qin , Michael Azzam , Maja Trebacz , Martin Polacek , Kashyap Krishnakumar , Shuo-yiin Chang , Matthew Tung , Ivo Penchev , Rishabh Joshi , Kate Olszewska , Carrie Muir , Mateo Wirth , Ale Jakse Hartman , Josh Newlan , Sheleem Kashem , Vijay Bolina , Elahe Dabir , Joost van Amersfoort , Zafarali Ahmed , James Cobon-Kerr , Aishwarya Kamath , Arnar Mar Hrafnkelsson , Le Hou , Ian Mackinnon , Alexandre Frechette , Eric Noland , Xiance Si , Emanuel Taropa , Dong Li , Phil Crone , Anmol Gulati , Sébastien Cevey , Jonas Adler , Ada Ma , David Silver , Simon Tokumine , Richard Powell , Stephan Lee , Michael Chang , Samer Hassan , Diana Mincu , Antoine Yang , Nir Levine , Jenny Brennan , Mingqiu Wang , Sarah Hodkinson , Jeffrey Zhao , Josh Lipschultz , Aedan Pope , Michael B. Chang , Cheng Li , Laurent El Shafey , Michela Paganini , Sholto Douglas , Bernd Bohnet , Fabio Pardo , Seth Odoom , Mihaela Rosca , Cicero Nogueira dos Santos , Kedar Soparkar , Arthur Guez , Tom Hudson , Steven Hansen , Chulayuth Asawaroengchai , Ravi Addanki , Tianhe Yu , Wojciech Stokowiec , Mina Khan , Justin Gilmer , Jaehoon Lee , Carrie Grimes Bostock , Keran Rong , Jonathan Caton , Pedram Pejman , Filip Pavetic , Geoff Brown , Vivek Sharma , Mario Lučić , Rajkumar Samuel , Josip Djolonga , Amol Mandhane , Lars Lowe Sjösund , Elena Buchatskaya , Elspeth White , Natalie Clay , Jiepu Jiang , Hyeontaek Lim , Ross Hemsley , Jane Labanowski , Nicola De Cao , David Steiner , Sayed Hadi Hashemi , Jacob Austin , Anita Gergely , Tim Blyth , Joe Stanton , Kaushik Shivakumar , Aditya Siddhant , Anders Andreassen , Carlos Araya , Nikhil Sethi , Rakesh Shivanna , Steven Hand , Ankur Bapna , Ali Khodaei , Antoine Miech , Garrett Tanzer , Andy Swing , Shantanu Thakoor , Zhufeng Pan , Zachary Nado , Stephanie Winkler , Dian Yu , Mohammad Saleh , Loren Maggiore , Iain Barr , Minh Giang , Thais Kagohara , Ivo Danihelka , Amit Marathe , Vladimir Feinberg , Mohamed Elhawaty , Nimesh Ghelani , Dan Horgan , Helen Miller , Lexi Walker , Richard Tanburn , Mukarram Tariq , Disha Shrivastava , Fei Xia , Chung-Cheng Chiu , Zoe Ashwood , Khuslen Baatarsukh , Sina Samangooei , Fred Alcober , Axel Stjerngren , Paul Komarek , Katerina Tsihlas , Anudhyan Boral , Ramona Comanescu , Jeremy Chen , Ruibo Liu , Dawn Bloxwich , Charlie Chen , Yanhua Sun , Fangxiaoyu Feng , Matthew Mauger , Xerxes Dotiwalla , Vincent Hellendoorn , Michael Sharman , Ivy Zheng , Krishna Haridasan , Gabe Barth-Maron , Craig Swanson , Dominika Rogozińska , Alek Andreev , Paul Kishan Rubenstein , Ruoxin Sang , Dan Hurt , Gamaleldin Elsayed , Renshen Wang , Dave Lacey , Anastasija Ilić , Yao Zhao , Lora Aroyo , Chimezie Iwuanyanwu , Vitaly Nikolaev , Balaji Lakshminarayanan , Sadegh Jazayeri , Raphaël Lopez Kaufman , Mani Varadarajan , Chetan Tekur , Doug Fritz , Misha Khalman , David Reitter , Kingshuk Dasgupta , Shourya Sarcar , Tina Ornduff , Javier Snaider , Fantine Huot , Johnson Jia , Rupert Kemp , Nejc Trdin , Anitha Vijayakumar , Lucy Kim , Christof Angermueller , Li Lao , Tianqi Liu , Haibin Zhang , David Engel , Somer Greene , Anaïs White , Jessica Austin , Lilly Taylor , Shereen Ashraf , Dangyi Liu , Maria Georgaki , Irene Cai , Yana Kulizhskaya , Sonam Goenka , Brennan Saeta , Kiran Vodrahalli , Christian Frank , Dario de Cesare , Brona Robenek , Harry Richardson , Mahmoud Alnahlawi , Christopher Yew , Priya Ponnapalli , Marco Tagliasacchi , Alex Korchemniy , Yelin Kim , Dinghua Li , Bill Rosgen , Zoe Ashwood , Kyle Levin , Jeremy Wiesner , Praseem Banzal , Praveen Srinivasan , Hongkun Yu , Çağlar Ünlü , David Reid , Zora Tung , Daniel Finchelstein , Ravin Kumar , Andre Elisseeff , Jin Huang , Ming Zhang , Rui Zhu , Ricardo Aguilar , Mai Giménez , Jiawei Xia , Olivier Dousse , Willi Gierke , Soheil Hassas Yeganeh , Damion Yates , Komal Jalan , Lu Li , Eri Latorre-Chimoto , Duc Dung Nguyen , Ken Durden , Praveen Kallakuri , Yaxin Liu , Matthew Johnson , Tomy Tsai , Alice Talbert , Jasmine Liu , Alexander Neitz , Chen Elkind , Marco Selvi , Mimi Jasarevic , Livio Baldini Soares , Albert Cui , Pidong Wang , Alek Wenjiao Wang , Xinyu Ye , Krystal Kallarackal , Lucia Loher , Hoi Lam , Josef Broder , Dan Holtmann-Rice , Nina Martin , Bramandia Ramadhana , Daniel Toyama , Mrinal Shukla , Sujoy Basu , Abhi Mohan , Nick Fernando , Noah Fiedel , Kim Paterson , Hui Li , Ankush Garg , Jane Park , DongHyun Choi , Diane Wu , Sankalp Singh , Zhishuai Zhang , Amir Globerson , Lily Yu , John Carpenter , Félix de Chaumont Quitry , Carey Radebaugh , Chu-Cheng Lin , Alex Tudor , Prakash Shroff , Drew Garmon , Dayou Du , Neera Vats , Han Lu , Shariq Iqbal , Alex Yakubovich , Nilesh Tripuraneni , James Manyika , Haroon Qureshi , Nan Hua , Christel Ngani , Maria Abi Raad , Hannah Forbes , Anna Bulanova , Jeff Stanway , Mukund Sundararajan , Victor Ungureanu , Colton Bishop , Yunjie Li , Balaji Venkatraman , Bo Li , Chloe Thornton , Salvatore Scellato , Nishesh Gupta , Yicheng Wang , Ian Tenney , Xihui Wu , Ashish Shenoy , Gabriel Carvajal , Diana Gage Wright , Ben Bariach , Zhuyun Xiao , Peter Hawkins , Sid Dalmia , Clement Farabet , Pedro Valenzuela , Quan Yuan , Chris Welty , Ananth Agarwal , Mia Chen , Wooyeol Kim , Brice Hulse , Nandita Dukkipati , Adam Paszke , Andrew Bolt , Elnaz Davoodi , Kiam Choo , Jennifer Beattie , Jennifer Prendki , Harsha Vashisht , Rebeca Santamaria-Fernandez , Luis C. Cobo , Jarek Wilkiewicz , David Madras , Ali Elqursh , Grant Uy , Kevin Ramirez , Matt Harvey , Tyler Liechty , Heiga Zen , Jeff Seibert , Clara Huiyi Hu , Mohamed Elhawaty , Andrey Khorlin , Maigo Le , Asaf Aharoni , Megan Li , Lily Wang , Sandeep Kumar , Alejandro Lince , Norman Casagrande , Jay Hoover , Dalia El Badawy , David Soergel , Denis Vnukov , Matt Miecnikowski , Jiri Simsa , Anna Koop , Praveen Kumar , Thibault Sellam , Daniel Vlasic , Samira Daruki , Nir Shabat , John Zhang , Guolong Su , Jiageng Zhang , Jeremiah Liu , Yi Sun , Evan Palmer , Alireza Ghaffarkhah , Xi Xiong , Victor Cotruta , Michael Fink , Lucas Dixon , Ashwin Sreevatsa , Adrian Goedeckemeyer , Alek Dimitriev , Mohsen Jafari , Remi Crocker , Nicholas FitzGerald , Aviral Kumar , Sanjay Ghemawat , Ivan Philips , Frederick Liu , Yannie Liang , Rachel Sterneck , Alena Repina , Marcus Wu , Laura Knight , Marin Georgiev , Hyo Lee , Harry Askham , Abhishek Chakladar , Annie Louis , Carl Crous , Hardie Cate , Dessie Petrova , Michael Quinn , Denese Owusu-Afriyie , Achintya Singhal , Nan Wei , Solomon Kim , Damien Vincent , Milad Nasr , Christopher A. Choquette-Choo , Reiko Tojo , Shawn Lu , Diego de Las Casas , Yuchung Cheng , Tolga Bolukbasi , Katherine Lee , Saaber Fatehi , Rajagopal Ananthanarayanan , Miteyan Patel , Charbel Kaed , Jing Li , Jakub Sygnowski , Shreyas Rammohan Belle , Zhe Chen , Jaclyn Konzelmann , Siim Põder , Roopal Garg , Vinod Koverkathu , Adam Brown , Chris Dyer , Rosanne Liu , Azade Nova , Jun Xu , Slav Petrov , Demis Hassabis , Koray Kavukcuoglu , Jeffrey Dean , Oriol Vinyals Share this with someone who'll enjoy it:
In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
View paper on
Share this with someone who'll enjoy it: