Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox


Training Deeper Neural Machine Translation Models with Transparent Attention

Sep 04, 2018
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, Yonghui Wu


Share this with someone who'll enjoy it:


While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.

* To appear in EMNLP 2018 


   Access Paper Source



Share this with someone who'll enjoy it: