Get our free extension to see links to code for papers anywhere online!

 Add to Chrome

 Add to Firefox

CatalyzeX Code Finder - Browser extension linking code for ML papers across the web! | Product Hunt Embed

Training Deeper Neural Machine Translation Models with Transparent Attention

Sep 04, 2018
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, Yonghui Wu



While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.

* To appear in EMNLP 2018 


Share this with someone who'll enjoy it:

   Access Paper Source



Share this with someone who'll enjoy it: