Greedy Ordering of Layer Weight Matrices in Transformers Improves Translation

Prior work has attempted to understand the internal structures and functionalities of Transformer-based encoder-decoder architectures on the level of multi-head attention and feed-forward sublayers. Interpretations have focused on the encoder and decoder, along with the combinatorial possibilities of the self-attention, cross-attention, and feed-forward sublayers. However, without examining the low-level structures, one gains limited understanding of the motivation behind sublayer reordering. Could we dive into the sublayer abstraction and permute layer weight matrices to improve the quality of translation? We propose AEIUOrder to greedily reorder layer weight matrices in the encoder by their well-trainedness, as measured by Heavy-Tailed Self-Regularization (HT-SR) metrics, and order the decoder matrices correspondingly. Our results suggest that greedily reordering layer weight matrices to maximize Total well-trainedness facilitates the model to learn representations and generate translations more effectively.

Weight, Is Attention All We Need? AEIUOrder: Greedy Ordering of Layer Weight Matrices in Transformer Improves Translation

Prior work has attempted to understand the internal structures and functionalities of Transformer-based encoder-decoder architectures on the level of multi-head attention and feed-forward sublayers. Interpretations have focused on the encoder and decoder, along with the combinatorial possibilities of the self-attention, cross-attention, and feed-forward sublayers. Could we improve the quality of translation by diving into the Transformer sublayer abstractions and permuting its layer weight matrices? We propose AEIUOrder to greedily reorder layer weight matrices in the encoder by their well-trainedness, as measured by Random Matrix Theory (RMT) metrics, and reverse the ordering scheme for the encoder. The objective is to maximize Total well-trainedness in the encoder while the decoder structure serves to represent the reverse process of encoding. On the standard Transformer (6 layers, model dimension 512), AEIUOrder achieves a BLEU score of 34.62 (baseline 34.31) on the IWSLT 2016 German-to-English translation task, and 27.95 BLEU on the WMT 2014 English-to-German translation task (baseline 27.91). AEIUOrder is also realized on Transformers with various depths and embedding dimensions, showing significant improvements on deeper, wider models than on their shallower, slimmer counterparts. For instance, the 8-layer, 768-dimension and the 4-layer, 1024-dimension Transformers achieve respective 29.1 and 29.31 BLEU scores on the IWSLT 2016 English-to-German translation task (28.53 and 28.97 on respective baselines). Our results suggest that the RMT-motivated approach to maximize \textit{Total well-trainedness}, by greedily reordering its layer weight matrices, facilitates the model to learn representations and generate translations more effectively.

NewsPod: Automatic and Interactive News Podcasts

News podcasts are a popular medium to stay informed and dive deep into news topics. Today, most podcasts are handcrafted by professionals. In this work, we advance the state-of-the-art in automatically generated podcasts, making use of recent advances in natural language processing and text-to-speech technology. We present NewsPod, an automatically generated, interactive news podcast. The podcast is divided into segments, each centered on a news event, with each segment structured as a Question and Answer conversation, whose goal is to engage the listener. A key aspect of the design is the use of distinct voices for each role (questioner, responder), to better simulate a conversation. Another novel aspect of NewsPod allows listeners to interact with the podcast by asking their own questions and receiving automatically generated answers. We validate the soundness of this system design through two usability studies, focused on evaluating the narrative style and interactions with the podcast, respectively. We find that NewsPod is preferred over a baseline by participants, with 80% claiming they would use the system in the future.