Alert button
Picture for Xing Niu

Xing Niu

Alert button

RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation

May 26, 2023
Gabriele Sarti, Phu Mon Htut, Xing Niu, Benjamin Hsu, Anna Currey, Georgiana Dinu, Maria Nadejde

Figure 1 for RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation
Figure 2 for RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation
Figure 3 for RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation
Figure 4 for RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation

Attribute-controlled translation (ACT) is a subtask of machine translation that involves controlling stylistic or linguistic attributes (like formality and gender) of translation outputs. While ACT has garnered attention in recent years due to its usefulness in real-world applications, progress in the task is currently limited by dataset availability, since most prior approaches rely on supervised methods. To address this limitation, we propose Retrieval and Attribute-Marking enhanced Prompting (RAMP), which leverages large multilingual language models to perform ACT in few-shot and zero-shot settings. RAMP improves generation accuracy over the standard prompting approach by (1) incorporating a semantic similarity retrieval component for selecting similar in-context examples, and (2) marking in-context examples with attribute annotations. Our comprehensive experiments show that RAMP is a viable approach in both zero-shot and few-shot settings.

* Accepted at ACL 2023 
Viaarxiv icon

Pseudo-Label Training and Model Inertia in Neural Machine Translation

May 19, 2023
Benjamin Hsu, Anna Currey, Xing Niu, Maria Nădejde, Georgiana Dinu

Figure 1 for Pseudo-Label Training and Model Inertia in Neural Machine Translation
Figure 2 for Pseudo-Label Training and Model Inertia in Neural Machine Translation
Figure 3 for Pseudo-Label Training and Model Inertia in Neural Machine Translation
Figure 4 for Pseudo-Label Training and Model Inertia in Neural Machine Translation

Like many other machine learning applications, neural machine translation (NMT) benefits from over-parameterized deep neural models. However, these models have been observed to be brittle: NMT model predictions are sensitive to small input changes and can show significant variation across re-training or incremental model updates. This work studies a frequently used method in NMT, pseudo-label training (PLT), which is common to the related techniques of forward-translation (or self-training) and sequence-level knowledge distillation. While the effect of PLT on quality is well-documented, we highlight a lesser-known effect: PLT can enhance a model's stability to model updates and input perturbations, a set of properties we call model inertia. We study inertia effects under different training settings and we identify distribution simplification as a mechanism behind the observed results.

* accepted ICLR 2023 
Viaarxiv icon

MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

Nov 02, 2022
Anna Currey, Maria Nădejde, Raghavendra Pappagari, Mia Mayer, Stanislas Lauly, Xing Niu, Benjamin Hsu, Georgiana Dinu

Figure 1 for MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation
Figure 2 for MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation
Figure 3 for MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation
Figure 4 for MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased. In particular, gender accuracy in translation can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MT-GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license.

* Accepted at EMNLP 2022. Data and code: https://github.com/amazon-research/machine-translation-gender-eval 
Viaarxiv icon

Sockeye 3: Fast Neural Machine Translation with PyTorch

Jul 12, 2022
Felix Hieber, Michael Denkowski, Tobias Domhan, Barbara Darques Barros, Celina Dong Ye, Xing Niu, Cuong Hoang, Ke Tran, Benjamin Hsu, Maria Nadejde, Surafel Lakew, Prashant Mathur, Marcello Federico, Anna Currey

Figure 1 for Sockeye 3: Fast Neural Machine Translation with PyTorch
Figure 2 for Sockeye 3: Fast Neural Machine Translation with PyTorch
Figure 3 for Sockeye 3: Fast Neural Machine Translation with PyTorch
Figure 4 for Sockeye 3: Fast Neural Machine Translation with PyTorch

Sockeye 3 is the latest version of the Sockeye toolkit for Neural Machine Translation (NMT). Now based on PyTorch, Sockeye 3 provides faster model implementations and more advanced features with a further streamlined codebase. This enables broader experimentation with faster iteration, efficient training of stronger and faster models, and the flexibility to move new ideas quickly from research to production. When running comparable models, Sockeye 3 is up to 126% faster than other PyTorch implementations on GPUs and up to 292% faster on CPUs. Sockeye 3 is open source software released under the Apache 2.0 license.

Viaarxiv icon

CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality

May 09, 2022
Maria Nădejde, Anna Currey, Benjamin Hsu, Xing Niu, Marcello Federico, Georgiana Dinu

Figure 1 for CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality
Figure 2 for CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality
Figure 3 for CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality
Figure 4 for CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality

The machine translation (MT) task is typically formulated as that of returning a single translation for an input segment. However, in many cases, multiple different translations are valid and the appropriate translation may depend on the intended target audience, characteristics of the speaker, or even the relationship between speakers. Specific problems arise when dealing with honorifics, particularly translating from English into languages with formality markers. For example, the sentence "Are you sure?" can be translated in German as "Sind Sie sich sicher?" (formal register) or "Bist du dir sicher?" (informal). Using wrong or inconsistent tone may be perceived as inappropriate or jarring for users of certain cultures and demographics. This work addresses the problem of learning to control target language attributes, in this case formality, from a small amount of labeled contrastive data. We introduce an annotated dataset (CoCoA-MT) and an associated evaluation metric for training and evaluating formality-controlled MT models for six diverse target languages. We show that we can train formality-controlled models by fine-tuning on labeled contrastive data, achieving high accuracy (82% in-domain and 73% out-of-domain) while maintaining overall quality.

* NAACL 2022 
Viaarxiv icon

Faithful Target Attribute Prediction in Neural Machine Translation

Sep 24, 2021
Xing Niu, Georgiana Dinu, Prashant Mathur, Anna Currey

Figure 1 for Faithful Target Attribute Prediction in Neural Machine Translation
Figure 2 for Faithful Target Attribute Prediction in Neural Machine Translation
Figure 3 for Faithful Target Attribute Prediction in Neural Machine Translation
Figure 4 for Faithful Target Attribute Prediction in Neural Machine Translation

The training data used in NMT is rarely controlled with respect to specific attributes, such as word casing or gender, which can cause errors in translations. We argue that predicting the target word and attributes simultaneously is an effective way to ensure that translations are more faithful to the training data distribution with respect to these attributes. Experimental results on two tasks, uppercased input translation and gender prediction, show that this strategy helps mirror the training data distribution in testing. It also facilitates data augmentation on the task of uppercased input translation.

* Withdrawn from Findings of ACL 2021 
Viaarxiv icon

Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation

Oct 07, 2020
Weijia Xu, Xing Niu, Marine Carpuat

Figure 1 for Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation
Figure 2 for Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation
Figure 3 for Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation
Figure 4 for Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation

While Iterative Back-Translation and Dual Learning effectively incorporate monolingual training data in neural machine translation, they use different objectives and heuristic gradient approximation strategies, and have not been extensively compared. We introduce a novel dual reconstruction objective that provides a unified view of Iterative Back-Translation and Dual Learning. It motivates a theoretical analysis and controlled empirical study on German-English and Turkish-English tasks, which both suggest that Iterative Back-Translation is more effective than Dual Learning despite its relative simplicity.

* Accepted at Findings of EMNLP 2020 
Viaarxiv icon

The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020

Aug 11, 2020
Tobias Domhan, Michael Denkowski, David Vilar, Xing Niu, Felix Hieber, Kenneth Heafield

Figure 1 for The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020
Figure 2 for The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020
Figure 3 for The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020
Figure 4 for The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020

We present Sockeye 2, a modernized and streamlined version of the Sockeye neural machine translation (NMT) toolkit. New features include a simplified code base through the use of MXNet's Gluon API, a focus on state of the art model architectures, distributed mixed precision training, and efficient CPU decoding with 8-bit quantization. These improvements result in faster training and inference, higher automatic metric scores, and a shorter path from research to production.

Viaarxiv icon

Evaluating Robustness to Input Perturbations for Neural Machine Translation

May 01, 2020
Xing Niu, Prashant Mathur, Georgiana Dinu, Yaser Al-Onaizan

Figure 1 for Evaluating Robustness to Input Perturbations for Neural Machine Translation
Figure 2 for Evaluating Robustness to Input Perturbations for Neural Machine Translation
Figure 3 for Evaluating Robustness to Input Perturbations for Neural Machine Translation
Figure 4 for Evaluating Robustness to Input Perturbations for Neural Machine Translation

Neural Machine Translation (NMT) models are sensitive to small perturbations in the input. Robustness to such perturbations is typically measured using translation quality metrics such as BLEU on the noisy input. This paper proposes additional metrics which measure the relative degradation and changes in translation when small perturbations are added to the input. We focus on a class of models employing subword regularization to address robustness and perform extensive evaluations of these models using the robustness measures proposed. Results show that our proposed metrics reveal a clear trend of improved robustness to perturbations when subword regularization methods are used.

* Accepted at ACL 2020 
Viaarxiv icon

Controlling Neural Machine Translation Formality with Synthetic Supervision

Nov 28, 2019
Xing Niu, Marine Carpuat

Figure 1 for Controlling Neural Machine Translation Formality with Synthetic Supervision
Figure 2 for Controlling Neural Machine Translation Formality with Synthetic Supervision
Figure 3 for Controlling Neural Machine Translation Formality with Synthetic Supervision
Figure 4 for Controlling Neural Machine Translation Formality with Synthetic Supervision

This work aims to produce translations that convey source language content at a formality level that is appropriate for a particular audience. Framing this problem as a neural sequence-to-sequence task ideally requires training triplets consisting of a bilingual sentence pair labeled with target language formality. However, in practice, available training examples are limited to English sentence pairs of different styles, and bilingual parallel sentences of unknown formality. We introduce a novel training scheme for multi-task models that automatically generates synthetic training triplets by inferring the missing element on the fly, thus enabling end-to-end training. Comprehensive automatic and human assessments show that our best model outperforms existing models by producing translations that better match desired formality levels while preserving the source meaning.

* Accepted at AAAI 2020 
Viaarxiv icon