Alert button
Picture for Eva Vanmassenhove

Eva Vanmassenhove

Alert button

Tailoring Domain Adaptation for Machine Translation Quality Estimation

Apr 18, 2023
Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, Frédéric Blain, Eva Vanmassenhove, Mirella De Sisto, Chris Emmery, Pieter Spronck

Figure 1 for Tailoring Domain Adaptation for Machine Translation Quality Estimation
Figure 2 for Tailoring Domain Adaptation for Machine Translation Quality Estimation
Figure 3 for Tailoring Domain Adaptation for Machine Translation Quality Estimation
Figure 4 for Tailoring Domain Adaptation for Machine Translation Quality Estimation

While quality estimation (QE) can play an important role in the translation process, its effectiveness relies on the availability and quality of training data. For QE in particular, high-quality labeled data is often lacking due to the high-cost and effort associated with labeling such data. Aside from the data scarcity challenge, QE models should also be generalizable, i.e., they should be able to handle data from different domains, both generic and specific. To alleviate these two main issues -- data scarcity and domain mismatch -- this paper combines domain adaptation and data augmentation within a robust QE system. Our method is to first train a generic QE model and then fine-tune it on a specific domain while retaining generic knowledge. Our results show a significant improvement for all the language pairs investigated, better cross-lingual inference, and a superior performance in zero-shot learning scenarios as compared to state-of-the-art baselines.

* Accepted to EAMT 2023 (main) 
Viaarxiv icon

The Ecological Footprint of Neural Machine Translation Systems

Feb 04, 2022
Dimitar Sherionov, Eva Vanmassenhove

Figure 1 for The Ecological Footprint of Neural Machine Translation Systems
Figure 2 for The Ecological Footprint of Neural Machine Translation Systems
Figure 3 for The Ecological Footprint of Neural Machine Translation Systems
Figure 4 for The Ecological Footprint of Neural Machine Translation Systems

Over the past decade, deep learning (DL) has led to significant advancements in various fields of artificial intelligence, including machine translation (MT). These advancements would not be possible without the ever-growing volumes of data and the hardware that allows large DL models to be trained efficiently. Due to the large amount of computing cores as well as dedicated memory, graphics processing units (GPUs) are a more effective hardware solution for training and inference with DL models than central processing units (CPUs). However, the former is very power demanding. The electrical power consumption has economical as well as ecological implications. This chapter focuses on the ecological footprint of neural MT systems. It starts from the power drain during the training of and the inference with neural MT models and moves towards the environment impact, in terms of carbon dioxide emissions. Different architectures (RNN and Transformer) and different GPUs (consumer-grate NVidia 1080Ti and workstation-grade NVidia P100) are compared. Then, the overall CO2 offload is calculated for Ireland and the Netherlands. The NMT models and their ecological impact are compared to common household appliances to draw a more clear picture. The last part of this chapter analyses quantization, a technique for reducing the size and complexity of models, as a way to reduce power consumption. As quantized models can run on CPUs, they present a power-efficient inference solution without depending on a GPU.

* 25 pages, 3 figures, 10 tables 
Viaarxiv icon

NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives

Sep 13, 2021
Eva Vanmassenhove, Chris Emmery, Dimitar Shterionov

Figure 1 for NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives
Figure 2 for NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives
Figure 3 for NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives
Figure 4 for NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives

Recent years have seen an increasing need for gender-neutral and inclusive language. Within the field of NLP, there are various mono- and bilingual use cases where gender inclusive language is appropriate, if not preferred due to ambiguity or uncertainty in terms of the gender of referents. In this work, we present a rule-based and a neural approach to gender-neutral rewriting for English along with manually curated synthetic data (WinoBias+) and natural data (OpenSubtitles and Reddit) benchmarks. A detailed manual and automatic evaluation highlights how our NeuTral Rewriter, trained on data generated by the rule-based approach, obtains word error rates (WER) below 0.18% on synthetic, in-domain and out-domain test sets.

Viaarxiv icon

GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena

Aug 31, 2021
Eva Vanmassenhove, Johanna Monti

Languages differ in terms of the absence or presence of gender features, the number of gender classes and whether and where gender features are explicitly marked. These cross-linguistic differences can lead to ambiguities that are difficult to resolve, especially for sentence-level MT systems. The identification of ambiguity and its subsequent resolution is a challenging task for which currently there aren't any specific resources or challenge sets available. In this paper, we introduce gENder-IT, an English--Italian challenge set focusing on the resolution of natural gender phenomena by providing word-level gender tags on the English source side and multiple gender alternative translations, where needed, on the Italian target side.

Viaarxiv icon

Generating Gender Augmented Data for NLP

Jul 13, 2021
Nishtha Jain, Maja Popovic, Declan Groves, Eva Vanmassenhove

Figure 1 for Generating Gender Augmented Data for NLP
Figure 2 for Generating Gender Augmented Data for NLP
Figure 3 for Generating Gender Augmented Data for NLP
Figure 4 for Generating Gender Augmented Data for NLP

Gender bias is a frequent occurrence in NLP-based applications, especially pronounced in gender-inflected languages. Bias can appear through associations of certain adjectives and animate nouns with the natural gender of referents, but also due to unbalanced grammatical gender frequencies of inflected words. This type of bias becomes more evident in generating conversational utterances where gender is not specified within the sentence, because most current NLP applications still work on a sentence-level context. As a step towards more inclusive NLP, this paper proposes an automatic and generalisable rewriting approach for short conversational sentences. The rewriting method can be applied to sentences that, without extra-sentential context, have multiple equivalent alternatives in terms of gender. The method can be applied both for creating gender balanced outputs as well as for creating gender balanced training data. The proposed approach is based on a neural machine translation (NMT) system trained to 'translate' from one gender alternative to another. Both the automatic and manual analysis of the approach show promising results for automatic generation of gender alternatives for conversational sentences in Spanish.

* 10 pages, 4 tables 
Viaarxiv icon

Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation

Jan 30, 2021
Eva Vanmassenhove, Dimitar Shterionov, Matthew Gwilliam

Figure 1 for Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation
Figure 2 for Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation
Figure 3 for Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation
Figure 4 for Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation

Recent studies in the field of Machine Translation (MT) and Natural Language Processing (NLP) have shown that existing models amplify biases observed in the training data. The amplification of biases in language technology has mainly been examined with respect to specific phenomena, such as gender bias. In this work, we go beyond the study of gender in MT and investigate how bias amplification might affect language in a broader sense. We hypothesize that the 'algorithmic bias', i.e. an exacerbation of frequently observed patterns in combination with a loss of less frequent ones, not only exacerbates societal biases present in current datasets but could also lead to an artificially impoverished language: 'machine translationese'. We assess the linguistic richness (on a lexical and morphological level) of translations created by different data-driven MT paradigms - phrase-based statistical (PB-SMT) and neural MT (NMT). Our experiments show that there is a loss of lexical and morphological richness in the translations produced by all investigated MT paradigms for two language pairs (EN<=>FR and EN<=>ES).

Viaarxiv icon

On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation

Mar 31, 2020
Eva Vanmassenhove

Figure 1 for On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation
Figure 2 for On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation
Figure 3 for On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation
Figure 4 for On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation

New machine translations (MT) technologies are emerging rapidly and with them, bold claims of achieving human parity such as: (i) the results produced approach "accuracy achieved by average bilingual human translators" (Wu et al., 2017b) or (ii) the "translation quality is at human parity when compared to professional human translators" (Hassan et al., 2018) have seen the light of day (Laubli et al., 2018). Aside from the fact that many of these papers craft their own definition of human parity, these sensational claims are often not supported by a complete analysis of all aspects involved in translation. Establishing the discrepancies between the strengths of statistical approaches to MT and the way humans translate has been the starting point of our research. By looking at MT output and linguistic theory, we were able to identify some remaining issues. The problems range from simple number and gender agreement errors to more complex phenomena such as the correct translation of aspectual values and tenses. Our experiments confirm, along with other studies (Bentivogli et al., 2016), that neural MT has surpassed statistical MT in many aspects. However, some problems remain and others have emerged. We cover a series of problems related to the integration of specific linguistic features into statistical and neural MT, aiming to analyse and provide a solution to some of them. Our work focuses on addressing three main research questions that revolve around the complex relationship between linguistics and MT in general. We identify linguistic information that is lacking in order for automatic translation systems to produce more accurate translations and integrate additional features into the existing pipelines. We identify overgeneralization or 'algorithmic bias' as a potential drawback of neural MT and link it to many of the remaining linguistic issues.

Viaarxiv icon

Getting Gender Right in Neural Machine Translation

Sep 11, 2019
Eva Vanmassenhove, Christian Hardmeier, Andy Way

Figure 1 for Getting Gender Right in Neural Machine Translation
Figure 2 for Getting Gender Right in Neural Machine Translation
Figure 3 for Getting Gender Right in Neural Machine Translation
Figure 4 for Getting Gender Right in Neural Machine Translation

Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996). One such difference is related to the way gender is expressed in a language. Saying "I am happy" in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either "Je suis heureux", for a male speaker or "Je suis heureuse" for a female one. Apart from morphological agreement, demographic factors (gender, age, etc.) also influence our use of language in terms of word choices or even on the level of syntactic constructions (Tannen, 1991; Pennebaker et al., 2003). We integrate gender information into NMT systems. Our contribution is two-fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.

* Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing  
* Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), October-November, 2018. Brussels, Belgium, pages 3003-3008, URL: https://www.aclweb.org/anthology/D18-1334, DOI: 10.18653/v1/D18-1334 
Viaarxiv icon

Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation

Jun 28, 2019
Eva Vanmassenhove, Dimitar Shterionov, Andy Way

Figure 1 for Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation
Figure 2 for Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation
Figure 3 for Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation
Figure 4 for Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation

This work presents an empirical approach to quantifying the loss of lexical richness in Machine Translation (MT) systems compared to Human Translation (HT). Our experiments show how current MT systems indeed fail to render the lexical diversity of human generated or translated text. The inability of MT systems to generate diverse outputs and its tendency to exacerbate already frequent patterns while ignoring less frequent ones, might be the underlying cause for, among others, the currently heavily debated issues related to gender biased output. Can we indeed, aside from biased data, talk about an algorithm that exacerbates seen biases?

* Accepted for publication at the 17th Machine Translation Summit (MTSummit2019), Dublin, Ireland, August 2019 
Viaarxiv icon