Alert button
Picture for Peter Grasch

Peter Grasch

Alert button

Model Stability with Continuous Data Updates

Jan 14, 2022
Huiting Liu, Avinesh P. V. S., Siddharth Patwardhan, Peter Grasch, Sachin Agarwal

Figure 1 for Model Stability with Continuous Data Updates
Figure 2 for Model Stability with Continuous Data Updates
Figure 3 for Model Stability with Continuous Data Updates
Figure 4 for Model Stability with Continuous Data Updates

In this paper, we study the "stability" of machine learning (ML) models within the context of larger, complex NLP systems with continuous training data updates. For this study, we propose a methodology for the assessment of model stability (which we refer to as jitter under various experimental conditions. We find that model design choices, including network architecture and input representation, have a critical impact on stability through experiments on four text classification tasks and two sequence labeling tasks. In classification tasks, non-RNN-based models are observed to be more stable than RNN-based ones, while the encoder-decoder model is less stable in sequence labeling tasks. Moreover, input representations based on pre-trained fastText embeddings contribute to more stability than other choices. We also show that two learning strategies -- ensemble models and incremental training -- have a significant influence on stability. We recommend ML model designers account for trade-offs in accuracy and jitter when making modeling choices.

Viaarxiv icon

Noise-robust Named Entity Understanding for Virtual Assistants

May 29, 2020
Deepak Muralidharan, Joel Ruben Antony Moniz, Sida Gao, Xiao Yang, Lin Li, Justine Kao, Stephen Pulman, Atish Kothari, Ray Shen, Yinying Pan, Vivek Kaul, Mubarak Seyed Ibrahim, Gang Xiang, Nan Dun, Yidan Zhou, Andy O, Yuan Zhang, Pooja Chitkara Xuan Wang, Alkesh Patel, Kushal Tayal, Roger Zheng, Peter Grasch, Jason Williams

Figure 1 for Noise-robust Named Entity Understanding for Virtual Assistants
Figure 2 for Noise-robust Named Entity Understanding for Virtual Assistants
Figure 3 for Noise-robust Named Entity Understanding for Virtual Assistants
Figure 4 for Noise-robust Named Entity Understanding for Virtual Assistants

Named Entity Understanding (NEU) plays an essential role in interactions between users and voice assistants, since successfully identifying entities and correctly linking them to their standard forms is crucial to understanding the user's intent. NEU is a challenging task in voice assistants due to the ambiguous nature of natural language and because noise introduced by speech transcription and user errors occur frequently in spoken natural language queries. In this paper, we propose an architecture with novel features that jointly solves the recognition of named entities (a.k.a. Named Entity Recognition, or NER) and the resolution to their canonical forms (a.k.a. Entity Linking, or EL). We show that by combining NER and EL information in a joint reranking module, our proposed framework improves accuracy in both tasks. This improved performance and the features that enable it, also lead to better accuracy in downstream tasks, such as domain classification and semantic parsing.

* 9 pages 
Viaarxiv icon