Alert button

Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification

May 24, 2023
Mujeen Sung, James Gung, Elman Mansimov, Nikolaos Pappas, Raphael Shu, Salvatore Romeo, Yi Zhang, Vittorio Castelli

Figure 1 for Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification
Figure 2 for Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification
Figure 3 for Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification
Figure 4 for Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification

Share this with someone who'll enjoy it:

Intent classification (IC) plays an important role in task-oriented dialogue systems as it identifies user intents from given utterances. However, models trained on limited annotations for IC often suffer from a lack of generalization to unseen intent classes. We propose a novel pre-training method for text encoders that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks. By applying this pre-training strategy, we also introduce the pre-trained intent-aware encoder (PIE). Specifically, we first train a tagger to identify key phrases within utterances that are crucial for interpreting intents. We then use these extracted phrases to create examples for pre-training a text encoder in a contrastive manner. As a result, our PIE model achieves up to 5.4% and 4.0% higher accuracy than the previous state-of-the-art pre-trained sentence encoder for the N-way zero- and one-shot settings on four IC datasets.

View paper onarxiv icon

Share this with someone who'll enjoy it: