Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox


HINT3: Raising the bar for Intent Detection in the Wild

Sep 29, 2020
Gaurav Arora, Chirag Jain, Manas Chaturvedi, Krupal Modi



Intent Detection systems in the real world are exposed to complexities of imbalanced datasets containing varying perception of intent, unintended correlations and domain-specific aberrations. To facilitate benchmarking which can reflect near real-world scenarios, we introduce 3 new datasets created from live chatbots in diverse domains. Unlike most existing datasets that are crowdsourced, our datasets contain real user queries received by the chatbots and facilitates penalising unwanted correlations grasped during the training process. We evaluate 4 NLU platforms and a BERT based classifier and find that performance saturates at inadequate levels on test sets because all systems latch on to unintended patterns in training data.

* Accepted to EMNLP-2020's Insights workshop. Camera ready version is WIP 


Share this with someone who'll enjoy it:

   Access Paper Source



Share this with someone who'll enjoy it: