Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox


HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Jun 01, 2021
Weixin Liang, Kai-Hui Liang, Zhou Yu


Share this with someone who'll enjoy it:


Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an annotation efficient framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manual labeling training samples, we first use a set of labeling heuristics to automatically label training samples. We then denoise the weakly labeled data using Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.

* ACL 2021. Code & data available at https://github.com/Weixin-Liang/HERALD/ 


   Access Paper Source



Share this with someone who'll enjoy it: