Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox


LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing

May 05, 2021
Yu Li, Josh Arnold, Feifan Yan, Weiyan Shi, Zhou Yu


Share this with someone who'll enjoy it:


We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.



   Access Paper Source



Share this with someone who'll enjoy it: