Alert button

AI and the Everything in the Whole Wide World Benchmark

Nov 26, 2021
Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, Alex Hanna

Figure 1 for AI and the Everything in the Whole Wide World Benchmark

Share this with someone who'll enjoy it:

There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks. These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems. State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals. In this position paper, we explore the limits of such benchmarks in order to reveal the construct validity issues in their framing as the functionally "general" broad measures of progress they are set up to be.

* Accepted in NeurIPS 2021 Benchmarks and Datasets track  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it: