Alert button

Thinking Beyond Distributions in Testing Machine Learned Models

Dec 06, 2021
Negar Rostamzadeh, Ben Hutchinson, Christina Greer, Vinodkumar Prabhakaran

Share this with someone who'll enjoy it:

Testing practices within the machine learning (ML) community have centered around assessing a learned model's predictive performance measured against a test dataset, often drawn from the same distribution as the training dataset. While recent work on robustness and fairness testing within the ML community has pointed to the importance of testing against distributional shifts, these efforts also focus on estimating the likelihood of the model making an error against a reference dataset/distribution. We argue that this view of testing actively discourages researchers and developers from looking into other sources of robustness failures, for instance corner cases which may have severe undesirable impacts. We draw parallels with decades of work within software engineering testing focused on assessing a software system against various stress conditions, including corner cases, as opposed to solely focusing on average-case behaviour. Finally, we put forth a set of recommendations to broaden the view of machine learning testing to a rigorous practice.

* NeurIPS 2021 workshop on Distribution Shifts  * Neural Information Processing System, NeurIPS 2021 workshop on Distribution Shifts  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it: