Evaluating Systemic Error Detection Methods using Synthetic Images

teaser

We introduce SpotCheck, a framework for generating synthetic datasets to use for evaluating methods for discovering blindspots (i.e., systemic errors) in image classifiers. We use SpotCheck to run controlled studies of how various factors influence the performance of blindspot discovery methods. Our experiments reveal several shortcomings of existing methods, such as relatively poor performance in settings with multiple blindspots and sensitivity to hyperparameters. Further, we find that a method based on dimensionality reduction, PlaneSpot, is competitive with existing methods, which has promising implications for the development of interactive tools.

Citation

Evaluating Systemic Error Detection Methods using Synthetic Images
Gregory Plumb, Nari Johnson, Ángel Alexander Cabrera, Marco Tulio Ribeiro, Ameet Talwalkar
ICML - Workshop on Spurious Correlations, Invariance and Stability. Baltimore, MD, 2022.

BibTex

@inproceedings{plumb2020expo, author = {Plumb, Gregory and Johnson, Nari and Cabrera, \'{A}ngel Alexander and Ribeiro, Marco Tulio and Talwalkar, Ameet}, title = {Evaluating Systemic Error Detection Methods using Synthetic Images}, year = {2022}, booktitle = {Workshop on Spurious Correlations, Invariance and Stability at International Conference on Machine Learning}, }