详情

原文传递 Estimating Classifier Accuracy Using Noisy Expert Labels.

题名：	Estimating Classifier Accuracy Using Noisy Expert Labels.
作者：	Holodnak, J. T.; Matterer, J. T.; Streilein, W. W.
关键词：	Algorithms, Artificial intelligence software, Bayesian networks, Simulations, Social media, Estimators, Artificial intelligence, Information processing, Information science, Computational science, Machine learning, Supervised machine learning, Iarpa collections, Nlp(natural language processing), Mv(majority vote), Agr(agreement-based estimators), Opt(optimization), Cov(covariance-based estimator)
摘要：	In this work, we present an empirical comparison of statistical methods that estimate the accuracy of a classifier using noisy expert labels. We are motivated by the application of machine learning to difficult problems for which even experts may be unable to provide an authoritative label for every data instance. Several estimators have been recently proposed in the literature, but prior empirical work to evaluate the applicability of these estimators to real-world problems is limited. We apply the estimators to labels simulated from three models of the expert labeling process and also four real datasets labeled by human experts. Our simulations reveal the importance of the accuracy of the classifier relative to the experts and confirm that conditional dependence between experts negatively impacts estimator performance. On two of the real datasets, the estimators clearly outperformed the baseline majority vote estimator, supporting their use in applications. We also briefly examine the utility, in terms of increasing or decreasing confidence in an estimators output, of a few diagnostics that can be applied to the expert labels.
报告类型：	科技报告