Where academic tradition
meets the exciting future

You are here: TUCS > PUBLICATIONS > Publication Search > A Comparison of AUC Estimators...

A Comparison of AUC Estimators in Small-Sample Studies

Antti Airola, Tapio Pahikkala, Willem Waegeman, Bernard De Baets, Tapio Salakoski, A Comparison of AUC Estimators in Small-Sample Studies . In: Sašo Geurts Pierre Rousu Juho Džeroski (Ed.), Machine Learning in Systems Biology, JMLR Workshop and Conference Proceedings 8, 3–13, MIT Press, 2010.

Abstract:

Reliable estimation of the classification performance of learned predictive models is difficult,
when working in the small sample setting. When dealing with biological data it is often the case
that separate test data cannot be afforded. Cross-validation is in this case a typical strategy for estimating the performance. Recent results, further supported by experimental evidence presented in this article, show that many standard approaches to cross-validation suffer from extensive bias or variance when the area under ROC curve (AUC) is used as performance measure. We advocate the use of leave-pair-out cross-validation (LPOCV) for performance estimation, as it avoids many of these problems. A method previously proposed by us can be used to efficiently calculate this estimate for regularized least-squares (RLS) based learners.

Files:

Full publication in PDF-format

BibTeX entry:

@INPROCEEDINGS{inpAiPaWaDeSa10a,
  title = {A Comparison of AUC Estimators in Small-Sample Studies },
  booktitle = {Machine Learning in Systems Biology},
  author = {Airola, Antti and Pahikkala, Tapio and Waegeman, Willem and De Baets, Bernard and Salakoski, Tapio},
  volume = {8},
  series = {JMLR Workshop and Conference Proceedings},
  editor = {Džeroski, Sašo Geurts Pierre Rousu Juho},
  publisher = {MIT Press},
  pages = {3–13},
  year = {2010},
  keywords = {Area under the ROC curve, Classifier performance estimation, Conditional AUC estimation, Cross-validation, Leave-pair-out cross-validation},
}

Belongs to TUCS Research Unit(s): Algorithmics and Computational Intelligence Group (ACI), Turku BioNLP Group

Publication Forum rating of this publication: level 1

Edit publication