Tools
Tools
Wainer, Jacques and Cawley, Gavin 
ORCID: https://orcid.org/0000-0002-4118-9095
(2021)
Nested cross-validation when selecting classifiers is overzealous for most practical applications.
Expert Systems with Applications, 182.
ISSN 0957-4174
Preview |
PDF (Accepted_Manuscript)
- Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (1MB) | Preview |
Abstract
When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset and the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure: hyperparameter selection is performed in the inner cross-validation, while the outer cross-validation computes an unbiased estimate of the expected accuracy of the algorithm with cross-validation based hyperparameter tuning. The alternative approach, which we shall call “flat cross-validation”, uses a single cross-validation step both to select the optimal hyperparameter values and to provide an estimate of the expected accuracy of the algorithm that, while biased, may nevertheless still be used to select the best learning algorithm. We tested both procedures using 12 different algorithms on 115 real-life binary datasets and conclude that using the less computationally costly flat cross-validation procedure will generally result in the selection of an algorithm that is, for all practical purposes, of similar quality to that selected via nested cross-validation, provided the learning algorithms have relatively few hyperparameters to be optimised.
| Item Type: | Article |
|---|---|
| Faculty \ School: | Faculty of Science > School of Computing Sciences |
| UEA Research Groups: | Faculty of Science > Research Groups > Centre for Ocean and Atmospheric Sciences Faculty of Science > Research Groups > Computational Biology Faculty of Science > Research Groups > Data Science and Statistics |
| Related URLs: | |
| Depositing User: | LivePure Connector |
| Date Deposited: | 19 May 2021 00:09 |
| Last Modified: | 20 Apr 2023 21:32 |
| URI: | https://ueaeprints.uea.ac.uk/id/eprint/80048 |
| DOI: | 10.1016/j.eswa.2021.115222 |
Downloads
Downloads per month over past year
Actions (login required)
 |
View Item |