Empirical evaluation of resampling procedures for optimising SVM hyperparameters

Tools

Wainer, Jacques and Cawley, Gavin ORCID: https://orcid.org/0000-0002-4118-9095 (2017) Empirical evaluation of resampling procedures for optimising SVM hyperparameters. Journal of Machine Learning Research, 18 (15). pp. 1-35. ISSN 1532-4435

Preview

PDF (JMLR_2017_18_1) - Published Version
Download (403kB) | Preview

Abstract

Tuning the regularisation and kernel hyperparameters is a vital step in optimising the generalisation performance of kernel methods, such as the support vector machine (SVM). This is most often performed by minimising a resampling/cross-validation based model selection criterion, however there seems little practical guidance on the most suitable form of resampling. This paper presents the results of an extensive empirical evaluation of resampling procedures for SVM hyperparameter selection, designed to address this gap in the machine learning literature. Wetested 15 different resampling procedures on 121 binary classification data sets in order to select the best SVM hyperparameters. Weused three very different statistical procedures to analyse the results: the standard multi-classifier/multidata set procedure proposed by Demˇsar, the confidence intervals on the excess loss of each procedure in relation to 5-fold cross validation, and the Bayes factor analysis proposed by Barber. We conclude that a 2-fold procedure is appropriate to select the hyperparameters of an SVM for data sets for 1000or more datapoints, while a 3-fold procedure is appropriate for smaller data sets.

Item Type:	Article
Uncontrolled Keywords:	hyperparameters,svm,resampling,cross-validation,k-fold,bootstrap
Faculty \ School:	Faculty of Science > School of Computing Sciences
UEA Research Groups:	Faculty of Science > Research Groups > Computational Biology Faculty of Science > Research Groups > Data Science and Statistics Faculty of Science > Research Groups > Centre for Ocean and Atmospheric Sciences
Related URLs:	https://jmlr.org/papers/v18/16-174.html
Depositing User:	Pure Connector
Date Deposited:	13 Jan 2017 00:05
Last Modified:	04 Aug 2023 16:30
URI:	https://ueaeprints.uea.ac.uk/id/eprint/62020
DOI:

Downloads

Downloads per month over past year

Actions (login required)

View Item