Empirical evaluation of resampling procedures for optimising SVM hyperparameters

Wainer, Jacques and Cawley, Gavin (2017) Empirical evaluation of resampling procedures for optimising SVM hyperparameters. Journal of Machine Learning Research, 18 (15). pp. 1-35. ISSN 1532-4435

[img] PDF (Accepted manuscript) - Submitted Version
Restricted to Repository staff only until 31 December 2099.

Download (915kB) | Request a copy
[img]
Preview
PDF (JMLR_2017_18_1) - Published Version
Download (403kB) | Preview

Abstract

Tuning the regularisation and kernel hyperparameters is a vital step in optimising the generalisation performance of kernel methods, such as the support vector machine (SVM). This is most often performed by minimising a resampling/cross-validation based model selection criterion, however there seems little practical guidance on the most suitable form of resampling. This paper presents the results of an extensive empirical evaluation of resampling procedures for SVM hyperparameter selection, designed to address this gap in the machine learning literature. Wetested 15 different resampling procedures on 121 binary classification data sets in order to select the best SVM hyperparameters. Weused three very different statistical procedures to analyse the results: the standard multi-classifier/multidata set procedure proposed by Demˇsar, the confidence intervals on the excess loss of each procedure in relation to 5-fold cross validation, and the Bayes factor analysis proposed by Barber. We conclude that a 2-fold procedure is appropriate to select the hyperparameters of an SVM for data sets for 1000or more datapoints, while a 3-fold procedure is appropriate for smaller data sets.

Item Type: Article
Uncontrolled Keywords: hyperparameters,svm,resampling,cross-validation,k-fold,bootstrap
Faculty \ School: Faculty of Science > School of Computing Sciences

University of East Anglia > Faculty of Science > Research Groups > Computational Biology (subgroups are shown below) > Machine learning in computational biology
Depositing User: Pure Connector
Date Deposited: 13 Jan 2017 00:05
Last Modified: 22 Apr 2020 02:21
URI: https://ueaeprints.uea.ac.uk/id/eprint/62020
DOI:

Actions (login required)

View Item View Item