Kernel learning at the first level of inference

Cawley, G.C. ORCID: https://orcid.org/0000-0002-4118-9095 and Talbot, N.L.C. (2014) Kernel learning at the first level of inference. Neural Networks, 53. pp. 69-80. ISSN 0893-6080

Preview

PDF (nn2014a) - Accepted Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (302kB) | Preview

Abstract

Kernel learning methods, whether Bayesian or frequentist, typically involve multiple levels of inference, with the coefficients of the kernel expansion being determined at the first level and the kernel and regularisation parameters carefully tuned at the second level, a process known as model selection. Model selection for kernel machines is commonly performed via optimisation of a suitable model selection criterion, often based on cross-validation or theoretical performance bounds. However, if there are a large number of kernel parameters, as for instance in the case of automatic relevance determination (ARD), there is a substantial risk of over-fitting the model selection criterion, resulting in poor generalisation performance. In this paper we investigate the possibility of learning the kernel, for the Least-Squares Support Vector Machine (LS-SVM) classifier, at the first level of inference, i.e.parameter optimisation. The kernel parameters and the coefficients of the kernel expansion are jointly optimised at the first level of inference, minimising a training criterion with an additional regularisation term acting on the kernel parameters. The key advantage of this approach is that the values of only two regularisation parameters need be determined in model selection, substantially alleviating the problem of over-fitting the model selection criterion. The benefits of this approach are demonstrated using a suite of synthetic and real-world binary classification benchmark problems, where kernel learning at the first level of inference is shown to be statistically superior to the conventional approach, improves on our previous work (Cawley and Talbot, 2007) and is competitive with Multiple Kernel Learning approaches, but with reduced computational expense.

Item Type:	Article
Faculty \ School:	Faculty of Science > School of Computing Sciences
UEA Research Groups:	Faculty of Science > Research Groups > Computational Biology Faculty of Science > Research Groups > Data Science and Statistics Faculty of Science > Research Groups > Centre for Ocean and Atmospheric Sciences
Related URLs:	http://www.scopus.com/inward/record.url?...
Depositing User:	Pure Connector
Date Deposited:	09 Mar 2015 07:31
Last Modified:	18 Apr 2023 22:33
URI:	https://ueaeprints.uea.ac.uk/id/eprint/52500
DOI:	10.1016/j.neunet.2014.01.011

Downloads

Actions (login required)

View Item