Kernel learning at the first level of inference

Cawley, G.C. ORCID: https://orcid.org/0000-0002-4118-9095 and Talbot, N.L.C. (2014) Kernel learning at the first level of inference. Neural Networks, 53. pp. 69-80. ISSN 0893-6080

[thumbnail of nn2014a]
Preview
PDF (nn2014a) - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives.

Download (302kB) | Preview

Abstract

Kernel learning methods, whether Bayesian or frequentist, typically involve multiple levels of inference, with the coefficients of the kernel expansion being determined at the first level and the kernel and regularisation parameters carefully tuned at the second level, a process known as model selection. Model selection for kernel machines is commonly performed via optimisation of a suitable model selection criterion, often based on cross-validation or theoretical performance bounds. However, if there are a large number of kernel parameters, as for instance in the case of automatic relevance determination (ARD), there is a substantial risk of over-fitting the model selection criterion, resulting in poor generalisation performance. In this paper we investigate the possibility of learning the kernel, for the Least-Squares Support Vector Machine (LS-SVM) classifier, at the first level of inference, i.e.parameter optimisation. The kernel parameters and the coefficients of the kernel expansion are jointly optimised at the first level of inference, minimising a training criterion with an additional regularisation term acting on the kernel parameters. The key advantage of this approach is that the values of only two regularisation parameters need be determined in model selection, substantially alleviating the problem of over-fitting the model selection criterion. The benefits of this approach are demonstrated using a suite of synthetic and real-world binary classification benchmark problems, where kernel learning at the first level of inference is shown to be statistically superior to the conventional approach, improves on our previous work (Cawley and Talbot, 2007) and is competitive with Multiple Kernel Learning approaches, but with reduced computational expense.

Item Type: Article
Faculty \ School: Faculty of Science > School of Computing Sciences


UEA Research Groups: Faculty of Science > Research Groups > Computational Biology
Faculty of Science > Research Groups > Data Science and Statistics
Faculty of Science > Research Groups > Centre for Ocean and Atmospheric Sciences
Related URLs:
Depositing User: Pure Connector
Date Deposited: 09 Mar 2015 07:31
Last Modified: 18 Apr 2023 22:33
URI: https://ueaeprints.uea.ac.uk/id/eprint/52500
DOI: 10.1016/j.neunet.2014.01.011

Actions (login required)

View Item View Item