Kernel learning at the first level of inference

Cawley, G.C. ORCID: and Talbot, N.L.C. (2014) Kernel learning at the first level of inference. Neural Networks, 53. pp. 69-80. ISSN 0893-6080

[thumbnail of nn2014a]
PDF (nn2014a) - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives.

Download (302kB) | Preview


Kernel learning methods, whether Bayesian or frequentist, typically involve multiple levels of inference, with the coefficients of the kernel expansion being determined at the first level and the kernel and regularisation parameters carefully tuned at the second level, a process known as model selection. Model selection for kernel machines is commonly performed via optimisation of a suitable model selection criterion, often based on cross-validation or theoretical performance bounds. However, if there are a large number of kernel parameters, as for instance in the case of automatic relevance determination (ARD), there is a substantial risk of over-fitting the model selection criterion, resulting in poor generalisation performance. In this paper we investigate the possibility of learning the kernel, for the Least-Squares Support Vector Machine (LS-SVM) classifier, at the first level of inference, i.e.parameter optimisation. The kernel parameters and the coefficients of the kernel expansion are jointly optimised at the first level of inference, minimising a training criterion with an additional regularisation term acting on the kernel parameters. The key advantage of this approach is that the values of only two regularisation parameters need be determined in model selection, substantially alleviating the problem of over-fitting the model selection criterion. The benefits of this approach are demonstrated using a suite of synthetic and real-world binary classification benchmark problems, where kernel learning at the first level of inference is shown to be statistically superior to the conventional approach, improves on our previous work (Cawley and Talbot, 2007) and is competitive with Multiple Kernel Learning approaches, but with reduced computational expense.

Item Type: Article
Faculty \ School: Faculty of Science > School of Computing Sciences

University of East Anglia > Faculty of Science > Research Groups > Computational Biology (subgroups are shown below) > Machine learning in computational biology
Related URLs:
Depositing User: Pure Connector
Date Deposited: 09 Mar 2015 07:31
Last Modified: 21 Oct 2022 00:38
DOI: 10.1016/j.neunet.2014.01.011

Actions (login required)

View Item View Item