Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters

Cawley, Gavin C. ORCID: and Talbot, Nicola L. C. (2007) Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters. Journal of Machine Learning Research, 8. pp. 841-861. ISSN 1533-7928

Full text not available from this repository. (Request a copy)


While the model parameters of a kernel machine are typically given by the solution of a convex optimisation problem, with a single global optimum, the selection of good values for the regularisation and kernel parameters is much less straightforward. Fortunately the leave-one-out cross-validation procedure can be performed or a least approximated very efficiently in closed form for a wide variety of kernel learning methods, providing a convenient means for model selection. Leave-one-out cross-validation based estimates of performance, however, generally exhibit a relatively high variance and are therefore prone to over-fitting. In this paper, we investigate the novel use of Bayesian regularisation at the second level of inference, adding a regularisation term to the model selection criterion corresponding to a prior over the hyper-parameter values, where the additional regularisation parameters are integrated out analytically. Results obtained on a suite of thirteen real-world and synthetic benchmark data sets clearly demonstrate the benefit of this approach.

Item Type: Article
Additional Information: Special Topic on Model Selection
Faculty \ School: Faculty of Science > School of Computing Sciences

UEA Research Groups: Faculty of Science > Research Groups > Data Science and Statistics
Faculty of Science > Research Groups > Computational Biology
Faculty of Science > Research Groups > Centre for Ocean and Atmospheric Sciences
Related URLs:
Depositing User: Vishal Gautam
Date Deposited: 07 Mar 2011 14:55
Last Modified: 22 Apr 2023 01:12

Actions (login required)

View Item View Item