Tools
Tools
Liza, Farhana Ferdousi 
ORCID: https://orcid.org/0000-0003-4854-5619 and Grzes, Marek
(2018)
Improving language modelling with noise contrastive estimation.
In:
Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI 2018.
32nd AAAI Conference on Artificial Intelligence, AAAI 2018
.
AAAI press, USA, pp. 5277-5284.
ISBN 9781577358008
Abstract
Neural language models do not scale well when the vocabulary is large. Noise contrastive estimation (NCE) is a sampling-based method that allows for fast learning with large vocabularies. Although NCE has shown promising performance in neural machine translation, its full potential has not been demonstrated in the language modelling literature. A sufficient investigation of the hyperparameters in the NCE-based neural language models was clearly missing. In this paper, we showed that NCE can be a very successful approach in neural language modelling when the hyperparameters of a neural network are tuned appropriately. We introduced the 'search-then-converge' learning rate schedule for NCE and designed a heuristic that specifies how to use this schedule. The impact of the other important hyperparameters, such as the dropout rate and the weight initialisation range, was also demonstrated. Using a popular benchmark, we showed that appropriate tuning of NCE in neural language models outperforms the state-of-the-art single-model methods based on standard dropout and the standard LSTM recurrent neural networks.
| Item Type: | Book Section |
|---|---|
| Additional Information: | Publisher Copyright: Copyright © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. |
| Uncontrolled Keywords: | artificial intelligence ,/dk/atira/pure/subjectarea/asjc/1700/1702 |
| Faculty \ School: | Faculty of Science > School of Computing Sciences |
| Related URLs: | |
| Depositing User: | LivePure Connector |
| Date Deposited: | 26 Sep 2024 16:30 |
| Last Modified: | 29 Sep 2024 06:30 |
| URI: | https://ueaeprints.uea.ac.uk/id/eprint/96822 |
| DOI: |
Actions (login required)
 |
View Item |