WordSeg: Standardizing unsupervised word form segmentation from text

Tools

Bernard, Mathieu, Thiolliere, Roland, Saksida, Amanda, Loukatou, Georgia R, Larsen, Elin, Johnson, Mark, Fibla, Laia, Dupoux, Emmanuel, Daland, Robert, Cao, Xuan Nga and Cristia, Alejandrina (2020) WordSeg: Standardizing unsupervised word form segmentation from text. Behavior Research Methods, 52 (1). 264–278. ISSN 1554-351X

Full text not available from this repository.

Abstract

A basic task in first language acquisition likely involves discovering the boundaries between words or morphemes in input where these basic units are not overtly segmented. A number of unsupervised learning algorithms have been proposed in the last 20 years for these purposes, some of which have been implemented computationally, but whose results remain difficult to compare across papers. We created a tool that is open source, enables reproducible results, and encourages cumulative science in this domain. WordSeg has a modular architecture: It combines a set of corpora description routines, multiple algorithms varying in complexity and cognitive assumptions (including several that were not publicly available, or insufficiently documented), and a rich evaluation package. In the paper, we illustrate the use of this package by analyzing a corpus of child-directed speech in various ways, which further allows us to make recommendations for experimental design of follow-up work. Supplementary materials allow readers to reproduce every result in this paper, and detailed online instructions further enable them to go beyond what we have done. Moreover, the system can be installed within container software that ensures a stable and reliable environment. Finally, by virtue of its modular architecture and transparency, WordSeg can work as an open-source platform, to which other researchers can add their own segmentation algorithms.

Item Type: Article
Faculty \ School: Faculty of Social Sciences > School of Psychology
UEA Research Groups: Faculty of Social Sciences > Research Groups > Developmental Science
Faculty of Social Sciences > Research Groups > Social Cognition Research Group
Faculty of Social Sciences > Research Groups > Cognition, Action and Perception
Related URLs:
Depositing User: LivePure Connector
Date Deposited: 13 Jun 2019 12:30
Last Modified: 22 Oct 2022 04:43
URI: https://ueaeprints.uea.ac.uk/id/eprint/71384
DOI: 10.3758/s13428-019-01223-3

Actions (login required)

View Item View Item