Tools
Tools
Bear, Helen and Harvey, Richard 
ORCID: https://orcid.org/0000-0001-9925-8316
(2019)
Alternative visual units for an optimized phoneme-based lipreading system.
Applied Sciences, 9 (18).
ISSN 2076-3417
Preview |
PDF (applsci-09-03870)
- Published Version
Available under License Creative Commons Attribution. Download (1MB) | Preview |
Abstract
Lipreading is understanding speech from observed lip movements. An observed series of lip motions is an ordered sequence of visual lip gestures. These gestures are commonly known, but as yet are not formally defined, as `visemes’. In this article, we describe a structured approach which allows us to create speaker-dependent visemes with a fixed number of visemes within each set. We create sets of visemes for sizes two to 45. Each set of visemes is based upon clustering phonemes, thus each set has a unique phoneme-to-viseme mapping. We first present an experiment using these maps and the Resource Management Audio-Visual (RMAV) dataset which shows the effect of changing the viseme map size in speaker-dependent machine lipreading and demonstrate that word recognition with phoneme classifiers is possible. Furthermore, we show that there are intermediate units between visemes and phonemes which are better still. Second, we present a novel two-pass training scheme for phoneme classifiers. This approach uses our new intermediary visual units from our first experiment in the first pass as classifiers; before using the phoneme-to-viseme maps, we retrain these into phoneme classifiers. This method significantly improves on previous lipreading results with RMAV speakers.
| Item Type: | Article |
|---|---|
| Faculty \ School: | Faculty of Science > School of Computing Sciences |
| UEA Research Groups: | Faculty of Science > Research Groups > Interactive Graphics and Audio Faculty of Science > Research Groups > Smart Emerging Technologies |
| Depositing User: | LivePure Connector |
| Date Deposited: | 16 Sep 2019 12:30 |
| Last Modified: | 20 Apr 2023 08:34 |
| URI: | https://ueaeprints.uea.ac.uk/id/eprint/72248 |
| DOI: | 10.3390/app9183870 |
Downloads
Downloads per month over past year
Actions (login required)
 |
View Item |