Deep dynamic neural networks for multimodal gesture segmentation and recognition

Tools

Wu, Di, Pigou, Lionel, Kindermans, Pieter-Jan, Le, Nam Do-Hoang, Shao, Ling, Dambre, Joni and Odobez, Jean-Marc (2016) Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8). pp. 1583-1597. ISSN 0162-8828

[thumbnail of Published manuscript]
Preview
 PDF (Published manuscript) - Published Version
Download (1MB) | Preview

Abstract

This paper describes a novel method called Deep Dynamic Neural Networks (DDNN) for multimodal gesture recognition. A semi-supervised hierarchical dynamic framework based on a Hidden Markov Model (HMM) is proposed for simultaneous gesture segmentation and recognition where skeleton joint information, depth and RGB images, are the multimodal input observations. Unlike most traditional approaches that rely on the construction of complex handcrafted features, our approach learns high-level spatiotemporal representations using deep neural networks suited to the input modality: a Gaussian-Bernouilli Deep Belief Network (DBN) to handle skeletal dynamics, and a 3D Convolutional Neural Network (3DCNN) to manage and fuse batches of depth and RGB images. This is achieved through the modeling and learning of the emission probabilities of the HMM required to infer the gesture sequence. This purely data driven approach achieves a Jaccard index score of 0.81 in the ChaLearn LAP gesture spotting challenge. The performance is on par with a variety of state-of-the-art hand-tuned feature-based approaches and other learning-based methods, therefore opening the door to the use of deep learning techniques in order to further explore multimodal time series data.

Item Type: Article
Uncontrolled Keywords: gesture recognition,deep learning,convolutional neural networks,deep belief networks,hidden markov models
Faculty \ School: Faculty of Science > School of Computing Sciences
Related URLs:
Depositing User: Pure Connector
Date Deposited: 09 Mar 2017 01:40
Last Modified: 03 Jul 2023 10:30
URI: https://ueaeprints.uea.ac.uk/id/eprint/62904
DOI: 10.1109/TPAMI.2016.2537340

Downloads

Downloads per month over past year

Actions (login required)

View Item View Item