The UEA Digital Humans entry to the GENEA Challenge 2023

Tools

Windle, Jonathan, Matthews, Iain, Milner, Ben and Taylor, Sarah (2023) The UEA Digital Humans entry to the GENEA Challenge 2023. In: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, 2023-10-09 - 2023-10-13.

Full text not available from this repository. (Request a copy)

Abstract

This paper describes our entry to the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. This year's challenge focuses on generating gestures in a dyadic setting - predicting a main-agent's motion from the speech of both the main-agent and an interlocutor. We adapt a Transformer-XL architecture for this task by adding a cross-attention module that integrates the interlocutor's speech with that of the main-agent. Our model is conditioned on speech audio (encoded using PASE+), text (encoded using FastText) and a speaker identity label, and is able to generate smooth and speech appropriate gestures for a given identity. We consider the GENEA Challenge user study results and present a discussion of our model strengths and where improvements can be made.

Item Type:	Conference or Workshop Item (Paper)
Uncontrolled Keywords:	3d pose prediction,cross-attention,self-attention,speech-to-gesture,transformer-xl,gesture generation,software,human-computer interaction,computer vision and pattern recognition,computer networks and communications ,/dk/atira/pure/subjectarea/asjc/1700/1712
Faculty \ School:	Faculty of Science > School of Computing Sciences Faculty of Science
UEA Research Groups:	Faculty of Science > Research Groups > Visual Computing and Signal Processing Faculty of Science > Research Groups > Smart Emerging Technologies (former - to 2025) Faculty of Science > Research Groups > Data Science and AI Faculty of Science > Research Groups > Cyber Intelligence and Networks
Related URLs:	http://www.scopus.com/inward/record.url?...
Depositing User:	LivePure Connector
Date Deposited:	09 Jan 2024 01:16
Last Modified:	06 Aug 2025 14:30
URI:	https://ueaeprints.uea.ac.uk/id/eprint/94078
DOI:	10.1145/3577190.3616116

Actions (login required)

View Item