Watching subtitled videos with the sound off affects viewers’ comprehension, cognitive load, immersion, enjoyment, and gaze patterns: A mixed-methods eye-tracking study

Szarkowska, Agnieszka, Ragni, Valentina, Szkriba, Sonia, Black, Sharon ORCID: https://orcid.org/0000-0001-8123-4383, Orrego-Carmona, David and Kruger, Jan-Louis (2024) Watching subtitled videos with the sound off affects viewers’ comprehension, cognitive load, immersion, enjoyment, and gaze patterns: A mixed-methods eye-tracking study. PLoS One, 19 (10). ISSN 1932-6203

Preview

PDF (PLOSOne_2024_e0306251) - Published Version Available under License Creative Commons Attribution. Download (1MB) | Preview

Abstract

Every day, millions of viewers worldwide engage with subtitled content, and an increasing number choose to watch without sound. In this mixed-methods study, we examine the impact of sound presence or absence on the viewing experience of both first-language (L1) and second-language (L2) viewers when they watch subtitled videos. We explore this novel phenomenon through comprehension and recall post-tests, self-reported cognitive load, immersion, and enjoyment measures, as well as gaze pattern analysis using eye tracking. We also investigate viewers’ motivations for opting for audiovisual content without sound and explore how the absence of sound impacts their viewing experience, using in-depth, semi-structured interviews. Our goal is to ascertain whether these effects are consistent among L2 and L1 speakers from different language varieties. To achieve this, we tested L1-British English, L1-Australian English and L2-English (L1-Polish) language speakers (n = 168) while they watched English-language audiovisual material with English subtitles with and without sound. The findings show that when watching videos without sound, viewers experienced increased cognitive load, along with reduced comprehension, immersion and overall enjoyment. Examination of participants’ gaze revealed that the absence of sound significantly affected the viewing experience, increasing the need for subtitles and thus increasing the viewers’ propensity to process them more thoroughly. The absence of sound emerged as a global constraint that made reading more effortful. Triangulating data from multiple sources made it possible to tap into some of the metacognitive strategies employed by viewers to maintain comprehension in the absence of sound. We discuss the implications within the context of the growing trend of watching subtitled videos without sound, emphasising its potential impact on cognitive processes and the viewing experience.

Item Type:	Article
Additional Information:	Data Availability: All relevant data are available here: https://osf.io/9nmfx/. Funding information: This study was conducted within the WATCH-ME project funded by the National Science Centre OPUS 19 programme 2020/37/B/HS2/00304. The principal investigator is Agnieszka Szarkowska, whereas Valentina Ragni is employed as a post-doc and Sonia Szkriba as a PhD candidate researcher in the project.
Faculty \ School:	Faculty of Arts and Humanities > School of Media, Language and Communication Studies
UEA Research Groups:	Faculty of Arts and Humanities > Research Groups > Area Studies
Depositing User:	LivePure Connector
Date Deposited:	08 Oct 2024 13:30
Last Modified:	17 Oct 2024 00:30
URI:	https://ueaeprints.uea.ac.uk/id/eprint/96946
DOI:	10.1371/journal.pone.0306251

Downloads

Actions (login required)

View Item