Multiverse simulation to explore the impact of analytical choices on type I and type II errors in a reaction time study

Bognar, Miklos, Varga, Marton A., van Ravenzwaaij, Don, Kekecs, Zoltan, Grange, James A., Gyurkovics, Máté and Aczel, Balazs (2025) Multiverse simulation to explore the impact of analytical choices on type I and type II errors in a reaction time study. Behavior Research Methods, 57 (10). ISSN 1554-351X

Microsoft Word (rba08-Gyurkovics_etal_Multivers) - Published Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

Researcher degrees of freedom in data analysis present significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose an example of an exploratory multiverse simulation to empirically compare various decision pathways to identify an effect’s sensitivity to different analytical choices. The approach is demonstrated on the congruency sequence effect (CSE), a well-studied phenomenon in cognitive control research. We reviewed existing literature to identify common non-theory-specific analytical decisions, such as outlier exclusion criteria and hypothesis testing methods, and incorporated these into our simulation framework. Using 20,000 simulated datasets, we compared the true positive rates (TPR) and false positive rates (FPR) across 50 different decision pathways, resulting in a total of 1 million analyses. Our results indicate substantial differences in power and type I error rates across the analytical pathways, with some posing a significant risk of producing high false positives. The findings underscore the importance of running extensive simulations to investigate different data handling and hypothesis testing approaches in certain research fields. This case study serves as an example for conducting similar simulation procedures in research fields characterized by high variability in analytical decisions when investigating an otherwise widely accepted effect.

Item Type:	Article
Uncontrolled Keywords:	reaction tim analysis,methodology,multiverse analysis,simulation,2* ,/dk/atira/pure/researchoutput/REFrank/2_
Faculty \ School:	Faculty of Social Sciences > School of Psychology
Depositing User:	LivePure Connector
Date Deposited:	26 Jan 2026 16:30
Last Modified:	26 Jan 2026 20:30
URI:	https://ueaeprints.uea.ac.uk/id/eprint/101729
DOI:	10.3758/s13428-025-02807-y

Downloads

Actions (login required)

View Item