Tools
Tools
Waddington, Hugh Sharma, Villar, Paul Fenton 
ORCID: https://orcid.org/0000-0001-8779-4189 and Valentine, Jeffrey C.
(2023)
Can non-randomised studies of interventions provide unbiased effect estimates? A systematic review of internal replication studies.
Evaluation Review, 47 (3).
pp. 563-593.
ISSN 0193-841X
Preview |
PDF (SharmaWaddington_etal_2023_ER)
- Published Version
Available under License Creative Commons Attribution. Download (720kB) | Preview |
Abstract
Background: Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. Objectives: We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. Research Design: We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. Results: We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least “some concerns” about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. Conclusions: The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.
| Item Type: | Article |
|---|---|
| Uncontrolled Keywords: | design replication,internal replication,meta-analysis,non-randomized study of interventions,quasi-experimental design,systematic review,within-study comparison,arts and humanities (miscellaneous),social sciences(all) ,/dk/atira/pure/subjectarea/asjc/1200/1201 |
| Faculty \ School: | Faculty of Social Sciences > School of Global Development (formerly School of International Development) |
| Related URLs: | |
| Depositing User: | LivePure Connector |
| Date Deposited: | 03 Mar 2023 14:30 |
| Last Modified: | 05 Aug 2023 00:57 |
| URI: | https://ueaeprints.uea.ac.uk/id/eprint/91353 |
| DOI: | 10.1177/0193841x221116721 |
Downloads
Downloads per month over past year
Actions (login required)
 |
View Item |