Article in Press
Development and preliminary evaluation of the Implementation Research Proposal Appraisal Criteria (ImpResPAC) tool: an international expert modified e-Delphi study
Received: 27 Mar 2025
Accepted: 16 Dec 2025
Louise Hull, Chloe Sweetnam, Rachel Davis, Zarnie Khadjesari, Andy Healey, Annette Boaz, Ioannis Bakolis, Nick Sevdalis, Lucy Goulding, Sara Ackerman, Prajakta Adsul, Miya Barnett, Barbara Barrett, Anna Bergström, Sarah Birken, Sabah Boufkhed, Louise Brown, Adithya Cattamanchi, Caroline Clarke, Tara Clinton-McHarg, Peter Craig,
Cite this article as: Hull, L., Sweetnam, C., Davis, R. et al. Development and preliminary evaluation of the Implementation Research Proposal Appraisal Criteria

Development and Preliminary Evaluation of the Implementation Research Proposal
Appraisal Criteria (ImpResPAC) tool: An International Expert Modified e-Delphi
Study
Louise Hull1,2*
Chloe Sweetnam3*
Rachel E Davis4
Zarnie Khadjesari5
Andy Healey1
6
Annette Boaz6
Ioannis Bakolis1
Nick Sevdalis7
Lucy Goulding8
ImpResPAC Expert Advisory Panel
ImpResPAC Expert Advisory Panel (listed in alphabetical order of their surnames): Sara L Ackerman, Prajakta Adsul, Miya Barnett, Barbara Barrett, Anna Bergström, Sarah Birken, Sabah Boufkhed, Louise Brown, Adithya Cattamanchi, Caroline Clarke, Tara Clinton-
McHarg, Peter Craig, Geoffrey Curran, Alex Dopp, Alicia G. Dugan, Jacquelyn S Favours,
Tracy Finch, Signe Flottorp, Robbie Foy, Hannah E. Frank, Kara Gray-Burrows, Claudia
Hanson, Ties Hoomans, Elaine C Khoong, Roman Kislov, Rebecca Lawton, Jennifer
Leeman, Rebecca Lengnick-Hall, Moira McNulty, Sara Malone, Graham Martin, Ruben G.
Martinez, Allison Metz, Aaloke Mody, Joanna Moullin, Janet Myers, Per Nilsen, Kathryn Oliver, Dennis Pérez, Lisa Pfadenhauer, Hilary Pinnock, Sebastian Potthoff, Byron Powell, Borsika Rabin, Angus I G Ramsay, Jamie Ross, Alex Rowe, Lisa Saldana, Jane Sandall, Joanna Schellenberg, Kelli Scott, Kate Seers, Christopher Shea, Justin D. Smith, Matthew A Spinelli, Nicole Stadnick, Taren Swindle, Carolyn Tarrant, Silia Vitoratou, Thomas J. Waltz, Michel Wensing, Michelle White, Cathleen Willging, Til Wykes, Brian Yates, Serene Yoong, Alexandra Ziemann.
Corresponding author: Louise Hull (louise.hull@kcl.ac.uk)
1 Centre for Mental Health Policy and Evaluation, Health Service and Population Research Department, King’s College London, London UK
2
2 Diabetes Research Centre, College of Life Sciences, University of Leicester, Leicester, UK.
3Petauri Kinect, New York, USA
4Patient-Centred Research Group, Evidera, inc PPD, part of Thermofisher Scientific, London, UK
5Behavioural and Implementation Science (BIS) Research Group, School of Health Sciences, University of East Anglia, UK.
6Health and Social Care Workforce Research Unit, The Policy Institute, King’s College London, UK
7Centre for Behavioural and Implementation Science Interventions, Yong Loo Lin School of Medicine, National University of Singapore, Singapore
8UCLPartners, London, UK
Keywords: Implementation science, Implementation research, Implementation proposals, Implementation protocols, Implementation evaluation, Appraisal criteria, Review criteria, Quality appraisal
ABSTRACT
Background: High-quality implementation research is fundamental to increasing the uptake of evidence-based interventions in healthcare and ultimately improving healthcare delivery and outcomes. However, the development of high-quality implementation research proposals and the identification of the ‘best’ implementation research proposals to fund are hampered by the lack of criteria for their appraisal. This study describes the development and evaluation of the Implementation Research Proposal Appraisal Criteria (ImpResPAC), a comprehensive tool developed to help research teams identify strengths and weaknesses of their implementation research proposals and provide grant reviews with clear criteria for appraising the conceptual and methodological quality of implementation research proposals.
Methods: A two-stage sequential mixed-methods design, including content development (stage 1) and a two-round modified e-Delphi (stage 2a & 2b), was employed. Stage 1: Informed by the structure and content of the Implementation Science Research Development (ImpRes) tool and supplementary guide, and a previous study in which five of the ten ImpRes domains were operationalised to develop five ImpResPAC domains, the ImpResPAC development team developed the initial content of ImpResPAC. Purposive and snowball sampling was used to recruit an Expert Advisory Panel (EAP), of individuals that have made a significant contribution to the conceptual and methodological advancement of implementation science. Stage 2a: The EAP provided feedback on ImpResPAC. Based on the feedback, the ImpResPAC development team made extensive revisions to ImpResPAC. Stage 2b: The refined ImpResPAC was shared with the EAP for further feedback and evaluation.
Results: 68 international experts formed the EAP. ImpResPAC includes 71 domain items, each indicative of high-quality implementation research. The domain items are organised over 10 domains each representing a core element of implementation research. Based on feedback, user instructions were refined, and a glossary of terms used in ImpResPAC was developed.
Conclusion: ImpResPAC is a comprehensive and transparent quantitative tool to appraise the conceptual and methodological quality of implementation research proposals in healthcare. Application of ImpResPAC has several potential and immediate benefits; it will support research teams identify strengths and weaknesses of their implementation research proposals and in turn develop high-quality proposals, enable grant reviewers to identify the ‘best’ implementation research proposals to fund, and provide a comprehensive tool to be used by educators to appraise implementation research proposals submitted as part of implementation science teaching and training initiatives.
Contributions to the literature
•	High-quality implementation research is critical to improving the uptake of evidencebased interventions, advancing the field, and improving healthcare delivery and outcomes.
•	Limited attention has been directed to developing criteria to appraise the conceptual and methodological quality of implementation research proposals in healthcare.
•	We have developed a new tool, Implementation Research Proposal Appraisal Criteria (ImpResPAC), in collaboration with 68 international experts, to appraise the conceptual and methodological quality of implementation research proposals.
•	ImpResPAC can be used by a wide range of implementation research proposal stakeholders including research teams, grant reviewers, and educators to appraise implementation research proposals.
BACKGROUND
Over the past two decades there has been significant and rapid growth in the recognition, from researchers and funders alike, that the integration of implementation science into applied health research is critical to the successful and timely implementation of evidencebased interventions. As such, developing high-quality implementation research proposals and identifying the ‘best’ implementation research proposals to allocate research funds to is critical, as funding is limited, and poor-quality implementation research is significantly less likely to result in the successful implementation of evidence-based interventions and improvements in healthcare delivery and outcomes.
As noted by Brownson et al., [1] many of the hallmarks of high-quality implementation research are shared with other research areas, however, several important differences exist that need to be considered when developing, and we argue appraising, implementation grant applications. More than a decade ago, Proctor et al., [2] proposed ten key ingredients that are important in implementation research grant proposals. Whilst several resources, including tools, recommendations and guidelines, have been developed to support the design of implementation research and the reporting of implementation studies [3-10], less attention has been directed to support the appraisal of implementation research proposals. Appraisal tools are valuable not only to grant reviewers aiming to identify the ‘best’ implementation research proposals for funding, but also research teams striving to develop high-quality competitive implementation research proposals. Research teams can use appraisal criteria to identify elements of their proposals that need to be strengthened prior to funding submission, including identifying elements that might have otherwise been unintentionally overlooked.
Research appraisal tools are critical to ensure that appraisals are standardised, transparent, objective (as far as possible) and fair. The need for such tools, specific to implementation research grant proposals, has been previously highlighted [11] but surprisingly has received limited attention to date. To the best of our knowledge, the only tool that currently exists is the ImplemeNtation and Improvement Science Proposals Evaluation CriTeria (INSPECT) scoring system developed by Crable et al., in 2018 [11]. Based on Proctor et al.’s [2] ten key ingredients that are important in implementation research grant proposals, and reflecting the team’s experience and expertise of writing for federal funding agencies in the United States, INSPECT was recently adapted to fit with the University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center (ACTRI DISC) context and to align with their request for applications [12]. As such, INSPECT’s application and generalisability beyond the United States and ACTRI DISC remains to be evaluated. Our work builds on INSPECT and the Implementation Science Research Development (ImpRes) tool and supplementary guide, designed to support researchers to design high-quality implementation research. Furthermore, ImpResPAC builds on the extensive and significant conceptual and methodological advancements the field has experienced since Proctor et al’s ten key ingredients to writing implementation research grant proposals and INSPECT were published in 2012 and 2018 respectively. A detailed comparison of ImpResPAC (initial mapping) and INSPECT can be found in the protocol paper detailing the development and psychometric evaluation of ImpResPAC.[13]
We previously developed the Implementation Science Research Development (ImpRes) tool and supplementary guide [3] to provide applied health research teams with a step-by-step approach to design high-quality implementation research. The ImpRes tool contains ten domains (including implementation theories, models and frameworks, implementation strategies and implementation outcomes) that represent core elements of implementation research. Informed by the structure and content of the ImpRes tool and guide, and a previous study in which five of the ten ImpRes domains were operationalised to develop five Implementation Research Proposal Appraisal Criteria (ImpResPAC) domains [14], the current study seeks to further develop, refine and evaluate the ImpResPAC tool: a comprehensive, quantitative tool to appraise the conceptual and methodological quality of implementation research proposals in healthcare.
METHODS
We conducted a two-round modified e-Delphi involving the collection and analysis of qualitative and quantitative data. A modified e-Delphi approach was selected because: (1) it is a rigorous consensus-building method; (2) it is flexible and can be designed to best suit study aims, objectives and scope; (3) it allows the participation of a large international panel of experts. We report our findings using the Delphi studies in social and health sciences – recommendations for an interdisciplinary standardized reporting (DELPHISTAR) guideline. [15] The protocol detailing the development and psychometric evaluation of ImpResPAC is published and describes the three-stage sequential mixed-methods design [13]. This paper reports the content development (stage 1), content validation and refinement of ImpResPAC (stage 2a/Delphi round 1) and further content validation, refinement and evaluation of ImpResPAC (stage 2b/Delphi round 2). The primary purpose of the first Delphi round was to elicit in-depth qualitative feedback, as well as limited quantitative feedback, relating to all aspects of ImpResPAC (i.e., the tool itself, user instructions, scoring anchors etc) to inform the refinement of ImpResPAC. In contrast, the primary purpose of the second Delphi round was to elicit a quantitative evaluation relating to the revised ImpResPAC, as well as providing the EAP with the opportunity to provide additional qualitative feedback if they wished to do so. No consultation regarding the methodology was undertaken.
This modified e-Delphi study aims to achieve consensus on the criteria indicative of high- quality implementation research proposals.
Stage 1: ImpResPAC Content development
The ImpResPAC development team (LH, CS, RD, ZK, AH, AB, IB, NS, LG) used the ImpRes tool and supplementary guide as a starting point to develop ImpResPAC [3]. ImpRes includes ten domains that represent core elements of implementation research: (1) implementation research characteristics, (2) implementation theories, frameworks and models, (3) determinants of implementation: contextual factors, (4) implementation strategies, (5) service and patient outcomes, (6) implementation outcomes, (7) unintended consequences, (8) economic evaluation, (9) stakeholder involvement and engagement and (10) patient and public involvement and engagement. As part of a previous Master’s dissertation study (undertaken by CS and supervised by members of the ImpResPAC development team (LH and LG) [14], five of the ten ImpRes domains (implementation research characteristics, implementation theories, models and frameworks, implementation strategies, implementation outcomes, and unintended consequences), were operationalised to develop five ImpResPAC domains. Due to time constraints and the scope of this Master’s dissertation study, this work focused on operationalising a sub-set of ImpRes domains. Each ImpResPAC domain contained several items indicative of high-quality implementation research. Inter-rater reliability (IRR) was assessed by applying this initial version of
ImpResPAC to 16 research protocols, published in Implementation Science.
In the present study, the five previously developed ImpResPAC domain items were reviewed and refined by the ImpResPAC development team and the remaining five ImpRes domains (service and patient outcomes, determinants of implementation: contextual factors, economic evaluation, stakeholder involvement and engagement and patient and public involvement and engagement) were operationalised for use within ImpResPAC with domain items developed. ImpRes is based on key conceptual and methodological literature containing research design guidance and recommendations, published up to March 2018. To ensure that ImpResPAC reflects state-of-the-science research design guidance and recommendations, more recently published key conceptual and methodological literature was identified by the ImpResPAC development team and incorporated within ImpResPAC. ImpResPAC user instructions, including scoring instructions, were also developed.
Stage 2: ImpResPAC Content validation and refinement
The number of Delphi rounds (n=2) was decided in advance.
Stage 2a/Delphi Round 1
A multistage sequential sampling strategy, incorporating purposive and snowball sampling was used to form an Expert Advisory Panel (EAP), consisting of academics across the world who have made a significant contribution to the conceptual and methodological advancement of implementation science. We defined an expert as ‘someone widely recognized as a reliable source of knowledge, technique, or skill whose judgement is accorded authority and status by the public or his or her peers’. [16] Snowball sampling, also referred to as “referral” or “chain” sampling, is a nonprobability sampling technique, and is applied when it is difficult

to access participants with the target characteristics.[17] It involves requesting existing participants to recruit additional participants, who meet the inclusion criteria, from their own social networks. It is considered an efficient and cost-effective method to access participants who would otherwise be difficult to find. [18] First, the ImpResPAC development team generated a list of experts that met the above criteria, based on their collective knowledge, and invited them to participate in the study. Second, once experts agreed to participate in the study, they were encouraged to nominate additional experts, (i.e., snowballing sampling), whose contribution to the development of ImpResPAC would be valuable. Experts were requested to provide the name(s) and email address(es) of additional experts. The ImpResPAC development team then invited the nominated experts to participate in the study.
Delphi Round 1 (December 2021)
A survey was sent to the EAP via email as a Word document attachment. The survey was designed to elicit both quantitative and qualitative feedback on the content, style and comprehensiveness of ImpResPAC domain items and user instructions. Open-ended questions were posed in order to elicit in-depth feedback and suggestions for the addition, removal or modification of domain items. Closed-end questions generated quantitative evaluation data. Along with survey questions focusing on ImpResPAC domains and domains items, the EAP were provided with an overview of ImpResPAC, including ImpResPAC user instructions, and the ImpResPAC domain items. The survey questions explored whether (i) ImpResPAC domain items represent high quality conceptual and methodological elements of implementation research proposals, (ii) user instructions are adequate and clear, (iii) the ImpResPAC scoring scale and associated anchors are appropriate and clear, (iv) there are any domain items missing from ImpResPAC, and (v) ImpResPAC domain item wording is clear. Members of the EAP were asked to only provide feedback on the domains and associated domain items that they have expertise in. A reminder email was sent 2 weeks after the survey was first sent and another reminder email was sent 1 week later. A four-week window was given for experts to complete the survey but there was flexibility with this if experts needed more time. The first survey closed in May 2022.
Stage 2b/Delphi Round 2: ImpResPAC Content validation, refinement and evaluation
Delphi Round 2 (December 2023)
A survey, including the refined ImpResPAC tool, user instruments and newly developed ImpResPAC glossary, was sent to the EAP via a RedCap survey link and included open- ended, multiple-choice questions (yes/no), and statements in which experts were required to indicate their level of agreement using Likert scales. We made the decision not to share the qualitative or quantitative feedback from the previous round on pragmatic grounds: we received over 100 pages of qualitative feedback on the instrument, and the quantitative evaluation was primarily designed to prompt the EAP to provide qualitative feedback and suggestions for improvement. Feedback was shared in the form of the refined version of ImpResPAC.
First, the survey collected information about the EAP, including their academic and/or healthcare institution affiliation(s), current position held, implementation science experience (type of expertise, years working in the field, years of experience reviewing implementation research proposals). Second, like the first survey, the second survey re-visited whether (i) ImpResPAC domain items represent high quality conceptual and methodological elements of implementation research proposals, (ii) the ImpResPAC overview is adequate and clear, (iii) user instructions are adequate and clear, (iv) the ImpResPAC scoring scale and associated
anchors are appropriate and clear, (v) there are any domain items missing from ImpResPAC, and (vi) ImpResPAC domain item wording is clear. These survey questions were presented as dichotomous questions (response format: yes/no). Third, the EAP were asked to rate their agreement with several statements including whether ImpResPAC provides a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research proposals and nine statements relating to the acceptability, feasibility and appropriateness of ImpResPAC for use by grant reviewers, researchers and practitioners, and educators. Fourth, the EAP were asked to identify funding bodies, and funding programmes, that they believed ImpResPAC would be of value to. Finally, the EAP was provided the opportunity to provide any additional comments on ImpResPAC.
Experts were given the option of providing feedback on the same domains that they provided feedback on in survey A, or on the entire tool. Experts were given 4 weeks to submit their responses. Two follow-up reminders were sent before the closing date.
Data analysis
Qualitative data analysis
Qualitative feedback on ImpResPAC domain items were collated in Microsoft word at the ImpResPAC domain level (i.e., one Microsoft word document for each ImpResPAC domain) and then further collated at the domain item level. Each expert’s feedback at the domain item level was presented individually. Two members of the ImpResPAC development team (LH, CS) reviewed and discussed the feedback and provided a response and recommendation on how to address the feedback. Given the qualitative nature of feedback that we sought from experts, unlike quantitative Delphi studies that typically define consensus in terms of percentage agreement, consensus for domain item inclusion was indicated if members of the
EAP did not explicitly suggest the removal or modification of a domain item. In cases where members of the EAP suggested the removal of a domain item, these were discussed among the ImpResPAC development team until consensus was reached in relation to whether the item was removed (as suggested) modified, or remained unchanged. We adopted this approach to all feedback and suggestions received from the EAP. This was then sent to the wider ImpResPAC development team (RD, ZK, AH, AB, IB, NS, LG) who independently reviewed the feedback, added comments and provided suggestions on LH and CS recommendations how to respond to the feedback. Based on the wider ImpResPAC development teams’ feedback, two members of the ImpResPAC development team (LH & CS) met again to discuss and reach consensus on ImpResPAC refinements, taking into account the wider teams feedback This process was followed for Delphi round 1 and 2 and for feedback on ImpResPAC user instructions and scoring system.
Quantitative data analysis
Descriptive statistics were calculated for closed-ended survey questions.
RESULTS
Stage 1
1.1	ImpResPAC content development
The ImpResPAC development team operationalised ImpRes to develop ImpResPAC domain items. This process resulted in the initial development of 55 domain items (indicative of high- quality implementation research) organised across the ten domains: implementation research characteristics (n=6 domain items), implementation theories, models and frameworks (n=6), implementation determinants (n=5), implementation strategies (n=7), service and patient outcomes (n=5), implementation outcomes (n=7), unintended consequences (n=2), economic evaluation (n=7), stakeholder involvement and engagement (n=5) and patient and public involvement and engagement (n=5). Supplementary Table 1 displays the ImpResPAC domain items developed across all stages of the study.
As part of the developmental work, ImpResPAC user instructions, including scoring instructions were developed. A five-point scoring system (1-5) with descriptive anchors was developed to be applied at a domain level to distinguish between ‘very poor’, ‘poor’, ‘satisfactory’, ‘good’ and ‘excellent’ implementation research proposals. A score of 1 (‘very poor’) was suggested for implementation research proposals that fail to adequately address all domain items; a score of 2 (‘poor’) for proposals that address only a few items adequately; a score of 3 (‘satisfactory’) for proposals that address some domain items adequately; a score of 4 (‘good’) for proposals that address most items adequately/fully, and; a score of 5 (‘excellent’) for proposals that fully address all domain items. Given the diversity and scope of implementation research proposals, it is possible that one or more ImpResPAC domains, and all associated items, may not be applicable. As such, a ‘not applicable’ option was also included in the scoring system.

Stage 2a: Content validation, refinement and initial

" 

Composition of the Expert Advisory Panel (EAP)
Two-hundred-and-six experts were invited, via purposive and snowballing sampling, to be members of the EAP. In total, 68 experts were recruited, representing a 33.0% response rate. Table 1 displays the composition of the EAP, including their affiliation(s), geographic location and the ImpResPAC domains they reviewed. Experts were based in the USA (n=34), UK (England and Scotland) (n=24), Australia (n=3), Sweden (n=3), Germany (n=2), Norway (n=1), and Cuba (n=1).
TABLE 1	HERE
94.2	% of the EAP held an academic appointment. 35.8% of the EAP had exclusive expertise in implementation science, 33.9% had expertise in implementation science and clinical practice. The EAP had a cumulative total of 734 years of expertise in implementation science.
94.3	% of the EAP had experience of reviewing implementation research proposals, with a cumulative total of 378 years of experience in reviewing implementation research proposals. A full summary of the characteristics of the EAP are presented in Table 2.
TABLE 2	HERE
2.2 Delphi Round 1 data collection
Sixty-eight experts completed the first survey. Members of the EAP provided feedback on the ImpResPAC domains that they felt they had the expertise to review, as such experts provided feedback on differing numbers of ImpResPAC domains; 34 (50.0%) experts provided feedback on 1 domain, 24 (35.3%) experts provided feedback on 2 domains, 6 (8.8%) experts provided feedback on 3 domains, 3 (4.4%) experts provided feedback on 4 domains, 1 (1.5%) expert provided feedback on 6 domains. The number of experts that provided feedback on each ImpResPAC domain was as follows: implementation research characteristics (15 experts); implementation theories, models and frameworks (12 experts); implementation determinants (14 experts); implementation strategies (15 experts); service and patient outcomes (10 experts); implementation outcomes (11 experts); unintended consequences (15 experts); stakeholder involvement and engagement (11 experts); economic evaluation (7 experts); patient and public involvement and engagement (6 experts).
Quantitative evaluative feedback from the EAP is reported in Supplementary Table 3. Across the ten ImpResPAC domains, the EAP responded variably, ranging from 33.3% in the patient and public engagement and involvement domain to 60.0% in the implementation research characteristics and implementation strategies domains, agreeing that the ImpResPAC domain items represent and reflect high-quality conceptual and methodological elements of implementation research proposals. Across the ten ImpResPAC domains, a substantial proportion of the EAP failed to respond to this question, ranging from 0.0% in the economic evaluation domain to 50.0% in the patient and public involvement and engagement domain. Across the ten ImpResPAC domains, the EAP responded variably, ranging from 8.3% in the implementation theories, models and frameworks domain to 66.7% in the patient and public involvement and engagement domain, agreeing that there are domain items missing from ImpResPAC domains. Again, across the ten ImpResPAC domains, a substantial proportion of the EAP failed to respond to this question, ranging from 0.0% in the economic evaluation domain to 50.0% in the implementation determinants domain.
Across the ten ImpResPAC domains, the EAP responded variably, ranging from 27.3%) in the implementation outcomes domain to 71.4% economic evaluation domain, respectively agreeing that the ImpResPAC domain item wording is clear. Again, across the ten ImpResPAC domains, a substantial proportion of the EAP failed to respond to this question, ranging from 0.0%) in the economic evaluation domain to (50.0% in the patient and public involvement and engagement domain.
57.4% of the EAP agreed that the ImpResPAC user instructions are adequate and clear.
25.0% of the EAP failed to respond to this question. 38.2% of the EAP agreed that the ImpResPAC scoring scale and associated anchors are appropriate and clear. 32.4% of the EAP failed to respond to this question.
Experts provided extensive feedback and suggestions for improvements relating to ImpResPAC domain items and user instructions, including scoring. To address feedback, extensive refinements were made to ImpResPAC domain items including modifications to domain items (n=41), the addition of domain items (n=20), the merger and modification of two domain items into one domain item (n=6), the removal of domain items (n=5) and the split of one domain item into two domain items (n=1). After extensive refinements, the number of ImpResPAC domain items increased from 55 to 67. Supplementary Table 1 displays the ImpResPAC domain items developed across all stages of the study. Supplementary Table 2 provides a full summary of the refinements made to ImpResPAC domain items across ImpResPAC domains and all stages of the study.
In addition to feedback and suggestions for improving ImpResPAC domain items, the EAP provided feedback and suggestions for improving other aspects of ImpResPAC. Comments and concerns relating to the lack of use of widely accepted terms and abbreviations, the focus of particular ImpResPAC domains, the scope of ImpResPAC’s application, scale descriptors and scoring anchors, the comparability of global ImpResPAC scores for proposals with differing number of applicable domain items, and concerns that some terms in ImpResPAC may be open to interpretation, prompted several key refinements. The refinements from Stage 2a are presented below:
1.	Renamed Implementation theories, frameworks and models domain to Implementation theories, models and frameworks. Refinement rationale: To ensure the domain name is in line with widely accepted and used word ordering/abbreviation (Theories-Models-Frameworks/TMFs).
2.	Renamed Implementation determinants: contextual factors domain to Implementation determinants. Refinement rationale: To ensure that the domain name is not misleading in relation to the focus of the domain (i.e., the. domain is not exclusively focused on contextual determinants to the exclusion of all other types of implementation determinants).
3.	Replaced the term project with proposal throughout ImpResPAC. Refinement rationale: To reflect the focus of ImpResPAC to appraise the quality of implementation research proposals (prospective appraisal) rather than the quality of implementation research proposals and completed implementation research studies (prospective and retrospective appraisal).
4.	Revised the mid-point scoring scale descriptor from satisfactory to fair. Refinement rationale: To better reflect the intended meaning of the descriptor.
5.	Revised scoring anchors to include suggested percentage cut-off ranges relating to the percentage of domain items partially or fully addressed. Refinement rationale: To support ImpResPAC domain rating decisions.
6.	Revised the global ImpResPAC scoring recommendations from the cumulative total of domain scores to median. Refinement rationale: To allow ImpResPAC users to easily compare the quality of implementation research proposals with differing number of applicable ImpResPAC domains.
7.	Developed a supplementary ImpResPAC glossary, containing terms and acronyms used within ImpResPAC. Refinement rationale: To ensure that terms are not open to interpretation. Some members of the ImpResPAC EAP expressed concerns that some terms, such as ‘quality gap’, ‘strength of evidence’, ‘stakeholder’, are open to interpretation.
Stage 2b
Delphi Round 2 data collection
42/68 members of the EAP, representing a response rate of 61.8%, completed Survey B.
73.8	% of the EAP based their evaluation of ImpResPAC on all 10 domains. The remaining 11 experts based their evaluation on 9 domains (n=3), 8 domains (n=2), 7 domains (n=1), 6 domains (n=1), 3 domains (n=2), 1 domain (n=2).
Evaluation of ImpResPAC
83.3	% of the EAP either agreed or strongly agreed that ImpResPAC provides a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research proposals.
Content, clarity and adequacy of domain items, user instructions and scoring
Quantitative evaluative feedback from the ImpResPAC EAP is reported in Table 3. The majority of the ImpResPAC EAP responded favourably to statements relating to the content and clarity of ImpResPAC domain items and the adequacy and clarity of ImpResPAC user instructions and scoring scale and associated anchors. 26.2% of the EAP thought that domain items were missing from ImpResPAC.
TABLE 3	HERE
Acceptability, feasibility and appropriateness of ImpResPAC
Quantitative evaluative feedback from the EAP, relating to the acceptability, feasibility and appropriateness of ImpResPAC for use by research teams, grant reviewers, and educators, to appraise the conceptual and methodological quality of implementation research proposals, is reported in Table 4. The EAP responded favourably, above the scale midpoint (3), to all statements relating to the acceptability, feasibility and appropriateness of ImpResPAC. Thereby indicating that the EAP viewed ImpResPAC as an acceptable, feasible and appropriate tool for use by research teams, grant reviewers, and educators.
TABLE 4	HERE
Whilst most of the EAP considered ImpResPAC to be an acceptable, feasible and appropriate tool to appraise the quality of implementation research proposals, some members of the EAP raised concerns relating to the acceptability, feasibility and appropriateness of the tool. For example, one expert stated, ‘The tool [ImpResPAC] is comprehensive but likely to be too detailed to be of value for researchers and reviewers.” However, another expert stated that ‘The length of the tool and scoring method are appropriate.” In addition, one expert stated, ‘I'm unconvinced of the role of tools of this kind in appraisal in general - it's not a specific critique of this tool in particular!”. However, in contrast, another expert stated that ImpResPAC “Provides a clear standardised way to equitably evaluate proposals.”
Additional feedback on ImpResPAC
Experts provided additional feedback, including suggestions for improvements relating to ImpResPAC domain items and user instructions, including scoring. To address feedback, modifications to domain items (n=17), the addition of domain items (n=4), and the split of one domain item into two domain items (n=2) were made. After refinements, the number of ImpResPAC domain items increased from 67 to 71. Supplementary Table 1 displays the ImpResPAC domain items developed across all stages of the study. Supplementary Table 2 provides a full summary of the refinements made to ImpResPAC domain items across ImpResPAC domains and all stages of the study. In addition to these modifications, the ImpResPAC development team made very minor amendments to several domain items to ensure consistency in language across and within domain items and the glossary (e.g., steering/advisory group was inconsistently referred to within and between the tool and glossary, this was amended).
In addition to feedback and suggestions for improving ImpResPAC domain items, the EAP provided feedback and suggestions for improving other aspects of ImpResPAC. Comments and concerns relating to the weighting of domains and domain items, the reference to specific terms, tools and methods with several domain items, the descript and scoring options of domain items, appropriate referencing of established guidance, and inclusion of additional terms to be included in the ImpResPAC glossary, prompted several refinements: The refinements from Stage 2b are presented below.
1.	Clarified that ImpResPAC domains and domain items are weighted equally rather than not weighted. Refinement rationale: In response to a member of the EAP highlighting that “In the [ImpResPAC] overview, it is stated that the items in each domain are not weighted. I don't think this is quite correct. I think they are weighted equally. If they were unweighted, that would imply it was up to the user to attach to them the weight they see as most appropriate - this is not what is happening here, since the instructions are clear that each item should be weighted equally in calculating a score.”
2.	Clarified that although some ImpResPAC domain items reference specific terms, tools and methods that are indicative of excellent conceptual and methodological elements of implementation research, it is possible that a proposal fully addresses an ImpResPAC domain item without explicitly referring to the terms, tools and methods referenced in ImpResPAC. In such cases, we recommend that ImpResPAC users use their judgement to determine whether a proposal fulfils the criteria. Refinement rationale: In response to a member of the EAP expressing concerns that ‘a proposal could be excellent despite not using the tools or terms mentioned ... for example an implementation strategy could be clearly described to allow replication yet not use the terms "actor, action, target, temporality, dose.”
3.	Added a ‘not applicable’ category option to domain item scoring category options of ‘not addressed’ and ‘partially/fully addressed’. Refinement rationale: In response to the recommendation that “maybe there should be a column to check which items are being considered as relevant for a given proposal to assess whether reviewers considered the same items as relevant.”
4.	Added an example of a domain item with more than one element to the ImpResPAC overview. Refinement rationale: In response to one member of the EAP reflecting that “the description of the distinction between 'partially' and 'fully' in the instructions is a bit unclear because the reader has not yet seen the domains or items to fully understand what this means. Perhaps this part of the instructions could include a specific example of an item with more than one element for context.”
5.	Added citations, where relevant, to ImpResPAC domain items. Refinement rationale: In response to concerns raised by one member of the EAP that “some things [domain elements] are referencing established guidance already represented in other publications…I would recommend citing thoroughly throughout this document [ImpResPAC] and any explanatory documents that will accompany it.”
6.	Additional terms added to the glossary (including contextual determinants, health equity, intermediate outcome, interquartile range, median range, perspective [economic evaluation], sensitivity analysis, net monetary benefit). Refinement rationale: These additions were based on suggestions from members of the EAP. Modifications were made to several terms (e.g., Engagement was changed to ‘Engagement [in research] and ‘Involvement’ was changed to ‘Involvement [in research] to improve clarity, some definitions were modified as the links to the original reference was no longer available. Perverse was removed from the glossary.
7.	Clarified that ImpResPAC domains are ‘distinct but interrelated’ and ‘that each domain represents a core element of implementation research.’ And clarification that ImpResPAC domain items are ‘indicative of high-quality implementation research.’ Refinement rationale: In response to one member of the EAP commenting “In the overview section, I think it would be worth explaining explicitly what is meant by a 'domain' and an 'item'. This only becomes clear later on.”
8.	Clarified that in addition to calculating the global ImpResPAC median score, it is recommended that ImpResPAC users consider individual domain scores. This is important because there may be instances in which the global ImpResPAC median score may be inflated by a number of ImpResPAC domains that score very highly but consist of a number of ImpResPAC domains that score very poorly.
Refinement rationale: In response to the following comment received from one member of the EAP: “One thought is that you may have a proposal that scores very highly on a given domain and that may outweigh other concerns in other domains. So in essence, I worry somewhat about the scoring more than I worry about the domains and items themselves. I think the scoring is potentially less useful than the guidance itself in the items.”
9.	Revised the term ‘project’ to ‘proposal’ in ImpResPAC. Refinement rationale: To reflect the evolution of ImpResPAC across the stages of development and reflect the sole use of the tool being the prospective application to implementation research proposal rather than the retrospective application to assess the quality of implementation research.
Funding bodies
The EAP identified more than 30 funders that they believed ImpResPAC would be of value to, including the National Institute for Health (NIH), the National Institute for Health and Care Research (NIHR), and the National Health and Medical Research Council (NHMRC). This information is presented in Supplementary Table 4. Two members of the EAP commented “It [ImpResPAC] would be valuable to any funding body that funds implementation research” and “They (appraisal criteria) seem generic enough to be valuable for a range of funders.”
Final ImpResPAC tool and user and scoring instructions
The final ImpResPAC tool includes 71 domain items, organised across 10 domains: implementation research characteristics (11 items); implementation theories, models and frameworks (6 items); implementation determinants (5 items); implementation strategies (9 items); service and patient outcomes (7 items); implementation outcomes (7 items); unintended consequences (5 items); economic evaluation (9 items); stakeholder involvement and engagement (6 items); patient and public involvement and engagement (6 items). Table 5 presents the 71 ImpResPAC domain items. ImpResPAC user and scoring instructions are presented in Figure 1.

FIGURE 1 HERE

Interactive ImpResPAC PDF
To support users in the application of ImpResPAC, we commissioned ICON [19], a creative design company, to develop a user-friendly, interactive ImpResPAC PDF. The PDF contains an introduction to ImpResPAC (developed by the ImpResPAC research team without the input of the EAP), comprehensive user and scoring instructions, ImpResPAC, a ImpResPAC worksheet, including further instructions on ‘what to do next’ once a proposal has been appraised using ImpResPAC (developed by the ImpResPAC research team, without the input of the EAP), and the ImpResPAC glossary (Supplementary File 5).
DISCUSSION
This study sought to develop a new tool to support research teams to identify strengths and weaknesses in their implementation research proposals, aid grant reviewers in identifying the ‘best’ proposals to fund and assist educators in appraising the quality of implementation research proposals in healthcare. To the best of our knowledge, ImpResPAC represents the first in-depth quantitative tool to appraise the conceptual and methodological quality of implementation research proposals in healthcare, incorporating state-of-the-science guidelines and recommendations, based on extensive input from a large international expert advisory panel.

To the best of our knowledge, the only other review criteria for implementation research proposals is the ImplemeNtation and Improvement Science Proposals Evaluation CriTeria (INSPECT) scoring system. This study builds on ImpRes, INSPECT and the significant conceptual and methodological advancements that have shaped the design of implementation research since Proctor et al’s ten key ingredients to writing implementation research grant proposals was published. The criteria included in ingredients to writing implementation research grant proposals was published. The criteria included in ImpResPAC and INSPECT overlap somewhat, which is unsurprising given their similar aim (i.e., to evaluate the quality of implementation research proposals). For example, both ImpResPAC and INSPECT include criteria relating to the care/quality gap (ImpResPAC: ‘The proposed study explicitly seeks to address an implementation problem; it clearly identifies and describes both the associated quality and/or coverage of care gap, and the evidence-based intervention selected to address the problem.’ INSPECT: ‘Clearly defined care/ quality/community gap or need is supported by local setting data (i.e., evidence of chart review or other preliminary data) and appropriate citations from the literature.’) However, there are cases where there is very little, if any, overlap between the criteria in ImpResPAC and INSPECT. For example, ImpResPAC includes a domain, containing four domain items, relating to the unintended consequences of implementation efforts, whereas INSPECT does not include any criteria relating explicitly to unintended consequences. Furthermore, INSPECT focuses solely on measurement and analysis of implementation outcomes whereas ImpResPAC provides a more comprehensive and detailed appraisal of the full spectrum of outcomes typically evaluated in implementation research, including implementation outcomes, service and patient outcomes, economic outcomes and unintended consequences/outcomes. In contrast, INSPECT contains four criteria relating to the setting’s readiness to adopt new services/treatment/programs, whereas ImpResPAC does not include criteria relating specifically to the setting’s readiness to adopt the evidence-based intervention. It is important to note that INSPECT operationalises Proctor et al.’s [2] “ten key ingredients” for grant writers, that were developed to map onto NIH criteria, potentially limiting its applicability beyond NIH funding programmes. In contrast, ImpResPAC is based on ImpRes (not developed to map onto an existing funder’s criteria) and the input of an international panel of experts. As such, ImpResPAC is expected to be applicable to a wider range of funders, globally. Our results support this expectation, members of the EAP identified more than 30 funding bodies across the world that ImpResPAC would be of value to, including the USA, UK, Canada, Australia, Sweden, Switzerland, and the Netherlands. In summary, whilst ImpResPAC and INSPECT both aim to appraise the quality of implementation research proposals, ImpResPAC provides a more comprehensive and indepth appraisal of implementation research proposals than INSPECT (ImpResPAC contains 71 domain items in comparison to INSPECT which contains 35 criteria, at the scale point 3 and is based on a more rigorous process of development, involving the input of a large international advisory panel with diverse and extensive expertise and experience in implementation science and reviewing implementation research proposals.
Our goal in developing ImpResPAC was to support the appraisal of the conceptual and methodological quality of implementation research proposals in healthcare. Results from our initial evaluation of ImpResPAC are promising: the vast majority of the ImpResPAC EAP agreed that ImpResPAC provides a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research proposals and that the domain items represent and reflect high-quality conceptual and methodological elements of implementation research proposals.
Supporting research teams to develop high-quality implementation research proposals
The majority of the ImResPAC EAP agreed or strongly agreed that ImpResPAC is an appropriate, acceptable and feasibility tool (91%, 86%, and 81% respectively) to be use by research teams to appraise the quality of their implementation research funding proposals.
We anticipate that research teams who use ImpResPAC will be more likely to identify elements of their implementation research proposals that are weak and need to be strengthened prior to funding submission or submission as part of educational initiatives. Using ImpResPAC in this way is likely to identify elements not only in need of improvement but also elements that might have otherwise been mistakenly overlooked, and ultimately lead to improvements in the quality of proposals submitted for funding For example, Francetic et al., [20] found that unintended consequences, also referred to as ‘spillover effects’, are rarely considered and tend not to be a pre-specified feature of implementation studies. ImpResPAC includes an ‘unintended consequences’ domain which is likely to serve as a prompt to research teams to explore whether anticipated and unanticipated unintended consequences occur as a result of implementation efforts.
Supporting grant reviewers to identify the ‘best’ implementation research proposals We envisage grant reviewers will be able to use ImpResPAC to inform funding decisions.
We recommend that grant reviewers consider the global ImpResPAC score as well as individual domain scores to identify and differentiate between poor and high-quality implementation research proposals. Applying ImpResPAC in this way will allow grant reviewers to provide feedback to research teams to either explain a funding decision and/or to provide targeted feedback to strengthen an implementation research proposal.
It is important to stress that ImpResPAC is not intended to replace the broad and varied criteria of funding bodies and programmes nor discourage the more nuanced aspects of the review process, rather it is intended to complement and strengthen the review process. The weight (i.e., the importance) assigned to ImpResPAC criteria, relative to other criteria, is likely to vary considerably across funding bodies and programmes. As such, we believe that funders/funding programmes using ImpResPAC to enhance their review processes are best placed to determine the weight they wish to assign to ImpResPAC criteria relative to other criteria. Similarly, ImpResPAC domains and items are equally weighted, however if deemed necessary, we do not discourage users placing greater weight on particular ImpResPAC domains and/or domains items. In such instances, we recommend that the rationale for differential weighting is clearly described.
Supporting educators to appraise the quality of implementation research proposals
We envisage educators applying ImpResPAC, to appraise implementation research proposals submitted as part of implementation teaching and training initiatives, can use ImpResPAC to score proposals and provide detailed feedback to students. In addition, we believe that ImpResPAC can be viewed as an educational tool in that it provides a list of markers of high- quality implementation research.
Length and feasibility of ImpResPAC
ImpResPAC contains 71 domain items, organised over 10 distinct, but interrelated, domains. Each domain represents a core element of implementation research. Domain items are indicative of high-quality implementation research. Although ImpResPAC contains a large number of domain items, given the diversity and scope of implementation research proposal, it is possible that one or more ImpResPAC domains, and all associated items, may not be applicable. In such cases, we suggest that users only consider the domains that are applicable to the proposed study in question. Similarly, not every ImpResPAC domain item, in applicable domains, may be applicable. Again, in such cases we suggest that users on only consider the domain items applicable to the proposed study in question.
Although the majority of the EAP agreed that ImpResPAC is a feasible tool for use by researchers and practitioners (81%) and educators (71%), fewer agreed that it is a feasible tool for use by grant reviewers (64%). Additionally, a small number of experts raised concerns regarding the length of ImpResPAC. ImpResPAC includes 71 domain items that we recommend are used to determine domain scores. This means that users are expected to provide a maximum of eleven scores, depending on the number of applicable domains, including a global ImpResPAC score, and ten domain scores. It is not essential to indicate the applicability (not applicable, not addressed, partially/fully addressed) of each of the 71 domain items. Ultimately, we encourage users to apply ImpResPAC to suit their needs.
Notwithstanding the above, these findings may suggest that a shorter and/or simplified version may improve its feasibility. Several considerations arise from this. First, the assumption that reducing the number of domains or domain items will enhance the pragmatic quality of ImpResPAC is not straightforward. Pragmatic quality is increasingly recognised as a multidimensional concept. [21, 22] For example, the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) developed by Lewis et al,[22] includes cost, language accessibility, assessor burden (both in terms of ease of training and ease of interpretation, as well as length as key components of pragmatism. Third, there is an inherent trade-off between comprehensiveness and practicality that must be approached with care. Future research may wish to explore the relative importance and/or desirability of ImpResPAC domains and domain items, potentially paving the way for a shorter and/or more simplified version of ImpResPAC.
Scope and purpose of ImpResPAC
ImpResPAC was developed to provide a comprehensive appraisal of the conceptual and methodological quality of implementation research proposals in healthcare. Some members of the EAP reflected that it would be useful to develop a version of ImpResPAC that could be applied to completed implementation research. Although beyond the scope of the present study, we see value in developing a version of ImpResPAC to appraise the conceptual and methodological quality of completed implementation research.
A few members of the EAP, questioned the necessity for an intervention to be evidencebased. Although we ultimately decided not to replace ‘evidence-based intervention’ with ‘intervention’ in ImpResPAC, we are aware of the pitfalls, as described by Beidas and colleagues [23], with only studying the implementation of evidence-based interventions. As such, we do not necessarily discourage the use of ImpResPAC to appraise the conceptual and methodological quality of implementation research that involves the implementation of an intervention that has not yet demonstrated effectiveness.
Some members of the EAP suggested that ImpResPAC could be applied to deimplementation research proposals. Although we appreciate that de-implementation is an important sub-field of implementation science, we are of the opinion that although there may well be similarities between high-quality implementation and de-implementation research proposals, there are likely to be important differences, that are not captured in ImpResPAC. As such, we anticipate that ImpResPAC will require adaptations and modifications to be appropriate to appraise the conceptual and methodological quality of de-implementation research proposals. Therefore, we do not recommend that ImpResPAC is used to appraise the quality of de-implementation research proposals. Similarly, implementation science is now being used in disciplines and research fields far beyond healthcare (e.g., social care and climate change policy) where an adapted version of ImpResPAC is likely to be needed. We decided to develop appraisal criteria for implementation research proposals in healthcare for two important reasons. First, healthcare is an extremely large and broad field. Second, our team’s expertise is predominately in the development and delivery of implementation research in healthcare. Similar to other implementation frameworks and taxonomies that have been developed for application in healthcare and subsequently adapted and modified for use in other contexts and fields[24, 25], we anticipate that ImpResPAC is likely to require adaptation to ensure it is appropriate for application outside of healthcare.
Strengths and limitations
This study is the first to develop a comprehensive implementation research proposal appraisal tool based on in-depth feedback from a large international expert advisory panel. Where available and applicable, the content of ImpResPAC is based on state-of-the-science implementation research guidelines and recommendations. For example, the content of several ImpResPAC implementation outcomes domain items are based on recommendations for improved implementation outcomes reporting published by Lengnick-Hall et al., [10] and the content of several implementation strategy domain items are based on recommendations for specifying and reporting implementation strategies published by Proctor et al [9]. To support the application of ImpResPAC, comprehensive user and scoring instructions and a glossary were developed. Furthermore, to support the use of ImpResPAC, we developed a user-friendly, interactive ImpResPAC PDF (Supplementary File 5). Despite the strengths of this study, there are limitations. First, whether ImpResPAC is a reliable and valid tool remains to be tested. Second, a minority of the EAP expressed concerns regarding the acceptability and feasibility of applying the tool. Third, the EAP mainly consisted of experts from high-income countries, with very few experts from low- and middle-income countries. Whilst this is unsurprising given that conceptual and methodological advances in implementation science (e.g., conceptualisation of implementation outcomes, implementation strategies, development of most-widely cited implementation theories, models and frameworks), have predominantly stemmed from research conducted in high-income countries, we acknowledge that a more inclusive and balanced international EAP would have been ideal.
Future Research
As detailed in the ImpResPAC protocol paper, we plan to apply ImpResPAC to 50 research protocols published in Implementation Science to evaluate the psychometric strength (i.e., reliability and validity) of ImpResPAC. Furthermore, future research should identify determinants (i.e., barriers and facilitators) to implementing ImpResPAC, from multiple stakeholder perspectives (i.e., grant reviewers, researchers and practitioners, and educators), in order to select and develop tailored implementation strategies to improve the adoption and
implementation of ImpResPAC. Furthermore, understanding how ImpResPAC is used, including breadth and depth of application, to appraise implementation research proposals warrants future evaluation. We encourage ImpResPAC users to report how they use ImpResPAC to appraise implementation research proposals and critique its use.
CONCLUSION
ImpResPAC was developed to support research teams, grant reviewers and educators appraise the conceptual and methodological quality of implementation research proposals in healthcare. In this study, ImpResPAC has been content validated as an acceptable, feasible and appropriate tool to enable grant reviewers to identify the ‘best’ implementation research proposals to fund, support researchers and practitioners to develop high-quality implementation research proposals and provide a comprehensive tool to be used by educators to appraise implementation research proposals submitted as part of implementation science teaching and training initiatives.
REFERENCES
1.	Brownson RC, Colditz GA, Dobbins M, et al. Concocting that Magic Elixir: Successful Grant Application Writing in Dissemination and Implementation Research. Clin Transl Sci. 2015;8(6):710-716.
2.	Proctor EK, Powell BJ, Baumann AA, et al. Writing implementation research grant proposals: ten key ingredients. Implement Sci. 2012;7:96.
3.	Hull L, Goulding L, Khadjesari Z, et al. Designing high-quality implementation research: development, application, feasibility and preliminary evaluation of the implementation science research development (ImpRes) tool and guide. Implement Sci. 2019;14(1):80.
4.	Birken SA, Rohweder CL, Powell BJ, et al. T-CaST: an implementation theory comparison and selection tool. Implement Sci. 2018;13(1):143.
5.	Curran GM, Landes SJ, McBain SA, et al. Reflections on 10 years of effectivenessimplementation hybrid studies. Front Health Serv. 2022;2:1053496.
6.	Smith JD, Li DH, Rafferty MR. The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. 2020;15(1):84.
7.	Wolfenden L, Foy R, Presseau J, et al: Designing and undertaking randomised implementation trials: guide for researchers. BMJ. 2021, 372:m3721.
8.	Pinnock H, Barwick M, Carpenter CR, et al: Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356:i6795.
9.	Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.
10.	Lengnick-Hall R, Gerke DR, Proctor EK, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci. 2022;17(1):16.
11.	Crable EL, Biancarelli D, Walkey AJ, et al. Standardizing an approach to the evaluation of implementation science proposals. Implement Sci. 2018;13(1):71.
12.	Stadnick NA, Viglione C, Crable EL, et al. Enhancing review criteria for dissemination and implementation science grants. Implement Sci Commun. 2023;4(1):17.
13.	Sweetnam C, Goulding L, Davis RE, et al. Development and psychometric evaluation of the Implementation Science Research Project Appraisal Criteria (ImpResPAC) tool: a study protocol. BMJ Open. 2022;12(12):e061209.
14.	Sweetnam C, Goulding, L Hull. Implementation science research development (ImpRes) tool protocol assessment criteria (ImpResPAC): development and evaluation. Proceedings from the 2nd Annual UK Implementation Science Research Conference, 'Advancing the science of scaling up: Improving efficiency and effectiveness of implementation strategies in healthcare': meeting abstracts. Implement Sci. 2019:14(Suppl 2). 69.
15.	Niederberger M, Schifano J, Deckert S, et al. Delphi studies in social and health sciences-Recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study. PLoS One;2024;9(8):e0304651.
16.	Ericsson KA. The Cambridge Handbook of Expertise and Expert Performance. UK: Cambridge University Press; 2006.
17.	Naderifar M, Goli H, Ghaljaie F. Snowball Sampling: A Purposeful Method of Sampling in Qualitative Research. Strides Dev Med Educ. 2017;14(3):e67670.
18.	Polit D, Beck C. Essentials of nursing research: Appraising evidence fornursing practice.Lippincott Williams Wilkins; 2010.
19.	ICON Creative Design. www.iconcreativedesign.com. Accessed 15 Jan 2025.
20.	Francetic I, Meacock R, Elliott J, et al. Framework for identification and measurement of spillover effects in policy implementation: intended non-intended targeted nontargeted spillovers (INTENTS). Implement Sci Commun. 2022;3(1):30.
21.	Hull L, Boulton R, Jones F, et al. Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research. Transl Behav Med. 2022;12(11):1049-1064.
22.	Lewis CC, Mettert KD, Stanick CF, et al. The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implement Res Pract. 2021; 2:26334895211037391.
23.	Beidas RS, Dorsey S, Lewis CC, et al. Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem. Implement Sci. 2022;17(1):55.
24.	Cook CR, Lyon AR, Locke J, et al. Adapting a Compilation of Implementation Strategies to Advance School-Based Implementation Research and Practice. Prev Sci. 2019;20(6):914-935.
25.	Means AR, Kemp CG, Gwayi-Chore MC, et al. Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: a systematic review. Implement Sci. 2020, 15(1):17.
Abbreviations
EAP: Expert Advisory Panel
ImpRes: Implementation Science Research Development tool
ImpResPAC: Implementation Research Proposal Appraisal Criteria
INSPECT: ImplemeNtation and Improvement Science Proposals Evaluation CriTeria
NIH: National Institute for Health
NIHR: National Institute for Health and Care Research
Acknowledgements
The authors gratefully acknowledge members of the Expert Advisory Panel for their time and generosity in sharing their extensive experience and expertise.
The Authors’ contributions
LH and CS took the lead in conducting the Delphi study and drafting the manuscript. LH, CS, RD, LH, CS, ZK, AH, AB, IB, NS, and LG made substantial contributions to the conception or design of the work and data analysis. RD, ZK, AH, AB, IB, NS, and LG provided critical revisions to the manuscript and approved the final version of the manuscript and agreed to be accountable for all aspects of the work.
Funding
This study is funded by the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. IB is part supported by the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust and by the NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London.
Availability of data and materials
Data are available upon reasonable request directed to the corresponding author.
Declarations
Ethics approval and consent to participate
This study was registered, and minimal risk ethical clearance granted by The Research Ethics Office, King’s College London (reference numbers: MRA-20/21-20807 & MRM-23/24- 20807). All study participants provided informed consent prior to participation in the study.
Consent for publication
Not applicable.
Competing interests
NS is the director of London Safety and Training Solutions ltd, which offers training and improvement and implementation solutions to healthcare organisations and the pharmaceutical industry. The other authors have no conflicts of interest to declare. All other authors declare that they have no competing interests.
Table 1: Composition of the ImpResPAC Expert Advisory Panel

Name	Academic and/or healthcare institution affiliation(s)	Geographical location (country)	ImpResPAC domain reviewed	
Sara L Ackerman**	University of California San Francisco	USA	PPIE	
Prajakta Adsul**	University of New Mexico	USA	SIE	
Miya Barnett**	University of California Santa Barbara	USA	EE	
Barbara Barrett**	King's College London	England, UK	EE	
Anna Bergström**	Uppsala University, Karolinska Institutet	Sweden	IRC, UC	
Sarah Birken**	Wake Forest University School of Medicine	USA	TMF, IS	
Sabah Boufkhed*	The University of Manchester	England, UK	IO, SPO	
Louise Brown**	University of Bath
Stellenbosch University	England, UK South Africa	IS, SIE	
Adithya Cattamanchi**	University of California Irvine	USA	IS	
Caroline Clarke**	University College London	England, UK	ID, IS	
Tara Clinton-McHarg**	The University of Melbourne	Australia	IO, SPO	
Peter Craig*	University of Glasgow	Scotland, UK	ID	
Geoffrey Curran*	University of Arkansas for Medical Sciences
University of Limerick	USA
Ireland	IRC, UC	
Alex Dopp**	RAND Corporation	USA	EE	
Alicia G. Dugan**	University of Connecticut	USA	ID	
Jacquelyn S Favours*	Vanderbilt School of Medicine and the Meharry- Vanderbilt Alliance	USA	PPIE	
Tracy Finch**	Northumbria University	England, UK	IRC, UC, TMF, SIE	
Signe Flottorp**	University of Oslo	Norway	ID	
Robbie Foy**	University of Leeds	England, UK	IRC, UC	
Hannah E. Frank**	The Warren Alpert Medical School of Brown University	USA	IS	
Kara Gray-Burrows**	University of Leeds	England, UK	PPIE	
Claudia Hanson**	Karolinska Institutet	Sweden	IRC, UC	
Ties Hoomans**	London School of Economics	England, UK	EE	
Elaine C Khoong**	University of California San Francisco	USA	TMF	
Roman Kislov**	Manchester Metropolitan University; The University of Manchester	England, UK	SIE	
Rebecca Lawton*	University of Leeds	England, UK	SIE	
Jennifer Leeman**	University of North Carolina at Chapel Hill	USA	IS	
Rebecca Lengnick-Hall*	Brown School, Washington University in St. Louis	USA	ID, IO, SPO	
Moira McNulty*	University of Chicago	USA	IRC, UC	
Sara Malone*	Washington University in St.	USA	IO, SPO	

	Louis			
Graham Martin**	University of Cambridge	England, UK	IRC, UC, SIE	
Ruben G. Martinez*	Brown University: Providence	USA	IO, SPO	
Allison Metz*	The University Of North Carolina At Chapel Hill	USA	SIE	
Aaloke Mody**	Washington University in St. Louis	USA	IRC, UC, IO, SPO	
Joanna Moullin**	Curtin University	Australia	TMF	
Janet Myers*	University of California San Francisco	USA	IO, SPO	
Per Nilsen**	Linköping University and Halmstad University	Sweden	TMF, ID, IS, IO, PPIE, SIE	
Kathryn Oliver*	London School of Hygiene & Tropical Medicine	England, UK	IRC, UC	
Dennis Pérez*	Instituto de Medicina Tropical Pedro Kourí	Cuba	TMF	
Lisa Pfadenhauer*	Ludwig-Maximilians- Universität München	Germany	ID	
Hilary Pinnock**	The University of Edinburgh	Scotland, UK	IRC, UC, TMF	
Sebastian Potthoff**	Northumbria University	England, UK	ID, IS, SIE	
Byron Powell**	Washington University in St. Louis	USA	TMF, IS	
Borsika Rabin*	University of California San Diego	USA	TMF	
Angus I G Ramsay**	University College London	England, UK	ID, IS	
Jamie Ross*	Queen Mary University of London	England, UK	ID, PPIE	
Alex Rowe*	Centers for Disease Control and Prevention, Atlanta	USA	IS, IO, SPO	
Lisa Saldana*	Oregon Social Learning Center	USA	EE	
Jane Sandall*	King’s College London	England, UK	ID	
Joanna Schellenberg**	London School of Hygiene & Tropical Medicine	England, UK	IRC, UC	
Kelli Scott*	Northwestern University Feinberg School of Medicine	USA	IRC, UC	
Kate Seers*	University of Warwick	USA	TMF, ID	
Christopher Shea*	University of North Carolina-Chapel Hill	USA	IS	
Justin D. Smith**	University of Utah	USA	IRC, UC	
Matthew A Spinelli**	University of California San Francisco	USA	ID	
Nicole Stadnick**	University of California San Diego	USA	ID, IS	
Taren Swindle**	University of Arkansas for Medical Sciences	USA	IS	
Carolyn Tarrant**	University of Leicester	England, UK	IRC, UC, SIE	
Silia Vitoratou**	King's College London	England, UK	IO, SPO	
Thomas J. Waltz**	Eastern Michigan University	USA	IS	
Michel Wensing*	Heidelberg University	Germany	EE	
Michelle White*	Great Ormond Street	England, UK	TMF	

	Hospital
Kings College London			
Cathleen Willging**	Pacific Institute for Research and Evaluation	USA	SIE	
Til Wykes**	King’s College London	England, UK	PPIE	
Brian Yates*	American University, Washington, DC	USA	EE	
Serene Yoong*	Deakin University	Australia	IRC, UC, IO, SPO	
Alexandra Ziemann**	University of Bath	England, UK	ID	
*Expert Advisory Panel members that completed Survey A; ** Expert Advisory Panel members that completed Survey A & B. Note: One member of the Expert Advisory Panel did not consent to be named and acknowledged in research outputs.

IRC=Implementation research characteristics; TMF=Theories, models and frameworks;
ID=Implementation determinants; IS=Implementation strategies; SPO=Service and patient outcomes;
IO=Implementation outcomes; UC=Unintended consequences; EE=Economic evaluation;
SIE=Stakeholder involvement and engagement; PPIE=Patient and public involvement and
engagement.

Table 2: Characteristics of the ImpResPAC Expert Advisory Panel
	Number (Percentage)	
Academic appointment		
Yes	50 (94.3%)	
No	3 (5.7%)	
Level of academic appointment (%)		
Lecturer (Assistant professor)	12 (22.6%)	
Senior Lecturer/Reader (Associate professor)	20 (37.7%)	
Professor (Full professor)	17 (32.0%)	
Other
Senior/Principal Research Fellow	2 (3.8%)
2 (3.8%)	
Type of expertise (%)		
Implementation science only	19 (35.8%)	
Clinical practice only	0 (0.0%)	
Implementation science and clinical practice	18 (33.9%)	
Other
Implementation Science + Other (including Public Health, Anthropology, Applied Health Research, Health Services Research, Social Science, Digital Health, Health and Social Care Economics, Community based)
Health Economics/Economic Evaluation in Healthcare
Evidence Use
Epidemiology and Statistics
Psychometrics/Statistics
Psychology
Implementation Science, Clinical Practice and Clinical Informatics	16 (30.2%)
9 (16.9%)
2 (3.8%)
1 (1.9%)
1 (1.9%)
1 (1.9%)
1 (1.9%)
1 (1.9%)	
	Years of experience	
Years worked in the field the field of implementation science/practice	13.8 years (mean)
734 years (cumulative total)	
Experience of reviewing implementation research funding proposals		
Yes	42 (79.2%)	
No	11 (20.8%)	
Years of experience reviewing implementation research funding proposals	7.1 years (mean)
378 (cumulative total)	
*Data based on information provided by 53/68 members of the EAP.

Table 3: Evaluation of ImpResPAC (Survey B)
Survey questions	Response
Count (%)	
	Yes	No	
Do ImpResPAC domain items represent and reflect high-quality conceptual and methodological elements of implementation research proposals?	40 (95.2%)	2 (4.8%)	
Are there any domain items missing from ImpResPAC?	11 (26.2%)	31 (73.8%)	
Is the ImpResPAC domain item wording clear?	39 (92.9%)	3 (7.1%)	
Is the ImpResPAC overview adequate and clear?	40 (95.2%)	2 (4.8%)	
Are the ImpResPAC user instructions adequate and clear?	37 (88.1%)	5 (11.9%)	
Is the ImpResPAC scoring scale and associated anchors appropriate and clear?	34 (81.0%)	8 (19.0%)	

Table 4: Evaluation of the acceptability, feasibility and appropriateness of ImpResPAC
Median (IQR)
Acceptability statements	
ImpResPAC is an acceptable tool to be used by reviewers to appraise the conceptual and methodological quality of implementation research proposals	4 (2)	
ImpResPAC is an acceptable tool to be used by researchers and practitioners to appraise the conceptual and methodological quality of their implementation funding proposals.	4 (1)	
ImpResPAC is an acceptable tool to be used for educators to appraise the conceptual and methodological quality of implementation research proposals submitted as part of implementation capacity building initiatives.	4 (1.25)	
Feasibility statements	
ImpResPAC is a feasible tool to be used by reviewers to appraise the conceptual and methodological quality of implementation research proposals	4 (1)	
ImpResPAC is a feasible tool to be used by researchers and practitioners to appraise the conceptual and methodological quality of their implementation funding proposals.	4 (0)	
ImpResPAC is a feasible tool to be used for educators to appraise the conceptual and methodological quality of implementation research proposals submitted as part of implementation capacity building initiatives.	4 (1)	
Appropriateness statements	
ImpResPAC is an appropriate tool to be used by reviewers to appraise the conceptual and methodological quality of implementation research proposals	4 (1)	
ImpResPAC is an appropriate tool to be used by researchers and practitioners to appraise the conceptual and methodological quality of their implementation funding proposals.	4 (1)	
ImpResPAC is an appropriate tool to be used for educators to appraise the conceptual and methodological quality of implementation research proposals submitted as part of implementation capacity building initiatives.	4 (1)	
1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree

Table 5: Implementation Research Proposal Appraisal Criteria (ImResPAC)
ImpResPAC domain	ImpResPAC domain items	
Implementation research characteristics	The proposed study explicitly seeks to address an implementation problem; it clearly identifies and describes both the associated quality and/or coverage of care gap, and the evidence-based intervention selected to address the problem.
Clear and strong justification is provided to support the selection of the evidence-based intervention to be implemented to address the quality and/or coverage of care gap (e.g., prior efficacy and/or effectiveness studies, patient preference). Literature and/or local data used to support the evidence-based intervention to be implemented is up to date and has been critically appraised. Implementation and wider study aims and objectives are explicitly and clearly articulated, and align with the proposed study design, methods, measures, outcomes and analysis plan.
Issues relating to scientific and health equity have been thoroughly considered and described in detail. Existing inequities in the quality and/or coverage of care gap have been considered with the aim of achieving equity and avoiding exacerbating existing inequities.
Design and methods of the proposed study are clearly stated and comprehensively described (e.g., qualitative, semi-structured interviews) and align appropriately with the aims and objectives of the study.
The conceptual linkages between all elements of the proposed study (i.e., the evidence-based intervention, implementation determinants, implementation strategies, mechanisms of action and outcomes) are comprehensively depicted in a programme theory, logic model or theory of change.
Stage(s) of implementation of the proposed study and the associated activities planned at each stage are described in detail.
Planned adaptations and modifications to the evidence-based intervention are clearly described. Reasons (e.g., available resources), goals (e.g., to reduce costs) and the process of adaptations and modifications are clearly described.1 Core components of the evidence-based intervention are retained. Clear intention to explore the impact of planned adaptations and modifications to the evidence-based intervention on service and patient outcomes and implementation outcomes is stated.
Clear intention to document unplanned adaptations and modifications to the evidence-based intervention that arise is described. Clear intention to explore the impact of unplanned adaptations and modifications on service and patient outcomes and implementation outcomes is stated.
How the proposed study will contribute to the conceptual and/or methodological advancement of the field is clearly articulated.
Taking into consideration the expertise and capacity of the research team, the proposed study can be accomplished within the available timeframe, requested budget and resources.	
Implementation theories, models and frameworks	Clear and strong justification is provided to support the selection of implementation theories, models and frameworks (TMFs). For example, by citing appropriate literature (e.g., the TMFs have been applied successfully to similar interventions), using implementation TMF criteria selection tools (e.g., T-CaST)2 and/or using data from implementation site(s) (e.g., pilot data	

	identifying relevant/important factors of implementation, which map well onto a particular implementation TMF). If pre-existing implementation TMFs are not applied, and the study aims to develop an implementation TMF, clear and strong justification is provided.
The chosen implementation TMF(s) inform and structure all relevant aspects of the proposed study (e.g., study design, aims and objectives, outcomes, analysis, interpretation of results).
If implementation TMFs are to be partially applied, clear and strong justification is provided.
Proposed adaptations and modifications (above and beyond partial application) including extensions and the innovative use of chosen implementation TMFs are clearly and comprehensively described, and an explanation is provided.
If more than one implementation TMF is selected, the unique contribution of each, and the benefits of combining them is described, and how they will be integrated is explained.
Constructs/domains of implementation TMFs are proposed to be evaluated, using appropriate methods (quantitative, qualitative, mixed- methods). If proposed to be measured quantitatively, psychometrically robust (e.g., valid and reliable) and pragmatic (i.e., practical) instruments are to be utilised, where available.	
Implementation determinants	The context in which the proposed study is to take place and implementation determinants, including contextual determinants (i.e., factors theorised and/or identified to influence implementation), are clearly described.
The proposed study aims to identify implementation determinants, operating at different levels, either pre-, during and/or post-implementation, as appropriate to address the study aims and objectives.
Relevant stakeholders (e.g., those responsible for implementation/delivery and those expected to benefit) have been, or will be, involved in the identification of implementation determinants.
Clear and detailed description of the methods to be used to identify (e.g., interviews, document analysis) and/or evaluate (e.g., quantitative instruments) implementation determinants is provided.
Clear intention to document changes to the context (e.g., policy change) that arise during implementation is described.	
Implementation strategies	The implementation strategies proposed to be used are described in sufficient detail to allow replication and evaluation (i.e., named, defined, specified/operationalised and actor, action, target, temporality, dose specified).3 If appropriate, implementation strategies proposed to be used are conceptually linked to an existing implementation strategy taxonomy (e.g., ERIC implementation strategy compilation,4 Strategies to support the use of research in clinical practice taxonomy5).
The implementation strategies proposed to be used have been, or will be, selected and tailored to address implementation determinants.
Implementation determinant-implementation strategy linkages and the hypothesized mechanisms through which implementation strategies are expected to work are explicitly stated using, for example, causal pathway models or mechanism mapping.
The process (e.g., implementation determinant identification → implementation determinant prioritisation → implementation strategy selection and tailoring) and methods (e.g., concept mapping, conjoint analysis) proposed to be used to select and tailor implementation strategies are clearly	

	described.
Relevant stakeholders (e.g., those responsible for implementation/delivery and those expected to benefit) have been, or will be, involved in the selection and tailoring of the proposed implementation strategies.
Implementation strategy selection and tailoring is theoretically, empirically (if evidence is available) and/or pragmatically (e.g., feasibility) justified.
The proximal outcome (e.g., increased screening rate), and implementation outcome(s) (e.g., screening feasibility) that are targeted for improvement by the proposed implementation strategy are explicitly stated.
The proposed monitoring (e.g., fidelity of delivery and/or receipt) and evaluation (e.g., impact on intermediate outcomes, implementation outcomes and/or proximal outcomes) of implementation strategies are clearly described.
Planned adaptations and modifications to the implementation strategy are clearly described. Reasons (e.g., available resources), goals (e.g., to reduce costs) and the process of adaptations and modifications are clearly described.6 Core components of the implementation strategy are retained. Clear intention to explore the impact of planned adaptations and modifications to the implementation strategy on service and patient outcomes and implementation outcomes is stated.
Clear intention to document unplanned adaptations and modifications to the implementation strategy that arise is described. Clear intention to explore the impact of unplanned adaptations and modifications to the implementation strategy on service and patient outcomes and implementation outcomes is stated.	
Service and patient outcomes	The proposed study includes the evaluation of service and/or patient outcome(s) as appropriate to address the study aims and objectives. Clear and explicit evidence that relevant stakeholders (e.g., those responsible for implementation/delivery and those expected to benefit) were involved, or will be involved, in the identification and selection of relevant and important service and/or patient outcomes to be evaluated.
The method(s) (qualitative/quantitative/mixed-methods) to be used to evaluate service and/or patient outcome(s) are clearly described and appropriate to address the study aims and objectives.
The proposed measurement method(s), level(s) of measurement/analysis, timing and frequency of service and/or patient outcome(s) data collection are appropriate to address the study aims and objectives. The time horizon over which service and patient outcomes are to be evaluated is clearly stated.
The degree of focus placed on evaluating service and/or patient outcomes (i.e., the effectiveness of the selected evidence-based intervention) is guided by the strength of evidence for the intervention in question, the extent of planned adaptations and modifications to the evidence-based intervention and/or implementation strategy, and the context in which implementation is to take place.7
Where quantitative service and/or patient outcome instrument(s) are proposed to be used to assess service and/or patient outcomes, justification for selection of an existing instrument or development of a new instrument is reported. Psychometric properties (e.g., validity and reliability) and/or	

	pragmatic (i.e., practical) qualities and/or previous application to the population of interest are stated to justify selection of an existing instrument or development of a new instrument. If a new instrument is being developed, there is clear intention to evaluate its psychometric properties.
A clear and detailed description of the service and/or patient outcome data analysis plan is presented. How the service and/or patient outcomes are to be treated (e.g., as a predictor variable) and how service and/or patient outcomes will be analysed (e.g., correlational analysis) relative to other constructs (e.g., implementation outcomes) is clearly stated. For qualitatively evaluated service and/or patient outcomes, how data will be analysed and interpreted (e.g., thematic analysis) is clearly stated.	
Implementation outcomes	The proposed study includes the evaluation of one or more implementation outcomes as appropriate to address the study aims and objectives.
Clear and explicit evidence that all relevant stakeholders (e.g., those responsible for implementation/delivery and those expected to benefit) were involved, or will be involved, in the identification and selection of relevant and important implementation outcomes to be evaluated.
Each implementation outcome to be evaluated is clearly stated and an operational definition provided.8
The method(s) (qualitative/quantitative/mixed-methods) to be used to evaluate implementation outcomes are clearly described and appropriate to address the study aims and objectives.
The proposed measurement method(s), level(s) of measurement/analysis, timing and frequency of implementation outcome data collection are appropriate to address the study aims and objectives.8 The time horizon over which implementation outcomes are to be evaluated is clearly stated.
Where quantitative implementation outcome instruments are proposed to be used to assess implementation outcomes, justification for the selection of an existing instrument or development of a new instrument is reported.
Psychometric properties (e.g., validity and reliability) and/or pragmatic (i.e., practical) qualities and/or previous application to the population of interest are stated to justify the selection of an existing instrument or development of a new instrument. If a new instrument is being developed, there is a clear intention to evaluate its psychometric properties.
A clear and detailed description of the implementation outcome data analysis plan is presented. How the implementation outcomes are to be treated (e.g., as a predictor variable) and how implementation outcomes will be analysed (e.g., correlational analysis) relative to other constructs (e.g., implementation determinants) is clearly stated.8 For qualitatively evaluated implementation outcomes, how data will be analyzed and interpreted (e.g., thematic analysis) is clearly stated.	
Unintended consequences	Clear and detailed discussion of the intention to explore whether anticipated and unanticipated unintended consequences (including anticipated drawbacks, unexpected benefits and unexpected drawbacks) occur as a result of implementation.9
If relevant, anticipated unintended consequences, particularly if negative, are stated and described.
The intention to explore unintended consequences (both anticipated and unanticipated) from different stakeholder perspectives (e.g., patients, healthcare providers) and at different levels (e.g., individual, team, organisation) is described.
The method(s) proposed to be used to detect, measure, analyse and better	

	understand why unintended consequences (both anticipated and unanticipated) occur are clearly described (e.g., semi-structured interviews with those responsible and/or affected by implementation). The phase(s) of implementation (e.g., during, post-implementation) that detection of unintended consequences is planned is clearly stated.
Ethical, moral and practical implications of how to respond to both anticipated and unanticipated unintended consequences (e.g., de-implementation or adaptation and/or modification to the evidence-based intervention and/or implementation strategy), to prevent or mitigate their effects, are described.	
Economic evaluation	The economic evaluation research question(s) and the type of economic evaluation (including relevant costs and consequences) proposed to be used are clearly articulated. The alternatives being compared (e.g., the evidence-based intervention compared to standard practice and/or different implementation strategies) are clearly described.
The perspective(s) (e.g., societal or health system perspective) of the economic evaluation is clearly stated and justified in relation to the context of the research and the time horizon over which costs and consequences are to be evaluated.
There is a clear statement of which outcomes are to be quantified and/or qualitatively explored where relevant (e.g., patient/population health outcomes, improvement in healthcare delivery processes/provider outcomes, spillovers/unintended cost impacts to non-targeted populations and sites) and these are appropriate to the aims and objectives of the economic evaluation and informed by stakeholder involvement.
The proposed design of the economic evaluation, where appropriate, has built in an allowance for describing/reporting the distributional implications of implementation (e.g., reducing health disparities).
The approach to measurement of costs (including quantification and valuation of resources to be used for implementation strategies and evidence-based intervention) is clearly stated, including reference to appropriate data sources and methods.
Clear and explicit recognition of implementation strategy costs, including those in the initial implementation phase and those related to sustainment and scale-up, if applicable.
If relevant to the wider research aims and study design, the economic evaluation allows the incremental costs and consequences specific to the implementation strategy and/or evidence-based intervention to be separately determined.
The planned approach to summarising implementation cost-effectiveness (e.g., use of incremental cost-effectiveness ratios, cost-benefit ratios, net monetary benefit) is clearly stated and appropriate.
The approach to sensitivity analysis to evaluate the robustness of conclusions to uncertainty (including those relating to sampling error) around the value of key implementation, clinical, epidemiological and economic parameters is clearly stated and appropriate.	
Stakeholder* involvement and engagement	Clear and explicit evidence that relevant stakeholders were meaningfully involved in developing the study proposal (e.g., shaping research questions) and, if appropriate, are members of the research team (e.g., as co-applicants or as members of a study steering/advisory group).
Clear and explicit evidence of the intention to meaningfully engage and/or involve stakeholders in all relevant later stages of the study (e.g., interpreting and disseminating research findings).
The purpose, anticipated benefits, and impact of planned engagement and/or involvement activities are clearly described. The level of engagement and/or involvement are appropriate to meet the purpose of engagement and/or involvement activities.
The proposed study is informed by feedback from stakeholders, including their preferences and priorities. It strives to be a meaningful partnership between researchers and relevant stakeholders.
Planned engagement and/or involvement methods are appropriate, clearly	
	described, and promote meaningful engagement and/or involvement.	
	There is a clear plan to give feedback to stakeholders engaged and/or involved in the proposed study on the impacts of their engagement and/or involvement.	
Patient and public involvement and engagement	Clear and explicit evidence that relevant patients, service users, carers and/or members of the public were meaningfully involved in developing the study proposal (e.g., shaping research questions) and, if appropriate, are members of the research team (e.g., as co-applicants or as members of a study steering/advisory group).	
	Clear and explicit evidence of the intention to meaningfully engage and/or involve patients, service users, carers and/or members of the public, in all relevant later stages of the study (e.g., undertaking interviews with research participants, interpreting and disseminating research findings).	
	The purpose, anticipated benefits and impact of planned engagement and/or involvement activities are clearly described. The level and nature of engagement and/or involvement activities are appropriate to meet the purpose of engagement and/or involvement activities.	
	The proposed study is informed by feedback from patients, service users, carers and/or members of the public, including their preferences and priorities. It strives to be a meaningful partnership between researchers and relevant patients, service users, carers and/or members of the public.	
	Planned engagement and/or involvement methods are appropriate, clearly described, and promote meaningful engagement and/or involvement.	
	There is a clear plan to give feedback to patients, service users, carers and/or members of the public engaged and/or involved in the proposed study on the impacts of their engagement and/or involvement.	
All terms/acronyms in bold are defined in the ImpResPAC glossary.

* Stakeholders: Stakeholders in this domain include healthcare providers, managers, commissioners, policy makers etc.
1 Content based on Stirman et al. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58.
2T-CaST: Implementation theory comparison and selection tool. Content refers to Birken et al. T- CaST: an implementation theory comparison and selection tool.Implement Sci. 2018;13(1):143.
3 Content based on Proctor et al. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.
4ERIC: Expert Recommendations for Implementing Change. Content refers to Powell et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.
5 Content refers to Boaz et al. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implement Sci. 2024;19(1):15.
6Content based on Miller et al. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci. 2021;16(1):36.
7 Content based on Curran et al. Reflections on 10 years of effectiveness-implementation hybrid studies. Front Health Serv. 2022;2:1053496.
8Content based on Lengnick-Hall et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci.2022;17(1):16.
9Content based on Toma et al. A balanced approach to identifying, prioritising and evaluating all potential consequences of quality improvement: modified Delphi study. BMJ Open.
2019;9(3):e023890.
10Stakeholders: Stakeholders in this domain include healthcare providers, managers, commissioners, policy makers etc.
Figure 1: ImpResPAC user and scoring instructions
ImpResPAC introduction
ImpResPAC is a comprehensive and in-depth quantitative appraisal tool to evaluate the conceptual and methodological quality of implementation research proposals in healthcare.
ImpResPAC domains
ImpResPAC contains 10 distinct, but interrelated, domains. Each domain represents a core element of implementation research.
ImpResPAC domains: (1) Implementation research characteristics, (2) Implementation theories, models and framework, (3) Implementation determinants, (4) Implementation strategies, (5) Service and patient outcomes, (6) Implementation outcomes, (7) Unintended consequences, (8) Economic evaluation, (9) Stakeholder involvement and engagement, and (10) Patient and public involvement and engagement.
ImpResPAC domains are equally weighted.
ImpResPAC domain items
Each ImpResPAC domain contains several items. Each item is indicative of high-quality implementation research.
Example domain item from the implementation research characteristics domain:
The proposed study explicitly seeks to address an implementation problem; it clearly identifies and describes both the associated quality and/or coverage of care gap, and the evidence-based intervention selected to address the problem.
ImpResPAC domains item are equally weighted.
ImpResPAC user instructions
ImpResPAC application and scoring instructions
Given the diversity and scope of implementation research, it is possible that one or more ImpResPAC domains, and all associated items, may not be
applicable. Only consider the domains that are applicable to the proposed study in question. Similarly, not every ImpResPAC domain item, in applicable domains, may be applicable. Only consider the domain items applicable to the proposed study in question. For each applicable item, consider whether the proposed study partially or fully addresses the content of the domain item. Based on the percentage of domain items that are partially or fully addressed, a score of 1-5 is assigned to each ImpResPAC domain:
1	= Very poor: Very few applicable items are partially or fully addressed (0-20%*)
2	= Poor: Few applicable items are partially or fully addressed (21-40%*)
3	= Fair: Some applicable items are partially or fully addressed (41-60%*)
4	= Good: Most applicable items are partially or fully addressed (61-80%*)
5	= Excellent: Almost all applicable items are partially or fully addressed (81-100%*)
*Please note the above suggested percentage cut-off ranges are for guidance only.
Partially versus fully addressed domain items
The distinction between ‘partially’ and ‘fully’ accounts for the fact that many domain items contain more than one element. Therefore, it is possible that research proposals may ‘partially’ or ‘fully’ address the content elements of each domain item.
Example of an ImpResPAC item containing more than one element: Proposed adaptations and modifications (above and beyond partial application), including extensions and the innovative use of chosen implementation theories, models and frameworks (TMFs) are clearly and comprehensively described, and an explanation is provided.

Research teams may address one element in the above ImpResPAC item (e.g., adaptations and modifications to TMFs are clearly and comprehensively described) but not the other (e.g., an explanation is not provided).
Note: Some ImpResPAC domain items reference specific terms, tools and methods that are indicative of excellent conceptual and methodological elements of implementation research. It is possible that a proposal fully addresses an ImpResPAC domain item without explicitly referring to the terms, tools and methods referenced in ImpResPAC. In such cases, we recommend that ImpResPAC users use their judgement to determine whether a proposal fulfils the criteria.
Calculating the global ImpResPAC score
Given the fact that not every ImpResPAC domain will be applicable in every implementation research proposal, we suggest that you calculate the global ImpResPAC median score, range and interquartile range of the applicable domain scores. This will allow you to compare the conceptual and methodological quality of implementation research proposals with varying numbers of applicable domains. The overall global ImpResPAC median score will range from 1-5, with higher scores indicating higher quality implementation research proposals.
Please note: In addition to calculating the global ImpResPAC median score, we recommend that you consider individual domain scores. This is important because there may be instances in which the global ImpResPAC median score may be inflated by a number of ImpResPAC domains that score very highly, but consist of a number of ImpResPAC domains that score very poorly.
ImpResPAC is intended to measure the conceptual and methodological quality of implementation research proposals in healthcare, rather than as a measure of reporting quality. As such, if domain(s) and/or item(s) deemed to be applicable, are not addressed, this should be viewed negatively. If being used for high-stakes assessment (e.g., funding decisions), we suggest that clarification is sought from research teams regarding ImpResPAC domain(s) and/or item(s) not addressed.