Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

Abstract

Background: The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

Methods: There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

Results: Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

Limitations: The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

Conclusions: This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

Future work: We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

Funding: This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

Keywords: COMPLEX INTERVENTION; COMPLEXITY; CONTEXT; DEVELOPMENT; EVALUATION; FEASIBILITY; IMPLEMENTATION; INTERVENTION REFINEMENT; PROGRAMME THEORY; STAKEHOLDERS; SYSTEMS; UNCERTAINTIES.

Figure 1. Key elements for developing and evaluating complex interventions (consultation version).

Figure 2. Framework for addressing complexity within evaluation (consultation version).

Figure 3

Main phases and core elements of complex intervention research. The diagram illustrates the non-linear, iterative pathway through the research process. The core elements should be repeatedly considered and may lead to further work at the same phase or movement to another phase. Reproduced with permission from Skivington et al. This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Figure 4

Two pathways of evidence generation. Reproduced with permission from Ogilvie et al. This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure.

Figure 5. Case study demonstrating how the FFIT programme moved through the research process. SPFL, Scottish Premier Football League.

Figure 6

Logic model of a complex intervention to improve payment of salaries to health workers. Reproduced with permission from Maini et al. This is an Open Access article distributed in accordance with the Creative Commons Attribution Non-Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/. The figure includes minor additions and formatting changes to the original figure. ASSP, Accès Aux Soins de Santé Primaires; CODESA, community health committee; DFID, Department for International Development; FP, family planning; HR, human resources; HRH, human resources for health; iHRIS, IntraHealth International (Chapel Hill, NC, USA); IMA, IMA World Health (Washington, DC, USA); MoH, Ministry of Health; SNIs, single-number indicators; WISN, Workload Indicator of Staffing Need. Sources included as per original.^–

Figure 7

Dark logic model for Communities in Charge of Alcohol. Light blue arrows, potential links between harms; orange arrows, links between logic model inputs and outputs; black arrows, dark logic model mechanism; dark blue arrows, main logic model mechanism. AHC, Alcohol Health Champion. Reproduced with permission from Cook et al. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. The figure includes minor additions and formatting changes to the original figure.

Figure 8

Conceptual system map for the UK SDIL. Context: modifying factors (e.g. age, gender, socioeconomic position and household composition) and macro-level factors (e.g. Brexit, economic climate, agricultural policy, global governance and trade). Adapted with permission from White et al.

Figure 9

Physical activity system map with interventions. Reproduced from Signal LN, Walton MD, Mhurchu CN, et al. Tackling ‘wicked’ health promotion problems: a New Zealand case study, Health Promotion International 2013;28(1):84–94, by permission of Oxford University Press.