Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

doi:10.1136/bmjopen-2024-084164

. 2024 Mar 11;14(3):e084164.

doi: 10.1136/bmjopen-2024-084164.

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Jack Wilkinson¹, Calvin Heal², George A Antoniou^{3

4}, Ella Flemyng⁵, Zarko Alfirevic⁶, Alison Avenell⁷, Ginny Barbour⁸, Nicholas J L Brown⁹, John Carlisle¹⁰, Mike Clarke¹¹, Patrick Dicker¹², Jo C Dumville^{13

14}, Andrew Grey¹⁵, Steph Grohmann⁵, Lyle Gurrin¹⁶, Jill Alison Hayden^{17

18}, James Heathers¹⁹, Kylie Elizabeth Hunter²⁰, Toby Lasserson⁵, Emily Lam²¹, Sarah Lensen²², Tianjing Li²³, Wentao Li²⁴, Elizabeth Loder^{25

26}, Andreas Lundh^{27

28}, Gideon Meyerowitz-Katz²⁹, Ben W Mol³⁰, Neil E O'Connell^{31

32}, Lisa Parker³³, Barbara K Redman³⁴, Lene Seidler²⁰, Kyle A Sheldrick³⁵, Emma Sydenham³⁶, David Torgerson³⁷, Madelon van Wely^{38

39}, Rui Wang³⁰, Lisa Bero⁴⁰, Jamie J Kirkham²

Affiliations

¹ Centre for Biostatistics, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK jack.wilkinson@manchester.ac.uk.
² Centre for Biostatistics, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.
³ Manchester Vascular Centre, Manchester University NHS Foundation Trust, Manchester, UK.
⁴ Division of Cardiovascular Sciences, School of Medical Sciences, Manchester Academic Health Science Centre, The University of Manchester Faculty of Biology Medicine and Health, Manchester, UK.
⁵ Evidence Production and Methods Directorate, Cochrane, London, UK.
⁶ Department of Women's and Children's Health, University of Liverpool, Liverpool, UK.
⁷ Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
⁸ Medical Journal of Australia, Sydney, New South Wales, Australia.
⁹ Department of Psychology, Linnaeus University, Växjö, Sweden.
¹⁰ Anaesthesia and Critical Care, Torbay Hospital, Torquay, UK.
¹¹ Northern Ireland Methodology Hub, Queen's University Belfast, Belfast, UK.
¹² Department of Epidemiology and Public Health, Royal College of Surgeons in Ireland, Dublin, Ireland.
¹³ Division of Nursing, Midwifery and Social Work, School of Health Sciences, The University of Manchester, Manchester, UK.
¹⁴ NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK.
¹⁵ Department of Medicine, University of Auckland, Auckland, New Zealand.
¹⁶ School of Population and Global Health, The University of Melbourne, Melbourne, Victoria, Australia.
¹⁷ Community Health & Epidemiology, Dalhousie University, Halifax, Nova Scotia, Canada.
¹⁸ Department of Community Health & Epidemiology, Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada.
¹⁹ SafeBeat Rx Inc, St Louis, Missouri, USA.
²⁰ Evidence Integration, NHMRC Clinical Trials Centre, University of Sydney, Sydney, New South Wales, Australia.
²¹ Independent Lay Member, Cheshire, UK.
²² Department of Obstetrics and Gynaecology, Royal Women's Hospital, University of Melbourne, Melbourne, Victoria, Australia.
²³ Department of Ophthalmology, University of Colorado, Denver, Colorado, USA.
²⁴ Department of Obstetrics and Gynecology, Monash University, Clayton, Victoria, Australia.
²⁵ BMJ Publishing, London, UK.
²⁶ Department of Neurology, Brigham and Women's Hospital, Boston, Massachusetts, USA.
²⁷ Centre of Evidence-Based Medicine Odense and Cochrane Denmark, Department of Clinical Research, University of Southern Denmark, Odense, Denmark.
²⁸ Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital-Bispebjerg and Frederiksberg, Kobenhavn, Denmark.
²⁹ School of Health and Society, University of Wollongong, Wollongong, New South Wales, Australia.
³⁰ Department of Obstetrics and Gynaecology, Monash University, Clayton, Victoria, Australia.
³¹ Department of Clinical Science, Brunel University, Uxbridge, UK.
³² Centre for Health and Wellbeing across the Lifecourse, Dept of Health Sciences, Brunel University, London, UK.
³³ School of Pharmacy, Charles Perkins Centre, The University of Sydney, Sydney, New South Wales, Australia.
³⁴ Division of Medical Ethics, New York University Grossman School of Medicine, New York, New York, USA.
³⁵ Faculty of Medicine, University of New South Wales, Sydney, New South Wales, Australia.
³⁶ Cochrane Central Production Service, Cochrane, London, UK.
³⁷ York Trials Unit, Dept of Health Sciences, University of York, York, UK.
³⁸ Cochrane Gynaecology and Fertility Satellite and Cochrane Sexually Transmitted Infection Group, Cochrane, Duivendrecht, The Netherlands.
³⁹ Reproduction and Development Research Institute, Amsterdam University Medical Center, Amsterdam, The Netherlands.
⁴⁰ Center for Bioethics and Humanities, University of Colorado-Anschutz Medical Campus, Denver, Colorado, USA.

PMID: 38471680
PMCID: PMC10936473
DOI: 10.1136/bmjopen-2024-084164

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Jack Wilkinson et al. BMJ Open. 2024.

. 2024 Mar 11;14(3):e084164.

doi: 10.1136/bmjopen-2024-084164.

Authors

Affiliations

¹ Centre for Biostatistics, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK jack.wilkinson@manchester.ac.uk.
² Centre for Biostatistics, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.
³ Manchester Vascular Centre, Manchester University NHS Foundation Trust, Manchester, UK.
⁴ Division of Cardiovascular Sciences, School of Medical Sciences, Manchester Academic Health Science Centre, The University of Manchester Faculty of Biology Medicine and Health, Manchester, UK.
⁵ Evidence Production and Methods Directorate, Cochrane, London, UK.
⁶ Department of Women's and Children's Health, University of Liverpool, Liverpool, UK.
⁷ Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
⁸ Medical Journal of Australia, Sydney, New South Wales, Australia.
⁹ Department of Psychology, Linnaeus University, Växjö, Sweden.
¹⁰ Anaesthesia and Critical Care, Torbay Hospital, Torquay, UK.
¹¹ Northern Ireland Methodology Hub, Queen's University Belfast, Belfast, UK.
¹² Department of Epidemiology and Public Health, Royal College of Surgeons in Ireland, Dublin, Ireland.
¹³ Division of Nursing, Midwifery and Social Work, School of Health Sciences, The University of Manchester, Manchester, UK.
¹⁴ NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK.
¹⁵ Department of Medicine, University of Auckland, Auckland, New Zealand.
¹⁶ School of Population and Global Health, The University of Melbourne, Melbourne, Victoria, Australia.
¹⁷ Community Health & Epidemiology, Dalhousie University, Halifax, Nova Scotia, Canada.
¹⁸ Department of Community Health & Epidemiology, Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada.
¹⁹ SafeBeat Rx Inc, St Louis, Missouri, USA.
²⁰ Evidence Integration, NHMRC Clinical Trials Centre, University of Sydney, Sydney, New South Wales, Australia.
²¹ Independent Lay Member, Cheshire, UK.
²² Department of Obstetrics and Gynaecology, Royal Women's Hospital, University of Melbourne, Melbourne, Victoria, Australia.
²³ Department of Ophthalmology, University of Colorado, Denver, Colorado, USA.
²⁴ Department of Obstetrics and Gynecology, Monash University, Clayton, Victoria, Australia.
²⁵ BMJ Publishing, London, UK.
²⁶ Department of Neurology, Brigham and Women's Hospital, Boston, Massachusetts, USA.
²⁷ Centre of Evidence-Based Medicine Odense and Cochrane Denmark, Department of Clinical Research, University of Southern Denmark, Odense, Denmark.
²⁸ Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital-Bispebjerg and Frederiksberg, Kobenhavn, Denmark.
²⁹ School of Health and Society, University of Wollongong, Wollongong, New South Wales, Australia.
³⁰ Department of Obstetrics and Gynaecology, Monash University, Clayton, Victoria, Australia.
³¹ Department of Clinical Science, Brunel University, Uxbridge, UK.
³² Centre for Health and Wellbeing across the Lifecourse, Dept of Health Sciences, Brunel University, London, UK.
³³ School of Pharmacy, Charles Perkins Centre, The University of Sydney, Sydney, New South Wales, Australia.
³⁴ Division of Medical Ethics, New York University Grossman School of Medicine, New York, New York, USA.
³⁵ Faculty of Medicine, University of New South Wales, Sydney, New South Wales, Australia.
³⁶ Cochrane Central Production Service, Cochrane, London, UK.
³⁷ York Trials Unit, Dept of Health Sciences, University of York, York, UK.
³⁸ Cochrane Gynaecology and Fertility Satellite and Cochrane Sexually Transmitted Infection Group, Cochrane, Duivendrecht, The Netherlands.
³⁹ Reproduction and Development Research Institute, Amsterdam University Medical Center, Amsterdam, The Netherlands.
⁴⁰ Center for Bioethics and Humanities, University of Colorado-Anschutz Medical Campus, Denver, Colorado, USA.

PMID: 38471680
PMCID: PMC10936473
DOI: 10.1136/bmjopen-2024-084164

Abstract

Introduction: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare-related interventions.

Methods and analysis: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: (1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, (2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, (3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in, (4) a consensus meeting to select checks to be included in a draft tool and to determine its format and (5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies and will help patients by protecting them from the influence of false data on their healthcare.

Ethics and dissemination: The University of Manchester ethics decision tool was used, and this returned the result that ethical approval was not required for this project (30 September 2022), which incorporates secondary research and surveys of professionals about subjects relating to their expertise. Informed consent will be obtained from all survey participants. All results will be published as open-access articles. The final tool will be made freely available.

Keywords: Randomized Controlled Trial; STATISTICS & RESEARCH METHODS; Systematic Review.

PubMed Disclaimer

Conflict of interest statement

Competing interests: JW, CH, GAA, LB and JJK declare funding from NIHR (NIHR203568) in relation to this work. JW additionally declares stats or methodological editor roles for BJOG, Fertility and Sterility, Reproduction and Fertility, Journal of Hypertension and for Cochrane Gynaecology and Fertility. CH declares a statistical editor role for Cochrane Colorectal. LB additionally declares a role as academic meta-research editor for PLoS Biology, and that The University of Colorado receives remuneration for service as senior research integrity editor, Cochrane. JJK additionally declares a statistical editor role for The BMJ. WL, LS and RW declare funding from Australian National Health and Medical Research Council Investigator Grants (GNT2016729, GNT2009432, GNT2009767). LS additionally declares roles as editorial board member for Cochrane Evidence Synthesis Methods and statistical editor for Cochrane Neonatal Group. RW additionally declares roles as deputy editor for Human Reproduction and editorial board member for BJOG and Cochrane Gynaecology and Fertility. WL additionally declares roles of associate editor for Human Reproduction Open and methodological editor for Fertility and Sterility and Fertility and Sterility Reviews. EF, SG and TLasserson declare employment by Cochrane. EF additionally declares a role as editorial board member for Cochrane Synthesis and Methods. TLasserson additionally declares authorship of a chapter in the Cochrane Handbook for Systematic Reviews of Interventions and that he is a developer of standards for Cochrane Intervention Reviews (MECIR). SG additionally declares an academic editor role for PLOS Global Public Health. ES declares that she was a member of the Cochrane scientific misconduct policy advisory group, and roles as sign-off editor and proposal editor of The Cochrane Library. ZA declares he is a member of the Cochrane Library Editorial Board, was co-ordinating editor for Cochrane Pregnancy and Childbirth Group 2002–2023 and is PI on a grant from Children Investment Foundation Fund to University of Liverpool to investigate research integrity of clinical trials related to nutritional supplements in pregnancy. TLi is supported by grant UG1 EY020522 from the National Eye Institute, National Institutes of Health. TLi also declares roles as editor-in-chief for Trials, statistical editor for Annals of Internal Medicine, review editor for JAMA Ophthalmology and editorial board member for Journal of Clinical Epidemiology. MC declares that he is Co-ordinating editor for the Cochrane Methodology Review Group, Editor in Chief of Journal of Evidence-Based Medicine, and Co-ordinating Editor of the James Lind Library. AA declares that The Health Services Research Unit, University of Aberdeen, is funded by the Health and Social Care Directorates of the Scottish Government. All other authors declare no competing interests. DT declares an RCT topic editor role for Research Methods in Medicine and Health Sciences. MvW declares roles as co-ordinating editor for Cochrane Gynaecology and Fertility and Sexually Transmitted Infections, methodology editor for Human Reproduction Update and editorial editor for Fertility and Sterility. NEO declares roles as co-ordinating editor for Cochrane Pain, Palliative and Supportive Care (PaPaS) Group, 2020–2023, as member of Cochrane Central Editorial Board and as editorial board member for Journal of Pain. JC declares an editor role for Anaesthesia. AL declares a role as editorial board member for BMC Medical Ethics. GAA declares a role as statistical editor for the European Journal of Vascular and Endovascular Surgery. NJLB declares roles as editorial board member for International Review of Social Psychology/Revue Internationale de Psychologie Sociale, statistical advisory board member for Mental Health Science and advisory board member for Meta-Psychology. BWM declares roles as editor for Cochrane Gynaecology and Fertility and Sexually Transmitted Infections and for Fertility and Sterility. SL declares roles as associate editor for Human Reproduction, methodological editor for Fertility and Sterility and editor for Cochrane Gynaecology and Fertility. EL is head of research and clinical editor for The BMJ.

Figures

**Figure 1**
INveStigating ProblEmatic Clinical Trials in Systematic Reviews development process.

See this image and copyright information in PMC

Update of

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions.
Wilkinson J, Heal C, Antoniou GA, Flemyng E, Alfirevic Z, Avenell A, Barbour V, Brown NJL, Carlisle J, Clarke M, Dicker P, Dumville J, Grey A, Grohmann S, Gurrin LC, Hayden JA, Heathers J, Hunter KE, Lasserson T, Lam E, Lensen S, Li T, Li W, Loder E, Lundh A, Meyerowitz-Katz G, Mol BW, O' Connell NE, Parker L, Redman BK, Seidler AL, Sheldrick KA, Sydenham E, Torgerson DJ, van Wely M, Wang R, Bero L, Kirkham JJ. Wilkinson J, et al. medRxiv [Preprint]. 2023 Nov 13:2023.09.21.23295626. doi: 10.1101/2023.09.21.23295626. medRxiv. 2023. Update in: BMJ Open. 2024 Mar 11;14(3):e084164. doi: 10.1136/bmjopen-2024-084164. PMID: 37873409 Free PMC article. Updated. Preprint.

Cited by

Investigation of trial registration as part of a research integrity assessment of randomised controlled trials in COVID-19 evidence syntheses: a meta-epidemiological study.
Pscheidl T, Weber F, Sydenham E, Meybohm P, Weibel S. Pscheidl T, et al. BMJ Open. 2025 May 11;15(5):e092243. doi: 10.1136/bmjopen-2024-092243. BMJ Open. 2025. PMID: 40350190 Free PMC article.
Investigation of ethics approval as part of a research integrity assessment of randomised controlled trials in COVID-19 evidence syntheses: a meta-epidemiological study.
Weber F, Pscheidl T, Sydenham E, Meybohm P, Weibel S. Weber F, et al. BMJ Open. 2025 Mar 24;15(3):e092244. doi: 10.1136/bmjopen-2024-092244. BMJ Open. 2025. PMID: 40132830 Free PMC article.
Trustworthiness assessment of published clinical trials: Literature review of domains and questions.
Alfirevic Z, Weeks J. Alfirevic Z, et al. Cochrane Evid Synth Methods. 2024 Aug 20;2(8):e12099. doi: 10.1002/cesm.12099. eCollection 2024 Aug. Cochrane Evid Synth Methods. 2024. PMID: 40475114 Free PMC article. Review.
Preoperative statins in cardiac surgery: a tale of small study bias or 'the truth, and nothing but the truth'.
Providencia R. Providencia R. Cochrane Database Syst Rev. 2024 Jul 22;7(7):ED000167. doi: 10.1002/14651858.ED000167. Cochrane Database Syst Rev. 2024. PMID: 39037830 Free PMC article. No abstract available.
Data-sharing and trustworthiness of trials evaluating cervical ripening in induction of labour: a meta-epidemiological study of randomised controlled trials.
Patabendige M, Rolnik DL, Li W, Mol BW, Li W. Patabendige M, et al. EClinicalMedicine. 2025 Jul 8;85:103346. doi: 10.1016/j.eclinm.2025.103346. eCollection 2025 Jul. EClinicalMedicine. 2025. PMID: 40686691 Free PMC article.

See all "Cited by" articles

References

1. Bryant A, Lawrie TA, Dowswell T, et al. . Ivermectin for prevention and treatment of COVID-19 infection: a systematic review, meta-analysis, and trial sequential analysis to inform clinical guidelines. Am J Ther 2021;28:e434–60. 10.1097/MJT.0000000000001402 - DOI - PMC - PubMed
1. Hill A, Garratt A, Levi J, et al. . Meta-analysis of randomized trials of ivermectin to treat SARS-Cov-2 infection. Open Forum Infect Dis 2021;8:ofab358. 10.1093/ofid/ofab358 - DOI - PMC - PubMed
1. Lawrence JM, Meyerowitz-Katz G, Heathers JAJ, et al. . The lesson of Ivermectin: meta-analyses based on summary data alone are inherently unreliable. Nat Med 2021;27:1853–4. 10.1038/s41591-021-01535-y - DOI - PubMed
1. Brown NJL. 2023. Available: http://steamtraen.blogspot.com/2021/07/Some-problems-with-the-data-from-...
1. Hill A, Mirchandani M, Pilkington V. Ivermectin for COVID-19: addressing potential bias and medical fraud. Open Forum Infect Dis 2022;9:ofab645. 10.1093/ofid/ofab645 - DOI - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

UG1 EY020522/EY/NEI NIH HHS/United States

LinkOut - more resources

Full Text Sources
Research Materials
- NCI CPTC Antibody Characterization Program

[1] Bryant A, Lawrie TA, Dowswell T, et al. . Ivermectin for prevention and treatment of COVID-19 infection: a systematic review, meta-analysis, and trial sequential analysis to inform clinical guidelines. Am J Ther 2021;28:e434–60. 10.1097/MJT.0000000000001402 - DOI - PMC - PubMed

[2] Bryant A, Lawrie TA, Dowswell T, et al. . Ivermectin for prevention and treatment of COVID-19 infection: a systematic review, meta-analysis, and trial sequential analysis to inform clinical guidelines. Am J Ther 2021;28:e434–60. 10.1097/MJT.0000000000001402 - DOI - PMC - PubMed

[3] Hill A, Garratt A, Levi J, et al. . Meta-analysis of randomized trials of ivermectin to treat SARS-Cov-2 infection. Open Forum Infect Dis 2021;8:ofab358. 10.1093/ofid/ofab358 - DOI - PMC - PubMed

[4] Hill A, Garratt A, Levi J, et al. . Meta-analysis of randomized trials of ivermectin to treat SARS-Cov-2 infection. Open Forum Infect Dis 2021;8:ofab358. 10.1093/ofid/ofab358 - DOI - PMC - PubMed

[5] Lawrence JM, Meyerowitz-Katz G, Heathers JAJ, et al. . The lesson of Ivermectin: meta-analyses based on summary data alone are inherently unreliable. Nat Med 2021;27:1853–4. 10.1038/s41591-021-01535-y - DOI - PubMed

[6] Lawrence JM, Meyerowitz-Katz G, Heathers JAJ, et al. . The lesson of Ivermectin: meta-analyses based on summary data alone are inherently unreliable. Nat Med 2021;27:1853–4. 10.1038/s41591-021-01535-y - DOI - PubMed

[7] Brown NJL. 2023. Available: http://steamtraen.blogspot.com/2021/07/Some-problems-with-the-data-from-...

[8] Brown NJL. 2023. Available: http://steamtraen.blogspot.com/2021/07/Some-problems-with-the-data-from-...

[9] Hill A, Mirchandani M, Pilkington V. Ivermectin for COVID-19: addressing potential bias and medical fraud. Open Forum Infect Dis 2022;9:ofab645. 10.1093/ofid/ofab645 - DOI - PMC - PubMed

[10] Hill A, Mirchandani M, Pilkington V. Ivermectin for COVID-19: addressing potential bias and medical fraud. Open Forum Infect Dis 2022;9:ofab645. 10.1093/ofid/ofab645 - DOI - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Affiliations

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Update of

Similar articles

Cited by

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Research Materials