Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project

doi:10.1016/j.jclinepi.2025.111824

. 2025 May 9:184:111824.

doi: 10.1016/j.jclinepi.2025.111824. Online ahead of print.

Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project

Jack Wilkinson¹, Calvin Heal², Georgios A Antoniou³, Ella Flemyng⁴, Love Ahnström⁵, Alessandra Alteri⁶, Alison Avenell⁷, Timothy Hugh Barker⁸, David N Borg⁹, Nicholas J L Brown¹⁰, Rob Buhmann¹¹, Jose A Calvache¹², Rickard Carlsson¹⁰, Lesley-Anne Carter¹, Aidan G Cashin¹³, Sarah Cotterill¹, Kenneth Färnqvist¹⁴, Michael C Ferraro¹³, Steph Grohmann⁴, Lyle C Gurrin¹⁵, Jill A Hayden¹⁶, Kylie E Hunter¹⁷, Natalie Hyltse¹⁰, Lukas Jung¹⁸, Ashma Krishan¹, Silvy Laporte¹⁹, Toby J Lasserson⁴, David R T Laursen²⁰, Sarah Lensen²¹, Wentao Li²², Tianjing Li²³, Jianping Liu²⁴, Clara Locher²⁵, Zewen Lu¹, Andreas Lundh²⁶, Antonia Marsden¹, Gideon Meyerowitz-Katz²⁷, Ben W Mol²¹, Zachary Munn²², Florian Naudet²⁵, David Nunan²⁸, Neil E O'Connell²⁹, Natasha Olsson⁵, Lisa Parker³⁰, Eleftheria Patetsini³¹, Barbara Redman³², Sarah Rhodes¹, Rachel Richardson⁴, Martin Ringsten³³, Ewelina Rogozińska³⁴, Anna Lene Seidler³⁵, Kyle Sheldrick³⁶, Katie Stocking¹, Emma Sydenham³⁷, Hugh Thomas³⁸, Sofia Tsokani³⁹, Constant Vinatier²⁵, Colby J Vorland⁴⁰, Rui Wang¹⁷, Bassel H Al Wattar⁴¹, Florencia Weber⁴², Stephanie Weibel⁴², Madelon van Wely⁴³, Chang Xu⁴⁴, Lisa Bero⁴⁵, Jamie J Kirkham¹

Affiliations

¹ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
² Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK. Electronic address: calvin.heal@manchester.ac.uk.
³ Manchester Vascular Centre, Manchester University NHS Foundation Trust, Manchester, UK; Division of Cardiovascular Sciences, School of Medical Sciences, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.
⁴ Evidence Production and Methods Directorate, Cochrane Central Executive, London, UK.
⁵ Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden.
⁶ Obstetrics and Gynaecology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy.
⁷ Aberdeen Centre for Evaluation, University of Aberdeen, Aberdeen, UK.
⁸ Health Evidence Synthesis, Recommendations and Impact, School of Public Health, The University of Adelaide, Adelaide, South Australia, Australia.
⁹ Australian Sports Commission, Australian Institute of Sport, Bruce, Australian Capital Territory, Australia; Queensland University of Technology, School of Exercise Nutrition Sciences, Brisbane, Queensland, Australia.
¹⁰ Department of Psychology, Linnaeus University, Växjö, Sweden.
¹¹ University of the Sunshine Coast, School of Health, Queensland, Australia.
¹² Department of Anesthesiology, Erasmus MC, Rotterdam, The Netherlands; Department of Anesthesiology, Universidad del Cauca, Popayán, Colombia.
¹³ Centre for Pain IMPACT, Neuroscience Research Australia, Randwick, Australia; School of Health Sciences, Faculty of Medicine and Health, University of New South Wales Sydney, Sydney, Australia.
¹⁴ Department of Molecular medicine and surgery, Karolinska Institute, Stockholm, Sweden.
¹⁵ School of Population and Global Health, The University of Melbourne, Melbourne, Victoria, Australia.
¹⁶ Department of Community Health & Epidemiology, Dalhousie University, Halifax, Canada.
¹⁷ NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia.
¹⁸ Independent Researcher, Heidelberg, Germany.
¹⁹ Clinical Pharmacology Unit, INSERM U1059 Sainbiose, University Hospital of Saint-Etienne, Saint-Etienne, France.
²⁰ Cochrane Denmark & Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark.
²¹ Department of Obstetrics, Gynaecology and Newborn Health, Royal Women's Hospital, University of Melbourne, Melbourne, Victoria, Australia.
²² National Perinatal Epidemiology and Statistics Unit, Centre for Big Data Research in Health and School of Women's and Children's Health, The University of New South Wales, Sydney, Australia.
²³ Department of Ophthalmology and Department of Epidemiology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
²⁴ Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China.
²⁵ Univ Rennes, CHU Rennes, Inserm, Irset [(Institut de recherche en santé, environnement et travail)] - UMR_S 1085, F-35000 Rennes, France.
²⁶ Cochrane Denmark & Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark; Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital, Copenhagen, Bispebjerg and Frederiksberg, Denmark.
²⁷ School of Nursing, University of Wollongong, Wollongong, New South Wales, Australia.
²⁸ Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK.
²⁹ Department of Health Sciences, Centre for Health and Wellbeing across the Lifecourse, Brunel University London, London, UK.
³⁰ Faculty of Medicine & Health, Charles Perkins Centre, University of Sydney, Sydney, New South Wales, Australia.
³¹ York Trials Unit, Department of Health Sciences, University of York, York, UK.
³² Division of Medical Ethics, New York University Grossman School of Medicine, New York, NY, USA.
³³ Cochrane Sweden, Skåne University Hospital, Lund University, Lund, Sweden.
³⁴ Meta-Analysis Group, MRC Clinical Trials Unit at UCL, Institute of Clinical Trials and Methodology, London, UK.
³⁵ NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia; Department of Child and Adolescent Psychiatry, University Medical Centre Rostock, Rostock, Germany; German Center for Child and Adolescent Health (DZKJ), partner site Greifswald/Rostock, Rostock, Germany.
³⁶ Faculty of Medicine, University of New South Wales, Sydney, New South Wales, Australia.
³⁷ Cochrane Central Editorial Service, Cochrane Database of Systematic Reviews, London, UK.
³⁸ The Lancet Gastroenterology & Hepatology, London, UK.
³⁹ Laboratory of Hygiene, Social & Preventive Medicine and Medical Statistics, School of Medicine, Aristotle University of Thessaloniki, Thessaloniki, Greece.
⁴⁰ Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA.
⁴¹ Clinical Trials Unit, Anglia Ruskin University, Chelmsford, UK.
⁴² Department of Anaesthesiology, Intensive Care, Emergency and Pain Medicine, University Hospital Würzburg, Würzburg, Germany.
⁴³ Centre for Reproductive Medicine, Department of Obstetrics and Gynaecology, Amsterdam University Medical Center, Amsterdam, The Netherlands.
⁴⁴ The Third Department of Hepatic Surgery, Eastern Hepatobiliary Surgery Hospital, Third Affiliated Hospital, Second Military Medical University, Naval Medical University, Shanghai, China; Proof of Concept Center, Eastern Hepatobiliary Surgery Hospital, Third Affiliated Hospital, Second Military Medical University, Naval Medical University, Shanghai, China.
⁴⁵ Center for Bioethics, and Humanities, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.

PMID: 40349737
DOI: 10.1016/j.jclinepi.2025.111824

Free article

Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project

Jack Wilkinson et al. J Clin Epidemiol. 2025.

Free article

. 2025 May 9:184:111824.

doi: 10.1016/j.jclinepi.2025.111824. Online ahead of print.

Authors

Affiliations

¹ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
² Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK. Electronic address: calvin.heal@manchester.ac.uk.
³ Manchester Vascular Centre, Manchester University NHS Foundation Trust, Manchester, UK; Division of Cardiovascular Sciences, School of Medical Sciences, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.
⁴ Evidence Production and Methods Directorate, Cochrane Central Executive, London, UK.
⁵ Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden.
⁶ Obstetrics and Gynaecology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy.
⁷ Aberdeen Centre for Evaluation, University of Aberdeen, Aberdeen, UK.
⁸ Health Evidence Synthesis, Recommendations and Impact, School of Public Health, The University of Adelaide, Adelaide, South Australia, Australia.
⁹ Australian Sports Commission, Australian Institute of Sport, Bruce, Australian Capital Territory, Australia; Queensland University of Technology, School of Exercise Nutrition Sciences, Brisbane, Queensland, Australia.
¹⁰ Department of Psychology, Linnaeus University, Växjö, Sweden.
¹¹ University of the Sunshine Coast, School of Health, Queensland, Australia.
¹² Department of Anesthesiology, Erasmus MC, Rotterdam, The Netherlands; Department of Anesthesiology, Universidad del Cauca, Popayán, Colombia.
¹³ Centre for Pain IMPACT, Neuroscience Research Australia, Randwick, Australia; School of Health Sciences, Faculty of Medicine and Health, University of New South Wales Sydney, Sydney, Australia.
¹⁴ Department of Molecular medicine and surgery, Karolinska Institute, Stockholm, Sweden.
¹⁵ School of Population and Global Health, The University of Melbourne, Melbourne, Victoria, Australia.
¹⁶ Department of Community Health & Epidemiology, Dalhousie University, Halifax, Canada.
¹⁷ NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia.
¹⁸ Independent Researcher, Heidelberg, Germany.
¹⁹ Clinical Pharmacology Unit, INSERM U1059 Sainbiose, University Hospital of Saint-Etienne, Saint-Etienne, France.
²⁰ Cochrane Denmark & Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark.
²¹ Department of Obstetrics, Gynaecology and Newborn Health, Royal Women's Hospital, University of Melbourne, Melbourne, Victoria, Australia.
²² National Perinatal Epidemiology and Statistics Unit, Centre for Big Data Research in Health and School of Women's and Children's Health, The University of New South Wales, Sydney, Australia.
²³ Department of Ophthalmology and Department of Epidemiology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
²⁴ Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China.
²⁵ Univ Rennes, CHU Rennes, Inserm, Irset [(Institut de recherche en santé, environnement et travail)] - UMR_S 1085, F-35000 Rennes, France.
²⁶ Cochrane Denmark & Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark; Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital, Copenhagen, Bispebjerg and Frederiksberg, Denmark.
²⁷ School of Nursing, University of Wollongong, Wollongong, New South Wales, Australia.
²⁸ Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK.
²⁹ Department of Health Sciences, Centre for Health and Wellbeing across the Lifecourse, Brunel University London, London, UK.
³⁰ Faculty of Medicine & Health, Charles Perkins Centre, University of Sydney, Sydney, New South Wales, Australia.
³¹ York Trials Unit, Department of Health Sciences, University of York, York, UK.
³² Division of Medical Ethics, New York University Grossman School of Medicine, New York, NY, USA.
³³ Cochrane Sweden, Skåne University Hospital, Lund University, Lund, Sweden.
³⁴ Meta-Analysis Group, MRC Clinical Trials Unit at UCL, Institute of Clinical Trials and Methodology, London, UK.
³⁵ NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia; Department of Child and Adolescent Psychiatry, University Medical Centre Rostock, Rostock, Germany; German Center for Child and Adolescent Health (DZKJ), partner site Greifswald/Rostock, Rostock, Germany.
³⁶ Faculty of Medicine, University of New South Wales, Sydney, New South Wales, Australia.
³⁷ Cochrane Central Editorial Service, Cochrane Database of Systematic Reviews, London, UK.
³⁸ The Lancet Gastroenterology & Hepatology, London, UK.
³⁹ Laboratory of Hygiene, Social & Preventive Medicine and Medical Statistics, School of Medicine, Aristotle University of Thessaloniki, Thessaloniki, Greece.
⁴⁰ Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA.
⁴¹ Clinical Trials Unit, Anglia Ruskin University, Chelmsford, UK.
⁴² Department of Anaesthesiology, Intensive Care, Emergency and Pain Medicine, University Hospital Würzburg, Würzburg, Germany.
⁴³ Centre for Reproductive Medicine, Department of Obstetrics and Gynaecology, Amsterdam University Medical Center, Amsterdam, The Netherlands.
⁴⁴ The Third Department of Hepatic Surgery, Eastern Hepatobiliary Surgery Hospital, Third Affiliated Hospital, Second Military Medical University, Naval Medical University, Shanghai, China; Proof of Concept Center, Eastern Hepatobiliary Surgery Hospital, Third Affiliated Hospital, Second Military Medical University, Naval Medical University, Shanghai, China.
⁴⁵ Center for Bioethics, and Humanities, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.

PMID: 40349737
DOI: 10.1016/j.jclinepi.2025.111824

Abstract

Background and objectives: The aim of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project is to develop a tool to identify problematic RCTs in systematic reviews. In stage 1 of the project, a list of potential trustworthiness checks was created. The checks on this list must be evaluated to determine which should be included in the INSPECT-SR tool.

Methods: We attempted to apply 72 trustworthiness checks to randomized controlled trials (RCTs) in 50 Cochrane reviews. For each, we recorded whether the check was passed, failed, or possibly failed or whether it was not feasible to complete the check. Following application of the checks, we recorded whether we had concerns about the authenticity of each RCT. We repeated each meta-analysis after removing RCTs flagged by each check and again after removing RCTs where we had concerns about authenticity to estimate the impact of trustworthiness assessment. Trustworthiness assessments were compared to Risk of Bias and Grading of Recommendations Assessment, Development and Evaluation (GRADE) assessments in the reviews.

Results: Ninety-five RCTs were assessed. Following application of the checks, assessors had some or serious concerns about the authenticity of 25% and 6% of the RCTs, respectively. Removing RCTs with either some or serious concerns resulted in 22% of meta-analyses having no remaining RCTs. However, many checks proved difficult to understand or implement, which may have led to unwarranted skepticism in some instances. Furthermore, we restricted assessment to meta-analyses with no more than five RCTs (54% contained only 1 RCT), which will distort the impact on results. No relationship was identified between trustworthiness assessment and Risk of Bias or GRADE.

Conclusion: This study supports the case for routine trustworthiness assessment in systematic reviews, as problematic studies do not appear to be flagged by Risk of Bias assessment. The study produced evidence on the feasibility and impact of trustworthiness checks. These results will be used, in conjunction with those from a subsequent Delphi process, to determine which checks should be included in the INSPECT-SR tool.

Plain language summary: Systematic reviews collate evidence from randomized controlled trials (RCTs) to find out whether health interventions are safe and effective. However, it is now recognized that the findings of some RCTs are not genuine, and some of these studies appear to have been fabricated. Various checks for these "problematic" RCTs have been proposed, but it is necessary to evaluate these checks to find out which are useful and which are feasible. We applied a comprehensive list of "trustworthiness checks" to 95 RCTs in 50 systematic reviews to learn more about them and to see how often performing the checks would lead us to classify RCTs as being potentially inauthentic. We found that applying the checks led to concerns about the authenticity of around 1 in three RCTs. However, we found that many of the checks were difficult to perform and could have been misinterpreted. This might have led us to be overly skeptical in some cases. The findings from this study will be used, alongside other evidence, to decide which of these checks should be performed routinely to try to identify problematic RCTs, to stop them from being mistaken for genuine studies and potentially being used to inform health care decisions.

Keywords: Critical appraisal; Evidence synthesis; Fabrication; Forensic analysis; Fraud; Misconduct; Randomized controlled trials; Research Integrity; Systematic reviews; Trustworthiness.

PubMed Disclaimer

Conflict of interest statement

Declaration of competing interest J.W., C.H., G.A.A., L.B., and J.J.K. declare funding from NIHR (NIHR203568) in relation to the current project. J.W. additionally declares Stats or Methodological Editor roles for BJOG, Fertility and Sterility, Reproduction and Fertility, Journal of Hypertension, and Cochrane Gynaecology and Fertility. C.H. declares a Statistical Editor role for Cochrane Colorectal. G.A.A. additionally declares a Statistical Reviewer role for the European Journal of Vascular and Endovascular Surgery. L.B. additionally declares a role as Academic Meta-Research Editor for PLoS Biology and that the University of Colorado receives remuneration for service as Senior Research Integrity Editor, Cochrane. J.J.K. additionally declares a Statistical Editor role for the BMJ. E.F. is employed by the Cochrane Collaboration and on the Editorial Board of Cochrane Evidence Synthesis and Methods. E.S. is a Senior (Sign-off) Editor for Cochrane and contributed to Cochrane's Policy for managing potentially problematic studies. S.L. is an editor for Cochrane Gynaecology and Fertility, Human Reproduction, and Fertility and Sterility. T.J.L. is the Deputy Editor-in-Chief of the Cochrane Library and is an employee of the Cochrane Collaboration. D.N.B. is an associate editor for Research Quarterly for Exercise and Sport and a section editor for Communications in Kinesiology. N.E.O. is a member of the Cochrane Editorial Board and holds an ERA-NET Neuron Co-Fund grant for a separate project. R.R. declares acting as an author and editor on Cochrane reviews. K.S. is an editor for Cochrane Gynecology and Fertility and Fertility and Sterility. M.v.W. declares to be coordinating editor for Cochrane Gynaecology and Fertility and Cochrane Sexually Transmitted Infections, methodological editor for Human Reproduction Update and Editorial Editor for Fertility & Sterility. H.T. is Deputy Editor of the Lancet Gastroenterology & Hepatology and is an employee of Elsevier. S.L. received funding from the French National Research Agency (ANR-23-CE36-0006-01). A.K. is an editorial board member for BJGP Open. T.Li. serves as the Principal Investigator on a grant from the National Eye Institute, National Institutes of Health that funds the work of Cochrane Eyes and Vision US Project. She also acts as a sign-off editor for the Cochrane Library. Z.M. is supported by an NHMRC Investigator Grant 1195676. Z.M. is an associate Editor for BMC Medical Research Methodology and is on the Editorial Board for Clinical and Public Health Guidelines. R.C. is Editor-in-Chief at Meta-Psychology. C.L. is a work-package leader for the doctoral network MSCA-DN SHARE-CTD (HORIZON-MSCA-2022-DN-01 101120360), funded by the EU. C.V. received funding as part of the OSIRIS project (Open Science to Increase Reproducibility in Science); the OSIRIS (Open Science to Increase Reproducibility in Science) project has received funding from the EU (grant agreement No. 101094725). F.N. received funding from the French National Research Agency (ANR-23-CE36-0006-01), the French Ministry of Health, and the French Ministry of Research. He is a work-package leader in the OSIRIS project (Open Science to Increase Reproducibility in Science). The OSIRIS project has received funding from the European Union's Horizon Europe research and innovation program under grant agreement No. 101094725. He is a work-package leader for the doctoral network MSCA-DN SHARE-CTD (HORIZON-MSCA-2022-DN-01 101120360), funded by the EU. D.N. declares having led/co-authored/co-authoring Cochrane reviews. He also declares having been part of the Cochrane Convenes initiative organized by Cochrane to consider the issue of misinformation, its impact on the health evidence ecosystem, and solutions to address it. LJ is the creator of the scrutiny package in R. W.L. is supported by an NHMRC Investigator grant (GNT2016729). R.W. is supported by an NHMRC Investigator Grant (2009767) and acts as a Deputy Editor for Human Reproduction, and an editorial board member for BJOG and Cochrane Gynecology and Fertility. E.F., S.G.T.La., and R.R. declare employment by Cochrane. T.La. additionally declares authorship of a chapter in the Cochrane Handbook for Systematic Reviews of Interventions and that he is a developer of standards for Cochrane intervention reviews (MECIR). A.L. is on the editorial board of BMC Medical Ethics.

LinkOut - more resources

Full Text Sources
- ClinicalKey
- Elsevier Science
Research Materials
- NCI CPTC Antibody Characterization Program

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project

Affiliations

Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project

Authors

Affiliations

Abstract

Conflict of interest statement

Similar articles

LinkOut - more resources

Full Text Sources

Research Materials

Abstract

Conflict of interest statement

Similar articles

Related information

LinkOut - more resources

Full Text Sources

Research Materials