A survey of experts to identify methods to detect problematic studies: stage 1 of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews project

Jack Wilkinson¹, Calvin Heal², George A Antoniou³, Ella Flemyng⁴, Alison Avenell⁵, Virginia Barbour⁶, Esmee M Bordewijk⁷, Nicholas J L Brown⁸, Mike Clarke⁹, Jo Dumville¹⁰, Steph Grohmann⁴, Lyle C Gurrin¹¹, Jill A Hayden¹², Kylie E Hunter¹³, Emily Lam¹⁴, Toby Lasserson⁴, Tianjing Li¹⁵, Sarah Lensen¹⁶, Jianping Liu¹⁷, Andreas Lundh¹⁸, Gideon Meyerowitz-Katz¹⁹, Ben W Mol²⁰, Neil E O'Connell²¹, Lisa Parker²², Barbara Redman²³, Anna Lene Seidler¹³, Kyle Sheldrick²⁴, Emma Sydenham²⁵, Darren L Dahly²⁶, Madelon van Wely⁷, Lisa Bero²⁷, Jamie J Kirkham²

Affiliations

¹ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK. Electronic address: jack.wilkinson@manchester.ac.uk.
² Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
³ Manchester Vascular Centre, Manchester University NHS Foundation Trust, Manchester, UK; Division of Cardiovascular Sciences, School of Medical Sciences, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.
⁴ Evidence Production and Methods Directorate, Cochrane Central Executive, London, UK.
⁵ Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
⁶ Medical Journal of Australia, Sydney, Australia.
⁷ Department of Obstetrics and Gynaecology, Centre for Reproductive Medicine, Amsterdam University Medical Center, Amsterdam, Netherlands.
⁸ Department of Psychology, Linnaeus University, Växjö, Sweden.
⁹ Northern Ireland Methodology Hub, Queen's University Belfast, Belfast, UK.
¹⁰ Division of Nursing, Midwifery & Social Work, School of Health Sciences, The University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK.
¹¹ School of Population and Global Health, The University of Melbourne, Melbourne, Australia.
¹² Department of Community Health & Epidemiology, Dalhousie University, Halifax, Canada.
¹³ NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia.
¹⁴ Independent Lay Member, Unaffiliated, Cheshire, UK.
¹⁵ Department of Ophthalmology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
¹⁶ Department of Obstetrics, Gynaecology and Newborth Health, Royal Women's Hospital, University of Melbourne, Melbourne, Australia.
¹⁷ Director, Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China.
¹⁸ Cochrane Denmark & Centre for Evidence-Based Medicine Odense, Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital - Bispebjerg and Frederiksberg, Copenhagen, Denmark.
¹⁹ School of Health and Society, University of Wollongong, Wollongong, Australia.
²⁰ Department of Obstetrics and Gynaecology, Monash University, Melbourne, Australia.
²¹ Department of Health Sciences, Centre for Wellbeing Across the Lifecourse, Brunel University London, London, UK.
²² Charles Perkins Centre, Sydney Medical School, University of Sydney, Sydney, Australia.
²³ New York University, New York, NY, USA.
²⁴ Faculty of Medicine, University of New South Wales, Sydney, Australia.
²⁵ Cochrane Central Editorial Service, London, UK.
²⁶ HRB Clinical Research Facility, University College Cork, Cork, Ireland.
²⁷ University of Colorado Anschutz Medical Campus, Aurora, CO, USA.

PMID: 39222724
DOI: 10.1016/j.jclinepi.2024.111512

Free article

A survey of experts to identify methods to detect problematic studies: stage 1 of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews project

Jack Wilkinson et al. J Clin Epidemiol. 2024 Nov.

Free article

. 2024 Nov:175:111512.

doi: 10.1016/j.jclinepi.2024.111512. Epub 2024 Aug 31.

Authors

Affiliations

¹ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK. Electronic address: jack.wilkinson@manchester.ac.uk.
² Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
³ Manchester Vascular Centre, Manchester University NHS Foundation Trust, Manchester, UK; Division of Cardiovascular Sciences, School of Medical Sciences, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.
⁴ Evidence Production and Methods Directorate, Cochrane Central Executive, London, UK.
⁵ Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
⁶ Medical Journal of Australia, Sydney, Australia.
⁷ Department of Obstetrics and Gynaecology, Centre for Reproductive Medicine, Amsterdam University Medical Center, Amsterdam, Netherlands.
⁸ Department of Psychology, Linnaeus University, Växjö, Sweden.
⁹ Northern Ireland Methodology Hub, Queen's University Belfast, Belfast, UK.
¹⁰ Division of Nursing, Midwifery & Social Work, School of Health Sciences, The University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK.
¹¹ School of Population and Global Health, The University of Melbourne, Melbourne, Australia.
¹² Department of Community Health & Epidemiology, Dalhousie University, Halifax, Canada.
¹³ NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia.
¹⁴ Independent Lay Member, Unaffiliated, Cheshire, UK.
¹⁵ Department of Ophthalmology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
¹⁶ Department of Obstetrics, Gynaecology and Newborth Health, Royal Women's Hospital, University of Melbourne, Melbourne, Australia.
¹⁷ Director, Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China.
¹⁸ Cochrane Denmark & Centre for Evidence-Based Medicine Odense, Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital - Bispebjerg and Frederiksberg, Copenhagen, Denmark.
¹⁹ School of Health and Society, University of Wollongong, Wollongong, Australia.
²⁰ Department of Obstetrics and Gynaecology, Monash University, Melbourne, Australia.
²¹ Department of Health Sciences, Centre for Wellbeing Across the Lifecourse, Brunel University London, London, UK.
²² Charles Perkins Centre, Sydney Medical School, University of Sydney, Sydney, Australia.
²³ New York University, New York, NY, USA.
²⁴ Faculty of Medicine, University of New South Wales, Sydney, Australia.
²⁵ Cochrane Central Editorial Service, London, UK.
²⁶ HRB Clinical Research Facility, University College Cork, Cork, Ireland.
²⁷ University of Colorado Anschutz Medical Campus, Aurora, CO, USA.

PMID: 39222724
DOI: 10.1016/j.jclinepi.2024.111512

Abstract

Background and objective: Randomized controlled trials (RCTs) inform health-care decisions. Unfortunately, some published RCTs contain false data, and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesize all RCTs which have been conducted on a given topic. This means that any of these 'problematic studies' are likely to be included, but there are no agreed methods for identifying them. The INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project is developing a tool to identify problematic RCTs in systematic reviews of health care-related interventions. The tool will guide the user through a series of 'checks' to determine a study's authenticity. The first objective in the development process is to assemble a comprehensive list of checks to consider for inclusion.

Methods: We assembled an initial list of checks for assessing the authenticity of research studies, with no restriction to RCTs, and categorized these into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; Inspecting the individual participant data. We implemented this list as an online survey, and invited people with expertise and experience of assessing potentially problematic studies to participate through professional networks and online forums. Participants were invited to provide feedback on the checks on the list, and were asked to describe any additional checks they knew of, which were not featured in the list.

Results: Extensive feedback on an initial list of 102 checks was provided by 71 participants based in 16 countries across five continents. Fourteen new checks were proposed across the five domains, and suggestions were made to reword checks on the initial list. An updated list of checks was constructed, comprising 116 checks. Many participants expressed a lack of familiarity with statistical checks, and emphasized the importance of feasibility of the tool.

Conclusion: A comprehensive list of trustworthiness checks has been produced. The checks will be evaluated to determine which should be included in the INSPECT-SR tool.

Keywords: Critical appraisal; Evidence synthesis; Fabrication; Forensic analysis; Fraud; Misconduct; Randomised controlled trials; Research integrity; Systematic reviews; Trustworthiness.

PubMed Disclaimer

Conflict of interest statement

Declaration of competing interest J.W., CH, GAA., L.B., and J.J.K. declare funding from NIHR (NIHR203568) in relation to the current project. J.W. additionally declares Stats or Methodological Editor roles for BJOG, Fertility and Sterility, Reproduction and Fertility, Journal of Hypertension, and for Cochrane Gynecology and Fertility. C.H. declares a Statistical Editor role for Cochrane Colorectal. L.B. additionally declares a role as Academic Meta-Research Editor for PLoS Biology, and that The University of Colorado receives remuneration for service as Senior Research Integrity Editor, Cochrane. J.J.K. additionally declares a Statistical Editor role for The BMJ. A.A. declares that The Health Services Research Unit, University of Aberdeen, is funded by the Health and Social Care Directorates of the Scottish Government. V.B. is EiC of the Medical Journal of Australia and on the Editorial Board of Research Integrity and Peer Review. N.J.L.B. declares roles as Editorial Board member for International Review of Social Pyschology/Revue Internationale de Psychologie Sociale, Statistical Advisory Board member for Mental Health Science, and Advisory Board member for Meta-Psychology. M.C. declares that he is Co-ordinating Editor for the Cochrane Methodology Review Group, Editor in Chief, Journal of Evidence-Based Medicine, and Coordinating Editor, James Lind Library. E.F., S.G., and T.La. declare employment by Cochrane. E.F. additionally declares a role as Editorial Board member for Cochrane Synthesis and Methods. T.La. additionally declares authorship of a chapter in the Cochrane Handbook for Systematic Reviews of Interventions and that he is a developer of standards for Cochrane intervention reviews (MECIR). T.Li. is funded by the National Eye Institute, National Institutes of Health (Grant #UG1 EY020522). S.L. is funded by NHMRC (APP1195189), and holds general or methodological editor positions for Cochrane Gynecology and Fertility, Fertility and Sterility, and Human Reproduction. A.L. is on the editorial board of BMC Medical Ethics. B.W.M. declares roles as Editor for Cochrane Gynecology and Fertility and Sexually Transmitted Infections, and for Fertility and Sterility. S.L. declares roles as Associate Editor for Human Reproduction, Methodological Editor for Fertility and Sterility, and Editor for Cochrane Gynecology and Fertility. N.O.C. is a member of the Cochrane Editorial Board and holds an ERA-NET Neuron Cofund grant for a separate project. A.L.S. declares funding from Australian National Health and Medical Research Council Investigator Grants (GNT2009432). E.S. is a Sign-off Editor for the Cochrane Library. M.v.W. is coordinating editor of Cochrane Gynecology and Fertility and Cochrane Sexually Transmitted Infections, Methodological Editor of Human Reproduction Update and editorial Editor of Fertility and Sterility. There are no competing interests for any other author.

Update of

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project.
Wilkinson J, Heal C, Antoniou GA, Flemyng E, Avenell A, Barbour V, Bordewijk EM, Brown NJL, Clarke M, Dumville J, Grohmann S, Gurrin LC, Hayden JA, Hunter KE, Lam E, Lasserson T, Li T, Lensen S, Liu J, Lundh A, Meyerowitz-Katz G, Mol BW, O'Connell NE, Parker L, Redman B, Seidler AL, Sheldrick K, Sydenham E, Dahly DL, van Wely M, Bero L, Kirkham JJ. Wilkinson J, et al. medRxiv [Preprint]. 2024 Mar 25:2024.03.18.24304479. doi: 10.1101/2024.03.18.24304479. medRxiv. 2024. Update in: J Clin Epidemiol. 2024 Nov;175:111512. doi: 10.1016/j.jclinepi.2024.111512. PMID: 38585914 Free PMC article. Updated. Preprint.

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- ClinicalKey
- Elsevier Science
Research Materials
- NCI CPTC Antibody Characterization Program

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A survey of experts to identify methods to detect problematic studies: stage 1 of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews project

Affiliations

A survey of experts to identify methods to detect problematic studies: stage 1 of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews project

Authors

Affiliations

Abstract

Conflict of interest statement

Update of

MeSH terms

LinkOut - more resources

Full Text Sources

Research Materials