Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2016 Sep 6;9(9):MR000007.
doi: 10.1002/14651858.MR000007.pub3.

Individual participant data meta-analyses compared with meta-analyses based on aggregate data

Catrin Tudur Smith  1 , Maura Marcucci, Sarah J Nolan, Alfonso Iorio, Maria Sudell, Richard Riley, Maroeska M Rovers, Paula R Williamson
Affiliations
Review

Individual participant data meta-analyses compared with meta-analyses based on aggregate data

Catrin Tudur Smith et al. Cochrane Database Syst Rev. .

Abstract

Background: Meta-analyses based on individual participant data (IPD-MAs) allow more powerful and uniformly consistent analyses as well as better characterisation of subgroups and outcomes, compared to those which are based on aggregate data (AD-MAs) extracted from published trial reports. However, IPD-MAs are a larger undertaking requiring greater resources than AD-MAs. Researchers have compared results from IPD-MA against results obtained from AD-MA and reported conflicting findings. We present a methodology review to summarise this empirical evidence .

Objectives: To review systematically empirical comparisons of meta-analyses of randomised trials based on IPD with those based on AD extracted from published reports, to evaluate the level of agreement between IPD-MA and AD-MA and whether agreement is affected by differences in type of effect measure, trials and participants included within the IPD-MA and AD-MA, and whether analyses were undertaken to explore the main effect of treatment or a treatment effect modifier.

Search methods: An electronic search of the Cochrane Library (includes Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effectiveness, CENTRAL, Cochrane Methodology Register, HTA database, NHS Economic Evaluations Database), MEDLINE, and Embase was undertaken up to 7 January 2016. Potentially relevant articles that were known to any of the review authors and reference lists of retrieved articles were also checked.

Selection criteria: Studies reporting an empirical comparison of the results of meta-analyses of randomised trials using IPD with those using AD. Studies were included if sufficient numerical data, comparing IPD-MA and AD-MA, were available in their reports.

Data collection and analysis: Two review authors screened the title and abstract of identified studies with full-text publications retrieved for those identified as eligible or potentially eligible. A 'quality' assessment was done and data were extracted independently by two review authors with disagreements resolved by involving a third author. Data were summarised descriptively for comparisons where an estimate of effect measure and corresponding precision have been provided both for IPD-MA and for AD-MA in the study report. Comparisons have been classified according to whether identical effect measures, identical trials and patients had been used in the IPD-MA and the AD-MA, and whether the analyses were undertaken to explore the main effect of treatment, or to explore a potential treatment effect modifier.Effect measures were transformed to a standardised scale (z scores) and scatter plots generated to allow visual comparisons. For each comparison, we compared the statistical significance (at the 5% two-sided level) of an IPD-MA compared to the corresponding AD-MA and calculated the number of discrepancies. We examined discrepancies by type of analysis (main effect or modifier) and according to whether identical trials, patients and effect measures had been used by the IPD-MA and AD-MA. We calculated the average of differences between IPD-MA and AD-MA (z scores, ratio effect estimates and standard errors (of ratio effects)) and 95% limits of agreement.

Main results: From the 9330 reports found by our searches, 39 studies were eligible for this review with effect estimate and measure of precision extracted for 190 comparisons of IPD-MA and AD-MA. We classified the quality of studies as 'no important flaws' (29 (74%) studies) or 'possibly important flaws' (10 (26%) studies).A median of 4 (interquartile range (IQR): 2 to 6) comparisons were made per study, with 6 (IQR 4 to 11) trials and 1225 (542 to 2641) participants in IPD-MAs and 7 (4 to 11) and 1225 (705 to 2541) for the AD-MAs. One hundred and forty-four (76%) comparisons were made on the main treatment effect meta-analysis and 46 (24%) made using results from analyses to explore treatment effect modifiers.There is agreement in statistical significance between the IPD-MA and AD-MA for 152 (80%) comparisons, 23 of which disagreed in direction of effect. There is disagreement in statistical significance for 38 (20%) comparisons with an excess proportion of IPD-MA detecting a statistically significant result that was not confirmed with AD-MA (28 (15%)), compared with 10 (5%) comparisons with a statistically significant AD-MA that was not confirmed by IPD-MA. This pattern of disagreement is consistent for the 144 main effect analyses but not for the 46 comparisons of treatment effect modifier analyses. Conclusions from some IPD-MA and AD-MA differed even when based on identical trials, participants (but not necessarily identical follow-up) and treatment effect measures. The average difference between IPD-MA and AD-MA in z scores, ratio effect estimates and standard errors is small but limits of agreement are wide and include important differences in both directions. Discrepancies between IPD-MA and AD-MA do not appear to increase as the differences between trials and participants increase.

Authors' conclusions: IPD offers the potential to explore additional, more thorough, and potentially more appropriate analyses compared to those possible with AD. But in many cases, similar results and conclusions can be drawn from IPD-MA and AD-MA. Therefore, before embarking on a resource-intensive IPD-MA, an AD-MA should initially be explored and researchers should carefully consider the potential added benefits of IPD.

PubMed Disclaimer

Conflict of interest statement

All authors are involved in the conduct of meta‐analyses using individual participant data. CTS and PW are responsible for at least one of the studies included in the review.

Figures

1
1
Study flow diagram.
2
2
Year of publication for 39 included studies comparing IPD‐MA and AD‐MA
3
3
Number of trials and participants in each empirical comparison of IPD‐MA and AD‐MA
4
4
Comparison of standardised effects (z scores ‐ main effect analyses)
5
5
Comparison of standardised effects (z scores ‐ interaction effect analyses)
6
6
Comparison of standardised effects split by data and effect measure (main effect analyses only)
7
7
Comparison of standardised effects split by data and effect measure (interaction effect analyses only)
8
8
Difference in standardised estimates (IPD‐AD) versus difference in participants (top panel) and trials (bottom panel) for 174 main effect and effect modifier analyses (IPD‐AD)
See this image and copyright information in PMC

Update of

References

References to studies included in this review

Berlin 2002 {published data only}
    1. Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI, Anti‐Lymphocyte Antibody Induction Therapy Study G. Individual patient‐ versus group‐level data meta‐regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Statistics in Medicine 2002;21(3):371‐87. - PubMed
Best 2000 {published data only}
    1. Best L, Simmonds P, Baughan C, Buchanan R, Davis C, Fentiman I, et al. Collaboration Colorectal Meta‐analysis. Palliative chemotherapy for advanced or metastatic colorectal cancer. Cochrane Database of Systematic Reviews 2000, Issue 1. [DOI: 10.1002/14651858.CD001545] - DOI - PMC - PubMed
Beveridge 2015 {published data only}
    1. Beveridge LA, Struthers AD, Khan F, Jorde R, Scragg R, Macdonald HM, et al. Effect of vitamin D supplementation on blood pressure: a systematic review and meta‐analysis incorporating individual patient data. JAMA Internal Medicine 2015;175(5):745‐54. - PMC - PubMed
Brouwer 2009 {published data only}
    1. Brouwer IA, Raitt MH, Dullemeijer C, Kraemer DF, Zock PL, Morris C, et al. Effect of fish oil on ventricular tachyarrhythmia in three studies in patients with implantable cardioverter defibrillators. European Heart Journal 2009;30(7):820‐6. - PMC - PubMed
Clarke 1998 {published data only}
    1. Clarke M, Godwin J. Systematic reviews using individual patient data: A map for the minefields?. Annals of Oncology 1998;9:827‐33. - PubMed
D'Amico 1998 {published data only}
    1. D'Amico R, Pifferi S, Leonetti, Torri V, Tinazzi A, Liberati A. Effectiveness of antibiotic prophylaxis in critically ill adult patients: systematic review of randomised controlled trials. BMJ 1998;316:1275–85. - PMC - PubMed
Duchateau 2001 {published data only}
    1. Duchateau L, Pignon JP, Bijnens L, Bertin S, Bourhis J, Sylvester R. Individual patient‐versus literature‐based meta‐analysis of survival data: time to event and event rate at a particular time can make a difference, an example based on head and neck cancer. Controlled Clinical Trials 2001;22(5):538‐47. - PubMed
Fortin 1995 {published data only}
    1. Fortin PR, Lew RA, Liang MH, Wright EA, Beckett LA, Chalmers TC, et al. Validation of a meta‐analysis: the effects of fish oil in rheumatoid arthritis. Journal of Clinical Epidemiology 1995;48(11):1379‐90. - PubMed
Franzosi 1997 {published data only}
    1. Franzosi MG, Santoro E, Santoro L. Prospective meta‐analysis using individual patient data vs meta‐analysis of published reports: the case of ACE‐inhibitors in myocardial infarction. Controlled Clinical Trials 1997;18:183s.
Ioannidis 1999 {published data only}
    1. Ioannidis JP, Contopoulos‐Ioannidis DG, Lau J. Recursive cumulative meta‐analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. Journal of Clinical Epidemiology 1999;52(4):281‐91. - PubMed
Jeng 1995 {published data only}
    1. Jeng GT, Scott JR, Burmeister LF. A comparison of meta‐analytic results using literature vs individual patient data: Paternal cell immunization for recurrent miscarriage. JAMA 1995;274(10):830‐6. - PubMed
Jones 2009 {published data only}
    1. Jones AP, Riley RD, Williamson PR, Whitehead A. Meta‐analysis of individual patient data versus aggregate data from longitudinal clinical trials. Clinical Trials 2009;6(1):16‐27. - PubMed
Kim 2010 {published data only}
    1. Kim PW, Wu YT, Cooper C, Rochester G, Valappil T, Wang Y, et al. Meta‐analysis of a possible signal of increased mortality associated with cefepime use. Clinical Infectious Diseases 2010;51(4):381‐9. - PubMed
Koopman 2008 {published data only}
    1. Koopman L, Heijden GJ, Hoes AW, Grobbee DE, Rovers MM. Empirical comparison of subgroup effects in conventional and individual patient data meta‐analyses. International Journal of Technology Assessment in Health Care 2008;24(3):358‐61. - PubMed
le Chevalier 1996 {published data only}
    1. Chevalier T. Chemotherapy for advanced NSCLC: Will meta‐analysis provide the answer?. Chest 1996;109(5 Suppl):107s‐9s. - PubMed
Legg 2003 {published data only}
    1. Legg L, Leonardi‐Bee J, Langhorne P, Walker M. Is getting individual patient data for meta‐analyses worthwhile?. XI Cochrane Colloquium: Evidence, Health Care and Culture. 2003.
Lindley 2005 {published data only}
    1. Lindley RI, Wardlaw JM, Sandercock PA. Alteplase and ischaemic stroke: have new reviews of old data helped?. Lancet Neurology 2005;4(4):249‐53. - PubMed
Lukka 2006 {published data only}
    1. Lukka H, Waldron T, Klotz L, Winquist E, Trachtenberg J. Maximal androgen blockade for the treatment of metastatic prostate cancer‐‐a systematic review. Current Oncology (Toronto, Ont.) 2006;13(3):81‐93. - PMC - PubMed
Michiels 2005 {published data only}
    1. Michiels S, Piedbois P, Burdett S, Syz N, Stewart L, Pignon JP. Meta‐analysis when only the median survival times are known: a comparison with individual patient data results. International Journal of Technology Assessment in Health Care 2005;21(1):119‐25. - PubMed
Myeloma 1998 {published data only}
    1. Myeloma Trialists' Collaborative Group. Combination chemotherapy versus melphalan plus prednisone as treatment for multiple myeloma: an overview of 6,633 patients from 27 randomized trials. Journal of Clinical Oncology 1998;16(12):3832‐42. - PubMed
Pignon 1992 {published data only}
    1. Pignon JP, Arriagada R. Role of thoracic radiotherapy in limited‐stage small‐cell lung cancer: quantitative review based on the literature versus meta‐analysis based on individual data. Journal of Clinical Oncology 1992;10(11):1819‐20. - PubMed
Rejnmark 2012 {published data only}
    1. Rejnmark L, Avenell A, Masud T, Anderson F, Meyer HE, Sanders KM, et al. Vitamin D with calcium reduces mortality: patient level pooled analysis of 70,528 patients from eight major vitamin D trials. Journal of Clinical Endocrinology 2012;97(8):2670‐81. - PMC - PubMed
Rothwell 2011 {published data only}
    1. Rothwell PM, Fowkes FG, Belch JF, Ogawa H, Warlow CP, Meade TW. Effect of daily aspirin on long‐term risk of death due to cancer: analysis of individual patient data from randomised trials. Lancet 2011;377(9759):31‐41. - PubMed
Saillourglenisson 2000 {published data only}
    1. Saillourglenisson F, Chene G, Salmi LR, Hafner R, Salamon R. Effect of dapsone on survival in HIV infected patients: a meta‐ analysis of finished trials [Effet de la dapsone sur la survie des patients infectés par le VIH : une métaanalyse des essais terminés]. Revue d'Epidemiologie et de Sante Publique 2000;48(1):17‐30. - PubMed
Schmid 2004 {published data only}
    1. Schmid CH, Stark PC, Berlin JA, Landais P, Lau J. Meta‐regression detected associations between heterogeneous treatment effects and study‐level, but not patient‐level, factors. Journal of Clinical Epidemiology 2004;57(7):683‐97. - PubMed
Shepperd 2009 {published data only}
    1. Shepperd S, Doll H, Broad J, Gladman JS, Langhorne P, Richards S, et al. Hospital at home early discharge. Cochrane Database of Systematic Reviews 2009, Issue 1. [DOI: 10.1002/14651858.CD000356.pub3] - DOI - PMC - PubMed
Spooner 1998 {published data only}
    1. Spooner C, Rowe BH, Saunders LD, Milner RA. Nedocromil sodium as treatment of exercise‐induced bronchoconstriction: a comparison of results from a meta‐analysis with individual patient data. 6th Cochrane Colloquium. 1998; Vol. A03.
Stewart 1993 {published data only}
    1. Stewart LA, Parmar MK. Meta‐analysis of the literature or of individual patient data: is there a difference?. Lancet 1993;341(8842):418‐22. - PubMed
Szczech 1998 {published data only}
    1. Szczech LA, Berlin JA, Feldman HI, for the Anti‐Lymphocyte Antibody Induction Therapy Study Group. The effect of antilymphocyte induction therapy on renal allograft survival. Annals of Internal Medicine 1998;128:817‐26. - PubMed
Teramukai 2004 {published data only}
    1. Teramukai S, Matsuyama Y, Mizuno S, Sakamoto J. Individual patient‐level and study‐level meta‐analysis for investigating modifiers of treatment effect. Japanese Journal of Clinical Oncology 2004;34(12):717‐21. - PubMed
Thompson 2001 {published data only}
    1. Thompson SG, Turner RM, Warn DE. Multilevel models for meta‐analysis, and their application to absolute risk differences. Statistical Methods in Medical Research 2001;10(6):375‐92. - PubMed
Tierney 2001 {published data only}
    1. Tierney J, Rydzewska L, Burdett S. Feasibility and reliability of using hazard ratios in meta‐analyses of published time‐to‐event data. 9th Cochrane Colloquium. 2001; Vol. O‐002.
Tonia 2011 {published data only}
    1. Tonia T, Bohlius J. Ten years of meta‐analyses on erythropoiesis‐stimulating agents in cancer patients. Cancer Treatment and Research 2011;157:217‐38. - PubMed
Tudur 2001 {published data only}
    1. Tudur C, Williamson PR, Khan SA, Best L. The value of the aggregate data approach in meta‐analysis with time‐to‐event outcomes. Journal of the Royal Statistical Society. Series a (General) 2001;164:357‐70.
Tudur Smith 2005 {published data only}
    1. Tudur Smith C, Williamson PR, Marson AG. An overview of methods and empirical comparison of aggregate data and individual patient data results for investigating heterogeneity in meta‐analysis of time‐to‐event outcomes. Journal of Evaluation in Clinical Practice 2005;11(5):468‐78. - PubMed
Turner 2000 {published data only}
    1. Turner RM, Omar RZ, Yang M, Goldstein H, Thompson SG. A multilevel model framework for meta‐analysis of clinical trials with binary outcomes. Statistics in Medicine 2000;19(24):3417‐32. - PubMed
Vansteenkiste 2012 {published data only}
    1. Vansteenkiste J, Glaspy J, Henry D, Ludwig H, Pirker R, Tomita D, et al. Benefits and risks of using erythropoiesis‐stimulating agents (ESAs) in lung cancer patients: study‐level and patient‐level meta‐analyses. Lung Cancer (Amsterdam, Netherlands) 2012;76(3):478‐85. - PubMed
Walwyn 2015 {published data only}
    1. Walwyn R, Roberts C. Meta‐analysis of absolute mean differences from randomised trials with treatment‐related clustering associated with care providers. Statistics in Medicine 2015;34(6):966‐83. - PubMed
Williamson 2000 {published data only}
    1. Williamson PR, Marson AG, Tudur C, Hutton JL, Chadwick D. Individual patient data meta‐analysis of randomized anti‐epileptic drug monotherapy trials. Journal of Evaluation in Clinical Practice 2000;6(2):205‐14. - PubMed

Additional references

Ahmed 2012
    1. Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta‐analyses using individual participant data: a database survey. BMJ 2012;3(344):d7762. - PubMed
Chalmers 1993
    1. Chalmers I. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences 1993;703:156‐65. - PubMed
Clarke 1997
    1. Clarke M, Stewart L. Individual patient data or published data meta‐analysis: a systematic review. Proceedings of the Fifth Cochrane Collaboration Colloquium. 1997:94, abstract 019.04.
Cools 2010
    1. Cools F, Askie L, Offringa M, Asselin M, Calvert JM, Courtney SA, et al. Elective high‐frequency oscillatory versus conventional ventilation in preterm infants: a systematic review and meta‐analysis of individual patients' data. Lancet 2010;375:2082‐91. - PubMed
Cooper 2009
    1. Cooper H, Patall EA. The relative benefits of meta‐analysis conducted with individual participant data versus aggregated data. Psychological Methods 2009;14(2):165‐76. - PubMed
Cope 2012
    1. Cope S, Zhang J, Williams J, Jansen JP. Efficacy of once‐daily indacaterol 75 μg relative to alternative bronchodilators in COPD: a study level and a patient level network meta‐analysis. BMC Pulmonary Medicine 2012;12:29. - PMC - PubMed
Donegan 2013
    1. Donegan S, Williamson P, D'Alessandro U, Garner P, Tudur Smith C. Combining individual patient data and aggregate data in mixed treatment comparison meta‐analysis: Individual patient data may be beneficial if only for a subset of trials. Statistics in Medicine 2013;32(6):914‐930. - PubMed
Horsley 2011
    1. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews 2011, Issue 8. [DOI: 10.1002/14651858.MR000026.pub2; PUBMED: 21833989] - DOI - PMC - PubMed
Lambert 2002
    1. Lambert PC, Sutton AJ, Abrams KR, Jones DR. A comparison of summary patient‐level covariates in meta‐regressionwith individual patient data meta‐analysis. Journal of Clinical Epidemiology 2002;55:86–94. - PubMed
Mathew 1999
    1. Mathew T, Nordström K. On the equivalence of meta‐analysis using literature and using individual patient data. Biometrics 1999;55(4):1221‐3. - PubMed
Mukhtar 2008
    1. Mukhtar MA. Incorporation of heterogeneity in meta‐analysis of randomised controlled trials. PhD Thesis 2008.
Olkin 1998
    1. Olkin I, Sampson A. Comparison of meta‐analysis versus analysis of variance of individual patient data. Biometrics 1998;54:317‐22. - PubMed
Steinberg 1997
    1. Steinberg KK, Smith SJ, Stroup DF, Olkin I, Lee NC, Williamson GD, et al. Comparison of effect estimates from a meta‐analysis of summary data from published studies and from a meta‐analysis using individual patient data for ovarian cancer studies. American Journal of Epidemiology 1997;145:917‐25. - PubMed

References to other published versions of this review

Clarke 2001
    1. Clarke M, Stewart L, Tierney J, Williamson P. Individual patient data meta‐analyses compared with meta‐analyses based on aggregate data. Cochrane Database of Systematic Reviews 2001, Issue 3. [DOI: 10.1002/14651858.MR000007] - DOI
Clarke 2007
    1. Clarke MJ, Stewart L, Tierney J, Williamson PR. Individual patient data meta‐analyses compared with meta‐analyses based on aggregate data. Cochrane Database of Systematic Reviews 2007, Issue 2. [DOI: 10.1002/14651858.MR000007.pub2] - DOI - PMC - PubMed