Quality of reporting of cancer prognostic marker studies: association with reported prognostic effect

Abstract

Background: Issues of reported study quality have not been addressed empirically with large-scale data in the cancer prognostic literature.

Methods: Eight quality measures pertaining to study design and assay methods (i.e., blinding, prospective versus retrospective design, power calculations, outcomes' definitions, time of enrollment, reporting of variables, assay description, and assay reference) were evaluated in cancer prognostic marker studies included in meta-analyses identified in Medline and EMBASE. To be eligible, meta-analyses had to include at least six studies and to examine binary outcomes. We estimated the ratios of relative risks, which compared the overall prognostic effects (summary relative risks) between poor-quality and good-quality studies for each quality item. Between-study heterogeneity was tested with the Q statistic (statistically significant at P<.10). All statistical tests were two-sided.

Results: We identified 20 meta-analyses that included 331 cancer prognostic marker studies published between 1987 and 2005. Only three (0.9%) of the 331 studies presented power calculations, 129 (39.0%) studies stated that analyses were blinded, and 73 (21.5%) stated that they were prospective. Time of enrollment was defined in 232 (70.0%), 234 (70.7%) gave lists of candidate variables, and 254 (76.7%) defined outcomes. The assay used was described in 317 (95.8%), but only 177 (53.5%) provided the assay reference. Estimates of prognostic effects from poor-quality studies varied considerably and could be larger or smaller than summary estimates derived from meta-analyses. Summary ratios of relative risks of poor- versus good-quality studies for the seven quality measures ranged from 0.95 to but 1.26, but none was statistically significantly. There was statistically significant heterogeneity (P<.10) between the ratios of relative risk estimates across meta-analyses for blinding, defining endpoints, and stating variables and assay references.

Conclusions: Among cancer prognostic marker studies, reporting quality of design and assay information often appears suboptimal, indicating that this literature may be largely unreliable. Given the potential clinical importance of prognostic marker information, improved design and reporting of these studies are warranted.