COMPOSITE SCORES FOR TRANSPLANT CENTER EVALUATION: A NEW INDIVIDUALIZED EMPIRICAL NULL METHOD

Abstract

Risk-adjusted quality measures are used to evaluate healthcare providers with respect to national norms while controlling for factors beyond their control. Existing healthcare provider profiling approaches typically assume that the between-provider variation in these measures is entirely due to meaningful differences in quality of care. However, in practice, much of the between-provider variation will be due to trivial fluctuations in healthcare quality, or unobservable confounding risk factors. If these additional sources of variation are not accounted for, conventional methods will disproportionately identify larger providers as outliers, even though their departures from the national norms may not be "extreme" or clinically meaningful. Motivated by efforts to evaluate the quality of care provided by transplant centers, we develop a composite evaluation score based on a novel individualized empirical null method, which robustly accounts for overdispersion due to unobserved risk factors, models the marginal variance of standardized scores as a function of the effective sample size, and only requires the use of publicly-available center-level statistics. The evaluations of United States kidney transplant centers based on the proposed composite score are substantially different from those based on conventional methods. Simulations show that the proposed empirical null approach more accurately classifies centers in terms of quality of care, compared to existing methods.

Keywords: Empirical null; End-stage renal disease; Provider profiling; Unmeasured confounders.

Fig 1.

Funnel plots for four measures, using different standardization methods. TRR: Transplant Rate Ratio, SAR: Standardized Acceptance Ratio, PSMR: Patient Standardized Mortality Ratio, SGFR: Standardized Graft Failure Ratio. Centers that fall outside of the control limit lines are considered outliers. The solid and dotted lines correspond to the control limits based on fixed-effects standardization and empirical null standardization, respectively. The dot-dashed line represents the null value of one. One center with extremely large measure values is excluded for visual clarity.

Fig 2.

Flagging probability for different quality of care effect sizes, with (a) no outliers and (b) 10% outliers. The solid, dashed, and dotted lines correspond to fixed-effects, method-of-moments, and empirical null standardization respectively. The fixed-effects approach does not account for overdispersion, the method-of-moments is an existing correction (Section 1), and the empirical null is the proposed correction.

Fig 3.

(a): Flagging probability and (b): Average estimate of ${\sigma }_{{\alpha }_k}^2$ (the variance of the unobserved quantity, ${\alpha }_{ik}$), for different tuning parameter $q$ values. The dashed and dotted lines correspond to method-of-moments and empirical null standardization respectively. In (b), the solid line corresponds to the true value of 0.14. The method-of-moments is an existing correction (Section 1), and the empirical null is the proposed correction.

Fig 4.

Flagging probabilities based on a simulated composite score of two measures, for four centers of interest: Center 1 with poor access and poor outcomes, Center 2 with good access and good outcomes, Center 3 with poor access and good outcomes, and Center 4 with good access and poor outcomes. The horizontal axis corresponds to the quality of care effect size with respect to the first measure (i.e., ${\gamma }_{i1}^{*}$ in Section 2.2). The solid, dashed, and dotted lines correspond to fixed-effects, method-of-moments, and empirical null standardization respectively. The fixed-effects approach does not account for overdispersion, the method-of-moments is an existing correction (Section 1), and the empirical null is the proposed correction.

Fig 5.

(Top Row): Descriptive analyses based on naive Z-scores, $Z_{FE}$, using fixed-effects standardization, and (Bottom Row): Descriptive analyses based on the proposed Z-scores, $Z_{EN}$, using individualized empirical null standardization. In each row, (Left): Histogram of Z-scores for the Transplant Rate Ratio measure; Solid curve: standard normal distribution. (Middle): Variance of Z-score within each group of transplant centers. (Right): Proportion of centers flagged for providing extremely poor or good care.

Fig 6.

Plots of (a) Transplant Rate Ratio, (b) naive fixed-effects $Z$-score $\left(Z_{FE}\right)$, and (c) proposed empirical null $Z$-score $\left(Z_{EN}\right)$, against effective sample size. The sizes of the points are proportional to the effective sample size on the horizontal axis. In all plots, the solid line corresponds to the null value of the statistics, which represents average care that is consistent with national expectations.

Fig 7.

Heat map of center performance for the 20 centers that would be flagged as poor-performers using historical methods (based on the PSMR and SGFR measures only). Lower $Z_{EN}$ values (and darker shades) correspond to worse quality of care for the corresponding measures. Centers that fall above the horizontal line are flagged as poor-performers by the proposed composite score approach. TRR: Transplant Rate Ratio, SAR: Standardized Acceptance Ratio, PSMR: Patient Standardized Mortality Ratio, SGFR: Standardized Graft Failure Ratio.

Fig 8.

National distributions of the standardized Z-scores $\left(Z_{EN}\right)$ for the composite score and each measure component. Lower Z-score values correspond to worse quality of care for the corresponding measures. The Z-score for Center 18 from Figure 7 is marked with a dashed line. The dark shaded region of the top panel represents the flagging region based on a threshold of −1.96. TRR: Transplant Rate Ratio, SAR: Standardized Acceptance Ratio, PSMR: Patient Standardized Mortality Ratio, SGFR: Standardized Graft Failure Ratio.