Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Mar;51(3):275-84.
doi: 10.1097/MLR.0b013e31827ff0dc.

The relationship between the C-statistic of a risk-adjustment model and the accuracy of hospital report cards: a Monte Carlo Study

Peter C Austin  1 , Mathew J Reeves
Affiliations

The relationship between the C-statistic of a risk-adjustment model and the accuracy of hospital report cards: a Monte Carlo Study

Peter C Austin et al. Med Care. 2013 Mar.

Abstract

Background: Hospital report cards, in which outcomes following the provision of medical or surgical care are compared across health care providers, are being published with increasing frequency. Essential to the production of these reports is risk-adjustment, which allows investigators to account for differences in the distribution of patient illness severity across different hospitals. Logistic regression models are frequently used for risk adjustment in hospital report cards. Many applied researchers use the c-statistic (equivalent to the area under the receiver operating characteristic curve) of the logistic regression model as a measure of the credibility and accuracy of hospital report cards.

Objectives: To determine the relationship between the c-statistic of a risk-adjustment model and the accuracy of hospital report cards.

Research design: Monte Carlo simulations were used to examine this issue. We examined the influence of 3 factors on the accuracy of hospital report cards: the c-statistic of the logistic regression model used for risk adjustment, the number of hospitals, and the number of patients treated at each hospital. The parameters used to generate the simulated datasets came from analyses of patients hospitalized with a diagnosis of acute myocardial infarction in Ontario, Canada.

Results: The c-statistic of the risk-adjustment model had, at most, a very modest impact on the accuracy of hospital report cards, whereas the number of patients treated at each hospital had a much greater impact.

Conclusions: The c-statistic of a risk-adjustment model should not be used to assess the accuracy of a hospital report card.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Relationship between the magnitude of the effect of the risk score on mortality and between−hospital variation in expected 30−day mortality rates For a given value of the ICC, the differences in the distribution of the expected hospital-specific mortality rate described by the lines of the graph reflect differences in case-mix across hospitals. These lines were generated in a setting in which there were no differences in hospital-performance: all differences in outcomes between hospital were solely due to differences in case-mix and random variation.
Figure 2
Figure 2
Relationship between AUC and the correlation between the O−E/P−E ratio and the hospital−specific random effect (ICI = 0.037)
Figure 3
Figure 3
Relationship between AUC and the correlation between the O−E/P−E ratio and the hospital−specific random effect (ICC = 0.10)
Figure 4
Figure 4
Relationship between the c−statistic and percentage of hospitals correctly classified (ICC = 0.037)
Figure 5
Figure 5
Relationship between the c−statistic and percentage of hospitals correctly classified (ICC = 0.10)
See this image and copyright information in PMC

Comment in

References

    1. Luft HS, Romano PS, Remy LL, Rainwater J. Annual Report of the California Hospital Outcomes Project. Sacramento, CA: California Office of Statewide Health Planning and Development; 1993.
    1. Pennsylvania Health Care Cost Containment Council. Focus on heart attack in Pennsylvania: research methods and results. Harrisburg, PA: Pennsylvania Health Care Cost Containment Council; 1996.
    1. Scottish Office. Clinical outcome indicators, 1994. Vol. 1995 Scottish Office; 1995.
    1. Naylor CD, Rothwell DM, Tu JV, Austin PC . the Cardiac Care Network Steering Committee. Outcomes of Coronary Artery Bypass Surgery in Ontario. In: Naylor CD, Slaughter PM, editors. Cardiovascular Health and Services in Ontario: An ICES Atlas. Institute for Clinical Evaluative Sciences; Toronto: 1999. pp. 189–198.
    1. Massachusetts Data Analysis Center. Adult Coronary Artery Bypass Graft Surgery in the Commonwealth of Massachusetts: Fiscal Year 2010 Report. Boston, MA: Department of Health Care Policy, Harvard Medical School; 2012.

Publication types