Performance of statistical models to predict mental health and substance abuse cost

Abstract

Background: Providers use risk-adjustment systems to help manage healthcare costs. Typically, ordinary least squares (OLS) models on either untransformed or log-transformed cost are used. We examine the predictive ability of several statistical models, demonstrate how model choice depends on the goal for the predictive model, and examine whether building models on samples of the data affects model choice.

Methods: Our sample consisted of 525,620 Veterans Health Administration patients with mental health (MH) or substance abuse (SA) diagnoses who incurred costs during fiscal year 1999. We tested two models on a transformation of cost: a Log Normal model and a Square-root Normal model, and three generalized linear models on untransformed cost, defined by distributional assumption and link function: Normal with identity link (OLS); Gamma with log link; and Gamma with square-root link. Risk-adjusters included age, sex, and 12 MH/SA categories. To determine the best model among the entire dataset, predictive ability was evaluated using root mean square error (RMSE), mean absolute prediction error (MAPE), and predictive ratios of predicted to observed cost (PR) among deciles of predicted cost, by comparing point estimates and 95% bias-corrected bootstrap confidence intervals. To study the effect of analyzing a random sample of the population on model choice, we re-computed these statistics using random samples beginning with 5,000 patients and ending with the entire sample.

Results: The Square-root Normal model had the lowest estimates of the RMSE and MAPE, with bootstrap confidence intervals that were always lower than those for the other models. The Gamma with square-root link was best as measured by the PRs. The choice of best model could vary if smaller samples were used and the Gamma with square-root link model had convergence problems with small samples.

Conclusion: Models with square-root transformation or link fit the data best. This function (whether used as transformation or as a link) seems to help deal with the high comorbidity of this population by introducing a form of interaction. The Gamma distribution helps with the long tail of the distribution. However, the Normal distribution is suitable if the correct transformation of the outcome is used.

Figure 1

Predictive ratios (PR) per decile of predicted cost in full sample (N = 525,620). PR is computed as the ratio of predicted cost to observed cost for deciles of predicted cost. For each decile, PR = 1 when mean predicted cost equals mean observed cost. Also shown, are 95% bias-corrected bootstrap confidence intervals.

Figure 2

95% root mean square error (RMSE) percentile intervals per model at each simulation of various sample sizes. RMSE=1n∑i=1n(yi−y^i)2 MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGsbGucqWGnbqtcqWGtbWucqWGfbqrcqGH9aqpdaGcaaqaamaalaaabaGaeGymaedabaGaemOBa4gaamaaqadabaGaeiikaGIaemyEaK3aaSbaaSqaaiabdMgaPbqabaGccqGHsislcuWG5bqEgaqcamaaBaaaleaacqWGPbqAaeqaaOGaeiykaKYaaWbaaSqabeaacqaIYaGmaaaabaGaemyAaKMaeyypa0JaeGymaedabaGaemOBa4ganiabggHiLdaaleqaaaaa@4557@ Large values indicate a poor fit. The simulations at each sample size are based on 100 samples with the exception of the simulations for the Gamma Square Root model. Samples for which the model did not converge are dropped: 50 when sampling 5,000 subjects, 15 for 10,000, 16 for 15,000, 8 for 20,000, 7 for 25,000 and 30,000, 5 for 35,000, and 1 for 50,000 and 55,000.

Figure 3

95% mean absolute prediction error (MAPE) percentile intervals per model at each simulation of various sample sizes. MAPE=1n∑i=1n|yi−y^i| MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGnbqtcqWGbbqqcqWGqbaucqWGfbqrcqGH9aqpdaWcaaqaaiabigdaXaqaaiabd6gaUbaadaaeWaqaamaaemaabaGaemyEaK3aaSbaaSqaaiabdMgaPbqabaGccqGHsislcuWG5bqEgaqcamaaBaaaleaacqWGPbqAaeqaaaGccaGLhWUaayjcSdaaleaacqWGPbqAcqGH9aqpcqaIXaqmaeaacqWGUbGBa0GaeyyeIuoaaaa@4570@ Large values indicate a poor fit. The simulations at each sample size are based on 100 samples with the exception of the simulations for the Gamma Square Root model. Samples for which the model did not converge are dropped: 50 when sampling 5,000 subjects, 15 for 10,000, 16 for 15,000, 8 for 20,000, 7 for 25,000 and 30,000, 5 for 35,000, and 1 for 50,000 and 55,000.

Figure 4

95% predicted ratio for decile 10 (PR10) percentile intervals per model at each simulation of various sample sizes. PR is computed as the ratio of predicted cost to observed cost whithin decile 10 of predicted cost. PR = 1 when mean predicted cost equals mean observed cost. The simulations at each sample size are based on 100 samples with the exception of the simulations for the Gamma Square Root model. Samples for which the model did not converge are dropped: 50 when sampling 5000 subjects, 15 for 10,000, 16 for 15,000, 8 for 20,000, 7 for 25,000 and 30,000, 5 for 35,000, and 1 for 50,000 and 55,000.