Assessing the performance of a computer-based policy model of HIV and AIDS

Abstract

Background: Model-based analyses, conducted within a decision analytic framework, provide a systematic way to combine information about the natural history of disease and effectiveness of clinical management strategies with demographic and epidemiological characteristics of the population. Among the challenges with disease-specific modeling include the need to identify influential assumptions and to assess the face validity and internal consistency of the model.

Methods and findings: We describe a series of exercises involved in adapting a computer-based simulation model of HIV disease to the Women's Interagency HIV Study (WIHS) cohort and assess model performance as we re-parameterized the model to address policy questions in the U.S. relevant to HIV-infected women using data from the WIHS. Empiric calibration targets included 24-month survival curves stratified by treatment status and CD4 cell count. The most influential assumptions in untreated women included chronic HIV-associated mortality following an opportunistic infection, and in treated women, the 'clinical effectiveness' of HAART and the ability of HAART to prevent HIV complications independent of virologic suppression. Good-fitting parameter sets required reductions in the clinical effectiveness of 1st and 2nd line HAART and improvements in 3rd and 4th line regimens. Projected rates of treatment regimen switching using the calibrated cohort-specific model closely approximated independent analyses published using data from the WIHS.

Conclusions: The model demonstrated good internal consistency and face validity, and supported cohort heterogeneities that have been reported in the literature. Iterative assessment of model performance can provide information about the relative influence of uncertain assumptions and provide insight into heterogeneities within and between cohorts. Description of calibration exercises can enhance the transparency of disease-specific models.

Figure 1. Base case natural history data with WIHS cohort characteristics.

Part A of Figure 1 shows the model-estimated survival of the WIHS cohort using natural history input parameters derived from the MACS. With the exception of the highest CD4 stratum (CD4 ≥350/µl), the model underestimates survival for individuals with initial CD4 cell counts <350/µl, particularly as follow-up time increased. Part B of Figure 1 shows the base-case model-estimated survival using natural history input parameters derived from the WIHS. The re-parameterized model more closely approximates the empiric data for the three lowest CD4 strata (generally within the 95% CI), although the model continues to marginally underestimate mean survival in CD4 strata CD4 50–199/µl and CD4 200–349/µl. Model-projected survival in the highest CD4 stratum (≥350/µl) is more significantly underestimated, with a better visual fit achieved using natural history inputs derived from the MACS (Figure 1, Part A). Part C of Figure 1 shows the impact of adjustment of CD4-specific attribution on model-projected survival. Part C of Figure 1 demonstrates that better consistency between model-projected survival and empiric data was best achieved with adjustment of CD4 stratum-specific attribution. Specifically, reduction of the incrementally increased probability of AIDS-related mortality in patients with a history of previous opportunistic infections (attribution) by 25% for CD4 50–199/µl and 50% for CD4 strata ≥200/µl resulted in better estimation of the empiric survival data.

Figure 2. Top 50 best fits of WIHS empiric survival for CD4 50–199/µl and CD4 <50/µl.

Part A of Figure 2 illustrates the top 50 best fits of the empiric survival data for CD4 50–199/µl. Nearly all runs in the top 50 combined changes in both ‘clinical effectiveness’ of HAART (a function of regimen efficacy, side effects or toxicity, adherence, and personal choice to remain on HAART) and estimates of CD4 gains while on effective HAART. The majority of the 50 best fits had a 2- to 3-fold increase in the rates of failure/switching/discontinuation of early lines of HAART in combination with an increase of 40%–75% in treatment efficacy in later lines of HAART. Part B of Figure 2 illustrates the top 50 best fits of the empiric survival data for CD4 <50/µl. Nearly all runs in the top 50 combined changes in both ‘clinical effectiveness’ of treatment (a function of regimen efficacy, side effects or toxicity, adherence, and personal choice to remain on HAART) and estimates of CD4 gains while on effective HAART. The majority of the 50 best fits had a 2- to 3-fold increase in the rates of failure/switching/discontinuation of early lines of HAART in combination with an increase of 30%–75% in treatment efficacy in later lines of HAART.

Figure 3. Impact of alternative assumptions on life expectancy.

Results from sensitivity analyses showed that the most influential variable on long-term outcomes was the probability of late treatment failure. Part A of Figure 3, shows the impact of varying our base case assumptions (0.021) from no late failure to a 2-fold increase in late failure. Depending on the baseline CD4 cell count, life-expectancy was increased by 14.8 to 30.9 months with no late failure (green), and was decreased by 2.8 to 6.6 months with 1.5-fold increase in late failure (orange), and by 5.1 to 11.0 months with a 2-fold increase in late failure (red). The magnitude of these changes was similar regardless of whether we assumed 4 lines or 5 lines of HAART. Part B of Figure 3 shows the impact of 5 lines of HAART versus 4 lines of HAART on life expectancy. The average life expectancy projected by the model calibrated to the 24-month short-term data (using the mean of the 50 best-fitting sets) was 140.9 months (range, 130.5-148.4 months) among the patients with CD4 50–199/µl and 80.1 months (range, 65.9–87.3 months) among those with CD4 <50/µl. Average life-expectancy projected by the uncalibrated model varied with different assumptions about the ART effect, ranging from 123.3 months (no ART effect) to 156.5 months (ART effect) in patients with CD4 50–199/µl, and from 73.2 months (no ART effect) to 100.1 months in patients with CD4 <50/µl (ART effect). Figure 3, Part B, shows the incremental gains provided by 5 lines of HAART versus 4 lines of HAART were greatest using the calibrated model (green bars), and lowest using the uncalibrated model assuming no ART eff ect (orange bars).