Validation and calibration of a computer simulation model of pediatric HIV infection

Abstract

Background: Computer simulation models can project long-term patient outcomes and inform health policy. We internally validated and then calibrated a model of HIV disease in children before initiation of antiretroviral therapy to provide a framework against which to compare the impact of pediatric HIV treatment strategies.

Methods: We developed a patient-level (Monte Carlo) model of HIV progression among untreated children <5 years of age, using the Cost-Effectiveness of Preventing AIDS Complications model framework: the CEPAC-Pediatric model. We populated the model with data on opportunistic infection and mortality risks from the International Epidemiologic Database to Evaluate AIDS (IeDEA), with mean CD4% at birth (42%) and mean CD4% decline (1.4%/month) from the Women and Infants' Transmission Study (WITS). We internally validated the model by varying WITS-derived CD4% data, comparing the corresponding model-generated survival curves to empirical survival curves from IeDEA, and identifying best-fitting parameter sets as those with a root-mean square error (RMSE) <0.01. We then calibrated the model to other African settings by systematically varying immunologic and HIV mortality-related input parameters. Model-generated survival curves for children aged 0-60 months were compared, again using RMSE, to UNAIDS data from >1,300 untreated, HIV-infected African children.

Results: In internal validation analyses, model-generated survival curves fit IeDEA data well; modeled and observed survival at 16 months of age were 91.2% and 91.1%, respectively. RMSE varied widely with variations in CD4% parameters; the best fitting parameter set (RMSE = 0.00423) resulted when CD4% was 45% at birth and declined by 6%/month (ages 0-3 months) and 0.3%/month (ages >3 months). In calibration analyses, increases in IeDEA-derived mortality risks were necessary to fit UNAIDS survival data.

Conclusions: The CEPAC-Pediatric model performed well in internal validation analyses. Increases in modeled mortality risks required to match UNAIDS data highlight the importance of pre-enrollment mortality in many pediatric cohort studies.

Figure 1. CEPAC-Pediatric model structure.

A schematic of the Cost-Effectiveness of Preventing AIDS Complications (CEPAC)-Pediatric natural history model (see Methods for details).

Figure 2. Internal validation of survival outcomes: Observed survival curves from the IeDEA East African region and projected results from the CEPAC-Pediatric Model.

The solid black stepped line represents observed survival in the IeDEA cohort based on Kaplan-Meier analysis, beginning at 5 months of age. Dashed black lines reflect the upper and lower bounds of the 95% confidence intervals for IeDEA-observed survival. The orange line shows CEPAC model-projected survival using the "IeDEA-WITS projection" data (RMSE = 0.0103). The best-fitting curve is shown with the red line, reflecting mean CD4% at birth of 45.0%, CD4% decline of 6.0%/month in infants <3 months of age, and 0.3%/month for children >3 months of age (RMSE = 0.00423).

Figure 3. Internal validation of clinical event risk outcomes: CEPAC-Pediatric model results compared to IeDEA data.

Risks of clinical events from 5-16 months of age, as observed among infants in the IeDEA East Africa region and projected by the CEPAC-Pediatric model. Simulated infants enter the model with the CD4% at birth identified in the best-fitting parameter set for the internal validation survival analyses (45.0%), and CD4% values decline as per the best-fitting parameter set (6.0%/month ages 0-3 months, 0.3%/month ages ≥3 months). Simulated infants face competing risks of all three types of clinical events, as well as "acute" and "chronic" mortality. Due to differing methods of reporting, IeDEA event risks (reported for three distinct CD4 strata) could not directly be compared to model-projected event risks (reported as a cohort average, where the cohort consists of a population with a unique distribution of CD4% each month). To generate a comparable IeDEA risk for each clinical event, we calculated an average of the three reported risks from IeDEA (CD4 <15%, CD4 15-25%, CD4 >25%) weighted by the proportion of the cohort in each CD4% strata during each month of the simulation. Model-generated rates are expected to be slightly lower than IeDEA-observed rates, due to 1) model accounting of OIs (which permits only one OI to be recorded each month), and 2) competing risks of other OIs and chronic HIV mortality in the model. TB: tuberculosis, PY: person-years.

Figure 4. CEPAC-Pediatric model calibration analyses: Projected survival (A).

Model-projected survival curves from age 0-60 months for: 1) Base-case IeDEA mortality data used in the internal validation analyses (purple line); 2) the empiric UNAIDS mortality data (black line); 3) 10 of the best-fitting parameter sets (with the lowest RMSE) identified in the calibration analyses (group of colored lines surrounding and almost completely overlapping with the black UNAIDS line); 4) the lowest-mortality risk parameter set from Table 3 (blue line) and 5) the highest-mortality risk parameter set from Table 3 (red line). The 10 sample best-fitting parameter sets from calibration analyses are almost entirely obscured by the UNAIDS survival data (black line) due to their extremely close fit to the calibration target. The IeDEA survival curve from internal validation analyses, and both the highest- and lowest-mortality risk parameter sets are all projected to 60 months of age for comparison only, as they did not meet the threshold of UNAIDS risk ±1% at 6 months and therefore were not formally evaluated at subsequent time points in the calibration analyses. B: A zoom plot, enlarging the results for months 0-6, shows the nearly-overlapping curves in larger detail.