Statistical Methods for Cohort Studies of CKD: Prediction Modeling

Abstract

Prediction models are often developed in and applied to CKD populations. These models can be used to inform patients and clinicians about the potential risks of disease development or progression. With increasing availability of large datasets from CKD cohorts, there is opportunity to develop better prediction models that will lead to more informed treatment decisions. It is important that prediction modeling be done using appropriate statistical methods to achieve the highest accuracy, while avoiding overfitting and poor calibration. In this paper, we review prediction modeling methods in general from model building to assessing model performance as well as the application to new patient populations. Throughout, the methods are illustrated using data from the Chronic Renal Insufficiency Cohort Study.

Keywords: C-statistic; Calibration; Cohort Studies; Disease Progression; Humans; ROC curve; Renal Insufficiency, Chronic; Risk; Sensitivity; Specificity.

Figure 1.

Plot of risk score against CKD progression within 2 years in the Chronic Renal Insufficiency Cohort (CRIC) Study data. The plots are for three different models, with increasing numbers of established predictors included from left to right. The risk score is log odds (logit) of the predicted probability. The outcome, progressive CKD (1= yes, 0= no), is jittered in the plots to make the points easier to see. ACE/ARB, angiotensin-converting enzyme/angiotensin II receptor blocker; BMI, body mass index; CVD, cardiovascular disease; diab., diabetes; educ., education; SBP, systolic BP.

Figure 2.

Plots of progressive CKD within 2 years by risk score from the model with all established predictors. The plot in the left panel is on the basis of a risk score cutoff of −4, and the plot in the right panel uses a cutoff of zero. The vertical line separates (left panel) low- and (right panel) high-risk patients. The horizontal line separates the groups with and without progressive CKD (1= yes, 0= no). By counting the number of subjects in each of the quadrants, we obtain a standard 2×2 table. For example, using the cutoff of −4, there were 12 patients who were classified as low risk and ended up with CKD progression within 2 years. The sensitivity of this classification approach can be estimated by taking the number of true positives (294) and dividing by the total number of high-risk patients (294+12=306). Thus, sensitivity is 96%. Similar calculations can be used to find the specificity of 50%. Given the same prediction model, a different classification cutpoint could be chosen. In the plot in the right panel, in which a risk score of zero is chosen as the cutpoint, sensitivity decreases to 43%, whereas specificity increases to 98%.

Figure 3.

Receiver operating characteristics (ROC) curves for three different prediction models. A is the ROC curve when urine neutrophil gelatinase–associated lipocalin (NGAL) is the only predictor variable included in the model. The c statistic is 0.8. B and C each include two ROC curves. In B, the blue curve (area under the curve [AUC] =0.69) is from the model that includes only demographic variables. The red curve (AUC=0.82) additionally includes urine NGAL. In C, the blue curve is for the model that includes all established predictors. The red curve includes all established predictors plus urine NGAL. The c statistics in C are both 0.9. This plots show that urine NGAL is a relatively good predictor variable alone and has incremental improvement over the demographics-only model but does not help to improve prediction beyond established predictors.

Figure 4.

Calibration plot for the model that includes established predictors and urine neutrophil gelatinase–associated lipocalin. The straight line is where a perfectly calibrated model would fall. The curve is the estimated relationship between the predicted and observed values. The shaded region is ±2 SEMs.

Figure 5.

Classification of patients in the event and nonevent groups in models with and without urine neutrophil gelatinase–associated lipocalin (NGAL). The counts in red are from patients who were reclassified in the wrong direction. The left panel shows the reclassification that occurs when urine NGAL is added to a model that includes only demographics. For events, 3+8+37=48 were reclassified in the right direction, whereas 16+16+5=37 were reclassified in the wrong direction. The net reclassification improvement (NRI) for events was (48/306)−(37/306)=3.6%. The NRI for nonevents was (1153/2727)−(243/2727)=33%. Thus, there was large improvement in risk prediction for nonevents when going from the demographics-only model to demographics and NGAL model. In the right panel, urine NGAL is added to a model that includes all established predictors. Overall, there was little reclassification for both events and nonevents. The NRI for events is simply (6/306)−(8/306)=−0.65%. The NRI for nonevents is (94/2727)−(79/2727)=0.55%.