Prediction Model for the Risk of HIV Infection among MSM in China: Validation and Stability

Abstract

The impact of psychosocial factors on increasing the risk of HIV infection among men who have sex with men (MSM) has attracted increasing attention. We aimed to develop and validate an integrated prediction model, especially incorporating emerging psychosocial variables, for predicting the risk of HIV infection among MSM. We surveyed and collected sociodemographic, psychosocial, and behavioral information from 547 MSM in China. The participants were split into a training set and a testing set in a 3:1 theoretical ratio. The prediction model was constructed by introducing the important variables selected with the least absolute shrinkage and selection operator (LASSO) regression, applying multivariate logistic regression, and visually assessing the risk of HIV infection through the nomogram. Receiver operating characteristic curves (ROC), Kolmogorov-Smirnov test, calibration plots, Hosmer-Lemeshow test and population stability index (PSI) were performed to test validity and stability of the model. Four of the 15 selected variables-unprotected anal intercourse, multiple sexual partners, involuntary subordination and drug use before sex-were included in the prediction model. The results indicated that the comprehensive prediction model we developed had relatively good predictive performance and stability in identifying MSM at high-risk for HIV infection, thus providing targeted interventions for high-risk MSM.

Keywords: HIV infection; involuntary subordination; machine learning; men who have sex with men; model validation; nomogram; psychosocial factors.

Figure 1

Flow diagram of study design.

Figure 2

Predictors selection using the least absolute shrinkage and selection operator (LASSO) binary logistic regression model: (a) A coefficient profile plot was constructed against the log (lambda) parameters. Four variables with nonzero coefficients were selected by deriving the optimal lambda; (b) Selection of optimal parameter (lambda) in the LASSO model used 10-fold cross-validation error curve and was based on 1 standard error of the minimum criteria (1-SE criteria). The partial likelihood deviance (binomial deviance) curve was plotted versus log (lambda). The dotted lines were drawn at the optimal values by the minimum criteria and 1-SE criteria.

Figure 3

Developed nomogram to assess the HIV infection risk among MSM.

Figure 4

Receiver operating characteristic (ROC) validation of the HIV infection risk nomogram prediction in training set and testing set: (a) The area under the receiver operating characteristic curve (AUC) represents the discrimination performance of the model in the training set; (b) The area under the receiver operating characteristic curve (AUC) represents the discrimination performance of the model in the testing set.

Figure 5

Kolmogorov–Smirnov test curve of the HIV risk prediction model in the training set and in the testing set. The horizontal coordinate of the curve is the “threshold” (the overall sample is divided into 10 equal parts in probability order), and the vertical coordinate is the value of TPR (true positive rate) or FPR (false positive rate), ranging from 0 to 1. The red and blue solid lines indicate the HIV-positive and negative cases in the training set, respectively. The maximum vertical distance between these two curves (gray dashed line) is the KS test value, and the corresponding horizontal coordinate is the threshold value that classifies the model best. The orange and green dashed lines indicate the HIV-positive and negative cases in the testing set, respectively.

Figure 6

Calibration curves for the prediction of HIV infection risk nomogram The y-axis represented the actual risk of HIV infection. The x-axis represented the predicted risk of HIV infection. The diagonal dotted line represents a perfect prediction by an ideal model, the solid line represents the performance of the training set (a) and testing set (b), with the results indicating that a closer fit to the diagonal dotted line represents a better prediction.

Figure 7

Population distribution plot of HIV infection risk prediction model in the training and testing sets. The horizontal coordinate indicates the 10 binning interval of the general distribution, and the vertical coordinate indicates the percentage of the population. The red and dark green squares indicate the actual population distribution in each binning interval for the training and testing sets, respectively. The green solid line and the orange dashed line indicate the expected population distribution in each binning interval for the training and testing sets, respectively.