Predictive models for musculoskeletal injury risk: why statistical approach makes all the difference

Abstract

Objective: Compare performance between an injury prediction model categorising predictors and one that did not and compare a selection of predictors based on univariate significance versus assessing non-linear relationships.

Methods: Validation and replication of a previously developed injury prediction model in a cohort of 1466 service members followed for 1 year after physical performance, medical history and sociodemographic variables were collected. The original model dichotomised 11 predictors. The second model (M2) kept predictors continuous but assumed linearity and the third model (M3) conducted non-linear transformations. The fourth model (M4) chose predictors the proper way (clinical reasoning and supporting evidence). Model performance was assessed with R², calibration in the large, calibration slope and discrimination. Decision curve analyses were performed with risk thresholds from 0.25 to 0.50.

Results: 478 personnel sustained an injury. The original model demonstrated poorer R² (original:0.07; M2:0.63; M3:0.64; M4:0.08), calibration in the large (original:-0.11 (95% CI -0.22 to 0.00); M2: -0.02 (95% CI -0.17 to 0.13); M3:0.03 (95% CI -0.13 to 0.19); M4: -0.13 (95% CI -0.25 to -0.01)), calibration slope (original:0.84 (95% CI 0.61 to 1.07); M2:0.97 (95% CI 0.86 to 1.08); M3:0.90 (95% CI 0.75 to 1.05); M4: 081 (95% CI 0.59 to 1.03) and discrimination (original:0.63 (95% CI 0.60 to 0.66); M2:0.90 (95% CI 0.88 to 0.92); M3:0.90 (95% CI 0.88 to 0.92); M4: 0.63 (95% CI 0.60 to 0.66)). At 0.25 injury risk, M2 and M3 demonstrated a 0.43 net benefit improvement. At 0.50 injury risk, M2 and M3 demonstrated a 0.33 net benefit improvement compared with the original model.

Conclusion: Model performance was substantially worse in the models with dichotomised variables. This highlights the need to follow established recommendations when developing prediction models.

Keywords: Injury; Neuromuscular; Prevention; Risk factor.

Figure 1

Calibration slope of original developed model (M1). Calibration is the relationship between predicted and actual probability of the event. The calibration slope plots the predicted risk graphically against the observed outcome; displaying the calibration intercept and calibration slope. perfect calibration would result in a 45° line. Within this calibration plot, risk does not begin until 0.20. What this clinically means is that everyone’s risk cannot be lower than 0.20. Individuals with little risk of injury may be inappropriately referred for clinical care.

Figure 2

Calibration slope of original developed model replicated with continuous predictors; linearity assumed (M2). Calibration is the relationship between predicted and actual probability of the event. The calibration slope plots the predicted risk graphically against the observed outcome; displaying the calibration intercept and calibration slope. perfect calibration would result in a 45° line. The predicted risk is lower between 0.20 and 0.40, compared with the actual risk for these individuals. What this means clinically is that these individuals would be under estimated for their true risk of injury, and potentially not referred to the appropriate clinical care or injury prevention strategies.

Figure 3

Calibration slope of original developed model replicated with continuous predictors and non-linear transformations (M3). Calibration is the relationship between predicted and actual probability of the event. The calibration slope plots the predicted risk graphically against the observed outcome; displaying the calibration intercept and calibration slope. Perfect calibration would result in a 45° line. Predicted risk between 0.20 and 0.40 is slightly lower than actual risk, which may alter clinical decisions for individuals within this risk threshold.

Figure 4

Calibration slope when using best practices to choose predictors (added several new predictors), but keeping them all dichotomised (M4). Calibration is the relationship between predicted and actual probability of the event. The calibration slope plots the predicted risk graphically against the observed outcome; displaying the calibration intercept and calibration slope. Perfect calibration would result in a 45° line. Within this calibration plot, risk does not begin until 0.20. What this clinically means is that everyone’s risk cannot be lower than 0.20. Individuals with little risk of injury may be inappropriately referred for clinical care.

Figure 5

Calibration slope of optimal model developed based on appropriately chosen predictors, keeping predictors continuous and conducting appropriate non-linear transformations (M5). Calibration is the relationship between predicted and actual probability of the event. Perfect calibration would result in a 45° line. The calibration slope plots the predicted risk graphically against the observed outcome; displaying the calibration intercept and calibration slope. This calibration model demonstrates risk from 0.00 to 1.0, and has the most uniform predicted risk to the actual risk. Predicted risk between 0.20 and 0.40 is lower than actual risk, which may alter clinical decisions for individuals with the least risk of injury.

Figure 6

Decision curve for the prediction models to predict injury risk in military personnel. The figure reports the expected net benefit compared with not predicting injuries. ‘Treat all’ assumes that all military personnel are at a high risk for injury and should be intervened on, while treat none assumes that all military personnel are at a low risk for injury and NO interventions should be performed. The threshold probability was defined as the population risk of injury within military personnel of 0.25–0.50. The models keeping predictors continuous (M2, M3, M5) non-linear transformations (M3, M5) all demonstrated improved net benefit (ie, correct injury identification) compared with ‘treat all’ and the original (M1) model and the original model with further added dichotomised predictors (M4) at these threshold probabilities.

Figure 7

Example of the multivariate injury dynamic nomogram. Probability of injury is approximately 22.5% (95% CI of 10.6% to 41.8%) when a 24-year-old male has one previous injury in the military, a recovery score of 70, was on profile the previous year, five movements reported pain, a 5° asymmetry in ankle dorsiflexion, 68% limb length on the Y-Balance anterior reach, 68% limb length on the Y-Balance upper quarter superolateral reach, and 7% limb length asymmetry on the Y-Balance upper quarter inferolateral reach and completes the 2 mile run in a time in 868 s (14 min, 29 s). (More examples in online supplemental appendix).