Exploratory Data Mining Techniques (Decision Tree Models) for Examining the Impact of Internet-Based Cognitive Behavioral Therapy for Tinnitus: Machine Learning Approach

Abstract

Background: There is huge variability in the way that individuals with tinnitus respond to interventions. These experiential variations, together with a range of associated etiologies, contribute to tinnitus being a highly heterogeneous condition. Despite this heterogeneity, a "one size fits all" approach is taken when making management recommendations. Although there are various management approaches, not all are equally effective. Psychological approaches such as cognitive behavioral therapy have the most evidence base. Managing tinnitus is challenging due to the significant variations in tinnitus experiences and treatment successes. Tailored interventions based on individual tinnitus profiles may improve outcomes. Predictive models of treatment success are, however, lacking.

Objective: This study aimed to use exploratory data mining techniques (ie, decision tree models) to identify the variables associated with the treatment success of internet-based cognitive behavioral therapy (ICBT) for tinnitus.

Methods: Individuals (N=228) who underwent ICBT in 3 separate clinical trials were included in this analysis. The primary outcome variable was a reduction of 13 points in tinnitus severity, which was measured by using the Tinnitus Functional Index following the intervention. The predictor variables included demographic characteristics, tinnitus and hearing-related variables, and clinical factors (ie, anxiety, depression, insomnia, hyperacusis, hearing disability, cognitive function, and life satisfaction). Analyses were undertaken by using various exploratory machine learning algorithms to identify the most influencing variables. In total, 6 decision tree models were implemented, namely the classification and regression tree (CART), C5.0, GB, XGBoost, AdaBoost algorithm and random forest models. The Shapley additive explanations framework was applied to the two optimal decision tree models to determine relative predictor importance.

Results: Among the six decision tree models, the CART (accuracy: mean 70.7%, SD 2.4%; sensitivity: mean 74%, SD 5.5%; specificity: mean 64%, SD 3.7%; area under the receiver operating characteristic curve [AUC]: mean 0.69, SD 0.001) and gradient boosting (accuracy: mean 71.8%, SD 1.5%; sensitivity: mean 78.3%, SD 2.8%; specificity: 58.7%, SD 4.2%; AUC: mean 0.68, SD 0.02) models were found to be the best predictive models. Although the other models had acceptable accuracy (range 56.3%-66.7%) and sensitivity (range 68.6%-77.9%), they all had relatively weak specificity (range 31.1%-50%) and AUCs (range 0.52-0.62). A higher education level was the most influencing factor for ICBT outcomes. The CART decision tree model identified 3 participant groups who had at least an 85% success probability following the undertaking of ICBT.

Conclusions: Decision tree models, especially the CART and gradient boosting models, appeared to be promising in predicting ICBT outcomes. Their predictive power may be improved by using larger sample sizes and including a wider range of predictive factors in future studies.

Keywords: artificial intelligence; cognitive behavioral therapy; data mining; decision tree; digital therapeutics; internet interventions; machine learning; random forest; tinnitus.

Figure 1

Feature importance based on the mean absolute SHAP values from (A) the best gradient model and (B) from the best classification and regression tree decision tree model. These SHAP values represent the absolute change in log odds. Relatively higher importance is indicated with larger SHAP values. CFQ: Cognitive Failures Questionnaire; HHIA: Hearing Handicap Inventory for Adults; SHAP: Shapley additive explanations; SWLS: Satisfaction with Life Scales; TFI: Tinnitus Functional Index.

Figure 2

The best gradient boosting model–based feature effects. Each graph represents a feature and their corresponding SHAP values. Plots labeled with “1” illustrate the impact that each feature has on achieving a successful treatment outcome (a 13-point or more reduction in TFI score). CFQ: Cognitive Failures Questionnaire; HHIA: Hearing Handicap Inventory for Adults; SHAP: Shapley additive explanations; SWLS: Satisfaction with Life Scales; TFI: Tinnitus Functional Index.

Figure 3

The best classification and regression tree decision tree model feature effects. Each graph represents a feature vs. corresponding SHAP value. Plots with “1” represent the effect that each feature has on achieving a successful treatment outcome. CFQ: Cognitive Failures Questionnaire; HHIA: Hearing Handicap Inventory for Adults; SHAP: Shapley additive explanations; SWLS: Satisfaction with Life Scales; TFI: Tinnitus Functional Index.

Figure 4

The best classification and regression tree decision tree model. The fitted tree has 10 terminal nodes (denote the decision criteria). The variable categories (please refer to Multimedia Appendix 2 for variable category labels) corresponding to each split are given at the top of each branch. Each terminal node contains the predicted treatment class ("1" indicates the successful treatment class and "0" indicates the unsuccessful treatment class), the percentage of subjects with successful treatment outcomes, and the percentage of participants with the given characteristics in the training set.