Idiographic Lapse Prediction With State Space Modeling: Algorithm Development and Validation Study

Abstract

Background: Many mental health conditions (eg, substance use or panic disorders) involve long-term patient assessment and treatment. Growing evidence suggests that the progression and presentation of these conditions may be highly individualized. Digital sensing and predictive modeling can augment scarce clinician resources to expand and personalize patient care. We discuss techniques to process patient data into risk predictions, for instance, the lapse risk for a patient with alcohol use disorder (AUD). Of particular interest are idiographic approaches that fit personalized models to each patient.

Objective: This study bridges 2 active research areas in mental health: risk prediction and time-series idiographic modeling. Existing work in risk prediction has focused on machine learning (ML) classifier approaches, typically trained at the population level. In contrast, psychological explanatory modeling has relied on idiographic time-series techniques. We propose state space modeling, an idiographic time-series modeling framework, as an alternative to ML classifiers for patient risk prediction.

Methods: We used a 3-month observational study of participants (N=148) in early recovery from AUD. Using once-daily ecological momentary assessment (EMA), we trained idiographic state space models (SSMs) and compared their predictive performance to logistic regression and gradient-boosted ML classifiers. Performance was evaluated using the area under the receiver operating characteristic curve (AUROC) for 3 prediction tasks: same-day lapse, lapse within 3 days, and lapse within 7 days. To mimic real-world use, we evaluated changes in AUROC when models were given access to increasing amounts of a participant's EMA data (15, 30, 45, 60, and 75 days). We used Bayesian hierarchical modeling to compare SSMs to the benchmark ML techniques, specifically analyzing posterior estimates of mean model AUROC.

Results: Posterior estimates strongly suggested that SSMs had the best mean AUROC performance in all 3 prediction tasks with ≥30 days of participant EMA data. With 15 days of data, results varied by task. Median posterior probabilities that SSMs had the best performance with ≥30 days of participant data for same-day lapse, lapse within 3 days, and lapse within 7 days were 0.997 (IQR 0.877-0.999), 0.999 (IQR 0.992-0.999), and 0.998 (IQR 0.955-0.999), respectively. With 15 days of data, these median posterior probabilities were 0.732, <0.001, and <0.001, respectively.

Conclusions: The study findings suggest that SSMs may be a compelling alternative to traditional ML approaches for risk prediction. SSMs support idiographic model fitting, even for rare outcomes, and can offer better predictive performance than existing ML approaches. Further, SSMs estimate a model for a patient's time-series behavior, making them ideal for stepping beyond risk prediction to frameworks for optimal treatment selection (eg, administered using a digital therapeutic platform). Although AUD was used as a case study, this SSM framework can be readily applied to risk prediction tasks for other mental health conditions.

Keywords: alcohol use disorder; digital health; digital therapeutics; mHealth; mental health; mobile health; personalized medicine; substance use disorder.

Figure 1

Visual representation of state space modeling for this alcohol use disorder study, following the time series for a single participant from day 1 to the end of their study period (day T). This modeling framework explains observable quantities (ie, a participant’s lapse decision and ecological momentary assessment [EMA] responses each day) in terms of hidden states (ie, the participant’s state of mind) that change over time. The observation equation explains how the participant’s hidden state produces EMA responses and lapses each day, while the transition equation explains how the hidden state evolves from one day to the next.

Figure 2

Illustration of the data availability fitting procedure used for state space models in this study. Data availability refers to the amount of the participant’s data that is available to use when training the model. As an example, this graphic considers 15 days of data availability, leading to models trained with data from days 1-15, 2-16, 3-17, and so on. The red bracketed case describes the scenario where a model is trained on ecological momentary assessment data from days 4-18 and lapse data from days 4-17. The trained model is used to make a same-day lapse prediction for day 18 and 2 window-style predictions for day 18 (ie, a lapse between days 18-21 and 18-25). This fitting procedure is repeated for each participant for different data availabilities (15, 30, 45, 60, and 75 days).

Figure 3

Histogram describing the distribution of ecological momentary assessment (EMA) adherence for the complete study cohort. We characterize adherence as the proportion of study days that each participant responded to the EMA after being prompted each morning. The median adherence was approximately 0.86.

Figure 4

Histogram describing the distribution of total lapses during the study for the study cohort. Specifically, this histogram summarizes the fraction of the study cohort that reported each number of total lapses for the complete study period. Note that nearly 45% of participants reported no lapses during the study. The median lapse count for a participant was 1.

Figure 5

Plots of posterior mean area under the receiver operating characteristic curve (AUROC) performance for 3 prediction tasks and 3 model types (state space model [SSM], logistic regression [LR], and extreme gradient boosting [XGB]). Medians are marked with dots, and 95% credible intervals (CrI) are provided as error bars. Each panel describes performance on a different prediction task (ie, same-day lapse, lapse within 3 days, and lapse within 7 days). The values reported for the 3 models at each x-value (number of days of data availability) are offset slightly by method for easier viewing of the median and CrIs. Note that there is substantial correlation between the performance of the different models in the posterior samples, making visual ranking comparisons using the CrIs incomplete. Model ranking is instead assessed by a separate summary calculation of posterior samples that accounts for this correlation.