Validating Biobehavioral Technologies for Use in Clinical Psychiatry

Abstract

The last decade has witnessed the development of sophisticated biobehavioral and genetic, ambulatory, and other measures that promise unprecedented insight into psychiatric disorders. As yet, clinical sciences have struggled with implementing these objective measures and they have yet to move beyond "proof of concept." In part, this struggle reflects a traditional, and conceptually flawed, application of traditional psychometrics (i.e., reliability and validity) for evaluating them. This paper focuses on "resolution," concerning the degree to which changes in a signal can be detected and quantified, which is central to measurement evaluation in informatics, engineering, computational and biomedical sciences. We define and discuss resolution in terms of traditional reliability and validity evaluation for psychiatric measures, then highlight its importance in a study using acoustic features to predict self-injurious thoughts/behaviors (SITB). This study involved tracking natural language and self-reported symptoms in 124 psychiatric patients: (a) over 5-14 recording sessions, collected using a smart phone application, and (b) during a clinical interview. Importantly, the scope of these measures varied as a function of time (minutes, weeks) and spatial setting (i.e., smart phone vs. interview). Regarding reliability, acoustic features were temporally unstable until we specified the level of temporal/spatial resolution. Regarding validity, accuracy based on machine learning of acoustic features predicting SITB varied as a function of resolution. High accuracy was achieved (i.e., ~87%), but only when the acoustic and SITB measures were "temporally-matched" in resolution was the model generalizable to new data. Unlocking the potential of biobehavioral technologies for clinical psychiatry will require careful consideration of resolution.

Keywords: biobehavioral; clinical science; digital phenotyping; psychiatric illness; psychometrics; serious mental illness.

Figure 1

Bocci ball analogy to demonstrate reliability, validity, and the importance of resolution. (A) Red player shows relatively low reliability, and validity. Green player shows relatively high reliability and low validity. Blue player shows relatively high reliability and high validity. (B) Image has relatively low spatial resolution. (C) Image has relatively low temporal resolution. (D) Image has relatively low spectral resolution.

Figure 2

Temporal stability of acoustic features across a variety temporal and spatial resolutions. Dotted midline reflects “fair” stability, defined at 0.50. See Table 2 for definitions.