Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Apr 1;44(3):289-299.
doi: 10.1093/jpepsy/jsy113.

Developing Machine Learning Models for Behavioral Coding

April Idalski Carcone  1 , Mehedi Hasan  1 , Gwen L Alexander  2 , Ming Dong  1 , Susan Eggly  3 , Kathryn Brogan Hartlieb  4 , Sylvie Naar  5 , Karen MacDonell  1 , Alexander Kotov  1
Affiliations

Developing Machine Learning Models for Behavioral Coding

April Idalski Carcone et al. J Pediatr Psychol. .

Abstract

Objective: The goal of this research is to develop a machine learning supervised classification model to automatically code clinical encounter transcripts using a behavioral code scheme.

Methods: We first evaluated the efficacy of eight state-of-the-art machine learning classification models to recognize patient-provider communication behaviors operationalized by the motivational interviewing framework. Data were collected during the course of a single weight loss intervention session with 37 African American adolescents and their caregivers. We then tested the transferability of the model to a novel treatment context, 80 patient-provider interactions during routine human immunodeficiency virus (HIV) clinic visits.

Results: Of the eight models tested, the support vector machine model demonstrated the best performance, achieving a .680 F1-score (a function of model precision and recall) in adolescent and .639 in caregiver sessions. Adding semantic and contextual features improved accuracy with 75.1% of utterances in adolescent and 73.8% in caregiver sessions correctly coded. With no modification, the model correctly classified 72.0% of patient-provider utterances in HIV clinical encounters with reliability comparable to human coders (k = .639).

Conclusions: The development of a validated approach for automatic behavioral coding offers an efficient alternative to traditional, resource-intensive methods with the potential to dramatically accelerate the pace of outcomes-oriented behavioral research. The knowledge gained from computer-driven behavioral research can inform clinical practice by providing clinicians with empirically supported communication strategies to tailor their conversations with patients. Lastly, automatic behavioral coding is a critical first step toward fully automated eHealth/mHealth (electronic/mobile Health) behavioral interventions.

Keywords: machine learning; motivational interviewing; qualitative research.

PubMed Disclaimer

References

    1. Adamou M., Antoniou G., Greasidou E., Lagani V., Charonyktakis P., Tsamardinos I. (2018). Mining free-text medical notes for suicide risk assessment. Paper presented at the Proceedings of the 10th Hellenic Conference on Artificial Intelligence, Patras, Greece.
    1. Armstrong R., Symons M., Scott J., Arnott W., Copland D., McMahon K., Whitehouse A. (2018). Predicting language difficulties in middle childhood from early developmental milestones: A comparison of traditional regression and machine learning techniques. Journal of Speech, Language, and Hearing Research: JSLHR, 61, 1926–1944. - PubMed
    1. Atkins D. C., Steyvers M., Imel Z. E., Smyth P. (2014). Scaling up the evaluation of psychotherapy: Evaluating motivational interviewing fidelity via statistical text classification. Implementation Science, 9, 1–11. doi:10.1186/1748-5908-9-49 - PMC - PubMed
    1. Bakeman R., Quera V. (1997). Observing interaction: An introduction to sequential analysis (2nd edn). New York, NY: Cambridge University Press.
    1. Bakeman R., Quera V. (2011). Sequential analysis and observational methods for the behavioral sciences. New York, NY: Cambridge University Press.

Publication types