Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health

Abstract

Mental illness is one of the most pressing public health issues of our time. While counseling and psychotherapy can be effective treatments, our knowledge about how to conduct successful counseling conversations has been limited due to lack of large-scale data with labeled outcomes of the conversations. In this paper, we present a large-scale, quantitative study on the discourse of text-message-based counseling conversations. We develop a set of novel computational discourse analysis methods to measure how various linguistic aspects of conversations are correlated with conversation outcomes. Applying techniques such as sequence-based conversation models, language model comparisons, message clustering, and psycholinguistics-inspired word frequency analyses, we discover actionable conversation strategies that are associated with better conversation outcomes.

Figure 1

Differences in counselor message length (in #tokens) over the course of the conversation are larger between more and less successful counselors (blue circle/red square) than between positive and negative conversations (solid/dashed). Error bars in all plots correspond to bootstrapped 95% confidence intervals using the member bootstrapping technique from Ren et al. (2010).

Figure 2

More successful counselors are more varied in their language across positive/negative conversations, suggesting they adapt more. All differences between more successful and less successful counselors except for the 0–20 bucket were found to be statistically significant (p < 0.05; bootstrap resampling test).

Figure 3

More ambiguous situations (length of situation setter) are less likely to result in positive conversations.

Figure 4

All counselors react to short, ambiguous messages by writing more (relative to the texter message) but more successful counselors do it more than less successful counselors.

Figure 5

More successful counselors use less common/templated responses (after the texter first explains the situation). This suggests that they respond in a more creative way. There is no significant difference between positive and negative conversations.

Figure 6

Our conversation model generates a particular conversation C_k by first generating a sequence of hidden states S₀, S₁,… according to a Markov model. Each state S_i then generates a message as a bag of words W_{i, 0}, W_{i, 1} … according a unigram language model W_Si.

Figure 7

Allowed state transitions for the conversation model. Counselor and texter messages are produced by distinct states and conversations must progress through the stages in increasing order.

Figure 8

More successful counselors are quicker to get to know texter and issue (stage 2) and use more of their time in the “problem solving” phase (stage 4).

Figure 9

A: Throughout the conversation there is a shift from talking about the past to future, where in positive conversations this shift is greater; B: Texters that talk more about others more often feel better after the conversation; C: More positive sentiment by the texter throughout the conversation is associated with successful conversations.

Figure 10

Prediction accuracies vs. percent of the conversation seen by the model (without texter features).