COMPASS: Computational mapping of patient-therapist alliance strategies with language modeling

Abstract

The therapeutic working alliance is a critical predictor of psychotherapy success. Traditionally, working alliance assessment relies on questionnaires completed by both therapists and patients. In this paper, we present COMPASS, a novel framework to directly infer the therapeutic working alliance from the natural language used in psychotherapy sessions. Our approach leverages advanced large language models (LLMs) to analyze session transcripts and map them to distributed representations. These representations capture the semantic similarities between the dialogues and psychometric instruments, such as the Working Alliance Inventory. Analyzing a dataset of over 950 sessions spanning diverse psychiatric conditions -- including anxiety (N = 498), depression (N = 377), schizophrenia (N = 71), and suicidal tendencies (N = 12) -- collected between 1970 and 2012, we demonstrate the effectiveness of our method in providing fine-grained mapping of patient-therapist alignment trajectories, offering interpretable insights for clinical practice, and identifying emerging patterns related to the condition being treated. By employing various deep learning-based topic modeling techniques in combination with prompting generative language models, we analyze the topical characteristics of different psychiatric conditions and how these topics evolve during each turn of the conversation. This integrated framework enhances the understanding of therapeutic interactions, enables timely feedback for therapists on the quality of therapeutic relationships, and provides clear, actionable insights to improve the effectiveness of psychotherapy.

Fig. 1. Analytical pipeline of the working alliance analysis.

The transcript is separated into turns by the therapist (T) and turns by the patients (P). These dyads of turns are compared separately by the working alliance inventories (WAI) for the clients and the therapists in the sentence embedding space, and the inferred WAI scores according to different inventory items are computed and then summarized into separate scales for Task, Bond and Goal. Topics and embeddings not biased by WAI are also computed for further analysis and interpretation through sequence modeling, a validating example of which is the diagnosis of psychiatric condition being treated only given the linguistic features of patient-doctor conversations.

Fig. 2. Flowcharts for topic modeling pipelines.

A Topics are extracted from the sessions including therapists and patients turns. To facilitate additional interpretability, we coarse-grained the topics using PCA. B Flowchart for identifying principal topics and their interpretation. The turns with largest projections on the principal topics are fed into the language modeling interpreter to gain insights. C Flowchart for identifying association between therapist topics and inferred patient working alliance. The estimation of how inferred patient working alliance is conditioned by the therapist: top therapist turns for each topic are used to select the corresponding patient turns.

Fig. 3. Working alliance scores in the patient and therapist sessions of different clinical conditions.

After standardizing the working alliance scores, we pooled the sessions into different psychiatric conditions and averaged the working alliance scores of the patients and therapists separately at each time step (i.e. dialogue turn). A The progression of the working alliance over the sessions can be observed as well as their distinctions across the clinical conditions their corresponding session belong to. B The differences between the working alliance scores of therapist and patient turns are also highlighted in boxplots, tested with T-test for the means of the two independent samples of scores (p-value notations: **** 1e–4, *** 1e–3, ** 1e–2, * 0.05, ns for “not significant”).

Fig. 4. The average 3d trajectories of different classes of psychiatric conditions in the alliance and topic space.

For each clinical condition, we averaged the time series of the therapists and patients over the sessions. We compute the patient-therapist discrepancy and their cumulative sum over time, both in terms of their inferred scores of working alliance A and topic scores B. In both the alliance space and the topic space, we mark the end points of the trajectories as a bigger dot. The coefficients of the three principal topics are shown as a heatmap in panel C.

Fig. 5. Patients’ working alliance differ when therapists chose different topics to discuss.

We compute the topic (T) and principal topic (PT) scores for all the therapist turns, and select the top 100 turns for each clinical condition and each topic; to estimate the effect of the topic on working alliance, we compute the average working alliance scores of the subsequent patients’ turns. We plot these averaged working alliance scores by the patients in a heatmap.