Evaluating therapist adherence in motivational interviewing by comparing performance with standardized and real patients

Abstract

Objective: The goal of measuring therapist adherence is to determine whether a therapist can perform a given treatment. Yet, the evaluation of therapist behaviors in most clinical trials is limited. Typically, randomized trials have few therapists and minimize therapist variability through training and supervision. Furthermore, therapist adherence is confounded with uncontrolled differences in patients across therapists. Consequently, the extent to which adherence measures capture differences in actual therapist adherence versus other sources of variance is unclear.

Method: We estimated intra-class correlations (ICCs) for therapist adherence in sessions with real and standardized patients (RPs and SPs), using ratings from a motivational interviewing (MI) dissemination trial (Baer et al., 2009) in which 189 therapists recorded 826 sessions with both patient types. We also examined the correlations of therapist adherence between SP and RP sessions, and the reliability of therapist level adherence scores with generalizability coefficients (GCs).

Results: ICCs for therapist adherence were generally large (average ICC for SPs = .44; average ICC for RPs = .40), meaning that a given therapist's adherence scores were quite similar across sessions. Both ICCs and GCs were larger for SP sessions compared to RPs on global measures of MI adherence, such as Empathy and MI Spirit. Correlations between therapist adherence with real and standardized patients were moderate to large on 3 of 5 adherence measures.

Conclusion: Differences in therapist-level adherence ratings were substantial, and standardized patients have promise as tools to evaluate therapist behavior.

Figure 1

Intra-class correlations (ICCs) and HPD intervals for therapists on each outcome derived from SP (black), RP sessions, (light gray), and the difference between SP and RP ICC estimates (dark grey).

Figure 2

Correlations (r) for therapist adherence estimates obtained from SPs and RPs. Correlations are derived from Bayesian mixed effects models and the error bars represent 95% HPD intervals.

Figure 3

Generalizability coefficients based on the average number of SP and RPs seen by therapists. SP (m=2.59) and RP (m=2.21)

Figure 4

Generalizability Coefficients (GCs) for each outcome as a function of the number observations per therapist (M). The horizontal dashed line indicates where the number of patients results in a GC of .80. The vertical dashed line indicates the GC with one session.