Implementation and evaluation of a communication coaching program: a CFIR-Informed qualitative analysis mapped onto a logic model

Abstract

Background: Coaching programs in graduate medical education have the potential to impact trainee development across multiple core competencies but require rigorous program evaluation to ensure effectiveness. We sought to qualitatively evaluate the implementation of a multi-departmental, faculty-led communication coaching program using a logic model framework.

Methods: Study participants were selected from four key stakeholder groups: resident coachees, faculty coaches, medical education leaders, and programmatic sponsors. 30-45 min semi-structured interviews were conducted via Zoom, transcribed, and de-identified for the analysis. Interviews captured stakeholders' perspectives on physicians' communication training needs, stakeholders perceived and actual roles, stakeholders' involvement in the program, factors influencing the implementation process, and strategies for programmatic improvement, sustainment, and spread. The Consolidated Framework of Implementation Research (CFIR) guided the codebook development and data analysis. A combined inductive/deductive approach was used to develop a 20-item codebook, followed by a team-based thematic analysis. A strong intercoder agreement (Cohen's kappa coefficient κ = 0.83) ensured coding consistency. The emerging themes were then mapped onto four domains of a logic model: Context, Inputs and Outputs, Outcomes, and Evaluation.

Results: 35 interviews were conducted between November 2021 and April 2022 with representation from all stakeholder groups, including 10 resident coachees (who received coaching), 10 faculty coaches (who served as coaches and underwent coaching-specific faculty development), 9 medical education leaders (who designed and implemented program), and programmatic sponsors (who provided financial support). We mapped 8 emergent themes onto the critical domains of a logic model for program evaluation. For the domain of Context, themes included (1) gap in communication education and (2) patient-centeredness. For the domain of Inputs/Outputs, themes included (1) investment in the program and (2) perceived program value. For the domain of Outcomes, themes included (1) learning-focused outcomes and (2) patient-related outcomes. For the domain of Evaluation, themes included (1) defining success and (2) challenges with evaluation.

Conclusions: Mapping CFIR-informed themes onto a logic model for program evaluation presents a novel strategy for integrating program implementation and evaluation, both of which are essential to effective educational programming. These findings can be used to guide future programmatic modifications to better meet the needs of key stakeholders.

Keywords: Communication Coaching; Graduate Medical Education (GME); Logic Model; Program Evaluation; Program Implementation; Qualitative Interview Study; The Consolidated Framework of Implementation Research.

Fig. 1

Five Domains Within the CFIR Framework . Note. Figure 1 depicts the 5 domains within the Consolidated Framework of Implementation Research (CFIR): outer setting, inner setting, process, intervention characteristics, and characteristics of individuals. A brief description of each domain within the context of the coaching program is also included

Fig. 2

Key Themes Mapped onto a Logic Model for Coaching Program Evaluation. Note. Figure 2 illustrates each of the key themes (italics) identified in the qualitative analysis, mapped onto a distinct domain of a logic model of program evaluation

Fig. 3

Framework for Success, Outcomes, and Challenges with Evaluation. Note. Figure 3 depicts the complex interplay between definitions of success, outcomes being measured, and evaluation challenges given the limitations of current outcomes metrics