How to Evaluate Online Education for General Practitioners: Development of a Tailored Questionnaire for Heart Failure Education

Abstract

Physician-oriented online education could be a pathway to improve care for patients with heart failure, however, it is difficult to measure the impact of such education. Self-efficacy is a potential outcome measure. In this article, we develop a methodology for analyzing an educational intervention for general practitioners (GPs) using self-efficacy as a concept. This study was partly conducted within the setting of an observational study, IMPACT-B, where we developed online education for GPs. We designed and refined a 24-item questionnaire using item analysis, and exploratory and confirmatory factor analysis. Ninety-one GPs completed the questionnaire before and after the online education. Follow-up data after 6 months was available for 13 GPs. Item analysis revealed a high degree of internal consistency (coefficient alpha 0.95) and validity. Each additional year of experience was associated with an average baseline self-efficacy score of 0.50 points (95% CI [0.21-0.80]), and each additional patient in HF follow-up with an average score of 2.0 points (95% CI [0.48-3.5]). Items that differentiated most between GPs with high and low self-efficacy were the treatment of congestion as well as titrating medication and MRA in heart failure with reduced ejection fraction. Factor analysis reduced the number of questions to 14, mapping to three factors (diagnosis, treatment, and follow-up), and improved the model fit as measured by the goodness-of-fit indicator comparative-fit-index (from 0.83 to 0.91). We demonstrated a method to assess the impact of online education on general practitioners. This led to a questionnaire that was reliable, valid, and convenient to use in an implementation context.

Keywords: Heart failure; PROM; general practitioner; qualitative design; self-efficacy.

Figure 1.

Questionnaire correlation plot indicating the degree of correlation between scores on each pair of questionnaire item responses. Items are ordered according to a hierarchical clustering algorithm with four different clusters (black squares). Hues of blue indicate the degree of correlation (Pearson's correlation coefficient) between items. Asterisks indicate the degree of significance: <.001 (***), <.01 (**), <.05 (*).

Figure 2.

Association between average self-efficacy score per item and measures of clinical experience for participating general practitioners (n = 91). Each dot represents the mean self-efficacy score per item. HF, heart failure.

Figure 3.

Longitudinal overview of average scores per item per factor across 3 different time points: before- and immediately after an online training session and after 6 months follow-up.

Figure 4.

Scree plot indicating eigenvalues of an exploratory analysis with 10 hypothesized factors and corresponding eigenvalues.

Figure 5.

Bar chart of factor loadings for each item on 3 hypothesized factors.