Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jan 6;21(1):21.
doi: 10.1186/s12909-020-02448-z.

Competency assessment of an undergraduate program using a third-party, objective pre-post examination

Lawrence Fulton  1 , Cristian Lieneck  2 , Zo Ramamonjiarivelo  2 , Clemens Scott Kruse  2 , Matthew S Brooks  2
Affiliations

Competency assessment of an undergraduate program using a third-party, objective pre-post examination

Lawrence Fulton et al. BMC Med Educ. .

Abstract

Background: Assessing competencies or program learning outcomes in educational programs is often a leadership challenge. This case study reports medical education program's efforts to document undergraduate competency attainment using a pre-post, third-party, objective testing service that allows for inter-university comparison, a testing service that is being adopted by some certification and accrediting bodies.

Methods: Students completed a pre-test after program acceptance and a post-test at the end of the last didactic semester (1.5 years later) just prior to their required internships. Scores and subscores were evaluated using t-tests (Holm-adjusted p-values). MANOVA models of sub-competency difference scores were also evaluated.

Results: Results indicate competency improvement for each of the 12 areas based on the n = 55 student sample, (p < .001 for all scores). These improvements were independent of ethnicity, age, gender, and grades. The average student improved by 12.85 points (95% CI of 10.52 to 15.18) with the largest improvements in strategic planning and leadership competency areas (21.30 and 18.33 percentage points, respectively).

Conclusions: The third-party pre-post has some face validity given that student performance improved after completing a related curriculum as would be expected. Congruent with earlier studies, we find that repeated testing helps document competency attainment and that a single method for assessment is insufficient. We further document limitations of this 3d-party exam.

Keywords: Competency assessment; Competency-based medical education (CBME); Peregrine testing; Pre-post testing; Quasi-experimental.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Fig. 1
Fig. 1
Distribution of students by race and gender
Fig. 2
Fig. 2
Distribution of age by gender (top) and by ethnicity / race (bottom)
Fig. 3
Fig. 3
95% confidence intervals for GPA based on gender and race
Fig. 4
Fig. 4
Scatterplot matrix with bivariate contour plots, histograms, kernel density estimates, and loess curve estimates of the bivariate scatterplot relationships
Fig. 5
Fig. 5
Raw test score improvements
Fig. 6
Fig. 6
Pre-post notched boxplots
Fig. 7
Fig. 7
Percentile difference scores
See this image and copyright information in PMC

References

    1. Frank JR, Snell L, Englander R, Holmboe ES, Collaborators I. Implementing competency-based medical education: moving forward. Med Teach. 2017;39(6):568–573. - PubMed
    1. Cellucci LW, Molinari C, Young J. Competency-based education in undergraduate programs certified by the association of university programs in health administration. J Health Adm Educ. 2018;35(2):175–185.
    1. Standish MP. Competency models in graduate healthcare management education: analysis of current practices and recommendations for getting to best practices. J Health Adm Educ. 2018;35(2):269–280.
    1. Mjåset C, Lawrence K, Lee T. Hybrid Physicians Create ‘Social Capital’for Health Care. NEJM Catalyst Innovations in Care Delivery. 2020;1(5). https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0271, Retrieved 12/19/20. - DOI
    1. Osbourne AG, Johnson RE, Hinish C, Ahmed K. The significance of an MBA degree for a podiatric physician—is it worth it? A survey study. J Am Podiatr Med Assoc. 2018;108(6):494–502. - PubMed

LinkOut - more resources