Implementation of a multi-level evaluation strategy: a case study on a program for international medical graduates
- PMID: 22259672
- PMCID: PMC3258549
- DOI: 10.3352/jeehp.2011.8.13
Implementation of a multi-level evaluation strategy: a case study on a program for international medical graduates
Abstract
Evaluation of educational interventions is often focused on immediate and/or short-term metrics associated with knowledge and/or skills acquisition. We developed an educational intervention to support international medical graduates working in rural Victoria. We wanted an evaluation strategy that included participants' reactions and considered transfer of learning to the workplace and retention of learning. However, with participants in distributed locations and limited program resources, this was likely to prove challenging. Elsewhere, we have reported the outcomes of this evaluation. In this educational development report, we describe our evaluation strategy as a case study, its underpinning theoretical framework, the strategy, and its benefits and challenges. The strategy sought to address issues of program structure, process, and outcomes. We used a modified version of Kirkpatrick's model as a framework to map our evaluation of participants' experiences, acquisition of knowledge and skills, and their application in the workplace. The predominant benefit was that most of the evaluation instruments allowed for personalization of the program. The baseline instruments provided a broad view of participants' expectations, needs, and current perspective on their role. Immediate evaluation instruments allowed ongoing tailoring of the program to meet learning needs. Intermediate evaluations facilitated insight on the transfer of learning. The principal challenge related to the resource intensive nature of the evaluation strategy. A dedicated program administrator was required to manage data collection. Although resource-intensive, we recommend baseline, immediate, and intermediate data collection points, with multi-source feedback being especially illuminating. We believe our experiences may be valuable to faculty involved in program evaluations.
Keywords: Educational measurement; Medical education; Medical students.
Conflict of interest statement
No potential conflict of interest relevant to this article was reported.
Similar articles
-
The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: a systematic review protocol.JBI Database System Rev Implement Rep. 2015 Jan;13(1):52-64. doi: 10.11124/jbisrir-2015-1919. JBI Database System Rev Implement Rep. 2015. PMID: 26447007
-
Feasibility and outcomes of paid undergraduate student nurse positions.Nurs Leadersh (Tor Ont). 2006 Sep;19(3):e1-14. doi: 10.12927/cjnl.2006.19032. Nurs Leadersh (Tor Ont). 2006. PMID: 19830923
-
Supporting international medical graduates in rural Australia: a mixed methods evaluation.Rural Remote Health. 2012;12:1897. Epub 2012 Jan 5. Rural Remote Health. 2012. PMID: 22233146
-
A Web-Based Dementia Education Program and its Application to an Australian Web-Based Dementia Care Competency and Training Network: Integrative Systematic Review.J Med Internet Res. 2020 Jan 22;22(1):e16808. doi: 10.2196/16808. J Med Internet Res. 2020. PMID: 32012077 Free PMC article.
-
Kirkpatrick's Evaluation of Teaching and Learning Approaches of Workplace Violence Education Programs for Undergraduate Nursing Students: A Systematic Review.J Nurs Educ. 2020 Aug 1;59(8):439-447. doi: 10.3928/01484834-20200723-04. J Nurs Educ. 2020. PMID: 32757007
Cited by
-
The evaluation of a home-based paediatric nursing service: concept and design development using the Kirkpatrick model.J Res Nurs. 2018 Sep;23(6):492-501. doi: 10.1177/1744987118786019. Epub 2018 Jul 31. J Res Nurs. 2018. PMID: 34394464 Free PMC article.
-
Multilevel Modeling of Individual and Group Level Influences on Critical Thinking and Clinical Decision-Making Skills among Registered Nurses: A Study Protocol.Healthcare (Basel). 2023 Apr 19;11(8):1169. doi: 10.3390/healthcare11081169. Healthcare (Basel). 2023. PMID: 37108003 Free PMC article.
-
A review of support for International Medical Graduates (IMGs): a scoping review.BMC Med Educ. 2025 Jul 1;25(1):924. doi: 10.1186/s12909-025-07389-z. BMC Med Educ. 2025. PMID: 40598001 Free PMC article.
-
Supporting international medical graduates' transition to their host-country: realist synthesis.Med Educ. 2016 Oct;50(10):1015-32. doi: 10.1111/medu.13071. Med Educ. 2016. PMID: 27628719 Free PMC article. Review.
-
Evaluation of a consulting training course for international development assistance for health.BMC Med Educ. 2018 Oct 11;18(1):234. doi: 10.1186/s12909-018-1339-3. BMC Med Educ. 2018. PMID: 30305087 Free PMC article.
References
-
- Calder J. Programme evaluation and quality: a comprehensive guide to setting up an evaluation system. London: Kogan Page Limited; 1995.
-
- Freeth D, Mammick M, Reeves S, Koppel I, Barr H. Effective interprofessional education: development, delivery and evaluation. Oxford: Blackwell; 2005.
-
- Garman K. Eastside, Westside. An exercise in applying document analysis techniques in educational evaluation. Research on evaluation program paper and report series (No. 78) Portland: Northwest Regional Educational Lab.; 1982.
-
- Joint Committee on Standards for Educational Evaluation. Standards for evaluation of educational programs, projects and materials. New York: McGraw Hill; 1981.
-
- Mertens D. Research and evaluation in education and psychology: integrating diversity with quantitative, qualitative, and mixed methods. Thousand Oaks: Sage Publications Inc.; 2005.
LinkOut - more resources
Full Text Sources