BMAT's predictive validity for medical school performance: A retrospective cohort study

Abstract

Background: Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.

Methods: This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.

Results: When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.

Discussion: Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

FIGURE 1

The standardised regression coefficients from univariable robust linear regressions investigating associations between individual BioMedical Admissions Test (BMAT) sections and assessment performance at ICSM. The bars indicate the 95% bootstrap CIs of the standardised coefficient (BMAT Section 1 = ‘thinking skills’, BMAT Section 2 = ‘scientific knowledge and applications’, BMAT Section 3a = writing content of the ‘writing task’)

FIGURE 2

The standardised regression coefficients from univariable robust linear regressions investigating associations between individual BioMedical Admissions Test (BMAT) sections and assessment performance at LKCMedicine. The bars indicate the 95% bootstrap confidence intervals of the standardised coefficient (BMAT Section 1 = ‘thinking skills’, BMAT Section 2 = ‘scientific knowledge and applications’, BMAT Section 3a = writing content of the ‘writing task’)

FIGURE 3

The standardised regression coefficients from multivariable robust linear regressions investigating how individual BioMedical Admissions Test (BMAT) sections and A‐level performance predicted assessment performance at Imperial College School of Medicine (ICSM). Only coefficients for associations between BMAT sections and assessment performance are shown, for clarity. The bars indicate the 95% bootstrap confidence intervals of the standardised coefficient. Triangles show the unadjusted coefficients from univariable analyses, for comparison. Circles show the coefficients after adjusting for A‐level performance (BMAT Section 1 = ‘thinking skills’, BMAT Section 2 = ‘scientific knowledge and applications’, BMAT Section 3a = writing content of the ‘writing task’)