The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools

Abstract

Background: Most UK medical schools use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the United Kingdom Clinical Aptitude Test (UKCAT), and its four sub-scales, along with measures of educational attainment, individual and contextual socio-economic background factors, as predictors of performance in the first year of medical school training.

Methods: A prospective study of 4,811 students in 12 UK medical schools taking the UKCAT from 2006 to 2008 as a part of the medical school application, for whom first year medical school examination results were available in 2008 to 2010.

Results: UKCAT scores and educational attainment measures (General Certificate of Education (GCE): A-levels, and so on; or Scottish Qualifications Authority (SQA): Scottish Highers, and so on) were significant predictors of outcome. UKCAT predicted outcome better in female students than male students, and better in mature than non-mature students. Incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by sex (male students performing less well), ethnicity (non-White students performing less well), and a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level (irrespective of public or private sector) performing less well. Multilevel modeling showed no differences between medical schools in predictive ability of the various measures. UKCAT sub-scales predicted similarly, except that Verbal Reasoning correlated positively with performance on Theory examinations, but negatively with Skills assessments.

Conclusions: This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training. UKCAT has predictive validity as a predictor of medical school outcome, particularly in mature applicants to medical school. UKCAT offers small but significant incremental validity which is operationally valuable where medical schools are making selection decisions based on incomplete measures of educational attainment. The study confirms the validity of using all the existing measures of educational attainment in full at the time of selection decision-making. Contextual measures provide little additional predictive value, except that students from high attaining secondary schools perform less well, an effect previously shown for UK universities in general.

Figure 1

OverallMark (a), TheoryMark (b) and SkillsMark (c) , and SkillsMark in relation to TheoryMark (d). The red line is the linear regression, and the green line a lowess curve.

Figure 2

Relationship of OverallMark at medical school to Educational Attainment (zEducationalAttainment). Scattergrams are shown separately for a) General Certificate of Education (GCE) qualifications (A-levels/AS-levels/GCSEs), and b) Scottish Qualifications Authority (SQA) qualifications (Scottish Highers and Advanced Highers). The red line is a linear regression, and the green line is a lowess curve. The slope of the line for SQA qualifications (b = .423) is significantly larger than that for GCE qualifications (b = .349; interaction term, t = 25.95, 3,428 df, P <.001).

Figure 3

a) Performance at medical school in relation to DfES average point score for secondary school attended. Performance of medical school entrants (vertical) is expressed as a standardised (z) score. DfES measure of average point score per examination entry (horizontal) is for the (English) secondary school which the entrant attended. Sections b) and c) show the distribution of average point scores for entrants from non-selective secondary schools (, in gray) and selective secondary schools (c, in blue). The gray and blue lines in a) show the fitted regression lines for non-selective secondary schools (gray) and selective secondary schools (blue), for candidates with AAA at A-level (top, thickest line), down through AAB and ABB to BBB (lowest, thinnest line). Average point scores are grouped into four groups, indicated by vertical dashed lines, and mean entry scores, with 95% CI, are shown for entrants from non-selective secondary schools (black squares) and selective secondary schools (blue circles), the largest squares/circles for AAA, the medium squares/circles for AAB, and the smallest squares/circles for ABB. Groups with small N and, hence, large CIs are omitted.

Figure 4

Scattergram showing relationship between OverallMark at medical school, and UKCAT score (standardised within medical schools). Mature students (green) and non-mature students (blue) are shown separately, along with fitted linear regression functions. The crossing of the two lines is at about 2.5 standard deviations below the mean, so that at almost all candidate ability levels, mature students outperform non-mature students, with a steeper slope for mature students.

Figure 5

Multilevel modeling of relationship of OverallMark at medical school to Educational Attainment (zEducationalAttainment). See text for details.

Figure 6

Multilevel modeling of relationship of OverallMark at medical school to UKCAT total score. See text for details.