A comparison of standardised patients, record abstraction and clinical vignettes for the purpose of measuring clinical practice

Abstract

Background: A recent review found standardised patient (SP) methodology to be the gold standard method for evaluating clinical care. Alternative methods include record abstraction and computerised clinical vignettes. SPs were compared to clinical records and to computerised vignettes in order to assess whether record abstraction and vignettes are accurate measures of clinical care provided within optometry.

Methods: A total of 111 community optometrists in the south east of the UK consented to be visited by unannounced actors for an eye examination. The SPs received extensive training to enable accurate reporting of the content of the eye examinations using checklists. Clinical records were requested from optometrists who chose to receive feedback following the SP visits. The SP checklists were used as a guide to extract relevant information from the clinical records. An opportunity was made available to all UK qualified optometrists, through publicity in the College of Optometrists' monthly newsletter, the Association of Optometrists' monthly newsletter (Blink) and the UK optometry e-mail discussion list to complete three computerised clinical vignettes by performing a virtual eye examination using an on-line optometric record card. The average differences were calculated between the record abstraction and vignette results compared to the standardised patient encounter findings for different domains of an eye examination. Chi-square analyses were performed on the tests which were of the greatest clinical significance for each scenario.

Results: The average overall difference for information gathered from record abstraction compared to the standardised patient encounter ranged from +2 to -26% (positive values indicate items that were recorded on the clinical records but not reported by the SP). For history and symptoms, the average difference ranged from -9 to -26%; for the proportion of tests performed during the examination this value ranged from +2 to -24% and for management issues the difference ranged from -1 to -4%. The average overall difference for the vignette data compared to the standardised patient encounter ranged from 0 to +26% (positive values indicate items that were not carried out in a clinical setting, as recorded by the SP, but were described by optometrists who completed the vignette as tests they would have carried out). For history and symptoms, the average difference for the vignette data ranged from +2 to +26%; from 0 to +20% for tests performed during the eye examination and from 0 to +11% for management.

Conclusion: Different methods of measuring clinical care capture different elements of clinical practice and are prone to different biases. This three-way comparison indicates that clinical records tend to under-estimate actual care provided, while vignette scores tend to over-estimate clinical performance. Low participation rates mean that the participating optometrists could be described as a 'self-selected sample'. This is a limitation of the research and the conclusions should be considered with this in mind. The significance of these findings for future research and for litigation and disciplinary cases is discussed.