Unbiased review of digital diagnostic images in practice: informatics prototype and pilot study

Abstract

Rationale and objectives: Clinical and contextual information associated with images may influence how radiologists draw diagnostic inferences, highlighting the need to control multiple sources of bias in the methodologic design of investigations involving radiologic interpretation. In the past, manual control methods to mask review films presented in practice have been used to reduce potential interpretive bias associated with differences between viewing images for patient care and reviewing images for the purposes of research, education, and quality improvement. These manual precedents from the film era raise the question whether similar methods to reduce bias can be implemented in the modern digital environment.

Materials and methods: A prototype application, CreateAPatient, was built for masking review case presentations within one institution's production radiology information system and picture archiving and communication system. To test whether CreateAPatient could be used to mask review images presented in practice, six board-certified radiologists participated in a pilot study. During pilot testing, seven digital chest radiographs, known to contain lung nodules and associated with fictitious patient identifiers, were mixed into the routine workloads of the participating radiologists while they covered general evening call shifts. The aim was to test whether it was possible to mask the presentation of these review cases, both by probing the interpreting radiologists to report detection and by conducting a forced-choice experiment on a separate cohort of 20 radiologists and information technology professionals.

Results: None of the participating radiologists reported awareness of review activity, and forced-choice detection was less than predicted at chance, suggesting that radiologists were effectively blinded. In addition, no evidence was identified of review reports unsafely propagating beyond their intended scope or otherwise interfering with patient care, despite integration of these records within production electronic work flow systems.

Conclusions: Information technology can facilitate the design of unbiased methods involving professional review of digital diagnostic images.

Figure 1

Review masking system components and their connections. Arrows indicate directionality of information flow. PACS = Picture Archiving and Communication System, RIS = Radiology Information System, DB = database.

Figure 2

Review masking workflow. Swimlane diagram illustrating process steps for radiologist reviewer and study administrator, with time flowing from top to bottom. Interpretations for masked review cases were blocked from entering the RIS DB. PACS = Picture Archiving and Communication System, RIS = Radiology Information System. DB = database.

Figure 3

User interface for the CreateAPatient research application.