ROCView: prototype software for data collection in jackknife alternative free-response receiver operating characteristic analysis

Abstract

ROCView has been developed as an image display and response capture (IDRC) solution to image display and consistent recording of reader responses in relation to the free-response receiver operating characteristic paradigm. A web-based solution to IDRC for observer response studies allows observations to be completed from any location, assuming that display performance and viewing conditions are consistent with the study being completed. The simplistic functionality of the software allows observations to be completed without supervision. ROCView can display images from multiple modalities, in a randomised order if required. Following registration, observers are prompted to begin their image evaluation. All data are recorded via mouse clicks, one to localise (mark) and one to score confidence (rate) using either an ordinal or continuous rating scale. Up to nine "mark-rating" pairs can be made per image. Unmarked images are given a default score of zero. Upon completion of the study, both true-positive and false-positive reports can be downloaded and adapted for analysis. ROCView has the potential to be a useful tool in the assessment of modality performance difference for a range of imaging methods.

Figure 1

The process used by readers attempting a study on ROCView.

Figure 2

The administration process used to create a study on ROCView.

Figure 3

The implication of acceptance radius. A larger acceptance radius could result in a greater number of incorrect true-positive results owing to the marking of a lesion mimic that falls within the acceptance radius. A small acceptance radius can reduce this risk and would increase the number of false-positive results owing to lesion mimics and reader inaccuracies.

Figure 4

Screenshot of ROCView. The main image (scaled) shows a reader mark that prompted the confidence scale to appear. Note the option to remove unwanted clicks. The image in the top right of the screen shows the main image at 100% size, without reader marks for re-evaluation of uncertain areas. Progress through the observation challenge is also displayed.

Figure 5

Screenshot of ROCView showing the continuous-style confidence scale variant. A mouse click prompts the slider bar to appear. The slider starts at the left side of the bar (score 1.0/low confidence) and can be moved to any point along it to a maximum of 10.0 (high confidence).