Theoretical vs. empirical discriminability: the application of ROC methods to eyewitness identification

Abstract

Receiver operating characteristic (ROC) analysis was introduced to the field of eyewitness identification 5 years ago. Since that time, it has been both influential and controversial, and the debate has raised an issue about measuring discriminability that is rarely considered. The issue concerns the distinction between empirical discriminability (measured by area under the ROC curve) vs. underlying/theoretical discriminability (measured by d' or variants of it). Under most circumstances, the two measures will agree about a difference between two conditions in terms of discriminability. However, it is possible for them to disagree, and that fact can lead to confusion about which condition actually yields higher discriminability. For example, if the two conditions have implications for real-world practice (e.g., a comparison of competing lineup formats), should a policymaker rely on the area-under-the-curve measure or the theory-based measure? Here, we illustrate the fact that a given empirical ROC yields as many underlying discriminability measures as there are theories that one is willing to take seriously. No matter which theory is correct, for practical purposes, the singular area-under-the-curve measure best identifies the diagnostically superior procedure. For that reason, area under the ROC curve informs policy in a way that underlying theoretical discriminability never can. At the same time, theoretical measures of discriminability are equally important, but for a different reason. Without an adequate theoretical understanding of the relevant task, the field will be in no position to enhance empirical discriminability.

Keywords: Discriminability; Eyewitness identification; ROC analysis; Sequential lineups; Simultaneous lineups.

Fig. 1

An illustration of two common eyewitness identification procedures. The left panel illustrates a showup in which the recognition memory test consists of a single photo – either the guilty suspect (the target) or an innocent suspect (the foil) – presented for a yes/no decision. The right panel illustrates a simultaneous lineup in which the recognition memory test consists of the presentation of a target-present array containing one guilty suspect (the target) and five fillers (foils) or a target-absent array containing one innocent suspect and five fillers (all foils). Suspect faces and filler faces from the Chicago Face database (Ma, Correll, & Wittenbrink, 2015)

Fig. 2

Equal-variance Gaussian signal detection model for a showup or a lineup. For a showup, the model operates in the same way that it does for a standard old/new recognition memory test. For a lineup, the simplest decision rule holds that a positive identification (ID) is made if the memory-strength of the strongest item in the array (considered in isolation) exceeds criterion, c₁. In that case, the confidence rating associated with the ID depends on the highest confidence criterion that is exceeded (e.g., the confidence rating is 5 if the strength of the most familiar face exceeds c₅)

Fig. 3

An equal-variance Gaussian signal detection model illustrating the placement of three different decision criteria (liberal, neutral and conservative)

Fig. 4

Hypothetical receiver operating characteristics (ROC) curve for a lineup procedure in which a 5-point confidence scale was used. The number above each point is the diagnosticity ratio for that correct and false positive identification (ID) rate pair. The region shaded in light gray represents the partial area under the ROC curve (pAUC) for the specified false ID rate range of 0 to FAR_max, which is equal to .057 in this case. The diagonal line represents chance performance (where correct ID rate = false ID rate)

Fig. 5

Hypothetical receiver operating characteristic (ROC) curves for two eyewitness identification procedures in which a 5-point confidence scale was used. The rightmost ROC point again represents the overall correct and false positive identification (ID) rates that are ordinarily used to compute the diagnosticity ratio. Note that the diagnosticity ratio for the rightmost point is higher for the sequential procedure, a result that, in the past, would have been interpreted to mean that the sequential procedure is diagnostically superior to the simultaneous procedure. The region shaded dark gray represents the partial area under the curve (pAUC) for the sequential procedure in the specified false ID rate range of 0 to FAR_max. That dark gray region plus the light gray region above it represents the pAUC for the simultaneous procedure over the same false ID rate range. The dashed line represents the line of chance performance

Fig. 6

The same receiver operating characteristic (ROC) data as in Fig. 5 except that the smooth curves generated by a theoretical (signal detection) model are drawn through the ROC data points. The dashed line represents chance performance. To generate these data, d' was set to 1.4 for the simultaneous procedure and to 1.6 for the sequential procedure. The confidence criteria for the simultaneous lineup ranged from 1.5 (the overall decision criterion, c₁) to 2.5 (the high-confidence decision criterion, c₅). The corresponding confidence criteria for the sequential lineup ranged from 2.0 to 3.0, which captures the widely held view that sequential lineups induce more conservative responding than simultaneous lineups. Finally, criterion variance (σ_C) was set to 0 for the simultaneous lineup and to 0.75 for the sequential lineup, which is why the sequential lineup, despite its higher d', yields a lower ROC than the simultaneous lineup

Fig. 7

Simulated receiver operating characteristic (ROC) data generated by a simultaneous lineup using the MAX decision rule, a sequential lineup using the “first-above-criterion” decision rule, and a showup. A showup is an old/new recognition memory task in which a single face is presented for an old/new decision. For all three procedures, d' was set to 1.4, the overall decision criterion was set to 1.7, and 100,000 simulated trials were run. The top panel shows the simulated results with criterion variability set to 0. The middle panel shows the simulated results with criterion variability set to 0.5. The bottom panel shows the simulated results with criterion variability set to 2.0 (extreme criterion variability). The confidence criteria were programmed to shift in lock step to prevent violations of monotonic order (lowest = 1 to highest = 5). The dashed line represents the line of chance performance