Referral Decision Making of General Practitioners: A Signal Detection Study

Abstract

Background: Signal detection theory (SDT) describes how respondents categorize ambiguous stimuli over repeated trials. It measures separately "discrimination" (ability to recognize a signal amid noise) and "criterion" (inclination to respond "signal" v. "noise"). This is important because respondents may produce the same accuracy rate for different reasons. We employed SDT to measure the referral decision making of general practitioners (GPs) in cases of possible lung cancer.

Methods: We constructed 44 vignettes of patients for whom lung cancer could be considered and estimated their 1-year risk. Under UK risk-based guidelines, half of the vignettes required urgent referral. We recruited 216 GPs from practices across England. Practices differed in the positive predictive value (PPV) of their urgent referrals (chance of referrals identifying cancer) and the sensitivity (chance of cancer patients being picked up via urgent referral from their practice). Participants saw the vignettes online and indicated whether they would refer each patient urgently or not. We calculated each GP's discrimination ( d ') and criterion ( c) and regressed these on practice PPV and sensitivity, as well as on GP experience and gender.

Results: Criterion was associated with practice PPV: as PPV increased, GPs' c also increased, indicating lower inclination to refer ( b = 0.06 [0.02-0.09]; P = 0.001). Female GPs were more inclined to refer than male GPs ( b = -0.20 [-0.40 to -0.001]; P = 0.049). Average discrimination was modest ( d' = 0.77), highly variable (range, -0.28 to 1.91), and not associated with practice referral performance.

Conclusions: High referral PPV at the organizational level indicates GPs' inclination to avoid false positives, not better discrimination. Rather than bluntly mandating increases in practice PPV via more referrals, it is necessary to increase discrimination by improving the evidence base for cancer referral decisions.

Keywords: cancer referral; conversion rate; detection rate; primary care; signal detection theory.

Figure 1

Hypothetical probability distributions of a decision variable, in units of standard deviation, across signal and noise trials. M_noise = 0, M_signal = 2, and d ′ = 2.

Figure 2

Screenshot of a vignette and the questions asked. All questions were compulsory, except for the last one. To avoid order effects, general practitioners (GPs) were randomly assigned to view the response categories in 1 of 2 orders: order 1 (depicted here) presented the response categories in the order of “yes,”“no” (chest X-ray question) and “refer to specialist urgently,”“refer to specialist routinely,”“not refer at this stage” (referral question). Order 2 presented the response categories in the opposite order for both questions. Order was held constant for a given GP across parts A and B.

Figure 3

Scatterplot of the hit and false alarm rates of the 216 general practitioners (GPs), with superimposed theoretical receiver operating characteristic curves produced by d ′ = 0, d ′ = 1, d ′ = 2, and d ′ = 3. The dots of the scatterplot are sized by frequency (number of GPs with the same hit and false alarm rates).

Figure 4

Distribution of the decision variable across positive and negative case presentations based on the standardized hit and false alarm rates of the 216 general practitioners. The distribution for negative cases has a mean of 0 and that for positive cases a mean of 0.77 (i.e., the separation is 0.77, which is equal to the sample’s d ′). The sample’s c is 0.50, which is the distance between the neutral point (0.77/2 = 0.39) and the criterion location (0.39 + 0.50 = 0.89).