Using patients like my patient for clinical decision support: institution-specific probability of celiac disease diagnosis using simplified near-neighbor classification

Abstract

Background: Interpretation of a diagnostic test result requires knowing what proportion of patients with a "similar" result has the condition in question. This information is often not readily available from the medical literature, or may be based on different clinical populations that make it nonapplicable. In certain settings, where correlated screening parameters and diagnostic data are available in electronic medical records, a representation of diagnostic test performance on "patients like my patient" can be obtained.

Objective: We sought to integrate patient demographic and physician practice information using a simplified nearest neighbor algorithm. We used this method to illustrate the relationship between tTG IgA test result and duodenal biopsy for celiac disease in a local diagnostic context.

Participants: We used a data set of 1,461 paired tissue transglutaminase (tTG) IgA and definitive duodenal biopsy results from Intermountain Healthcare with data on patient age and ordering physician specialty. This was split into a discovery set of 1,000 and a validation set of 461 paired results.

Main measures: Accuracy of the local discovery data set in predicting probability of positive duodenal biopsy and confidence intervals around predicted probability in the test data compared to probabilities of positive biopsy implied from published logistic regression and from published sensitivity and specificity studies.

Key results: The near-neighbor method could estimate probability of clinical outcomes with predictive performance equivalent to other methods while adjusting probability estimates and confidence intervals to fit specific clinical situations.

Conclusions: Data from clinical encounters obtained from electronic medical records can yield prediction estimates that are tailored to the individual patient, local population, and healthcare delivery processes. Local analysis of diagnostic probability may be more clinically meaningful than probabilities inferred from published studies. This local utility may come at the expense of external validity and generalizability.

Figure 1.

Illustration of the near-neighbor method for two-dimensional analysis to predict probability of duodenal biopsy using only tTG IgA and patient age in a discovery sample of 1,000 Intermountain Healthcare patients. Natural log-transformed tTG IgA values are plotted by age. The orange box corresponds to the near-neighbor space surrounding a 17-year-old patient with a tTG IgA of 3 units. Of the 161 points inside the orange box (some of which are plotted on top of others), six represent positive biopsies, indicating a 3.9 % chance this patient will have a positive biopsy (95 % CI [1.4 % to 8.2 %]). The green box corresponds to the near-neighbor space surrounding a 58-year-old patient with a tTG IgA of 55 units. Of the 16 points in this box, ten represent positive biopsies indicating a 63 % chance (95 % CI [35 % to 85 %]) this patient will have a positive biopsy. (Note that tTG IgA values of 0 were transformed to −1, since ln(0) is undefined).

Figure 2.

Illustration of the effect of the size of the near-neighbor space on the probability of a positive biopsy as a function of tTG IgA. For large, middle, and small near-neighbor spaces, tTG IgA space length was 0.77, 0.53, and 0.26 ln(tTG IgA) units, respectively. Near-neighbor analysis using very large and very small cubes is shown as dotted black and dotted gray lines, respectively. In near-neighbor analysis, smaller cube lengths created less smooth probability estimate curves and larger cube lengths decreased the predictive power at high tTG IgA levels. This figure illustrates that box size for clinically meaningful probabilities and adequate smoothness can be determined empirically.

Figure 3.

Comparison of probability estimates of positive duodenal biopsy from near-neighbor analysis with those derived from published sensitivity and specificity and from published logistic regression. Black points—near-neighbor probability estimates for 461 individuals in the validation data set using tTG IgA and age, with near-neighbor estimates derived from the Intermountain Healthcare discovery data set. Red lines —probability estimates inferred from sensitivity and specificity reported by meta-analysis of tTG IgA studies. Blue line—probability of positive biopsy using published logistic regression. Orange vertical lines show the tTG IgA manufacturer’s cutoffs for “weak positive” and “moderate to strong positive.”

Figure 4.

ROC analysis comparing relative accuracy of near-neighbor analysis by using predicted probabilities as risk scores.

Figure 5.

Bland–Altman analyses. a Near-neighbor probabilities are compared to binary probability estimates derived from reported sensitivity and specificity. b Near-neighbor probabilities are compared to estimates based on published logistic regression. Logistic regression appears to overestimate probability of positive biopsy for most patients.