Performance evaluation of a prescription medication image classification model: an observational cohort

Abstract

Technology assistance of pharmacist verification tasks through the use of machine intelligence has the potential to detect dangerous and costly pharmacy dispensing errors. National Drug Codes (NDC) are unique numeric identifiers of prescription drug products for the United States Food and Drug Administration. The physical form of the medication, often tablets and capsules, captures the unique features of the NDC product to help ensure patients receive the same medication product inside their prescription bottle as is found on the label from a pharmacy. We report and evaluate using an automated check to predict the shape, color, and NDC for images showing a pile of pills inside a prescription bottle. In a test set containing 65,274 images of 345 NDC classes, overall macro-average precision was 98.5%. Patterns of incorrect NDC predictions based on similar colors, shapes, and imprints of pills were identified and recommendations to improve the model are provided.

Fig. 1. Precision-recall curves for prediction of medication image components.

AUC-PR is used as a measure of how well the classifier distinguishes between classes. In our work, the higher the AUC, the better the models are at predicting the NDC, color, or shape. The NDC and the shape model achieved an AUC-PR of almost 1.00 indicating that decision thresholds could be established to minimize both false-positive and false-negative events.

Fig. 2. Confidence histogram and reliability diagram for the National Drug Code prediction model on the test dataset.

a The confidence histogram shows the percentage of samples falling into each forecasted probability bin, while (b) the reliability diagram plots the observed predicted probability accuracy against the expected probability, where the range of forecasted probabilities is divided into ten bins (i.e., 0–0.1, 0.1–0.2, 0.2–0.3, etc.). Error bars in the reliability diagram represent 95% confidence intervals for each bin and the numbers of images in each bin are 45, 17, 36, 115, 153, 285, 287, 471, 1063, 62,802, respectively.

Fig. 3. Confusion matrix normalized by proportion of all predicted and reference National Drug Code images.

Proportion equal to the predicted National Drug Code (NDC) count for given reference image divided by the count of all reference images of that NDC). 1 = the same NDC was predicted each time; 0 = no cases of reference image being a particular predicted NDC.

Fig. 4. Frequency of the medication National Drug Code incorrectly predicted by the machine sharing similar characteristics.

The left-hand columns show an incorrect National Drug Code (NDC) predicted image along with description of the prescription label. The right-hand columns show an example image of the NDC predicted by the machine.