Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2003 Nov-Dec;10(6):563-72.
doi: 10.1197/jamia.M1338. Epub 2003 Aug 4.

Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score

Padmanabhan Ramnarayan  1 , Ritika R Kapoor, Michael Coren, Vasantha Nanduri, Amanda L Tomlinson, Paul M Taylor, Jeremy C Wyatt, Joseph F Britto
Affiliations

Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score

Padmanabhan Ramnarayan et al. J Am Med Inform Assoc. 2003 Nov-Dec.

Abstract

Objective: Few previous studies evaluating the benefits of diagnostic decision support systems have simultaneously measured changes in diagnostic quality and clinical management prompted by use of the system. This report describes a reliable and valid scoring technique to measure the quality of clinical decision plans in an acute medical setting, where diagnostic decision support tools might prove most useful.

Design: Sets of differential diagnoses and clinical management plans generated by 71 clinicians for six simulated cases, before and after decision support from a Web-based pediatric differential diagnostic tool (ISABEL), were used.

Measurements: A composite quality score was calculated separately for each diagnostic and management plan by considering the appropriateness value of each component diagnostic or management suggestion, a weighted sum of individual suggestion ratings, relevance of the entire plan, and its comprehensiveness. The reliability and validity (face, concurrent, construct, and content) of these two final scores were examined.

Results: Two hundred fifty-two diagnostic and 350 management suggestions were included in the interrater reliability analysis. There was good agreement between raters (intraclass correlation coefficient, 0.79 for diagnoses, and 0.72 for management). No counterintuitive scores were demonstrated on visual inspection of the sets. Content validity was verified by a consultation process with pediatricians. Both scores discriminated adequately between the plans of consultants and medical students and correlated well with clinicians' subjective opinions of overall plan quality (Spearman rho 0.65, p < 0.01). The diagnostic and management scores for each episode showed moderate correlation (r = 0.51).

Conclusion: The scores described can be used as key outcome measures in a larger study to fully assess the value of diagnostic decision aids, such as the ISABEL system.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
Scale for diagnostic suggestion ratings and proposed weighting scheme.
Figure 2.
Figure 2.
Scale for management suggestion ratings and proposed weighting scheme.
Figure 3.
Figure 3.
Creation of a fictitious aggregate list from panel members' and subjects' suggestions. Diagnostic suggestions are in Title case, management suggestions are in UPPER case.
Figure 4.
Figure 4.
Example of scoring procedure for one subject's post-DDSS diagnostic and management plan.
Figure 5.
Figure 5.
Graphical depiction of entire scoring scheme. Rectangles represent tasks and measurements, ovals represent actions, and the diamond represents a simple task such as tallying. The gray shaded oval indicates a step that did not directly contribute to the procedure. Adapted with permission from: Hripcsak G, Wilcox A. Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance. J Am Med Inform Assoc. 2002;9:1–15.
Figure 6.
Figure 6.
Concurrent validity: correlation between DQS/MQS and overall subjective score.
Figure 7.
Figure 7.
Correlation between DQS and MQS.
Figure 8.
Figure 8.
Concurrent validity assessed after the omission of weighting and relevance steps.
See this image and copyright information in PMC

Comment in

References

    1. De Dombal FT, Leaper DJ, Staniland JR, McCann AP, Horrocks JC. Computer-aided diagnosis of acute abdominal pain. Br Med J. 1972;2:9–13. - PMC - PubMed
    1. Barnett GO, Cimino JJ, Hupp JA, Hoffer EP. DXplain: an evolving diagnostic decision-support system. JAMA. 1987;258:67–74. - PubMed
    1. Miller R, Masarie FE, Myers JD. Quick medical reference (QMR) for diagnostic assistance. MD Comput. 1986;3(5):34–48. - PubMed
    1. Rubeck RF. ILIAD: a medical diagnostic expert system. Teach Learn Med. 1989;1:221–2.
    1. Warner HR, Haug P, Bouhaddou O, et al. ILIAD as an expert consultant to teach differential diagnosis. In: Proc Annu Symp Comput Appl Med Care. 1988:371–6.

Publication types