Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Dec;42(12):2167-2176.
doi: 10.1007/s00296-022-05202-4. Epub 2022 Sep 10.

Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy

Markus Gräf #  1   2 , Johannes Knitza #  3   4   5 , Jan Leipe  6 , Martin Krusche  7 , Martin Welcker  8 , Sebastian Kuhn  9 , Johanna Mucke  10 , Axel J Hueber  1   11 , Johannes Hornig  12 , Philipp Klemm  13 , Stefan Kleinert  14 , Peer Aries  15 , Nicolas Vuillerme  16   17   18 , David Simon  1   2 , Arnd Kleyer  1   2 , Georg Schett  1   2 , Johanna Callhoff  19   20
Affiliations

Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy

Markus Gräf et al. Rheumatol Int. 2022 Dec.

Abstract

Symptom checkers are increasingly used to assess new symptoms and navigate the health care system. The aim of this study was to compare the accuracy of an artificial intelligence (AI)-based symptom checker (Ada) and physicians regarding the presence/absence of an inflammatory rheumatic disease (IRD). In this survey study, German-speaking physicians with prior rheumatology working experience were asked to determine IRD presence/absence and suggest diagnoses for 20 different real-world patient vignettes, which included only basic health and symptom-related medical history. IRD detection rate and suggested diagnoses of participants and Ada were compared to the gold standard, the final rheumatologists' diagnosis, reported on the discharge summary report. A total of 132 vignettes were completed by 33 physicians (mean rheumatology working experience 8.8 (SD 7.1) years). Ada's diagnostic accuracy (IRD) was significantly higher compared to physicians (70 vs 54%, p = 0.002) according to top diagnosis. Ada listed the correct diagnosis more often compared to physicians (54 vs 32%, p < 0.001) as top diagnosis as well as among the top 3 diagnoses (59 vs 42%, p < 0.001). Work experience was not related to suggesting the correct diagnosis or IRD status. Confined to basic health and symptom-related medical history, the diagnostic accuracy of physicians was lower compared to an AI-based symptom checker. These results highlight the potential of using symptom checkers early during the patient journey and importance of access to complete and sufficient patient information to establish a correct diagnosis.

Keywords: Artificial intelligence; Diagnosis; Diagnostic decision support system; Rheumatology; Symptom checker; Telemedicine.

PubMed Disclaimer

Conflict of interest statement

JK has received research support from Novartis Pharma GmbH. SK and MW are members of RheumaDatenRhePort. JK is a member of the scientific board of RheumaDatenRhePort.

Figures

Fig. 1
Fig. 1
Example of the Ada symptom assessment report excerpt presented to physicians (adapted from original report and translated to English)
Fig. 2
Fig. 2
Percentage of correctly classified IRD status by diagnosis rank, vignette difficulty and IRD status
Fig. 3
Fig. 3
Percentage of correct exact diagnoses by diagnosis rank, vignette difficulty and IRD status
Fig. 4
Fig. 4
Probabilities of diagnosis. The bars show the interquartile range. Correct and incorrect refers to the top diagnosis compared to the actual diagnosis
See this image and copyright information in PMC

References

    1. Burgers LE, Raza K, van der Helm-van Mil AH. Window of opportunity in rheumatoid arthritis - definitions and supporting evidence: from old to new perspectives. RMD Open. 2019;5:e000870. doi: 10.1136/rmdopen-2018-000870. - DOI - PMC - PubMed
    1. Benesova K, Lorenz H-M, Lion V, et al. Early recognition and screening consultation: a necessary way to improve early detection and treatment in rheumatology? : Overview of the early recognition and screening consultation models for rheumatic and musculoskeletal diseases in Germany. Z Rheumatol. 2019;78:722–742. doi: 10.1007/s00393-019-0683-y. - DOI - PubMed
    1. Sørensen J, Hetland ML. Diagnostic delay in patients with rheumatoid arthritis, psoriatic arthritis and ankylosing spondylitis: results from the Danish nationwide DANBIO registry. Ann Rheum Dis. 2015;74:e12–e12. doi: 10.1136/annrheumdis-2013-204867. - DOI - PMC - PubMed
    1. Feuchtenberger M, Nigg AP, Kraus MR, Schäfer A. Rate of proven rheumatic diseases in a large collective of referrals to an outpatient rheumatology clinic under routine conditions. Clin Med Insights Arthritis Musculoskelet Disord. 2016;9:181–187. doi: 10.4137/CMAMD.S40361. - DOI - PMC - PubMed
    1. Knitza J, Mohn J, Bergmann C, et al. Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res Ther. 2021;23:112. doi: 10.1186/s13075-021-02498-8. - DOI - PMC - PubMed