Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Randomized Controlled Trial
. 2024 Jul 23:26:e55542.
doi: 10.2196/55542.

Diagnostic Accuracy of a Mobile AI-Based Symptom Checker and a Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial

Johannes Knitza  1   2   3   4 , Koray Tascilar  3   4 , Franziska Fuchs  3   4 , Jacob Mohn  3   4 , Sebastian Kuhn  1 , Daniela Bohr  3   4 , Felix Muehlensiepen  2   5   6 , Christina Bergmann  3   4 , Hannah Labinsky  3   4 , Harriet Morf  3   4 , Elizabeth Araujo  3   4 , Matthias Englbrecht  7 , Wolfgang Vorbrüggen  7   8 , Cay-Benedict von der Decken  8   9   10 , Stefan Kleinert  8   11 , Andreas Ramming  3   4 , Jörg H W Distler  3   4   12 , Peter Bartz-Bazzanella  8   10 , Nicolas Vuillerme  2   13   14 , Georg Schett  3   4 , Martin Welcker #  8   15 , Axel Hueber #  3   4   16
Affiliations
Randomized Controlled Trial

Diagnostic Accuracy of a Mobile AI-Based Symptom Checker and a Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial

Johannes Knitza et al. J Med Internet Res. .

Abstract

Background: The diagnosis of inflammatory rheumatic diseases (IRDs) is often delayed due to unspecific symptoms and a shortage of rheumatologists. Digital diagnostic decision support systems (DDSSs) have the potential to expedite diagnosis and help patients navigate the health care system more efficiently.

Objective: The aim of this study was to assess the diagnostic accuracy of a mobile artificial intelligence (AI)-based symptom checker (Ada) and a web-based self-referral tool (Rheport) regarding IRDs.

Methods: A prospective, multicenter, open-label, crossover randomized controlled trial was conducted with patients newly presenting to 3 rheumatology centers. Participants were randomly assigned to complete a symptom assessment using either Ada or Rheport. The primary outcome was the correct identification of IRDs by the DDSSs, defined as the presence of any IRD in the list of suggested diagnoses by Ada or achieving a prespecified threshold score with Rheport. The gold standard was the diagnosis made by rheumatologists.

Results: A total of 600 patients were included, among whom 214 (35.7%) were diagnosed with an IRD. Most frequent IRD was rheumatoid arthritis with 69 (11.5%) patients. Rheport's disease suggestion and Ada's top 1 (D1) and top 5 (D5) disease suggestions demonstrated overall diagnostic accuracies of 52%, 63%, and 58%, respectively, for IRDs. Rheport showed a sensitivity of 62% and a specificity of 47% for IRDs. Ada's D1 and D5 disease suggestions showed a sensitivity of 52% and 66%, respectively, and a specificity of 68% and 54%, respectively, concerning IRDs. Ada's diagnostic accuracy regarding individual diagnoses was heterogenous, and Ada performed considerably better in identifying rheumatoid arthritis in comparison to other diagnoses (D1: 42%; D5: 64%). The Cohen κ statistic of Rheport for agreement on any rheumatic disease diagnosis with Ada D1 was 0.15 (95% CI 0.08-0.18) and with Ada D5 was 0.08 (95% CI 0.00-0.16), indicating poor agreement for the presence of any rheumatic disease between the 2 DDSSs.

Conclusions: To our knowledge, this is the largest comparative DDSS trial with actual use of DDSSs by patients. The diagnostic accuracies of both DDSSs for IRDs were not promising in this high-prevalence patient population. DDSSs may lead to a misuse of scarce health care resources. Our results underscore the need for stringent regulation and drastic improvements to ensure the safety and efficacy of DDSSs.

Trial registration: German Register of Clinical Trials DRKS00017642; https://drks.de/search/en/trial/DRKS00017642.

Keywords: artificial intelligence; decision; decision support; decision support system; diagnosis; diagnostic; diagnostic decision support system; eHealth; randomized controlled trial; resources; rheumatologists; rheumatology; support; support system; symptom assessment; symptom checker; tool.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: JK and MW have received research support from Novartis Pharma GmbH. Qinum and RheumaDatenRhePort developed and hold the rights for Rheport. WV, CBvdD, S Kleinert, PB-B, and MW are members of RheumaDatenRhePort. WV, CBvdD, and PB-B were involved in the development of Rheport. JK is a member of the scientific board of RheumaDatenRhePort. S Kuhn is a founder and shareholder of MED.digital. PB-B has received speaker honoraria from AbbVie, Boehringer Ingelheim, Chugai/Roche, Janssen-Cilag, Novartis, and Pfizer.

Figures

Figure 1
Figure 1
Patient flow diagram.
Figure 2
Figure 2
Diagnostic properties of Ada and Rheport regarding IRDs according to the order of usage. Ada D1: Ada's top diagnosis; Ada D5: Ada's top 5 suggestions; IRD: inflammatory rheumatic disease; DDSS: diagnostic decision support system.
Figure 3
Figure 3
Cumulative overall diagnostic accuracy of Ada for selected diseases according to diagnostic rank. FM: fibromyalgia; IRD: inflammatory rheumatic diseases; OA: osteoarthritis; overall: all patients with a final medical diagnosis; RA: rheumatoid arthritis; SpA: spondyloarthritis.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Simons G, Belcher J, Morton C, Kumar K, Falahee M, Mallen CD, Stack RJ, Raza K. Symptom recognition and perceived urgency of help-seeking for rheumatoid arthritis and other diseases in the general public: a mixed method approach. Arthritis Care Res (Hoboken) 2017 May;69(5):633–641. doi: 10.1002/acr.22979. https://europepmc.org/abstract/MED/27389847 - DOI - PMC - PubMed
    1. Ehrenstein B, Pongratz G, Fleck M, Hartung W. The ability of rheumatologists blinded to prior workup to diagnose rheumatoid arthritis only by clinical assessment: a cross-sectional study. Rheumatology (Oxford) 2018 Sep 01;57(9):1592–1601. doi: 10.1093/rheumatology/key127.5023843 - DOI - PubMed
    1. Sørensen J, Hetland ML, All Departments of Rheumatology in Denmark Diagnostic delay in patients with rheumatoid arthritis, psoriatic arthritis and ankylosing spondylitis: results from the Danish nationwide DANBIO registry. Ann Rheum Dis. 2015 Mar;74(3):e12. doi: 10.1136/annrheumdis-2013-204867. http://ard.bmj.com/lookup/pmidlookup?view=long&pmid=24534758 annrheumdis-2013-204867 - DOI - PMC - PubMed
    1. Burgers LE, Raza K, van der Helm-van Mil AH. Window of opportunity in rheumatoid arthritis - definitions and supporting evidence: from old to new perspectives. RMD Open. 2019;5(1):e000870. doi: 10.1136/rmdopen-2018-000870. https://rmdopen.bmj.com/lookup/pmidlookup?view=long&pmid=31168406 rmdopen-2018-000870 - DOI - PMC - PubMed
    1. Ceney A, Tolond S, Glowinski A, Marks B, Swift S, Palser T. Accuracy of online symptom checkers and the potential impact on service utilisation. PLoS One. 2021 Jul 15;16(7):e0254088. doi: 10.1371/journal.pone.0254088. https://dx.plos.org/10.1371/journal.pone.0254088 PONE-D-20-21627 - DOI - DOI - PMC - PubMed

Publication types

LinkOut - more resources