Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jul 15;16(7):e0254088.
doi: 10.1371/journal.pone.0254088. eCollection 2021.

Accuracy of online symptom checkers and the potential impact on service utilisation

Adam Ceney  1 , Stephanie Tolond  1 , Andrzej Glowinski  1 , Ben Marks  1 , Simon Swift  1   2 , Tom Palser  1   3   4
Affiliations

Accuracy of online symptom checkers and the potential impact on service utilisation

Adam Ceney et al. PLoS One. .

Abstract

Objectives: The aims of our study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems.

Setting: Publicly available symptom checkers for patient use.

Participants: Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes were developed and systematically run through each system by a non-clinical researcher.

Primary and secondary outcome measures: System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette.

Results: Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). Safety of disposition advice decreased with condition urgency (being 71.8% for emergency cases vs 87.3% for non-urgent cases). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems.

Conclusions: There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.

PubMed Disclaimer

Conflict of interest statement

All authors have completed the Unified Competing Interest form at http://www.icmje.org/disclosure-of-interest/. (available on request from the corresponding author) and declare: Doctorlink (funder) engaged Methods Analytics (a commercial entity) to undertake this research but were not involved in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. Employees of Doctorlink had sight of draft manuscripts but all decisions were made independently by the authorship team. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Figures

Fig 1
Fig 1. Correlation of mean number of questions to diagnostic accuracy listed first.
Fig 2
Fig 2. Correlation of mean number of questions to diagnostic accuracy listed in the top five.
Fig 3
Fig 3. Correlation of mean number of questions to triage accuracy.
Fig 4
Fig 4. Correlation of mean number of questions to triage safety.
Fig 5
Fig 5. Correlation of triage safety and accuracy of disposition.
Fig 6
Fig 6. Correlation of additional resource units for all vignettes and self-care specific.
Fig 7
Fig 7. Correlation of additional resource units for self-care specific vignettes.
See this image and copyright information in PMC

References

    1. Global Health Workforce Alliance and World Health Organization, A Universal Truth: No Health Without a Workforce Third Global Forum on Human Resources for Health Report, 2013. Available from: https://www.who.int/workforcealliance/knowledge/resources/GHWA-a_univers...
    1. Dall TM, West T, Chakrabarti R, Iacobucci W, Brunec P. The complexities of physician supply and demand: Projections from 2013 to 2025 final report association of American medical colleges. Association of American Medical Colleges; 2015. doi: 10.13140/RG.2.2.13111.57764 - DOI
    1. Aggarwal M, Hutchison B.G (2013). Toward a primary care strategy for Canada. Ottawa, Ont, Canadian Foundation for Healthcare Improvement [Internet]. Deslibris.ca. [cited 2020 Apr 8] Available from: https://www.deslibris.ca/ID/235686.
    1. Beech J, Bottery S, Charlesworth A, Evans H, Gershlick B, Hemmings N, et al.. Closing the Gap: Key Areas for Action on the Health and Care Workforce. London: The Health Foundation, The King’s Fund & Nuffield Trust, 2019. [Internet] KingsFund [cited 2020 Apr 12]. Available from: https://www.kingsfund.org.uk/publications/closing-gap-health-care-workforce
    1. Coster JE, Turner JK, Bradbury D, Cantrell A. Why Do People Choose Emergency and Urgent Care Services? A Rapid Review Utilizing a Systematic Literature Search and Narrative Synthesis. Acad Emerg Med. 2017;24(9):1137–1149. doi: 10.1111/acem.13220 - DOI - PMC - PubMed

MeSH terms