Detection Accuracy of Collective Intelligence Assessments for Skin Cancer Diagnosis

Abstract

Importance: Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy.

Objective: To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision.

Design, setting, and participants: Evaluations were obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015.

Main outcomes and measures: For both collective intelligence rules, we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 - sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates.

Results: One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size.

Conclusions and relevance: Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer-related mortality.