Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 May 9;2(7):681-9.
doi: 10.1016/j.ebiom.2015.05.009. eCollection 2015 Jul.

Crowdsourcing the General Public for Large Scale Molecular Pathology Studies in Cancer

Francisco J Candido Dos Reis  1 , Stuart Lynn  2 , H Raza Ali  3 , Diana Eccles  4 , Andrew Hanby  5 , Elena Provenzano  6 , Carlos Caldas  3 , William J Howat  3 , Leigh-Anne McDuffus  3 , Bin Liu  3 , Frances Daley  7 , Penny Coulson  7 , Rupesh J Vyas  8 , Leslie M Harris  8 , Joanna M Owens  8 , Amy F M Carton  8 , Janette P McQuillan  8 , Andy M Paterson  8 , Zohra Hirji  8 , Sarah K Christie  8 , Amber R Holmes  8 , Marjanka K Schmidt  9 , Montserrat Garcia-Closas  7 , Douglas F Easton  10 , Manjeet K Bolla  11 , Qin Wang  11 , Javier Benitez  12 , Roger L Milne  13 , Arto Mannermaa  14 , Fergus Couch  15 , Peter Devilee  16 , Robert A E M Tollenaar  17 , Caroline Seynaeve  18 , Angela Cox  19 , Simon S Cross  20 , Fiona M Blows  21 , Joyce Sanders  9 , Renate de Groot  9 , Jonine Figueroa  22 , Mark Sherman  22 , Maartje Hooning  18 , Hermann Brenner  23 , Bernd Holleczek  24 , Christa Stegmaier  24 , Chris Lintott  2 , Paul D P Pharoah  10
Affiliations

Crowdsourcing the General Public for Large Scale Molecular Pathology Studies in Cancer

Francisco J Candido Dos Reis et al. EBioMedicine. .

Abstract

Background: Citizen science, scientific research conducted by non-specialists, has the potential to facilitate biomedical research using available large-scale data, however validating the results is challenging. The Cell Slider is a citizen science project that intends to share images from tumors with the general public, enabling them to score tumor markers independently through an internet-based interface.

Methods: From October 2012 to June 2014, 98,293 Citizen Scientists accessed the Cell Slider web page and scored 180,172 sub-images derived from images of 12,326 tissue microarray cores labeled for estrogen receptor (ER). We evaluated the accuracy of Citizen Scientist's ER classification, and the association between ER status and prognosis by comparing their test performance against trained pathologists.

Findings: The area under ROC curve was 0.95 (95% CI 0.94 to 0.96) for cancer cell identification and 0.97 (95% CI 0.96 to 0.97) for ER status. ER positive tumors scored by Citizen Scientists were associated with survival in a similar way to that scored by trained pathologists. Survival probability at 15 years were 0.78 (95% CI 0.76 to 0.80) for ER-positive and 0.72 (95% CI 0.68 to 0.77) for ER-negative tumors based on Citizen Scientists classification. Based on pathologist classification, survival probability was 0.79 (95% CI 0.77 to 0.81) for ER-positive and 0.71 (95% CI 0.67 to 0.74) for ER-negative tumors. The hazard ratio for death was 0.26 (95% CI 0.18 to 0.37) at diagnosis and became greater than one after 6.5 years of follow-up for ER scored by Citizen Scientists, and 0.24 (95% CI 0.18 to 0.33) at diagnosis increasing thereafter to one after 6.7 (95% CI 4.1 to 10.9) years of follow-up for ER scored by pathologists.

Interpretation: Crowdsourcing of the general public to classify cancer pathology data for research is viable, engages the public and provides accurate ER data. Crowdsourced classification of research data may offer a valid solution to problems of throughput requiring human input.

Keywords: Breast cancer; Citizen science; Crowd science; Crowdsourcing.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Flowchart of study design showing details of score calculation (sub-image, image and tumor), comparison with pathologist evaluation and survival analysis.
Fig. 2
Fig. 2
Engagement in Cell Slider by Citizen Scientists. (A) Number of classifications by day since project launch; (B) histogram of number of classifications done by each Citizen Scientist.
Fig. 3
Fig. 3
Histogram of user performance final scores.
Fig. 4
Fig. 4
Accuracy of Citizen Scientist classifications. (A) Receiver operator characteristic curve for identification of cancer cells by Citizen Scientists in 3038 tumors from the SEARCH study compared to classification by a pathologist. (B) Frequency weighted scatterplot of Citizen Scientist pseudo-Allred score against the pathologist assigned Allred score for 2121 tumors from the SEARCH study. (C) Receiver operator characteristic curve for classification of ER status based on Citizen Scientist pseudo-Allred score against the pathologist classification for 2121 tumors from the SEARCH study; (D) receiver operator characteristic curve for classification of ER status based on Citizen Scientist pseudo-Allred score against the ER status recorded in the Breast Cancer Association Consortium data base from a variety of sources for 10,679 tumors from ten studies.
Fig. 5
Fig. 5
Kaplan–Meier estimates of cumulative survival of 4947 patients. (A) ER status classified by Citizen Scientists. (B) ER status as recorded in BCAC database.
See this image and copyright information in PMC

Comment in

References

    1. Ali A.M., Dawson S.J., Blows F.M., Provenzano E., Ellis I.O., Baglietto L., Huntsman D., Caldas C., Pharoah P.D. Comparison of methods for handling missing data on immunohistochemical markers in survival analysis of breast cancer. Br. J. Cancer. 2011;104(4):693–699. - PMC - PubMed
    1. Ali H.R., Irwin M., Morris L., Dawson S.J., Blows F.M., Provenzano E., Mahler-Araujo B., Pharoah P.D., Walton N.A., Brenton J.D., Caldas C. Astronomical algorithms for automated analysis of tissue protein expression in breast cancer. Br. J. Cancer. 2013;108(3):602–612. - PMC - PubMed
    1. Ali H.R., Provenzano E., Dawson S.J., Blows F.M., Liu B., Shah M., Earl H.M., Poole C.J., Hiller L., Dunn J.A., Bowden S.J., Twelves C., Bartlett J.M., Mahmoud S.M., Rakha E., Ellis I.O., Liu S., Gao D., Nielsen T.O., Pharoah P.D., Caldas C. Association between CD8 + T-cell infiltration and breast cancer survival in 12,439 patients. Ann. Oncol. 2014;25(8):1536–1543. - PubMed
    1. Badve S.S., Baehner F.L., Gray R.P., Childs B.H., Maddala T., Liu M.L., Rowley S.C., Shak S., Perez E.A., Perez E.D., Shulman L.J., Martino S., Davidson N.E., Sledge G.W., Goldstein L.J., Sparano J.A. Estrogen- and progesterone-receptor status in ECOG 2197: comparison of immunohistochemistry by local and central laboratories and quantitative reverse transcription polymerase chain reaction by central laboratory. J. Clin. Oncol. 2008;26(15):2473–2481. - PubMed
    1. Blows F.M., Driver K.E., Schmidt M.K., Broeks A., van Leeuwen F.E., Wesseling J., Cheang M.C., Gelmon K., Nielsen T.O., Blomqvist C., Heikkilä P., Heikkinen T., Nevanlinna H., Akslen L.A., Bégin L.R., Foulkes W.D., Couch F.J., Wang X., Cafourek V., Olson J.E., Baglietto L., Giles G.G., Severi G., McLean C.A., Southey M.C., Rakha E., Green A.R., Ellis I.O., Sherman M.E., Lissowska J., Anderson W.F., Cox A., Cross S.S., Reed M.W., Provenzano E., Dawson S.J., Dunning A.M., Humphreys M., Easton D.F., García-Closas M., Caldas C., Pharoah P.D., Huntsman D. Subtyping of breast cancer by immunohistochemistry to investigate a relationship between subtype and short and long term survival: a collaborative analysis of data for 10,159 cases from 12 studies. PLoS Med. 2010;7(5):e1000279. - PMC - PubMed

Publication types

Substances