Inter-annotator reliability of medical events, coreferences and temporal relations in clinical narratives by annotators with varying levels of clinical expertise
- PMID: 23304416
- PMCID: PMC3540452
Inter-annotator reliability of medical events, coreferences and temporal relations in clinical narratives by annotators with varying levels of clinical expertise
Abstract
The manual annotation of clinical narratives is an important step for training and validating the performance of automated systems that utilize these clinical narratives. We build an annotation specification to capture medical events, and coreferences and temporal relations between medical events in clinical text. Unfortunately, the process of clinical data annotation is both time consuming and costly. Many annotation efforts have used physicians to annotate the data. We investigate using annotators that are current students or graduates from diverse clinical backgrounds with varying levels of clinical experience. In spite of this diversity, the annotation agreement across our team of annotators is high; the average inter-annotator kappa statistic for medical events, coreferences, temporal relations, and medical event concept unique identifiers was 0.843, 0.859, 0.833, and 0.806, respectively. We describe methods towards leveraging the annotations to support temporal reasoning with medical events.
Figures


Similar articles
-
Inter-rater agreement for the annotation of neurologic signs and symptoms in electronic health records.Front Digit Health. 2023 Jun 13;5:1075771. doi: 10.3389/fdgth.2023.1075771. eCollection 2023. Front Digit Health. 2023. PMID: 37383943 Free PMC article.
-
Is the Juice Worth the Squeeze? Costs and Benefits of Multiple Human Annotators for Clinical Text De-identification.Methods Inf Med. 2016 Aug 5;55(4):356-64. doi: 10.3414/ME15-01-0122. Epub 2016 Jul 13. Methods Inf Med. 2016. PMID: 27405787 Free PMC article.
-
A framework for enhancing spatial and temporal granularity in report-based health surveillance systems.BMC Med Inform Decis Mak. 2010 Jan 12;10:1. doi: 10.1186/1472-6947-10-1. BMC Med Inform Decis Mak. 2010. PMID: 20067612 Free PMC article.
-
RIL-Contour: a Medical Imaging Dataset Annotation Tool for and with Deep Learning.J Digit Imaging. 2019 Aug;32(4):571-581. doi: 10.1007/s10278-019-00232-0. J Digit Imaging. 2019. PMID: 31089974 Free PMC article. Review.
-
Temporal reasoning with medical data--a review with emphasis on medical natural language processing.J Biomed Inform. 2007 Apr;40(2):183-202. doi: 10.1016/j.jbi.2006.12.009. Epub 2007 Jan 11. J Biomed Inform. 2007. PMID: 17317332 Free PMC article. Review.
Cited by
-
Constructing fine-grained entity recognition corpora based on clinical records of traditional Chinese medicine.BMC Med Inform Decis Mak. 2020 Apr 6;20(1):64. doi: 10.1186/s12911-020-1079-2. BMC Med Inform Decis Mak. 2020. PMID: 32252745 Free PMC article.
-
The sitting active and prone passive lag test: a validity study in a symptomatic knee population.J Phys Ther Sci. 2023 May;35(5):312-319. doi: 10.1589/jpts.35.312. Epub 2023 May 1. J Phys Ther Sci. 2023. PMID: 37131358 Free PMC article.
-
EMR2vec: Bridging the gap between patient data and clinical trial.Comput Ind Eng. 2021 Jun;156:107236. doi: 10.1016/j.cie.2021.107236. Epub 2021 Mar 15. Comput Ind Eng. 2021. PMID: 33746344 Free PMC article.
-
Temporal data representation, normalization, extraction, and reasoning: A review from clinical domain.Comput Methods Programs Biomed. 2016 May;128:52-68. doi: 10.1016/j.cmpb.2016.02.007. Epub 2016 Feb 23. Comput Methods Programs Biomed. 2016. PMID: 27040831 Free PMC article. Review.
-
Construction of an Emotional Lexicon of Patients With Breast Cancer: Development and Sentiment Analysis.J Med Internet Res. 2023 Sep 12;25:e44897. doi: 10.2196/44897. J Med Internet Res. 2023. PMID: 37698914 Free PMC article.
References
-
- Zhou L, Melton GB, Parsons S, Hripcsak G. A temporal constraint structure for extracting temporal information from clinical narrative. J Biomed Inform. 2006 Aug;39(4):424–39. - PubMed
-
- Pustejovsky J, Castaño J, Ingria R, et al. TimeML: Robust Specification of Event and Temporal Expressions in Text. Fifth International Workshop on Computational Semantics (IWCS-5); 2003; Tilburg, The Netherlands. 2003.
-
- Pustejovsky J, Verhagen M, Sauri R, et al. TimeBank 1.2. Philadelphia: Linguistic Data Consortium; 2006.
-
- Galescu L, Blaylock N. A corpus of clinical narratives annotated with temporal information. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium; Miami, Florida, USA: ACM; 2012. pp. 715–20.
Publication types
MeSH terms
Grants and funding
- 8UL1TR000090-05/TR/NCATS NIH HHS/United States
- UL1 TR000090/TR/NCATS NIH HHS/United States
- R01 LM011116/LM/NLM NIH HHS/United States
- TL1RR025753/RR/NCRR NIH HHS/United States
- UL1RR025755/RR/NCRR NIH HHS/United States
- 8TL1TR000091-05/TR/NCATS NIH HHS/United States
- UL1 RR025755/RR/NCRR NIH HHS/United States
- 8KL2TR000112-05/TR/NCATS NIH HHS/United States
- KL2RR025754/RR/NCRR NIH HHS/United States
- KL2 RR025754/RR/NCRR NIH HHS/United States
- TL1 RR025753/RR/NCRR NIH HHS/United States
- KL2 TR000112/TR/NCATS NIH HHS/United States
- TL1 TR000091/TR/NCATS NIH HHS/United States
LinkOut - more resources
Full Text Sources