A large dataset of annotated incident reports on medication errors

Abstract

Incident reports of medication errors are valuable learning resources for improving patient safety. However, pertinent information is often contained within unstructured free text, which prevents automated analysis and limits the usefulness of these data. Natural language processing can structure this free text automatically and retrieve relevant past incidents and learning materials, but to be able to do so requires a large, fully annotated and validated corpus of incident reports. We present a corpus of 58,658 machine-annotated incident reports of medication errors that can be used to advance the development of information extraction models and subsequent incident learning. We report the best F1-scores for the annotated dataset: 0.97 and 0.76 for named entity recognition and intention/factuality analysis, respectively, for the cross-validation exercise. Our dataset contains 478,175 named entities and differentiates between incident types by recognising discrepancies between what was intended and what actually occurred. We explain our annotation workflow and technical validation and provide access to the validation datasets and machine annotator for labelling future incident reports of medication errors.

Fig. 1

Data creation workflow. (a) Building the basic dataset, (b) building the machine annotator, which labels named entities, intention/factuality and incident types, and (c) technical validation via internal validation, external validation and error analysis.

Fig. 2

Distribution of incident reports of medication errors (58,658). Y-axis: the number of reports. X-axis: the number of words.

Fig. 3

Example of annotation of free text from an incident report. The legend indicates how to distinguish between types of named entity, the results of intention/factuality analysis and relation status. The type of incident inferred by the model is shown on the right.

Fig. 4

Example of annotation of free text from an incident report. The legend indicates how to distinguish between types of named entity, the results of intention/factuality analysis and relation status. The type of incident inferred by the model is shown on the right.

Fig. 5

The development of the multi-task BERT machine annotator. (a) Pre-training phase, (b) fine-tuning phase 1 and (c) fine-tuning phase 2.

Fig. 6

An overview of an annotated incident report.

Fig. 7

Technical validation summary. Reported F-1 scores for the cross validation, internal validation and external validation exercises.