Transformer-based structuring of free-text radiology report databases

Abstract

Objectives: To provide insights for on-site development of transformer-based structuring of free-text report databases by investigating different labeling and pre-training strategies.

Methods: A total of 93,368 German chest X-ray reports from 20,912 intensive care unit (ICU) patients were included. Two labeling strategies were investigated to tag six findings of the attending radiologist. First, a system based on human-defined rules was applied for annotation of all reports (termed "silver labels"). Second, 18,000 reports were manually annotated in 197 h (termed "gold labels") of which 10% were used for testing. An on-site pre-trained model (T_mlm) using masked-language modeling (MLM) was compared to a public, medically pre-trained model (T_med). Both models were fine-tuned on silver labels only, gold labels only, and first with silver and then gold labels (hybrid training) for text classification, using varying numbers (N: 500, 1000, 2000, 3500, 7000, 14,580) of gold labels. Macro-averaged F1-scores (MAF1) in percent were calculated with 95% confidence intervals (CI).

Results: T_mlm,gold (95.5 [94.5-96.3]) showed significantly higher MAF1 than T_med,silver (75.0 [73.4-76.5]) and T_mlm,silver (75.2 [73.6-76.7]), but not significantly higher MAF1 than T_med,gold (94.7 [93.6-95.6]), T_med,hybrid (94.9 [93.9-95.8]), and T_mlm,hybrid (95.2 [94.3-96.0]). When using 7000 or less gold-labeled reports, T_mlm,gold (N: 7000, 94.7 [93.5-95.7]) showed significantly higher MAF1 than T_med,gold (N: 7000, 91.5 [90.0-92.8]). With at least 2000 gold-labeled reports, utilizing silver labels did not lead to significant improvement of T_mlm,hybrid (N: 2000, 91.8 [90.4-93.2]) over T_mlm,gold (N: 2000, 91.4 [89.9-92.8]).

Conclusions: Custom pre-training of transformers and fine-tuning on manual annotations promises to be an efficient strategy to unlock report databases for data-driven medicine.

Key points: • On-site development of natural language processing methods that retrospectively unlock free-text databases of radiology clinics for data-driven medicine is of great interest. • For clinics seeking to develop methods on-site for retrospective structuring of a report database of a certain department, it remains unclear which of previously proposed strategies for labeling reports and pre-training models is the most appropriate in context of, e.g., available annotator time. • Using a custom pre-trained transformer model, along with a little annotation effort, promises to be an efficient way to retrospectively structure radiological databases, even if not millions of reports are available for pre-training.

Keywords: Deep learning; Intensive care units; Natural language processing; Radiology; Thorax.

Fig. 1

Overview of the presented study. The dataset of the presented study includes a total of 93,368 free-text chest X-ray reports of intensive care unit patients. For a subset of the dataset, human annotations were generated for the occurrence of six findings within the reports to create gold-labeled training, validation, and test datasets. Furthermore, a rule-based system was applied for silver label generation. The use of an on-site pre-trained model using masked-language modeling (T_mlm) was compared to a public, medically pre-trained model (T_med) when adapting to silver labels only, gold labels only, and to first with silver and then gold labels (hybrid). To also give insights into which pre-training and labeling strategy is most appropriate in context of available human annotation time, the models were developed using varying numbers of gold-labeled reports

Fig. 2

Model performances for different numbers of gold-labeled reports. F1-scores in % (y-axis) are displayed for the rule-based (RB) system in black, as well as for T_med,gold (blue), T_mlm,gold (orange), TFIDF_gold (green), and T_mlm,hybrid (red) using various numbers of gold-labeled reports for training (x-axis)