Clinical concept annotation with contextual word embedding in active transfer learning environment

Abstract

Objective: The study aims to present an active learning approach that automatically extracts clinical concepts from unstructured data and classifies them into explicit categories such as Problem, Treatment, and Test while preserving high precision and recall and demonstrating the approach through experiments using i2b2 public datasets.

Methods: Initially labeled data are acquired from a lexical-based approach in sufficient amounts to perform an active learning process. A contextual word embedding similarity approach is adopted using BERT base variant models such as ClinicalBERT, DistilBERT, and SCIBERT to automatically classify the unlabeled clinical concept into explicit categories. Additionally, deep learning and large language model (LLM) are trained on acquiring label data through active learning.

Results: Using i2b2 datasets (426 clinical notes), the lexical-based method achieved precision, recall, and F1-scores of 76%, 70%, and 73%. SCIBERT excelled in active transfer learning, yielding precision of 70.84%, recall of 77.40%, F1-score of 73.97%, and accuracy of 69.30%, surpassing counterpart models. Among deep learning models, convolutional neural networks (CNNs) trained with embeddings (BERTBase, DistilBERT, SCIBERT, ClinicalBERT) achieved training accuracies of 92-95% and testing accuracies of 89-93%. These results were higher compared to other deep learning models. Additionally, we individually evaluated these LLMs; among them, ClinicalBERT achieved the highest performance, with a training accuracy of 98.4% and a testing accuracy of 96%, outperforming the others.

Conclusions: The proposed methodology enhances clinical concept extraction by integrating active learning and models like SCIBERT and CNN. It improves annotation efficiency while maintaining high accuracy, showcasing potential for clinical applications.

Keywords: Clinical concept extraction; active transfer learning; clinical concept annotation; contextual word embedding; information extraction; large language models.

Figure 1.

The proposed system workflow towards clinical concept classification consists of three modules: (a) label data preparation; (b) active transfer learning; (c) deep learning and large language model (LLM).

Figure 2.

Clinical concept extraction and classification: an example case study scenario workflow.

Figure 3.

Presented clinical concepts embedding similarity between candidate (unlabeled) concepts and known (label) concepts and active learning process for automatically boosting labeled concepts.

Figure 4.

Comprehensive details of datasets and their utilization.

Figure 5.

Proposed approaches evaluation and experimental setup.

Figure 6.

Threshold value identification for embedding similarity using cosine similarity approach.

Figure 7.

Upset analysis to measure the LLM performance in ensemble learning environment towards clinical concept classification.

Figure 8.

Training and validation accuracy and loss curves for CNN models trained on embeddings generated from four BERT-based variants (BERTBase, DistilBERT, ClinicalBERT, and SCIBERT). The plots demonstrate the model's performance over epochs, highlighting the efforts to mitigate overfitting and ensure robust generalization. Training accuracy and loss are represented by solid lines, while validation accuracy and loss are depicted by dashed lines for each model variant.

Figure 9.

Ablation testing results for the CNN model utilizing BERT-based embeddings (BERTBase, DistilBERT, ClinicalBERT, and SCIBERT), evaluating the impact of sequence length, regularization, dropout, and batch size on model performance.

Figure 10.

Presents performance comparison impact of two learning rates Longest Valley and Min Numerical Gradient on the training and validation metrics of BERTBase, DistilBERT, ClinicalBERT, and SCIBERT.

Figure 11.

Upset Analysis plot presented statistical analysis of individual LLMs toward clinical concept classification.

Figure A1.

Receiver operating characteristic and area under the curve (ROC_AUC) to identify optimal deep learning model towards clinical concept classification over various BERT base version embeddings.