A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation

Abstract

Background: The COVID-19 pandemic has highlighted the need for robust and adaptable diagnostic tools capable of detecting the disease from diverse and continuously evolving data sources. Machine learning models, particularly convolutional neural networks, are promising in this regard. However, the dynamic nature of real-world data can lead to model drift, where the model's performance degrades over time, as the underlying data distribution changes due to evolving disease characteristics, demographic shifts, and variations in recording conditions. Addressing this challenge is crucial to maintaining the accuracy and reliability of these models in ongoing diagnostic applications.

Objective: This study aims to develop a comprehensive framework that not only monitors model drift over time but also uses adaptation mechanisms to mitigate performance fluctuations in COVID-19 detection models trained on dynamic cough audio data.

Methods: Two crowdsourced COVID-19 audio datasets, namely COVID-19 Sounds and Coswara, were used for development and evaluation purposes. Each dataset was divided into 2 distinct periods, namely the development period and postdevelopment period. A baseline convolutional neural network model was initially trained and evaluated using data (ie, coughs from COVID-19 Sounds and shallow coughs from Coswara dataset) from the development period. To detect changes in data distributions and the model's performance between these periods, the maximum mean discrepancy distance was used. Upon detecting significant drift, a retraining procedure was triggered to update the baseline model. The study explored 2 model adaptation approaches, unsupervised domain adaptation and active learning, both of which were comparatively assessed.

Results: The baseline model achieved an area under the receiver operating characteristic curve of 69.13% and a balanced accuracy of 63.38% on the development test set of the COVID-19 Sounds dataset, while for the Coswara dataset, the corresponding values were 66.8% and 61.64%. A decline in performance was observed when the model was evaluated on data from the postdevelopment period, indicating the presence of model drift. The application of the unsupervised domain adaptation approach led to performance improvement in terms of balanced accuracy by up to 22% and 24% for the COVID-19 Sounds and Coswara datasets, respectively. The active learning approach yielded even greater improvement, corresponding to a balanced accuracy increase of up to 30% and 60% for the 2 datasets, respectively.

Conclusions: The proposed framework successfully addresses the challenge of model drift in COVID-19 detection by enabling continuous adaptation to evolving data distributions. This approach ensures sustained model performance over time, contributing to the development of robust and adaptable diagnostic tools for COVID-19 and potentially other infectious diseases.

Keywords: COVID-19 detection; active learning; data distribution shift; machine learning; maximum mean discrepancy; model degradation; unsupervised domain adaptation.

Figure 1

Overview of the proposed framework. The data are split into a labeled development set for training the baseline model and an unlabeled postdevelopment set for evaluation. The framework includes three modules: (1) a baseline model trained for binary classification, (2) a drift detection mechanism that monitors model performance in postdevelopment data, and (3) an adaptation module that retrains the model using unsupervised domain adaptation (UDA) or active learning (AL) when drift is detected. CUSUM: cumulative sum; MMD: maximum mean discrepancy.

Figure 2

COVID-19 Sounds (A) and Coswara (B) data streams over time. A 70:30 partition of the data into development and postdevelopment sets is applied, marked by the red line. The development set was divided into training, validation, and test subsets (60:20:20). Care was taken to avoid participant overlap across all subsets and between development and postdevelopment periods.

Figure 3

Baseline model architecture.

Figure 4

Proposed drift detection mechanism. Data are processed in chronological order based on their acquisition time point (t).

Figure 5

Unsupervised domain adaptation (UDA) process. The model was fed with a batch of postdevelopment data samples, along with a batch of samples from the development set and was then trained jointly (1) to correctly classify the labeled development data and (2) to minimize the maximum mean discrepancy distance between the embeddings of the development and postdevelopment batches.

Figure 6

Performance evaluation of the baseline model on the development and postdevelopment data from the COVID-19 Sounds and Coswara datasets. The model’s performance was assessed using the area under the receiver operating characteristic curve (AUC-ROC), accuracy, balanced accuracy, sensitivity, specificity, and F1-score. The test subset from development data is referenced as Dev, while Post refers to the entire postdevelopment period.

Figure 7

The obtained balanced accuracy score across the data batches of the entire postdevelopment period of the COVID-19 Sounds dataset using unsupervised domain adaptation. The orange dashed line is used to indicate the performance on the test subset of the development period (benchmark). Vertical red and blue dotted lines indicate the start and end of each alert period.

Figure 8

Balanced accuracy score through the entire postdevelopment period on Coswara dataset using unsupervised domain adaptation. The orange dashed line is used to indicate the performance on the test subset of the development period. Vertical red and blue dotted lines indicate the start and end of each alert period.

Figure 9

Balanced accuracy score through the entire postdevelopment period on the COVID-19 Sounds dataset using active learning. The orange dashed line indicates the performance on the test subset of the development period. Vertical red and blue dotted lines indicate the start and end of each alert period.

Figure 10

Balanced accuracy score through the entire postdevelopment period on the Coswara dataset using active learning. The orange dashed line indicates the performance on the test subset of the development period. Vertical red and blue dotted lines indicate the start and the end of each alert period.

Figure 11

Descriptive statistics for COVID-19 Sounds development and postdevelopment data reveal moderate changes. The disease exhibited moderate shifts in both its prevalence and the frequency of related symptoms. The 2 data subsets shared similar characteristics in terms of age, gender, and medical history of individuals. COPD: chronic obstructive pulmonary disease.

Figure 12

Descriptive statistics for Coswara development and postdevelopment data reveal profound differences in demographic characteristics, symptoms, and preexisting medical conditions between the development and postdevelopment periods. The representation of positive and negative classes in the development data is reversed in the postdevelopment data.

Figure 13

Box plots of the balanced accuracy scores across the entire postdevelopment period using the baseline model, the unsupervised domain adaptation (UDA) approach, the active learning (AL) approach, and the random sampling approach for the COVID-19 Sounds (A) and Coswara (B) datasets. The orange dashed line indicates the performance of the baseline model on the test subset of the development period (benchmark).