A summary of the ComParE COVID-19 challenges

Abstract

The COVID-19 pandemic has caused massive humanitarian and economic damage. Teams of scientists from a broad range of disciplines have searched for methods to help governments and communities combat the disease. One avenue from the machine learning field which has been explored is the prospect of a digital mass test which can detect COVID-19 from infected individuals' respiratory sounds. We present a summary of the results from the INTERSPEECH 2021 Computational Paralinguistics Challenges: COVID-19 Cough, (CCS) and COVID-19 Speech, (CSS).

Keywords: COVID-19; Digital Health; computer audition; deep learning; machine learning.

Figure 1

(A) Is a cumulative plot detailing when COVID-19 positive and negative submission to both the CCS and CSS were made. (B) Details the age and sex distribution of COVID-19 positive and negative participants for the CCS and CSS Sub-Challenges.

Figure 2

Team performance on the held out test set for the COVID-19 Cough Sub-Challenge.

Figure 3

Team performance on the held out test set for the COVID-19 Speech Sub-Challenge.

Figure 4

Two-sided significance test on the COVID-19 Cough (A) and Speech (B) test sets with various levels of significance according to a two-sided $Z$-test.

Figure 5

Schematic detailing the level of agreement between teams for each test instance for the COVID-19 Cough Sub-Challenge. Each row represents a team’s submission results. The teams have been ordered by Unweighted Average Recall, from the bottom up (team Casanova et al.’s predictions represent the highest scoring submission). Each column represents all teams predictions, across the competition, for one test instance. The test instances appear in the order in which they are in the test set. (A) Details all the test instances, (B) details only the test instances which were experiencing symptoms at the time of recording, and (C) details only the test instances which were experiencing no symptoms at the time of recording.

Figure 6

Schematic detailing the level of agreement between teams for each test instance for the COVID-19 Speech Sub-Challenge. Each row represents a team’s submission results. The teams have been ordered by Unweighted Average Recall (UAR), from the bottom up (team yoshiharuyamamoto’s predictions represent the highest scoring submission). Each column represents all teams’ predictions, across the competition, for one test instance. The test instances appear in the order which they are in the test set. note: There are more test cases in the COVID-19 Speech Sub-Challenge than in the COVID-19 Cough Sub-Challenge. (A) Details all the test instances, (B) details only the test instances which were experiencing symptoms at the time of recording, and (C) details only the test instances which were experiencing no symptoms at the time of recording.

Figure A1

Team performance on the full test set (NoControl) and two curated test sets featuring only test instances where the participants either had at least one symptom (AnySymptoms) or were displaying no symptoms at all (NoSymptoms). The metric reported is recall for positive cases. 95% confidence intervals are shown, calculated via the normal approximation method. (A) Corresponds to the COVID-19 Cough Sub-Challenge, CCS, and (B) the COVID-19 Speech Sub-Challenge, CSS.

Figure A2

The performance of the fusion model of $n$-best models for the COVID-19 Cough Sub-Challenge using majority voting.

Figure A3

The performance of the fusion model of $n$-best models for the COVID-19 Speech Sub-Challenge using majority voting.

Figure A4

Schematic detailing the level of agreement as in Figure 5 with test instances with either a low sample rate (below 12 kHz) (A) or high sample rate (above 12 kHz) (B).

Figure A5

Schematic detailing the level of agreement as in Figure 6 with test instances with either a low sample rate (below 12 kHz) (A) or high sample rate (above 12 kHz) (B).

Figure A6

Team performance on two curated test sets from the COVID-19 Cough Sub-Challenge. (A) Controls for test samples with a sample rate of greater than 12 kHz and (B) controls for test samples with a sample rate of 12 kHz and below. The metric reported is recall for positive cases. 95% confidence intervals are shown, calculated via the normal approximation method.

Figure A7

Team performance on two curated test sets from the COVID-19 Speech Sub-Challenge. (A) Controls for test samples with a sample rate of greater than 12 kHz and (B) controls for test samples with a sample rate of 12 kHz and below. The metric reported is recall for positive cases. 95% confidence intervals are shown, calculated via the normal approximation method.