Improvement of an interobserver agreement of ARDS diagnosis by adding additional imaging and a confidence scale

Abstract

Acute respiratory distress syndrome (ARDS) often is not recognized in clinical practice, largely due to variation in the interpretation of chest x-ray (CXR) leading to poor interobserver reliability. We hypothesized that the agreement in the interpretation of chest imaging for the diagnosis of ARDS in invasively ventilated intensive care unit patients between experts improves when using an 8-grade confidence scale compared to using a dichotomous assessment and that the agreement increases after adding chest computed tomography (CT) or lung ultrasound (LUS) to CXR. Three experts scored ARDS according to the Berlin definition based on case records from an observational cohort study using a dichotomous assessment and an 8-grade confidence scale. The intraclass correlation (ICC), imaging modality, and the scoring method were calculated per day and compared using bootstrapping. A consensus judgement on the presence of ARDS was based on the combined confidence grades of the experts, followed by a consensus meeting for conflicting scores. In total, 401 patients were included in the analysis. The best ICC was found using an 8-grade confidence scale for LUS (ICC: 0.49; 95%-CI: 0.29-0.63) and CT evaluation (ICC: 0.49; 95%-CI: 0.34-0.61). The ICC of CXR increased by 0.022 and of CT by 0.065 when 8-grade scoring was used instead of the dichotomous assessment. Adding information from LUS or chest CT increased the ICC by 0.25 when using the 8-grade confidence assessment. An agreement on the diagnosis of ARDS can increase substantially by adapting the scoring system from a dichotomous assessment to an 8-grade confidence scale and by adding additional imaging modalities such as LUS or chest CT. This suggests that a simple assessment of the diagnosis of ARDS with a chart review by one assessor is insufficient to define ARDS in future studies.

Clinical trial registration: Trialregister.nl (identifier NL8226).

Keywords: ARDS; CT; chest X-ray; confidence; diagnosis; imaging.

Figure 1

Overview of ARDS scoring and classification, divided into three parts. (A) The 8-grade confidence scale on which the expert scored a certain grade for each available image. The upper row displays the grades the expert scored, where the middle row indicates how the confidence of this grade should be interpreted. The row below indicates the corresponding diagnosis. (B) Schedule for indexing final classification of ARDS diagnosis. Based on the three grades of the expert panel (three times a value between 1 and 8), a subject was classified in one of the three outcomes: “No ARDS,” “Uncertain diagnosis,” or “ARDS.” The option bars above the outcome indicate the grades given by the experts. The figure shows possible combinations between the three raters in the vertical direction. The boxes show the values that were allowed to come to a score of ARDS or no ARDS. When there was uncertainty between the raters or highly confident conflicting scores, an uncertain classification was given. (C) Order of assessing the available grades. For the first CT, if a definitive outcome was available, this was regarded as the final decision. If no was CT available or there was no definitive outcome, the grades of CXR1 were assessed. If this did not give a definitive outcome, the second CXR grades were assessed. Also, if no definitive outcome was available, one should proceed in the direction of the arrow, ending up in the “consensus meeting.” Importantly, this was not the order in which the images were scored by the experts. ARDS, acute respiratory distress syndrome; CXR, chest x-ray.

Figure 2

Confidence for several imaging modalities. (A) Shift in scoring on the 8-grade confidence scale for the situation CXR1 to CT. (B) Shift in scoring on the 8-grade confidence scale for the situation CXR1 to LUS1. The confidence on the 8-grade scale corresponds to the confidence of the ARDS diagnosis, ranging from high confidence no ARDS (grade 1) to high confidence ARDS on both extremes (grade 8), see Figure 1A.

Figure 3

Agreement on scoring ARDS for all imaging modalities. ICCs with a 95% confidence interval displaying the agreement for dichotomous scoring of bilateral opacities to diagnose ARDS and for scoring of confidence for ARDS diagnosis on an 8-grade scale. The confidence on the 8-grade scale corresponds to the confidence of the ARDS diagnosis, ranging from high confidence no ARDS (grade 1) to high confidence ARDS on both extremes (grade 8), see Figure 1A. CXR, chest x-ray; ICC, intra class correlation coefficient; LUS, lung ultrasound.

Figure 4

Relationship between cumulative counts per confidence grade and ARDS category in the diagnosis of ARDS. On the x-axis, the confidence score on the 8-grade confidence scale. The confidence on the 8-grade scale corresponds to the confidence of the ARDS diagnosis, ranging from high confidence no ARDS (grade 1) to high confidence ARDS on both extremes (grade 8), see indicator below graph. Cumulative review counts: CXR1: 1,025, LUS1: 1,104, CT: 643. ARDS, acute respiratory distress syndrome; CXR, chest x-ray; LUS, lung ultrasound.

Figure 5

Face validity of ARDS for the expert panel, stratified for researchers and clinical diagnoses. On the left, patient characteristics as recognized by the research team. The research team recognized 66% of the patients classified as ARDS by the expert panel. PaO₂/FiO₂ 108 [77, 143] in the ARDS group and 167 [109, 223] the in no ARDS group and PEEP of 10 (8, 12) in the ARDS group and 8 (6, 10) in the no ARDS group [numbers represent median (IQR)]. On the right, patient characteristics as recognized by the clinical team. The clinical team recognized 86% of the patients classified as ARDS by the expert panel. PaO₂/FiO₂ 85 [68, 100] in the ARDS group and 144 [98, 201] in the no ARDS group and PEEP of 12 (8, 13) in the ARDS group and 8 (7, 10) in the no ARDS group [numbers represent median (IQR)]. ARDS, acute respiratory distress syndrome; FiO₂, fraction of inspired oxygen; PaO₂, partial pressure of oxygen; PEEP, positive end-expiratory pressure.