Study of interobserver reliability in clinical assessment of RSV lower respiratory illness: a Pediatric Investigators Collaborative Network for Infections in Canada (PICNIC) study

Abstract

Randomized trials of ribavirin therapy have used clinical scores to assess illness severity. Little information on agreement for these findings between observers has been published. We decided to determine interobserver agreement for (1) a history for apnea or respiratory failure; (2) assessment of cyanosis, respiratory rate, retractions, and oximetry; and (3) determination of reason for hospitalization (requirement for medications, supportive care, underlying illness, poor home environment). At eight centers 137 RSV-infected patients were assessed by two observers blinded to the assessments by others with no interventions made between assessments. Observations were categorized, and agreement was summarized as percentage of observed agreement, Pearson correlation, or as a kappa statistic. Observed agreement for a history of either apnea or a respiratory arrest was at least 90% at all centers, with seven of the eight centers in total agreement. At all centers except one, the agreement on the reason why the patient remained in hospital was at least 80%. The observed agreement for assessing cyanosis was at least 94% at all eight centers. The correlation coefficient for respiratory rate varied from 0.42 to 0.97 across centers. The kappa values for agreement beyond chance for retractions varied from 0.05 to 1.00. The kappa values for oxygen saturation measures varied from 0.31 to 0.70. Although not statistically significant, there appeared to be more variation as the time between assessments increased. In conclusion, agreement for historical findings and assessment of cyanosis was high. However, there was wide variation in agreement in the other assessments. Training to ensure consistent and reproducible assessment by different examiners will be necessary if these findings are to be used as outcome variables in clinical trials.