Do practicing clinicians agree with expert ratings of neonatal intensive care unit quality measures?

Abstract

Objective: To assess the level of agreement when selecting quality measures for inclusion in a composite index of neonatal intensive care quality (Baby-MONITOR) between two panels: one comprised of academic researchers (Delphi) and another comprised of academic and clinical neonatologists (clinician).

Study design: In a modified Delphi process, a panel rated 28 quality measures. We assessed clinician agreement with the Delphi panel by surveying a sample of 48 neonatal intensive care practitioners. We asked the clinician group to indicate their level of agreement with the Delphi panel for each measure using a five-point scale (much too high, slightly too high, reasonable, slightly too low and much too low). In addition, we asked clinicians to select measures for inclusion in the Baby-MONITOR based on a yes or no vote and a pre-specified two-thirds majority for inclusion.

Result: In all, 23 (47.9%) of the clinicians responded to the survey. We found high levels of agreement between the Delphi and clinician panels, particularly across measures selected for the Baby-MONITOR. Clinicians selected the same nine measures for inclusion in the composite as the Delphi panel. For these nine measures, 74% of clinicians indicated that the Delphi panel rating was 'reasonable'.

Conclusion: Practicing clinicians agree with an expert panel on the measures that should be included in the Baby-MONITOR, enhancing face validity.

Figure 1. Deviation among clinicians’ responses evaluating ratings of 27 measures of NICU care quality

Each midpoint (triangle) represents the mean rating from the clinician panel. The tails represent +/− 1 standard deviation. Among the measures selected for the Baby-MONITOR were those with the smallest standard deviation. Measures not selected for the Baby-MONITOR: A. VLBW volume; B. IVH; C. Cystic PVL; D. Assisted ventilation (AV); E. O₂ at 28 days; F. O₂ at discharge; G. AV at discharge; H. Retinopathy of Prematurity Surgery; I. Any NEC; J. NEC surgery; K. Human milk only at discharge; L. Length of stay; M. Retinopathy of Prematurity>stage 2; N. IVH >grade 2; O. Temp 1 hour; P. Surfactant within 2 hours; Q. Steroids for chronic lung disease; R. Mortality <28 days Measures selected for the Baby-MONITOR: S. O₂ at 36 weeks; T. Any human milk at discharge; U. Hospital mortality; V. Hypothermia at admission; W. Pneumothorax; X. Growth velocity; K. Antenatal steroids; Z Retinopathy of Prematurity Exam; AA. Late onset sepsis