Validation of a new reference standard for the diagnosis of vasospasm

Abstract

Rationale and objectives: The purpose of our study is to perform an internal validation of a new reference standard for vasospasm diagnosis in aneurysmal subarachnoid hemorrhage (A-SAH) patients.

Materials and methods: A retrospective study was performed on A-SAH patients between January 2002 and May 2009. All patients were applied to this new reference standard using a multistage hierarchical approach incorporating clinical and imaging criteria. An internal validation method was performed in two phases to compare the new reference standard with digital subtraction angiography (DSA) and to assess accuracy. In Phase I, the diagnostic outcomes from DSA at the primary level were compared with the secondary/tertiary levels in the reference standard. In Phase II, the new reference standard was compared with chart diagnosis. Accuracy test characteristics, agreement rates, kappa values, and bias indices were calculated.

Results: In Phase I (n = 85), there was 87% agreement rate, 0.674 kappa, and 0.12 bias index. However, there was 100% agreement in patients diagnosed with vasospasm by DSA. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 100%, 61%, 83%, and 100% respectively. In Phase II (n = 137), there was 91% agreement rate, 0.824 kappa, and 0.04 bias index. Sensitivity, specificity, PPV, and NPV were 88%, 95%, 96%, and 87%, respectively.

Conclusion: Performing validation methods for a new reference standard is an evolving and ongoing process because limitations and bias in the reference standard are identified. Based on the results of this internal validation, a modification in the new reference standard is made at the primary level, resulting in improvement in its accuracy and classification of A-SAH patients.

Figure 1

Flow Chart for Reference Standard To Determine the Study Outcome of Vasospasm Diagnosis Reprinted from Academic Radiology, 16/5, Reichman, et al., “Developing Patient-centered Outcome Measures for Evaluating Vasospasm in Aneurysmal Subarachnoid Hemorrhage,” pg. 541-545, 2009, with permission from Elsevier

Figure 2

Revised Flow Chart for Reference Standard To Determine the Study Outcome of Vasospasm Diagnosis