Performance of physician-certified verbal autopsies: multisite validation study using clinical diagnostic gold standards

Abstract

Background: Physician review of a verbal autopsy (VA) and completion of a death certificate remains the most widely used approach for VA analysis. This study provides new evidence about the performance of physician-certified verbal autopsy (PCVA) using defined clinical diagnostic criteria as a gold standard for a multisite sample of 12,542 VAs. The study was also designed to analyze issues related to PCVA, such as the impact of a second physician reader on the cause of death assigned, the variation in performance with and without household recall of health care experience (HCE), and the importance of local information for physicians reading VAs.

Methods: The certification was performed by 24 physicians. The assignment of VA was random and blinded. Each VA was certified by one physician. Half of the VAs were reviewed by a different physician with household recall of health care experience included. The completed death certificate was processed for automated ICD-10 coding of the underlying cause of death. PCVA was compared to gold standard cause of death assignment based on strictly defined clinical diagnostic criteria that are part of the Population Health Metrics Research Consortium (PHMRC) gold standard verbal autopsy study.

Results: For individual cause assignment, the overall chance-corrected concordance for PCVA against the gold standard cause of death is less than 50%, with substantial variability by cause and physician. Physicians assign the correct cause around 30% of the time without HCE, and addition of HCE improves performance in adults to 45% and slightly higher in children to 48%. Physicians estimate cause-specific mortality fractions (CSMFs) with considerable error for adults, children, and neonates. Only for neonates for a cause list of six causes with HCE is accuracy above 0.7. In all three age groups, CSMF accuracy improves when household recall of health care experience is available.

Conclusions: Results show that physician coding for cause of death assignment may not be as robust as previously thought. The time and cost required to initially collect the verbal autopsies must be considered in addition to the analysis, as well as the impact of diverting physicians from servicing immediate health needs in a population to review VAs. All of these considerations highlight the importance and urgency of developing better methods to more reliably analyze past and future verbal autopsies to obtain the highest quality mortality data from populations without reliable death certification.

Figure 1

Illustration of the review framework used for physician certification.

Figure 2

Diagram of the process for physician review and data analysis.

Figure 3

Median chance-corrected concordance (%), by adult cause with and without HCE.

Figure 4

Median chance-corrected concordance (%), by child cause with and without HCE.

Figure 5

Median chance-corrected concordance (%), by neonate cause with and without HCE.

Figure 6

Random effect logistic regression odds ratios (OR) and standard errors (SE) by physician, of assigning the correct cause as a function of the physician reading the VA for adult, child, and neonatal causes.

Figure 7

Estimated versus true CSMFs across 500 Dirichlet splits, for adult bite of venomous animal without HCE.

Figure 8

Estimated versus true CSMFs across 500 Dirichlet splits, for adult bite of venomous animal with HCE.

Figure 9

Estimated versus true CSMFs across 500 Dirichlet splits, for adult asthma without HCE.

Figure 10

Estimated versus true CSMFs across 500 Dirichlet splits, for adult asthma with HCE.

Figure 11

Estimated versus true CSMFs across 500 Dirichlet splits, for adult other noncommunicable diseases without HCE.

Figure 12

Estimated versus true CSMFs across 500 Dirichlet splits, for adult other noncommunicable diseases with HCE.

Figure 13

Estimated versus true CSMFs across 500 Dirichlet splits, for child falls without HCE.

Figure 14

Estimated versus true CSMFs across 500 Dirichlet splits, for child falls with HCE.