Assessing the quality of clinical data in a computer-based record for calculating the pneumonia severity index

Abstract

Objective: This study examined whether clinical data routinely available in a computerized patient record (CPR) can be used to drive a complex guideline that supports physicians in real time and at the point of care in assessing the risk of mortality for patients with community-acquired pneumonia.

Setting: Emergency department of a tertiary-care hospital.

Design: Retrospective analysis with medical chart review.

Patients: All 241 inpatients during a 17-month period (Jun 1995 to Nov 1996) who presented to the emergency department and had a primary discharge diagnosis of community-acquired pneumonia. METHODS/MAIN OUTCOME MEASURES: The 20 guideline variables were extracted from the CPR (HELP System) and the paper chart. The risk score and the risk class of the Pneumonia Severity Index were computed using data from the CPR alone and from a reference standard of all data available in the paper chart and the CPR at the time of the emergency department encounters. Availability and concordance were quantified to determine data quality. The type and cause of errors were analyzed depending on the source and format of the clinical variables.

Results: Of the 20 guideline variables, 12 variables were required to be present for every computer-charted emergency department patient, seven variables were required for selected patients only, and one variable was not typically available in the HELP System during a patient's encounter. The risk class was identical for 86.7 percent of the patients. The majority of patients with different risk classes were assigned too low a risk class. The risk scores were identical for 72.1 percent of the patients. The average availability was 0.99 for the data elements that were required to be present and 0.79 for the data elements that were not required to be present. The average concordance was 0.98 when all a patient's variables were taken into account. The cause of error was attributed to the nurse charting in 77 percent of the cases and to the computerized evaluation in 23 percent. The type of error originated from the free-text fields in 64 percent, from coded fields in 21 percent, from vital signs in 14 percent, and from laboratory results in 1 percent.

Conclusion: From a clinical perspective, the current level of data quality in the HELP System supports the automation and the prospective evaluation of the Pneumonia Severity Index as a computerized decision support tool.

Figure 1

First step of the Pneumonia Severity Index (PSI) scoring algorithm. On the basis of the patient's historical data and vital signs, the clinician assigns the patient to either risk class 1 or a higher risk class. For patients assigned to a higher risk class, a risk score is calculated and used by the clinician to determine the appropriate risk class. (Reprinted with permission from Fine et al. © 1997, New England Journal of Medicine.)

Figure 2

Analysis of risk classes. The chart shows patients' true risk classes as derived from the reference standard. For each risk class, the number of patients with identical risk classes (dark shading) and the number with different risk classes (light shading) are graphed. Solid triangles indicate the percentage of patients assigned to different classes. A different risk class was derived from the CPR data for 20 patients (13.3 percent). The proportion of misclassification was greatest in risk class 5.

Figure 3

Analysis of risk scores by risk class. The chart shows patients' true risk classes as derived from the reference standard. For each risk class, the number of patients with identical risk scores (dark shading) and the number with different risk scores (light shading) are graphed. Solid triangles indicate the percentage of patients assigned to different classes. An identical risk score was derived from the CPR data for 163 patients (72.1 percent). In risk classes 4 and 5, the proportion of incorrect scores was greatest.

Figure 4

Analysis of types of errors and their causes. The most prevalent type of error occurred in the free-text category and accounted for 64 percent of all errors. The laboratory data had the most correct data. The cause of error was attributed to emergency department nurse charting (light shading) for 77 percent of errors and to system errors (dark shading) for the rest.