Combining Rule-based NLP-lite with Rapid Iterative Chart Adjudication for Creation of a Large, Accurately Curated Cohort from EHR data: A Case Study in the Context of a Clinical Trial Emulation

Abstract

The aim of this work was to create a gold-standard curated cohort of 10,000+ cases from the Veteran Affairs (VA) corporate data warehouse (CDW) for virtual emulation of a randomized clinical trial (CSP#592). The trial had six inclusion/exclusion criteria lacking adequate structured data. We therefore used a hybrid computer/human approach to extract information from clinical notes. Rule-based NLP output was iteratively adjudicated by a panel of trained non-clinician content experts and non-experts using an easy-to-use spreadsheet-based rapid adjudication display. This group-adjudication process iteratively sharpened both the computer algorithm and clinical decision criteria, while simultaneously training the non-experts. The cohort was successfully created with each inclusion/exclusion decision backed by a source document. Less than 0.5% of cases required referral to specialist clinicians. It is likely that such curated datasets capturing specialist reasoning and using a process-supervised approach will acquire greater importance as training tools for future clinical AI applications.

Figure 1:

Rapid Adjudication Display: This screenshot displays a part of our final LVEF adjudication for the simplest cases. The first 7 columns (“Ser No” through “Report Button”) were algorithm-generated, and the last 4 columns were filled in by human adjudicators. The adjudicators had to do the following: i) Type “Y” in the “LVEF Correct?” column if the numbers in the “Low Value” and “High Value” columns corresponded to the highlighted EF, LVEF or Ejection fraction values in the “Snippet” column, otherwise type “N”. ii) Type “Y” in the “LVEF qualifies” column if the mean of the true LVEF values in the snippet was <=35%, else type “N”. iii) Type “Y’ if the dates in the “Proc Date” and “Doc Date” column were within 15 days of each other, else type “N” – dates are ‘x’-ed out here for privacy reasons. iv) Optionally type in comments in case of ambiguity, or to provide feedback. The adjudicators could click the “Report Button” to view the entire report if a decision could not be made based on the snippet alone.

Figure 2:

Rapid Adjudication Display for BiV exclusion showing the implementation of the complex clinical decision tree. All the columns except the last one, were algorithm-generated. The ‘reason’ column shows the algorithm’s justification for its Y/N decision. The adjudicator was required to flag patients who needed to be rejected.