Symptom severity prediction from neuropsychiatric clinical records: Overview of 2016 CEGS N-GRID shared tasks Track 2

Abstract

The second track of the CEGS N-GRID 2016 natural language processing shared tasks focused on predicting symptom severity from neuropsychiatric clinical records. For the first time, initial psychiatric evaluation records have been collected, de-identified, annotated and shared with the scientific community. One-hundred-ten researchers organized in twenty-four teams participated in this track and submitted sixty-five system runs for evaluation. The top ten teams each achieved an inverse normalized macro-averaged mean absolute error score over 0.80. The top performing system employed an ensemble of six different machine learning-based classifiers to achieve a score 0.86. The task resulted to be generally easy with the exception of two specific classes of records: records with very few but crucial positive valence signals, and records describing patients predominantly affected by negative rather than positive valence. Those cases proved to be very challenging for most of the systems. Further research is required to consider the task solved. Overall, the results of this track demonstrate the effectiveness of data-driven approaches to the task of symptom severity classification.

Keywords: CEGS N-GRID; Clinical NLP; Neuropsychiatric records; Shared tasks; Symptom severity classification.

Figure 1

Excerpts of an initial psychiatric evaluation record. Whitespace characters are highlighted.

Figure 2

The RDoC matrix for the positive valence domain with some of its genetic, molecular and behavioral traits.

Figure 3

Distribution of classes between training and test set.

Figure 4

Distribution of classes between annotators and gold standard.

Figure 5

Confusion matrixes for the top 5 best runs.

Figure 6

Number of test set records grouped by absolute distance. The distance is averaged across the top 10 best systems predictions.