International survey-based assessment of the reliability, validity, and interpretability of the TDN grade for neurosurgical adverse events

Abstract

Neurosurgical adverse events (AE) are frequent and may have dramatic consequences on quality of life. The lack of a standardized classification of their severity hinders evaluation and improvement of the safety of procedures. The Therapy-Disability-Neurology (TDN) grade, validated in 2021 on 6071 interventions, overcomes limitations of previous grading-systems by addressing the severity of neurologic and disabling AEs. The aim of the current study is to assess the reliability, validity and applicability of the TDN grade. We conducted an online survey involving participants with varying levels of neurosurgical expertise. Participants assessed the TDN grade for 16 case vignettes and reviewed the validity, interpretability, logicality, simplicity, and usefulness of the grading-system. The TDN grade showed substantial inter-rater (α = 0.66) and intra-rater (α = 0.79) reliability. Most participants recommended reporting its separate dimensions, which demonstrated substantial to almost perfect reliability (inter-rater: α = 0.74; intra-rater: α = 0.85). Online calculation tools significantly improved agreement and participants' scores. The TDN grade was considered fairly useful, very logical, fairly simple to use and interpret, and its separate dimensions were considered a very valid measure of the severity of AEs. Neurosurgical AEs should be systematically reported, and surveyed neurosurgeons recommend the use of the TDN grade along its separate dimensions for this purpose.

Keywords: Classification; Complication; Complications; Cranial; Outcome; Spinal.

Fig. 1

The Therapy-Disability-Neurology (TDN) grade. The TDN grade as well as its separate dimensions can be measured using this flowchart. First, choose the severity of the Therapy (T) dimension (based on the Clavien-Dindo-Classification, CDG, and Landriel-Ibañez Classification, LIC)^, and follow the arrow. Second, answer the question of the Disability (D, based on the modified Rankin Scale, mRS) dimension (follow green arrow to the next dimension if answer is “no”, follow red arrow if answer is “yes”). Third, answer to the question of the Neurology (N) dimension and follow the corresponding arrow to get the overall TDN grade. In order to report dimensions separately, use the flowchart separately for each dimension, while ignoring other dimensions (or use Table 1). It should be noted that grade 5 adverse events always correspond to T5D5N2.

Fig. 2

Results of the survey (Case vignettes). Each panel corresponds to a case vignette (see Supplementary Information 7 [Table—Case vignettes and authors answers] online). Participants were asked to assess the severity of AEs using the TDN grade and its separate dimensions (Therapy: “T”, Disability: “D”, Neurology: “N”). All individual answers are displayed with black points. For each case vignette, participants were asked the question “How well does the TDN grade reflect severity in this case”. Original answers ranged from 0 to 100 and were rescaled for the Fig. (0 = “very badly” = red, 5 = “very well” = green). Big green circles represent the authors’ recommended answer and horizontal bars participants’ median answer (black when concordant with author’s recommendations, red when discordant).

Fig. 3

Inter- and intra-rater reliability of the TDN grade and its separate dimensions. Krippendorff’s α is provided for each sub-analysis and displayed along the interpretation of agreement coefficients as proposed by Landis et al. TDN TDN grade, T/D/N each separate dimensions of the TDN grade together, T Therapy dimension, D Disability dimension, N Neurology dimension, CI 95% Confidence Interval, IQR Interquartile range.

Fig. 4

Differences in participants’ score according to the use of online tools or artificial intelligence. Participant scores were measured through a comparison of their responses (overall TDN grade, not individual dimensions) with author’s responses (percentage of correct answers), with unanswered questions excluded from the scoring. The difference in the median score of participants who did or did not use online calculating tools was compared using the Wilcoxon rank sum test. Violin plots present the density of scores, where broader sections indicate higher density. Boxplots within each violin offer key summary statistics (median, quartiles, outliers). Furthermore, after receiving instructions concerning the functioning of the TDN grade, ChatGPT 3.5 (OpenAI ©) was prompted to determine the grade of AEs for each case vignette and its score was calculated similarly as for other participants.