Local inconsistency detection using the Kullback-Leibler divergence measure

Abstract

Background: The standard approach to local inconsistency assessment typically relies on testing the conflict between the direct and indirect evidence in selected treatment comparisons. However, statistical tests for inconsistency have low power and are subject to misinterpreting a p-value above the significance threshold as evidence of consistency.

Methods: We propose a simple framework to interpret local inconsistency based on the average Kullback-Leibler divergence (KLD) from approximating the direct with the corresponding indirect estimate and vice versa. Our framework uses directly the mean and standard error (or posterior mean and standard deviation) of the direct and indirect estimates obtained from a local inconsistency method to calculate the average KLD measure for selected comparisons. The average KLD values are compared with a semi-objective threshold to judge the inconsistency as acceptably low or material. We exemplify our novel interpretation approach using three networks with multiple treatments and multi-arm studies.

Results: Almost all selected comparisons in the networks were not associated with statistically significant inconsistency at a significance level of 5%. The proposed interpretation framework indicated 14%, 66%, and 75% of the selected comparisons with an acceptably low inconsistency in the corresponding networks. Overall, information loss was more notable when approximating the posterior density of the indirect estimates with that of the direct estimates, attributed to indirect estimates being more imprecise.

Conclusions: Using the concept of information loss between two distributions alongside a semi-objectively defined threshold helped distinguish target comparisons with acceptably low inconsistency from those with material inconsistency when statistical tests for inconsistency were inconclusive.

Keywords: Consistency; Information loss; Kullback–Leibler divergence; Network meta-analysis.

Fig. 1

Network plots on a thrombolytics (first example) [19, 20], b smoking cessation (second example) [21], and c Parkinson’s disease (third example) [22]. Each node refers to a treatment and each edge to an observed (direct) comparison. The nodes’ size and the edges’ thickness are proportional to the number of randomised participants in the respective treatments and the number of studies investigating the respective comparisons. Numbers on the edges refer to the number of studies. Coloured loops indicate multi-arm studies. Acc t-PA, accelerated alteplase; t-PA, alteplase; ASPAC, anistreplase; PTCA, percutaneous transluminal coronary angioplasty; r-PA, reteplase; SK, streptokinase; SK + t-PA, streptokinase plus alteplase; TNK, tenecteplase; UK, urokinase

Fig. 2

a Probability densities of the direct (blue line) and indirect (black line) log ORs for a fictional target comparison assuming ${\widehat{\mu }}_D-{\widehat{\mu }}_I=1.17\tau$ (inconsistency evidence) and $\tau =0.1$ (low statistical heterogeneity [23]), yielding an average information loss ($D$) of 0.64. b Probability densities of the direct (blue line) and indirect (black line) log ORs for a fictional target comparison assuming ${\widehat{\mu }}_D-{\widehat{\mu }}_I=0$ (consistency evidence) and $\tau =0.1$ (low statistical heterogeneity [23]), yielding an average information loss of 0.13 (stricter threshold)

Fig. 3

Posterior densities of the direct (blue line) and indirect (black line) log ORs for 14 target comparisons from the thrombolytics network (first example). The grey area and vertical line indicate the inconsistency’s 95% interval (approximated using the reported posterior mean and standard deviation) and posterior mean. The average information loss ($D^j$) appear at the top left of each plot. The plots have been sorted in ascending order of the $D^j$ values. The x-axis and y-axis values vary across all plots. Green and orange $D^j$ values indicate acceptably low and material inconsistency. The threshold of 0.64 was employed

Fig. 4

Bar plots with the percentage contribution of approximating direct posterior density (blue bars, $D_{D,I}^j$) and indirect posterior density (black bars, $D_{I,D}^j$) to their total information loss ($D_{D,I}^j+D_{I,D}^j$) for each target comparison (x-axis) from the thrombolytics network (first example). Percentage contributions appear outside the parenthesis. The plots have been sorted in ascending order of the $D^j$ values. The $D_{D,I}^j$ and $D_{I,D}^j$ values appear in the parentheses

Fig. 5

Posterior densities of the direct (blue line) and indirect (black line) log ORs for six target comparisons from the smoking cessation network (second example). The grey area and vertical line indicate the inconsistency’s 95% interval (approximated using the reported posterior mean and standard deviation) and posterior mean. The average information loss ($D^j$) appear at the top left of each plot. The plots have been sorted in ascending order of the $D^j$ values. The x-axis and y-axis values vary across all plots. Green and orange $D^j$ values indicate acceptably low and material inconsistency. The threshold of 0.64 was employed

Fig. 6

Posterior densities of the direct (blue line) and indirect (black line) MD for four target comparisons from the Parkinson’s disease network (third example). The grey area and vertical line indicate the inconsistency’s 95% credible interval and posterior mean. The average information loss ($D^j$) appear at the top left of each plot. The plots have been sorted in ascending order of the $D^j$ values. The x-axis and y-axis values vary across all plots. Green and orange $D^j$ values indicate acceptably low and material inconsistency. The threshold of 0.64 was employed