EA3: A softmax algorithm for evidence appraisal aggregation

Abstract

Real World Evidence (RWE) and its uses are playing a growing role in medical research and inference. Prominently, the 21st Century Cures Act-approved in 2016 by the US Congress-permits the introduction of RWE for the purpose of risk-benefit assessments of medical interventions. However, appraising the quality of RWE and determining its inferential strength are, more often than not, thorny problems, because evidence production methodologies may suffer from multiple imperfections. The problem arises to aggregate multiple appraised imperfections and perform inference with RWE. In this article, we thus develop an evidence appraisal aggregation algorithm called EA3. Our algorithm employs the softmax function-a generalisation of the logistic function to multiple dimensions-which is popular in several fields: statistics, mathematical physics and artificial intelligence. We prove that EA3 has a number of desirable properties for appraising RWE and we show how the aggregated evidence appraisals computed by EA3 can support causal inferences based on RWE within a Bayesian decision making framework. We also discuss features and limitations of our approach and how to overcome some shortcomings. We conclude with a look ahead at the use of RWE.

Fig 1. Behaviour of EA3(c@k,1k@k) for varying β, where β·(1-1k) is the second factor within the scope of the exponential function.

The smaller parameter β and the greater the number of appraisals (the greater k), the closer ${\text{EA}}^3(c@k,\frac{1}{k}@k)$ gets to the identity map. This graph clearly displays the monotonicity of these functions.

Fig 2. Aggregated appraisals of Karimi et al. (2006) [75] according to De Pretis et al. (2019) [78] (solid line) and EA³ (dash-dot line).

The latter, lower, curve displays the behaviour with respect to the cautiousness parameter β. Both curves agree for β = 0 where EA³ equals the weighted mean.

Fig 3. Similarly to Fig 2, the upper panel shows the aggregated appraisals of Newson et al. (2000) [72], Amberbir et al. (2011) [76] and Beasley et al. (2011) [77] according to De Pretis et al. (2019) [78] (solid line) and EA³ (dash-dot line).

The lower panel depicts the aggregated appraisals for Lesko and Mitchell (1999) [71] and Lesko et al. (2002) [73].

Fig 4. Posterior probability of the causal hypothesis ©, considering Beasley et al. (2011) [77] as evidence E1, and computed in agreement with De Pretis et al. (2019) [78] (solid line) and EA³ (dash-dot, dotted and dashed lines).

For EA³, different lines represent different priors $P\left({\mathcal{E}}_1\right)$, whereas the prior P(©) is always set to 1%. All curves agree for ν_f = 1 where De Pretis et al. (2019) [78] becomes a special case of EA³.

Fig 5. Similarly to Fig 4, the graph pictures the posterior probability of the causal hypothesis ©, considering Beasley et al. (2011) [77] as evidence E1, and computed in agreement with De Pretis et al. (2019) [78] (solid line) and EA³ (dash-dot, dotted and dashed lines).

In this plot, the prior P(©) is set to 0.5%.