SEPIA: simulation-based evaluation of prioritization algorithms

Abstract

Background: The ability to prioritize people living with HIV (PLWH) by risk of future transmissions could aid public health officials in optimizing epidemiological intervention. While methods exist to perform such prioritization based on molecular data, their effectiveness and accuracy are poorly understood, and it is unclear how one can directly compare the accuracy of different methods. We introduce SEPIA (Simulation-based Evaluation of PrIoritization Algorithms), a novel simulation-based framework for determining the effectiveness of prioritization algorithms. SEPIA expands upon prior related work by defining novel metrics of effectiveness with which to compare prioritization techniques, as well as by creating a simulation-based tool with which to perform such effectiveness comparisons. Under several metrics of effectiveness that we propose, we compare two existing prioritization approaches: one phylogenetic (ProACT) and one distance-based (growth of HIV-TRACE transmission clusters).

Results: Using all proposed metrics, ProACT consistently slightly outperformed the transmission cluster growth approach. However, both methods consistently performed just marginally better than random, suggesting that there is significant room for improvement in prioritization tools.

Conclusion: We hope that, by providing ways to quantify the effectiveness of prioritization methods in simulation, SEPIA will aid researchers in developing novel risk prioritization tools for PLWH.

Keywords: FAVITES; HIV; Metrics; Phylogenetic; Prioritization; SEPIA; Simulation-based evaluation.

Fig. 1

Metric 2 is the slope of the best-fit line (red solid line) of the step function of the number of times a given individual has transmitted (red dashed lines), regressed between the time of the individual’s first transmission event (“Start”) and present day (“Present”)

Fig. 2

Given simulated epidemic data and a prioritization of the individuals in the simulated epidemic, SEPIA computes the user-selected effectiveness metric for each person in the prioritization. Then, to construct an overall effectiveness score for this prioritization, SEPIA computes the Kendall Tau-b correlation coefficient between the ordered list of effectiveness values and the theoretical optimum

Fig. 3

Effectiveness of prioritization using ProACT and HIV-TRACE transmission cluster growth across all metrics on datasets simulated by FAVITES. Each column represents a single experimental condition, and each violin plot depicts the Kendall Tau-b correlation coefficients computed by SEPIA across 20 simulation replicates. The experimental conditions are varied by altering 3 parameters: expected number of contacts per individual $\left(E_d\right)$, rate of starting ART $\left({\lambda }_{+}\right)$, and rate of stopping ART $\left({\lambda }_{-}\right)$