Assessing the variability in experimental hut trials evaluating insecticide-treated nets against malaria vectors

Abstract

Experimental hut trials (EHTs) are used to evaluate indoor vector control interventions against malaria vectors in a controlled setting. The level of variability present in the assay will influence whether a given study is well powered to answer the research question being considered. We utilised disaggregated data from 15 previous EHTs to gain insight into the behaviour typically observed. Using simulations from generalised linear mixed models to obtain power estimates for EHTs, we show how factors such as the number of mosquitoes entering the huts each night and the magnitude of included random effects can influence study power. A wide variation in behaviour is observed in both the mean number of mosquitoes collected per hut per night (ranging from 1.6 to 32.5) and overdispersion in mosquito mortality. This variability in mortality is substantially greater than would be expected by chance and should be included in all statistical analyses to prevent false precision of results. We utilise both superiority and non-inferiority trials to illustrate our methodology, using mosquito mortality as the outcome of interest. The framework allows the measurement error of the assay to be reliably assessed and enables the identification of outlier results which could warrant further investigation. EHTs are increasingly playing an important role in the evaluation and regulation of indoor vector control interventions so it is important to ensure that these studies are adequately powered.

Keywords: Anopheles; Experimental hut trials; Insecticide-treated nets; Long-lasting insecticidal nets; Power analysis; Vector control.

Graphical abstract

Fig. 1

Data from selected arms of an experimental hut trial, showing nightly variability in mosquito mortality, blood-feeding, and numbers caught. A Breakdown of mosquito numbers in the control arm (untreated net) over the course of the trial. The height of each bar indicates the total number of mosquitoes entering the hut each night. Bar colour denotes the mosquito mortality and blood-feeding status 24 ​h following collection (see the legend in panel D for the description of the mosquito status). B Breakdown of mosquito numbers in huts with unwashed PermaNet 3.0 nets. C Breakdown of mosquito numbers in huts in which unwashed PermaNet 3.0 nets were used. Panels A–C include pie-charts showing the aggregated mosquito data for each trial arm (as a percentage of total mosquitoes collected in that trial arm). D Summary measures over the whole trial for the outcome of a single feeding event by a blood-feeding mosquito: being deterred from entering the hut (green shading, calculated by the difference in the number of mosquitoes caught in the control arm relative to panels B and C), mosquitoes being alive and unfed, unfed and dead, fed and dead or successfully blood-fed and alive (red, note the percentage is different from pie charts in panels b and c due to the actions of deterrence). E and F Daily mortality estimates (and 95% confidence intervals, vertical line) for unwashed (E) and washed (F) PermaNet 3.0 nets, indicating the wide variation observed over the course of the trial. In these panels, the larger circles indicate mortality estimates with narrower confidence intervals (i.e. higher numbers of mosquitoes present). The colour of the points indicates the hut number. The data shown here were reported by Ngufor et al. (2022) from a study carried out in Benin in 2017.

Fig. 2

Illustration of the non-inferiority assessment. Here we show estimates of mosquito mortality after 24 ​h in a simulated experimental hut trial. Points (with 68% and 95% confidence interval estimates on the horizontal lines) show unwashed ITN (red), washed ITN (yellow) and an untreated control net (grey). Overall estimate of the ITN mortality is shown in orange and is calculated by combining data from washed and unwashed nets. We show two scenarios, one in which a candidate product is deemed to be non-inferior to an ITN already evaluated in a randomised control trial (top panels: A, B), and another in which it is not non-inferior (lower panels: C, D). Both unwashed and washed nets are included in the simulated trials: the averaged mortality across unwashed and washed nets was used to calculate the odds ratio between the two products. The right-hand panels (B, D) show the odds ratios for the two scenarios, along with their confidence intervals. If the lower confidence interval is greater than the pre-selected non-inferiority margin (NIM), the candidate product is judged to be non-inferior to the pre-evaluated ITN. Here we use an NIM of 0.7, as recommended by the WHO (WHO, 2018). We note that the same underlying parameters were used to generate the two synthetic datasets, illustrating the variation present in the assay.

Fig. 3

Sources of variability in data from previous experimental hut trials. A The mean number of mosquitoes entering each hut each night, across the 15 studies considered here (Supplementary Table S1). B-D The three sources of variation present in experimental hut trials (huts, sleepers, and observations, respectively). Here we visualise the values of the variances of the random effects in the regression models (Supplementary Table S1). Individual-level information on huts and sleepers was not available for all trials, hence fewer data points are available. This means that some of the observation-level variation recorded for those trials (shown by the darker colour in panel D) could be attributed to variation between huts and or sleepers.

Fig. 4

Power estimates for testing for superiority in an experimental hut trial. Here we examine a number of factors that can influence the power of a superiority trial: the difference in mortality between two products, the number of mosquitoes entering the huts and the magnitude of the observation-level variation in the data (characterised by the variance of the corresponding random effect in the GLMM). We considered a scenario in which we tested for superiority of a novel ITN, compared to an ITN with an induced mosquito mortality of 0.25 (i.e. one that would, on average, kill a quarter of all mosquitoes entering a hut in an EHT). We simulated three values for the underlying mortality induced by the novel net: 0.3, 0.35, and 0.4 (panels A–C, respectively, which state the corresponding percentage difference in induced mortalities between the nets). We varied the mean number of mosquitoes caught per hut per night (fixing the value of the dispersion parameter equal to 2) and the observation-level variation (x-axes), estimating the power using 1000 simulations for each parameter combination (solid lines). The dashed lines show the power estimates that would be obtained if the same synthetic datasets were analysed using a regression analysis without the observation-level random effect (these results can be misleading, as discussed in the Results section). All results were generated using one full rotation of a seven-arm trial, containing 343 data points in total (therefore, 49 data points each for the two ITNs evaluated in this scenario). The grey horizontal line in each panel indicates a power of 80%.

Fig. 5

Variability in experimental hut trial results and identification of possible outlier results. Here we use a recently obtained statistical relationship between the mortality induced by pyrethroid-PBO ITNs and pyrethroid-only ITNs (Sherrard-Smith et al., 2022b), fitted to data from experimental hut trials. This relationship is shown by the black curve (shown in all three panels) with the 95% credible intervals for the relationship indicated by the light-blue shaded area in panel A. Mosquito mortality is simulated for theoretical hut trials to allows us to illustrate the level of variation present in the assay. A We selected one point on the black curve (pyrethroid-PBO mortality equal to 0.75, pyrethroid-only mortality equal to 0.45) indicated by the green dot, and simulated 1000 trials (the mean number of mosquitoes per hut per night was 15, observation-level variance was 1.0). For each trial, point estimates of the observed mortality are shown by the dark blue dots, indicating the variation present in the assay. We next simulated 50,000 more trials, each time randomly selecting the pyrethroid-only mortality and then using the statistical relationship (this time incorporating the uncertainty in the relationship, represented by the light-blue area in panel A) to determine the pyrethroid-PBO mortality. We carried out this procedure, varying observation-level variation present in the data (B) and the average number of mosquitoes entering each hut (C). In panels B and C, the coloured lines indicate the 95% interval of the measured mortality of the pyrethroid-only and pyrethroid-PBO ITNs, indicating that results become more variable as the observation-level variation increases (B) and the mean number of mosquitoes per night per hut decreases (C). In panels B and C, the triangles indicate the observed values used to fit the best-fit curve. Points that fall outside the coloured lines could therefore be considered as outside the range of variability caused by the assay and so could be identified as outliers. All results generated here were from simulations of one full rotation of a seven-arm trial.

Fig. 6

Factors influencing the statistical power of a non-inferiority trial. Here we examined how the number of mosquitoes entering the huts, the magnitude of the observation-level variation in the data (characterised by the variance of the corresponding random effect in the GLMM), and the non-inferiority margin can influence the power of a non-inferiority trial. We assessed whether a candidate net was non-inferior (for mosquito mortality) to the active comparator product, assuming that the two products kill the same proportion of mosquitoes on average (we used a value of 0.3). We varied the mean number of mosquitoes (fixing the value of the dispersion parameter equal to 2) and the observation-level variation, estimating the power using 1000 simulations for each parameter combination. All results were generated using either one full rotation (panels A–C) or two full rotations (panels D–F) of a seven-arm trial: one rotation generates 343 data points in total. As illustrated in Fig. 3, the non-inferiority assessment was made from grouping together data from unwashed and washed ITNs of the same type (meaning that each simulated dataset contained 98 data points for each ITN, per rotation).