Identifying the Italian provinces with increased mortality during COVID-19 epidemics using the data made available by the Italian National Institute of Statistics. A methodological challenge

Abstract

Objectives: to identify the Italian provinces with excess mortality during the COVID-19 epidemics using the mortality data provided in April 2020 by the Italian National Institute of Statistics (Istat) that, by design, included only the municipalities with at least 20% mortality increase compared to the same period in 2015-19. Inference with the aim to identify increased mortality at provincial level was a very important task when the Istat data were released in April, but the naïve aggregation of the selected municipalities was not sensible to due to the selection criteria of the municipalities used by Istat.

Design: use of a permutation-based approach to identify the Italian provinces with excess mortality during the first month of the COVID-19 epidemics using the data made available from Istat and taking into account the biased inclusion criteria.

Setting and participants: the number of deaths from any cause from 1 January was available for each year of the 2015-2020 period. Data were stratified by municipality, sex and 21 age categories. The third data release (R3) included 1,686 of the 7,904 Italian municipalities with increased mortality in 2020, covering about 40% of the Italian population. Results were compared with those obtainable with the fifth data release (R5), made available in June, when the selection of the municipalities was no longer based on increased mortality and which included more than 90% of the Italian population. R5 was considered the gold standard.

Main outcome measures: excess of deaths from any cause in the Italian provinces between 1 March and 4 April; relative risk (RR); permutation p-values; permutation-based adjusted relative risk; population coverage.

Results: the results of this study, which are based on two different test statistics, identify 17 and 33 provinces (out of 103) with increased overall mortality, respectively, controlling the family-wise error rate at 0.05 level. Most of the identified provinces are neighbouring provinces in the northern regions of Lombardy, Emilia-Romagna, Piedmont, Liguria, Marche and Tuscany, where most of the COVID-19 cases and deaths were identified. The comparison with data from R5 shows that all the identified provinces had an increase in overall mortality, mostly (31/34) above 25%. On average, the adjusted RR slightly underestimates the RR from R5, underestimating the large RR and overestimating the small RR.

Conclusions: this was, to the best of the authors' knowledge, the first attempt to aggregate the Istat data at province level and obtain a reliable and generalizable statistical inference. This permutation-based approach provides a feasible approach to take into account the selection bias that was present in the data and could be used for analysing other types of data that present some type of selection bias.

Keywords: COVID-19; overall mortality; permutation test; selection bias; Istat.