Large-scale assessment of antimicrobial resistance marker databases for genetic phenotype prediction: a systematic review

Abstract

Background: Antimicrobial resistance (AMR) is a rising health threat with 10 million annual casualties estimated by 2050. Appropriate treatment of infectious diseases with the right antibiotics reduces the spread of antibiotic resistance. Today, clinical practice relies on molecular and PCR techniques for pathogen identification and culture-based antibiotic susceptibility testing (AST). Recently, WGS has started to transform clinical microbiology, enabling prediction of resistance phenotypes from genotypes and allowing for more informed treatment decisions. WGS-based AST (WGS-AST) depends on the detection of AMR markers in sequenced isolates and therefore requires AMR reference databases. The completeness and quality of these databases are material to increase WGS-AST performance.

Methods: We present a systematic evaluation of the performance of publicly available AMR marker databases for resistance prediction on clinical isolates. We used the public databases CARD and ResFinder with a final dataset of 2587 isolates across five clinically relevant pathogens from PATRIC and NDARO, public repositories of antibiotic-resistant bacterial isolates.

Results: CARD and ResFinder WGS-AST performance had an overall balanced accuracy of 0.52 (±0.12) and 0.66 (±0.18), respectively. Major error rates were higher in CARD (42.68%) than ResFinder (25.06%). However, CARD showed almost no very major errors (1.17%) compared with ResFinder (4.42%).

Conclusions: We show that AMR databases need further expansion, improved marker annotations per antibiotic rather than per antibiotic class and validated multivariate marker panels to achieve clinical utility, e.g. in order to meet performance requirements such as provided by the FDA for clinical microbiology diagnostic testing.

Figure 1.

The bACC is shown for every species as the average of all bACCs for the species–antibiotic combinations mentioned in the main text. bACC was chosen as the evaluation criterion as it avoids performance inflation and provides a balanced representation of false-positive and false-negative rates even in the case of dataset class imbalance. Error bars indicate SD. CARD predicted all E. coli and P. aeruginosa isolates to be resistant to all tested antibiotics, resulting in a constant bACC of 0.5 and the absence of the error bars.

Figure 2.

bACC measures using ResFinder. The heatmap shows analysed antibiotics versus pathogens. White rectangles represent species–antibiotic pairs that were not analysed due to absent or insufficient AST data.

Figure 3.

Evaluation of ResFinder and CARD (RGI) antibiotic resistance prediction performance on E. coli. (a) ResFinder prediction performance across 17 antibiotics. (b) CARD prediction performance across 17 antibiotics. (c) ResFinder and PointFinder prediction performance for ciprofloxacin, cefotaxime and ceftazidime. (d) CARD prediction performance excluding predictions based on markers related to efflux pump mechanism. ResFinder shows overall better prediction performance than CARD; PointFinder predictions improve ResFinder predictions and, excluding predictions based on efflux-related markers, improve CARD predictions except for tetracycline.

Figure 4.

Differences in resistance profiles for β-lactamases and AMEs in K. pneumoniae. (a) KPC β-lactamases and NDM β-lactamases are consistently good predictors of resistance across all the analysed cephalosporins whereas OKP β-lactamases and CTX-M β-lactamases show variable resistance prediction performance. (b) AACs, aminoglycoside phosphotransferases (APHs) and ANTs consistently show lower PPVs for amikacin than tobramycin or gentamicin.

Figure 5.

bACC of ResFinder predictions based on ResFinder compound–class-level annotation versus compound-level annotation in K. pneumoniae samples. Compound-level prediction consistently performs better and the increase in bACC ranges between 0.01 for tobramycin resistance prediction and 0.26 for cefepime resistance prediction.