Quality indices for topic model selection and evaluation: a literature review and case study

Abstract

Background: Topic models are a class of unsupervised machine learning models, which facilitate summarization, browsing and retrieval from large unstructured document collections. This study reviews several methods for assessing the quality of unsupervised topic models estimated using non-negative matrix factorization. Techniques for topic model validation have been developed across disparate fields. We synthesize this literature, discuss the advantages and disadvantages of different techniques for topic model validation, and illustrate their usefulness for guiding model selection on a large clinical text corpus.

Design, setting and data: Using a retrospective cohort design, we curated a text corpus containing 382,666 clinical notes collected between 01/01/2017 through 12/31/2020 from primary care electronic medical records in Toronto Canada.

Methods: Several topic model quality metrics have been proposed to assess different aspects of model fit. We explored the following metrics: reconstruction error, topic coherence, rank biased overlap, Kendall's weighted tau, partition coefficient, partition entropy and the Xie-Beni statistic. Depending on context, cross-validation and/or bootstrap stability analysis were used to estimate these metrics on our corpus.

Results: Cross-validated reconstruction error favored large topic models (K ≥ 100 topics) on our corpus. Stability analysis using topic coherence and the Xie-Beni statistic also favored large models (K = 100 topics). Rank biased overlap and Kendall's weighted tau favored small models (K = 5 topics). Few model evaluation metrics suggested mid-sized topic models (25 ≤ K ≤ 75) as being optimal. However, human judgement suggested that mid-sized topic models produced expressive low-dimensional summarizations of the corpus.

Conclusions: Topic model quality indices are transparent quantitative tools for guiding model selection and evaluation. Our empirical illustration demonstrated that different topic model quality indices favor models of different complexity; and may not select models aligning with human judgment. This suggests that different metrics capture different aspects of model goodness of fit. A combination of topic model quality indices, coupled with human validation, may be useful in appraising unsupervised topic models.

Keywords: Clinical text data; Cross-validation; Electronic medical record; Internal validation; Non-negative matrix factorization; Stability analysis; Topic model.

Fig. 1

Average training/testing reconstruction error of NMF models of complexity K = (5,10,25,50,100,150,200,250) estimated using five-fold Monte Carlo cross-validation

Fig. 2

Average topic coherence of NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis. The left-hand panel plot uses the UCI topic coherence score, and the right-hand panel plot uses the UMASS topic coherence score

Fig. 3

Average rank-biased overlap of $\varphi$ over NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis

Fig. 4

Average rank-biased overlap of $\theta$ over NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis

Fig. 5

Average Kendall’s weighted tau of $\varphi$ over NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis

Fig. 6

Average Kendall’s weighted tau of $\theta$ over NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis

Fig. 7

Average partition coefficient and partition entropy scores of NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis. The left-hand panel plot uses the partition coefficient score, and the right-hand panel plot uses partition entropy score. Both the partition coefficient and the partition entropy suggest that smaller models result in more “crisp” clustering solutions; whereas, larger models result in more “fuzzy/admixed” clustering solutions

Fig. 8

Average Xie-Beni scores of NMF models of complexity K = (5,10,25,50,75,100) estimated using five-fold stability analysis