Tuning parameter selection for a penalized estimator of species richness

Abstract

Our goal is to estimate the true number of classes in a population, called the species richness. We consider the case where multiple frequency count tables have been collected from a homogeneous population, and investigate a penalized maximum likelihood estimator under a negative binomial model. Because high probabilities of unobserved classes increase the variance of species richness estimates, our method penalizes the probability of a class being unobserved. Tuning the penalization parameter is challenging because the true species richness is never known, and so we propose and validate four novel methods for tuning the penalization parameter. We illustrate and contrast the performance of the proposed methods by estimating the strain-level microbial diversity of Lake Champlain over 3 consecutive years, and global human host-associated species-level microbial richness.

Keywords: diversity; ecology; maximum likelihood; microbiome; regularization.

Figure 1.

Estimates of C and their root-MSE over λ when $\eta =({10}^{-1},{10}^{-1})$ and C = 1000. Results are based on 100 simulations per λ.

Figure 2.

Estimates of C and their root-MSE over λ when $\eta =({10}^{-2},{10}^{-5})$ and C = 1000. Results are based on 100 simulations per λ.

Figure 3.

Simulation results for all proposed methods when $\eta =({10}^{-1},{10}^{-1})$, and when $\eta =({10}^{-2},{10}^{-5})$.

Figure 4.

Simulation results for Methods 0 and 3 when $\eta =({10}^{-1},{10}^{-3})$, $\eta =({10}^{-1},{10}^{-5})$ and $C\in \{500,1000,2000\}$. The distribution of $\hat{C}$ is shown over 100 draws. The true value of C is indicated with a solid horizontal line.