Deconvolution estimation of mixture distributions with boundaries

Abstract

In this paper, motivated by an important problem in evolutionary biology, we develop two sieve type estimators for distributions that are mixtures of a finite number of discrete atoms and continuous distributions under the framework of measurement error models. While there is a large literature on deconvolution problems, only two articles have previously addressed the problem taken up in our article, and they use relatively standard Fourier deconvolution. As a result the estimators suggested in those two articles are degraded seriously by boundary effects and negativity. A major contribution of our article is correct handling of boundary effects; our method is asymptotically unbiased at the boundaries, and also is guaranteed to be nonnegative. We use roughness penalization to improve the smoothness of the resulting estimator and reduce the estimation variance. We illustrate the performance of the proposed estimators via our real driving application in evolutionary biology and two simulation studies. Furthermore, we establish asymptotic properties of the proposed estimators.

Keywords: Boundary effect; maximum likelihood; measurement error; mixture distribution; penalization; sieve method.

Fig 1

Simulation Study I. Density plot of the continuous component in Model (4.1).

Fig 2

The motivating virus lineage application. Panel (a) shows various density estimators of the deleterious mutation effects. Panel (b) shows the selection for the penalty parameter λ.

Fig 3

Validation of the exponential assumption: the density-envelope plots for various values of the roughness penalty parameter λ. The light grey curves form the envelope to show the natural estimation variation, the black dash-dotted curve shows the exponential density which generates the simulation samples, the black solid curve is the penalized sieve density estimator obtained from the data, and the dark grey solid curve is the average of the 100 grey curves.