Search for exotic decays of the Higgs boson to a pair of pseudoscalars in the μμbb and ττbb final states

Abstract

A search for exotic decays of the Higgs boson ($\text{H}$) with a mass of 125$\text{Ge}\text{V}$ to a pair of light pseudoscalars ${\text{a}}_1$ is performed in final states where one pseudoscalar decays to two $\text{b}$ quarks and the other to a pair of muons or $\tau$ leptons. A data sample of proton-proton collisions at $\sqrt{s}=13\text{Te}\text{V}$ corresponding to an integrated luminosity of 138${\text{fb}}^{-1}$ recorded with the CMS detector is analyzed. No statistically significant excess is observed over the standard model backgrounds. Upper limits are set at 95% confidence level ($\text{CL}$) on the Higgs boson branching fraction to $\mu \mu \text{b}\text{b}$ and to $\tau \tau \text{b}\text{b},$ via a pair of ${\text{a}}_1$s. The limits depend on the pseudoscalar mass $m_{{\text{a}}_1}$ and are observed to be in the range (0.17-3.3) $\times {10}^{-4}$ and (1.7-7.7) $\times {10}^{-2}$ in the $\mu \mu \text{b}\text{b}$ and $\tau \tau \text{b}\text{b}$ final states, respectively. In the framework of models with two Higgs doublets and a complex scalar singlet (2HDM+S), the results of the two final states are combined to determine upper limits on the branching fraction $B(\text{H}\to {\text{a}}_1{\text{a}}_1\to \ell \ell \text{b}\text{b})$ at 95% $\text{CL}$, with $\ell$ being a muon or a $\tau$ lepton. For different types of 2HDM+S, upper bounds on the branching fraction $B(\text{H}\to {\text{a}}_1{\text{a}}_1)$ are extracted from the combination of the two channels. In most of the Type II 2HDM+S parameter space, $B(\text{H}\to {\text{a}}_1{\text{a}}_1)$ values above 0.23 are excluded at 95% $\text{CL}$ for $m_{{\text{a}}_1}$ values between 15 and 60$\text{Ge}\text{V}$.

Fig. 1

The distributions of leading and subleading (upper) muon $p_{\text{T}}$ and (lower) b jet $p_{\text{T}}$ in the selected events. The uncertainty band in the lower panel represents the limited size of the simulated samples together with a 30% uncertainty in the low-mass DY cross section. Simulated samples are normalized using the corresponding theoretical cross sections. To evaluate the normalization of the signal, SM Higgs boson cross sections are multiplied by the $\mathcal{B}({\text{a}}_1{\text{a}}_1\to \mathrm{\mu }\mathrm{\mu }\text{b}\text{b})$ value that is calculated in the Type III model with $tan\beta =2$

Fig. 2

The $p_{\text{T}}$ distributions of the (upper) dimuon systems and (lower) di-b-jet system. The uncertainty band in the lower panel represents the limited size of the simulated samples together with a 30% uncertainty in the low-mass DY cross section. Simulated samples are normalized to using the corresponding theoretical cross sections. To evaluate the normalization of the signal, SM Higgs boson cross sections are multiplied by the $\mathcal{B}({\text{a}}_1{\text{a}}_1\to \mathrm{\mu }\mathrm{\mu }\text{b}\text{b})$ value that is calculated in the Type III model with $tan\beta =2$

Fig. 3

The distribution of ${\chi }_{\text{b}\text{b}}$ versus ${\chi }_{\text{H}}$ as defined in Eq. (1) for (upper) simulated background processes and (lower) the signal process with $m_{{\text{a}}_1}=40\text{Ge}\text{V}.$ The contours indicate lines of constant ${\chi }_{\text{tot}}^2.$ The gray scale represents the expected yields in data. To evaluate the yield of the signal, SM Higgs boson cross sections are multiplied by the $\mathcal{B}({\text{a}}_1{\text{a}}_1\to \mathrm{\mu }\mathrm{\mu }\text{b}\text{b})$ value that is calculated in the Type III model with $tan\beta =2$

Fig. 4

Signal $(m_{{\text{a}}_1}=40\text{Ge}\text{V})$ versus background efficiency for different thresholds on ${\chi }_{\text{tot}}^2$ (gray) and ${\chi }_{\text{d}}^2$ (red) variables. The black star indicates signal efficiency versus that of background for the optimized ${\chi }_{\text{d}}^2$ requirement

Fig. 5

Pre-fit distributions of the DNN score for the $\mathrm{\mu }{\mathrm{\tau }}_{\text{h}}$ channel divided into events with one (upper) or at least two (lower) b jets. The shape of the $\text{H}\to {\text{a}}_1{\text{a}}_1$ signal, where $m_{{\text{a}}_1}=35\text{Ge}\text{V},$ is indicated assuming $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \mathrm{\tau }\mathrm{\tau }\text{b}\text{b})$ to be 10%. The lower panel shows the ratio of the observed data to the expected yields. The gray band represents the unconstrained statistical and systematic uncertainties

Fig. 6

The best fit background models for the $\mathrm{\mu }\mathrm{\mu }\text{b}\text{b}$ channel together with a 68% CL uncertainty band from the fit to the data under the background-only hypothesis for the (upper left) Low $p_{\text{T}}$ category, (middle left) VBF category, (middle right) TL category, (lower left) TM category, and (lower right) TT category. For comparison, the signal-plus-background is shown for the (upper right) Low $p_{\text{T}}$ category for a signal with $m_{{\text{a}}_1}=40\text{Ge}\text{V}.$ The expected signal yield is evaluated assuming the SM production of the Higgs boson and $\mathcal{B}({\text{a}}_1{\text{a}}_1\to \mathrm{\mu }\mathrm{\mu }\text{b}\text{b})=0.2\%,$ as predicted in the Type III 2HDM+S with $tan\beta =2.$ The bin widths depend on statistics, irrelevant for the final fit

Fig. 7

Post-fit distributions of $m_{\mathrm{\tau }\mathrm{\tau }}$ for the $\mathrm{\mu }{\mathrm{\tau }}_{\text{h}}$ channel signal regions in events with exactly one b tagged jet: SR1 (upper), SR2 (middle), and SR3 (lower). The shape of the $\text{H}\to {\text{a}}_1{\text{a}}_1$ signal, where $m_{{\text{a}}_1}=35\text{Ge}\text{V},$ is indicated assuming $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \mathrm{\tau }\mathrm{\tau }\text{b}\text{b})$ to be 10%

Fig. 8

Post-fit distributions of the $m_{\mathrm{\tau }\mathrm{\tau }}$ for the $\mathrm{\mu }{\mathrm{\tau }}_{\text{h}}$ channel signal regions in events with at least two b tagged jets: SR1 (upper) and SR2 (lower). The shape of the $\text{H}\to {\text{a}}_1{\text{a}}_1$ signal, where $m_{{\text{a}}_1}=35\text{Ge}\text{V},$ is indicated assuming $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \mathrm{\tau }\mathrm{\tau }\text{b}\text{b})$ to be 10%

Fig. 9

Post-fit distributions of the $m_{\mathrm{\tau }\mathrm{\tau }}$ for the $\mathrm{\mu }{\mathrm{\tau }}_{\text{h}}$ channel control regions in events with exactly one b tagged jet (upper) and at least two b tagged jets (lower). The contamination from the $\text{H}\to {\text{a}}_1{\text{a}}_1$ signal, where $m_{{\text{a}}_1}=35\text{Ge}\text{V},$ is barely visible assuming $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \mathrm{\tau }\mathrm{\tau }\text{b}\text{b})$ to be 10%

Fig. 10

Observed and expected upper limits at 95% $\text{CL}$ on $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \mathrm{\mu }\mathrm{\mu }\text{b}\text{b})$ as functions of $m_{{\text{a}}_1}$. The inner and outer bands indicate the regions containing the distribution of limits located within 68 and 95% confidence intervals, respectively, of the expectation under the background-only hypothesis

Fig. 11

Observed and expected 95% $\text{CL}$ exclusion limits on $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \mathrm{\tau }\mathrm{\tau }\text{b}\text{b})$ in percent, for the (upper left) $\mathrm{\mu }{\mathrm{\tau }}_{\text{h}}$, (upper right) $\text{e}{\mathrm{\tau }}_{\text{h}}$, (lower left) $\text{e}\mathrm{\mu }$ channels, and (lower right) for the combination of all the channels

Fig. 12

Observed and expected 95% CL upper limits on $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1\to \ell \ell \text{b}\text{b})$ in %, where $\ell$ stands for muons or $\mathrm{\tau }$ leptons, obtained from the combination of the $\mathrm{\mu }\mathrm{\mu }\text{b}\text{b}$ and $\mathrm{\tau }\mathrm{\tau }\text{b}\text{b}$ channels. The results are obtained as functions of $m_{{\text{a}}_1}$ for 2HDM+S models, independent of the type and $tan\beta$ parameter

Fig. 13

Observed and expected 95% CL upper limits on $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1)$ in %, obtained from the combination of the $\mathrm{\mu }\mathrm{\mu }\text{b}\text{b}$ and $\mathrm{\tau }\mathrm{\tau }\text{b}\text{b}$ channels. The results are obtained as functions of $m_{{\text{a}}_1}$ for 2HDM+S Type I (independent of $tan\beta$), Type II $(tan\beta =2.0),$ Type III $(tan\beta =2.0),$ and Type IV $(tan\beta =0.6),$ respectively

Fig. 14

Observed 95% CL upper limits on $\mathcal{B}(\text{H}\to {\text{a}}_1{\text{a}}_1)$ in %, for the combination of the $\mathrm{\mu }\mathrm{\mu }\text{b}\text{b}$ and $\mathrm{\tau }\mathrm{\tau }\text{b}\text{b}$ channels for Type III (upper) and Type IV (lower) 2HDM+S in the $tan\beta$ vs. $m_{{\text{a}}_1}$ parameter space. The limits are calculated in a grid of 5$\text{Ge}\text{V}$ in $m_{{\text{a}}_1}$ and 0.1–0.5 in $tan\beta$, interpolating the points in between. The contours corresponding to branching fractions of 100 and 16% are drawn using dashed lines, where 16% refers to the combined upper limit on Higgs boson to undetected particle decays from previous Run 2 results [16]. All points inside the contour are allowed within that upper limit