Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition

Abstract

Audio-visual speech recognition (AVSR) can significantly improve performance over audio-only recognition for small or medium vocabularies. However, current AVSR, whether hybrid or end-to-end (E2E), still does not appear to make optimal use of this secondary information stream as the performance is still clearly diminished in noisy conditions for large-vocabulary systems. We, therefore, propose a new fusion architecture-the decision fusion net (DFN). A broad range of time-variant reliability measures are used as an auxiliary input to improve performance. The DFN is used in both hybrid and E2E models. Our experiments on two large-vocabulary datasets, the Lip Reading Sentences 2 and 3 (LRS2 and LRS3) corpora, show highly significant improvements in performance over previous AVSR systems for large-vocabulary datasets. The hybrid model with the proposed DFN integration component even outperforms oracle dynamic stream-weighting, which is considered to be the theoretical upper bound for conventional dynamic stream-weighting approaches. Compared to the hybrid audio-only model, the proposed DFN achieves a relative word-error-rate reduction of 51% on average, while the E2E-DFN model, with its more competitive audio-only baseline system, achieves a relative word error rate reduction of 43%, both showing the efficacy of our proposed fusion architecture.

Keywords: audio-visual speech recognition; decision fusion net; end-to-end recognition; hybrid models; reliability measures.

Figure 1

Audio-visual fusion based on the DFN, applied to one stream of audio and two streams of video features.

Figure 2

Audio encoder (left), video encoder (middle) and reliability measure encoder (right) for both modalities $i\in \mathrm{A},\mathrm{VI}$. The blue blocks are used to align video features with audio features; the turquoise block shows the transformer encoder.

Figure 3

Transformer encoder for both modalities $i\in \mathrm{A},\mathrm{VI}$. The blue block shows the sub-sampling, whereas the turquoise blocks comprise the the transformer encoder.

Figure 4

Transformer decoder (left) and CTC decoder (right) for both modalities $i\in \mathrm{A},\mathrm{VI}$.

Figure 5

DFN fusion topology for E2E model, $type\in s2s,ctc$.

Figure 6

Decision fusion net structure for the hybrid model. The turquoise block indicates the successively repeated layers.

Figure 7

${\mathrm{DFN}}_{ctc}$ (left) and ${\mathrm{DFN}}_{s2s}$ (right). The turquoise blocks indicate the successively repeated layers.

Figure 8

Estimated log-posteriors of sentence S2 for the target state $s_t^{*}$, with additive noise at $-9$ dB. All abbreviations are the same as in Table 3. The whiskers show the maximum and minimum values; the upper and lower bounds of the green blocks represent the respective 25th and 75th percentile; the yellow line in the center of the green block indicates the median.

Figure 9

WER (%) on the test set of the LRS2 corpus in different noise conditions.