Recently, the end-to-end system has made significant breakthroughs
in the field of speech recognition. However, this single
end-to-end architecture is not especially robust to the input
variations interfered of noises and reverberations, resulting
in performance degradation dramatically in reality. To alleviate
this issue, the mainstream approach is to use a well-designed
speech enhancement module as the front-end of ASR. However,
enhancement modules would result in speech distortions
and mismatches to training, which sometimes degrades the ASR
performance. In this paper, we propose a jointly adversarial
enhancement training to boost robustness of end-to-end systems.
Specifically, we use a jointly compositional scheme of maskbased
enhancement network, attention-based encoder-decoder
network and discriminant network during training. The discriminator
is used to distinguish between the enhanced features from
enhancement network and clean features, which could guide enhancement network to output towards the realistic distribution.
With the joint optimization of the recognition, enhancement and
adversarial loss, the compositional scheme is expected to learn
more robust representations for the recognition task automatically.
Systematic experiments on AISHELL-1 show that the
proposed method improves the noise robustness of end-to-end
systems and achieves the relative error rate reduction of 4.6%
over the multi-condition training.