In realistic environments, speech is usually interfered by various
noise and reverberation, which dramatically degrades the performance
of automatic speech recognition (ASR) systems. To alleviate
this issue, the commonest way is to use a well-designed speech
enhancement approach as the front-end of ASR. However, more
complex pipelines, more computations and even higher hardware
costs (microphone array) are additionally consumed for this kind of
methods. In addition, speech enhancement would result in speech
distortions and mismatches to training. In this paper, we propose
an adversarial training method to directly boost noise robustness
of acoustic model. Specifically, a jointly compositional scheme of
generative adversarial net (GAN) and neural network-based acoustic
model (AM) is used in the training phase. GAN is used to generate
clean feature representations from noisy features by the guidance of
a discriminator that tries to distinguish between the true clean signals
and generated signals. The joint optimization of generator, discriminator
and AM concentrates the strengths of both GAN and AM for
speech recognition. Systematic experiments on CHiME-4 show that
the proposed method significantly improves the noise robustness of
AM and achieves the average relative error rate reduction of 23.38%
and 11.54% on the development and test set, respectively.