Attention-based encoder-decoder (AED) models
have achieved promising performance in speech recognition.
However, because the decoder predicts text tokens (such as
characters or words) in an autoregressive manner, it is difficult
for an AED model to predict all tokens in parallel. This makes
the inference speed relatively slow. In contrast, we propose an
end-to-end non-autoregressive speech recognition model called
LASO (Listen Attentively, and Spell Once). The model aggre-
gates encoded speech features into the hidden representations
corresponding to each token with attention mechanisms. Thus,
the model can capture the token relations by self-attention on
the aggregated hidden representations from the whole speech
signal rather than autoregressive modeling on tokens. Without
explicitly autoregressive language modeling, this model predicts
all tokens in the sequence in parallel so that the inference is
efficient. Moreover, we propose a cross-modal transfer learning
method to use a text-modal language model to improve the
performance of speech-modal LASO by aligning token semantics.
We conduct experiments on two scales of public Chinese speech
datasets AISHELL-1 and AISHELL-2. Experimental results
show that our proposed model achieves a speedup of about 50×
and competitive performance, compared with the autoregressive
transformer models. And the cross-modal knowledge transferring
from the text-modal model can improve the performance of the
speech-modal model.