End-to-end autonomous driving provides a feasible
way to automatically maximize overall driving system performance
by directly mapping the raw pixels from a front-facing
camera to control signals. Recent advanced methods construct
a latent world model to map the high dimensional observations
into compact latent space. However, the latent states embedded by
the world model proposed in previous works may contain a large
amount of task-irrelevant information, resulting in low sampling
efficiency and poor robustness to input perturbations. Meanwhile,
the training data distribution is usually unbalanced, and the
learned policy is challenging to cope with the corner cases during
the driving process. To solve the above challenges, we present a
SEMantic Masked recurrent world model (SEM2), which introduces
a semantic filter to extract key driving-relevant features
and make decisions via the filtered features, and is trained with
a multi-source data sampler, which aggregates common data and
multiple corner case data in a single batch, to balance the data
distribution. Extensive experiments on CARLA show our method
outperforms the state-of-the-art approaches in terms of sample
efficiency and robustness to input permutations.