In this paper, we aim to automatically identify and track the interesting social event from vast amounts of social media data. However, there are two existing challenges: 1) how to model multimodal social event data over time and visualize the topic evolution and 2) how to alleviate the tracking drift problem to boost social event tracking accuracy. We propose a novel online multimodal multiexpert learning algorithm for social event tracking. Compared with existing methods, the proposed model has several advantages: First, it has a nonparametric online multimodal tracking module, which is able to not only automatically learn the number of topics from data over time, but also exploit the multimodal property of the social event. Second, it adopts a novel multiexpert minimization restoration scheme and allows the tracked model to evolve backwards to undo undesirable model updates, which helps alleviate the model drift problem of social event tracking. Third, it is able to not only effectively track the multimodal social event, but also automatically exploit the topic evolution of the social event for a deep understanding with multimodal topics. To evaluate the proposed model, we collect a real-world dataset for research on social event tracking with multimodality information. We have conducted extensive experiments, and both qualitative and quantitative evaluation results have demonstrated the effectiveness of the proposed model.