Online interactive behaviors between Web users often make some social media contents go viral. The popularity of social media contents can help us understand public interest and attention behind user in- teractions, thus popularity prediction of online contents has become a key task in social media analytics and can facilitate many applications in different domains. However, it is a difficult task for two main reasons. Firstly, popularity can be affected by many factors such as user, text content and time. Secondly, social media data is often noisy, which may degrade the performance of the prediction model. To over- come these difficulties, in this paper, we design a deep learning based popularity prediction model, which extracts and fuses the rich information of text content, user and time series in a data-driven fashion. To deal with the noise in social media data, we incorporate attention mechanism to focus on more informa- tive parts and suppress noisy ones. Experiments on real world datasets demonstrate the effectiveness of our proposed model.