Meteorological forecasting is a typical and fundamental problem in the remote sensing field. Although many brilliant forecasting methods have been developed, long-term (a few days ahead) meteorological prediction still relies on traditional Numerical Weather Prediction (NWP) that is not competent for the oncoming flood of meteorological data. To improve the forecasting ability faced with meteorological big data, this article adopts the Automated Machine Learning (AutoML) technique and proposes a deep learning framework to model the dynamics of multi-modal meteorological data along spatial and temporal dimensions. Spatially, a convolution based network is developed to extract the spatial context of multi-modal meteorological data. Considering the complex relationship between different modalities, the Neural Architecture Search (NAS) technique is introduced to automate the designing procedure of the fusion network in a purely data-driven manner. As for the temporal dimension, an encoder-decoder structure is built to exhaustively model the temporal dynamics of the embedding sequence. Specializing for the numerical sequence representation transformation, the multi-head attention module endows the proposed model with the ability to forecast future data. Generally speaking, the whole framework could be optimized with the standard back-propagation, yielding an end-to-end learning mechanism. To investigate its feasibility, the proposed model is evaluated with four typical meteorological modalities including temperature, relative humidity, and two components of wind, which are all restricted under the region whose latitude and longitude range from to N and E to E, respectively. Experiments on two datasets with different resolutions verify that deep learning is effective as an operational technique for the meteorological forecasting task.