Quick and high-fidelity updates about aerodynamic loads of large-scale structures, from trains, planes, and automobiles to many civil infrastructures, serving under the influence of a broad range of crosswinds are of practical significance for their design and in-use safety assessment. Herein, we demonstrate that data-driven machine learning (ML) modeling, in combination with conventional computational methods, can fulfill the goal of fast yet faithful aerodynamic prediction for moving objects subject to crosswinds. Taking a full-scale high-speed train, we illustrate that our data-driven model, trained with a small amount of data from simulations, can readily predict with high fidelity pressure and viscous stress distributions on the train surface in a wide span of operating speed and crosswind velocity. By exploring the dependence of aerodynamic coefficients on yaw angles from ML-based predictions, a rapid update of aerodynamic forces is realized, which can be effectively generalized to trains operating at higher speed levels and subject to harsher crosswinds. The method introduced here paves the way for high-fidelity yet efficient predictions to capture the aerodynamics of engineering structures and facilitates their safety assessment with enormous economic and social significance.