Stroke order recovery from off-line multi-stroke characters is a great challenge due to the ambiguity in intersection and connection among strokes. In this paper, we propose a novel framework for handwriting trajectory recovery from off-line handwritten characters by deep neural network (DNN) based ordering prediction and heuristic search, where several DNN modules are designed to extract stroke skeleton, ambiguous zones and starting points, respectively. Then, the ordering matrix $M_o$ among all the stroke segments is calculated by a pointer network (Ptr-Net). Besides, a convolutional neural network (CNN) is used to measure the time adjacency between two arbitrary segments. Based on these necessary measurements, the final writing order is decided via searching for the optimal permutation by heuristic $A^*$ search. Experiments on handwriting images synthesized from the public online handwriting datasets CASIA-OLHWDB1.1, ICDAR13-Online and UNIPEN show that the proposed method yields superior performance on Chinese and English/Arabic handwriting.