Humans are able to simultaneously identify a person and recognize his or her action based on biological motions. Previous work usually treats action recognition and person identification from motions as two separate tasks with different objectives. In this paper, we present an end-to-end framework to perform these two tasks together. Inspired by the recent success of deep recurrent neural networks (RNN) for skeleton based action recognition, we propose a new pipeline to recognize both actions and persons from skeletons extracted by RGBD sensors. The structure includes two subnets and is end-to-end trainable. The former is skeleton transformation, which accommodates viewpoint changes and noise. The latter is multi-task RNN for joint learning and various architectures are explored including a novel architecture that learns the joint probability between the two output variables. Experiments on 3D action recognition benchmark datasets demonstrate the benefits of multi-task learning and our method dramatically outperforms existing state-of-the-art in action recognition.