Recognize an object and detect a good grasp in unstructured scenes is still a challenge. In this paper, the problem of detecting robotic grasps is expressed by a two-point representation in an unstructured scene with an RGB-D camera. A deep Convolutional Neural Network is designed to predict good grasps in real-time on GTX1080, with using region proposal techniques. A contribution of this work is our proposed network framework can perform classification, location and grasp detection simultaneously so that in a single step, it not only recognizes the category and bounding-box of the object, but also finds a good grasp line. Besides, in training process, we minimize a multi-task loss objective function of object classification, location and grasp detection in order to train the network endto-end. Our experimental evaluation on a real robotic manipulator demonstrates that the robotic manipulator can fulfill the grasping task effectively.