Autonomous robot navigation in unpredictable and crowded environments requires a guarantee of safety and a stronger ability to pass through a narrow passage. However, it’s challenging to plan safe, dynamically-feasible trajectories in real-time. Previous approaches, such as Reachability-based Trajectory Design (RTD), focus on safety guarantee, but the lack of online strategy always makes the robot fail to pass through a narrow passage. This paper proposes to learn a policy that guides the robot to make successful plans using deep Reinforcement Learning (RL). We train a deep network based on the RTD method to create cost functions in realtime. The created cost function is expected to help the online planner optimize the robot’s feasible trajectory, satisfying its kino-dynamics model and collision avoidance constraints. In crowded simulated environments, our approach substantially improves the planning success rate compared to RTD and some other methods.