In recent years, control under urban intersection scenarios has become an emerging research topic. In such scenarios, the autonomous vehicle confronts complicated situations since it must deal with the interaction with social vehicles timely while obeying the traffic rules. Generally, the autonomous vehicle is supposed to avoid collisions while pursuing better efficiency. The existing work fails to provide a framework that emphasizes the integrity of the scenarios while deploying and testing reinforcement learning(RL) methods. Specifically, we propose a benchmark for training and testing RL-based autonomous driving agents in complex intersection scenarios, which is called RL-CIS. Then, a set of baselines consisting various algorithms are deployed. The test benchmark and baselines provide a fair and comprehensive training and testing platform for the study of RL for autonomous driving in the intersection scenario, advancing RL-based methods for autonomous driving control. The code of our proposed framework can be found at https://github.com/liuyufi123/ComplexUrbanScenarios.