3D perception is one of the most important tasks of autonomous vehicles. Both methods based on expensive LiDAR and stereo cameras, and methods based on monocular cameras, have achieved great success in 3D object detection from vehicle view. The roadside view, as an important component of the entire intelligent transportation system, has distinct features from the vehicle’s forward view. The 3D object detection from the roadside view has enormous research and application value. However, current research on 3D object detection from roadside view lags far behind the research on 3D object detection from vehicle view. Based on the work of M3D-RPN, we analyze the differences in sample space between roadside view and vehicle view. We fnd that although the post-optimization based on 2D 3D geometric consistency can improve 3D detection performance in the front view of the vehicle, it can reduce the performance of 3D detection in the roadside view. At the same time, to adapt to the characteristics of the roadside view, we propose a novel ray-aware convolution to replace the depth-aware convolution for the vehicle view. Compared to the M3D-RPN, our proposed M3D-RA-RPN improves the performance of monocular 3D object detection and BEV object detection on the Rope3d dataset.