The instability of continental slopes damages marine engineering equipment, such as submarine pipelines, resulting in the generation of tsunamis, which endangers the safety of nearshore personnel. Therefore, research on the instability of continental slopes where submarine landslides usually occur is crucial to the risk evaluation of deepwater drilling. Previous studies were mainly based on simplified 2D and 3D models, which extend the 2D model applied on submarine slopes with complex topography. In this study, a numerical model with bathymetric data from the Qiongdongnan Basin was established. Furthermore, 3D slope stability analysis and static and dynamic analyses were conducted. The static analysis found two discussions where slopes are most likely to occur. Through the analysis of different seismic forces, the dynamic result showed that an instability area is added to the two positions where the static analysis is unstable. Topography scatters and transmits seismic waves and controls the accumulation and diffusion of seismic energy. 3D calculations and analysis revealed that the direction of slope instability is closely related to terrain inclination, slope, terrain effect, and terrain curvature. Data showed that instability situations could not be derived from a single direction or profile data. Such situations are an important factor in slope stability analysis and are critical to the prediction and evaluation of marine geological disasters.