Understanding the geological factors behind the physical and elastic properties of marine sediments and unconsolidated rock is essential for the interpretation of geophysical measurements, hazard assessment, and ocean engineering applications. Core and well logging data from the six drilling sites of the Ocean Drilling Program/International Ocean Discovery Program (ODP/IODP) were used to analyze the rock physical characteristics in the South China sea. The depositional environment plays a significant role in affecting the physical properties of marine sediments. The sediments deposited under shallow water conditions show a higher velocity than the basin, slope, and deeper shelf carbonate deposits. Moreover, the non-depositional hiatus along the Oligocene-Miocene boundary displays a notable control on the variation of rock physical properties. It is found that the lithofacies and physical compaction remarkably influence the elastic characteristics of P-impedance and Vp/Vs ratio. The calcareous-rich sediment and ooze have very low P-impedance and high Vp/Vs ratio, whereas the siltstone and coarse sand present high P-impedance and low Vp/Vs ratio characteristics. With the enhancement of the consolidation degree, the Vp/Vs ratio significantly decreases from 6 to less than 2, suggesting that the shear wave velocity is highly sensitive to physical compactions. The basalt at site U1431 is considerably lower in its P-wave velocity than that at the site of U1433, which is probably caused by the intense fracturing occurring at the site of U1431 associated with different tectonic environments. We establish the link between geological factors and elastic characteristics of marine sediments of SCS, laying the foundation for characterizing depositional environments, lithofacies, and compaction degrees using geophysical measurements.