Coastal cetaceans are recognized as ecologically important species and have been the target for environmental monitoring programs and conservation strategies. Although supervised models have increasingly been used to better monitor complexity relationships between cetaceans and their habitats, the development of unsupervised learning techniques that have significant advantages in visualizing, grouping and reducing the dimensionality of data has been overlooked. Here, using the unsupervised artificial neural network of self-organizing map (SOM), we examined the intraspecific variation of habitat utilization among three geographically neighboring populations (waters southwest of Hainan Island abbreviated as WS Hainan, Sanniang Bay, and Zhanjiang waters) of the Indo-Pacific humpback dolphin (Sousa chinensis) in the northern South China Sea. The results showed that the population inhabiting in WS Hainan occupied a largest area with scattered patterns comparing to the other two in Sanniang Bay and Zhanjiang waters. The SOM analysis further showed that the examined populations in different waters have distinct habitat characteristics. S. chinensis in WS Hainan was sighted in deeper water with higher salinity, whereas the population in Sanniang Bay inhabited in estuary with lower pH and salinity. More complicated distributed patterns and environmental heterogeneity were observed in Zhanjiang waters, where the dolphins distributed contractively in a small area at the entrance of Zhanjiang Port and dispersed in another large area in Leizhou Bay. Correspondently, part of the sightings in Zhanjiang waters had similar habitat with those in WS Hainan, while others were reported in more inshore, deeper and muddied waters. Based on these findings, we hypothesized that S. chinensis can be divided into two ecotypes: estuarine and non-estuarine. Here, we confirmed that the SOM can well identify the habitat differentiation in S. chinensis, and therefore suggested it is a powerful modeling tool for cetacean monitoring program. Our findings contribute to a specific habitat conservation strategy that one integrated protected area would be better for S. chinensis in Sanniang Bay and Zhanjiang waters, while several small protected areas with connecting corridors were more suitable for population in WS Hainan.