Swarm systems with simple, homogeneous and autonomous individuals can efficiently accomplish specified complex tasks. Recent works have shown the power of deep reinforcement learning (DRL) methods to learn cooperative policies for swarm systems. However, most of them show poor adaptability when applied to new environments or tasks. In this paper, we propose a novel semantic perception swarm policy with DRL for distributed swarm systems. This policy implements innovative semantic perception, which enables agents to under- stand their observation information, yielding semantic information, to promote agents’ adaptability. In particular, semantic disentangled representation with posterior distribution and semantic mixture representation with network mapping are realized to represent semantic information of agents’ observations. Moreover, in the semantic representation, heterogeneous graph attention network is adopted to effectively model individual-level and group-level relational information. The distributed and transferable swarm policy can perceive the information of uncertain number of agents in swarm environments. Various simulations and real-world experiments on several challenging tasks, i.e., sheep food collection and wolves predation, demonstrate the superior effectiveness and adaptability performance of our method compared with existing methods.