Multi-agent systems often face the challenge of adapting to dynamic team composition and variable partial observability, which can hinder the generalization ability of agent policies. This research introduces a novel method, Mutual Information-guided Multi-Agent coordination (MIMA), to address these issues. MIMA utilizes a hierarchical structure that includes a meta-controller, an information extractor, and agents acting as controllers. The meta-controller partitions the team into distinct groups, while the information extractor uses this partition to extract relevant information. The controllers then make decisions based on this information. We propose two objectives based on mutual information to learn individual and group-specific information. The information extractor uses individual information to form inner-group information, addressing the variable partial observability challenge. It also extracts group-specific information to improve the agents' adaptability to scenarios with dynamic team composition. Both types of information guide agents' distributed execution and influence policy updates during centralized training. Our experiments in multi-agent particle environments and StarCraft II micromanagement tasks show that MIMA improves the zero-shot generalization ability by a large margin, demonstrating its effectiveness in handling dynamic team composition and variable partial observability.