Cognitive control is essential to resolve conflict in stimulus-response compatibility (SRC) tasks. The SRC effect in the current trial is reduced after an incongruent trial as compared with a congruent trial, a phenomenon being termed conflict adaptation (CA). The CA effect is found to be domain-specific, such that it occurs when adjacent trials contain the same type of conflict, but disappears when the conflicts are of different types. Similar patterns have been observed when tasks involve different modalities, but the modality-specific effect may have been confounded by task switching. In the current study, we investigated whether or not cognitive control could transfer across auditory and visual conflicts when taskswitching was controlled. Participants were asked to respond to a visual or auditory (Experiments 1A/B) stimulus, with conflict coming from either the same or a different modality. CA effects showed modality-specific patterns. To account for potential confounding effects caused by differences in task-irrelevant properties, we specifically examined the influence of task-irrelevant properties on CA effects within the visual modality (Experiments 2A/B). Significant CA effects were observed across different conflicts from distinct task-irrelevant properties, ruling out that the lack of cross-modal CA effects in Experiments 1A/B resulted from differences in task-irrelevant information. Task-irrelevant properties were further matched in Experiments 3A/B to examine the pure effect of modality. Results replicated Experiments 1A/B showing robust modality-specific CA effects. Taken together, we provide supporting evidences that modality affects cognitive control in conflict resolution, which should be taken into account in theories of cognitive control.