In this paper, we propose Collaborative Adversarial Networks (CAN) to enable simultaneous forward synthesis and backward segmentation of X-ray breast mass image. The proposed CAN consists of a generator (G), an inverter (I) and a discriminator (D). G aims to reconstruct mass images from corresponding annotated masks, while I is trained for mapping images back to accurate segmentation masks. All the obtained mask-image pairs are fed to D trained in an adversarial learning scheme. Through the collaborative adversarial training using a joint loss function, G synthesizes realistic mass images consistent with provided masks and I effectively segments the tumor regions from the images. Qualitative and quantitative evaluations on publicly available INbreast database demonstrate the effectiveness of our model. Furthermore, different from conventional GANs-based methods that can only perform either image synthesis or segmentation, the proposed model can be generalized to other bidirectional image-to-image translation of multimodal medical data.