deep learning is skilled at learning representation from raw data, which are embedded in the semantic space. traditional multimodal networks take advantage of this, and maximize the joint distribution over the representations of different modalities. however, the similarity among the representations are not emphasized, which is an important property for multimodal data. in this paper, we will introduce a novel learning method for multimodal networks, named as semantic similarity learning (ssl), which aims at training the model via enhancing the similarity between the highlevel features of different modalities. sets of experiments are conducted for evaluating the method on different multimodal networks and multiple tasks. the experimental results demonstrate the effectiveness of ssl in keeping the shared information and improving the discrimination. particularly, ssl shows its ability in encouraging each modality to learn transferred knowledge from the other one when faced with missing data. © 2016 acm.