The rising demand for deploying convolutional neural networks (CNNs) to mobile applications has promoted the booming of compact networks. Two parallel mainstream techniques include network compression and lightweight architecture design. Despite these two techniques can theoretically work together, the naive combination results in dramatic accuracy degradation. In this paper, we present Binarized MobileNet-Sp for mobile applications, by compression-architecture co-design. We first reveal the connection between MobileNets and low-rank decomposition, showing that decomposition-based architecture is not quantization friendly. Then, by adopting the view of sparsity, we propose the Binarized MobileNet-Sp, which significantly enhances the robustness to binarization. Experiments on ImageNet show that the proposed Binarized MobileNet-Sp achieves 61.2% top-1 accuracy, outperforming the naive binarization method by about 10% higher top-1 accuracy. Compared to the Bi-Real net which achieves 56.4% top-1 accuracy on the more heavy-weight and redundant ResNet-18 (which has comparable baseline accuracy with MobileNet in full-precision representation), the Binarized MobileNet-Sp achieves much higher accuracy with a significant reduction in computing complexity.