in recent years, convolutional neural networks (cnns) have achieved remarkable success in various applications such as image classification, object detection, object parsing and face alignment. such cnn models are extremely powerful to deal with massive amounts of training data by using millions and billions of parameters. however, these models are typically deficient due to the heavy cost in model storage, which prohibits their usage on resource-limited applications like mobile or embedded devices. in this paper, we target at compressing cnn models to an extreme without significantly losing their discriminability. our main idea is to explicitly model the output reconstruction error between the original and compressed cnns, which error is minimized to pursuit a satisfactory rate-distortion after compression. in particular, a global error reconstruction method termed ger is presented, which firstly leverages an svd-based low-rank approximation to coarsely compress the parameters in the fully connected layers in a layerwise manner. subsequently, such layer-wise initial compressions are jointly optimized in a global perspective via back-propagation. the proposed ger method is evaluated on the ilsvrc2012 image classification benchmark, with implementations on two widely-adopted convolutional neural networks, i.e., the alexnet and vggnet-19. comparing to several state-of-the-art and alternative methods of cnn compression, the proposed scheme has demonstrated the best rate-distortion performance on both networks.