pedestrian detection based on the combination of convolutional neural network (cnn) and traditional handcrafted features (i.e., hog+luv) has achieved great success. in general, hog+luv are used to generate the candidate proposals and then cnn classifies these proposals. despite its success, there is still room for improvement. for example, cnn classifies these proposals by the fully connected layer features, while proposal scores and the features in the inner-layers of cnn are ignored. in this paper, we propose a unifying framework called multi-layer channel features (mcf) to overcome the drawback. it first integrates hog+luv with each layer of cnn into a multi-layer image channels. based on the multi-layer image channels, a multi-stage cascade adaboost is then learned. the weak classifiers in each stage of the multi-stage cascade are learned from the image channels of corresponding layer. experiments on caltech data set, inria data set, eth data set, tud-brussels data set, and kitti data set are conducted. with more abundant features, an mcf achieves the state of the art on caltech pedestrian data set (i.e., 10.40% miss rate). using new and accurate annotations, an mcf achieves 7.98% miss rate. as many non-pedestrian detection windows can be quickly rejected by the first few stages, it accelerates detection speed by 1.43 times. by eliminating the highly overlapped detection windows with lower scores after the first stage, it is 4.07 times faster than negligible performance loss.