Multi-scale features are of great importance in modern convolutional neural networks and show consistent performance gains on many vision tasks. Therefore, many plug-and-play blocks are introduced to upgrade existing convolutional neural networks for stronger multi-scale representation ability. However, the design of plug-and-play blocks is getting more complex and these manually designed blocks are not optimal. In this work, we propose PP-NAS to develop plug-and-play blocks based on neural architecture search. Specifically, we design a new search space and develop the corresponding search algorithm. Extensive experiments on CIFAR10, CIFAR100, and ImageNet show that PP-NAS can find a series of novel blocks that outperform manually designed ones. Transfer learning results on representative computer vision tasks including object detection and semantic segmentation further verify the superiority of the PP-NAS over the state-of-the-art CNNs (e.g., ResNet, Res2Net). Our code will be made avaliable at https://github.com/sbl1996/PP-NAS.