While existing person search methods have achieved good performance, they require the images used for training contain labels about the identity and bounding box location of each person. However, it is expensive and difficult to manually annotate these labels in the large scale scenario. To overcome this issue, we consider weakly supervised person search. The weakly supervised setting means during training we only know which identities appear in the image set and how many individuals present in each image, without any identity or location information on the image. Facing this challenge, we propose a clustering and patch based weakly supervised learning (CPBWSL) framework, which separately addresses two sub-tasks including pedestrian detection and person re-identification. Particularly, we introduce multiple detectors to provide more detection results as well as fuzzy c-means clustering algorithm to cluster these results and remove low membership ones. Moreover, a patch based learning network is designed to generate different patches and learn discriminative patch features. Extensive experiments on two benchmarks indicate that the proposed weakly supervised setting is feasible and our method can achieve performance comparable to some fully supervised person search methods.