Pedestrians in crowd scenes tend to connect with each other and form coherent groups. In order to investigate the collective behaviors in crowds, plenty of studies have been conducted on group detection. However, most of the existing methods are limited to discover the underlying semantic priors of individuals. By segmenting the crowd image into patches, this paper proposes the Patch-based Topic Model (PTM) for group detection. The main contributions of this study are threefold: (1) the crowd dynamics are represented by patch-level descriptor, which provides a macroscopic-level representation; (2) the semantic topic label of each patch are inferred by integrating the Latent Dirichlet Allocation (LDA) model and the Markov Random Fields (MRF); (3) the optimal group number is determined automatically with an intro-class distance evaluation criterion. Experimental results on real-world crowd videos demonstrate the superior performance of the proposed method over the state-of-the-arts.