To our knowledge, optical 4f systems have been widely used as a convolutional layer to perform convolutional computation in free-space optical neural networks (ONNs), which makes ONNs too bulky to be easily applied to miniaturized smart systems. Hence, we propose a compact lensless optoelectronic convolutional neural network (LOE-CNN) architecture in which a single optimized diffractive phase mask acts as an analog convolution processor to perform convolutional operation without a Fourier lens or lenslet array. We demonstrate that this LOE-CNN can be functionally comparable to existing electronic counterparts in classification performance, achieving a classification accuracy of 98.07% and 95% over the Modified National Institute of Standards and Technology dataset in simulation and experiment, respectively, which not only opens new application prospects for free-space ONNs based on a compact single-chip convolution processor, but also facilitates the development of ONN-based smart devices.