Aiming at the problem that the algorithm of rice feature extraction and classification is difficult to determine, this paper improves the LeNet-5 convolutional neural network model and studies its performance on rice sorting. In this paper, the rice original image was preprocessed and the image of single grain rice was extracted to establish a rice image sample database. Then the original LeNet-5 model was improved and tested. Finally, the improved model and several traditional classification methods, 3 lightweight convolutional neural network models were compared. The improved LeNet-5 model has a rice shape selection accuracy of 97.2%, a color selection accuracy of 90.6%, and a processing speed of about 5300 particles·s^-1. The experimental results show that the improved LeNet-5 model can efficiently sort broken rice and chalky rice, and can effectively reduce the workload of preparation before actual sorting.