Embodiments of the invention provide a processing method and device video movement classification. The method comprises the steps of reading to-be-identified video frames and extracting light stream images of the video frames; selecting one video frame as the start frame, extracting light stream images in the x direction and the y direction of m video frames following the start frame, and forming the images and an RGB image of the start frame into a sample; inputting the light stream images and the RGB image of the start frame of each sample into an SCNN and a TCNN at the same time to separately obtain convolution projections calculated by the highest convolution layers of the SCNN and the TCNN; according to the convolution projections and a fusion module of multi-scale convolution kernels, acquiring a space-time fusion feature projection of video movement; performing calculation on the space-time fusion feature projection through convolution layers, the maximum pooling layer and a full connection layer, and obtain the classification of the video movement according to the calculation results and a classifier. The device executes the method. The processing method and device for video movement classification can increase accuracy of identification of people movement in complicated scenes.