Abstract: In order to obtain the behavior information in video more efficiently, we propose a human action recognition method based on temporal convolutional neural network and dual-stream convolutional neural network. Multi-layer temporal convolution was used to obtain dynamic information from the video and obtain two-dimensional depth dynamic features. A dual-stream CNN was constructed, and depth dynamic features were used as input to the motion information stream instead of optical flow features. The dual-stream classification scores were fused in a weighted average to obtain a determination of the video action category. The algorithm was tested on public data set UCF101, HMDB51 and NTU-RGBD-60, with the highest accuracy of 94.2%, 70.9% and 89.1% (cross-object experiments). When the accuracy is similar to the classical algorithms, such as ECO and TSM, the average parallel speed is increased by a factor of 2.1 and 3.6 respectively. The proposed algorithm improves the computational efficiency and is more practical.