Abstract: Aimed at that the existing CNN network model only pays attention to the output of the last layer of the network without fully utilizing the features of the middle layer, which always contains much useful information, a driver distraction detection model is proposed, which extracts the output features of the multi-stage middle network layer end-to-end and integrates with HOG features. The parameter number of our model was only 3.6M. We used L2 weight regularization, Dropout, and batch regularization to improve model performance. The network was verified by the two public datasets State Farm Distracted Driver Detection (SFD3) and AUC Distracted Driver (AUCD2). The accuracy of SDF3 is 99.78%, which is about 3 percentage points higher than those existing methods, and the number of network parameters is reduced by about 95%. The accuracy of AUCD2 is 95.15%, which is about 2 percentage points higher than those existing methods, the number of network parameters is reduced by about 60%.