Abstract: We propose a video commands learning framework based on multi-stage atrous pyramid network(MS-APN)for generating robot manipulation instructions from untrimmed videos.Specifically,we introduced an atrous convolution pyramid module to capture multi-scale action features and a multi-stage architecture to refine the segmentation results.The untrimmed video was divided into a series of video segments,and action features were extracted.We applied the object detection model to extract the object features,and they were fused with the action features for inputting into two classifiers to recognize the subject and patient object.A command quadruplet was defined to represent robot commands.Experiments conducted on the MPII Cooking 2 dataset show that the accuracy of the action segmentation,object classification,and robot commands generation reach 84.1％,76.5％,62.4％,respectively.And we successfully deploy our system on a Baxter robot for further verifying the effectiveness of our framework.