Abstract: Aiming at the disadvantages of high-power consumption and no portability in the deployment of existing speech emotion recognition system, this paper proposes a design of FPGA speech emotion recognition system based on neural network accelerator. Mel frequency cepstrum coefficient (MFCC) feature extraction of speech was realized on FPGA, which was convenient for recognition. The instruction generation algorithm was designed for the neural network accelerator, and the network model was deployed in the neural network accelerator to realize speech emotion recognition. The main hardware resource consumption of the whole system is 37 078 LUTs and 153 DSPs, which supports the deployment on the mainstream FPGA platform. After testing, the instruction operation error of speech emotion recognition system is less than 0.06, and the output error is less than 0.000 4, which meets the needs of speech emotion recognition.