Abstract: Face to the demand for regional high-density data transmission brought by massive power information collection device in smart park, ultra-dense networking technology was introduced to improve network coverage. The access decision optimization problem was modeled as a long-term throughput optimization problem. The long-term constraint and long-term optimization problem were transformed into a series of short-term problems by Lyapunov optimization. Based on the complex and changeable network environment and the coupling between multi-device access decisions, this paper used the reinforcement learning algorithm with competition coefficient to maximize the weighted utility of throughput and queue drift. Simulation results show that the proposed algorithm can take the stability of data queue and energy deficit queue into account, and much improve throughput with better convergence when meeting the long-term queue stability constraints.