Abstract:
Traditional data partitioning strategies suffer from coarse granularity and fail to adapt to complex analytical workloads. On the other hand, existing fine-grained partitioning approaches tend to overfit to historical workloads and struggle to handle workload variations. To address these challenges, this study delves into intelligent data partitioning and proposes an adaptive partitioning algorithm. The algorithm modelled and predicted the system workload, enabling proactive optimization of the partition structure based on predicted workload to enhance query performance. Extensive comparative experiments on the TPC-H dataset demonstrate the effectiveness and superiority of the proposed model, showcasing a 40% reduction in average query response time and over a twofold decrease in average disk I/O in specific scenarios.