基于查询批处理优化的自适应数据分区算法研究

ADAPTIVE DATA PARTITIONING ALGORITHM BASED ON QUERY BATCHING OPTIMIZATION

  • 摘要: 大数据背景下传统数据分区策略粒度过粗,不能适应复杂分析负载,而现有细粒度分区对历史负载极度过拟合,无法应对工作负载变化。对智能数据分区进行进一步探索,提出一种自适应分区算法。对分区问题背景下的系统工作负载进行建模及预测,进而实现基于预期负载对分区结构进行主动调优以提高查询性能。在TPC-H数据集上进行大量对比实验,所提模型在特定场景下查询平均响应时间减少40%,平均磁盘I/O减少2倍以上,证明了该模型的有效性和优越性。

     

    Abstract: Traditional data partitioning strategies suffer from coarse granularity and fail to adapt to complex analytical workloads. On the other hand, existing fine-grained partitioning approaches tend to overfit to historical workloads and struggle to handle workload variations. To address these challenges, this study delves into intelligent data partitioning and proposes an adaptive partitioning algorithm. The algorithm modelled and predicted the system workload, enabling proactive optimization of the partition structure based on predicted workload to enhance query performance. Extensive comparative experiments on the TPC-H dataset demonstrate the effectiveness and superiority of the proposed model, showcasing a 40% reduction in average query response time and over a twofold decrease in average disk I/O in specific scenarios.

     

/

返回文章
返回