在写入Clickhouse时, clickhouse报错如下:
代码语言:javascript复制 message: Too many partitions for single INSERT block (more than xxx).
The limit is controlled by 'max_partitions_per_insert_block' setting.
Large number of partitions is a common misconception.
It will lead to severe negative performance impact, including slow server startup,
slow INSERT queries and slow SELECT queries. Recommended total number of partitions
for a table is under xxx..xxx. Please note, that partitioning is not intended to
speed up SELECT queries (ORDER BY key is sufficient to make range queries fast).
Partitions are intended for data manipulation (DROP PARTITION, etc)
原因
clickhouse要求每次写入的数据不能跨越特别多的partitions, 具体受max_partitions_per_insert_block控制。
处理办法
一般调大该配置即可。修改users.xml,按照如下添加配置:
思考
一次写入太多的分区应该对clickhouse本身不友好, 所以调整的时候注意不要特别大。至于max_partitions_per_insert_block特别大对clickhouse有哪些坏处大家可以留言评论。