We have a table which is partition by HOUR. However, QuestDB is creating multiple partitions for the same hour. It seems questdb is partitioning further by 20GB size. Can someone explain why 20GB is the limit and if there is any way to change this setting in QuestDB ?
Is it your most recent partition that is split into multiple parts? If so, most likely it is the out-of-order merge optimisation. This splits the partition into smaller chunks to reduce how much data gets re-written. On partition rollover, it will be compacted. See: Time Partitions | QuestDB
If not, then can you run table_partitions('table_name'); and post the result, please?
Yes, it is the latest partition that is split. When the partition rolls over (i.e. you insert from 1800 onwards), it will compact.
If you won’t insert more data, then you can use:
ALTER TABLE table_name SQUASH PARTITIONS
As for why it is consistently shaped, we don’t have a ‘max-size’ setting or anything. The logic aims to balance number of splits versus size of split, so if you write data out-of-order, it minimises the amount of data rewritten.
Presumably, that happens to be even 5 minute splits in this case.