Losing active partitions on power loss

Community
1

Hello!

I noticed that I’m losing data in case of power failure (on a dev system, luckily), and I would like to learn more about what happens. It seems that latest partitions are always either missing or somehow corrupted after crash.

There are no suspended WAL tables. If the last partition exists, querying gives this error: “i.q.c.TableReader open partition failed, partition does not exist on the disk”

table_partitions() shows that there is an active partition with rows in it, but the min/max timestamps are NULL.

When new rows have been inserted, the error is gone and everything works perfectly again. Except significant amount of data has been lost silently…

Is there something I can do to prevent losing the whole partition, or to recover data?

I’m using QuestDB 8.2.2 (with default config) on Windows.

Thanks.

2

Hi @J-T ,

There are two disaster scenarios that you must avoid - power loss, and running out of disk space. Generally, other scenarios will not result in data loss. Power loss or write failures mean that dirty pages in the OS page cache cannot be persisted to disk.

You should avoid these scenarios at all cost and always stop the database gracefully.

You could choose to run the database in sync mode. This will slowdown ingestion and increase CPU usage, but improve the chances of avoiding table corruption.

If you have a multi-node setup, as with Enterprise, then the data loss risk is much lower - only data which has not yet been replicated may be lost.