Hi Team,
We are facing an issue with WAL tables after importing data into QuestDB.
Issue Summary
-
We observe that sequencerTxn is higher than writerTxn in some WAL tables.
-
At other times, we see the opposite: writerTxn > sequencerTxn.
-
When we re-run the same query after a few minutes, the mismatch sometimes disappears, but then appears again for other tables.
-
Writer counts are much lower than sequencer counts in several tables, which seems to indicate the writer thread is lagging.
-
Because of this, our latest data is delayed by almost 2 days, even though inserts are happening with current timestamps.
Background
-
We imported large datasets from CSV into QuestDB.
-
During import, we disabled WAL, and after the import finished we re-enabled WAL using
ALTER TABLE <name> SET WAL ON. -
Could it be after enabling WAL, these transaction mismatches started appearing?
Environment
-
QuestDB Version: 9.2.0
-
OS: Ubuntu 22.04
-
Instance: AWS EC2
-
CPU: 8 vCPU
-
RAM: 32 GB
-
Storage: GP3 EBS (3000 IOPS, 125 MB/s throughput)
-
-
CPU Load Average: 5.37, 5.76, 5.94
Problem We Are Trying to Solve
Our current data (latest timestamps) is falling behind by ~2 days, and we suspect WAL replay or writer lag is the reason.
We are unable to clearly understand:
-
Why sequencerTxn and writerTxn become unaligned
-
Whether this is normal behavior or an indication of corruption/lag
-
Why writer is not catching up with sequencer
-
Whether disabling WAL → bulk import → enabling WAL can cause such issues
-
How we can fix the WAL lag and data delay
Request
Can someone help explain:
-
What causes sequencerTxn and writerTxn mismatch?
-
Is this expected under load, or does it indicate WAL replay issues?
-
How to bring writer in sync and eliminate the 2-day lag?
-
Any recommended checks or configuration tuning?
Please let me know if more logs or metrics are needed.
Thank you!