WAL sequencerTxn > writerTxn (and sometimes writerTxn > sequencerTxn) Data lagging by 2 days in QuestDB 9.2.0

Community
1

Hi Team,
We are facing an issue with WAL tables after importing data into QuestDB.
Issue Summary

  • We observe that sequencerTxn is higher than writerTxn in some WAL tables.

  • At other times, we see the opposite: writerTxn > sequencerTxn.

  • When we re-run the same query after a few minutes, the mismatch sometimes disappears, but then appears again for other tables.

  • Writer counts are much lower than sequencer counts in several tables, which seems to indicate the writer thread is lagging.

  • Because of this, our latest data is delayed by almost 2 days, even though inserts are happening with current timestamps.

Background

  • We imported large datasets from CSV into QuestDB.

  • During import, we disabled WAL, and after the import finished we re-enabled WAL using ALTER TABLE <name> SET WAL ON.

  • Could it be after enabling WAL, these transaction mismatches started appearing?

Environment

  • QuestDB Version: 9.2.0

  • OS: Ubuntu 22.04

  • Instance: AWS EC2

    • CPU: 8 vCPU

    • RAM: 32 GB

    • Storage: GP3 EBS (3000 IOPS, 125 MB/s throughput)

  • CPU Load Average: 5.37, 5.76, 5.94

Problem We Are Trying to Solve
Our current data (latest timestamps) is falling behind by ~2 days, and we suspect WAL replay or writer lag is the reason.
We are unable to clearly understand:

  1. Why sequencerTxn and writerTxn become unaligned

  2. Whether this is normal behavior or an indication of corruption/lag

  3. Why writer is not catching up with sequencer

  4. Whether disabling WAL → bulk import → enabling WAL can cause such issues

  5. How we can fix the WAL lag and data delay

Request
Can someone help explain:

  • What causes sequencerTxn and writerTxn mismatch?

  • Is this expected under load, or does it indicate WAL replay issues?

  • How to bring writer in sync and eliminate the 2-day lag?

  • Any recommended checks or configuration tuning?

Please let me know if more logs or metrics are needed.
Thank you!

2

Let’s pick it up on Slack.