Unrecoverable storage corruption detected: column version mismatch

Community
1

Hi everyone.
A few days ago I created this table:

CREATE TABLE 'sensor_data' ( 
	index_time TIMESTAMP,
	DT VARCHAR,
	DPB DOUBLE,
	DPH DOUBLE,
	RS DOUBLE,
	HP DOUBLE,
	SPP DOUBLE,
	CH DOUBLE,
	PS1 DOUBLE,
	PS2 DOUBLE,
	PS3 DOUBLE,
	PS4 DOUBLE,
	CIV DOUBLE,
	CDO DOUBLE,
	CDI DOUBLE
) timestamp(index_time) PARTITION BY DAY
DEDUP UPSERT KEYS(index_time);

Then I populated it with 242653 rows.

Today when I start this database I read the message:

2026-04-07T08:47:40.587057Z C i.q.c.p.WriterPool could not open [table=sensor_data~807, thread=35, msg=Unrecoverable storage corruption detected: column version mismatch [table=sensor_data~807, txnVersion=157032, actualFileVersion=156258], errno=0]
2026-04-07T08:47:40.591406668Z 2026-04-07T08:47:40.587205Z C i.q.c.w.ApplyWal2TableJob job failed, table suspended [table=sensor_data~807, error=io.questdb.cairo.CairoException: [0] Unrecoverable storage corruption detected: column version mismatch [table=sensor_data~807, txnVersion=157032, actualFileVersion=156258]
2026-04-07T08:47:40.591411127Z 	at io.questdb.cairo.CairoException.instance(CairoException.java:387)
2026-04-07T08:47:40.591413210Z 	at io.questdb.cairo.CairoException.critical(CairoException.java:80)
2026-04-07T08:47:40.591414668Z 	at io.questdb.cairo.TableWriter.<init>(TableWriter.java:454)
2026-04-07T08:47:40.591416043Z 	at io.questdb.cairo.TableWriter.<init>(TableWriter.java:357)
2026-04-07T08:47:40.591417335Z 	at io.questdb.cairo.pool.WriterPool.createWriter(WriterPool.java:400)
2026-04-07T08:47:40.591418543Z 	at io.questdb.cairo.pool.WriterPool.getWriterEntry(WriterPool.java:446)
2026-04-07T08:47:40.591419752Z 	at io.questdb.cairo.pool.WriterPool.get(WriterPool.java:149)
2026-04-07T08:47:40.591466168Z 	at io.questdb.cairo.CairoEngine.getWriterUnsafe(CairoEngine.java:1264)
2026-04-07T08:47:40.591469377Z 	at io.questdb.cairo.wal.ApplyWal2TableJob.applyWal(ApplyWal2TableJob.java:929)
2026-04-07T08:47:40.591470752Z 	at io.questdb.cairo.wal.ApplyWal2TableJob.doRun(ApplyWal2TableJob.java:989)
2026-04-07T08:47:40.591472085Z 	at io.questdb.mp.AbstractQueueConsumerJob.run(AbstractQueueConsumerJob.java:50)
2026-04-07T08:47:40.591473377Z 	at io.questdb.mp.Worker.run(Worker.java:152)

Even running
ALTER TABLE sensor_data RESUME WAL;
doesn’t fix it.

How can I solve it?
Thank you.

2

Hey @BepTheWolf , did you encounter some sort of failure? Storage died, ran out of disk, power loss etc.?

3

No, nothing strange.
I turned off the PC normally and turned it back on today. I’m testing with a MacPro, so I don’t have any power outages and the disk still has 90GB free.

I turned it off after the program inserted the 242653 rows and maybe there were some aborted transactions left?

with the query
SELECT * FROM wal_tables() WHERE name='sensor_data';
I see

image
image1306×212 14.4 KB

4

That’s strange, if it wasn’t a hard fault, a corruption is unexpected. Did you perhaps hit open file limits? Mac is notorious for being difficult to raise them.

Perhaps you can find some ‘ E ‘ or ‘ C ‘ errors in the logs? Or feel free to DM them to me to take a look. Definitely want to identify a cause here, this is abnormal behaviour.

5

Hi,
If it helps, on April 2nd I loaded the rows in four parts with a test program.
In the first part, the program used ‘INSERT’ because the table was empty.
The other three parts used ‘UPDATE’ for existing rows and some ‘INSERT’ for new rows.
After the test program loaded all the rows into the DB, I shut down the computer.
It was probably still incrementing ‘writerTnx’.

The log for April 2nd is this:

2026-04-02T07:33:58.580408Z A server-main QuestDB 9.3.4. Copyright (C) 2014-2026, all rights reserved.
2026-04-02T07:33:58.580899Z A server-main linux-aarch64 [Vanilla,0, 64 bits, 8 processors]
2026-04-02T07:33:58.580957Z A server-main fs.file-max checked [limit=1048576]
2026-04-02T07:33:58.581030Z A server-main vm.max_map_count checked [limit=262144]
2026-04-02T07:33:58.617485Z A server-main Server config: /root/.questdb/conf/server.conf
2026-04-02T07:33:58.772503Z A server-main Config:
2026-04-02T07:33:58.772592Z A server-main  - http.enabled : true
2026-04-02T07:33:58.772615Z A server-main  - tcp.enabled  : true
2026-04-02T07:33:58.772625Z A server-main  - pg.enabled   : true
2026-04-02T07:33:58.772633Z A server-main  - attach partition suffix: .attachable
2026-04-02T07:33:58.772651Z A server-main  - open database [38930525-eec3-b8e7-03a6-e2bc97464c4a]
2026-04-02T07:33:58.772846Z A server-main  - db root: [path=/root/.questdb/db, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED (SYSTEM COULD BE UNSTABLE)
2026-04-02T07:33:58.772990Z A server-main  - .checkpoint root: [path=/root/.questdb/.checkpoint, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-02T07:33:58.773286Z A server-main  - snapshot root: [path=/root/.questdb/snapshot, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-02T07:33:58.773402Z A server-main  - sql copy input root: [path=/root/.questdb/import, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-02T07:33:58.773564Z A server-main  - sql copy input worker root: [path=/root/.questdb/tmp, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-02T07:33:58.775020Z A server-main  - SQL JIT compiler mode: on
2026-04-02T07:33:58.775050Z A server-main  - configured ram.usage.limit.bytes: 0 (no limit)
2026-04-02T07:33:58.775072Z A server-main  - configured ram.usage.limit.percent: 90
2026-04-02T07:33:58.775091Z A server-main  - system RAM: 9.7 GiB
2026-04-02T07:33:58.775363Z A server-main  - resolved RAM usage limit: 8.7 GiB
2026-04-02T07:33:58.785193Z A server-main Metrics are disabled, health check endpoint will not consider unhandled errors
2026-04-02T07:33:58.798962Z A i.q.c.CairoEngine loading functions from io.questdb
2026-04-02T07:33:59.207760Z A i.q.c.CairoEngine loaded 1039 functions
2026-04-02T07:33:59.637120Z A i.q.TelemetryConfigLogger instance [id=0x0620074186715000000ad2959be769, enabled=false]
2026-04-02T07:33:59.652694Z A http-server listening on 0.0.0.0:9000 [fd=672162382061 backlog=256]
2026-04-02T07:33:59.662702Z A pg-server listening on 0.0.0.0:8812 [fd=697932185841 backlog=64]
2026-04-02T07:33:59.663595Z A tcp-line-server listening on 0.0.0.0:9009 [fd=706522120435 backlog=256]
2026-04-02T07:33:59.705159Z A server-main server is ready to be started
2026-04-02T07:33:59.739971Z E i.q.c.m.MatViewRefreshJob marking materialized view as invalid [view=a_tmp~631, reason=base table does not exist, ts=2026-04-02T07:33:59.739896Z]
2026-04-02T07:33:59.756641Z A server-main data id: 3b0698af-8e58-5b21-42cf-be0d32519b47
2026-04-02T07:33:59.784117Z A server-main enjoy
2026-04-02T07:54:20.489733Z E i.q.g.e.QueryProgress err [id=-1, sql=`SELECT*
FROM(
SELECT
index_time,
	MAX(ColA) AS ColA,
	MAX(ColB) AS ColB,
	MAX(ColC) AS ColC,
	MAX(ColD) AS ColD,
	MAX(ColE) AS ColE,
	MAX(ColF) AS ColF,
	MAX(ColG) AS ColG,
	MAX(ColH) AS ColH,
	MAX(ColI) AS ColI,
	MAX(ColJ) AS ColJ
FROM MyTab_a
WHERE index_time IN '[2023-01-02]'
SAMPLE BY 1h)
PIVOT (
	avg(ColA), 
	avg(ColB), 
	avg(ColC), 
	avg(ColD), 
	avg(ColE), 
	avg(ColF), 
	avg(ColG), 
	avg(ColH), 
	avg(ColI), 
	avg(ColJ) 
)`, principal=admin, cache=false, jit=true, time=7847292, msg=expected FOR, errno=0, pos=468]
2026-04-02T08:46:53.946287Z E i.q.g.e.QueryProgress err [id=-1, sql=`truncare table 
 sensor_data`, principal=admin, cache=false, jit=true, time=1291209, msg=table and column names that are SQL keywords have to be enclosed in double quotes, such as "table", errno=0, pos=9]
2026-04-02T10:11:14.972315Z E i.q.g.e.QueryProgress err [id=-1, sql=`SELECT writerTxn, sequencerTxn FROM wal_tables() WHERE "sensor_data"`, principal=admin, cache=false, jit=true, time=1845750, msg=Invalid column: sensor_data, errno=0, pos=55]
2026-04-02T10:13:09.868199Z E i.q.g.e.QueryProgress err [id=-1, sql=`WHERE index_time >= '2013-03-06T10:57'`, principal=admin, cache=false, jit=true, time=586417, msg=table and column names that are SQL keywords have to be enclosed in double quotes, such as "WHERE", errno=0, pos=0]

The log for April 4th, when I restarted the computer, is this:

2026-04-04T09:22:36.220823Z A server-main QuestDB 9.3.4. Copyright (C) 2014-2026, all rights reserved.
2026-04-04T09:22:36.221865Z A server-main linux-aarch64 [Vanilla,0, 64 bits, 8 processors]
2026-04-04T09:22:36.221933Z A server-main fs.file-max checked [limit=1048576]
2026-04-04T09:22:36.222016Z A server-main vm.max_map_count checked [limit=262144]
2026-04-04T09:22:36.240533Z A server-main Server config: /root/.questdb/conf/server.conf
2026-04-04T09:22:36.548070Z A server-main Config:
2026-04-04T09:22:36.548142Z A server-main  - http.enabled : true
2026-04-04T09:22:36.548162Z A server-main  - tcp.enabled  : true
2026-04-04T09:22:36.548170Z A server-main  - pg.enabled   : true
2026-04-04T09:22:36.548353Z A server-main  - attach partition suffix: .attachable
2026-04-04T09:22:36.548378Z A server-main  - open database [38930525-eec3-b8e7-03a6-e2bc97464c4a]
2026-04-04T09:22:36.549340Z A server-main  - db root: [path=/root/.questdb/db, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED (SYSTEM COULD BE UNSTABLE)
2026-04-04T09:22:36.552019Z A server-main  - .checkpoint root: [path=/root/.questdb/.checkpoint, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-04T09:22:36.556623Z A server-main  - snapshot root: [path=/root/.questdb/snapshot, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-04T09:22:36.557977Z A server-main  - sql copy input root: [path=/root/.questdb/import, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-04T09:22:36.560903Z A server-main  - sql copy input worker root: [path=/root/.questdb/tmp, magic=0x6a656a63, fs=unknown] -> UNSUPPORTED
2026-04-04T09:22:36.570429Z A server-main  - SQL JIT compiler mode: on
2026-04-04T09:22:36.570496Z A server-main  - configured ram.usage.limit.bytes: 0 (no limit)
2026-04-04T09:22:36.570534Z A server-main  - configured ram.usage.limit.percent: 90
2026-04-04T09:22:36.570614Z A server-main  - system RAM: 9.7 GiB
2026-04-04T09:22:36.570905Z A server-main  - resolved RAM usage limit: 8.7 GiB
2026-04-04T09:22:36.593899Z A server-main Metrics are disabled, health check endpoint will not consider unhandled errors
2026-04-04T09:22:36.627321Z A i.q.c.CairoEngine loading functions from io.questdb
2026-04-04T09:22:37.239322Z A i.q.c.CairoEngine loaded 1039 functions
2026-04-04T09:22:37.845198Z A i.q.TelemetryConfigLogger instance [id=0x0620074186715000000ad2959be769, enabled=false]
2026-04-04T09:22:37.862260Z A http-server listening on 0.0.0.0:9000 [fd=723701989634 backlog=256]
2026-04-04T09:22:37.874421Z A pg-server listening on 0.0.0.0:8812 [fd=740881858822 backlog=64]
2026-04-04T09:22:37.875528Z A tcp-line-server listening on 0.0.0.0:9009 [fd=749471793416 backlog=256]
2026-04-04T09:22:37.934900Z A server-main server is ready to be started
2026-04-04T09:22:38.017679Z A server-main data id: 3b0698af-8e58-5b21-42cf-be0d32519b47
2026-04-04T09:22:38.055265Z A server-main enjoy
2026-04-04T09:22:38.171523Z C i.q.c.p.WriterPool could not open [table=sensor_data~807, thread=35, msg=Unrecoverable storage corruption detected: column version mismatch [table=sensor_data~807, txnVersion=157032, actualFileVersion=156258], errno=0]
2026-04-04T09:22:38.171793Z C i.q.c.w.ApplyWal2TableJob job failed, table suspended [table=sensor_data~807, error=io.questdb.cairo.CairoException: [0] Unrecoverable storage corruption detected: column version mismatch [table=sensor_data~807, txnVersion=157032, actualFileVersion=156258]
	at io.questdb.cairo.CairoException.instance(CairoException.java:387)
	at io.questdb.cairo.CairoException.critical(CairoException.java:80)
	at io.questdb.cairo.TableWriter.<init>(TableWriter.java:454)
	at io.questdb.cairo.TableWriter.<init>(TableWriter.java:357)
	at io.questdb.cairo.pool.WriterPool.createWriter(WriterPool.java:400)
	at io.questdb.cairo.pool.WriterPool.getWriterEntry(WriterPool.java:446)
	at io.questdb.cairo.pool.WriterPool.get(WriterPool.java:149)
	at io.questdb.cairo.CairoEngine.getWriterUnsafe(CairoEngine.java:1264)
	at io.questdb.cairo.wal.ApplyWal2TableJob.applyWal(ApplyWal2TableJob.java:929)
	at io.questdb.cairo.wal.ApplyWal2TableJob.doRun(ApplyWal2TableJob.java:989)
	at io.questdb.mp.AbstractQueueConsumerJob.run(AbstractQueueConsumerJob.java:50)
	at io.questdb.mp.Worker.run(Worker.java:152)
]

QuestDB is running on docker

  questdb:
    image: questdb/questdb:9.3.4
    container_name: questdb
    restart: on-failure
    env_file:
      - ./questdb/.env
    ports:
      - 9900:9000
      - 9909:9009
      - 8812:8812
      - 9903:9003
    volumes:
      - ./questdb:/root/.questdb
    environment:
      - QDB_LOG_W_STDOUT_LEVEL=ERROR
      - QDB_LOG_W_FILE_LEVEL=ERROR
      - QDB_LOG_W_HTTP_MIN_LEVEL=ERROR
      - QDB_SHARED_WORKER_COUNT=4
      - QDB_LINE_TCP_WRITER_WORKER_COUNT=4
      - QDB_PG_NET_CONNECTION_TIMEOUT=3600000
      - QDB_PG_RECV_BUFFER_SIZE=16M
      - QDB_PG_SEND_BUFFER_SIZE=16M
      - QDB_TELEMETRY_ENABLED=false
      - QDB_CAIRO_O3_MAX_LAG=5000
      - QDB_CAIRO_O3_MIN_LAG=500
      - JVM_OPTS=-Xmx8g -Xms2g -XX:-MaxFDLimit
6

I tried repeating the operations with a new table, following the same steps, but I no longer encountered any errors.
It’s probably some special condition.

7

When you shut down the machine, did QuestDB first log a magic number? If not, it was unlikely to be a safe shutdown. To be absolutely certain, you can use a system-wide sync to ensure the pages are flush to disk, or cycle a CHECKPOINT (which will run an fsync whilst creating the checkpoint).

8

When you turned off the machine, did QuestDB record a magic number first?
How can I check this?

Otherwise it would hardly have been a safe arrest.
Shutdown was performed normally with the MacBook shutdown procedure.

To be absolutely sure, you can use a system-wide synchronization to ensure that pages are downloaded to disk, or loop through a CHECKPOINT (which will perform an fsync while creating the checkpoint).

Assuming we have a system crash, is it possible to recover the database at least up to the last point where it was written consistently?

9

If you check the server logs, during a safe shutdown, you should see ‘SIGTERM’ received or similar, plus potentially a magic number.

In terms of recovering the state, its tricky. For the type of corruption you’ve reported, its is unlikely, you’d need to restore from a backup. If corruption is confined to WAL, you can usually scrap the backed up WAL and the table will be ok.

10

Okay, I’ve tried several times, and the result is that when I shut down my PC properly, the insert and update queries have completed successfully and everything appears to be fine, but the writerTnx and sequencerTnx counters are actually misaligned, then my table is corrupted.
In some cases, the error is “bulk update failed and will be rolled back,” more often “Unrecoverable storage corruption detected: column version mismatch.”

11

Is there a way to reconstruct the corrupted table from existing files up to the last valid point?

12

Maybe I solved it. My fault.
The problem was here.
In Docker, I had set the data folder to Mac.
Now I’ve moved it to Docker.

image
image1478×1188 205 KB