Multithreaded ingestion and out of order

Community
1

Hi everyone

I’m trying to save some trades data into a daily partitionned QuestDB table.

In order to do as keep time ordering, i sort my data and ingest day by day:

  • Get all the day data needed
  • Ingest it
  • Wait for no WAL txs left
  • Advance to Day +1

The ingestion is done by chunking the daily data and using multithreading (32 threads) to insert it (one long lived sender per thread), in chunks of 500 000 rows.

This was way faster than using a single sender.

However, since data between threads does not arrive in order, this creates large out of order operations, my median amp is 9x.

Is there a design guideline on how to ingest with multi threading to avoid O3 operations?

Waiting for the WAL txs to be 0 seems the right choice but hurts the performance a lot due to the amount of time it takes.

QDB is configured to have 20 threads for WAL processing. Machine has 128 GB RAM and 64 threads.

2

Hi @Pablo , how much data is it?

We’re about to release a new wire protocol, which is dramatically faster. You shouldn’t need more than one connection to saturate network bandwidth, in all likelihood, which then avoids the multi-threading issue.

If you can hold off for a few days - or you can use the QuestDB nightly container plus java-questdb-client >= 1.2.0 to leverage it sooner.

3

Hi Nick,

thanks for your awnser.

It’s about 12 TiB, so indeed it can take a while. I saw the news about QWP being noticeably faster, those are great great news!

Unfortunately, I use rust, and from what i saw the repo was not yet updated.

Is there any chance you can give a rough ballpark on when the update will land in QDB?

4

Server support is already there and working with java-questdb-client 1.2.0. You can use a nightly container to test.

The rest of the clients should ship in the next couple of weeks.