Disabling O3 Engine for Ordered Data Ingestion

Community
,
1

I’m working with a scenario where my data is guaranteed to arrive in strict timestamp order. Therefore, I believe the out-of-order (O3) engine’s functionality is redundant and potentially adds unnecessary overhead in my case.

Is there a way to completely disable the O3 engine, or configure it to bypass its sorting and merging logic, assuming the data is already ordered? I’m looking to optimize ingestion performance by eliminating the O3 overhead.

I am using ILP Over TCP to ingest data.

2

Hi @CS-Aditya-Rawat ,

There isn’t an overhead unless you actually perform an O3 write. In the normal appending case, no splits/merge happen.

But if you use DEDUP, that does have to go through a similar process.

So as long as you avoid DEDUP and don’t perform any O3 writes, you should not notice a performance drop.

P.S. In our testing, DEDUP overhead was approximately 8%, so not crazy.