We are looking at using QDB as a part of a replacement of a part of our pipeline that uses Kafka for both streaming and storage, decoupling the storage part and using QDB for that.
What we are interested in at the moment is documentation around how to scale QDB if we grow and if dynamic scaling is supported. We are an Azure shop if that makes a difference.
Thanks,
Glenn
There are different ways of scaling QuestDB. You can add more CPUs/memory to a single instance and use it for both writes and reads, or you can configure (using QuestDB Enterprise) multiple read-replicas, so you can scale-out reads if needed. Having said so, we see users that process thousands of queries per second on a single instance, so it depends a lot on your use case.
For scaling storage, you can configure QuestDB Enterprise to send older partitions automatically to object storage. Those partitions remain available for querying (with initial increased latency), but they free storage on your local filesystem.
Many thanks for the detailed reply and the options available.
I found QuestDB surprisingly scalable compared to Google BigQuery.
I’m using a relatively low powered orange pi 5 and 8GB of ram.
A huge performance boost was using ZFS with compression, it nearly tripled my read speed and likely similarly the write. It’s using sata SSD storage, so it’s heavily bottlenecked by the I/O.
Hi @Nom , we have benchmarked the database ourselves on RPi 5 and it can indeed do well for in many cases. Our userbase has some substantial RPi 4 deployments too. We also support parquet compression for users who can’t leverage zfs. Some queries are sped up by parquet too by using metadata we don’t ordinarily store in the native format.