Different TTL between primary and replica

Community
1

Hello,

Could you confirm if we can configure with a different (longer TTL) dans the primary base ?

For instance, we would like to set TTL at 6 months on the primary and 12 months on the replica. Eventually, we would even store as PARQUET partitions between 12 and 36 months.

Can we replicate only part of the tables ? Typically, could we replicate only the materialized view and not its base table ?

Thank you.

2

Hi @Gabriel ,

Tables and Materialized Views can have separate lifecycles. The lifecycle is the same across primaries and replicas. So your mat view should always be consistent, but it can have a longer history e.g. 30 day TTL vs your base table’s 7 day TTL. Same story with Storage Policy on Enterprise.

Just make sure not to invalidate the view by running a manual DROP or UPDATE against the base table.

3

Thank you for your answer. For our use case, I won’t be able to rely on the replication parameters the.
The architecture we need to set up is:

  • Several instances of QuestDB running on small computer capable of 6 months data retention.
  • A centralized DB gathering data (not every tables) from all small instances with a retention period of 12 months (hot), 36 months (warm).
  • This “aggregated” DB has one or two clones.

At the stage, I understand we shall handle the replication between the dispached instances and one centralized instance by ourselves but that the other clones can be replicated from the first one with the replication mechanism.

4

Yes, for now you’d need to do the edge replication yourself, and we have a quite a few users doing this.

Then your central database would be replicated over shared disk (S3, NFS etc.) with read-replicas reading from that same disk. They can be same region or different regions, and you can migrate the primary to a different instance upon failures.