Yup dropping the index boosts the query a lot! Selecting the count in the interval can returns also more then 200mln of rows so I guess removing it is fine… I don’t even need other complex queries, I just need to stream ticks from quest to my application which is a simulator running on VPS with 12 vCore (I know not too much, so trying to keep things as much efficient as possibile)…
Union is not a solution btw, it’s incredibly slow, amost 30sec to merge 2000 minutes of series, most of the time it goes in timeout.
So I think the final solution will be to make a big table for a single broker and put all symbols there, drop index and use the proper WHERE clause… Thanks a lot for the help!
May I ask you if you can review my snippet to execute queries? I think it’s fine but maybe I might have forgot some important performance consideration ![]()
long startTime = currentTime - (timeframe.millis * (count + 1));
String query = "SELECT timestamp AS tick_time, first(bid) AS open, last(bid) AS close, min(bid) AS low, max(bid) AS high FROM " + symbol.dataSource + " WHERE timestamp >= " + startTime + " AND timestamp <= " + currentTime + "000 AND symbol = '" + symbol.symbol + "' SAMPLE BY " + tf + " ALIGN TO CALENDAR ORDER BY tick_time DESC LIMIT " + count;
System.out.println(query);
int cores = Runtime.getRuntime().availableProcessors();
System.out.println("Number of cores: " + cores);
try (SqlExecutionContext ctx = new SqlExecutionContextImpl(cairoEngine, cores)
.with(AllowAllSecurityContext.INSTANCE, null)) {
try (RecordCursorFactory factory = cairoEngine.select(query, ctx)) {
try (RecordCursor cursor = factory.getCursor(ctx)) {
final Record record = cursor.getRecord();
while (cursor.hasNext()) {
Period period = new Period();
period.period = timeframe;
period.symbol = symbolId;
period.startDate = record.getLong(0) / 1000;
period.endDate = period.startDate + period.period.millis;
period.open = record.getDouble(1);
period.close = record.getDouble(2);
period.low = record.getDouble(3);
period.high = record.getDouble(4);
periods.insert(0, period);
}
}
}
} catch (SqlException e) {
throw new RuntimeException(e);
}