## Summary
Native SQL drivers with a high-performance binary protocol for query… results, replacing the PostgreSQL wire protocol for applications that need maximum throughput. Uses the same binary protocol as the new ingestion layer (#105). The wire format is Arrow IPC compatible — received bytes are directly usable as Apache Arrow record batches with zero conversion overhead on the client.
## Current State
- Queries use PostgreSQL wire protocol
- Text-based encoding adds serialization overhead
- Limited by PG protocol constraints
- Clients must parse text, convert types, and build columnar arrays — redundant work
## Design Goal: Zero Conversion Anywhere
The QuestDB Wire Protocol (QWP) encodes query results as Arrow IPC record batches on the wire. This is a deliberate design choice: the fastest possible data delivery means **no conversion at any point in the path**.
```
QuestDB server Client
┌────────────┐ ┌────────────────────┐
│ Query │ │ │
│ engine │ │ Received bytes │
│ │ │ Arrow IPC on wire │ ═══════════ │
│ ▼ │ ─────────────────────► │ ARE the Arrow │
│ Serialize │ │ RecordBatch │
│ to Arrow │ │ │ │
│ IPC │ │ ├─► .to_numpy() │ zero-copy
└────────────┘ │ ├─► .to_pandas() │ zero-copy
│ ├─► polars.from_ │ zero-copy
│ │ arrow() │
│ └─► torch.from_ │ zero-copy
│ numpy() │
└────────────────────┘
```
The server does one serialization (internal format → Arrow IPC). The client does none. The received network buffer is memory-mapped directly as Arrow arrays. Column data pointers reference the receive buffer — no parsing loop, no field-by-field deserialization, no intermediate representation.
This matters because:
- Arrow IPC is the lingua franca of the data ecosystem — Pandas, Polars, DuckDB, Spark, PyTorch, TensorFlow all consume Arrow natively
- Any Arrow-aware tool can consume QWP results without a QuestDB-specific driver implementation
- The conversion cost that normally sits in every client driver is eliminated entirely
## What This Enables
| Feature | Description |
|---------|-------------|
| **Arrow IPC wire format** | Query results are Arrow record batches on the wire — no client-side conversion |
| **Native drivers** | Purpose-built drivers for Python, Java, Go, etc. — thin wrappers, not parsers |
| **True zero-copy** | Client memory-maps received bytes directly as Arrow arrays |
| **Unified protocol** | Same binary format for both ingestion (#105) and egress |
| **Cursor-based streaming** | Stream large result sets as a sequence of Arrow record batches without loading everything into memory |
| **Ecosystem compatibility** | Any Arrow-aware tool (Polars, DuckDB, Spark, Pandas) can consume results without custom integration |
## Benefits
- **Fastest possible egress** — One serialization on the server, zero on the client. No conversion overhead anywhere.
- **Lower bandwidth** — Binary columnar format vs text-based PG protocol
- **Consistent API** — Single protocol for read and write operations
- **Better type fidelity** — Arrow types preserve QuestDB semantics without text conversion edge cases
- **Thin drivers** — Client drivers become Arrow IPC readers with QWP framing, not full parsers. Less code, fewer bugs, easier to maintain across languages.
## Use Cases
### High-frequency data retrieval
Trading systems, real-time dashboards, and monitoring applications that poll QuestDB frequently benefit from the reduced serialization overhead and compact binary format.
### Bulk data export and ETL pipelines
Large-scale data extraction for warehousing, compliance reporting, or analytics. Cursor-based streaming of Arrow record batches keeps memory bounded even for multi-billion row exports. Downstream tools (Spark, Polars, DuckDB) consume the batches directly.
### ML training data pipeline
Native drivers are the egress foundation for ML training workflows in physical AI and other domains. Since the wire format is Arrow IPC, the path from database to tensor is zero-copy end to end:
```python
import questdb
import torch
from torch.utils.data import DataLoader, IterableDataset
class QuestDBDataset(IterableDataset):
def __init__(self, client, episode_query, data_query):
self.client = client
self.episode_query = episode_query
self.data_query = data_query
def __iter__(self):
# Find matching episodes
episodes = self.client.query(self.episode_query)
random.shuffle(episodes)
for ep in episodes:
# Stream episode data as Arrow record batches
# Wire bytes ARE Arrow — no conversion
# Compound sort keys (#119) make per-episode reads sequential
batches = self.client.query_streaming(
self.data_query,
params={"episode_id": ep.episode_id},
batch_size=1024
)
for batch in batches:
# Arrow → NumPy → PyTorch (zero-copy all the way)
positions = torch.from_numpy(batch.column("positions").to_numpy())
velocities = torch.from_numpy(batch.column("velocities").to_numpy())
torques = torch.from_numpy(batch.column("torques").to_numpy())
yield {"positions": positions, "velocities": velocities, "torques": torques}
dataset = QuestDBDataset(
client=questdb.connect("questdb://central:9000"),
episode_query="""
SELECT episode_id FROM episodes
WHERE outcome = 'success' AND model_version = 'v2.3'
""",
data_query="""
SELECT ts, positions, velocities, torques
FROM joint_telemetry
WHERE episode_id = :episode_id
ORDER BY ts
"""
)
dataloader = DataLoader(dataset, batch_size=32, num_workers=4)
for batch in dataloader:
loss = model(batch)
loss.backward()
```
Why this matters:
- **No export step** — train directly against QuestDB, no intermediate Parquet files
- **Always fresh** — new episodes from today's robot runs are available in the next training epoch
- **No storage duplication** — data lives in QuestDB, not copied to a separate training store
- **Zero-copy end to end** — Arrow IPC on wire → Arrow RecordBatch → NumPy view → PyTorch tensor, no serialization at any step
- **Streaming** — cursor-based delivery keeps memory bounded regardless of dataset size
- **Parallelizable** — multiple DataLoader workers query different episodes concurrently
For multi-modal training data (scalar telemetry + LiDAR + camera), this driver provides the scalar egress layer. File resolution is handled by the multi-modal data management client (#122), which uses the native driver for metadata queries and fetches files from object storage.
### Foundation for distributed queries
Database links (#110) use QWP for remote-to-remote communication. Arrow IPC on the wire means the receiving QuestDB instance can process remote result batches without deserialization — it reads the Arrow buffers directly into its query engine.
## Related
- #105 — Binary ingestion protocol (same binary format, unified for ingestion and egress)
- #119 — Compound sort keys (efficient per-episode, per-series reads that the driver streams)
- #122 — Multi-modal data management (uses native driver for metadata queries, extends with file resolution)
- #110 — Database links (uses QWP/Arrow IPC for remote queries — zero-copy between instances)
- Supersedes #51 (Apache Arrow ADBC) — QWP delivers the same zero-copy goals with tighter QuestDB integration, and the Arrow IPC wire format provides equivalent ecosystem compatibility