Turbine¶
Turbine is a batch-first stream processor for Python, backed by a Rust engine. All Kafka I/O, JSON/Avro serialization, and state management run in Rust; Python is invoked only for business logic on Apache Arrow RecordBatches.
Why Turbine¶
⚡ Performance — Rust handles the hot path (broker I/O, serde, state). Python never sees individual messages — only columnar batches via Apache Arrow zero-copy. Throughput scales with partitions: a single partition typically sustains several hundred thousand messages per second on a stateful pipeline with S3 checkpoints, and aggregate throughput grows roughly linearly as you add partitions. Real-world numbers depend on the cost of the user callback and the workload (decoding, aggregation, sink); the gap between a no-op pipeline and a complex one can be wide.
🐍 Free-threaded Python (3.14t) — Turbine targets the free-threaded build of CPython 3.14, so multiple partitions execute in true parallel within a single process — no GIL serialization between worker threads. PyArrow compute kernels and your user callbacks actually run on multiple cores, which is what makes the per-partition throughput compose as you add partitions instead of flattening on the GIL. Stock CPython works too, but you lose the intra-process scaling.
💾 Stateful processing with checkpoints — Each partition gets its own RocksDB state store. State is periodically checkpointed to durable storage (local filesystem or S3). On restart, the worker restores from the latest snapshot and resumes from the exact offset — no data loss, no reprocessing.
📦 Batch-level acknowledgment — Offsets are committed only after the entire batch has been processed, state flushed, and output produced. A crash mid-batch means the batch is replayed from the last committed offset, giving you at-least-once delivery with minimal duplication.
🛡️ High availability — In Raft cluster mode, partition assignments are replicated across nodes via consensus. If a node crashes, the cluster automatically redistributes its partitions to surviving nodes within a configurable grace period (default 60s). When the node recovers, it rejoins and partitions are rebalanced — no manual intervention needed.
When to Use Turbine¶
Turbine is not the right tool for every streaming workload. If you process messages one at a time with lightweight business logic (routing, enrichment, request/reply), a per-message framework like FastStream is simpler and more appropriate.
Legend: ✅ best choice — 🟡 not optimal — ❌ bad choice.
| Use case | FastStream | Turbine |
|---|---|---|
| Per-message routing / enrichment | ✅ Better — natural async model, low per-message latency | 🟡 Batching overhead unnecessary |
| Request/reply, RPC over broker | ✅ Better — built-in pattern | ❌ Not supported |
| Multi-broker (RabbitMQ, Redis, NATS + Kafka) | ✅ Better — backends included | ❌ Kafka only (NATS JetStream planned) |
| High-volume aggregation / analytics | ❌ Slow — Python serde per message | ✅ Better — Arrow batches in Python, Rust on the hot path |
| Stateful processing (counters, windows, joins) | ❌ No built-in state store; adding one through an external database is possible but at a significant performance cost | ✅ Better — RocksDB per partition + checkpoints |
| Exactly-once delivery (no output duplicates across crashes) | ❌ Not a framework primitive — handlers must be idempotent, or you wire aiokafka transactions by hand | ✅ Better — opt-in per subscribe via processing_guarantee="exactly_once", configurable crash-recovery policy |
| Automatic high availability | 🟡 Kafka consumer groups only | ✅ Better — Raft consensus, automatic failover even for stateful processing |
| Controlled rolling upgrades | 🟡 Depends on broker | ✅ Better — drain/rejoin workflow |
| Columnar workloads (group-by, filter, compute) | ❌ Slow — per-message Python serde | ✅ Better — PyArrow columnar, zero-copy |
Rule of thumb: per-message logic with lightweight transforms → FastStream. High-volume aggregation, stateful processing, or cluster-level HA → Turbine.
A First Example¶
A Turbine handler receives a RecordBatch (Apache Arrow) and an optional TurbineState. The example below counts incoming messages per partition, persisting the running total in RocksDB:
from turbine import RecordBatch, Turbine, TurbineState
app = Turbine(brokers="localhost:9092")
@app.subscribe("events", publish_to="counts")
def aggregate(batch: RecordBatch, state: TurbineState) -> RecordBatch:
total = state.get("total") or 0
total += batch.num_rows
state.put("total", total)
return batch
app.run()
TurbineState exposes get(key), put(key, value), delete(key), get_bytes(key), put_bytes(key, value). State is checkpointed automatically and survives restarts.
See Installation to set up a local environment and walk through a full first application.
Why Turbine is not async¶
User code is plain def process(batch, state) — no async/await anywhere on the user surface. The reasoning:
- Parallelism is already covered. On free-threaded Python (3.14t), each partition runs on its own thread without GIL contention. If one partition blocks on I/O, the others keep going. Async would duplicate that story rather than extend it — and an event loop is single-threaded by default, so it would compete with free-threading instead of complementing it.
- The hot-path I/O isn't in Python. Kafka producer/consumer is in Rust, the state store is local RocksDB. The user never awaits broker traffic; there is no per-message network round-trip to overlap.
- Async has a fixed per-await cost. Coroutine scheduling adds measurable overhead at the rate Turbine calls your handler — a cost paid in every workload, for a benefit that only matters in a narrow one.
The narrow case is per-batch external lookups (Redis enrichment, HTTP enrichment, secondary DB). Today this is solved without async by batching the lookup itself (redis.mget(keys_of_batch), httpx-with-thread-pool, etc.) — the batch model makes those naturally vectorisable. If a real workload ever shows async would actually win, we'd add it as opt-in (async def process()) rather than retrofit the whole API.