Installation

Prerequisites

Free-threaded Python is required

Turbine's per-partition parallelism relies on the free-threaded (no-GIL) CPython build, and the published wheels target it exclusively — cp314t (Python 3.14t). On a standard GIL interpreter there is no compatible wheel and no source fallback on PyPI, so the install fails outright. The t suffix denotes the free-threaded build; uv requests and pins it as 3.14t. Python 3.13's free-threaded build was experimental and is not supported (free-threading became official in 3.14, PEP 779).

• Python 3.14t (free-threaded) — see the warning above.
• Linux x86_64. The wheels are manylinux_2_28_x86_64; macOS, Windows and arm aren't published yet — build from source there (see From source).
• A Kafka-compatible broker, reachable before you run an app — a local Redpanda or Kafka (Redpanda is officially tested; the repo ships docker/dev/docker-compose.yml), or any reachable broker.
• (Optional) an object store (S3, MinIO, GCS) for durable checkpoints. The local-filesystem default is fine for a first run.

These instructions use uv — install it first if you don't have it.

TODO

Document a turn-key local stack: docker compose up for Redpanda and MinIO, plus the matching Turbine(brokers=…, checkpoint_url=…) boilerplate. Reference compose file: docker/dev/docker-compose.yml.

Install

The PyPI package is turbine-stream; the import name is turbine.

In a uv project (recommended)

uv init my-pipeline && cd my-pipeline
uv python pin 3.14t          # free-threaded is mandatory
uv add --prerelease=allow turbine-stream

uv python pin writes a .python-version and downloads the free-threaded interpreter if you don't already have it; uv add then resolves the matching cp314t wheel and pulls the runtime deps (pyarrow, pydantic, python-dotenv). --prerelease=allow is required while Turbine is published only as a pre-release.

Into a standalone virtualenv

uv venv --python 3.14t       # free-threaded interpreter
uv pip install --prerelease=allow turbine-stream

Either way, confirm the GIL is really off before going further — if this prints True, you're on a standard build and Turbine won't scale:

uv run python -c "import sys; print(sys._is_gil_enabled())"   # must print: False

Type-checking note: pyarrow ships no type stubs. If you import it in your handlers and run a type checker (ty, mypy, pyright…), add stubs to silence "missing library stubs or py.typed marker" warnings:

uv add --dev pyarrow-stubs

From source (development, or clustering / other platforms)

The published wheel bundles the python, state-rocksdb and checkpoint features — enough for single-node stateful pipelines with durable checkpoints. The clustering features (cluster, cluster-raft) aren't in it, and there are no wheels for non-Linux/arm platforms, so build from a clone for either case (needs the Rust toolchain and maturin):

uv venv --python 3.14t
maturin develop --features python,state-rocksdb,checkpoint,cluster,cluster-raft

This compiles the Rust core (including RocksDB), so the first build takes several minutes. Inside the repo, prefer the project's just recipes (just py-build) over raw cargo.

Smoke Test

A 10-line app that reads from a topic and prints batch sizes — enough to confirm Turbine is wired up and the broker is reachable.

1. Create a test topic and push a few messages (Redpanda CLI shown):

   docker compose -f docker/dev/docker-compose.yml exec redpanda \
     rpk topic create smoke-test
   docker compose -f docker/dev/docker-compose.yml exec redpanda \
     bash -c 'for i in 1 2 3; do echo "{\"i\":$i}"; done | rpk topic produce smoke-test'
   

2. Save the snippet below as smoke.py and run it:

   from turbine import KafkaBroker, RecordBatch, Turbine
   
   app = Turbine(brokers="localhost:9092", from_earliest=True)
   kafka = KafkaBroker(bootstrap="localhost:9092")
   
   @app.subscribe(kafka.topic("smoke-test"))
   def hello(batch: RecordBatch) -> None:
       print(f"Got {batch.num_rows} messages")
   
   app.run()
   

   uv run python smoke.py
   

You should see Got 3 messages after a moment. If you do, Turbine is installed correctly and the broker is reachable. Continue to Quick Start.