Installation

Prerequisites

Turbine consumes from and produces to Kafka. You need a broker reachable from your dev machine before installing Turbine — either:

• A local Redpanda or Kafka instance (Redpanda is officially tested; the project ships a docker/dev/docker-compose.yml).
• Any reachable Kafka-compatible broker.

For stateful workloads with durable checkpoints you'll also want an object store later (S3, MinIO, GCS). For your first run, the local filesystem default is fine.

TODO

Document a turn-key local stack: docker compose up for Redpanda and MinIO, plus the matching Turbine(brokers=…, checkpoint_url=…) boilerplate. Reference compose file: docker/dev/docker-compose.yml.

Install

pip install turbine
# or, for development:
maturin develop --features python,state-rocksdb,checkpoint,cluster,cluster-raft

Type-checking note: pyarrow does not ship type stubs. If you import it in your handlers and use a type checker (mypy, pyright, ty…), add pyarrow-stubs to your dev dependencies to silence "missing library stubs or py.typed marker" warnings:

uv add --dev pyarrow-stubs   # or: pip install pyarrow-stubs

Smoke Test

A 10-line app that reads from a topic and prints batch sizes — enough to confirm Turbine is wired up and the broker is reachable.

1. Create a test topic and push a few messages (Redpanda CLI shown):

   docker compose -f docker/dev/docker-compose.yml exec redpanda \
     rpk topic create smoke-test
   docker compose -f docker/dev/docker-compose.yml exec redpanda \
     bash -c 'for i in 1 2 3; do echo "{\"i\":$i}"; done | rpk topic produce smoke-test'
   

2. Save the snippet below as smoke.py and run it:

   from turbine import RecordBatch, Turbine
   
   app = Turbine(brokers="localhost:9092", from_earliest=True)
   
   @app.subscribe("smoke-test")
   def hello(batch: RecordBatch) -> None:
       print(f"Got {batch.num_rows} messages")
   
   app.run()
   

   python smoke.py
   

You should see Got 3 messages after a moment. If you do, Turbine is installed correctly and the broker is reachable. Continue to Quick Start.