Console & REST API

Web Console

TODO

Document the Turbine web console (React + TypeScript SPA in console/): how to build it (pnpm --prefix console build), how to point it at a node's REST API, and what views it exposes (cluster topology, partition assignments, state browsing, metrics).

REST API

Every Turbine node exposes an HTTP API (default port: 8400, configurable via api_port or TURBINE_API_PORT).

Status & monitoring

Endpoint	Method	Description
/health	GET	Node status, index, worker count
/assignments	GET	Partition assignments for this node
/metrics	GET	Prometheus metrics (see below)

# Check node health
curl http://localhost:8400/health

# List assigned partitions
curl http://localhost:8400/assignments

# Scrape Prometheus metrics
curl http://localhost:8400/metrics

Exposed metrics

Metric	Type	Description
turbine_messages_total	counter	Total messages processed
turbine_batches_total	counter	Total batches processed
turbine_throughput_10s	gauge	Messages/second over the last 10s
turbine_throughput_60s	gauge	Messages/second over the last 60s
turbine_state_ops_total	counter	Total state operations committed (user puts/deletes + offset)
turbine_state_ops_10s	gauge	State operations/second over the last 10s
turbine_state_ops_60s	gauge	State operations/second over the last 60s
turbine_consumer_lag	gauge	Consumer lag (messages behind high watermark)
turbine_poll_seconds	histogram	Time waiting for messages from the broker
turbine_decode_seconds	histogram	JSON → Arrow deserialization time
turbine_process_seconds	histogram	User business logic time
turbine_produce_seconds	histogram	Encoding + sending to the output topic
turbine_commit_seconds	histogram	Writing state + offset to RocksDB
turbine_batch_duration_seconds	histogram	Total batch time (all phases)

All metrics are labeled with worker="work-{sub}-p{partition}".

Cluster management (Raft mode)

These endpoints are available when running in Raft cluster mode. They support ForwardToLeader: if you hit a follower, you get a redirect response with the leader's address.

Endpoint	Method	Description	Body
/cluster/add_learner	POST	Add a node as Raft learner	{"node_id": 4, "addr": "http://10.0.0.4:8400"}
/cluster/change_membership	POST	Promote learners to voters	{"add_voter_ids": [4]}
/cluster/remove_voter	POST	Remove a voter (drain before maintenance)	{"remove_voter_ids": [3]}

Response format:

// Success
{"status": "Ok"}

// Redirect (hit a follower)
{"status": "ForwardToLeader", "leader_id": 1, "leader_addr": "http://10.0.0.1:8400"}

// Error
{"status": "Error", "message": "..."}

State introspection

Query the RocksDB state stores across workers for debugging, alerting dashboards, or operational visibility.

POST /state — per-node query

Query the state stores on a single node. Available in both standalone and cluster mode.

Request body:

Field	Type	Default	Description
worker_id	string \| null	null	Filter to a specific worker. null = all workers on this node.
prefix	string \| null	null	Only return entries whose key starts with this prefix. null = no filter.
limit	int \| null	100	Maximum entries per worker. Capped at 1000.
include_internal	bool \| null	false	Include _turbine_* keys (offsets, timer state).

Response:

{
  "workers": [
    {
      "worker_id": "work-00-p00",
      "topic": "dev-input",
      "partition": 0,
      "truncated": false,
      "entries": [
        {"key": "acc|error_rate_short|tenant_1|region_eu|1700000000000", "value": "{\"sum\":450.2,\"count\":10}"},
        {"key": "cool|error_rate_short|tenant_1|region_eu", "value": "{\"severity\":\"warning\"}"}
      ]
    }
  ]
}

• truncated is true when the worker has more entries than limit. Increase limit or narrow prefix to see more.
• value is returned as UTF-8 text when the bytes are valid UTF-8, otherwise as b64:<base64>.
• Keys starting with _turbine_ are excluded by default (these store offsets and timer metadata). Set include_internal: true to see them.

Examples:

# All state across all workers on this node
curl -X POST http://localhost:8400/state \
  -H 'Content-Type: application/json' -d '{}'

# Accumulators for a specific rule, limiting results
curl -X POST http://localhost:8400/state \
  -H 'Content-Type: application/json' \
  -d '{"prefix": "acc|error_rate_short", "limit": 50}'

# All state for one worker, including internal keys
curl -X POST http://localhost:8400/state \
  -H 'Content-Type: application/json' \
  -d '{"worker_id": "work-00-p00", "include_internal": true}'

POST /cluster/state — cluster-wide query

Fan-out to all Raft members, aggregate their /state responses. Only available in Raft cluster mode.

Same request body as /state. The response includes an additional unreachable field listing node addresses that did not respond within 5 seconds.

Response:

{
  "workers": [
    {"worker_id": "work-00-p00", "topic": "dev-input", "partition": 0, "truncated": false, "entries": [...]},
    {"worker_id": "work-01-p00", "topic": "dev-input", "partition": 1, "truncated": true, "entries": [...]}
  ],
  "unreachable": ["10.0.1.3:8400"]
}

Example:

# All aggregate state across the entire cluster
curl -X POST http://any-node:8400/cluster/state \
  -H 'Content-Type: application/json' \
  -d '{"prefix": "agg|"}'

State key conventions

The alerting app (examples/alerting_app.py) uses the following key layout. Prefix-based queries make it easy to find data per tenant or rule:

Key pattern	Prefix for queries	Content
agg\|{rule}\|{group}\|{window_start}	agg\| or agg\|{rule}\|	JSON aggregate state (sum, count, etc.)
cool\|{rule}\|{group}	cool\| or cool\|{rule}\|	Cooldown state machine (severity, ts)
_turbine_offset	_turbine_	Last committed offset (internal)
_turbine_timer:{id}	_turbine_timer:	Scheduled timer fire times (internal)