Control Plane API

The control plane holds the cluster’s metadata (tenants, namespaces, streams, caches and nodes) behind a REST API, decides shard placement, and publishes the assignment feed brokers follow. It is never on the data path. This page documents its endpoints and how brokers and operators use them.

Every route is plain HTTP and JSON under /v1. The control plane also serves its OpenAPI document at /v1/openapi.json. It serves no UI for it; see Viewing the API document.

Authentication and Token Exchange (HTTP)

Felix uses upstream OIDC JWTs for authentication and exchanges them for tenant-scoped Felix tokens. Brokers validate Felix tokens locally.

POST /v1/tenants/{tenant_id}/token/exchange

Exchange an upstream OIDC token for a Felix token.

Request:

POST /v1/tenants/{tenant_id}/token/exchange
Authorization: Bearer <oidc_jwt>
Content-Type: application/json

{
  "requested": ["stream.publish", "stream.subscribe", "cache.read"],
  "resources": ["namespace:t1/payments", "stream:t1/payments/orders/*"]
}

Response:

{
  "felix_token": "<jwt>",
  "expires_in": 900,
  "token_type": "Bearer",
  "refresh_token": "<opaque>",
  "refresh_expires_in": 2592000
}

requested and resources narrow the permissions RBAC grants and never widen them, as a cross product of actions and resources. permissions narrows by action:object pairs instead, each on its own, so a token can subscribe to one stream and publish to another:

{
  "requested": [],
  "permissions": [
    "stream.subscribe:stream:t1/rooms/a",
    "stream.publish:stream:t1/rooms/b"
  ]
}

permissions must come with "requested": [] and no non-empty resources, or the request is a 400; a pair that does not parse is a 400 too. The empty requested makes an older control plane, which ignores permissions, refuse rather than mint full rights. With the Raft store a pair narrowing is a 409 until every member supports it. If nothing is left, the exchange returns 403. audience picks who the token is for: felix-broker (the default) or felix-controlplane for this API. A token is accepted by one of the two, never both.

actor_token (RFC 8693) is for a gateway that will delegate the token it is exchanging for: its own felix-controlplane token, holding token.delegate:tenant:{tenant_id}. The broker token then carries may_act: {"sub": "<gateway principal>"}, and so does every token refreshed from it, and only that gateway can pass it to /token/delegate. actor_token_type is optional and must be urn:ietf:params:oauth:token-type:jwt. An actor_token with "audience": "felix-controlplane" is a 400. One that does not verify for the tenant (bad signature, expired, another tenant or audience), lacks token.delegate, was issued before a revocation of the gateway, or belongs to a gateway current RBAC no longer grants token.delegate is a 403 with code actor_refused. /token/delegate answers the same code when the caller’s own token was issued before a revocation of it or RBAC no longer grants it token.delegate. Either way, mint a new gateway token and retry:

{"code": "actor_refused", "message": "invalid actor token", "request_id": null}

That code means the gateway’s own token was refused. A user who lacks permissions is still a 403 with code forbidden, so a gateway can tell the two apart by code without reading message. With the Raft store it is a 409 until every member supports it.

A delegating gateway gets the broker token its own shared connections authenticate with by sending {"requested": [], "permissions": []}. For a principal that holds token.delegate:tenant:{tenant_id}, that mints a connection-only token: the gateway’s sub and no permissions, so it opens a connection and can do nothing on it. Refreshing it checks token.delegate again. For anyone else, asking for nothing is still a 403 with code forbidden, as is a gateway’s ordinary exchange that ends up empty.

POST /v1/tenants/{tenant_id}/token/refresh

Trade a refresh token for a new Felix token without going back to the IdP.

POST /v1/tenants/{tenant_id}/token/refresh
Content-Type: application/json

{ "refresh_token": "<opaque>" }

The answer has the same shape as the exchange’s. A refresh token is single-use: each refresh returns its replacement. It lasts FELIX_REFRESH_TOKEN_TTL_SECONDS (30 days by default). The token keeps the audience its exchange chose, and naming a different audience is a 400.

POST /v1/tenants/{tenant_id}/token/delegate

An RFC 8693 token exchange for a gateway acting for its users: trade a user’s broker token for one that names the caller as its actor, so a broker that binds tokens to client certificates accepts it on the gateway’s certificate.

POST /v1/tenants/{tenant_id}/token/delegate
Authorization: Bearer <caller's felix-controlplane token>
Content-Type: application/json

{
  "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
  "subject_token": "<user's felix-broker token>",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "permissions": ["stream.subscribe:stream:t1/app/alice-feed"]
}

{
  "access_token": "<jwt>",
  "issued_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_type": "Bearer",
  "expires_in": 900
}

The caller needs token.delegate:tenant:{tenant_id}; tenant.manage does not imply it. The subject token must be a broker token of the same tenant that is not already delegated, and its may_act must name the caller: it has to come from an exchange (or a refresh of one) that sent the caller’s token as actor_token. A subject token with no may_act is refused unless the control plane runs with FELIX_CONTROLPLANE_DELEGATE_UNBOUND_TOKENS=true. The result has the subject token’s sub, act: {"sub": "<caller>"}, its permissions or the subset permissions keeps (never wider), and expires no later than the subject token. subject_token_type and permissions are optional. No refresh token is issued.

400 for a wrong grant_type or subject_token_type or a malformed permission, 401 for a missing or invalid caller token, 403 when the caller lacks the permission, the subject token does not verify, is already delegated or was not minted for the caller, or nothing is left after narrowing.

Configuring Allowed IdPs

IdP allowlists are stored per tenant in the control plane database (idp_issuers table) and can be managed via the admin HTTP endpoints below (or directly in the store for tests/dev).

Required fields:

issuer (iss)
audiences (allowed aud values)
subject_claim (default sub)
optional groups_claim
either discovery_url or jwks_url

Admin API: IdP Issuers

IdP issuer admin endpoints require tenant.manage on tenant:{tenant_id}. That is enough to add an issuer no other tenant trusts. Changing an existing issuer’s keys (jwks_url, discovery_url), audiences or claim_mappings, registering an issuer another tenant already trusts, and deleting an issuer also need tenant.manage:cluster:*, because those settings decide which principals and groups the issuer’s tokens become.

Create or update an issuer for a tenant:

POST /v1/tenants/{tenant_id}/idp-issuers
Content-Type: application/json

{
  "issuer": "https://login.microsoftonline.com/<tenant>/v2.0",
  "audiences": ["api://felix-controlplane"],
  "discovery_url": null,
  "jwks_url": null,
  "claim_mappings": {
    "subject_claim": "sub",
    "groups_claim": "groups"
  }
}

Discovery and JWKS URLs must be https (plain http only on a loopback host, unless FELIX_CONTROLPLANE_OIDC_ALLOW_INSECURE_HTTP=true), and the issuer must not contain #.

Delete an issuer:

DELETE /v1/tenants/{tenant_id}/idp-issuers/{issuer}

Admin API: RBAC

RBAC endpoints are split by capability:

GET /v1/tenants/{tenant_id}/rbac/policies -> requires rbac.view
GET /v1/tenants/{tenant_id}/rbac/groupings -> requires rbac.view
POST /v1/tenants/{tenant_id}/rbac/policies -> requires rbac.policy.manage
POST /v1/tenants/{tenant_id}/rbac/groupings -> requires rbac.assignment.manage
DELETE /v1/tenants/{tenant_id}/rbac/policies -> requires rbac.policy.manage (body: the rule, as for POST; 404 if absent)
DELETE /v1/tenants/{tenant_id}/rbac/groupings -> requires rbac.assignment.manage (body: the grouping, as for POST; 404 if absent)

Removing a policy or a grouping also revokes the broker tokens of everyone the rule reached, so their open streams end and they authenticate again under the rules as they are now. A rule that reached an IdP group revokes the whole tenant, and so does deleting an IdP issuer or narrowing or replacing what it trusts (an audience dropped, the subject or groups claim changed, or jwks_url or discovery_url changed; adding an audience revokes nobody). The change and its revocation are one write, so a failed request changed nothing and is safe to retry. refresh-tokens/revoke with "principal_id": "*" revokes a whole tenant by hand. See Revoking a principal.

GET /v1/principal-revocations/changes?since=<n> -> requires node.view:cluster:*. The feed brokers poll: {"items": [{"seq", "tenant_id", "principal_id", "revoked_at"}], "next_seq"}, each principal once at its latest revocation, principal_id * for a whole tenant.

Admin API: Refresh-token revocation and signing keys

Both require tenant.manage on tenant:{tenant_id}.

POST /v1/tenants/{tenant_id}/refresh-tokens/revoke with {"principal_id": "..."} ends every refresh chain the principal holds and revokes its broker tokens: brokers end the streams they opened within about one sync interval. 409 when some control-plane member predates revocations; the call is safe to repeat.
GET /v1/tenants/{tenant_id}/signing-keys lists the current key and the keys that only verify.
POST /v1/tenants/{tenant_id}/signing-keys stages a new key: published, not signing.
POST /v1/tenants/{tenant_id}/signing-keys/{kid}/activate makes it sign.
DELETE /v1/tenants/{tenant_id}/signing-keys/{kid} retires a key that no longer signs (409 for the current one).

The waits between the steps are in the auth design doc’s rotation section.

Every admin endpoint needs a token minted with "audience": "felix-controlplane".

Scope is enforced server-side. Callers can only mutate rules/assignments within their own RBAC scope (tenant/namespace/stream/cache).

Canonical object grammar for RBAC policy payloads:

tenant:{tenant_id}
namespace:{tenant_id}/{namespace} or namespace:{tenant_id}/*
stream:{tenant_id}/{namespace}/{stream} or stream:{tenant_id}/{namespace}/*
cache:{tenant_id}/{namespace}/{cache} or cache:{tenant_id}/{namespace}/*
cache:{tenant_id}/{namespace}/{cache}/{key} or .../{key_prefix}*, with cache.read or cache.write only. The namespace and cache must be named, and the key may end in one * and contain none elsewhere.
cluster:*, the cluster itself; see Cluster membership

Rejected on write:

tenant:*
non-tenant-scoped wildcards such as stream:*/*
cluster:* from any tenant-scoped caller, since no tenant scope contains it

Cluster membership

GET /v1/nodes lists registered brokers; GET /v1/nodes/{node_id} fetches one. Both require node.view:cluster:*.

GET /v1/nodes?lifecycle=live&region=us-west-2&label=rack%3Da1
Authorization: Bearer <felix-token>

Filters intersect, and repeating label requires all of them. Each entry pairs the node record with why it is or is not a placement candidate. routable says whether brokers may still send it requests for the shards it leads: true for a live or draining node whose heartbeat is inside the window.

{ "items": [ { "node": { "node_id": "broker-1", "spec": { "advertise_addr": "10.0.0.4:7000", "region": "us-west-2" },
                         "status": { "lifecycle": "live", "incarnation": 3 } },
               "placement": { "eligible": false, "routable": false, "heartbeat_age_ms": 41200,
                              "reasons": ["last heartbeat was 41200ms ago, past the 15000ms timeout; expiry has not run yet"] } } ] }

A node’s spec may also carry client_addr, the host:port Felix clients connect to, and kafka_addr, the host:port Kafka clients are told to connect to (present only on a broker running the Kafka listener; a hostname is allowed, an IPv6 host must be bracketed). Both are omitted when unset. It may also carry zone, the failure domain the broker registered from FELIX_NODE_ZONE; placement spreads each shard’s copies across zones. A broker that sends none has no zone and is placed as if alone in its own. The zone is read at registration only, so changing it means restarting the broker.

A node’s status also carries features, the fleet features the broker reported when it registered (omitted when none). GET /v1/fleet/features, with the same permission, answers with the ones every live or draining broker supports and the ones an operator has enabled:

{ "supported": ["generation_start", "jump_hash_routing", "lease_free_reads", "majority_ack"],
  "enabled": [], "serving_nodes": 3 }

This build implements four features: generation_start, majority_ack, lease_free_reads and jump_hash_routing.

Support turns nothing on. POST /v1/fleet/features/{feature}/finalize (node.manage:cluster:*) enables a feature, and is refused with 409 unless every live or draining broker supports it. It is also refused before the features it only works alongside: majority_ack before generation_start, and lease_free_reads or fenced_caches before both. With ?dry_run=true it changes nothing and answers whether it would be accepted, listing brokers that lack the feature as lacking and features to finalize first as requires:

{ "feature": "jump_hash_routing", "dry_run": true, "enabled": false,
  "would_enable": false, "lacking": ["broker-3"], "requires": [],
  "serving_nodes": 3 }

Finalizing is one-way: after it, a broker that registers without the feature is refused with 409 naming it. felix-controlplane admin features and admin features finalize <feature> [--dry-run] drive the same routes. See Upgrades and compatibility.

cluster:* sits outside the tenant hierarchy and no tenant scope contains it, so a tenant admin cannot grant themselves cluster access. The tenant comes from the token’s own tid claim rather than a path segment, and only selects which signing keys to verify against.

POST /v1/nodes/{node_id}/drain marks a broker as leaving: it keeps serving, and placement moves every shard it leads to brokers that are staying, one handoff at a time. A shard being moved shows its destination as successor and, once the leader has been told to stop, "state": "draining".

PATCH /v1/nodes/{node_id} changes region, labels or capacity, and moves lifecycle between live and draining. {"lifecycle": "live"} cancels a drain. Nothing observed is patchable: a down or left broker is revived only by registering, so a patch cannot claim a silent broker is alive.

POST /v1/nodes/{node_id}/deregister marks a broker left. Its heartbeats stop renewing its lease, but what it leads is not handed on until the lease it already holds has run out: placement treats it as draining until its last heartbeat is older than the expiry timeout plus the regrant margin.

DELETE /v1/nodes/{node_id} removes a broker’s record. It needs node.manage on cluster:*, and is refused (409) while the broker is live or draining or while any shard names it as leader or replica. See Adding, draining and removing brokers.

The registration, heartbeat, drain, deregister and patch endpoints require node.manage over the node being changed. A broker’s credential is scoped to node:{its own id}, so it cannot act for another broker; an operator holding cluster:* can manage the whole fleet. Registration authorises the identity in the request body, so a broker cannot claim a name its credential does not cover.

Shard moves and placement

What an operator uses to steer shard moves; the walk-through is Moving shards by hand, and felix-controlplane admin is a command-line client of these endpoints. Reads take node.view:cluster:*; the rest take node.manage:cluster:*.

Endpoint	What it does
`GET /v1/shard-moves`	moves and follower replacements in progress, and whether placement is paused
`GET /v1/placement/plan`	what the next placement pass would write, without writing it
`GET /v1/placement/replication`	each shard’s copies on serving brokers (`current_replicas`) against its replication factor (`desired_replicas`), the members whose broker is not serving, the members whose leader has stopped shipping to them (`halted`: `node_id`, `reason`, `generation`, `since_millis`), and the copy being added by a restore; `?under_replicated=true` lists only the shards that are short
`POST /v1/shard-moves`	start moving a shard’s leadership to a node
`DELETE /v1/shard-moves/{tenant_id}/{namespace}/{name}/{shard}`	cancel a shard’s move; `?kind=cache` for a cache shard
`POST /v1/placement/pause`, `POST /v1/placement/resume`	stop and restart placement’s own moves
`POST /v1/placement/abandon/{tenant_id}/{namespace}/{name}/{shard}`	give up a stranded durable shard’s log and place the shard afresh. This loses data: records only the old leader held are gone, acknowledged ones included. 409 `not_stranded` if the leader is serving or a replica can take over

POST /v1/shard-moves
Authorization: Bearer <felix-token>
Content-Type: application/json

{ "tenant_id": "t1", "namespace": "ns", "stream": "orders", "shard": 0, "destination": "broker-3" }

{ "step": "stage",
  "assignment": { "tenant_id": "t1", "namespace": "ns", "stream": "orders", "shard": 0, "kind": "stream",
                  "leader": "broker-1", "replicas": ["broker-3"], "generation": 12, "state": "active",
                  "successor": "broker-3", "move_started_at_millis": 1790000000000, "move_reason": "operator" } }

With "dry_run": true in the request the move is decided and answered but not started, and the response carries "dry_run": true. When any broker the shard may use reports a zone, the response also carries zones_before and zones_after: the zones the shard’s live copies span now and are expected to span once the move cuts over, counting a broker without a zone as one of its own. A move that narrows the spread is started anyway and logged as a warning; see Zones.

A start is refused where placement would not make the move: 404 unknown_shard or unknown_node, or 409 destination_not_live, already_leader, at_capacity, already_moving, leader_unavailable or move_limit. A cancel answers cancel before the fence and retake after it, when the leader that stopped serves again at a new generation; with no move in progress it is 409 not_moving.

GET /v1/shard-moves lists each move’s step (staged, fenced, replacing, restoring), reason (drain, balance, operator, replace, restore), start time, and from the leader’s latest report lag_records, caught_up and drained. GET /v1/placement/plan lists each shard the next pass would act on with its action (place, a move step, waiting, settling or unplaceable) and the assignment it would write or the reason it cannot. settling is a new shard held back while a young cluster has fewer live brokers than its replication factor (FELIX_PLACEMENT_SETTLE_MS).

Tenants, namespaces, streams and caches

Every resource endpoint takes a Felix bearer token, checked before anything else. A tenant that does not exist has no signing keys, so a request against it answers 401 whatever the token says. A 404 would reveal whether the tenant exists.

Endpoint	Requires
`GET`/`POST /v1/tenants`, `DELETE /v1/tenants/{id}`	`tenant.manage:cluster:*`
`/v1/tenants/{t}/namespaces[/{ns}]`	`ns.manage` over `namespace:{t}/{ns}`, from a `t` token
`/v1/tenants/{t}/namespaces/{ns}/streams[/{s}]`	`stream.manage` over `stream:{t}/{ns}/{s}`, from a `t` token
`/v1/tenants/{t}/namespaces/{ns}/caches[/{c}]`	`cache.manage` over `cache:{t}/{ns}/{c}`, from a `t` token
`POST /v1/tenants/{t}/namespaces/{ns}/resources`	`stream.manage` and `cache.manage` over every item in the batch, from a `t` token
`/v1/{tenants,namespaces,streams,caches}/{snapshot,changes}`	`node.view:cluster:*`

POST /v1/tenants/t1/namespaces/payments/streams
Authorization: Bearer <felix-token with stream.manage over stream:t1/payments/orders>
Content-Type: application/json

{ "stream": "orders", "kind": "Stream", "shards": 1, "replication_factor": 1,
  "retention": { "max_age_seconds": null, "max_size_bytes": null },
  "consistency": "Leader", "delivery": "AtLeastOnce", "durable": true }

retention bounds a durable stream’s logs on every broker that holds one: max_size_bytes per shard log, max_age_seconds by the newest record of each segment. A bound left null is the broker’s own (FELIX_DURABLE_RETENTION_BYTES, FELIX_DURABLE_RETENTION_SECONDS), zero is refused, and a PATCH takes effect without restarting a broker.

A stream may also name a region, such as "region": "eu-west-1", and is then placed only on brokers in that region or in one the control plane’s FELIX_REGION_BRIDGES bridges it to. That variable takes comma-separated source>dest pairs, such as eu-west-1>us-east-1, and each pair allows one direction only. The region is fixed at creation, an empty one is refused with 400, and omitting it places the stream anywhere. An operator move to a broker outside the allowed regions is refused with 409 and code region_not_allowed.

A stream also records how routing keys map to its shards, fixed at creation. Omitted, routing is "jump_hash" (jump consistent hashing, which would move only about 1/n of the keys were a stream grown to n shards) once the fleet has finalized jump_hash_routing, and "modulo" before. "routing": "modulo" always works; "routing": "jump_hash" before the finalize is refused with 409. Finalizing never changes an existing stream, so no live stream’s keys move. Stream answers and shard assignments omit routing when it is modulo.

Once the fleet has finalized stream_incarnations, a created stream also carries an incarnation, higher for every creation of the same name. Brokers keep each incarnation’s data apart, so a stream deleted and created again starts empty. It is left out when 0, which is every stream created before the finalize, and a request cannot set it. Shard assignments carry the incarnation of the stream they were written for.

Before that finalize, creating a stream under a name that was deleted on the cluster is refused with 409, alone or in a batch, because without an incarnation the new stream would reopen the deleted one’s data on the brokers. The message names the finalize to run first (POST /v1/fleet/features/stream_incarnations/finalize). Caches work the same way with cache_incarnations.

After the finalize, deleting a stream (directly or through its namespace or tenant) records a tombstone for its name: every incarnation up to the one it holds is deleted. Tombstones are kept for good. GET /v1/streams/snapshot lists them as tombstones, left out while empty:

{ "items": [], "next_seq": 42,
  "tombstones": [
    { "tenant_id": "acme", "namespace": "default", "stream": "orders",
      "incarnation": 1760430000123456 } ] }

and a stream’s Deleted change carries the same value as deleted_through. Brokers delete a stream’s data only on one of these.

Caches get the same once the fleet has finalized cache_incarnations: a created cache carries an incarnation (left out when 0), cache shard assignments carry it, and after the finalize a cache delete records a tombstone, listed by GET /v1/caches/snapshot as tombstones with a cache field in place of stream, and carried on the cache’s Deleted change as deleted_through.

A cache takes consistency the same way, "Leader" when omitted. Under "Quorum", puts, deletes and counter adds are acknowledged only once a majority of the shard’s replicas hold them. How gets confirm the leader is set by the broker’s FELIX_QUORUM_READS.

POST /v1/tenants/t1/namespaces/payments/caches
Content-Type: application/json

{ "cache": "sessions", "display_name": "Sessions", "shards": 4,
  "replication_factor": 3, "consistency": "Quorum" }

Creating a stream or cache that already exists with the same configuration answers 200 with the existing one, so a create is safe to retry. A different configuration under the same name is 409. A stream’s routing counts only when the request names it.

A tenant admin’s token already carries the manage actions: exchange expands tenant.manage:tenant:t1 to ns.manage:namespace:t1/*, stream.manage:stream:t1/*/* and cache.manage:cache:t1/*/*. Listings return only what the caller could manage.

Listings are paged. limit is 1 to 10000, default 1000; a response with more to come carries next_cursor, which goes back as cursor for the next page. This applies to tenants, namespaces, streams, caches, /v1/nodes and /v1/shard-assignments. The RBAC policy and grouping listings answer with the full bare array when neither parameter is given, and with { "items": [...], "next_cursor": ... } when either is.

GET /v1/tenants/t1/namespaces/payments/streams?limit=100&cursor=InMyIg

The tenant catalog (which tenants exist) is cluster metadata, so creating, listing and deleting tenants takes the same kind of operator credential as managing the fleet, and deleting is operator-only even for the tenant’s own admin. The feeds are what brokers seed from, and take the broker’s own credential (FELIX_NODE_TOKEN), the same one that reads the shard-assignment watch. An operator credential comes out of bootstrap the same way a broker’s does: a policy granting the cluster actions to a role, and an exchange.

A create, update or delete of a tenant, namespace, stream or cache can answer 503. Code busy means other writes to the same feed held it for more than 5 s, for example a large tenant delete, and nothing was written, so retry it. Code unavailable is Raft: no quorum committed the write in time and it may still land, so a retried create can answer 409 and a retried delete 404. felixctl retries only busy, a few times before failing.

Creating several at once

POST /v1/tenants/{t}/namespaces/{ns}/resources creates up to 256 streams and caches in one namespace, all of them or none. Each item has the body its own create endpoint takes.

POST /v1/tenants/t1/namespaces/room-42/resources
Content-Type: application/json

{ "streams": [ { "stream": "chat", "kind": "Stream", "shards": 1,
                 "retention": { "max_age_seconds": null, "max_size_bytes": null },
                 "consistency": "Leader", "delivery": "AtLeastOnce", "durable": true } ],
  "caches":  [ { "cache": "cursors", "display_name": "Cursors" },
               { "cache": "board", "display_name": "Board" } ] }

Answer	When	Created
`201`	at least one item was new	the new items
`200`	every item already existed as asked	nothing
`400`	empty, over 256 items, an invalid item, or a name given twice	nothing
`403`	the token lacks the manage action over any one item	nothing
`404`	the tenant or namespace does not exist	nothing
`409`	an item exists with a different configuration (the message names it), or `jump_hash` routing before the finalize	nothing
`503`	`upgrade_in_progress`: a Raft member has not been upgraded yet	nothing
`503`	`busy`: on Postgres, other writes to the stream or cache feed held it for more than 5 s; safe to retry	nothing
`503`	`unavailable`: on Raft, no quorum committed the write in time; it may still land	unknown

A success lists every item in request order with "status": "created" or "unchanged", so a caller that failed part-way sends the same batch again. Permission is checked for every item before anything is read. The new items are written in one Postgres transaction, or one Raft log entry, so a concurrent create of the same name cannot leave part of a batch behind. See Creating streams and caches together for the details.

Internal Bootstrap API (Day-0)

Used once per tenant to seed auth before any admin tokens exist. Disabled by default and bound to a separate internal address when enabled; the listener can additionally require mTLS (see Security).

POST /internal/bootstrap/tenants/{tenant_id}/initialize
X-Felix-Bootstrap-Token: <secret>
Content-Type: application/json

{
  "display_name": "Tenant One",
  "idp_issuers": [...],
  "initial_admin_principals": ["p:alice"]
}

Initialization is atomic and exactly-once per tenant, across every control-plane instance: exactly one concurrent call wins and returns 200 with the tenant’s signing-key id; every other returns 409 already_initialized. A failed call leaves the tenant retryable, because the bootstrapped flag only commits together with a complete seed.

Status	Meaning
`200`	This call performed the initialization; the response carries `kid` and the tenant JWKS URL
`400`	Validation failed (empty display name, no admin principals, blank issuer)
`401`	Missing or wrong `X-Felix-Bootstrap-Token`
`404`	Bootstrap is not enabled on this control plane
`409`	The tenant is already initialized, by an earlier call or a concurrent one that won

GET /v1/tenants/{tenant_id}/.well-known/jwks.json

Fetch tenant signing keys (public JWKS) used by brokers to verify Felix tokens.

Response:

{
  "keys": [
    {
      "kty": "OKP",
      "kid": "k1",
      "alg": "EdDSA",
      "use": "sig",
      "crv": "Ed25519",
      "x": "..."
    }
  ]
}

Other endpoints

The endpoints not covered above:

Endpoint	What it does	Requires
`PATCH /v1/tenants/{t}/namespaces/{ns}/streams/{s}`	change `retention`, `consistency`, `delivery` or `durable`; every field is optional	`stream.manage`
`PATCH /v1/tenants/{t}/namespaces/{ns}/caches/{c}`	change `display_name`	`cache.manage`
`POST /v1/nodes`	register a broker; the answer carries the heartbeat interval, the expiry timeout and the enabled fleet features	`node.manage` over the node
`POST /v1/nodes/{node_id}/heartbeat`	renew a broker’s liveness; the body is `{"incarnation": n}` from its last registration, plus `"suspects"`, the leaders it cannot reach, when there are any	`node.manage` over the node
`POST /v1/nodes/{node_id}/replica-status`	a leader reports which replicas hold each shard it leads, which failover reads to pick a successor, and per shard `by_round` when it serves without its lease, which placement needs before it acts on its followers’ word (kept only when the body’s `incarnation` is the node’s current one); 409 if any shard’s report was refused	`node.manage` over the node
`GET /v1/shard-assignments`	list shard assignments, paged	`node.view:cluster:*`
`GET /v1/shard-assignments/{snapshot,changes}`	the assignment feed brokers follow	`node.view:cluster:*`
`GET /v1/regions`, `GET /v1/regions/{region_id}`	the region this control plane serves (`FELIX_REGION_ID`, default `local`)	none
`GET /v1/system/info`	region, API version and feature flags	none
`GET /v1/system/live`	200 while the process runs	none
`GET /v1/system/ready`, `GET /v1/system/health`	200 when the instance can serve metadata, 503 `not_ready` otherwise	none
`GET /v1/openapi.json`	the OpenAPI document	none

Errors share one body: {"code": "...", "message": "...", "request_id": ...}.

Storage backends

The control plane keeps its metadata in one of three backends, chosen with FELIX_CONTROLPLANE_STORAGE_BACKEND. The API is the same on all of them.

memory is the default. It keeps everything in the process and loses it on restart, so it is for tests and local runs.
postgres keeps the metadata in a shared database. The instances are stateless and don’t know about each other, so you run several against one highly available database. Each answers /v1/system/ready only when it can reach a database whose schema matches its build.
raft keeps the metadata in an openraft group embedded in the instances (FELIX_RAFT_NODE_ID, FELIX_RAFT_DATA_DIR, FELIX_RAFT_PEERS). Writes go through the Raft leader, and a follower forwards the writes it receives. The group survives losing a minority of its members without losing an acknowledged write, and there is no external database. See Metadata Raft.

graph TB
    subgraph CP["Control plane instances"]
        CP1["controlplane-0"]
        CP2["controlplane-1"]
        CP3["controlplane-2"]
    end
    Store[("Postgres, or the Raft group<br/>the instances form")]
    Brokers["Brokers"]
    Operators["Operators<br/>(felix-controlplane admin, curl)"]

    CP1 --> Store
    CP2 --> Store
    CP3 --> Store
    Brokers -->|register, heartbeat, poll feeds| CP
    Operators -->|REST| CP

Stream payloads and cache entries are not control-plane data. Brokers keep them in their own logs and replicate them between themselves.

How brokers follow the metadata

A broker registers with POST /v1/nodes at startup, then heartbeats every FELIX_NODE_HEARTBEAT_INTERVAL_MS (5 s by default). A broker silent for FELIX_NODE_EXPIRY_TIMEOUT_MS (15 s by default) is marked down, and must register again when its next heartbeat answer says so.

Metadata reaches brokers through snapshot and changes feeds, one pair per resource: tenants, namespaces, streams, caches and shard assignments. Each change carries a seq, an op (created, updated or deleted), the key, and the resource as it now stands:

GET /v1/streams/changes?since=41
Authorization: Bearer <FELIX_NODE_TOKEN>

{ "items": [ { "seq": 42, "op": "created",
               "key": { "tenant_id": "t1", "namespace": "payments", "stream": "orders" },
               "stream": { "tenant_id": "t1", "namespace": "payments", "stream": "orders", "...": "..." } } ],
  "next_seq": 43 }

A broker loads each snapshot once, then asks for changes from the snapshot’s next_seq. The snapshot’s items and next_seq are read at one point, so the two together describe every committed change exactly once. A page holds at most FELIX_CONTROLPLANE_CHANGES_LIMIT changes (1000 by default); a full page’s next_seq is the seq after its last change, and the broker keeps asking from next_seq until a page comes back empty. It polls the tenant, namespace, stream and cache feeds every FELIX_CONTROLPLANE_SYNC_INTERVAL_MS (2 s by default), so a new stream can take that long to reach every broker. The shard-assignment feed is a long poll: wait_ms (at most 25000) holds the request open until a change lands, so a leadership change reaches brokers without waiting for the next poll.

The control plane is not on the data path, so publishes and subscriptions never wait on it. A broker that can’t reach it keeps the metadata it last saw. New streams and placement changes reach it once it can reach the control plane again.

Consistency

A write is acknowledged once the backend has it: committed in Postgres, or committed by a majority of the Raft group. On Raft, reads are answered by the instance that receives them, so a follower can be a moment behind the leader. Brokers see changes later still, by up to the poll interval above.

Audit records are written for bootstrap only: every initialization attempt, accepted or rejected, is logged with the tenant and the outcome. Other operations are traced but not audit-logged.

Running it well

On Raft, run an odd number of instances (3 or 5) so the group keeps a majority through a failure, give them persistent volumes, and keep them off the broker nodes so data-plane load can’t starve consensus. On Postgres, run two or more instances and make the database itself highly available. The workload is metadata only and light. Disruption budgets, failover drills and the Postgres-to-Raft migration are in Control-plane HA.

Viewing the API document

The control plane serves the OpenAPI document but no viewer, which keeps a browser UI and its build-time download out of the service. Any OpenAPI viewer can read the document. The control plane sends no CORS headers, so give the viewer the file rather than the URL:

curl -s http://localhost:8443/v1/openapi.json -o openapi.json
docker run --rm -p 8081:8080 -e SWAGGER_JSON=/spec/openapi.json \
  -v "$PWD/openapi.json:/spec/openapi.json:ro" swaggerapi/swagger-ui

Then open http://localhost:8081. Podman works the same with podman in place of docker.