Skip to content

Felix 0.4.0

Released 2026-09-18. The full history is in CHANGELOG.md.

The multi-node hardening release. 0.3.0 made a cache into something you can subscribe to; 0.4.0 makes the cluster something you can authenticate, package, and reason about formally. Every broker-to-broker link can now be mutually authenticated, every control-plane endpoint requires a credential, the whole deployment ships as a Helm chart, and a TLA+ model of the lease and promotion protocol found a safety defect that reading had not.

Felix also stopped being a Rust-only project: there is a Python client, a binding over the Rust one rather than a reimplementation, gated by a conformance catalogue that the next language will have to pass too.

No wire-protocol break. felix-wire VERSION remains 1 and INTERNAL_VERSION remains 1. The one new capability is a negotiated feature bit, FEATURE_IDEMPOTENT_PRODUCER. A 0.3.x client and a 0.4.0 broker interoperate byte-for-byte on everything they both know.

  • A TLA+ model of one shard’s lease, replication and promotion protocol, at docs/formal/FelixShard.tla, checked with TLC by task tla:check and in CI. It holds no-two-leaders, acknowledged-records-survive, acknowledged-records-agree and no-truncation-below-the-mark under clock drift, lost heartbeats, lost reports, and a broker paused between admitting a write and committing it. Three configurations are expected to fail and are held to it: one with no safety interval and one with the commit-time lease check removed, which show each is load-bearing, and one with promotion as the design writes it, which finds that a leader report older than the last acknowledgement can promote a replica missing an acknowledged Quorum record. Promotion by (last generation, length) passes, and is the rule to move to (#420).

  • Idempotent producers (#422). A client asks the broker for a producer id (producer_init) and sends its batches as publish_idempotent with a per-shard sequence; the shard’s leader appends the sequence it expects, answers a re-send of one it already holds with the original outcome and no second append, and refuses a gap, an unknown producer, or a sequence older than it remembers with a typed publish_refused. A batch for a shard led elsewhere is refused with the leader’s address rather than forwarded. Negotiated as FEATURE_IDEMPOTENT_PRODUCER. ClusterClient::idempotent_producer and Client::idempotent_producer re-send under the same sequence after an ambiguous outcome, follow a not-leader refusal, and end on any other. The sequences live in the leader’s memory: a new leader answers unknown_producer, so a batch in flight across a failover is reported rather than silently landed or dropped.

  • A leader rebuilds a halted follower itself. A follower whose log diverged from the leader’s, or that refused a bootstrap because it held records of its own, is told to discard its copy of the shard (ReplicateRebuild, kind 24) and shipped again from the leader’s oldest surviving record. It happens under a policy: FELIX_REPLICATION_REBUILD_MAX_CONCURRENT caps how many followers a broker rebuilds at once (default 1; 0 leaves every halt to an operator, as before) and FELIX_REPLICATION_REBUILD_BYTES_PER_SEC paces the transfer. A fenced halt is never rebuilt, and a follower that predates the message stays halted until it is upgraded (#424).

  • Broker-to-broker mTLS (#125). With FELIX_INTERNAL_TLS_CERT, FELIX_INTERNAL_TLS_KEY and FELIX_INTERNAL_TLS_CA set (all three or none), every peer connection is mutually authenticated against the CA, and the certificate’s DNS name is the broker’s identity: a dialler verifies the listener’s certificate against the node id it dials, and the listener checks the node id a peer claims in Hello against the certificate it presented. A peer with no certificate, an untrusted or expired one, or one issued to a different name is refused. The certificate and key are re-read every 30s, so a renewal is picked up by the next handshake without a restart and without dropping connections already up. Without the three variables the peer link is encrypted but unauthenticated, as before, and startup now warns. Node ids must be valid DNS names under mTLS. The cluster test harness runs every cluster test under mTLS.

  • A Helm chart, at deploy/helm/felix, for the control plane and a broker cluster. The control plane is a Deployment over Postgres, rolling one instance at a time with none unavailable, or a StatefulSet under the Raft backend with a volume per member and the peers map derived from the release. Brokers are a StatefulSet whose pod name is the node id, with a volume each, the pod IP advertised to peers and the pod’s DNS name to clients, the credential and Postgres URL taken from Secrets the operator creates, probes and a preStop-plus-drain grace period the chart derives, PodDisruptionBudgets that keep a replication-factor-three quorum, anti- affinity and zone spread, a NetworkPolicy that admits the internal port from brokers only, and optional peer mTLS issued per pod by cert-manager’s CSI driver. Value combinations that are each fine alone and wrong together (an even Raft group, a budget wider than the replica count, a drain longer than its grace period, a backend without its store) refuse to render. task chart:check lints and renders it every way ci/ describes and asserts those properties on the output; CI runs it (#131, #132).

  • A Python client (#392, #393, #395), as a binding over the Rust client rather than a reimplementation, so the two cannot drift in behaviour. Both a synchronous and an asyncio surface, covering publish and subscribe, consumer groups, cache watches and multi-shard subscriptions. It is gated by a conformance catalogue (scenarios.toml) that every future client must pass: Python claims every required scenario and leaves two optional ones unclaimed: at_least_once does not carry a routing key, and a prefix watch over a multi-shard cache needs one watch per shard. Release wheels are built in CI.

  • Container images for the broker and the control plane (#405), built and smoke-tested in CI, which is what the Helm chart above deploys.

  • Refresh tokens and POST /token/refresh (#402). 0.3.1 raised the exchanged-token TTL as a stopgap because a broker read its credential once and held it forever; this is the real fix. A refresh rotates within a family, re-evaluates RBAC on every use (so a grant removed since the last exchange stops working without waiting for a re-exchange), and a replayed token revokes the whole chain. Brokers refresh their node credential instead of holding it for their lifetime (#404).

  • Configuration that refuses to be wrong quietly. An unrecognised FELIX_* variable is reported as a typo instead of silently taking a default (#499), and settings that are each fine alone and wrong together are refused at startup (#509). --print-config prints the effective configuration with every credential redacted, so it can be pasted into an issue (#500).

  • Day-0 bootstrap is audited (#506), with its scope containment pinned by test. It logged nothing at all before.

  • A broker names the replicas replication has stopped for (#485), so a halted follower is visible instead of inferred from lag.

  • Segments record where each leadership generation began (#466), which is what lets a follower resume at a generation boundary and a leader drop a divergent suffix.

  • An unknown internal frame kind is refused rather than fatal (#496). A newer peer sending a kind an older one does not know no longer ends its control loop.

  • --slow-subscribers in the load generator (#382), for measuring slow-consumer isolation rather than asserting it.

  • The protocol decoders are fuzzed, with every target run in CI (#480).

  • Replication stopped being a timer. A pass ships on a durable append instead of waiting for the next tick (#457), ships every shard concurrently rather than one after another (#471), and advances the quorum mark at the majority instead of at the slowest follower (#478). A pass’s replica reports now share one control-plane request (#494). Together these are what make a Quorum acknowledgement a measurement of replication rather than of a 2s sweep.
  • The cache write path group-commits like the stream path (#390).
  • A commit turn is released to one publisher rather than all of them (#511), so a wake-up does not thunder.
  • Placement uses bounded-load rendezvous hashing (#388), so shards spread evenly instead of piling onto whichever node scores highest.
  • Replica reports (what a shard’s leader says about which replicas hold its log) are now written to the control plane’s store instead of kept in the memory of the instance that received them. With several control-plane instances over one Postgres, promotion could run on an instance that had never seen the report, so a Quorum acknowledgement released on it could not be made good at failover (#409). Under Raft the report is a log command, restamped with the leader’s clock like a heartbeat. A report is stamped with the store’s clock and judged against it, so freshness is no longer a subtraction between two hosts’ clocks under Postgres; a deleted assignment now takes its report with it.
  • A Quorum acknowledgement could outrun what the replicas held. A follower can no longer confirm past the batch it was sent (#427), the leader holds the quorum mark when a replica report did not land (#434), and a broker may only report positions for shards it leads (#430).
  • Clocks are no longer compared across hosts. The lease is anchored at the heartbeat’s send, not its response (#426); liveness expiry is judged by one clock rather than one per instance (#439); the leader stamps a heartbeat rather than whichever instance received it (#475); and a replica report is judged on the clock that stamped it (#484).
  • Storage durability gaps. A failed fsync now poisons the segment writer rather than letting the next append proceed as if the disk had kept its word (#428); the compaction directory swap is crash-safe (#429); and a sparse index entry may not point inside the segment header (#483).
  • A follower resumes at the generation boundary rather than at zero (#477), and a leader drops a divergent suffix left by a previous generation (#467).
  • A subscriber id is never reused (#397).
  • A forwarded publish is bounded by the ack budget (#399).
  • An unknown stream reports zero shards rather than one (#400).
  • A bootstrap whose ranges meet is accepted, rather than requiring an exact match (#432).
  • Shard assignments cascade on delete in the in-memory store too (#398).
  • felix-common builds on its own default features (#468).
  • A forwarded publish or cache operation now carries the client’s own bearer token, and the owning broker verifies it before writing: against the tenant’s keys, for stream.publish on that stream or cache.read / cache.write on that cache. The owner used to re-check ownership and generation only, so anything that could reach the internal port could append to any tenant’s stream with no credential (#503). The credential rides two new internal kinds, AuthorizedForwardPublish (22) and AuthorizedForwardCacheOp (23); the legacy kinds still decode and are refused Unauthorized. During a rolling upgrade, an upgraded broker falls back to the legacy kind toward an owner that predates it, while an old broker’s forwards to an upgraded owner fail until it is upgraded. See the upgrade notes.
  • The control-plane resource API now requires a Felix bearer token on every endpoint. Tenants were created, listed and deleted, and namespaces, streams and caches managed, with no credential at all, while /v1/nodes next to them returned 401. Namespaces, streams and caches take ns.manage, stream.manage or cache.manage over the object from a token minted for that tenant, with listings filtered to the caller’s scope; the tenant catalog takes tenant.manage:cluster:*; the snapshot and changes feeds take node.view:cluster:*, the broker credential that already read the shard-assignment watch. The credential is checked before existence, so a tenant with no keys answers 401 rather than a 404 that says whether it exists.
  • Inbound connections on the internal listener are bounded (#505), by FELIX_INTERNAL_MAX_INBOUND_CONNECTIONS and FELIX_INTERNAL_MAX_INBOUND_PER_SOURCE. This outlives mTLS: an authenticated peer looping on a bug exhausts a listener exactly as a hostile one does.
  • Every refused credential is counted and logged (#519), so a misconfigured broker or an attacker is visible rather than silent.
  • A broker presents FELIX_NODE_TOKEN / FELIX_NODE_TOKEN_FILE on the metadata sync as well, and accepts one without FELIX_NODE_ID: a standalone broker pointed at a control plane needs it to learn any stream. Without one it still starts, warns once, and has its sync refused.
  • The RBAC object grammar accepts stream:{tenant}/*/* and cache:{tenant}/*/*, the tenant-wide forms token exchange already expanded tenant.manage to; the control plane’s own parser had refused them. A wildcard namespace under a named leaf is still refused.
  • The workspace was reorganised to Rust conventions (#391): foo.rs + foo/ throughout, with unreachable_pub and mod_module_files enforced through [workspace.lints].
  • The replica-report shapes are shared rather than redeclared, and the lifecycle is typed (#447).
  • Promotion can pick a replica missing an acknowledged Quorum record (#527). Promotion reads the leader’s last report, so a leader that acknowledges and then dies before its next report leaves a fresh report naming a replica that never received it. Report expiry does not close it: the report is recent, only older than the acknowledgement. The guarantee still holds against every fault the suite injects. The TLA+ model added in this release is what found the interleaving the suite does not reach. Promotion by greatest (last generation, length) closes it and is the rule to move to.
  • Shard rebalancing is not implemented (#130). A shard whose leader is alive is never moved, so adding a broker adds capacity for new placements only. Scaling a broker StatefulSet up will not migrate existing shards onto the new pods.
  • A retried Raft proposal can answer 409 for a write that succeeded (#529), so a provisioning script can be told a tenant it just created already exists.
  • There is no rate limiting in the control plane (#524), and an unauthenticated caller can force a JWKS fetch per request by presenting a token with an unknown kid.
  • Published throughput and latency figures remain RF=1 Leader. The harness can now seed Quorum streams (#526), but the numbers are not taken yet (#425, #375).