Felix 0.4.0
Released 2026-09-18. The full history is in CHANGELOG.md.
The multi-node hardening release. 0.3.0 made a cache into something you can subscribe to; 0.4.0 makes the cluster something you can authenticate, package, and reason about formally. Every broker-to-broker link can now be mutually authenticated, every control-plane endpoint requires a credential, the whole deployment ships as a Helm chart, and a TLA+ model of the lease and promotion protocol found a safety defect that reading had not.
Felix also stopped being a Rust-only project: there is a Python client, a binding over the Rust one rather than a reimplementation, gated by a conformance catalogue that the next language will have to pass too.
No wire-protocol break. felix-wire VERSION remains 1 and
INTERNAL_VERSION remains 1. The one new capability is a negotiated feature
bit, FEATURE_IDEMPOTENT_PRODUCER. A 0.3.x client and a 0.4.0 broker
interoperate byte-for-byte on everything they both know.
-
A TLA+ model of one shard’s lease, replication and promotion protocol, at
docs/formal/FelixShard.tla, checked with TLC bytask tla:checkand in CI. It holds no-two-leaders, acknowledged-records-survive, acknowledged-records-agree and no-truncation-below-the-mark under clock drift, lost heartbeats, lost reports, and a broker paused between admitting a write and committing it. Three configurations are expected to fail and are held to it: one with no safety interval and one with the commit-time lease check removed, which show each is load-bearing, and one with promotion as the design writes it, which finds that a leader report older than the last acknowledgement can promote a replica missing an acknowledgedQuorumrecord. Promotion by (last generation, length) passes, and is the rule to move to (#420). -
Idempotent producers (#422). A client asks the broker for a producer id (
producer_init) and sends its batches aspublish_idempotentwith a per-shard sequence; the shard’s leader appends the sequence it expects, answers a re-send of one it already holds with the original outcome and no second append, and refuses a gap, an unknown producer, or a sequence older than it remembers with a typedpublish_refused. A batch for a shard led elsewhere is refused with the leader’s address rather than forwarded. Negotiated asFEATURE_IDEMPOTENT_PRODUCER.ClusterClient::idempotent_producerandClient::idempotent_producerre-send under the same sequence after an ambiguous outcome, follow a not-leader refusal, and end on any other. The sequences live in the leader’s memory: a new leader answersunknown_producer, so a batch in flight across a failover is reported rather than silently landed or dropped. -
A leader rebuilds a halted follower itself. A follower whose log diverged from the leader’s, or that refused a bootstrap because it held records of its own, is told to discard its copy of the shard (
ReplicateRebuild, kind 24) and shipped again from the leader’s oldest surviving record. It happens under a policy:FELIX_REPLICATION_REBUILD_MAX_CONCURRENTcaps how many followers a broker rebuilds at once (default1;0leaves every halt to an operator, as before) andFELIX_REPLICATION_REBUILD_BYTES_PER_SECpaces the transfer. A fenced halt is never rebuilt, and a follower that predates the message stays halted until it is upgraded (#424). -
Broker-to-broker mTLS (#125). With
FELIX_INTERNAL_TLS_CERT,FELIX_INTERNAL_TLS_KEYandFELIX_INTERNAL_TLS_CAset (all three or none), every peer connection is mutually authenticated against the CA, and the certificate’s DNS name is the broker’s identity: a dialler verifies the listener’s certificate against the node id it dials, and the listener checks the node id a peer claims inHelloagainst the certificate it presented. A peer with no certificate, an untrusted or expired one, or one issued to a different name is refused. The certificate and key are re-read every 30s, so a renewal is picked up by the next handshake without a restart and without dropping connections already up. Without the three variables the peer link is encrypted but unauthenticated, as before, and startup now warns. Node ids must be valid DNS names under mTLS. The cluster test harness runs every cluster test under mTLS. -
A Helm chart, at
deploy/helm/felix, for the control plane and a broker cluster. The control plane is a Deployment over Postgres, rolling one instance at a time with none unavailable, or a StatefulSet under the Raft backend with a volume per member and the peers map derived from the release. Brokers are a StatefulSet whose pod name is the node id, with a volume each, the pod IP advertised to peers and the pod’s DNS name to clients, the credential and Postgres URL taken from Secrets the operator creates, probes and a preStop-plus-drain grace period the chart derives, PodDisruptionBudgets that keep a replication-factor-three quorum, anti- affinity and zone spread, a NetworkPolicy that admits the internal port from brokers only, and optional peer mTLS issued per pod by cert-manager’s CSI driver. Value combinations that are each fine alone and wrong together (an even Raft group, a budget wider than the replica count, a drain longer than its grace period, a backend without its store) refuse to render.task chart:checklints and renders it every wayci/describes and asserts those properties on the output; CI runs it (#131, #132). -
A Python client (#392, #393, #395), as a binding over the Rust client rather than a reimplementation, so the two cannot drift in behaviour. Both a synchronous and an asyncio surface, covering publish and subscribe, consumer groups, cache watches and multi-shard subscriptions. It is gated by a conformance catalogue (
scenarios.toml) that every future client must pass: Python claims every required scenario and leaves two optional ones unclaimed:at_least_oncedoes not carry a routing key, and a prefix watch over a multi-shard cache needs one watch per shard. Release wheels are built in CI. -
Container images for the broker and the control plane (#405), built and smoke-tested in CI, which is what the Helm chart above deploys.
-
Refresh tokens and
POST /token/refresh(#402). 0.3.1 raised the exchanged-token TTL as a stopgap because a broker read its credential once and held it forever; this is the real fix. A refresh rotates within a family, re-evaluates RBAC on every use (so a grant removed since the last exchange stops working without waiting for a re-exchange), and a replayed token revokes the whole chain. Brokers refresh their node credential instead of holding it for their lifetime (#404). -
Configuration that refuses to be wrong quietly. An unrecognised
FELIX_*variable is reported as a typo instead of silently taking a default (#499), and settings that are each fine alone and wrong together are refused at startup (#509).--print-configprints the effective configuration with every credential redacted, so it can be pasted into an issue (#500). -
Day-0 bootstrap is audited (#506), with its scope containment pinned by test. It logged nothing at all before.
-
A broker names the replicas replication has stopped for (#485), so a halted follower is visible instead of inferred from lag.
-
Segments record where each leadership generation began (#466), which is what lets a follower resume at a generation boundary and a leader drop a divergent suffix.
-
An unknown internal frame kind is refused rather than fatal (#496). A newer peer sending a kind an older one does not know no longer ends its control loop.
-
--slow-subscribersin the load generator (#382), for measuring slow-consumer isolation rather than asserting it. -
The protocol decoders are fuzzed, with every target run in CI (#480).
Performance
Section titled “Performance”- Replication stopped being a timer. A pass ships on a durable append
instead of waiting for the next tick (#457), ships every shard concurrently
rather than one after another (#471), and advances the quorum mark at the
majority instead of at the slowest follower (#478). A pass’s replica
reports now share one control-plane request (#494). Together these are what
make a
Quorumacknowledgement a measurement of replication rather than of a 2s sweep. - The cache write path group-commits like the stream path (#390).
- A commit turn is released to one publisher rather than all of them (#511), so a wake-up does not thunder.
- Placement uses bounded-load rendezvous hashing (#388), so shards spread evenly instead of piling onto whichever node scores highest.
- Replica reports (what a shard’s leader says about which replicas hold its
log) are now written to the control plane’s store instead of kept in the
memory of the instance that received them. With several control-plane
instances over one Postgres, promotion could run on an instance that had
never seen the report, so a
Quorumacknowledgement released on it could not be made good at failover (#409). Under Raft the report is a log command, restamped with the leader’s clock like a heartbeat. A report is stamped with the store’s clock and judged against it, so freshness is no longer a subtraction between two hosts’ clocks under Postgres; a deleted assignment now takes its report with it. - A
Quorumacknowledgement could outrun what the replicas held. A follower can no longer confirm past the batch it was sent (#427), the leader holds the quorum mark when a replica report did not land (#434), and a broker may only report positions for shards it leads (#430). - Clocks are no longer compared across hosts. The lease is anchored at the heartbeat’s send, not its response (#426); liveness expiry is judged by one clock rather than one per instance (#439); the leader stamps a heartbeat rather than whichever instance received it (#475); and a replica report is judged on the clock that stamped it (#484).
- Storage durability gaps. A failed fsync now poisons the segment writer rather than letting the next append proceed as if the disk had kept its word (#428); the compaction directory swap is crash-safe (#429); and a sparse index entry may not point inside the segment header (#483).
- A follower resumes at the generation boundary rather than at zero (#477), and a leader drops a divergent suffix left by a previous generation (#467).
- A subscriber id is never reused (#397).
- A forwarded publish is bounded by the ack budget (#399).
- An unknown stream reports zero shards rather than one (#400).
- A bootstrap whose ranges meet is accepted, rather than requiring an exact match (#432).
- Shard assignments cascade on delete in the in-memory store too (#398).
felix-commonbuilds on its own default features (#468).
Security
Section titled “Security”- A forwarded publish or cache operation now carries the client’s own bearer
token, and the owning broker verifies it before writing: against the
tenant’s keys, for
stream.publishon that stream orcache.read/cache.writeon that cache. The owner used to re-check ownership and generation only, so anything that could reach the internal port could append to any tenant’s stream with no credential (#503). The credential rides two new internal kinds,AuthorizedForwardPublish(22) andAuthorizedForwardCacheOp(23); the legacy kinds still decode and are refusedUnauthorized. During a rolling upgrade, an upgraded broker falls back to the legacy kind toward an owner that predates it, while an old broker’s forwards to an upgraded owner fail until it is upgraded. See the upgrade notes. - The control-plane resource API now requires a Felix bearer token on every
endpoint. Tenants were created, listed and deleted, and namespaces, streams
and caches managed, with no credential at all, while
/v1/nodesnext to them returned401. Namespaces, streams and caches takens.manage,stream.manageorcache.manageover the object from a token minted for that tenant, with listings filtered to the caller’s scope; the tenant catalog takestenant.manage:cluster:*; thesnapshotandchangesfeeds takenode.view:cluster:*, the broker credential that already read the shard-assignment watch. The credential is checked before existence, so a tenant with no keys answers401rather than a404that says whether it exists. - Inbound connections on the internal listener are bounded (#505), by
FELIX_INTERNAL_MAX_INBOUND_CONNECTIONSandFELIX_INTERNAL_MAX_INBOUND_PER_SOURCE. This outlives mTLS: an authenticated peer looping on a bug exhausts a listener exactly as a hostile one does. - Every refused credential is counted and logged (#519), so a misconfigured broker or an attacker is visible rather than silent.
Changed
Section titled “Changed”- A broker presents
FELIX_NODE_TOKEN/FELIX_NODE_TOKEN_FILEon the metadata sync as well, and accepts one withoutFELIX_NODE_ID: a standalone broker pointed at a control plane needs it to learn any stream. Without one it still starts, warns once, and has its sync refused. - The RBAC object grammar accepts
stream:{tenant}/*/*andcache:{tenant}/*/*, the tenant-wide forms token exchange already expandedtenant.manageto; the control plane’s own parser had refused them. A wildcard namespace under a named leaf is still refused. - The workspace was reorganised to Rust conventions (#391):
foo.rs+foo/throughout, withunreachable_pubandmod_module_filesenforced through[workspace.lints]. - The replica-report shapes are shared rather than redeclared, and the lifecycle is typed (#447).
Known limitations
Section titled “Known limitations”- Promotion can pick a replica missing an acknowledged
Quorumrecord (#527). Promotion reads the leader’s last report, so a leader that acknowledges and then dies before its next report leaves a fresh report naming a replica that never received it. Report expiry does not close it: the report is recent, only older than the acknowledgement. The guarantee still holds against every fault the suite injects. The TLA+ model added in this release is what found the interleaving the suite does not reach. Promotion by greatest (last generation, length) closes it and is the rule to move to. - Shard rebalancing is not implemented (#130). A shard whose leader is alive is never moved, so adding a broker adds capacity for new placements only. Scaling a broker StatefulSet up will not migrate existing shards onto the new pods.
- A retried Raft proposal can answer
409for a write that succeeded (#529), so a provisioning script can be told a tenant it just created already exists. - There is no rate limiting in the control plane (#524), and an
unauthenticated caller can force a JWKS fetch per request by presenting a
token with an unknown
kid. - Published throughput and latency figures remain RF=1
Leader. The harness can now seedQuorumstreams (#526), but the numbers are not taken yet (#425, #375).
