Felix 0.6.0-preview.4
Released 2026-10-09 as a preview. The full history is in CHANGELOG.md.
The fourth preview of 0.6.0, mostly fixes. A Quorum stream with a tight
retention bound could acknowledge a publish whose records no broker still
held; retention now stops at the commit offset on every replica, and that hold
survives a restart. A promoted cache or counter shard whose log fails to open
no longer opens unfenced. A consumer group’s cursor no longer moves backwards
after a cancelled commit, and a late nack can no longer take a record from the
next consumer. An offload outage no longer fails durable logs at open, and
offload keys now start with the node id, so two brokers sharing a directory
cannot overwrite each other’s copies. A shared connection’s subscription and
publish limits now apply per identity, and a gateway can trade a user’s token
for a delegated one carrying act, which subject binding accepts. A durable
Quorum stream with majority_ack and lease_free_reads finalized fails
over on its followers’ word in about 5 s instead of waiting out the lease. A
subscriber that registers just as its shard moves is refused rather than left
attached to nothing, and a publish forwarded to a shard its new owner is still
fencing is held rather than refused. felix-broker inspect segments checks a
data directory offline and reports what startup would do. The docs site covers
felix-gateway and the projects built on Felix. The Rust API has breaking
changes, listed under Changed. pip will not install it without --pre.
- Failover on the followers’ word (part of #1009). Each broker pings the
leaders of the shards it follows (new internal kinds
Ping38 andPong39, capabilityPING,1 << 7) and names in its heartbeat the ones that have not answered forFELIX_LEADER_SUSPECT_AFTER_MS(5 s;0turns it off). Once the fleet has finalizedmajority_ackandlease_free_reads, placement promotes a reported follower of a durableQuorumstream as soon as a majority of its set names the leader, without waiting for the leader to be marked down, so failover takes about 5 s instead ofL + margin, and a leader cut off from its followers while still heartbeating is replaced too. The promotion keeps the set and fences as before.Leaderstreams and caches still fail over on the lease, and with the control plane down a dead leader is still not replaced. The heartbeat request gains an optionalsuspectslist; the store gainsrecord_suspicion/list_suspicions(Postgres migration0024_node_suspicions, Raft leader soft state).run_heartbeatandmembership::spawntake the broker’sSuspects. TLA+:Suspicionsconstant,FelixShardSuspicion.cfg(pass) andFelixShardSuspicionLease.cfg(violatesAtMostOneServing). felix-broker inspect segments <data-dir> [TENANT/NAMESPACE/NAME/SHARD](part of #1077): a broker’s data directory read from disk, by the broker binary without starting a broker; it binds nothing and reads no configuration. For every shard of every store (streams,caches/,groups/,dead-letters/,counters/) it lists the segments, checks every record’s checksum and each index against one rebuilt from its segment, and gives the verdict startup would reach: clean, repair (and what it would cut or remove), or refuse (and the segment and byte). Strictly read-only: files are opened for reading and nothing is created, cut, removed or re-indexed.--repair-checksum-tail,--index-spacingand--verify-all-on-openmatch the broker’s settings,--kindnarrows to one store,--segmentslists every shard’s segments and--jsonprints one line per shard. It exits 0 when every shard would open as it is, 6 when startup would repair one, and 7 when it would refuse one or a record fails its checksum. Diagnosing a cluster covers a broker that will not start, on Kubernetes, Docker and Podman, and on a copy. Decoding records offline (inspect records) comes next.- A cache shard’s compaction pass now writes its key index to
keys.idxin the shard directory, and an open loads it and replays only the log past it instead of the whole log (part of #1073). The snapshot is checked against the log (CRC-32C, the offset range it covers, the checksum of the last record it saw) and ignored for a full replay when anything disagrees; a log truncation andforget_indexremove it. Memory is unchanged: the index is still held whole in memory. Format indocs/storage-format.md.
Changed
Section titled “Changed”-
FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTESandFELIX_MAX_SUBSCRIPTIONS_PER_CONNare now per identity (tenant and token subject) on a connection, which for a plain client is the whole connection. Two new settings cap a connection across its identities:FELIX_BROKER_PUBLISH_CONN_TOTAL_INFLIGHT_BYTES(default four times the per-identity budget, at mostFELIX_BROKER_PUBLISH_INFLIGHT_BYTES) andFELIX_MAX_SUBSCRIPTIONS_PER_CONN_TOTAL(default four times the per-identity cap), also settable in the config file. Startup refuses either below its per-identity value. A subscription past the connection’s ceiling is refused withmax subscriptions per connection exceeded across identities. -
Client::group_nackandClusterClient::group_nackin felix-client take theGroupRecordinstead of its offset, and sendrecord.attemptsto a broker advertisingFEATURE_GROUP_CLAIM_CONTROL. The Pythongroup_nacktakes theGroupRecordand the NodegroupNacktakes the record (any object withoffsetandattempts). felixctl’sgroup nacksends the attempt count of aSHARD:OFFSET:ATTEMPTSclaim with or without--delay-ms. -
felix-storage:
OffloadTarget::LocalDiris nowLocalDir { dir, node }, andLogConfig::validaterefuses a node id that is not one path segment (letters, digits,-,_,., not starting with.). In felix-broker-service,DurableStorageConfig::stream_logtakes the cluster node id and returns aResult(#1109). -
Startup recovery of a shard now plans every write before it makes any (
plan_recoveryin felix-storage’sdisk_log/recovery/plan.rs), then makes them in the same order as before. What recovery does is unchanged; the plan is whatfelix_storage::inspectandfelix-broker inspect segmentsreport. An I/O error while planning now stops before any repair is written, where it used to stop partway. Thesegment_recoveryfuzz target also checks that plan and recovery agree.
- A publish forwarded to a moved shard’s new owner while it was still fencing
the replicas was refused as
not_readyinstead of held, so a planned move or shutdown handoff whose fence took longer than the forwarding broker’s retries could refuse a publish (#1120). The owner now holds it until the shard serves, as it already did for a publish sent to it directly. - A subscription or cache watch that registered just after a lease lapse, deposal or release ended its shard’s readers is now refused instead of being left attached to a shard the broker no longer serves, where it got nothing and never heard the shard moved (#1117). The broker checks again after registering, and a broker deposed at a shard’s generation now refuses new readers of it even while its lease holds.
- A retention sweep after a restart could delete records above a
Quorumshard’s commit offset, on an idle leader or a follower cut off from its leader, until replication reached the shard again (#1109). The hold at the commit offset is now kept in the shard’sreplicafile (a flag in what was a reserved field, which older builds ignore) and restored at open, before retention starts. A stream or cache raised toQuorumis held as the broker applies the change, not from the first replication pass. A leader whose shard has no followers left lifts the hold, and so does a broker with neither membership nor a peer transport, at open, since nothing there advances a commit offset. In felix-storage and felix-broker,hold_retention_at_commitis now async and returns whether the change was written; felix-storage addsLogConfig::retention_hold,DiskLogProvider::set_stream_retention_holdandStorageApi::set_retention_hold. - A consumer-group commit cancelled after it handed its write to a task, but before that task first ran, could still let the next commit read the old cursor and move the group backwards (part of #1109; the case #1097 left open). A log-backed cache put or delete that checks no condition now marks its key in flight before it spawns the write, so a settled read waits for it.
- A plain nack can no longer take a record from the consumer it was handed to
next (part of #1109).
group_nacknames the delivery by its attempt count, asgroup_nack_afterdoes, so a late one is refused withstale_claim. - Two brokers offloading to one directory could write the same object key,
because keys held only the shard directory name, which every replica of a
shard shares, so one broker could replace a copy the other had recorded
(#1109). Keys now start with the node id:
<node id>/<shard directory>/<base>-<segment id>.segment. The node id isFELIX_NODE_ID, or, on a broker without one, an id generated once and kept innode-idunderFELIX_DURABLE_STORAGE_DIR. Copies made by 0.6.0-preview.3 keep their old keys, which their manifest entries still name. - One client acting for many users over a shared connection
(
Client::with_identity) no longer lets one user starve the rest (part of #1098). The broker’s subscription cap (FELIX_MAX_SUBSCRIPTIONS_PER_CONN) and publish byte budget (FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES) now apply per tenant and token subject on a connection, so one user at either limit is refused or slowed while the others keep their room. A user’s share is dropped once nothing of theirs is open or in flight on the connection. A client that authenticates every stream as one identity sees the same limits, refusals and frames as before. - Subject binding (
FELIX_TLS_CLIENT_CERT_BIND_SUBJECT) refused every user’s token a gateway presented over its own certificate (part of #1098). The control plane has a new RFC 8693 token exchange,POST /v1/tenants/{tenant_id}/token/delegate: a caller holding the newtoken.delegate:tenant:{tenant_id}permission trades a user’s broker token for one with the same subject, no wider permissions and no later expiry, andact: {"sub": "<caller>"}. The broker binds such a token to a certificate issued to its actor, on QUIC and Kafka SASL alike; a token withoutactbinds only to its subject, as before.token.delegateis never put in a broker token, so brokers that predate it are unaffected. - The docs said a delegated identity had its own publish budget when only the client’s was per identity; they now describe what the broker enforces.
- An in-memory cache’s versions restarted at 0 with the broker, so a
cache_put_iforcache_delete_ifholding a version read before a restart could apply against a value written after it (#1098). Versions now start each run at the wall-clock time in microseconds. Log-backed caches were not affected: their versions are log offsets. - A late nack can no longer take a record from the consumer it was handed to
next (#1098).
group_nacktakes an optionalattemptsthat names the delivery, as ongroup_extend, and is refused withstale_claimonce that claim has lapsed or the record has gone out again. Without it, a delayed nack arriving after a redelivery parked the new delivery behind its delay. Left out at0, so a nack that does not name a delivery is the frame it always was; a broker that predates the field ignores it. felix-client’sgroup_nack_afternow takes theGroupRecordrather than its offset (onClientandClusterClient), and sendsattemptsto a broker advertisingFEATURE_GROUP_CLAIM_CONTROL. felixctl sends it for anack --delay-msclaim given asSHARD:OFFSET:ATTEMPTS. In felix-broker,GroupReader::nack_aftertakes the attempt count to check and nacks at once for a zero delay. felixctlsaved its config through a temporary file with a predictable name, opened withoutO_EXCL, so a symlink planted at that path in a shared directory was followed and the target overwritten (#1098). The temporary file now gets a random name and is created exclusively, owner-only on Unix, before it is renamed over the config.- A first segment whose header and records had all been zeroed was deleted at startup as a segment whose creation never finished, and the log came back empty (part of #1098). Creation syncs the header before anything appends, so only a file no longer than a header can be an unfinished creation. Recovery now refuses a longer one as corruption.
- Retention could acknowledge a
Quorumpublish for records no broker held (#1094). With both followers of a shard down, a tight retention bound deleted records only the leader had, including ones publishes were still waiting on. The followers came back below the leader’s new base, were rebuilt there, and their answers moved the quorum mark over the deleted records. Retention and compaction’s head trims now stop at the commit offset on every replica of aQuorumshard, and the leader will not bootstrap or rebuild a follower at a base above it. A stream whose followers are down can therefore grow past its retention bound until a majority is back.DiskLog::hold_retention_at_commitandStreamLog::hold_retention_at_commitare new. - A consumer group’s cursor can no longer move backwards after a cancelled commit (#1097). The cancelled commit’s write still finished on its own task, but the next commit read the position from before it and wrote over it. A commit now waits for a write to the cursor that is still applying. Finishing a redrive had the same race against a cancelled return to dead and could drop the dead letter; it waits the same way.
- A missing or read-only offload directory no longer fails every durable
stream log at open. The directory is opened by the offload pass, which
reports a failure through
felix_storage_offload_failures_total, the newfelix_storage_offload_failing_logsandfelix_storage_offload_held_bytesgauges and a backed-off log line, and retries on the next tick. Retention still never deletes a segment without a recorded copy, so an outage grows local disk; deletion resumes once the directory is back (#1096). - A promoted cache shard whose cache log or counter log failed to open (the shard closing, too many open files, a failed recovery) opened on the lease without a fence, even once caches acknowledge by their followers and the lease no longer stops the old leader. It now stays closed and the next pass tries again (#1095).
Documentation
Section titled “Documentation”- The docs said an in-memory cache’s versions never repeat across a restart.
They can if the clock steps back across the restart or the last run averaged
more than one write per microsecond;
docs/protocol.mdand the cache page now say so (#1109). That cache runs only on a broker with no data directory, so there is nowhere to record where its counter stopped. - The docs site documents felix-gateway, the WebSocket relay that lets browsers use Felix, under Clients, and has a Built on Felix section for felix-canvas, felix-webhook-relay and felix-arena that says which Felix features each one uses and what it relies on Felix to guarantee.
