Skip to content

Felix 0.6.0-preview.4

Released 2026-10-09 as a preview. The full history is in CHANGELOG.md.

The fourth preview of 0.6.0, mostly fixes. A Quorum stream with a tight retention bound could acknowledge a publish whose records no broker still held; retention now stops at the commit offset on every replica, and that hold survives a restart. A promoted cache or counter shard whose log fails to open no longer opens unfenced. A consumer group’s cursor no longer moves backwards after a cancelled commit, and a late nack can no longer take a record from the next consumer. An offload outage no longer fails durable logs at open, and offload keys now start with the node id, so two brokers sharing a directory cannot overwrite each other’s copies. A shared connection’s subscription and publish limits now apply per identity, and a gateway can trade a user’s token for a delegated one carrying act, which subject binding accepts. A durable Quorum stream with majority_ack and lease_free_reads finalized fails over on its followers’ word in about 5 s instead of waiting out the lease. A subscriber that registers just as its shard moves is refused rather than left attached to nothing, and a publish forwarded to a shard its new owner is still fencing is held rather than refused. felix-broker inspect segments checks a data directory offline and reports what startup would do. The docs site covers felix-gateway and the projects built on Felix. The Rust API has breaking changes, listed under Changed. pip will not install it without --pre.

  • Failover on the followers’ word (part of #1009). Each broker pings the leaders of the shards it follows (new internal kinds Ping 38 and Pong 39, capability PING, 1 << 7) and names in its heartbeat the ones that have not answered for FELIX_LEADER_SUSPECT_AFTER_MS (5 s; 0 turns it off). Once the fleet has finalized majority_ack and lease_free_reads, placement promotes a reported follower of a durable Quorum stream as soon as a majority of its set names the leader, without waiting for the leader to be marked down, so failover takes about 5 s instead of L + margin, and a leader cut off from its followers while still heartbeating is replaced too. The promotion keeps the set and fences as before. Leader streams and caches still fail over on the lease, and with the control plane down a dead leader is still not replaced. The heartbeat request gains an optional suspects list; the store gains record_suspicion/list_suspicions (Postgres migration 0024_node_suspicions, Raft leader soft state). run_heartbeat and membership::spawn take the broker’s Suspects. TLA+: Suspicions constant, FelixShardSuspicion.cfg (pass) and FelixShardSuspicionLease.cfg (violates AtMostOneServing).
  • felix-broker inspect segments <data-dir> [TENANT/NAMESPACE/NAME/SHARD] (part of #1077): a broker’s data directory read from disk, by the broker binary without starting a broker; it binds nothing and reads no configuration. For every shard of every store (streams, caches/, groups/, dead-letters/, counters/) it lists the segments, checks every record’s checksum and each index against one rebuilt from its segment, and gives the verdict startup would reach: clean, repair (and what it would cut or remove), or refuse (and the segment and byte). Strictly read-only: files are opened for reading and nothing is created, cut, removed or re-indexed. --repair-checksum-tail, --index-spacing and --verify-all-on-open match the broker’s settings, --kind narrows to one store, --segments lists every shard’s segments and --json prints one line per shard. It exits 0 when every shard would open as it is, 6 when startup would repair one, and 7 when it would refuse one or a record fails its checksum. Diagnosing a cluster covers a broker that will not start, on Kubernetes, Docker and Podman, and on a copy. Decoding records offline (inspect records) comes next.
  • A cache shard’s compaction pass now writes its key index to keys.idx in the shard directory, and an open loads it and replays only the log past it instead of the whole log (part of #1073). The snapshot is checked against the log (CRC-32C, the offset range it covers, the checksum of the last record it saw) and ignored for a full replay when anything disagrees; a log truncation and forget_index remove it. Memory is unchanged: the index is still held whole in memory. Format in docs/storage-format.md.
  • FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES and FELIX_MAX_SUBSCRIPTIONS_PER_CONN are now per identity (tenant and token subject) on a connection, which for a plain client is the whole connection. Two new settings cap a connection across its identities: FELIX_BROKER_PUBLISH_CONN_TOTAL_INFLIGHT_BYTES (default four times the per-identity budget, at most FELIX_BROKER_PUBLISH_INFLIGHT_BYTES) and FELIX_MAX_SUBSCRIPTIONS_PER_CONN_TOTAL (default four times the per-identity cap), also settable in the config file. Startup refuses either below its per-identity value. A subscription past the connection’s ceiling is refused with max subscriptions per connection exceeded across identities.

  • Client::group_nack and ClusterClient::group_nack in felix-client take the GroupRecord instead of its offset, and send record.attempts to a broker advertising FEATURE_GROUP_CLAIM_CONTROL. The Python group_nack takes the GroupRecord and the Node groupNack takes the record (any object with offset and attempts). felixctl’s group nack sends the attempt count of a SHARD:OFFSET:ATTEMPTS claim with or without --delay-ms.

  • felix-storage: OffloadTarget::LocalDir is now LocalDir { dir, node }, and LogConfig::validate refuses a node id that is not one path segment (letters, digits, -, _, ., not starting with .). In felix-broker-service, DurableStorageConfig::stream_log takes the cluster node id and returns a Result (#1109).

  • Startup recovery of a shard now plans every write before it makes any (plan_recovery in felix-storage’s disk_log/recovery/plan.rs), then makes them in the same order as before. What recovery does is unchanged; the plan is what felix_storage::inspect and felix-broker inspect segments report. An I/O error while planning now stops before any repair is written, where it used to stop partway. The segment_recovery fuzz target also checks that plan and recovery agree.

  • A publish forwarded to a moved shard’s new owner while it was still fencing the replicas was refused as not_ready instead of held, so a planned move or shutdown handoff whose fence took longer than the forwarding broker’s retries could refuse a publish (#1120). The owner now holds it until the shard serves, as it already did for a publish sent to it directly.
  • A subscription or cache watch that registered just after a lease lapse, deposal or release ended its shard’s readers is now refused instead of being left attached to a shard the broker no longer serves, where it got nothing and never heard the shard moved (#1117). The broker checks again after registering, and a broker deposed at a shard’s generation now refuses new readers of it even while its lease holds.
  • A retention sweep after a restart could delete records above a Quorum shard’s commit offset, on an idle leader or a follower cut off from its leader, until replication reached the shard again (#1109). The hold at the commit offset is now kept in the shard’s replica file (a flag in what was a reserved field, which older builds ignore) and restored at open, before retention starts. A stream or cache raised to Quorum is held as the broker applies the change, not from the first replication pass. A leader whose shard has no followers left lifts the hold, and so does a broker with neither membership nor a peer transport, at open, since nothing there advances a commit offset. In felix-storage and felix-broker, hold_retention_at_commit is now async and returns whether the change was written; felix-storage adds LogConfig::retention_hold, DiskLogProvider::set_stream_retention_hold and StorageApi::set_retention_hold.
  • A consumer-group commit cancelled after it handed its write to a task, but before that task first ran, could still let the next commit read the old cursor and move the group backwards (part of #1109; the case #1097 left open). A log-backed cache put or delete that checks no condition now marks its key in flight before it spawns the write, so a settled read waits for it.
  • A plain nack can no longer take a record from the consumer it was handed to next (part of #1109). group_nack names the delivery by its attempt count, as group_nack_after does, so a late one is refused with stale_claim.
  • Two brokers offloading to one directory could write the same object key, because keys held only the shard directory name, which every replica of a shard shares, so one broker could replace a copy the other had recorded (#1109). Keys now start with the node id: <node id>/<shard directory>/<base>-<segment id>.segment. The node id is FELIX_NODE_ID, or, on a broker without one, an id generated once and kept in node-id under FELIX_DURABLE_STORAGE_DIR. Copies made by 0.6.0-preview.3 keep their old keys, which their manifest entries still name.
  • One client acting for many users over a shared connection (Client::with_identity) no longer lets one user starve the rest (part of #1098). The broker’s subscription cap (FELIX_MAX_SUBSCRIPTIONS_PER_CONN) and publish byte budget (FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES) now apply per tenant and token subject on a connection, so one user at either limit is refused or slowed while the others keep their room. A user’s share is dropped once nothing of theirs is open or in flight on the connection. A client that authenticates every stream as one identity sees the same limits, refusals and frames as before.
  • Subject binding (FELIX_TLS_CLIENT_CERT_BIND_SUBJECT) refused every user’s token a gateway presented over its own certificate (part of #1098). The control plane has a new RFC 8693 token exchange, POST /v1/tenants/{tenant_id}/token/delegate: a caller holding the new token.delegate:tenant:{tenant_id} permission trades a user’s broker token for one with the same subject, no wider permissions and no later expiry, and act: {"sub": "<caller>"}. The broker binds such a token to a certificate issued to its actor, on QUIC and Kafka SASL alike; a token without act binds only to its subject, as before. token.delegate is never put in a broker token, so brokers that predate it are unaffected.
  • The docs said a delegated identity had its own publish budget when only the client’s was per identity; they now describe what the broker enforces.
  • An in-memory cache’s versions restarted at 0 with the broker, so a cache_put_if or cache_delete_if holding a version read before a restart could apply against a value written after it (#1098). Versions now start each run at the wall-clock time in microseconds. Log-backed caches were not affected: their versions are log offsets.
  • A late nack can no longer take a record from the consumer it was handed to next (#1098). group_nack takes an optional attempts that names the delivery, as on group_extend, and is refused with stale_claim once that claim has lapsed or the record has gone out again. Without it, a delayed nack arriving after a redelivery parked the new delivery behind its delay. Left out at 0, so a nack that does not name a delivery is the frame it always was; a broker that predates the field ignores it. felix-client’s group_nack_after now takes the GroupRecord rather than its offset (on Client and ClusterClient), and sends attempts to a broker advertising FEATURE_GROUP_CLAIM_CONTROL. felixctl sends it for a nack --delay-ms claim given as SHARD:OFFSET:ATTEMPTS. In felix-broker, GroupReader::nack_after takes the attempt count to check and nacks at once for a zero delay.
  • felixctl saved its config through a temporary file with a predictable name, opened without O_EXCL, so a symlink planted at that path in a shared directory was followed and the target overwritten (#1098). The temporary file now gets a random name and is created exclusively, owner-only on Unix, before it is renamed over the config.
  • A first segment whose header and records had all been zeroed was deleted at startup as a segment whose creation never finished, and the log came back empty (part of #1098). Creation syncs the header before anything appends, so only a file no longer than a header can be an unfinished creation. Recovery now refuses a longer one as corruption.
  • Retention could acknowledge a Quorum publish for records no broker held (#1094). With both followers of a shard down, a tight retention bound deleted records only the leader had, including ones publishes were still waiting on. The followers came back below the leader’s new base, were rebuilt there, and their answers moved the quorum mark over the deleted records. Retention and compaction’s head trims now stop at the commit offset on every replica of a Quorum shard, and the leader will not bootstrap or rebuild a follower at a base above it. A stream whose followers are down can therefore grow past its retention bound until a majority is back. DiskLog::hold_retention_at_commit and StreamLog::hold_retention_at_commit are new.
  • A consumer group’s cursor can no longer move backwards after a cancelled commit (#1097). The cancelled commit’s write still finished on its own task, but the next commit read the position from before it and wrote over it. A commit now waits for a write to the cursor that is still applying. Finishing a redrive had the same race against a cancelled return to dead and could drop the dead letter; it waits the same way.
  • A missing or read-only offload directory no longer fails every durable stream log at open. The directory is opened by the offload pass, which reports a failure through felix_storage_offload_failures_total, the new felix_storage_offload_failing_logs and felix_storage_offload_held_bytes gauges and a backed-off log line, and retries on the next tick. Retention still never deletes a segment without a recorded copy, so an outage grows local disk; deletion resumes once the directory is back (#1096).
  • A promoted cache shard whose cache log or counter log failed to open (the shard closing, too many open files, a failed recovery) opened on the lease without a fence, even once caches acknowledge by their followers and the lease no longer stops the old leader. It now stays closed and the next pass tries again (#1095).
  • The docs said an in-memory cache’s versions never repeat across a restart. They can if the clock steps back across the restart or the last run averaged more than one write per microsecond; docs/protocol.md and the cache page now say so (#1109). That cache runs only on a broker with no data directory, so there is nowhere to record where its counter stopped.
  • The docs site documents felix-gateway, the WebSocket relay that lets browsers use Felix, under Clients, and has a Built on Felix section for felix-canvas, felix-webhook-relay and felix-arena that says which Felix features each one uses and what it relies on Felix to guarantee.