Skip to content

Felix 0.6.0-preview.8

Released 2026-10-16 as a preview. The full history is in CHANGELOG.md.

The eighth preview of 0.6.0. A broker now has a storage identity that its node registers, and placement counts a follower toward promotion only while the identity its leader reported matches, so a follower that lost its disk is never promoted on an old report. Streams get incarnations, and caches get them too, so a stream or cache deleted and created again under the same name starts empty on every broker; both are behind finalized features. A leader that crashed under FsyncMode::Periodic or None no longer reopens its old generation with a log that may be short of what it acknowledged (#1294, #1305). The single-publisher fanout path is back to preview.7’s throughput, with large-payload fanout faster (#1309).

Upgrade notes.

  • A Raft control plane needs every member upgraded before these take effect, each at its own metadata level. Until then the proposer leaves the new data out, the same on every member:
    • Level 9: an auth change and its revocation, and a refresh rotation, are single entries. Below it they are written as the two older entries they combine.
    • Level 10: stream incarnations (an upgraded member refuses the stream_incarnations finalize with 409), and deletes that take shard assignments and replica reports.
    • Level 11: a replica report’s by_round. Below it, leaders are failed over on their lease.
    • Level 12: storage identity. Below it, followers read as unverified.
    • Level 13: a report’s leader and node_incarnations. Below it, placement reads reports as at level 12.
    • Level 14: cache incarnations (an upgraded member refuses the cache_incarnations finalize with 409). Below it, caches are created at incarnation 0 and no cache delete records a tombstone.
    • Level 15: a report’s given_up. Below it, placement replaces a leader that gave a shard up only on its storage identity.
  • A Postgres control plane runs migrations 0027 through 0033 on start.
  • stream_incarnations and cache_incarnations are finalized by an operator after the upgrade: every control-plane member at level 10 and 14, and every broker on this release, first. With Postgres, upgrade every instance before finalizing, since an older instance still running after the finalize cannot create streams or caches. See the upgrades page.
  • A cluster that ran an unreleased build at metadata level 10 to 13 and logged a stream create that names no incarnation, or its delete, replays it differently now and must rebuild those members from a snapshot (#1287).
  • The Python and TypeScript SDKs read their CA file with rustls-pki-types instead of the unmaintained rustls-pemfile (RUSTSEC-2025-0134), and deny.toml no longer ignores that advisory (#1255).
  • A revoked gateway can no longer delegate with a felix-controlplane token it got before the revocation (#1205). /token/delegate checked the revocation cutoffs against the user’s token only, and trusted token.delegate from the caller’s token, which stays valid for its TTL. Now /token/delegate and actor_token on /token/exchange refuse, with actor_refused, a gateway whose token predates a revocation of it or that current RBAC no longer grants token.delegate. A refresh of a user chain bound to a gateway is refused, and the chain ended, once RBAC no longer grants the gateway token.delegate.
  • Revocation cutoffs are read on the Raft leader when minting or delegating, so a member that has not applied a revocation yet no longer mints past it.
  • Deleting an IdP issuer, or narrowing or replacing what it trusts (an audience dropped, the subject or groups claim changed, or jwks_url or discovery_url changed), revokes the tenant’s sessions, as removing a rule that reached an IdP group does. Adding an audience revokes nobody.
  • felix_client::Seed and ClusterClient::connect_seeds (#1227): a cluster client’s seeds can be host:port names as well as addresses. connect and connect_with_policy still take SocketAddrs and are unchanged.
  • Brokers export felix_revocation_feed_last_success_seconds (set when they start following the feed, then at each good poll) and felix_revocation_feed_poll_errors_total, and log a warning, repeated every minute, while they cannot poll the revocation feed. docs/auth.md now states how revocation fails open meanwhile, and the clock-skew bound on revocation cutoffs across control-plane members.
  • Replay rings and cache watch queues are bounded by bytes as well as entries (#1201). FELIX_REPLAY_RING_MAX_BYTES (64 MiB per stream shard) drops a ring’s oldest records first; FELIX_CACHE_WATCH_MAX_BYTES (64 MiB per watch) ends a watch that would go over as lagged, as a full queue does. Each has an optional broker-wide _TOTAL. New gauges: felix_replay_ring_bytes, felix_cache_watch_queue_bytes.
  • Brokers exchange storage identities in the peer handshake, behind a new capability bit, STORAGE_IDENTITY (1 << 10), so an older peer sees the frames it always did (#1266). A leader asks for each follower’s identity at the start of every exchange and starts the follower’s cursors over when it changes, so a follower that lost its disk is no longer reported caught up on an idle shard. Replica reports carry the identity each follower was counted under as replica_storage_ids, omitted when empty. Placement does not read it yet.
  • Each broker data directory now has a storage identity (#1266): 16 random bytes kept in <FELIX_DURABLE_STORAGE_DIR>/storage-id, created on the first start, kept across restarts, and replaced when the file is missing or does not decode. A broker sends it as storage_id with its registration; a broker without durable storage sends a new one each start. felix-broker restore-point writes a new one.
  • Changelog entries are now collected from changelog.d/ fragments, one file per change, so pull requests no longer conflict on CHANGELOG.md. scripts/changelog_assemble.py folds them into the release section, and the release workflow refuses a tag while fragments remain. Entries written directly in CHANGELOG.md still work. See changelog.d/README.md.
  • A subscription’s feeder takes room in its writer-lane share without a lock or a wakeup registration, and a written frame notifies only when the feeder is waiting. The byte gauges (felix_sub_egress_bytes, felix_sub_queue_bytes, felix_cache_watch_queue_bytes, felix_replay_ring_bytes) are looked up once per subscription or ring instead of on every batch, and a replay ring reports once per publish.
  • YAML parser: serde_yaml_ng replaced by serde-saphyr (#1211). The broker, control plane and felix-client read their config files with serde-saphyr 1.3, a pure-Rust parser. serde_yaml_ng pulled in unsafe-libyaml, which is unmaintained (RUSTSEC-2024-0449). felix-client users get the new dependency in place of serde_yaml_ng. A config file that loaded before loads to the same values, and the 1.1 spellings that were refused still are: yes/no/on/off for a boolean, 0777, a << merge key, a duplicate key, an unknown key. Some inputs behave differently:
    • Now accepted: a quoted number or boolean (max_frame_bytes: "1024", ack_on_commit: "true"), 1_000, a file that is only ~ or null, and a tab before the first key.
    • Now refused: an explicit tag that contradicts the setting’s type (!!str 5 for a number, !!int "5" for a string); a tag the parser does not know (!custom x, which used to read as x); and a bare or null tls.cert_path/key_path in the control plane file (cert_path: or cert_path: ~), which used to be read as the path "" or ~. A quoted cert_path: '' is still accepted, as before. !!binary on a string setting is still kept as written, not decoded.
    • Parse errors name the key and give a line and column. A wrong-typed value is no longer quoted, since it could be a token; an unknown enum value still is.
    • --print-config puts double quotes, not single quotes, around strings that need them, and also quotes yes and 1_000. The output still reads back to the same values.
  • An auth change that takes rights back (an RBAC rule removed, an issuer deleted or narrowed) and its revocation are one store write: one Raft entry (metadata level 9) or one Postgres transaction. A failed request changed nothing and is safe to retry. Below level 9 the Raft proposer writes them separately, as before.
  • /token/refresh does its store reads before spending the refresh token, and spends it in the same write that stores the replacement (Raft level 9), so a failed refresh no longer turns the client’s retry into a replay that ends the chain.
  • /token/delegate reads the subject’s and the caller’s revocation cutoffs in one leader read.
  • A spent refresh token presented again with the right secret within FELIX_REFRESH_REUSE_GRACE_SECONDS (30 by default; 0 turns it off) of its rotation, while its replacement is unused, gets a working token instead of revoking the chain. This covers a refresh whose answer was lost and two concurrent refreshes. Replay detection is blind for that window. Postgres migration 0027_refresh_token_rotation; under Raft, metadata level 9.
  • Deleting an IdP issuer the tenant does not have answers 404 and revokes nobody.

Upgrade note. A Raft control plane needs every member at metadata level 9 before auth changes and their revocations, and refresh rotations, are single entries.

  • FELIX_SUB_QUEUE_MAX_BYTES now counts only what waits in a subscriber’s broker queue, and its default drops from 256 MiB to 128 MiB (#1195). Frames in the writer lanes count against a new per-connection bound, FELIX_SUB_EGRESS_MAX_BYTES (256 MiB), shared out equally among the connection’s subscriptions, so one whose stream stalls stops at its share and never holds up the others. It is not a hard ceiling: a subscription with nothing in the lanes always admits one batch, so each can hold the larger of its share and one batch. FELIX_SUB_QUEUE_MAX_BYTES_TOTAL still covers a batch until it is written. New gauge: felix_sub_egress_bytes.
  • Upgrade notes for the follower-suspicion fixes (#1247). The order check on finalizing fleet features runs on the control-plane member that serves the finalize request. During a rolling control-plane upgrade a member still on the previous release skips it, so finish upgrading every member before finalizing majority_ack, lease_free_reads or fenced_caches, or finalize them in order: generation_start first, then majority_ack, then the other two. A Raft control plane records a report’s by_round only once every member is at metadata level 11; until then placement fails a leader over on its lease, not its followers’ word. A Postgres control plane runs migration 0029 on start. Remove FELIX_QUORUM_READS=lease before upgrading a fleet that has finalized lease_free_reads; otherwise each upgraded broker is refused at registration. The refusal names the variable: unset it and restart the broker. The PR performance comment now marks a change that is 10% or more worse but not statistically significant as inconclusive rather than passing, and shows each row’s trial spread. Before this, a noisy cell could more than double its latency and still get a green mark.
  • A leader that crashed under FsyncMode::Periodic (the default) or None and restarted before it was failed over no longer reopens its old generation with a log that may be short of what it acknowledged (#1295). Under Quorum it and a follower that lagged behind the lost records made a majority and could write over acknowledged offsets. A start that rotated the storage id now leads no replicated shard from a copy that may be short: the copy it led, and one it followed at an older generation than it is named leader at until a leader ships to it again. The open is refused before it writes anything, and the copy is given up on disk as a ballot naming no leader (DiskLog::give_up_generation), so no later start leads it either. Replication passes over such a shard meanwhile, so nothing ships, reports or counts toward a mark at the old generation, and the broker refuses a successor’s promotion fence from that copy in any replica set but two. Instead it asks every follower where its copy ends and reports that, flagged given_up (an optional report field, stored by the control plane at Raft metadata level 15 and in Postgres migration 0033). Replica reports name the leader’s own storage id (replica_storage_ids). Placement promotes a follower the report names caught up once the leader is registered under another id or reports it gave the shard up, without waiting for its lease. When a fresh report from the restarted leader measures every follower and none holds more than it does, placement names it leader again at a new generation, so a Leader stream whose followers lagged does not wait for an operator. Otherwise the shard waits. A shard with no followers is led from its only copy as before, and a clean restart is unchanged. TLA+: FelixShardLeaderCrashKeepsGen.cfg and FelixShardLeaderCrashReclaimBlind.cfg (violate AckedSurvive), FelixShardLeaderCrashGivesUp.cfg and FelixShardLeaderCrashReclaim.cfg (pass).
  • A follower that crashed under FsyncMode::Periodic (the default) or None is no longer counted, or promoted, on records it acknowledged before the crash and may have lost (#1284). The broker now keeps <data dir>/run-state: it records a run under way before any store opens, and a clean stop once a drain has flushed every store with nothing cut short and no log poisoned. A start that finds no clean stop after a Periodic or None run writes a new storage id before it registers, so leaders copy the follower again and placement stops counting its old reports. A clean stop, and a crash under OnCommit, keep the id. The first start on this release rotates only if startup would repair a shard, so an orderly roll keeps every id. A leader’s own crash is covered by #1295, below. felix-broker restore-point records a clean stop when it finishes. The shutdown drain now flushes the cache, consumer-group, dead-letter and counter logs too, not only stream logs (Broker::shutdown_storage, StorageApi::shutdown). TLA+: FelixShardCrashKeepsId.cfg (the passing case with a partial suffix loss is too large for CI and is not model-checked).
  • Placement no longer counts a wiped follower from a report its leader made over a connection to the follower’s old process (#1281). The window was open only while brokers or control-plane instances were being rolled and the follower’s registration lost its storage identity. A follower now gives its node incarnation in the peer handshake beside its storage identity (the new NODE_INCARNATION peer bit), the leader starts a follower’s copy over when its storage identity changes, or its incarnation when it sends no storage identity, and reports carry the incarnation it measured (replica_incarnations). A follower that sends none still gets the incarnation registered when the report arrives. PeerRequester::storage_identity is now identity, returning a PeerIdentity.
  • Deleting many streams at once, as a namespace delete does, starts one thread per store to remove the retired directories instead of one per directory, which raced on the same entries and logged spurious warnings. Directory renames and stream marker and floor writes during a delete no longer run on the async workers. Broker::restore_stream_marker is now async.
  • With FELIX_REPLAY_RING_MAX_BYTES_TOTAL set, a busy stream could keep every other stream’s replay ring empty, since a ring made room only from its own records. A ring with nothing left to give up now keeps its newest batch, exceeding the total by at most that batch.
  • The power-loss test model no longer mixes up a file with an earlier one that had the same inode number (#1221). A file created after another was deleted could get its inode number, and the model then spliced the new file’s bytes onto the old one’s, or gave a deleted name whose unlink was not flushed the new file’s bytes after a crash. Files are now told apart by birth time and inode generation as well.
  • A broker following the control plane’s catalog no longer misses tenant, namespace, stream or cache changes (#1224). Three ways to lose one are closed, on Postgres and in memory: a snapshot now reads its items and its next_seq at one point, so a delete landing between the two reads is no longer counted but unseen; Postgres hands out each change table’s seq in commit order, so a reader can no longer pass a change that is still committing; and a full page of changes (FELIX_CONTROLPLANE_CHANGES_LIMIT, 1000 by default) answers with the seq after its last change instead of the feed’s tail, which skipped everything past the first page. The node and shard-assignment feeds had the same page bug and are fixed too. Brokers now follow each feed page by page within a sync pass, up to 64 pages. The wire format is unchanged; an older broker against a fixed control plane catches up one page per sync interval.
  • A broker no longer moves a change-feed cursor backwards (#1224). On Raft a control-plane member answers from its own applied state, so a broker whose request reached a lagging member could replay changes it had already applied (a stale delete removing a stream created again since) or, for shard assignments, re-snapshot onto the member’s older view. The catalog sync and the shard watch now ignore an answer from behind what they applied and ask again. Once a feed has answered from behind for 60 s it is taken as reset, and the broker takes a new snapshot of it.
  • A Postgres catalog write that waits more than 5 s for other writes to the same change feed now gives up with a retryable 503 unavailable, and writes queued in one control-plane instance no longer hold pooled connections while they wait, so a long cascading delete cannot starve reads (#1224). The commit-order guarantee holds once every control-plane instance runs this release. A Postgres write that gives up waiting answers 503 busy (nothing written, safe to retry); felixctl retries only busy, up to three times over about two seconds. A Raft 503 unavailable may have committed, so it is not retried.
  • The durable mark no longer moves backward (#1220). A background roll’s seal that finished after a flush had already covered the new segment set the mark back to that segment’s header, and on an idle shard nothing moved it forward again. After a power loss, rot in acknowledged records of that segment was then repaired as a torn tail instead of refused. The mark now only moves forward, except on a truncation or reset, and a seal that started before a truncation leaves the mark alone.
  • A ClusterClient recovers when every broker changes IP at once, as in a rollout that restarts every pod (#1227). Reconnect dialled only the addresses it already held, and discovery needs one of them to answer, so the client failed every operation until it was rebuilt. When none of them answers, it now resolves its seed names, and the names brokers last advertised, again and dials the new addresses. The names are looked up at the same time, each bounded to 2 s, without holding the lock every operation reads the client through, and at most once a second for a client and the identities derived from it, including when callers give up waiting. Discovery’s names are looked up the same way. A seed name that stops resolving keeps its last addresses. felixctl and the Python, TypeScript and C clients pass the configured host:port through as a name instead of resolving it once at startup; a name that does not resolve at connect is still refused as an invalid argument.
  • A ClusterClient reconnect replaces the client only if it is still the one the failing operation used. Every operation that failed on the same broker used to throw away whatever client was in place, including a fresh replacement, and dial again with the endpoint list it had read before waiting for the lock.
  • A history campaign client no longer keeps publishing over a connection to brokers that restarted on new ports (#1223). A successful read or cache op, which opens its own connection, reset the client’s count of failures in a row, so a client could spend most of the outage campaign waiting out 10 s publish timeouts, and the campaign fell short of its floor of cache puts. A failed op over the connection now reconnects at once when the address book changed since the client connected, and only ops over the connection count toward reconnecting.
  • The power-loss test model no longer loses a sync it saw (#1218). When two flushes of one file overlapped, the one that read the file first could record its read last, so the model held less than the device would. A flush that read the active segment just before a background roll retired it could undo the seal’s sync of that segment, and the nightly sweep then built crashes that cut a sealed segment short and failed when recovery refused them. Each capture now takes a ticket before it reads, and an older read never replaces a newer one. Storage itself is unchanged.
  • A client can no longer split one shard across two writers by routing with the width of a stream that was deleted and created again at another width (#1067). The client declares the width it routed each keyed batch by in a new publish_width message, sent once per width per publish stream, and a broker advertising the new extended feature FEATURE_PUBLISH_WIDTH refuses a keyed batch routed by a width the stream does not have: acknowledged publishes fail with the new error code stale_width (retry, 17), and the client drops the width so the retry asks for the current one; fire-and-forget publishes are dropped. Against such a broker the Rust client also asks a stream’s width again every 5 seconds while it publishes to it, so a fire-and-forget or idle client catches up. A ClusterClient’s widths and an idempotent producer’s numbering follow the same rules, and their batches are declared by the width that routed or numbered them. Under a new width a producer starts the stream’s sequences again under a fresh producer id for that stream, except while a batch on it is in doubt: that batch keeps the width and id it was numbered by until it settles, and a re-send under a width that has since gone is refused with an error saying so. The new IdempotentProducer::abandon_in_doubt(tenant, namespace, stream) gives such batches up and numbers the stream afresh under a new id; an abandoned batch may land twice. IdempotentProducer::producer_id keeps returning the first id; the new IdempotentProducer::producer_id_for(tenant, namespace, stream) returns the id a stream’s batches go out under. Older brokers and clients exchange the frames they always did.
  • FELIX_SUB_QUEUE_MAX_BYTES_TOTAL and FELIX_SUB_QUEUE_MAX_BYTES above 4 GiB were silently cut to 4 GiB. Larger budgets now hold as configured; above 4 GiB they count in units of 2 bytes (4-8 GiB), 4 bytes (8-16 GiB), 8 bytes (16-32 GiB) and so on, each batch rounded up to a whole unit (#1200).
  • A consumer group could pass a record that was neither finished nor listed as dead. The dead letter is written before the cursor passes the record, but replication shipped cursors first, the two logs flush separately, and a follower’s dead letters can be replaced by a newer leader’s, so a copy could hold a cursor without the dead letter it relied on and serve it once promoted. Each cursor now records the dead-letter log it needs (its end and the generation that wrote it) and the group’s origin; dead letters are synced before a cursor that needs them; and before a broker serves a shard’s groups, a cursor that no longer stands is put back to the latest one that does or to its origin, never removed. Dead letters ship before cursors. A promoted stream leader fences both group logs in one round beside its records, takes the dead-letter log of the replica furthest ahead and merges that replica’s cursors into its own, on replicas that offer the new internal capability GROUP_FENCE (1 << 8). Group logs carry generation labels like the records, and a Quorum stream’s report leaves out a follower short of the pair a majority holds. The cursor value is a new 33-byte layout, written once the fleet finalizes the new group_cursor_dependencies feature, and by a single broker from the start; an older build cannot read it (#1202).
  • A long counter-log catch-up now reports progress as it goes (#1199). Counters ship in the same five-second slices as records, so placement no longer drops a restore that is still moving as stalled.
  • A stream deleted and created again under the same name no longer reopens the deleted stream’s logs once the fleet finalizes the new stream_incarnations feature (#1208). Each created stream then gets an incarnation, higher for every creation of its name (the control plane’s clock, raised above the name’s last deleted one), carried in the catalog and the shard assignments, and brokers keep each incarnation’s records, group positions and dead letters in directories named with it. Every stream delete after the finalize, cascades included, records a tombstone for the name, kept for good, served in GET /v1/streams/snapshot as tombstones and on the Deleted change as deleted_through. A broker deletes a stream’s data only on one of those, or on an operator’s removal: every incarnation up to the one named goes from every store, through .deleted/ and a marker under .streams/ that lets a delete cut short finish at the next start. A broker that was down through a delete and a recreate deletes the old data from the snapshot’s tombstones. It never moves a name to a lower incarnation or one the control plane deleted, and refuses a record without the incarnation for a name it has seen incarnated, counting both in felix_broker_stream_incarnation_refused_total. A delete from before the finalize keeps the data as before, so a stream created again at incarnation 0 reopens it consistently with the quorum marks kept for it. The first assignment of an incarnated stream’s shard takes its change’s sequence number as its generation. A control-plane restore can still make that equal to a deleted incarnation’s, so leaders keep follower positions and quorum marks per incarnation too, replica reports carry an optional stream_incarnation that the control plane checks against the assignment’s, and a stream delete takes its shards’ replica reports in the in-memory and Raft stores as Postgres already did. A follower refuses replication for an incarnation it does not hold (the new internal Incarnated frame, kind 40). The catalog sync moves past each change it applies, a shard assigned before its stream registers waits without holding up the others, and a stream whose namespace or tenant was removed first is still removed when its cascading delete arrives. Existing streams are incarnation 0 and keep their directory names. felix-broker inspect segments takes --incarnation; backup points carry the incarnation, and restore-point sets each stream’s back to the point’s. A broker joining a fleet with many deleted names records their tombstones with one flush, and POST /v1/resources answers with the incarnation the store gave each stream. Once incarnations are finalized, a create proposes the clock even while a Raft member has not yet heard from a peer since it restarted. Such a peer counts at level 10 then; a peer that answers with a lower level still holds level-10 commands back.
  • Upgrade notes for stream_incarnations (#1208): it needs every Raft control-plane member at metadata level 10 and every broker on this release before it can be finalized. A Postgres control plane runs migration 0028 once for the database. An older Postgres instance still running after the finalize cannot create streams, and the database stamps its shard assignments with the stream’s incarnation, but brokers refuse its catalog answers for incarnated streams, so upgrade every instance before finalizing.
  • A cache deleted and created again under the same name no longer serves or replicates the deleted cache’s entries and counters once the fleet finalizes the new cache_incarnations feature (#1209). Each created cache then gets an incarnation, carried in the catalog and its shard assignments, and brokers keep each incarnation’s cache and counter logs in directories named with it. Every cache delete after the finalize, cascades included, records a tombstone served in GET /v1/caches/snapshot as tombstones and on the Deleted change as deleted_through, and brokers delete a cache’s data, from both stores, only on one of those or on an operator’s removal, so a broker that was down through the delete still deletes it. Cache and counter replication requests name the incarnation and a follower holding another refuses them. Stale registrations are refused and counted in felix_broker_cache_incarnation_refused_total. Finalizing needs every Raft member at metadata version 14; Postgres gets migration 0032. Existing caches stay at incarnation 0 with their directories and data. A broker without durable storage drops a deleted cache’s entries.
  • Broker::remove_cache is now an operator’s removal and deletes the cache’s data; the catalog path uses Broker::remove_cache_from_catalog.
  • Raft members on different builds no longer diverge when a finalize of cache_incarnations or stream_incarnations is committed through a member that predates it. Create and delete entries now carry an incarnations flag (metadata level 14), set only once every member is at level 14, and apply decides incarnations and tombstones from the entry alone (see docs/metadata-raft-design.md).
  • Rolling a v0.6.0-preview.7 Raft control plane to this release no longer splits it if stream_incarnations is finalized through a preview.7 member (which accepts the unknown finalize): a stream create from an older proposer that names no incarnation is stored at 0, and its delete records no tombstone, on every member. Clusters that ran an unreleased build at metadata level 10 to 13 and logged such an entry replay it differently now and must rebuild those members from a snapshot.
  • A broker that held a deleted stream’s shard now takes the recreated stream’s assignment even when its generation is lower, as after a control-plane restore rewinds the change seq (#1209). Shard ownership is ordered by (incarnation, generation), as the shard lifecycle already was. Deleting a namespace or tenant on the in-memory or Raft control plane now also removes its streams’ and caches’ shard assignments and replica reports, as deleting the stream itself and Postgres’s cascade already did; before, a stream created again under the name was assigned as an update of the deleted one’s shard. On Raft, deletes take shard assignments and replica reports this way only once every member is at metadata version 10, the same gate as stream incarnations; below it every member keeps what the older release keeps, so members never disagree.
  • Under Raft, the leader refuses to add a control-plane member whose build is below the metadata version the group already uses, or that reports none; the error names both versions. That version is the one a finalized fleet feature needed (10 for stream_incarnations), or the newest one any committed entry needed, which the state machine now tracks and carries in its snapshot (#1209). A member already in the group that comes back on an older build, as after a lost volume, is no longer sent entries or a snapshot once it reports an older version, or while it has lost its log and not yet reported one; the leader logs it once and sets felix_meta_raft_member_below_floor{member} to 1. A held member that is behind gets no heartbeats either, so it may call elections the others refuse, and after a long hold it can cost one extra election once released. A member that is slow or unable to answer is not held. Either member would refuse entries the others apply and diverge from them. A held voter is one fewer failure the group survives until it is upgraded. A group that ran unreleased builds of main may have members whose tracked level differs, since those builds applied some entries without raising it; while such a member leads, it can admit or feed a member the others would hold.
  • Truncation, reset and restore are refused on a log that has failed (#1222). A flush that failed to sync a retired segment marked the roll failed while its seal was still running, and a truncation could then cut that segment and report success before the seal padded it back with zeros.
  • A truncation or reset no longer races a background segment roll (#1222). The roll’s seal could put back the length a truncation had cut, or cut off records appended after it. Truncation, reset and restore now wait for the seal.
  • Placement takes a follower’s word against a leader only when that leader’s own replica report says it serves the shard by round (by_round): it acknowledges by its followers and confirms its reads by a round (#1247). Before, a leader still reading on its lease could be replaced while that lease was valid and keep serving consumer-group writes and state reads, either because it ran with FELIX_QUORUM_READS=lease or because it had not yet heard on a heartbeat that the fleet finalized lease_free_reads. A report without the field, from an older broker, reads as the lease. The flag counts only from the leader’s current node incarnation, so a leader’s report from before a restart no longer counts once it registers again. A placement pass that read the old incarnation can still act on that report, so in a fleet that reads by round a restarted leader refuses consumer-group writes, which it would acknowledge on its lease alone, until a replication pass over the shard made after it registered. A pass that ran before registration answered saw no fleet features and does not count. Under Postgres it counts only with the report that set it, so a control plane on the previous release cannot leave a stale one behind. Postgres migration 0029_replica_report_by_round; under Raft, metadata level 11.
  • A broker started with FELIX_QUORUM_READS=lease no longer reports lease_free_reads. The finalize dry run names it, and the finalize is refused while it serves; once the feature is finalized such a broker is refused at registration, like one with FELIX_INTERNAL_FENCE=false.
  • Placement fails a Quorum stream over on its followers’ word only once generation_start, majority_ack and lease_free_reads are all finalized, which is what brokers need to acknowledge and read without the lease (#1247). Before, a fleet with the last two but not generation_start could have a live leader replaced while it still served reads on its lease. The control plane now refuses with 409 to finalize majority_ack before generation_start, and lease_free_reads or fenced_caches before both; the finalize dry run lists what is missing as requires. A fleet that already finalized them out of order fails over on the lease until generation_start is finalized.
  • A failed replica commit-offset write is retried once a second until it lands, rather than only when the leader’s next batch arrives, which never happens on an idle shard. A closed log now refuses to write its replica state (#1248).
  • A replica commit-offset write that failed was not retried until the offset rose. The next batch carrying the same offset now writes it again. The follower still acknowledges the batch when the write fails, since no acknowledged record depends on the offset (#1248).
  • felix_broker_promotions_opened_total now counts a promotion when the shard is let go to serve rather than when its fence finishes, so a refused open that is retried is counted once (#1250).
  • group_describe and the dead-letter list now check that the broker still leads the shard, as state reads do (#1252). A deposed leader could answer them from stale state. They are refused with shard_unavailable on a lapsed lease, or leadership_lost when a round finds a newer leader.
  • A broker promoted before its registration answered no longer opens a shard on the lease (#1261). Until then its fleet view is empty, so on a fleet that finalized generation_start and majority_ack, an unreachable old leader read as a peer without the fence and the shard opened without fencing a majority, which could lose records the old leader acknowledged. The shard now stays closed until the broker has its fleet view, then fences as usual.
  • A Raft control plane restarted with one member down no longer drops storage identities from registrations and reports until that member answers. A member that has not answered counts at the level the log already depends on, not 0. Before, a wiped follower’s registration then cleared its identity and an older report could promote it empty.
  • A follower without a storage identity now counts only from a report taken at its current incarnation. The control plane stamps each report with the leader that made it and every named node’s registered incarnation.
  • A leader that dies partway through a move no longer leaves its last report to expire because the move’s steps raised the generation (#1250). The report stays believed while its leader is still the assignment’s leader.
  • A Leader shard is no longer promoted on a previous leader’s report once another node leads: what the new leader acknowledged may be on it alone. A new leader that dies before its first report leaves the shard waiting for it; before, it was promoted on the old report for a while and then stalled. Quorum shards still promote on it. Postgres migration 0031_replica_report_stamp; under Raft, metadata level 13.
  • Placement no longer promotes a follower whose storage changed since its leader’s last report (#1266). The control plane keeps each broker’s registered storage identity and the identity each report names for every follower, and counts a follower only while the two match. A follower that lost its disk comes back under a new identity and is not promoted, even on a fresh report. One that only restarted keeps its identity and still counts. A follower with no identity on either side (an older broker or control plane) is judged by the report’s age alone, as before.
  • A dead leader’s last replica report no longer expires while placement or the control plane is slow, which left the shard unplaceable until an operator abandoned it (#1250). While the assignment is still at the report’s generation and the leader is down, the report’s age is measured to the leader’s last heartbeat, for followers whose identity matches. A live leader’s report still expires. Registration answers 400 for a malformed storage_id. Postgres migration 0030_storage_identity; under Raft, metadata level 12.
  • A Quorum leader counts a follower’s confirmed position toward the mark only after the follower has answered in the same pass, over a connection carrying the storage identity the position was learned under. A follower level with the tail is sent its last record again while the mark lags its position. A follower wiped after confirming more than the mark, beside another that failed fast, could release records only the leader held. The counter mark follows the same rule (#1266).
  • A leader that crashed under periodic or none and gave its shard up is replaced by a follower that holds at least what it kept, and named leader again only when every follower holds less (#1314). Its probe now reports a follower level with it as caught up, so the outcome no longer depends on whether the control plane sees the leader’s new storage id or its probe first. A follower counts from the probe only when measured under the identity and incarnation it is registered at now.
  • A subscription’s feeder gives a batch back to FELIX_SUB_QUEUE_MAX_BYTES once its frame is in the writer lanes, not before encoding it. The earlier release sat on the delivery path and cost single-publisher streams 1-2% throughput against 0.6.0-preview.7.
  • Without FELIX_SUB_QUEUE_MAX_BYTES_TOTAL, each subscriber can hold up to FELIX_SUB_QUEUE_MAX_BYTES or its largest batch, whichever is larger, and with the per-subscriber bound at 0 only queue capacity times batch size limits it; the docs now say so and recommend setting the total in production. They also say that under block one subscriber that stops reading holds a shard’s publishers with no timeout, and that the replay ring and cache watch queues are bounded by entry count only (#1201).
  • The replication design no longer says a lagging non-successor follower is skipped while a shard drains: it is not gated on, but the drained pass still ships to it and waits.
  • Design for replicas electing a leader while the control plane is down (#1009), not built yet: “Electing without the control plane” in docs/replication-design.md, with pre-vote, stickiness, a joint fence on every set change, kept with the set and acknowledged on jointly until the control plane flips to the new set, which it does only after the joint fence; cut-overs that wait for the drain; and the failover bound. The TLA+ model gains SetCheck, JointFence, Adopt, AdoptMatch, Swaps and PreVote, the persisted setGen, RefuseStaleSet, CpAdopt, the Swap cut-over and the AdoptedMatchesOpened and SetChangesJoint invariants, and AckedHeldByLeader is checked per generation where replicas elect. New configurations: FelixShardElectSwap.cfg and its no-joint twin, FelixShardElectAdopt.cfg (two writes nightly), FelixShardElectAdoptPromote.cfg, FelixShardElectPreVote.cfg, FelixShardElectStepDown.cfg, FelixShardElectNoBallotAcks.cfg, and the FelixShardWalkElect.cfg and FelixShardWalkElectSwap.cfg walks with their twins. FelixShardElectHandoff.cfg now runs with the set check on.