Felix 0.6.0-preview.8
Released 2026-10-16 as a preview. The full history is in CHANGELOG.md.
The eighth preview of 0.6.0. A broker now has a storage identity that its node
registers, and placement counts a follower toward promotion only while the
identity its leader reported matches, so a follower that lost its disk is
never promoted on an old report. Streams get incarnations, and caches get them
too, so a stream or cache deleted and created again under the same name starts
empty on every broker; both are behind finalized features. A leader that
crashed under FsyncMode::Periodic or None no longer reopens its old
generation with a log that may be short of what it acknowledged (#1294,
#1305). The single-publisher fanout path is back to preview.7’s throughput,
with large-payload fanout faster (#1309).
Upgrade notes.
- A Raft control plane needs every member upgraded before these take effect,
each at its own metadata level. Until then the proposer leaves the new data
out, the same on every member:
- Level 9: an auth change and its revocation, and a refresh rotation, are single entries. Below it they are written as the two older entries they combine.
- Level 10: stream incarnations (an upgraded member refuses the
stream_incarnationsfinalize with 409), and deletes that take shard assignments and replica reports. - Level 11: a replica report’s
by_round. Below it, leaders are failed over on their lease. - Level 12: storage identity. Below it, followers read as unverified.
- Level 13: a report’s
leaderandnode_incarnations. Below it, placement reads reports as at level 12. - Level 14: cache incarnations (an upgraded member refuses the
cache_incarnationsfinalize with 409). Below it, caches are created at incarnation 0 and no cache delete records a tombstone. - Level 15: a report’s
given_up. Below it, placement replaces a leader that gave a shard up only on its storage identity.
- A Postgres control plane runs migrations
0027through0033on start. stream_incarnationsandcache_incarnationsare finalized by an operator after the upgrade: every control-plane member at level 10 and 14, and every broker on this release, first. With Postgres, upgrade every instance before finalizing, since an older instance still running after the finalize cannot create streams or caches. See the upgrades page.- A cluster that ran an unreleased build at metadata level 10 to 13 and logged a stream create that names no incarnation, or its delete, replays it differently now and must rebuild those members from a snapshot (#1287).
Security
Section titled “Security”- The Python and TypeScript SDKs read their CA file with
rustls-pki-typesinstead of the unmaintainedrustls-pemfile(RUSTSEC-2025-0134), anddeny.tomlno longer ignores that advisory (#1255). - A revoked gateway can no longer delegate with a
felix-controlplanetoken it got before the revocation (#1205)./token/delegatechecked the revocation cutoffs against the user’s token only, and trustedtoken.delegatefrom the caller’s token, which stays valid for its TTL. Now/token/delegateandactor_tokenon/token/exchangerefuse, withactor_refused, a gateway whose token predates a revocation of it or that current RBAC no longer grantstoken.delegate. A refresh of a user chain bound to a gateway is refused, and the chain ended, once RBAC no longer grants the gatewaytoken.delegate. - Revocation cutoffs are read on the Raft leader when minting or delegating, so a member that has not applied a revocation yet no longer mints past it.
- Deleting an IdP issuer, or narrowing or replacing what it trusts (an
audience dropped, the subject or groups claim changed, or
jwks_urlordiscovery_urlchanged), revokes the tenant’s sessions, as removing a rule that reached an IdP group does. Adding an audience revokes nobody.
felix_client::SeedandClusterClient::connect_seeds(#1227): a cluster client’s seeds can behost:portnames as well as addresses.connectandconnect_with_policystill takeSocketAddrs and are unchanged.- Brokers export
felix_revocation_feed_last_success_seconds(set when they start following the feed, then at each good poll) andfelix_revocation_feed_poll_errors_total, and log a warning, repeated every minute, while they cannot poll the revocation feed.docs/auth.mdnow states how revocation fails open meanwhile, and the clock-skew bound on revocation cutoffs across control-plane members. - Replay rings and cache watch queues are bounded by bytes as well as entries
(#1201).
FELIX_REPLAY_RING_MAX_BYTES(64 MiB per stream shard) drops a ring’s oldest records first;FELIX_CACHE_WATCH_MAX_BYTES(64 MiB per watch) ends a watch that would go over as lagged, as a full queue does. Each has an optional broker-wide_TOTAL. New gauges:felix_replay_ring_bytes,felix_cache_watch_queue_bytes. - Brokers exchange storage identities in the peer handshake, behind a new
capability bit,
STORAGE_IDENTITY(1 << 10), so an older peer sees the frames it always did (#1266). A leader asks for each follower’s identity at the start of every exchange and starts the follower’s cursors over when it changes, so a follower that lost its disk is no longer reported caught up on an idle shard. Replica reports carry the identity each follower was counted under asreplica_storage_ids, omitted when empty. Placement does not read it yet. - Each broker data directory now has a storage identity (#1266): 16 random
bytes kept in
<FELIX_DURABLE_STORAGE_DIR>/storage-id, created on the first start, kept across restarts, and replaced when the file is missing or does not decode. A broker sends it asstorage_idwith its registration; a broker without durable storage sends a new one each start.felix-broker restore-pointwrites a new one. - Changelog entries are now collected from
changelog.d/fragments, one file per change, so pull requests no longer conflict onCHANGELOG.md.scripts/changelog_assemble.pyfolds them into the release section, and the release workflow refuses a tag while fragments remain. Entries written directly inCHANGELOG.mdstill work. Seechangelog.d/README.md.
Changed
Section titled “Changed”- A subscription’s feeder takes room in its writer-lane share without a lock
or a wakeup registration, and a written frame notifies only when the feeder
is waiting. The byte gauges (
felix_sub_egress_bytes,felix_sub_queue_bytes,felix_cache_watch_queue_bytes,felix_replay_ring_bytes) are looked up once per subscription or ring instead of on every batch, and a replay ring reports once per publish. - YAML parser:
serde_yaml_ngreplaced byserde-saphyr(#1211). The broker, control plane andfelix-clientread their config files withserde-saphyr1.3, a pure-Rust parser.serde_yaml_ngpulled inunsafe-libyaml, which is unmaintained (RUSTSEC-2024-0449).felix-clientusers get the new dependency in place ofserde_yaml_ng. A config file that loaded before loads to the same values, and the 1.1 spellings that were refused still are:yes/no/on/offfor a boolean,0777, a<<merge key, a duplicate key, an unknown key. Some inputs behave differently:- Now accepted: a quoted number or boolean (
max_frame_bytes: "1024",ack_on_commit: "true"),1_000, a file that is only~ornull, and a tab before the first key. - Now refused: an explicit tag that contradicts the setting’s type
(
!!str 5for a number,!!int "5"for a string); a tag the parser does not know (!custom x, which used to read asx); and a bare or nulltls.cert_path/key_pathin the control plane file (cert_path:orcert_path: ~), which used to be read as the path""or~. A quotedcert_path: ''is still accepted, as before.!!binaryon a string setting is still kept as written, not decoded. - Parse errors name the key and give a line and column. A wrong-typed value is no longer quoted, since it could be a token; an unknown enum value still is.
--print-configputs double quotes, not single quotes, around strings that need them, and also quotesyesand1_000. The output still reads back to the same values.
- Now accepted: a quoted number or boolean (
- An auth change that takes rights back (an RBAC rule removed, an issuer deleted or narrowed) and its revocation are one store write: one Raft entry (metadata level 9) or one Postgres transaction. A failed request changed nothing and is safe to retry. Below level 9 the Raft proposer writes them separately, as before.
/token/refreshdoes its store reads before spending the refresh token, and spends it in the same write that stores the replacement (Raft level 9), so a failed refresh no longer turns the client’s retry into a replay that ends the chain./token/delegatereads the subject’s and the caller’s revocation cutoffs in one leader read.- A spent refresh token presented again with the right secret within
FELIX_REFRESH_REUSE_GRACE_SECONDS(30 by default; 0 turns it off) of its rotation, while its replacement is unused, gets a working token instead of revoking the chain. This covers a refresh whose answer was lost and two concurrent refreshes. Replay detection is blind for that window. Postgres migration0027_refresh_token_rotation; under Raft, metadata level 9. - Deleting an IdP issuer the tenant does not have answers
404and revokes nobody.
Upgrade note. A Raft control plane needs every member at metadata level 9 before auth changes and their revocations, and refresh rotations, are single entries.
FELIX_SUB_QUEUE_MAX_BYTESnow counts only what waits in a subscriber’s broker queue, and its default drops from 256 MiB to 128 MiB (#1195). Frames in the writer lanes count against a new per-connection bound,FELIX_SUB_EGRESS_MAX_BYTES(256 MiB), shared out equally among the connection’s subscriptions, so one whose stream stalls stops at its share and never holds up the others. It is not a hard ceiling: a subscription with nothing in the lanes always admits one batch, so each can hold the larger of its share and one batch.FELIX_SUB_QUEUE_MAX_BYTES_TOTALstill covers a batch until it is written. New gauge:felix_sub_egress_bytes.- Upgrade notes for the follower-suspicion fixes (#1247). The order check
on finalizing fleet features runs on the control-plane member that serves the
finalizerequest. During a rolling control-plane upgrade a member still on the previous release skips it, so finish upgrading every member before finalizingmajority_ack,lease_free_readsorfenced_caches, or finalize them in order:generation_startfirst, thenmajority_ack, then the other two. A Raft control plane records a report’sby_roundonly once every member is at metadata level 11; until then placement fails a leader over on its lease, not its followers’ word. A Postgres control plane runs migration0029on start. RemoveFELIX_QUORUM_READS=leasebefore upgrading a fleet that has finalizedlease_free_reads; otherwise each upgraded broker is refused at registration. The refusal names the variable: unset it and restart the broker. The PR performance comment now marks a change that is 10% or more worse but not statistically significant as inconclusive rather than passing, and shows each row’s trial spread. Before this, a noisy cell could more than double its latency and still get a green mark.
- A leader that crashed under
FsyncMode::Periodic(the default) orNoneand restarted before it was failed over no longer reopens its old generation with a log that may be short of what it acknowledged (#1295). UnderQuorumit and a follower that lagged behind the lost records made a majority and could write over acknowledged offsets. A start that rotated the storage id now leads no replicated shard from a copy that may be short: the copy it led, and one it followed at an older generation than it is named leader at until a leader ships to it again. The open is refused before it writes anything, and the copy is given up on disk as a ballot naming no leader (DiskLog::give_up_generation), so no later start leads it either. Replication passes over such a shard meanwhile, so nothing ships, reports or counts toward a mark at the old generation, and the broker refuses a successor’s promotion fence from that copy in any replica set but two. Instead it asks every follower where its copy ends and reports that, flaggedgiven_up(an optional report field, stored by the control plane at Raft metadata level 15 and in Postgres migration 0033). Replica reports name the leader’s own storage id (replica_storage_ids). Placement promotes a follower the report names caught up once the leader is registered under another id or reports it gave the shard up, without waiting for its lease. When a fresh report from the restarted leader measures every follower and none holds more than it does, placement names it leader again at a new generation, so aLeaderstream whose followers lagged does not wait for an operator. Otherwise the shard waits. A shard with no followers is led from its only copy as before, and a clean restart is unchanged. TLA+:FelixShardLeaderCrashKeepsGen.cfgandFelixShardLeaderCrashReclaimBlind.cfg(violateAckedSurvive),FelixShardLeaderCrashGivesUp.cfgandFelixShardLeaderCrashReclaim.cfg(pass). - A follower that crashed under
FsyncMode::Periodic(the default) orNoneis no longer counted, or promoted, on records it acknowledged before the crash and may have lost (#1284). The broker now keeps<data dir>/run-state: it records a run under way before any store opens, and a clean stop once a drain has flushed every store with nothing cut short and no log poisoned. A start that finds no clean stop after aPeriodicorNonerun writes a new storage id before it registers, so leaders copy the follower again and placement stops counting its old reports. A clean stop, and a crash underOnCommit, keep the id. The first start on this release rotates only if startup would repair a shard, so an orderly roll keeps every id. A leader’s own crash is covered by #1295, below.felix-broker restore-pointrecords a clean stop when it finishes. The shutdown drain now flushes the cache, consumer-group, dead-letter and counter logs too, not only stream logs (Broker::shutdown_storage,StorageApi::shutdown). TLA+:FelixShardCrashKeepsId.cfg(the passing case with a partial suffix loss is too large for CI and is not model-checked). - Placement no longer counts a wiped follower from a report its leader made
over a connection to the follower’s old process (#1281). The window was
open only while brokers or control-plane instances were being rolled and
the follower’s registration lost its storage identity. A follower now gives
its node incarnation in the peer handshake beside its storage identity (the
new
NODE_INCARNATIONpeer bit), the leader starts a follower’s copy over when its storage identity changes, or its incarnation when it sends no storage identity, and reports carry the incarnation it measured (replica_incarnations). A follower that sends none still gets the incarnation registered when the report arrives.PeerRequester::storage_identityis nowidentity, returning aPeerIdentity. - Deleting many streams at once, as a namespace delete does, starts one
thread per store to remove the retired directories instead of one per
directory, which raced on the same entries and logged spurious warnings.
Directory renames and stream marker and floor writes during a delete no
longer run on the async workers.
Broker::restore_stream_markeris now async. - With
FELIX_REPLAY_RING_MAX_BYTES_TOTALset, a busy stream could keep every other stream’s replay ring empty, since a ring made room only from its own records. A ring with nothing left to give up now keeps its newest batch, exceeding the total by at most that batch. - The power-loss test model no longer mixes up a file with an earlier one that had the same inode number (#1221). A file created after another was deleted could get its inode number, and the model then spliced the new file’s bytes onto the old one’s, or gave a deleted name whose unlink was not flushed the new file’s bytes after a crash. Files are now told apart by birth time and inode generation as well.
- A broker following the control plane’s catalog no longer misses tenant,
namespace, stream or cache changes (#1224). Three ways to lose one are
closed, on Postgres and in memory: a snapshot now reads its items and its
next_seqat one point, so a delete landing between the two reads is no longer counted but unseen; Postgres hands out each change table’sseqin commit order, so a reader can no longer pass a change that is still committing; and a full page of changes (FELIX_CONTROLPLANE_CHANGES_LIMIT, 1000 by default) answers with the seq after its last change instead of the feed’s tail, which skipped everything past the first page. The node and shard-assignment feeds had the same page bug and are fixed too. Brokers now follow each feed page by page within a sync pass, up to 64 pages. The wire format is unchanged; an older broker against a fixed control plane catches up one page per sync interval. - A broker no longer moves a change-feed cursor backwards (#1224). On Raft a control-plane member answers from its own applied state, so a broker whose request reached a lagging member could replay changes it had already applied (a stale delete removing a stream created again since) or, for shard assignments, re-snapshot onto the member’s older view. The catalog sync and the shard watch now ignore an answer from behind what they applied and ask again. Once a feed has answered from behind for 60 s it is taken as reset, and the broker takes a new snapshot of it.
- A Postgres catalog write that waits more than 5 s for other writes to the
same change feed now gives up with a retryable 503
unavailable, and writes queued in one control-plane instance no longer hold pooled connections while they wait, so a long cascading delete cannot starve reads (#1224). The commit-order guarantee holds once every control-plane instance runs this release. A Postgres write that gives up waiting answers 503busy(nothing written, safe to retry);felixctlretries onlybusy, up to three times over about two seconds. A Raft 503unavailablemay have committed, so it is not retried. - The durable mark no longer moves backward (#1220). A background roll’s seal that finished after a flush had already covered the new segment set the mark back to that segment’s header, and on an idle shard nothing moved it forward again. After a power loss, rot in acknowledged records of that segment was then repaired as a torn tail instead of refused. The mark now only moves forward, except on a truncation or reset, and a seal that started before a truncation leaves the mark alone.
- A
ClusterClientrecovers when every broker changes IP at once, as in a rollout that restarts every pod (#1227). Reconnect dialled only the addresses it already held, and discovery needs one of them to answer, so the client failed every operation until it was rebuilt. When none of them answers, it now resolves its seed names, and the names brokers last advertised, again and dials the new addresses. The names are looked up at the same time, each bounded to 2 s, without holding the lock every operation reads the client through, and at most once a second for a client and the identities derived from it, including when callers give up waiting. Discovery’s names are looked up the same way. A seed name that stops resolving keeps its last addresses. felixctl and the Python, TypeScript and C clients pass the configuredhost:portthrough as a name instead of resolving it once at startup; a name that does not resolve at connect is still refused as an invalid argument. - A
ClusterClientreconnect replaces the client only if it is still the one the failing operation used. Every operation that failed on the same broker used to throw away whatever client was in place, including a fresh replacement, and dial again with the endpoint list it had read before waiting for the lock. - A history campaign client no longer keeps publishing over a connection to brokers that restarted on new ports (#1223). A successful read or cache op, which opens its own connection, reset the client’s count of failures in a row, so a client could spend most of the outage campaign waiting out 10 s publish timeouts, and the campaign fell short of its floor of cache puts. A failed op over the connection now reconnects at once when the address book changed since the client connected, and only ops over the connection count toward reconnecting.
- The power-loss test model no longer loses a sync it saw (#1218). When two flushes of one file overlapped, the one that read the file first could record its read last, so the model held less than the device would. A flush that read the active segment just before a background roll retired it could undo the seal’s sync of that segment, and the nightly sweep then built crashes that cut a sealed segment short and failed when recovery refused them. Each capture now takes a ticket before it reads, and an older read never replaces a newer one. Storage itself is unchanged.
- A client can no longer split one shard across two writers by routing with
the width of a stream that was deleted and created again at another width
(#1067). The client declares the width it routed each keyed batch by in a
new
publish_widthmessage, sent once per width per publish stream, and a broker advertising the new extended featureFEATURE_PUBLISH_WIDTHrefuses a keyed batch routed by a width the stream does not have: acknowledged publishes fail with the new error codestale_width(retry, 17), and the client drops the width so the retry asks for the current one; fire-and-forget publishes are dropped. Against such a broker the Rust client also asks a stream’s width again every 5 seconds while it publishes to it, so a fire-and-forget or idle client catches up. AClusterClient’s widths and an idempotent producer’s numbering follow the same rules, and their batches are declared by the width that routed or numbered them. Under a new width a producer starts the stream’s sequences again under a fresh producer id for that stream, except while a batch on it is in doubt: that batch keeps the width and id it was numbered by until it settles, and a re-send under a width that has since gone is refused with an error saying so. The newIdempotentProducer::abandon_in_doubt(tenant, namespace, stream)gives such batches up and numbers the stream afresh under a new id; an abandoned batch may land twice.IdempotentProducer::producer_idkeeps returning the first id; the newIdempotentProducer::producer_id_for(tenant, namespace, stream)returns the id a stream’s batches go out under. Older brokers and clients exchange the frames they always did. FELIX_SUB_QUEUE_MAX_BYTES_TOTALandFELIX_SUB_QUEUE_MAX_BYTESabove 4 GiB were silently cut to 4 GiB. Larger budgets now hold as configured; above 4 GiB they count in units of 2 bytes (4-8 GiB), 4 bytes (8-16 GiB), 8 bytes (16-32 GiB) and so on, each batch rounded up to a whole unit (#1200).- A consumer group could pass a record that was neither finished nor listed
as dead. The dead letter is written before the cursor passes the record,
but replication shipped cursors first, the two logs flush separately, and
a follower’s dead letters can be replaced by a newer leader’s, so a copy
could hold a cursor without the dead letter it relied on and serve it once
promoted. Each cursor now records the dead-letter log it needs (its end
and the generation that wrote it) and the group’s origin; dead letters
are synced before a cursor that needs them; and before a broker serves a
shard’s groups, a cursor that no longer stands is put back to the latest
one that does or to its origin, never removed. Dead letters ship before
cursors. A promoted stream leader fences both group logs in one round
beside its records, takes the dead-letter log of the replica furthest
ahead and merges that replica’s cursors into its own, on replicas that
offer the new internal capability
GROUP_FENCE(1 << 8). Group logs carry generation labels like the records, and aQuorumstream’s report leaves out a follower short of the pair a majority holds. The cursor value is a new 33-byte layout, written once the fleet finalizes the newgroup_cursor_dependenciesfeature, and by a single broker from the start; an older build cannot read it (#1202). - A long counter-log catch-up now reports progress as it goes (#1199). Counters ship in the same five-second slices as records, so placement no longer drops a restore that is still moving as stalled.
- A stream deleted and created again under the same name no longer reopens
the deleted stream’s logs once the fleet finalizes the new
stream_incarnationsfeature (#1208). Each created stream then gets anincarnation, higher for every creation of its name (the control plane’s clock, raised above the name’s last deleted one), carried in the catalog and the shard assignments, and brokers keep each incarnation’s records, group positions and dead letters in directories named with it. Every stream delete after the finalize, cascades included, records a tombstone for the name, kept for good, served inGET /v1/streams/snapshotastombstonesand on theDeletedchange asdeleted_through. A broker deletes a stream’s data only on one of those, or on an operator’s removal: every incarnation up to the one named goes from every store, through.deleted/and a marker under.streams/that lets a delete cut short finish at the next start. A broker that was down through a delete and a recreate deletes the old data from the snapshot’s tombstones. It never moves a name to a lower incarnation or one the control plane deleted, and refuses a record without the incarnation for a name it has seen incarnated, counting both infelix_broker_stream_incarnation_refused_total. A delete from before the finalize keeps the data as before, so a stream created again at incarnation 0 reopens it consistently with the quorum marks kept for it. The first assignment of an incarnated stream’s shard takes its change’s sequence number as its generation. A control-plane restore can still make that equal to a deleted incarnation’s, so leaders keep follower positions and quorum marks per incarnation too, replica reports carry an optionalstream_incarnationthat the control plane checks against the assignment’s, and a stream delete takes its shards’ replica reports in the in-memory and Raft stores as Postgres already did. A follower refuses replication for an incarnation it does not hold (the new internalIncarnatedframe, kind 40). The catalog sync moves past each change it applies, a shard assigned before its stream registers waits without holding up the others, and a stream whose namespace or tenant was removed first is still removed when its cascading delete arrives. Existing streams are incarnation 0 and keep their directory names.felix-broker inspect segmentstakes--incarnation; backup points carry the incarnation, andrestore-pointsets each stream’s back to the point’s. A broker joining a fleet with many deleted names records their tombstones with one flush, andPOST /v1/resourcesanswers with the incarnation the store gave each stream. Once incarnations are finalized, a create proposes the clock even while a Raft member has not yet heard from a peer since it restarted. Such a peer counts at level 10 then; a peer that answers with a lower level still holds level-10 commands back. - Upgrade notes for
stream_incarnations(#1208): it needs every Raft control-plane member at metadata level 10 and every broker on this release before it can be finalized. A Postgres control plane runs migration0028once for the database. An older Postgres instance still running after the finalize cannot create streams, and the database stamps its shard assignments with the stream’s incarnation, but brokers refuse its catalog answers for incarnated streams, so upgrade every instance before finalizing. - A cache deleted and created again under the same name no longer serves or
replicates the deleted cache’s entries and counters once the fleet finalizes
the new
cache_incarnationsfeature (#1209). Each created cache then gets anincarnation, carried in the catalog and its shard assignments, and brokers keep each incarnation’s cache and counter logs in directories named with it. Every cache delete after the finalize, cascades included, records a tombstone served inGET /v1/caches/snapshotastombstonesand on theDeletedchange asdeleted_through, and brokers delete a cache’s data, from both stores, only on one of those or on an operator’s removal, so a broker that was down through the delete still deletes it. Cache and counter replication requests name the incarnation and a follower holding another refuses them. Stale registrations are refused and counted infelix_broker_cache_incarnation_refused_total. Finalizing needs every Raft member at metadata version 14; Postgres gets migration 0032. Existing caches stay at incarnation 0 with their directories and data. A broker without durable storage drops a deleted cache’s entries. Broker::remove_cacheis now an operator’s removal and deletes the cache’s data; the catalog path usesBroker::remove_cache_from_catalog.- Raft members on different builds no longer diverge when a finalize of
cache_incarnationsorstream_incarnationsis committed through a member that predates it. Create and delete entries now carry anincarnationsflag (metadata level 14), set only once every member is at level 14, and apply decides incarnations and tombstones from the entry alone (seedocs/metadata-raft-design.md). - Rolling a
v0.6.0-preview.7Raft control plane to this release no longer splits it ifstream_incarnationsis finalized through a preview.7 member (which accepts the unknown finalize): a stream create from an older proposer that names no incarnation is stored at 0, and its delete records no tombstone, on every member. Clusters that ran an unreleased build at metadata level 10 to 13 and logged such an entry replay it differently now and must rebuild those members from a snapshot. - A broker that held a deleted stream’s shard now takes the recreated stream’s
assignment even when its generation is lower, as after a control-plane
restore rewinds the change seq (#1209). Shard ownership is ordered by
(incarnation, generation), as the shard lifecycle already was. Deleting a namespace or tenant on the in-memory or Raft control plane now also removes its streams’ and caches’ shard assignments and replica reports, as deleting the stream itself and Postgres’s cascade already did; before, a stream created again under the name was assigned as an update of the deleted one’s shard. On Raft, deletes take shard assignments and replica reports this way only once every member is at metadata version 10, the same gate as stream incarnations; below it every member keeps what the older release keeps, so members never disagree. - Under Raft, the leader refuses to add a control-plane member whose build is
below the metadata version the group already uses, or that reports none;
the error names both versions. That version is the one a finalized fleet
feature needed (10 for
stream_incarnations), or the newest one any committed entry needed, which the state machine now tracks and carries in its snapshot (#1209). A member already in the group that comes back on an older build, as after a lost volume, is no longer sent entries or a snapshot once it reports an older version, or while it has lost its log and not yet reported one; the leader logs it once and setsfelix_meta_raft_member_below_floor{member}to 1. A held member that is behind gets no heartbeats either, so it may call elections the others refuse, and after a long hold it can cost one extra election once released. A member that is slow or unable to answer is not held. Either member would refuse entries the others apply and diverge from them. A held voter is one fewer failure the group survives until it is upgraded. A group that ran unreleased builds of main may have members whose tracked level differs, since those builds applied some entries without raising it; while such a member leads, it can admit or feed a member the others would hold. - Truncation, reset and restore are refused on a log that has failed (#1222). A flush that failed to sync a retired segment marked the roll failed while its seal was still running, and a truncation could then cut that segment and report success before the seal padded it back with zeros.
- A truncation or reset no longer races a background segment roll (#1222). The roll’s seal could put back the length a truncation had cut, or cut off records appended after it. Truncation, reset and restore now wait for the seal.
- Placement takes a follower’s word against a leader only when that leader’s
own replica report says it serves the shard by round (
by_round): it acknowledges by its followers and confirms its reads by a round (#1247). Before, a leader still reading on its lease could be replaced while that lease was valid and keep serving consumer-group writes and state reads, either because it ran withFELIX_QUORUM_READS=leaseor because it had not yet heard on a heartbeat that the fleet finalizedlease_free_reads. A report without the field, from an older broker, reads as the lease. The flag counts only from the leader’s current node incarnation, so a leader’s report from before a restart no longer counts once it registers again. A placement pass that read the old incarnation can still act on that report, so in a fleet that reads by round a restarted leader refuses consumer-group writes, which it would acknowledge on its lease alone, until a replication pass over the shard made after it registered. A pass that ran before registration answered saw no fleet features and does not count. Under Postgres it counts only with the report that set it, so a control plane on the previous release cannot leave a stale one behind. Postgres migration0029_replica_report_by_round; under Raft, metadata level 11. - A broker started with
FELIX_QUORUM_READS=leaseno longer reportslease_free_reads. The finalize dry run names it, and the finalize is refused while it serves; once the feature is finalized such a broker is refused at registration, like one withFELIX_INTERNAL_FENCE=false. - Placement fails a
Quorumstream over on its followers’ word only oncegeneration_start,majority_ackandlease_free_readsare all finalized, which is what brokers need to acknowledge and read without the lease (#1247). Before, a fleet with the last two but notgeneration_startcould have a live leader replaced while it still served reads on its lease. The control plane now refuses with 409 to finalizemajority_ackbeforegeneration_start, andlease_free_readsorfenced_cachesbefore both; the finalize dry run lists what is missing asrequires. A fleet that already finalized them out of order fails over on the lease untilgeneration_startis finalized. - A failed replica commit-offset write is retried once a second until it lands, rather than only when the leader’s next batch arrives, which never happens on an idle shard. A closed log now refuses to write its replica state (#1248).
- A replica commit-offset write that failed was not retried until the offset rose. The next batch carrying the same offset now writes it again. The follower still acknowledges the batch when the write fails, since no acknowledged record depends on the offset (#1248).
felix_broker_promotions_opened_totalnow counts a promotion when the shard is let go to serve rather than when its fence finishes, so a refused open that is retried is counted once (#1250).group_describeand the dead-letter list now check that the broker still leads the shard, as state reads do (#1252). A deposed leader could answer them from stale state. They are refused withshard_unavailableon a lapsed lease, orleadership_lostwhen a round finds a newer leader.- A broker promoted before its registration answered no longer opens a shard
on the lease (#1261). Until then its fleet view is empty, so on a fleet that
finalized
generation_startandmajority_ack, an unreachable old leader read as a peer without the fence and the shard opened without fencing a majority, which could lose records the old leader acknowledged. The shard now stays closed until the broker has its fleet view, then fences as usual. - A Raft control plane restarted with one member down no longer drops storage identities from registrations and reports until that member answers. A member that has not answered counts at the level the log already depends on, not 0. Before, a wiped follower’s registration then cleared its identity and an older report could promote it empty.
- A follower without a storage identity now counts only from a report taken at its current incarnation. The control plane stamps each report with the leader that made it and every named node’s registered incarnation.
- A leader that dies partway through a move no longer leaves its last report to expire because the move’s steps raised the generation (#1250). The report stays believed while its leader is still the assignment’s leader.
- A
Leadershard is no longer promoted on a previous leader’s report once another node leads: what the new leader acknowledged may be on it alone. A new leader that dies before its first report leaves the shard waiting for it; before, it was promoted on the old report for a while and then stalled.Quorumshards still promote on it. Postgres migration0031_replica_report_stamp; under Raft, metadata level 13. - Placement no longer promotes a follower whose storage changed since its leader’s last report (#1266). The control plane keeps each broker’s registered storage identity and the identity each report names for every follower, and counts a follower only while the two match. A follower that lost its disk comes back under a new identity and is not promoted, even on a fresh report. One that only restarted keeps its identity and still counts. A follower with no identity on either side (an older broker or control plane) is judged by the report’s age alone, as before.
- A dead leader’s last replica report no longer expires while placement or the
control plane is slow, which left the shard unplaceable until an operator
abandoned it (#1250). While the assignment is still at the report’s
generation and the leader is down, the report’s age is measured to the
leader’s last heartbeat, for followers whose identity matches. A live
leader’s report still expires. Registration answers 400 for a malformed
storage_id. Postgres migration0030_storage_identity; under Raft, metadata level 12. - A
Quorumleader counts a follower’s confirmed position toward the mark only after the follower has answered in the same pass, over a connection carrying the storage identity the position was learned under. A follower level with the tail is sent its last record again while the mark lags its position. A follower wiped after confirming more than the mark, beside another that failed fast, could release records only the leader held. The counter mark follows the same rule (#1266). - A leader that crashed under
periodicornoneand gave its shard up is replaced by a follower that holds at least what it kept, and named leader again only when every follower holds less (#1314). Its probe now reports a follower level with it as caught up, so the outcome no longer depends on whether the control plane sees the leader’s new storage id or its probe first. A follower counts from the probe only when measured under the identity and incarnation it is registered at now.
Performance
Section titled “Performance”- A subscription’s feeder gives a batch back to
FELIX_SUB_QUEUE_MAX_BYTESonce its frame is in the writer lanes, not before encoding it. The earlier release sat on the delivery path and cost single-publisher streams 1-2% throughput against 0.6.0-preview.7.
Documentation
Section titled “Documentation”- Without
FELIX_SUB_QUEUE_MAX_BYTES_TOTAL, each subscriber can hold up toFELIX_SUB_QUEUE_MAX_BYTESor its largest batch, whichever is larger, and with the per-subscriber bound at0only queue capacity times batch size limits it; the docs now say so and recommend setting the total in production. They also say that underblockone subscriber that stops reading holds a shard’s publishers with no timeout, and that the replay ring and cache watch queues are bounded by entry count only (#1201). - The replication design no longer says a lagging non-successor follower is skipped while a shard drains: it is not gated on, but the drained pass still ships to it and waits.
- Design for replicas electing a leader while the control plane is down
(#1009), not built yet: “Electing without the control plane” in
docs/replication-design.md, with pre-vote, stickiness, a joint fence on every set change, kept with the set and acknowledged on jointly until the control plane flips to the new set, which it does only after the joint fence; cut-overs that wait for the drain; and the failover bound. The TLA+ model gainsSetCheck,JointFence,Adopt,AdoptMatch,SwapsandPreVote, the persistedsetGen,RefuseStaleSet,CpAdopt, theSwapcut-over and theAdoptedMatchesOpenedandSetChangesJointinvariants, andAckedHeldByLeaderis checked per generation where replicas elect. New configurations:FelixShardElectSwap.cfgand its no-joint twin,FelixShardElectAdopt.cfg(two writes nightly),FelixShardElectAdoptPromote.cfg,FelixShardElectPreVote.cfg,FelixShardElectStepDown.cfg,FelixShardElectNoBallotAcks.cfg, and theFelixShardWalkElect.cfgandFelixShardWalkElectSwap.cfgwalks with their twins.FelixShardElectHandoff.cfgnow runs with the set check on.
