Felix 0.7.0-preview.2
Released 2026-11-01 as a preview. The full history is in CHANGELOG.md.
0.7.0-preview.2 adds the conditional head trim: a client can now remove the
oldest records of a stream, but only at the tail it expects, so a stale caller
is refused instead of cutting data it has not seen. It works through felixctl stream trim and the Rust, C, C#, Python and TypeScript clients (trim_before),
survives a leader change, and needs a broker and fleet that have turned on the
head_trim feature. The Rust client’s Client is now the cluster-aware client
(a breaking rename for Rust callers; the migration steps are below), a client
given several broker addresses tries the next one after 250 ms instead of
waiting out a silent one, and a failed publish_if or commit now says whether
it may have written. A just-promoted Quorum leader refuses reads and Latest
subscribes with a retryable not_ready until a majority holds its first record
(#1718). The replica-set change work for #1009 lands here switched off: its
control-plane, replica and fencing pieces ship dormant until a fleet can
finalize them.
- The control plane adopts a leader that a shard’s replicas elected under
replica_election(#1009). A broker’s replica report can name its election (elected); on the set the control plane records the broker becomes the leader at the elected generation, on any other set only the generation is taken, and every later generation is above it. Promotion on the followers’ word waitsFELIX_SHARD_ELECTION_HOLD_MS(10 s) under the feature. Nothing changes untilreplica_electionis finalized, which no broker supports yet. Under Raft adoption takes metadata level 21. - Joint fences, dormant behind the
joint_fencefleet feature, which no fleet can finalize yet: a leader whose replica set changed fences a majority of the set it leaves as well as of the new one before it serves, never counting a learner, and the generation right after one a broker serves is fenced too (#1009). - While a replica set change is pending, a leader acknowledges a record only when a majority of the new set and a majority of the set being left hold it, and reports its generation-start record as started so the control plane can take the new set (#1009). Dormant behind the
joint_fencefleet feature, which no fleet can finalize yet. - The control plane keeps a cut-over’s new replica set pending under
replica_election(#1009). The old set stays the control plane’s until the leader reports its generation’s start record acknowledged on a majority of both, and a failover, a cancel, a consistency change orFELIX_SHARD_PENDING_SET_TIMEOUT_MS(ten minutes) drops it. Nothing changes untilreplica_electionis finalized, which no broker supports yet. Postgres runs migration0038; under Raft it takes metadata level 21. - Replicas keep the replica set they answer at, refuse a fence or batch on an older set (
StaleSet), refuse a candidate while they still hear their leader, and a leader that answers a newer fence stops counting. Dormant: only between brokers run with the test switchFELIX_INTERNAL_ELECTION(#1009). Internal frames that wrap a batch no longer copy its body. - The C ABI, C#, Python and TypeScript clients send the conditional head
trim, step 8 of #1662.
felix_client_trim_beforefills aFelixTrimOutcome(a stale tail is an answer withtrimmedfalse and the tail);FelixClient.TrimBeforeAsyncreturns aConditionalTrim; Python’strim_before(sync and async) returns aConditionalTrimand TypeScript’strimBeforeresolves to one. A broker withoutFEATURE_HEAD_TRIMis refused locally:FELIX_STATUS_UNSUPPORTED,FelixStatus.Unsupported, Python’sTrimErrorand TypeScript’sTrimError. A trim that timed out is outcome unknown in every binding.client-fixturegains atrimmer_tokenand atrim_stream, finalizeshead_trim, and fourhead_trim.*conformance scenarios cover the bindings. The bindings’ cursor errors still carry only the floor, not the trim’s headers. A publish refusal other than a stale tail, answered to a trim, is now aBrokerError(invalid_request) instead of aPublishRefused. - The broker writes a conditional head trim in process, the third piece of
#1662.
Broker::trim_before_atappends the trim record only at the tail the caller expects, refuses a floor past the record, a stale no-op and a stream holding atomic commits, and drops the replay ring’s entries below the floor in the record’s turn. Every reader now starts at the readable base, including group cursors and the Kafka listener’s fetch and earliest offset, andCursorTooOldcarries the setting trim’s headers.StreamLog::base_offsetis nowphysical_base, besidereadable_base. A trim is answered only once the ring has it, and a no-op on aQuorumstream waits for the committed mark, and a trim’s waits end inTrimTimedOut. Nothing on the wire writes a trim yet. - The Rust client and
felixctlsend the conditional head trim, step 7 of #1662.Client::trim_beforeandConnection::trim_beforetake a shard, the new floor, the expected next offset and optional headers. A stale offset comes back asConditionalTrim::Refused { tail }and is not retried; a timed-out trim is aBrokerErrorwithOutcomeUnknown; a broker withoutFEATURE_HEAD_TRIMis refused locally withTrimError::Unsupported.SubscribeCursorErrorgainstrim_headers, so it is no longerCopy.felixctl stream trimprints where the trim landed and where reads start. A cluster test shows a trim and its floor surviving the leader. - Storage applies a trim record’s floor, the second piece of the conditional
head trim (#1662). Once a trim is durable, and committed on a
Quorumlog,read_rangebelow its floor returnsTrimmedwith the trim’s headers,DiskLog::readable_basereports where reads start, and the sealed segments below the floor are deleted in the background. The floor is derived from the trim records and rebuilt at open, with a newheadfile as a cache.read_rangenow returns aReadPagewhosenextsays where to carry on, since a page can be empty below the tail when only trims remain. Nothing above storage writes a trim yet. Seedocs/durable-storage.md, “Conditional head trim”. - Storage has a trim record, the first piece of the conditional head trim
(#1662). It is a new record kind that holds a floor and the caller’s
headers, and it needs segment format v7. A normal read leaves trim records
out, and
read_range_with_trimsreturns them. Nothing writes one yet, so a log stays on v6 or older and its bytes are unchanged. Seedocs/storage-format.md, “Trim records”. - Replication ships head trims, the fourth piece of #1662. A trim record
travels with mark byte
5and its floor, and a follower of aQuorumstream applies the floor only once the leader’s commit offset passes the record. A promoted leader raises its log’s commit offset before it publishes the quorum mark, so a trim it inherited applies before its first read. A new fleet feature,head_trim, gates trims in a cluster, and a new peer capability,HEAD_TRIM, keeps a batch holding a trim from any follower that did not offer it; shipping counts that as thetrim_unsupportedoutcome offelix_broker_replication_shipped_total, and logs it at most once a minute. Trims now fail closed:Broker::allow_head_trims_whenreplacesset_replication_enabled, and a broker refuses every trim withHeadTrimNotEnabled(formerlyTrimNotReplicatedYet) until startup says it runs alone or its fleet has finalizedhead_trim. A new cluster finalizeshead_trimat its first registration.mark_to_wireno longer fails, andUnshippableMarkis gone. - Clients can send a conditional head trim, the fifth piece of #1662.
trim_beforenames a shard, the floor and the expected tail, and is answered withtrim_ok(where the record landed and the new floor) orpublish_refusedwithoffset_mismatch. It is negotiated asFEATURE_HEAD_TRIM, the fifth bit of the extended feature word. Two new error codes,feature_not_enabled(number 18) andstream_has_commits(19), and the reasonstrim_past_tail,not_durableandtrim_timed_outname each refusal.subscribe_cursor_errorcarriestrim_headersbelow a trim’s floor, only to a client that offered the bit. A new action,stream.trim, allows it, andstream.managegrants it too; the control plane mintsstream.triminto broker tokens only once the fleet has finalizedhead_trim. The first trim writes ahead-trimmedmarker in the data directory, and a broker holding it refuses to join a fleet that has not finalizedhead_trim.
Changed
Section titled “Changed”-
A client given several broker addresses no longer waits out a silent one before trying the next (#1232).
Connection::connect_anyand aClient’s connect and reconnect start each next address 250 ms after the one before it, or at once when that one fails. The first to connect wins and the rest are dropped. The Python, TypeScript and C bindings inherit this. -
Breaking: in the Rust
felix-clientcrate,Clientis now the cluster-aware client (#1470). Applications should always use it: it sends each request to the broker that owns the shard and reconnects when a broker fails. A plain connection to a cluster forwards publishes for shards its broker does not lead, which cut in-memory throughput 2.4x and durable quorum-acked throughput 4.4x in the v0.6.0 three-node run. The old single-connection type is nowConnection, kept public for tools that address one named broker. The Python, TypeScript, C and C# clients already exposed the cluster client under the nameClient, so only Rust callers change.Migrating:
- Rename in this order, or the second rename undoes the first: first
ClienttoConnection, thenClusterClienttoClient. - After both,
use felix_client::{Client, Connection}brings in the cluster-aware client (wasClusterClient) and the single-connection type (wasClient).Client::connect(&seeds, server_name, config)andClient::connect_seedskeep their arguments. - Code that held one broker address should move to
Clientwith that address as the only seed; a single broker needs nothing else. UseConnection::connect(addr, server_name, config)only for tools that address one named broker. ClusterSubscription,ClusterCacheWatchand the otherCluster*types keep their names.
- Rename in this order, or the second rename undoes the first: first
-
A failed
publish_iforcommitnow says whether it may have written. The Rust client wraps any failure after the send, other than a coded refusal, inAckLost, andfelix_client::outcome_unknownis true for that and for a broker error classedoutcome_unknown, false for a refusal, validation error orforbidden. A multi-recordpublish_ifon aQuorumstream is documented as not all or nothing: after an unknown outcome any prefix may exist, so re-read from the expected offset. Usecommitfor a batch that must be all or nothing. The Python, TypeScript and C bindings already mapAckLostto their outcome-unknown error, so they inherit this. (#1630) -
An RF 2 shard kept closed because the failover’s fence ids name a copy beside the set of two now says so (#1635). The crashed leader’s copy logs a warning naming the extra id (a staged or joining successor the failover left out), counts
felix_broker_fence_refused_ids_beyond_two_total, and its refusal reaches the promoted leader’sfelixctl inspect shardreason. A fence refusal’s detail now shows there in general. Replication itself is unchanged. -
A crashed leader takes a fence under the set-of-two exception only when the fence is at the failover’s generation and comes from that set’s leader (#1638).
-
Built with Rust 1.99.1; the minimum supported Rust version stays 1.97.
- A
Quorumleader that was just promoted no longer answers astream_readwith an empty page, or starts aLatestsubscribe at offset 0, before a majority holds a record of its generation. It refuses them asnot_ready, which is retryable (#1718).
Documentation
Section titled “Documentation”- The shard model now covers a staged destination that votes beside an RF 2
failover (#1635). A leader restarted cleanly counts the staged copy toward a
Quorumwrite, so the promoted leader’s refusal of the crashed leader’s fence is what keeps that write. The model checks that today’s fence ids keep it and that leaving the staged copy out of them loses it. - docs.getfelix.dev now serves the newest stable release’s docs at the root and
the upcoming release’s docs (from
main) under/next/.
