Felix 0.6.0-preview
Released 2026-10-02 as a preview. The full history is in CHANGELOG.md.
A preview of 0.6.0. Shards move between live brokers without refusing
writes, Quorum streams can acknowledge by their followers and serve reads
without the lease once an operator finalizes the matching fleet features,
storage recovers from power loss and failed fsyncs without losing
acknowledged records, and Kafka clients can produce and consume. pip will not install it without --pre.
A note on the version string. Cargo and npm carry 0.6.0-preview
verbatim. PEP 440 normalises it to 0.6.0rc0 (preview is one of its
spellings of rc), so the wheel’s version differs from the crate’s and the
npm package’s on purpose. It sorts before 0.6.0 on all three.
Upgrade notes. These need action, and each is marked Breaking below:
- Raft control planes need
FELIX_RAFT_CLUSTER_ID,FELIX_RAFT_BIND_ADDRandFELIX_RAFT_PEER_TOKEN(orFELIX_RAFT_INSECURE_PEERS=true);FELIX_RAFT_PEERSnow lists peer-listener addresses, and a new group needsFELIX_RAFT_INITIAL_CLUSTER_STATE=new. The Helm chart needs a peer-token Secret. - Clustered brokers need peer mTLS (
FELIX_INTERNAL_TLS_*) orFELIX_INTERNAL_ALLOW_UNAUTHENTICATED=true; the chart needsbroker.peerTls.enabledorbroker.peerTls.allowUnauthenticated. - A broker outside Helm with
FELIX_QUIC_LISTENERSunset binds up to four client listeners on consecutive ports. A container that publishes only port 5000 needsFELIX_QUIC_LISTENERS=1. - A replication-factor-1 durable shard now waits for its owner instead of failing over empty.
- Publish scheduling settings (
FELIX_BROKER_PUB_*) keep their names but change meaning. - Control-plane listings return at most 1000 entries per page.
stream.subscribealone no longer allows redriving or discarding dead letters.- New segments are written in storage format v3 and later, which 0.5.0 cannot read.
- Prefix cache watches on multi-shard caches must name a shard.
- Several metrics, Rust APIs and crate paths changed; see Changed.
Security
Section titled “Security”- A refresh cannot widen its exchange. The exchange’s
requested,resourcesandaudienceare stored with the refresh chain and re-applied on every refresh; a refresh naming another audience is a400. Postgres gains a nullablerefresh_tokens.narrowingcolumn; older tokens refresh as before. On Raft, roll every member before relying on it. - Changing an IdP issuer’s trust takes cluster rights. Changing an
existing issuer’s
jwks_url/discovery_url, audiences or claim mapping, registering an issuer another tenant already uses, or deleting an issuer needstenant.manage:cluster:*. A tenant admin can still add an issuer no other tenant uses. (#755, #852) IdP fetches no longer follow redirects and refuse private, link-local and unique-local addresses unlessFELIX_CONTROLPLANE_OIDC_ALLOW_PRIVATE_IDPis set. - Control-plane authorization can shrink.
DELETE .../rbac/policiesand.../rbac/groupingsremove rules (same scope as adding them);POST /v1/tenants/{t}/refresh-tokens/revokeends a principal’s refresh chains;/v1/tenants/{t}/signing-keysstages, activates and retires tenant signing keys. On Raft they are refused (409) until every member runs this release. - IdP groups are scoped by issuer (
group:{issuer}#{name}), so an IdP a tenant admin registers cannot claim another IdP’s groups. Baregroup:{name}groupings need migrating; seedocs/auth.md(FELIX_CONTROLPLANE_LEGACY_UNSCOPED_GROUPSbridges the migration). - The control-plane API refuses broker-audience tokens. Exchange and
refresh take
"audience": "felix-controlplane"for API callers, including broker node credentials;FELIX_CONTROLPLANE_ACCEPT_BROKER_AUDIENCEbridges the migration. - IdP discovery and JWKS URLs must be HTTPS (plain HTTP on loopback, or
with
FELIX_CONTROLPLANE_OIDC_ALLOW_INSECURE_HTTP), and a discovery document must name the configured issuer. - Token exchange
resourcesnow narrow grants instead of passing a broader grant through unchanged. - Tenant, namespace, stream and cache names are validated at creation;
*,/,:and similar are refused. - Postgres
delete_nodeand assignment writes lock the node row, so a racing delete can no longer leave an assignment to a deleted node; a node that is still a replica can no longer be deleted. - Live tokens no longer appear in
Debugoutput, and migration exports are written mode0600. - Breaking: Raft peers are served on their own authenticated listener.
Raft RPCs used to sit on the public API port with no authentication, where
one
proposecould replace the metadata store. They are now served only onFELIX_RAFT_BIND_ADDR, and every request carries the cluster id andFELIX_RAFT_PEER_TOKEN(at least 32 characters;FELIX_RAFT_INSECURE_PEERS=trueskips it).FELIX_RAFT_PEERSnow lists peer-listener addresses. Optional peer mTLS:FELIX_RAFT_TLS_CERT,FELIX_RAFT_TLS_KEY,FELIX_RAFT_TLS_CA, all three or none.felix-controlplane migrate importauthenticates as a peer and must target a peer address. Helm gainscontrolplane.storage.raft.peerPort(8444),clusterId,peerToken.existingSecret/tokenKeyandinsecurePeers, and refuses to render without a token Secret. (#727) - Breaking: clustered brokers require peer mTLS. A broker with
FELIX_NODE_IDset refuses to start withoutFELIX_INTERNAL_TLS_CERT,FELIX_INTERNAL_TLS_KEYandFELIX_INTERNAL_TLS_CA, unlessFELIX_INTERNAL_ALLOW_UNAUTHENTICATED=true. This holds at replication factor 1 too, since writes are forwarded over that port. The Helm chart refuses to render unlessbroker.peerTls.enabledorbroker.peerTls.allowUnauthenticatedis set. (#739) - Unauthenticated QUIC connections are bounded. A peer could make the
broker reserve 16 MiB per stream from a 12-byte header, on up to 2,048
streams, with no deadline. Before a connection authenticates, frames are
capped at
FELIX_PREAUTH_MAX_FRAME_BYTES(default 64 KiB) and at mostFELIX_PREAUTH_MAX_STREAMS_PER_CONN(default 16) streams are read. A connection with no authenticated stream afterFELIX_AUTH_TIMEOUT_MS(default 10 s,0off) is closed.FELIX_MAX_CLIENT_CONNECTIONS(default 8192) caps connections per broker, and a stalled handshake no longer blocks the accept loop. New metricsfelix_quic_connections_refused_total,felix_quic_handshake_failures_totalandfelix_quic_auth_timeout_total. A client older than this release that leaves an event connection idle for over 10 s before subscribing loses it; raise or disableFELIX_AUTH_TIMEOUT_MSfor those. (#736) - The broker’s pre-auth JWKS fetch cannot be used to flood the control plane. Implausible tenant ids are refused without a request, unknown tenants are refused locally once the first catalog sync lands, concurrent misses share one fetch, failures are cached for 5 s, and at most 8 fetches run at once with a 5 s timeout. (#736)
- Tokens can be bound to the client certificate. With
FELIX_TLS_CLIENT_CERT_BIND_SUBJECT=true(which needsFELIX_TLS_CLIENT_CA), a client that presents a certificate must use a token whosesubthe certificate names: a URI SANfelix:principal:<sub>, or a DNS or IP SAN equal tosub. Control-plane principal ids match only through the URI SAN. The check covers QUICauthand Kafka SASL/PLAIN over TLS. (#751, #758, #771) - The vendored Kafka decoder no longer trusts counts from the wire. A 66-byte produce batch could make it reserve over 3 GB, and a 1 MiB Metadata frame about 72 MiB, before decoding anything. Reservations are now capped by the bytes left in the frame. A record whose offset or timestamp delta overflows the batch base is refused instead of panicking or wrapping. (#762, #768, #821)
- Breaking: consumer groups have their own authorization actions.
group.consumecovers poll, ack, nack and listing dead letters;group.managecovers redrive and discard.stream.subscribestill grantsgroup.consumeandstream.managestill grantsgroup.manage, but a principal with onlystream.subscribecan no longer redrive or discard. Both can be granted on one group,group:{tenant}/{namespace}/{stream}/{group}; where a principal holds a group grant for an action on a stream, only its group grants count there. Upgrade brokers before writing policies that use the new actions, since an older broker refuses tokens that carry them. (#731, #744)
-
felixctl, a command-line tool (#872).pub(from an argument, a file or stdin; keyed, unacknowledged or idempotent),sub(fromlatest,earliestor an offset, one shard or all),cache get|put|del|watch,topology(shard count, owners and brokers), read-onlytenant,namespace,stream,cache ls|info,nodeandshardlistings that follownext_cursor, andbench ingest|latency|fanout|cache. Named contexts infelixctl/config.toml, overridden by newFELIX_*variables (FELIX_CLI_CONFIG,FELIX_CONTEXT,FELIX_BROKERS,FELIX_NAMESPACE,FELIX_AUTH_TOKEN_FILE,FELIX_CONTROLPLANE_TOKEN,FELIX_CONTROLPLANE_TOKEN_FILE,FELIX_CA_FILE,FELIX_CLIENT_CERT_FILE,FELIX_CLIENT_KEY_FILE,FELIX_SERVER_NAME), overridden by flags.--jsoneverywhere, distinct exit statuses, shell completions and man pages. -
Releases ship felixctl. A tag attaches
felixctl-<tag>-<target>archives for Linux x86_64 and aarch64, macOS aarch64 and x86_64, and Windows x86_64, each with a.sha256, completions and man pages; pushesghcr.io/<owner>/felixctlfor amd64 and arm64; and publishesfelix-loadgenandfelixctlto crates.io after the client. The crates.io job packages every crate before uploading any, and uses trusted publishing when noCARGO_REGISTRY_TOKENis set.release.ymltakesdry_run(build everything, publish nothing) andref(build a branch as if it weretag, dry runs only), so a release can be rehearsed before its tag exists. -
felix-loadgenis also a library.felix_loadgen::runruns a scenario and returns itsLOADGEN_JSONobject;felixctl benchuses it. The binary’s flags and output are unchanged. The crate is now publishable. -
felix-cluster upcreates a cache namedusers, and its session file names each broker’s certificate (cert_file), so a client can verify the brokers it connects to. -
felix-loadgeningest:--in-flight <n>,--duration-secs,--start-at.--in-flightsends acked batches withnoutstanding per publisher and reports their ack latency, so a run measures what a client that waits for durability gets and uses the publish window. The other two size a run by time and start every generator at the same instant. The Azure drivers use all three, and gained ashapesstep (payload x batch x acked) and, in session C, lease and lease-free passes. -
felix-loadgen pubsubpublishes to the shard’s owner. Single publishes went through the first broker, so latency depended on which broker led the shard and a relayed publish paid an extra round trip to the owner.--via-entrykeeps the old path. -
Publish acknowledgements say where the batch landed. A client that offers the new frame flag
FLAG_BINARY_PUBLISH_ACK_OFFSET(0x1000) gets the offset of the batch’s first record on a successful binary ack, and asoffseton a JSONpublish_ok. The broker sends it only when it answers after the write; a client that did not offer the bit gets byte-identical frames. A duplicate idempotent batch reports the original’s offset, and a forwarded batch reports the owner’s, through a new peer capabilityFORWARD_OFFSETS.Publisher::publish,publish_batchand friends,ClusterClient::publishandIdempotentProducer::publishnow returnResult<Option<u64>>instead ofResult<()>; the Python and Node clients return the offset too. The history checker now checks every acknowledged append against the offset it was acknowledged at (#876). -
Placement sees halted replicas (#874). A leader’s replica report names the followers it has stopped shipping to, with the reason (
divergedorneeds_bootstrap), as an optionalhaltedfield that is omitted when empty. The control plane stores them with the report, keeping when each halt began: Postgres gainsreplica_reports.halted(migration 0023), and on Raft the field is metadata level 4, dropped until every member is upgraded. Placement never picks a halted copy as a move, replacement or restore destination, gives up a move whose destination halts (stephalted) and places it elsewhere, counts a halted copy as missing, and replaces one halted forFELIX_SHARD_RESTORE_AFTER_MS. An operator’s move to one is refused withdestination_halted, and a drain with nowhere else to go says which copy is halted.GET /v1/placement/replicationandfelix-controlplane admin replicationlist halted copies, andfelix_shard_replicas_haltedcounts them. -
Placement restores a shard’s replication factor. A follower whose broker has been down or gone for
FELIX_SHARD_RESTORE_AFTER_MS(shard_restore_after_ms, five minutes by default,0to turn it off) is replaced by a copy on a live broker. A replica set a failover left short of its factor is topped up once a broker is free, which covers a cache orLeaderstream whose failover dropped the dead leader. The copy joins and is seated the way a drain’s replacement is, so aQuorumstream keeps the old set’s majority rule. The restore is stored in the assignment (move reasonrestore) and resumes across control-plane restarts, and a destination that dies mid-copy is replaced. A short set grows before a lost follower in it is replaced, nothing starts until the leader has reported at its generation, and a failover keeps a copy that was joining a set of two, since the leader counted it. New:felix_shards_under_replicated,felix_shard_replicas_missing,GET /v1/placement/replicationandfelix-controlplane admin replication. The move stepSeatnow names the follower it replaces as optional, andMoveStep::RestoreandMoveReason::Restoreare new. A control plane that predatesrestorecannot read an assignment with that move reason, so upgrade every control-plane instance before relying on it. -
Adversarial history campaign and liveness checks. The history checker’s nemesis gains compound faults on four brokers: two brokers killed at once, a leader isolated from its peers and the control plane until it fails over, a partition beside a delayed link, two random faults together, a move whose source or destination is killed, restart loops, torn segment writes, and drains that replace follower copies. After every heal the campaign now waits for each shard to serve again and fails naming any shard that is stuck and why.
FELIX_HISTORY_NEMESISpicks the nemesis, and the nightly workflow runs both. -
Power loss, control-plane crashes and delivery checks in the history campaign. The adversarial nemesis can cut power to every broker at once, keeping only what each one flushed, and crash the control plane while a move, drain or failover is in flight. Live subscribers now follow each list for the whole run, and three new rules check that delivery follows offset order, that nothing delivered is later lost, and that every record is either delivered or skipped visibly. Debug and
fault-injectionbrokers take the power loss fromFELIX_STORAGE_POWER_LOSS_ROOTand the storage fault file, on Linux. The harness keeps broker storage instorage/under each node’s data directory. -
Atomic commits on one shard.
Client::commit(tenant, namespace, entity_key, [publish | enqueue, put, delete])writes an event and state updates as one record on the stream shardentity_keyroutes to, andstate_getreads that state with the commit’s offset as its version. No reader sees part of a commit, across failover and promotion. Operations on another stream are refused withCommitError::NotOnOwningShard, never split. New wire requestscommit/state_getbehindFEATURE_ATOMIC_COMMIT; storage format v5 (a segment is written at v5 only to hold a commit record); replication mark byte4; fleet featureatomic_commit, which a cluster must finalize before commits are accepted. Seedocs/atomic-commit.md. -
Jump-hash stream routing once the
jump_hash_routingfleet feature is finalized. A stream’s key-to-shard mapping (routing:moduloorjump_hash) is fixed at creation; streams created after the finalize get jump consistent hashing, and existing streams keep modulo, so no key moves. Postgres gains a nullablestreams.routingcolumn (migration 0022). Shard assignments andstream_shards_viewcarryroutingonly for jump-hash streams, the broker resolves keyed publishes by it, andClusterClientroutes by it.Client::stream_routingis new;felix_router::Placedgains aroutingfield. -
Readers and consumer groups on
Quorumshards without the lease oncelease_free_readsis finalized (#885). Subscriptions, replay, Kafka fetches and cache watches on a replicatedQuorumshard keep going when the lease lapses, since they see only the committed mark. They end when a follower refuses the leader for a newer one, or when no majority has confirmed it for a lease duration (a round runs every quarter lease while the lease is lapsed). Group polls, acks, nacks and dead-letter changes skip the lease and are confirmed by a read-index round after they are durable and before they are acknowledged; a refused round answersleadership_lost. Kafka produce now takes its lease check from the shard fence, like a native publish, so amajority_ackshard takes Kafka writes without the lease too.BrokerCluster::with_writesloses its lease argument. TLA+:FelixShardSessions.tla, withSubscriberandGroupRoundpassingNoLostDeliveryandNoStaleGroupCommit, andPastMark,GroupNoRoundandGroupLeaseviolating them. Seedocs/replication-design.md, “Readers and group sessions without the lease”. -
Quorumcache reads without the lease once thelease_free_readsfleet feature is finalized (withmajority_ackandgeneration_start). A get or counter get on a replicatedQuorumcache, local or forwarded, takes its value and then confirms leadership read-index style: the promotion fence at the broker’s own generation goes to every replica, and the read is answered once a majority, the broker included, has taken it. Concurrent reads share rounds, only ever one that started after them. A cache replica also refuses the round when its counter log accepted a newer leader. A same-generation fence is now counted asfence_confirmedand not logged. NewFELIX_QUORUM_READS(majorityorlease) keeps a broker on the lease. No wire change. TLA+:FelixShardReads.tla, withFelixShardReadsRoundpassingNoStaleReadunder drift and no margin andFelixShardReadsNoRound/FelixShardReadsLeaseviolating it;check_tla.shtakesTLC_WORKERS. Seedocs/replication-design.md, “Reads without the lease”. -
Quorumstreams acknowledge by their followers once themajority_ackfleet feature is finalized. A write is acknowledged when a majority of the replica set, the leader included, has answeredReplicateOkfor it at the leader’s generation; the control-plane report goes out behind the mark for placement only, and the lease is off the write’s path (admission, commit and acknowledgement). A leader counts itself only while it has accepted no newer generation, and a promoted leader never opens a stream shard on the lease. Needsgeneration_startfinalized too. A broker running withFELIX_INTERNAL_FENCE=falsedoes not report the feature.Leaderstreams and cache writes keep the lease;lease_free_readstakes it off reads and group operations. TLA+:HeldAtGencounts follower answers (confirmed), andFelixShardFigure8FollowerAcks/FelixShardFigure8FollowerAcksNoStartRecordjoincheck_tla.sh;FelixShardFencedAcknow includes the start record. Seedocs/replication-design.md, “Acknowledging by the followers”. -
scripts/check_tla.shtakes configuration names to check only those. -
Fleet features: cross-broker behaviour turns on only when an operator finalizes it. A broker reports the features it implements at registration (
features, optional), and the control plane tracks the set every live or draining broker supports. Support enables nothing:POST /v1/fleet/features/{feature}/finalize(orfelix-controlplane admin features finalize <feature> [--dry-run]) enables one, refused until every serving broker supports it. Registration and every heartbeat answer with the enabled set asfleet_features;GET /v1/fleet/featuresandadmin featuresshow supported and enabled. Brokers check it throughfelix_common::fleet::FleetGateand exportfelix_broker_fleet_feature_enabled. Before a finalize any broker can be rolled back; after it, finalizing is one-way and a broker without the feature is refused at registration (409).generation_start,majority_ack,lease_free_reads,jump_hash_routingandatomic_commituse it. Postgres gainsnodes.featuresand afleet_featurestable; on Raft the control plane keeps features and accepts a finalize only once every member is at metadata version 2. -
Operator moves report their zone impact.
POST /v1/shard-movesanswerszones_beforeandzones_afterwhen brokers report zones, takes"dry_run": trueto decide a move without starting it, and logs a move that narrows a shard’s spread as a warning instead of refusing it.felix-controlplane admin movegains--dry-runand prints the zones. -
The Helm chart runs brokers per zone.
broker.zonesrenders one broker StatefulSet per zone, pinned to its nodes and settingFELIX_NODE_ZONE, with one disruption budget across all of them. -
Pipelined publishes. A client that offers
FEATURE_PUBLISH_PIPELINEis granted a per-connectionpublish_windowinauth_ok(FELIX_BROKER_PUBLISH_WINDOW, default 256,0to turn off). The broker answers that client’s acked publishes on each stream in request order and stops reading its publishes while the window is full. The Rust client offers it and caps each connection at the window, andIdempotentProducer::publish_batcheskeeps up to 64 batches in flight, re-sending from the first unanswered one after a failure. Older clients see no change. -
Connection-fault conformance scenarios. A catalogue scenario can carry a
stepthat drops, resets or stalls the client’s link mid-publish or mid-subscribe.felix_conformance::linkis the UDP interposer that does it;felix-cluster client-fixtureserves one atlink_addr, breaks it onPOST /link, and lists the steps underfaults. The protocol suite runs them with the Rust client, and the Python and TypeScript suites run them in CI. A subscription must resume with no gap or raise; ending as if the stream had finished fails. -
Placement spreads a shard’s copies across zones. A broker registers the failure domain it is in with
FELIX_NODE_ZONE(the node’s new optionalzone). Followers go to zones the shard has no copy in, drains and rebalances never narrow a shard’s zones when they can avoid it, and a follower sharing a zone is replaced by one in a missing zone once a broker there has room. Where the copies cannot be spread they are placed anyway, with a warning andfelix_shards_zone_unspread. A broker without a zone shares one with nobody, so clusters that report none are placed as before. Postgres gains a nullablenodes.zonecolumn. -
A promoted leader fences a majority before it serves. A broker promoted to lead a stream shard now waits in a
fencingphase: it sendsFenceto every replica, opens once a majority (itself included) has persisted its generation, and first takes the log of the replica furthest ahead by (last generation, length) with the newReplicateFetch(kind 31, capabilityTAIL_FETCH). A deposed leader’s ships are then refused by that majority even where the control plane never reached the follower. Applies only when every replica offers both capabilities; otherwise the shard opens on the lease as before, except that undermajority_acka stream shard never opens on the lease.felix_broker_promotions_opened_total{path}says which. A replica that takesFencepersists the leader’s generation before it answers with its log end, commit offset and last record’s generation, and refuses every older leader from then on. Brokers offer capability bits in the peer handshake (HelloCapable, kinds 27 and 28), and a peer that did not offerFENCEis never sent one;FELIX_INTERNAL_FENCE=falsewithdraws the offer. Untilmajority_ackis finalized, acknowledgements still go by the report and the lease. Each peer connection opens one more stream, for the handshake and the fence’s requests only, so they never wait behind a forwarded publish. (#794, #798) -
Clients can offer the
felix/1ALPN.felix_client::quic_client_configbuilds the QUIC TLS config and offersfelix/1when asked; the Python (offer_alpn=True) and TypeScript (offerAlpn) clients expose the same switch. Off by default, since a broker older than ALPN support refuses a client that offers one. Before this no shipped client could connect to a broker withFELIX_TLS_REQUIRE_ALPN=true. -
The licence split is checked on the dependency graph.
scripts/check_license_graph.py, run bytask publish:check, fails if an Apache-2.0 or MIT first-party crate has a normal or build dependency, direct or transitive, on an AGPL one. -
Advisories reach the vendored
kafka-protocol. A patched-in path crate has no registry source, so cargo-deny skipped it.vendor/VENDORED.tomlrecords the upstream version (0.18.0) andtask denychecks advisories against it, failing if the record and the vendored copy disagree. -
Kafka producers can write to durable streams.
Produce(v3-9) on the Kafka listener decodes v2 record batches, compressed with gzip, snappy, lz4 or zstd, and publishes them through the broker’s publish path on the shard’s leader, withstream.publishchecked as for a QUIC publish; any other broker answersNOT_LEADER_OR_FOLLOWER.acks=allwaits for the stream’s own consistency: a majority on aQuorumstream, the leader on aLeaderone.InitProducerIdgives idempotent producers a Felix producer id, and the newBroker::publish_records_idempotentstores each record as a one-record producer batch so Kafka’s per-record sequences are the log’s own: a batch re-sent after a failover or a move is answered with its original offset. Transactions are refused withTRANSACTIONAL_ID_AUTHORIZATION_FAILEDand a message; legacy v0/v1 message sets withUNSUPPORTED_FOR_MESSAGE_FORMAT. Keys and headers are dropped and counted; producer timestamps are dropped (a record’s timestamp is its append time). New metrics:felix_kafka_produce_records_total,felix_kafka_produce_bytes_total,felix_kafka_produce_duplicate_records_total,felix_kafka_produce_errors_totalandfelix_kafka_produce_dropped_total.FELIX_KAFKA_ANONYMOUS_TENANTnow allows writes as well as reads. The docs-site page “Reading with Kafka Clients” is now “Kafka Compatibility”. -
Kafka consumers can read durable streams. A broker started with
FELIX_KAFKA_LISTENserves the Kafka protocol for consumers that assign their own partitions: kcat, librdkafka programs and JavaKafkaConsumers. A durable stream is topic<namespace>.<stream>, partition N is shard N, and offsets are Felix’s log offsets. It speaksApiVersions, SASL/PLAIN (tenant id as username, Felix token as password, reads checked asstream.subscribe),Metadata,ListOffsetsand a long-pollingFetch, and a consumer follows a partition to its new leader after a move or failover.FELIX_KAFKA_ADVERTISE_ADDR,FELIX_KAFKA_TLS(on by default, with the broker’s certificate),FELIX_KAFKA_ANONYMOUS_TENANT,FELIX_KAFKA_DEFAULT_NAMESPACEandFELIX_KAFKA_MAX_CONNECTIONSconfigure it. Each broker registers its advertised address as the node’s newkafka_addrfield so any broker can name any partition’s leader; migration0018_node_kafka_addr.sqladds the column. Consumer groups are refused:FindCoordinatoranswersGROUP_AUTHORIZATION_FAILEDwith a message saying to assign partitions, so a group consumer fails with a reason instead of hanging. Metrics are underfelix_kafka_*.docs/kafka-compatibility.mdreplaces the spike write-up, and the spike crate underspikes/is removed. -
Streams can be kept in a region. A stream created with
"region"has its leader and every replica only on brokers in that region, or in one the new directional allowlistFELIX_REGION_BRIDGES(source>destpairs) bridges it to: on first placement, rebalancing and drain moves, follower replacement and failover. A copy found outside, after a broker’s region changes or a bridge is removed, is moved back in; with no broker in an allowed region the shard is left unplaced rather than placed elsewhere. An operator move out of the region is refused withregion_not_allowed. Brokers read the sameFELIX_REGION_BRIDGESand forward only to leaders in bridged regions, refusing the rest asshard_unavailablewith the new reasonregion_not_routableinstead ofowner_unavailable. Streams without a region, and every cache, are placed as before. Migration0017_stream_region.sqladds a nullablestreams.regioncolumn. -
Consumer-group calls follow the shard’s leader.
ClusterClientgainsgroup_poll,group_poll_wait,group_ack,group_nack,group_dead_letters,group_discardandgroup_redrive, which follow the broker’sNotLeaderredirect to the shard’s leader and remember it. The Python and TypeScript clients’ group calls now use them, so they no longer fail withShardUnavailableError(not_leader) when connected to a broker that does not lead the shard, or after a shard move.Client’s group calls still returnNotLeaderErrorrather than follow it. -
A broker hands its shards off before it stops. On SIGTERM a clustered broker turns readiness off, drains itself through the control plane and keeps serving until it leads no shard, then shuts down as before, so a rolling restart moves shards instead of failing them over: no publish is refused and subscriptions follow the shard. Bounded by
FELIX_SHUTDOWN_HANDOFF_TIMEOUT_MS(default 30 s,0turns it off); skipped when there is nobody to hand to or the control plane does not answer, and ended early by a second signal. Metrics:felix_broker_shutdown_handoffs_total{outcome},felix_broker_shutdown_handoff_shards_total,felix_broker_shutdown_handoff_duration_ms. The Helm chart addsbroker.shutdown.handoffTimeoutMsand counts it in the derived grace period; a grace period set by hand must now cover it too. -
Cache, counter and consumer-group operations are no longer refused during a shard move. A cache put, delete or read and a counter add or read that arrives between a move’s fence and its cut-over is held and sent to the new owner, as a publish already was, on the broker it reached and on an owner a forward reached. A consumer-group poll, ack, nack or dead-letter change is held and then answered with
NotLeadernaming the new owner, whichClusterClient::group_shardedfollows; group operations are still served only by the shard’s leader. A group poll waiting for records when its shard moves away now answers with no records instead of an error. Bounded by the sameFELIX_SHARD_MOVE_HOLD_MSandFELIX_SHARD_MOVE_HOLD_MAX. -
routablein the node listing.GET /v1/nodesnow says whether other brokers may send a node requests for the shards it leads: live or draining, heartbeat inside the window. Brokers forward on it instead of oneligible, so writes through another broker to a draining broker’s shards are no longer refused before each shard’s turn to move. -
Idempotent producers keep their sequences across a leader change (#608). On a durable stream each record of a producer’s batch is now stored with its producer id and sequence, and replicated with them, so a leader promoted after a failover, a planned move’s destination and a restarted broker answer a re-send of the batch in flight from the records they hold, and the producer carries on instead of being refused as
unknown_producer. A batch whose leader stopped partway through it is finished by the re-send rather than written twice. A producer is remembered while any of its batches is in the log, so retention now decides when one is forgotten, on every replica alike. In-memory streams keep sequences in the leader’s memory as before. Opening a shard reads no more than before: the producer state is saved at each rollover and the rest comes from the active segment recovery already scans. New metricfelix_storage_producer_state_rebuilt_totalcounts opens that had to read sealed segments to rebuild it. -
Subscriptions follow a moved shard. A broker that stops serving a shard now ends each of its subscriptions and cache watches with a final
shard_movedframe: the offset to resume from, the broker taking the shard when known, and the generation that moved it. Only a client that offersFEATURE_SHARD_MOVED(0x1000) gets it; any other sees the stream end byte for byte as before. For a durable streamresume_fromis exact, so resuming atmax(last delivered + 1, resume_from)neither repeats nor skips a record. AClusterClientsubscription does that itself: it resubscribes on the new owner, retrying until the move has cut over, and an in-memory stream resumes at the new owner’s tail. AClientsubscription ends withSubscription::shard_moved()set, and cache watches deliverCacheWatchItem::ShardMoved. Sharded subscriptions and watches reportShardEvent::ShardMovedandShardedCacheWatchItem::ShardMoved. Python and TypeScript subscriptions follow too, and surface the move asShardMoved/CacheWatchShardMoved(Python) andshardMoved(Node). See “Shard moves” indocs/protocol.md. -
Publishes are not refused while their shard moves. A publish that reaches a broker between a move’s fence and its cut-over is held, before it is accepted, until the broker’s routes show the new owner, and is then sent there. It is refused only if the move takes longer than
FELIX_SHARD_MOVE_HOLD_MS(default 2000) orFELIX_SHARD_MOVE_HOLD_MAX(default 1024) publishes are already waiting, withshard_unavailableand the new reasonmoving(retry classretry, with aretry_after_mshint). New metricsfelix_broker_shard_move_held_total,felix_broker_shard_move_hold_secondsandfelix_broker_shard_move_hold_refused_total{reason}. -
A move’s destination does not count toward the quorum while it copies. A
Quorumpublish waits for a majority of the replicas the stream asked for, not for the copy, and the copy is shipped in slices so it never holds a replication pass for long. -
Shard moves are paced. Every copy holds a move slot, including a follower being replaced on a draining broker, which is now copied in beside the follower it replaces (named
joining) and seated once it has caught up, so a shard never drops below its replication factor while it fills.FELIX_SHARD_MOVES_MAX_PER_NODEbounds copies into or out of one broker. Drains get slots before rebalancing. A move fences once its destination is withinFELIX_SHARD_MOVE_FENCE_MAX_LAG_RECORDS(default 1000) of the leader’s tail rather than exactly level, which under steady writes it may never be; replica reports carry the leader’s tail (leader_offset) for this. A move that has not reached its fence withinFELIX_SHARD_MOVE_TIMEOUT_MS(default 30 min) is abandoned as atimed_outstep, counted infelix_shard_moves_timed_out_total, and waits behind other shards for its next slot; a fenced move is always finished. On the broker,FELIX_SHARD_MOVE_BYTES_PER_SEClimits what a leader ships to move destinations that the quorum does not need (felix_broker_replication_move_throttled_bytes_total). Migration0014_shard_move_pacingadds the assignment and report columns. -
Operators can steer shard moves.
GET /v1/shard-moveslists the moves in progress with their step, the reason they started (drain,balance,operator,replace, stored asmove_reasonon the assignment), start time and lag;GET /v1/placement/planshows what the next placement pass would do without doing it.POST /v1/shard-movesstarts a move to a named broker, held to the move limits.DELETE /v1/shard-moves/{tenant_id}/{namespace}/{name}/{shard}cancels one: before the fence the destination is dropped; after it the fenced leader serves again at a new generation (retake) with every write it accepted, and held publishes and followed subscriptions find it again; after the cut-over it is a 409.POST /v1/placement/pauseand/resumestop and restart placement’s own moves on every instance; moves in flight finish, and new shards and failovers are still placed. Reads takenode.view:cluster:*, changesnode.manage:cluster:*.felix-controlplane admindoes the same from a shell (moves,plan,move,cancel,pause,resume, with--json). Migration0015_operator_movesaddsmove_reasonand theplacement_settingstable; the Raft backend gains aset_moves_pausedcommand, which an older member refuses.felix_shard_move_steps_totalgains thecancelandretakesteps. -
Faster shard move switch-over, control-plane side.
GET /v1/shard-assignments/changestakes an optionalwait_ms: with nothing newer thansince, the request waits up to that long (capped at 25 s) and answers as soon as a change lands. It holds no store connection while waiting, and withoutwait_msit behaves exactly as before. A replica report that a move is waiting for (a drained leader, or a caught-up staged successor) now wakes placement at once instead of at its next tick; wakes coalesce and one pass runs at a time. New histogramsfelix_shard_move_duration_secondsandfelix_shard_move_fence_secondstime moves from the steps each control-plane instance writes. -
Typed error codes on the client wire. A client that offers
FEATURE_ERROR_CODES(0x0800) gets acode, aretryclass and an optionaldetailon everyerrorandpublish_error, and on a failed binary publish ack when it also offersFLAG_BINARY_PUBLISH_ACK_CODE(0x0200). The retry class says whether the request may have been applied: a quorum timeout is nowquorum_timeoutwithoutcome_unknown, distinct from a refusal, andshard_unavailablenames its reason. An unknown code decodes with its retry class instead of failing the frame. Clients that do not offer the bit get byte-identical frames. A broker that is draining now answersauthon a new control stream withdrainingfor clients that offered the bit. The Rust client offers it and exposes the code onfelix_client::BrokerError. See “Error codes” indocs/protocol.md. -
The Rust
ClusterClientacts on the broker’s error codes. Retry, reroute or fail is decided by the retry class instead of by matching the message. A cached shard owner that answersshard_unavailable(includingfenced),drainingornot_leaderis forgotten and the publish goes to the entry broker at once, frompublishtoo, since nothing was applied.outcome_unknownis never re-sent bypublish;publish_at_least_onceand the idempotent producer re-send it.retry_afterhonoursretry_after_ms, andnot_foundis retried for 5 s from the first one rather than for the whole attempt budget. A fatal code such asinvalid_requestis no longer retried.publishreconnects only ondrainingor a dead connection, not on every coded refusal. A subscribe or cache-watch refusal is now a typedBrokerError(it was the debug text of the frame), and a subscribe, cache watch or group request whose redirect target answersshard_unavailablegoes back to the entry broker once. Peers that did not negotiate codes get the old handling. -
Python and TypeScript errors carry the broker’s error code. Both have
code,retryanddetail. The class is chosen from the code when the broker sent one, and from the message only when it did not. New classes:ShardUnavailableError(shard_unavailable,not_leader; retryable),OverloadedError, andOutcomeUnknownError(quorum_timeout,leadership_lost,unacknowledged, or anything sent asoutcome_unknown); a draining broker is aConnectionError. In TypeScript,retryablefollows the broker’s retry class when there is one, so aNotFoundErrorsent asretry_afteris now retryable. Two conformance scenarios,error.shard_unavailable_is_retryableanderror.quorum_timeout_is_outcome_unknown, hold both bindings and the Rust client to this, against faultsfelix-cluster client-fixturenow serves on a localhost control endpoint (POST /fence,/partition,/heal). -
Online shard rebalancing (#130). A shard whose leader is alive is now moved instead of reassigned. The control plane stages the destination as a replica and lets the leader catch it up. Once the copy is close to the leader’s tail it fences the leader (the assignment goes
draining, and a broker never serves a draining assignment), waits for the leader’s drained report, and only then names the destination leader at a new generation. Each step is an assignment write, so any control-plane instance resumes a half-done move from the store.Two triggers. A drained broker (
POST /v1/nodes/{id}/drain) hands off everything it leads and is replaced as a follower wherever it holds a copy, so it can then be deleted. A broker leading more than its share of shards gives them to one under its share, which is what makes a broker that joins a running cluster take work. A cluster where a control-plane restart left every shard on the first broker to register now corrects itself. Moves only go from over share to under share and count moves in flight as done, so placement converges instead of oscillating.FELIX_SHARD_MOVES_MAX_CONCURRENTbounds moves in flight, one by default.A destination that dies before it leads is passed over; a leader that dies mid-move is an ordinary failover with the destination as a candidate. Writes that arrive between the fence and the new owner opening are held and sent on (see “Publishes are not refused while their shard moves”). Real-process tests cover drain, join, a destination dying mid-transfer, a drain and a join at once, and publishes arriving throughout.
The fence is model-checked.
docs/formal/FelixShardHandoff.cfgadds the planned move to the TLA+ spec and explores 2.7M states without a violation;FelixShardHandoffNoWait.cfg, which cuts over as soon as the fence is written, finds two brokers serving one shard in seven steps.The assignment carries an optional
successor, the replica report an optionaldrained; both are omitted when unset, so nothing changes on the wire for a cluster that never moves a shard. Postgres gains migration0012_shard_moves.felix_shard_move_steps_total{step}andfelix_shard_moves_waitingreport progress. The harness gainedadd_node,drain_node,undrain_nodeanddrain_until_empty. -
A broker can bind several client listeners.
FELIX_QUIC_LISTENERSbinds that many QUIC listeners on consecutive ports from theFELIX_QUIC_BINDport, each with its own socket and accept loop. A broker with more than one names their ports inauth_ok(listener_ports, omitted otherwise), and the client spreads its publish, cache and event pools over them, always keeping the host it dialled. The Helm chart gainsbroker.ports.listeners(default1) and refuses to render ifbroker.ports.internalfalls inside the range. See Changed for the default count. (#595) -
ClusterClientsends publishes straight to the shard’s owner. It learns each shard’s owner from the publish ack and sends later batches there, keyed or not, instead of having the entry broker forward every one. It falls back to forwarding when the owner is unknown or unreachable.felix_client::publishes_forwarded()counts acks that came back forwarded. (#600) -
PATCHandDELETE /v1/nodes/{id}. Both were documented but answered 405.PATCHchanges a node’s region, labels, capacity, or lifecycle betweenliveanddraining(which cancels a drain); it cannot mark a down node live.DELETEneedsnode.manageoncluster:*and is refused with 409 while the node is live or draining or any shard still names it. (#644) -
FELIX_SHUTDOWN_PREDRAIN_MS. A stopping broker keeps accepting connections for this long after/readyturns false, so a load balancer polling/readynotices first (default0; a second SIGTERM ends the wait). In Kubernetes the chart’s preStop sleep already does this. (#645) -
A cache can be created with
Quorumconsistency.CacheandCacheCreateRequestgainconsistency(LeaderorQuorum, defaultLeader), set at creation only. On aQuorumcache a put, delete or counter update is acknowledged once a majority of the shard’s replicas hold it, and survives the loss of its leader. Postgres migration0013_cache_consistency.sql.felix_broker::CacheMetadatagains aconsistencyfield. (#648, #750) -
One consumer group across every shard of a stream.
ClusterClient::group_shardedreturns aShardedGroupthat finds each shard’s leader by following redirects, polls the shards in turn, and sends acks and nacks to the right leader. Ordering holds per shard. A broker now answers group requests for a shard it does not lead withNotLeaderto clients that offerFEATURE_REDIRECT; a client that offered the bit but predates this sees an unexpected-answer error instead of the old refusal. (#649) -
The Rust client refreshes its token for streams opened after connect.
ClientConfig::token_provideris asked for a token whenever a stream authenticates, and overridesauth_token.RefreshingTokenwraps a fetch function and refreshes once two thirds of the token’s life has passed. Before, subscriptions, watches, group requests and reconnects failed once the connect-time token expired. (#652) -
Watch a cache prefix across every shard.
ClusterClient::watch_cache_shardedandwatch_cache_sharded_retainedopen one prefix watch per shard on its owner and merge them. A retained watch emitsStateCompleteonce every shard has delivered its retained values, andresume_offsets()gives one offset per shard. New requestcache_shards, feature bitFEATURE_CACHE_SHARDS(0x0400) andClient::cache_shards. Rust only for now. (#654) -
A resumed subscription reports where it joined. A subscribe with a start position on a durable stream answers
subscribedwithstart_offset(the first record delivered) andlive_offset(the tail when it was registered), to clients that negotiatedFLAG_EVENT_BATCH_OFFSETSonly.Subscription::start_offset()andlive_offset()expose them. (#655) -
Idempotent publishes use the binary frame. The negotiated flag
FLAG_BINARY_PUBLISH_IDEMPOTENT(0x0100) carries a producer id and sequence on an acked binary publish.Publisher::publish_idempotent_batchuses it when the broker offers it and falls back to JSON otherwise. (#656) -
A different batch under a reused producer sequence is refused. A leader answered any batch under a sequence it already held as a duplicate, so a producer that reused a sequence got
okfor data never written. Each held batch now keeps a digest of its payloads, and a mismatch is refused with reasonsequence_reusedto clients that offerFEATURE_SEQUENCE_REUSED(0x4000); older clients still get the duplicate answer. The RustIdempotentProduceroffers it. (#764) -
Brokers negotiate the
felix/1ALPN. Client listeners selectfelix/1when offered and refuse clients that offer only other protocols. Clients that offer none are accepted unlessFELIX_TLS_REQUIRE_ALPN=true. (#751) -
Unknown requests get an answer instead of a closed stream. A client that offers
FEATURE_UNSUPPORTEDgetsunsupportedfor a request type the broker does not know, and the stream stays open. (#751) -
Brokers can serve a real certificate and require client certificates.
FELIX_TLS_CERT/FELIX_TLS_KEYserve a certificate from files, re-read every 30 s.FELIX_TLS_CLIENT_CAmakes the QUIC and Kafka listeners require client certificates. Without a certificate the broker still generates a self-signed one and warns;FELIX_TLS_REQUIRE_CERT=truerefuses to start instead. Helm:broker.clientTls.*. The Python and TypeScript clients cannot present client certificates yet. (#739) -
The control-plane API can serve TLS.
FELIX_CONTROLPLANE_TLS_CERTandFELIX_CONTROLPLANE_TLS_KEY(ortls:in the config file) serve the REST API over TLS, reloaded on rotation. Brokers andfelix-controlplane admintrust a private CA fromFELIX_CONTROLPLANE_CA. Helm:controlplane.tls.*. (#739) -
Per-IP connection caps and Kafka pre-auth limits.
FELIX_MAX_CONNECTIONS_PER_IP(default 512,0unlimited) caps QUIC connections from one source address per broker, andFELIX_KAFKA_MAX_CONNECTIONS_PER_IP(default 128) does the same on the Kafka listener, where requests before SASL are capped at 64 KiB and SASL must finish withinFELIX_KAFKA_AUTH_TIMEOUT_MS(default 10 s). Clients behind one NAT or proxy address may need a higher cap. (#749) -
Per-tenant publish quotas.
FELIX_TENANT_PUBLISH_BYTES_PER_SECandFELIX_TENANT_PUBLISH_MSGS_PER_SEC(default0, unlimited) set a per-tenant token bucket on each broker, withFELIX_TENANT_PUBLISH_BURST_MS(default 1000) and per-tenant overrides inFELIX_TENANT_PUBLISH_QUOTAS(tenant:bytes:msgs,...). An acked publish over quota is refused asoverloadedwith reasontenant_quotaandretry_after_ms; Kafka produce is throttled withthrottle_time_ms. (#749) -
Per-tenant metrics.
felix_tenant_published_{messages,bytes}_totalandfelix_tenant_delivered_{messages,bytes}_total. Tenants past the firstFELIX_TENANT_METRICS_MAX(default 100) share the label_overflow. (#749) -
Control-plane listings are paged. Tenants, namespaces, streams, caches, nodes, shard assignments and the RBAC listings take
limit(1-10000, default 1000) andcursor, and carrynext_cursorwhen more remain. Breaking: a caller that ignoresnext_cursornow sees only the first 1000. The RBAC listings still return a bare array when called with neither parameter. (#767) -
Consistent backups across brokers.
felix-controlplane admin backup-point <name>records one committed offset per shard log from each leader after a single barrier, and writes a JSON manifest. Brokers serve the offsets atGET /backup/offsetson the metrics listener. After copying the leaders’ shard directories,felix-broker restore-point --point FILEcuts every copied log back to the point, offline. (#799) -
Giving up a durable shard’s records is an explicit step.
POST /v1/placement/abandon/{tenant_id}/{namespace}/{name}/{shard}(?kind=cachefor a cache) orfelix-controlplane admin abandonlets placement move a shard it is holding for a dead owner (see Changed). It needsnode.manage:cluster:*and is refused with 409 unless placement is holding the shard. (#728) -
The Helm chart forms a Raft group once, and can encrypt the peer port. A post-install hook creates the ConfigMap
<fullname>-controlplane-raft-formedonce the API is ready; members start withFELIX_RAFT_INITIAL_CLUSTER_STATE=newonly while it is absent, so a member that later loses its volume waits instead of starting an empty group.bootstrapTimeoutSeconds(default 270) bounds the hook; keephelm --timeoutabove it.controlplane.storage.raft.tls.*turns on peer mTLS. A new NetworkPolicy (controlplane.networkPolicy.enabled, on by default under Raft) admits the peer port only from members andraftPeerFrom; add a migration tool’s pods there. (#761)
Changed
Section titled “Changed”-
Durable publishes queued on one shard are written as one append. When an executor takes a durable publish, it also takes the plain publishes queued behind it on the same shard (up to 64, or 1 MiB) and claims them together: one write, one flush wait and one fanout, with each publish still answered with its own offset, in order. The fence is checked per publish; a failed append or flush fails every publish in the claim.
felix_broker_publish_claim_jobsreports the group size. Group commit waiters now also watch the durable bound, so one flush wakes all of them at once instead of one at a time through the flush lock. -
A lone subscriber event is sent without waiting for a batch. A subscription batch takes what is already queued and flushes at once; it waits up to
FELIX_EVENT_BATCH_MAX_DELAY_USfor more only after the previous batch found events queued behind its first. With one message in flight, delivery no longer sits out the batch delay. (part of #926) -
Breaking: the broker’s default listener count follows its cores. With
FELIX_QUIC_LISTENERSunset, a broker bindsmax(1, min(cores / 2, 4))client listeners, using the cores it may run on (cgroup limits included). The count is shortened so the range stops beforeFELIX_INTERNAL_BIND, so a cluster member on the default ports keeps one. An explicit value still wins. A container that publishes only port5000, or remaps it, needsFELIX_QUIC_LISTENERS=1, because clients dial the advertised ports. The Helm chart still sets the count (default1), since it must list the ports. (#720) -
Binary publish payloads are no longer copied on decode.
felix_wire::binary::PublishBatch.payloadsis nowVec<Bytes>, each record a slice of the frame it arrived in, which drops one copy and one allocation per record on the broker’s ingest path. A retained record keeps its whole frame alive. The broker reads frame bodies with quinn’sread_chunks, taking up to 32 chunks per call instead of one; bytes read past a frame wait in the newserving::quic::FrameScratch, which replaces theBytesMutargument ofread_frame_limited_intoandread_message_limitedand must stay with one stream. The encoders still take&[Vec<u8>]. -
aws-lc-rs is the only TLS crypto provider.
ringis gone from the build: quinn, rcgen and reqwest now use aws-lc-rs, which rustls and jsonwebtoken already did. The broker and control plane install it as the process-wide rustls default at startup. reqwest moves to 0.13, so HTTPS calls to the control plane and to IdPs are verified against the system trust store (plusFELIX_CONTROLPLANE_CA) instead of a bundled copy of the Mozilla roots; the container images already shipca-certificates. -
Replication and the broker-to-broker transport are their own crate,
felix-replication.felix_broker_service::peeris nowfelix_replication::peer, andfelix_broker_service::replication::*is at the root offelix_replication.ShardKeyandShardKindare defined there and still re-exported fromfelix_broker_service::shards. No behaviour change. -
ClusterClientholds one connection per broker. Every role a broker plays (entry, shard owner, redirect target, producer leader) shares one client, and every stream to it is multiplexed on one QUIC connection, where each role used to open a fullClientof 20 connections. More open only when the streams on it saturate, up tocluster_conn_pool(FELIX_CLUSTER_CONN_POOL, default 8);cluster_streams_per_conn(FELIX_CLUSTER_STREAMS_PER_CONN, default 1024) sets the stream budget.ClusterClient::connections_per_nodeandClient::connection_countreport the counts.Client::listeners_in_usenow returns aVec. -
felix-conformanceis AGPL-3.0-only. It links the broker, storage and authz crates to run its suite, so a build of it was AGPL whatever its label said. It is not published; runningverifyover a client’s results puts no obligation on that client. -
OTLP tracing export is off unless an endpoint is set. The broker and the control plane used to install the OpenTelemetry layer unconditionally, so a deployment with no collector exported to
localhost:4317and logged a failed export for every batch. SetOTEL_EXPORTER_OTLP_ENDPOINT(orOTEL_EXPORTER_OTLP_TRACES_ENDPOINT) to turn it on. -
IdP group claims are always prefixed. A group claim value
Xnow maps to the RBAC subjectgroup:Xeven whenXalready starts withgroup:, so an IdP group namedgroup:operatorsis no longer treated as theoperatorsgroup. Where an IdP was set up to emit pre-prefixed values, rename the groupings (group:group:<name>) or emit the bare name. -
Token exchange and refresh refusals are counted. Both now land in
felix_controlplane_auth_rejected_total; unusable refresh tokens use the newrefresh_refusedreason. The refresh replay, bad-secret, revoked and issued counters are now in the control plane’s metrics table. -
Breaking for Rust callers:
MovePolicyis no longerCopy. It gainsregions,max_per_node,fence_max_lag_records,timeout_millis,restore_after_millisandpaused; clone it where it was copied. The control-plane config holds it asshard_moves. The control-planeStreammodel gainsregion: Option<String>, and the broker’sMembershipConfiggainsregion_bridges. -
Cache watches follow a moved shard.
ClusterClient::watch_cacheandwatch_cache_retainednow return aClusterCacheWatch(and takeself: &Arc<Self>). When the shard moves it hands outCacheWatchItem::ShardMovedas a notice and reopens the watch on the new owner atmax(offset after the last change, resume_from), so the caller no longer reopens it and sees no change twice or missing. A sharded cache watch follows each shard the same way;ShardedCacheWatchItem::ShardMovednow means the shard is being followed, and aShardClosedfollows if it could not be. The Python and TypeScript watches follow too. AClientwatch still ends with the frame. -
A drain moves leaders before it replaces followers. Move slots on a draining broker go to the shards it leads first, then to its follower copies, so with the default limit of one a follower’s copy no longer holds the slot while the broker’s leaderships wait.
-
Online rebalancing is marked done on the status page. The row now cites the tests behind each claim. Load-aware placement has its own row. Pages that still said a move refuses publishes or ends subscriptions rather than letting them follow were corrected, and the docs-site semantics page no longer says a cache cannot declare
Quorumor that metadata lives only in Postgres. -
Breaking for downgrades: storage format v3. A record’s length word now carries two flag bits and, for the first record of an idempotent producer’s batch, a 20-byte tag (see
docs/storage-format.md, “Producer marks”). v2 segments are still read and an unmarked record is byte for byte a v2 record, but new segments are written as v3, which a v2 build refuses to open: downgrading past this needs the data directory discarded. A shard directory may also hold aproducerssnapshot besideepochs. -
Storage format v4. The generation-start record is flag bit 29 of a record’s
payload_len, and only a v4 segment may hold one. Segments stay v3 until a log’s first generation-start record, so an upgrade stays reversible untilgeneration_startis finalized; after that an older build cannot open the logs. v2 and v3 segments are still read. No migration is needed. (Format v5, for commit records, is under “Atomic commits”.) -
Internal protocol kind 25,
ReplicateMarkedRecords, ships records together with their producer marks. A batch without marks still travels asReplicateRecords, unchanged; a follower that predates the kind refuses it and is halted rather than storing records without their marks. The API changed with it:ReplicateRecordshas amarksfield,batch_checksumtakes the marks,AppendRecordandLogRecordhave amark, andfelix_broker::replication::applytakes the marks. -
Breaking for Rust callers of the control-plane crate:
AppStatehas amove_policyfield,ShardAssignmentamove_reasonfield, andControlPlaneStorethemoves_pausedandset_moves_pausedmethods.Default::default()fills the fields. -
Breaking for Rust callers:
ClusterClient::subscribeandsubscribe_fromtakeself: &Arc<Self>and return aClusterSubscriptioninstead of a(client, Subscription)pair, so the subscription can follow its shard.ShardEventis#[non_exhaustive], andCacheWatchItemandShardedCacheWatchItemhave a newShardMovedvariant, so exhaustive matches need an arm. On the broker,Broker::end_subscriptionsandCacheWatchHub::end_shardtake an optional argument saying where the shard went. -
max_shardscaps roles when choosing a move’s destination. It was compared against leaders only, so a node full of follower roles could be given a shard to lead. Placement’s load now counts existing followers too. -
A replica report leaves out followers the leader could not reach, as it already left out halted ones, so an unreachable move destination is never fenced on its last position.
-
Breaking for TypeScript callers:
err.codeis now the broker’s error code. It used to hold this client’s own kind (FELIX_AUTH,FELIX_CONNECTION, …), which moves toerr.kind.codenow matches the Python exceptions and the protocol (forbidden,shard_unavailable, …) and isundefinedwhen the broker sent no code, as areretryanddetail;kindis always set. Code that switched onerr.code === "FELIX_AUTH"should switch onerr.kind, or on the class. -
Fresh placement bounds leaders as well as roles. With a replication factor equal to the node count every node holds a role for every shard, so the role bound was satisfied with every leader on one node, and the new rebalancer would then move them apart again. A cluster placed from scratch now needs no move to be balanced.
-
A
drainingassignment is reachable fromassigningas well asactive, since nothing reportsactiveyet and the fence should not cost a move an extra generation. A drained shard still leaves only through a freshassigning. -
A node marked draining no longer has its shards reassigned on the next placement pass to brokers that hold none of their log. Its shards are moved instead, which takes a few passes; the harness’s
move_shardsteps placement until the move completes. -
felix_router::shardis no longer public. Everything in it was already re-exported at the crate root, sofelix_router::ShardRouterand friends are the paths to use. -
felix-commondrops what nothing used:NodeConfig,LimitsConfig,Error::Config, and every id type butRegionId. -
felix-broker’scache_watch,consumer_groups,dead_letters,durable,group_deliveryandgroup_readermodules are private. Their public types are at the crate root (felix_broker::ConsumerGroups,felix_broker::GroupReader,felix_broker::DurableStorage, …), and the group tracker thatgroup_deliveryexposed is internal.replicationandtimingsstay public modules.ClaimedPublishis now exported, sinceBroker::claim_publishreturns it. The registry keys (CacheKey,NamespaceKey,StreamKey,TopicKey) are no longer public; nothing outside the broker used them. -
The workspace is grouped by role. Crates live under
crates/{protocol,server,sdk,testing}/and the service packages are renamedfelix-broker-serviceandfelix-controlplane-service(binaries unchanged), socargo test -p brokeris nowcargo test -p felix-broker-service. Crate internals are reorganised by domain;CONTRIBUTING.mdhas the rules. -
felix-storagepaths:CacheOpisfelix_storage::cache::CacheOp,Epochisfelix_storage::log::Epoch, theCorruption*types are only at the crate root, and the modules nothing outside the crate used (commit_order,segment::io,disk_log::{epochs, recovery, retention, segments, sync}) are private. -
felix-controlplane-servicepaths: the router andAppStateare inapi(wasapp), readiness isapi::readiness, TLS isserver::tls,membershipandplacement(withReplicaPositions) are undercluster, the Raft store isstore::raftwithcommandandstate_machinebeneath it, metadata export isstore::export, andnow_millisisclock::now_millis. Two integration tests are renamed:readiness_pgispg_readinessandmeta_raftisraft_state_machine. -
felix-broker-servicemodules are grouped by job, so their paths changed:quic/transport::quic,auth(withauth_demoasauth::demo) andcore_shardsare underserving::, as is forwarding to a shard’s owner (serving::forward, frompeer::forwardandpeer::handler).credential,membership,lease,node_catalogandclient_endpointsare undercluster::, andcontrolplaneiscluster::catalog_sync.shard_watch,shard_lifecycleandshard_routingareshards::watch,shards::lifecycleandshards::routing.peer::replicaisreplication::replica,peer::dispatchisnode::peer_dispatch,durable_configisconfig::durable, andtimingsisobservability::timings.peeris now only the broker-to-broker transport.- Startup moved out of the binary into
node::run_with_shutdown.cache_routingandgroup_opsare no longer public. - Log targets follow module paths, so a
RUST_LOGfilter such asfelix_broker_service::quic=debugbecomesfelix_broker_service::serving::quic=debug.
-
Only
felix-wire,felix-transportandfelix-clientare published to crates.io. The server crates were only there because of thein-processfeature below. -
Breaking: Raft members need an explicit cluster id and never form an empty group by accident.
FELIX_RAFT_CLUSTER_IDis required and recorded in the data directory; a member refuses a store recorded under another id, and an existing data directory adopts the configured one.FELIX_RAFT_INITIAL_CLUSTER_STATE(neworexisting, defaultexisting): empty members form a group only undernew, so a brand-new group needsnew(the Helm chart’s hook sets it). The chart’s StatefulSet now creates members in parallel. (#727, #761) -
Breaking: a durable shard with no caught-up replica waits for its owner. A durable shard with no replicas, which is every
replication_factor: 1stream, used to be placed on another broker about 15 s after its owner went quiet, where it served an empty log. Placement now holds it until the owner returns, which keeps its records at the same generation. It shows asowner_unavailableinGET /v1/placement/planand infelix_shards_unplaceable; giving the records up is the newabandonstep. In-memory shards still move at once. (#728) -
Breaking: publishes are scheduled per shard and shared fairly across tenants. The process-wide pool of four publish workers, hashed by stream, is replaced by one ordered lane per shard (or per remote shard being forwarded to), run on
FELIX_BROKER_PUB_WORKERS_PER_CONNexecutors (default 4) and picked by byte-weighted round robin across tenants, so a slow shard, a forward or a quorum wait no longer holds up other streams. Each tenant is guaranteedFELIX_BROKER_PUB_QUEUE_DEPTHslots (default 64) out of depth x workers and may borrow idle room.FELIX_BROKER_PUB_FLUSH_CONCURRENCYnow counts per shard. A full queue answers an acked publishoverloadedwith reasonpublish_queue_full. Existing settings keep their names but change meaning, so revisit tuned values. New metricfelix_tenant_publish_queue_full_total. (#774, #802) -
Every write path checks the lease. Forwarded publishes, cache puts and deletes, counter adds, group acks, nacks and dead-letter changes, and Kafka produce go through the same per-shard fence as a direct publish, which re-checks the lease when a queued write claims its offsets; a lapsed lease answers
shard_unavailable/fenced. Oncemajority_ackis finalized,Quorumstream writes are acknowledged by their followers instead. (#738) -
Replica reports get an answer per shard.
POST /v1/nodes/{id}/replica-statusanswers one outcome per shard (accepted,stale,not_leader,unassigned,future_generation) with 200 when all were accepted and 409 otherwise, where it used to answer 204. The control plane writes a report only if the node still leads the shard at that generation, checked in the same step, so a deposed leader’s report is no longer stored. Old and new brokers and control planes interoperate. (#738, #757) -
Readers of a
Quorumshard see only committed records. Subscriptions, replay, cache gets, counter gets, cache watches, group polls and KafkaFetchon a replicatedQuorumshard stop at the quorum mark, and Kafka reports the mark as the high watermark. Untillease_free_readsis finalized, a lapsed lease ends these readers withshard_movedand refuses cache reads asshard_unavailable/fenced.Leaderstreams are unchanged. (#738, #750) -
Followers remember the newest leader they accepted. Each shard directory holds a
replicafile with the highest accepted generation and the commit offset, synced before the first batch at a new generation, so a restarted follower still refuses an older leader. AQuorumleader sends its mark with each batch (peer kind 26), and a follower refuses a truncation below it. (#743) -
Node expiry tolerates clock steps. The control plane marks a silent broker down only after
FELIX_NODE_EXPIRY_TIMEOUT_MSplusFELIX_NODE_REGRANT_MARGIN_MS(default a quarter of the timeout), measured on its own monotonic clock. A dead broker is placed elsewhere a little later than before. On Linux the broker times its lease onCLOCK_BOOTTIME, so a host suspended past the lease no longer wakes up holding it. (#743, #758) -
Under Raft, heartbeats and the placement lease stay off the log. The leader keeps them as soft state, answers a heartbeat only after a quorum confirms it still leads, and writes only expiries and a heartbeat checkpoint every 5 s. The placement lease now expires on Raft as on Postgres. (#745)
-
Raft upgrades no longer depend on rollout order. Each member reports the metadata version it supports, and the leader proposes nothing above the group’s minimum: new commands answer 409 until every member is upgraded, and new fields are either left out on every member or the command is refused. A member that receives an entry it would drop refuses it and counts it in
felix_meta_raft_unsupported_commands_total, so a rolled-back member stops with an error instead of diverging. (#761, #857) -
Faster shard move switch-over, broker side. The broker long-polls
GET /v1/shard-assignments/changesand acts on a change at once instead of on the nextFELIX_CONTROLPLANE_SYNC_INTERVAL_MStick, and a move’s successor opens the shard ahead of the cut-over. On a local cluster the switch-over drops from about 8 s to under 100 ms. New metricsfelix_broker_shard_move_secondsandfelix_broker_shard_switchover_seconds. (#673) -
Breaking: a prefix cache watch on a multi-shard cache must name its shard. Without
shardit read shard 0 and saw only the keys that hash there; it is now refused with the shard count. Open one watch per shard or useClusterClient::watch_cache_sharded. (#647) -
Each durable log flushes on its own thread instead of tokio’s shared blocking pool, where it queued behind other shards’ work. The thread stops after 10 s idle. With
FELIX_STORAGE_IO_URING=1, new work wakes the ring at once instead of waiting behind another log’s fsync. (#692, #699, #716) -
Cache and counter compaction runs in the background. The write that crossed the threshold used to rewrite the whole live set under the shard lock (about 20 s with 20 MiB live). A write now only starts a pass, which copies live entries forward and deletes the sealed segments below the cut.
FELIX_STORAGE_COMPACTION_BYTES_PER_SEC(default 64 MiB/s,0unlimited) paces it. (#789) -
Opening a log no longer opens its sealed segments. Recovery reads each sealed segment’s header and last index entry only; files open on first read, with at most 256 open per log (
felix_storage_open_sealed_segments). Retention no longer holds the segment lock while it unlinks. A broker also closes a shard’s logs once it has no role in the shard, and opens shards in parallel. (#746, #765) -
Breaking: connection-labelled subscriber metrics are gone.
felix_sub_connection_subscriberslost itsconnection_idlabel, andfelix_sub_conn_queue_len{connection_id}is replaced by the histogramfelix_sub_conn_queue_depth. The demo signing key is behind a newdemocargo feature, sopubsub-demo-simple,cache-demoandlatency-demoneed--features demo. (#747) -
Breaking: unknown frame flag bits are refused on every stream.
FrameHeader::decoderefuses any bit outsideKNOWN_FLAGSwith the newfelix_wire::Error::UnknownFlags, which an exhaustive match must handle. (#737) -
MTU overrides are clamped on Linux.
FELIX_INITIAL_MTUandFELIX_MTU_UPPER_BOUNDabove 6550 are lowered with a warning, so ten GSO segments fit one UDP datagram. (#737, #754) -
A keep-alive that cannot keep the connection open is replaced. A
FELIX_KEEPALIVE_MSof zero, or not below half ofFELIX_MAX_IDLE_TIMEOUT_MS, becomes a third of the idle timeout, with a warning. (#751) -
A Raft write that cannot reach a quorum answers 503 instead of 500. The outcome is unknown, so a retried create may answer 409. (#813)
-
Group polls and in-memory group state are bounded. One poll returns at most 1,000 records and about 4 MiB. A group untouched for 10 minutes (or twice its visibility timeout) is dropped from memory and rebuilt from disk.
FELIX_GROUP_MAX_IN_FLIGHT(default 10000) caps a group’s unsettled records; a poll at the cap returns empty, counted infelix_group_polls_capped_total. Long polls wake on appends instead of re-checking every 20 ms. (#731, #744) -
felix-loadgeningest stops at its deadline and spreads keys over every shard.--duration-secsnow bounds each send and ack wait, and the JSON report gainscompleted_records,acked_records,acked_throughput_*andcut_off_records. (#924) -
npm publishes through trusted publishing instead of a token, and each package is published with an explicit
npm publish, platform packages first. A manualnpm_stageinput stages versions for a maintainer to promote. (#580, #581, #582)
Removed
Section titled “Removed”felix-client’sin-processfeature andInProcessClient. It wrapped an embeddedfelix_broker::Brokerfor two smoke tests and nothing else used it, but it madefelix-brokerandfelix-storageoptional dependencies of the client, and so forced them onto crates.io. Test against a broker over QUIC instead;felix-clusterstarts one.Broker::in_flight_publishes. It counted claimed publishes for the drained report, which now reads the broker service’s per-shard write fence instead.
-
In-memory replicated streams open after
generation_startis finalized (#930). A promoted shard of an in-memory stream tried to append a generation-start record, which only a durable log can take, so it never opened for writes and its promotion retried five times a second. In-memory streams now skip the record: their publishes take no log offsets and are not replicated, so there is nothing inherited for it to cover. A promoted shard kept closed after its fence now backs off, doubling from 200 ms to 2 s. -
A cache put or delete ships without waiting for the replication tick (#928). Stream publishes and counter updates woke the replication driver, but cache writes waited for its next tick (
controlplane_sync_interval_ms, 2 s by default), so aQuorumcache put took about 840 ms. Every cache write now starts a replication pass, forLeadercaches too. -
A broker’s rotated refresh token is no longer world-readable (#913). Each rotation wrote
FELIX_NODE_REFRESH_TOKEN_FILEthrough a temporary created with the default mode, so 0644 under the usual umask, even when the operator had created the original 0600. The temporary is now created 0600, and a stale one left by a crash is removed first rather than reused with its old mode. -
FELIX_EVENT_BATCH_MAX_DELAY_USbounds the whole batch (#719). The subscriber feeder restarted the delay on every event, so a publisher sending faster than one event per delay held each subscriber’s events until 64 had built up or 64 KiB filled: about 6 ms at one event per 190 µs. The delay now counts from the batch’s first event, as documented. -
An io_uring submit error no longer closes files under in-flight flushes (#722). With
FELIX_STORAGE_IO_URING=1, a failed submit dropped every outstanding flush’s file while the kernel could still be syncing it, or had not yet taken its entry, so the sync could land on a reused descriptor. Each file is now held until its completion is reaped; the waiters are still failed with the error.EAGAINandEBUSY, which only ask for completions to be reaped first, no longer fail any flush. -
A power loss on a fresh node keeps its store roots (#888). The stream, group, dead-letter, cache and counter roots were created without syncing the directory that holds them, so a power loss before anything else synced it could drop a root and the acknowledged records in it. Every new directory is now synced into its parent, and opening an existing root syncs it again.
-
A new copy of a trimmed shard is placed without a rebuild. The leader’s first batch opened an empty log at 0 on the new broker, which then refused the leader’s bootstrap offer as if it held records. The copy waited for a rebuild slot, or for ever with
FELIX_REPLICATION_REBUILD_MAX_CONCURRENT=0. An empty log now takes the offered base. -
A subscription follows a moved shard off a draining entry broker. A
ClusterClientreader toldshard_movedwith no new owner named (a lapsed lease or a deposed leader ends readers that way) asked only its entry broker. When that was the old owner shutting down, it answereddrainingand then nothing, and the reader failed with “the new owner did not take it before the deadline”. The follow now moves to another known broker when the entry broker answersdrainingor not at all, and a reconnect tries the broker it was using last. Seedocs/multi-node-client.md. -
A broker handed a shard back after dropping held batches takes writes (#881). Liveness only: no acknowledged write was lost. A lease-mode leader that lost its lease with
Quorumbatches still held dropped them, followed the new leader, and answered a batch shipped again with that batch’s end. Its stream’s commit order moved back below records already in its log, so once the shard moved back to it every publish timed out withpublish commit timeout. The commit order now never moves backwards on a replicated batch. Seedocs/replication-design.md, “Replication”. -
A new leader names no follower short of the log it inherited (#882). Until a majority held a record of the new generation, a leader measured its replica report against offset 0 and named every follower caught up, answered or not. In lease mode, where promotion trusts the report, two failovers in quick succession could promote a follower missing a record the first leader acknowledged. Stream shards promoted with the fence were covered by it;
Quorumcaches and unfenced stream promotions were not. The report, and a cache’s counter level, now never name a follower short of where the leader’s generation begins. Right after a failover, a new leader that dies before any follower has copied its inherited log leaves the shard unplaced until a broker holding that log returns. Seedocs/replication-design.md, “Who may be promoted”. -
A leader counts a follower only for what it has answered (#878). The quorum mark and the replica report counted a follower the leader had not yet heard from at the offset its cursor starts at, one record below where the leader’s own generation begins. A new leader, or a move’s source, that could reach none of its followers then published a mark over records it inherited and no follower held: its commit offset covered them, and readers could be handed them as committed. Deposed, the broker refused to drop them and refused every rebuild, so it never rejoined. Both now count
FollowerCursor::confirmed, what the follower answered holding at this generation. No acknowledged write was affected: a publish waiting on the mark is dropped when its generation ends. Seedocs/replication-design.md, “Replica reports and the committed mark”. -
A
Quorumfailover no longer promotes a move’s destination. Its broker opened the shard as the move’s cut-over, without the promotion fence, and undermajority_ackthe report that named it caught up could predate a record the old leader acknowledged on its followers, so that record was lost. Failover on a durableQuorumstream now ends the move instead. A promoted leader that gets a new generation while still fencing also keeps replication paused until it reopens, rather than shipping its unfenced log at the old generation in between. -
A cache leader that loses its leadership rejoins as a follower (#863). Cache leaders now record where their generation began, on the cache and counter logs, and accept the generation on both. Before, a cache leader that died holding a record no majority had came back with no generation history, halted on that record, and refused the new leader’s rebuild for the rest of the generation, so a later drain could wait out the 30-minute move timeout. A rebuild now keeps the follower’s records below its commit offset and drops only what lies past them; the leader re-ships from its base, so those records are compared as they arrive, and a committed record that disagrees halts loudly (
below_commit) and refuses further rebuilds from that leader. A refused rebuild is asked again after a backoff (5 s, doubling, capped at 5 min) instead of never.FollowerCursor::rebuild_refusedis now anOption<RebuildBackoff>. Seedocs/replication-design.md, “Rebuilding a halted follower”. -
A
Quorumfailover keeps its replica set. Failover used to rebuild the followers from live brokers, dropping the dead leader and adding a broker that had never held the shard. Withmajority_ackfinalized the promoted leader fences a majority of that new set, which it could make with the newcomer alone, or with nobody when no other broker was live, and so open without a write the old leader and another follower acknowledged. A durableQuorumstream now keeps the previous set, the dead leader in it, so the fence needs a majority of the set that acknowledged. The dead member stays a follower until it returns or is drained. Upgrade the control plane before finalizingmajority_ack. -
Replacing a
Quorumfollower keeps what the old set acknowledged. A drain seats the replacement and drops the old follower once the replacement is within the move lag bound, or once a report names it caught up, which a leader does for every follower at a new generation before anything is counted there. Seated early, the replacement and a lagging follower were a majority of the new set without a record the leader and the old follower acknowledged, and a failover then lost it. On a durableQuorumstream the seat now also waits until the replacement holds what a majority of the set holds. -
A subscription’s
live_offsetis reachable after a leadership change. It was the raw log tail, so when the log ended in a new leader’s generation-start record, which is never delivered, a reader waiting to reachlive_offsetwaited until the next client write. It now stops short of trailing generation-start records, never belowstart_offset. -
A cache write the store refused is no longer acknowledged. A put or delete whose log write failed (a failed fsync poisons the shard) was logged and answered as a success, so a
Quorumcache could acknowledge writes that later reads did not return. It is now a storage error, and a failed read is an error rather than a miss.felix_storage::StorageApi’sput,getanddeletereturnResult. -
A Raft leader elected after a clock step back expires a dead broker in one window. The leader judged a broker it had not heard from by the log’s stamp when that was later than its own start, so a stamp left in the future by a step back, or by an old leader whose clock ran ahead, kept a dead broker placeable until real time caught up. It now ages every broker on its monotonic clock only, from its election or the broker’s registration, and lists a stamp ahead of its clock as now. Separately, a leader that died was counted at metadata level 0 by its successor, which then fell back to heartbeats and expiry through the log for as long as the old leader stayed down. The member probing a peer’s level now states its own, so every member knows the leader’s.
-
A record a new leader inherited is no longer lost after it was acknowledged (Raft’s Figure 8), once an operator finalizes the new fleet feature
generation_start. A leader then appends a generation-start record whenever it starts leading a stream shard (after a promotion’s fence, at either end of a move, on a cancelled move’s hand-back) and its quorum mark counts a majority only once it reaches a record of the leader’s own generation, so an inherited record is acknowledged only when a later fence cannot prefer a log without it. Acknowledged inherited records are readable at the new leader without waiting for a client write. Until the finalize, brokers count as before. Readers skip the record’s offset; seedocs/protocol.mdfor how a subscriber tells it from a drop. Finalize it only once every broker runs this version; it is one-way (see the upgrades page). -
Opening a
ClusterClientsubscription waits out a shard that is still opening.subscribe,subscribe_fromandsubscribe_shardedfailed at once onshard_unavailable/not_ready, which a leader answers while it fences its replicas after a promotion, including at cluster start. They now retryretry-class refusals with theReconnectPolicyattempts and backoff, as a publish does; afatalrefusal still returns at once. -
A cache or counter shard that an older build left mid-compaction opens whole again. In 0.5.0 and earlier, compaction swapped a shard directory with
<shard>.retiredand<shard>.compactingsiblings. Background compaction dropped that swap and its recovery, so on upgrade a shard stopped between the two renames opened empty. Opening a shard now settles the old swap first: it restores<shard>.retiredwhen the shard directory is missing, and it deletes leftover siblings when the directory is there. The cleanup assumes that a shard directory next to<shard>.retiredis complete, and deletes.retired. That holds for every release. It fails only for someone who ran an unreleased main build from between #789 and #797 on a half-swapped shard: that build opened the missing directory as a new, empty shard, and.retiredheld the only copy of the data. If you did, replace the shard directory with its.retiredsibling by hand before starting this build. -
A power loss during a replication truncation or reset no longer leaves a log that refuses to open. Truncation unlinked the segments past the cut (sealed ones newest first, the active one last) and synced the directory once at the end. A reset created its new segment before deleting the old ones. So a power loss could leave segments that do not meet, and recovery reported corruption. Both now delete newest first with a directory sync after each unlink, and a reset creates its new segment only after the old ones are gone. An interrupted pass leaves a longer log, which replication cuts again. A head gap that an older build’s retention left behind still needs a manual fix; see “A gap at the head left by an older build” in
docs/durable-storage.md. -
A power loss during a retention sweep no longer leaves a log that refuses to open. Retention unlinked a sweep’s segments without syncing the directory, so the device could keep a newer unlink and lose an older one. Recovery then found an offset gap and reported corruption. Segments are now unlinked oldest first with a directory sync after each, so an interrupted sweep leaves a longer log instead.
-
A
ClusterSubscriptionread cancelled mid-resume no longer ends the subscription. After a lost connection,next_eventcleared the loss before resubscribing, so a caller that dropped the call partway (a read undertokio::time::timeout, and every Pythonnext_event(timeout=...)) came back to the dead subscription and gotNone, as if the broker had closed the stream. The loss is now kept until the resubscribe lands, and the resubscribe (after a loss or a shard move) runs on a task of its own, so a caller polling with a timeout shorter than a reconnect no longer restarts it on every call and never finishes. -
Python’s
closedtells a timeout from the end of the stream. It was only set byclose(), so afternext_event(timeout=...)returnedNonethere was no way to tell the two apart, which its documentation said there was. It is now also set once the broker ends the stream. -
A publish in flight when its shard’s log is reset no longer reaches readers. A publish waiting behind earlier ones when the broker became a follower (or its log was rebuilt) was woken by the reset and went on to put its offsets from the old log into the cleared replay ring and fan them out. It now fails with
unacknowledgedand applies nothing. -
Acks for group claims made before a failover, move or eviction are settled. A new owner refused acks and nacks for records an earlier owner had handed out, as
invalid_request, which is fatal. It now accepts any ack below the log tail it inherited. An ack for a record written after that and not yet polled is refused with the newstale_claimcode (retry classretry; an older client sees an unknown code with that class). An offset past the log tail is stillinvalid_request. (#760, #776) -
A traced publish no longer panics a stream handler. The binary batch, JSON publish, JSON batch and subscribe handlers held a span guard across an
.await. When the task resumed on another worker the guard exited there, leaving a stale span on the first worker’s stack; the next span created on that worker could then clone the closed span and panic with “tried to clone a span that already closed”, killing the publish stream. The binary batch span isinfo, so this hit the default filter under load, and a slowon_closein the OpenTelemetry layer widened the window. The handlers now use.instrument(span), andclippy.tomlrejects span guards held across an await. -
felix_publish_requests_totalandfelix_publish_bytes_totalare recorded in a default build. Both went through thetelemetry-gated macros, so a broker built without that feature exported neither while serving traffic. Bytes are now counted for fire-and-forget (accepted) publishes too, as the payload sum of the request, and no longer for a batch whose acknowledgement timed out. The observability page marks which of the listed metrics still need thetelemetryfeature. -
A
Periodicfsync tick skips a log with nothing to flush. Every open log fsynced on every tick, written or not: a broker with ~200 idle shard, cache and counter logs issued ~800 fsyncs/s. -
Clients publishing one stream spread across a broker’s listeners. The stream-to-publish-stream hash used a seed fixed for the whole process, so every
Clientin a process sent a given stream down the same pool slot, and so to the same listener. On a four-listener broker fed by four load generators, two ports carried all the publish data (3.1 GB and 9.4 GB) and the other two only handshakes. The seed is now per client; one client still keeps each stream on one writer. -
Concurrent “ensure signing keys” calls agree on one key set. Postgres and in-memory control-plane stores checked for a tenant’s keys and wrote new ones in separate steps, so racing callers each generated keys and all but the last returned a set that was then overwritten; tokens signed with it would not verify. Postgres now decides under the tenant row lock, the way auth bootstrap does, and the in-memory store under one write lock. The Raft store was already safe: install-if-absent is applied through the log.
-
A failover no longer costs a publisher 30 seconds. A publish in flight to a leader that was killed waited out the 30 s ack timeout, because a killed broker sends no QUIC close and the connection only died at the 30 s idle timeout. The default QUIC idle timeout is now 6 s and keep-alives go every 2 s (
FELIX_MAX_IDLE_TIMEOUT_MS,FELIX_KEEPALIVE_MS), so the dead connection fails the waiting publish andClusterClientretries on the new leader in about 6 s. Quiet connections stay open on the keep-alives. A keep-alive not below half the idle timeout is replaced (see Changed). -
A publish to a shard that cannot be served is no longer called “stream not found”. The broker kept the right code but replaced the message with
stream not found: ...for every refused publish, so a shard that was moving, fenced or still opening read as a missing stream to operators and to clients without error codes. Only a stream that does not resolve says that now; any other refusal keeps its own code and text, e.g.publish to t1/ns/orders refused: this broker is still opening the shard. Clients that matched on the old text for these cases will see the new one; retry behaviour of coded clients is unchanged, since the code was alwaysshard_unavailable. -
Binary publish acks carry the error’s detail. A failed binary ack had the code but dropped
detail, so a binary publisher never learnt ashard_unavailablereason or amovingshard’s suggested wait. The new flag bitFLAG_BINARY_PUBLISH_ACK_DETAIL(0x0400), negotiated likeFLAG_BINARY_PUBLISH_ACK_CODE, adds it after the code;felix-clientoffers it and fillsBrokerError::detailfrom it. The newencode_publish_ack_bytes_detailedencodes it;PublishAckgains adetailfield. -
Forged upstream tokens can no longer make the control plane hammer an IdP. A token with an allowlisted issuer and an unknown
kidmade/token/exchangere-fetch that issuer’s JWKS before any signature check, one fetch per request, with no credential needed. Each JWKS URL is now fetched at most once per 30 seconds, concurrent misses share the fetch in progress, and JWKS and discovery requests time out after 10 seconds. A key the IdP rotates in is accepted within 30 seconds of its first use. -
An assignment change refreshes the node catalog. A broker read
/v1/nodeson a wake only when an assignment named a node it did not know. A broker that restarted comes back under the same id on new ports, so it was known and its old address kept: the first ship or forward to it after a move waited for the next catalog tick (FELIX_CONTROLPLANE_SYNC_INTERVAL_MS). Every wake now reads the catalog, alongside reconcile rather than before it, and each read is bounded at 5 s so a control plane that never answers cannot hold the feed (a_wake_refreshes_the_address_of_a_node_that_moved, and infelix-cluster,a_move_onto_a_restarted_broker_does_not_wait_for_the_catalog_tick). -
A leader ships to a restarted follower at its new address. A follower’s cursor kept the address it was created with for the whole generation, so a broker that restarted on new ports, still in the replica set, was dialled at its old one until the shard’s generation changed. A move staged onto a broker that had just restarted never caught up and sat at
stageduntil the move timeout. The cursor now takes each pass’s address from the route, so the follower is reached within one catalog refresh (FELIX_CONTROLPLANE_SYNC_INTERVAL_MS) (a_follower_that_moved_address_is_shipped_to_at_the_new_one, and infelix-cluster,a_restarted_broker_takes_shards_again). -
A broker that took a shard over replays all of it. A follower’s replay ring was filled when the first replicated batch opened the stream and was never touched again, so after a failover or a planned move a reader from the start got that first batch and then the live edge, skipping everything replicated in between. The records were on disk; a cursor-based subscribe had the same hole. The ring is now emptied as replicated records move the tail, and a reader is served from the log (
records_replicated_after_the_first_batch_replay_from_the_start, and infelix-cluster,a_handoff_that_times_out_loses_no_acknowledged_record). -
Client: out-of-order publish acks no longer break the stream. The broker answers acked publishes as each completes, so on a publish stream carrying several at once a
Quorumor forwarded publish can be answered after one sent behind it. The client required acks in request order and treated any other as a protocol error, failing every publish outstanding on that stream and reconnecting. Each answer now goes to the request it names (acks_answered_out_of_order_reach_their_own_requests). -
A Postgres-backed control plane keeps a stream’s replication factor. Reading a stream back always reported a replication factor of 1, so a replicated stream was placed as leader-only, and any patch to the stream (retention, consistency, delivery) wrote the 1 back over the stored value. The store’s Postgres tests, which would have caught it, never ran:
task testdid not enable their feature, and when enabled they skipped themselves, because resetting the schema failed on a foreign key and the tests sharing it deadlocked. They now run with the databasetask teststarts, serially, and a setup failure fails them. -
A
Quorumshard is replaced when its leader dies mid-publish. A leader reported a follower caught up only when it held the leader’s whole log, so a publish landing between shipping and the report left every follower one record short and the report named nobody. A leader killed right then left the control plane no replica it may promote, and the shard stayed down for good while its producers were toldstream not found. UnderQuoruma follower now counts as caught up when it holds everything up to the offset a majority holds, which is every record that can have been acknowledged.Leaderstreams and moves still require an exact copy. -
The move limits hold across control-plane instances. Each placement write was conditional only on its own shard’s generation, so two Postgres-backed instances, or a pass and an operator’s request on another instance, could each read the last free slot and start moves on two different shards. One instance now runs the timed placement passes, the holder of a lease in the store that lasts three reconcile intervals, is renewed every pass and is released on shutdown; under Raft the leader holds it. Every placement write, operator steps included, is also fenced by a placement token read before the pass or request decided: a write lands only if no other placement write landed since, and a new holder advances the token, so an instance that paused past its lease writes nothing after a takeover. Passes woken by a report still run where the report arrived. Postgres gains migration
0016_placement_lease.sql; the Raft log gainsput_shard_assignment_fencedandtake_placement_leasecommands, which a follower running an older build refuses, so upgrade every member before the leader.ControlPlaneStore::put_shard_assignment_iftakes the token. Metrics:felix_placement_lease_held,felix_placement_lease_takeovers_total,felix_placement_writes_fenced_total. -
A redirect to a draining broker carries its address. A
not_leaderanswer or publish-ack owner hint naming a broker that is being drained, but still leads the shard, omitted the client address because only brokers eligible for new work were listed. Routable brokers’ addresses are now used for redirects;topologystill lists only eligible brokers. -
One refused publish failed later publishes on the same connection. A publish the broker refused (an unknown stream, a forbidden one, overload) stopped the client’s publish worker that sent it, and every later publish routed to that worker, to any stream, got the same refusal back without a code.
ClusterClientthen took the uncoded error for a dead connection and reconnected. A refusal now answers only the publish it belongs to; a broken or timed-out ack stream still ends the worker. -
A long replay lost records and could look stalled. A subscription resumed from an early offset dropped history even when the application read every event promptly: the broker writes history as fast as it reads it, and the client drained it into its bounded queues under the default
drop_newpolicy. A replay of 5,000 records typically delivered the first 256 and a scattered few after; when the tail was among the drops, the subscription went quiet until the next live publish. History below the subscription’slive_offsetnow waits for room in the client’s queues whatever the policy, so a replay is paced by its reader and never dropped. Live records pastlive_offsetkeep the configured policy. -
A subscription’s last events could be dropped when it ended. The subscriber’s connection writer dropped deliveries queued in the same batch as its unregister, and the feeder forgot the subscriber’s connection before deliveries still in its lane queue were routed. Both now deliver them first, so a subscriber ended by a shard move receives everything the broker committed.
-
A
Leaderpublish acknowledged on enqueue could be dropped without a trace. Withack_on_commitoff (the default) the ack goes out when the publish is queued; if the broker’s lease lapsed before the worker wrote it, the worker refused the write and the refusal went nowhere. The refusal is right (another broker may lead the shard), so admission now reads the lease clock for such a publish and, with less left than the publish queue wait plus the ack wait (capped at half the lease), waits for the write, so a lapse reaches the client asshard_unavailable. A pause that starts after the ack can still strand one; it is counted in the newfelix_broker_acked_publishes_dropped_total{reason}and logged at warn. The docs no longer claim aLeaderack means the record is durable under the default. (#671) -
A
Quorumstream placed with one replica refused every publish on a cluster withleadership_lost: nothing ships for such a shard, so no quorum mark was ever published. The leader alone is its majority now. -
An acknowledged write could be lost in a planned shard move. Admission checked that the broker served the shard, but an admitted publish could wait in the publish queue and claim its offsets after the old leader had reported
drainedand the control plane had cut over; it was then committed and acknowledged on a broker the new owner never copied it from. The drained report guessed at this with the tail holding still for two passes, which a deep enough queue outlasts. The same gap existed for cache puts and deletes, counter adds, consumer-group polls, acks, nacks and dead-letter changes, and publishes forwarded to the old leader.Every such write now enters a per-shard write fence right before it claims its place in the log and holds it until it is durable and fanned out. The shard lifecycle closes the fence as soon as it sees a move, and a write that reaches the fence afterwards is refused the way a write to a shard the broker does not serve is. The drained report goes out only once the fence is closed with nothing inside it, which makes it exact. The readers a moved shard’s old leader ends now receive every record it committed first. The TLA+ model splits admission from the claim, and
FelixShardHandoffNoClaimFence.cfgshows the loss without the check. -
A planned shard move could lose a dead letter or a counter add. The old leader reported
drainedon the shard’s own log and shipped its consumer groups’ cursors and dead letters, and its cache’s counters, only afterwards, logging a failure and moving on. A dead letter the new owner lacked was a record its group silently skipped, and a counter add it lacked was an acknowledged add gone from the sum. The drained pass now ships those logs first and reportsdrainedonly once the move’s successor holds them; any other replica still missing them is left out of the report’s caught-up list rather than holding the move. A move that lost its successor waits for every replica level on the shard’s log. A move held on them staysDraining, andfelix_broker_replication_drain_withheld_total{log}and a warning naming the shard and follower say why. Brokers now read the assignment’ssuccessor. -
A broker shutting down in the middle of a credential refresh could exit after the control plane had rotated its refresh token but before writing the replacement, so the next start presented a spent token and the chain was revoked. Shutdown now waits, within the drain deadline, for a refresh in flight to finish.
-
A failed fsync poisons the log. Linux can drop the dirty pages behind a failed fsync, so a later successful sync could acknowledge lost data. Any flush, seal, truncate or reset failure, including a failed directory or epoch-file sync, now stops the log’s durable offset, and every later write returns the error until the broker restarts. Readers of a poisoned
Leaderstream stop at the durable offset. (#726, #777, #851) -
Recovery repairs what a power loss leaves. A zero-filled tail is cut back as a torn tail; a roll interrupted by power loss, a segment cut short while it was sealed, and an empty segment left by a lost background roll are repaired; a sealed segment whose index points at the wrong place is rescanned. Each log keeps a
durable.markfile, and damage past it is repaired as a torn tail, so a power loss underPeriodicorNonefsync no longer leaves a shard that refuses to start. Damage before the mark is still fatal. (#726, #746, #754, #759, #775) -
A cancelled cache put or delete is still applied. A write cancelled after it claimed its offset released its commit turn before its fsync finished, so readers could see a record that was not yet durable. (#726)
-
A follower’s commit offset is on disk before it acknowledges, under
OnCommit. It was written at most once a second, so a crashed follower could read back a lower offset and allow a deeper truncation. (#769) -
A panicking publish worker is restarted, and only the job it was running fails (
felix_broker_publish_worker_restarts_total). Graceful shutdown now waits for queued publishes, including ones acknowledged on enqueue, to be written. (#747) -
One stalled subscription no longer blocks the others on its connection. Each subscriber has its own queue in the connection writer and drops only its own frames (
felix_sub_queue_dropped_total). ALatestsubscriber no longer gets live records from below its start offset. (#747) -
An idempotent producer no longer reuses a sequence after an ambiguous failure. The next, different batch went out under the same sequence and was answered as a duplicate without being written. Such a batch is now held in doubt: only the same payloads may be re-sent under it. (#737)
-
A lost connection ends a subscription with an error, not as the end of the stream. It surfaces as
felix_client::SubscriptionLost, andClusterSubscriptionresubscribes from the next offset until the retry policy’s deadline. A publish larger than the 4 MiB in-flight budget now goes out alone instead of never. (#737) -
A redrive survives a failover. It is one durable write, so a new leader delivers redriven entries again, and a failed dead-letter write is retried on the next poll instead of stalling the group. (#731)
-
A consumer group cannot redeliver acked records after its shard moves back. The in-flight tracker now resets when a broker starts a new term leading the shard. (#688)
-
A follower’s leftover tail from an older leader is compared before a new leader ships past it, so a dead leader’s unacknowledged write cannot survive at an offset the new leader filled differently. A dropped suffix is dropped from the replay ring too. (#779, #782)
-
A follower keeps the generation each record was written at. It labelled every record with the sender’s generation, so a copy of an inherited record looked newer and a later promotion could prefer that log and lose an acknowledged record. Batches now carry per-record generations (peer kinds 32 and 33, capability
GENERATION_LABELS); the fix holds once every broker runs this version. (#814, #837) -
A
Quorumpublish no longer fails while a move is staged. It waits for the new generation’s first quorum mark instead of being refused. (#796) -
A slow peer no longer holds other quorum marks. One unresponsive peer could delay every
Quorumacknowledgement on the broker by about 2 s; each shard now replicates on its own schedule. (#804) -
One slow peer request no longer stalls every peer. Broker-to-broker handlers ran inline on the QUIC I/O thread. (#767)
-
Brokers rejoin after being marked down, and a control-plane outage no longer expires the fleet. The expiry sweep waits a full window after start, an election or a lost store, and broker calls to the control plane have deadlines. (#730)
-
Deregistering a live broker no longer allows two leaders. A
leftnode is treated as draining until one expiry window plus the regrant margin has passed since its last heartbeat. (#766) -
A dead broker expires in one window after the control plane’s clock steps back on the memory and Postgres stores. (#785)
-
Replica reports survive a Raft snapshot and the
migrateexport. (#849) -
A retried Raft write no longer answers 409 for its own success. Each write carries a request id fixed before the first attempt. (#602)
-
A Raft member that lost its volume does not vote until it has caught up, so it cannot win an election and truncate acknowledged metadata writes. (#670)
-
A control-plane member that cannot verify a token says so with
503 cannot_verifyinstead of401 invalid token. (#603) -
Two control-plane instances could undo each other’s shard moves. Each instance wrote what it planned unconditionally, so one that stalled could hand a shard back to its old leader after another had cut over. Every placement, promotion and move step is now written only if the shard is still at the generation it was planned from (
ControlPlaneStore::put_shard_assignment_if, Raft commandPutShardAssignmentIf); a skipped write is counted infelix_shard_assignment_write_conflicts_totaland re-planned. (#660) -
The docs said the clients were not installable.
felix-clientis on crates.io, PyPI and npm; the client pages and binding READMEs now say how to install it. (#585) -
Release pipeline fixes. The npm job published nothing and reported success; it now moves each binary into its platform package, publishes the main package last, and checks the registry. The Node addon’s
x86_64-apple-darwinleg uses the pinned toolchain. The PyPI step skips files already uploaded, the crates.io job waits out the new-crate rate limit, and every npm command pins the public registry.scripts/npm_first_publish.shclaims a new package name, which CI cannot. (#578, #579, #580, #582, #583, #584)
Known limitations
Section titled “Known limitations”Deliberately not in this preview:
- A single acked publish on the fsync path still takes longer to be acknowledged than the comparable NATS JetStream setup (#926).
- Claiming a lane’s queued publishes as one durable append is still being measured (#932).
- A publish connection’s QUIC receive windows can buffer far more than the ingress budget before backpressure applies (#923).
Quorumcache and counter shards still open on the lease, not on a fenced majority (#933).
