Skip to content

Felix 0.6.0-preview

Released 2026-10-02 as a preview. The full history is in CHANGELOG.md.

A preview of 0.6.0. Shards move between live brokers without refusing writes, Quorum streams can acknowledge by their followers and serve reads without the lease once an operator finalizes the matching fleet features, storage recovers from power loss and failed fsyncs without losing acknowledged records, and Kafka clients can produce and consume. pip will not install it without --pre.

A note on the version string. Cargo and npm carry 0.6.0-preview verbatim. PEP 440 normalises it to 0.6.0rc0 (preview is one of its spellings of rc), so the wheel’s version differs from the crate’s and the npm package’s on purpose. It sorts before 0.6.0 on all three.

Upgrade notes. These need action, and each is marked Breaking below:

  • Raft control planes need FELIX_RAFT_CLUSTER_ID, FELIX_RAFT_BIND_ADDR and FELIX_RAFT_PEER_TOKEN (or FELIX_RAFT_INSECURE_PEERS=true); FELIX_RAFT_PEERS now lists peer-listener addresses, and a new group needs FELIX_RAFT_INITIAL_CLUSTER_STATE=new. The Helm chart needs a peer-token Secret.
  • Clustered brokers need peer mTLS (FELIX_INTERNAL_TLS_*) or FELIX_INTERNAL_ALLOW_UNAUTHENTICATED=true; the chart needs broker.peerTls.enabled or broker.peerTls.allowUnauthenticated.
  • A broker outside Helm with FELIX_QUIC_LISTENERS unset binds up to four client listeners on consecutive ports. A container that publishes only port 5000 needs FELIX_QUIC_LISTENERS=1.
  • A replication-factor-1 durable shard now waits for its owner instead of failing over empty.
  • Publish scheduling settings (FELIX_BROKER_PUB_*) keep their names but change meaning.
  • Control-plane listings return at most 1000 entries per page.
  • stream.subscribe alone no longer allows redriving or discarding dead letters.
  • New segments are written in storage format v3 and later, which 0.5.0 cannot read.
  • Prefix cache watches on multi-shard caches must name a shard.
  • Several metrics, Rust APIs and crate paths changed; see Changed.
  • A refresh cannot widen its exchange. The exchange’s requested, resources and audience are stored with the refresh chain and re-applied on every refresh; a refresh naming another audience is a 400. Postgres gains a nullable refresh_tokens.narrowing column; older tokens refresh as before. On Raft, roll every member before relying on it.
  • Changing an IdP issuer’s trust takes cluster rights. Changing an existing issuer’s jwks_url/discovery_url, audiences or claim mapping, registering an issuer another tenant already uses, or deleting an issuer needs tenant.manage:cluster:*. A tenant admin can still add an issuer no other tenant uses. (#755, #852) IdP fetches no longer follow redirects and refuse private, link-local and unique-local addresses unless FELIX_CONTROLPLANE_OIDC_ALLOW_PRIVATE_IDP is set.
  • Control-plane authorization can shrink. DELETE .../rbac/policies and .../rbac/groupings remove rules (same scope as adding them); POST /v1/tenants/{t}/refresh-tokens/revoke ends a principal’s refresh chains; /v1/tenants/{t}/signing-keys stages, activates and retires tenant signing keys. On Raft they are refused (409) until every member runs this release.
  • IdP groups are scoped by issuer (group:{issuer}#{name}), so an IdP a tenant admin registers cannot claim another IdP’s groups. Bare group:{name} groupings need migrating; see docs/auth.md (FELIX_CONTROLPLANE_LEGACY_UNSCOPED_GROUPS bridges the migration).
  • The control-plane API refuses broker-audience tokens. Exchange and refresh take "audience": "felix-controlplane" for API callers, including broker node credentials; FELIX_CONTROLPLANE_ACCEPT_BROKER_AUDIENCE bridges the migration.
  • IdP discovery and JWKS URLs must be HTTPS (plain HTTP on loopback, or with FELIX_CONTROLPLANE_OIDC_ALLOW_INSECURE_HTTP), and a discovery document must name the configured issuer.
  • Token exchange resources now narrow grants instead of passing a broader grant through unchanged.
  • Tenant, namespace, stream and cache names are validated at creation; *, /, : and similar are refused.
  • Postgres delete_node and assignment writes lock the node row, so a racing delete can no longer leave an assignment to a deleted node; a node that is still a replica can no longer be deleted.
  • Live tokens no longer appear in Debug output, and migration exports are written mode 0600.
  • Breaking: Raft peers are served on their own authenticated listener. Raft RPCs used to sit on the public API port with no authentication, where one propose could replace the metadata store. They are now served only on FELIX_RAFT_BIND_ADDR, and every request carries the cluster id and FELIX_RAFT_PEER_TOKEN (at least 32 characters; FELIX_RAFT_INSECURE_PEERS=true skips it). FELIX_RAFT_PEERS now lists peer-listener addresses. Optional peer mTLS: FELIX_RAFT_TLS_CERT, FELIX_RAFT_TLS_KEY, FELIX_RAFT_TLS_CA, all three or none. felix-controlplane migrate import authenticates as a peer and must target a peer address. Helm gains controlplane.storage.raft.peerPort (8444), clusterId, peerToken.existingSecret/tokenKey and insecurePeers, and refuses to render without a token Secret. (#727)
  • Breaking: clustered brokers require peer mTLS. A broker with FELIX_NODE_ID set refuses to start without FELIX_INTERNAL_TLS_CERT, FELIX_INTERNAL_TLS_KEY and FELIX_INTERNAL_TLS_CA, unless FELIX_INTERNAL_ALLOW_UNAUTHENTICATED=true. This holds at replication factor 1 too, since writes are forwarded over that port. The Helm chart refuses to render unless broker.peerTls.enabled or broker.peerTls.allowUnauthenticated is set. (#739)
  • Unauthenticated QUIC connections are bounded. A peer could make the broker reserve 16 MiB per stream from a 12-byte header, on up to 2,048 streams, with no deadline. Before a connection authenticates, frames are capped at FELIX_PREAUTH_MAX_FRAME_BYTES (default 64 KiB) and at most FELIX_PREAUTH_MAX_STREAMS_PER_CONN (default 16) streams are read. A connection with no authenticated stream after FELIX_AUTH_TIMEOUT_MS (default 10 s, 0 off) is closed. FELIX_MAX_CLIENT_CONNECTIONS (default 8192) caps connections per broker, and a stalled handshake no longer blocks the accept loop. New metrics felix_quic_connections_refused_total, felix_quic_handshake_failures_total and felix_quic_auth_timeout_total. A client older than this release that leaves an event connection idle for over 10 s before subscribing loses it; raise or disable FELIX_AUTH_TIMEOUT_MS for those. (#736)
  • The broker’s pre-auth JWKS fetch cannot be used to flood the control plane. Implausible tenant ids are refused without a request, unknown tenants are refused locally once the first catalog sync lands, concurrent misses share one fetch, failures are cached for 5 s, and at most 8 fetches run at once with a 5 s timeout. (#736)
  • Tokens can be bound to the client certificate. With FELIX_TLS_CLIENT_CERT_BIND_SUBJECT=true (which needs FELIX_TLS_CLIENT_CA), a client that presents a certificate must use a token whose sub the certificate names: a URI SAN felix:principal:<sub>, or a DNS or IP SAN equal to sub. Control-plane principal ids match only through the URI SAN. The check covers QUIC auth and Kafka SASL/PLAIN over TLS. (#751, #758, #771)
  • The vendored Kafka decoder no longer trusts counts from the wire. A 66-byte produce batch could make it reserve over 3 GB, and a 1 MiB Metadata frame about 72 MiB, before decoding anything. Reservations are now capped by the bytes left in the frame. A record whose offset or timestamp delta overflows the batch base is refused instead of panicking or wrapping. (#762, #768, #821)
  • Breaking: consumer groups have their own authorization actions. group.consume covers poll, ack, nack and listing dead letters; group.manage covers redrive and discard. stream.subscribe still grants group.consume and stream.manage still grants group.manage, but a principal with only stream.subscribe can no longer redrive or discard. Both can be granted on one group, group:{tenant}/{namespace}/{stream}/{group}; where a principal holds a group grant for an action on a stream, only its group grants count there. Upgrade brokers before writing policies that use the new actions, since an older broker refuses tokens that carry them. (#731, #744)
  • felixctl, a command-line tool (#872). pub (from an argument, a file or stdin; keyed, unacknowledged or idempotent), sub (from latest, earliest or an offset, one shard or all), cache get|put|del|watch, topology (shard count, owners and brokers), read-only tenant, namespace, stream, cache ls|info, node and shard listings that follow next_cursor, and bench ingest|latency|fanout|cache. Named contexts in felixctl/config.toml, overridden by new FELIX_* variables (FELIX_CLI_CONFIG, FELIX_CONTEXT, FELIX_BROKERS, FELIX_NAMESPACE, FELIX_AUTH_TOKEN_FILE, FELIX_CONTROLPLANE_TOKEN, FELIX_CONTROLPLANE_TOKEN_FILE, FELIX_CA_FILE, FELIX_CLIENT_CERT_FILE, FELIX_CLIENT_KEY_FILE, FELIX_SERVER_NAME), overridden by flags. --json everywhere, distinct exit statuses, shell completions and man pages.

  • Releases ship felixctl. A tag attaches felixctl-<tag>-<target> archives for Linux x86_64 and aarch64, macOS aarch64 and x86_64, and Windows x86_64, each with a .sha256, completions and man pages; pushes ghcr.io/<owner>/felixctl for amd64 and arm64; and publishes felix-loadgen and felixctl to crates.io after the client. The crates.io job packages every crate before uploading any, and uses trusted publishing when no CARGO_REGISTRY_TOKEN is set. release.yml takes dry_run (build everything, publish nothing) and ref (build a branch as if it were tag, dry runs only), so a release can be rehearsed before its tag exists.

  • felix-loadgen is also a library. felix_loadgen::run runs a scenario and returns its LOADGEN_JSON object; felixctl bench uses it. The binary’s flags and output are unchanged. The crate is now publishable.

  • felix-cluster up creates a cache named users, and its session file names each broker’s certificate (cert_file), so a client can verify the brokers it connects to.

  • felix-loadgen ingest: --in-flight <n>, --duration-secs, --start-at. --in-flight sends acked batches with n outstanding per publisher and reports their ack latency, so a run measures what a client that waits for durability gets and uses the publish window. The other two size a run by time and start every generator at the same instant. The Azure drivers use all three, and gained a shapes step (payload x batch x acked) and, in session C, lease and lease-free passes.

  • felix-loadgen pubsub publishes to the shard’s owner. Single publishes went through the first broker, so latency depended on which broker led the shard and a relayed publish paid an extra round trip to the owner. --via-entry keeps the old path.

  • Publish acknowledgements say where the batch landed. A client that offers the new frame flag FLAG_BINARY_PUBLISH_ACK_OFFSET (0x1000) gets the offset of the batch’s first record on a successful binary ack, and as offset on a JSON publish_ok. The broker sends it only when it answers after the write; a client that did not offer the bit gets byte-identical frames. A duplicate idempotent batch reports the original’s offset, and a forwarded batch reports the owner’s, through a new peer capability FORWARD_OFFSETS. Publisher::publish, publish_batch and friends, ClusterClient::publish and IdempotentProducer::publish now return Result<Option<u64>> instead of Result<()>; the Python and Node clients return the offset too. The history checker now checks every acknowledged append against the offset it was acknowledged at (#876).

  • Placement sees halted replicas (#874). A leader’s replica report names the followers it has stopped shipping to, with the reason (diverged or needs_bootstrap), as an optional halted field that is omitted when empty. The control plane stores them with the report, keeping when each halt began: Postgres gains replica_reports.halted (migration 0023), and on Raft the field is metadata level 4, dropped until every member is upgraded. Placement never picks a halted copy as a move, replacement or restore destination, gives up a move whose destination halts (step halted) and places it elsewhere, counts a halted copy as missing, and replaces one halted for FELIX_SHARD_RESTORE_AFTER_MS. An operator’s move to one is refused with destination_halted, and a drain with nowhere else to go says which copy is halted. GET /v1/placement/replication and felix-controlplane admin replication list halted copies, and felix_shard_replicas_halted counts them.

  • Placement restores a shard’s replication factor. A follower whose broker has been down or gone for FELIX_SHARD_RESTORE_AFTER_MS (shard_restore_after_ms, five minutes by default, 0 to turn it off) is replaced by a copy on a live broker. A replica set a failover left short of its factor is topped up once a broker is free, which covers a cache or Leader stream whose failover dropped the dead leader. The copy joins and is seated the way a drain’s replacement is, so a Quorum stream keeps the old set’s majority rule. The restore is stored in the assignment (move reason restore) and resumes across control-plane restarts, and a destination that dies mid-copy is replaced. A short set grows before a lost follower in it is replaced, nothing starts until the leader has reported at its generation, and a failover keeps a copy that was joining a set of two, since the leader counted it. New: felix_shards_under_replicated, felix_shard_replicas_missing, GET /v1/placement/replication and felix-controlplane admin replication. The move step Seat now names the follower it replaces as optional, and MoveStep::Restore and MoveReason::Restore are new. A control plane that predates restore cannot read an assignment with that move reason, so upgrade every control-plane instance before relying on it.

  • Adversarial history campaign and liveness checks. The history checker’s nemesis gains compound faults on four brokers: two brokers killed at once, a leader isolated from its peers and the control plane until it fails over, a partition beside a delayed link, two random faults together, a move whose source or destination is killed, restart loops, torn segment writes, and drains that replace follower copies. After every heal the campaign now waits for each shard to serve again and fails naming any shard that is stuck and why. FELIX_HISTORY_NEMESIS picks the nemesis, and the nightly workflow runs both.

  • Power loss, control-plane crashes and delivery checks in the history campaign. The adversarial nemesis can cut power to every broker at once, keeping only what each one flushed, and crash the control plane while a move, drain or failover is in flight. Live subscribers now follow each list for the whole run, and three new rules check that delivery follows offset order, that nothing delivered is later lost, and that every record is either delivered or skipped visibly. Debug and fault-injection brokers take the power loss from FELIX_STORAGE_POWER_LOSS_ROOT and the storage fault file, on Linux. The harness keeps broker storage in storage/ under each node’s data directory.

  • Atomic commits on one shard. Client::commit(tenant, namespace, entity_key, [publish | enqueue, put, delete]) writes an event and state updates as one record on the stream shard entity_key routes to, and state_get reads that state with the commit’s offset as its version. No reader sees part of a commit, across failover and promotion. Operations on another stream are refused with CommitError::NotOnOwningShard, never split. New wire requests commit/state_get behind FEATURE_ATOMIC_COMMIT; storage format v5 (a segment is written at v5 only to hold a commit record); replication mark byte 4; fleet feature atomic_commit, which a cluster must finalize before commits are accepted. See docs/atomic-commit.md.

  • Jump-hash stream routing once the jump_hash_routing fleet feature is finalized. A stream’s key-to-shard mapping (routing: modulo or jump_hash) is fixed at creation; streams created after the finalize get jump consistent hashing, and existing streams keep modulo, so no key moves. Postgres gains a nullable streams.routing column (migration 0022). Shard assignments and stream_shards_view carry routing only for jump-hash streams, the broker resolves keyed publishes by it, and ClusterClient routes by it. Client::stream_routing is new; felix_router::Placed gains a routing field.

  • Readers and consumer groups on Quorum shards without the lease once lease_free_reads is finalized (#885). Subscriptions, replay, Kafka fetches and cache watches on a replicated Quorum shard keep going when the lease lapses, since they see only the committed mark. They end when a follower refuses the leader for a newer one, or when no majority has confirmed it for a lease duration (a round runs every quarter lease while the lease is lapsed). Group polls, acks, nacks and dead-letter changes skip the lease and are confirmed by a read-index round after they are durable and before they are acknowledged; a refused round answers leadership_lost. Kafka produce now takes its lease check from the shard fence, like a native publish, so a majority_ack shard takes Kafka writes without the lease too. BrokerCluster::with_writes loses its lease argument. TLA+: FelixShardSessions.tla, with Subscriber and GroupRound passing NoLostDelivery and NoStaleGroupCommit, and PastMark, GroupNoRound and GroupLease violating them. See docs/replication-design.md, “Readers and group sessions without the lease”.

  • Quorum cache reads without the lease once the lease_free_reads fleet feature is finalized (with majority_ack and generation_start). A get or counter get on a replicated Quorum cache, local or forwarded, takes its value and then confirms leadership read-index style: the promotion fence at the broker’s own generation goes to every replica, and the read is answered once a majority, the broker included, has taken it. Concurrent reads share rounds, only ever one that started after them. A cache replica also refuses the round when its counter log accepted a newer leader. A same-generation fence is now counted as fence_confirmed and not logged. New FELIX_QUORUM_READS (majority or lease) keeps a broker on the lease. No wire change. TLA+: FelixShardReads.tla, with FelixShardReadsRound passing NoStaleRead under drift and no margin and FelixShardReadsNoRound / FelixShardReadsLease violating it; check_tla.sh takes TLC_WORKERS. See docs/replication-design.md, “Reads without the lease”.

  • Quorum streams acknowledge by their followers once the majority_ack fleet feature is finalized. A write is acknowledged when a majority of the replica set, the leader included, has answered ReplicateOk for it at the leader’s generation; the control-plane report goes out behind the mark for placement only, and the lease is off the write’s path (admission, commit and acknowledgement). A leader counts itself only while it has accepted no newer generation, and a promoted leader never opens a stream shard on the lease. Needs generation_start finalized too. A broker running with FELIX_INTERNAL_FENCE=false does not report the feature. Leader streams and cache writes keep the lease; lease_free_reads takes it off reads and group operations. TLA+: HeldAtGen counts follower answers (confirmed), and FelixShardFigure8FollowerAcks / FelixShardFigure8FollowerAcksNoStartRecord join check_tla.sh; FelixShardFencedAck now includes the start record. See docs/replication-design.md, “Acknowledging by the followers”.

  • scripts/check_tla.sh takes configuration names to check only those.

  • Fleet features: cross-broker behaviour turns on only when an operator finalizes it. A broker reports the features it implements at registration (features, optional), and the control plane tracks the set every live or draining broker supports. Support enables nothing: POST /v1/fleet/features/{feature}/finalize (or felix-controlplane admin features finalize <feature> [--dry-run]) enables one, refused until every serving broker supports it. Registration and every heartbeat answer with the enabled set as fleet_features; GET /v1/fleet/features and admin features show supported and enabled. Brokers check it through felix_common::fleet::FleetGate and export felix_broker_fleet_feature_enabled. Before a finalize any broker can be rolled back; after it, finalizing is one-way and a broker without the feature is refused at registration (409). generation_start, majority_ack, lease_free_reads, jump_hash_routing and atomic_commit use it. Postgres gains nodes.features and a fleet_features table; on Raft the control plane keeps features and accepts a finalize only once every member is at metadata version 2.

  • Operator moves report their zone impact. POST /v1/shard-moves answers zones_before and zones_after when brokers report zones, takes "dry_run": true to decide a move without starting it, and logs a move that narrows a shard’s spread as a warning instead of refusing it. felix-controlplane admin move gains --dry-run and prints the zones.

  • The Helm chart runs brokers per zone. broker.zones renders one broker StatefulSet per zone, pinned to its nodes and setting FELIX_NODE_ZONE, with one disruption budget across all of them.

  • Pipelined publishes. A client that offers FEATURE_PUBLISH_PIPELINE is granted a per-connection publish_window in auth_ok (FELIX_BROKER_PUBLISH_WINDOW, default 256, 0 to turn off). The broker answers that client’s acked publishes on each stream in request order and stops reading its publishes while the window is full. The Rust client offers it and caps each connection at the window, and IdempotentProducer::publish_batches keeps up to 64 batches in flight, re-sending from the first unanswered one after a failure. Older clients see no change.

  • Connection-fault conformance scenarios. A catalogue scenario can carry a step that drops, resets or stalls the client’s link mid-publish or mid-subscribe. felix_conformance::link is the UDP interposer that does it; felix-cluster client-fixture serves one at link_addr, breaks it on POST /link, and lists the steps under faults. The protocol suite runs them with the Rust client, and the Python and TypeScript suites run them in CI. A subscription must resume with no gap or raise; ending as if the stream had finished fails.

  • Placement spreads a shard’s copies across zones. A broker registers the failure domain it is in with FELIX_NODE_ZONE (the node’s new optional zone). Followers go to zones the shard has no copy in, drains and rebalances never narrow a shard’s zones when they can avoid it, and a follower sharing a zone is replaced by one in a missing zone once a broker there has room. Where the copies cannot be spread they are placed anyway, with a warning and felix_shards_zone_unspread. A broker without a zone shares one with nobody, so clusters that report none are placed as before. Postgres gains a nullable nodes.zone column.

  • A promoted leader fences a majority before it serves. A broker promoted to lead a stream shard now waits in a fencing phase: it sends Fence to every replica, opens once a majority (itself included) has persisted its generation, and first takes the log of the replica furthest ahead by (last generation, length) with the new ReplicateFetch (kind 31, capability TAIL_FETCH). A deposed leader’s ships are then refused by that majority even where the control plane never reached the follower. Applies only when every replica offers both capabilities; otherwise the shard opens on the lease as before, except that under majority_ack a stream shard never opens on the lease. felix_broker_promotions_opened_total{path} says which. A replica that takes Fence persists the leader’s generation before it answers with its log end, commit offset and last record’s generation, and refuses every older leader from then on. Brokers offer capability bits in the peer handshake (HelloCapable, kinds 27 and 28), and a peer that did not offer FENCE is never sent one; FELIX_INTERNAL_FENCE=false withdraws the offer. Until majority_ack is finalized, acknowledgements still go by the report and the lease. Each peer connection opens one more stream, for the handshake and the fence’s requests only, so they never wait behind a forwarded publish. (#794, #798)

  • Clients can offer the felix/1 ALPN. felix_client::quic_client_config builds the QUIC TLS config and offers felix/1 when asked; the Python (offer_alpn=True) and TypeScript (offerAlpn) clients expose the same switch. Off by default, since a broker older than ALPN support refuses a client that offers one. Before this no shipped client could connect to a broker with FELIX_TLS_REQUIRE_ALPN=true.

  • The licence split is checked on the dependency graph. scripts/check_license_graph.py, run by task publish:check, fails if an Apache-2.0 or MIT first-party crate has a normal or build dependency, direct or transitive, on an AGPL one.

  • Advisories reach the vendored kafka-protocol. A patched-in path crate has no registry source, so cargo-deny skipped it. vendor/VENDORED.toml records the upstream version (0.18.0) and task deny checks advisories against it, failing if the record and the vendored copy disagree.

  • Kafka producers can write to durable streams. Produce (v3-9) on the Kafka listener decodes v2 record batches, compressed with gzip, snappy, lz4 or zstd, and publishes them through the broker’s publish path on the shard’s leader, with stream.publish checked as for a QUIC publish; any other broker answers NOT_LEADER_OR_FOLLOWER. acks=all waits for the stream’s own consistency: a majority on a Quorum stream, the leader on a Leader one. InitProducerId gives idempotent producers a Felix producer id, and the new Broker::publish_records_idempotent stores each record as a one-record producer batch so Kafka’s per-record sequences are the log’s own: a batch re-sent after a failover or a move is answered with its original offset. Transactions are refused with TRANSACTIONAL_ID_AUTHORIZATION_FAILED and a message; legacy v0/v1 message sets with UNSUPPORTED_FOR_MESSAGE_FORMAT. Keys and headers are dropped and counted; producer timestamps are dropped (a record’s timestamp is its append time). New metrics: felix_kafka_produce_records_total, felix_kafka_produce_bytes_total, felix_kafka_produce_duplicate_records_total, felix_kafka_produce_errors_total and felix_kafka_produce_dropped_total. FELIX_KAFKA_ANONYMOUS_TENANT now allows writes as well as reads. The docs-site page “Reading with Kafka Clients” is now “Kafka Compatibility”.

  • Kafka consumers can read durable streams. A broker started with FELIX_KAFKA_LISTEN serves the Kafka protocol for consumers that assign their own partitions: kcat, librdkafka programs and Java KafkaConsumers. A durable stream is topic <namespace>.<stream>, partition N is shard N, and offsets are Felix’s log offsets. It speaks ApiVersions, SASL/PLAIN (tenant id as username, Felix token as password, reads checked as stream.subscribe), Metadata, ListOffsets and a long-polling Fetch, and a consumer follows a partition to its new leader after a move or failover. FELIX_KAFKA_ADVERTISE_ADDR, FELIX_KAFKA_TLS (on by default, with the broker’s certificate), FELIX_KAFKA_ANONYMOUS_TENANT, FELIX_KAFKA_DEFAULT_NAMESPACE and FELIX_KAFKA_MAX_CONNECTIONS configure it. Each broker registers its advertised address as the node’s new kafka_addr field so any broker can name any partition’s leader; migration 0018_node_kafka_addr.sql adds the column. Consumer groups are refused: FindCoordinator answers GROUP_AUTHORIZATION_FAILED with a message saying to assign partitions, so a group consumer fails with a reason instead of hanging. Metrics are under felix_kafka_*. docs/kafka-compatibility.md replaces the spike write-up, and the spike crate under spikes/ is removed.

  • Streams can be kept in a region. A stream created with "region" has its leader and every replica only on brokers in that region, or in one the new directional allowlist FELIX_REGION_BRIDGES (source>dest pairs) bridges it to: on first placement, rebalancing and drain moves, follower replacement and failover. A copy found outside, after a broker’s region changes or a bridge is removed, is moved back in; with no broker in an allowed region the shard is left unplaced rather than placed elsewhere. An operator move out of the region is refused with region_not_allowed. Brokers read the same FELIX_REGION_BRIDGES and forward only to leaders in bridged regions, refusing the rest as shard_unavailable with the new reason region_not_routable instead of owner_unavailable. Streams without a region, and every cache, are placed as before. Migration 0017_stream_region.sql adds a nullable streams.region column.

  • Consumer-group calls follow the shard’s leader. ClusterClient gains group_poll, group_poll_wait, group_ack, group_nack, group_dead_letters, group_discard and group_redrive, which follow the broker’s NotLeader redirect to the shard’s leader and remember it. The Python and TypeScript clients’ group calls now use them, so they no longer fail with ShardUnavailableError (not_leader) when connected to a broker that does not lead the shard, or after a shard move. Client’s group calls still return NotLeaderError rather than follow it.

  • A broker hands its shards off before it stops. On SIGTERM a clustered broker turns readiness off, drains itself through the control plane and keeps serving until it leads no shard, then shuts down as before, so a rolling restart moves shards instead of failing them over: no publish is refused and subscriptions follow the shard. Bounded by FELIX_SHUTDOWN_HANDOFF_TIMEOUT_MS (default 30 s, 0 turns it off); skipped when there is nobody to hand to or the control plane does not answer, and ended early by a second signal. Metrics: felix_broker_shutdown_handoffs_total{outcome}, felix_broker_shutdown_handoff_shards_total, felix_broker_shutdown_handoff_duration_ms. The Helm chart adds broker.shutdown.handoffTimeoutMs and counts it in the derived grace period; a grace period set by hand must now cover it too.

  • Cache, counter and consumer-group operations are no longer refused during a shard move. A cache put, delete or read and a counter add or read that arrives between a move’s fence and its cut-over is held and sent to the new owner, as a publish already was, on the broker it reached and on an owner a forward reached. A consumer-group poll, ack, nack or dead-letter change is held and then answered with NotLeader naming the new owner, which ClusterClient::group_sharded follows; group operations are still served only by the shard’s leader. A group poll waiting for records when its shard moves away now answers with no records instead of an error. Bounded by the same FELIX_SHARD_MOVE_HOLD_MS and FELIX_SHARD_MOVE_HOLD_MAX.

  • routable in the node listing. GET /v1/nodes now says whether other brokers may send a node requests for the shards it leads: live or draining, heartbeat inside the window. Brokers forward on it instead of on eligible, so writes through another broker to a draining broker’s shards are no longer refused before each shard’s turn to move.

  • Idempotent producers keep their sequences across a leader change (#608). On a durable stream each record of a producer’s batch is now stored with its producer id and sequence, and replicated with them, so a leader promoted after a failover, a planned move’s destination and a restarted broker answer a re-send of the batch in flight from the records they hold, and the producer carries on instead of being refused as unknown_producer. A batch whose leader stopped partway through it is finished by the re-send rather than written twice. A producer is remembered while any of its batches is in the log, so retention now decides when one is forgotten, on every replica alike. In-memory streams keep sequences in the leader’s memory as before. Opening a shard reads no more than before: the producer state is saved at each rollover and the rest comes from the active segment recovery already scans. New metric felix_storage_producer_state_rebuilt_total counts opens that had to read sealed segments to rebuild it.

  • Subscriptions follow a moved shard. A broker that stops serving a shard now ends each of its subscriptions and cache watches with a final shard_moved frame: the offset to resume from, the broker taking the shard when known, and the generation that moved it. Only a client that offers FEATURE_SHARD_MOVED (0x1000) gets it; any other sees the stream end byte for byte as before. For a durable stream resume_from is exact, so resuming at max(last delivered + 1, resume_from) neither repeats nor skips a record. A ClusterClient subscription does that itself: it resubscribes on the new owner, retrying until the move has cut over, and an in-memory stream resumes at the new owner’s tail. A Client subscription ends with Subscription::shard_moved() set, and cache watches deliver CacheWatchItem::ShardMoved. Sharded subscriptions and watches report ShardEvent::ShardMoved and ShardedCacheWatchItem::ShardMoved. Python and TypeScript subscriptions follow too, and surface the move as ShardMoved / CacheWatchShardMoved (Python) and shardMoved (Node). See “Shard moves” in docs/protocol.md.

  • Publishes are not refused while their shard moves. A publish that reaches a broker between a move’s fence and its cut-over is held, before it is accepted, until the broker’s routes show the new owner, and is then sent there. It is refused only if the move takes longer than FELIX_SHARD_MOVE_HOLD_MS (default 2000) or FELIX_SHARD_MOVE_HOLD_MAX (default 1024) publishes are already waiting, with shard_unavailable and the new reason moving (retry class retry, with a retry_after_ms hint). New metrics felix_broker_shard_move_held_total, felix_broker_shard_move_hold_seconds and felix_broker_shard_move_hold_refused_total{reason}.

  • A move’s destination does not count toward the quorum while it copies. A Quorum publish waits for a majority of the replicas the stream asked for, not for the copy, and the copy is shipped in slices so it never holds a replication pass for long.

  • Shard moves are paced. Every copy holds a move slot, including a follower being replaced on a draining broker, which is now copied in beside the follower it replaces (named joining) and seated once it has caught up, so a shard never drops below its replication factor while it fills. FELIX_SHARD_MOVES_MAX_PER_NODE bounds copies into or out of one broker. Drains get slots before rebalancing. A move fences once its destination is within FELIX_SHARD_MOVE_FENCE_MAX_LAG_RECORDS (default 1000) of the leader’s tail rather than exactly level, which under steady writes it may never be; replica reports carry the leader’s tail (leader_offset) for this. A move that has not reached its fence within FELIX_SHARD_MOVE_TIMEOUT_MS (default 30 min) is abandoned as a timed_out step, counted in felix_shard_moves_timed_out_total, and waits behind other shards for its next slot; a fenced move is always finished. On the broker, FELIX_SHARD_MOVE_BYTES_PER_SEC limits what a leader ships to move destinations that the quorum does not need (felix_broker_replication_move_throttled_bytes_total). Migration 0014_shard_move_pacing adds the assignment and report columns.

  • Operators can steer shard moves. GET /v1/shard-moves lists the moves in progress with their step, the reason they started (drain, balance, operator, replace, stored as move_reason on the assignment), start time and lag; GET /v1/placement/plan shows what the next placement pass would do without doing it. POST /v1/shard-moves starts a move to a named broker, held to the move limits. DELETE /v1/shard-moves/{tenant_id}/{namespace}/{name}/{shard} cancels one: before the fence the destination is dropped; after it the fenced leader serves again at a new generation (retake) with every write it accepted, and held publishes and followed subscriptions find it again; after the cut-over it is a 409. POST /v1/placement/pause and /resume stop and restart placement’s own moves on every instance; moves in flight finish, and new shards and failovers are still placed. Reads take node.view:cluster:*, changes node.manage:cluster:*. felix-controlplane admin does the same from a shell (moves, plan, move, cancel, pause, resume, with --json). Migration 0015_operator_moves adds move_reason and the placement_settings table; the Raft backend gains a set_moves_paused command, which an older member refuses. felix_shard_move_steps_total gains the cancel and retake steps.

  • Faster shard move switch-over, control-plane side. GET /v1/shard-assignments/changes takes an optional wait_ms: with nothing newer than since, the request waits up to that long (capped at 25 s) and answers as soon as a change lands. It holds no store connection while waiting, and without wait_ms it behaves exactly as before. A replica report that a move is waiting for (a drained leader, or a caught-up staged successor) now wakes placement at once instead of at its next tick; wakes coalesce and one pass runs at a time. New histograms felix_shard_move_duration_seconds and felix_shard_move_fence_seconds time moves from the steps each control-plane instance writes.

  • Typed error codes on the client wire. A client that offers FEATURE_ERROR_CODES (0x0800) gets a code, a retry class and an optional detail on every error and publish_error, and on a failed binary publish ack when it also offers FLAG_BINARY_PUBLISH_ACK_CODE (0x0200). The retry class says whether the request may have been applied: a quorum timeout is now quorum_timeout with outcome_unknown, distinct from a refusal, and shard_unavailable names its reason. An unknown code decodes with its retry class instead of failing the frame. Clients that do not offer the bit get byte-identical frames. A broker that is draining now answers auth on a new control stream with draining for clients that offered the bit. The Rust client offers it and exposes the code on felix_client::BrokerError. See “Error codes” in docs/protocol.md.

  • The Rust ClusterClient acts on the broker’s error codes. Retry, reroute or fail is decided by the retry class instead of by matching the message. A cached shard owner that answers shard_unavailable (including fenced), draining or not_leader is forgotten and the publish goes to the entry broker at once, from publish too, since nothing was applied. outcome_unknown is never re-sent by publish; publish_at_least_once and the idempotent producer re-send it. retry_after honours retry_after_ms, and not_found is retried for 5 s from the first one rather than for the whole attempt budget. A fatal code such as invalid_request is no longer retried. publish reconnects only on draining or a dead connection, not on every coded refusal. A subscribe or cache-watch refusal is now a typed BrokerError (it was the debug text of the frame), and a subscribe, cache watch or group request whose redirect target answers shard_unavailable goes back to the entry broker once. Peers that did not negotiate codes get the old handling.

  • Python and TypeScript errors carry the broker’s error code. Both have code, retry and detail. The class is chosen from the code when the broker sent one, and from the message only when it did not. New classes: ShardUnavailableError (shard_unavailable, not_leader; retryable), OverloadedError, and OutcomeUnknownError (quorum_timeout, leadership_lost, unacknowledged, or anything sent as outcome_unknown); a draining broker is a ConnectionError. In TypeScript, retryable follows the broker’s retry class when there is one, so a NotFoundError sent as retry_after is now retryable. Two conformance scenarios, error.shard_unavailable_is_retryable and error.quorum_timeout_is_outcome_unknown, hold both bindings and the Rust client to this, against faults felix-cluster client-fixture now serves on a localhost control endpoint (POST /fence, /partition, /heal).

  • Online shard rebalancing (#130). A shard whose leader is alive is now moved instead of reassigned. The control plane stages the destination as a replica and lets the leader catch it up. Once the copy is close to the leader’s tail it fences the leader (the assignment goes draining, and a broker never serves a draining assignment), waits for the leader’s drained report, and only then names the destination leader at a new generation. Each step is an assignment write, so any control-plane instance resumes a half-done move from the store.

    Two triggers. A drained broker (POST /v1/nodes/{id}/drain) hands off everything it leads and is replaced as a follower wherever it holds a copy, so it can then be deleted. A broker leading more than its share of shards gives them to one under its share, which is what makes a broker that joins a running cluster take work. A cluster where a control-plane restart left every shard on the first broker to register now corrects itself. Moves only go from over share to under share and count moves in flight as done, so placement converges instead of oscillating. FELIX_SHARD_MOVES_MAX_CONCURRENT bounds moves in flight, one by default.

    A destination that dies before it leads is passed over; a leader that dies mid-move is an ordinary failover with the destination as a candidate. Writes that arrive between the fence and the new owner opening are held and sent on (see “Publishes are not refused while their shard moves”). Real-process tests cover drain, join, a destination dying mid-transfer, a drain and a join at once, and publishes arriving throughout.

    The fence is model-checked. docs/formal/FelixShardHandoff.cfg adds the planned move to the TLA+ spec and explores 2.7M states without a violation; FelixShardHandoffNoWait.cfg, which cuts over as soon as the fence is written, finds two brokers serving one shard in seven steps.

    The assignment carries an optional successor, the replica report an optional drained; both are omitted when unset, so nothing changes on the wire for a cluster that never moves a shard. Postgres gains migration 0012_shard_moves. felix_shard_move_steps_total{step} and felix_shard_moves_waiting report progress. The harness gained add_node, drain_node, undrain_node and drain_until_empty.

  • A broker can bind several client listeners. FELIX_QUIC_LISTENERS binds that many QUIC listeners on consecutive ports from the FELIX_QUIC_BIND port, each with its own socket and accept loop. A broker with more than one names their ports in auth_ok (listener_ports, omitted otherwise), and the client spreads its publish, cache and event pools over them, always keeping the host it dialled. The Helm chart gains broker.ports.listeners (default 1) and refuses to render if broker.ports.internal falls inside the range. See Changed for the default count. (#595)

  • ClusterClient sends publishes straight to the shard’s owner. It learns each shard’s owner from the publish ack and sends later batches there, keyed or not, instead of having the entry broker forward every one. It falls back to forwarding when the owner is unknown or unreachable. felix_client::publishes_forwarded() counts acks that came back forwarded. (#600)

  • PATCH and DELETE /v1/nodes/{id}. Both were documented but answered 405. PATCH changes a node’s region, labels, capacity, or lifecycle between live and draining (which cancels a drain); it cannot mark a down node live. DELETE needs node.manage on cluster:* and is refused with 409 while the node is live or draining or any shard still names it. (#644)

  • FELIX_SHUTDOWN_PREDRAIN_MS. A stopping broker keeps accepting connections for this long after /ready turns false, so a load balancer polling /ready notices first (default 0; a second SIGTERM ends the wait). In Kubernetes the chart’s preStop sleep already does this. (#645)

  • A cache can be created with Quorum consistency. Cache and CacheCreateRequest gain consistency (Leader or Quorum, default Leader), set at creation only. On a Quorum cache a put, delete or counter update is acknowledged once a majority of the shard’s replicas hold it, and survives the loss of its leader. Postgres migration 0013_cache_consistency.sql. felix_broker::CacheMetadata gains a consistency field. (#648, #750)

  • One consumer group across every shard of a stream. ClusterClient::group_sharded returns a ShardedGroup that finds each shard’s leader by following redirects, polls the shards in turn, and sends acks and nacks to the right leader. Ordering holds per shard. A broker now answers group requests for a shard it does not lead with NotLeader to clients that offer FEATURE_REDIRECT; a client that offered the bit but predates this sees an unexpected-answer error instead of the old refusal. (#649)

  • The Rust client refreshes its token for streams opened after connect. ClientConfig::token_provider is asked for a token whenever a stream authenticates, and overrides auth_token. RefreshingToken wraps a fetch function and refreshes once two thirds of the token’s life has passed. Before, subscriptions, watches, group requests and reconnects failed once the connect-time token expired. (#652)

  • Watch a cache prefix across every shard. ClusterClient::watch_cache_sharded and watch_cache_sharded_retained open one prefix watch per shard on its owner and merge them. A retained watch emits StateComplete once every shard has delivered its retained values, and resume_offsets() gives one offset per shard. New request cache_shards, feature bit FEATURE_CACHE_SHARDS (0x0400) and Client::cache_shards. Rust only for now. (#654)

  • A resumed subscription reports where it joined. A subscribe with a start position on a durable stream answers subscribed with start_offset (the first record delivered) and live_offset (the tail when it was registered), to clients that negotiated FLAG_EVENT_BATCH_OFFSETS only. Subscription::start_offset() and live_offset() expose them. (#655)

  • Idempotent publishes use the binary frame. The negotiated flag FLAG_BINARY_PUBLISH_IDEMPOTENT (0x0100) carries a producer id and sequence on an acked binary publish. Publisher::publish_idempotent_batch uses it when the broker offers it and falls back to JSON otherwise. (#656)

  • A different batch under a reused producer sequence is refused. A leader answered any batch under a sequence it already held as a duplicate, so a producer that reused a sequence got ok for data never written. Each held batch now keeps a digest of its payloads, and a mismatch is refused with reason sequence_reused to clients that offer FEATURE_SEQUENCE_REUSED (0x4000); older clients still get the duplicate answer. The Rust IdempotentProducer offers it. (#764)

  • Brokers negotiate the felix/1 ALPN. Client listeners select felix/1 when offered and refuse clients that offer only other protocols. Clients that offer none are accepted unless FELIX_TLS_REQUIRE_ALPN=true. (#751)

  • Unknown requests get an answer instead of a closed stream. A client that offers FEATURE_UNSUPPORTED gets unsupported for a request type the broker does not know, and the stream stays open. (#751)

  • Brokers can serve a real certificate and require client certificates. FELIX_TLS_CERT/FELIX_TLS_KEY serve a certificate from files, re-read every 30 s. FELIX_TLS_CLIENT_CA makes the QUIC and Kafka listeners require client certificates. Without a certificate the broker still generates a self-signed one and warns; FELIX_TLS_REQUIRE_CERT=true refuses to start instead. Helm: broker.clientTls.*. The Python and TypeScript clients cannot present client certificates yet. (#739)

  • The control-plane API can serve TLS. FELIX_CONTROLPLANE_TLS_CERT and FELIX_CONTROLPLANE_TLS_KEY (or tls: in the config file) serve the REST API over TLS, reloaded on rotation. Brokers and felix-controlplane admin trust a private CA from FELIX_CONTROLPLANE_CA. Helm: controlplane.tls.*. (#739)

  • Per-IP connection caps and Kafka pre-auth limits. FELIX_MAX_CONNECTIONS_PER_IP (default 512, 0 unlimited) caps QUIC connections from one source address per broker, and FELIX_KAFKA_MAX_CONNECTIONS_PER_IP (default 128) does the same on the Kafka listener, where requests before SASL are capped at 64 KiB and SASL must finish within FELIX_KAFKA_AUTH_TIMEOUT_MS (default 10 s). Clients behind one NAT or proxy address may need a higher cap. (#749)

  • Per-tenant publish quotas. FELIX_TENANT_PUBLISH_BYTES_PER_SEC and FELIX_TENANT_PUBLISH_MSGS_PER_SEC (default 0, unlimited) set a per-tenant token bucket on each broker, with FELIX_TENANT_PUBLISH_BURST_MS (default 1000) and per-tenant overrides in FELIX_TENANT_PUBLISH_QUOTAS (tenant:bytes:msgs,...). An acked publish over quota is refused as overloaded with reason tenant_quota and retry_after_ms; Kafka produce is throttled with throttle_time_ms. (#749)

  • Per-tenant metrics. felix_tenant_published_{messages,bytes}_total and felix_tenant_delivered_{messages,bytes}_total. Tenants past the first FELIX_TENANT_METRICS_MAX (default 100) share the label _overflow. (#749)

  • Control-plane listings are paged. Tenants, namespaces, streams, caches, nodes, shard assignments and the RBAC listings take limit (1-10000, default 1000) and cursor, and carry next_cursor when more remain. Breaking: a caller that ignores next_cursor now sees only the first 1000. The RBAC listings still return a bare array when called with neither parameter. (#767)

  • Consistent backups across brokers. felix-controlplane admin backup-point <name> records one committed offset per shard log from each leader after a single barrier, and writes a JSON manifest. Brokers serve the offsets at GET /backup/offsets on the metrics listener. After copying the leaders’ shard directories, felix-broker restore-point --point FILE cuts every copied log back to the point, offline. (#799)

  • Giving up a durable shard’s records is an explicit step. POST /v1/placement/abandon/{tenant_id}/{namespace}/{name}/{shard} (?kind=cache for a cache) or felix-controlplane admin abandon lets placement move a shard it is holding for a dead owner (see Changed). It needs node.manage:cluster:* and is refused with 409 unless placement is holding the shard. (#728)

  • The Helm chart forms a Raft group once, and can encrypt the peer port. A post-install hook creates the ConfigMap <fullname>-controlplane-raft-formed once the API is ready; members start with FELIX_RAFT_INITIAL_CLUSTER_STATE=new only while it is absent, so a member that later loses its volume waits instead of starting an empty group. bootstrapTimeoutSeconds (default 270) bounds the hook; keep helm --timeout above it. controlplane.storage.raft.tls.* turns on peer mTLS. A new NetworkPolicy (controlplane.networkPolicy.enabled, on by default under Raft) admits the peer port only from members and raftPeerFrom; add a migration tool’s pods there. (#761)

  • Durable publishes queued on one shard are written as one append. When an executor takes a durable publish, it also takes the plain publishes queued behind it on the same shard (up to 64, or 1 MiB) and claims them together: one write, one flush wait and one fanout, with each publish still answered with its own offset, in order. The fence is checked per publish; a failed append or flush fails every publish in the claim. felix_broker_publish_claim_jobs reports the group size. Group commit waiters now also watch the durable bound, so one flush wakes all of them at once instead of one at a time through the flush lock.

  • A lone subscriber event is sent without waiting for a batch. A subscription batch takes what is already queued and flushes at once; it waits up to FELIX_EVENT_BATCH_MAX_DELAY_US for more only after the previous batch found events queued behind its first. With one message in flight, delivery no longer sits out the batch delay. (part of #926)

  • Breaking: the broker’s default listener count follows its cores. With FELIX_QUIC_LISTENERS unset, a broker binds max(1, min(cores / 2, 4)) client listeners, using the cores it may run on (cgroup limits included). The count is shortened so the range stops before FELIX_INTERNAL_BIND, so a cluster member on the default ports keeps one. An explicit value still wins. A container that publishes only port 5000, or remaps it, needs FELIX_QUIC_LISTENERS=1, because clients dial the advertised ports. The Helm chart still sets the count (default 1), since it must list the ports. (#720)

  • Binary publish payloads are no longer copied on decode. felix_wire::binary::PublishBatch.payloads is now Vec<Bytes>, each record a slice of the frame it arrived in, which drops one copy and one allocation per record on the broker’s ingest path. A retained record keeps its whole frame alive. The broker reads frame bodies with quinn’s read_chunks, taking up to 32 chunks per call instead of one; bytes read past a frame wait in the new serving::quic::FrameScratch, which replaces the BytesMut argument of read_frame_limited_into and read_message_limited and must stay with one stream. The encoders still take &[Vec<u8>].

  • aws-lc-rs is the only TLS crypto provider. ring is gone from the build: quinn, rcgen and reqwest now use aws-lc-rs, which rustls and jsonwebtoken already did. The broker and control plane install it as the process-wide rustls default at startup. reqwest moves to 0.13, so HTTPS calls to the control plane and to IdPs are verified against the system trust store (plus FELIX_CONTROLPLANE_CA) instead of a bundled copy of the Mozilla roots; the container images already ship ca-certificates.

  • Replication and the broker-to-broker transport are their own crate, felix-replication. felix_broker_service::peer is now felix_replication::peer, and felix_broker_service::replication::* is at the root of felix_replication. ShardKey and ShardKind are defined there and still re-exported from felix_broker_service::shards. No behaviour change.

  • ClusterClient holds one connection per broker. Every role a broker plays (entry, shard owner, redirect target, producer leader) shares one client, and every stream to it is multiplexed on one QUIC connection, where each role used to open a full Client of 20 connections. More open only when the streams on it saturate, up to cluster_conn_pool (FELIX_CLUSTER_CONN_POOL, default 8); cluster_streams_per_conn (FELIX_CLUSTER_STREAMS_PER_CONN, default 1024) sets the stream budget. ClusterClient::connections_per_node and Client::connection_count report the counts. Client::listeners_in_use now returns a Vec.

  • felix-conformance is AGPL-3.0-only. It links the broker, storage and authz crates to run its suite, so a build of it was AGPL whatever its label said. It is not published; running verify over a client’s results puts no obligation on that client.

  • OTLP tracing export is off unless an endpoint is set. The broker and the control plane used to install the OpenTelemetry layer unconditionally, so a deployment with no collector exported to localhost:4317 and logged a failed export for every batch. Set OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) to turn it on.

  • IdP group claims are always prefixed. A group claim value X now maps to the RBAC subject group:X even when X already starts with group:, so an IdP group named group:operators is no longer treated as the operators group. Where an IdP was set up to emit pre-prefixed values, rename the groupings (group:group:<name>) or emit the bare name.

  • Token exchange and refresh refusals are counted. Both now land in felix_controlplane_auth_rejected_total; unusable refresh tokens use the new refresh_refused reason. The refresh replay, bad-secret, revoked and issued counters are now in the control plane’s metrics table.

  • Breaking for Rust callers: MovePolicy is no longer Copy. It gains regions, max_per_node, fence_max_lag_records, timeout_millis, restore_after_millis and paused; clone it where it was copied. The control-plane config holds it as shard_moves. The control-plane Stream model gains region: Option<String>, and the broker’s MembershipConfig gains region_bridges.

  • Cache watches follow a moved shard. ClusterClient::watch_cache and watch_cache_retained now return a ClusterCacheWatch (and take self: &Arc<Self>). When the shard moves it hands out CacheWatchItem::ShardMoved as a notice and reopens the watch on the new owner at max(offset after the last change, resume_from), so the caller no longer reopens it and sees no change twice or missing. A sharded cache watch follows each shard the same way; ShardedCacheWatchItem::ShardMoved now means the shard is being followed, and a ShardClosed follows if it could not be. The Python and TypeScript watches follow too. A Client watch still ends with the frame.

  • A drain moves leaders before it replaces followers. Move slots on a draining broker go to the shards it leads first, then to its follower copies, so with the default limit of one a follower’s copy no longer holds the slot while the broker’s leaderships wait.

  • Online rebalancing is marked done on the status page. The row now cites the tests behind each claim. Load-aware placement has its own row. Pages that still said a move refuses publishes or ends subscriptions rather than letting them follow were corrected, and the docs-site semantics page no longer says a cache cannot declare Quorum or that metadata lives only in Postgres.

  • Breaking for downgrades: storage format v3. A record’s length word now carries two flag bits and, for the first record of an idempotent producer’s batch, a 20-byte tag (see docs/storage-format.md, “Producer marks”). v2 segments are still read and an unmarked record is byte for byte a v2 record, but new segments are written as v3, which a v2 build refuses to open: downgrading past this needs the data directory discarded. A shard directory may also hold a producers snapshot beside epochs.

  • Storage format v4. The generation-start record is flag bit 29 of a record’s payload_len, and only a v4 segment may hold one. Segments stay v3 until a log’s first generation-start record, so an upgrade stays reversible until generation_start is finalized; after that an older build cannot open the logs. v2 and v3 segments are still read. No migration is needed. (Format v5, for commit records, is under “Atomic commits”.)

  • Internal protocol kind 25, ReplicateMarkedRecords, ships records together with their producer marks. A batch without marks still travels as ReplicateRecords, unchanged; a follower that predates the kind refuses it and is halted rather than storing records without their marks. The API changed with it: ReplicateRecords has a marks field, batch_checksum takes the marks, AppendRecord and LogRecord have a mark, and felix_broker::replication::apply takes the marks.

  • Breaking for Rust callers of the control-plane crate: AppState has a move_policy field, ShardAssignment a move_reason field, and ControlPlaneStore the moves_paused and set_moves_paused methods. Default::default() fills the fields.

  • Breaking for Rust callers: ClusterClient::subscribe and subscribe_from take self: &Arc<Self> and return a ClusterSubscription instead of a (client, Subscription) pair, so the subscription can follow its shard. ShardEvent is #[non_exhaustive], and CacheWatchItem and ShardedCacheWatchItem have a new ShardMoved variant, so exhaustive matches need an arm. On the broker, Broker::end_subscriptions and CacheWatchHub::end_shard take an optional argument saying where the shard went.

  • max_shards caps roles when choosing a move’s destination. It was compared against leaders only, so a node full of follower roles could be given a shard to lead. Placement’s load now counts existing followers too.

  • A replica report leaves out followers the leader could not reach, as it already left out halted ones, so an unreachable move destination is never fenced on its last position.

  • Breaking for TypeScript callers: err.code is now the broker’s error code. It used to hold this client’s own kind (FELIX_AUTH, FELIX_CONNECTION, …), which moves to err.kind. code now matches the Python exceptions and the protocol (forbidden, shard_unavailable, …) and is undefined when the broker sent no code, as are retry and detail; kind is always set. Code that switched on err.code === "FELIX_AUTH" should switch on err.kind, or on the class.

  • Fresh placement bounds leaders as well as roles. With a replication factor equal to the node count every node holds a role for every shard, so the role bound was satisfied with every leader on one node, and the new rebalancer would then move them apart again. A cluster placed from scratch now needs no move to be balanced.

  • A draining assignment is reachable from assigning as well as active, since nothing reports active yet and the fence should not cost a move an extra generation. A drained shard still leaves only through a fresh assigning.

  • A node marked draining no longer has its shards reassigned on the next placement pass to brokers that hold none of their log. Its shards are moved instead, which takes a few passes; the harness’s move_shard steps placement until the move completes.

  • felix_router::shard is no longer public. Everything in it was already re-exported at the crate root, so felix_router::ShardRouter and friends are the paths to use.

  • felix-common drops what nothing used: NodeConfig, LimitsConfig, Error::Config, and every id type but RegionId.

  • felix-broker’s cache_watch, consumer_groups, dead_letters, durable, group_delivery and group_reader modules are private. Their public types are at the crate root (felix_broker::ConsumerGroups, felix_broker::GroupReader, felix_broker::DurableStorage, …), and the group tracker that group_delivery exposed is internal. replication and timings stay public modules. ClaimedPublish is now exported, since Broker::claim_publish returns it. The registry keys (CacheKey, NamespaceKey, StreamKey, TopicKey) are no longer public; nothing outside the broker used them.

  • The workspace is grouped by role. Crates live under crates/{protocol,server,sdk,testing}/ and the service packages are renamed felix-broker-service and felix-controlplane-service (binaries unchanged), so cargo test -p broker is now cargo test -p felix-broker-service. Crate internals are reorganised by domain; CONTRIBUTING.md has the rules.

  • felix-storage paths: CacheOp is felix_storage::cache::CacheOp, Epoch is felix_storage::log::Epoch, the Corruption* types are only at the crate root, and the modules nothing outside the crate used (commit_order, segment::io, disk_log::{epochs, recovery, retention, segments, sync}) are private.

  • felix-controlplane-service paths: the router and AppState are in api (was app), readiness is api::readiness, TLS is server::tls, membership and placement (with ReplicaPositions) are under cluster, the Raft store is store::raft with command and state_machine beneath it, metadata export is store::export, and now_millis is clock::now_millis. Two integration tests are renamed: readiness_pg is pg_readiness and meta_raft is raft_state_machine.

  • felix-broker-service modules are grouped by job, so their paths changed:

    • quic/transport::quic, auth (with auth_demo as auth::demo) and core_shards are under serving::, as is forwarding to a shard’s owner (serving::forward, from peer::forward and peer::handler).
    • credential, membership, lease, node_catalog and client_endpoints are under cluster::, and controlplane is cluster::catalog_sync.
    • shard_watch, shard_lifecycle and shard_routing are shards::watch, shards::lifecycle and shards::routing.
    • peer::replica is replication::replica, peer::dispatch is node::peer_dispatch, durable_config is config::durable, and timings is observability::timings. peer is now only the broker-to-broker transport.
    • Startup moved out of the binary into node::run_with_shutdown. cache_routing and group_ops are no longer public.
    • Log targets follow module paths, so a RUST_LOG filter such as felix_broker_service::quic=debug becomes felix_broker_service::serving::quic=debug.
  • Only felix-wire, felix-transport and felix-client are published to crates.io. The server crates were only there because of the in-process feature below.

  • Breaking: Raft members need an explicit cluster id and never form an empty group by accident. FELIX_RAFT_CLUSTER_ID is required and recorded in the data directory; a member refuses a store recorded under another id, and an existing data directory adopts the configured one. FELIX_RAFT_INITIAL_CLUSTER_STATE (new or existing, default existing): empty members form a group only under new, so a brand-new group needs new (the Helm chart’s hook sets it). The chart’s StatefulSet now creates members in parallel. (#727, #761)

  • Breaking: a durable shard with no caught-up replica waits for its owner. A durable shard with no replicas, which is every replication_factor: 1 stream, used to be placed on another broker about 15 s after its owner went quiet, where it served an empty log. Placement now holds it until the owner returns, which keeps its records at the same generation. It shows as owner_unavailable in GET /v1/placement/plan and in felix_shards_unplaceable; giving the records up is the new abandon step. In-memory shards still move at once. (#728)

  • Breaking: publishes are scheduled per shard and shared fairly across tenants. The process-wide pool of four publish workers, hashed by stream, is replaced by one ordered lane per shard (or per remote shard being forwarded to), run on FELIX_BROKER_PUB_WORKERS_PER_CONN executors (default 4) and picked by byte-weighted round robin across tenants, so a slow shard, a forward or a quorum wait no longer holds up other streams. Each tenant is guaranteed FELIX_BROKER_PUB_QUEUE_DEPTH slots (default 64) out of depth x workers and may borrow idle room. FELIX_BROKER_PUB_FLUSH_CONCURRENCY now counts per shard. A full queue answers an acked publish overloaded with reason publish_queue_full. Existing settings keep their names but change meaning, so revisit tuned values. New metric felix_tenant_publish_queue_full_total. (#774, #802)

  • Every write path checks the lease. Forwarded publishes, cache puts and deletes, counter adds, group acks, nacks and dead-letter changes, and Kafka produce go through the same per-shard fence as a direct publish, which re-checks the lease when a queued write claims its offsets; a lapsed lease answers shard_unavailable/fenced. Once majority_ack is finalized, Quorum stream writes are acknowledged by their followers instead. (#738)

  • Replica reports get an answer per shard. POST /v1/nodes/{id}/replica-status answers one outcome per shard (accepted, stale, not_leader, unassigned, future_generation) with 200 when all were accepted and 409 otherwise, where it used to answer 204. The control plane writes a report only if the node still leads the shard at that generation, checked in the same step, so a deposed leader’s report is no longer stored. Old and new brokers and control planes interoperate. (#738, #757)

  • Readers of a Quorum shard see only committed records. Subscriptions, replay, cache gets, counter gets, cache watches, group polls and Kafka Fetch on a replicated Quorum shard stop at the quorum mark, and Kafka reports the mark as the high watermark. Until lease_free_reads is finalized, a lapsed lease ends these readers with shard_moved and refuses cache reads as shard_unavailable/fenced. Leader streams are unchanged. (#738, #750)

  • Followers remember the newest leader they accepted. Each shard directory holds a replica file with the highest accepted generation and the commit offset, synced before the first batch at a new generation, so a restarted follower still refuses an older leader. A Quorum leader sends its mark with each batch (peer kind 26), and a follower refuses a truncation below it. (#743)

  • Node expiry tolerates clock steps. The control plane marks a silent broker down only after FELIX_NODE_EXPIRY_TIMEOUT_MS plus FELIX_NODE_REGRANT_MARGIN_MS (default a quarter of the timeout), measured on its own monotonic clock. A dead broker is placed elsewhere a little later than before. On Linux the broker times its lease on CLOCK_BOOTTIME, so a host suspended past the lease no longer wakes up holding it. (#743, #758)

  • Under Raft, heartbeats and the placement lease stay off the log. The leader keeps them as soft state, answers a heartbeat only after a quorum confirms it still leads, and writes only expiries and a heartbeat checkpoint every 5 s. The placement lease now expires on Raft as on Postgres. (#745)

  • Raft upgrades no longer depend on rollout order. Each member reports the metadata version it supports, and the leader proposes nothing above the group’s minimum: new commands answer 409 until every member is upgraded, and new fields are either left out on every member or the command is refused. A member that receives an entry it would drop refuses it and counts it in felix_meta_raft_unsupported_commands_total, so a rolled-back member stops with an error instead of diverging. (#761, #857)

  • Faster shard move switch-over, broker side. The broker long-polls GET /v1/shard-assignments/changes and acts on a change at once instead of on the next FELIX_CONTROLPLANE_SYNC_INTERVAL_MS tick, and a move’s successor opens the shard ahead of the cut-over. On a local cluster the switch-over drops from about 8 s to under 100 ms. New metrics felix_broker_shard_move_seconds and felix_broker_shard_switchover_seconds. (#673)

  • Breaking: a prefix cache watch on a multi-shard cache must name its shard. Without shard it read shard 0 and saw only the keys that hash there; it is now refused with the shard count. Open one watch per shard or use ClusterClient::watch_cache_sharded. (#647)

  • Each durable log flushes on its own thread instead of tokio’s shared blocking pool, where it queued behind other shards’ work. The thread stops after 10 s idle. With FELIX_STORAGE_IO_URING=1, new work wakes the ring at once instead of waiting behind another log’s fsync. (#692, #699, #716)

  • Cache and counter compaction runs in the background. The write that crossed the threshold used to rewrite the whole live set under the shard lock (about 20 s with 20 MiB live). A write now only starts a pass, which copies live entries forward and deletes the sealed segments below the cut. FELIX_STORAGE_COMPACTION_BYTES_PER_SEC (default 64 MiB/s, 0 unlimited) paces it. (#789)

  • Opening a log no longer opens its sealed segments. Recovery reads each sealed segment’s header and last index entry only; files open on first read, with at most 256 open per log (felix_storage_open_sealed_segments). Retention no longer holds the segment lock while it unlinks. A broker also closes a shard’s logs once it has no role in the shard, and opens shards in parallel. (#746, #765)

  • Breaking: connection-labelled subscriber metrics are gone. felix_sub_connection_subscribers lost its connection_id label, and felix_sub_conn_queue_len{connection_id} is replaced by the histogram felix_sub_conn_queue_depth. The demo signing key is behind a new demo cargo feature, so pubsub-demo-simple, cache-demo and latency-demo need --features demo. (#747)

  • Breaking: unknown frame flag bits are refused on every stream. FrameHeader::decode refuses any bit outside KNOWN_FLAGS with the new felix_wire::Error::UnknownFlags, which an exhaustive match must handle. (#737)

  • MTU overrides are clamped on Linux. FELIX_INITIAL_MTU and FELIX_MTU_UPPER_BOUND above 6550 are lowered with a warning, so ten GSO segments fit one UDP datagram. (#737, #754)

  • A keep-alive that cannot keep the connection open is replaced. A FELIX_KEEPALIVE_MS of zero, or not below half of FELIX_MAX_IDLE_TIMEOUT_MS, becomes a third of the idle timeout, with a warning. (#751)

  • A Raft write that cannot reach a quorum answers 503 instead of 500. The outcome is unknown, so a retried create may answer 409. (#813)

  • Group polls and in-memory group state are bounded. One poll returns at most 1,000 records and about 4 MiB. A group untouched for 10 minutes (or twice its visibility timeout) is dropped from memory and rebuilt from disk. FELIX_GROUP_MAX_IN_FLIGHT (default 10000) caps a group’s unsettled records; a poll at the cap returns empty, counted in felix_group_polls_capped_total. Long polls wake on appends instead of re-checking every 20 ms. (#731, #744)

  • felix-loadgen ingest stops at its deadline and spreads keys over every shard. --duration-secs now bounds each send and ack wait, and the JSON report gains completed_records, acked_records, acked_throughput_* and cut_off_records. (#924)

  • npm publishes through trusted publishing instead of a token, and each package is published with an explicit npm publish, platform packages first. A manual npm_stage input stages versions for a maintainer to promote. (#580, #581, #582)

  • felix-client’s in-process feature and InProcessClient. It wrapped an embedded felix_broker::Broker for two smoke tests and nothing else used it, but it made felix-broker and felix-storage optional dependencies of the client, and so forced them onto crates.io. Test against a broker over QUIC instead; felix-cluster starts one.
  • Broker::in_flight_publishes. It counted claimed publishes for the drained report, which now reads the broker service’s per-shard write fence instead.
  • In-memory replicated streams open after generation_start is finalized (#930). A promoted shard of an in-memory stream tried to append a generation-start record, which only a durable log can take, so it never opened for writes and its promotion retried five times a second. In-memory streams now skip the record: their publishes take no log offsets and are not replicated, so there is nothing inherited for it to cover. A promoted shard kept closed after its fence now backs off, doubling from 200 ms to 2 s.

  • A cache put or delete ships without waiting for the replication tick (#928). Stream publishes and counter updates woke the replication driver, but cache writes waited for its next tick (controlplane_sync_interval_ms, 2 s by default), so a Quorum cache put took about 840 ms. Every cache write now starts a replication pass, for Leader caches too.

  • A broker’s rotated refresh token is no longer world-readable (#913). Each rotation wrote FELIX_NODE_REFRESH_TOKEN_FILE through a temporary created with the default mode, so 0644 under the usual umask, even when the operator had created the original 0600. The temporary is now created 0600, and a stale one left by a crash is removed first rather than reused with its old mode.

  • FELIX_EVENT_BATCH_MAX_DELAY_US bounds the whole batch (#719). The subscriber feeder restarted the delay on every event, so a publisher sending faster than one event per delay held each subscriber’s events until 64 had built up or 64 KiB filled: about 6 ms at one event per 190 µs. The delay now counts from the batch’s first event, as documented.

  • An io_uring submit error no longer closes files under in-flight flushes (#722). With FELIX_STORAGE_IO_URING=1, a failed submit dropped every outstanding flush’s file while the kernel could still be syncing it, or had not yet taken its entry, so the sync could land on a reused descriptor. Each file is now held until its completion is reaped; the waiters are still failed with the error. EAGAIN and EBUSY, which only ask for completions to be reaped first, no longer fail any flush.

  • A power loss on a fresh node keeps its store roots (#888). The stream, group, dead-letter, cache and counter roots were created without syncing the directory that holds them, so a power loss before anything else synced it could drop a root and the acknowledged records in it. Every new directory is now synced into its parent, and opening an existing root syncs it again.

  • A new copy of a trimmed shard is placed without a rebuild. The leader’s first batch opened an empty log at 0 on the new broker, which then refused the leader’s bootstrap offer as if it held records. The copy waited for a rebuild slot, or for ever with FELIX_REPLICATION_REBUILD_MAX_CONCURRENT=0. An empty log now takes the offered base.

  • A subscription follows a moved shard off a draining entry broker. A ClusterClient reader told shard_moved with no new owner named (a lapsed lease or a deposed leader ends readers that way) asked only its entry broker. When that was the old owner shutting down, it answered draining and then nothing, and the reader failed with “the new owner did not take it before the deadline”. The follow now moves to another known broker when the entry broker answers draining or not at all, and a reconnect tries the broker it was using last. See docs/multi-node-client.md.

  • A broker handed a shard back after dropping held batches takes writes (#881). Liveness only: no acknowledged write was lost. A lease-mode leader that lost its lease with Quorum batches still held dropped them, followed the new leader, and answered a batch shipped again with that batch’s end. Its stream’s commit order moved back below records already in its log, so once the shard moved back to it every publish timed out with publish commit timeout. The commit order now never moves backwards on a replicated batch. See docs/replication-design.md, “Replication”.

  • A new leader names no follower short of the log it inherited (#882). Until a majority held a record of the new generation, a leader measured its replica report against offset 0 and named every follower caught up, answered or not. In lease mode, where promotion trusts the report, two failovers in quick succession could promote a follower missing a record the first leader acknowledged. Stream shards promoted with the fence were covered by it; Quorum caches and unfenced stream promotions were not. The report, and a cache’s counter level, now never name a follower short of where the leader’s generation begins. Right after a failover, a new leader that dies before any follower has copied its inherited log leaves the shard unplaced until a broker holding that log returns. See docs/replication-design.md, “Who may be promoted”.

  • A leader counts a follower only for what it has answered (#878). The quorum mark and the replica report counted a follower the leader had not yet heard from at the offset its cursor starts at, one record below where the leader’s own generation begins. A new leader, or a move’s source, that could reach none of its followers then published a mark over records it inherited and no follower held: its commit offset covered them, and readers could be handed them as committed. Deposed, the broker refused to drop them and refused every rebuild, so it never rejoined. Both now count FollowerCursor::confirmed, what the follower answered holding at this generation. No acknowledged write was affected: a publish waiting on the mark is dropped when its generation ends. See docs/replication-design.md, “Replica reports and the committed mark”.

  • A Quorum failover no longer promotes a move’s destination. Its broker opened the shard as the move’s cut-over, without the promotion fence, and under majority_ack the report that named it caught up could predate a record the old leader acknowledged on its followers, so that record was lost. Failover on a durable Quorum stream now ends the move instead. A promoted leader that gets a new generation while still fencing also keeps replication paused until it reopens, rather than shipping its unfenced log at the old generation in between.

  • A cache leader that loses its leadership rejoins as a follower (#863). Cache leaders now record where their generation began, on the cache and counter logs, and accept the generation on both. Before, a cache leader that died holding a record no majority had came back with no generation history, halted on that record, and refused the new leader’s rebuild for the rest of the generation, so a later drain could wait out the 30-minute move timeout. A rebuild now keeps the follower’s records below its commit offset and drops only what lies past them; the leader re-ships from its base, so those records are compared as they arrive, and a committed record that disagrees halts loudly (below_commit) and refuses further rebuilds from that leader. A refused rebuild is asked again after a backoff (5 s, doubling, capped at 5 min) instead of never. FollowerCursor::rebuild_refused is now an Option<RebuildBackoff>. See docs/replication-design.md, “Rebuilding a halted follower”.

  • A Quorum failover keeps its replica set. Failover used to rebuild the followers from live brokers, dropping the dead leader and adding a broker that had never held the shard. With majority_ack finalized the promoted leader fences a majority of that new set, which it could make with the newcomer alone, or with nobody when no other broker was live, and so open without a write the old leader and another follower acknowledged. A durable Quorum stream now keeps the previous set, the dead leader in it, so the fence needs a majority of the set that acknowledged. The dead member stays a follower until it returns or is drained. Upgrade the control plane before finalizing majority_ack.

  • Replacing a Quorum follower keeps what the old set acknowledged. A drain seats the replacement and drops the old follower once the replacement is within the move lag bound, or once a report names it caught up, which a leader does for every follower at a new generation before anything is counted there. Seated early, the replacement and a lagging follower were a majority of the new set without a record the leader and the old follower acknowledged, and a failover then lost it. On a durable Quorum stream the seat now also waits until the replacement holds what a majority of the set holds.

  • A subscription’s live_offset is reachable after a leadership change. It was the raw log tail, so when the log ended in a new leader’s generation-start record, which is never delivered, a reader waiting to reach live_offset waited until the next client write. It now stops short of trailing generation-start records, never below start_offset.

  • A cache write the store refused is no longer acknowledged. A put or delete whose log write failed (a failed fsync poisons the shard) was logged and answered as a success, so a Quorum cache could acknowledge writes that later reads did not return. It is now a storage error, and a failed read is an error rather than a miss. felix_storage::StorageApi’s put, get and delete return Result.

  • A Raft leader elected after a clock step back expires a dead broker in one window. The leader judged a broker it had not heard from by the log’s stamp when that was later than its own start, so a stamp left in the future by a step back, or by an old leader whose clock ran ahead, kept a dead broker placeable until real time caught up. It now ages every broker on its monotonic clock only, from its election or the broker’s registration, and lists a stamp ahead of its clock as now. Separately, a leader that died was counted at metadata level 0 by its successor, which then fell back to heartbeats and expiry through the log for as long as the old leader stayed down. The member probing a peer’s level now states its own, so every member knows the leader’s.

  • A record a new leader inherited is no longer lost after it was acknowledged (Raft’s Figure 8), once an operator finalizes the new fleet feature generation_start. A leader then appends a generation-start record whenever it starts leading a stream shard (after a promotion’s fence, at either end of a move, on a cancelled move’s hand-back) and its quorum mark counts a majority only once it reaches a record of the leader’s own generation, so an inherited record is acknowledged only when a later fence cannot prefer a log without it. Acknowledged inherited records are readable at the new leader without waiting for a client write. Until the finalize, brokers count as before. Readers skip the record’s offset; see docs/protocol.md for how a subscriber tells it from a drop. Finalize it only once every broker runs this version; it is one-way (see the upgrades page).

  • Opening a ClusterClient subscription waits out a shard that is still opening. subscribe, subscribe_from and subscribe_sharded failed at once on shard_unavailable/not_ready, which a leader answers while it fences its replicas after a promotion, including at cluster start. They now retry retry-class refusals with the ReconnectPolicy attempts and backoff, as a publish does; a fatal refusal still returns at once.

  • A cache or counter shard that an older build left mid-compaction opens whole again. In 0.5.0 and earlier, compaction swapped a shard directory with <shard>.retired and <shard>.compacting siblings. Background compaction dropped that swap and its recovery, so on upgrade a shard stopped between the two renames opened empty. Opening a shard now settles the old swap first: it restores <shard>.retired when the shard directory is missing, and it deletes leftover siblings when the directory is there. The cleanup assumes that a shard directory next to <shard>.retired is complete, and deletes .retired. That holds for every release. It fails only for someone who ran an unreleased main build from between #789 and #797 on a half-swapped shard: that build opened the missing directory as a new, empty shard, and .retired held the only copy of the data. If you did, replace the shard directory with its .retired sibling by hand before starting this build.

  • A power loss during a replication truncation or reset no longer leaves a log that refuses to open. Truncation unlinked the segments past the cut (sealed ones newest first, the active one last) and synced the directory once at the end. A reset created its new segment before deleting the old ones. So a power loss could leave segments that do not meet, and recovery reported corruption. Both now delete newest first with a directory sync after each unlink, and a reset creates its new segment only after the old ones are gone. An interrupted pass leaves a longer log, which replication cuts again. A head gap that an older build’s retention left behind still needs a manual fix; see “A gap at the head left by an older build” in docs/durable-storage.md.

  • A power loss during a retention sweep no longer leaves a log that refuses to open. Retention unlinked a sweep’s segments without syncing the directory, so the device could keep a newer unlink and lose an older one. Recovery then found an offset gap and reported corruption. Segments are now unlinked oldest first with a directory sync after each, so an interrupted sweep leaves a longer log instead.

  • A ClusterSubscription read cancelled mid-resume no longer ends the subscription. After a lost connection, next_event cleared the loss before resubscribing, so a caller that dropped the call partway (a read under tokio::time::timeout, and every Python next_event(timeout=...)) came back to the dead subscription and got None, as if the broker had closed the stream. The loss is now kept until the resubscribe lands, and the resubscribe (after a loss or a shard move) runs on a task of its own, so a caller polling with a timeout shorter than a reconnect no longer restarts it on every call and never finishes.

  • Python’s closed tells a timeout from the end of the stream. It was only set by close(), so after next_event(timeout=...) returned None there was no way to tell the two apart, which its documentation said there was. It is now also set once the broker ends the stream.

  • A publish in flight when its shard’s log is reset no longer reaches readers. A publish waiting behind earlier ones when the broker became a follower (or its log was rebuilt) was woken by the reset and went on to put its offsets from the old log into the cleared replay ring and fan them out. It now fails with unacknowledged and applies nothing.

  • Acks for group claims made before a failover, move or eviction are settled. A new owner refused acks and nacks for records an earlier owner had handed out, as invalid_request, which is fatal. It now accepts any ack below the log tail it inherited. An ack for a record written after that and not yet polled is refused with the new stale_claim code (retry class retry; an older client sees an unknown code with that class). An offset past the log tail is still invalid_request. (#760, #776)

  • A traced publish no longer panics a stream handler. The binary batch, JSON publish, JSON batch and subscribe handlers held a span guard across an .await. When the task resumed on another worker the guard exited there, leaving a stale span on the first worker’s stack; the next span created on that worker could then clone the closed span and panic with “tried to clone a span that already closed”, killing the publish stream. The binary batch span is info, so this hit the default filter under load, and a slow on_close in the OpenTelemetry layer widened the window. The handlers now use .instrument(span), and clippy.toml rejects span guards held across an await.

  • felix_publish_requests_total and felix_publish_bytes_total are recorded in a default build. Both went through the telemetry-gated macros, so a broker built without that feature exported neither while serving traffic. Bytes are now counted for fire-and-forget (accepted) publishes too, as the payload sum of the request, and no longer for a batch whose acknowledgement timed out. The observability page marks which of the listed metrics still need the telemetry feature.

  • A Periodic fsync tick skips a log with nothing to flush. Every open log fsynced on every tick, written or not: a broker with ~200 idle shard, cache and counter logs issued ~800 fsyncs/s.

  • Clients publishing one stream spread across a broker’s listeners. The stream-to-publish-stream hash used a seed fixed for the whole process, so every Client in a process sent a given stream down the same pool slot, and so to the same listener. On a four-listener broker fed by four load generators, two ports carried all the publish data (3.1 GB and 9.4 GB) and the other two only handshakes. The seed is now per client; one client still keeps each stream on one writer.

  • Concurrent “ensure signing keys” calls agree on one key set. Postgres and in-memory control-plane stores checked for a tenant’s keys and wrote new ones in separate steps, so racing callers each generated keys and all but the last returned a set that was then overwritten; tokens signed with it would not verify. Postgres now decides under the tenant row lock, the way auth bootstrap does, and the in-memory store under one write lock. The Raft store was already safe: install-if-absent is applied through the log.

  • A failover no longer costs a publisher 30 seconds. A publish in flight to a leader that was killed waited out the 30 s ack timeout, because a killed broker sends no QUIC close and the connection only died at the 30 s idle timeout. The default QUIC idle timeout is now 6 s and keep-alives go every 2 s (FELIX_MAX_IDLE_TIMEOUT_MS, FELIX_KEEPALIVE_MS), so the dead connection fails the waiting publish and ClusterClient retries on the new leader in about 6 s. Quiet connections stay open on the keep-alives. A keep-alive not below half the idle timeout is replaced (see Changed).

  • A publish to a shard that cannot be served is no longer called “stream not found”. The broker kept the right code but replaced the message with stream not found: ... for every refused publish, so a shard that was moving, fenced or still opening read as a missing stream to operators and to clients without error codes. Only a stream that does not resolve says that now; any other refusal keeps its own code and text, e.g. publish to t1/ns/orders refused: this broker is still opening the shard. Clients that matched on the old text for these cases will see the new one; retry behaviour of coded clients is unchanged, since the code was always shard_unavailable.

  • Binary publish acks carry the error’s detail. A failed binary ack had the code but dropped detail, so a binary publisher never learnt a shard_unavailable reason or a moving shard’s suggested wait. The new flag bit FLAG_BINARY_PUBLISH_ACK_DETAIL (0x0400), negotiated like FLAG_BINARY_PUBLISH_ACK_CODE, adds it after the code; felix-client offers it and fills BrokerError::detail from it. The new encode_publish_ack_bytes_detailed encodes it; PublishAck gains a detail field.

  • Forged upstream tokens can no longer make the control plane hammer an IdP. A token with an allowlisted issuer and an unknown kid made /token/exchange re-fetch that issuer’s JWKS before any signature check, one fetch per request, with no credential needed. Each JWKS URL is now fetched at most once per 30 seconds, concurrent misses share the fetch in progress, and JWKS and discovery requests time out after 10 seconds. A key the IdP rotates in is accepted within 30 seconds of its first use.

  • An assignment change refreshes the node catalog. A broker read /v1/nodes on a wake only when an assignment named a node it did not know. A broker that restarted comes back under the same id on new ports, so it was known and its old address kept: the first ship or forward to it after a move waited for the next catalog tick (FELIX_CONTROLPLANE_SYNC_INTERVAL_MS). Every wake now reads the catalog, alongside reconcile rather than before it, and each read is bounded at 5 s so a control plane that never answers cannot hold the feed (a_wake_refreshes_the_address_of_a_node_that_moved, and in felix-cluster, a_move_onto_a_restarted_broker_does_not_wait_for_the_catalog_tick).

  • A leader ships to a restarted follower at its new address. A follower’s cursor kept the address it was created with for the whole generation, so a broker that restarted on new ports, still in the replica set, was dialled at its old one until the shard’s generation changed. A move staged onto a broker that had just restarted never caught up and sat at staged until the move timeout. The cursor now takes each pass’s address from the route, so the follower is reached within one catalog refresh (FELIX_CONTROLPLANE_SYNC_INTERVAL_MS) (a_follower_that_moved_address_is_shipped_to_at_the_new_one, and in felix-cluster, a_restarted_broker_takes_shards_again).

  • A broker that took a shard over replays all of it. A follower’s replay ring was filled when the first replicated batch opened the stream and was never touched again, so after a failover or a planned move a reader from the start got that first batch and then the live edge, skipping everything replicated in between. The records were on disk; a cursor-based subscribe had the same hole. The ring is now emptied as replicated records move the tail, and a reader is served from the log (records_replicated_after_the_first_batch_replay_from_the_start, and in felix-cluster, a_handoff_that_times_out_loses_no_acknowledged_record).

  • Client: out-of-order publish acks no longer break the stream. The broker answers acked publishes as each completes, so on a publish stream carrying several at once a Quorum or forwarded publish can be answered after one sent behind it. The client required acks in request order and treated any other as a protocol error, failing every publish outstanding on that stream and reconnecting. Each answer now goes to the request it names (acks_answered_out_of_order_reach_their_own_requests).

  • A Postgres-backed control plane keeps a stream’s replication factor. Reading a stream back always reported a replication factor of 1, so a replicated stream was placed as leader-only, and any patch to the stream (retention, consistency, delivery) wrote the 1 back over the stored value. The store’s Postgres tests, which would have caught it, never ran: task test did not enable their feature, and when enabled they skipped themselves, because resetting the schema failed on a foreign key and the tests sharing it deadlocked. They now run with the database task test starts, serially, and a setup failure fails them.

  • A Quorum shard is replaced when its leader dies mid-publish. A leader reported a follower caught up only when it held the leader’s whole log, so a publish landing between shipping and the report left every follower one record short and the report named nobody. A leader killed right then left the control plane no replica it may promote, and the shard stayed down for good while its producers were told stream not found. Under Quorum a follower now counts as caught up when it holds everything up to the offset a majority holds, which is every record that can have been acknowledged. Leader streams and moves still require an exact copy.

  • The move limits hold across control-plane instances. Each placement write was conditional only on its own shard’s generation, so two Postgres-backed instances, or a pass and an operator’s request on another instance, could each read the last free slot and start moves on two different shards. One instance now runs the timed placement passes, the holder of a lease in the store that lasts three reconcile intervals, is renewed every pass and is released on shutdown; under Raft the leader holds it. Every placement write, operator steps included, is also fenced by a placement token read before the pass or request decided: a write lands only if no other placement write landed since, and a new holder advances the token, so an instance that paused past its lease writes nothing after a takeover. Passes woken by a report still run where the report arrived. Postgres gains migration 0016_placement_lease.sql; the Raft log gains put_shard_assignment_fenced and take_placement_lease commands, which a follower running an older build refuses, so upgrade every member before the leader. ControlPlaneStore::put_shard_assignment_if takes the token. Metrics: felix_placement_lease_held, felix_placement_lease_takeovers_total, felix_placement_writes_fenced_total.

  • A redirect to a draining broker carries its address. A not_leader answer or publish-ack owner hint naming a broker that is being drained, but still leads the shard, omitted the client address because only brokers eligible for new work were listed. Routable brokers’ addresses are now used for redirects; topology still lists only eligible brokers.

  • One refused publish failed later publishes on the same connection. A publish the broker refused (an unknown stream, a forbidden one, overload) stopped the client’s publish worker that sent it, and every later publish routed to that worker, to any stream, got the same refusal back without a code. ClusterClient then took the uncoded error for a dead connection and reconnected. A refusal now answers only the publish it belongs to; a broken or timed-out ack stream still ends the worker.

  • A long replay lost records and could look stalled. A subscription resumed from an early offset dropped history even when the application read every event promptly: the broker writes history as fast as it reads it, and the client drained it into its bounded queues under the default drop_new policy. A replay of 5,000 records typically delivered the first 256 and a scattered few after; when the tail was among the drops, the subscription went quiet until the next live publish. History below the subscription’s live_offset now waits for room in the client’s queues whatever the policy, so a replay is paced by its reader and never dropped. Live records past live_offset keep the configured policy.

  • A subscription’s last events could be dropped when it ended. The subscriber’s connection writer dropped deliveries queued in the same batch as its unregister, and the feeder forgot the subscriber’s connection before deliveries still in its lane queue were routed. Both now deliver them first, so a subscriber ended by a shard move receives everything the broker committed.

  • A Leader publish acknowledged on enqueue could be dropped without a trace. With ack_on_commit off (the default) the ack goes out when the publish is queued; if the broker’s lease lapsed before the worker wrote it, the worker refused the write and the refusal went nowhere. The refusal is right (another broker may lead the shard), so admission now reads the lease clock for such a publish and, with less left than the publish queue wait plus the ack wait (capped at half the lease), waits for the write, so a lapse reaches the client as shard_unavailable. A pause that starts after the ack can still strand one; it is counted in the new felix_broker_acked_publishes_dropped_total{reason} and logged at warn. The docs no longer claim a Leader ack means the record is durable under the default. (#671)

  • A Quorum stream placed with one replica refused every publish on a cluster with leadership_lost: nothing ships for such a shard, so no quorum mark was ever published. The leader alone is its majority now.

  • An acknowledged write could be lost in a planned shard move. Admission checked that the broker served the shard, but an admitted publish could wait in the publish queue and claim its offsets after the old leader had reported drained and the control plane had cut over; it was then committed and acknowledged on a broker the new owner never copied it from. The drained report guessed at this with the tail holding still for two passes, which a deep enough queue outlasts. The same gap existed for cache puts and deletes, counter adds, consumer-group polls, acks, nacks and dead-letter changes, and publishes forwarded to the old leader.

    Every such write now enters a per-shard write fence right before it claims its place in the log and holds it until it is durable and fanned out. The shard lifecycle closes the fence as soon as it sees a move, and a write that reaches the fence afterwards is refused the way a write to a shard the broker does not serve is. The drained report goes out only once the fence is closed with nothing inside it, which makes it exact. The readers a moved shard’s old leader ends now receive every record it committed first. The TLA+ model splits admission from the claim, and FelixShardHandoffNoClaimFence.cfg shows the loss without the check.

  • A planned shard move could lose a dead letter or a counter add. The old leader reported drained on the shard’s own log and shipped its consumer groups’ cursors and dead letters, and its cache’s counters, only afterwards, logging a failure and moving on. A dead letter the new owner lacked was a record its group silently skipped, and a counter add it lacked was an acknowledged add gone from the sum. The drained pass now ships those logs first and reports drained only once the move’s successor holds them; any other replica still missing them is left out of the report’s caught-up list rather than holding the move. A move that lost its successor waits for every replica level on the shard’s log. A move held on them stays Draining, and felix_broker_replication_drain_withheld_total{log} and a warning naming the shard and follower say why. Brokers now read the assignment’s successor.

  • A broker shutting down in the middle of a credential refresh could exit after the control plane had rotated its refresh token but before writing the replacement, so the next start presented a spent token and the chain was revoked. Shutdown now waits, within the drain deadline, for a refresh in flight to finish.

  • A failed fsync poisons the log. Linux can drop the dirty pages behind a failed fsync, so a later successful sync could acknowledge lost data. Any flush, seal, truncate or reset failure, including a failed directory or epoch-file sync, now stops the log’s durable offset, and every later write returns the error until the broker restarts. Readers of a poisoned Leader stream stop at the durable offset. (#726, #777, #851)

  • Recovery repairs what a power loss leaves. A zero-filled tail is cut back as a torn tail; a roll interrupted by power loss, a segment cut short while it was sealed, and an empty segment left by a lost background roll are repaired; a sealed segment whose index points at the wrong place is rescanned. Each log keeps a durable.mark file, and damage past it is repaired as a torn tail, so a power loss under Periodic or None fsync no longer leaves a shard that refuses to start. Damage before the mark is still fatal. (#726, #746, #754, #759, #775)

  • A cancelled cache put or delete is still applied. A write cancelled after it claimed its offset released its commit turn before its fsync finished, so readers could see a record that was not yet durable. (#726)

  • A follower’s commit offset is on disk before it acknowledges, under OnCommit. It was written at most once a second, so a crashed follower could read back a lower offset and allow a deeper truncation. (#769)

  • A panicking publish worker is restarted, and only the job it was running fails (felix_broker_publish_worker_restarts_total). Graceful shutdown now waits for queued publishes, including ones acknowledged on enqueue, to be written. (#747)

  • One stalled subscription no longer blocks the others on its connection. Each subscriber has its own queue in the connection writer and drops only its own frames (felix_sub_queue_dropped_total). A Latest subscriber no longer gets live records from below its start offset. (#747)

  • An idempotent producer no longer reuses a sequence after an ambiguous failure. The next, different batch went out under the same sequence and was answered as a duplicate without being written. Such a batch is now held in doubt: only the same payloads may be re-sent under it. (#737)

  • A lost connection ends a subscription with an error, not as the end of the stream. It surfaces as felix_client::SubscriptionLost, and ClusterSubscription resubscribes from the next offset until the retry policy’s deadline. A publish larger than the 4 MiB in-flight budget now goes out alone instead of never. (#737)

  • A redrive survives a failover. It is one durable write, so a new leader delivers redriven entries again, and a failed dead-letter write is retried on the next poll instead of stalling the group. (#731)

  • A consumer group cannot redeliver acked records after its shard moves back. The in-flight tracker now resets when a broker starts a new term leading the shard. (#688)

  • A follower’s leftover tail from an older leader is compared before a new leader ships past it, so a dead leader’s unacknowledged write cannot survive at an offset the new leader filled differently. A dropped suffix is dropped from the replay ring too. (#779, #782)

  • A follower keeps the generation each record was written at. It labelled every record with the sender’s generation, so a copy of an inherited record looked newer and a later promotion could prefer that log and lose an acknowledged record. Batches now carry per-record generations (peer kinds 32 and 33, capability GENERATION_LABELS); the fix holds once every broker runs this version. (#814, #837)

  • A Quorum publish no longer fails while a move is staged. It waits for the new generation’s first quorum mark instead of being refused. (#796)

  • A slow peer no longer holds other quorum marks. One unresponsive peer could delay every Quorum acknowledgement on the broker by about 2 s; each shard now replicates on its own schedule. (#804)

  • One slow peer request no longer stalls every peer. Broker-to-broker handlers ran inline on the QUIC I/O thread. (#767)

  • Brokers rejoin after being marked down, and a control-plane outage no longer expires the fleet. The expiry sweep waits a full window after start, an election or a lost store, and broker calls to the control plane have deadlines. (#730)

  • Deregistering a live broker no longer allows two leaders. A left node is treated as draining until one expiry window plus the regrant margin has passed since its last heartbeat. (#766)

  • A dead broker expires in one window after the control plane’s clock steps back on the memory and Postgres stores. (#785)

  • Replica reports survive a Raft snapshot and the migrate export. (#849)

  • A retried Raft write no longer answers 409 for its own success. Each write carries a request id fixed before the first attempt. (#602)

  • A Raft member that lost its volume does not vote until it has caught up, so it cannot win an election and truncate acknowledged metadata writes. (#670)

  • A control-plane member that cannot verify a token says so with 503 cannot_verify instead of 401 invalid token. (#603)

  • Two control-plane instances could undo each other’s shard moves. Each instance wrote what it planned unconditionally, so one that stalled could hand a shard back to its old leader after another had cut over. Every placement, promotion and move step is now written only if the shard is still at the generation it was planned from (ControlPlaneStore::put_shard_assignment_if, Raft command PutShardAssignmentIf); a skipped write is counted in felix_shard_assignment_write_conflicts_total and re-planned. (#660)

  • The docs said the clients were not installable. felix-client is on crates.io, PyPI and npm; the client pages and binding READMEs now say how to install it. (#585)

  • Release pipeline fixes. The npm job published nothing and reported success; it now moves each binary into its platform package, publishes the main package last, and checks the registry. The Node addon’s x86_64-apple-darwin leg uses the pinned toolchain. The PyPI step skips files already uploaded, the crates.io job waits out the new-crate rate limit, and every npm command pins the public registry. scripts/npm_first_publish.sh claims a new package name, which CI cannot. (#578, #579, #580, #582, #583, #584)

Deliberately not in this preview:

  • A single acked publish on the fsync path still takes longer to be acknowledged than the comparable NATS JetStream setup (#926).
  • Claiming a lane’s queued publishes as one durable append is still being measured (#932).
  • A publish connection’s QUIC receive windows can buffer far more than the ingress budget before backpressure applies (#923).
  • Quorum cache and counter shards still open on the lease, not on a fenced majority (#933).