Skip to content

Felix 0.6.0-preview.3

Released 2026-10-08 as a preview. The full history is in CHANGELOG.md.

The third preview of 0.6.0. felixctl now manages a cluster as well as using it: it creates, changes and deletes tenants, namespaces, streams and caches, works consumer groups, counters and RBAC rules, and shows a shard’s or a subscriber’s live state with felixctl inspect. It installs with Homebrew. Every change of a shard’s leader is fenced, and replicas keep a ballot with each generation they accept. Sealed segments can be copied to an object store before retention deletes them, the first step toward tiered storage. felix-capi adds a C ABI over the Rust client, the base for Go and C# SDKs. Caches gain conditional writes and key-scoped grants, and a client can read a bounded range of a stream. Development works with Docker or Podman. Several move, failover and lease-timing bugs are fixed. The Rust API has breaking changes, listed under Changed. pip will not install it without --pre.

  • A Homebrew formula for felixctl: brew install getfelix/tap/felixctl on macOS and Linux, with completions and man pages (#1005). release.yml writes it from each release’s archive checksums and pushes it to <owner>/homebrew-tap when PUBLISH_HOMEBREW is true, using the HOMEBREW_TAP_DEPLOY_KEY secret (a write deploy key on the tap); dry runs keep it as an artifact and nightlies skip it.

  • felixctl inspect subs and the subscriptions_list request (part of #1077). A broker advertising FEATURE_INSPECT now also answers subscriptions_list with subscriptions_list_info: one page of the subscriptions it serves, each with its stream and shard, subscriber and subscription ids, connection id, client address and principal, overflow policy, queue depth and capacity in batches, records dropped, position and the shard’s tail. The filter narrows by tenant, namespace, stream, shard, principal, or to subscriptions that have dropped records. A page is 100 by default and at most 1000, and a cursor of shard and subscriber id continues it. It needs node.view:cluster:*. The broker reads each shard’s fanout snapshot and never touches a queue. On the delivery path, a subscriber’s receiver now stores one past the offset of each batch it takes (a relaxed store), and a drop adds to a per-subscriber count beside the existing metric. felixctl asks every broker, or only --node, and prints one table or one JSON line per broker, with --principal, --dropping, --limit and --cursor. felix-broker adds Broker::list_subscriptions, SubscriberStats, SubscriberOwner and Subscription::set_owner; felix-client adds Client::list_subscriptions and SubscriptionsPage, and its inspect calls now report a broker’s unsupported answer as such. Diagnosing a cluster covers a subscriber missing records and one falling behind.

  • Stream logs can copy their sealed segments to an object store before retention deletes them, the first part of tiered storage (#172). Off by default; FELIX_DURABLE_OFFLOAD_DIR turns it on, and the only backend is a directory, through the object_store crate’s local filesystem backend. A pass on the retention timer uploads each sealed segment, reads it back to check size and CRC-32, and records it in a per-shard offload.manifest, fsynced. Retention then deletes only recorded segments, so a crash at any step leaves the local segment or a recorded copy. Recovery accepts a gap in the local chain when the manifest covers it. Nothing reads the copies yet: a read below the local head is still Trimmed. New metrics felix_storage_offload_segments_total, felix_storage_offload_bytes_total and felix_storage_offload_failures_total. Breaking for code that builds felix-storage’s LogConfig without ..Default::default(): it gains offload.

  • felixctl inspect shard and the shard_inspect request (part of #1077). A broker advertising FEATURE_INSPECT, the first bit of the extended feature word (server_features_hi 0x1), answers shard_inspect with shard_inspect_info: its own view of one shard, with its phase and generation, whether it serves and why not (opening, fencing, failed with the open error, draining, lease_lapsed, not_assigned_here, behind_generation), which replicas took a promoted leader’s fence and how many attempts it has made, its lease, tail and commit mark, and as leader each follower’s next offset, lag and state. It needs node.view:cluster:* and may name any tenant. The answer is read from snapshots the lifecycle and the replication driver publish (the new felix_replication::status::ShardStatusBoard), so it takes neither’s lock and opens no log. felixctl asks the leader and every replica and prints a table, or one JSON line per shard with --json; a broker that cannot be reached is listed as unreachable. felix-client adds Client::inspect_shard and Client::supports_inspect. A new docs page, Diagnosing a cluster, goes through shard, replication, subscriber, auth and startup problems by symptom.

  • felix-capi, a C ABI over the Rust client and the base for the Go and C# SDKs (part of #618). It builds libfelix as a shared and a static library with a cbindgen header checked in at crates/sdk/felix-capi/include/felix.h, and covers connect, publish and a polled subscribe (felix_client_connect, felix_client_publish, felix_client_subscribe, felix_subscription_next_event). Handles are opaque with a free function each, every call returns a status code whose classes match the other SDKs, the message for a failure is per thread (felix_last_error_message), panics never cross the boundary, and each client owns its Tokio runtime. A test fails while the header is stale, and CI runs a C program against the conformance fixture. Not published to crates.io.

  • Replicas keep a ballot with each accepted generation (part of #1009): the leader they accepted it from, by the node id its Hello gave. At that generation a replica refuses a fence, a batch, a bootstrap, a rebuild or a tail fetch from any other node with FencedEpoch, and a broker will not lead a generation it already accepted from another node. The ballot is a new ballot file in the shard directory, fsynced before anything from that leader is answered and read back on open; builds without ballots ignore it, so rolling back still opens the shard. A new internal capability, BALLOTS (1 << 6), is offered with the fence, and FELIX_INTERNAL_FENCE=false turns ballots off with it. Nothing changes while the control plane names every leader, since it never names two at one generation; this is the safety layer replica elections will need. The TLA+ model gains Ballots, Elections and OneLeaderPerGeneration, with FelixShardElect.cfg and two configurations that must fail: FelixShardElectNoBallot.cfg (two leaders at one generation) and FelixShardElectStaleSet.cfg (a replica standing on a set it has left). Breaking for callers of felix-storage’s DiskLog::accept_generation and felix-broker’s StreamLog::accept_generation, which take the leader, of felix-replication’s ReplicaHandler entry points, which take the sender, and for code matching GenerationCheck, which gains Promised and is no longer Copy.

  • felixctl group and felixctl counter (#1005). group create|describe|seek|rm manage a consumer group on every shard of a stream, or on --shard. group poll claims records and prints each with its claim, SHARD:OFFSET:ATTEMPTS, which group ack, nack (with --delay-ms), extend and dead-letters add take. group dead-letters ls|redrive|discard work the dead-letter list. group rm, dead-letters discard and a seek that moves a cursor back over finished records ask [y/N] at a terminal and need --yes elsewhere. counter get|add read and add to counters. All go through ClusterClient. felix-client now re-exports GroupRecord, which group_extend takes.

  • Nightly TLC simulation of long random walks over replica-set changes (#934). FelixShardWalkSpares, FelixShardWalkMoves and FelixShardWalkHandoff lift the bounds the exhaustive configurations stop at (time to 40, six writes, six moves) and sample traces up to 300 steps; each has a negative twin that must find its violation within the budget. scripts/check_tla.sh --simulate (task tla:walk) runs them, and tla-walk.yml runs them nightly and uploads TLC’s trace on a failure. Tick in FelixShard.tla enumerates clocks over the drift window only, with the same states.

  • felixctl writes to the control plane (part of #1005). tenant, namespace, stream and cache gain create and rm; stream set changes consistency, delivery, durability and retention, keeping a retention bound not given, and cache set the display name; each prints what changed. node drain and node deregister, shard move NAME SHARD --to NODE (with --dry-run and --cache) and shard move cancel, and placement pause, resume and abandon drive placement. Deletes, drains and deregistrations ask on a terminal and need --yes anywhere else, stopping with status 2 without it; placement abandon never asks and always needs --yes.

  • felixctl rbac policy ls|add|rm and felixctl rbac grouping ls|add|rm list, grant and revoke the current tenant’s RBAC policies and role assignments through the control plane (#1005). policy add checks the object against the RBAC grammar first, including cache key and key-prefix objects and their cache.read/cache.write-only rule, and stops with status 2 and the reason; action names and delegation scope are left to the control plane, whose refusal is printed with status 4. rm asks on a terminal and needs --yes elsewhere, and exits 5 when the rule does not exist. felix-cluster adds Credentials::rbac_admin_token.

  • A subscriber can choose its own broker-side queue capacity on a broker advertising FEATURE_SUBSCRIBE_QUEUE (0x4000_0000) (#1019). subscribe takes an optional queue_capacity, counted in published batches; the broker clamps it to 1..=FELIX_SUBSCRIBER_QUEUE_CAPACITY_MAX (new, default 4096) and echoes the grant as queue_capacity on subscribed. A subscribe without it, and the answer, are byte-identical to before. The overflow policy stays the stream’s: a subscriber that could choose Block could stall every publisher on its shard. felix-client adds ClientConfig::broker_sub_queue_capacity and Subscription::queue_capacity; felix-broker adds Broker::subscribe_sized and Broker::subscribe_from_sized. ClientConfig gains a public field, so code building it with a struct literal must set it.

  • One Rust Client can act for many users over the same connections (#969). Client::with_identity(tenant_id, token_provider) and Client::with_identity_token return a client that shares the parent’s connections and authenticates every stream it opens with the user’s token, so the broker checks each user’s publish, subscribe, cache and group requests against that user’s grants alone. It opens one publish and one cache stream and no connections. One user’s token expiring or being revoked stops only that user’s streams. No wire change: the broker already authenticates per stream. A refused publish or cache request still ends the stream it came on, so a gateway should build a new identity after one. ClusterClient does not offer it yet.

  • A write can be made only at the offset its writer expects, on a broker advertising FEATURE_PUBLISH_CONDITIONAL (0x1000_0000) (#1017). publish_if appends a batch only if it would start at expected_offset, the shard’s next; commit takes the same optional expected_offset as a compare-and-set on the whole shard. The check is made where the log assigns offsets, under the same lock, so of two writers at one offset exactly one is written. A refusal is publish_refused with the new offset_mismatch reason and the shard’s tail; it writes nothing, consumes no offset and holds up no later publish. The tail counts a new leader’s generation-start record, so a writer’s expected offset goes stale on failover. Served by the shard’s leader only: a non-leader answers not_leader. An in-memory stream refuses it. A commit without expected_offset is byte-identical to before. publish_if is charged against the tenant’s publish quota and the ingress byte budgets like any acked publish, and refused over quota the same way. felix-client adds publish_if and commit_if on Client and ClusterClient, answering ConditionalWrite; felix-storage adds DiskLog::append_claimed_at, and felix-broker Broker::claim_publish_at, publish_batch_at and commit_to_handle_at. Per-key version preconditions are #1051.

  • A second feature word for when the u32 feature set runs out (#1055). FEATURE_EXTENDED (0x8000_0000) in client_features or server_features says the peer sends and reads client_features_hi / server_features_hi. A peer sends the marker and the word only when it knows a feature there, and a broker answers with its word only to a client that set the marker, so every existing frame is unchanged. No feature uses the word yet. felix-wire adds FEATURE_EXTENDED, KNOWN_FEATURES_HI, offer_features, answer_features and peer_features_hi. Breaking for code that builds Message::Auth or Message::AuthOk: both have a new field.

  • A client can read a bounded range of a durable stream and stop (#1018). A broker advertising FEATURE_STREAM_READ (0x2000_0000) answers stream_read (shard, from, optional end, max_records, max_bytes) with stream_records: one page of committed records and the next_offset to continue from, which steps over generation-start records. No subscriber is registered and the read never waits. A page stops at the committed mark on a Quorum shard and, under FsyncMode::OnCommit, at the durable offset, and is capped at FELIX_DURABLE_MAX_RECORDS_PER_READ records and 4 MiB of payload. A from below retention or past the tail gets subscribe_cursor_error. Needs stream.subscribe; only the shard’s leader answers. In felix-client, Client::read and ClusterClient::read return a StreamPage; felix-broker has Broker::read_range. Frames to clients that do not send the request are unchanged. An end on Subscribe is tracked in #1053.

  • A consumer can manage its own claims on a broker advertising FEATURE_GROUP_CLAIM_CONTROL (0x800_0000) (#974). group_extend keeps a claim standing while the work goes on, answered with group_extended; it names the delivery by offset and attempts, so once the claim lapses and the record goes out again it is refused with stale_claim. delay_ms on group_nack makes the record owed only after a delay, holding a place under the in-flight cap until then. group_dead_letter lists a record as a dead letter and finishes it, written to the dead-letter log first as when the broker gives up, so group_redrive works on it. visibility_ms on group_poll sets how long that poll’s claims stand. All need group.consume. Durations are capped by the new FELIX_GROUP_MAX_VISIBILITY_MS (12 hours, never below the visibility timeout). Both new fields are left out at zero, so nacks and polls that do not use them are byte-identical to before. A group’s in-flight state is no longer evicted while a claim stands. felix-client adds group_poll_with and GroupPollOptions, group_extend, group_nack_after, group_dead_letter and supports_group_claim_control on Client and ClusterClient, and extend, nack_after and dead_letter on ShardedGroup. Breaking for callers of felix-broker’s GroupReader::poll_below, which takes the claims’ visibility, and for code matching felix-wire’s GroupPoll and GroupNack, which gain fields.

  • Streams and caches can be created together, all or none (#967). POST /v1/tenants/{t}/namespaces/{ns}/resources takes up to 256 stream and cache create bodies and writes the new ones in one Postgres transaction or one Raft log entry. A missing permission, an invalid item, a name given twice, or an item that already exists with a different configuration fails the whole batch and creates nothing. An item that already exists as asked is reported unchanged, so a batch can be sent again after a failure. Under Raft this is metadata level 5; the endpoint answers 503 until every control-plane member runs a release that has it.

  • Cache grants can name a key or a key prefix (#966). A fourth segment on a cache object, cache:{tenant}/{ns}/{cache}/{key} or .../{prefix}*, limits cache.read or cache.write to that key, or to keys starting with the prefix (a plain string prefix: user:1* covers user:10). The broker checks get, put, delete, conditional writes, counters and key watches against the key, a prefix watch against its prefix, and a forwarded op again at the owner. Grants without a key segment still cover the whole cache. The control plane refuses a key with * anywhere but the end, a bare *, a wildcard namespace or cache under a key, and any other action on a key object; token exchange narrows a whole-cache grant to a key hint. No wire change. Mixed versions fail closed: an older broker refuses requests a key grant would allow. felix-authz adds PermissionMatcher::allows_cache_keys and CacheKeys; the control plane’s ParsedObject::Cache gains a key field.

  • Token exchange can narrow by action and resource pairs (#968). permissions: ["stream.subscribe:stream:t1/ns/a", "stream.publish:stream:t1/ns/b"] narrows each pair on its own against the grants with exactly that action, where requested and resources apply every action to every resource. It never widens what RBAC grants. It must be sent with "requested": [] (a 400 otherwise), so a control plane that predates the field refuses rather than minting full rights, and the refresh record stores the same [] for the same reason. Refresh keeps the pairs, and the dev-token route accepts them too. Under the Raft store the field is metadata version 5: an exchange with pairs is a 409 until every member reports it.

  • Delivered events and group records can carry the record’s append time, and a client can look up an offset by time (#975). A subscriber that offers FLAG_EVENT_BATCH_TIMESTAMPS (0x4000) gets a u64 of microseconds before each event on batches from a durable stream, the same live and on replay. A broker advertising FEATURE_RECORD_TIMESTAMPS (0x400_0000) answers offset_for_time with offset_value: the first offset on a shard appended at or after a time, found by the binary search the Kafka listener’s ListOffsets now shares through StreamLog::offset_for_time. A client that offers the feature gets timestamp_micros on group records. Nobody else’s frames change, and no storage format changes. In felix-client, set ClientConfig::timestamps and read Event::timestamp_micros; call Client::offset_for_time and subscribe at the answer. Times are the clock of the broker that led when the record was written, and replicas keep them (#1045). Breaking for callers of felix-broker’s StreamLog::begin_append, begin_append_marked and continue_batch, which now take the batch’s timestamp_micros; felix-wire’s EventBatchMeta, EventBatch, SharedEventBatch and GroupRecord gain a field.

  • felix-loadgen --scenario subscribe (#980): subscribers only, at the live tail, counting deliveries for --duration-secs while another generator publishes. --fanout subscriptions are spread over --concurrency cluster clients, every shard of a sharded stream is read, and the run reports delivered events and delivered_throughput_msg_s, offset gaps (records dropped) and, with --stamp-send-time, delivery latency. It reports no publish throughput, since it publishes nothing. ingest --stamp-send-time writes the wall-clock send time into each payload’s first 8 bytes for it; that latency compares two machines’ clocks, so it needs them in sync.

  • Nightly builds. nightly.yml builds the newest green commit on main through release.yml once a day, smoke-tests the images with a publish and subscribe, and publishes to GitHub only: images as nightly and nightly-YYYYMMDD on GHCR, and the binaries, wheels and Node addons on a rolling nightly pre-release. release.yml can now be called as a reusable workflow.

  • Delivered events can say who published them (#1010). A subscriber that offers FLAG_EVENT_BATCH_PUBLISHER (0x2000) gets the principal (the token’s sub) on each event batch, and a group consumer that offers FEATURE_GROUP_PUBLISHER (0x80_0000) gets it as publisher on each record. Nobody else’s frames change. In felix-client, set ClientConfig::publishers and read Event::publisher. A durable stream stores the principal with each record (storage format v6, a byte-length trailer on the body), and replication ships it, so replay, promoted replicas and consumer groups report the same value. Storing it is off until enabled: finalize the publisher_principal fleet feature in a cluster, or set FELIX_RECORD_PUBLISHERS=true on a single broker. Either is one-way for the data directory, since an older broker cannot open a v6 segment.

  • A consumer group has a lifecycle. group_seek moves a group’s cursor on one shard to earliest, latest or an offset, backwards or forwards; with if_new it creates the group there and leaves an existing one alone. group_describe reports the committed cursor, the shard’s committed tail, and what the group has in flight and owed; group_delete removes the cursor and dead letters. Advertised as FEATURE_GROUP_ADMIN (0x100_0000). Seek and delete need group.manage, describe needs group.consume. A seek voids the claims standing when it lands, so a late ack cannot move the new cursor or finish a record the group now owes. felix-client adds group_create, group_seek, group_describe and group_delete on Client and ClusterClient, and ClusterClient::group_*_stream to make the call on every shard of a stream. (#973)

  • Cache shards are fenced on promotion like stream shards. A promoted cache leader fences a majority of the replica set on the cache log, then on the counter log, takes the furthest ahead of each by (last generation, length), and writes a generation-start record on each before it serves; with generation_start finalized, the cache and counter marks count only past their own log’s record. A new internal capability, CACHE_FENCE (1 << 4), says a peer answers Fence and ReplicateFetch for the counter log and reads labelled cache and counter batches; a cache shard is fenced only when every replica offers it, and opens on the lease otherwise. A failover of a Quorum cache keeps its replica set, the dead leader in it, as a Quorum stream’s does. The new fenced_caches fleet feature, finalized with majority_ack and generation_start, has replicated Quorum caches acknowledge puts, deletes and counter adds on their followers’ answers without the report or the lease, and never open a promoted cache shard on the lease. Upgrade the control plane before finalizing it. Model-checked in FelixShardFencedCache.cfg. (#933)

  • Conditional cache writes. cache_put_if stores a value only if the key is absent or at a given version, and cache_delete_if removes it only at a given version; both answer cache_condition_result with whether the write was made and the key’s version. A get’s cache_value carries the version for a client that offered the bit. The condition check and the write are atomic per key, including against a write to the key still waiting on its fsync. Negotiated as FEATURE_CACHE_CONDITIONAL (0x200_0000); forwarded between brokers as a new internal kind that an older owner refuses rather than applying unconditionally. felix-client and ClusterClient gain cache_put_if, cache_delete_if and cache_get_versioned. (#976)

  • Containers run under Docker or Podman. scripts/container_engine.sh picks the engine (CONTAINER_ENGINE, else whichever of docker and podman answers) for task test, task coverage, task pg:* and task tla:check; the kcat tests read the same variable and run host.containers.internal under Podman. Dockerfiles, the control plane’s compose file and the docs name images fully qualified (docker.io/library/...). A new docs page, Docker or Podman, covers setup and the differences. felix_cluster::ports::docker_host is now felix_cluster::container::host.
  • felix_replication::promotion::Outcome::Pending is a struct variant carrying why and took, the replicas that took the fence in that attempt, and driver::Published gains status. ShardLifecycle::open_failed takes the error. felix_wire::KNOWN_FEATURES_HI is now FEATURE_INSPECT, so felix-client sets FEATURE_EXTENDED and sends client_features_hi in its auth; an older broker ignores the field.
  • Every change of a shard’s leader is fenced, not only a promotion (part of #1009). A move’s cut-over, a failover that names a move’s destination, a cancelled move’s hand-back, and a generation of a shard its leader serves that skips one (a promotion elsewhere that the broker only saw coalesced away) now wait in fencing until a majority of the replica set takes the new generation, and take the answer furthest ahead, before they serve. The generation right after one the leader serves (a move’s staging or a follower replacement step) still opens at once: nobody led in between. Each was a generation the control plane picked from its own view, opened at once; a leader it did not know about, from a second planner or a later replica election, could keep writing beside it. A write that reaches the broker while it fences is held, within FELIX_SHARD_MOVE_HOLD_MS, rather than refused. The lease fallback is unchanged: when a replica does not offer the fence the shard opens on the lease as before. felix_broker_promotions_opened_total{path} now counts every new leadership, not only promotions. The TLA+ model gains FenceEveryChange, with FelixShardElectHandoff.cfg (passes; an hour, so nightly), FelixShardElectHandoffLeaders.cfg (the same without a write, per PR) and FelixShardElectHandoffUnfenced.cfg (an unfenced cut-over opens a second leader at an elected generation). The CI model check now runs on seven jobs, filled by each configuration’s measured time, to stay under the hour.
  • A publish acknowledged on commit is answered by the task that sees it commit (#926). The commit task, or the executor for an in-memory write, puts the ack straight onto the control stream’s writer queue, instead of completing a oneshot that a per-stream ack-waiter task awaited under a per-publish timer. The ack timeout is now one deadline sweep per control stream, and the writer sends every answer already queued in one write. An ack is still never sent before the publish is durable under the stream’s fsync mode, a pipelining client still gets answers in request order, and a failed, dropped or timed-out publish still gets its error. Not yet measured; numbers come from an Azure nats-latency.sh run. The test-only subscriber event_writer is removed; its byte-cap test moved onto the lane feeder. felix_broker_ack_waiter_queue_full_total is gone with the waiter queue.
  • A plain Client spreads one stream’s shards over its connections, as a ClusterClient already did (#724). Its publishers learn a stream’s width from the broker on the first keyed publish to it (StreamShards, one round trip, kept for the client’s life) and put each shard on a publish stream of its own, placed on the least-loaded connection, so one client’s hot multi-shard stream no longer rides one connection and one listener. Order is kept per shard: every publish a client makes to one shard still goes through one writer and one QUIC stream. Unkeyed publishes are shard 0, so a stream published without keys, or a single-shard stream, stays on one stream for total order. A stream whose width cannot be learned keeps all its keyed publishes on shard 0’s stream. The writer for a key always comes from the width the client keeps, never from a caller’s own shard, so plain publishes, a ClusterClient and an idempotent producer through one client share a key’s writer even if they disagree about the width. The kept width is dropped when a publish is refused with not_found, so a stream deleted and created with another width is routed by its new width. A ClusterClient now asks each broker it publishes keyed records through for the stream’s width once. Unkeyed publishes through a plain Client move from a pooled stream to shard 0’s own, which costs one stream open per stream on the first publish. FELIX_PUB_SHARD_STREAMS now applies to plain clients too; 0 restores the old routing.
  • A follower stores the leader’s append time with each record it replicates, instead of stamping it with its own clock, so after a failover the new leader reports the same record times, and offset_for_time gives the same answer, as before (#1045). Brokers offer the new RECORD_TIMES peer capability (1 << 5); a leader sends a follower that offered it ReplicateTimedRecords (internal kind 36), a labelled batch with one time per record, and a promoted leader taking a replica’s tail asks with ReplicateTimedFetch (kind 37). An older follower is sent what it reads, and a newer follower receiving from an older leader uses its own clock, as before. Breaking for callers of felix-broker’s StreamLog::begin_append_marked_at and replication::apply, which take the records’ times (empty for this broker’s clock), and of felix-wire’s ReplicateRecords and ReplicateFetch, which gain times and timed.
  • Breaking for Rust callers (#1017): felix_wire::Message::Commit has an expected_offset field, and PublishRefusalReason and felix_broker::BrokerError have new variants, so struct literals and exhaustive matches need updating.
  • A durable stream no longer reserves a whole 256 MiB segment of disk per shard when it is created (#1016). The active segment reserves 1 MiB (or a sixteenth of the segment size, if smaller) and doubles the reservation each time its records pass half of it, up to the segment size. Each extension runs on a blocking thread after the append that earned it and is best effort: a failure is logged, counted in felix_storage_segment_reserve_failed_total, and never fails an append. Reserving still leaves the file size alone, so recovery is unchanged. SegmentWriter::reopen takes a reservation limit.
  • Creating a stream or cache that already exists with the same configuration answers 200 with the existing one instead of 409 (#967). A different configuration under the same name is still 409. A stream’s routing only counts when the request names it. A caller that took 409 to mean “already there” should accept 200 as well.
  • Breaking, Rust API: Broker::claim_publish, publish_batch_with_outcome, claim_batch_idempotent and commit_to_handle take the publisher; ResumedSubscription::backlog is a Vec<RingRecord>; AppendRecord and LogRecord have a publisher field; StreamLog’s append methods and replication::apply take per-record publishers; ReplicateRecords has publishers and batch_checksum covers them.
  • A compacted cache log can no longer be read by an older broker. Compaction now writes each copied value as a version 2 cache record, which carries the value’s original version; a broker from before this change refuses those records as corruption, so a node cannot be downgraded once it has compacted a cache. Records a client writes are unchanged. (#976)
  • felix_storage::StorageApi has three new required methods (put_if, delete_if, get_versioned), and CacheOp::Put a version field. (#976)
  • Breaking, Rust API: IdempotentProducer owns its client and has no lifetime parameter. Client::idempotent_producer and ClusterClient::idempotent_producer take self: &Arc<Self>, so wrap the client in an Arc first. The producer can now be stored or moved into a task. (#977)
  • A request the broker refuses and then closes the stream for, such as a group describe without permission, reaches the client as the broker’s coded error (#1078). The broker’s response writer could see the stream’s cancellation before the queued error and finish the stream without it, so the client reported “the broker closed the group stream” instead of forbidden.
  • A broker notices its lease lapse within a sixteenth of the lease the control plane hands out, not of the 10s placeholder it starts on (#1071). The refresh that clears the cached admission flag, sets felix_broker_lease_held and ends a deposed leader’s readers was paced once, at startup, so under a 1s lease it ran every 470ms instead of every 47ms. The commit-time lease check was always exact; this was admission and the metric running late. It also made the cluster harness’s clock faults land late, since that refresh is what reads the clock on an idle broker, which failed a_frozen_leader_is_refused_by_the_majority_its_successor_fenced.
  • A write that reaches a shard’s new owner while it is still opening the shard, or just after its fence settles but before it publishes that it serves, is held until it serves instead of being refused with shard_unavailable / not_ready (#1085). Publishes, cache writes and counter adds were all exposed; a move’s cut-over showed it as a refused counter add.
  • The Rust client’s cache worker keeps its stream after the broker refuses a cache or counter request (#1085). It used to stop on any error, so one refusal turned the requests that later landed on that worker into cache worker closed.
  • A Quorum publish no longer waits out an unreachable follower when the other follower already holds the record but is still answering an earlier replication pass (#1080). The pass waiting for a majority had shipped only to the unreachable one, and the busy follower’s answer reached the driver, not that pass, so the mark stayed put until the peer request timed out (5 s by default, past most publish timeouts). The answer now ends that wait and the next pass counts it.
  • felix_broker_shard_phase reports fencing. The gauge left the phase out, so a promoted shard waiting for its fence was counted in no phase.
  • A broker accepts a token carrying node.view or node.manage. Neither was an action it knew, so an operator token that also served the control plane failed to authenticate there.
  • The docs no longer list DropOld as a working overflow policy (#1019). It is accepted at the broker, writer-lane and client stages but behaves as DropNew at each: the arriving batch is dropped, not the oldest. The status table, configuration reference and client docs now say so.
  • JSON numbers survive a decode and re-encode exactly. serde_json’s default float parser could land a long literal one ulp off, so an extension body the broker passed on carried a different number. Nightly fuzzing found it.
  • Dropping an IdempotentProducer publish no longer ends the producer. The producer sends from a task of its own and a call waits on it, so a call dropped by a timeout, a select! or a hung-up handler still runs to its answer and the next batch goes out under the next sequence. A batch left in doubt with nobody waiting is re-sent by the producer, under its sequence, before the next batch on that shard. Dropping the producer lets what it was handed finish; IdempotentProducer::close waits for it. (#977)
  • The history campaign could report a record delivered past the end of the final log (lost-delivery). Since #977 a dropped producer still sends what it was handed, so an append recorded as unknown could land after the final read. A stopping client now closes its producer and finishes its publishes before the final read is taken. IdempotentProducer::close_within(timeout) is new: close, but the producer’s task is stopped at the timeout, so nothing it held is sent later. (#1075)
  • felixctl sub, cache get and cache watch escape binary payloads on a terminal (\x00, \u{85}) instead of writing raw bytes that garble it; piped output is unchanged. felixctl bench latency payloads showed it.
  • A log-backed cache’s expiry could delete a value a put had just refreshed, when the put was waiting on its fsync as the expiry was written. The expiry now waits for a write to the key in flight before it checks. (#976)
  • felix-cluster up held for more than an hour lost every control-plane call, teardown included, because its tokens and the brokers’ node tokens were minted once with a one-hour life. The harness now mints tokens on use, rewrites each broker’s node token file and the session file every half hour. Cluster’s client_token, admin_token, operator_token, subscribe_only_token and group_operator_token fields are now methods, and ControlPlane’s token methods moved to felix_cluster::Credentials.
  • A cache follower that dropped a divergent suffix of its cache log, or of its counter log, kept the in-memory index or counter sums built from the dropped records, so after a promotion a key could read the value of whatever record replaced its offset. Both are now rebuilt from the log as it is. (#933)
  • Leader mode’s single writer is stated as resting on the lease: which faults let a deposed Leader leader acknowledge a write its successor never sees (a pause past the lease margins, a broker clock running slow against the control plane’s), and that a clock step and Quorum with majority_ack are not exposed. A cluster test pins that a Leader stream refuses writes on a lapsed lease after majority_ack is finalized. (#1008)
  • A publish acknowledged on enqueue (a Leader stream with ack_on_commit off, the default) is not readable until it is written, so a group poll, history read or replay sent right after the ack can miss it. Stated in the semantics and queues pages and on Client::group_poll. The end-to-end group tests in cache_durability now publish with commit acks; they assumed otherwise and failed intermittently. (#1025)
  • Corrected four replication claims. Counter adds on a Quorum cache wait for a majority like puts and deletes; the log-backed cache stays Partial for its in-memory key index (#1073). A Quorum publish with no reachable majority is not refused up front: it waits, then fails with quorum_timeout. Under majority_ack a move’s destination cut off from its followers at the cut-over waits for a majority instead of opening on the lease. A promoted leader’s fence retries back off from 200 ms to 2 s. (#1074)