Felix 0.6.0-preview.3
Released 2026-10-08 as a preview. The full history is in CHANGELOG.md.
The third preview of 0.6.0. felixctl now manages a cluster as well as using
it: it creates, changes and deletes tenants, namespaces, streams and caches,
works consumer groups, counters and RBAC rules, and shows a shard’s or a
subscriber’s live state with felixctl inspect. It installs with Homebrew.
Every change of a shard’s leader is fenced, and replicas keep a ballot with
each generation they accept. Sealed segments can be copied to an object store
before retention deletes them, the first step toward tiered storage.
felix-capi adds a C ABI over the Rust client, the base for Go and C# SDKs.
Caches gain conditional writes and key-scoped grants, and a client can read a
bounded range of a stream. Development works with Docker or Podman. Several
move, failover and lease-timing bugs are fixed. The Rust API has breaking
changes, listed under Changed. pip will not install it without --pre.
-
A Homebrew formula for felixctl:
brew install getfelix/tap/felixctlon macOS and Linux, with completions and man pages (#1005).release.ymlwrites it from each release’s archive checksums and pushes it to<owner>/homebrew-tapwhenPUBLISH_HOMEBREWistrue, using theHOMEBREW_TAP_DEPLOY_KEYsecret (a write deploy key on the tap); dry runs keep it as an artifact and nightlies skip it. -
felixctl inspect subsand thesubscriptions_listrequest (part of #1077). A broker advertisingFEATURE_INSPECTnow also answerssubscriptions_listwithsubscriptions_list_info: one page of the subscriptions it serves, each with its stream and shard, subscriber and subscription ids, connection id, client address and principal, overflow policy, queue depth and capacity in batches, records dropped, position and the shard’s tail. The filter narrows by tenant, namespace, stream, shard, principal, or to subscriptions that have dropped records. A page is 100 by default and at most 1000, and a cursor of shard and subscriber id continues it. It needsnode.view:cluster:*. The broker reads each shard’s fanout snapshot and never touches a queue. On the delivery path, a subscriber’s receiver now stores one past the offset of each batch it takes (a relaxed store), and a drop adds to a per-subscriber count beside the existing metric. felixctl asks every broker, or only--node, and prints one table or one JSON line per broker, with--principal,--dropping,--limitand--cursor. felix-broker addsBroker::list_subscriptions,SubscriberStats,SubscriberOwnerandSubscription::set_owner; felix-client addsClient::list_subscriptionsandSubscriptionsPage, and its inspect calls now report a broker’sunsupportedanswer as such. Diagnosing a cluster covers a subscriber missing records and one falling behind. -
Stream logs can copy their sealed segments to an object store before retention deletes them, the first part of tiered storage (#172). Off by default;
FELIX_DURABLE_OFFLOAD_DIRturns it on, and the only backend is a directory, through theobject_storecrate’s local filesystem backend. A pass on the retention timer uploads each sealed segment, reads it back to check size and CRC-32, and records it in a per-shardoffload.manifest, fsynced. Retention then deletes only recorded segments, so a crash at any step leaves the local segment or a recorded copy. Recovery accepts a gap in the local chain when the manifest covers it. Nothing reads the copies yet: a read below the local head is stillTrimmed. New metricsfelix_storage_offload_segments_total,felix_storage_offload_bytes_totalandfelix_storage_offload_failures_total. Breaking for code that builds felix-storage’sLogConfigwithout..Default::default(): it gainsoffload. -
felixctl inspect shardand theshard_inspectrequest (part of #1077). A broker advertisingFEATURE_INSPECT, the first bit of the extended feature word (server_features_hi0x1), answersshard_inspectwithshard_inspect_info: its own view of one shard, with its phase and generation, whether it serves and why not (opening,fencing,failedwith the open error,draining,lease_lapsed,not_assigned_here,behind_generation), which replicas took a promoted leader’s fence and how many attempts it has made, its lease, tail and commit mark, and as leader each follower’s next offset, lag and state. It needsnode.view:cluster:*and may name any tenant. The answer is read from snapshots the lifecycle and the replication driver publish (the newfelix_replication::status::ShardStatusBoard), so it takes neither’s lock and opens no log. felixctl asks the leader and every replica and prints a table, or one JSON line per shard with--json; a broker that cannot be reached is listed as unreachable. felix-client addsClient::inspect_shardandClient::supports_inspect. A new docs page, Diagnosing a cluster, goes through shard, replication, subscriber, auth and startup problems by symptom. -
felix-capi, a C ABI over the Rust client and the base for the Go and C# SDKs (part of #618). It buildslibfelixas a shared and a static library with a cbindgen header checked in atcrates/sdk/felix-capi/include/felix.h, and covers connect, publish and a polled subscribe (felix_client_connect,felix_client_publish,felix_client_subscribe,felix_subscription_next_event). Handles are opaque with a free function each, every call returns a status code whose classes match the other SDKs, the message for a failure is per thread (felix_last_error_message), panics never cross the boundary, and each client owns its Tokio runtime. A test fails while the header is stale, and CI runs a C program against the conformance fixture. Not published to crates.io. -
Replicas keep a ballot with each accepted generation (part of #1009): the leader they accepted it from, by the node id its
Hellogave. At that generation a replica refuses a fence, a batch, a bootstrap, a rebuild or a tail fetch from any other node withFencedEpoch, and a broker will not lead a generation it already accepted from another node. The ballot is a newballotfile in the shard directory, fsynced before anything from that leader is answered and read back on open; builds without ballots ignore it, so rolling back still opens the shard. A new internal capability,BALLOTS(1 << 6), is offered with the fence, andFELIX_INTERNAL_FENCE=falseturns ballots off with it. Nothing changes while the control plane names every leader, since it never names two at one generation; this is the safety layer replica elections will need. The TLA+ model gainsBallots,ElectionsandOneLeaderPerGeneration, withFelixShardElect.cfgand two configurations that must fail:FelixShardElectNoBallot.cfg(two leaders at one generation) andFelixShardElectStaleSet.cfg(a replica standing on a set it has left). Breaking for callers of felix-storage’sDiskLog::accept_generationand felix-broker’sStreamLog::accept_generation, which take the leader, of felix-replication’sReplicaHandlerentry points, which take the sender, and for code matchingGenerationCheck, which gainsPromisedand is no longerCopy. -
felixctl groupandfelixctl counter(#1005).group create|describe|seek|rmmanage a consumer group on every shard of a stream, or on--shard.group pollclaims records and prints each with its claim,SHARD:OFFSET:ATTEMPTS, whichgroup ack,nack(with--delay-ms),extendanddead-letters addtake.group dead-letters ls|redrive|discardwork the dead-letter list.group rm,dead-letters discardand aseekthat moves a cursor back over finished records ask[y/N]at a terminal and need--yeselsewhere.counter get|addread and add to counters. All go throughClusterClient. felix-client now re-exportsGroupRecord, whichgroup_extendtakes. -
Nightly TLC simulation of long random walks over replica-set changes (#934).
FelixShardWalkSpares,FelixShardWalkMovesandFelixShardWalkHandofflift the bounds the exhaustive configurations stop at (time to 40, six writes, six moves) and sample traces up to 300 steps; each has a negative twin that must find its violation within the budget.scripts/check_tla.sh --simulate(task tla:walk) runs them, andtla-walk.ymlruns them nightly and uploads TLC’s trace on a failure.TickinFelixShard.tlaenumerates clocks over the drift window only, with the same states. -
felixctl writes to the control plane (part of #1005).
tenant,namespace,streamandcachegaincreateandrm;stream setchanges consistency, delivery, durability and retention, keeping a retention bound not given, andcache setthe display name; each prints what changed.node drainandnode deregister,shard move NAME SHARD --to NODE(with--dry-runand--cache) andshard move cancel, andplacement pause,resumeandabandondrive placement. Deletes, drains and deregistrations ask on a terminal and need--yesanywhere else, stopping with status 2 without it;placement abandonnever asks and always needs--yes. -
felixctl rbac policy ls|add|rmandfelixctl rbac grouping ls|add|rmlist, grant and revoke the current tenant’s RBAC policies and role assignments through the control plane (#1005).policy addchecks the object against the RBAC grammar first, including cache key and key-prefix objects and theircache.read/cache.write-only rule, and stops with status 2 and the reason; action names and delegation scope are left to the control plane, whose refusal is printed with status 4.rmasks on a terminal and needs--yeselsewhere, and exits 5 when the rule does not exist.felix-clusteraddsCredentials::rbac_admin_token. -
A subscriber can choose its own broker-side queue capacity on a broker advertising
FEATURE_SUBSCRIBE_QUEUE(0x4000_0000) (#1019).subscribetakes an optionalqueue_capacity, counted in published batches; the broker clamps it to1..=FELIX_SUBSCRIBER_QUEUE_CAPACITY_MAX(new, default 4096) and echoes the grant asqueue_capacityonsubscribed. A subscribe without it, and the answer, are byte-identical to before. The overflow policy stays the stream’s: a subscriber that could chooseBlockcould stall every publisher on its shard. felix-client addsClientConfig::broker_sub_queue_capacityandSubscription::queue_capacity; felix-broker addsBroker::subscribe_sizedandBroker::subscribe_from_sized.ClientConfiggains a public field, so code building it with a struct literal must set it. -
One Rust
Clientcan act for many users over the same connections (#969).Client::with_identity(tenant_id, token_provider)andClient::with_identity_tokenreturn a client that shares the parent’s connections and authenticates every stream it opens with the user’s token, so the broker checks each user’s publish, subscribe, cache and group requests against that user’s grants alone. It opens one publish and one cache stream and no connections. One user’s token expiring or being revoked stops only that user’s streams. No wire change: the broker already authenticates per stream. A refused publish or cache request still ends the stream it came on, so a gateway should build a new identity after one.ClusterClientdoes not offer it yet. -
A write can be made only at the offset its writer expects, on a broker advertising
FEATURE_PUBLISH_CONDITIONAL(0x1000_0000) (#1017).publish_ifappends a batch only if it would start atexpected_offset, the shard’s next;committakes the same optionalexpected_offsetas a compare-and-set on the whole shard. The check is made where the log assigns offsets, under the same lock, so of two writers at one offset exactly one is written. A refusal ispublish_refusedwith the newoffset_mismatchreason and the shard’s tail; it writes nothing, consumes no offset and holds up no later publish. The tail counts a new leader’s generation-start record, so a writer’s expected offset goes stale on failover. Served by the shard’s leader only: a non-leader answersnot_leader. An in-memory stream refuses it. A commit withoutexpected_offsetis byte-identical to before.publish_ifis charged against the tenant’s publish quota and the ingress byte budgets like any acked publish, and refused over quota the same way. felix-client addspublish_ifandcommit_ifonClientandClusterClient, answeringConditionalWrite; felix-storage addsDiskLog::append_claimed_at, and felix-brokerBroker::claim_publish_at,publish_batch_atandcommit_to_handle_at. Per-key version preconditions are #1051. -
A second feature word for when the
u32feature set runs out (#1055).FEATURE_EXTENDED(0x8000_0000) inclient_featuresorserver_featuressays the peer sends and readsclient_features_hi/server_features_hi. A peer sends the marker and the word only when it knows a feature there, and a broker answers with its word only to a client that set the marker, so every existing frame is unchanged. No feature uses the word yet. felix-wire addsFEATURE_EXTENDED,KNOWN_FEATURES_HI,offer_features,answer_featuresandpeer_features_hi. Breaking for code that buildsMessage::AuthorMessage::AuthOk: both have a new field. -
A client can read a bounded range of a durable stream and stop (#1018). A broker advertising
FEATURE_STREAM_READ(0x2000_0000) answersstream_read(shard,from, optionalend,max_records,max_bytes) withstream_records: one page of committed records and thenext_offsetto continue from, which steps over generation-start records. No subscriber is registered and the read never waits. A page stops at the committed mark on aQuorumshard and, underFsyncMode::OnCommit, at the durable offset, and is capped atFELIX_DURABLE_MAX_RECORDS_PER_READrecords and 4 MiB of payload. Afrombelow retention or past the tail getssubscribe_cursor_error. Needsstream.subscribe; only the shard’s leader answers. In felix-client,Client::readandClusterClient::readreturn aStreamPage; felix-broker hasBroker::read_range. Frames to clients that do not send the request are unchanged. AnendonSubscribeis tracked in #1053. -
A consumer can manage its own claims on a broker advertising
FEATURE_GROUP_CLAIM_CONTROL(0x800_0000) (#974).group_extendkeeps a claim standing while the work goes on, answered withgroup_extended; it names the delivery byoffsetandattempts, so once the claim lapses and the record goes out again it is refused withstale_claim.delay_msongroup_nackmakes the record owed only after a delay, holding a place under the in-flight cap until then.group_dead_letterlists a record as a dead letter and finishes it, written to the dead-letter log first as when the broker gives up, sogroup_redriveworks on it.visibility_msongroup_pollsets how long that poll’s claims stand. All needgroup.consume. Durations are capped by the newFELIX_GROUP_MAX_VISIBILITY_MS(12 hours, never below the visibility timeout). Both new fields are left out at zero, so nacks and polls that do not use them are byte-identical to before. A group’s in-flight state is no longer evicted while a claim stands. felix-client addsgroup_poll_withandGroupPollOptions,group_extend,group_nack_after,group_dead_letterandsupports_group_claim_controlonClientandClusterClient, andextend,nack_afteranddead_letteronShardedGroup. Breaking for callers of felix-broker’sGroupReader::poll_below, which takes the claims’ visibility, and for code matching felix-wire’sGroupPollandGroupNack, which gain fields. -
Streams and caches can be created together, all or none (#967).
POST /v1/tenants/{t}/namespaces/{ns}/resourcestakes up to 256 stream and cache create bodies and writes the new ones in one Postgres transaction or one Raft log entry. A missing permission, an invalid item, a name given twice, or an item that already exists with a different configuration fails the whole batch and creates nothing. An item that already exists as asked is reportedunchanged, so a batch can be sent again after a failure. Under Raft this is metadata level 5; the endpoint answers503until every control-plane member runs a release that has it. -
Cache grants can name a key or a key prefix (#966). A fourth segment on a cache object,
cache:{tenant}/{ns}/{cache}/{key}or.../{prefix}*, limitscache.readorcache.writeto that key, or to keys starting with the prefix (a plain string prefix:user:1*coversuser:10). The broker checks get, put, delete, conditional writes, counters and key watches against the key, a prefix watch against its prefix, and a forwarded op again at the owner. Grants without a key segment still cover the whole cache. The control plane refuses a key with*anywhere but the end, a bare*, a wildcard namespace or cache under a key, and any other action on a key object; token exchange narrows a whole-cache grant to a key hint. No wire change. Mixed versions fail closed: an older broker refuses requests a key grant would allow.felix-authzaddsPermissionMatcher::allows_cache_keysandCacheKeys; the control plane’sParsedObject::Cachegains akeyfield. -
Token exchange can narrow by action and resource pairs (#968).
permissions: ["stream.subscribe:stream:t1/ns/a", "stream.publish:stream:t1/ns/b"]narrows each pair on its own against the grants with exactly that action, whererequestedandresourcesapply every action to every resource. It never widens what RBAC grants. It must be sent with"requested": [](a400otherwise), so a control plane that predates the field refuses rather than minting full rights, and the refresh record stores the same[]for the same reason. Refresh keeps the pairs, and the dev-token route accepts them too. Under the Raft store the field is metadata version 5: an exchange with pairs is a409until every member reports it. -
Delivered events and group records can carry the record’s append time, and a client can look up an offset by time (#975). A subscriber that offers
FLAG_EVENT_BATCH_TIMESTAMPS(0x4000) gets au64of microseconds before each event on batches from a durable stream, the same live and on replay. A broker advertisingFEATURE_RECORD_TIMESTAMPS(0x400_0000) answersoffset_for_timewithoffset_value: the first offset on a shard appended at or after a time, found by the binary search the Kafka listener’sListOffsetsnow shares throughStreamLog::offset_for_time. A client that offers the feature getstimestamp_microson group records. Nobody else’s frames change, and no storage format changes. In felix-client, setClientConfig::timestampsand readEvent::timestamp_micros; callClient::offset_for_timeand subscribe at the answer. Times are the clock of the broker that led when the record was written, and replicas keep them (#1045). Breaking for callers of felix-broker’sStreamLog::begin_append,begin_append_markedandcontinue_batch, which now take the batch’stimestamp_micros; felix-wire’sEventBatchMeta,EventBatch,SharedEventBatchandGroupRecordgain a field. -
felix-loadgen --scenario subscribe(#980): subscribers only, at the live tail, counting deliveries for--duration-secswhile another generator publishes.--fanoutsubscriptions are spread over--concurrencycluster clients, every shard of a sharded stream is read, and the run reports delivered events anddelivered_throughput_msg_s, offset gaps (records dropped) and, with--stamp-send-time, delivery latency. It reports no publish throughput, since it publishes nothing.ingest --stamp-send-timewrites the wall-clock send time into each payload’s first 8 bytes for it; that latency compares two machines’ clocks, so it needs them in sync. -
Nightly builds.
nightly.ymlbuilds the newest green commit on main throughrelease.ymlonce a day, smoke-tests the images with a publish and subscribe, and publishes to GitHub only: images asnightlyandnightly-YYYYMMDDon GHCR, and the binaries, wheels and Node addons on a rollingnightlypre-release.release.ymlcan now be called as a reusable workflow. -
Delivered events can say who published them (#1010). A subscriber that offers
FLAG_EVENT_BATCH_PUBLISHER(0x2000) gets the principal (the token’ssub) on each event batch, and a group consumer that offersFEATURE_GROUP_PUBLISHER(0x80_0000) gets it aspublisheron each record. Nobody else’s frames change. In felix-client, setClientConfig::publishersand readEvent::publisher. A durable stream stores the principal with each record (storage format v6, a byte-length trailer on the body), and replication ships it, so replay, promoted replicas and consumer groups report the same value. Storing it is off until enabled: finalize thepublisher_principalfleet feature in a cluster, or setFELIX_RECORD_PUBLISHERS=trueon a single broker. Either is one-way for the data directory, since an older broker cannot open a v6 segment. -
A consumer group has a lifecycle.
group_seekmoves a group’s cursor on one shard toearliest,latestor an offset, backwards or forwards; withif_newit creates the group there and leaves an existing one alone.group_describereports the committed cursor, the shard’s committed tail, and what the group has in flight and owed;group_deleteremoves the cursor and dead letters. Advertised asFEATURE_GROUP_ADMIN(0x100_0000). Seek and delete needgroup.manage, describe needsgroup.consume. A seek voids the claims standing when it lands, so a late ack cannot move the new cursor or finish a record the group now owes. felix-client addsgroup_create,group_seek,group_describeandgroup_deleteonClientandClusterClient, andClusterClient::group_*_streamto make the call on every shard of a stream. (#973) -
Cache shards are fenced on promotion like stream shards. A promoted cache leader fences a majority of the replica set on the cache log, then on the counter log, takes the furthest ahead of each by (last generation, length), and writes a generation-start record on each before it serves; with
generation_startfinalized, the cache and counter marks count only past their own log’s record. A new internal capability,CACHE_FENCE(1 << 4), says a peer answersFenceandReplicateFetchfor the counter log and reads labelled cache and counter batches; a cache shard is fenced only when every replica offers it, and opens on the lease otherwise. A failover of aQuorumcache keeps its replica set, the dead leader in it, as aQuorumstream’s does. The newfenced_cachesfleet feature, finalized withmajority_ackandgeneration_start, has replicatedQuorumcaches acknowledge puts, deletes and counter adds on their followers’ answers without the report or the lease, and never open a promoted cache shard on the lease. Upgrade the control plane before finalizing it. Model-checked inFelixShardFencedCache.cfg. (#933) -
Conditional cache writes.
cache_put_ifstores a value only if the key is absent or at a given version, andcache_delete_ifremoves it only at a given version; both answercache_condition_resultwith whether the write was made and the key’s version. A get’scache_valuecarries the version for a client that offered the bit. The condition check and the write are atomic per key, including against a write to the key still waiting on its fsync. Negotiated asFEATURE_CACHE_CONDITIONAL(0x200_0000); forwarded between brokers as a new internal kind that an older owner refuses rather than applying unconditionally. felix-client andClusterClientgaincache_put_if,cache_delete_ifandcache_get_versioned. (#976)
Changed
Section titled “Changed”- Containers run under Docker or Podman.
scripts/container_engine.shpicks the engine (CONTAINER_ENGINE, else whichever ofdockerandpodmananswers) fortask test,task coverage,task pg:*andtask tla:check; the kcat tests read the same variable and runhost.containers.internalunder Podman. Dockerfiles, the control plane’s compose file and the docs name images fully qualified (docker.io/library/...). A new docs page, Docker or Podman, covers setup and the differences.felix_cluster::ports::docker_hostis nowfelix_cluster::container::host. felix_replication::promotion::Outcome::Pendingis a struct variant carryingwhyandtook, the replicas that took the fence in that attempt, anddriver::Publishedgainsstatus.ShardLifecycle::open_failedtakes the error.felix_wire::KNOWN_FEATURES_HIis nowFEATURE_INSPECT, so felix-client setsFEATURE_EXTENDEDand sendsclient_features_hiin itsauth; an older broker ignores the field.- Every change of a shard’s leader is fenced, not only a promotion (part of
#1009). A move’s cut-over, a failover that names a move’s destination, a
cancelled move’s hand-back, and a generation of a shard its leader serves
that skips one (a promotion elsewhere that the broker only saw coalesced
away) now wait in
fencinguntil a majority of the replica set takes the new generation, and take the answer furthest ahead, before they serve. The generation right after one the leader serves (a move’s staging or a follower replacement step) still opens at once: nobody led in between. Each was a generation the control plane picked from its own view, opened at once; a leader it did not know about, from a second planner or a later replica election, could keep writing beside it. A write that reaches the broker while it fences is held, withinFELIX_SHARD_MOVE_HOLD_MS, rather than refused. The lease fallback is unchanged: when a replica does not offer the fence the shard opens on the lease as before.felix_broker_promotions_opened_total{path}now counts every new leadership, not only promotions. The TLA+ model gainsFenceEveryChange, withFelixShardElectHandoff.cfg(passes; an hour, so nightly),FelixShardElectHandoffLeaders.cfg(the same without a write, per PR) andFelixShardElectHandoffUnfenced.cfg(an unfenced cut-over opens a second leader at an elected generation). The CI model check now runs on seven jobs, filled by each configuration’s measured time, to stay under the hour. - A publish acknowledged on commit is answered by the task that sees it
commit (#926). The commit task, or the executor for an in-memory write, puts
the ack straight onto the control stream’s writer queue, instead of
completing a oneshot that a per-stream ack-waiter task awaited under a
per-publish timer. The ack timeout is now one deadline sweep per control
stream, and the writer sends every answer already queued in one write. An
ack is still never sent before the publish is durable under the stream’s
fsync mode, a pipelining client still gets answers in request order, and a
failed, dropped or timed-out publish still gets its error. Not yet measured;
numbers come from an Azure
nats-latency.shrun. The test-only subscriberevent_writeris removed; its byte-cap test moved onto the lane feeder.felix_broker_ack_waiter_queue_full_totalis gone with the waiter queue. - A plain
Clientspreads one stream’s shards over its connections, as aClusterClientalready did (#724). Its publishers learn a stream’s width from the broker on the first keyed publish to it (StreamShards, one round trip, kept for the client’s life) and put each shard on a publish stream of its own, placed on the least-loaded connection, so one client’s hot multi-shard stream no longer rides one connection and one listener. Order is kept per shard: every publish a client makes to one shard still goes through one writer and one QUIC stream. Unkeyed publishes are shard 0, so a stream published without keys, or a single-shard stream, stays on one stream for total order. A stream whose width cannot be learned keeps all its keyed publishes on shard 0’s stream. The writer for a key always comes from the width the client keeps, never from a caller’s own shard, so plain publishes, aClusterClientand an idempotent producer through one client share a key’s writer even if they disagree about the width. The kept width is dropped when a publish is refused withnot_found, so a stream deleted and created with another width is routed by its new width. AClusterClientnow asks each broker it publishes keyed records through for the stream’s width once. Unkeyed publishes through a plainClientmove from a pooled stream to shard 0’s own, which costs one stream open per stream on the first publish.FELIX_PUB_SHARD_STREAMSnow applies to plain clients too;0restores the old routing. - A follower stores the leader’s append time with each record it replicates,
instead of stamping it with its own clock, so after a failover the new
leader reports the same record times, and
offset_for_timegives the same answer, as before (#1045). Brokers offer the newRECORD_TIMESpeer capability (1 << 5); a leader sends a follower that offered itReplicateTimedRecords(internal kind 36), a labelled batch with one time per record, and a promoted leader taking a replica’s tail asks withReplicateTimedFetch(kind 37). An older follower is sent what it reads, and a newer follower receiving from an older leader uses its own clock, as before. Breaking for callers of felix-broker’sStreamLog::begin_append_marked_atandreplication::apply, which take the records’ times (empty for this broker’s clock), and of felix-wire’sReplicateRecordsandReplicateFetch, which gaintimesandtimed. - Breaking for Rust callers (#1017):
felix_wire::Message::Commithas anexpected_offsetfield, andPublishRefusalReasonandfelix_broker::BrokerErrorhave new variants, so struct literals and exhaustive matches need updating. - A durable stream no longer reserves a whole 256 MiB segment of disk per
shard when it is created (#1016). The active segment reserves 1 MiB (or a
sixteenth of the segment size, if smaller) and doubles the reservation each
time its records pass half of it, up to the segment size. Each extension runs
on a blocking thread after the append that earned it and is best effort: a
failure is logged, counted in
felix_storage_segment_reserve_failed_total, and never fails an append. Reserving still leaves the file size alone, so recovery is unchanged.SegmentWriter::reopentakes a reservation limit. - Creating a stream or cache that already exists with the same configuration
answers
200with the existing one instead of409(#967). A different configuration under the same name is still409. A stream’sroutingonly counts when the request names it. A caller that took409to mean “already there” should accept200as well. - Breaking, Rust API:
Broker::claim_publish,publish_batch_with_outcome,claim_batch_idempotentandcommit_to_handletake the publisher;ResumedSubscription::backlogis aVec<RingRecord>;AppendRecordandLogRecordhave apublisherfield;StreamLog’s append methods andreplication::applytake per-record publishers;ReplicateRecordshaspublishersandbatch_checksumcovers them. - A compacted cache log can no longer be read by an older broker. Compaction now writes each copied value as a version 2 cache record, which carries the value’s original version; a broker from before this change refuses those records as corruption, so a node cannot be downgraded once it has compacted a cache. Records a client writes are unchanged. (#976)
felix_storage::StorageApihas three new required methods (put_if,delete_if,get_versioned), andCacheOp::Putaversionfield. (#976)- Breaking, Rust API:
IdempotentProducerowns its client and has no lifetime parameter.Client::idempotent_producerandClusterClient::idempotent_producertakeself: &Arc<Self>, so wrap the client in anArcfirst. The producer can now be stored or moved into a task. (#977)
- A request the broker refuses and then closes the stream for, such as a
group describe without permission, reaches the client as the broker’s coded
error (#1078). The broker’s response writer could see the stream’s
cancellation before the queued error and finish the stream without it, so the
client reported “the broker closed the group stream” instead of
forbidden. - A broker notices its lease lapse within a sixteenth of the lease the control
plane hands out, not of the 10s placeholder it starts on (#1071). The refresh
that clears the cached admission flag, sets
felix_broker_lease_heldand ends a deposed leader’s readers was paced once, at startup, so under a 1s lease it ran every 470ms instead of every 47ms. The commit-time lease check was always exact; this was admission and the metric running late. It also made the cluster harness’s clock faults land late, since that refresh is what reads the clock on an idle broker, which faileda_frozen_leader_is_refused_by_the_majority_its_successor_fenced. - A write that reaches a shard’s new owner while it is still opening the
shard, or just after its fence settles but before it publishes that it
serves, is held until it serves instead of being refused with
shard_unavailable/not_ready(#1085). Publishes, cache writes and counter adds were all exposed; a move’s cut-over showed it as a refused counter add. - The Rust client’s cache worker keeps its stream after the broker refuses a
cache or counter request (#1085). It used to stop on any error, so one
refusal turned the requests that later landed on that worker into
cache worker closed. - A
Quorumpublish no longer waits out an unreachable follower when the other follower already holds the record but is still answering an earlier replication pass (#1080). The pass waiting for a majority had shipped only to the unreachable one, and the busy follower’s answer reached the driver, not that pass, so the mark stayed put until the peer request timed out (5 s by default, past most publish timeouts). The answer now ends that wait and the next pass counts it. felix_broker_shard_phasereportsfencing. The gauge left the phase out, so a promoted shard waiting for its fence was counted in no phase.- A broker accepts a token carrying
node.viewornode.manage. Neither was an action it knew, so an operator token that also served the control plane failed to authenticate there. - The docs no longer list
DropOldas a working overflow policy (#1019). It is accepted at the broker, writer-lane and client stages but behaves asDropNewat each: the arriving batch is dropped, not the oldest. The status table, configuration reference and client docs now say so. - JSON numbers survive a decode and re-encode exactly. serde_json’s default float parser could land a long literal one ulp off, so an extension body the broker passed on carried a different number. Nightly fuzzing found it.
- Dropping an
IdempotentProducerpublish no longer ends the producer. The producer sends from a task of its own and a call waits on it, so a call dropped by a timeout, aselect!or a hung-up handler still runs to its answer and the next batch goes out under the next sequence. A batch left in doubt with nobody waiting is re-sent by the producer, under its sequence, before the next batch on that shard. Dropping the producer lets what it was handed finish;IdempotentProducer::closewaits for it. (#977) - The history campaign could report a record delivered past the end of the
final log (
lost-delivery). Since #977 a dropped producer still sends what it was handed, so an append recorded as unknown could land after the final read. A stopping client now closes its producer and finishes its publishes before the final read is taken.IdempotentProducer::close_within(timeout)is new:close, but the producer’s task is stopped at the timeout, so nothing it held is sent later. (#1075) felixctl sub,cache getandcache watchescape binary payloads on a terminal (\x00,\u{85}) instead of writing raw bytes that garble it; piped output is unchanged.felixctl bench latencypayloads showed it.- A log-backed cache’s expiry could delete a value a put had just refreshed, when the put was waiting on its fsync as the expiry was written. The expiry now waits for a write to the key in flight before it checks. (#976)
felix-cluster upheld for more than an hour lost every control-plane call, teardown included, because its tokens and the brokers’ node tokens were minted once with a one-hour life. The harness now mints tokens on use, rewrites each broker’s node token file and the session file every half hour.Cluster’sclient_token,admin_token,operator_token,subscribe_only_tokenandgroup_operator_tokenfields are now methods, andControlPlane’s token methods moved tofelix_cluster::Credentials.- A cache follower that dropped a divergent suffix of its cache log, or of its counter log, kept the in-memory index or counter sums built from the dropped records, so after a promotion a key could read the value of whatever record replaced its offset. Both are now rebuilt from the log as it is. (#933)
Documentation
Section titled “Documentation”Leadermode’s single writer is stated as resting on the lease: which faults let a deposedLeaderleader acknowledge a write its successor never sees (a pause past the lease margins, a broker clock running slow against the control plane’s), and that a clock step andQuorumwithmajority_ackare not exposed. A cluster test pins that aLeaderstream refuses writes on a lapsed lease aftermajority_ackis finalized. (#1008)- A publish acknowledged on enqueue (a
Leaderstream withack_on_commitoff, the default) is not readable until it is written, so a group poll, history read or replay sent right after the ack can miss it. Stated in the semantics and queues pages and onClient::group_poll. The end-to-end group tests incache_durabilitynow publish with commit acks; they assumed otherwise and failed intermittently. (#1025) - Corrected four replication claims. Counter adds on a
Quorumcache wait for a majority like puts and deletes; the log-backed cache stays Partial for its in-memory key index (#1073). AQuorumpublish with no reachable majority is not refused up front: it waits, then fails withquorum_timeout. Undermajority_acka move’s destination cut off from its followers at the cut-over waits for a majority instead of opening on the lease. A promoted leader’s fence retries back off from 200 ms to 2 s. (#1074)
