Felix 0.6.0
Released 2026-10-23. The full history is in CHANGELOG.md.
0.6.0, the build the release performance runs measured. The previews below
describe most of the series; this section lists what changed since
0.6.0-preview.9. A durable stream now acknowledges a publish only after it
commits, a new cluster finalizes its fleet features at its first broker
registration, ClusterClient publishes batches straight to each shard’s
leader, and brokers forwarding Quorum publishes to each other no longer
deadlock their replication. The publish ingress path got a deeper default
queue, one wake per freed slot, and waits that no longer stop a control stream
from reading.
Upgrade notes.
- Acked publishes on durable streams now wait for the commit and cost a write’s
latency.
FELIX_ACK_ON_COMMIT=0restores acknowledging on queue. FELIX_BROKER_PUB_QUEUE_DEPTHdefaults to 512 (was 64) andFELIX_PUB_SHARD_STREAMSto 4 (was 16).- A Postgres control plane runs migration
0035on start. Under Raft, the recreated-stream fix needs every member at metadata level 17 and the record of deleted names needs level 18. - Creating a stream or cache under a deleted name returns 409 until
stream_incarnations(orcache_incarnations) is finalized. - On a new cluster, start every broker with the same
FELIX_QUORUM_READSandFELIX_INTERNAL_FENCE.
Known issues.
- With
pub_ingress_waiton, fire-and-forget batches of 256 B messages run about 13% slower than on the build the performance runs measured (#1469). Larger messages and acked publishes measured the same as that build. Fixed in 0.6.1.
Security
Section titled “Security”- Updated
rustls-webpkito 0.103.16 andwebpki-rootsandwebpki-root-certsto 1.0.10 for RUSTSEC-2026-0382 and RUSTSEC-2026-0383, in the workspace and every out-of-workspace lockfile.
ClusterClient::offset_for_timelooks up the offset for a time on one shard of a durable stream, asking whichever broker leads the shard and followingNotLeaderredirects, asClusterClient::readdoes.ClusterClientpublishes batches owner-direct:publish_batch,publish_batch_keyed, andpublish_records/publish_records_at_least_once, which split records by shard and send each shard’s ordered batch to its leader, reporting failures per shard in aBatchReport. Before, only single records were leader-direct, so batched publishing on a cluster was forwarded by a non-leader.felix-loadgen ingest --clusteruses it, and a loadgen publisher now survives a broker loss and reportsreconnects,failed_sendsand per-shard acked counts.ClusterClientnow looks up shard leaders from the cluster (FEATURE_SHARD_OWNERS) on a stream’s first publish, every few seconds, and after a publish finds one wrong, so the first batch and publishes that ask for no ack go leader-direct. Before, a leader was learned only from the ack of a forwarded batch, so fire-and-forget publishing was forwarded for as long as it ran. A cached leader learned under no stream width is no longer trusted for a keyed publish.
Changed
Section titled “Changed”- Behaviour change: a durable stream now acknowledges a publish after it commits by default
(#1421). Before, the broker answered when the publish was queued, so a broker crash or a lapsed
lease could lose a record the client had been told was stored. Committed means written to the
stream’s log under
FELIX_DURABLE_FSYNC_MODE: with the defaultperiodicmode the bytes are in the log file (they survive a broker crash, and a machine crash can lose the last 250 ms), andon_commitflushes to the device first. In-memory streams still answer when the publish is queued. Acks now carry their offset on durable streams, and acked publishes cost a write’s latency.FELIX_ACK_ON_COMMIT=0(orack_on_commit: false) restores the old behaviour for every stream;FELIX_ACK_ON_COMMIT=1makes in-memory streams wait too. The wire is unchanged, so a client that predates commit-ack negotiation works as before. FELIX_ACK_ON_COMMITnow reads1/true/yes/onand0/false/no/off(trimmed, any case). Any other value keeps the default and logs a warning instead of turning commit acks off. A stream with more than 1024 answers owed now refuses an acked publish before queueing it, so a retry cannot write it twice.- The control plane refuses with 409 to create a stream or cache under a name that was deleted on
the cluster until
stream_incarnations(orcache_incarnations) is finalized, and the message says to finalize first (#1447). Without an incarnation, a stream created again opened the deleted stream’s logs on every broker, so a recreatedQuorumstream’s followers held old committed records that the new leader’s writes disagreed with, and they halted as diverged. Every store records deleted names (Postgres migration 0035; on Raft once every member is at metadata level 18), and they are part of the exported state. - A new cluster finalizes
stream_incarnationsandcache_incarnationsat its first broker registration, so a stream or cache deleted and created again there always starts empty on every broker (#1447). The control plane treats a cluster as new when no stream or cache was ever created and no broker has registered twice, and finalizes through the same checks as an operator’s finalize, so a broker without the features keeps them off or is refused afterwards. A cluster upgraded from an older release still waits for the operator to finalize. - The one-command
compose.yamlstack now runs its broker withFELIX_DURABLE_STORAGE_DIR, soordersis a durable stream, consumer groups work, and cache watches work. It also creates the cachesconfigandstats, and its development tokens can use caches, counters and groups. CI exercises these commands through the stack. - A new cluster finalizes every fleet feature its brokers report at its first
broker registration, not only the incarnations:
generation_start,majority_ack,lease_free_reads,fenced_caches,jump_hash_routing,atomic_commit,group_cursor_dependencies,stream_incarnationsandcache_incarnations, in that order. A fresh RF=3 cluster used to acknowledgeQuorumwrites on the control plane’s replica report until an operator finalizedmajority_ack; it now acknowledges on its followers from the first stream.publisher_principalstays an operator step, since it grows every record by its publisher’s id. Each finalize goes through the operator’s checks, so a feature some broker does not report stays off with the features that need it, and under the Raft backend nothing is finalized until every member runs this release; a finalize held back that way is retried in the background for up to ten minutes. Start every broker with the sameFELIX_QUORUM_READSandFELIX_INTERNAL_FENCE: a broker that lacks a feature the first one finalized is refused, and the refusal now namesFELIX_INTERNAL_FENCEas well asFELIX_QUORUM_READS. An upgraded cluster still waits for the operator.
- Placement now spreads each stream’s and cache’s leaders evenly across brokers, not only the cluster-wide total (#1429). A 12-shard stream on three brokers could be placed 0/0/12 beside other streams that already filled two brokers to their share, and rebalancing left it at 1/1/10 because the totals were even, so the stream’s writes ran on one broker. New shards now go to the broker leading fewest of their stream first, a broker leading two or more of a stream’s shards more than another hands one over, and a move for the cluster-wide balance never makes a stream’s spread wider. The totals are now evened to within one, where before a broker was only moved off when it led more than its rounded-up share. Existing clusters even out after the upgrade, one move at a time under the usual move limits.
finish()and broker connects no longer hang a client forever when a broker stalls (#1433).Publisher::finishandClusterClient::finishgive up afterfinish_timeout(FELIX_FINISH_TIMEOUT_MS, default 60 s) and abort the streams, and each connect, including a cluster reconnect, fails afterconnect_timeout(FELIX_CONNECT_TIMEOUT_MS, default 10 s). A cluster reconnect now dials outside the client lock and within three timesconnect_timeout,ClusterClient::finishfinishes brokers concurrently, and the topology, shard-count and subscribe exchanges give up afterconnect_timeout.- A fenced shard move whose leader is live but never reports drained no longer holds the shard
without writes forever (#1434). After
FELIX_SHARD_FENCE_TIMEOUT_MS(two minutes by default,shard_fence_timeout_msin the config file) placement gives the move up the way an operator’s cancel does: the leader serves again at a new generation, holding every write it accepted, and the move counts infelix_shard_moves_timed_out_total. The shard’s next move waits the timeout, doubling with each give-up in a row up to an hour, so a leader that cannot drain does not cut its shard’s writes every few minutes; repeated give-ups log a warning.0keeps the old behaviour. - A stream deleted and created again under the same name without incarnations no longer leaves a
zombie leader behind (#1435). The delete now puts an
unassignedchange for each shard on the shard change log, so a broker leading one lets it go. The recreated stream’s shards start at the change seq instead of generation 0, because brokers keep an unincarnated stream’s logs and the generations those logs accepted: a broker that had taken a shard by a move refused to lead or follow the new shard at generation 0. Under Raft both need every member at metadata level 17. A broker also takes a shard’s first assignment over whatever it still holds, which covers control planes that don’t record the unassignments yet. - Brokers forwarding durable
Quorumpublishes to each other no longer deadlock their replication (#1458). Replication shared peer streams with forwarded publishes, and a peer answered each stream one request at a time, so a forward waiting on its quorum held up the replication that quorum needed until every wait timed out. Replication now has its own peer streams and in-flight allowance, and a broker answers forwarded publishes and cache operations as they finish rather than in arrival order. Both ends are fixed independently, so a mixed cluster improves as soon as either side of a link is upgraded. - A forwarded publish now waits
FELIX_PUBLISH_QUORUM_TIMEOUT_MS+ 1 s for the owner’s answer (wasFELIX_INTERNAL_REQUEST_TIMEOUT_MS, 5 s), so a slow commit returns the owner’s result instead of an indeterminate one. The publish ack wait floor rises to match: the larger ofFELIX_ACK_WAIT_TIMEOUT_MSandFELIX_PUBLISH_QUORUM_TIMEOUT_MS+ 1.5 s (6.5 s by default). - A peer stream whose requests stopped being answered no longer stays wedged for the life of
the connection (#1458). A requester replaces a stream after two timeouts in a row
(
felix_broker_peer_lanes_replaced_total) and offers the newLANE_RENEWALcapability, and to such a requester a broker drops a stream once a request on it has run pastFELIX_INTERNAL_REQUEST_TIMEOUT_MS. Streams from an older requester are never dropped. After a forwarding storm, peer requests and quorum waits on an affected link kept timing out for more than ten minutes with no load. - A broker serves at most
FELIX_INTERNAL_MAX_SERVED_FORWARDS(4096) forwarded publishes and cache operations at once, and 256 per peer stream; past that a forward is refused asOverload, which nothing applied and the forwarder may retry (#1458). cargo bench --workspace --no-runbuilds again: the broker’s unit tests that slow the fsync now getfelix-storage’sfault-injectionfeature in a release build too.
Performance
Section titled “Performance”- Per-tenant publish counters and the subscriber-queue metrics are resolved once and reused
instead of looked up in the metrics registry on every publish (#1406). On a broker publishing
small batches this removed about 3% of CPU spent hashing and registering metric keys.
felix_broker_ingress_queue_depthis now refreshed when the depth has moved by 16 or more since the last refresh and when it reaches zero, so while jobs are queued it can differ from the true depth by up to 15; an emptied queue reads zero. - A freed publish-queue slot now admits one waiting publisher, in arrival order, instead of waking
every publisher blocked on a full queue (#1406). With
pub_ingress_waiton and thousands of publishes in flight, each pop used to wake all of them to contend for one slot. - The default
FELIX_BROKER_PUB_QUEUE_DEPTHis now 512 (was 64), so the publish queue holds 2048 jobs with the default executors instead of 256 (#1406). With many pipelining publishers the queue filled first, and a full queue stops the publishing connection from being read. With 64 publisher connections and spread keys, 256 B unbatched acked publishes went from 95 to 163 MB/s (p50 12.6 to 7.5 ms), and 4 KiB batch-64 p99 fell from about 460-600 ms to about 360-380 ms; throughput at large batches is unchanged (about 4.7 GB/s). Set the variable to go back to a shallower queue. FELIX_PUB_SHARD_STREAMS(publish_shard_streams) now defaults to 4 instead of 16 (#1406). With keys spread over 48 shards, 256 B acked publishes measured about 125 MB/s at 4 and about 97 MB/s at 16. Behaviour change: only the first four shards a client publishes to on a broker (first come, across all streams) get a dedicated stream; later shards use the shared pool, where a stalled shard can hold up the other pooled shards. Raise the variable to the total number of shards the client publishes to on a broker for full isolation.- A publish that finds the broker’s publish queue full no longer stops its
control stream from reading (#1416). Under
FELIX_PUB_INGRESS_WAIT=1, and for acked publishes underack_on_commit, the publish waits in line off the read loop, so publishes behind it to shards with room still go through. A stream stops reading only once it has 1024 publishes waiting or its byte budget is spent. Order per shard, the queue’s first-come admission and the wait timeout are unchanged. - The load generator’s
ingestscenario now stops issuing at the deadline, waits for outstanding acks, and callsfinish()on every publisher instead of dropping the connections. Publishes still queued at the broker used to be cancelled, so published bytes exceeded accepted bytes by a few batches per connection. The drain is reported asfinish_msinLOADGEN_JSON(withfinish_failed) and is excluded fromduration_sand the throughput figures. Acked runs no longer remove in-flight sends frompublishedat the deadline; only sends still unanswered after the 60 s drain bound count ascut_off_records.
