Felix 0.6.0-preview.2
Released 2026-10-04 as a preview. The full history is in CHANGELOG.md.
The second preview of 0.6.0, and the first release from the GetFelix
organization: images are under ghcr.io/getfelix, and the crates, npm and
Python packages point at GetFelix/felix. Subscribers learn when they fall
behind and where to resume, consumer groups report skipped offsets, a slow
disk or a stalled shard no longer holds up the rest of a broker, and several
failover and many-subscription delivery bugs are fixed. pip will not install
it without --pre.
-
A durable-stream subscription that falls behind is told so, even when nothing is published after the drop. A broker ends it with
subscription_lagged(FEATURE_SUBSCRIPTION_LAGGED,0x20_0000) naming the first offset its queue for that subscriber dropped, after delivering what was queued before it. felix-client does the same for drops in its own queue, andSubscription::next_eventreturns aSubscriptionLagged { resume_from }error.ClusterSubscriptionresubscribes after its last event, which replays the dropped records from the log, andShardedSubscriptionreports the shard lost and recovers it the same way, also after its last event. Neither resumes atresume_from: the broker’s connection writer can drop a frame for a slow subscription without reporting it, below that offset. Drops a resume already filled from disk do not count. (#965) -
A client can ask for commit acks, and the offsets they carry, on its own connections:
ClientConfig::ack_on_commitoffersFEATURE_ACK_ON_COMMIT(0x8_0000), and a broker that honours it answers that connection’s acknowledged publishes after the write, asFELIX_ACK_ON_COMMIT=truedoes for every client.Client::supports_ack_on_commitsays whether a broker does. Other clients on the broker are unchanged. (#956) -
A consumer-group record says how many offsets directly below it were settled without delivery (generation-start records, records retention removed first):
skipped_beforeonGroupRecord, in the wire’sgroup_records, and in the Python and Node clients (skippedBefore). Negotiated asFEATURE_GROUP_SKIPPED(0x40_0000), which the Felix clients offer: a client that did not offer it, or a record with nothing skipped, gets the frame without the field. (#963) Left out when zero, so older clients get the same frames. (#963) -
A consumer-group member can name itself and take back what a previous process under its name held:
group_pollacceptsconsumerandreclaim(FEATURE_GROUP_CONSUMER,0x10_0000),Client::group_poll_asandClusterClient::group_poll_astake aGroupMember, and the Python and Nodegroup_polltakeconsumerandreclaim. A restarted member no longer waits out the visibility timeout for its records. A member is the name and the connection’s principal, and a name is 1 to 128 bytes. A connection’s first poll withreclaimreserves the claims the member holds from older connections; they go back to it before anything else, over as many polls as it takes, until taken back or lapsed. Later reclaims on that connection, and reclaims from older connections, do nothing. (#962) -
Development tokens without an identity provider. With
FELIX_BOOTSTRAP_DEV_TOKENS=truethe bootstrap listener servesPOST /internal/bootstrap/tenants/{tenant}/dev-token, which mints a Felix token (and refresh token) for any principal of an initialized tenant with what RBAC grants it. Startup refuses it unless bootstrap is on a loopback bind. (#954) -
felix-clientre-exportsAckMode,BrokerEndpointandShardRouting, so an application no longer needsfelix-wireas a direct dependency to call it. (#937) -
quic_client_config_with_identitypresents a client certificate, andClientIdentity::from_pem_filesandroot_store_from_pem_fileread one and a CA from PEM files, so mutual TLS no longer means building the rustls config by hand. felixctl uses them and drops itsrustlsandfelix-wiredependencies. (#937) -
ClusterClient::subscribe_shardsubscribes to one chosen shard and follows it to its owner and through moves, assubscribe_fromdoes for shard 0.felixctl sub --sharduses it, so it now follows a moved shard too. (#937) -
ClusterClient::finishflushes unacknowledged publishes on every broker the client holds before the process exits.felixctl pub --ack nonenow publishes through the cluster client and calls it. (#937) -
IdempotentProducer::publish_keyedandpublish_batch_keyedpublish idempotently with a routing key. The producer keeps a sequence per shard, which is what the leader checks, rather than one per stream.felixctl pubaccepts--idempotentwith--key. (#937) -
Brokers answer
shard_owners(FEATURE_SHARD_OWNERS,0x4_0000): which broker owns each shard of a stream or cache, with its client address and generation.Client::shard_ownersasks it.felixctl topologytakes owners from the broker and needs a control-plane URL only for replicas and state. (#937) -
ClusterClienthascache_put,cache_get,cache_delete,counter_addandcounter_get. Each goes to the key’s shard owner when the client knows it (asked once per cache withshard_owners), else through the broker in use, which forwards; a failed read is asked again once, a write is not sent twice.felixctl cacheuses them. (#937) -
The storage fault file takes
write_delay_ms=<n>, which delays every segment write the way a throttledwrite()does. (#909)
Changed
Section titled “Changed”-
Breaking: a felix-client subscription to a durable stream no longer carries on past a dropped record. Under
DropNeworDropOldit ends withSubscriptionLaggedat the first drop instead, in this client’s queue or the broker’s. In-memory streams, which have no offset to resume from, keep dropping as before. (#965) -
Connection ids (
felix_transport::ConnectionId) count up from 1 per process instead of reusing quinn’sstable_id, which is an address and can repeat once a connection is freed. -
The control plane accepts RS256 ID tokens from an upstream identity provider by default, next to ES256. Most providers sign with RS256, so a first setup against Dex, Keycloak, Auth0, Entra ID, Google or Okta no longer fails with an unsupported algorithm.
FELIX_CONTROLPLANE_OIDC_ALLOWED_ALGORITHMSstill narrows it. (#984) -
A subscribe with no start position, from a client that negotiated event offsets, is now
latest: on a durable stream itssubscribedcarriesstart_offsetandlive_offset, so the client knows where live delivery began. Before, only an explicitlatestreported them. Clients without offsets get the frame they always did. Like an explicitlatest, such a subscribe can now be refused with a retryablenot_readyorfencedon aQuorumshard that is settling or refusing reads. And aClusterSubscriptionthat loses its connection before delivering anything now resumes from that exact offset rather than the new tail: no gap, but it can fail with “cursor too old” if retention passed it meanwhile. (#961) -
A cache entry whose TTL passes is now a change a watch receives. The shard’s leader writes a delete for it within about a second, through the write fence and replicated like any write, so a watcher no longer has to run its own timers. Reads still treat an entry as absent the moment it lapses.
StorageApigainsopen_shardsandexpire_due. (#960) -
felixctl pubsays how many acknowledgements came back without an offset and why, instead of leaving it to anullin--json. An owner that acks on enqueue (ack_on_commitoff) answers before the record has an offset, while a publish forwarded through another broker is answered after the write and carries one. The protocol and client docs now state that rule (#941). -
The broker’s client listeners take their QUIC receive windows from new settings,
pub_conn_recv_windowandpub_stream_recv_window(FELIX_BROKER_PUB_CONN_RECV_WINDOW,FELIX_BROKER_PUB_STREAM_RECV_WINDOW). Both default topub_conn_inflight_bytes(16 MiB), raised tomax_frame_bytesif that is larger. They used the cache windows, 256 MiB per connection and 64 MiB per stream, so a client whose publishes waited on ingress could leave far more unread data in the broker than the ingress budget allows, and learned about a slow disk minutes late (#923). Startup refuses a window belowmax_frame_bytesor a stream window above the connection window. Larger windows can still help on a path with a large bandwidth-delay product; set both settings to the old values to get them back. -
Breaking: the broker no longer reads
cache_conn_recv_windowandcache_stream_recv_windowfrom its config file, which now refuses them as unknown keys, and ignoresFELIX_CACHE_CONN_RECV_WINDOWandFELIX_CACHE_STREAM_RECV_WINDOW. Those remain client settings.
-
A stream’s retention from the control plane now bounds its logs. It was stored and never read: every durable stream got the broker-wide
FELIX_DURABLE_RETENTION_BYTES/FELIX_DURABLE_RETENTION_SECONDS. A stream’smax_size_bytesandmax_age_secondsnow bound its shard logs, each falling back to the broker’s setting when unset, and a patch reaches open logs without a restart. The control plane refuses a zero bound.StreamMetadatagainsretention, andDiskLog::set_retentionandDiskLogProvider::set_stream_retentionset bounds after a log opens. (#964) -
The Docker Compose, installation and Kubernetes pages pull the
0.6.0-previewimages instead of0.5.0, and a release tag now failscheck_release_version.pywhile any doc pins aghcr.io/getfelix/felix*image at another version. (#957) -
felixctlsaves its config through a temporary file that is synced and renamed into place, and makes it owner-only even when it already existed with wider permissions. A crash mid-save no longer leaves an empty config. -
The npm client’s Linux addons load on glibc 2.28. They were built against the release runners’ glibc 2.38 and failed to load on Debian bookworm, RHEL 8 and
node:*-bookwormimages. The release now links them against glibc 2.28 with cargo-zigbuild and fails if an addon needs a newer glibc symbol. (#981) -
A broker promoted after it moved a shard away is fenced again. A broker that had drained a shard into a move kept that fact after the move finished, and when a later failover gave it the shard back it took that for a cancelled move and opened without fencing. The previous leader, cut off rather than gone, could then still commit a write with a follower that had not heard of the promotion, at the offset the new leader started its generation at. The history campaign saw one value at two offsets and a subscriber told the first held no event. Only the generation right after the draining one now counts as a hand-back. (#971)
-
Cluster discovery reaches brokers that advertise a DNS name.
ClusterClientkept only advertised client addresses that parsed as IP addresses, silently, so a cluster advertising names looked like its seeds alone, and shard owners, cache owners, redirects and moved-shard hints given by name fell back to the entry broker. It now resolves a name each time it is handed one, logs one that does not resolve, and checks that broker’s certificate against the name. The broker refuses aFELIX_CLIENT_ADVERTISE_ADDRthat is nothost:portat startup. (#982) -
The npm and Python clients’ cache, counter and stream-shard calls fail over. They called the broker the client entered by directly, so once it died they kept going to it while other seeds were up. They now go through
ClusterClientlike publishes do, which routes cache calls to the key’s owner and moves to another broker when the one in use is gone.ClusterClient::stream_shardsis new. The conformance catalogue gainsfault.cache_through_a_reset_link. (#979) -
A full disk is reported as a full disk. When the disk filled while a log was being created, its first segment was left with no header, and every later open reported corruption, even once space was freed. Group polls on a new shard hit it first, since a shard’s cursor and dead-letter logs are made on demand. A segment whose creation fails is now removed, and recovery starts a log whose only segment never got a header afresh.
ENOSPCand quota errors areStorageError::FullandBrokerError::StorageFull, counted byfelix_storage_full_total, and clients seeoverloaded(retry after) with nothing written. Breaking for code that matches either enum exhaustively. (#983) -
A cache watch that reaches a broker just after it stopped serving the shard is refused instead of left waiting, as subscribes already were. One registered in that moment was never ended and looked like a quiet key. It is now answered
shard_unavailable(moving), and the client retries where the shard is served. (#986) -
Many subscriptions on one connection no longer lose deliveries at the default queue bound. A publish fanned out to every subscription on a connection put one entry per subscription on the connection’s writer lane and on its connection writer queue, both 64 deep by default, and under
drop_newthe overflow was dropped, the connection queue’s without a count. Those two hand-offs now wait when full. Drops happen only in each subscription’s own queues and are all counted infelix_sub_queue_dropped_total.felix_subscriber_lane_dropped_totalis gone, since nothing is dropped where it counted. (#978) -
A standalone broker keeps its control-plane credential current. A broker without
FELIX_NODE_IDread its node token once at startup and never refreshed it, so its catalog sync was refused once an exchanged token expired, fifteen minutes in. It now re-readsFELIX_NODE_TOKEN_FILEand honoursFELIX_NODE_REFRESH_TOKEN_FILEthe way cluster members do. The two settings moved fromMembershipConfigtoBrokerConfig. (#955) -
A subscribe that reaches a broker just after it stopped serving the shard is refused instead of left waiting. A broker ends a shard’s readers before its routes catch up with the move or failover, so for that moment it still accepted subscribes, and one registered then was never ended: it delivered nothing more, for good. It is now answered
shard_unavailable(moving), and the client retries where the shard is served. -
A
Quorumstream’s held batch is no longer stranded when the last mark that covers it arrives while the shard’s bound is refused or settling. The release now rechecks the bound while anything is held, so a route or lease change that makes the bound cover it delivers it to live subscribers. Before, a subscriber could stop short of records every read of the log returned. -
A slow or throttled disk no longer stalls the broker’s runtime. A durable append’s
write()ran on the Tokio worker that called it, under a synchronous lock every reader of that log also took, and once Linux throttles a process that dirties pages faster than the device takes them,write()blocks for up to hundreds of milliseconds. Each log now appends on a thread of its own, started on demand and stopped after 10 s idle like its flush thread, and the write runs with the lock released. A lone unbatched publisher loses 4 to 6% of its throughput to the hand-off, one sending batches of 16 gains about a tenth, and several publishers on one log get about twice the throughput with p99 in tens of microseconds. Sparse index entries are written 256 at a time instead of with each append that crosses an interval.felix_storage_sync_batch_appendsnow counts the records each flush covered rather than the callers waiting when it started, so publishes the broker merges into one append count once each, and a client batch of N records counts N. Seedocs/storage-performance.md. (#909) -
Breaking:
StreamLog::begin_append,begin_append_markedandcontinue_batchtake the stream’sCommitSequencerand return the claimedCommitTurnwith thePendingAppend.DiskLog::append_claimedandcontinue_claimed(which replacescontinue_pending) make that claim on the append thread, so a caller cancelled before it hears its offsets still releases its range. (#909) -
A stalled shard no longer stalls publishes to healthy shards on the same connection. The pipelined publish window was counted per connection, so a stream whose publishes waited on one stuck shard (a quorum wait, say) could take every slot and stop the connection’s other streams. The broker now grants
publish_windowper stream and says so with a new feature bit,FEATURE_STREAM_PUBLISH_WINDOW(0x2_0000). The Rust client gives each stream its own window against such a broker and keeps sharing one per connection against an older broker. Older clients keep working: they share one window across their streams, which stays inside every stream’s. (#843) -
Through a
ClusterClient, a stalled shard no longer holds up other shards of the same stream, or streams that share its QUIC stream. A pipelining stream is answered in request order, and the Rust client hashed every shard of a stream onto one of its pooled QUIC streams, so a stuck shard held back answers the other shards had already committed.ClusterClientknows the shard of every publish it makes, so it now gives each shard a QUIC stream of its own: plain, keyed and idempotent publishes alike. Up topublish_shard_streamsof them per broker it talks to (FELIX_PUB_SHARD_STREAMS, default 16), opened on a shard’s first publish and kept; shards past that share the pool as before, and0turns them off. A plainClientdoes not know a stream’s width and routes exactly as before, one writer per stream. No wire change. (#843) -
The control plane’s Raft leader no longer retries a down or misconfigured member in a tight loop. Refused connections, timeouts and non-2xx replies now back off (one heartbeat interval, doubling to one second), and the error names the HTTP status instead of a JSON parse failure. Unit tests in the control-plane crate no longer print their logs to stdout.
