Skip to content

Felix 0.6.0-preview.2

Released 2026-10-04 as a preview. The full history is in CHANGELOG.md.

The second preview of 0.6.0, and the first release from the GetFelix organization: images are under ghcr.io/getfelix, and the crates, npm and Python packages point at GetFelix/felix. Subscribers learn when they fall behind and where to resume, consumer groups report skipped offsets, a slow disk or a stalled shard no longer holds up the rest of a broker, and several failover and many-subscription delivery bugs are fixed. pip will not install it without --pre.

  • A durable-stream subscription that falls behind is told so, even when nothing is published after the drop. A broker ends it with subscription_lagged (FEATURE_SUBSCRIPTION_LAGGED, 0x20_0000) naming the first offset its queue for that subscriber dropped, after delivering what was queued before it. felix-client does the same for drops in its own queue, and Subscription::next_event returns a SubscriptionLagged { resume_from } error. ClusterSubscription resubscribes after its last event, which replays the dropped records from the log, and ShardedSubscription reports the shard lost and recovers it the same way, also after its last event. Neither resumes at resume_from: the broker’s connection writer can drop a frame for a slow subscription without reporting it, below that offset. Drops a resume already filled from disk do not count. (#965)

  • A client can ask for commit acks, and the offsets they carry, on its own connections: ClientConfig::ack_on_commit offers FEATURE_ACK_ON_COMMIT (0x8_0000), and a broker that honours it answers that connection’s acknowledged publishes after the write, as FELIX_ACK_ON_COMMIT=true does for every client. Client::supports_ack_on_commit says whether a broker does. Other clients on the broker are unchanged. (#956)

  • A consumer-group record says how many offsets directly below it were settled without delivery (generation-start records, records retention removed first): skipped_before on GroupRecord, in the wire’s group_records, and in the Python and Node clients (skippedBefore). Negotiated as FEATURE_GROUP_SKIPPED (0x40_0000), which the Felix clients offer: a client that did not offer it, or a record with nothing skipped, gets the frame without the field. (#963) Left out when zero, so older clients get the same frames. (#963)

  • A consumer-group member can name itself and take back what a previous process under its name held: group_poll accepts consumer and reclaim (FEATURE_GROUP_CONSUMER, 0x10_0000), Client::group_poll_as and ClusterClient::group_poll_as take a GroupMember, and the Python and Node group_poll take consumer and reclaim. A restarted member no longer waits out the visibility timeout for its records. A member is the name and the connection’s principal, and a name is 1 to 128 bytes. A connection’s first poll with reclaim reserves the claims the member holds from older connections; they go back to it before anything else, over as many polls as it takes, until taken back or lapsed. Later reclaims on that connection, and reclaims from older connections, do nothing. (#962)

  • Development tokens without an identity provider. With FELIX_BOOTSTRAP_DEV_TOKENS=true the bootstrap listener serves POST /internal/bootstrap/tenants/{tenant}/dev-token, which mints a Felix token (and refresh token) for any principal of an initialized tenant with what RBAC grants it. Startup refuses it unless bootstrap is on a loopback bind. (#954)

  • felix-client re-exports AckMode, BrokerEndpoint and ShardRouting, so an application no longer needs felix-wire as a direct dependency to call it. (#937)

  • quic_client_config_with_identity presents a client certificate, and ClientIdentity::from_pem_files and root_store_from_pem_file read one and a CA from PEM files, so mutual TLS no longer means building the rustls config by hand. felixctl uses them and drops its rustls and felix-wire dependencies. (#937)

  • ClusterClient::subscribe_shard subscribes to one chosen shard and follows it to its owner and through moves, as subscribe_from does for shard 0. felixctl sub --shard uses it, so it now follows a moved shard too. (#937)

  • ClusterClient::finish flushes unacknowledged publishes on every broker the client holds before the process exits. felixctl pub --ack none now publishes through the cluster client and calls it. (#937)

  • IdempotentProducer::publish_keyed and publish_batch_keyed publish idempotently with a routing key. The producer keeps a sequence per shard, which is what the leader checks, rather than one per stream. felixctl pub accepts --idempotent with --key. (#937)

  • Brokers answer shard_owners (FEATURE_SHARD_OWNERS, 0x4_0000): which broker owns each shard of a stream or cache, with its client address and generation. Client::shard_owners asks it. felixctl topology takes owners from the broker and needs a control-plane URL only for replicas and state. (#937)

  • ClusterClient has cache_put, cache_get, cache_delete, counter_add and counter_get. Each goes to the key’s shard owner when the client knows it (asked once per cache with shard_owners), else through the broker in use, which forwards; a failed read is asked again once, a write is not sent twice. felixctl cache uses them. (#937)

  • The storage fault file takes write_delay_ms=<n>, which delays every segment write the way a throttled write() does. (#909)

  • Breaking: a felix-client subscription to a durable stream no longer carries on past a dropped record. Under DropNew or DropOld it ends with SubscriptionLagged at the first drop instead, in this client’s queue or the broker’s. In-memory streams, which have no offset to resume from, keep dropping as before. (#965)

  • Connection ids (felix_transport::ConnectionId) count up from 1 per process instead of reusing quinn’s stable_id, which is an address and can repeat once a connection is freed.

  • The control plane accepts RS256 ID tokens from an upstream identity provider by default, next to ES256. Most providers sign with RS256, so a first setup against Dex, Keycloak, Auth0, Entra ID, Google or Okta no longer fails with an unsupported algorithm. FELIX_CONTROLPLANE_OIDC_ALLOWED_ALGORITHMS still narrows it. (#984)

  • A subscribe with no start position, from a client that negotiated event offsets, is now latest: on a durable stream its subscribed carries start_offset and live_offset, so the client knows where live delivery began. Before, only an explicit latest reported them. Clients without offsets get the frame they always did. Like an explicit latest, such a subscribe can now be refused with a retryable not_ready or fenced on a Quorum shard that is settling or refusing reads. And a ClusterSubscription that loses its connection before delivering anything now resumes from that exact offset rather than the new tail: no gap, but it can fail with “cursor too old” if retention passed it meanwhile. (#961)

  • A cache entry whose TTL passes is now a change a watch receives. The shard’s leader writes a delete for it within about a second, through the write fence and replicated like any write, so a watcher no longer has to run its own timers. Reads still treat an entry as absent the moment it lapses. StorageApi gains open_shards and expire_due. (#960)

  • felixctl pub says how many acknowledgements came back without an offset and why, instead of leaving it to a null in --json. An owner that acks on enqueue (ack_on_commit off) answers before the record has an offset, while a publish forwarded through another broker is answered after the write and carries one. The protocol and client docs now state that rule (#941).

  • The broker’s client listeners take their QUIC receive windows from new settings, pub_conn_recv_window and pub_stream_recv_window (FELIX_BROKER_PUB_CONN_RECV_WINDOW, FELIX_BROKER_PUB_STREAM_RECV_WINDOW). Both default to pub_conn_inflight_bytes (16 MiB), raised to max_frame_bytes if that is larger. They used the cache windows, 256 MiB per connection and 64 MiB per stream, so a client whose publishes waited on ingress could leave far more unread data in the broker than the ingress budget allows, and learned about a slow disk minutes late (#923). Startup refuses a window below max_frame_bytes or a stream window above the connection window. Larger windows can still help on a path with a large bandwidth-delay product; set both settings to the old values to get them back.

  • Breaking: the broker no longer reads cache_conn_recv_window and cache_stream_recv_window from its config file, which now refuses them as unknown keys, and ignores FELIX_CACHE_CONN_RECV_WINDOW and FELIX_CACHE_STREAM_RECV_WINDOW. Those remain client settings.

  • A stream’s retention from the control plane now bounds its logs. It was stored and never read: every durable stream got the broker-wide FELIX_DURABLE_RETENTION_BYTES / FELIX_DURABLE_RETENTION_SECONDS. A stream’s max_size_bytes and max_age_seconds now bound its shard logs, each falling back to the broker’s setting when unset, and a patch reaches open logs without a restart. The control plane refuses a zero bound. StreamMetadata gains retention, and DiskLog::set_retention and DiskLogProvider::set_stream_retention set bounds after a log opens. (#964)

  • The Docker Compose, installation and Kubernetes pages pull the 0.6.0-preview images instead of 0.5.0, and a release tag now fails check_release_version.py while any doc pins a ghcr.io/getfelix/felix* image at another version. (#957)

  • felixctl saves its config through a temporary file that is synced and renamed into place, and makes it owner-only even when it already existed with wider permissions. A crash mid-save no longer leaves an empty config.

  • The npm client’s Linux addons load on glibc 2.28. They were built against the release runners’ glibc 2.38 and failed to load on Debian bookworm, RHEL 8 and node:*-bookworm images. The release now links them against glibc 2.28 with cargo-zigbuild and fails if an addon needs a newer glibc symbol. (#981)

  • A broker promoted after it moved a shard away is fenced again. A broker that had drained a shard into a move kept that fact after the move finished, and when a later failover gave it the shard back it took that for a cancelled move and opened without fencing. The previous leader, cut off rather than gone, could then still commit a write with a follower that had not heard of the promotion, at the offset the new leader started its generation at. The history campaign saw one value at two offsets and a subscriber told the first held no event. Only the generation right after the draining one now counts as a hand-back. (#971)

  • Cluster discovery reaches brokers that advertise a DNS name. ClusterClient kept only advertised client addresses that parsed as IP addresses, silently, so a cluster advertising names looked like its seeds alone, and shard owners, cache owners, redirects and moved-shard hints given by name fell back to the entry broker. It now resolves a name each time it is handed one, logs one that does not resolve, and checks that broker’s certificate against the name. The broker refuses a FELIX_CLIENT_ADVERTISE_ADDR that is not host:port at startup. (#982)

  • The npm and Python clients’ cache, counter and stream-shard calls fail over. They called the broker the client entered by directly, so once it died they kept going to it while other seeds were up. They now go through ClusterClient like publishes do, which routes cache calls to the key’s owner and moves to another broker when the one in use is gone. ClusterClient::stream_shards is new. The conformance catalogue gains fault.cache_through_a_reset_link. (#979)

  • A full disk is reported as a full disk. When the disk filled while a log was being created, its first segment was left with no header, and every later open reported corruption, even once space was freed. Group polls on a new shard hit it first, since a shard’s cursor and dead-letter logs are made on demand. A segment whose creation fails is now removed, and recovery starts a log whose only segment never got a header afresh. ENOSPC and quota errors are StorageError::Full and BrokerError::StorageFull, counted by felix_storage_full_total, and clients see overloaded (retry after) with nothing written. Breaking for code that matches either enum exhaustively. (#983)

  • A cache watch that reaches a broker just after it stopped serving the shard is refused instead of left waiting, as subscribes already were. One registered in that moment was never ended and looked like a quiet key. It is now answered shard_unavailable (moving), and the client retries where the shard is served. (#986)

  • Many subscriptions on one connection no longer lose deliveries at the default queue bound. A publish fanned out to every subscription on a connection put one entry per subscription on the connection’s writer lane and on its connection writer queue, both 64 deep by default, and under drop_new the overflow was dropped, the connection queue’s without a count. Those two hand-offs now wait when full. Drops happen only in each subscription’s own queues and are all counted in felix_sub_queue_dropped_total. felix_subscriber_lane_dropped_total is gone, since nothing is dropped where it counted. (#978)

  • A standalone broker keeps its control-plane credential current. A broker without FELIX_NODE_ID read its node token once at startup and never refreshed it, so its catalog sync was refused once an exchanged token expired, fifteen minutes in. It now re-reads FELIX_NODE_TOKEN_FILE and honours FELIX_NODE_REFRESH_TOKEN_FILE the way cluster members do. The two settings moved from MembershipConfig to BrokerConfig. (#955)

  • A subscribe that reaches a broker just after it stopped serving the shard is refused instead of left waiting. A broker ends a shard’s readers before its routes catch up with the move or failover, so for that moment it still accepted subscribes, and one registered then was never ended: it delivered nothing more, for good. It is now answered shard_unavailable (moving), and the client retries where the shard is served.

  • A Quorum stream’s held batch is no longer stranded when the last mark that covers it arrives while the shard’s bound is refused or settling. The release now rechecks the bound while anything is held, so a route or lease change that makes the bound cover it delivers it to live subscribers. Before, a subscriber could stop short of records every read of the log returned.

  • A slow or throttled disk no longer stalls the broker’s runtime. A durable append’s write() ran on the Tokio worker that called it, under a synchronous lock every reader of that log also took, and once Linux throttles a process that dirties pages faster than the device takes them, write() blocks for up to hundreds of milliseconds. Each log now appends on a thread of its own, started on demand and stopped after 10 s idle like its flush thread, and the write runs with the lock released. A lone unbatched publisher loses 4 to 6% of its throughput to the hand-off, one sending batches of 16 gains about a tenth, and several publishers on one log get about twice the throughput with p99 in tens of microseconds. Sparse index entries are written 256 at a time instead of with each append that crosses an interval. felix_storage_sync_batch_appends now counts the records each flush covered rather than the callers waiting when it started, so publishes the broker merges into one append count once each, and a client batch of N records counts N. See docs/storage-performance.md. (#909)

  • Breaking: StreamLog::begin_append, begin_append_marked and continue_batch take the stream’s CommitSequencer and return the claimed CommitTurn with the PendingAppend. DiskLog::append_claimed and continue_claimed (which replaces continue_pending) make that claim on the append thread, so a caller cancelled before it hears its offsets still releases its range. (#909)

  • A stalled shard no longer stalls publishes to healthy shards on the same connection. The pipelined publish window was counted per connection, so a stream whose publishes waited on one stuck shard (a quorum wait, say) could take every slot and stop the connection’s other streams. The broker now grants publish_window per stream and says so with a new feature bit, FEATURE_STREAM_PUBLISH_WINDOW (0x2_0000). The Rust client gives each stream its own window against such a broker and keeps sharing one per connection against an older broker. Older clients keep working: they share one window across their streams, which stays inside every stream’s. (#843)

  • Through a ClusterClient, a stalled shard no longer holds up other shards of the same stream, or streams that share its QUIC stream. A pipelining stream is answered in request order, and the Rust client hashed every shard of a stream onto one of its pooled QUIC streams, so a stuck shard held back answers the other shards had already committed. ClusterClient knows the shard of every publish it makes, so it now gives each shard a QUIC stream of its own: plain, keyed and idempotent publishes alike. Up to publish_shard_streams of them per broker it talks to (FELIX_PUB_SHARD_STREAMS, default 16), opened on a shard’s first publish and kept; shards past that share the pool as before, and 0 turns them off. A plain Client does not know a stream’s width and routes exactly as before, one writer per stream. No wire change. (#843)

  • The control plane’s Raft leader no longer retries a down or misconfigured member in a tight loop. Refused connections, timeouts and non-2xx replies now back off (one heartbeat interval, doubling to one second), and the error names the HTTP status instead of a JSON parse failure. Unit tests in the control-plane crate no longer print their logs to stdout.