What Felix Is For

Status markers last verified against the code on 2026-09-13. The Sharding, Multi-node clustering and Leader failover rows were re-verified on 2026-09-28, and the Quorum acknowledgement and Replication factor restore rows on 2026-09-30, and the Token-based authorization, Durable streams, Tiered storage and Multi-node clustering rows on 2026-10-09, and the Leader failover timings and Replication factor restore row on 2026-10-10, and the Leader failover row on 2026-10-15, and the Durable streams, Fleet-wide feature gate, Quorum acknowledgement, Linearizable Quorum reads, Online rebalancing, Client surviving a broker failure, Control plane metadata service and Record times rows and the delivery semantics on 2026-10-23, for 0.6.0. This page goes stale the moment a 🎯 row lands, and stale status markers are worse than no status markers. A reader who catches one wrong row stops trusting the other twenty. Treat updating it as part of shipping any capability listed here. There is precedent for drift: docs/semantics.md claimed authorization was unimplemented long after it was enforced on the control stream, and described a single-node at-most-once system long after replication shipped. Both were corrected when that document was rewritten against the tests.

✅ and 🚧 rows were confirmed by reading the implementation. “Not started” on 🎯 rows means no implementation was found, which is a weaker check.

The one-sentence answer

Today: Felix is a QUIC-based replicated log that serves stream, cache, and queue semantics from one storage engine, optimized for high-fanout delivery with predictable tail latency and strict slow-consumer isolation.

Shards of a stream or cache are placed across brokers, replicated by leader leases and log shipping, and survive the loss of a leader; a publish can be made to wait for a quorum of the replica set before it is acknowledged. What is not yet true of a cluster is spelled out row by row in section 2. Most of the multi-node rows are ✅; the ones still 🚧 each say what is missing.

The slow-consumer isolation half of that sentence is runnable: task demo:slow-consumer stalls one consumer and measures what happens to the others, under both queue policies.

Target: Felix is a distributed data plane offering coordination-store watch semantics (read the current value, then receive every subsequent change with no gap) at messaging-system fanout and throughput. The point of that combination is keeping large populations of services, agents, and edge nodes synchronized with rapidly changing state.

The gap between those two sentences is the roadmap. The rest of this page makes that gap explicit.

1. The problem Felix is aimed at

Many distributed systems need a large number of independent processes to agree on the same continuously changing state: configuration, policy, routing tables, task assignments, cluster membership, device state.

The common implementation stitches together several systems: a database for the authoritative value, Redis for fast reads, Kafka or RabbitMQ for change events, WebSockets or polling to reach the consumers. Each one solves part of the problem with different semantics, different failure modes, and no shared consistency story between them.

Felix’s thesis is that this class of workload deserves a single coherent abstraction: current state, the stream of changes to it, and the transport that distributes both.

Concretely, the traffic looks like a namespace of small, frequently-changing values with a change feed beside each one:

config/current              # the authoritative value
config/changes              # the deltas since
routing/table
policy/authz
cluster/membership
device/{id}/state
agent/{id}/tasks
agent/{id}/events

carrying events like certificate rotated, node joined, route changed, policy updated, feature enabled, task assigned, run cancelled. The defining property is that each consumer needs to end up holding an accurate local copy. Seeing that something happened is not enough.

flowchart TD
    P["Producer / control plane"] --> F["Felix"]
    F --> C1["Consumer"]
    F --> C2["Consumer"]
    F --> C3["Consumer (slow)"]
    C1 -.-> S1["local view of state"]
    C2 -.-> S2["local view of state"]
    C3 -.-> S3["local view of state"]

That is the target. What exists today is the transport and fanout layer underneath it, plus the first watch primitives on top: a keyed cache watch (subscribe to one key or prefix, resume by offset, loud loss) and retained delivery on it, which is exactly the “current value plus every subsequent change, gap-free” contract this section describes, for cache keys (see the table below for exactly how far it goes).

Why not etcd, Consul, or ZooKeeper?

This is the first question the thesis has to survive, because “current value plus every subsequent change, gap-free” is not a new idea. etcd’s watch does exactly this (read at a revision, watch from that revision), and Kubernetes is built on it. Consul and ZooKeeper have watches. NATS JetStream’s KV store has watch. Kafka’s stream-table duality is the same concept applied to a log. Anyone evaluating Felix will reach for one of these first, and for many workloads they should.

The gap Felix is aimed at is the scale at which those semantics are available. Coordination stores are consistency-first: they run a consensus quorum over a comparatively small dataset, and they trade write throughput and watch fanout to get correctness. etcd is a well-known example: the Kubernetes apiserver maintains its own watch cache in front of etcd in large part because etcd cannot serve that many watchers directly. When people need watch semantics at high fanout, they build a fanout tier in front of the coordination store.

Felix’s bet is that this tier should be a system rather than a bespoke component rebuilt per deployment: watch semantics as a first-class primitive on infrastructure designed for fanout and throughput from the start, rather than bolted in front of a quorum store.

That bet is unproven, and it is the load-bearing claim of this entire document. It fails if coordination stores turn out to be fast enough for most real workloads, or if the consistency Felix gives up to get fanout turns out to be the part that mattered. Both are live possibilities.

2. What Felix does today

These are shipped and measured. If you need one of these, Felix is usable now.

Capability	Status	Notes
QUIC transport, TLS 1.3 always on	✅ Today	Multiplexed streams, no head-of-line blocking, tuned path-MTU/cwnd/socket buffers. The broker serves an operator-supplied certificate (`FELIX_TLS_CERT`), re-read on rotation, with optional client certificates (`FELIX_TLS_CLIENT_CA`); without one it generates a self-signed development certificate and warns
Publish / subscribe with high fanout	✅ Today	Shared-frame fanout encodes a publish batch once regardless of subscriber count
Batched publish and batched delivery	✅ Today	Count- and time-bounded, JSON or binary framing
Gap-free “current state + subsequent changes” subscribe	✅ Today	A subscribe with a start position on a durable stream reports `start_offset` (the first record delivered) and `live_offset` (the tail at join). Everything below `live_offset` was already there, everything from it on is new, and nothing is skipped between them, including under concurrent publishes. A stream has no per-key current value, so “current state” here is a position rather than a snapshot
Idempotent producers	✅ Today	A producer takes an id from the broker and numbers its batches per shard (`FEATURE_IDEMPOTENT_PRODUCER`), keyed or not. The leader answers a re-send of a sequence it already holds without appending it, and refuses a gap or a forgotten producer with a typed reason. A different batch sent under a sequence it holds is never written. On a durable stream the sequences are stored in the log and replicated, so they survive a failover, a planned move and a restart. `ClusterClient::idempotent_producer` re-sends across reconnects and follows the leader. The producer owns its client and sends from its own task, so a call dropped by a timeout or a hung-up handler still runs to its answer and the producer stays usable (`a_dropped_publish_leaves_the_producer_usable`). An in-memory stream loses its sequences with its leader. Kafka idempotent producers get the same checks. See Idempotent producers
Conditional publish and commit (single-writer fence)	✅ Today	`publish_if` appends a batch only if it would start at the offset the writer expects, the shard’s next, and `commit` takes the same `expected_offset` as a compare-and-set on the whole shard. The check is made under the lock that assigns offsets, so of two writers at one offset exactly one is written (`of_writers_racing_at_one_offset_exactly_one_wins`). A refusal is `publish_refused` with `offset_mismatch` and the tail; it writes nothing and consumes no offset (`a_refused_publish_leaves_no_record_and_no_gap`). The tail counts a new leader’s generation-start record, so a writer’s expected offset goes stale on failover (`a_generation_start_moves_the_expected_offset`). Negotiated as `FEATURE_PUBLISH_CONDITIONAL`. Served by the shard’s leader and never forwarded; on a `Leader` stream the fence is only as strong as the lease. No per-key version precondition yet (#1051), and a lost answer has to be resolved by reading the shard. In the Rust client only so far. See Conditional publishes
Bounded per-subscriber queues	✅ Today	Explicit depth limits at every stage, a byte bound on each subscriber’s broker queue (`FELIX_SUB_QUEUE_MAX_BYTES`), and a per-connection byte bound on the writer lanes behind it, shared out so one stalled subscription cannot hold up the others under the default lane policy (`FELIX_SUB_EGRESS_MAX_BYTES`, with one batch always admitted per subscription; under `FELIX_SUB_LANE_QUEUE_POLICY=block` a subscription with a full frame queue stops its connection’s writer), with an optional broker-wide `FELIX_SUB_QUEUE_MAX_BYTES_TOTAL`. Replay rings and cache watch queues are byte-bounded too (`FELIX_REPLAY_RING_MAX_BYTES`, `FELIX_CACHE_WATCH_MAX_BYTES`, each with a `_TOTAL`). A subscriber can ask for its own broker-side queue capacity on subscribe; the broker clamps it to `FELIX_SUBSCRIBER_QUEUE_CAPACITY_MAX` and answers with what it granted (`FEATURE_SUBSCRIBE_QUEUE`). The overflow policy stays the stream’s. See Choosing a queue size
Slow-consumer isolation	✅ Today	`DropNew` (default) or `Block`, set by the operator; see `SubQueuePolicy`. `DropOld` is accepted but behaves as `DropNew`: the arriving record is dropped, not the oldest
Key/value cache with TTL	✅ Today	Scoped `(tenant, namespace, cache, key)`, expiry against an absolute expiry time that survives a restart; on a log-backed cache the leader writes a delete when an entry lapses, so watches see it go
Conditional cache writes (compare-and-set)	✅ Today	`cache_put_if` writes only if the key is absent or at the version the caller read, and `cache_delete_if` removes it only at that version, for leases, idempotency keys and read-modify-write edits. The check and the write are atomic per key on its owner, including against a write still waiting on its fsync (`put_if_absent_waits_for_a_staged_write_to_the_key`). A refusal answers with the key’s current version. A version is the offset of the put that wrote the value; compaction carries it and a restart replays it. Both cache backends; negotiated as `FEATURE_CACHE_CONDITIONAL`. In the Rust client only so far. See Conditional writes
Keyed cache watch	✅ Today	`cache_watch` delivers every applied write to one key or key prefix, deletes included, in the shard’s write order with its log offset. A watch resumes by offset with no gap or duplicate, and retained delivery (`FEATURE_CACHE_WATCH_RETAINED`) starts from each key’s current value. A watch that falls behind is ended with the offset to re-watch from, so a drop is never silent. It survives a leader failover. Log-backed caches only. `ClusterClient::watch_cache_sharded` watches a prefix across every shard, in the Rust client only. See Keyed watch
Counters (deltas folded into a durable sum)	✅ Today	`counter_add` applies a signed delta and answers with the sum including it: one round trip to increment and know where you stand, for rate limits, quotas, and live tallies. Scoped and routed exactly like cache keys, stored beside the cache; the sum survives restart (rebuilt by folding the log), compaction (deltas collapse into a checkpoint without renumbering the log; regression-tested), and leader failover (the counter log replicates with its cache shard, and the promoted replica keeps counting from the true sum). At-least-once, stated honestly: a retried add after a lost acknowledgement double-counts, and the decision is recorded in `docs/projections.md`
Multi-tenant scoping	✅ Today	Tenant and namespace required on all data-plane operations
Per-tenant quotas and connection limits	🚧 Partial	A tenant’s publish rate (bytes and messages per second) is capped per broker, across QUIC and Kafka, with a typed retryable refusal for acked QUIC publishes and `throttle_time_ms` for Kafka; one source address may hold a bounded number of connections; publish and delivery are counted per tenant under a label cap. Publishes queue per shard behind a per-tenant fair queue (deficit round robin) with a guaranteed share per tenant, so one tenant flooding a broker is refused with a retryable `overloaded` while others still get in, and a slow shard, forward or quorum wait holds up only its own shard. Not built: quotas on subscriptions, cache or storage, and quotas stored in the control plane rather than each broker’s environment
Token-based authorization	✅ Today	OIDC token exchange (narrowable by action, resource, or per action and resource pair), tenant-scoped JWTs, broker-side permission checks on publish/subscribe/cache, with cache grants down to one key or key prefix. Tokens are checked per QUIC stream, so one Rust `Client` can act for many users over the same connections (`Client::with_identity`), each request checked against its own user’s token and each user given their own subscription cap and publish byte budget on the connection under a shared ceiling of four users’ worth by default. `ClusterClient::with_identity` does the same across a cluster, following moved shards over the shared connections, and the Python and TypeScript clients expose it with a fixed token. Where tokens are bound to client certificates, such a client first trades each user’s token at the control plane (`POST /v1/tenants/{tenant_id}/token/delegate`) for one naming it as the actor (`act`), which the broker accepts on the client’s own certificate. Revoking a principal, or removing an RBAC rule that reached it, ends its open streams, subscriptions, cache watches and Kafka connections on every broker within about one control-plane sync interval, with a `revoked` error code. Broker-to-broker mTLS with the certificate’s name bound to the node id, required for a cluster member unless explicitly waived
Publisher identity on delivered events	✅ Today	A subscriber that offers `FLAG_EVENT_BATCH_PUBLISHER`, and a group consumer that offers `FEATURE_GROUP_PUBLISHER`, is told the principal (token `sub`) the broker accepted each write from. A durable stream stores it with the record, so replay, promoted replicas and consumer groups report the same value, once enabled: the `publisher_principal` fleet feature, or `FELIX_RECORD_PUBLISHERS` on a single broker. It vouches for who wrote, not for the payload. The Rust client asks with `ClientConfig::publishers`; the Python and TypeScript bindings do not expose it yet
Record times and replay from a time	✅ Today	A subscriber that offers `FLAG_EVENT_BATCH_TIMESTAMPS`, and a group consumer that offers `FEATURE_RECORD_TIMESTAMPS`, gets each durable record’s append time, the same live and on replay. `offset_for_time` (on `Client` and `ClusterClient`, which asks the shard’s leader) answers the first offset on a shard appended at or after a time, the same binary search the Kafka listener’s `ListOffsets` uses, and a client subscribes there. Times are the clock of the broker that led when the record was written. A follower stores the leader’s time with each record it replicates (`RECORD_TIMES` between brokers), so a promoted replica reports the same times after a failover; during a rolling upgrade, records replicated between an older broker and a newer one take the follower’s clock. A clock that steps back makes the lookup approximate. In-memory streams have no times
Control plane metadata service	✅ Today	REST + OpenAPI, tenant/namespace/stream/cache CRUD, an all-or-nothing batch create of a namespace’s streams and caches (one Postgres transaction or one Raft log entry, safe to retry), snapshot and changes feeds, every one of them behind a Felix token (tenant resources by the tenant’s own manage permissions, the tenant catalog and the feeds by cluster scope), with in-memory or Postgres backing. Any number of identical instances over one HA Postgres: a rolling restart of every instance, with broker heartbeats and watches flowing throughout, serves every call. This is proven by test, with the database’s own HA an explicit operational contract (`docs/ha-postgres.md`). Tenant bootstrap is atomic and exactly-once across instances, with token rotation and optional mTLS on the bootstrap listener. The API serves TLS when given a certificate (`FELIX_CONTROLPLANE_TLS_CERT`) and is plain HTTP otherwise. A stream or cache created under a deleted name starts empty on every broker once `stream_incarnations` (or `cache_incarnations`) is finalized, as on a new cluster; before that the name is refused with 409 rather than reopening the deleted one’s logs (#1447)
Prometheus metrics, health endpoints	✅ Today	Plus opt-in `telemetry` feature for per-stage timings
Kubernetes packaging	✅ Today	A Helm chart at `deploy/helm/felix`: the control plane as a Deployment over Postgres or a StatefulSet under the Raft backend, brokers as a StatefulSet whose pod name is the node id with a volume each, probes and a preStop-plus-drain grace period derived from the values, PodDisruptionBudgets that keep a replication-factor-three quorum, anti-affinity and zone spread, a NetworkPolicy admitting the internal port from brokers only, and optional peer mTLS issued per pod by cert-manager’s CSI driver. Value combinations that are each fine alone and wrong together (an even Raft group, a budget wider than the replica count, a drain longer than its grace period) refuse to render rather than deploying something broken. `task chart:check` lints and renders it every way `ci/` describes and CI runs it
Durable streams	✅ Today	Opt-in per stream via `durable: true`, and only when the broker runs with `FELIX_DURABLE_STORAGE_DIR`. Segmented crash-safe log: CRC-verified records, torn-tail recovery, group commit, three fsync policies. A durable shard is replicated to followers and survives losing its leader. Retention is available and off by default: per stream from the control plane (`max_size_bytes`, `max_age_seconds`, applied without a restart), with `FELIX_DURABLE_RETENTION_BYTES` / `FELIX_DURABLE_RETENTION_SECONDS` for a bound a stream leaves unset; with none, a log grows without bound. On a `Quorum` shard retention never deletes past the commit offset, on the leader or any follower, and that hold is restored at open, before retention runs, so a stream whose followers are down grows past its bound until a majority is back. An acknowledged publish is answered after it is written to the log, under the broker’s fsync mode, unless the broker runs with `FELIX_ACK_ON_COMMIT=0`; under the default `periodic` mode it survives a broker crash, and a machine crash can lose the last 250 ms (#1421). That ack says which offset its batch landed at (`FLAG_BINARY_PUBLISH_ACK_OFFSET`), so a producer can checkpoint without reading it back
Tiered storage	🚧 Partial	Offload only. With `FELIX_DURABLE_OFFLOAD_DIR` set, a stream log’s sealed segments are copied to that directory through the `object_store` crate’s local backend, verified by size and checksum, and recorded in a per-shard manifest before retention may delete them locally. Object keys start with the broker’s node id, so brokers sharing one directory never overwrite each other’s copies. A missing or unwritable directory does not stop a log from opening: the pass reports the failure in metrics and retries, and retention keeps every segment without a recorded copy meanwhile. Off by default. Partial because nothing reads the copies yet (a read below the local head still gets `CursorTooOld`), and there is no cloud backend and no cold cache. See Durable storage
Resumable subscriptions	✅ Today	`Subscribe` takes `latest` / `earliest` / an offset; every delivered event carries its offset (`Event.offset`) so an application can checkpoint and resume at `offset + 1`. Stored history joins live delivery with no gap, backfilling from disk if the live queue overflowed. Durable streams only; bounded by retention when it is configured, unbounded otherwise
Bounded range reads	✅ Today	`stream_read` returns one page of a durable stream shard’s committed records from a start offset, stopping before an optional end, with the offset to continue from. It registers no subscriber and never waits, so it suits loading a known slice of history and stopping. Pages are capped in records and bytes; a start below retention or past the tail is a cursor error. Needs `stream.subscribe`. JSON over the control stream, one round trip per page, so a subscription is still the faster way to export a large range. The Rust client has `Client::read` and `ClusterClient::read`; the Python and TypeScript bindings do not expose it yet
Queue semantics (consumer groups, acks, redelivery)	🚧 Partial	A client polls a group, acknowledges a record or hands it back. An unanswered record is redelivered after the visibility timeout and dead-lettered after `max_attempts`. Cursors and dead letters replicate with the shard, so a promoted replica resumes each group where it was. A group can be created at the earliest record, the tail or an offset, moved, asked for its cursor and in-flight counts, and deleted. A consumer can claim for its own visibility, extend a claim it is still working on, nack with a delay, and dead-letter a record itself. Partial because a group is bound to one shard: `ClusterClient::group_sharded` merges shards on the client, and the broker serves one shard’s group at a time. The default visibility timeout and the attempt bound are broker settings, not per stream or per group. See Queues
Graceful shutdown	🚧 Partial	Readiness flips before the listener closes, and both the control plane and the broker can keep serving across a configurable hold-off so a load balancer can act on it. Bounded drain against one shared deadline, with a forced termination reported rather than logged as clean. A client that negotiated error codes and opens a control stream while its connection drains is told `draining`, so it can go elsewhere at once. Publish workers are supervised and drained explicitly: once connections are gone they write everything still queued, including publishes already acknowledged on enqueue, before replication catches up and the logs are flushed. Partial because, inside a connection, acknowledgement waiters and subscription writers are still not told to wind down early: the drain waits for each connection task as a whole
Sharding	✅ Today	A stream has a shard count, and the control plane assigns each shard an owner. A publish may carry a routing key that picks the shard, or name the shard outright (`FEATURE_EXPLICIT_SHARD`, on publishes, `publish_if`, commits and state reads; Rust client only), which is refused rather than wrapped when the stream does not have it (`a_shard_out_of_range_is_refused`). Streams created after the fleet finalizes `jump_hash_routing` map keys with jump consistent hashing, and older streams keep modulo, so no existing key moves. A publish sent to a broker that does not own the shard is forwarded, and `ClusterClient` learns the owner and sends there directly. Any broker names each shard’s owner on request (`shard_owners`), which is how `ClusterClient` sends cache and counter operations to the owner too. Ordering is per shard, so per key; a single-shard durable stream has total order. A subscription reads one shard, and `ClusterClient::subscribe_sharded` merges all of them on the client. A live stream’s shard count cannot be changed yet
Multi-node clustering and replication	🚧 Partial	Brokers register and are health-checked, shard leaders replicate to followers, and a publish or subscribe that reaches the wrong broker is forwarded or redirected. Partial because the control plane still decides every failover, so with it down a dead leader is not replaced, and `Leader` streams and caches fail over only once the leader’s lease lapses (see Leader failover)
Fleet-wide feature gate	✅ Today	Each broker reports the cross-broker features it implements when it registers, and the control plane tracks which ones every live or draining broker supports. A feature turns on only once it is finalized, which is refused until every serving broker supports it. A new cluster, one where no stream or cache was ever created and no broker has registered twice, finalizes every feature its brokers report at its first broker registration, except `publisher_principal`, which grows every record and stays an operator step; under the Raft backend that waits until every member runs a release that does it. A cluster upgraded from an older release waits for the operator (`felix-controlplane admin features finalize`), so a rolling upgrade can be rolled back broker by broker right up to the finalize. Finalizing is one-way: after it a broker without the feature is refused at registration, serialized with the finalize on every store backend. Readable at `GET /v1/fleet/features` and `felix_broker_fleet_feature_enabled`. It gates jump-hash stream routing, the generation-start record, majority acknowledgement, lease-free reads, fenced caches, atomic commits, group cursor dependencies, stream and cache incarnations, and recording the publisher with each record. See Upgrades and compatibility
Leader failover	✅ Today	A lost leader is replaced by a replica that held the log when it last reported, under the storage identity it is registered with now: a follower that lost its disk, or whose machine crashed under `Periodic` or `None` fsync and so may have lost its unsynced tail, or whose volume was put back from a snapshot taken before its last registration (#1334), comes back under a new identity and, once it has registered again, is not promoted on what it reported before; one that stopped cleanly, or, on Linux, whose process alone was killed in the same boot of its machine, keeps its identity (#1367). A wiped follower has also lost the fence promises it gave, so it answers a new leader’s fence only once its routing view has the shard, and at that generation only for the leader named there (#1267). A promoted leader’s fence counts a replica only under the storage identity the failover’s report named for it, so a follower that came back short after a crash or a wipe does not stand in a majority for records it lost (#1335). If the old leader is gone for good and the remaining followers cannot make such a majority, the shard waits rather than open on the short copy. Under `Periodic`, the default, that includes a leader and a follower whose machines go down together: the follower comes back under a new identity even when nothing was lost, so the shard no longer serves on its return, and waits for an operator; the way out is to stop the promoted broker and run `felixctl placement abandon`, which places the shard on live brokers that keep their copies, takes the furthest one, and loses only what no copy in the new set holds. A leader whose machine crashes that way and restarts before failover gives up the shards it led instead of reopening them with the shorter log, and is replaced by a follower that holds more, without waiting for its lease, or named again at a new generation once it has heard from every follower and none holds more; short of that, the shard waits (#1295). One wiped or crashed before it is marked down, whose leader dies before it registers again, still matches its old identity. A follower without an identity (a broker or control plane on an older release) counts only from a fresh report naming the incarnation it is registered at now, which its leader learns in the peer handshake, so one that restarted since, even one its leader still reaches over a connection to the old process, is not promoted on it. That leaves one window while brokers or control-plane instances are being rolled: when no incarnation was measured on the leader’s connection, because the leader is on an older release, the follower’s process at the time of that connection was (including one wiped and restarted on the current release onto a fresh volume), or the report reaches an older control-plane instance, the control plane stamps the incarnation registered when the report arrives. A wiped follower whose registration lost its identity can then still be counted on a report its leader makes from the follower’s old connection, until the leader redials it. A `Leader` shard whose new leader dies before its first report waits for it rather than promote on the previous leader’s report. A dead leader’s last report stays believed for followers whose identity matches, including across the steps of a move it was leading, so a slow control plane does not leave the shard unplaceable. With the defaults: a broker stopped with SIGTERM first hands each shard it leads to a follower as a planned move, which clients follow without a refusal, for up to `FELIX_SHUTDOWN_HANDOFF_TIMEOUT_MS` (30 s). After a crash, a `Leader` stream or a cache moves once the control plane marks the leader down, 18.75 s after its last heartbeat (`FELIX_NODE_EXPIRY_TIMEOUT_MS` 15 s plus `FELIX_NODE_REGRANT_MARGIN_MS` 3.75 s), checked every 2 s (`FELIX_NODE_EXPIRY_SWEEP_INTERVAL_MS`), and the next placement pass promotes a follower (`FELIX_SHARD_RECONCILE_INTERVAL_MS`, 5 s). Heartbeats go every 5 s (`FELIX_NODE_HEARTBEAT_INTERVAL_MS`), so that is roughly 13 to 26 s after the crash. A shard a stopping broker still leads when its handoff times out waits the same way. A lease-free `Quorum` stream is faster, below. The cluster tests shorten these windows to about a second. A shard with no such replica stays unavailable rather than served empty, which includes a durable shard with `replication_factor: 1`. When every replica supports it, every new leader fences a majority of its replicas before it serves, so a deposed leader cannot acknowledge a write the cluster has lost: a promoted one, a move’s destination at its cut-over, a leader handed a shard back by a cancelled move, and a leader given a generation of a shard it serves that skips one, so another leader may have come between. Cache shards are fenced the same way, on their cache log and their counter log, taking the furthest ahead of each. A `Quorum` promotion, stream or cache, keeps the shard’s replica set, the dead leader in it, so that fence always needs a majority of the replicas that acknowledged; with a majority of them down the shard waits rather than opening without them. Under `majority_ack` (and `fenced_caches` for a cache), a move’s destination cut off from its followers at the cut-over waits too, with no lease fallback: it keeps retrying the fence and serves once it reaches a majority, or is replaced when placement promotes a replica that holds the log. Without them it opens on the lease. Otherwise, and for `Leader` streams and caches, the lease does that job. With `generation_start`, `majority_ack` and `lease_free_reads` finalized, a `Quorum` stream does not wait for a dead leader to be marked down: followers ping the leaders they follow and name the silent ones in their heartbeat, and once a majority of the set names its leader the control plane promotes a follower, about 5 s after the leader went quiet with the defaults (`FELIX_LEADER_SUSPECT_AFTER_MS`), whether it crashed or stopped. That also replaces a leader cut off from its followers but still heartbeating, which the lease never did. The control plane still makes the call, so with it down a dead leader is not replaced, and caches still wait for the lease (#1009). So a `Leader` shard’s single writer rests on lease timing: a deposed leader suspended past the lease margins, or one whose clock runs slow against the control plane’s, can acknowledge a write its successor never sees. A clock step cannot, and `Quorum` with `majority_ack` is not exposed (see where the guarantees stop). Tested against kill, graceful stop, freeze, partition, one-way and delayed links, and clock faults, and run nightly through the history checker, including two brokers down at once, a leader cut off from its peers and the control plane until it fails over, every broker losing power at once, and the control plane crashing in the middle of a move or failover. The promotion protocol is model-checked in TLA+. Both paths go through the campaign: the nightly run uses the lease-free path (`majority_ack` and `generation_start`), every pull request covers both, and 55-minute runs of every single fault family have passed in each mode
Pipelined publishes	✅ Today	A client that offers `FEATURE_PUBLISH_PIPELINE` is granted a per-stream `publish_window` (`FELIX_BROKER_PUBLISH_WINDOW`, 256 by default; `0` turns it off). The broker answers that client’s acked publishes on each stream in the order it sent them, and stops reading a stream’s publishes while its window is full, so a pipelining client is slowed by flow control rather than filling its tenant’s share of the publish queue (`a_full_window_stops_reading_until_an_answer_frees_a_slot`, `a_pipelining_stream_is_answered_in_request_order`). Each stream has its own window, so a stream stuck behind a stalled shard does not stall the others on its connection (`a_stuck_stream_does_not_stall_another_on_the_same_connection`). Old clients get the frames they always got. The Rust client caps each stream at the window, or each connection against a broker that predates per-stream windows; `IdempotentProducer::publish_batches` keeps up to 64 batches in flight and, on a failure, re-sends from the first unanswered one, so a connection dropped, reset or stalled with a full window loses and repeats nothing (the conformance runner’s `fault.publish_*` scenarios run with pipelining too, and `a_pipelining_producer_loses_and_repeats_nothing_through_its_leaders_death`). Where it stops: the Python and TypeScript bindings sit on the Rust client, so their publishes are windowed too, but neither wraps the idempotent producer; and `publish_batch` still waits for each batch
Online rebalancing	✅ Today	A shard with a live leader is moved, not reassigned: the destination copies the log as a replica, the leader is fenced once the destination has nearly caught up, and the switch-over takes about 90 ms on a local debug build. Publishes, cache writes, counter adds and group operations during a move are held and forwarded, never refused, and nothing acknowledged is lost or stored twice. Subscriptions, cache watches and consumer groups follow the shard to its new owner in `ClusterClient`. A draining broker hands off all its shards, and a joining one takes its share. Operators can list, preview, start, cancel and pause moves. The single-broker `Client` returns a redirect as `NotLeaderError` rather than following it. A fenced move whose leader never reports drained is given up after `FELIX_SHARD_FENCE_TIMEOUT_MS` (two minutes), the leader serving again with every write it accepted, and the shard’s next move backs off, doubling up to an hour (#1434). Placement balances shard counts, not load: each stream’s leaders are spread evenly across brokers, then the cluster’s total, so one stream’s writes are not left on one broker (#1429). See Moving shards by hand
Replication factor restore	✅ Today	A shard returns to its replication factor without an operator. A follower whose broker has been down or gone for `FELIX_SHARD_RESTORE_AFTER_MS` (five minutes by default, so a restart is not mistaken for a loss) is replaced by a copy on a live broker, and a replica set left short, by a failover or by placement while too few brokers were live (down to the leader alone), is topped up once one is free. A cluster started all at once is not left short to begin with: in its first `FELIX_PLACEMENT_SETTLE_MS` (10 s) a new shard waits for as many live brokers as its factor. A top-up of a shard holding at most `FELIX_SHARD_MOVE_FENCE_MAX_LAG_RECORDS` records takes no move slot, so many short shards fill at once. The copy joins beside the set and is seated once it has caught up; on a `Quorum` stream it must hold what a majority of the old set held first, as for a drain. The restore is stored in the assignment, so it resumes across control-plane restarts, and a destination that dies mid-copy is replaced by another. A follower its leader reports halted (diverged, or needing records retention removed) is not counted as a copy, is never picked as a move’s destination, and is replaced once it has been halted for the same delay; a move whose destination halts is given up and started elsewhere. `felix_shards_under_replicated`, `felix_shard_replicas_halted` and `GET /v1/placement/replication` show each shard’s current against desired copies and any halted ones with the reason. Model-checked in TLA+ (`FelixShardFencedAckGrow.cfg`) and tested by killing a follower and then its first replacement (`a_lost_follower_is_restored_even_when_its_first_replacement_dies`), and by halting a follower and watching it replaced (`a_halted_follower_is_shown_and_replaced`). See Moving shards
Consistent backup point	🚧 Partial	`felix-controlplane admin backup-point` records, after one barrier instant, each shard log’s committed offset on its leader (`GET /backup/offsets` on the broker’s metrics listener): the quorum mark for a `Quorum` shard, the acknowledged tail otherwise, re-collected if any leader or generation changed meanwhile. Leaders’ shard directories are copied while they run and `felix-broker restore-point` cuts each copy back to the point, below its own commit offset. Proven under load: every record acknowledged before the barrier is in the point and the restored copy, and a `Quorum` shard whose leader can reach no follower is not cut past its mark (`a_point_under_load_is_committed_and_misses_nothing_acknowledged_before_it`, `a_live_copy_restored_to_the_point_keeps_every_ack_before_it_and_nothing_after`). Where it stops: offsets are read one broker at a time with no write pause, so a record written during the collection can be in the point on one shard and not another; a `Leader` shard is cut at what its leader held; a cache, group or counter log compacted after the point cannot be cut back to it and the restore refuses the copy; and control-plane metadata is backed up separately. See Backup and restore
Zone-aware placement	✅ Today	A broker registers its failure domain with `FELIX_NODE_ZONE`, and placement puts each shard’s followers in zones the shard has no copy in yet. A drain goes where the shard keeps the most zones, a rebalance only where it loses none, and the cut-over keeps the copies that hold the spread. A follower sharing a zone with another copy is replaced, paced like any move, once a broker in a missing zone has room, so shards placed before zones were reported are spread too. When a shard cannot be spread (every broker in a missing zone is full) it is placed anyway, with a warning and `felix_shards_zone_unspread`. A broker without a zone is treated as alone in its own, so a cluster that reports none is placed as before. `a_shards_copies_span_every_zone_and_a_drain_keeps_them_spread` runs four brokers in three zones. An operator’s move is never refused on zone grounds: it reports `zones_before` and `zones_after`, can be previewed with `dry_run` (`felix-controlplane admin move --dry-run`), and a narrowing one is logged as a warning. The Helm chart’s `broker.zones` runs one StatefulSet per zone, pinned to it and reporting it. A zone is read when the broker registers, so changing it takes a restart
Quorum acknowledgement	✅ Today	A `Quorum` publish is acknowledged once a majority of the replica set, counting the leader, holds it durably, whichever broker the client reached. One unreachable replica does not stall it. With no reachable majority the publish waits up to `FELIX_PUBLISH_QUORUM_TIMEOUT_MS` (5 s by default), then fails with `quorum_timeout`: the record is on the leader and may still survive, so the outcome is unknown, not refused. Until the `majority_ack` fleet feature is finalized (on an upgraded cluster, by the operator), the leader also reports which replicas hold the record, and the report lands before the acknowledgement is released. With it finalized, as on a new cluster, a `Quorum` stream acknowledges on follower answers at the leader’s generation alone, so a leader cut off from the control plane keeps acknowledging. With `fenced_caches` finalized too, a replicated `Quorum` cache acknowledges puts, deletes and counter adds the same way, and a cache leader cut off from the control plane past its lease loses none it acknowledged (`a_cache_leader_cut_off_from_the_control_plane_loses_nothing_it_acknowledged`, `FelixShardFencedCache.cfg`). `Leader` streams and caches keep the report and the lease. Model-checked in TLA+, through failover, a second promotion, planned moves and a cancelled move, and run nightly and on every pull request through the history checker with `majority_ack` finalized
Linearizable `Quorum` reads	✅ Today	Once the `lease_free_reads` fleet feature is finalized (with `majority_ack` and `generation_start`; a new cluster does so at its first broker registration, an upgraded one waits for the operator), a get or counter get on a replicated `Quorum` cache confirms its broker still leads read-index style, with no clock: after taking its value the broker sends the promotion fence at its own generation to the shard’s replicas and answers only once a majority, itself included, confirms no newer leader reached it. A leader cut off from its replicas cannot serve a read once a new leader has taken writes, whatever its lease says (`a_cut_off_leader_cannot_serve_a_quorum_read_once_a_new_leader_took_writes`), and reads keep working with the control plane partitioned (`quorum_reads_continue_while_the_control_plane_is_partitioned`). Model-checked in TLA+ (`FelixShardReadsRound.cfg`, drifting clocks and no lease margin); the same model without the round, or with the lease in its place, finds the stale read. `FELIX_QUORUM_READS=lease` keeps a broker on the lease. The same feature takes readers and consumer groups off the lease. Subscriptions, replay, Kafka fetches and cache watches on a replicated `Quorum` shard deliver up to the committed mark through a lapsed lease (`readers_and_groups_survive_a_control_plane_partition`) and end once a replica refuses their broker for a newer leader or no majority has confirmed it for a lease duration, after which `ClusterClient` follows the shard (`a_deposed_leader_ends_its_subscriptions_and_readers_follow`). A group poll, ack, nack or dead-letter change is confirmed by a round before it is acknowledged, so a replaced coordinator cannot acknowledge one (`a_deposed_coordinator_refuses_a_group_ack`); modelled in `FelixShardSessions.tla`. Until the feature is finalized, and on `Leader` streams and unreplicated shards, reads and readers follow the lease
Client surviving a broker failure	✅ Today	`ClusterClient` takes several broker addresses and rebuilds its connection from the rest when the one it is using fails, so an application outlives the broker it happened to connect to. It holds one QUIC connection per broker, shared by every role that broker plays (entry, shard owner, redirect target, producer leader) with every stream multiplexed on it, and opens more only when those streams saturate it, up to a ceiling; a connection that dies fails only its own streams and is replaced (`one_connection_per_broker_under_light_mixed_role_use`, `a_dead_connection_is_replaced_and_only_its_streams_fail`). `publish` reports the failure with the connection already replaced; `publish_at_least_once` also resends, and documents that a resend can duplicate. A client given one broker address asks it which brokers exist and adds them to what it will try, so the address it was configured with stops being a single point of failure; the configured seeds are always kept, so a wrong or stale answer cannot leave it worse off. When no address it holds answers, as after a rollout that gives every broker a new IP, it resolves its `host:port` seeds and the names brokers advertised again and dials what is new, at most once a second for a client and its identities, each lookup bounded to 2 s (`a_name_is_resolved_again_when_every_broker_has_moved`, `concurrent_resolutions_share_one_lookup`, `a_cancelled_caller_does_not_cancel_the_resolution`, `a_hanging_lookup_neither_holds_the_client_nor_hides_other_names`, `a_hung_system_lookup_is_shared_and_blocks_no_other_name`). A stalled broker cannot hang the client: each connect gives up after `connect_timeout` (10 s), and `finish` after `finish_timeout` (60 s) (#1433). A reconnect replaces the client only if it is still the one that failed (`a_reconnect_for_a_client_already_replaced_does_nothing`); a client given only IPs, against brokers that advertise only IPs, has nothing to resolve. A publish in flight when the leader fails is resolved by the idempotent producer on a durable stream: it re-sends until the promoted leader answers, and the sequence stored in the log makes the re-send land once. `a_producer_publishing_through_its_leaders_death_loses_and_repeats_nothing` kills the leader mid-stream and finds every record exactly once, in order. `publish_at_least_once` still documents that a resend can duplicate: that is its contract, and the idempotent producer is the path without the duplicate. A subscription whose broker dies is not ended quietly: `next_event` returns `SubscriptionLost`, and a `ClusterClient` subscription on a durable stream resubscribes from the offset after the last one it delivered (`a_subscription_resumes_after_its_broker_is_killed`). The same holds when the network rather than the broker fails: the conformance suite drops, resets and stalls the client’s link mid-publish and mid-subscribe, and the Rust, Python and TypeScript clients each resume with no gap or report the loss, never end the subscription as if the stream had finished (`fault.*` in the catalogue)
Kafka wire compatibility: consumers	✅ Today	With `FELIX_KAFKA_LISTEN` set, every broker serves Kafka consumers that assign their own partitions: durable streams appear as `<namespace>.<stream>` topics, partitions are shards, offsets are Felix’s log offsets, and a fetch follows its partition to a new leader after a move or failover. SASL/PLAIN with a Felix token. Tested with kcat. Consumer groups are refused with an error that says why, so Connect, Streams and ksqlDB do not work. See Kafka compatibility
Kafka Produce	✅ Today	Kafka producers write to durable streams through the broker’s publish path on the shard’s leader: gzip, snappy, lz4 and zstd; `acks` 0, 1 and all, where all waits for the stream’s own consistency (a majority on `Quorum`, the leader on `Leader`). Idempotent producers get a Felix producer id, and their sequences are kept in the log, so a batch re-sent after a failover or a move is answered as a duplicate, not written twice. Keys, headers and producer timestamps are dropped. Transactions are refused with an error that says why. Tested with kcat and across a failover and a move. See Kafka compatibility
Log-backed cache (one core log, many semantics)	🚧 Partial	A cache is a log: writes append records, reads go through an index of key → offset rebuilt from the log at startup (from the snapshot the last compaction wrote, replaying only the records after it), and compaction reclaims superseded and expired records without ever rewriting one, on a background task with its own I/O budget, so no write waits for it. Entries survive a restart when the broker has `FELIX_DURABLE_STORAGE_DIR`; without one the cache is in memory, because there is nowhere to write a log. Cache operations are routed: a key hashes to a shard, that shard has one owner, and a broker that receives an operation for a key it does not own forwards it there, so a value written through any broker is readable through every other. A cache’s shards are replicated and survive the loss of their leader. Put, get, delete and their conditional forms are all on the wire, with delete reporting the value it removed, and the log is what makes the keyed watch above possible, since every change has an offset to resume from. A cache declares `Leader` or `Quorum` like a stream, and under `Quorum` a put, delete or counter add is acknowledged only once a majority of its shard’s replicas hold it. Partial because each shard keeps its whole key index in memory: memory grows with the key count and nothing caps or reports it (#1073). A shard that has never compacted, or whose snapshot does not match its log, still rebuilds from the whole log on its first request
Atomic event + state commits (one shard)	🚧 Partial	`commit(entity_key, [publish or enqueue, put, delete])` writes an event and keyed state updates as one record on the stream shard the entity routes to, and `state_get` reads the state with the commit’s offset as its version. No reader sees part of a commit: subscribers, history, consumer groups and the Kafka listener see the event at its offset, and the state changes under the same lock, held with the event on a `Quorum` stream until the mark passes it. Modelled across failover and promotion in TLA+ (`FelixAtomicCommit`), and checked under faults by the history checker’s partial-commit rule. An operation on another stream is refused with a typed error and never split: atomicity across logs is a non-goal. A cluster must finalize the `atomic_commit` fleet feature first. Partial because a commit is not idempotent (a re-sent commit may land twice), state is rebuilt from the retained log with no snapshot, and there is no read at an older version. See `docs/atomic-commit.md`
Raft consensus for cluster metadata	🚧 Partial	Complete as a capability: `FELIX_RAFT_NODE_ID`/`_DATA_DIR`/`_PEERS` select a Raft-replicated metadata store with no external database: an openraft group embedded in the control-plane instances behind a seam, a deterministic command-driven state machine, invisible follower→leader write forwarding, leader-gated sweep and placement, quorum-aware readiness, a documented Postgres migration and DR path, and a chaos suite in which three real binaries survive rolling restarts, a SIGKILLed leader, a frozen-then-thawed leader, and a wiped follower volume with zero failed calls and every acknowledged write on every member. A member that comes back with a wiped volume does not vote until it has caught up with the group, because voting from an empty log can elect a member that is missing acknowledged writes. Partial for the same reason leader failover is: the injected faults are the ones a single machine can produce, plus the known pre-0.10-openraft bound that restarting the leader pauses writes one election (~1.2s). Postgres remains fully supported per `docs/ha-postgres.md`. Design and findings: `docs/metadata-raft-design.md`. Raft here is for metadata only, deliberately not for stream records: that is leases plus log shipping per `docs/replication-design.md`
Cross-region routing and data sovereignty enforcement	🚧 Partial	A stream created with a `region` has its leader and every replica only on brokers in that region, or in a region the directional allowlist `FELIX_REGION_BRIDGES` bridges it to. That holds for first placement, rebalancing and drain moves, failover promotion and operator moves, which are refused as `region_not_allowed`; a copy found outside the allowed regions, because a broker’s region changed or a bridge was removed, is moved back in the way a draining broker’s copies are. A broker forwards a request only to a leader in its own region or one its `FELIX_REGION_BRIDGES` reaches, and otherwise refuses it as `shard_unavailable` with reason `region_not_routable`. `a_stream_homed_in_one_region_is_refused_by_another` runs two regions with no bridge and checks both. Partial because: a stream created without a region, and every cache, is placed anywhere; the allowlist is per process, so the control plane and each broker must be given the same one; brokers ship to the replica set the control plane assigned without checking its regions themselves; a follower already outside the region stays until a broker inside it can replace it; a client connecting straight to a leader is not asked where it is; and there is no bridge agent, per-region encryption or audit log of cross-region movement
Non-Rust client SDKs	🚧 Partial	Python ships, as a binding over the Rust client rather than a reimplementation, and passes every required scenario in the client conformance catalogue (publish and subscribe, consumer groups, cache watches and multi-shard subscribe, on both a synchronous and an asyncio surface). Two optional scenarios stay unclaimed rather than passing: `at_least_once` does not carry a routing key, and a prefix watch over a multi-shard cache needs one watch per shard. TypeScript ships too (`crates/sdk/felix-typescript`, a napi-rs addon over the same client) with the same surface, and passes every required scenario as well; it leaves idempotent producers and `error.bad_offset_is_typed` unclaimed rather than passing. Both raise errors typed from the broker’s error code, with the code and retry class on the error, so a refusal during a shard move reads as retryable and a quorum timeout as outcome-unknown. All three publish to their language’s registry: `felix-client` on crates.io, PyPI and npm, the same name in each. Neither binding wraps conditional writes, idempotent producers, named-shard commits or merged prefix watches yet; each client page lists what is missing. C# is next, and Go after it. The C ABI C# will bind to has started (`crates/sdk/felix-capi`): connect, publish and a polled subscribe behind a generated header, driven from C against the conformance fixture in CI. It is brought up to the Rust client’s surface before the C# SDK starts. Each SDK will be gated on the same suite. See Clients in Other Languages
Command-line tool	🚧 Partial	`felixctl` publishes, subscribes from an offset or a shard, reads, writes and watches cache keys, shows shard owners, lists the control plane’s tenants, namespaces, streams, caches, brokers and shard assignments, creates, changes and deletes tenants, namespaces, streams and caches, drains and deregisters brokers, moves shards and pauses placement, lists, adds and removes RBAC policies and role assignments, works consumer groups (create, describe, seek, delete, poll, ack, nack, extend, dead letters) and counters, shows a shard’s live state from each of its brokers with `felixctl inspect shard` and each broker’s subscriptions with `felixctl inspect subs`, and runs felix-loadgen’s scenarios as `felixctl bench`, with `--json` throughout. Deletes and drains ask first on a terminal and need `--yes` elsewhere. Each release ships it as binaries for Linux, macOS and Windows, a Homebrew formula, a container image and a crate. State reads are not built yet (#1005)
Operator inspection	🚧 Partial	`felixctl inspect shard` asks the leader and every replica of a shard for its own view, over the authenticated client listener with `node.view:cluster:*`: whether the leader serves and why not (opening, fencing, failed with the open error, draining, lease lapsed, behind the assignment’s generation), which replicas took a promoted leader’s fence, the lease, the tail and commit mark, and each follower’s position, lag and state. `felixctl inspect subs` asks each broker for the subscriptions it serves, a bounded page at a time: the stream and shard, the connection, client address and principal, the overflow policy, queue depth and capacity, records dropped, and the position against the shard’s tail. Read-only, never forwarded. `felix-broker inspect segments` reads a broker’s data directory from disk without starting a broker, strictly read-only: every shard’s segments, whether their records and indexes verify, and the verdict startup would reach (open, repair a torn tail, or refuse and where), from the same recovery plan startup runs. Partial because a connection listing, the control plane’s decision log, decoding a data directory’s records offline and locating one record are not built yet (#1077). See Diagnosing a cluster

Measured performance

The current numbers are on v0.6.0 Performance, measured on Azure over a real network with TLS, with every condition stated. In short, on one 8 vCPU broker with local NVMe:

3.6 GB/s of in-memory 4 KiB messages fire-and-forget without losing any, and 4.4 GB/s acked.
1.40 to 1.48 GB/s durable with an fsync before every ack, which is the disk’s own rate.
About 0.3 ms for one fsynced durable publish when nothing else is running, and 160 to 180 µs in memory.

The same page covers a three-broker cluster with three copies of every durable record, and a comparison with NATS JetStream on the same machines. The loopback harness and its methodology are under Benchmarks; its numbers catch regressions in Felix’s own code and are not a measure of a deployment.

Delivery semantics today

At-most-once for a plain subscription. A subscriber is delivered to best-effort: no redelivery, and no publisher-visible confirmation that anyone received anything. This is the right default for the fanout workloads Felix is aimed at, where the next update supersedes the one that was dropped.
At-least-once through a consumer group. Polling a group is the other shape: a record is claimed rather than pushed, redelivered if it is not answered for within the visibility timeout, and dead-lettered once it has been attempted too many times. A consumer must expect the same record twice, because a crash after handling and before acknowledging is indistinguishable from a crash before handling. See Queues.
Idempotent producers, not exactly-once delivery. A producer that takes an id from the broker and numbers its batches can re-send a publish whose acknowledgement never arrived and have it land once. The shard’s leader answers a sequence it already holds without appending it, and on a durable stream that holds across a failover or a move too. Delivery stays one of the two shapes above: a consumer can still see a record twice on redelivery. Deduplicate there, keyed on something in the record.
Per-shard ordering preserved for a given subscriber on a durable stream, which means per key on a multi-shard stream and total order on a single-shard one. No ordering across streams. An in-memory stream’s fanout is not ordered against concurrent publishes, so different subscribers of the same in-memory stream can see a racing pair in different orders.
Resumable subscriptions over the wire, for durable streams. Subscribe carries an optional start (latest, the default and what every older client sends; earliest; or an exact offset), and every delivered event carries its offset, so an application checkpoints what it handled and resumes at the next one. History read from disk joins live delivery with no gap and no duplicate. Asking for an offset retention has discarded is a typed error, not a silent restart at the tail. A non-durable stream stays tail-only beyond its bounded replay ring, because there is nothing older to read.
Slow subscribers drop under the default policy. On a durable stream the subscription ends at the first drop with the offset to resume from, and the cluster client resumes there from the log; on an in-memory stream the subscriber is not told. Publishers never block on subscriber speed.
Publishes can be shed under overload. With the defaults, when the broker’s publish queue or byte budget is full, a fire-and-forget publish is dropped (at most once) and an acknowledged one is refused with a retryable overloaded that the application must handle; on a durable stream it waits up to publish_queue_wait_timeout_ms (2 s) first. With pub_ingress_wait, fire-and-forget publishers wait for room instead. A waiting publish is parked off its connection’s read loop, so publishes behind it to other shards go on (#1416).
Ephemeral by default. Nothing survives a broker restart unless the stream was registered with durable: true, which writes each record to its log before acknowledging it (unless the broker runs with FELIX_ACK_ON_COMMIT=0) and replays it afterwards. Durable storage is opt-in per stream and off unless the broker is started with FELIX_DURABLE_STORAGE_DIR. Retention is available but off by default: a stream’s own retention, or the broker’s FELIX_DURABLE_RETENTION_BYTES / FELIX_DURABLE_RETENTION_SECONDS for a bound it leaves unset. With none, a durable stream grows without bound; set, the oldest records are discarded and a resume below them fails with a typed error naming the oldest retained offset. Tiered storage stops at offload: sealed segments can be copied to a directory before retention deletes them, but nothing reads the copies back (#172).

3. What Felix is being built to become

None of the following exists today. It is listed so the intent is legible, not so it can be planned around. Capabilities move up into the table above when they ship rather than being marked off here.

Capability	Status	Current state of the code
Load-aware placement	🎯 Target	Not implemented. Placement spreads each stream’s leaders evenly across brokers, then the cluster’s, and moves a shard off a broker only when that broker leads more than its share of a stream’s shards or of all shards, counted, or is draining. It reads no traffic, disk or CPU figures, so a broker leading its share of hot shards is left as it is. An operator can move a hot shard by hand (`felix-controlplane admin move`)

Target scale, stated as ambition and nothing more: a single update fanning out to tens of thousands of subscribers across a multi-node cluster. Measured fanout today is 10, on loopback, single-node. There is no benchmark, model, or napkin-math projection behind the target figure yet. Treat the gap between 10 and “tens of thousands” as unexplored engineering. Nobody has validated a scaling curve for it.

The target architecture is described in full in Design and System Design.

4. Workload fit

Two separate questions, deliberately kept apart. “Fits the architecture” is about whether the workload matches where Felix is going. “Usable today” is about whether you could build it on the current release.

Workload	Fits the architecture	Usable today	Why
High-fanout real-time event distribution	Strong	Yes	Fanout, isolation, and tail latency are the shipped strengths
Live operational dashboards, telemetry feeds	Strong	Yes	Tolerant of at-most-once and of loss under lag
Ephemeral coordination between services	Strong	Yes	Low-latency, no durability needed
Internal service event bus	Moderate	Mostly	Works, but NATS and RabbitMQ serve this well already; weak differentiation
Distributed live-state synchronization	Strong	Partly	On log-backed caches the primitive exists: a retained `cache_watch` starts from current values, resumes by offset with no gap, and a watch that falls behind is ended with the offset to re-watch from rather than dropping silently. In-memory caches and non-durable streams still drop under lag with no way to resynchronize (a durable stream’s subscription is ended with the offset to resume from), and only the Rust client watches a prefix across shards
Infrastructure / control-plane state distribution	Strong	Partly	The same primitive, on a multi-node story that is still 🚧 (see section 2)
AI-agent coordination and shared state	Strong	Partly	Ephemeral coordination works. Durable task state replicates to followers, and a lost leader fails over to one that holds the log
Edge and disconnected operation	Strong	No	Durability, resumable subscriptions, and replication exist; retention is available but bounded by one machine’s disk, and there is no store-and-forward between sites
Durable event log, replay, event sourcing	Weak	Partly	A durable log with offset replay exists, records replicate to followers, and retention can bound growth. Sealed segments can be copied to a directory before retention deletes them, but nothing reads the copies back yet; use Kafka for anything that needs readable history to outlive one machine today
Primary datastore	Weak	No	Use a database
General-purpose key-value store	Weak	No	Use Redis or Valkey
Complex broker routing, workflow messaging	Weak	No	Use RabbitMQ
Strongly-consistent coordination, leader election, locks	Weak	No	Use etcd, Consul, or ZooKeeper

Read the two columns together and the current position is uncomfortable but worth stating plainly: the workloads Felix is differentiated for are the ones it cannot serve yet, and the workloads it serves today are ones several mature systems already serve well. That is a normal place for an early project to be. It is not a place to make strong adoption claims from.

What the target workloads actually carry

The Strong-fit rows that are not fully usable yet are the ones the architecture is aimed at. What each moves:

Infrastructure and control planes: distributed configuration, feature flags, service discovery, routing tables, authorization policy, deployment state, cluster membership, certificate rotation. Kubernetes control planes, service meshes, fleet management, private and hybrid cloud. The traffic is live infrastructure state rather than durable business events.
Agent fleets: task assignments and status, tool results, model configuration, resource availability, cancellation signals, intermediate results, shared scratch state. Attractive because agent systems naturally produce many concurrent producers and consumers, high event rates, fanout, ephemeral data, and wildly heterogeneous consumer speeds, which is close to Felix’s intended operating point.
Edge and device fleets: site- and device-level state pushed outward rather than polled inward, under intermittent connectivity, constrained bandwidth, and high latency. The model is that the edge holds a local copy and reconciles when a link returns. This is the furthest from what exists today.
High-fanout real-time data: dashboards, telemetry, IoT, multiplayer and collaborative applications, live-event and market-data-style feeds. This one is largely usable now, because it tolerates loss.

The open design question

The state-synchronization thesis and the current delivery semantics are in real tension, and it has not been resolved.

For an event feed, a dropped message means a consumer missed one update. For a consumer maintaining a local copy of state, a dropped message means its local copy is permanently wrong with no signal that would let it recover on its own. At-most-once delivery plus DropNew is a correct design for the first case and an incorrect one for the second.

Closing this requires at least one of: gap-free snapshot-plus-stream subscribe, resumable subscriptions with a durable log behind them, or an explicit resynchronization protocol triggered by the lag signal subscribers already receive.

The second of those exists for durable streams: a subscriber that records the offsets it handles can resume from them. Because offsets are contiguous apart from the generation starts the broker reports, it can also detect a drop rather than diverging silently. That narrows the case rather than closing it. It does not help a non-durable stream, it requires the application to checkpoint, and a resume reaches only as far back as retention keeps: unbounded when retention is unset, and no further than the configured bound when it is.

For keyed state the first and third exist too, on log-backed caches: a retained cache watch delivers current values and then changes with no gap, and a watch that falls behind is ended with the offset to re-watch from, which is the resynchronization signal. What is left is everything that is not a log-backed cache: an in-memory cache or stream still drops under lag with nothing to resynchronize from. A durable stream’s subscription now gets the same signal a cache watch does: it is ended at its first drop with the offset to resume from.

5. What Felix is deliberately not

Felix should not be positioned against any of these, now or later.

Not a Kafka replacement. Kafka is excellent at durable event history, long-term retention, replay, and data integration. Felix’s durability is meant to serve live distribution rather than long retention. Sealed segments can be copied to a directory before retention deletes them, but nothing reads them back yet, and retention is off unless it is configured. Speaking part of Kafka’s protocol does not change that: it lets Kafka producers and partition-assigning consumers reach Felix streams, not run Kafka’s ecosystem.
Not a Redis replacement. Cache semantics in Felix exist as part of a state-distribution model, not as a general-purpose data-structure server.
Not a RabbitMQ replacement. Elaborate routing topologies and traditional enterprise queueing are not the differentiator.
Not a database. Felix is not the system of record.
Not a NATS competitor on general-purpose messaging. NATS is fast, mature, broadly deployed, and already covers general high-performance pub/sub, with JetStream and its KV store overlapping parts of Felix’s target surface. This is the closest neighbor, and “we are like NATS but faster” is not a position.
Not “a faster message broker.” That is a crowded category and the claim would rest on single-node loopback numbers.

Likewise, “Rust” and “QUIC” are implementation choices rather than the value proposition, though QUIC is load-bearing for the eventual edge story (connection migration, 0-RTT resumption, no head-of-line blocking).

Where that leaves Felix

What each system is best at:

System	Owns
Databases	Storing authoritative state
Kafka	Durable event history and replay
Redis / Valkey	Fast data structures and caching
RabbitMQ	Traditional broker routing and workflow queueing
NATS	General-purpose high-performance messaging
etcd / Consul / ZooKeeper	Strongly-consistent coordination and watch, at modest scale
Felix (target)	Watch semantics at fanout and throughput coordination stores don’t reach

The nearest neighbour is the coordination-store row rather than the messaging rows. See Why not etcd, Consul, or ZooKeeper? for the argument and its failure modes. The snapshot-plus-stream primitive exists for log-backed caches; Felix becomes credible against that row only once the multi-node story is real and fanout has been measured somewhere well past 10.

6. How to decide whether Felix fits you

Felix is a strong fit today if your workload has most of these:

one producer or few producers, many consumers;
updates that matter in milliseconds, not seconds;
consumers that run at genuinely different speeds, where one slow consumer must not affect the others;
either tolerance for at-most-once delivery, or work that suits a consumer group’s poll-acknowledge-redeliver shape;
durability that is opt-in per stream rather than assumed everywhere.

Felix is usable but early if you need multi-node operation. Placement, replication, quorum acknowledgement, leader failover and client redirection are all implemented and tested. Faults are injected into the process, the network (including one-way and delayed links), the clock, the control plane and the disk, and the nightly history campaign draws from all of them, on both the lease and the lease-free acknowledgement path. All of it runs on one machine. There is no cluster-scale latency budget (#136). Run it where you can tolerate finding the next bug.

On a stream or an in-memory cache, Felix is a strong fit for the target architecture but not yet usable if you need a consumer to reconstruct keyed state after a disconnect in one call. Asking a durable stream for changes since an offset is gap-free, but a stream has no per-key current value, and stitching one together yourself races. A retained watch on a log-backed cache does give current state and every change after it.

Felix is the wrong tool if you need long-term event history, tiered storage, transactional guarantees, exactly-once processing, or a mature multi-language ecosystem. There are three clients, Rust, Python and TypeScript, and the latter two wrap the first without covering every surface.

Two things are not goals at all. Felix does not encrypt payloads end to end: TLS protects each hop, but a broker reads payloads in the clear, so a client that must hide them from brokers encrypts them itself. And Felix makes no compliance claim.

7. Why this framing, and not a broader one

One reason to organize the project around this specific question (how do you keep a large distributed population synchronized with rapidly changing state?) is that the requirements it generates are the ones Felix has been building anyway, on both sides of the today/target line:

shipped because the thesis demands them: high-performance transport, persistent connections, efficient serialization, encode-once fanout, bounded subscriber queues, slow-consumer isolation, snapshot-plus-change subscriptions, durability, replication, resumable subscriptions;
targeted because the thesis demands it: edge operation.

A generic message broker would not obviously need that set, or build it in that order. The framing is useful because it explains and constrains the roadmap.