Coming from NATS
A team runs NATS for service messaging. Core NATS carries live updates and fire-and-forget events, JetStream holds the events that must not be lost, and a KV bucket holds configuration. Each has a direct home on Felix.
Subjects become streams, JetStream streams become durable streams, queue groups and consumers become consumer groups, KV becomes the cache, and accounts become tenants. All of them share one log and one set of offsets, and high-fanout delivery keeps a slow subscriber from slowing anyone else.
NATS also covers request-reply, headers, an object store, MQTT and edge topologies that Felix does not, or only partly. The limits say what to do about each, and for some workloads the right answer is to keep NATS.
Try it
Section titled “Try it”The Quickstart starts a broker and a control
plane with a durable stream named orders. From the directory holding
compose.yaml, subscribe in one terminal and publish from another, the Felix
equivalent of nats sub and nats pub:
docker compose run --rm felixctl sub ordersdocker compose run --rm felixctl pub orders 'hello'The same stream also works as a JetStream-style queue. Create a group, publish a job and claim it:
docker compose run --rm felixctl group create orders workers-1 --from latestdocker compose run --rm felixctl pub orders 'job-1'docker compose run --rm felixctl group poll orders workers-1 --max 1 --wait-ms 5000Work queues finishes the walk-through with ack and
nack, and the sections below show how to create streams of the other kinds.
Why Felix fits
Section titled “Why Felix fits”Streams, consumer groups, the cache and counters run on the same log and offsets, so a queue and a key-value bucket do not need a second system that must agree with the stream. A Felix subscriber gets its own bounded queue, so a slow one reaches its own bound and does not hold the others back, and each event on a durable stream carries its offset, so a drop shows as a jump. Accounts become tenants, and tokens are scoped to a tenant and can be narrowed by action and resource.
How it maps
Section titled “How it maps”| NATS | Felix |
|---|---|
| Account | Tenant. Namespaces split a tenant further |
| Subject | A stream, addressed as tenant/namespace/stream, plus an optional routing key |
| Core pub/sub | An ephemeral stream (--durable false), in memory |
| Queue group on core NATS | A consumer group on a stream. It adds acknowledgements, so it is closer to JetStream than to core |
| JetStream stream | A durable stream, written to a CRC-checked log and replayable by offset |
| JetStream consumer | A subscription that resumes from an offset, or a consumer group |
AckWait, nak, term, in-progress |
Visibility timeout, group_nack with an optional delay, giving up on a record, extending a claim |
MaxDeliver |
FELIX_GROUP_MAX_ATTEMPTS, with a dead-letter list when it is reached |
| KV bucket | A cache, with versions, conditional writes, TTLs and watches |
| R3 stream | replication_factor 3 with Quorum consistency |
| WebSocket listener | felix-gateway |
Subjects become streams
Section titled “Subjects become streams”A NATS subject is a dotted string, and a subscriber can use * and > to match
many of them. Felix has no subject tree and no wildcard subscriptions. A stream
has a name, and a subscription names exactly one stream.
What to do instead depends on why you used the wildcard:
- Hierarchy for organisation, such as
orders.retail.placedandorders.wholesale.placed. Use a namespace for the coarse part (orders) and one stream per kind of event (retail-placed). A subscriber that wants both opens two subscriptions. - Fanning a family of subjects into one consumer, such as
orders.>. Put the events in one stream and carry the kind in the payload. The consumer reads one stream and filters in code. - Splitting by key, such as
orders.<customer-id>. Publish with a routing key on a stream with several shards. A key always lands on the same shard, so records for one customer stay in order. Ordering holds per key, not across the stream. See Pub/Sub Streaming. - Per-entity channels at very large scale, such as a subject per device or per user. A stream per entity is a heavier object than a subject. Keep these as routing keys on a shared stream, or keep NATS for that traffic.
Streams are created ahead of time, through the control plane or felixctl.
Nothing is created by publishing to a new name.
felixctl -n orders stream create placed --shards 4 --replication 3 --consistency quorumfelixctl -n orders stream create ticks --durable false --delivery at-most-onceCore pub/sub becomes an ephemeral stream
Section titled “Core pub/sub becomes an ephemeral stream”An ephemeral stream lives in memory and is lost on restart, the same bargain as core NATS. Each subscriber gets its own bounded queue with an overflow policy, so a slow subscriber drops records and never holds up the publisher or the other subscribers. NATS handles a slow subscriber differently: the server disconnects it as a slow consumer. A Felix subscriber stays connected and misses records, and on a durable stream the offsets on delivered events show exactly what was missed. See Isolation and backpressure.
Queue groups and JetStream consumers become consumer groups
Section titled “Queue groups and JetStream consumers become consumer groups”A core NATS queue group picks a random member for each message. If the member dies before it finishes, the message is gone. A Felix consumer group is closer to a JetStream pull consumer shared by several workers:
- A consumer polls, gets a record on a visibility timeout, and acknowledges it.
nackhands it back at once or after a delay. A claim that lapses is redelivered.- Each delivery carries
attempts. AfterFELIX_GROUP_MAX_ATTEMPTS(5 by default) the record is put on a dead-letter list and the group moves on. The record stays in the log. An operator can redrive it or discard it. - The group’s position and dead-letter list replicate with the shard, so a leader failover resumes where the group was.
NATS’s MaxDeliver defaults to unlimited, and when it is reached the server
stops delivering and raises an advisory. Felix keeps the dead-letter list for
you to inspect and redrive. A NATS consumer can also be told to give up on a
message early with a term, and Felix has the matching dead-letters add.
Two differences matter for the design:
- A group is bound to one shard. There is no coordinator that spreads shards across a group’s workers. On a stream with four shards you run at least one worker per shard. A NATS consumer, by contrast, hands any message to any worker.
- Order is not preserved across workers, in either system. Redeliveries reorder. JetStream deduplicates publishes by message id within a window and offers double-ack for exactly-once-style consumption. Felix deduplicates re-sent batches through idempotent producers, but delivers to consumers at least once, so a handler has to tolerate seeing a record again. See Semantics.
See Queues and Consumer Groups and the work queue scenario.
JetStream streams become durable streams
Section titled “JetStream streams become durable streams”A JetStream stream captures messages from the subjects it is bound to, with limits-based, interest-based or work-queue retention. A Felix durable stream is a log you publish to by name. Subscribers read from the start, from the tail or from a stored offset, and each event carries its offset. A consumer’s position in JetStream lives on the server; in Felix a plain subscription keeps no server-side position, so the client stores the offset it last saw. A consumer group is the option when you want the server to hold the position.
Retention is by size or age, and off until you set it. Replication is per
stream: replication_factor sets the number of copies, and the consistency
mode sets what an acknowledgement means.
Leaderacknowledges once the leader has the record. It is fast, and a record the leader acknowledged can be lost if the leader fails before its followers have it.Quorumacknowledges once a majority hold the record. This is the closest to an R3 JetStream stream, which commits by quorum.
Felix does not use per-shard Raft for the data path. It uses a leader per shard
with log shipping and, for Quorum, majority acknowledgement. The
Leader vs Quorum demo shows the difference under
a partition, and Semantics states the guarantees.
KV becomes the cache
Section titled “KV becomes the cache”A Felix cache holds keys under a tenant, namespace and cache name.
- A put can carry a TTL. Entries survive a restart only when the broker runs
with
FELIX_DURABLE_STORAGE_DIR; without it the cache is in memory. The log-backed cache is marked Partial in What Felix Is For because each shard keeps its whole key index in memory, so memory grows with the key count. cache_get_versionedreturns a version, andcache_put_ifandcache_delete_ifwrite only if the key is absent or still at that version. This is the same idea as KV’s revision-based create and update, and it is what leases and locks are built from. See Leases, locks and idempotency.- A watch on a key or a key prefix delivers each change with a resumable offset, and a retained watch delivers the current values first and then the changes. If a watcher falls behind it is told and can resume without a gap.
- Counters are a separate primitive with a durable increment.
Differences from a KV bucket: there is no per-key history, so you cannot read
earlier revisions of a key. A conditional write checks one key, and there are no
multi-key operations. A prefix watch reads one shard, and the cluster client
merges the shards for you. Keys in a cache are not subjects, so you cannot watch
config.>. See Distributed Cache.
Accounts become tenants
Section titled “Accounts become tenants”A NATS account is an isolated subject space with its own users. A Felix tenant
is the top of a hierarchy, tenant / namespace / stream or cache, and the scope
is part of every operation. Tokens are tenant-scoped, and RBAC grants name
objects such as stream:acme/orders/placed or cache:acme/prod/sessions/user-*,
with actions like stream.subscribe or group.consume. Grants can be narrowed to
one stream, one consumer group or a key prefix.
Cross-account import and export has no direct counterpart. Sharing between tenants is not a built-in feature. Tenants authenticate through the control plane, which can exchange an identity provider’s OIDC token for a Felix token. Per-tenant publish rate limits exist, set in each broker’s environment. See Security and Multi-tenant SaaS.
Browsers
Section titled “Browsers”NATS can serve the NATS protocol over WebSocket. Felix clients speak QUIC, which browsers cannot, so browsers go through felix-gateway: a WebSocket relay that signs the user in, holds a token narrowed to one scope, and forwards JSON messages. It is a separate process you run, and it is pre-1.0.
A migration path
Section titled “A migration path”Move the workloads that gain from the log first, and leave RPC on NATS.
- Pick a stream of live updates with many subscribers. Create an ephemeral stream, publish to it from the same code path that publishes to the NATS subject, and move subscribers over one at a time.
- Move a JetStream stream that holds events you replay. Create a durable
stream with
Quorum, dual-publish, and start the consumers from the offset that matches where they were. Switch consumers, then stop publishing to NATS. - Move work queues to consumer groups, one worker per shard, and check the handlers tolerate a repeat.
- Move configuration from KV into a cache and replace the watchers with cache watches. Use conditional writes where you used revisions.
- Leave request-reply, headers-dependent flows and edge or MQTT connectivity on NATS.
Compared with NATS
Section titled “Compared with NATS”Felix is built for very high fanout to many live subscribers with isolation between them, and for a durable log, work queues, caches and counters that share one log and one set of offsets, so you do not run a second system for state.
Request-reply, RPC, a very wide set of client libraries and edge topologies stay on NATS, which has years in production where Felix is pre-1.0. JetStream is a capable log, and per-key TTL and watches make KV a good fit for configuration. Latency and throughput figures, with their conditions, are on Benchmarks. Do not compare them with a NATS number from another setup.
Limits you can plan around
Section titled “Limits you can plan around”Request-reply. There is no built-in request pattern, reply subject or scatter-gather. Build one from two streams and a correlation id in the payload, accepting that latency and setup cost are not those of a NATS request. Keep NATS for RPC-style traffic.
Message headers. A record is a payload and, on durable streams, an offset and
append time. A broker can carry a key and named headers with each record for
the Rust client, and a cluster does once its fleet finalizes record_headers. Consumer groups and the other clients do not
yet (see What Felix Is For). Elsewhere, put
metadata in the payload.
Subject wildcards and subject mapping. Described above. Use namespaces, one stream per kind, or routing keys. There is no server-side rewriting or weighted routing of subjects.
Object store. Keep blobs elsewhere and publish a reference.
MQTT. A broker can accept MQTT connections on a listener with token authentication, and a device can publish at QoS 0 and 1 to a mounted stream. Subscribing over it is not built yet, so keep MQTT subscribers on NATS for now (MQTT). A Kafka listener serves producers and self-assigning consumers; see Coming from Kafka.
Leaf nodes, superclusters and mirrors. Felix brokers form one cluster, and there is no leaf topology for edge sites, no gateway between independent clusters and no stream mirroring between clusters. Pinning a stream’s replicas to a region is possible; see Data residency.
Follower reads. Every read goes to the shard’s leader.
Deployment. A Felix deployment has brokers and a control plane, backed by Postgres or its Raft backend, so it takes more to run than a single NATS binary.
