Skip to content

Felix 0.5.0

Released 2026-09-19. The full history is in CHANGELOG.md.

A third client, and the release where the data path stopped paying for compatibility.

A Node.js / TypeScript client, a napi-rs addon over the same Rust client the Python binding wraps, passing every required scenario in the conformance catalogue with CI and the release pipeline both gated on it.

The data path is binary. A routing key now rides in the binary publish frame, so a keyed publish no longer falls back to JSON (which measured 645.8 MB/s against 917 for the same workload), and the JSON publish surface is deprecated for removal in 0.6.0. A forwarded publish now says so on its ack and names the shard’s owner, which makes the cost of publishing to the wrong broker visible for the first time.

Two defaults changed on measurement. MTU discovery is bounded below Linux’s UDP GSO ceiling, where the old bound could stall delivery outright on a jumbo-frame network; and a broker joining a cluster refuses to start with a credential nothing can renew, instead of running fine until the token expires and the lease lapses under it.

Upgrade notes. A deployment passing an expiring FELIX_NODE_TOKEN by value must now set FELIX_NODE_REFRESH_TOKEN_FILE or FELIX_NODE_TOKEN_FILE. Two docs URLs moved, listed below.

Wire protocol: two new frame flags, FLAG_BINARY_PUBLISH_KEYED (0x0040) and FLAG_BINARY_PUBLISH_ACK_OWNER (0x0080). VERSION remains 1, INTERNAL_VERSION remains 1, and no feature bit is added. Both are negotiated on the handshake, so a client and broker that disagree about either exchange the same bytes they did before.

  • A forwarded publish says so on its ack, and names the shard’s owner (#536). A publish for a shard the receiving broker does not own is forwarded to the owner and acknowledged once the owner has written it. That is correct, but it was invisible, so a client kept publishing to the same entry broker forever while every record was decrypted, re-encrypted and decrypted again on the way. A perf session put the cost at roughly half the throughput per core: ~250 MB/s per busy vCPU direct against ~140 forwarded.

    FLAG_BINARY_PUBLISH_ACK_OWNER (0x0080) is a modifier on the publish ack, the shape FLAG_BINARY_PUBLISH_KEYED established. The bit’s presence is the signal that forwarding happened; the payload carries the owner’s node_id, the address it serves clients on, and the ownership generation.

    It is only a hint. The publish already succeeded, so a client that ignores it is exactly as correct as before, and just as slow. That makes it safe to add, since nothing depends on a client acting on it. It is only set for a client that advertised the bit, and an ack with no owner is byte-identical to one from before the bit existed.

    Clients count it as felix_client_publish_forwarded_total, labelled by owner, and felix_client::publishes_forwarded() exposes the same number without the telemetry feature, so a build compiled without measurement can still tell whether it is paying for forwarding.

    Routing on the hint is not built yet. Caching shard → owner and sending the next batch straight there needs a connection per owner and a client-side shard_for(key), neither of which exists. That is a separate change, and this one makes its benefit measurable.

  • A routing key rides in the binary publish frame (#549). A keyed publish used to force the JSON encoding, because the binary layouts were fixed and had nowhere to put a key. That fallback measured 645.8 MB/s against 917 MB/s unkeyed on the same rig, with user CPU up from 20% to 28%. Routing a record cost roughly 30% of throughput, so sharding carried a real cost.

    0x0040 is a modifier on 0x0001, the shape FLAG_BINARY_PUBLISH_ACKED already established: the body is prefixed with a u16 key length and the key bytes. With both bits set the correlation prefix still comes first, so a broker reads request_id at offset 0 whether or not a key follows.

    A broker that predates the bit would read key_len as tenant_len, so the client sends the keyed binary frame only to a broker that advertised it and uses JSON otherwise. The fallback costs throughput and never correctness.

  • A Node.js / TypeScript client (crates/felix-typescript), a napi-rs addon over felix-client instead of a reimplementation of the protocol. The reasoning matches the Python binding, and so does the surface: publish (keyed, or at-least-once), subscribe, sharded subscribe with per-shard resume, cache get/put/delete, counters, cache watches and consumer groups. Every call returns a Promise, and every handle is disposable. Errors arrive as typed classes (ConnectionError, AuthError, NotFoundError, CursorError, InvalidArgumentError) that mirror the Python binding’s exceptions, so an application branches on the error class instead of the message text.

    It passes the client conformance suite, and CI and the release pipeline are both gated on that: the suite drives a real three-node fixture cluster, names the catalogue scenario each test demonstrates, and felix-conformance verify checks the results against the catalogue, so a green run that quietly skipped a required scenario still fails. Not published to npm: the addon is built per platform and attached to the GitHub release, with the npm step behind PUBLISH_NPM for the same reason PyPI is behind PUBLISH_PYPI.

  • task release:check, and the release refuses a tag that disagrees with the tree. A tag and a version that disagree ship artifacts labelled with neither, and nothing else notices: the archive is named from the tag, each crate is built from its own manifest, and both succeed. The wheel, the npm package and the Helm chart each carry a version of their own, so bumping the workspace was never enough. CI asserts the fields agree with each other, which is the half a pull request can be wrong about; the release job additionally asserts they match the tag, before anything is published.

  • task lock:refresh, and CI fails when a build updates a lockfile the commit did not include. Six crates declare their own [workspace] (the four under demos/, plus the Python and TypeScript bindings), so the repository workspace never touches their Cargo.lock. All four demo locks had sat at 0.4.0-preview through two releases and had never heard of io-uring, because CI regenerated them on every run and nothing looked at what changed.

  • felix-loadgen --keys <n> spreads the ingest scenario’s batches over n routing keys. The scenario published unkeyed, and an unkeyed record resolves to shard 0, so every “multi-shard” measurement taken with it was really a single-shard one. A 12-shard and a 1-shard stream measured identically (920 vs 923 MB/s) because both exercised the same log. Default 0 keeps the old behaviour, so existing runs stay comparable.

  • A crates.io publish job (#574), behind PUBLISH_CRATES like the PyPI and npm ones. The Rust client is what the other two bindings wrap, and it was the one registry with no job at all.

    The order is the mechanism: crates.io resolves each dependency against the registry as the crate uploads, so a crate published ahead of something it depends on fails halfway, with the earlier crates already permanent. The job walks a topological sort of the workspace’s internal edges and skips any version already on the registry, so a re-run after a partial failure completes rather than erroring on what already landed. task publish:check asserts that list is every publishable crate and is really topological, because nothing about adding a crate to the workspace forces anyone to revisit a workflow file.

    felix-broker and felix-storage are published too, AGPL-3.0 and all: felix-client’s in-process feature declares them optional, and crates.io resolves an optional dependency like any other: cargo publish --dry-run refuses felix-client without them. The licence split is unchanged; a default felix-client build still pulls no AGPL-3.0 code.

  • The npm publish can run (#575). napi ships one package per platform (the main package declares them as optional dependencies and npm installs the matching one), and none of that existed: no npm/ directory, no optionalDependencies, and a loader that only looked for a file beside itself. napi prepublish had nothing to publish.

    Five platform packages now cover exactly the five targets the release builds, and the loader resolves the installed one, detecting musl rather than assuming glibc. A checkout is unaffected: a local build still wins, which is what the conformance suite runs against. check_npm_packages.py ties the build matrix, the declared triples and the packages on disk together, because drift between them publishes cleanly and fails at npm install on someone else’s machine, against a version that is permanent. The napi CLI is pinned to @2, matching the napi crate, whose v3 renamed the config keys this package uses.

    Registry metadata went with it: neither binding shipped a LICENSE despite both declaring Apache-2.0, the Python package had no py.typed so type checkers ignored the stubs beside it, and every crate inherited the workspace readme, so the repository README, roadmap and all, rendered on eight library pages. Each publishable crate has its own now.

  • The Quickstart’s first command did not work, and neither did the one it became. cargo run --release -p broker asked which of nine binaries to run, because eight of them are demos and nothing set default-run. Corrected to --bin felix-broker, it starts and then exits: FELIX_CONTROLPLANE_URL must be set for auth. The page claimed you would see QUIC listening on 0.0.0.0:5000 and that “the broker is now ready to accept connections”, and printed that command three times.

    default-run = "felix-broker" fixes the first half. The second half is intended. A broker validates client tokens against keys it fetches from the control plane and registers itself there for shard placement, so there is no unauthenticated mode. The docs had never said so.

    The Quickstart now opens with felix-cluster up, which starts a control plane, mints the credentials and brings up three brokers, then publishes through a non-owner and receives from the owner. That command existed the whole time and no getting-started page mentioned it. Every command and every block of output on the page was run to produce it.

    Also corrected: the landing page and Installation both told you to run the broker alone; the container instructions did the same with docker run; and Troubleshooting told you to set FELIX_METRICS_BIND, which nothing reads – the broker warns about that exact name, and the variable is FELIX_BROKER_METRICS_BIND.

  • Four timing-sensitive tests stopped reporting a slow machine as a broken invariant. Each was a wall-clock assertion on a shared runner, each failed on a branch that could not have caused it, and between them they cost several investigations.

    concurrent_durable_appends_share_a_flush asserted a wall-clock speedup from group commit. A ratio cannot tell “the flushes coalesced” from “this machine could not put sixteen appends in flight for them to”, which is how it read 0.70x on a two-core runner with nothing wrong. DiskLog::flushes() now exposes the flush count (one relaxed increment against an fsync), and the test asserts the thing itself: 64 appends produce 64 flushes serially and 5 concurrently. Counting does not measure the machine.

    an_inline_rollover_does_not_park_every_worker allowed a fifth of one rollover for scheduling noise, described as “far above the scheduling noise even on a loaded box”. CI falsified that twice, measuring 137 ms. The bug it catches parks every worker for the whole rollover, so half keeps a 2x margin on both sides instead of sitting next to the noise floor.

    FELIX_TEST_TIMEOUT_SCALE multiplies the setup deadlines in the cluster harness and the Raft chaos suite. Unset means 1, so a developer’s run is unchanged and still fails fast on a genuine hang; CI sets 3 and the coverage job 5. Raising the constants instead would have bought the same green while never noticing a hang on the machines fast enough to.

    Deliberately untouched: losing_quorum_fails_writes_loudly_not_silently waits on a leader noticing it has lost quorum, and that wait is the subject. Widening it would only make the test slower. It is serialised instead, which is what its module already says.

  • A cancelled idempotent publish no longer loses records silently. IdempotentProducer::publish_batch advanced its sequence only after the broker answered. Dropping the future in between (a timeout, a losing select! branch) left the cursor pointing at a sequence the batch may already have been appended under. The next batch then went out under that spent number, and the broker’s contract is to answer a remembered sequence from memory without appending it: the caller was told Ok and its records were discarded, with nothing reported anywhere.

    The producer now notices the cancellation and refuses to publish again, naming the reason and telling the caller to take a fresh producer id. The batch in doubt is the only one whose fate is unknown, and that is recoverable; silently dropping the ones after it was not. Covered by a cluster test that cancels a real publish and fails without the fix.

    Only reachable from Rust: neither the Python nor the TypeScript binding wraps idempotent producers.

  • The Node package is felix-client, unscoped. @felix on npm was already taken (there is an unscoped felix package, and the scope with it), so the six packages the release publishes are felix-client and one per platform, felix-client-darwin-arm64 and friends. All six names were confirmed free.

    Unscoped rather than hunting for another scope: it is the same name the crate has on crates.io and the wheel has on PyPI, so one string covers every install instruction, and an unscoped name is first-come rather than colliding with a namespace someone else may hold. publishConfig.access went with the scope, since only a scoped package defaults to restricted.

    Nothing had been published, so this costs nothing but the rename.

  • The loopback MTU guarantee’s buffer gate no longer moves with the MTU knobs. It checks whether the host was tuned, as a proxy, because Linux clamps SO_RCVBUF to a stock ~208 KB and an untuned host cannot absorb the bursts the pin exists to survive. It was measured against the size about to be pinned, so when the discovery bound’s default dropped to 4096 below the requirement fell from ~1 MiB of socket buffer to 256 KiB and every stock Linux host silently started pinning the loopback MTU. A routed-path hazard fix had no business changing who gets a loopback pin. The gate now reads the jumbo payload and nothing configurable.

    One behaviour goes with it: setting FELIX_MTU_UPPER_BOUND low on an untuned host no longer buys the guarantee. Asking for a smaller pin is not evidence of headroom.

  • MTU discovery is bounded below Linux’s UDP GSO ceiling by default (4096, was 16384; macOS keeps 16384). Linux packs a whole sendmsg batch into one IP datagram, so MTU × segments must stay under 65,535, and quinn batches up to 10, which puts the real ceiling at 6,553 bytes. Above it the kernel rejects every batch with EMSGSIZE, which quinn does not recognise as a GSO failure (it falls back only on EIO/EINVAL), so the transmit is dropped after quinn has counted it as sent and delivery stalls permanently rather than degrading.

    Loopback was capped at 4096 when this was diagnosed; a routed path was not. That made the old default harmless on a 1500-byte network and fatal on a jumbo-frame one, and jumbo frames are what you buy for throughput. Every perf session set FELIX_MTU_UPPER_BOUND=4096 by hand; that is now the default.

    4096 rather than the exact 6553: quinn’s MAX_TRANSMIT_SEGMENTS is private to it, so the ceiling cannot be derived through its API, and 6,553 breaks the moment that number rises. Two tests pin the invariant, and they check the non-macOS value from either host, so the check cannot quietly pass just because of the machine doing the editing.

  • A broker says so when the OS clamps its UDP socket buffers. Linux accepts an oversized SO_RCVBUF/SO_SNDBUF and silently clamps it to net.core.rmem_max/wmem_max, which ship at around 208 KB against the 8 MiB Felix asks for. Bursts then overflow the socket and surface as QUIC retransmits, so the broker runs at a fraction of the host’s capacity and looks healthy doing it. Nothing in Felix can raise a host limit, so the warning names the sysctls instead.

    Deliberately unchanged: publish_conn_pool (4) and publish_sharding (HashStream). Both were swept in an Azure session and both looked promising, but those runs were void. The generator was ignoring client environment config (#553), so the overrides never applied and the runs measured the defaults. Re-tested on a fixed generator, round-robin landed at 912.6 MB/s, inside the 842–926 band every valid configuration occupied, and raising the broker’s publish worker pool 4 → 16 measured slightly worse. The reasoning is now recorded next to each default so it is not re-litigated from the retracted numbers.

  • A Clients section in the docs, with a page per client. Rust had its own page and everything else shared one, where TypeScript got two paragraphs. Two URLs moved: /api/client-sdk/ is now /clients/rust/, and /api/clients/ is now /clients/overview/. The Rust page also lost an error handling example that could never have compiled. It matched on felix_common::Error variants that do not exist, in a crate that is not one of felix-client’s dependencies.

  • The container images are published. ghcr.io/gabloe/felix-broker and ghcr.io/gabloe/felix-controlplane went out with 0.4.1 and pull without credentials. Each release tags the full version, the minor series, and latest on non-prereleases.

    The docs said they were not published and told you to build your own; that was true when written and is not now. The Docker Compose, Kubernetes and installation pages now pull instead, and keep the build commands for running something unreleased.

    Images are signed with cosign, keyless, over the digest and never the tag, because a tag can be moved and a signature over one would follow it. The Kubernetes page carries the cosign verify invocation. The chart’s image tag defaults to its appVersion, so a default install resolves to a published image with nothing to configure.

  • The data path is binary; JSON is compatibility only (#550). Now that #549 put the routing key in the binary publish frame, the JSON encoding has no remaining reason to carry data-plane traffic. It measured 645.8 MB/s against 917 for the same keyed workload on the same rig, with user CPU up from 20% to 28%, and it buys nothing the binary frames do not now cover.

    Publisher::publish_json and Publisher::publish_batch_json are deprecated and will be removed in 0.6.0. Nothing routes through them any more: publish_batch’s fallback now calls the private keyed form directly, so it keeps working once the public surface goes.

    The broker still accepts JSON publishes and will keep accepting them. ORIGINAL_V1_FLAGS is frozen, so a client older than FLAG_BINARY_PUBLISH_ACKED or FLAG_BINARY_PUBLISH_KEYED is entitled to send them forever. What changed is that no current Felix client emits one except as a fallback it chooses itself, against a broker that did not advertise the frame it wanted. felix_broker_json_publishes_total{frame="publish"|"publish_batch"} counts what still arrives that way, which is the evidence a deployment would need before the arm could ever be dropped.

    publish_idempotent is unaffected. It is JSON because no binary layout carries a producer id and sequence yet, not for compatibility.

  • A broker refuses to start when its credential will expire with nothing able to renew it. The broker’s control-plane calls all read one token, and the heartbeat is among them, and the heartbeat is the lease renewal. A broker with an expired credential stops serving the shards it leads once the lease lapses. That is correct behaviour, and still an outage nobody chose.

    Joining a cluster (FELIX_NODE_ID set) with a token that carries an exp and neither FELIX_NODE_REFRESH_TOKEN_FILE nor FELIX_NODE_TOKEN_FILE now fails startup, naming both ways out. Previously it logged at info and ran fine until the token died. Upgrade note: a deployment passing an expiring FELIX_NODE_TOKEN by value must set one of the two files.

  • FELIX_NODE_TOKEN_FILE is re-read, so a credential rotated by something outside the broker (a Vault agent, SPIRE, a sidecar) takes effect without a restart. It was read once at startup and never again, while the refresh token file was deliberately re-read every time; the asymmetry was the surprising half. A replacement that has already expired is declined rather than adopted, because swapping a working credential for a dead one turns someone else’s rotation bug into this broker’s outage.

  • Device flushes can go through io_uring on Linux (#548), behind FELIX_STORAGE_IO_URING=1, default off. IORING_OP_FSYNC removes the spawn_blocking hand-off rather than shrinking it, and unlike running the sync inline it keeps the await as a yield point, so background rollover and retention still get scheduled. A perf session measured 956.7 MB/s against 917.2 with it on. Every run was better, with no overlap between the distributions.

    The blocking pool stays as the fallback: a kernel too old for the opcode, or a container that forbids the syscall, falls back rather than failing. Durability must not depend on an optimisation being available.

  • Docs stopped calling shipped capabilities unbuilt. The README, the docs site landing page, the overview, why-felix, the FAQ, the components and project-structure pages, docs/architecture.md, docs/auth.md and docs/internal-protocol.md all still listed broker-to-broker mTLS as future work; it shipped in M8 (#125, #126). Alongside it: docs/architecture.md said no metadata rides the control plane’s Raft group, which M13 closed; how-felix-works.md named Raft and mTLS as unbuilt; the threat model called peer connection exhaustion unmitigated after #504 capped it, and credited #422 with closing forwarded-publish replay, which it does not: ForwardPublish carries no producer identity, so that case stands open. A wrong status marker on a security control misleads readers about what protects them.