Upgrading to 0.7
Read Upgrades and compatibility for the rules these notes rely on: the five compatibility surfaces, the upgrade order, fleet features and rollback.
Breaking changes
Section titled “Breaking changes”- Lease-token templates take over braces in RBAC objects. A policy object
that holds
{or}is now a lease template. A grant on a literal cache key with braces, such ascache:t1/prod/c/{leases}, stops granting that key. It instead lets the holder of keyKin cachet1/prod/leasesget a lease token forc/K. Before upgrading, list your grants for objects with braces. Rename those keys so they hold none, and update the grants. The control plane also refuses to write a malformed template. - Rust client, renamed.
ClusterClientis nowClient, and the old single-connectionClientis nowConnection. Rename in this order, or the second rename undoes the first:ClienttoConnection, thenClusterClienttoClient. Code that connected to one address with the oldClientshould use the newClientwith that address as its only seed. - Rust crates that embed Felix.
StreamLog::base_offsetis nowphysical_base, besidereadable_base.Broker::allow_head_trims_whenreplacesset_replication_enabled.SubscribeCursorErrorgainstrim_headersand is no longerCopy. Thefelix-wiremessage types also gained fields (record_headers,client_flags_ext,server_flags_ext), so struct literals and exhaustive matches need them. The wire bytes for a client that does not use these are unchanged. - Storage format 7 and 8.
FORMAT_VERSIONwas 6 in 0.6.2, is 7 in preview.2 and is 8 in 0.7.0.- Version 7 holds the head-trim record. Version 8 holds a record’s key and headers.
- A broker moves a log onto a version only to hold such a record: v7 at the
first trim (in a cluster, once
head_trimis finalized), v8 at the first publish with headers. Until then the log stays readable by 0.6.2. - After that, 0.6.2 refuses to open the log. There is no rollback except restoring a backup.
FELIX_RECORD_HEADERS=falsekeeps v8 off. It does not affect v7.- A single broker accepts headers from the start. A cluster accepts them
once it finalizes
record_headers(seerecord_headers). - A single broker allows trims from the start and writes a
head-trimmedmarker. That data directory cannot join a cluster until the cluster has finalizedhead_trim(seehead_trim).
Wire protocol
Section titled “Wire protocol”Record headers are negotiated, so 0.6.2 clients and brokers keep working. A
frame flag, FLAG_EXTENDED, opens an extension word, and Auth and AuthOk
carry client_flags_ext and server_flags_ext to agree on
EXT_RECORD_HEADERS. A peer that does not offer it exchanges the same bytes as
before. Every client offers it. Header names starting with felix. are reserved.
A pipelining client can also offer FEATURE_PUBLISH_ANSWER_ON_COMPLETION.
A broker that grants it a publish window then answers plain acked
publishes as they complete and keeps request order only among idempotent
ones, so a shard stalled during a failover no longer holds back the answers
of every other shard on its stream. Older clients and brokers exchange the
same frames as before. The Rust client offers it; nothing to configure.
New settings
Section titled “New settings”None of these is required. Each default keeps the 0.6.2 behavior, except
FELIX_SHARD_STUCK_LEADER_MS and FELIX_UNIFIED_FENCE, which are on by
default.
| Setting | What it does |
|---|---|
FELIX_RECORD_HEADERS |
Single broker only. false refuses publishes that carry headers, which keeps logs at v7 or older. Turning it off later leaves stored headers readable. A cluster follows the record_headers fleet feature instead. |
FELIX_KAFKA_RECORD_HEADERS |
true stores a Kafka producer’s keys and headers with the record, which moves the logs to v8. Off by default, so the Kafka listener alone never does. |
FELIX_MQTT_LISTEN, FELIX_MQTT_WS_LISTEN |
Bind the MQTT listener, plain and over WebSockets. Unset leaves it off, and the broker is then unchanged. Devices can publish; subscribing arrives in 0.8.0. Related FELIX_MQTT_* limits are in the reference. |
FELIX_LEASE_TOKEN_KEY_FILE, FELIX_LEASE_TOKEN_MAX_TTL_SECONDS |
Key file and longest lifetime for lease tokens. Without the key file a broker neither mints nor accepts them. Every broker needs the same file. |
FELIX_INTERNAL_ELECTION |
Broker. true makes the broker offer replica elections. Every broker needs it before joint_fence and replica_election can be finalized, and keeps needing it after. See Turning on failover without the control plane. |
FELIX_SHARD_ELECTION_HOLD_MS, FELIX_SHARD_PENDING_SET_TIMEOUT_MS |
Control-plane timers for replica_election. Nothing uses them until that feature is finalized. |
FELIX_SHARD_STUCK_LEADER_MS |
Control plane, 70 s by default. A Quorum shard whose leader is on a live broker but shows no progress at its generation for this long is named again one generation up, on the same set. In 0.6.2 such a shard stayed stuck until something else changed its assignment. 0 turns it off. See “The control plane’s backstop” in docs/replication-design.md. |
FELIX_SHARD_SUSPECT_AFTER_MS |
Broker, 120 s by default, used only under replica_election. A follower whose copy holds a generation that never started, and that has heard nothing from the shard’s leader for this long, may stand for that one shard though the leader’s broker is up. 0 turns it off. |
FELIX_UNIFIED_FENCE |
Broker, on by default. A broker promoted to lead a shard counts its fence by what each replica attests about its copy: whether it kept its ballots and whether it holds every record it acknowledged. That lets a failover open past a copy doubted after a crash, which the older rules could leave waiting, as long as the trusted copies still cover every majority. Answers from 0.6.2 brokers count as they always did, so a rolling upgrade is safe in either order. A memory-only broker counts as before. A broker started without the control plane (FELIX_START_WITHOUT_CONTROLPLANE) does not vote or count toward a promotion until it reaches the control plane, so a failover that needs its vote waits for the control plane. 0 turns it off and restores the older pin-based rule, under which that vote counts. See “Fencing a promotion” in docs/replication-design.md. |
In a cluster, headers are refused until the fleet finalizes record_headers,
whatever FELIX_RECORD_HEADERS says.
Upgrade order
Section titled “Upgrade order”Before you start, list your RBAC grants for objects with braces (see the first breaking change).
The ordinary order applies: control plane, then brokers one at a time, then clients. Back up the data directory first if you may use headers or head trim, because both write formats 0.6.2 cannot read. The first MQTT publish to a mounted stream also stores record headers, so it counts as using headers.
The head_trim and record_headers fleet features are new. They stay off after
the brokers are upgraded, and nothing needs them unless you want to trim a
stream or store record keys and headers in a cluster. To turn one on, once
every broker runs the new build:
felixctl features finalize head_trim --dry-runfelixctl features finalize head_trim --yesUse record_headers in place of head_trim for headers. The two do not
depend on each other or on the fleet features from 0.6.
The dry run names any live or draining broker that lacks the feature. A broker
that is down is not counted and is refused later if it returns on the old
build, so replace or retire it first. After the finalize, watch
felix_broker_fleet_feature_enabled{feature="head_trim"} (or record_headers) reach 1 on each
broker, or read GET /v1/fleet/features.
Finalizing is one-way. A cluster created on the new build finalizes it at the
first broker registration. See head_trim
and record_headers
for what each writes and what an older broker does with it, and
Rollback for the limits above.
A rollback can also trip on a shard’s replica set file. A 0.7 broker that has
not yet read whether joint_fence is finalized, for example right after a
restart, keeps older replica sets in that file, because dropping them is unsafe
if the fleet did finalize it. An older build refuses such a file and fails to
open that shard’s log. Start the broker on 0.7 again, let it learn that
joint_fence is off and write the shard’s next replica set, which drops them,
then roll it back.
Control plane changes
Section titled “Control plane changes”A leader promoted by a failover now reports, for each follower it has not reached yet, the storage identity that its own promotion named the follower by. The next failover names the follower by that identity, so a second failover soon after the first no longer leaves the shard closed (#1574).
- With the Postgres backend, the control plane runs migration
0037when it starts. Nothing else is needed. - With the Raft backend, the new field is kept only once every member is at metadata level 20. Until the last member is upgraded, a follower named this way does not have its fence answer counted, as in 0.6.2, and a shard that needs that answer waits. Upgrade every control-plane member before the brokers, and see Rollback for the limit on going back once level 20 entries are written.
The control plane can now keep a cut-over’s new replica set pending until
the leader reports it on both the old and the new set, and adopt a leader
its replicas elected, both of which replica_election needs (#1009).
Nothing uses them until that feature is finalized.
- With the Postgres backend, the control plane runs migration
0038when it starts. Finish upgrading every control-plane instance before finalizingreplica_election: the control plane cannot see another instance’s version, and one that predates this drops a pending set when it writes the shard. - With the Raft backend, both need every member at metadata level 21.
Finalizing
replica_electionneeds level 26 (below).
The control plane also changed after preview.2:
- Every placement write names each member’s registered storage id
(
member_storage_ids): Postgres migration0039, Raft metadata level 22. - A copy whose storage was replaced is a learner until it is re-admitted, and
a leader reports where its generation started (
holds_start): migration0040, level 23. Finalizingjoint_fenceneeds level 25 (below), andgeneration_start,stream_incarnationsandcache_incarnationsfinalized first. A broker reportsjoint_fenceonly withFELIX_INTERNAL_ELECTION=true, so it is part of turning on failover without the control plane (below). - A stream can carry an MQTT mount: migration
0041, level 24. - A leader that could not serve a generation because it had promised it to
another node reports the refusal, and placement names it again at a higher
generation (#1730): migration
0042, level 25. - After
joint_fenceis finalized, a leader restarted at a generation that started before the finalize cannot learn the replica set that generation replaced. It reports the generation refused, and placement names it again one generation up. The report uses therefusedfield above with no node named, so it needs migration0042and level 25, and finalizingjoint_fenceunder Raft waits for level 25. Rolling restarts after the finalize need no extra step. A new shard a broker first sees in a snapshot opens on the snapshot’sfirst_assignmentmark, which a 0.7 control plane sends. With the usual upgrade order (control plane first) every broker gets it. - An elected leader is adopted on the replica set it was elected on: level
26, with no Postgres migration. Finalizing
replica_electionneeds that level. Level 27 does the same when the elected leader is the one the control plane already names; below it, the same report moves the shard past the election at once, and the next report is only a fallback. - A leader that answered another candidate’s fence, when the control plane
never adopted that candidate, reports the candidate’s ballot with its refusal, and
placement names it again one past that ballot (#1951): migration
0043, level 28. Below it the leader is named one generation up, as before, each rename waits the same 10 s, and it can take several renames to get past the ballot. - A placement write that lands more than one generation up records the
generation it replaced (
previous_generation, #2024), so a leader renamed past a withdrawn candidate’s ballot does not leave its joint fence pending for minutes: migration0045, level 29. Below it the broker learns no set for the skipped generation, as before. - Postgres refuses a mount on a non-durable stream in the schema itself:
migration
0044, with no Raft level. - When the leader of a set that re-admitted a replaced copy reports it caught
up, the control plane records that on the assignment (
readmitted): migration0046, level 30. Below level 30 nothing is recorded.
In short, Postgres runs migrations 0037 to 0046 on start (0039 to
0046 are new since preview.2), and under Raft these features wait for metadata
levels 20 to 30. Upgrade every control-plane member before the brokers.
Turning on failover without the control plane
Section titled “Turning on failover without the control plane”In 0.7.0 a shard’s replicas can elect a new leader themselves when the old one dies while the control plane is down. It stays off after the upgrade. What it does and its limits are in Failover without the control plane. To turn it on, after the upgrade above:
-
Finish upgrading every control-plane member. Under Raft the finalizes below check it (level 25 for
joint_fence, 26 forreplica_election). Under Postgres nothing checks it, so make sure no instance runs an older build. -
Set
FELIX_INTERNAL_ELECTION=trueon every broker and restart them one at a time. Nothing changes yet. -
Finalize, if they are not already,
generation_start,majority_ack,lease_free_reads, andfenced_cachesif you want caches elected. Each has its own cautions in Upgrades and compatibility. -
Finalize
stream_incarnationsandcache_incarnations.joint_fenceis refused until both are finalized. -
Finalize
joint_fence, dry run first:Terminal window felixctl features finalize joint_fence --dry-runfelixctl features finalize joint_fence --yesFrom here a broker without
FELIX_INTERNAL_ELECTION=trueis refused at registration, so keep it set on every broker, new ones included. -
Finalize
replica_electionthe same way. -
Check that
felix_broker_fleet_feature_enabled{feature="replica_election"}is 1 on every broker. A broker withFELIX_LEADER_SUSPECT_AFTER_MS=0still never stands, so leave that setting at its default or above zero.
A cluster created on 0.7.0 has already finalized steps 3 and 4 at its first broker registration, so it needs steps 2, 5, 6 and 7 only.
Clients
Section titled “Clients”The Rust client has the rename above. The other clients are additive: every
binding takes four reconnect options (an attempt count, a deadline for one
retried call, and the backoff and max backoff between attempts), unset values
keep the Rust defaults, and trim_before is new. Cursor errors in every binding also carry
the reason, the nearest available offsets and the trim’s headers. The
C struct FelixConnectOptions gained reconnect_attempts,
reconnect_deadline_ms, reconnect_backoff_ms and reconnect_max_backoff_ms
after token_release; a program built against the older felix.h keeps the
defaults, since zero means default.
Cache and consumer-group calls on the Rust Client now follow
ReconnectPolicy, as publishes already did. The limits bound the whole call,
and a write is sent again only after a refusal that says nothing was applied.
A Connection opens its pooled connections as its traffic needs them.
A subscriber with default settings now holds 5 broker connections instead of 20,
so about 100 fit under the default FELIX_MAX_CONNECTIONS_PER_IP=512 instead
of about 25. No action is needed; the pool sizes are ceilings.
What else is new
Section titled “What else is new”A client can trim a stream’s oldest records at the tail it expects
(trim_before, felixctl stream trim); a cursor error now says where to resume
and carries the trim’s headers; a single broker accepts and delivers record
headers, on a cluster once it finalizes record_headers; every client has
lease tokens; and a broker can serve MQTT, where a device can publish at QoS 0
and 1. See
what Felix is for for the status of each.
Packaging
Section titled “Packaging”Each GitHub release now carries prebuilt libfelix C libraries, shared and
static, with felix.h and a LINKING.txt, for Linux, macOS and Windows on x64
and arm64. See the C client.
