Failover without the control plane
Normally the control plane decides every failover. When a shard’s leader dies, the control plane picks a follower and names it the new leader. If the control plane is down at that moment, nobody can name one, and the shard stops taking writes until the control plane is back.
With the replica_election fleet feature finalized, a shard’s replicas elect a
new leader among themselves instead. The new leader serves writes while the
control plane is still down, and the control plane takes it as the shard’s
leader when it returns. No acknowledged record is lost on the way.
This ships in 0.7.0. It is off until you turn it on: see Turning it on.
Which shards it covers
Section titled “Which shards it covers”Elections cover the shards whose control plane can already replace a leader as soon as a majority of its followers report they cannot reach it, without waiting for the leader’s lease to run out:
- durable streams that acknowledge with
Quorum, oncemajority_ackandlease_free_readsare finalized; Quorumcaches, oncefenced_cachesis finalized as well.
Leader streams and in-memory streams still wait for the old leader’s lease
to run out and the control plane to name a new one, as before. So do consumer
groups: a group on a shard with an elected leader waits until the
control plane is back (issue #1197).
A shard can elect only while a majority of its copies are running. By default a copy that restarted since the control plane went down does not count. See Limits.
What happens when a leader dies
Section titled “What happens when a leader dies”A few terms first.
- The replica set is the list of brokers that hold a copy of the shard. Each copy keeps the set on disk.
- A generation numbers each leadership of a shard. Every new leader takes a higher one than the last.
- A ballot is a broker’s promise, kept on disk, that it will follow only the leader of a given generation. A broker never promises one generation to two leaders.
- A fence is how a new leader takes over: it asks the replica set for their ballots at its generation, and it serves only once a majority has given one. A replica that answered stops taking writes from the old leader.
The election and the fence follow the prepare and promise steps of Paxos. The generation is the ballot number. A fence is the prepare round, and it needs a majority. A replica’s ballot on disk is its promise never to follow an older leader. The new leader then catches up from the copy furthest ahead among the answers, ranked by the generation of its last record and then its length, the way Paxos adopts the value accepted at the highest ballot. Felix does not run Multi-Paxos for the log, though. After the fence, the leader ships records and counts majority acks. And normally the control plane names leaders, not an election. See Leslie Lamport, “The Part-Time Parliament” (ACM TOCS, 1998) and “Paxos Made Simple” (2001).
Say a shard has three copies on brokers A, B and C, with A leading, and the control plane is down. A crashes.
- B and C notice. Each follower pings its leader. A leader that has not
answered for
FELIX_LEADER_SUSPECT_AFTER_MS(5 s by default) is suspected. - Pre-vote. B asks the other members of its replica set whether they would back it. C says yes only if it cannot hear A either, it does not lead the shard itself, B’s log is at least as far ahead as its own, and B’s replica set is no older than the one C knows. A follower with a bad network link therefore cannot unseat a leader the others still hear. Nothing is written to disk for a pre-vote. B needs a majority, counting itself.
- Waiting its place. The replicas are ordered by how far ahead their logs are. B waits 300 ms for each replica ahead of it, so the one with the most data usually stands alone. A replica that waited asks again and stops if a member that said yes now says no. One ahead of it that has already stood now says no, so the two do not stand against each other.
- Ballot. B writes its ballot for the next generation to disk.
- Fence. B fences the replica set as a candidate. C gives its ballot and records B as the new leader. With B and C, B has a majority of three. A gets the fence later, when it comes back.
- Serving. B takes the most complete log among the answers, writes a start record for its generation, and opens the shard for writes. C now takes B’s batches and sends clients that reach it to B.
- The control plane returns. B’s next report says it was elected. The control plane names B the leader at that generation, and from then on everything runs as before.
sequenceDiagram
participant A as Broker A (leader)
participant B as Broker B
participant C as Broker C
participant CP as Control plane
Note over CP: down
Note over A: crashes
Note over B,C: no answer from A for 5 s
B->>C: pre-vote for the next generation
C-->>B: yes (C cannot hear A either)
Note over B: waits its place, then writes its ballot
B->>C: fence as a candidate
C-->>B: ballot given
Note over B: majority (B, C): opens the shard
B->>C: batches at the new generation
Note over CP: back
B->>CP: report: elected at this generation
CP-->>B: B is the shard's leader
Note over A: restarts
B->>A: fence again
Note over A: follows B
If two followers stand at once, the ballots decide: a replica gives its ballot for a generation to one of them only, so at most one gets a majority. The other backs off for 200 to 800 ms (doubling to at most 3 s) and tries again. A standing that has not opened within 3 s of its route being published, and has no fence or catch-up still running, is withdrawn, and the next try is at a higher generation.
The time to a new leader is the suspicion window, plus one pre-vote round, plus 300 ms per replica ahead of the winner and a second pre-vote round if it waited, plus one fence round.
While the control plane is up
Section titled “While the control plane is up”Replicas elect whether or not the control plane is reachable. So that the two
do not race, the control plane waits FELIX_SHARD_ELECTION_HOLD_MS (10 s) after
followers first suspect a leader before it promotes on their word. Promotion
after a dead leader’s node expires is not held; with the default
FELIX_NODE_EXPIRY_TIMEOUT_MS that comes later than the hold anyway, so if you
lower it, keep it above the hold. If both end up
naming a leader at the same generation, ballots let only one of them open
there. If the elected one opened, the control plane moves the shard to it on
its next report.
Most elections start because followers cannot reach a leader’s broker. A shard
can also get stuck on a broker that is up: a fence that never finishes, or a
copy that stopped accepting the leader. When the control plane sees no
progress from such a leader for FELIX_SHARD_STUCK_LEADER_MS (70 s), it
names the same leader again one generation up. The leader fences its
followers again before it serves, so no acknowledged write is lost. An idle
shard counts as progressing, a shard that never opened gets twice as long,
and each control-plane instance renames one shard at most four times an
hour. When many leaders look stuck at once the fault may not be theirs, so
they are renamed one a pass. With the control plane down, a follower stands for
such a shard instead, after a longer bound (see
Turning it on).
What you see
Section titled “What you see”On the brokers, at info:
| Log message | Meaning |
|---|---|
standing for election: the leader is unreachable or stuck and a majority agreed |
This broker won its pre-vote and wrote its ballot. The line has stream, shard, generation, the old leader, and broker_suspected (false when it stood on one stuck shard). |
a majority took the fence; opening the shard for writes |
The shard opened at generation. Same line as for any failover. |
the standing did not open in time; withdrawing it |
The shard had not opened 3 s after the route naming this broker was published (or FELIX_ELECTION_DEFER_FOR_MS after the ballot, if it never was), and no fence or catch-up was still running. The broker tries again. |
not standing for election |
The broker suspects the leader but will not stand. why says which rule held it back, for example the pre-vote found no majority with the answers it got. Logged once per reason. |
On the control plane, an elected leader reported; the control plane took its generation, with adopted=true when the elected broker became the shard’s
leader.
A stuck leader named again logs a warning, this shard's leader shows no progress at its generation on a live broker; naming it again one generation up, with reason (never_opened, stopped_reporting or not_committing),
and counts in felix_shard_move_steps_total{step="stuck"}.
Metrics:
felix_broker_promotions_opened_total{path="fenced"}goes up on the broker that won, as for any failover.felix_broker_fleet_feature_enabled{feature="replica_election"}is 1 on a broker that has the feature on. It stands for election only ifFELIX_LEADER_SUSPECT_AFTER_MSis not0as well.
Limits
Section titled “Limits”By default, a broker that restarts while the control plane is down cannot vote. It waits for the control plane before it opens its data, so it neither votes nor stands. It does not answer pings either, so if it led a shard, its followers treat it as down and elect one of themselves. A shard with three copies then elects only while two of them have kept running since the outage began.
Set
FELIX_START_WITHOUT_CONTROLPLANE=true
to let it rejoin. It then starts from what the control plane last told it,
kept in its data directory, votes, and follows its shards’ leaders. With the unified fence on, the default, that vote does not
count toward a promotion: the broker attests nothing until it reaches the
control plane and its restore check has run. A failover that needs its vote
waits for the control plane. FELIX_UNIFIED_FENCE=0 restores the older rule,
where the restarted broker’s vote counts during the outage. Until the
control plane is back and it registers, it serves no client and does not stand
for election. It does not answer pings either, so followers of a shard it led
treat it as down and elect one of themselves. It starts this way only if the
control plane is down rather than refusing it, and only from the data
directory the control plane last registered.
It also starts this way only after a restart that keeps /dev/shm: the broker
process restarting on the same Linux machine without a reboot, or its
container restarting inside a live Kubernetes pod. A fresh container
(docker restart, a Compose restart, a new pod), a reboot, and macOS all wait
for the control plane. Each start leaves a small file in
/dev/shm naming the data directory as that start left it. A directory put
back from a backup does not match it, and a reboot clears it. Without a match
the broker cannot rule out that its data was restored and has lost ballots it
gave, so it waits for the control plane. Peers learn a broker’s address only from the control
plane, so a broker restarted while it is down votes only if it comes back at
its registered address. See
Starting while the control plane is down.
16 generations per outage. The control plane adopts an election only up to 16 generations past the last one it wrote for the shard. Each election uses a generation, and so does each failed round after its ballot. A shard that reaches the limit stops electing until the control plane is back.
Brokers outside the replica set do not hear about it. Only the shard’s replicas learn the new leader. Every other broker keeps the old route until the control plane is back.
Some shards wait for the control plane. A replica does not stand while its replica set has a change pending (most moves mid-way), or while the control plane records a different set from the one the replica keeps. Neither is unsafe to skip; the shard simply waits for the control plane as before.
Mixed fleets do nothing. Every serving broker must offer elections before
the control plane lets you finalize joint_fence or replica_election, and
until replica_election is finalized no broker stands, even one that offers
it.
Turning it on
Section titled “Turning it on”You need:
- every broker on 0.7.0 with
FELIX_INTERNAL_ELECTION=true, restarted to pick it up. It works only withFELIX_INTERNAL_FENCEleft on (the default), and withFELIX_LEADER_SUSPECT_AFTER_MSnot0; - every control-plane member on 0.7.0. Under Raft the control plane checks
this:
joint_fenceneeds metadata level 25 on every member, andreplica_electionlevel 26. Under Postgres it cannot see other instances’ versions, so finish the control-plane upgrade before you start.
Then finalize these fleet features, in this order:
generation_start, thenmajority_ack, thenlease_free_reads(andfenced_cachesif you want caches elected).stream_incarnationsandcache_incarnations.joint_fenceis refused until both are finalized.joint_fence. From here on, a change to a replica set (a move, a replacement) needs a majority of both the old and the new set before the new leader serves.replica_election. From here on, replicas elect.
A new cluster finalizes steps 1 and 2 by itself at its first broker
registration (see A new cluster starts with them
finalized),
so you finalize only joint_fence and replica_election. An upgraded
cluster finalizes whichever of steps 1 and 2 it has not done yet, with the
cautions in Upgrades and
compatibility.
For each feature, dry-run first:
felixctl features finalize joint_fence --dry-runfelixctl features finalize joint_fence --yesfelixctl features finalize replica_election --dry-runfelixctl features finalize replica_election --yesThe dry run says either that the feature can be finalized, or what stops it: the features to finalize first, or the live or draining brokers that do not support it. Under Raft the metadata level is checked only by the real finalize, so it can still be refused after a clean dry run; the refusal says what is missing.
Finalizing is one-way. Once joint_fence is finalized, a broker that registers
without it is refused and exits, and a broker reports joint_fence only with
FELIX_INTERNAL_ELECTION=true. Keep the setting on every broker from then on,
including any broker you add or replace later.
The defaults suit most clusters. FELIX_LEADER_SUSPECT_AFTER_MS sets how long
followers wait before they suspect a leader; a shorter window fails over faster
and replaces a slow leader more often. Keep FELIX_SHARD_ELECTION_HOLD_MS on
the control plane at twice that value. FELIX_ELECTION_DEFER_FOR_MS (60 s)
bounds how long a newly elected leader that never opens holds its followers
back before one of them may stand again.
A shard can also get stuck on a leader whose broker is alive, for example one
that keeps fencing after a restart while its copies have promised a newer
generation elsewhere. Pings get answered, so no follower suspects that broker.
A follower that has heard nothing from the leader for that shard for
FELIX_SHARD_SUSPECT_AFTER_MS (120 s), and whose copy holds a generation that
never started, may stand for that one shard. An idle shard is not suspected,
since its generation started on every copy. The one exception: a copy left
holding a ballot from a candidate that gave up refuses its leader’s batches,
and standing is how it gets back in, at the cost of one generation. With the control plane up, it
renames the stuck leader before that bound runs out, so keep the value above
the control plane’s stuck-leader bound, FELIX_SHARD_STUCK_LEADER_MS (70 s). Set it to 0 to turn per-shard
suspicion off.
The same bound covers a leader that won an election just as the control plane named someone else at the same generation and then went down. The other broker’s route names itself, so its copy refuses the elected leader. Once that has lasted the bound, the elected leader fences its replica set again one generation up, and the refused copy follows it.
It also covers a leader that was elected, then restarted while the control plane was down. The restart forgets the election, so the broker goes back to the control plane’s older route for that shard, and it cannot open there: its own copy already promised the newer generation to itself. After the bound, it stands for the shard again one generation up, and its followers vote for it.
How it is checked
Section titled “How it is checked”The election is modelled in TLA+ (docs/formal/), and each rule above has a
test. Some of them:
a_granted_pre_vote_puts_the_ballot_on_disk_then_records_the_standing,a_refused_pre_vote_writes_nothing,the_leader_refuses_a_pre_vote_against_itself,a_pending_set_change_is_not_stood_on,a_set_the_control_plane_does_not_record_is_not_stood_on,a_candidate_does_not_stand_past_the_adoption_window,a_cut_off_leader_takes_the_elected_leaders_fence,an_elected_leaders_report_is_adopted_once_replica_election_is_finalized,joint_fence_is_refused_while_a_serving_broker_lacks_election,replica_election_is_refused_until_joint_fence_and_the_rest_are_finalized.
The full design, with every rule and its reason, is in the replication design notes.
