Skip to content

Building Resilient Clients

Brokers get restarted, shards move, tokens expire and networks drop packets. Felix’s clients absorb most of that, and this page covers the part they leave to you. Each section says what the client already does, then shows the code an application adds, in every language that has the API. The per-language pages (Rust, Python, TypeScript, C#, C) are the reference for everything else.

The examples read credentials from the environment or a file. They rely on the client’s own timeouts (10 s to connect to a broker, 30 s for a publish acknowledgement, 30 s for a token fetch) and pass a cancellation token where the client takes one. Names such as handle, charge and checkpoint stand for your own code.

The C examples use the waiter and wait_for helpers from A minimal example to turn a callback into a blocking call, and on_connect, on_publish, on_subscribe, on_event and on_done are the one-line callbacks that pass their result to finish. A production C client would drive the callbacks from its own event loop instead.

Pick your language in any tab below and the whole page switches with it.

For which error maps to which type in your language, see Errors Across Clients. For tokens and TLS options side by side, see Auth and TLS.

Give the client more than one seed. It asks whichever seed answers which brokers exist, adds them, and keeps your seeds whatever the answer, so a stale seed cannot leave it worse off. One reachable seed is enough to start, and listing several lets the first connection survive one being down.

Prefer names to IP addresses. When no broker the client knows answers, it resolves the names again, which is how it gets back in after a rollout has given every broker a new address. In Rust, a seed is an address or a host:port name (Seed). Every other client takes host:port strings, and a name that does not resolve at connect time is an error (ValueError, an InvalidArg rejection, a FelixException with status InvalidArgument, FELIX_STATUS_INVALID_ARGUMENT). server_name is the name the broker’s certificate carries, and TLS is always on.

use felix_client::{Client, ReconnectPolicy, Seed};
// Names are resolved again whenever no known broker answers.
let seeds: Vec<Seed> = vec![
"felix-0.felix.svc:5000".parse()?,
"felix-1.felix.svc:5000".parse()?,
"felix-2.felix.svc:5000".parse()?,
];
// The connect gives up after `config.connect_timeout` per broker (10 s by default).
let client = Arc::new(
Client::connect_seeds(&seeds, "felix.example.com", config, ReconnectPolicy::default())
.await
.context("connect to the Felix cluster")?,
);
tracing::info!(brokers = client.endpoints().await.len(), "connected");
  • A broker fails. A call waiting on it fails once the connection has been silent for the idle timeout (6 s by default), with the connection already rebuilt from the brokers that remain. Reads and writes the broker refused without applying are retried on another broker. A write that may have been applied is reported as an unknown outcome and not sent again (Writes whose outcome is unknown). A cache watch open on that broker is reopened on whichever broker owns its shard now, from the offset after the last change it delivered, under the same policy as a subscription. If no broker takes it before the deadline, the watch ends and reports why. A cache request that cannot connect to the shard’s owner goes through another broker and asks again who owns the shard, rather than waiting on the dead owner every time.
  • A shard moves. A publish that reaches the old owner is redirected to the new one, and the client remembers where each shard lives. A subscription follows its shard to the new owner and carries on.
  • A broker fails over. Publishes ride out a leader failover inside the retry budget below. A leader change costs the client about the idle timeout.

The client does not hide a failure it cannot recover from. Those arrive as the errors in Errors and retry classes.

Publishes, opening and following subscriptions, commits, conditional publishes, state reads, cache calls and consumer-group calls retry under one policy. A read retries on any error that is not terminal. A write, including a group poll, which claims records, retries only on a broker’s refusal that says nothing was applied.

  • Attempts is how many times a call tries to reach a broker. Each attempt tries every broker the client knows. The default is 5.
  • Deadline is a ceiling on one call across every attempt and every wait. It is off by default. It is checked before each backoff sleep, so a call can last the deadline plus one attempt’s own timeout.
  • Backoff and max backoff shape the wait between attempts: the first retry waits a random time up to the backoff, the ceiling doubles each attempt up to the maximum, and the jitter keeps every client of a cluster from retrying in step. The defaults are 200 ms and 2 s, and every wait is capped at the max backoff, even when the backoff is larger.

A deadline shorter than one attempt’s own timeout means no retry happens: the first attempt spends the whole budget. Set it to cover at least one attempt plus the backoff you can afford, or leave it off to wait out a failover.

Language Attempts Deadline Backoff
Rust ReconnectPolicy.attempts deadline: Option<Duration> backoff, max_backoff
Python reconnect_attempts reconnect_deadline (seconds) reconnect_backoff, reconnect_max_backoff (seconds)
TypeScript reconnectAttempts reconnectDeadline (milliseconds) reconnectBackoff, reconnectMaxBackoff (milliseconds)
C# ReconnectAttempts ReconnectDeadline (TimeSpan) ReconnectBackoff, ReconnectMaxBackoff (TimeSpan)
C reconnect_attempts reconnect_deadline_ms reconnect_backoff_ms, reconnect_max_backoff_ms
use felix_client::{Client, ReconnectPolicy};
// `seeds` is the Vec<Seed> built in the previous section.
let policy = ReconnectPolicy {
attempts: 8,
backoff: Duration::from_millis(250),
max_backoff: Duration::from_secs(4),
// The longest any one call may spend retrying.
deadline: Some(Duration::from_secs(30)),
};
let client = Client::connect_seeds(&seeds, "felix.example.com", config, policy).await?;

A client made for another user with with_identity (Rust, Python, TypeScript) inherits the policy.

A token lasts 15 minutes by default, and the broker checks it each time a stream authenticates, including the streams a reconnect opens. A process that outlives one token needs a token provider. The client calls it lazily: when it holds no token, when the held one has passed two thirds of its life, or when a broker has just refused it. An idle client makes no calls, and the client never runs two calls at once. If a fetch fails while the held token is still valid, the client keeps using it and asks again about five seconds later. A fetch that does not answer within 30 seconds counts as failed.

The provider returns a token and nothing else, so it can read a file a sidecar keeps fresh, call a secrets service, or run the control plane’s refresh flow. Each language page has the full exchange and refresh example for the control plane: Rust, Python, TypeScript, C# and C. A refresh token works once, so the provider must store the replacement before it returns.

use felix_client::RefreshingToken;
// The provider is async, so it awaits its I/O and never blocks the runtime.
// `tokio::fs` needs tokio's `fs` feature.
let tokens = RefreshingToken::new(|| async {
let token = tokio::fs::read_to_string("/var/run/secrets/felix/token").await?;
anyhow::Ok(token.trim().to_owned())
});
config.auth_tenant_id = Some("acme".into());
config.token_provider = Some(Arc::new(tokens));
let client = Client::connect_seeds(&seeds, "felix.example.com", config, policy).await?;

A refused token is one retry, not a loop. The client asks the provider again, and only if it returns a different token does it try again. If the broker refuses that one too, the call fails with the language’s auth error. The client does not retry that error: a provider call can rotate a refresh token, so each retry would cost a rotation. The next operation asks the provider again.

An auth error is therefore final for that call. Log it, alert on it, and stop sending the request; a retry cannot grant a permission. A revoked token, or a refresh token that has expired, needs a new login, after which you connect a new client.

use felix_client::{AckMode, AuthError, AuthErrorKind};
if let Err(err) = client
.publish("acme", "prod", "orders", payload, AckMode::PerMessage)
.await
{
match err.downcast_ref::<AuthError>() {
// The provider could not produce a token: fix the token source.
Some(auth) if auth.kind == AuthErrorKind::Token => {
tracing::error!(error = %auth.message, "token source failed");
}
// The broker refused a token the provider did return: log in again.
Some(auth) => tracing::error!(error = %auth.message, "token refused"),
None => return Err(err),
}
}

A broker that sends error codes attaches a code and a retry class to every refusal. The class says what you may do about it, and it is the field to branch on. Never match a message where a code is available: it is prose, and it gets reworded. C# and C carry no code, which the stale-claim example below has to work around. The codes and their classes are in the wire protocol and the error code table.

Class What it means What to do
retry Nothing was applied. The shard is moving, the broker is draining, or another broker owns the shard. Send it again. The cluster client already did, with backoff, before the error reached you.
retry_after Nothing was applied. The broker is shedding load. Wait at least retry_after_ms from the error’s detail, then send it again.
redirect Nothing was applied. Another broker owns the shard. Send it again. The cluster client follows the redirect itself.
outcome_unknown The write may have been applied, as after a quorum timeout. Do not resend a plain write. Use an idempotent producer or a conditional publish, or read to find out.
fatal It cannot succeed as sent: bad credentials, a missing stream, a limit. Fix the request or the credentials. Do not retry.

The cluster client has already acted on the class before an error reaches you: a fatal error is returned at once, an outcome_unknown write is never re-sent, and the other classes are retried inside the attempts and deadline of the reconnect policy. An error that reaches your code has exhausted that budget, so a retry loop of your own is for outages longer than it.

How each client shows the class:

Language Where to read it
Rust BrokerError { code, retry, detail }, recovered with downcast_ref. None means no code.
Python err.retry, err.code, err.detail, plus an exception class per kind.
TypeScript err.retry, err.code, err.detail, err.retryable, plus a class per kind. retryable is also true for a connection error.
C# FelixException.Status, IsRetryable and OutcomeUnknown. The class is folded into the status, and IsRetryable is also true for a connection error.
C The FelixStatus of the callback. The class is folded into the status.

A lost acknowledgement is an unknown outcome. A publish that was written before the connection failed reports outcome_unknown (Rust AckLost, Python and TypeScript OutcomeUnknownError, FELIX_STATUS_OUTCOME_UNKNOWN), not a connection error. A connection error that remains means the write was not sent, or the failure came before it. Resend an unknown outcome only with an idempotent producer, or after checking, with a conditional publish or by reading the stream back. C#’s IsRetryable and TypeScript’s retryable suit reads, and the write loops below test the class or the specific statuses instead.

A stale settle is final. The cluster client does not retry a stale_claim refusal, and reports its retry class as fatal whatever the broker sent, so every client sees it at once. Log it and move on (Consumer groups under failure).

A broker that predates error codes sends none. Rust then gives you no BrokerError, and the other clients choose the exception class or status from the message. Treat a missing code as “no information”, not as success.

use felix_client::{AckMode, BrokerError, RetryClass};
enum Verdict {
// Nothing was applied: send it again after the pause.
Resend(Duration),
// It may have landed: find out before writing again.
Reconcile,
// Sending it again cannot help.
GiveUp,
}
fn judge(err: &anyhow::Error) -> Verdict {
let Some(refusal) = err.downcast_ref::<BrokerError>() else {
// No code: a lost connection, which is ambiguous for a write, or a broker
// that predates error codes. Do not resend a plain write.
return Verdict::GiveUp;
};
match refusal.retry {
RetryClass::Retry | RetryClass::Redirect => Verdict::Resend(Duration::from_millis(200)),
RetryClass::RetryAfter => {
let wait = refusal.detail.as_ref().and_then(|detail| detail.retry_after_ms);
Verdict::Resend(Duration::from_millis(wait.unwrap_or(1_000)))
}
RetryClass::OutcomeUnknown => Verdict::Reconcile,
_ => Verdict::GiveUp,
}
}
let mut attempt = 1;
let offset = loop {
match client
.publish("acme", "prod", "orders", payload.clone(), AckMode::PerMessage)
.await
{
Ok(offset) => break offset,
Err(err) if attempt < 4 => match judge(&err) {
Verdict::Resend(pause) => {
tracing::warn!(attempt, "publish refused; sending again: {err:#}");
tokio::time::sleep(pause).await;
attempt += 1;
}
Verdict::Reconcile | Verdict::GiveUp => return Err(err),
},
Err(err) => return Err(err),
}
};
tracing::info!(?offset, "published");

A publish that fails after the broker may have written it has an unknown outcome: a connection dropped before the acknowledgement, or the leader wrote the record and a majority did not confirm it in time. By default no client sends it again, because nothing downstream could tell two copies apart. You choose one of three answers.

  • Idempotent producer. The producer numbers its batches, the shard’s leader remembers the last few, and a batch carrying a sequence it already holds is answered from memory instead of being appended. A resend after a lost acknowledgement lands once, across reconnects and leader changes. Every client has it.
  • Conditional publish. publish_if writes only if the shard’s next offset is the one you expect. It suits a single writer that must not append after it has been replaced, and a refusal is an answer, not an error. Rust, Python, TypeScript, C# and C have it.
  • At-least-once. The client resends an ambiguous publish to another broker. The record is certain to land and may land twice, so the consumers must tolerate a duplicate, usually by carrying an id in the payload. Every client has it, with or without a routing key.

With an idempotent producer, a failure other than a broker refusal leaves the batch in doubt. The next call on that shard must carry the same batch and key; it goes out under the same sequence and lands once. A different batch is refused without being sent until the first one settles. A refusal, such as the shard no longer knowing the producer, ends the producer on that stream.

use felix_client::PublishRefused;
let producer = client.idempotent_producer().await?;
let mut attempt = 1;
let offset = loop {
match producer.publish("acme", "prod", "orders", payload.clone()).await {
Ok(offset) => break offset,
// The shard refuses this producer (a sequence gap, an unknown producer):
// retrying the same batch will not change that.
Err(err) if err.downcast_ref::<PublishRefused>().is_some() => {
return Err(err.context("the broker refused the producer"));
}
// In doubt: the same call again is safe, and lands once.
Err(err) if attempt < 5 => {
tracing::warn!(attempt, "publish in doubt; sending the same batch again: {err:#}");
tokio::time::sleep(Duration::from_secs(1)).await;
attempt += 1;
}
Err(err) => return Err(err),
}
};
tracing::info!(?offset, "published exactly once");
producer.close_within(Duration::from_secs(10)).await?;

The C felix_client_publish takes FELIX_DELIVERY_AT_LEAST_ONCE where the examples above pass FELIX_DELIVERY_AT_MOST_ONCE, and the C# PublishAsync takes delivery: Delivery.AtLeastOnce. Python passes at_least_once=True and TypeScript the last positional argument. In Rust, publish_at_least_once is the call.

publish_if answers with where the write landed or, when the shard has moved on, the shard’s tail. Do not resend one whose answer was lost. Read the shard at the expected offset to find out whether it landed.

use felix_client::ConditionalWrite;
// The offset this writer expects to write next.
let mut next = 0;
for tick in ticks {
match client
.publish_if("acme", "games", "match-7", None, vec![tick], next)
.await?
{
ConditionalWrite::Written { offset } => next = offset + 1,
ConditionalWrite::Refused { tail } => {
anyhow::bail!("another writer has the stream; its next offset is {tail}")
}
}
}

A subscriber’s queue sheds records under the default policy rather than blocking the publisher, so a subscriber can miss records. On a durable stream every delivered event carries its log offset, and that is what makes losing your place detectable and fixable.

Checkpoint the offset you handled and resume at the next one. start is the first record you have not seen, so a resuming consumer passes offset + 1. Handle the record before you save the checkpoint, so a crash between the two repeats one record instead of losing it.

Check the offsets as they arrive. A jump between consecutive offsets is a drop, with one exception the broker tells you about: a new leader writes a generation-start record that takes an offset and is never delivered, and the event after it reports the count in skipped_before. The first offset an event accounts for is offset - skipped_before.

A subscribe can also fail with a cursor error, for one of two reasons. Too old means retention discarded the checkpoint’s offset, and the only recovery is to accept the gap and say so in a log: a silent restart at the tail loses records with nothing reported. In the future means the checkpoint is past the stream’s tail, so it came from another stream or was corrupted. That is a bug in the caller’s checkpoint, and the loop should fail loudly. Rust’s error carries the reason and available, the nearest offset that would have worked. The other clients raise the same error without the reason, so their loops fail on it unless the caller opted into skipping the gap (allow_gap), and the error message says which case it was.

use felix_client::{AuthError, BrokerError, CursorErrorReason, RetryClass};
use felix_client::{StartPosition, SubscribeCursorError};
// Too old: take the smallest gap. In the future: the checkpoint is not this stream's.
fn after_cursor_error(cursor: &SubscribeCursorError) -> Result<StartPosition> {
match cursor.reason {
CursorErrorReason::TooOld => {
tracing::error!(
requested = cursor.requested,
resuming_at = cursor.available,
"checkpoint is past retention",
);
Ok(StartPosition::Offset(cursor.available))
}
_ => anyhow::bail!(
"checkpoint {} is beyond the stream's tail ({})",
cursor.requested,
cursor.available
),
}
}
// A failure that waiting and subscribing again could outlast.
fn worth_resuming(err: &anyhow::Error) -> bool {
if err.downcast_ref::<AuthError>().is_some() {
return false;
}
err.downcast_ref::<BrokerError>()
.is_none_or(|refusal| refusal.retry != RetryClass::Fatal)
}
let mut start = match checkpoint.load().await? {
Some(offset) => StartPosition::Offset(offset + 1),
None => StartPosition::Earliest,
};
let mut expected: Option<u64> = None;
loop {
let mut subscription = match client
.subscribe_from("acme", "prod", "events", Some(start))
.await
{
Ok(subscription) => subscription,
Err(err) => match err.downcast_ref::<SubscribeCursorError>() {
Some(cursor) => {
start = after_cursor_error(cursor)?;
expected = None;
continue;
}
None => return Err(err),
},
};
loop {
match subscription.next_event().await {
Ok(Some(event)) => {
if let (Some(want), Some(got)) = (expected, event.offset) {
let first = got - event.skipped_before;
if first > want {
tracing::warn!(dropped = first - want, "records were dropped");
}
}
handle(&event.payload).await?;
if let Some(offset) = event.offset {
checkpoint.save(offset).await?;
start = StartPosition::Offset(offset + 1);
expected = Some(offset + 1);
}
}
Ok(None) if subscription.stream_deleted() => return Ok(()),
// The broker ended it for another reason: wait, then subscribe again from `start`.
Ok(None) => {
tokio::time::sleep(Duration::from_secs(1)).await;
break;
}
Err(err) => {
if let Some(cursor) = err.downcast_ref::<SubscribeCursorError>() {
start = after_cursor_error(cursor)?;
expected = None;
} else if worth_resuming(&err) {
// The client already tried other brokers. Resuming from `start` is lossless.
tracing::warn!("subscription lost; resuming: {err:#}");
tokio::time::sleep(Duration::from_secs(1)).await;
} else {
return Err(err);
}
break;
}
}
}
}

The cluster client, and every binding, resubscribes by itself when a connection drops mid-stream: on a durable stream it resumes from the offset after the last one it delivered, and where it cannot it raises an error rather than ending as though the stream had finished. The loops above are for a restart of your process, and for the cases where the client gave up. In C, call read_from again when it returns FELIX_STATUS_CONNECTION, after a pause. A FELIX_STATUS_CURSOR result is for the caller to fail on, or to answer by setting *next to the start you chose and subscribing with FELIX_START_EARLIEST, if you accept the gap.

A subscription reads one shard. subscribe_sharded opens one per shard, follows each shard’s own owner, and merges them. Order holds within a shard and nowhere across shards. Resuming is a map from shard to last offset, because offsets are per shard, and a single number carried across shards replays on every shard but one.

A lost shard is an item in the stream, not an error. The other shards carry on, so a consumer that ignored the item would read part of the stream while believing it read all of it. Alert on it.

A shard that has delivered nothing has no position. On resume, such a shard starts wherever start says, which is the live tail by default. Pass earliest alongside the positions if records published to it while you were away matter.

use felix_client::{ShardEvent, StartPosition};
let mut stream = match checkpoint.load().await? {
Some(positions) => {
client
.resubscribe_sharded("acme", "prod", "orders", positions, Some(StartPosition::Earliest))
.await?
}
None => {
client
.subscribe_sharded("acme", "prod", "orders", Some(StartPosition::Earliest))
.await?
}
};
while let Some(item) = stream.next().await {
match item {
ShardEvent::Record { shard, event } => {
handle(shard, &event).await?;
checkpoint.save(stream.positions()).await?;
}
ShardEvent::ShardLost { shard, error } => tracing::error!(shard, %error, "shard lost"),
ShardEvent::ShardRecovered { shard } => tracing::info!(shard, "shard recovered"),
ShardEvent::ShardMoved { shard, .. } => tracing::info!(shard, "shard moved; followed"),
ShardEvent::StreamDeleted { shard } => tracing::info!(shard, "stream deleted"),
_ => {} // `ShardEvent` is non-exhaustive
}
}

Three signals look alike from the outside and mean different things.

Signal What happened What the client does
A jump in offsets, less skipped_before The subscriber’s queue overflowed and records were dropped. Nothing. This is the only trace a broker without subscription_lagged leaves, so check for it.
subscription_lagged A durable stream’s queue on the broker dropped records, and the broker ended the subscription after delivering what was queued. resume_from is the first dropped offset. The cluster client and every binding resubscribe after the last event they delivered and replay the gap from the log. A bare Rust Connection subscription ends with SubscriptionLagged and you resubscribe.
shard_moved A rebalance or a drain moved the shard. The old owner delivered everything it committed, then said where the shard went. The cluster client and every binding follow the shard to its new owner. On a durable stream nothing is repeated or skipped. A bare Rust Connection subscription ends and shard_moved() says where.

Resume after the last event you received, not at resume_from: the broker can also drop a frame on its way out to a slow subscriber without reporting it, and that frame can sit below resume_from. An in-memory stream has no offsets, so it keeps dropping, and a moved shard resumes at the new owner’s tail.

Python, TypeScript, C# and C never show you subscription_lagged or a plain shard move; the subscription handles both. A sharded subscription reports a move as an item for information.

A subscriber that cannot keep up loses records, but only its own. Each subscription has a bounded queue on the broker, and a publisher never waits on a subscriber under the default overflow policy (drop_new). The stream’s policy lives on the broker. A subscriber picks only the size, because a subscriber that could pick block could stall every publisher on the shard just by reading slowly.

Two queues matter to a client, and the Rust client sets both:

  • broker_sub_queue_capacity asks the broker for a different queue size for this client’s subscriptions, counted in published batches. The broker clamps it to FELIX_SUBSCRIBER_QUEUE_CAPACITY_MAX, and queue_capacity() on a Client subscription reports what it granted (None from a broker that predates the option, which keeps the stream’s default).
  • client_sub_queue_capacity and client_sub_queue_policy size the client’s own queue per subscription. ClientSubQueuePolicy::DropNew drops the arriving item. Block stops reading the event stream until the application catches up, which pushes the wait back to the broker’s queue and its policy. DropOld currently behaves as DropNew.

A larger queue makes drops rarer and does not change what a drop is. On a durable stream a drop ends the subscription with subscription_lagged, and the client resubscribes and replays the gap from the log, so a reader that must not miss records uses a durable stream. An ephemeral stream keeps dropping. Python, TypeScript, C# and C take the stream’s and the client’s defaults and do not expose these settings.

// A reader with bursty processing asks for more room.
config.broker_sub_queue_capacity = Some(4096);
config.client_sub_queue_capacity = 8192;
let client = Arc::new(
Client::connect_seeds(&seeds, "felix.example.com", config, ReconnectPolicy::default())
.await?,
);
let mut subscription = client.subscribe("acme", "prod", "webhooks").await?;
// Hand slow work to a task over a bounded channel instead of doing it in the
// read loop, where it holds up the next event.
let (queued, mut work) = tokio::sync::mpsc::channel::<Event>(100);
let worker = tokio::spawn(async move {
while let Some(event) = work.recv().await {
slow_work(event).await?;
}
anyhow::Ok(())
});
while let Some(event) = subscription.next_event().await? {
if queued.send(event).await.is_err() {
break; // the worker stopped; its result below says why
}
}
// Closing the channel lets the worker drain what is queued and finish.
drop(queued);
worker.await??;

In C, read events on one thread and hand them to a worker over a bounded queue of your own, with the same shape as the examples above. A callback that blocks stalls the client, so never do the slow work inside one.

A consumer group hands each record to one consumer and takes it back if nobody settles it. Records are pulled, because only the consumer knows when it has capacity. Four things go wrong, and each has a tool.

  • A consumer dies mid-record. The record’s claim lapses after the visibility timeout (FELIX_GROUP_VISIBILITY_TIMEOUT_MS, 30 s by default) and the record goes to whoever polls next. The same record can therefore be seen twice, so handling must be idempotent. attempts counts deliveries, this one included, so 1 is a first attempt and anything higher is a redelivery.
  • The work takes longer than the timeout. Extend the claim before it lapses. extend is for the delivery you hold: once the claim has lapsed and the record has gone out again, it is refused with stale_claim, even if your worker is still running. Every client has extend, named group_extend in Rust and Python and groupExtend in TypeScript.
  • A consumer restarts. A claim belongs to the group, not the process, so a restarted consumer would wait out the timeout for its old records. Poll with a stable consumer name (1 to 128 bytes, unique per live process) and set reclaim after a restart, and the records that name still holds come back at once. Only the first poll on a connection reclaims, so leaving it set on every poll is harmless. Claims are the leader’s memory, so this holds while the shard’s leader stays put; after a failover the group resumes from its durable position.
  • A settle comes too late. An ack, nack or extend for a claim that no longer stands is refused with stale_claim. Nothing was applied, and the record will be delivered again. The cluster client retries a stale ack or nack with backoff first, so the refusal surfaces only after the reconnect policy’s attempts. Log it and move on. Rust, Python and TypeScript show the code. C# and C report it as the general error status, so the C# example recognises it by the broker’s message, on settles only.

nack hands a record back for immediate redelivery, or after a delay on a broker with claim control: group_nack_after in Rust and Python, groupNackAfter in TypeScript. A record that keeps failing is dead-lettered after FELIX_GROUP_MAX_ATTEMPTS. See Queues for dead letters, redrive and the broker’s side of all of this.

use felix_client::{BrokerError, ErrorCode, GroupMember};
fn is_stale_claim(err: &anyhow::Error) -> bool {
err.downcast_ref::<BrokerError>()
.is_some_and(|refusal| refusal.code == ErrorCode::StaleClaim)
}
// A claim that lapsed means the record belongs to the group again.
fn tolerate_stale(result: Result<()>) -> Result<()> {
match result {
Err(err) if is_stale_claim(&err) => {
tracing::warn!("claim lapsed; the record will be delivered again");
Ok(())
}
other => other,
}
}
let member = GroupMember { consumer: "billing-worker-1".into(), reclaim: true };
while !stop.load(std::sync::atomic::Ordering::Relaxed) {
let records = client
.group_poll_as("acme", "prod", "orders", 0, "billing", Some(&member), 16, Duration::from_secs(5))
.await?;
for record in records {
// Ask for time before the slow step.
if let Err(err) = client
.group_extend("acme", "prod", "orders", 0, "billing", &record, Duration::from_secs(60))
.await
{
if is_stale_claim(&err) {
continue; // someone else has it now
}
return Err(err);
}
match charge(&record.payload, record.attempts).await {
Ok(()) => tolerate_stale(
client.group_ack("acme", "prod", "orders", 0, "billing", record.offset).await,
)?,
Err(err) => {
tracing::warn!(offset = record.offset, "charge failed: {err:#}");
// Back off before the retry instead of redelivering at once.
tolerate_stale(
client
.group_nack_after(
"acme", "prod", "orders", 0, "billing", &record, Duration::from_secs(30),
)
.await,
)?;
}
}
}
}

Dropping or closing a handle ends it, and the one thing closing cannot do is wait. Wait for what you care about first.

  • Stop reading. Close subscriptions and watches. A client’s subscriptions hold their own connection and keep running after the client is closed, so closing the client does not stop them.
  • Flush what you wrote. A publish with AckMode::None returns once it is queued, and records still queued are lost with the process. Rust’s finish() waits for everything queued on every broker and then ends publishing through that client. An idempotent producer’s close waits for the batches it was handed, and a close with a timeout gives up after that long.
  • Close the client last. Whatever it still has pending is cancelled or fails, and a publish that may have been sent is reported as an unknown outcome.

Shutting down your process is not the same as a broker shutting down (see Process lifecycle and graceful shutdown). A broker that is told to stop marks itself draining so load balancers stop routing to it, asks the control plane to move its shards elsewhere, stops accepting connections, and gives streams already in flight a bounded grace period before it closes each connection. Nothing in your code is needed for that to work: a subscription follows its shard to the new owner, a publish refused with draining or shard_unavailable is retried on another broker, and a connection the broker closes is replaced from the brokers that remain. What you see is a short pause. The only publishes you must treat with care are those in flight when the connection closed, and those arrive as an unknown outcome. A rolling restart needs no special client settings. If you set a reconnect deadline, keep it longer than a broker takes to drain; FELIX_SHUTDOWN_HANDOFF_TIMEOUT_MS bounds how long it spends handing its shards over.

// Stop reading. Dropping a subscription closes it on the broker.
drop(subscription);
// Let the producer's batches finish, but not for ever.
producer.close_within(Duration::from_secs(10)).await?;
// Wait for AckMode::None publishes to be written, then end publishing.
tokio::time::timeout(Duration::from_secs(15), client.finish())
.await
.context("finish took longer than 15 seconds")??;
  • Choosing a Client covers how the bindings relate and what the conformance suite checks.
  • Delivery semantics has the guarantees each stream type gives and the failure scenarios behind this page.
  • Queues and Pub/Sub cover consumer groups and subscriber queues from the broker’s side.
  • Wire protocol lists the frames and error codes the clients act on.