Skip to content

Error Codes Reference

A broker that refuses a request says why with a code and says what the client may do about it with a retry class. This page lists every code, retry class and reason, what usually causes it, what to do, and what you catch in each client. The frame layout is in the wire protocol and the protocol specification. A script checks this page against crates/protocol/felix-wire/src/client/error_code.rs and message/fields.rs (scripts/check_error_codes_reference.py, run by task docs:evidence), so a code, class or reason added there without a row here fails CI.

Codes are sent only to a client that offered FEATURE_ERROR_CODES when it authenticated. Every official client does. Without the feature the broker sends the same failures as plain text and the client falls back to reading the message, which is why a missing code does not mean success.

The code and the retry class travel separately. A client that does not know a code still knows from the class whether sending again is safe, so an unknown code never fails the frame. Act on the class first and on the code second. A broker may send a class other than the default for a code when it knows better, and the class it sends is the one that counts.

An error can also be the last frame on a subscription’s event stream or a cache watch, for example when the credential it was opened with is revoked. The broker writes the error before it ends the stream.

Class Number Meaning What to do
retry 1 Nothing was applied. Sending the request again is safe. Retry, ideally against a refreshed route.
retry_after 2 Nothing was applied. Wait before sending again. detail.retry_after_ms may say how long. Back off for at least the hint, then retry.
redirect 3 Nothing was applied. Another broker owns the shard. Send it to the owner.
outcome_unknown 4 The request may have been applied. Resend only an idempotent request. Otherwise read back the state, or accept a possible duplicate.
fatal 5 Sending it again fails the same way. Stop and fix the request or the credential.

A class this version does not know is read as fatal, the one reading that cannot duplicate a write. The number is what a binary publish ack carries.

The class is the default the broker sends. Number is the u16 a binary publish ack carries.

Code Number Default class Meaning Typical cause What the client should do
unauthenticated 1 fatal The stream has not authenticated, or the credential was refused. A request before auth, or a token that does not verify or has expired. Get a new token. A client with a token provider does this once per refused stream. Do not retry with the same token.
forbidden 2 fatal The credential does not grant this operation. A missing permission, or a tenant other than the token’s. Fix the grant. Retrying cannot succeed.
not_found 3 retry_after The tenant, namespace, stream, cache or group does not exist on this broker. A typo, or a stream that was never created. On a cluster, a broker promoted a moment ago has not yet learned a stream it is about to serve. Retry briefly, as the Rust cluster client does for 5 seconds. If it persists, create the resource or fix the name.
invalid_request 4 fatal The request is malformed or can never succeed as sent. A malformed frame or batch, unknown frame flags, a missing request_id, a second auth, a bad watch filter, a shard at or past the stream’s shard count. A head trim past its own record (detail.reason trim_past_tail) or on an in-memory stream (not_durable). Fix the request. Nothing was written.
shard_unavailable 5 retry Nobody can serve the shard right now. A shard being moved, opened or failed over. detail.reason says which; see shard unavailable reasons. Retry with backoff, honoring retry_after_ms when present. Nothing was applied.
not_leader 6 redirect Another broker owns the shard. The client went to a broker that does not lead the shard, and the dedicated not_leader message could not be sent: to a client without FEATURE_REDIRECT, or for a publish the broker cannot forward. Go to the owner. The cluster client does this itself.
quorum_timeout 7 outcome_unknown The leader wrote the batch and a majority did not confirm it in time. It may survive. A write to a Quorum stream or cache while followers are slow or down. Do not resend a plain publish. Use an idempotent producer, or reconcile by reading the stream.
leadership_lost 8 outcome_unknown Leadership moved after the leader wrote the batch and before a majority held it. A write to a Quorum stream or cache during a failover or move. As for quorum_timeout.
unacknowledged 9 outcome_unknown The broker stopped waiting for the write’s outcome. A forwarded batch whose answer never came, a commit that outlasted the ack wait, a publish whose shard log was reset while it waited behind earlier publishes, or a head trim that waited 30 seconds to commit (detail.reason trim_timed_out). As for quorum_timeout. The publish case reached no reader. A timed-out trim may still land, and a retry with the same expected offset is then refused with offset_mismatch: read the shard to tell.
overloaded 10 retry_after The broker is shedding load. A full ingress queue or ack path, a full disk under durable storage, a full per-tenant publish queue, or a tenant over its quota. See overloaded reasons. Back off and retry. Sent as outcome_unknown when the batch was already queued before the broker ran out of room to track its ack.
limit_exceeded 11 fatal The request exceeds a configured limit. Too many subscriptions on one connection, or a payload over the size limit. Reduce the request or raise the limit. It will keep failing.
draining 12 retry The broker is shutting down and takes no new work. A new control stream opened while the connection drains. Connect to another broker.
internal 13 outcome_unknown Something failed inside the broker. Anything not covered by another code. Sent as retry for reads and failures before any write, and fatal for configuration the request cannot change. Follow the class sent. Report a persistent one with the broker log.
storage 14 outcome_unknown The storage layer failed. A durable write, a group’s state or a cache log could not be written or read. Sent as retry for reads. Follow the class sent. Check the broker’s disk and the felix_storage_* metrics.
stale_claim 15 retry, shown as fatal by the clients The group has no record of handing this offset out. Nothing was applied, and the record will be delivered again. A group_ack or group_nack after the group’s in-memory state was rebuilt, by eviction while idle or a shard failover. Sent as fatal to a group_extend, or a group_nack naming attempts, whose claim no longer stands. Drop the settlement and poll again. The record comes back.
revoked 16 fatal The credential was revoked: its principal, or the actor of a delegated token. Answer to auth with such a token, or the last frame of an open stream when the revocation reaches the broker. Stop using the token. A token minted after the revocation may be accepted, so a client with a token provider can authenticate again.
stale_width 17 retry The keyed batch was routed by a width the stream does not have on this broker. Nothing was written. The stream was deleted and created again with a different shard count after the client last asked for its width. Ask stream_shards again, then retry. Only a client that sent publish_width gets it.
feature_not_enabled 18 fatal The request needs a fleet feature that has not been finalized. detail.reason names it. Nothing was applied. A head trim on a cluster that has not finalized head_trim (detail.reason head_trim), or a publish with record headers to a single broker run with FELIX_RECORD_HEADERS=false (detail.reason record_headers). A publish with headers forwarded to an owner that predates them, or that has not yet seen the finalize, is refused the same way. For a head trim, finalize the feature (felixctl features finalize head_trim) once every broker is upgraded. It keeps failing until then.
stream_has_commits 19 fatal The stream holds atomic commits, whose state lives in the records a trim would remove. Nothing was written. A head trim on a stream that has taken a commit. Do not trim that stream. Retention still applies to it.

A code this client does not know is kept as an unknown code with its retry class. 0 is never sent as a number.

detail.reason on a shard_unavailable error. It is a string, so a reason added later reaches an older client as text and the error is handled like any other shard_unavailable.

Reason Meaning What to do
not_assigned Placement has not assigned the shard. Retry. If it persists, check that the control plane is placing shards.
owner_unavailable The owner is known but unreachable or not live. Retry. The shard is served again once the owner returns or placement moves it.
not_ready The owner has not finished opening the shard. Also what a client sees when a forwarded write was refused because the owner’s epoch was fenced, since the broker-to-broker answer does not say which. Retry.
stale This broker’s routing view is behind the request’s. Retry. The view catches up from the control plane.
fenced The owner’s epoch was superseded, or its lease lapsed, so it no longer leads the shard. A write refused for this reason was not written. Retry against the new owner.
moving The shard is moving to another broker and the move had not cut over in time. Retry after detail.retry_after_ms.
region_not_routable The leader is in a region the broker has no bridge to, so it will not forward there. Connecting to the leader directly is not refused. Connect to a broker in the leader’s region, or configure the bridge.

detail.reason on an overloaded error, when the broker has one. detail.retry_after_ms suggests how long to wait.

Reason Meaning
publish_queue_full The broker’s publish queue had no room for the publish’s tenant. Nothing was queued.
tenant_quota The publish’s tenant is over its publish quota. Nothing was queued. retry_after_ms says when the tenant is back under.

detail.reason on a refused head trim (trim_before), beside its code. A stale expected offset is not among them: it is answered with publish_refused and offset_mismatch, like publish_if.

Reason Code Meaning What to do
head_trim feature_not_enabled The cluster has not finalized head_trim. Nothing was written. Finalize it once every broker is upgraded.
trim_past_tail invalid_request before is past the expected offset, where the trim record would land. Nothing was written. Trim to at most the expected offset.
not_durable invalid_request The stream is in memory and keeps no log to trim. Retention and the replay ring bound an in-memory stream instead.
trim_timed_out unacknowledged The broker waited 30 seconds for the trim to commit. It may still land. Read the shard: below before is too_old once it landed. A retry with the same expected offset is refused with offset_mismatch if it did.

detail.reason on a feature_not_enabled error that is not a head trim (those are under Head trim reasons).

Reason Code Meaning What to do
record_headers feature_not_enabled A publish carried record headers to a single broker whose FELIX_RECORD_HEADERS switch is off, or was forwarded to a shard owner that predates headers or has not yet seen the finalize. Nothing was written. Publish without headers, or turn the switch back on. Behind a finalize, retry after a moment. A client of a cluster member learns at connect time that headers are unavailable and never sends them.

A conditional or idempotent write that the broker will not append is answered with publish_refused, not an error code, because each reason names a different remedy. It answers a publish_idempotent, a publish_if, a commit that carried expected_offset, or a head trim (trim_before). In the Rust client it arrives as a PublishRefused error carrying the reason. The Python and TypeScript bindings have no idempotent or conditional publish, so they never see it. The C ABI reports a refused conditional write as FELIX_STATUS_OK with written false and the shard’s tail.

Reason Carries Meaning What to do
sequence_gap expected The sequence is past the next one expected. What was skipped is lost to the broker. Stop. Do not carry on as if nothing was skipped.
unknown_producer none The broker holds no sequence for this producer on this shard and the batch was not its first. Start again with a new producer id.
sequence_expired none The sequence is older than the window the broker keeps, so whether it was appended cannot be told. Treat the outcome as unknown and reconcile.
sequence_reused none The broker already holds a batch under this sequence with different payloads. Only sent to a client that offered FEATURE_SEQUENCE_REUSED. The producer reused a number it had spent. Fix the producer.
offset_mismatch tail A publish_if, conditional commit or head trim expected a different next offset. Nothing was written and no offset consumed. Read the shard, then try again from tail if you still own it.
not_leader node_id, addr This broker does not lead the shard, and only the leader holds the sequences. Send the batch to the broker named.

A subscribe that cannot start where it asked is answered with subscribe_cursor_error, also not an error code. It carries the offset requested and the nearest available one. The Rust client returns it as a SubscribeCursorError with both offsets. Python raises CursorError, TypeScript CursorError (kind FELIX_CURSOR) and C and C# report FELIX_STATUS_CURSOR; those three carry the class and not the offsets.

Reason Meaning What to do
too_old Retention discarded the offset, a head trim hid it, or it fell out of an in-memory stream’s replay ring. available is the oldest readable. Below a trim’s floor, a client that offered FEATURE_HEAD_TRIM also gets trim_headers, the headers of the trim that set it. Decide whether to accept the gap and resume from earliest or available.
in_future The offset is past the end of the stream. available is the current tail. Usually a checkpoint written for a different stream. Fix the checkpoint.

All clients pick the exception class from the code, except that an outcome_unknown class always wins. Whatever went wrong, “this may have been written” decides what the caller does next. Rows below show the default class; internal and storage become the outcome-unknown type when the broker sends them that way, as it does for a write.

Code Rust Python TypeScript C and C# status
unauthenticated BrokerError AuthError AuthError FELIX_STATUS_AUTH (Auth)
forbidden BrokerError AuthError AuthError FELIX_STATUS_AUTH (Auth)
not_found BrokerError NotFoundError NotFoundError FELIX_STATUS_NOT_FOUND (NotFound)
invalid_request BrokerError FelixError FelixError FELIX_STATUS_ERROR (Error)
shard_unavailable BrokerError ShardUnavailableError ShardUnavailableError FELIX_STATUS_SHARD_UNAVAILABLE (ShardUnavailable)
not_leader BrokerError, or NotLeaderError from a redirect ShardUnavailableError ShardUnavailableError FELIX_STATUS_SHARD_UNAVAILABLE (ShardUnavailable)
quorum_timeout BrokerError OutcomeUnknownError OutcomeUnknownError FELIX_STATUS_OUTCOME_UNKNOWN (OutcomeUnknown)
leadership_lost BrokerError OutcomeUnknownError OutcomeUnknownError FELIX_STATUS_OUTCOME_UNKNOWN (OutcomeUnknown)
unacknowledged BrokerError OutcomeUnknownError OutcomeUnknownError FELIX_STATUS_OUTCOME_UNKNOWN (OutcomeUnknown)
overloaded BrokerError OverloadedError OverloadedError FELIX_STATUS_OVERLOADED (Overloaded)
limit_exceeded BrokerError FelixError FelixError FELIX_STATUS_ERROR (Error)
draining BrokerError ConnectionError ConnectionError FELIX_STATUS_CONNECTION (Connection)
internal BrokerError OutcomeUnknownError OutcomeUnknownError FELIX_STATUS_OUTCOME_UNKNOWN (OutcomeUnknown)
storage BrokerError OutcomeUnknownError OutcomeUnknownError FELIX_STATUS_OUTCOME_UNKNOWN (OutcomeUnknown)
stale_claim BrokerError FelixError FelixError FELIX_STATUS_ERROR (Error)
revoked BrokerError AuthError AuthError FELIX_STATUS_AUTH (Auth)
stale_width BrokerError FelixError FelixError FELIX_STATUS_ERROR (Error)
feature_not_enabled BrokerError FelixError FelixError FELIX_STATUS_ERROR (Error)
stream_has_commits BrokerError FelixError FelixError FELIX_STATUS_ERROR (Error)

A few errors come from the client rather than the broker and have no code:

Failure Rust Python TypeScript C and C# status
Token source failed, or the broker refused the credential AuthError AuthError AuthError FELIX_STATUS_AUTH
Start offset gone or ahead of the tail SubscribeCursorError CursorError CursorError FELIX_STATUS_CURSOR
Publish written, acknowledgement lost to a dropped connection AckLost OutcomeUnknownError OutcomeUnknownError FELIX_STATUS_OUTCOME_UNKNOWN
Subscription connection dropped SubscriptionLost ConnectionError ConnectionError FELIX_STATUS_CONNECTION
Subscription fell behind and records were dropped SubscriptionLagged SubscriptionLagged Rust client only Rust client only
Idempotent publish refused (a conditional publish returns its refusal as a result) PublishRefused PublishRefusedError with reason PublishRefusedError with reason (FELIX_PUBLISH_REFUSED kind) No typed case: the status is read from the refusal’s message as for an error without a code, so it is FELIX_STATUS_ERROR unless the message names a class (for example “not leader” reads as FELIX_STATUS_CONNECTION). C# surfaces that status in FelixException.Status and has no refusal reason
Commit refused before sending CommitError CommitError, NotOnOwningShardError, EventCountError FELIX_COMMIT, FELIX_NOT_ON_OWNING_SHARD, FELIX_EVENT_COUNT kinds FELIX_STATUS_UNSUPPORTED or FELIX_STATUS_INVALID_ARGUMENT
Head trim refused before sending (the broker did not advertise FEATURE_HEAD_TRIM) TrimError::Unsupported TrimError TrimError (FELIX_TRIM kind) FELIX_STATUS_UNSUPPORTED
Bad argument to the client none ValueError InvalidArgumentError FELIX_STATUS_INVALID_ARGUMENT

Calls return anyhow::Result. Recover a broker refusal with err.downcast_ref::<BrokerError>(), which carries code, retry, detail and message, and reason() for the detail reason. Branch on retry. Client acts on the class itself before an error reaches you: it returns fatal at once, never resends an outcome_unknown publish except through publish_at_least_once or an idempotent producer, and retries retry, redirect and retry_after. See Errors you can act on.

Every exception carries code, retry and detail, all None when the broker sent no code or the failure was local. felix.ConnectionError is Felix’s own class, not the built-in. See Errors you can act on.

Every error carries code, retry, detail and a retryable flag that follows retry when the broker sent one. err.kind is the client’s own name for the class (FELIX_AUTH, FELIX_SHARD_UNAVAILABLE and so on) and is set whether or not the broker sent a code. See Errors you can act on.

The C ABI reports a status per call, not the broker’s code, so several codes share a status. The statuses are in crates/sdk/felix-capi/include/felix.h and listed with their values under Status codes. The C# client’s FelixException.Status is the same value, IsRetryable is true for Connection, ShardUnavailable and Overloaded, and a cancelled call is an OperationCanceledException.

Status Value Raised for
FELIX_STATUS_OK 0 The call succeeded.
FELIX_STATUS_TIMEOUT 1 A bounded wait ended with nothing. The handle is still usable.
FELIX_STATUS_CLOSED 2 The broker ended the subscription, or the producer is closed.
FELIX_STATUS_CANCELLED 3 The call was cancelled. A publish cancelled after it may have been sent reports FELIX_STATUS_OUTCOME_UNKNOWN instead.
FELIX_STATUS_INVALID_ARGUMENT 10 A null pointer, non-UTF-8 string or out-of-range value. Nothing was sent.
FELIX_STATUS_PANIC 11 The library panicked. A bug worth reporting.
FELIX_STATUS_ERROR 20 A failure with no more specific class: invalid_request, limit_exceeded, stale_claim, stale_width and unknown codes.
FELIX_STATUS_CONNECTION 21 draining, an unreachable broker, or a lost connection.
FELIX_STATUS_AUTH 22 unauthenticated, forbidden and revoked.
FELIX_STATUS_NOT_FOUND 23 not_found.
FELIX_STATUS_CURSOR 24 A cursor error.
FELIX_STATUS_SHARD_UNAVAILABLE 25 shard_unavailable and not_leader.
FELIX_STATUS_OVERLOADED 26 overloaded.
FELIX_STATUS_OUTCOME_UNKNOWN 27 Any code sent with the outcome_unknown class.
FELIX_STATUS_UNSUPPORTED 28 The broker does not advertise atomic commits.