Skip to content

Rust: Subscribing

Part of the Rust client guide. Setup and configuration are on that page.

let mut subscription = client.subscribe("acme", "prod", "events").await?;
while let Some(event) = subscription.next_event().await? {
process_event(event).await?;
}

None means the broker ended the stream. A lost connection is an error, SubscriptionLost, so the loop above stops with ? rather than exiting as if the stream had finished. A Client subscription resubscribes by itself from the offset after the last one it delivered; on an in-memory stream there is no offset to resume from, so it returns the error. next_event is cancel-safe, so wrapping it in tokio::time::timeout is fine: a resubscribe in progress keeps running in the background and the next call picks it up.

use bytes::Bytes;
use felix_client::RecordHeaders;
use std::sync::Arc;
pub struct Event {
pub tenant_id: Arc<str>,
pub namespace: Arc<str>,
pub stream: Arc<str>,
pub payload: Bytes,
/// The log offset on a durable stream. `None` on an in-memory one, and
/// against a broker that did not negotiate offsets.
pub offset: Option<u64>,
/// Offsets just before `offset` that hold no event. Non-zero only on the
/// first event after a leader change.
pub skipped_before: u64,
/// Who published it, when `ClientConfig::publishers` asked and the broker
/// recorded one.
pub publisher: Option<Arc<str>>,
/// When the broker appended it, in microseconds since the Unix epoch,
/// when `ClientConfig::timestamps` asked. `None` on an in-memory stream.
pub timestamp_micros: Option<u64>,
/// The key and headers it was published with. `None` for a record that
/// had neither, and against a broker that did not accept record headers.
pub headers: Option<RecordHeaders>,
}

Set publishers: true in the ClientConfig and each event says who published it: event.publisher is the subject of the token the broker accepted the write from. Group records carry the same value as publisher. Off by default, since it adds the principal to every batch.

let mut config = ClientConfig::from_env_or_yaml(quic_client_config(None, false)?, None)?;
config.publishers = true;
let client = Arc::new(Client::connect(&seeds, "localhost", config).await?);
let mut inputs = client.subscribe("arena", "match-7", "inputs").await?;
while let Some(event) = inputs.next_event().await? {
let Some(player) = event.publisher.as_deref() else {
continue; // written before the broker recorded publishers
};
apply_input(player, &event.payload);
}

That is the broker vouching that the write came from that principal; it says nothing about the payload’s contents. An in-memory stream always reports it. A durable stream reports it only for records written once the broker stores publishers (the publisher_principal fleet feature, or FELIX_RECORD_PUBLISHERS=true on a single broker), and then on replay and to consumer groups as well.

Sizing the broker’s queue for a subscription

Section titled “Sizing the broker’s queue for a subscription”

Each subscription has a bounded queue on the broker. Set broker_sub_queue_capacity in the ClientConfig to ask for a different size than the stream’s default, counted in published batches. The broker clamps it to its FELIX_SUBSCRIBER_QUEUE_CAPACITY_MAX and the subscription reports what it got:

config.broker_sub_queue_capacity = Some(4096);
let client = Arc::new(Client::connect(&seeds, "localhost", config).await?);
let subscription = client.subscribe("acme", "prod", "webhooks").await?;
// `None` from a broker that predates the option: the stream's default applies.
let granted = subscription.queue_capacity();
tracing::info!(?granted, "subscription queue");

It applies to every subscription the client opens, including the ones a Client reopens after a move or a lag. A larger queue makes drops rarer; it does not change what happens on a drop, which the stream’s policy decides (see When a subscription falls behind).

When an event was written, and replaying from a time

Section titled “When an event was written, and replaying from a time”

Set timestamps: true in the ClientConfig and each event from a durable stream carries event.timestamp_micros: when the broker appended it, in microseconds since the Unix epoch. Group records carry the same value as timestamp_micros. Off by default, since it adds eight bytes to every event.

To replay from a time, look up the offset and subscribe there. offset_for_time works whether or not timestamps is set:

let start = client
.offset_for_time("acme", "prod", "webhooks", 0, since_micros)
.await?;
let position = match start {
Some(offset) => StartPosition::Offset(offset),
// Nothing that recent yet.
None => StartPosition::Latest,
};
let replay = client
.subscribe_from("acme", "prod", "webhooks", Some(position))
.await?;

The answer is the first record on that shard appended at or after the time, and the oldest record when the time is older than all of them. Times come from the leading broker’s clock, so the search is exact while that clock only moves forward, and close otherwise. client.supports_offset_for_time() says whether the broker answers it. Client::offset_for_time asks whichever broker leads the shard, following redirects. The Python and TypeScript bindings wrap it.

subscribe_from keeps going into live delivery. To load a fixed slice, such as a match’s events up to a snapshot offset, read it page by page instead. No subscription is opened:

let mut from = start;
while from < end {
let page = client
.read("acme", "prod", "matches", 0, from, Some(end), 500)
.await?;
if page.records.is_empty() {
break; // nothing committed past here yet
}
for record in &page.records {
apply(record.offset, &record.payload);
}
from = page.next_offset;
}

Continue from next_offset rather than the last offset plus one, since offsets that hold no record are passed over. A page holds at most max_records (0 for the broker’s cap) and is capped in size by the broker. A start below what retention kept, or past the tail, fails with SubscribeCursorError. Client::read does the same against whichever broker leads the shard, and client.supports_read() says whether the broker answers it.

Subscriber queues shed under the default policy rather than blocking the publisher, so a subscriber can silently miss records. A jump between consecutive offsets is a drop, with one exception the broker tells you about: a leader that takes or reopens the shard writes a generation-start record that takes an offset and is never delivered, and the event after it says so in skipped_before.

let mut expected: Option<u64> = None;
while let Some(event) = subscription.next_event().await? {
if let (Some(want), Some(got)) = (expected, event.offset) {
// The first offset this event accounts for, past any that hold no event.
let first = got - event.skipped_before;
if first > want {
tracing::warn!(dropped = first - want, "subscriber queue overflowed");
}
}
expected = event.offset.map(|offset| offset + 1);
handle(&event.payload);
}

Worth writing even if you never resume from offsets. It is the only signal the queue overflowed. A broker older than the skip count reports none, so against one a leader change reads as a drop of one.

Checkpoint what you handled and resume at the next one. start is the first record you have not seen, so a resuming consumer passes offset + 1.

use felix_client::{StartPosition, SubscribeCursorError};
let mut start = match checkpoint.load().await? {
Some(offset) => StartPosition::Offset(offset + 1),
None => StartPosition::Earliest,
};
loop {
let mut subscription = client
.subscribe_from("acme", "prod", "events", Some(start))
.await?;
loop {
match subscription.next_event().await {
Ok(Some(event)) => {
handle(&event.payload).await?;
if let Some(offset) = event.offset {
checkpoint.save(offset).await?;
start = StartPosition::Offset(offset + 1);
}
}
// Nothing left to read.
Ok(None) if subscription.stream_deleted() => return Ok(()),
// Ended for another reason: resubscribe from `start`.
Ok(None) => break,
Err(err) => {
if let Some(cursor) = err.downcast_ref::<SubscribeCursorError>() {
// Retention discarded it. `available` is the nearest offset
// that would have worked, so this takes the smallest gap
// rather than restarting at the beginning, and says so,
// because a silent restart at the tail loses records with
// nothing reported.
tracing::error!(
requested = cursor.requested,
resuming_at = cursor.available,
"checkpoint is past retention",
);
start = StartPosition::Offset(cursor.available);
break;
}
return Err(err);
}
}
}
}

next_event is cancel-safe: it awaits an mpsc receive, so racing it in a tokio::select! consumes nothing when another branch wins. You can put a timeout around it without losing a record.

On a durable stream, a subscribe with a start position tells you where it joined:

let subscription = client
.subscribe_from("acme", "prod", "orders", Some(StartPosition::Offset(1_000)))
.await?;
let first = subscription.start_offset(); // Some(1000)
let live = subscription.live_offset(); // the tail when you subscribed

Events below live_offset() are catch-up; events from it on are new, and none are skipped between the two. A reader that has received an event at live_offset() - 1, or asked to start at live_offset(), has caught up: the offset leaves out any generation-start records at the end of the log, which never arrive as events. From Latest the two offsets are equal and both stop short of those records, so reading history up to live_offset() - 1 after a Latest subscribe ends on an event that exists. Both are None for a plain tail subscribe, an in-memory stream, or an older broker.

A rebalance or a drain can move a stream’s shard to another broker. The old owner delivers everything it committed, then ends the subscription with a shard_moved frame saying where the shard went and where to resume.

Client::subscribe and subscribe_from (shard 0), and subscribe_shard (a shard you name), return a ClusterSubscription that follows the shard on its own: next_event resubscribes on the new owner and carries on. On a durable stream it resumes at max(last delivered offset + 1, resume_from), so nothing is repeated or skipped; an in-memory stream resumes at the new owner’s tail. moves() counts how often it followed, and client() is the connection it is on now.

A Connection subscription ends instead, and says why:

while let Some(event) = subscription.next_event().await? {
handle(&event);
}
if let Some(moved) = subscription.shard_moved() {
// Resubscribe at the larger of `moved.resume_from` and your last offset
// + 1, on `moved.addr` if it is set, or via any broker, which redirects.
} else if subscription.stream_deleted() {
// The stream was deleted: there is nothing to resubscribe to.
}

A ClusterSubscription does not follow a deleted stream: next_event returns None and stream_deleted() says why.

See Multi-node client.

Under DropNew or DropOld, a subscriber that cannot keep up loses records, in the broker’s queue for it or in the client’s own. On a durable stream the subscription then ends: next_event returns the events queued before the drop and then a SubscriptionLagged error whose resume_from is where the broker’s queue first dropped. It comes as soon as the drop happens, not with the next publish. Resubscribe after the last event you received, not at resume_from: the broker can also drop a frame on its way out to a slow subscriber without reporting it, and that frame can sit below resume_from.

use felix_client::{StartPosition, SubscriptionLagged};
let mut last = None;
let resume_at = loop {
match subscription.next_event().await {
Ok(Some(event)) => {
last = event.offset;
handle(&event);
}
Ok(None) => return Ok(()),
Err(err) => match err.downcast_ref::<SubscriptionLagged>() {
// Subscribe again after the last record; the log still has the rest.
Some(lagged) => break last.map_or(lagged.resume_from, |last| last + 1),
None => return Err(err),
},
}
};
let subscription = client
.subscribe_from("acme", "prod", "events", Some(StartPosition::Offset(resume_at)))
.await?;

ClusterSubscription does this itself, so a slow reader catches up from the log instead of losing records. An in-memory stream has no offsets, so it keeps dropping as before; felix_client_sub_queue_dropped_total and felix_client_sub_dispatch_dropped_total count those drops.

Handle multiple streams concurrently:

let mut orders = client.subscribe("acme", "prod", "orders").await?;
let mut inventory = client.subscribe("acme", "prod", "inventory").await?;
let mut logs = client.subscribe("acme", "staging", "logs").await?;
// `next_event` is cancel-safe, so the branches that lose a round lose nothing.
// Stops when any of the three ends.
loop {
tokio::select! {
event = orders.next_event() => match event? {
Some(event) => handle_order(event).await?,
None => break,
},
event = inventory.next_event() => match event? {
Some(event) => handle_inventory(event).await?,
None => break,
},
event = logs.next_event() => match event? {
Some(event) => handle_log(event).await?,
None => break,
},
}
}

Slow work inside the read loop holds up the next event, and a reader that falls behind loses records (see When a subscription falls behind). Hand the work to a task over a bounded channel instead. A task per event would also keep up, but with no limit on how many run at once or on the memory they hold.

use tokio::sync::mpsc;
let (events, mut queued) = mpsc::channel(100);
let worker = tokio::spawn(async move {
while let Some(event) = queued.recv().await {
expensive_processing(event).await?;
}
anyhow::Ok(())
});
while let Some(event) = subscription.next_event().await? {
if events.send(event).await.is_err() {
break; // the worker stopped; its result below says why
}
}
// Closing the channel lets the worker drain what is queued and finish.
drop(events);
worker.await??;

Once the channel is full, send waits, and the reader slows to the worker’s pace like any slow reader.

let mut subscription = client.subscribe("acme", "prod", "events").await?;
// Read up to 100 events, then stop.
for _ in 0..100 {
let Some(event) = subscription.next_event().await? else {
break;
};
process(event);
}
// Dropping the subscription closes it on the broker.
drop(subscription);