Skip to content

Your first stream

You will write a small program that publishes records to a durable stream and a second mode of the same program that reads them back. The reader remembers the last offset it handled in a file, so after you stop it and start it again it carries on with the first record it has not seen.

A running broker and the environment variables from the tutorials overview. That page starts the development stack from the quickstart and writes the broker address, a token and the broker’s certificate to an env file you source in each terminal. You also need the client for your language, set up as the overview’s “Set up once” tabs describe. The steps show Rust, Python, TypeScript and C#. The C client is callback based, so its version is one program at the end.

A stream has to exist before anything can publish to it. Create one with felixctl:

Terminal window
docker compose run --rm felixctl stream create tutorial-events

With an installed felixctl, drop the docker compose run --rm felixctl or podman compose run --rm felixctl prefix. A new stream is durable: its records are written to disk before the broker acknowledges them, and each one gets an offset, its position in the stream’s log. Offsets only grow, and they are what lets a reader say where it stopped.

The program takes one argument, publish or read. This part is shared by both. The broker address, token and certificate come from the environment, so none of them is written into the code, and every call to the broker has a time limit (see Time limits).

use std::future::Future;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use felix_client::{ClientConfig, Client, quic_client_config, root_store_from_pem_file};
// Must match FELIX_AUTH_TENANT.
const TENANT: &str = "dev";
const NAMESPACE: &str = "default";
const STREAM: &str = "tutorial-events";
// Every call to the broker gets this long.
const CALL_TIMEOUT: Duration = Duration::from_secs(10);
async fn bounded<T>(what: &str, limit: Duration, call: impl Future<Output = Result<T>>) -> Result<T> {
tokio::time::timeout(limit, call)
.await
.with_context(|| format!("{what} did not finish within {limit:?}"))?
}
async fn connect() -> Result<Arc<Client>> {
let ca_file = std::env::var("FELIX_CA_FILE").context("set FELIX_CA_FILE")?;
let roots = Arc::new(root_store_from_pem_file(&ca_file)?);
// Reads FELIX_AUTH_TENANT and FELIX_AUTH_TOKEN.
let config = ClientConfig::from_env_or_yaml(quic_client_config(Some(roots), false)?, None)?;
let broker = std::env::var("FELIX_BROKERS").unwrap_or_else(|_| "127.0.0.1:5000".into());
let seeds: Vec<SocketAddr> = vec![broker.parse().context("FELIX_BROKERS must be ip:port")?];
let connecting = Client::connect(&seeds, "localhost", config);
let client = bounded("connecting", CALL_TIMEOUT, connecting).await?;
Ok(Arc::new(client))
}

The broker makes its own certificate, which is why a CA file is passed. A broker with a certificate from a public CA needs none.

Each publish waits for the broker’s acknowledgement, which on a durable stream comes after the record is on disk. It returns the offset the record was written at.

use felix_client::AckMode;
async fn publish_three(client: &Client) -> Result<()> {
for i in 1..=3 {
let payload = format!("event {i}").into_bytes();
let publish = client.publish(TENANT, NAMESPACE, STREAM, payload, AckMode::PerMessage);
let offset = bounded("publish", CALL_TIMEOUT, publish)
.await?
.context("a durable stream returns an offset")?;
println!("published event {i} at offset {offset}");
}
Ok(())
}

A subscription starts at the tail of the stream unless you say otherwise. The reader asks for a start position instead: the beginning the first time, and the offset after the last one it handled on every run after that. A start offset is the first record you have not seen, so a resuming reader passes the last offset plus one.

The reader saves the offset after it has handled the record. If it dies between handling and saving, the record comes again on the next run, so whatever a handler does should be safe to repeat. The reader also stops after three idle seconds so the tutorial ends on its own; a service would keep reading.

use std::path::Path;
use felix_client::StartPosition;
use tokio::io::AsyncWriteExt;
async fn load_checkpoint(path: &Path) -> Result<Option<u64>> {
match tokio::fs::read_to_string(path).await {
Ok(text) => Ok(Some(text.trim().parse().context("the checkpoint is not an offset")?)),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(err) => Err(err).context("read the checkpoint"),
}
}
// Sync a temporary file and rename it over the checkpoint, so a crash leaves
// the old checkpoint or the new one, never half of one.
async fn save_checkpoint(path: &Path, offset: u64) -> Result<()> {
let tmp = path.with_extension("tmp");
let mut file = tokio::fs::File::create(&tmp).await.context("create the checkpoint")?;
file.write_all(offset.to_string().as_bytes()).await.context("write the checkpoint")?;
file.sync_all().await.context("sync the checkpoint")?;
drop(file);
tokio::fs::rename(&tmp, path).await.context("replace the checkpoint")
}
async fn read_new(client: &Arc<Client>, checkpoint: &Path) -> Result<()> {
let start = match load_checkpoint(checkpoint).await? {
Some(offset) => StartPosition::Offset(offset + 1),
None => StartPosition::Earliest,
};
let subscribe = client.subscribe_from(TENANT, NAMESPACE, STREAM, Some(start));
let mut events = bounded("subscribe", CALL_TIMEOUT, subscribe).await?;
loop {
match tokio::time::timeout(Duration::from_secs(3), events.next_event()).await {
Err(_) => {
println!("nothing new for 3 seconds, stopping");
return Ok(());
}
Ok(Err(err)) => return Err(err),
Ok(Ok(None)) => {
println!("the subscription ended");
return Ok(());
}
Ok(Ok(Some(event))) => {
let offset = event.offset.context("the stream has no offsets; is it durable?")?;
println!("offset {offset}: {}", String::from_utf8_lossy(&event.payload));
save_checkpoint(checkpoint, offset).await?;
}
}
}
}

The last part picks the mode from the command line.

use anyhow::bail;
#[tokio::main]
async fn main() -> Result<()> {
let client = connect().await?;
match std::env::args().nth(1).as_deref() {
Some("publish") => publish_three(&client).await,
Some("read") => read_new(&client, Path::new("checkpoint")).await,
_ => bail!("usage: first_stream publish|read"),
}
}

The program takes one argument, publish or read. Run each in a terminal where you have run source /path/to/felix-tutorial.env from the overview. The reader writes its checkpoint file to the directory you run it from, so run both from the same one.

From the project directory:

Terminal window
# publish three records
cargo run --bin first_stream -- publish
# read them back
cargo run --bin first_stream -- read

After publish, the program prints:

published event 1 at offset 1
published event 2 at offset 2
published event 3 at offset 3

Your offsets may start somewhere else. A new stream’s first record is often not at offset 0, because a leader writes a marker of its own first. After read, it prints:

offset 1: event 1
offset 2: event 2
offset 3: event 3
nothing new for 3 seconds, stopping

Run read again and it prints only the idle line, because checkpoint holds 3 and the reader asked to start at 4. Run publish and then read again, and only the new records appear:

offset 4: event 1
offset 5: event 2
offset 6: event 3
nothing new for 3 seconds, stopping

The stream still holds all six. Delete the checkpoint file and the reader starts from the beginning and prints every one of them. That is the difference from a subscription that starts at the tail: a durable stream is a log you can read from any retained offset, as many times as you like, until its retention bound removes old records.

If read fails with a cursor error, the offset in your checkpoint is older than the stream’s retention. See A consumer that survives a restart for how to handle that in a long-running reader.

Each C call returns at once and reports to a callback, so the program below wraps each one in a small waiter that blocks until the callback has run or ten seconds have passed. It is the same two modes as above. See How calls work for the model.

#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include "felix.h"
#define TENANT "dev"
#define NAMESPACE "default"
#define STREAM "tutorial-events"
#define CHECKPOINT "checkpoint"
struct waiter {
pthread_mutex_t mu;
pthread_cond_t cv;
bool done;
FelixStatus status;
void *handle;
bool has_offset;
uint64_t offset;
char message[256];
};
static void finish(struct waiter *w, FelixStatus status, const char *message, void *handle) {
pthread_mutex_lock(&w->mu);
w->done = true;
w->status = status;
w->handle = handle;
snprintf(w->message, sizeof w->message, "%s", message ? message : "");
pthread_cond_signal(&w->cv);
pthread_mutex_unlock(&w->mu);
}
/* A refused call never runs its callback, so its error comes from the thread.
A call the broker does not answer within limit_ms ends the program: its
callback may still run later, so nothing it uses can be freed. */
static FelixStatus wait_for_ms(struct waiter *w, FelixStatus started, long limit_ms) {
if (started != FELIX_STATUS_OK) {
snprintf(w->message, sizeof w->message, "%s", felix_last_error_message());
return started;
}
struct timespec deadline;
clock_gettime(CLOCK_REALTIME, &deadline);
deadline.tv_sec += limit_ms / 1000;
deadline.tv_nsec += (limit_ms % 1000) * 1000000L;
if (deadline.tv_nsec >= 1000000000L) {
deadline.tv_sec++;
deadline.tv_nsec -= 1000000000L;
}
pthread_mutex_lock(&w->mu);
while (!w->done) {
if (pthread_cond_timedwait(&w->cv, &w->mu, &deadline) == ETIMEDOUT) {
fprintf(stderr, "no answer from the broker within %ld ms\n", limit_ms);
exit(1);
}
}
w->done = false;
FelixStatus status = w->status;
pthread_mutex_unlock(&w->mu);
return status;
}
/* Every call to the broker gets this long. */
#define CALL_LIMIT_MS 10000
#define wait_for(w, started) wait_for_ms((w), (started), CALL_LIMIT_MS)
static void on_connect(void *user_data, FelixStatus status, const char *message,
FelixClient *client) {
finish(user_data, status, message, client);
}
static void on_publish(void *user_data, FelixStatus status, const char *message,
bool has_offset, uint64_t offset) {
struct waiter *w = user_data;
w->has_offset = has_offset;
w->offset = offset;
finish(w, status, message, NULL);
}
static void on_subscribe(void *user_data, FelixStatus status, const char *message,
FelixSubscription *subscription) {
finish(user_data, status, message, subscription);
}
static void on_event(void *user_data, FelixStatus status, const char *message,
FelixEvent *event) {
finish(user_data, status, message, event);
}
static int publish_three(const FelixClient *client, struct waiter *w) {
for (int i = 1; i <= 3; i++) {
char payload[32];
int len = snprintf(payload, sizeof payload, "event %d", i);
FelixStatus status = wait_for(
w, felix_client_publish(client, TENANT, NAMESPACE, STREAM, (const uint8_t *)payload,
(size_t)len, NULL, 0, FELIX_ACK_PER_MESSAGE,
FELIX_DELIVERY_AT_MOST_ONCE, NULL, on_publish, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "publish: %s\n", w->message);
return 1;
}
if (w->has_offset) {
printf("published event %d at offset %llu\n", i, (unsigned long long)w->offset);
} else {
printf("published event %d\n", i);
}
}
return 0;
}
/* The offset in the checkpoint file, or false when there is none yet. */
static bool load_checkpoint(uint64_t *offset) {
FILE *f = fopen(CHECKPOINT, "r");
if (!f) return false;
unsigned long long value = 0;
bool ok = fscanf(f, "%llu", &value) == 1;
fclose(f);
*offset = value;
return ok;
}
/* Synced, then renamed over the checkpoint, so a crash leaves the old one or
the new one, never half of one. */
static bool save_checkpoint(uint64_t offset) {
FILE *f = fopen(CHECKPOINT ".tmp", "w");
if (!f) return false;
bool ok = fprintf(f, "%llu\n", (unsigned long long)offset) > 0;
ok = ok && fflush(f) == 0 && fsync(fileno(f)) == 0;
ok = (fclose(f) == 0) && ok;
return ok && rename(CHECKPOINT ".tmp", CHECKPOINT) == 0;
}
static int read_new(const FelixClient *client, struct waiter *w) {
uint64_t saved = 0;
bool resume = load_checkpoint(&saved);
FelixSubscription *sub = NULL;
int exit_code = 1;
FelixStatus status = wait_for(
w, felix_client_subscribe(client, TENANT, NAMESPACE, STREAM, 0,
resume ? FELIX_START_OFFSET : FELIX_START_EARLIEST,
resume ? saved + 1 : 0, NULL, on_subscribe, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "subscribe: %s\n", w->message);
return 1;
}
sub = w->handle;
for (;;) {
status = wait_for_ms(w, felix_subscription_next(sub, 3000, NULL, on_event, w),
3000 + CALL_LIMIT_MS);
if (status == FELIX_STATUS_TIMEOUT) {
printf("nothing new for 3 seconds, stopping\n");
exit_code = 0;
break;
}
if (status == FELIX_STATUS_CLOSED) {
printf("the subscription ended\n");
exit_code = 0;
break;
}
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "read: %s\n", w->message);
break;
}
FelixEvent *event = w->handle;
uint64_t offset = 0;
bool has_offset = false;
const uint8_t *data = NULL;
size_t len = 0;
if (felix_event_offset(event, &offset, &has_offset) != FELIX_STATUS_OK ||
felix_event_payload(event, &data, &len) != FELIX_STATUS_OK || !has_offset) {
fprintf(stderr, "the event has no offset or payload; is the stream durable?\n");
felix_event_free(event);
break;
}
printf("offset %llu: %.*s\n", (unsigned long long)offset, (int)len, (const char *)data);
felix_event_free(event);
if (!save_checkpoint(offset)) {
fprintf(stderr, "could not save the checkpoint\n");
break;
}
}
felix_subscription_free(sub);
return exit_code;
}
int main(int argc, char **argv) {
const char *token = getenv("FELIX_AUTH_TOKEN");
const char *ca_file = getenv("FELIX_CA_FILE");
const char *broker = getenv("FELIX_BROKERS");
if (argc < 2 || (strcmp(argv[1], "publish") != 0 && strcmp(argv[1], "read") != 0)) {
fprintf(stderr, "usage: first-stream publish|read\n");
return 1;
}
if (!token || !*token || !ca_file || !*ca_file) {
fprintf(stderr, "set FELIX_AUTH_TOKEN and FELIX_CA_FILE\n");
return 1;
}
struct waiter w = {.done = false};
pthread_mutex_init(&w.mu, NULL);
pthread_cond_init(&w.cv, NULL);
int exit_code = 1;
FelixStatus status = wait_for(
&w, felix_client_connect(broker && *broker ? broker : "127.0.0.1:5000", TENANT,
token, "localhost", ca_file, NULL, on_connect, &w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "connect: %s\n", w.message);
} else {
FelixClient *client = w.handle;
exit_code = strcmp(argv[1], "publish") == 0 ? publish_three(client, &w)
: read_new(client, &w);
/* Every call has completed, so no callback still holds &w. */
felix_client_free(client);
}
pthread_cond_destroy(&w.cv);
pthread_mutex_destroy(&w.mu);
return exit_code;
}

Save it as first-stream.c in the unpacked release directory from the overview, and compile and link it there:

Terminal window
cc -std=c11 -pthread -I include first-stream.c -L lib -lfelix \
-Wl,-rpath,"$PWD/lib" -o first-stream

This links lib/libfelix.dylib on macOS. To build from source instead, see the C client page. Run it with the commands in the C tab of Run it.

A work queue reads a stream as jobs, so each record goes to one worker and a failed one comes back. The Rust client page covers subscribing in more depth, including ranges and what to do when a subscription falls behind, and Durable storage explains how the log, offsets and retention fit together.