Skip to content

Build an app

You will build a small order pipeline from the three pieces the earlier tutorials introduced. A submit command stands in for your API: it records an order’s status in a cache and publishes the order to a stream. Workers in a consumer group take orders off the stream, fulfil them and write the result to the cache. A dashboard watches the cache and prints every status change.

flowchart LR
    A["submit"] -->|"order:id = queued"| C[("cache")]
    A -->|"the order"| S[("stream")]
    S -->|"consumer group"| W["worker"]
    W -->|"processing, then done"| C
    C -->|"watch order:*"| D["dashboard"]

One program has three modes (submit, work and dashboard) so the whole thing stays in one file. In a real system they would be separate services, and submit would sit behind an HTTP handler.

A running broker, the env file and the client for your language, all set up as the tutorials overview describes. The earlier tutorials introduce each call used here, so Your first stream, A work queue and Cache, leases and watches are worth doing first. This tutorial has Rust, Python, TypeScript and C# tabs. The C calls are the same as in the earlier tutorials.

Terminal window
docker compose run --rm felixctl stream create tutorial-orders
docker compose run --rm felixctl cache create tutorial-cache

Use felixctl directly if you installed it. If you did the previous tutorial, the cache exists and the command says so.

An order is a small JSON document. Submitting one does two writes, and their order matters. The status goes into the cache first and the order is published second. Done the other way round, a quick worker could write done and then have the API overwrite it with queued. Status keys expire after an hour so the cache does not collect old orders.

use std::future::Future;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use bytes::Bytes;
use felix_client::{AckMode, ClientConfig, Client, quic_client_config, root_store_from_pem_file};
// Must match FELIX_AUTH_TENANT.
const TENANT: &str = "dev";
const NAMESPACE: &str = "default";
const STREAM: &str = "tutorial-orders";
const GROUP: &str = "fulfilment";
const CACHE: &str = "tutorial-cache";
const STATUS_TTL_MS: u64 = 60 * 60 * 1000;
// Every call to the broker gets this long. A poll waits on purpose, so it gets
// its wait on top.
const CALL_TIMEOUT: Duration = Duration::from_secs(10);
const POLL_WAIT: Duration = Duration::from_secs(5);
async fn bounded<T>(what: &str, limit: Duration, call: impl Future<Output = Result<T>>) -> Result<T> {
tokio::time::timeout(limit, call)
.await
.with_context(|| format!("{what} did not finish within {limit:?}"))?
}
async fn connect() -> Result<Arc<Client>> {
let ca_file = std::env::var("FELIX_CA_FILE").context("set FELIX_CA_FILE")?;
let roots = Arc::new(root_store_from_pem_file(&ca_file)?);
// Reads FELIX_AUTH_TENANT and FELIX_AUTH_TOKEN.
let config = ClientConfig::from_env_or_yaml(quic_client_config(Some(roots), false)?, None)?;
let broker = std::env::var("FELIX_BROKERS").unwrap_or_else(|_| "127.0.0.1:5000".into());
let seeds: Vec<SocketAddr> = vec![broker.parse().context("FELIX_BROKERS must be ip:port")?];
let connecting = Client::connect(&seeds, "localhost", config);
let client = bounded("connecting", CALL_TIMEOUT, connecting).await?;
Ok(Arc::new(client))
}
async fn set_status(client: &Client, id: &str, status: &str) -> Result<()> {
let value = Bytes::copy_from_slice(status.as_bytes());
let key = format!("order:{id}");
let put = client.cache_put(TENANT, NAMESPACE, CACHE, &key, value, Some(STATUS_TTL_MS));
bounded("writing the status", CALL_TIMEOUT, put).await
}
async fn submit(client: &Client, count: u32) -> Result<()> {
for n in 1..=count {
let id = format!("o-{n}");
let order = serde_json::to_vec(&serde_json::json!({ "id": id, "qty": n }))?;
// The status goes first, so a fast worker's result is never overwritten.
set_status(client, &id, "queued").await?;
let publish = client.publish(TENANT, NAMESPACE, STREAM, order, AckMode::PerMessage);
bounded("publish", CALL_TIMEOUT, publish).await?;
println!("accepted {id}");
}
Ok(())
}

Workers poll the fulfilment group one order at a time. For each order the worker reads the id and quantity, marks the order processing, does the work (here a two second pause and a price of 5 per unit) and writes done. The acknowledgement comes last, after the result is in the cache. If a worker dies before acknowledging, the order comes back and the same result is written again, which is why writing the result is safe to repeat.

The worker separates two kinds of failure. An order that does not parse, or whose quantity is outside 1 to 1,000,000, will never succeed, so retrying it only delays the orders behind it. The worker acknowledges it and moves on. Anything else, such as a cache write that failed, may work on the next attempt, so the worker nacks the record and the broker hands it out again. The Rust and C# workers nack with a one second delay, so a failing order is not retried at full speed; the Python and TypeScript bindings nack without one. A record that keeps failing ends up as a dead letter, as in A work queue.

Every call to the broker has a time limit. See Time limits.

use anyhow::bail;
const PRICE: u64 = 5;
const MAX_QTY: u64 = 1_000_000;
async fn fulfil(client: &Client, id: &str, total: u64) -> Result<()> {
set_status(client, id, "processing").await?;
tokio::time::sleep(Duration::from_secs(2)).await;
set_status(client, id, &format!("done total={total}")).await
}
// The order's id and its total, or None for an order that can never be filled.
fn parse_order(payload: &[u8]) -> Option<(String, u64)> {
let order: serde_json::Value = serde_json::from_slice(payload).ok()?;
let qty = order.get("qty")?.as_u64()?;
if qty == 0 || qty > MAX_QTY {
return None;
}
Some((order.get("id")?.as_str()?.to_owned(), qty.checked_mul(PRICE)?))
}
async fn work(client: &Client) -> Result<()> {
loop {
let poll = client.group_poll_wait(TENANT, NAMESPACE, STREAM, 0, GROUP, 1, POLL_WAIT);
let records = bounded("poll", CALL_TIMEOUT + POLL_WAIT, poll).await?;
for record in records {
let Some((id, total)) = parse_order(&record.payload) else {
// It will never parse, so settle it rather than retry it.
println!("rejected a malformed order at offset {}", record.offset);
let ack = client.group_ack(TENANT, NAMESPACE, STREAM, 0, GROUP, record.offset);
bounded("ack", CALL_TIMEOUT, ack).await?;
continue;
};
match fulfil(client, &id, total).await {
Ok(()) => {
let ack = client.group_ack(TENANT, NAMESPACE, STREAM, 0, GROUP, record.offset);
bounded("ack", CALL_TIMEOUT, ack).await?;
println!("fulfilled {id} (attempt {})", record.attempts);
}
Err(err) => {
println!("{id} failed on attempt {}: {err:#}", record.attempts);
let nack = client.group_nack_after(
TENANT, NAMESPACE, STREAM, 0, GROUP, &record, Duration::from_secs(1),
);
bounded("nack", CALL_TIMEOUT, nack).await?;
}
}
}
}
}

The dashboard is a retained prefix watch on order:. A retained watch first delivers the current value of every matching key and then the live changes, so a dashboard started in the middle of the day shows the state of every order that is still in the cache instead of only what changes afterwards. If the watch falls behind, the broker ends it and says where to watch again from.

use felix_client::{CacheWatchFilter, CacheWatchItem};
async fn dashboard(client: &Arc<Client>) -> Result<()> {
let start = client.watch_cache_retained(TENANT, NAMESPACE, CACHE, CacheWatchFilter::Prefix("order:".into()));
let mut watch = bounded("starting the watch", CALL_TIMEOUT, start).await?;
while let Some(item) = watch.recv().await {
match item {
CacheWatchItem::Change(change) => {
let status = change.value.as_deref().map(String::from_utf8_lossy);
println!("{} = {}", change.key, status.as_deref().unwrap_or("(expired)"));
}
CacheWatchItem::Lagged { resume_from } => {
bail!("the dashboard fell behind; watch again from offset {resume_from}")
}
CacheWatchItem::ShardMoved(_) => println!("the shard moved; the watch follows it"),
}
}
Ok(())
}
#[tokio::main]
async fn main() -> Result<()> {
let client = connect().await?;
let args: Vec<String> = std::env::args().skip(1).collect();
match args.first().map(String::as_str) {
Some("submit") => {
let count = args.get(1).map_or(Ok(3), |n| n.parse()).context("count must be a number")?;
submit(&client, count).await
}
Some("work") => work(&client).await,
Some("dashboard") => dashboard(&client).await,
_ => bail!("usage: build_an_app submit [count] | work | dashboard"),
}
}

Use four terminals. Run source /path/to/felix-tutorial.env from the overview in each terminal first. submit takes an optional order count, 3 by default.

From the project directory:

Terminal window
# terminal 1: the dashboard
cargo run --bin build_an_app -- dashboard
# terminal 2: first worker
cargo run --bin build_an_app -- work
# terminal 3: second worker
cargo run --bin build_an_app -- work
# terminal 4: submit four orders
cargo run --bin build_an_app -- submit 4

With dashboard running in one terminal and work in two others, submit 4 in a fourth prints:

accepted o-1
accepted o-2
accepted o-3
accepted o-4

The dashboard follows every order from queued to done. The lines for different orders interleave, and their order varies from run to run:

order:o-1 = queued
order:o-2 = queued
order:o-3 = queued
order:o-4 = queued
order:o-1 = processing
order:o-2 = processing
order:o-1 = done total=5
order:o-3 = processing
order:o-2 = done total=10
order:o-4 = processing
order:o-3 = done total=15
order:o-4 = done total=20

The two workers split the orders between them, as in the work queue, so the four orders take about four seconds instead of eight. Stop the dashboard and start it again after a few seconds. It prints the current status of each order first, then follows any new ones.

To see the failure paths, stop a worker with Ctrl-C while it is fulfilling an order. The other worker finishes the order about 30 seconds later, on attempt 2, and the dashboard shows it reach done. To see the rejection path, publish text that is not an order:

Terminal window
docker compose run --rm felixctl pub tutorial-orders 'not json'

A worker prints rejected a malformed order and carries on.

An order that fails every attempt becomes a dead letter, and the worker never writes its final status. Its key stays at processing until the one hour TTL removes it. List the group’s dead letters, as in A work queue, to find orders stuck that way. Redriving one runs it again.

Each piece here has more to it. Queues and consumer groups covers extending a claim for long jobs, delayed retries and dead letters, and Cache covers counters and conditional writes. To run several workers that must not duplicate an effect, combine the work queue with the lease from Cache, leases and watches. The demos run larger systems on Felix, and Choosing a client compares the language bindings.