Skip to content

Cache, leases and watches

You will use a Felix cache for three things. First you store a value that expires on its own. Then you build a lease, a name that only one process holds at a time, with a conditional write, and use its version as a fencing token. Last you watch a set of keys and print each change as it happens.

A cache is a set of keys with values, optional expiry times, and a version on every value. Unlike a stream, it keeps the latest value per key, and a watch tells you when one changes. See Cache for the full model.

A running broker, the env file and the client for your language, all set up as the tutorials overview describes. The steps show Rust, Python, TypeScript and C#, and the C version is one program at the end.

The lease step has tabs for Rust, C# and C, and the Python and TypeScript programs have no lease mode. Both bindings have the same conditional writes; their Python and TypeScript pages show the calls.

Terminal window
docker compose run --rm felixctl cache create tutorial-cache

Use felixctl cache create tutorial-cache if you installed it. With these defaults the cache is not replicated, and the leader acknowledges a write alone. A lease you rely on in production belongs on a cache created with --replication 3 --consistency quorum, so that a claim a broker acknowledged survives losing that broker. See Leases, locks and idempotency.

2. Connect, and store a value that expires

Section titled “2. Connect, and store a value that expires”

The program takes a mode: ttl, lease <name>, watch, set <key> <value> or delete <key>. The first stores a value with a five second time to live, reads it, waits, and reads it again.

use std::future::Future;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use bytes::Bytes;
use felix_client::{ClientConfig, Client, quic_client_config, root_store_from_pem_file};
// Must match FELIX_AUTH_TENANT.
const TENANT: &str = "dev";
const NAMESPACE: &str = "default";
const CACHE: &str = "tutorial-cache";
// Every call to the broker gets this long.
const CALL_TIMEOUT: Duration = Duration::from_secs(10);
async fn bounded<T>(what: &str, limit: Duration, call: impl Future<Output = Result<T>>) -> Result<T> {
tokio::time::timeout(limit, call)
.await
.with_context(|| format!("{what} did not finish within {limit:?}"))?
}
async fn connect() -> Result<Arc<Client>> {
let ca_file = std::env::var("FELIX_CA_FILE").context("set FELIX_CA_FILE")?;
let roots = Arc::new(root_store_from_pem_file(&ca_file)?);
// Reads FELIX_AUTH_TENANT and FELIX_AUTH_TOKEN.
let config = ClientConfig::from_env_or_yaml(quic_client_config(Some(roots), false)?, None)?;
let broker = std::env::var("FELIX_BROKERS").unwrap_or_else(|_| "127.0.0.1:5000".into());
let seeds: Vec<SocketAddr> = vec![broker.parse().context("FELIX_BROKERS must be ip:port")?];
let connecting = Client::connect(&seeds, "localhost", config);
let client = bounded("connecting", CALL_TIMEOUT, connecting).await?;
Ok(Arc::new(client))
}
async fn ttl(client: &Client) -> Result<()> {
// The TTL is in milliseconds.
let put = client.cache_put(TENANT, NAMESPACE, CACHE, "greeting", Bytes::from_static(b"hello"), Some(5_000));
bounded("put", CALL_TIMEOUT, put).await?;
let get = client.cache_get(TENANT, NAMESPACE, CACHE, "greeting");
println!("right after the put: {:?}", bounded("get", CALL_TIMEOUT, get).await?);
tokio::time::sleep(Duration::from_secs(6)).await;
let get = client.cache_get(TENANT, NAMESPACE, CACHE, "greeting");
println!("six seconds later: {:?}", bounded("get", CALL_TIMEOUT, get).await?);
Ok(())
}

A key that is missing or expired reads as nothing. The expiry time is stored with the value, so it holds across a broker restart.

A watch pushes each change to a set of keys: one key, or every key that starts with a prefix. A change is a new value or, when the value is absent, a delete. This watch follows keys that start with status: and starts from now.

Two things to handle are in the loop. A watch that falls behind is ended by the broker, and it says where to watch again from so that no change is missed. A shard that moves to another broker is followed for you, and the watch tells you it happened.

use anyhow::bail;
use felix_client::{CacheWatchFilter, CacheWatchItem};
async fn watch(client: &Arc<Client>) -> Result<()> {
let start = client.watch_cache(TENANT, NAMESPACE, CACHE, CacheWatchFilter::Prefix("status:".into()), None);
let mut watch = bounded("starting the watch", CALL_TIMEOUT, start).await?;
println!("watching status:*");
while let Some(item) = watch.recv().await {
match item {
CacheWatchItem::Change(change) => match change.value {
Some(value) => println!("{} = {}", change.key, String::from_utf8_lossy(&value)),
None => println!("{} deleted", change.key),
},
CacheWatchItem::Lagged { resume_from } => {
bail!("fell behind; watch again from offset {resume_from}")
}
CacheWatchItem::ShardMoved(_) => println!("the shard moved; the watch follows it"),
}
}
Ok(())
}
async fn set(client: &Client, key: &str, value: &str) -> Result<()> {
let put = client.cache_put(TENANT, NAMESPACE, CACHE, key, Bytes::copy_from_slice(value.as_bytes()), None);
bounded("put", CALL_TIMEOUT, put).await
}
async fn delete(client: &Client, key: &str) -> Result<()> {
let delete = client.cache_delete(TENANT, NAMESPACE, CACHE, key);
let removed = bounded("delete", CALL_TIMEOUT, delete).await?;
println!("{}", if removed.is_some() { "deleted" } else { "there was nothing to delete" });
Ok(())
}

Several workers run the same job, and exactly one should at a time. The job is named by a cache key, and holding the lease means holding the key.

A conditional write claims it. Absent writes only if the key has no live entry, and the check and the write are one atomic step, so two workers racing for the key cannot both win. The write carries a TTL, so a holder that dies loses the lease on its own. A refusal is an answer (applied is false), not an error.

The answer to a claim that applied carries the key’s version. A version is the log offset of the write, so it only grows, across renewals, restarts and compaction. That makes it a fencing token. A worker passes it along with everything it does as the lease holder, and the system it writes to remembers the highest token it has seen and refuses a lower one. Without that, a worker that stalls past its TTL, say in a long pause, would wake up and write after the lease passed to someone else.

The holder renews with Version(token), so a renewal fails if the lease has since changed hands, and releases with a delete at that version for the same reason. It releases on every path, including after an error, so the next worker does not wait out the TTL. If the release itself fails the lease still expires on its own. A worker that does not hold the lease watches the key, opened before its first claim so a release in between is not missed, and tries again when the lease is released or expires, or after one TTL, whichever comes first. The expired entry is deleted by the broker, so the watch hears about it, up to about a second after the TTL.

use felix_client::{CacheCondition, ClusterCacheWatch};
const LEASE_KEY: &str = "lease:report";
const LEASE_TTL_MS: u64 = 10_000;
async fn lease(client: &Arc<Client>, name: &str) -> Result<()> {
// Watch before the first claim, so a release that lands between a refusal
// and the wait is not missed.
let start = client.watch_cache(TENANT, NAMESPACE, CACHE, CacheWatchFilter::Key(LEASE_KEY.into()), None);
let mut watch = bounded("starting the watch", CALL_TIMEOUT, start).await?;
loop {
let claim = client.cache_put_if(
TENANT, NAMESPACE, CACHE, LEASE_KEY,
Bytes::copy_from_slice(name.as_bytes()),
Some(LEASE_TTL_MS),
CacheCondition::Absent,
);
let claim = bounded("claim", CALL_TIMEOUT, claim).await?;
if claim.applied {
let token = claim.version.context("the broker did not report a version to fence with")?;
drop(watch);
return hold(client, name, token).await;
}
println!("{name}: the lease is taken, waiting");
wait_for_release(&mut watch).await?;
}
}
async fn hold(client: &Client, name: &str, mut token: u64) -> Result<()> {
println!("{name}: holds the lease, fencing token {token}");
let worked = renew_rounds(client, name, &mut token).await;
// Release on every path, so the next worker does not wait out the TTL. If
// this fails the lease still expires on its own.
let release = client.cache_delete_if(TENANT, NAMESPACE, CACHE, LEASE_KEY, token);
match bounded("release", CALL_TIMEOUT, release).await {
Ok(released) => {
println!("{name}: released ({})", if released.applied { "ok" } else { "it had already moved on" });
}
Err(err) => println!("{name}: could not release, the lease expires on its own: {err:#}"),
}
worked
}
async fn renew_rounds(client: &Client, name: &str, token: &mut u64) -> Result<()> {
for round in 1..=3 {
// Do a unit of work here, passing `token` to whatever it writes.
tokio::time::sleep(Duration::from_millis(LEASE_TTL_MS / 3)).await;
let renew = client.cache_put_if(
TENANT, NAMESPACE, CACHE, LEASE_KEY,
Bytes::copy_from_slice(name.as_bytes()),
Some(LEASE_TTL_MS),
CacheCondition::Version(*token),
);
let renewed = bounded("renew", CALL_TIMEOUT, renew).await?;
match renewed.version {
Some(version) if renewed.applied => *token = version,
_ => bail!("{name}: lost the lease; stop working"),
}
println!("{name}: renewed (round {round}), fencing token {token}");
}
Ok(())
}
// Returns when the lease is deleted, or after one TTL.
async fn wait_for_release(watch: &mut ClusterCacheWatch) -> Result<()> {
let deleted = async {
while let Some(item) = watch.recv().await {
if matches!(item, CacheWatchItem::Change(ref change) if change.value.is_none()) {
return true;
}
}
false
};
// A timeout is expected while the holder keeps renewing.
match tokio::time::timeout(Duration::from_millis(LEASE_TTL_MS), deleted).await {
Ok(false) => bail!("the watch ended"),
_ => Ok(()),
}
}
#[tokio::main]
async fn main() -> Result<()> {
let client = connect().await?;
let args: Vec<String> = std::env::args().skip(1).collect();
let arg = |i: usize| args.get(i).map(String::as_str);
match (arg(0), arg(1), arg(2)) {
(Some("ttl"), _, _) => ttl(&client).await,
(Some("watch"), _, _) => watch(&client).await,
(Some("lease"), Some(name), _) => lease(&client, name).await,
(Some("set"), Some(key), Some(value)) => set(&client, key, value).await,
(Some("delete"), Some(key), _) => delete(&client, key).await,
_ => bail!("usage: cache_leases_watches ttl | watch | lease <name> | set <key> <value> | delete <key>"),
}
}

Run source /path/to/felix-tutorial.env from the overview in each terminal first. ttl, watch, set and delete are in every language. lease is in the Rust, C# and C programs only.

From the project directory:

Terminal window
# value with an expiry
cargo run --bin cache_leases_watches -- ttl
# terminal 1: watch the status keys
cargo run --bin cache_leases_watches -- watch
# terminal 2: change keys
cargo run --bin cache_leases_watches -- set status:api up
cargo run --bin cache_leases_watches -- set status:worker degraded
cargo run --bin cache_leases_watches -- delete status:api
# stop the watch with Ctrl-C, then in terminal 1 take the lease
cargo run --bin cache_leases_watches -- lease a
# terminal 2, quickly: wait for it
cargo run --bin cache_leases_watches -- lease b

After ttl, the program shows the value and then its absence:

right after the put: Some(b"hello")
six seconds later: None

Other languages print the value in their own way. For the watch, start watch in one terminal and run the three set and delete commands from another:

watching status:*
status:api = up
status:worker = degraded
status:api deleted

A key without the status: prefix produces nothing, because the watch filters on the broker.

For the lease, run lease a and then, quickly, lease b in another terminal. The Python and TypeScript programs have no lease mode. Your version numbers will differ. They only grow:

a: holds the lease, fencing token 12
a: renewed (round 1), fencing token 14
a: renewed (round 2), fencing token 15
a: renewed (round 3), fencing token 17
a: released (ok)
b: the lease is taken, waiting
b: holds the lease, fencing token 19
b: renewed (round 1), fencing token 21
...

b sat on a watch while a held the lease and took over as soon as a released it. To see expiry instead, stop a with Ctrl-C partway through. b takes over when the lease’s TTL passes, at most ten seconds after a’s last renewal, and its token is higher than any a held.

Each C call returns at once and reports to a callback, so this program wraps each one in a waiter that blocks until the callback has run or ten seconds have passed. It has the same modes as above. Values and keys handed to a callback are valid only while it runs, so the callbacks copy what the program needs.

#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include "felix.h"
#define TENANT "dev"
#define NAMESPACE "default"
#define CACHE "tutorial-cache"
#define LEASE_KEY "lease:report"
#define LEASE_TTL_MS 10000
struct waiter {
pthread_mutex_t mu;
pthread_cond_t cv;
bool done;
FelixStatus status;
void *handle;
bool found;
bool applied;
bool has_version;
uint64_t version;
int kind;
bool has_value;
char key[128];
char value[128];
char message[256];
};
static void finish(struct waiter *w, FelixStatus status, const char *message, void *handle) {
pthread_mutex_lock(&w->mu);
w->done = true;
w->status = status;
w->handle = handle;
snprintf(w->message, sizeof w->message, "%s", message ? message : "");
pthread_cond_signal(&w->cv);
pthread_mutex_unlock(&w->mu);
}
/* A refused call never runs its callback, so its error comes from the thread.
A call the broker does not answer within limit_ms ends the program: its
callback may still run later, so nothing it uses can be freed. */
static FelixStatus wait_for_ms(struct waiter *w, FelixStatus started, long limit_ms) {
if (started != FELIX_STATUS_OK) {
snprintf(w->message, sizeof w->message, "%s", felix_last_error_message());
return started;
}
struct timespec deadline;
clock_gettime(CLOCK_REALTIME, &deadline);
deadline.tv_sec += limit_ms / 1000;
deadline.tv_nsec += (limit_ms % 1000) * 1000000L;
if (deadline.tv_nsec >= 1000000000L) {
deadline.tv_sec++;
deadline.tv_nsec -= 1000000000L;
}
pthread_mutex_lock(&w->mu);
while (!w->done) {
if (pthread_cond_timedwait(&w->cv, &w->mu, &deadline) == ETIMEDOUT) {
fprintf(stderr, "no answer from the broker within %ld ms\n", limit_ms);
exit(1);
}
}
w->done = false;
FelixStatus status = w->status;
pthread_mutex_unlock(&w->mu);
return status;
}
/* Every call to the broker gets this long. */
#define CALL_LIMIT_MS 10000
#define wait_for(w, started) wait_for_ms((w), (started), CALL_LIMIT_MS)
static void copy_text(char *to, size_t cap, const void *from, size_t len) {
size_t n = len < cap - 1 ? len : cap - 1;
if (n > 0) memcpy(to, from, n);
to[n] = '\0';
}
static void on_connect(void *user_data, FelixStatus status, const char *message,
FelixClient *client) {
finish(user_data, status, message, client);
}
static void on_done(void *user_data, FelixStatus status, const char *message) {
finish(user_data, status, message, NULL);
}
static void on_value(void *user_data, FelixStatus status, const char *message, bool found,
const uint8_t *value, size_t len) {
struct waiter *w = user_data;
w->found = status == FELIX_STATUS_OK && found;
if (w->found) copy_text(w->value, sizeof w->value, value, len);
finish(w, status, message, NULL);
}
static void on_condition(void *user_data, FelixStatus status, const char *message,
bool applied, bool has_version, uint64_t version) {
struct waiter *w = user_data;
w->applied = applied;
w->has_version = has_version;
w->version = version;
finish(w, status, message, NULL);
}
static void on_watch_open(void *user_data, FelixStatus status, const char *message,
FelixCacheWatch *watch) {
finish(user_data, status, message, watch);
}
static void on_watch_item(void *user_data, FelixStatus status, const char *message,
const FelixWatchItem *item) {
struct waiter *w = user_data;
if (status == FELIX_STATUS_OK && item) {
w->kind = item->kind;
w->has_value = item->has_value;
copy_text(w->key, sizeof w->key, item->key, item->key_len);
copy_text(w->value, sizeof w->value, item->value, item->value_len);
}
finish(w, status, message, NULL);
}
static int ttl(const FelixClient *client, struct waiter *w) {
static const uint8_t hello[] = "hello";
FelixStatus status = wait_for(
w, felix_client_cache_put(client, TENANT, NAMESPACE, CACHE, "greeting", hello,
sizeof hello - 1, 5000 /* ttl_ms */, NULL, on_done, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "put: %s\n", w->message);
return 1;
}
for (int pass = 0; pass < 2; pass++) {
status = wait_for(w, felix_client_cache_get(client, TENANT, NAMESPACE, CACHE, "greeting",
NULL, on_value, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "get: %s\n", w->message);
return 1;
}
printf("%s: %s\n", pass == 0 ? "right after the put" : "six seconds later",
w->found ? w->value : "(none)");
if (pass == 0) sleep(6);
}
return 0;
}
static int set_key(const FelixClient *client, struct waiter *w, const char *key,
const char *value) {
FelixStatus status = wait_for(
w, felix_client_cache_put(client, TENANT, NAMESPACE, CACHE, key, (const uint8_t *)value,
strlen(value), 0 /* no expiry */, NULL, on_done, w));
if (status != FELIX_STATUS_OK) fprintf(stderr, "put: %s\n", w->message);
return status == FELIX_STATUS_OK ? 0 : 1;
}
static int delete_key(const FelixClient *client, struct waiter *w, const char *key) {
FelixStatus status = wait_for(
w, felix_client_cache_delete(client, TENANT, NAMESPACE, CACHE, key, NULL, on_value, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "delete: %s\n", w->message);
return 1;
}
printf("%s\n", w->found ? "deleted" : "there was nothing to delete");
return 0;
}
static int watch_prefix(const FelixClient *client, struct waiter *w) {
FelixStatus status = wait_for(
w, felix_client_watch_cache(client, TENANT, NAMESPACE, CACHE, FELIX_WATCH_PREFIX,
"status:", false, false, 0, NULL, on_watch_open, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "watch: %s\n", w->message);
return 1;
}
FelixCacheWatch *watch = w->handle;
int exit_code = 1;
printf("watching status:*\n");
for (;;) {
status = wait_for_ms(w, felix_cache_watch_next(watch, 30000, NULL, on_watch_item, w),
30000 + CALL_LIMIT_MS);
if (status == FELIX_STATUS_TIMEOUT) continue;
if (status == FELIX_STATUS_CLOSED) {
exit_code = 0;
break;
}
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "watch: %s\n", w->message);
break;
}
if (w->kind == FELIX_WATCH_CHANGE && w->has_value) {
printf("%s = %s\n", w->key, w->value);
} else if (w->kind == FELIX_WATCH_CHANGE) {
printf("%s deleted\n", w->key);
} else if (w->kind == FELIX_WATCH_LAGGED) {
fprintf(stderr, "fell behind; watch again\n");
break;
} else {
printf("the shard moved; the watch follows it\n");
}
}
felix_cache_watch_free(watch);
return exit_code;
}
/* Returns when the lease key is deleted, or after one TTL. */
static int wait_for_release(struct waiter *w, FelixCacheWatch *watch) {
for (;;) {
FelixStatus status = wait_for_ms(
w, felix_cache_watch_next(watch, LEASE_TTL_MS, NULL, on_watch_item, w),
LEASE_TTL_MS + CALL_LIMIT_MS);
if (status == FELIX_STATUS_TIMEOUT) return 0;
if (status == FELIX_STATUS_CLOSED) {
fprintf(stderr, "the watch ended\n");
return 1;
}
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "watch: %s\n", w->message);
return 1;
}
if (w->kind == FELIX_WATCH_CHANGE && !w->has_value) return 0;
}
}
static int hold(const FelixClient *client, struct waiter *w, const char *name, uint64_t token) {
int exit_code = 0;
FelixStatus status;
printf("%s: holds the lease, fencing token %llu\n", name, (unsigned long long)token);
for (int round = 1; round <= 3; round++) {
/* Do a unit of work here, passing token to whatever it writes. */
struct timespec pause = {LEASE_TTL_MS / 3 / 1000, (LEASE_TTL_MS / 3 % 1000) * 1000000L};
nanosleep(&pause, NULL);
status = wait_for(
w, felix_client_cache_put_if(client, TENANT, NAMESPACE, CACHE, LEASE_KEY,
(const uint8_t *)name, strlen(name), LEASE_TTL_MS,
FELIX_IF_VERSION, token, NULL, on_condition, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "renew: %s\n", w->message);
exit_code = 1;
break;
}
if (!w->applied || !w->has_version) {
fprintf(stderr, "%s: lost the lease; stop working\n", name);
exit_code = 1;
break;
}
token = w->version;
printf("%s: renewed (round %d), fencing token %llu\n", name, round,
(unsigned long long)token);
}
/* Release on every path, so the next worker does not wait out the TTL. If
this fails the lease still expires on its own. */
status = wait_for(
w, felix_client_cache_delete_if(client, TENANT, NAMESPACE, CACHE, LEASE_KEY, token, NULL,
on_condition, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "%s: could not release, the lease expires on its own: %s\n", name,
w->message);
} else {
printf("%s: released (%s)\n", name, w->applied ? "ok" : "it had already moved on");
}
return exit_code;
}
static int lease(const FelixClient *client, struct waiter *w, const char *name) {
/* Watch before the first claim, so a release that lands between a refusal
and the wait is not missed. */
FelixStatus status = wait_for(
w, felix_client_watch_cache(client, TENANT, NAMESPACE, CACHE, FELIX_WATCH_KEY,
LEASE_KEY, false, false, 0, NULL, on_watch_open, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "watch: %s\n", w->message);
return 1;
}
FelixCacheWatch *watch = w->handle;
for (;;) {
status = wait_for(
w, felix_client_cache_put_if(client, TENANT, NAMESPACE, CACHE, LEASE_KEY,
(const uint8_t *)name, strlen(name), LEASE_TTL_MS,
FELIX_IF_ABSENT, 0, NULL, on_condition, w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "claim: %s\n", w->message);
break;
}
if (w->applied) {
if (!w->has_version) {
fprintf(stderr, "the broker did not report a version to fence with\n");
break;
}
felix_cache_watch_free(watch);
return hold(client, w, name, w->version);
}
printf("%s: the lease is taken, waiting\n", name);
if (wait_for_release(w, watch) != 0) break;
}
felix_cache_watch_free(watch);
return 1;
}
int main(int argc, char **argv) {
const char *token = getenv("FELIX_AUTH_TOKEN");
const char *ca_file = getenv("FELIX_CA_FILE");
const char *broker = getenv("FELIX_BROKERS");
const char *mode = argc > 1 ? argv[1] : "";
bool ok = (!strcmp(mode, "ttl") || !strcmp(mode, "watch")) ||
(!strcmp(mode, "lease") && argc > 2) || (!strcmp(mode, "set") && argc > 3) ||
(!strcmp(mode, "delete") && argc > 2);
if (!ok) {
fprintf(stderr, "usage: cache-leases-watches ttl | watch | lease <name> | set <key> <value> | delete <key>\n");
return 1;
}
if (!token || !*token || !ca_file || !*ca_file) {
fprintf(stderr, "set FELIX_AUTH_TOKEN and FELIX_CA_FILE\n");
return 1;
}
struct waiter w = {.done = false};
pthread_mutex_init(&w.mu, NULL);
pthread_cond_init(&w.cv, NULL);
int exit_code = 1;
FelixStatus status = wait_for(
&w, felix_client_connect(broker && *broker ? broker : "127.0.0.1:5000", TENANT,
token, "localhost", ca_file, NULL, on_connect, &w));
if (status != FELIX_STATUS_OK) {
fprintf(stderr, "connect: %s\n", w.message);
} else {
FelixClient *client = w.handle;
if (!strcmp(mode, "ttl")) {
exit_code = ttl(client, &w);
} else if (!strcmp(mode, "watch")) {
exit_code = watch_prefix(client, &w);
} else if (!strcmp(mode, "lease")) {
exit_code = lease(client, &w, argv[2]);
} else if (!strcmp(mode, "set")) {
exit_code = set_key(client, &w, argv[2], argv[3]);
} else {
exit_code = delete_key(client, &w, argv[2]);
}
/* Every call has completed, so no callback still holds &w. */
felix_client_free(client);
}
pthread_cond_destroy(&w.cv);
pthread_mutex_destroy(&w.mu);
return exit_code;
}

Save it as cache-leases-watches.c in the unpacked release directory from the overview, and compile and link it there:

Terminal window
cc -std=c11 -pthread -I include cache-leases-watches.c -L lib -lfelix \
-Wl,-rpath,"$PWD/lib" -o cache-leases-watches

This links lib/libfelix.dylib on macOS. To build from source instead, see the C client page. Run it with the commands in the C tab of Run it.

Build an app puts a stream, a work queue and a cache together. For leases in production, including a single writer of a stream with publish_if and what each consistency level guarantees after a failover, read Leases, locks and idempotency. The cache page covers counters, retained watches that start from current state, and resuming a watch from an offset.