Skip to content

felixd terminology

The felixd pages use these words without stopping to define them. Each entry says what the term means in felixd and links to where it is used. Terms are grouped by area. Words that Felix itself defines, such as stream, shard, offset and cache, are in the Felix terminology.

A program felixd runs, plus the spec that says what it reads and under which limits. Its only inputs and outputs are Felix streams. It is identified by an id of 1 to 64 characters of a-z, 0-9 and -, which names its streams. See How it works.

The JSON value in the workloads cache that says what should run: the module’s hash, the input stream, the output stream, the snapshot cadence, the limits, the state and, if the workload failed, why. The limits are in the spec so every node runs the workload under the same ones. See The spec.

The WebAssembly artifact a workload is built as. A component carries typed interfaces, and felixd accepts one that implements the workload world and imports nothing. See The workload contract.

A component as stored. Its sha256 in hex is its identity, and it is kept in the modules cache under that hash. A spec names its module by hash.

The WebAssembly Interface Type language that describes a component’s imports and exports. The workload contract is written in WIT, in wit/0.3/workload.wit.

The interface between felixd and a workload and what each side promises. These pages teach 0.3. felixd still runs 0.2, which differs only in that a step never sees the publisher. A component built against 0.1 is refused. See The workload contract.

The workload function felixd calls once per input record, in input order. It returns outputs, or a rejection, or poisons the workload. See What step returns.

The workload function that returns its state as JSON. felixd replay prints it.

One record on the workload’s input stream, with metadata: the stream, the incarnation, the offset, the broker’s append time, the key, the payload and the publisher. When the input is another workload’s output stream, only the outputs addressed to out are inputs.

The principal the broker authenticated an input record’s write from, such as p:alice. A step receives it with the record, and it is none when the broker recorded none or when the input is another workload’s output. It says who wrote the record, not what is in it. See The publisher.

The stream wl.<id>.merge of a workload with several inputs. The lease holder appends the order it chose to step their records in before it steps them, so a new owner, a replay and felixd check use the same order.

A felixd workload that decides which submitted workloads run, by who asked. After felixd init, felixd run sends a request to it and nodes run only what it accepted. See Running felixd.

The offset of a record on the input stream. An output records the input offset of the step that produced it, a snapshot records the last input offset its state includes, and a replay’s target can be an input offset. The state as of an input offset includes that input.

A record a step produces, written to the workload’s output stream. It has a name, an optional key, a payload and a header of input offset, index and epoch. Outputs are written exactly once. See One step.

The label on an output. 1 to 64 bytes of lowercase letters, digits and -. The default is out, and a downstream workload reads the outputs named out.

An optional byte string on an output. A downstream workload receives it as the input’s key.

An output’s position among the outputs of one step, from 0, whatever their names. Together with the input offset it names an output uniquely.

The result of a step that returns reject: the input is bad and the workload carries on. It is recorded on the output stream as that input’s only output, with an error code and detail. See What step returns.

The result of a step that returns poison, which says the workload cannot go on. The word also names the class of failure that comes from an input the workload deterministically fails on. It fails the workload and is not retried. There is no recorded way past a poison input (#98).

A result that breaks the contract: an invalid output name, an effect request, a chunk over 4 MiB, more than 65,536 chunks, or more than 8 MiB of output from one step. It would break the same way on any node, so it fails the workload. See What counts as a failure.

A WebAssembly runtime error in the workload, such as an unreachable instruction or an out of bounds access. A trap fails the workload. Running out of fuel is a trap.

A workload that no node can run through no fault of its own and that a person has to fix: a module built against contract 0.1, or a restore that ran out of snapshot_fuel. It is not marked failed. The node writes a needs-operator record, with a reason, to the output stream and releases the lease. felixd ps shows it, and no node runs the workload until its spec changes.

failed is a value of a spec’s state. A holder writes it after the failure is recorded on the output stream, and it stays until felixd run writes a new spec. degraded is recorded when a node keeps failing on a transient error and releases the workload so another node can try. Counting that across the cluster is not built (#99). See When a workload fails.

wasmtime’s measure of work: the runtime deducts from a counter as WebAssembly instructions execute, roughly one unit per instruction, and traps when it reaches zero. It counts instructions and not time, so the same code on the same input burns the same fuel on any machine. felixd uses it to fail a runaway step at the same point on every node and to bound a call that would otherwise run for ever. Fuel is set again before each step; a snapshot or a restore gets one budget for all of its calls. Running out in a step fails the workload, in a snapshot skips the snapshot, and in a restore leaves the workload to an operator. See What fuel is.

The fuel given to each step and each inspect. The default is 50,000,000, set with felixd run --fuel.

The fuel given to a whole snapshot or a whole restore, every chunk call together. The default is 500,000,000, ten times the step default, set with felixd run --snapshot-fuel.

The most linear memory the workload may grow to. The default is 64 MiB.

A snapshot is written after this many inputs. The default is 1000.

Also a snapshot at the first input of each interval of this many milliseconds, by the inputs’ broker append times. Absent unless set.

A machine running a felixd agent. A node is named by its node id, which defaults to the host name and process id.

The felixd agent process on a node. It watches the workloads cache, claims and runs workloads, renews their leases and announces the node in the nodes cache.

The right to run a workload: an entry in the leases cache with a TTL, holding the owner’s node id and epoch. A lease decides who should run the workload. It is not what keeps the output correct, because it depends on clocks. See Claiming a workload.

The time a lease lives without renewal. The default is 10 seconds and the minimum is 2 seconds, set with felixd agent --lease-ttl-ms. The holder renews every third of the TTL and stops by its own clock two thirds of the TTL after its last successful renewal.

Two things, one on each side of the lease. A claim of a lease is a cache_put_if on “absent” that wins when the broker counts the previous lease as gone. A claim record is the first record a new owner appends to the output stream, carrying its epoch and node id, which fences the stream.

A Felix conditional cache write that applies only if the key is absent or at an expected version. felixd uses it to claim and renew leases, to take epochs and to write failed into a spec.

A number that grows with each claim of a workload. A claim takes an epoch one above every earlier claim’s and every epoch on the workload’s streams, and the owner stamps it on every record it writes. A record from a later epoch is how a stale owner learns it has been replaced. See Claiming a workload.

What stops a stale owner from writing. A new owner appends a claim record to the output stream, so a stale owner’s publish_if at its old tail is refused, and every snapshot and failure record is written the same way. See Epochs and the fence.

A node that no longer holds a workload but does not know it yet, for example one that was frozen past its lease. The fence refuses its writes.

A Felix conditional publish that applies only if the stream’s tail is at the offset the writer expects. Every write felixd makes to a workload’s streams is one. It is a compare-and-set on the whole shard, which is why a workload’s streams have one shard.

A stream’s next offset. A writer expects the tail it last saw, and a publish_if at a different tail is refused with the real one. The tail can be past the last output because claims and the generation-start records Felix appends take offsets that hold no output.

A stream deleted and created again under the same name is a different incarnation of it. The contract carries an incarnation in each input’s source. It is always 0 today, because Felix does not expose incarnations to clients yet (#133).

Another node taking over a workload after its holder stopped renewing. It is the same operation as resume. See How it works.

Which node claims which workload. Today it is first claim wins: after a random delay a node with room claims any free workload. See Claiming a workload.

A workload’s state saved on wl.<id>.snap as chunks followed by a manifest. It lets a new owner start from a recent point. Only a manifest makes a snapshot. See Snapshots.

A piece of a snapshot, at most 4 MiB, with its sha256. A workload returns its state as a list of chunks. A chunk that did not change since the last snapshot is not written again.

The record that makes a snapshot. It holds the input offset the state includes, the output stream’s next offset at that moment, and the sha256 and the stream offset of each chunk.

What a new owner does to pick a workload up: claim, read the latest snapshot manifest and the last output, compute the fold, restore the snapshot if there is one, and subscribe to the input. See Resume.

The computation that decides how to resume. From the latest snapshot’s header, the last output record and the output stream’s tail it returns whether to restore, which input offset to subscribe from, and which outputs the stream already holds. Replay uses the same fold.

Loading a snapshot’s chunks into a fresh instance with restore and restore-chunk, under snapshot_fuel.

Deciding what to do when a write is refused or its answer is lost: compare what is on the stream at the expected tail with the batch, then skip what landed, stop if a later epoch wrote, or send again.

The state of an owner that is running inputs whose outputs are already on the stream. It takes no snapshot until it is past them.

Rebuilding a workload’s state at an input offset or time by restoring the nearest earlier snapshot and running the inputs after it. felixd replay does it, takes no lease and writes nothing. See Replay.

A replayed output that differs from the recorded one. During a resume it fails the workload. felixd check reports it as a record that differs, and felixd replay --diff reports the first one with the input offset, the epoch, the node and both digests.

felixd check: replay each workload over its whole input and compare the result with its output stream. See Checking.

The property that a workload given the same inputs produces the same outputs and the same state on any node. Failover, exactly-once output and replay all rest on it. felixd enforces it by letting a workload import nothing, canonicalizing NaN bit patterns and applying fixed limits. What breaks it: a different felixd build, a changed module or input under a workload that has output, and possibly a different machine architecture. See Why the guarantees hold.

Each input’s outputs appear once on the output stream, in order, across takeovers, lost answers and partial batches.

A request from a workload to the outside world, written as an output and answered by a recorded input. The contract reserves the shape. Nothing performs an effect, and a workload that returns a request fails (#71).

The component that would perform an effect and answer it. It would use the request’s id, which is the output header, as an idempotency key, and refuse a request whose epoch is below the highest it has seen. No executor exists yet (#71).

The proposed name for the distributed operating system felixd is a first step toward. It is not decided, and felixd would stay the name of the agent on each node. See the discussion.