Docker Compose Deployment

Running Felix under Docker Compose, for local development and testing. With Podman, run the same files with podman compose; see Docker or Podman for the provider it needs and the few places the two differ.

The images are published to GHCR and are pullable without credentials:

docker pull ghcr.io/getfelix/felix-broker:0.6.0
docker pull ghcr.io/getfelix/felix-controlplane:0.6.0

Releases before 0.6.0-preview.2 are under ghcr.io/gabloe, the project’s previous owner.

Each release publishes the full version (0.6.0). A release without a pre-release suffix also publishes its minor series (0.6) and latest. Use a full version tag in anything you deploy, because the other two move. A client negotiates its features with the broker it connects to, so a newer client against older images runs without whatever the images predate.

To build them yourself instead, for a change you have not released or an architecture the release does not build:

docker build -f docker/broker.Dockerfile -t felix-broker:dev .
docker build -f docker/controlplane.Dockerfile -t felix-controlplane:dev .

Both build from the repository root. The binaries are workspace members, so cargo needs the whole workspace to resolve them. Each Dockerfile takes one build argument, BIN, naming the binary to build. Docker warns about any other --build-arg and ignores it. podman build needs --format docker, or it drops the images’ HEALTHCHECK.

Try it in one command

compose.yaml at the root of the repository runs a control plane, one broker and development credentials from the released images. It is a single file, so it works from a clone or on its own:

curl -fsSLO https://raw.githubusercontent.com/GetFelix/felix/main/compose.yaml
docker compose up

It sits at the root because compose.yaml is the name Compose looks for, so docker compose up in a fresh clone finds it with no -f.

With Podman, run it through the docker-compose provider (2.20 or later), and use podman compose wherever this section says docker compose:

export PODMAN_COMPOSE_PROVIDER=docker-compose
podman compose up

podman-compose does not work here. It cannot run a service that sits in a profile, as the felixctl service does, and it starts the broker even when felix-dev-auth fails, because it treats that dependency as merely stopped.

The images default to the release the file was last bumped for. Set FELIX_VERSION to run another tag (FELIX_VERSION=0.6.0 docker compose up).

Once the broker is ready, the stack prints its connection settings. In a second terminal, from the same directory:

docker compose run --rm felixctl sub orders

and in a third:

docker compose run --rm felixctl pub orders 'hello'

The subscriber prints hello. That felixctl runs inside the stack, with the settings already in its environment. To use one installed on your machine instead, copy the token and the broker’s certificate out and add a context:

docker compose exec -T felix-broker cat /var/lib/felix/client.jwt > felix-dev.jwt
docker compose exec -T felix-broker cat /var/lib/felix/controlplane.jwt > felix-dev-cp.jwt
docker compose exec -T felix-broker cat /var/lib/felix/broker-cert.pem > felix-dev.pem
felixctl context add felix-dev --brokers 127.0.0.1:5000 --tenant dev \
  --token-file felix-dev.jwt --ca-file felix-dev.pem \
  --controlplane-url http://127.0.0.1:8443 --controlplane-token-file felix-dev-cp.jwt --use
felixctl pub orders 'hello'

The broker makes a new certificate every time it starts, so copy broker-cert.pem again after a restart.

What it runs:

Service	What it does	Host ports
`felix-controlplane`	The control plane, with its store in memory	`8443` (REST), `8081` (metrics, `/ready`)
`felix-dev-auth`	Runs once: initializes tenant `dev`, creates namespace `default`, the durable stream `orders` and the caches `config` and `stats`, and mints the tokens	none
`felix-broker`	One broker, with durable storage in the `felix-dev` volume	`5000-5003/udp` (QUIC), `8080` (metrics, `/ready`)
`felix-dev-ready`	Prints the connection settings once the broker is ready	none
`felixctl`	Only for `docker compose run`	none

Every host port is bound to 127.0.0.1, so nothing in the stack is reachable from another machine. The control plane keeps its state in memory. The broker sets FELIX_DURABLE_STORAGE_DIR to a directory in the felix-dev volume, so streams, consumer groups and caches are on disk and cache watches work, and the tokens and certificate sit in the same volume. docker compose down -v discards all of it, and the next up starts clean. Without -v the volume survives, but the control plane forgets its streams and the next up overwrites the tokens, so run down -v before starting again.

The development credentials

felix-dev-auth uses the control plane’s development tokens through the ordinary bootstrap API, with no identity provider. None of this is a mode the control plane or broker runs in by default:

The bootstrap token, felix-dev-only-bootstrap-token, is fixed in the file. With FELIX_BOOTSTRAP_DEV_TOKENS on, its holder can mint a token for any principal. The control plane refuses to start with dev tokens unless the bootstrap listener is on loopback, and logs a warning that they are on. The listener is 127.0.0.1:9095 inside the control plane’s container and is not published; felix-dev-auth reaches it only because it shares that container’s network namespace.
The broker’s node token and refresh token (node.jwt, node-refresh.jwt) carry node.view:cluster:*, enough to read the catalog. The broker renews its own token with the refresh token.
The client token (client.jwt) can publish and subscribe in tenant dev, and the control-plane token (controlplane.jwt) can manage its namespaces and streams. Both last a day (FELIX_EXCHANGE_TOKEN_TTL_SECONDS=86400), so a copy on the host outlives a working session. After that, docker compose down -v and up mint new ones.

The tokens are signed by keys the in-memory control plane generates at start and verified only by this stack, so they mean nothing anywhere else. Never use this file, or FELIX_BOOTSTRAP_DEV_TOKENS, for anything shared. The rest of this page builds a real one.

CI brings the stack up on every change, from the images that change builds, and does one round trip through it (scripts/ci/compose_smoke.sh, which takes CONTAINER_ENGINE=podman). A release bumps the default tag in compose.yaml along with the other pins scripts/check_release_version.py checks.

What a broker needs

A broker cannot run on its own. Before writing a compose file, know what it expects:

FELIX_CONTROLPLANE_URL. Required. The broker fetches the keys that verify client tokens from the control plane, and it exits at startup without the URL:
```
Error: FELIX_CONTROLPLANE_URL must be set for auth
```
A node credential, as FELIX_NODE_TOKEN_FILE (or FELIX_NODE_TOKEN). The control plane’s metadata feeds (tenants, namespaces, streams, caches) answer only a Felix token carrying node.view:cluster:*. Without one the broker starts but never learns that any stream exists. See Broker credential.
A storage directory, if you want durable streams. With FELIX_DURABLE_STORAGE_DIR unset the broker keeps nothing on disk. The image declares /var/lib/felix as a volume for this. Point the variable at it.

The images run as uid and gid 65532. A named volume inherits the right ownership from the image. A bind mount must be writable by that uid.

Broker and control plane

A control plane over Postgres and one durable broker:

services:
  postgres:
    image: docker.io/library/postgres:16-alpine
    environment:
      POSTGRES_USER: felix
      POSTGRES_PASSWORD: felix
      POSTGRES_DB: felix
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U felix -d felix"]
      interval: 5s
      timeout: 3s
      retries: 5

  felix-controlplane:
    image: ghcr.io/getfelix/felix-controlplane:0.6.0
    environment:
      - FELIX_CONTROLPLANE_POSTGRES_URL=postgres://felix:felix@postgres:5432/felix
      # Day 0 only: the bootstrap API is how the broker credential is made.
      # Remove these three once it exists.
      - FELIX_BOOTSTRAP_ENABLED=true
      - FELIX_BOOTSTRAP_BIND_ADDR=0.0.0.0:9095
      - FELIX_BOOTSTRAP_TOKEN=change-me
      - RUST_LOG=info
    ports:
      - "8443:8443"            # REST API
      - "127.0.0.1:9095:9095"  # bootstrap, loopback only
    depends_on:
      postgres:
        condition: service_healthy

  felix-broker:
    image: ghcr.io/getfelix/felix-broker:0.6.0
    environment:
      - FELIX_CONTROLPLANE_URL=http://felix-controlplane:8443
      - FELIX_NODE_TOKEN_FILE=/run/secrets/felix-node-token
      - FELIX_DURABLE_STORAGE_DIR=/var/lib/felix
      - RUST_LOG=info
    ports:
      - "5000-5003:5000-5003/udp"  # client QUIC, one port per listener
      - "8080:8080"      # metrics, /live, /ready
    volumes:
      - felix-data:/var/lib/felix
    secrets:
      - felix-node-token
    depends_on:
      felix-controlplane:
        condition: service_healthy
    restart: unless-stopped

secrets:
  felix-node-token:
    file: ./felix-node-token

volumes:
  felix-data:

To run your own build, replace an image: line with build: { context: ., dockerfile: docker/broker.Dockerfile } (or the control-plane Dockerfile).

Both images carry a HEALTHCHECK against /ready on port 8080, so service_healthy works without declaring one. A durable broker’s /ready answers 503 until it has synced its catalog from the control plane, so a broker whose credential is missing or wrong stays unhealthy.

Start it and watch the broker come up:

docker compose up -d
docker compose logs -f felix-broker
curl http://localhost:8080/ready

Broker credential

The credential is a Felix token carrying node.view:cluster:*. Cluster scope cannot be granted by a tenant admin, so it comes out of bootstrap: initialize a tenant with a broker role, assign the broker’s principal to it, then exchange an IdP token for that principal.

curl -sS -X POST http://127.0.0.1:9095/internal/bootstrap/tenants/ops/initialize \
  -H 'X-Felix-Bootstrap-Token: change-me' \
  -H 'Content-Type: application/json' -d '{
    "display_name": "Operations",
    "idp_issuers": [ ... ],
    "initial_admin_principals": ["p:admin"],
    "policies": [
      { "subject": "role:broker", "object": "cluster:*", "action": "node.view" }
    ],
    "groupings": [
      { "user": "p:broker", "role": "role:broker" }
    ]
  }'

The token exchange needs an identity provider that the tenant trusts (idp_issuers). For a development stack with no identity provider, set FELIX_BOOTSTRAP_DEV_TOKENS=true and ask the bootstrap listener for a development token for p:broker with "audience": "felix-controlplane" instead. Otherwise exchange with "audience": "felix-controlplane" and write the Felix token to ./felix-node-token. The bootstrap flow and token exchange cover the request bodies.

The token expires like any Felix token, after 15 minutes for an exchanged one. Every broker, this one included, re-reads FELIX_NODE_TOKEN_FILE every 30 seconds, so whatever mints the credential can rewrite the file. Or give the broker a refresh token at FELIX_NODE_REFRESH_TOKEN_FILE (a writable path) and it renews its own token before it expires; see Kubernetes.

To try Felix without any of this, compose.yaml mints development credentials for you (see Try it in one command), and task cluster:up starts a control plane and brokers on your machine from source. See Local development.

Adding Prometheus

The repository’s docker/prometheus/prometheus.yml scrapes felix-broker:8080 and felix-controlplane:8080, the service names used above. Add a service that mounts it:

services:
  prometheus:
    image: docker.io/prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    depends_on:
      - felix-broker

volumes:
  prometheus-data:

The file also lists an otel-collector:8889 target, which stays down unless you run a collector. The broker exports traces over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; see Observability.

Some queries to start from, at http://localhost:9090:

# Publish rate (a relayed publish is also counted as forwarded)
rate(felix_publish_requests_total{result!="forwarded"}[1m])

# Publish failures, by what went wrong: `error`, `not_owner`, `unroutable`,
# `dropped`. The same counter carries the successes, under `ok` and
# `accepted`; `forwarded` is counted as well when a broker relays a publish.
rate(felix_publish_requests_total{result=~"error|not_owner|unroutable|dropped"}[1m])

felix_publish_latency_ms exists only in a broker built with --features telemetry. The release images are built without it. Observability lists the rest of the metrics.

Configuration

Environment variables

services:
  felix-broker:
    environment:
      # Network
      - FELIX_QUIC_BIND=0.0.0.0:5000
      - FELIX_BROKER_METRICS_BIND=0.0.0.0:8080

      # Control plane
      - FELIX_CONTROLPLANE_URL=http://felix-controlplane:8443
      - FELIX_CONTROLPLANE_SYNC_INTERVAL_MS=2000
      - FELIX_NODE_TOKEN_FILE=/run/secrets/felix-node-token

      # Publishing
      # Durable streams ack after commit by default. FELIX_ACK_ON_COMMIT=0 acks
      # on accept instead: faster, but an acked record can be lost in a crash.
      - FELIX_MAX_FRAME_BYTES=16777216
      - FELIX_PUBLISH_QUEUE_WAIT_MS=2000

      # Event batching
      - FELIX_EVENT_BATCH_MAX_EVENTS=64
      - FELIX_EVENT_BATCH_MAX_BYTES=262144
      - FELIX_EVENT_BATCH_MAX_DELAY_US=250
      - FELIX_FANOUT_BATCH=64

      # Client listener flow control (defaults follow the publish budget)
      - FELIX_BROKER_PUB_CONN_RECV_WINDOW=16777216
      - FELIX_BROKER_PUB_STREAM_RECV_WINDOW=16777216

      - RUST_LOG=info

The broker warns at startup about any FELIX_* variable it does not read, so a typo shows up in the logs. The environment reference lists them all.

Config file

The same kind of settings as YAML:

quic_bind: "0.0.0.0:5000"
metrics_bind: "0.0.0.0:8080"
event_batch_max_events: 64
event_batch_max_delay_us: 250
pub_conn_recv_window: 16777216

services:
  felix-broker:
    volumes:
      - ./config/broker.yml:/etc/felix/broker.yml:ro
    environment:
      - FELIX_BROKER_CONFIG=/etc/felix/broker.yml

The file is applied over the environment. An unknown key fails startup.

More than one broker

A broker that joins a cluster needs more than the single broker above:

FELIX_NODE_ID, unique per broker.
FELIX_NODE_ADVERTISE_ADDR, the internal address other brokers dial, as IP:port. A hostname is refused, so under Compose each broker needs a fixed IP on the network.
FELIX_INTERNAL_BIND for the broker-to-broker listener (default 0.0.0.0:5001).
Peer mTLS (FELIX_INTERNAL_TLS_CERT, FELIX_INTERNAL_TLS_KEY, FELIX_INTERNAL_TLS_CA), or FELIX_INTERNAL_ALLOW_UNAUTHENTICATED=true when only brokers can reach the internal port. A broker with a node id refuses to start with neither.
A credential that may also register the node (node.manage).
FELIX_CLIENT_ADVERTISE_ADDR, the address clients are sent to, when that differs from the bind address.

That is a lot to write by hand in Compose. For a local cluster, task cluster:up starts a control plane and three brokers wired this way (see Local development). For a real one, the Helm chart renders all of it.

Persistence

FELIX_DURABLE_STORAGE_DIR=/var/lib/felix with a volume at that path, as in the example above, is all persistence needs. Logs go to stdout. To use a host directory instead of a named volume, make it writable by uid 65532 first:

sudo chown 65532:65532 /path/to/host/data
# Rootless Podman maps container uids, so change it from inside its namespace:
podman unshare chown 65532:65532 /path/to/host/data

services:
  felix-broker:
    volumes:
      - /path/to/host/data:/var/lib/felix

Backups

A tar of a running broker’s volume is not a consistent backup: it is taken at a different moment from every other broker’s, and it can hold records no majority acknowledged. Take a backup point and copy the leaders’ shard directories against it instead; see Backup and restore. A tar of a stopped single broker’s volume is fine.

Networking

Compose puts every service on one bridge network, where services reach each other by service name. Clients outside Docker reach the broker on the published UDP port.

Host networking avoids Docker’s port forwarding:

services:
  felix-broker:
    network_mode: host
    environment:
      - FELIX_QUIC_BIND=0.0.0.0:5000

Resource limits

services:
  felix-broker:
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 4G
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

Health checks

The images already probe /ready. To change the timing:

services:
  felix-broker:
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/ready"]
      interval: 10s
      timeout: 2s
      retries: 6
      start_period: 10s

Use /ready, not /live. /ready turns 503 when the broker starts draining, while /live keeps answering until the process exits.

Troubleshooting

The broker exits at startup

docker compose logs felix-broker

The last line says why. FELIX_CONTROLPLANE_URL must be set for auth means the variable is missing. With a bind mount, check that uid 65532 can write it.

The broker never turns healthy

It has not synced its catalog. Check that the control plane is reachable from the broker’s container and that the credential file holds a valid token:

docker compose exec felix-broker wget -qO- http://felix-controlplane:8443/v1/system/live
docker compose logs felix-broker | grep -i -e warn -e error

Port conflicts

port is already allocated means another process holds a host port. Change the host side of the mapping. Clients dial the broker’s other listeners on the ports it advertises, which are its container ports, so a remapped broker needs one listener:

environment:
  - FELIX_QUIC_LISTENERS=1
ports:
  - "5001:5000/udp"
  - "8081:8080"

Inspecting a container

docker compose exec felix-broker /bin/sh
docker compose top felix-broker
docker stats
docker inspect "$(docker compose ps -q felix-broker)"

Next steps

Production deployment: Kubernetes Guide
Performance tuning: Performance Guide
Full configuration reference: Configuration Reference
Monitoring setup: Observability Guide