Fleets & Multi-Host

What is a fleet?

A fleet is the set of hosts resolved together in a single pipeline run. All hosts declared in a Den flake form a fleet automatically and share one scope tree. Hosts of the same system are sibling scopes, which pipe.collect reaches across; pipe.collectAll reaches hosts on every system.

The scope tree

Every entity in the pipeline gets a scope. Scopes form a tree rooted at the flake:

flowchart TD
  flake["flake (root)"]
  flake --> fs["flake-system {system=x86_64-linux}"]
  fs --> igloo["host {host=igloo}"]
  fs --> iceberg["host {host=iceberg}"]
  igloo --> tux["user {host=igloo, user=tux}"]
  iceberg --> alice["user {host=iceberg, user=alice}"]

The built-in flake policies drive this tree:

Policy	Effect
`flake-to-systems`	Creates a `flake-system` scope per system in `den.systems`
`system-to-os-outputs`	Creates a `host` scope per host, plus an instantiate request
`system-to-hm-outputs`	Creates a `home` scope per standalone home
`host-to-users`	Creates a `user` scope per user on each host

All hosts of the same system are siblings — they share the same flake-system parent scope. This is the key relationship that enables pipe.collect. Hosts on different systems have different parents, so they are not siblings: an x86_64-linux host and an aarch64-linux host cannot see each other through pipe.collect. See Across systems.

Cross-host data flow

The simplest cross-host pattern: every host produces some data, and every host (or a specific host) consumes the aggregated result.

Example: load balancer backends

# Declare the quirk
den.quirks.http-backends = {
  description = "HTTP backend endpoints";
};

# Each host produces its backend info
den.aspects.iceberg = {
  http-backends = { addr = "10.0.0.2"; port = 80; };
};
den.aspects.igloo = {
  http-backends = { addr = "10.0.0.1"; port = 8080; };
};

# Collect policy: each host sees all hosts' backends
den.policies.fleet-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collect ({ host, ... }: true))
    ])
  ];

den.schema.host.includes = [ den.policies.fleet-backends ];

# Consumer aspect
den.aspects.haproxy = {
  nixos = { http-backends, lib, ... }: {
    services.haproxy.config = lib.concatMapStringsSep "\n"
      (b: "server ${b.addr} ${b.addr}:${toString b.port}")
      http-backends;
  };
};

den.aspects.igloo.includes = [ den.aspects.haproxy ];

The igloo host sees http-backends containing entries from both itself and iceberg.

How `pipe.collect` works

pipe.collect receives a predicate that filters sibling scopes. Siblings are scopes sharing the same parent in the scope tree. The predicate receives each sibling’s context attrset.

# Collect from all host siblings
pipe.collect ({ host, ... }: true)

# Collect only from hosts named "web-*"
pipe.collect ({ host, ... }: lib.hasPrefix "web-" host.name)

Entity kind filtering is automatic: required args in the predicate (host in these examples) are checked against schema entity kinds. Only sibling scopes with matching entity kinds are considered — user scopes won’t match a { host, ... } predicate.

Filtering collected data

Transform stages after pipe.collect operate on the combined pool:

pipe.from "http-backends" [
  (pipe.collect ({ host, ... }: true))
  (pipe.filter (b: b.port != 8080))
]

Tracking provenance

Use pipe.withProvenance to know which host each entry came from:

den.policies.fleet-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collect ({ host, ... }: true))
      pipe.withProvenance
    ])
  ];

Consumers receive { value; source; } records where source is the full context attrset of the originating scope:

den.aspects.haproxy = {
  nixos = { http-backends, lib, ... }: {
    services.haproxy.config = lib.concatMapStringsSep "\n"
      (entry: "server ${entry.source.host.name} ${entry.value.addr}:${toString entry.value.port}")
      http-backends;
  };
};

Across systems: `pipe.collectAll`

pipe.collect stops at the flake-system boundary. With hosts on two systems:

flowchart TD
  flake["flake (root)"]
  flake --> x86["flake-system {system=x86_64-linux}"]
  flake --> arm["flake-system {system=aarch64-linux}"]
  x86 --> igloo["host {host=igloo}"]
  arm --> iceberg["host {host=iceberg}"]

igloo and iceberg are not siblings, so a pipe.collect on igloo sees only its own entry. pipe.collectAll takes the same predicate but matches scopes anywhere in the tree:

den.hosts.x86_64-linux.igloo.users.tux = {};
den.hosts.aarch64-linux.iceberg.users.alice = {};

den.policies.fleet-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collectAll ({ host, ... }: true))
    ])
  ];

Now igloo sees entries from both hosts. Entity kind filtering, filter stages, and pipe.withProvenance work the same as with pipe.collect.

pipe.broadcast reaches across systems too, pushing from the senders instead of pulling at the receivers; pipe.expose does not. See Crossing system boundaries for when to use each.

Bridging specific systems

The predicate receives each candidate’s context, and every host carries its system, so selecting by host.system bridges chosen flake-system subtrees and leaves the rest out:

den.hosts.x86_64-linux.igloo.users.tux = {};
den.hosts.aarch64-linux.iceberg.users.alice = {};
den.hosts.aarch64-darwin.floe.users.bob = {};

den.policies.linux-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collectAll ({ host, ... }:
        builtins.elem host.system [ "x86_64-linux" "aarch64-linux" ]))
    ])
  ];

igloo and iceberg see each other’s backends; floe is excluded. Any host attribute works the same way, so the bridge can follow tags or environments instead of systems. To make hosts siblings structurally rather than per policy, see Fleet entity grouping.

Config-dependent cross-host data

Quirk values can depend on a host’s evaluated NixOS config. For cross-host collection, these thunks are resolved eagerly against the source host’s instantiated config:

den.aspects.webserver = {
  nixos.services.nginx.enable = true;
  http-backends = { config, host, ... }: {
    addr = host.name;
    port = config.services.nginx.defaultHTTPListenPort;
  };
};

When igloo collects this via pipe.collect, the thunk runs against iceberg’s evaluated NixOS config, producing the correct port value.

Fleet entity grouping

By default, all hosts of the same system are siblings under flake-system. If you need a different grouping — for example, splitting hosts into production and staging fleets — you can define a fleet entity:

# Define a fleet policy that groups hosts under a parent scope
den.policies.to-fleet = _: [
  (den.lib.policy.resolve.to "fleet" {
    fleet = { name = "production"; };
  })
];

den.policies.fleet-to-hosts = { fleet, ... }:
  lib.concatMap (system:
    lib.concatMap (hostName:
      let host = den.hosts.${system}.${hostName}; in
      [ (den.lib.policy.resolve.to "host" { inherit host; })
        (den.lib.policy.instantiate host)
      ]
    ) (builtins.attrNames (den.hosts.${system} or {}))
  ) (builtins.attrNames (den.hosts or {}));

den.schema.flake.includes = [ den.policies.to-fleet ];
den.schema.fleet.includes = [ den.policies.fleet-to-hosts ];

This creates the scope tree:

flowchart TD
  flake["flake"]
  flake --> fleet["fleet {fleet=production}"]
  fleet --> igloo["host {host=igloo}"]
  fleet --> iceberg["host {host=iceberg}"]

Now hosts are siblings under the fleet scope rather than under flake-system, and pipe.collect sees them as peers.

Upward flow: user data to host

User-scope data reaches the host via pipe.expose. This is orthogonal to fleet patterns but composes with them:

# User produces preferences
den.aspects.tux = {
  prefs = [{ editor = "vim"; }];
};

# Expose policy pushes user data to host scope
den.policies.expose-prefs = { host, user, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "prefs" [ pipe.expose ]) ];

den.default.includes = [ den.policies.expose-prefs ];

# Host consumer sees all users' preferences
den.aspects.host-config = {
  nixos = { prefs, lib, ... }: {
    environment.etc."editors.txt".text =
      lib.concatMapStringsSep "\n" (p: p.editor) prefs;
  };
};

Exposed and local data merge: if the host also emits prefs, consumers see both.

Walk-then-instantiate

All hosts resolve in a single pipeline walk. After the walk completes, per-host assembly extracts each host’s subtree from the shared state:

Walk — the pipeline walks all entities (flake → systems → hosts → users), collecting aspects, pipe data, routes, and instantiate requests into scope-partitioned state.
Pipe assembly — assemblePipes runs post-walk, merging exposed data and applying pipe effects. Cross-host pipe.collect reads directly from sibling scopes’ already-walked data.
Per-host extraction — applyInstantiates filters the shared state to each host’s subtree (the host scope + its descendants + ancestor scopes like flake-system whose routes and provides must remain visible), then re-runs class wrapping, provides, and routes with the host as the root scope.
Instantiation — each extracted subtree is passed to lib.nixosSystem, darwinSystem, or homeManagerConfiguration.

This architecture means pipe.collect always sees all hosts’ data — there’s no walk-order sensitivity.