Cross-Scope Pipes

Source

templates/ci/modules/public-api/pipe-scope.nix — templates/ci/modules/public-api/pipe-broadcast.nix

By default a quirk value stays in the scope that emitted it (see Scoping). This page covers the pipe stages that move data between scopes. It assumes the basics from the Quirks & Pipes guide.

Stage	Direction	Use it for
pipe.expose	child → parent	a user’s data reaching its host
pipe.collect	pull from siblings	hosts in one environment seeing each other
pipe.collectAll	pull from every matching scope	fleet-wide rosters, cluster-level aggregation
pipe.broadcast	push to every matching scope	per-user meshes, hub-and-spoke

flowchart TB
  F["flake"] --> E1["environment: prod"]
  F --> E2["environment: dev"]
  E1 --> H1["host: igloo"]
  E1 --> H2["host: iceberg"]
  E2 --> H3["host: floe"]
  H1 --> U1["user: tux"]
  U1 -- "pipe.expose: up to the parent" --> H1
  H2 -. "pipe.collect: pull from siblings" .-> H1
  H3 -. "pipe.collectAll: pull from anywhere" .-> H1
  U1 == "pipe.broadcast: push to every match" ==> H3
  linkStyle 0,1,2,3,4,5 stroke:#bbb,stroke-width:1px

Grey edges are the scope tree; labelled edges are the pipe stages, each drawn from the scope that holds the data to the scope that reads it.

Upward: pipe.expose

Push child-scope data to the parent. Without this, a user’s emits stay in the user scope (see Scoping). A user’s preferences reach the host:

den.quirks.prefs = { description = "User preferences"; };

den.aspects.tux = {
  prefs = [{ editor = "vim"; }];
};

den.aspects.igloo = {
  includes = [ den.aspects.host-consumer ];
};

den.aspects.host-consumer = {
  nixos = { prefs, lib, ... }: {
    networking.hostName =
      lib.concatMapStringsSep "-" (p: p.editor) prefs;
  };
};

den.policies.expose-prefs = { host, user, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "prefs" [ pipe.expose ]) ];

den.default.includes = [ den.policies.expose-prefs ];

Result: host consumer sees [{ editor = "vim"; }].

Exposed data merges with host-local data. If the host aspect also emits prefs, consumers see both host-local and exposed user entries.

Transform stages run before expose:

pipe.from "items" [
  (pipe.filter (i: i.keep))
  (pipe.transform (i: { label = "x-${i.name}"; }))
  pipe.expose
]

Lateral: pipe.collect

Harvest data from sibling scopes. Siblings are scopes sharing the same parent in the scope tree. This is how you do cross-host aggregation:

den.hosts.x86_64-linux.igloo.users.tux = {};
den.hosts.x86_64-linux.iceberg.users.alice = {};

den.quirks.http-backends = {
  description = "HTTP backends";
};

den.aspects.iceberg = {
  http-backends = { addr = "10.0.0.2"; port = 80; };
};

den.aspects.igloo = {
  includes = [ den.aspects.lb-consumer ];
  http-backends = { addr = "10.0.0.1"; port = 8080; };
};

den.aspects.lb-consumer = {
  nixos = { http-backends, lib, ... }: {
    environment.etc."lb/backends".text =
      lib.concatMapStringsSep "\n" (b: "${b.addr}:${toString b.port}")
        http-backends;
  };
};

den.policies.fleet-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collect ({ host, ... }: true))
    ])
  ];

den.schema.host.includes = [ den.policies.fleet-backends ];

The predicate in pipe.collect filters which sibling scopes to harvest from.

The predicate must name the scope's entity kind

A scope is considered only when its own entity kind is a required argument of the predicate. ({ host, ... }: true) matches host scopes and never user scopes (a user scope also has host in its context, but its kind is user). A bare (_: true) names no kind, so it matches no entity scope at all. To collect from other users, write ({ user, ... }: true).

“Siblings” follows the scope tree, which makes collect a natural boundary. By default, hosts sit under a flake-system scope per system, so a host’s siblings are the hosts on the same system: an x86_64-linux host and an aarch64-linux host never see each other through collect (see Bridging specific systems). If hosts are resolved under a grouping entity (for example an environment kind whose policy fans out to its hosts), a host’s siblings are the hosts in the same environment, so a collect scopes itself to that environment without any predicate logic. Use pipe.collectAll when you want to cross that boundary.

Collect with provenance

Track where collected data came from:

pipe.from "http-backends" [
  (pipe.collect ({ host, ... }: true))
  pipe.withProvenance
]

Consumers receive [{ value = <data>; source = <context>; }] where source is the full context attrset ({ host = ...; }) of the source scope.

Collect with filtering

Stages after pipe.collect operate on the combined pool:

pipe.from "http-backends" [
  (pipe.collect ({ host, ... }: true))
  (pipe.filter (b: b.port != 8080))
]

Fleet-wide pull: pipe.collectAll

pipe.collectAll is pipe.collect without the sibling restriction: it harvests from every matching scope in the fleet. It is the right choice for a record every host needs about every other host (an /etc/hosts roster), and for an aggregating scope that sits above the hosts, such as a cluster or an environment:

den.quirks.host-addrs.description = "Address records for every host";

den.aspects.addr-record.host-addrs =
  { host, ... }: { name = host.name; addr = host.addr; };

# every host sees every host, across environments
den.policies.collect-host-addrs = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "host-addrs" [ (pipe.collectAll ({ host, ... }: true)) ]) ];

den.schema.host.includes = [
  den.aspects.addr-record
  den.policies.collect-host-addrs
];

den.aspects.etc-hosts.nixos = { host-addrs, lib, ... }: {
  networking.extraHosts =
    lib.concatMapStringsSep "\n" (h: "${h.addr} ${h.name}") host-addrs;
};

The same stage works from a non-host scope: a policy taking { cluster, ... } with (pipe.collectAll ({ host, ... }: true)) gives the cluster’s own class modules every host’s records. (host.addr here is a user-defined den.schema.host option.)

Bridging specific systems

The predicate receives each candidate’s context, and every host carries its system, so selecting on host.system joins chosen flake-system subtrees and leaves the rest out:

den.hosts.x86_64-linux.igloo.users.tux = {};
den.hosts.aarch64-linux.iceberg.users.alice = {};
den.hosts.aarch64-darwin.floe.users.bob = {};

den.policies.linux-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collectAll ({ host, ... }:
        builtins.elem host.system [ "x86_64-linux" "aarch64-linux" ]))
    ])
  ];

igloo and iceberg see each other’s backends; floe does not take part. Any host attribute works the same way, so the bridge can follow tags or environments instead of systems.

Fleet-wide: pipe.broadcast

Push a scope’s own pipe value to every other scope matching a predicate — in one stage, in any direction (user→user, user→host, host→user, fleet-wide). The push dual of pipe.expose: instead of one parent receiving, every matching scope receives.

A common shape: every user publishes a device record, and every user’s home sees the whole fleet’s roster:

den.hosts.x86_64-linux.igloo.users.tux = {};
den.hosts.x86_64-linux.iceberg.users.alice = {};

# Both users get a home-manager home (see the Home Manager guide):
den.schema.user.classes = lib.mkDefault [ "homeManager" ];

den.quirks.peer-dev = {
  description = "per-user device records";
};

den.aspects.tux = {
  peer-dev = [ { who = "tux@igloo"; } ];
  homeManager = { peer-dev, lib, ... }: {
    home.sessionVariables.PEERS =
      lib.concatStringsSep "," (map (p: p.who) peer-dev);
  };
};
den.aspects.alice = {
  peer-dev = [ { who = "alice@iceberg"; } ];
  homeManager = { peer-dev, lib, ... }: {
    home.sessionVariables.PEERS =
      lib.concatStringsSep "," (map (p: p.who) peer-dev);
  };
};

# Every user scope broadcasts its own peer-dev to all other user scopes.
den.policies.broadcast-peers = { host, user, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "peer-dev" [ (pipe.broadcast ({ user, ... }: true)) ]) ];

den.schema.user.includes = [ den.policies.broadcast-peers ];

PEERS on both tux’s and alice’s homes now lists both users — each scope sees its own emits plus everything broadcast to it. Transform stages before pipe.broadcast apply to the value as it leaves; config-dependent thunks are resolved against the source host’s config when they cross hosts. Note the asymmetry: a broadcaster pushes its own raw emits only — data exposed up into it (e.g. a host broadcaster’s users’ pipe.expose data) does not ride along.

Crossing system boundaries

Hosts on different systems sit under different flake-system scopes, so pipe.collect never reaches across them. Two stages do:

Method	Who decides	Use when
pipe.collectAll	the receiver picks its sources	one consumer policy aggregates a roster; sources need no policy of their own
pipe.broadcast	the sender picks its receivers	producers control where their data goes, or receivers differ per sender

Both give every host the fleet’s entries when the predicate matches all hosts.

Pull with collectAll, one policy on the receivers:

den.policies.fleet-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.collectAll ({ host, ... }: true))
    ]) ];

Sources can be narrowed with the predicate, as in Bridging specific systems. Collected data includes what each source’s users exposed up into it.

Push with broadcast, one policy on the senders:

den.policies.share-backends = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  [ (pipe.from "http-backends" [
      (pipe.broadcast ({ host, ... }: true))
    ]) ];

The predicate filters receivers, and because it is a closure over the sender’s context it can route per sender (see the recipe below). A broadcast carries the sender’s raw emits only, not data its users exposed into it.

pipe.expose does not bridge systems

Exposing host data up to flake-system and collecting between the system scopes looks like it should join the subtrees, but pipe.expose only moves data up: nothing carries the system-level pool back down to the hosts, so a host consumer still sees only its own system. Use collectAll or broadcast instead.

Recipe: per-user mesh and hub

A broadcast predicate is an ordinary closure, so it can compare the receiving scope against the sending one. This keeps each user’s devices in a mesh of their own (tux’s hosts see tux’s devices, never alice’s):

den.policies.broadcast-sync-peers = { user, ... }:
  let
    inherit (den.lib.policy) pipe;
    srcUser = user.name;
  in
  [ (pipe.from "sync-peers" [
      (pipe.broadcast ({ user, ... }: user.name == srcUser))
    ]) ];
den.schema.user.includes = [ den.policies.broadcast-sync-peers ];

A policy can also decide whether to broadcast at all. Here, only the host marked as the hub advertises itself, and every user scope receives it:

den.policies.broadcast-hub = { host, ... }:
  let inherit (den.lib.policy) pipe; in
  lib.optionals (host.isSyncHub or false) [
    (pipe.from "sync-peers" [ (pipe.broadcast ({ user, ... }: true)) ])
  ];
den.schema.host.includes = [ den.policies.broadcast-hub ];

A transform before the broadcast stamps the sender onto each record, which the receiver would otherwise not know:

(pipe.from "shares" [
  (pipe.transform (e: e // { user = srcUser; }))
  (pipe.broadcast ({ host, ... }: host.isSyncHub or false))
])

See also

• Quirks & Pipes guide — producing, consuming, scoping, transforms
• den.quirks reference — exact semantics of each stage
• Quirk Recipes — cross-cutting aspects built on quirks
• Case Study: Kubernetes — collect vs collectAll from a cluster scope