token.nu

packages/swarm-mcp/src/token.nu — cluster identity + authenticity from one shared --token.

Every node in a cluster is launched with the same --token. The token does two things, both derived deterministically so no node has to coordinate:

1. ISOLATION — token_group_id hashes the token into the 32-byte relay

multicast group id. Clusters with different tokens join different relay groups, so their HELLO/job traffic is mutually invisible even when they share one relay. Without the token you cannot derive the group id, so a stranger cannot even see (let alone join) the cluster's gossip.

1. AUTHENTICITY — token_key derives a 32-byte HMAC key. Every compute

payload (a kernel chunk) and every result is tagged with HMAC-SHA256 (token_tag) and verified on receipt (token_untag, constant-time). A worker only acts on token-authentic jobs; a coordinator only accepts token-authentic results. Combined with group isolation this gives mutual authentication of all compute traffic over the dumb (opaque) relay.

The relay itself never needs the token — it forwards opaque bytes — so the secret never leaves the nodes that own it.

API

@ token_tag_len → i

HMAC tag length prepended to authenticated messages (truncated SHA-256 MAC).

@ token_group_id s token → ( Vec u )

32-byte relay multicast group id for this cluster (token isolation).

@ token_key s token → ( Vec u )

32-byte HMAC key authenticating compute payloads + results.

@ token_tag ( Vec u ) key ( Vec u ) payload → ( Vec u )

Prepend a 16-byte HMAC-SHA256 tag to payload. Returns a NEW vec; the caller owns it (and still owns payload).

@ token_untag ( Vec u ) key ( Vec u ) tagged → ?( Vec u )

Verify the tag and strip it. Some(payload copy) on a valid tag, None on a missing/short/forged tag. The compare is constant-time. Caller owns the returned payload on the Some arm.

---

census.nu

packages/swarm/src/census.nu — lightweight membership gossip for the swarm.

The compute layer (dist/job over dist/ring) needs every node to agree on the live worker set: a worker must know it owns a key, and the coordinator must know which pubkeys to route to. The full SWIM table (net/membership.nu) is the heavy, churn-hardened answer; this is the small one a compute cluster actually needs — a HELLO announcement that feeds the consistent-hash ring.

• a node joins by broadcasting HELLO(role, want=1) to the relay group;
• everyone who hears it adds the node to their ring (workers only) and, if

want, replies with their own HELLO so the newcomer learns them too;

• the ring converges, and "join the cluster" is literally "run the binary".

Roles: a WORKER owns keys and executes handlers, so it joins everyone's ring; a CLIENT (the coordinator) only submits, so it is never added to the ring — otherwise it would own keys it has no handler for and silently drop results.

The codec and the membership set are pure; only the pump touches transport.

API

@ census_hello_t → i

@ role_client → i

@ role_worker → i

@ hello_build i id i role i want ( Vec u ) pubkey → ( Vec u )

HELLO wire: [3][id:8][role:1][want:1][pklen:2][pubkey…]

: Hello { i id i role i want ( Vec u ) pubkey }

@ hello_free Hello h → v

@ hello_decode ( Vec u ) buf → Hello

: Member { ( Vec u ) pubkey i id }

: Roster { ( Vec s ) members } // *Member

@ roster_new → *Roster

@ roster_free * Roster r → v

@ roster_has * Roster r ( Vec u ) pubkey → b

@ roster_count * Roster r → i

@ roster_add * Roster r * Ring ring ( Vec u ) pubkey i id i vnodes → b

Fold a worker into the roster + ring, once. Returns T if newly added.

---

work.nu

packages/swarm-mcp/src/work.nu — the distributed map-reduce the cluster runs.

A task is: evaluate an expression kernel (expr.nu) over an integer range [lo, hi) and fold the results with a reduce op. The coordinator shards the range; dist/ring routes each chunk to its owning worker; the worker parses the kernel once and folds it over its sub-range; the coordinator combines the partial folds. Every reduce op is associative, so sharding is exact.

reduce op : 0 sum · 1 product · 2 min · 3 max · 4 count (of truthy map) chunk : [op:1][lo:8 BE][hi:8 BE][expr bytes…] result : [value:8 BE]

API

@ red_sum → i

@ red_product → i

@ red_min → i

@ red_max → i

@ red_count → i

@ red_id i op → i

The identity (empty-range) value for a reduce op.

@ red_fold i op i acc i v → i

Fold one mapped value into the accumulator (per element).

@ red_combine i op i a i b → i

Combine two partial folds (across chunks). count combines by summing the per-chunk counts; every other op combines like its element fold.

@ chunk_payload i op i lo i hi ( Vec u ) expr → ( Vec u )

@ result_encode i value → ( Vec u )

@ result_decode ( Vec u ) p → i

@ kernel_handler ( Vec u ) key → ( @ ( Vec u ) ( Vec u ) )

: Chunk { i lo i hi }

@ shard i lo i hi i n → ( Vec s )

@ shard_free ( Vec s ) chunks → v

@ chunk_key i idx → ( Vec u )

A ring key for chunk index i: distinct chunks hash to distinct ring points.

@ reduce_op_of s name i fallback → i

True for a recognised reduce-op name; sets *op. Keeps the MCP layer thin.

@ reduce_op_name i op → s

---

expr.nu

packages/swarm-mcp/src/expr.nu — the phase-1 "kernel": a small, regular integer-expression language the cluster evaluates per element.

A workload is a map-reduce: the coordinator ships an expression in one variable x plus a range and a reduce op; each worker parses the expression once, evaluates it for every x in its sub-range, and folds the results. The language is deliberately small and regular so a language model can write it without docs — and it is the natural precursor to phase 2, where the kernel is arbitrary NURL compiled to wasm instead of interpreted here.

expr := ternary ternary := logic ( '?' expr ':' ternary )? logic := compare ( ('&'|'|') compare ) compare := addsub ( ('<'|'<='|'>'|'>='|'=='|'!=') addsub )? addsub := muldiv ( ('+'|'-') muldiv ) muldiv := unary ( (''|'/'|'%') unary ) unary := '-' unary | primary primary := INT | 'x' | '(' expr ')' | ('min'|'max') '(' expr ',' expr ')' | 'abs' '(' expr ')'

All arithmetic is i64 (truncated division; division/mod by zero yields 0). Comparisons and '&' '|' yield 0/1; any non-zero is "true".

Tokens are kept in two parallel int vectors; the parser builds a flat stride-4 node arena (tag,a,b,c) and returns the root index — the resp.nu arena pattern, so nesting needs no per-node allocation.

API

@ expr_tokenize ( Vec u ) src ( Vec i ) tk ( Vec i ) tv → b

: EParser

: EParser {
    ( Vec i ) tk
    ( Vec i ) tv
    i pos
    ( Vec i ) arena  // stride-4: tag,a,b,c
    b ok
}

@ expr_parse ( Vec u ) src * EParser p → i

Parse src → (arena, root). On any error, ok=0; the caller checks it. The arena is returned via the EParser; the root index via the return value.

@ eparser_free * EParser p → v

Frees the parser's arena/token vectors AND the struct itself.

@ expr_eval * EParser p i node i x → i

---

buildwasm.nu

packages/swarm-mcp/src/buildwasm.nu — compile NURL source to a wasm module via the NURL build API, so the MCP server itself can accept a kernel as source (compute_submit_kernel) instead of requiring the caller to pre-compile.

POST {source, filename} to <NURL_BUILD_API>/build_wasm. A direct nurlapi can answer with the raw wasm bytes; the public playground proxy answers with JSON carrying wasm_base64 (and nurlc_errors on failure). Both are handled.

$NURL_BUILD_API build service base URL (default https://play.nurl-lang.org)

API

@ build_api_url → String

@ wrap_kernel s source i op → String

Wrap a bare kernel into a complete wasm-ready program for reduce op op.

@ compile_to_wasm s source → !( Vec u ) String

Compile NURL source to a wasm module. Ok = module bytes; Err = a human-readable transport/compile error (suitable to hand back to the model).

---

main.nu

packages/swarm-mcp/src/main.nu — swarm-mcp: an MCP-controlled distributed compute engine. A language model sets a workload over MCP — an expression kernel in x plus a range and a reduce op — and the cluster evaluates it distributed; the model reads running tasks and finished results back.

swarm-mcp relay 0.0.0.0 47700 [--v] # the meeting point (one per cluster) swarm-mcp worker <host> <port> # join as a compute node swarm-mcp mcp <host> <port> # MCP server (stdio) → drives the cluster swarm-mcp submit <host> <port> <reduce> <lo> <hi> <expr> # manual CLI submit

The cluster layer (membership → ring → job dispatch) is the swarm package's; here every worker registers ONE generic kernel handler (work.nu) that interprets the submitted expression (expr.nu), so any normal-operation workload runs without recompiling a worker. Phase 2 swaps the interpreter for a NURL→wasm kernel; the protocol and MCP surface stay the same.

API

@ swarm_vnodes → i

@ kind_kernel → i

@ pk_from_id i id → ( Vec u )

A 32-byte opaque routing pubkey derived deterministically from a node id.

: Swarm

: Swarm {
    s transport  // *Transport
    s ring  // *Ring
    s roster  // *Roster
    s job  // *JobNode
    ( Vec u ) self_pk
    i self_id
    i role
    ( Vec u ) group  // token-derived 32-byte relay multicast group (cluster isolation)
    ( Vec u ) key  // token-derived HMAC key (compute authenticity)
}

@ swarm_new RelayClient rc i id i role s token → *Swarm

@ swarm_free * Swarm sw → v

@ swarm_join_group * Swarm sw → v

@ swarm_announce * Swarm sw i want → v

@ swarm_on_hello * Swarm sw Hello h → v

@ swarm_pump * Swarm sw i max → v

@ swarm_discover * Swarm sw i rounds → v

@ relay_dial_retry s host i port i tries → !RelayClient NetErr

Dial the relay, retrying briefly so a co-located relay that is still binding (or a relay that restarts) doesn't lose its local roles to a startup race.

@ node_relay s host i port i vflag → v

RELAY role: the dumb rendezvous point. Owns the fiber reactor on this thread.

@ node_worker s host i port s token i vflag → v

WORKER role: join the cluster, register the kernel + wasm handlers (every worker can run wasm modules), and drain cluster traffic forever. Each worker thread takes a fresh random identity, so --workers N spins up N independent ring members in one process.

@ cluster_submit * Swarm sw i op i lo i hi ( Vec u ) expr i nchunks → ( Vec i )

@ cluster_submit_wasm * Swarm sw i lo i hi ( Vec u ) wasm i nchunks → ( Vec i )

Shard a wasm-kernel task: ship the compiled module + each sub-range to its ring owner under kind_wasm. The module bytes ride every chunk (workers cache by content hash, so it is written once per worker).

@ tids_ready * Swarm sw ( Vec i ) tids → b

All chunk results present? (cluster job results are recorded by task-id.)

@ tids_combine * Swarm sw i op ( Vec i ) tids → i

Combine all chunk results with the reduce op (assumes ready).

@ nchunks_for i nworkers → i

@ run_submit s host i port i op i lo i hi s expr s token → i

@ nchunks_wasm i nworkers → i

Chunk count for a wasm task: fewer chunks than the expression path (each chunk ships the module and spawns a wasmtime process), but enough to spread across the ring.

@ run_runwasm s host i port i op i lo i hi s wasmpath s token → i

: Task

: Task {
    i id
    ( Vec u ) expr  // kernel bytes
    i lo
    i hi
    i op
    i nchunks
    ( Vec i ) tids
    i done
    i result
}

: McpState

: McpState {
    s swarm  // *Swarm coordinator
    ( Vec s ) tasks  // *Task
    i next_id
}

@ task_new i id ( Vec u ) expr i lo i hi i op i nchunks ( Vec i ) tids → *Task

Constructor. The params share the field names (lo, hi, …); = . t lo lo is a field store — the field-store/local-shadow miscompile this used to work around was fixed in the compiler (NURL v0.10.4).

: ~ i g_mcp 0

@ mcp_swarm → *Swarm

@ mcp_pump i rounds → v

@ task_refresh * Task t → v

Refresh one task's status; combine if every chunk has landed.

@ task_find i id → s

@ task_to_json * Task t → Json

Build the JSON-as-text result body for one task (LLM-readable + parseable).

@ tool_result_json Json o → Json

@ tool_submit Json args → Json

@ tool_run_wasm Json args → Json

@ tool_submit_kernel Json args → Json

compute_submit_kernel: the server compiles NURL source → wasm itself (via the build API), then runs it — the model hands over a kernel as plain NURL.

@ tool_list Json args → Json

@ tool_result Json args → Json

@ ms_prop Json props s name s ty s desc → v

@ ms_schema_submit → Json

@ ms_schema_result → Json

@ ms_schema_empty → Json

@ ms_schema_run_wasm → Json

@ ms_schema_submit_kernel → Json

@ build_tools_list → ( Vec Json )

@ dispatch_tool s name Json args → Json

@ handle_initialize Json id → Json

@ handle_ping Json id → Json

@ handle_tools_list Json id → Json

@ handle_tools_call Json id Json params → Json

@ handle_unknown Json id s method → Json

@ dispatch Json req → ?Json

@ node_mcp s rhost i rport s mcp_host i mcp_port s cert s key s token → v

@ arg_int i idx → i

@ arg_eq i idx s lit → b

@ has_flag s name → b

True if name appears anywhere in argv (a boolean role/verbose flag).

@ flag_val s name s deflt → String

The token following --name in argv, or a copy of deflt. Owned String.

@ flag_int s name i deflt → i

: HostPort { String host i port }

@ parse_hostport String hp s dhost i dport → HostPort

Parse "host:port", "host", ":port", or "" with host/port defaults.

@ ensure_cert s cert_path s key_path → b

Ensure a TLS cert+key exist at the given paths, auto-minting a self-signed EC P-256 pair with openssl if absent (so --mcp works out of the box; pass --tls-cert/--tls-key for a real cert). Returns T once both files exist.

@ cli_submit → i

@ cli_runwasm → i

@ usage → v

@ run_node → i

@ main → i

---

wasmkernel.nu

packages/swarm-mcp/src/wasmkernel.nu — phase 2: run a NURL-compiled wasm kernel over a chunk.

The language model writes a kernel as ordinary NURL — anything, not just an expression — and compiles it (via the NURL build service) to a wasm32-wasi module whose main reads lo hi from argv, folds the kernel over [lo, hi), and prints the partial. The coordinator ships that module plus a sub-range to each worker; the worker runs it under wasmtime and returns the partial, which the coordinator combines with the reduce op exactly as in phase 1.

chunk : [lo:8 BE][hi:8 BE][wasm module bytes…] (the reduce op is baked into the module by the wrapper, so the partial is final per chunk)

The module is cached on each worker by a content hash, so re-running the same kernel (every chunk of a task, and across tasks) writes the .wasm only once.

API

@ kind_wasm → i

@ wasm_chunk_payload i lo i hi ( Vec u ) wasm → ( Vec u )

@ wasm_handler ( Vec u ) key → ( @ ( Vec u ) ( Vec u ) )

The worker handler for a wasm chunk: verify the cluster HMAC tag, cache the module by content hash, run it under wasmtime, return the partial tagged for the coordinator. An untrusted/forged payload yields a tagged zero — a stranger can't make a worker fetch-and-run an arbitrary module. key is the cluster HMAC key (token_key), captured by the handler closure.