main.nu

packages/swarm/src/main.nu — swarm: a distributed compute cluster you join by installing it.

nurlpkg install swarm # drops the swarm binary on $PATH swarm relay 0.0.0.0 47700 # the meeting point (one per cluster) swarm worker <host> <port> # join as a compute node — that's the join swarm submit <host> <port> primes 1 1000000 # place a real workload

A worker needs no recompile and no member list: it announces itself over the relay group, every node folds it into the consistent-hash ring (census.nu), and from then on it owns its share of the keyspace and runs the registered handlers (work.nu). The coordinator discovers the live workers the same way, shards a real numeric range across them by key (dist/ring → dist/job), and sums the partial results. Add a worker → it takes load on the next submit.

Built entirely on the standard distributed stack: net/relay (reach), net/transport (the pubkey seam), dist/ring (ownership), dist/job (dispatch).

API

@ swarm_vnodes → i

@ swarm_group_id → ( Vec u )

The relay multicast group every node joins. It is a fixed 32 bytes on purpose: the relay's documented group-id contract is 32 bytes, and a 32-byte id round-trips correctly under every shipped relay framing — so a worker built against any toolchain release still finds the cluster. ("swarm" + zero padding keeps it recognisable on the wire.)

@ pk_from_id i id → ( Vec u )

A 32-byte opaque routing pubkey derived deterministically from a node id. Over the relay leg the pubkey is just the address the relay forwards by, so a spread-out deterministic value is all the routing needs (the real X25519 identity belongs to the direct securedgram leg, not used here).

: SwarmImpl

: SwarmImpl {
    Transport transport
    Ring ring
    Roster roster
    JobNode job
    ( Vec u ) self_pk
    i self_id
    i role
    ( Vec u ) group
}

: Swarm { s ctl }

A Swarm is a handle on the node bundle in an rcbox (stdlib/core/rcbox.nu): every copy is the same node, and the last owner releases it — the transport, ring, roster and job node with it.

@ Swarm_share Swarm h → Swarm

@ Swarm_drop sink Swarm h → v

@ swarm_new RelayClient rc i id i role → Swarm

@ swarm_free sink Swarm sw → v

Let go of sw now rather than at the end of its owner's scope (optional).

@ swarm_worker_count Swarm sw__h → i

How many workers this node has folded into its ring.

@ swarm_join_group Swarm sw__h → v

@ swarm_announce Swarm sw__h i want → v

Announce ourselves to the group. want asks hearers to reply so a newcomer learns the existing members.

@ swarm_on_hello Swarm sw__h Hello h → v

@ swarm_pump Swarm sw__h i max → v

Drain inbound transport messages, dispatching census HELLO and job traffic.

@ run_relay s host i port i vflag → i

@ swarm_register_handlers Swarm sw__h → v

@ run_worker s host i port i id i rounds → i

@ swarm_discover Swarm sw i rounds → v

Discover the live workers: announce, then pump a short window collecting the HELLO replies that fold workers into the ring.

@ run_submit s host i port i kind i lo i hi → i

@ usage → v

@ arg_int i idx → i

@ arg_eq i idx s lit → b

@ arg_is_verbose i idx → b

True if arg idx is the verbose flag in either spelling.

@ main → i

---

work.nu

packages/swarm/src/work.nu — the real workloads the cluster runs.

A workload is a pure function over a numeric range that the coordinator shards into many independent chunks; dist/ring routes each chunk to its owning worker, the worker runs the registered handler, and the coordinator sums the partial results. The work is genuine CPU — primes are counted by trial division, not looked up — so a bigger range is a bigger real load, and adding workers measurably shares it.

Two kinds, both verifiable by a closed form so a test can pin the answer:

• PRIMES count of primes in [lo, hi) (π(10^6) = 78498)
• SUMSQ Σ k² for k in [lo, hi), wrapping i64 (n(n+1)(2n+1)/6 closed form)

chunk payload : [lo:8 BE][hi:8 BE] result : [value:8 BE]

API

@ kind_primes → i

@ kind_sumsq → i

@ is_prime i n → b

Is n prime? Trial division to √n — deliberately the honest O(√n) work.

@ count_primes i lo i hi → i

Count primes in [lo, hi).

@ sum_squares i lo i hi → i

Σ k² for k in [lo, hi), wrapping (matches the i64 the closed form is taken mod).

@ chunk_payload i lo i hi → ( Vec u )

@ chunk_lo ( Vec u ) p → i

@ chunk_hi ( Vec u ) p → i

@ result_encode i value → ( Vec u )

@ result_decode ( Vec u ) p → i

@ primes_handler → ( @ ( Vec u ) ( Vec u ) )

@ sumsq_handler → ( @ ( Vec u ) ( Vec u ) )

: Chunk { i lo i hi }

@ shard i lo i hi i n → ( Vec Chunk )

@ shard_free sink ( Vec Chunk ) chunks → v

The chunks are plain values their Vec drops; this lets go of them now rather than at the end of the owner's scope (optional).

@ chunk_key i idx → ( Vec u )

A ring key for chunk index i: 8 BE bytes so distinct chunks hash to distinct ring points (FNV-1a/64 in ring_owner) and spread across the worker set.

---

census.nu

packages/swarm/src/census.nu — lightweight membership gossip for the swarm.

The compute layer (dist/job over dist/ring) needs every node to agree on the live worker set: a worker must know it owns a key, and the coordinator must know which pubkeys to route to. The full SWIM table (net/membership.nu) is the heavy, churn-hardened answer; this is the small one a compute cluster actually needs — a HELLO announcement that feeds the consistent-hash ring.

• a node joins by broadcasting HELLO(role, want=1) to the relay group;
• everyone who hears it adds the node to their ring (workers only) and, if

want, replies with their own HELLO so the newcomer learns them too;

• the ring converges, and "join the cluster" is literally "run the binary".

Roles: a WORKER owns keys and executes handlers, so it joins everyone's ring; a CLIENT (the coordinator) only submits, so it is never added to the ring — otherwise it would own keys it has no handler for and silently drop results.

The codec and the membership set are pure; only the pump touches transport.

API

@ census_hello_t → i

@ role_client → i

@ role_worker → i

@ hello_build i id i role i want ( Vec u ) pubkey → ( Vec u )

HELLO wire: [3][id:8][role:1][want:1][pklen:2][pubkey…]

: Hello { i id i role i want ( Vec u ) pubkey }

@ hello_free sink Hello h → v

A Hello owns only its pubkey, which its owner drops; this lets go of it now rather than at the end of the owner's scope (optional).

@ hello_decode ( Vec u ) buf → Hello

: Member { ( Vec u ) pubkey i id }

: RosterImpl { ( Vec Member ) members }

: Roster { s ctl }

A Roster is a handle on its member list in an rcbox (stdlib/core/rcbox.nu): every copy is the same roster, and the last owner releases it.

@ Roster_share Roster h → Roster

@ Roster_drop sink Roster h → v

@ roster_new → Roster

@ roster_free sink Roster r → v

Let go of r now rather than at the end of its owner's scope (optional).

@ roster_has Roster r__h ( Vec u ) pubkey → b

@ roster_count Roster r__h → i

@ roster_add Roster r__h Ring ring ( Vec u ) pubkey i id i vnodes → b

Fold a worker into the roster + ring, once. Returns T if newly added.