census.nu

packages/swarm/src/census.nu — lightweight membership gossip for the swarm.

The compute layer (dist/job over dist/ring) needs every node to agree on the live worker set: a worker must know it owns a key, and the coordinator must know which pubkeys to route to. The full SWIM table (net/membership.nu) is the heavy, churn-hardened answer; this is the small one a compute cluster actually needs — a HELLO announcement that feeds the consistent-hash ring.

• a node joins by broadcasting HELLO(role, want=1) to the relay group;
• everyone who hears it adds the node to their ring (workers only) and, if

want, replies with their own HELLO so the newcomer learns them too;

• the ring converges, and "join the cluster" is literally "run the binary".

Roles: a WORKER owns keys and executes handlers, so it joins everyone's ring; a CLIENT (the coordinator) only submits, so it is never added to the ring — otherwise it would own keys it has no handler for and silently drop results.

The codec and the membership set are pure; only the pump touches transport.

API

@ census_hello_t → i

@ role_client → i

@ role_worker → i

@ hello_build i id i role i want ( Vec u ) pubkey → ( Vec u )

HELLO wire: [3][id:8][role:1][want:1][pklen:2][pubkey…]

: Hello { i id i role i want ( Vec u ) pubkey }

@ hello_free Hello h → v

@ hello_decode ( Vec u ) buf → Hello

: Member { ( Vec u ) pubkey i id }

: Roster { ( Vec s ) members } // *Member

@ roster_new → *Roster

@ roster_free * Roster r → v

@ roster_has * Roster r ( Vec u ) pubkey → b

@ roster_count * Roster r → i

@ roster_add * Roster r * Ring ring ( Vec u ) pubkey i id i vnodes → b

Fold a worker into the roster + ring, once. Returns T if newly added.

---

work.nu

packages/swarm/src/work.nu — the real workloads the cluster runs.

A workload is a pure function over a numeric range that the coordinator shards into many independent chunks; dist/ring routes each chunk to its owning worker, the worker runs the registered handler, and the coordinator sums the partial results. The work is genuine CPU — primes are counted by trial division, not looked up — so a bigger range is a bigger real load, and adding workers measurably shares it.

Two kinds, both verifiable by a closed form so a test can pin the answer:

• PRIMES count of primes in [lo, hi) (π(10^6) = 78498)
• SUMSQ Σ k² for k in [lo, hi), wrapping i64 (n(n+1)(2n+1)/6 closed form)

chunk payload : [lo:8 BE][hi:8 BE] result : [value:8 BE]

API

@ kind_primes → i

@ kind_sumsq → i

@ is_prime i n → b

Is n prime? Trial division to √n — deliberately the honest O(√n) work.

@ count_primes i lo i hi → i

Count primes in [lo, hi).

@ sum_squares i lo i hi → i

Σ k² for k in [lo, hi), wrapping (matches the i64 the closed form is taken mod).

@ chunk_payload i lo i hi → ( Vec u )

@ chunk_lo ( Vec u ) p → i

@ chunk_hi ( Vec u ) p → i

@ result_encode i value → ( Vec u )

@ result_decode ( Vec u ) p → i

@ primes_handler → ( @ ( Vec u ) ( Vec u ) )

@ sumsq_handler → ( @ ( Vec u ) ( Vec u ) )

: Chunk { i lo i hi }

@ shard i lo i hi i n → ( Vec s )

@ shard_free ( Vec s ) chunks → v

@ chunk_key i idx → ( Vec u )

A ring key for chunk index i: 8 BE bytes so distinct chunks hash to distinct ring points (FNV-1a/64 in ring_owner) and spread across the worker set.

---

main.nu

packages/swarm/src/main.nu — swarm: a distributed compute cluster you join by installing it.

nurlpkg install swarm # drops the swarm binary on $PATH swarm relay 0.0.0.0 47700 # the meeting point (one per cluster) swarm worker <host> <port> # join as a compute node — that's the join swarm submit <host> <port> primes 1 1000000 # place a real workload

A worker needs no recompile and no member list: it announces itself over the relay group, every node folds it into the consistent-hash ring (census.nu), and from then on it owns its share of the keyspace and runs the registered handlers (work.nu). The coordinator discovers the live workers the same way, shards a real numeric range across them by key (dist/ring → dist/job), and sums the partial results. Add a worker → it takes load on the next submit.

Built entirely on the standard distributed stack: net/relay (reach), net/transport (the pubkey seam), dist/ring (ownership), dist/job (dispatch).

API

@ swarm_vnodes → i

@ swarm_group_id → ( Vec u )

The relay multicast group every node joins. It is a fixed 32 bytes on purpose: the relay's documented group-id contract is 32 bytes, and a 32-byte id round-trips correctly under every shipped relay framing — so a worker built against any toolchain release still finds the cluster. ("swarm" + zero padding keeps it recognisable on the wire.)

@ pk_from_id i id → ( Vec u )

A 32-byte opaque routing pubkey derived deterministically from a node id. Over the relay leg the pubkey is just the address the relay forwards by, so a spread-out deterministic value is all the routing needs (the real X25519 identity belongs to the direct securedgram leg, not used here).

: Swarm

: Swarm {
    s transport  // *Transport
    s ring  // *Ring
    s roster  // *Roster
    s job  // *JobNode
    ( Vec u ) self_pk
    i self_id
    i role
    ( Vec u ) group
}

@ swarm_new RelayClient rc i id i role → *Swarm

@ swarm_free * Swarm sw → v

@ swarm_join_group * Swarm sw → v

@ swarm_announce * Swarm sw i want → v

Announce ourselves to the group. want asks hearers to reply so a newcomer learns the existing members.

@ swarm_on_hello * Swarm sw Hello h → v

@ swarm_pump * Swarm sw i max → v

Drain inbound transport messages, dispatching census HELLO and job traffic.

@ run_relay s host i port → i

@ swarm_register_handlers * Swarm sw → v

@ run_worker s host i port i id i rounds → i

@ swarm_discover * Swarm sw i rounds → v

Discover the live workers: announce, then pump a short window collecting the HELLO replies that fold workers into the ring.

@ run_submit s host i port i kind i lo i hi → i

@ usage → v

@ arg_int i idx → i

@ arg_eq i idx s lit → b

@ main → i