pull.nu

packages/nurllama/src/pull.nu — streaming model download with resume.

The whole transfer is constant-memory: HTTP chunks flow straight to disk through file_write_chunk while the incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<name>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and a manifest records name → digest.

( nl_resolve_url src ) → String hf.co/ORG/REPO/FILE shorthand → the HF resolve URL; http(s):// passes through ( nl_default_name src ) → String basename minus .gguf ( nl_pull root src name ) → !v String

API

@ nl_resolve_url s src → String

hf.co/ORG/REPO/path/to/file.gguf → https://huggingface.co/ORG/REPO/resolve/main/path/to/file.gguf

@ nl_default_name s src → String

last path segment, ".gguf" stripped — the default store name

@ nl_pull String root s src s name_override → !v String

---

start.nu

packages/nurllama/src/start.nu — the nurllama start wizard.

Interactive setup: pick a model from the models directory (and the pull store), choose whether the server binds to localhost only or is reachable from the network, whether a network server is open or gated by a bearer token, and the port. The answers are written to $NURLLAMA_HOME/config.json (see config.nu) and the server starts.

A second nurllama start finds the config and offers to reuse it; nurllama start -y reuses it without asking.

API

: ModelEntry

: ModelEntry {
    String value
    String label
}

@ nurllama_start ArgParser p → i

nurllama start [-y]. Returns the server's exit code.

---

convert.nu

packages/nurllama/src/convert.nu — HF checkpoint → GGUF, streaming.

( nurllama_convert hfdir outpath otype ) → i 0 = ok

Converts a Hugging Face checkpoint directory (config.json + tokenizer.json + model*.safetensors shards) into a GGUF that llama.cpp would recognise — same architecture name, metadata keys, tensor names, shapes and fusions — so the file interoperates in both directions. Currently supported: llada2_moe (LLaDA2.x diffusion MoE), written as the llada2 architecture following llama.cpp's BailingMoeV2 conventions:

• the fused attention.query_key_value tensor stays FUSED

(blk.N.attn_qkv.weight, HF row order, no permutation — NEOX rotary semantics are preserved by construction)

• the 256 per-expert mlp.experts.E.{gate,up,down}_proj tensors

fuse into one 3D blk.N.ffn_{gate,up,down}_exps.weight each

• mlp.gate.weight → blk.N.ffn_gate_inp.weight (kept F32: the

reference model routes on an fp32 sigmoid, and the router is ~2 MB per layer — precision is free here)

• mlp.gate.expert_bias → blk.N.exp_probs_b.bias (F32)
• shared experts → blk.N.ffn_{gate,up,down}_shexp.weight
• tokenizer.json (byte-level BPE) → tokenizer.ggml.* (gpt2 model,

pre = bailingmoe2, id-ordered tokens, merges, added tokens as CONTROL, [PAD…] filler up to config vocab_size)

The conversion STREAMS: shards are mmapped, each tensor is dequantised (bf16→f32) and encoded (Q8_0/F16/BF16/F32) in row chunks straight into the gguf streaming writer — peak memory is one chunk, never the model. A 32 GB checkpoint converts on a 16 GB machine.

API

: i CV_F32 0

ggml tensor type ids (gguf.nu names them; the writer wants the raw id)

: i CV_F16 1

: i CV_Q8_0 8

: i CV_BF16 30

: i CV_TOK_NORMAL 1

token types (llama.cpp enum)

: i CV_TOK_CONTROL 3

: i CV_TOK_USER 4

: i CV_TOK_UNUSED 5

: CvJobs

: CvJobs {
    ( Vec String ) names
    ( Vec i ) gts
    ( Vec i ) nds
    ( Vec i ) d0s
    ( Vec i ) d1s
    ( Vec i ) d2s
    ( Vec String ) srcs
}

One conversion job = one GGUF tensor. src is the HF tensor name, or the marker experts:<layer>:<proj> for a 256-way fusion.

: Cv

: Cv {
    ( Vec i ) sts
    ( Vec String ) stpaths
    Json cfg
    b cfg_ok
    i n_layer
    i n_embd
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    i n_ctx
    i n_expert
    i n_expert_used
    i n_group
    i topk_group
    i n_shared
    i first_dense
    i moe_ff
    f rope_base
    f eps
    f route_scale
    b norm_topk
    f partial_rotary
}

: CvVocab

: CvVocab {
    ( Vec String ) tokens
    ( Vec i ) types
    ( Vec String ) merges
    i bos_id
    i eos_id
    i pad_id
    i mask_id
    b ok
}

@ nurllama_convert s hfdir s outpath s otype → i

@ _cv_geti ( Vec i ) v i k → i

---

webui.nu

packages/nurllama/src/webui.nu — the embedded web chat UI.

A single self-contained page (no external requests — CSP-friendly) served at / to browsers. It talks to the /web/* JSON endpoints, carries the per-client GUID in the nl_client cookie the server sets, and (when the server was started with token auth) prompts for the token once and sends it as a Bearer header on every request.

The markup avoids backtick and backslash characters so it embeds cleanly in a NURL backtick string.

API

@ webui_html → s

---

config.nu

packages/nurllama/src/config.nu — the nurllama start config file.

A small JSON file at $NURLLAMA_HOME/config.json (default ~/.nurllama/config.json) that remembers the wizard's choices so a second nurllama start can reuse them:

{ "model": "<path or store name>", "host": "127.0.0.1" | "0.0.0.0", "port": 11434, "auth": "open" | "token", "token": "<bearer token, when auth=token>", "models_dir": "<directory the wizard scans for *.gguf>" }

( cfg_path root ) → String the config file path (owned) ( cfg_exists root ) → b ( cfg_load root ) → ?NlConfig ( cfg_save root cfg ) → b ( cfg_free cfg ) → v

API

: NlConfig

: NlConfig {
    String model
    String host
    i port
    String auth
    String token
    String models_dir
}

@ cfg_free NlConfig c → v

@ cfg_path String root → String

@ cfg_exists String root → b

@ cfg_load String root → ?NlConfig

Load the config, or None when it is absent or unparseable. The caller owns the returned NlConfig (cfg_free).

@ cfg_save String root NlConfig c → b

Serialise + write. Returns T on success. Creates $NURLLAMA_HOME if it does not exist (the store already does, but a fresh box may not).

---

model.nu

packages/nurllama/src/model.nu — the llama-architecture forward pass.

Loads a GGUF model onto the device (CUDA, or the CPU/OpenMP backend via NURL_GPU=cpu — same kernels either way) and runs autoregressive decode with a device-resident KV cache:

( llm_open path n_ctx ) → !*Llm String ( llm_eval m token pos ) → v one decode step; logits land in the reused host buffer ( llm_logit m i ) → f read logit i of the last eval ( llm_n_vocab m ) ( llm_tok m ) the model's tokenizer rides along ( llm_close m )

Weights stay QUANTISED on the device: a matvec whose weight type has a device kernel (F32/F16/Q4_0/Q8_0/Q4_K/Q6_K) uploads the GGUF bytes verbatim and decodes each block inside the matmul, so a Q4_K model needs ~7× less device memory than its f32 expansion (and the memory-bound matvec reads ~7× fewer bytes). Any other type falls back to host dequantisation through gguf_dequant — correctness first, always. Activations, KV cache and logits are f32.

The token embedding is dequantised host-side once (its rows upload one at a time), which costs host RAM but no device memory. The prompt is processed in CHUNKS: the matmuls run over up to LM_CHUNK positions at once, so a 300-token prompt costs a few hundred kernel launches instead of a few tens of thousands, and each weight is read once per chunk instead of once per token. Attention still runs per position (it is not the bottleneck, and the causal window differs per position anyway) — over a cache the same chunk already filled.

Architectures: the llama shape — RMSNorm → GQA attention with NORM-style RoPE → SwiGLU FFN, residuals around both — which llama and qwen2 both are. Hyperparameters are read under the model's own key prefix (llama.* / qwen2.*), and qwen2's per-layer Q/K/V biases are applied when present (llama models simply have none). Tied-embedding models (no output.weight) reuse token_embd as the output projection.

API

: Llm

: Llm {
    Gpu g
    LlmKernels ks
    * Tok tok
    i n_embd
    i n_layer
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    i n_ctx
    i rope_dim
    i rope_style
    f eps
    f rope_base
    // gemma: the attention projections do NOT have to land back on n_embd —
    // gemma3-270m has n_embd 640, 4 heads and head_dim 256, so q is 1024 wide
    // and `wo` maps 1024 → 640. q_dim is nh*head_dim, never assumed = n_embd.
    i q_dim
    f embd_scale
    f attn_scale
    b ffn_gelu
    i swa_window
    i swa_period
    f rope_base_swa
    // ── llada2 (diffusion MoE) ──────────────────────────────────────
    // bidir: attention is NOT causal — the diffusion decode evaluates a
    // whole block window at once and every query sees the full window.
    b bidir
    i mask_id
    i first_dense
    i n_expert
    i n_expert_used
    i n_group
    i n_group_used
    i moe_ff
    i shexp_ff
    f route_scale
    b weights_norm
    ( Vec i ) gate_inp
    ( Vec i ) tq_gate_inp
    ( Vec f ) exp_bias
    ( Vec i ) gate_exps
    ( Vec i ) up_exps
    ( Vec i ) down_exps
    ( Vec i ) tq_gate_exps
    ( Vec i ) tq_up_exps
    ( Vec i ) tq_down_exps
    ( Vec i ) gate_shexp
    ( Vec i ) up_shexp
    ( Vec i ) down_shexp
    ( Vec i ) tq_gate_shexp
    ( Vec i ) tq_up_shexp
    ( Vec i ) tq_down_shexp
    i rlogitsd
    * u router_host
    i moe_outd
    // batched-MoE scratch: the assignment table (expert id + weight per
    // (position, selected-expert) pair) and the expert hidden/down buffers
    i moe_expid
    i moe_wts
    i moe_gate
    i moe_up
    i moe_downb
    // all layers' expert bias as f32 on the device (n_layer · n_expert),
    // read at offset L · n_expert by the on-device router
    i bias_all
    i gg
    // A SECOND weight source: the same model, in the container Hugging Face
    // ships it in. The GGUF still supplies the hyperparameters and the
    // tokenizer; the tensors come from here when it is set (0 = not set).
    // This is how the safetensor reader is proven: run the same model from
    // both containers and require the logits to agree.
    i st
    b st_norm_add1
    i st_embd
    i embd_idx
    ( Vec u ) embd_host
    ( Vec i ) wdptr
    ( Vec i ) wbytes
    ( Vec i ) tq_attn_norm
    ( Vec i ) tq_wq
    ( Vec i ) tq_wk
    ( Vec i ) tq_wv
    ( Vec i ) bq
    ( Vec i ) bk
    ( Vec i ) bv
    ( Vec i ) tq_wo
    ( Vec i ) tq_ffn_norm
    ( Vec i ) tq_gate
    ( Vec i ) tq_down
    ( Vec i ) tq_up
    i tq_output
    ( Vec i ) attn_norm
    ( Vec i ) q_norm
    ( Vec i ) k_norm
    ( Vec i ) post_attn_norm
    ( Vec i ) post_ffn_norm
    ( Vec i ) wq
    ( Vec i ) wk
    ( Vec i ) wv
    ( Vec i ) wo
    ( Vec i ) ffn_norm
    ( Vec i ) w_gate
    ( Vec i ) w_down
    ( Vec i ) w_up
    i out_norm
    i w_output
    ( Vec i ) kcache
    ( Vec i ) vcache
    i xd
    i xnd
    i qd
    i kd
    i vd
    i aod
    i tmpd
    // rmsnorm reads the whole row to form its sum of squares, so it can never
    // write back into its own input: another block would already have
    // overwritten the row under it. gemma's Q/K and post-block norms
    // therefore need somewhere to land.
    i qn2d
    i kn2d
    i tn2d
    i gd
    i ud
    i scored
    i logitsd
    * u logits_host
    // greedy argmax + confidence per window position (device + host)
    i amid_d
    i amprob_d
    * u amid_h
    * u amprob_h
    // coarse phase profiler (NURLLAMA_PROF): sync+timer per phase
    b prof_on
    i prof_attn_ns
    i prof_moe_ns
    i prof_shexp_ns
    i prof_dense_ns
    i prof_out_ns
}

@ _lm_geti ( Vec i ) v i k → i

@ string_eq_raw String a s raw → b

: ~ i __lm_last_type -1

Publishes the uploaded tensor's ggml type (0 when it was host-dequantised to f32), so the caller records which matvec kernel the weight needs.

: ~ i __lm_up_ns 0

: ~ i __lm_rp_ns 0

: ~ b __lm_alloc_failed F

Set when any device allocation comes back null (out of device memory). Checked once at the end of llm_open_st: a silent failure here used to surface as an all-zero forward pass — the model "ran" and produced token 0 forever — instead of an error naming the cause.

: i LM_CHUNK 64

Positions per prefill chunk. Bounds the batched activation scratch (chunk × n_ff floats) while keeping the launches amortised.

@ llm_open s path i want_ctx → !*Llm String

: ~ b __lm_lprof F

Same model, weights from a safetensors file instead of the GGUF's own tensors. The GGUF still supplies the hyperparameters and the tokenizer — safetensors carries neither (they live in config.json and tokenizer.json next to it) — so this is a weight SOURCE, not a second model format.

It exists to be checked: run one model through both containers and the logits must agree. If the safetensors reader got a dtype, an offset or a row order wrong, that check fails here rather than silently in whatever depends on it next. Load-phase stopwatch (NURLLAMA_PROF): prints [load] <what>=<ms>ms.

: ~ i __lm_lt 0

@ llm_open_st s path s weights i want_ctx → !*Llm String

@ llm_close * Llm m → v

@ llm_tok * Llm m → *Tok

@ llm_n_vocab * Llm m → i

@ llm_n_ctx * Llm m → i

@ llm_logit * Llm m i idx → f

@ llm_eval * Llm m i token i pos → v

One decode step: token at position pos (0-based; caller feeds positions sequentially). Logits for the NEXT token land in logits_host. One decode step: token at position pos.

@ llm_eval_n * Llm m ( Vec i ) ids i first i count i pos0 → v

Evaluate count tokens — ids[first .. first+count) — starting at absolute position pos0. The matmuls run over the whole batch; attention runs per position over the cache this same batch already filled (its causal window differs per position). Logits are produced for the LAST position only, which is all a prefill or a decode step needs.

@ llm_eval_win * Llm m ( Vec i ) ids i first i count i pos0 i kvlen b want_logits → v

Diffusion window evaluation (llada2): the batch is a block window at positions pos0.., every query attends the SAME fixed cache range [0, kvlen) — bidirectional inside the window, all earlier blocks visible — and logits come back for EVERY position when wanted (the denoise step samples all of them). The window's K/V rows land in the cache at pos0.., overwriting the previous step's; earlier (frozen) blocks keep theirs, which is exactly the block-diffusion mask.

@ llm_win_id * Llm m i row → i

After a mode-3 (greedy) window eval: the argmax token id and its softmax probability for window row row.

@ llm_win_prob * Llm m i row → f

@ llm_win_greedy * Llm m ( Vec i ) ids i first i count i pos0 i kvlen → v

Greedy window eval: run the forward and reduce to (id, prob) per position on the device (no full-vocab download).

@ llm_logit_at * Llm m i row i idx → f

Logit idx of window row row after an all-positions eval (llm_eval_win with logits). Row 0 = the window's first position.

@ llm_mask_id * Llm m → i

@ llm_is_diffusion * Llm m → b

@ llm_chunk → i

The largest chunk a single llm_eval_n call may take.

@ llm_prefill * Llm m ( Vec i ) ids → v

Prefill: run the whole prompt through the model in chunks, leaving the KV cache filled and the logits set for the LAST prompt token — the distribution the first generated token is drawn from.

---

chat.nu

packages/nurllama/src/chat.nu — chat templates.

Turns a message list into the exact prompt string a chat-tuned model was trained on. The template is chosen from the model's own GGUF metadata: tokenizer.chat_template is matched against the shapes we know, and the architecture/BOS conventions fill the rest — so a model carries its own dialect and the caller never guesses.

( chat_style_of g ) → i CHAT_* below ( chat_render style msgs ) → String prompt for generation ( chat_stop_matches style piece ) → b stop-token text guard

Styles: CHAT_LLAMA2 [INST] <<SYS>>…<</SYS>> user [/INST] assistant </s> CHAT_LLAMA3 <|start_header_id|>role<|end_header_id|>…<|eot_id|> CHAT_CHATML <|im_start|>role\n…<|im_end|> (qwen, many finetunes) CHAT_PHI3 <|user|>\n…<|end|>\n<|assistant|>\n (phi-3/3.5/4) CHAT_GEMMA <start_of_turn>user\n…<end_of_turn> (gemma; the assistant's role is spelled "model", and gemma has NO system role — the system prompt folds into the first user turn, which is what its own template does) CHAT_PLAIN no template — the messages are concatenated; a base (non-chat) model gets its raw prompt, which is the honest thing to do rather than inventing turns.

A ChatMsg's role is one of system / user / assistant.

API

: i CHAT_PLAIN 0

: i CHAT_LLAMA2 1

: i CHAT_LLAMA3 2

: i CHAT_CHATML 3

: i CHAT_GEMMA 4

: i CHAT_PHI3 5

: i CHAT_BAILING 6

: ChatMsg

: ChatMsg {
    String role
    String content
}

@ chat_msg s role s content → ChatMsg

@ chat_msg_free ChatMsg m → v

@ chat_msgs_free ( Vec ChatMsg ) v → v

@ chat_style_of * Gguf g → i

Pick the style from the model's own chat template (substring match on the marker tokens each dialect must contain), falling back to PLAIN when the model ships no template — a base model, honestly served.

@ chat_render i style ( Vec ChatMsg ) msgs → String

@ chat_stop_matches i style s piece → b

Some chat models emit their turn terminator as ordinary text rather than the EOS id (a finetune whose eos_token_id was left at the base value). Treat the dialect's terminator as a stop sequence too.

---

history.nu

packages/nurllama/src/history.nu — the conversation store (SQLite).

Every chat a web client has with a model is persisted, scoped to the client's cookie GUID, in $NURLLAMA_HOME/history.db. Following the registry's proven idiom, the DB is opened FRESH per operation (its handle is % Drop and auto-closes at the arm's scope exit — never closed by hand) with WAL + a busy timeout, so concurrent requests serialise cheaply and nothing has to keep a Drop handle alive across the globals-are-scalars boundary. The schema is IF NOT EXISTS, so the first open on a fresh box creates it.

( hist_db_path root ) → String ( hist_ensure_client path guid now ) → b ( hist_new_conversation path guid model now ) → i rowid, -1 on error ( hist_add_message path conv role content now ) → b ( hist_conversation_owner path conv ) → String ("" = none) ( hist_list_conversations path guid ) → Json array ( hist_get_messages path conv ) → Json array ( hist_rename_conversation path conv guid title ) → b ( hist_delete_conversation path conv guid ) → b

API

@ hist_db_path String root → String

@ hist_now → i

@ hist_open s path → !Database SqliteErr

Open + harden + migrate. The Database is % Drop: the caller uses it inside the ?? arm and lets it auto-close — NEVER sqlite_close it.

@ hist_ensure_client s path s guid i now → b

@ hist_new_conversation s path s guid s model i now → i

@ hist_add_message s path i conv s role s content i now → b

@ hist_conversation_owner s path i conv → String

The GUID that owns conv, or "" when the conversation does not exist. The web layer checks this before returning any conversation's contents.

@ hist_list_conversations s path s guid → Json

A JSON array of this client's conversations, newest first: [ { "id":N, "model":"…", "title":"…", "created_at":T }, … ]

@ hist_get_messages s path i conv → Json

A JSON array of a conversation's messages in order: [ { "role":"user|assistant", "content":"…", "created_at":T }, … ]

@ hist_delete_conversation s path i conv s guid → b

Delete a conversation and its messages — only when guid owns it.

---

diffuse.nu

packages/nurllama/src/diffuse.nu — the block-diffusion decode loop (LLaDA2.x "JointThreshold + editing").

( dz_generate m prompt gen_len block_len threshold edit_thr max_post temp rng ) → ( Vec i )

The sequence is a template of gen_len mask tokens after the prompt, processed one block_len window at a time. Each denoise step evaluates the whole active window bidirectionally (llm_eval_win) and samples every position at once:

• masked positions whose confidence clears threshold are

committed (all of them; if none clears it, the single most confident one is) — the parallel-decode half

• already-committed positions (except the prompt's) are REVISED

when the model now prefers a different token above edit_thr — the LLaDA2.1 editing half

• the step loop ends when a pass finds no masks and makes no

edits, or after max_post post-resolution passes

Completed blocks freeze: one final KV-only evaluation writes the block's settled content into the cache, and later windows attend to it without re-evaluating — the reference implementation recomputes the whole prefix every step; the cache makes each step O(window) instead of O(prefix).

Generation stops early when a completed block leaves no masks and an EOS has appeared (eos_early_stop), and the returned ids are cut BEFORE the first EOS — callers get exactly the text tokens.

API

& libm @ exp f x → f

@ _dz_geti ( Vec i ) v i k → i

@ _dz_getf ( Vec f ) v i k → f

@ dz_generate * Llm m ( Vec i ) prompt i gen_len0 i block_len0 f threshold f edit_thr i max_post f temp Rng rng → ( Vec i )

---

kernels.nu

packages/nurllama/src/kernels.nu — the LLM decode ops as GPU kernels.

CUDA-C sources compiled once through packages/gpu (NVRTC on the CUDA backend; the SAME source runs on the CPU/OpenMP backend, so every kernel below deliberately avoids shared / syncthreads — each thread computes its output independently, exactly like the onnx op set). Decode-step shapes (one token at a time) keep the naive forms honest: the matvecs dominate and each row is one thread's serial dot product.

Layout contracts (ggml row-major, ne0 fastest): weight W[out,in] → W[rin + c], matvec y[r] = Σc W[rin+c]·x[c] K/V cache → cache[t(n_kv·hd) + khhd + d] q/k activations → x[h*hd + d]

API

: LlmKernels

: LlmKernels {
    GpuKernel matvec
    GpuKernel mv_q4_0
    GpuKernel mv_q8_0
    GpuKernel mv_f32_b
    GpuKernel mv_q8_0_b
    GpuKernel mv_q5_0
    GpuKernel mv_q5_1
    GpuKernel mv_q4_k
    GpuKernel mv_q5_k
    GpuKernel mv_q6_k
    GpuKernel mv_f16
    GpuKernel w_f32
    GpuKernel w_q4_0
    GpuKernel w_q8_0
    GpuKernel w_q4_k
    GpuKernel w_q6_k
    GpuKernel w_q5_0
    GpuKernel w_q5_1
    GpuKernel w_q5_k
    b warp
    GpuKernel rmsnorm
    GpuKernel rope
    GpuKernel rope_neox
    GpuKernel attn
    GpuKernel attn_sc
    GpuKernel attn_out
    GpuKernel silumul
    GpuKernel gelumul
    GpuKernel scale
    GpuKernel addv
    GpuKernel addrow
    GpuKernel copyat
    GpuKernel axpy
    GpuKernel mv_exp_f32
    GpuKernel mv_exp_q8_0
    GpuKernel moe_scatter
    GpuKernel moe_route
    GpuKernel argmax_conf
    GpuKernel argmax_conf_w
    GpuKernel q8_repack
    b ok
}

@ lk_build Gpu g → LlmKernels

@ lk_free LlmKernels ks → v

@ lk_matvec LlmKernels ks i wd i xd i yd i rows i cols i batch → v

@ lk_matvec_q LlmKernels ks i gt i wd i xd i yd i rows i cols i batch → b

Quantised matvec dispatch: gt is the ggml tensor type of W. Returns F when the type has no device kernel (the caller then falls back to the f32 path — correctness first, always).

@ lk_experts_supported LlmKernels ks i gt → b

True when the batched-expert matvec has a kernel for weight type gt. Only F32 (0) and Q8_0 (8) so far — the two the converter produces and the reference path uses; other types fall back to the per-position loop in model.nu.

@ lk_matvec_experts LlmKernels ks i gt i wd i xd i yd i expid_dev i rows i cols i xk i n_assign → b

One batched matvec over all n_assign expert assignments. Output row a uses expert expid_dev[a] and input row a/xk of x. Returns F when gt has no batched kernel (caller falls back).

@ lk_moe_route LlmKernels ks i logitsd i biasd i expid_dev i wts_dev i n_expert i n_group i topk_group i n_used f scale i norm i count → v

On-device routing: fills expid_dev / wts_dev [count·n_used] from the router logits and the expert bias. One thread per position.

@ lk_q8_repack LlmKernels ks i srcd i dstd i nb i nblk → v

Greedy argmax + softmax-prob per position, on device. Repack one uploaded q8_0 tensor into the split per-row layout (see __lk_q8_repack). nb = quant blocks per row, nblk = blocks total.

@ lk_argmax_conf LlmKernels ks i logitsd i outid i outprob i vocab i count → v

@ lk_moe_scatter LlmKernels ks i downd i wtsd i outd i K i dim i count → v

out[pos] += Σ_j wts[posK+j] · down[(posK+j)] over dim floats.

@ lk_rmsnorm LlmKernels ks i xd i wd i yd i n f eps i batch → v

@ lk_rope LlmKernels ks i xd i nh i hd i rd i pos f base i style i batch → v

style: 0 = NORM (llama), 1 = NEOX (qwen2)

@ lk_attn LlmKernels ks i qd i kcd i vcd i outd i scd i nh i nkv i hd i npos0 i batch i maxpos f qscale i window i causal → v

npos0 = the causal window of the FIRST batch element (element b sees npos0 + b rows). maxpos strides the per-(batch,head) score scratch. Two-pass attention: scores, then softmax + value-weighted sum.

@ lk_axpy LlmKernels ks i yd i xd f a i n → v

y ← y + a·x — the routed-expert accumulator (each selected expert's down-projection lands weighted onto the residual stream).

@ lk_gelumul LlmKernels ks i gd i ud i n → v

GeGLU: g ← gelu(g) * u, in place. Same shape as lk_silumul.

@ lk_scale LlmKernels ks i xd f s i n → v

x ← x * s, in place.

@ lk_silumul LlmKernels ks i gd i ud i n → v

@ lk_addv LlmKernels ks i ad i bd i n → v

@ lk_addrow LlmKernels ks i ad i vd i n i batch → v

a[b][i] += v[i] for every row of the batch

@ lk_copyat LlmKernels ks i dstd i srcd i off i n → v

---

api.nu

packages/nurllama/src/api.nu — an ollama-compatible HTTP API.

Speaks enough of ollama's wire protocol that existing clients work unchanged:

POST /api/generate { model, prompt, stream?, options? } POST /api/chat { model, messages[{role,content}], stream?, options? } → NDJSON: one JSON object per token while generating, then a final object with done:true (+ token counts). stream:false returns a single aggregated object. GET /api/tags the local store, as ollama's model list POST /api/show { model } → the model's metadata GET / "nurllama is running"

Streaming rides the http package's stream hook (http_app_stream): the handler owns the TcpConn and writes chunked NDJSON itself, so a token reaches the client the moment it is decoded.

One model is resident at a time (LRU of one): a request naming a different model swaps it in. Generation is serialised — the decode loop owns the device — which is honest for a single-GPU host; the server is single-worker so that serialisation is structural rather than a lock.

API

: ~ i g_api_llm 0

The resident model + the store root, reachable from the handlers. Globals may only carry scalar/s literals, so the two strings are plain owned C-strings (strdup'd in, nurl_free'd out) rather than String structs.

: ~ i g_api_style 0

: ~ s g_api_name ``

: ~ s g_api_root ``

: ~ i g_api_auth 0

Config for the wizard-started server (nurllama start): the default model the web UI chats with, the bearer token (empty = open access), and the SQLite history path. All scalar/s (globals hold no structs).

: ~ s g_api_token ``

: ~ s g_api_model ``

: ~ s g_api_histpath ``

@ api_set_root String root → v

@ api_serve String root s host i port → i

Serve on host:port. Single worker: the decode loop owns the device, so serialisation is structural rather than a lock.

@ api_serve_cfg String root NlConfig cfg → i

Serve with a full wizard config: a default model for the web UI, an optional bearer token, and the conversation-history database.

---

tokenizer.nu

packages/nurllama/src/tokenizer.nu — the GGUF vocabulary loader.

The tokenizer ENGINE now lives in packages/tokenizer: SentencePiece merging, byte-level BPE, inline special tokens, decoding — none of which has anything to do with the container the vocabulary arrived in.

What is left here is the part that IS about GGUF: pulling the pieces, scores, token types and merge rules out of tokenizer.ggml.* metadata and handing them to tok_build. whisper does the same from vocab.json + merges.txt, with the same engine behind it (deps/tokenizer/src/hf.nu).

( tok_new gguf ) → !*Tok String

Everything else — tok_encode / tok_decode / tok_piece / tok_free / tok_bos / tok_eos / tok_unk / tok_n_vocab — comes from the package.

API

@ tok_new * Gguf g → !*Tok String

---

main.nu

nurllama — run language models locally.

nurllama start [-y] interactive setup wizard: pick a model, choose the bind scope (localhost / network) and access (open / bearer token) and the port, saved to config.json; launches the web chat server. A second run reuses the config (-y = without asking). The server hosts a web chat UI with per- browser cookie sessions and a SQLite conversation history. nurllama pull <src> [--name N] download into ~/.nurllama (hf.co/ORG/REPO/FILE.gguf or a URL; resumes a broken pull) nurllama list · rm <name> · verify <name> the local model store nurllama serve [--host H] [--port N] ollama-compatible HTTP API nurllama chat <model|name> interactive chat (model's own template from GGUF) nurllama run <model|name> <prompt> generate (stream to stdout) -n N (default 64) · --temp F (0 = greedy, default 0.8) --topk N · --topp F · --seed N · --ctx N diffusion models add: --block N · --threshold F · --edit-threshold F · --post-steps N (temp defaults to 0) nurllama convert <hf-dir> <out.gguf> HF checkpoint → GGUF --type q8_0 (default) | f16 | bf16 | f32 nurllama logits <model.gguf> <prompt> final-position logits, one per line (verification tap) nurllama dlogits / dgen diffusion verification taps (window logits / denoise ids) nurllama tokenize <model.gguf> <text> encode → token ids nurllama detok <model.gguf> <id> [id …] decode ids → bytes nurllama vocab <model.gguf> [n] dump the first n pieces nurllama selftest synthetic SPM + BPE vocab round-trips, bit-exact

Everything is loaded from the model file alone (tokenizer.ggml. + llama. metadata + tensors via packages/gguf); compute runs on packages/gpu — CUDA when present, NURL_GPU=cpu for the host backend.

API

: NlCnt

: NlCnt {
    i pass
    i fail
}

@ main → i

---

sample.nu

packages/nurllama/src/sample.nu — host-side sampling over the logits.

( sample_greedy m ) → i argmax ( sample_next m rng temp topk topp ) → i softmax(T) → top-k → top-p → draw

temp ≤ 0 collapses to greedy. top-k ≤ 0 disables the k filter; topp ≥ 1 disables the nucleus filter. Deterministic under a seeded Rng.

API

@ sample_greedy * Llm m → i

& libm @ exp f x → f

host-side softmax needs exp — bind libm directly, the stdlib idiom

@ sample_next * Llm m Rng rng f temp i topk f topp → i

---

store.nu

packages/nurllama/src/store.nu — the local model store (~/.nurllama).

Content-addressed, cas-shaped, but streaming end to end — a blob is never fully resident:

<root>/blobs/sha256-<64hex> the model files, named by content <root>/manifests/<name> one small JSON per model name

A manifest records { name, url, digest, size } — the name is the user-facing handle (nurllama run <name>), the digest is the proof. Two names may point at the same blob (dedup by content); rm only deletes a blob once no manifest references it. verify re-hashes the blob through the incremental sha256 and refuses drift.

( nl_store_root ) → String $NURLLAMA_HOME | ~/.nurllama ( nl_resolve root nameOrPath ) → ?String path to a model file ( nl_store_add root name url hex size ) → !v String ( nl_store_list root ) → v prints the table ( nl_store_rm root name ) → !v String ( nl_store_verify root name ) → !v String ( nl_blob_path root hex ) ( nl_manifest_path root name ) ( nl_hash_file path ) → !String String streaming sha256 hex

API

@ nl_store_root → String

@ nl_blob_path String root s hex → String

@ nl_manifest_path String root s name → String

@ nl_store_init String root → !v String

@ nl_hash_file s path → !String String

Streaming sha256 of a file — constant memory whatever the size.

@ nl_store_add String root s name s url s hex i size → !v String

@ nl_resolve String root s arg → ?String

Resolve a model argument: an existing file path wins; otherwise the store's manifest name → blob path. None = unknown.

@ nl_store_list String root → v

@ nl_store_rm String root s name → !v String

@ nl_store_verify String root s name → !v String

Re-hash the blob and compare with the manifest digest — the store's integrity loop (mirrors cas_get's refuse-on-mismatch, streaming).