NURLNURL registrynurl-lang.org →

← nurllama

nurllama 0.22.0 API

pull.nu

packages/nurllama/src/pull.nu — streaming model download with resume.

The whole transfer is constant-memory: HTTP chunks flow straight to disk through file_write_chunk while the incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<name>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and a manifest records name → digest.

( nl_resolve_url src ) → String hf.co/ORG/REPO/FILE shorthand → the HF resolve URL; http(s):// passes through ( nl_default_name src ) → String basename minus .gguf ( nl_pull root src name ) → !v String

API

@ nl_resolve_url s src → String

hf.co/ORG/REPO/path/to/file.gguf → https://huggingface.co/ORG/REPO/resolve/main/path/to/file.gguf

@ nl_default_name s src → String

last path segment, ".gguf" stripped — the default store name

@ nl_pull String root s src s name_override → !v String

start.nu

packages/nurllama/src/start.nu — the nurllama start wizard.

Interactive setup: pick a model from the models directory (and the pull store), choose whether the server binds to localhost only or is reachable from the network, whether a network server is open or gated by a bearer token, and the port. The answers are written to $NURLLAMA_HOME/config.json (see config.nu) and the server starts.

A second nurllama start finds the config and offers to reuse it; nurllama start -y reuses it without asking.

API

: ModelEntry

: ModelEntry {
    String value
    String label
}

@ nurllama_start ArgParser p → i

nurllama start [-y]. Returns the server's exit code.

finetune.nu

nurllama/finetune.nu — LoRA finetuning over the grad tape (M6b).

Loads a GGUF model's weights as f64 HOST tensors (gguf_dequant_f64 — independent of the quantised GPU inference path) and builds the whole transformer forward + cross-entropy loss as ONE grad-tape graph, with LoRA adapter pairs (A[in,r], B[r,out], y = x·W0 + (α/r)·(x·A)·B) on q/k/v/o and gate/up/down as the only parameters. Train by capturing the episode with gput and replaying on the device; the frozen base weights cost no gradient memory or backward compute (grad 0.3.0's requires-grad propagation).

Architectures: llama-family, qwen2 (bias'd q/k/v, NEOX rope) and qwen3 (no biases, NEOX rope, per-head Q/K RMSNorm before the rotation). The llama family's NORM-style rope rotates ADJACENT lanes (2j, 2j+1), which a contiguous-slice tape cannot express — so the loader UN-PERMUTES the q/k projection columns per head (evens first, odds second) and the graph applies half-split NEOX rope: every pair sees the identical rotation at a permuted lane, and attention scores are permutation-invariant, so the values downstream are exactly the model's. (v is untouched — context and o-proj stay in the original layout.)

Layouts: GGUF stores [out, in] row-major; the tape multiplies x[T,in] · W[in,out], so every weight is transposed once at load. Embedding lookup happens HOST-side (the rows enter the tape as a const; embeddings are frozen, so no gather op is needed). Tied-embedding models reuse token_embd for the lm_head (transposed the other way).

API

: FtV

: FtV {
    b has
    ( Vec f ) v
}

An optional per-layer 1-D tensor (norm weight / bias): has says whether the model carries it (llama has no attn_q_norm), v holds its values (empty for a shape-only placeholder).

: FtW

: FtW {
    i rows
    i cols
    ( Vec f ) data
}

One weight in tape layout: data [rows=in, cols=out] flat.

: FtModelImpl

: FtModelImpl {
    b ok
    i n_embd
    i n_layer
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    f eps
    f rope_base
    i rope_dim
    i rope_style  // 0 llama/NORM (columns un-permuted at load) · 1 NEOX
    ( Vec f ) embd  // [n_vocab, n_embd] row-major (host lookup table)
    FtW wout  // lm_head [n_embd, n_vocab] (tied → transposed embd)
    ( Vec f ) norm_f  // output_norm [n_embd]
    ( Vec FtW ) wq ( Vec FtW ) wk ( Vec FtW ) wv ( Vec FtW ) wo
    ( Vec FtW ) wg ( Vec FtW ) wu ( Vec FtW ) wd
    ( Vec FtV ) bq ( Vec FtV ) bk ( Vec FtV ) bv  // qwen2's biases (absent elsewhere)
    ( Vec FtV ) an ( Vec FtV ) fn  // attn_norm / ffn_norm per layer
    // qwen3's per-head Q/K RMSNorm weights [head_dim]. Absent for
    // llama/qwen2 — an absent one simply skips the norm.
    ( Vec FtV ) qn ( Vec FtV ) kn
    String src_path  // the GGUF path — so the per-layer base matrices can be
    // freed after device capture (ft_drop_base) and streamed back for the
    // merge (ft_reload_base), keeping host RAM off the base during training
    // STREAMING mode (ft_set_stream). The base weights are never all resident
    // host-side: ft_open reads their SHAPES only, ft_graph declares them as
    // lazy consts, and ft_stream_upload fills each one straight into its
    // device buffer after the capture, freeing it again before the next.
    // Host peak becomes one layer instead of the whole model — the
    // difference between training a 4B model on a 31 GB box and not.
    b stream
    ( Vec i ) lz_node  // tape node id of each lazy base const
    ( Vec i ) lz_key  // its identity: layer * 16 + slot (__ft_base_name)
    ? Gguf mgg  // a Gguf held open across a streamed merge
}

: FtModel { ( Locked FtModelImpl ) l }

An FtModel is a handle on its state in a Locked (stdlib/std/thread.nu): every copy is the same state, a call takes it for its duration and gives it back, and the last owner releases it.

@ FtModel_share FtModel h → FtModel

@ FtModel_drop sink FtModel h → v

: ~ b __ft_stream_on F

Streaming base upload, OFF by default: the finetune tests compare the CPU tape's loss against the device's, and a lazy const has no host values to compare with. nurllama finetune --stream turns it on for models whose f64 host copy would not fit.

@ ft_set_stream b on → v

@ ft_free sink FtModel m → v

Let go of m now rather than at the end of its owner's scope.

@ ft_drop_base FtModel h → v

Drop the per-layer base matrices + biases/norms (the bulk of host RAM), emptying their vecs (clearing a vec drops its elements). embd / wout / norm_f are KEPT (embd feeds the per-window input recompute; the rest are small). Idempotent — a second call over empty vecs is a no-op — and reversible via ft_reload_base.

@ ft_reload_base FtModel h → b

Re-stream the per-layer base from the source GGUF into m's (empty) vecs — the reverse of ft_drop_base, using the SAME loader path so the NORM-rope un-permute is identical byte-for-byte. Also restores the embedding table when ft_merge_st consumed it: training again after a merge would otherwise embed every token as a zero row. T on success.

@ ft_open s path → !FtModel String

@ ft_n_embd FtModel h → i

The model's shape, for callers that size buffers or print it.

@ ft_n_layer FtModel h → i

@ ft_n_head FtModel h → i

@ ft_n_kv FtModel h → i

@ ft_head_dim FtModel h → i

@ ft_n_vocab FtModel h → i

@ ft_rope_style FtModel h → i

@ ft_has_qk_norm FtModel h i L → b

Whether layer L carries qwen3's per-head Q and K norms (both of them).

@ ft_embed FtModel h ( Vec i ) ids → ( Vec f )

Host-side embedding lookup: token ids → [T, n_embd] rows.

: FtG

: FtG {
    GVar loss
    GVar logits
    GVar xin  // the input-rows const — refresh per window via gput_set_input
    GVar ohin  // the one-hot target const — refreshed together with xin
    i n_pairs  // LoRA pairs registered (7 per layer)
}

The result handles of one built graph.

@ ft_graph FtModel h inout GTape tp ( Vec i ) ids i r f alpha i seed ( Vec i ) pids → FtG

Build the full forward + next-token CE loss on tp. ids is one sequence (T tokens; loss over positions 0..T-2 predicting 1..T-1). pids receives the 2·7·n_layer LoRA parameter ids (A, B per slot); LoRA params register FIRST.

@ ft_onehot FtModel h ( Vec i ) ids → ( Vec f )

The one-hot rows for a window's ids (row t = ids[t+1]; last row zero).

: FtTrain

: FtTrain {
    b ok
    f l0
    f l1
    ( Vec f ) aflat
    ( Vec f ) bflat
}

One trained run's result: per-slot adapter values, flat in slot order (slot = layer·7 + {q k v o gate up down}); A blocks then B blocks.

@ ft_stream_upload FtModel h inout GProg pg → b

Fill every lazy base const's device buffer. Call once, right after the capture; T when every tensor landed.

@ ft_auto_stride i nwin → i

The golden-ratio point of [1, nwin), nudged up to the nearest value coprime with nwin — consecutive steps land maximally far apart.

@ ft_win_at i st i nwin i stride → i

@ ft_train FtModel m__h ( Vec i ) corpus i win i r f alpha i seed i steps f lr i dtype b verbose → FtTrain

@ ft_train_ck FtModel h ( Vec i ) corpus i win i r f alpha i seed i steps f lr i dtype b verbose s ckptp i ckevery b resume i wstride → FtTrain

wstride: 1 = sequential legacy order · 0 = auto (golden-ratio coprime, ft_auto_stride) · else used as given, mod nwin.

@ ft_adapters_save s path FtModel h FtTrain t i r → !v String

Save trained adapters as a safetensors file: per slot, blk.<L>.<which>.lora_a [in,r] and .lora_b [r,out], F32.

@ ft_adapters_load s path FtModel h inout i rank → !FtTrain String

Read a saved adapters file back into an FtTrain (per-slot A blocks then B blocks, flat, exactly as ft_train downloads them) — the merge path's input when the training happened in ANOTHER process: finetune --merge-only after a crash between the adapter save and the merge, or merging an adapter file someone shipped. The rank is read from the file's own blk.0.q.lora_a [in, r] shape and stored into rank.

@ ft_merge_st s path FtModel m__h FtTrain t i r f alpha → !v String

Merge every adapter into its base weight and write the FULL model as a safetensors file — run it with nurllama's --weights path (llm_open_st).

@ ft_merge_st_mask s path FtModel h FtTrain t i r f alpha i mask → !v String

mask bit0 = embeddings, bit1 = attention projections, bit2 = mlp projections (a bisect handle for the merged-path diagnostics).

@ nurllama_finetune s modp s datap s outp s mergedp i steps f lr i rank f alpha i seq i seed b f32 b mixed s ckptp i ckevery b resume i wstride b mergeonly → i

── the CLI driver ─────────────────────────────────────────────────── nurllama finetune <model.gguf> <data.txt>: tokenize the corpus with the model's own tokenizer, LoRA-train on the DEVICE over the first seq tokens (v1 trains one fixed window — the captured graph has a fixed shape; multi-window scheduling lands with gput_set_input plumbing), save the adapters, and optionally write a merged full-model safetensors runnable via nurllama run model.gguf PROMPT --weights merged.st.

convert.nu

packages/nurllama/src/convert.nu — HF checkpoint → GGUF, streaming.

( nurllama_convert hfdir outpath otype ) → i 0 = ok

Converts a Hugging Face checkpoint directory (config.json + tokenizer.json + model*.safetensors shards) into a GGUF that llama.cpp would recognise — same architecture name, metadata keys, tensor names, shapes and fusions — so the file interoperates in both directions. Currently supported: llada2_moe (LLaDA2.x diffusion MoE), written as the llada2 architecture following llama.cpp's BailingMoeV2 conventions:

(blk.N.attn_qkv.weight, HF row order, no permutation — NEOX rotary semantics are preserved by construction)

fuse into one 3D blk.N.ffn_{gate,up,down}_exps.weight each

reference model routes on an fp32 sigmoid, and the router is ~2 MB per layer — precision is free here)

pre = bailingmoe2, id-ordered tokens, merges, added tokens as CONTROL, [PAD…] filler up to config vocab_size)

The conversion STREAMS: shards are mmapped, each tensor is dequantised (bf16→f32) and encoded (Q8_0/F16/BF16/F32) in row chunks straight into the gguf streaming writer — peak memory is one chunk, never the model. A 32 GB checkpoint converts on a 16 GB machine.

API

: i CV_F32 0

ggml tensor type ids (gguf.nu names them; the writer wants the raw id)

: i CV_F16 1

: i CV_Q8_0 8

: i CV_BF16 30

: i CV_TOK_NORMAL 1

token types (llama.cpp enum)

: i CV_TOK_CONTROL 3

: i CV_TOK_USER 4

: i CV_TOK_UNUSED 5

: CvJobs

: CvJobs {
    ( Vec String ) names
    ( Vec i ) gts
    ( Vec i ) nds
    ( Vec i ) d0s
    ( Vec i ) d1s
    ( Vec i ) d2s
    ( Vec String ) srcs
}

One conversion job = one GGUF tensor. src is the HF tensor name, or the marker experts:<layer>:<proj> for a 256-way fusion.

: Cv

: Cv {
    ( Vec St ) sts
    ( Vec String ) stpaths
    Json cfg
    b cfg_ok
    i n_layer
    i n_embd
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    i n_ctx
    i n_expert
    i n_expert_used
    i n_group
    i topk_group
    i n_shared
    i first_dense
    i moe_ff
    f rope_base
    f eps
    f route_scale
    b norm_topk
    f partial_rotary
}

: CvVocab

: CvVocab {
    ( Vec String ) tokens
    ( Vec i ) types
    ( Vec String ) merges
    i bos_id
    i eos_id
    i pad_id
    i mask_id
    b ok
}

@ nurllama_convert s hfdir s outpath s otype → i

@ _cv_geti ( Vec i ) v i k → i

webui.nu

packages/nurllama/src/webui.nu — the embedded web chat UI.

A single self-contained page (no external requests — CSP-friendly) served at / to browsers. It talks to the /web/* JSON endpoints, carries the per-client GUID in the nl_client cookie the server sets, and (when the server was started with token auth) prompts for the token once and sends it as a Bearer header on every request.

The markup avoids backtick and backslash characters so it embeds cleanly in a NURL backtick string.

API

@ webui_html → s

config.nu

packages/nurllama/src/config.nu — the nurllama start config file.

A small JSON file at $NURLLAMA_HOME/config.json (default ~/.nurllama/config.json) that remembers the wizard's choices so a second nurllama start can reuse them:

{ "model": "<path or store name>", "host": "127.0.0.1" | "0.0.0.0", "port": 11434, "auth": "open" | "token", "token": "<bearer token, when auth=token>", "models_dir": "<directory the wizard scans for *.gguf>" }

( cfg_path root ) → String the config file path (owned) ( cfg_exists root ) → b ( cfg_load root ) → ?NlConfig ( cfg_save root cfg ) → b

API

: NlConfig

: NlConfig {
    String model
    String host
    i port
    String auth
    String token
    String models_dir
}

@ cfg_path String root → String

@ cfg_exists String root → b

@ cfg_load String root → ?NlConfig

Load the config, or None when it is absent or unparseable.

@ cfg_save String root NlConfig c → b

Serialise + write. Returns T on success. Creates $NURLLAMA_HOME if it does not exist (the store already does, but a fresh box may not).

model.nu

packages/nurllama/src/model.nu — the llama-architecture forward pass.

Loads a GGUF model onto the device (CUDA, or the CPU/OpenMP backend via NURL_GPU=cpu — same kernels either way) and runs autoregressive decode with a device-resident KV cache:

( llm_open path n_ctx ) → !Llm String ( llm_eval m token pos ) → v one decode step; logits land in the reused host buffer ( llm_logit m i ) → f read logit i of the last eval ( llm_n_vocab m ) ( llm_tok m ) the model's tokenizer rides along Llm is a handle: its last owner releases the weights, the KV cache and the device context; ( llm_close m ) is an optional early release.

Weights stay QUANTISED on the device: a matvec whose weight type has a device kernel (F32/F16/Q4_0/Q8_0/Q4_K/Q6_K) uploads the GGUF bytes verbatim and decodes each block inside the matmul, so a Q4_K model needs ~7× less device memory than its f32 expansion (and the memory-bound matvec reads ~7× fewer bytes). Any other type falls back to host dequantisation through gguf_dequant — correctness first, always. Activations, KV cache and logits are f32.

The token embedding is dequantised host-side once (its rows upload one at a time), which costs host RAM but no device memory. The prompt is processed in CHUNKS: the matmuls run over up to LM_CHUNK positions at once, so a 300-token prompt costs a few hundred kernel launches instead of a few tens of thousands, and each weight is read once per chunk instead of once per token. Attention still runs per position (it is not the bottleneck, and the causal window differs per position anyway) — over a cache the same chunk already filled.

Architectures: the llama shape — RMSNorm → GQA attention with NORM-style RoPE → SwiGLU FFN, residuals around both — which llama and qwen2 both are. Hyperparameters are read under the model's own key prefix (llama.* / qwen2.*), and qwen2's per-layer Q/K/V biases are applied when present (llama models simply have none). qwen3 is that same shape with two differences, both already tensor-driven: it drops the Q/K/V biases, and it RMSNorms every head's Q and K vector before the rotation (attn_q_norm/attn_k_norm, the per-head norm gemma3 introduced). Nothing about it is special-cased beyond the NEOX rotary style — a weight that is absent is simply not applied. Tied-embedding models (no output.weight) reuse token_embd as the output projection.

Device memory is gpukit's: every weight, cache and scratch buffer is a GkBuf (a weight also as its GGUF bytes, GkBBuf) and every kernel a checked one (kernels.nu). Nothing here handles a device address.

API

: LmW { LkW w i off }

A weight and the byte offset where it starts inside its tensor — a row RANGE of a fused tensor (phi3's attn_qkv, ffn_up) or one MoE expert.

: LmPlan

: LmPlan {
    ( Vec GkLaunch ) l
    i count
    b causal
    i mode
    ( Vec i ) rope
    ( Vec i ) copy
    ( Vec i ) attn
}

One recorded evaluation: its launches, the call shape they were recorded for (count positions, causal or not, logits mode), and the launches whose scalars move with the position — every RoPE (pos0), every KV-cache append (its offset) and every attention (npos0, two launches).

: i LM_PLANS 4

The plans kept: a prompt's chunk and remainder, decode, one more.

: LlmImpl

: LlmImpl {
    GpuKit kit
    b warp  // the CUDA kernel shapes (warp shuffles): F on the CPU backend
    Tok tok
    i n_embd
    i n_layer
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    i n_ctx
    i rope_dim
    i rope_style
    f eps
    f rope_base
    // gemma: the attention projections do NOT have to land back on n_embd —
    // gemma3-270m has n_embd 640, 4 heads and head_dim 256, so q is 1024 wide
    // and `wo` maps 1024 → 640. q_dim is nh*head_dim, never assumed = n_embd.
    i q_dim
    f embd_scale
    f attn_scale
    b ffn_gelu
    i swa_window
    i swa_period
    f rope_base_swa
    // ── llada2 (diffusion MoE) ──────────────────────────────────────
    // bidir: attention is NOT causal — the diffusion decode evaluates a
    // whole block window at once and every query sees the full window.
    b bidir
    i mask_id
    i first_dense
    i n_expert
    i n_expert_used
    i n_group
    i n_group_used
    i moe_ff
    i shexp_ff
    f route_scale
    b weights_norm
    ( Vec LmW ) gate_inp
    ( Vec f ) exp_bias
    ( Vec LmW ) gate_exps
    ( Vec LmW ) up_exps
    ( Vec LmW ) down_exps
    ( Vec LmW ) gate_shexp
    ( Vec LmW ) up_shexp
    ( Vec LmW ) down_shexp
    GkBuf rlogitsd
    GpuHost router_host
    GkBuf moe_outd
    // batched-MoE scratch: the assignment table (expert id — i64 — and
    // weight per (position, selected-expert) pair) and the expert
    // hidden/down buffers
    GkBuf moe_expid
    GkBuf moe_wts
    GkBuf moe_gate
    GkBuf moe_up
    GkBuf moe_downb
    // all layers' expert bias as f32 on the device (n_layer · n_expert),
    // read at offset L · n_expert by the on-device router
    GkBuf bias_all
    Gguf gg
    // A SECOND weight source: the same model, in the container Hugging Face
    // ships it in. The GGUF still supplies the hyperparameters and the
    // tokenizer; the tensors come from here when there is one (T).
    // This is how the safetensor reader is proven: run the same model from
    // both containers and require the logits to agree.
    ? St st
    b st_norm_add1
    i st_embd
    i embd_idx
    ( Vec u ) embd_host
    i dev_bytes  // device memory the model holds (NURLLAMA_VERBOSE)
    b alloc_failed  // a device allocation came back empty while loading
    ( Vec GkBuf ) attn_norm
    ( Vec GkBuf ) q_norm  // none where the model has none
    ( Vec GkBuf ) k_norm
    ( Vec GkBuf ) post_attn_norm
    ( Vec GkBuf ) post_ffn_norm
    ( Vec GkBuf ) bq  // qwen2's Q/K/V biases (none elsewhere)
    ( Vec GkBuf ) bk
    ( Vec GkBuf ) bv
    ( Vec LmW ) wq
    ( Vec LmW ) wk
    ( Vec LmW ) wv
    ( Vec LmW ) wo
    ( Vec GkBuf ) ffn_norm
    ( Vec LmW ) w_gate
    ( Vec LmW ) w_down
    ( Vec LmW ) w_up
    GkBuf out_norm
    LmW w_output
    ( Vec GkBuf ) kcache
    ( Vec GkBuf ) vcache
    GkBuf xd
    GkBuf xnd
    GkBuf qd
    GkBuf kd
    GkBuf vd
    GkBuf aod
    GkBuf tmpd
    // rmsnorm reads the whole row to form its sum of squares, so it can never
    // write back into its own input: another block would already have
    // overwritten the row under it. gemma's Q/K and post-block norms
    // therefore need somewhere to land.
    GkBuf qn2d
    GkBuf kn2d
    GkBuf tn2d
    GkBuf gd
    GkBuf ud
    GkBuf scored
    GkBuf logitsd
    GpuHost logits_host
    // greedy argmax + confidence per window position (device + host)
    GkBuf amid_d  // i64 ids
    GkBuf amprob_d
    ( Vec i ) amid_v
    GpuHost amprob_h
    // The recorded evaluations (__lm_eval_core), the oldest first: one per
    // call shape — a prompt alternates a full chunk and its
    // remainder, then decode is one position — and, while __lm_emit
    // records one, where in it the position-dependent scalars sit.
    ( Vec LmPlan ) plans
    ( Vec i ) pl_rope
    ( Vec i ) pl_copy
    ( Vec i ) pl_attn
    // coarse phase profiler (NURLLAMA_PROF): sync+timer per phase
    b prof_on
    i prof_attn_ns
    i prof_moe_ns
    i prof_shexp_ns
    i prof_dense_ns
    i prof_out_ns
}

: LlmInfo

: LlmInfo {
    Tok tok
    i n_vocab
    i n_ctx
    i mask_id
    b bidir
}

What a loaded model is and never changes: read without waiting for an evaluation.

: Llm

: Llm {
    ( Locked LlmImpl ) l
    ( Arc LlmInfo ) info
}

A Llm is a handle: the engine in a Locked — an evaluation takes it for the call and gives it back, so two threads using one model take turns instead of racing on its caches and scratch — beside its dimensions in an Arc. Every copy is the same model; the last owner releases it.

@ Llm_share Llm h → Llm

@ Llm_drop sink Llm h → v

@ _lm_geti ( Vec i ) v i k → i

@ string_eq_raw String a s raw → b

: ~ i __lm_up_ns 0

: ~ i __lm_rp_ns 0

: i LM_CHUNK 64

Positions per prefill chunk. Bounds the batched activation scratch (chunk × n_ff floats) while keeping the launches amortised.

@ llm_open s path i want_ctx → !Llm String

: ~ b __lm_lprof F

Same model, weights from a safetensors file instead of the GGUF's own tensors. The GGUF still supplies the hyperparameters and the tokenizer — safetensors carries neither (they live in config.json and tokenizer.json next to it) — so this is a weight SOURCE, not a second model format.

It exists to be checked: run one model through both containers and the logits must agree. If the safetensors reader got a dtype, an offset or a row order wrong, that check fails here rather than silently in whatever depends on it next. Load-phase stopwatch (NURLLAMA_PROF): prints [load] <what>=<ms>ms.

: ~ i __lm_lt 0

@ llm_open_st s path s weights i want_ctx → !Llm String

% Drop LlmImpl

% Drop LlmImpl {
    @ drop LlmImpl m → v {
        ? . m prof_on {
            : String pm ( string_from `[prof] attn=` )
            ( string_push_int pm / . m prof_attn_ns 1000000 )
            ( string_push_str pm `ms shexp=` )
            ( string_push_int pm / . m prof_shexp_ns 1000000 )
            ( string_push_str pm `ms moe_total=` )
            ( string_push_int pm / . m prof_moe_ns 1000000 )
            ( string_push_str pm `ms (experts=` )
            ( string_push_int pm / - . m prof_moe_ns . m prof_shexp_ns 1000000 )
            ( string_push_str pm `ms) dense=` )
            ( string_push_int pm / . m prof_dense_ns 1000000 )
            ( string_push_str pm `ms out=` )
            ( string_push_int pm / . m prof_out_ns 1000000 )
            ( string_push_str pm `ms` )
            ( nurl_eprintln ( string_data pm ) )
        } {}
    }
}

The profiler's totals (NURLLAMA_PROF) are printed when the model goes: its last owner drops the LlmImpl, and the compiler then drops every weight buffer, the tokenizer, the mappings and the device context.

@ llm_close sink Llm m → v

Early release (optional): the model's last owner gives everything back.

@ llm_tok Llm h → Tok

@ llm_n_vocab Llm h → i

@ llm_n_ctx Llm h → i

@ llm_logit Llm h i idx → f

@ llm_eval Llm h i token i pos → v

One decode step: token at position pos (0-based; caller feeds positions sequentially). Logits for the NEXT token land in logits_host. One decode step: token at position pos.

@ llm_eval_n Llm h ( Vec i ) ids i first i count i pos0 → v

Evaluate count tokens — ids[first .. first+count) — starting at absolute position pos0. The matmuls run over the whole batch; attention runs per position over the cache this same batch already filled (its causal window differs per position). Logits are produced for the LAST position only, which is all a prefill or a decode step needs.

@ llm_eval_win Llm h ( Vec i ) ids i first i count i pos0 i kvlen b want_logits → v

Diffusion window evaluation (llada2): the batch is a block window at positions pos0.., every query attends the SAME fixed cache range [0, kvlen) — bidirectional inside the window, all earlier blocks visible — and logits come back for EVERY position when wanted (the denoise step samples all of them). The window's K/V rows land in the cache at pos0.., overwriting the previous step's; earlier (frozen) blocks keep theirs, which is exactly the block-diffusion mask.

@ llm_win_id Llm h i row → i

After a mode-3 (greedy) window eval: the argmax token id and its softmax probability for window row row.

@ llm_win_prob Llm h i row → f

@ llm_win_greedy Llm h ( Vec i ) ids i first i count i pos0 i kvlen → v

Greedy window eval: run the forward and reduce to (id, prob) per position on the device (no full-vocab download).

@ llm_logit_at Llm h i row i idx → f

Logit idx of window row row after an all-positions eval (llm_eval_win with logits). Row 0 = the window's first position.

@ llm_mask_id Llm h → i

@ llm_is_diffusion Llm h → b

@ llm_logits_row Llm h i row → ( Vec f )

The logits of window row row (0 after a single-token eval), all n_vocab of them, copied out with the engine held once — a sampler that reads every logit should not take the lock per logit.

@ llm_logits Llm h → ( Vec f )

@ llm_chunk → i

The largest chunk a single llm_eval_n call may take.

@ llm_prefill Llm h ( Vec i ) ids → v

Prefill: run the whole prompt through the model in chunks, leaving the KV cache filled and the logits set for the LAST prompt token — the distribution the first generated token is drawn from.

chat.nu

packages/nurllama/src/chat.nu — chat templates.

Turns a message list into the exact prompt string a chat-tuned model was trained on. The template is chosen from the model's own GGUF metadata: tokenizer.chat_template is matched against the shapes we know, and the architecture/BOS conventions fill the rest — so a model carries its own dialect and the caller never guesses.

( chat_style_of g ) → i CHAT_* below ( chat_render style msgs ) → String prompt for generation ( chat_stop_matches style piece ) → b stop-token text guard

Styles: CHAT_LLAMA2 [INST] <<SYS>>…<</SYS>> user [/INST] assistant </s> CHAT_LLAMA3 <|start_header_id|>role<|end_header_id|>…<|eot_id|> CHAT_CHATML <|im_start|>role\n…<|im_end|> (qwen, many finetunes) CHAT_PHI3 <|user|>\n…<|end|>\n<|assistant|>\n (phi-3/3.5/4) CHAT_GEMMA <start_of_turn>user\n…<end_of_turn> (gemma; the assistant's role is spelled "model", and gemma has NO system role — the system prompt folds into the first user turn, which is what its own template does) CHAT_PLAIN no template — the messages are concatenated; a base (non-chat) model gets its raw prompt, which is the honest thing to do rather than inventing turns.

A ChatMsg's role is one of system / user / assistant.

API

: i CHAT_PLAIN 0

: i CHAT_LLAMA2 1

: i CHAT_LLAMA3 2

: i CHAT_CHATML 3

: i CHAT_GEMMA 4

: i CHAT_PHI3 5

: i CHAT_BAILING 6

: ChatMsg

: ChatMsg {
    String role
    String content
}

@ chat_msg s role s content → ChatMsg

@ chat_style_of Gguf g → i

Pick the style from the model's own chat template (substring match on the marker tokens each dialect must contain), falling back to PLAIN when the model ships no template — a base model, honestly served.

@ chat_render i style ( Vec ChatMsg ) msgs → String

@ chat_stop_matches i style s piece → b

Some chat models emit their turn terminator as ordinary text rather than the EOS id (a finetune whose eos_token_id was left at the base value). Treat the dialect's terminator as a stop sequence too.

history.nu

packages/nurllama/src/history.nu — the conversation store (SQLite).

Every chat a web client has with a model is persisted, scoped to the client's cookie GUID, in $NURLLAMA_HOME/history.db. Following the registry's proven idiom, the DB is opened FRESH per operation (its handle is % Drop and auto-closes at the arm's scope exit — never closed by hand) with WAL + a busy timeout, so concurrent requests serialise cheaply and nothing has to keep a Drop handle alive across the globals-are-scalars boundary. The schema is IF NOT EXISTS, so the first open on a fresh box creates it.

( hist_db_path root ) → String ( hist_ensure_client path guid now ) → b ( hist_new_conversation path guid model now ) → i rowid, -1 on error ( hist_add_message path conv role content now ) → b ( hist_conversation_owner path conv ) → String ("" = none) ( hist_list_conversations path guid ) → Json array ( hist_get_messages path conv ) → Json array ( hist_rename_conversation path conv guid title ) → b ( hist_delete_conversation path conv guid ) → b

API

@ hist_db_path String root → String

@ hist_now → i

@ hist_open s path → !Database SqliteErr

Open + harden + migrate. The Database is % Drop: the caller uses it inside the ?? arm and lets it auto-close — NEVER sqlite_close it.

@ hist_ensure_client s path s guid i now → b

@ hist_new_conversation s path s guid s model i now → i

@ hist_add_message s path i conv s role s content i now → b

@ hist_conversation_owner s path i conv → String

The GUID that owns conv, or "" when the conversation does not exist. The web layer checks this before returning any conversation's contents.

@ hist_list_conversations s path s guid → Json

A JSON array of this client's conversations, newest first: [ { "id":N, "model":"…", "title":"…", "created_at":T }, … ]

@ hist_get_messages s path i conv → Json

A JSON array of a conversation's messages in order: [ { "role":"user|assistant", "content":"…", "created_at":T }, … ]

@ hist_delete_conversation s path i conv s guid → b

Delete a conversation and its messages — only when guid owns it.

diffuse.nu

packages/nurllama/src/diffuse.nu — the block-diffusion decode loop (LLaDA2.x "JointThreshold + editing").

( dz_generate m prompt gen_len block_len threshold edit_thr max_post temp rng ) → ( Vec i )

The sequence is a template of gen_len mask tokens after the prompt, processed one block_len window at a time. Each denoise step evaluates the whole active window bidirectionally (llm_eval_win) and samples every position at once:

committed (all of them; if none clears it, the single most confident one is) — the parallel-decode half

when the model now prefers a different token above edit_thr — the LLaDA2.1 editing half

edits, or after max_post post-resolution passes

Completed blocks freeze: one final KV-only evaluation writes the block's settled content into the cache, and later windows attend to it without re-evaluating — the reference implementation recomputes the whole prefix every step; the cache makes each step O(window) instead of O(prefix).

Generation stops early when a completed block leaves no masks and an EOS has appeared (eos_early_stop), and the returned ids are cut BEFORE the first EOS — callers get exactly the text tokens.

API

@ _dz_geti ( Vec i ) v i k → i

@ _dz_getf ( Vec f ) v i k → f

@ dz_generate Llm m ( Vec i ) prompt i gen_len0 i block_len0 f threshold f edit_thr i max_post f temp Rng rng → ( Vec i )

kernels.nu

packages/nurllama/src/kernels.nu — the LLM decode ops as GPU kernels.

Every kernel is a CHECKED kernel (packages/gpukit): it reaches device memory only through LD / ST (and LD16 / LD32 / LD4) over the buffers its launcher binds, and every index is checked against that buffer's length. A launcher takes the buffers themselves — GkBuf for floats, GkBBuf for a weight's GGUF bytes — never their addresses, so a wrong size or offset can give a wrong number but never a read or write outside the model's memory: every launcher is a plain safe function, and so is the engine that calls them. The same kernels run on gpukit's CPU backend.

A launcher RECORDS its launch and appends it to a plan (a Vec GkLaunch); the engine records one evaluation's launches and replays them, giving the next call only the scalars that changed (its position).

Layout contracts (ggml row-major, ne0 fastest): weight W[out,in] → W[rin + c], matvec y[r] = Σc W[rin+c]·x[c] K/V cache → cache[t(n_kv·hd) + khhd + d] q/k activations → x[h*hd + d]

Offsets are scalars, not views: woff (bytes of a quantised W, elements of an f32 / f16 one), xoff and yoff (elements) move a launch inside its buffers — a MoE expert's rows, the last position's activation — without a view whose start would break the alignment the word loads rely on.

API

: LkW { i gt GkBBuf q GkBuf f GkHBuf h }

A weight tensor on the device, in its GGUF form: gt the ggml type and q its bytes (the quantised blocks — or, for the float types, the f32 / f16 elements' bytes), and for the float types the same memory as f32 (f, gt 0) or halves (h, gt 1). Q8_0 is held in the SPLIT row layout lk_q8_repack produces.

@ lk_w_none → LkW

@ lk_w_ok LkW w → b

@ lk_run ( Vec GkLaunch ) plan → b

Replay plan, in order. F on a device error.

@ lk_set inout ( Vec GkLaunch ) plan i at s name i x → b

Give scalar name of the plan's launch at the value x.

@ lk_matvec GpuKit kit b warp LkW w i woff GkBuf x i xoff GkBuf y i yoff i rows i cols i batch inout ( Vec GkLaunch ) plan → b

The matvec dispatch: w.gt is the ggml type of W. F when the type has no device kernel (the caller then falls back — correctness first). warp = the CUDA warp shapes; without it, the portable kernels.

@ lk_experts_supported i gt → b

True when the batched-expert matvec has a kernel for weight type gt: F32 (0) and Q8_0 (8) — the two the converter produces; other types fall back to the per-position loop in model.nu.

@ lk_matvec_experts GpuKit kit LkW w GkBuf x GkBuf y GkBuf expid i rows i cols i xk i n_assign inout ( Vec GkLaunch ) plan → b

One batched matvec over all n_assign expert assignments. F when gt has no batched kernel (caller falls back). (Two "better" shapes were measured SLOWER and stay deleted: warp-per-row — the shuffle beats the coalescing at these short rows — and expert-grouped tiles; the phase is not weight-bandwidth-bound.)

@ lk_moe_route GpuKit kit GkBuf logits GkBuf bias i boff GkBuf expid GkBuf wts i n_expert i n_group i topk_group i n_used f scale i norm i count inout ( Vec GkLaunch ) plan → b

@ lk_moe_scatter GpuKit kit GkBuf down GkBuf wts GkBuf out i K i dim i count inout ( Vec GkLaunch ) plan → b

Scatter-accumulate the experts' down-projections onto the residual: out[pos·dim + d] += Σ_j wts[pos·K+j] · down[(pos·K+j)·dim + d]. One thread per (pos, dim); a position's K assignments are contiguous, so no atomics.

@ lk_argmax_conf GpuKit kit b warp GkBuf logits GkBuf out_id GkBuf out_prob i vocab i count inout ( Vec GkLaunch ) plan → b

── Greedy argmax + confidence, on device ──────────────────────────

One position per thread (portable) or per warp (CUDA): the argmax token id and its softmax probability (1 / Σ exp(logit − max)), so only count·2 small values come back instead of count·vocab. out_id is an i64 buffer. The lowest-index tie-break reproduces the serial first-max argmax.

@ lk_rmsnorm GpuKit kit GkBuf x GkBuf w GkBuf y i n f eps i batch inout ( Vec GkLaunch ) plan → b

RMSNorm. On a GPU every thread recomputes its row's sum of squares — O(n²) a row, but norms are dwarfed by the matvecs at decode shapes and it needs no cross-thread sync. The CPU backend runs threads one after another, so there a thread takes a whole row: the sum once (in the same order — the same bits), then the row. gemma's RMSNorm is documented as x * inv * (1 + w), but the GGUF converter folds that +1 into the stored weights, so this is the plain form for every architecture. y must not be x: another thread would overwrite the row under the sum.

@ lk_rope GpuKit kit GkBuf x i nh i hd i rd i pos0 f base i style i batch inout ( Vec GkLaunch ) plan → b

Rotary embedding of the first rd dims of every head, at position pos0+b for batch element b. style 0 = ggml NORM (llama: adjacent pairs 2j, 2j+1), 1 = NEOX (qwen2, phi, …: the halves j, j + rd/2). Feeding a model the wrong one still runs and still produces fluent-looking garbage, which is why the architecture selects it rather than a heuristic.

@ lk_attn GpuKit kit b coop GkBuf q GkBuf kc GkBuf vc GkBuf out GkBuf sc i nh i nkv i hd i npos0 i batch i maxpos f qscale i window i causal inout ( Vec GkLaunch ) plan → b

Attention, two passes: one thread per (position, head, cache row) for the raw score, then the softmax over the row's scores and the value-weighted sum, a thread per (position, head, head-dim). coop (a backend whose block barrier is cheap — a GPU): a block per (position, head) shares the weights through shared memory; otherwise every thread walks its own row (the CPU backend runs a barrier's threads as fibers). qscale is the query scale (1/sqrt(head_dim) for llama; gemma states its own). window > 0 is a sliding window: a key older than window positions is not attended to (gemma3 on five layers of six). causal 1: batch element b attends its own window, npos0 + b cache rows (autoregressive decode); 0: every element attends the same npos0 rows (block diffusion). GQA: query head h reads kv head h/(nh/nkv). Appends two launches; npos0 is a scalar of both (lk_set).

: b sc2 ? coop ( __lk_rec kit c nl_attn_sc long long total = nh batch maxpos;`

: ~ GkCall c2 ( gk_call_new )

@ lk_axpy GpuKit kit GkBuf y i yoff GkBuf x i xoff f a i n inout ( Vec GkLaunch ) plan → b

y ← y + a·x — a routed expert's weighted down-projection onto the residual (yoff / xoff: the position's rows).

@ lk_silumul GpuKit kit GkBuf g GkBuf u i n inout ( Vec GkLaunch ) plan → b

SwiGLU gate: g = silu(g) · u, in place on g.

@ lk_gelumul GpuKit kit GkBuf g GkBuf u i n inout ( Vec GkLaunch ) plan → b

GeGLU — gemma's FFN gate. The tanh approximation is what gemma was trained with (gelu_pytorch_tanh), so the exact erf form would be the wrong function here, not a more accurate one.

@ lk_scale GpuKit kit GkBuf x f s i n inout ( Vec GkLaunch ) plan → b

x ← x · s (gemma scales the embedding row by sqrt(n_embd)).

@ lk_addv GpuKit kit GkBuf a GkBuf b i n inout ( Vec GkLaunch ) plan → b

Residual add, in place: a += b.

@ lk_addrow GpuKit kit GkBuf a GkBuf v i n i batch inout ( Vec GkLaunch ) plan → b

Broadcast add: every row of a batch gets the same vector (a bias).

@ lk_copyat GpuKit kit GkBuf dst GkBuf src i off i n inout ( Vec GkLaunch ) plan → b

Element copy into an offset — appends k/v rows into the cache (off moves with the position: lk_set).

@ lk_q8_repack GpuKit kit GkBBuf src GkBBuf dst i nb i nblk inout ( Vec GkLaunch ) plan → b

Repack an uploaded q8_0 tensor from ggml's interleaved 34-byte blocks ([d q0..q31] [d q0..q31] …) into the split per-row layout [d0 … d_{nb-1}] [q-block0 … q-block_{nb-1}], same nb·34-byte row stride. With an even nb the quant region is 4-byte aligned, which is what lets the matvecs read words and float4s instead of bytes (their limit is load-instruction count, not bandwidth). One thread per block, run once at model load.

api.nu

packages/nurllama/src/api.nu — an ollama-compatible HTTP API.

Speaks enough of ollama's wire protocol that existing clients work unchanged:

POST /api/generate { model, prompt, stream?, options? } POST /api/chat { model, messages[{role,content}], stream?, options? } → NDJSON: one JSON object per token while generating, then a final object with done:true (+ token counts). stream:false returns a single aggregated object. GET /api/tags the local store, as ollama's model list POST /api/show { model } → the model's metadata GET / "nurllama is running"

Streaming rides the http package's stream hook (http_app_stream): the handler owns the TcpConn and writes chunked NDJSON itself, so a token reaches the client the moment it is decoded.

One model is resident at a time (LRU of one): a request naming a different model swaps it in. Generation is serialised — the decode loop owns the device — which is honest for a single-GPU host; the server is single-worker so that serialisation is structural rather than a lock.

API

: ( Locked ? Llm ) g_api_llm ( __api_llm_blank ) // the resident model (F = none)

: ~ i g_api_style 0

: ~ i g_api_auth 0

Config for the wizard-started server (nurllama start): the default model the web UI chats with, the bearer token (empty = open access), and the SQLite history path.

: ApiCfg

: ApiCfg {
    String name
    String root
    String token
    String model
    String histpath
    String weights  // serve --weights (empty = none)
}

The server's text settings: the resident model's name (it changes as requests load and unload models), the model store's root, and the wizard's default model, bearer token, history path and --weights file. Handlers run on many threads, so they live in a Locked struct and are read as copies; as plain string globals, a model switch freed the name another request was comparing against.

: ( Locked ApiCfg ) g_api_cfg ( __api_cfg0 )

@ api_set_root String root → v

@ api_set_weights s w → v

serve --weights: when set, every model this server opens takes its tensors from this safetensors file instead of the GGUF's own (the GGUF still supplies the hyperparameters and the tokenizer) — the API twin of run --weights, for serving a finetuned/merged checkpoint. Empty = none.

@ api_serve String root s host i port s model → i

Serve on host:port. Single worker: the decode loop owns the device, so serialisation is structural rather than a lock. model (may be empty) is the default: the web UI chats with it, and an /api request that names no model of its own falls back to it.

@ api_serve_cfg String root NlConfig cfg → i

Serve with a full wizard config: a default model for the web UI, an optional bearer token, and the conversation-history database.

tokenizer.nu

packages/nurllama/src/tokenizer.nu — the GGUF vocabulary loader.

The tokenizer ENGINE now lives in packages/tokenizer: SentencePiece merging, byte-level BPE, inline special tokens, decoding — none of which has anything to do with the container the vocabulary arrived in.

What is left here is the part that IS about GGUF: pulling the pieces, scores, token types and merge rules out of tokenizer.ggml.* metadata and handing them to tok_build. whisper does the same from vocab.json + merges.txt, with the same engine behind it (deps/tokenizer/src/hf.nu).

( tok_new gguf ) → !Tok String

Everything else — tok_encode / tok_decode / tok_piece / tok_free / tok_bos / tok_eos / tok_unk / tok_n_vocab — comes from the package.

API

@ tok_new Gguf g → !Tok String

main.nu

nurllama — run language models locally.

nurllama start [-y] interactive setup wizard: pick a model, choose the bind scope (localhost / network) and access (open / bearer token) and the port, saved to config.json; launches the web chat server. A second run reuses the config (-y = without asking). The server hosts a web chat UI with per- browser cookie sessions and a SQLite conversation history. nurllama pull <src> [--name N] download into ~/.nurllama (hf.co/ORG/REPO/FILE.gguf or a URL; resumes a broken pull) nurllama list · rm <name> · verify <name> the local model store nurllama serve [model] [--host H] [--port N] ollama-compatible HTTP API

model is the default: the web UI chats with it, /api requests may name their own nurllama chat <model|name|hf-ref> interactive chat (model's own template from GGUF) nurllama run <model|name|hf-ref> <prompt> generate (stream to stdout) a model that is neither a local file nor a name in the store is a Hugging Face ref (org/repo/file.gguf) fetched on demand into the shared ~/.nurl/models cache via the hub package — no separate pull -n N (default 64) · --temp F (0 = greedy, default 0.8) --topk N · --topp F · --seed N · --ctx N diffusion models add: --block N · --threshold F · --edit-threshold F · --post-steps N (temp defaults to 0) nurllama convert <hf-dir|hf-ref> <out.gguf> HF checkpoint → GGUF (the checkpoint may be a local dir or an org/repo ref fetched via hub) nurllama finetune <model.gguf> <data.txt> LoRA finetune on the GPU [--out a.st] [--merged m.st] [--steps N] [--lr X] [--rank R] [--alpha A] [--seq T] [--seed S] [--f32] [--checkpoint c.st] [--save-every N] [--resume] [--window-stride N] (0 = spread over the corpus) --type q8_0 (default) | f16 | bf16 | f32 nurllama logits <model.gguf> <prompt> final-position logits, one per line (verification tap) nurllama dlogits / dgen diffusion verification taps (window logits / denoise ids) nurllama tokenize <model.gguf> <text> encode → token ids nurllama detok <model.gguf> <id> [id …] decode ids → bytes nurllama vocab <model.gguf> [n] dump the first n pieces nurllama selftest synthetic SPM + BPE vocab round-trips, bit-exact

Everything is loaded from the model file alone (tokenizer.ggml. + llama. metadata + tensors via packages/gguf); compute runs on packages/gpu — CUDA when present, NURL_GPU=cpu for the host backend.

API

: NlCnt

: NlCnt {
    i pass
    i fail
}

@ main → i

sample.nu

packages/nurllama/src/sample.nu — host-side sampling over the logits.

( sample_greedy m ) → i argmax ( sample_next m rng temp topk topp ) → i softmax(T) → top-k → top-p → draw

temp ≤ 0 collapses to greedy. top-k ≤ 0 disables the k filter; topp ≥ 1 disables the nucleus filter. Deterministic under a seeded Rng.

API

@ sample_greedy Llm m → i

@ sample_next Llm m Rng rng f temp i topk f topp → i

store.nu

packages/nurllama/src/store.nu — the local model store (~/.nurllama).

Content-addressed, cas-shaped, but streaming end to end — a blob is never fully resident:

<root>/blobs/sha256-<64hex> the model files, named by content <root>/manifests/<name> one small JSON per model name

A manifest records { name, url, digest, size } — the name is the user-facing handle (nurllama run <name>), the digest is the proof. Two names may point at the same blob (dedup by content); rm only deletes a blob once no manifest references it. verify re-hashes the blob through the incremental sha256 and refuses drift.

( nl_store_root ) → String $NURLLAMA_HOME | ~/.nurllama ( nl_resolve root nameOrPath ) → ?String path to a model file ( nl_store_add root name url hex size ) → !v String ( nl_store_list root ) → v prints the table ( nl_store_rm root name ) → !v String ( nl_store_verify root name ) → !v String ( nl_blob_path root hex ) ( nl_manifest_path root name ) ( nl_hash_file path ) → !String String streaming sha256 hex

API

@ nl_store_root → String

@ nl_blob_path String root s hex → String

@ nl_manifest_path String root s name → String

@ nl_store_init String root → !v String

@ nl_hash_file s path → !String String

Streaming sha256 of a file — constant memory whatever the size.

@ nl_store_add String root s name s url s hex i size → !v String

@ nl_resolve String root s arg → ?String

Resolve a model argument: an existing file path wins; otherwise the store's manifest name → blob path. None = unknown.

@ nl_store_list String root → v

@ nl_store_rm String root s name → !v String

@ nl_store_verify String root s name → !v String

Re-hash the blob and compare with the manifest digest — the store's integrity loop (mirrors cas_get's refuse-on-mismatch, streaming).