NURL registrynurl-lang.org →packages/nurllama/src/pull.nu — streaming model download with resume.
The whole transfer is constant-memory: HTTP chunks flow straight to disk through file_write_chunk while the incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<name>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and a manifest records name → digest.
( nl_resolve_url src ) → String hf.co/ORG/REPO/FILE shorthand → the HF resolve URL; http(s):// passes through ( nl_default_name src ) → String basename minus .gguf ( nl_pull root src name ) → !v String
@ nl_resolve_url s src → Stringhf.co/ORG/REPO/path/to/file.gguf → https://huggingface.co/ORG/REPO/resolve/main/path/to/file.gguf
@ nl_default_name s src → Stringlast path segment, ".gguf" stripped — the default store name
@ nl_pull String root s src s name_override → !v Stringpackages/nurllama/src/start.nu — the nurllama start wizard.
Interactive setup: pick a model from the models directory (and the pull store), choose whether the server binds to localhost only or is reachable from the network, whether a network server is open or gated by a bearer token, and the port. The answers are written to $NURLLAMA_HOME/config.json (see config.nu) and the server starts.
A second nurllama start finds the config and offers to reuse it; nurllama start -y reuses it without asking.
: ModelEntry: ModelEntry {
String value
String label
}@ nurllama_start ArgParser p → inurllama start [-y]. Returns the server's exit code.
nurllama/finetune.nu — LoRA finetuning over the grad tape (M6b).
Loads a GGUF model's weights as f64 HOST tensors (gguf_dequant_f64 — independent of the quantised GPU inference path) and builds the whole transformer forward + cross-entropy loss as ONE grad-tape graph, with LoRA adapter pairs (A[in,r], B[r,out], y = x·W0 + (α/r)·(x·A)·B) on q/k/v/o and gate/up/down as the only parameters. Train by capturing the episode with gput and replaying on the device; the frozen base weights cost no gradient memory or backward compute (grad 0.3.0's requires-grad propagation).
Architectures: llama-family, qwen2 (bias'd q/k/v, NEOX rope) and qwen3 (no biases, NEOX rope, per-head Q/K RMSNorm before the rotation). The llama family's NORM-style rope rotates ADJACENT lanes (2j, 2j+1), which a contiguous-slice tape cannot express — so the loader UN-PERMUTES the q/k projection columns per head (evens first, odds second) and the graph applies half-split NEOX rope: every pair sees the identical rotation at a permuted lane, and attention scores are permutation-invariant, so the values downstream are exactly the model's. (v is untouched — context and o-proj stay in the original layout.)
Layouts: GGUF stores [out, in] row-major; the tape multiplies x[T,in] · W[in,out], so every weight is transposed once at load. Embedding lookup happens HOST-side (the rows enter the tape as a const; embeddings are frozen, so no gather op is needed). Tied-embedding models reuse token_embd for the lm_head (transposed the other way).
: FtV: FtV {
b has
( Vec f ) v
}An optional per-layer 1-D tensor (norm weight / bias): has says whether the model carries it (llama has no attn_q_norm), v holds its values (empty for a shape-only placeholder).
: FtW: FtW {
i rows
i cols
( Vec f ) data
}One weight in tape layout: data [rows=in, cols=out] flat.
: FtModelImpl: FtModelImpl {
b ok
i n_embd
i n_layer
i n_head
i n_kv
i head_dim
i n_ff
i n_vocab
f eps
f rope_base
i rope_dim
i rope_style // 0 llama/NORM (columns un-permuted at load) · 1 NEOX
( Vec f ) embd // [n_vocab, n_embd] row-major (host lookup table)
FtW wout // lm_head [n_embd, n_vocab] (tied → transposed embd)
( Vec f ) norm_f // output_norm [n_embd]
( Vec FtW ) wq ( Vec FtW ) wk ( Vec FtW ) wv ( Vec FtW ) wo
( Vec FtW ) wg ( Vec FtW ) wu ( Vec FtW ) wd
( Vec FtV ) bq ( Vec FtV ) bk ( Vec FtV ) bv // qwen2's biases (absent elsewhere)
( Vec FtV ) an ( Vec FtV ) fn // attn_norm / ffn_norm per layer
// qwen3's per-head Q/K RMSNorm weights [head_dim]. Absent for
// llama/qwen2 — an absent one simply skips the norm.
( Vec FtV ) qn ( Vec FtV ) kn
String src_path // the GGUF path — so the per-layer base matrices can be
// freed after device capture (ft_drop_base) and streamed back for the
// merge (ft_reload_base), keeping host RAM off the base during training
// STREAMING mode (ft_set_stream). The base weights are never all resident
// host-side: ft_open reads their SHAPES only, ft_graph declares them as
// lazy consts, and ft_stream_upload fills each one straight into its
// device buffer after the capture, freeing it again before the next.
// Host peak becomes one layer instead of the whole model — the
// difference between training a 4B model on a 31 GB box and not.
b stream
( Vec i ) lz_node // tape node id of each lazy base const
( Vec i ) lz_key // its identity: layer * 16 + slot (__ft_base_name)
Gguf mgg // a Gguf held open across a streamed merge (gguf_none = none)
}: FtModel { s ctl }An FtModel is a handle on its state in an rcbox (stdlib/core/rcbox.nu): every copy is the same state, and the last owner releases it.
@ FtModel_drop sink FtModel h → v: ~ b __ft_stream_on FStreaming base upload, OFF by default: the finetune tests compare the CPU tape's loss against the device's, and a lazy const has no host values to compare with. nurllama finetune --stream turns it on for models whose f64 host copy would not fit.
@ ft_set_stream b on → v@ ft_free sink FtModel m → vLet go of m now rather than at the end of its owner's scope.
: FtG: FtG {
GVar loss
GVar logits
GVar xin // the input-rows const — refresh per window via gput_set_input
GVar ohin // the one-hot target const — refreshed together with xin
i n_pairs // LoRA pairs registered (7 per layer)
}The result handles of one built graph.
: FtTrain: FtTrain {
b ok
f l0
f l1
( Vec f ) aflat
( Vec f ) bflat
}One trained run's result: per-slot adapter values, flat in slot order (slot = layer·7 + {q k v o gate up down}); A blocks then B blocks.
@ ft_auto_stride i nwin → iThe golden-ratio point of [1, nwin), nudged up to the nearest value coprime with nwin — consecutive steps land maximally far apart.
@ ft_win_at i st i nwin i stride → i@ ft_train FtModel m__h ( Vec i ) corpus i win i r f alpha i seed i steps f lr i dtype b verbose → FtTrain@ ft_merge_st s path FtModel m__h FtTrain t i r f alpha → !v StringMerge every adapter into its base weight and write the FULL model as a safetensors file — run it with nurllama's --weights path (llm_open_st).
@ nurllama_finetune s modp s datap s outp s mergedp i steps f lr i rank f alpha i seq i seed b f32 b mixed s ckptp i ckevery b resume i wstride b mergeonly → i── the CLI driver ─────────────────────────────────────────────────── nurllama finetune <model.gguf> <data.txt>: tokenize the corpus with the model's own tokenizer, LoRA-train on the DEVICE over the first seq tokens (v1 trains one fixed window — the captured graph has a fixed shape; multi-window scheduling lands with gput_set_input plumbing), save the adapters, and optionally write a merged full-model safetensors runnable via nurllama run model.gguf PROMPT --weights merged.st.
packages/nurllama/src/convert.nu — HF checkpoint → GGUF, streaming.
( nurllama_convert hfdir outpath otype ) → i 0 = ok
Converts a Hugging Face checkpoint directory (config.json + tokenizer.json + model*.safetensors shards) into a GGUF that llama.cpp would recognise — same architecture name, metadata keys, tensor names, shapes and fusions — so the file interoperates in both directions. Currently supported: llada2_moe (LLaDA2.x diffusion MoE), written as the llada2 architecture following llama.cpp's BailingMoeV2 conventions:
(blk.N.attn_qkv.weight, HF row order, no permutation — NEOX rotary semantics are preserved by construction)
fuse into one 3D blk.N.ffn_{gate,up,down}_exps.weight each
reference model routes on an fp32 sigmoid, and the router is ~2 MB per layer — precision is free here)
pre = bailingmoe2, id-ordered tokens, merges, added tokens as CONTROL, [PAD…] filler up to config vocab_size)
The conversion STREAMS: shards are mmapped, each tensor is dequantised (bf16→f32) and encoded (Q8_0/F16/BF16/F32) in row chunks straight into the gguf streaming writer — peak memory is one chunk, never the model. A 32 GB checkpoint converts on a 16 GB machine.
: i CV_F32 0ggml tensor type ids (gguf.nu names them; the writer wants the raw id)
: i CV_F16 1: i CV_Q8_0 8: i CV_BF16 30: i CV_TOK_NORMAL 1token types (llama.cpp enum)
: i CV_TOK_CONTROL 3: i CV_TOK_USER 4: i CV_TOK_UNUSED 5: CvJobs: CvJobs {
( Vec String ) names
( Vec i ) gts
( Vec i ) nds
( Vec i ) d0s
( Vec i ) d1s
( Vec i ) d2s
( Vec String ) srcs
}One conversion job = one GGUF tensor. src is the HF tensor name, or the marker experts:<layer>:<proj> for a 256-way fusion.
: Cv: Cv {
( Vec St ) sts
( Vec String ) stpaths
Json cfg
b cfg_ok
i n_layer
i n_embd
i n_head
i n_kv
i head_dim
i n_ff
i n_vocab
i n_ctx
i n_expert
i n_expert_used
i n_group
i topk_group
i n_shared
i first_dense
i moe_ff
f rope_base
f eps
f route_scale
b norm_topk
f partial_rotary
}: CvVocab: CvVocab {
( Vec String ) tokens
( Vec i ) types
( Vec String ) merges
i bos_id
i eos_id
i pad_id
i mask_id
b ok
}@ nurllama_convert s hfdir s outpath s otype → i@ _cv_geti ( Vec i ) v i k → ipackages/nurllama/src/webui.nu — the embedded web chat UI.
A single self-contained page (no external requests — CSP-friendly) served at / to browsers. It talks to the /web/* JSON endpoints, carries the per-client GUID in the nl_client cookie the server sets, and (when the server was started with token auth) prompts for the token once and sends it as a Bearer header on every request.
The markup avoids backtick and backslash characters so it embeds cleanly in a NURL backtick string.
@ webui_html → spackages/nurllama/src/config.nu — the nurllama start config file.
A small JSON file at $NURLLAMA_HOME/config.json (default ~/.nurllama/config.json) that remembers the wizard's choices so a second nurllama start can reuse them:
{ "model": "<path or store name>", "host": "127.0.0.1" | "0.0.0.0", "port": 11434, "auth": "open" | "token", "token": "<bearer token, when auth=token>", "models_dir": "<directory the wizard scans for *.gguf>" }
( cfg_path root ) → String the config file path (owned) ( cfg_exists root ) → b ( cfg_load root ) → ?NlConfig ( cfg_save root cfg ) → b
: NlConfig: NlConfig {
String model
String host
i port
String auth
String token
String models_dir
}@ cfg_path String root → String@ cfg_exists String root → b@ cfg_load String root → ?NlConfigLoad the config, or None when it is absent or unparseable.
@ cfg_save String root NlConfig c → bSerialise + write. Returns T on success. Creates $NURLLAMA_HOME if it does not exist (the store already does, but a fresh box may not).
packages/nurllama/src/model.nu — the llama-architecture forward pass.
Loads a GGUF model onto the device (CUDA, or the CPU/OpenMP backend via NURL_GPU=cpu — same kernels either way) and runs autoregressive decode with a device-resident KV cache:
( llm_open path n_ctx ) → !Llm String ( llm_eval m token pos ) → v one decode step; logits land in the reused host buffer ( llm_logit m i ) → f read logit i of the last eval ( llm_n_vocab m ) ( llm_tok m ) the model's tokenizer rides along Llm is a handle: its last owner releases the weights, the KV cache and the device context; ( llm_close m ) is an optional early release.
Weights stay QUANTISED on the device: a matvec whose weight type has a device kernel (F32/F16/Q4_0/Q8_0/Q4_K/Q6_K) uploads the GGUF bytes verbatim and decodes each block inside the matmul, so a Q4_K model needs ~7× less device memory than its f32 expansion (and the memory-bound matvec reads ~7× fewer bytes). Any other type falls back to host dequantisation through gguf_dequant — correctness first, always. Activations, KV cache and logits are f32.
The token embedding is dequantised host-side once (its rows upload one at a time), which costs host RAM but no device memory. The prompt is processed in CHUNKS: the matmuls run over up to LM_CHUNK positions at once, so a 300-token prompt costs a few hundred kernel launches instead of a few tens of thousands, and each weight is read once per chunk instead of once per token. Attention still runs per position (it is not the bottleneck, and the causal window differs per position anyway) — over a cache the same chunk already filled.
Architectures: the llama shape — RMSNorm → GQA attention with NORM-style RoPE → SwiGLU FFN, residuals around both — which llama and qwen2 both are. Hyperparameters are read under the model's own key prefix (llama.* / qwen2.*), and qwen2's per-layer Q/K/V biases are applied when present (llama models simply have none). qwen3 is that same shape with two differences, both already tensor-driven: it drops the Q/K/V biases, and it RMSNorms every head's Q and K vector before the rotation (attn_q_norm/attn_k_norm, the per-head norm gemma3 introduced). Nothing about it is special-cased beyond the NEOX rotary style — a weight that is absent is simply not applied. Tied-embedding models (no output.weight) reuse token_embd as the output projection.
: LlmImpl: LlmImpl {
Gpu g
LlmKernels ks
Tok tok
i n_embd
i n_layer
i n_head
i n_kv
i head_dim
i n_ff
i n_vocab
i n_ctx
i rope_dim
i rope_style
f eps
f rope_base
// gemma: the attention projections do NOT have to land back on n_embd —
// gemma3-270m has n_embd 640, 4 heads and head_dim 256, so q is 1024 wide
// and `wo` maps 1024 → 640. q_dim is nh*head_dim, never assumed = n_embd.
i q_dim
f embd_scale
f attn_scale
b ffn_gelu
i swa_window
i swa_period
f rope_base_swa
// ── llada2 (diffusion MoE) ──────────────────────────────────────
// bidir: attention is NOT causal — the diffusion decode evaluates a
// whole block window at once and every query sees the full window.
b bidir
i mask_id
i first_dense
i n_expert
i n_expert_used
i n_group
i n_group_used
i moe_ff
i shexp_ff
f route_scale
b weights_norm
( Vec i ) gate_inp
( Vec i ) tq_gate_inp
( Vec f ) exp_bias
( Vec i ) gate_exps
( Vec i ) up_exps
( Vec i ) down_exps
( Vec i ) tq_gate_exps
( Vec i ) tq_up_exps
( Vec i ) tq_down_exps
( Vec i ) gate_shexp
( Vec i ) up_shexp
( Vec i ) down_shexp
( Vec i ) tq_gate_shexp
( Vec i ) tq_up_shexp
( Vec i ) tq_down_shexp
i rlogitsd
GpuHost router_host
i moe_outd
// batched-MoE scratch: the assignment table (expert id + weight per
// (position, selected-expert) pair) and the expert hidden/down buffers
i moe_expid
i moe_wts
i moe_gate
i moe_up
i moe_downb
// all layers' expert bias as f32 on the device (n_layer · n_expert),
// read at offset L · n_expert by the on-device router
i bias_all
Gguf gg
// A SECOND weight source: the same model, in the container Hugging Face
// ships it in. The GGUF still supplies the hyperparameters and the
// tokenizer; the tensors come from here when it is open (st_is_open).
// This is how the safetensor reader is proven: run the same model from
// both containers and require the logits to agree.
St st
b st_norm_add1
i st_embd
i embd_idx
( Vec u ) embd_host
( Vec i ) wdptr
( Vec i ) wbytes
( Vec GpuBuffer ) wbufs // the owners of wdptr's allocations (released with them)
( Vec i ) tq_attn_norm
( Vec i ) tq_wq
( Vec i ) tq_wk
( Vec i ) tq_wv
( Vec i ) bq
( Vec i ) bk
( Vec i ) bv
( Vec i ) tq_wo
( Vec i ) tq_ffn_norm
( Vec i ) tq_gate
( Vec i ) tq_down
( Vec i ) tq_up
i tq_output
( Vec i ) attn_norm
( Vec i ) q_norm
( Vec i ) k_norm
( Vec i ) post_attn_norm
( Vec i ) post_ffn_norm
( Vec i ) wq
( Vec i ) wk
( Vec i ) wv
( Vec i ) wo
( Vec i ) ffn_norm
( Vec i ) w_gate
( Vec i ) w_down
( Vec i ) w_up
i out_norm
i w_output
( Vec i ) kcache
( Vec i ) vcache
i xd
i xnd
i qd
i kd
i vd
i aod
i tmpd
// rmsnorm reads the whole row to form its sum of squares, so it can never
// write back into its own input: another block would already have
// overwritten the row under it. gemma's Q/K and post-block norms
// therefore need somewhere to land.
i qn2d
i kn2d
i tn2d
i gd
i ud
i scored
i logitsd
GpuHost logits_host
// greedy argmax + confidence per window position (device + host)
i amid_d
i amprob_d
GpuHost amid_h
GpuHost amprob_h
// coarse phase profiler (NURLLAMA_PROF): sync+timer per phase
b prof_on
i prof_attn_ns
i prof_moe_ns
i prof_shexp_ns
i prof_dense_ns
i prof_out_ns
}: Llm { s ctl }A Llm is a handle on its state in an rcbox (stdlib/core/rcbox.nu): every copy is the same state, and the last owner releases it.
@ Llm_drop sink Llm h → v@ _lm_geti ( Vec i ) v i k → i@ string_eq_raw String a s raw → b: ~ i __lm_last_type -1Publishes the uploaded tensor's ggml type (0 when it was host-dequantised to f32), so the caller records which matvec kernel the weight needs.
: ~ i __lm_up_ns 0: ~ i __lm_rp_ns 0: ~ b __lm_alloc_failed FSet when any device allocation comes back null (out of device memory). Checked once at the end of llm_open_st: a silent failure here used to surface as an all-zero forward pass — the model "ran" and produced token 0 forever — instead of an error naming the cause.
: i LM_CHUNK 64Positions per prefill chunk. Bounds the batched activation scratch (chunk × n_ff floats) while keeping the launches amortised.
@ llm_open s path i want_ctx → !Llm String: ~ b __lm_lprof FSame model, weights from a safetensors file instead of the GGUF's own tensors. The GGUF still supplies the hyperparameters and the tokenizer — safetensors carries neither (they live in config.json and tokenizer.json next to it) — so this is a weight SOURCE, not a second model format.
It exists to be checked: run one model through both containers and the logits must agree. If the safetensors reader got a dtype, an offset or a row order wrong, that check fails here rather than silently in whatever depends on it next. Load-phase stopwatch (NURLLAMA_PROF): prints [load] <what>=<ms>ms.
: ~ i __lm_lt 0% Drop LlmImpl% Drop LlmImpl {
@ drop LlmImpl m → v {
? . m prof_on {
: String pm ( string_from `[prof] attn=` )
( string_push_int pm / . m prof_attn_ns 1000000 )
( string_push_str pm `ms shexp=` )
( string_push_int pm / . m prof_shexp_ns 1000000 )
( string_push_str pm `ms moe_total=` )
( string_push_int pm / . m prof_moe_ns 1000000 )
( string_push_str pm `ms (experts=` )
( string_push_int pm / - . m prof_moe_ns . m prof_shexp_ns 1000000 )
( string_push_str pm `ms) dense=` )
( string_push_int pm / . m prof_dense_ns 1000000 )
( string_push_str pm `ms out=` )
( string_push_int pm / . m prof_out_ns 1000000 )
( string_push_str pm `ms` )
( nurl_eprintln ( string_data pm ) )
} {}
}
}The profiler's totals (NURLLAMA_PROF) are printed when the model goes: its last owner drops the LlmImpl, and the compiler then drops every weight buffer, the tokenizer, the mappings and the device context.
@ llm_close sink Llm m → vEarly release (optional): the model's last owner gives everything back.
@ llm_into_word sink Llm m → iA model kept in a module global (a server's resident model) as one word: llm_into_word makes the word one owner of the model, a view of it is # Llm w, and llm_word_release gives that owner back.
@ llm_word_release i w → v@ llm_eval Llm m__h i token i pos → vOne decode step: token at position pos (0-based; caller feeds positions sequentially). Logits for the NEXT token land in logits_host. One decode step: token at position pos.
@ llm_eval_n Llm m__h ( Vec i ) ids i first i count i pos0 → vEvaluate count tokens — ids[first .. first+count) — starting at absolute position pos0. The matmuls run over the whole batch; attention runs per position over the cache this same batch already filled (its causal window differs per position). Logits are produced for the LAST position only, which is all a prefill or a decode step needs.
@ llm_eval_win Llm m__h ( Vec i ) ids i first i count i pos0 i kvlen b want_logits → vDiffusion window evaluation (llada2): the batch is a block window at positions pos0.., every query attends the SAME fixed cache range [0, kvlen) — bidirectional inside the window, all earlier blocks visible — and logits come back for EVERY position when wanted (the denoise step samples all of them). The window's K/V rows land in the cache at pos0.., overwriting the previous step's; earlier (frozen) blocks keep theirs, which is exactly the block-diffusion mask.
@ llm_win_greedy Llm m__h ( Vec i ) ids i first i count i pos0 i kvlen → vGreedy window eval: run the forward and reduce to (id, prob) per position on the device (no full-vocab download).
@ llm_chunk → iThe largest chunk a single llm_eval_n call may take.
@ llm_prefill Llm m__h ( Vec i ) ids → vPrefill: run the whole prompt through the model in chunks, leaving the KV cache filled and the logits set for the LAST prompt token — the distribution the first generated token is drawn from.
packages/nurllama/src/chat.nu — chat templates.
Turns a message list into the exact prompt string a chat-tuned model was trained on. The template is chosen from the model's own GGUF metadata: tokenizer.chat_template is matched against the shapes we know, and the architecture/BOS conventions fill the rest — so a model carries its own dialect and the caller never guesses.
( chat_style_of g ) → i CHAT_* below ( chat_render style msgs ) → String prompt for generation ( chat_stop_matches style piece ) → b stop-token text guard
Styles: CHAT_LLAMA2 [INST] <<SYS>>…<</SYS>> user [/INST] assistant </s> CHAT_LLAMA3 <|start_header_id|>role<|end_header_id|>…<|eot_id|> CHAT_CHATML <|im_start|>role\n…<|im_end|> (qwen, many finetunes) CHAT_PHI3 <|user|>\n…<|end|>\n<|assistant|>\n (phi-3/3.5/4) CHAT_GEMMA <start_of_turn>user\n…<end_of_turn> (gemma; the assistant's role is spelled "model", and gemma has NO system role — the system prompt folds into the first user turn, which is what its own template does) CHAT_PLAIN no template — the messages are concatenated; a base (non-chat) model gets its raw prompt, which is the honest thing to do rather than inventing turns.
A ChatMsg's role is one of system / user / assistant.
: i CHAT_PLAIN 0: i CHAT_LLAMA2 1: i CHAT_LLAMA3 2: i CHAT_CHATML 3: i CHAT_GEMMA 4: i CHAT_PHI3 5: i CHAT_BAILING 6: ChatMsg: ChatMsg {
String role
String content
}@ chat_msg s role s content → ChatMsg@ chat_style_of Gguf g → iPick the style from the model's own chat template (substring match on the marker tokens each dialect must contain), falling back to PLAIN when the model ships no template — a base model, honestly served.
@ chat_render i style ( Vec ChatMsg ) msgs → String@ chat_stop_matches i style s piece → bSome chat models emit their turn terminator as ordinary text rather than the EOS id (a finetune whose eos_token_id was left at the base value). Treat the dialect's terminator as a stop sequence too.
packages/nurllama/src/history.nu — the conversation store (SQLite).
Every chat a web client has with a model is persisted, scoped to the client's cookie GUID, in $NURLLAMA_HOME/history.db. Following the registry's proven idiom, the DB is opened FRESH per operation (its handle is % Drop and auto-closes at the arm's scope exit — never closed by hand) with WAL + a busy timeout, so concurrent requests serialise cheaply and nothing has to keep a Drop handle alive across the globals-are-scalars boundary. The schema is IF NOT EXISTS, so the first open on a fresh box creates it.
( hist_db_path root ) → String ( hist_ensure_client path guid now ) → b ( hist_new_conversation path guid model now ) → i rowid, -1 on error ( hist_add_message path conv role content now ) → b ( hist_conversation_owner path conv ) → String ("" = none) ( hist_list_conversations path guid ) → Json array ( hist_get_messages path conv ) → Json array ( hist_rename_conversation path conv guid title ) → b ( hist_delete_conversation path conv guid ) → b
@ hist_db_path String root → String@ hist_now → i@ hist_open s path → !Database SqliteErrOpen + harden + migrate. The Database is % Drop: the caller uses it inside the ?? arm and lets it auto-close — NEVER sqlite_close it.
@ hist_ensure_client s path s guid i now → b@ hist_new_conversation s path s guid s model i now → i@ hist_add_message s path i conv s role s content i now → b@ hist_conversation_owner s path i conv → StringThe GUID that owns conv, or "" when the conversation does not exist. The web layer checks this before returning any conversation's contents.
@ hist_list_conversations s path s guid → JsonA JSON array of this client's conversations, newest first: [ { "id":N, "model":"…", "title":"…", "created_at":T }, … ]
@ hist_get_messages s path i conv → JsonA JSON array of a conversation's messages in order: [ { "role":"user|assistant", "content":"…", "created_at":T }, … ]
@ hist_delete_conversation s path i conv s guid → bDelete a conversation and its messages — only when guid owns it.
packages/nurllama/src/diffuse.nu — the block-diffusion decode loop (LLaDA2.x "JointThreshold + editing").
( dz_generate m prompt gen_len block_len threshold edit_thr max_post temp rng ) → ( Vec i )
The sequence is a template of gen_len mask tokens after the prompt, processed one block_len window at a time. Each denoise step evaluates the whole active window bidirectionally (llm_eval_win) and samples every position at once:
threshold arecommitted (all of them; if none clears it, the single most confident one is) — the parallel-decode half
when the model now prefers a different token above edit_thr — the LLaDA2.1 editing half
edits, or after max_post post-resolution passes
Completed blocks freeze: one final KV-only evaluation writes the block's settled content into the cache, and later windows attend to it without re-evaluating — the reference implementation recomputes the whole prefix every step; the cache makes each step O(window) instead of O(prefix).
Generation stops early when a completed block leaves no masks and an EOS has appeared (eos_early_stop), and the returned ids are cut BEFORE the first EOS — callers get exactly the text tokens.
& libm @ exp f x → f@ _dz_geti ( Vec i ) v i k → i@ _dz_getf ( Vec f ) v i k → f@ dz_generate Llm m ( Vec i ) prompt i gen_len0 i block_len0 f threshold f edit_thr i max_post f temp Rng rng → ( Vec i )packages/nurllama/src/kernels.nu — the LLM decode ops as GPU kernels.
CUDA-C sources compiled once through packages/gpu (NVRTC on the CUDA backend; the SAME source runs on the CPU/OpenMP backend, so every kernel below deliberately avoids shared / syncthreads — each thread computes its output independently, exactly like the onnx op set). Decode-step shapes (one token at a time) keep the naive forms honest: the matvecs dominate and each row is one thread's serial dot product.
Layout contracts (ggml row-major, ne0 fastest): weight W[out,in] → W[rin + c], matvec y[r] = Σc W[rin+c]·x[c] K/V cache → cache[t(n_kv·hd) + khhd + d] q/k activations → x[h*hd + d]
: LlmKernels: LlmKernels {
GpuKernel matvec
GpuKernel mv_q4_0
GpuKernel mv_q8_0
GpuKernel mv_f32_b
GpuKernel mv_q8_0_b
GpuKernel mv_q5_0
GpuKernel mv_q5_1
GpuKernel mv_q4_k
GpuKernel mv_q5_k
GpuKernel mv_q6_k
GpuKernel mv_f16
GpuKernel w_f32
GpuKernel w_q4_0
GpuKernel w_q8_0
GpuKernel w_q4_k
GpuKernel w_q6_k
GpuKernel w_q5_0
GpuKernel w_q5_1
GpuKernel w_q5_k
b warp
GpuKernel rmsnorm
GpuKernel rope
GpuKernel rope_neox
GpuKernel attn
GpuKernel attn_sc
GpuKernel attn_out
GpuKernel silumul
GpuKernel gelumul
GpuKernel scale
GpuKernel addv
GpuKernel addrow
GpuKernel copyat
GpuKernel axpy
GpuKernel mv_exp_f32
GpuKernel mv_exp_q8_0
GpuKernel moe_scatter
GpuKernel moe_route
GpuKernel argmax_conf
GpuKernel argmax_conf_w
GpuKernel q8_repack
b ok
}@ lk_build Gpu g → LlmKernels@ lk_matvec LlmKernels ks i wd i xd i yd i rows i cols i batch → v@ lk_matvec_q LlmKernels ks i gt i wd i xd i yd i rows i cols i batch → bQuantised matvec dispatch: gt is the ggml tensor type of W. Returns F when the type has no device kernel (the caller then falls back to the f32 path — correctness first, always).
@ lk_experts_supported LlmKernels ks i gt → bTrue when the batched-expert matvec has a kernel for weight type gt. Only F32 (0) and Q8_0 (8) so far — the two the converter produces and the reference path uses; other types fall back to the per-position loop in model.nu.
@ lk_matvec_experts LlmKernels ks i gt i wd i xd i yd i expid_dev i rows i cols i xk i n_assign → bOne batched matvec over all n_assign expert assignments. Output row a uses expert expid_dev[a] and input row a/xk of x. Returns F when gt has no batched kernel (caller falls back).
@ lk_moe_route LlmKernels ks i logitsd i biasd i expid_dev i wts_dev i n_expert i n_group i topk_group i n_used f scale i norm i count → vOn-device routing: fills expid_dev / wts_dev [count·n_used] from the router logits and the expert bias. One thread per position.
@ lk_q8_repack LlmKernels ks i srcd i dstd i nb i nblk → vGreedy argmax + softmax-prob per position, on device. Repack one uploaded q8_0 tensor into the split per-row layout (see __lk_q8_repack). nb = quant blocks per row, nblk = blocks total.
@ lk_argmax_conf LlmKernels ks i logitsd i outid i outprob i vocab i count → v@ lk_moe_scatter LlmKernels ks i downd i wtsd i outd i K i dim i count → vout[pos] += Σ_j wts[posK+j] · down[(posK+j)] over dim floats.
@ lk_rmsnorm LlmKernels ks i xd i wd i yd i n f eps i batch → v@ lk_rope LlmKernels ks i xd i nh i hd i rd i pos f base i style i batch → vstyle: 0 = NORM (llama), 1 = NEOX (qwen2)
@ lk_attn LlmKernels ks i qd i kcd i vcd i outd i scd i nh i nkv i hd i npos0 i batch i maxpos f qscale i window i causal → vnpos0 = the causal window of the FIRST batch element (element b sees npos0 + b rows). maxpos strides the per-(batch,head) score scratch. Two-pass attention: scores, then softmax + value-weighted sum.
@ lk_axpy LlmKernels ks i yd i xd f a i n → vy ← y + a·x — the routed-expert accumulator (each selected expert's down-projection lands weighted onto the residual stream).
@ lk_gelumul LlmKernels ks i gd i ud i n → vGeGLU: g ← gelu(g) * u, in place. Same shape as lk_silumul.
@ lk_scale LlmKernels ks i xd f s i n → vx ← x * s, in place.
@ lk_silumul LlmKernels ks i gd i ud i n → v@ lk_addv LlmKernels ks i ad i bd i n → v@ lk_addrow LlmKernels ks i ad i vd i n i batch → va[b][i] += v[i] for every row of the batch
@ lk_copyat LlmKernels ks i dstd i srcd i off i n → vpackages/nurllama/src/api.nu — an ollama-compatible HTTP API.
Speaks enough of ollama's wire protocol that existing clients work unchanged:
POST /api/generate { model, prompt, stream?, options? } POST /api/chat { model, messages[{role,content}], stream?, options? } → NDJSON: one JSON object per token while generating, then a final object with done:true (+ token counts). stream:false returns a single aggregated object. GET /api/tags the local store, as ollama's model list POST /api/show { model } → the model's metadata GET / "nurllama is running"
Streaming rides the http package's stream hook (http_app_stream): the handler owns the TcpConn and writes chunked NDJSON itself, so a token reaches the client the moment it is decoded.
One model is resident at a time (LRU of one): a request naming a different model swaps it in. Generation is serialised — the decode loop owns the device — which is honest for a single-GPU host; the server is single-worker so that serialisation is structural rather than a lock.
: ~ i g_api_llm 0The resident model + the store root, reachable from the handlers. Globals may only carry scalar/s literals, so the two strings are plain owned C-strings (strdup'd in, nurl_free'd out) rather than String structs.
: ~ i g_api_style 0: ~ s g_api_name ``: ~ s g_api_root ``: ~ i g_api_auth 0Config for the wizard-started server (nurllama start): the default model the web UI chats with, the bearer token (empty = open access), and the SQLite history path. All scalar/s (globals hold no structs).
: ~ s g_api_token ``: ~ s g_api_model ``: ~ s g_api_histpath ``: ~ i g_api_weights 0serve --weights: when set, every model this server opens takes its tensors from this safetensors file instead of the GGUF's own (the GGUF still supplies the hyperparameters and the tokenizer) — the API twin of run --weights, for serving a finetuned/merged checkpoint. The pointed string is the CLI's; it outlives the server loop.
@ api_set_weights s w → v@ api_serve String root s host i port s model → iServe on host:port. Single worker: the decode loop owns the device, so serialisation is structural rather than a lock. model (may be empty) is the default: the web UI chats with it, and an /api request that names no model of its own falls back to it.
@ api_serve_cfg String root NlConfig cfg → iServe with a full wizard config: a default model for the web UI, an optional bearer token, and the conversation-history database.
packages/nurllama/src/tokenizer.nu — the GGUF vocabulary loader.
The tokenizer ENGINE now lives in packages/tokenizer: SentencePiece merging, byte-level BPE, inline special tokens, decoding — none of which has anything to do with the container the vocabulary arrived in.
What is left here is the part that IS about GGUF: pulling the pieces, scores, token types and merge rules out of tokenizer.ggml.* metadata and handing them to tok_build. whisper does the same from vocab.json + merges.txt, with the same engine behind it (deps/tokenizer/src/hf.nu).
( tok_new gguf ) → !Tok String
Everything else — tok_encode / tok_decode / tok_piece / tok_free / tok_bos / tok_eos / tok_unk / tok_n_vocab — comes from the package.
@ tok_new Gguf g → !Tok Stringnurllama — run language models locally.
nurllama start [-y] interactive setup wizard: pick a model, choose the bind scope (localhost / network) and access (open / bearer token) and the port, saved to config.json; launches the web chat server. A second run reuses the config (-y = without asking). The server hosts a web chat UI with per- browser cookie sessions and a SQLite conversation history. nurllama pull <src> [--name N] download into ~/.nurllama (hf.co/ORG/REPO/FILE.gguf or a URL; resumes a broken pull) nurllama list · rm <name> · verify <name> the local model store nurllama serve [model] [--host H] [--port N] ollama-compatible HTTP API
model is the default: the web UI chats with it, /api requests may name their own nurllama chat <model|name|hf-ref> interactive chat (model's own template from GGUF) nurllama run <model|name|hf-ref> <prompt> generate (stream to stdout) a model that is neither a local file nor a name in the store is a Hugging Face ref (org/repo/file.gguf) fetched on demand into the shared ~/.nurl/models cache via the hub package — no separate pull -n N (default 64) · --temp F (0 = greedy, default 0.8) --topk N · --topp F · --seed N · --ctx N diffusion models add: --block N · --threshold F · --edit-threshold F · --post-steps N (temp defaults to 0) nurllama convert <hf-dir|hf-ref> <out.gguf> HF checkpoint → GGUF (the checkpoint may be a local dir or an org/repo ref fetched via hub) nurllama finetune <model.gguf> <data.txt> LoRA finetune on the GPU [--out a.st] [--merged m.st] [--steps N] [--lr X] [--rank R] [--alpha A] [--seq T] [--seed S] [--f32] [--checkpoint c.st] [--save-every N] [--resume] [--window-stride N] (0 = spread over the corpus) --type q8_0 (default) | f16 | bf16 | f32 nurllama logits <model.gguf> <prompt> final-position logits, one per line (verification tap) nurllama dlogits / dgen diffusion verification taps (window logits / denoise ids) nurllama tokenize <model.gguf> <text> encode → token ids nurllama detok <model.gguf> <id> [id …] decode ids → bytes nurllama vocab <model.gguf> [n] dump the first n pieces nurllama selftest synthetic SPM + BPE vocab round-trips, bit-exact
Everything is loaded from the model file alone (tokenizer.ggml. + llama. metadata + tensors via packages/gguf); compute runs on packages/gpu — CUDA when present, NURL_GPU=cpu for the host backend.
: NlCnt: NlCnt {
i pass
i fail
}packages/nurllama/src/sample.nu — host-side sampling over the logits.
( sample_greedy m ) → i argmax ( sample_next m rng temp topk topp ) → i softmax(T) → top-k → top-p → draw
temp ≤ 0 collapses to greedy. top-k ≤ 0 disables the k filter; topp ≥ 1 disables the nucleus filter. Deterministic under a seeded Rng.
@ sample_greedy Llm m → i& libm @ exp f x → fhost-side softmax needs exp — bind libm directly, the stdlib idiom
packages/nurllama/src/store.nu — the local model store (~/.nurllama).
Content-addressed, cas-shaped, but streaming end to end — a blob is never fully resident:
<root>/blobs/sha256-<64hex> the model files, named by content <root>/manifests/<name> one small JSON per model name
A manifest records { name, url, digest, size } — the name is the user-facing handle (nurllama run <name>), the digest is the proof. Two names may point at the same blob (dedup by content); rm only deletes a blob once no manifest references it. verify re-hashes the blob through the incremental sha256 and refuses drift.
( nl_store_root ) → String $NURLLAMA_HOME | ~/.nurllama ( nl_resolve root nameOrPath ) → ?String path to a model file ( nl_store_add root name url hex size ) → !v String ( nl_store_list root ) → v prints the table ( nl_store_rm root name ) → !v String ( nl_store_verify root name ) → !v String ( nl_blob_path root hex ) ( nl_manifest_path root name ) ( nl_hash_file path ) → !String String streaming sha256 hex
@ nl_store_root → String@ nl_blob_path String root s hex → String@ nl_manifest_path String root s name → String@ nl_store_init String root → !v String@ nl_hash_file s path → !String StringStreaming sha256 of a file — constant memory whatever the size.
@ nl_store_add String root s name s url s hex i size → !v String@ nl_resolve String root s arg → ?StringResolve a model argument: an existing file path wins; otherwise the store's manifest name → blob path. None = unknown.
@ nl_store_list String root → v@ nl_store_rm String root s name → !v String@ nl_store_verify String root s name → !v StringRe-hash the blob and compare with the manifest digest — the store's integrity loop (mirrors cas_get's refuse-on-mismatch, streaming).