packages/nurllama/src/pull.nu — streaming model download with resume.
The whole transfer is constant-memory: HTTP chunks flow straight to disk through file_write_chunk while the incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<name>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and a manifest records name → digest.
( nl_resolve_url src ) → String hf.co/ORG/REPO/FILE shorthand → the HF resolve URL; http(s):// passes through ( nl_default_name src ) → String basename minus .gguf ( nl_pull root src name ) → !v String
@ nl_resolve_url s src → Stringhf.co/ORG/REPO/path/to/file.gguf → https://huggingface.co/ORG/REPO/resolve/main/path/to/file.gguf
@ nl_default_name s src → Stringlast path segment, ".gguf" stripped — the default store name
@ nl_pull String root s src s name_override → !v Stringpackages/nurllama/src/convert.nu — HF checkpoint → GGUF, streaming.
( nurllama_convert hfdir outpath otype ) → i 0 = ok
Converts a Hugging Face checkpoint directory (config.json + tokenizer.json + model*.safetensors shards) into a GGUF that llama.cpp would recognise — same architecture name, metadata keys, tensor names, shapes and fusions — so the file interoperates in both directions. Currently supported: llada2_moe (LLaDA2.x diffusion MoE), written as the llada2 architecture following llama.cpp's BailingMoeV2 conventions:
(blk.N.attn_qkv.weight, HF row order, no permutation — NEOX rotary semantics are preserved by construction)
fuse into one 3D blk.N.ffn_{gate,up,down}_exps.weight each
reference model routes on an fp32 sigmoid, and the router is ~2 MB per layer — precision is free here)
pre = bailingmoe2, id-ordered tokens, merges, added tokens as CONTROL, [PAD…] filler up to config vocab_size)
The conversion STREAMS: shards are mmapped, each tensor is dequantised (bf16→f32) and encoded (Q8_0/F16/BF16/F32) in row chunks straight into the gguf streaming writer — peak memory is one chunk, never the model. A 32 GB checkpoint converts on a 16 GB machine.
: i CV_F32 0ggml tensor type ids (gguf.nu names them; the writer wants the raw id)
: i CV_F16 1: i CV_Q8_0 8: i CV_BF16 30: i CV_TOK_NORMAL 1token types (llama.cpp enum)
: i CV_TOK_CONTROL 3: i CV_TOK_USER 4: i CV_TOK_UNUSED 5: CvJobs: CvJobs {
( Vec String ) names
( Vec i ) gts
( Vec i ) nds
( Vec i ) d0s
( Vec i ) d1s
( Vec i ) d2s
( Vec String ) srcs
}
One conversion job = one GGUF tensor. src is the HF tensor name, or the marker experts:<layer>:<proj> for a 256-way fusion.
: Cv: Cv {
( Vec i ) sts
( Vec String ) stpaths
Json cfg
b cfg_ok
i n_layer
i n_embd
i n_head
i n_kv
i head_dim
i n_ff
i n_vocab
i n_ctx
i n_expert
i n_expert_used
i n_group
i topk_group
i n_shared
i first_dense
i moe_ff
f rope_base
f eps
f route_scale
b norm_topk
f partial_rotary
}
: CvVocab: CvVocab {
( Vec String ) tokens
( Vec i ) types
( Vec String ) merges
i bos_id
i eos_id
i pad_id
i mask_id
b ok
}
@ nurllama_convert s hfdir s outpath s otype → i@ _cv_geti ( Vec i ) v i k → ipackages/nurllama/src/model.nu — the llama-architecture forward pass.
Loads a GGUF model onto the device (CUDA, or the CPU/OpenMP backend via NURL_GPU=cpu — same kernels either way) and runs autoregressive decode with a device-resident KV cache:
( llm_open path n_ctx ) → !*Llm String ( llm_eval m token pos ) → v one decode step; logits land in the reused host buffer ( llm_logit m i ) → f read logit i of the last eval ( llm_n_vocab m ) ( llm_tok m ) the model's tokenizer rides along ( llm_close m )
Weights stay QUANTISED on the device: a matvec whose weight type has a device kernel (F32/F16/Q4_0/Q8_0/Q4_K/Q6_K) uploads the GGUF bytes verbatim and decodes each block inside the matmul, so a Q4_K model needs ~7× less device memory than its f32 expansion (and the memory-bound matvec reads ~7× fewer bytes). Any other type falls back to host dequantisation through gguf_dequant — correctness first, always. Activations, KV cache and logits are f32.
The token embedding is dequantised host-side once (its rows upload one at a time), which costs host RAM but no device memory. The prompt is processed in CHUNKS: the matmuls run over up to LM_CHUNK positions at once, so a 300-token prompt costs a few hundred kernel launches instead of a few tens of thousands, and each weight is read once per chunk instead of once per token. Attention still runs per position (it is not the bottleneck, and the causal window differs per position anyway) — over a cache the same chunk already filled.
Architectures: the llama shape — RMSNorm → GQA attention with NORM-style RoPE → SwiGLU FFN, residuals around both — which llama and qwen2 both are. Hyperparameters are read under the model's own key prefix (llama.* / qwen2.*), and qwen2's per-layer Q/K/V biases are applied when present (llama models simply have none). Tied-embedding models (no output.weight) reuse token_embd as the output projection.
: Llm: Llm {
Gpu g
LlmKernels ks
* Tok tok
i n_embd
i n_layer
i n_head
i n_kv
i head_dim
i n_ff
i n_vocab
i n_ctx
i rope_dim
i rope_style
f eps
f rope_base
// gemma: the attention projections do NOT have to land back on n_embd —
// gemma3-270m has n_embd 640, 4 heads and head_dim 256, so q is 1024 wide
// and `wo` maps 1024 → 640. q_dim is nh*head_dim, never assumed = n_embd.
i q_dim
f embd_scale
f attn_scale
b ffn_gelu
i swa_window
i swa_period
f rope_base_swa
// ── llada2 (diffusion MoE) ──────────────────────────────────────
// bidir: attention is NOT causal — the diffusion decode evaluates a
// whole block window at once and every query sees the full window.
b bidir
i mask_id
i first_dense
i n_expert
i n_expert_used
i n_group
i n_group_used
i moe_ff
i shexp_ff
f route_scale
b weights_norm
( Vec i ) gate_inp
( Vec i ) tq_gate_inp
( Vec f ) exp_bias
( Vec i ) gate_exps
( Vec i ) up_exps
( Vec i ) down_exps
( Vec i ) tq_gate_exps
( Vec i ) tq_up_exps
( Vec i ) tq_down_exps
( Vec i ) gate_shexp
( Vec i ) up_shexp
( Vec i ) down_shexp
( Vec i ) tq_gate_shexp
( Vec i ) tq_up_shexp
( Vec i ) tq_down_shexp
i rlogitsd
* u router_host
i moe_outd
i gg
// A SECOND weight source: the same model, in the container Hugging Face
// ships it in. The GGUF still supplies the hyperparameters and the
// tokenizer; the tensors come from here when it is set (0 = not set).
// This is how the safetensor reader is proven: run the same model from
// both containers and require the logits to agree.
i st
b st_norm_add1
i st_embd
i embd_idx
( Vec u ) embd_host
( Vec i ) wdptr
( Vec i ) wbytes
( Vec i ) tq_attn_norm
( Vec i ) tq_wq
( Vec i ) tq_wk
( Vec i ) tq_wv
( Vec i ) bq
( Vec i ) bk
( Vec i ) bv
( Vec i ) tq_wo
( Vec i ) tq_ffn_norm
( Vec i ) tq_gate
( Vec i ) tq_down
( Vec i ) tq_up
i tq_output
( Vec i ) attn_norm
( Vec i ) q_norm
( Vec i ) k_norm
( Vec i ) post_attn_norm
( Vec i ) post_ffn_norm
( Vec i ) wq
( Vec i ) wk
( Vec i ) wv
( Vec i ) wo
( Vec i ) ffn_norm
( Vec i ) w_gate
( Vec i ) w_down
( Vec i ) w_up
i out_norm
i w_output
( Vec i ) kcache
( Vec i ) vcache
i xd
i xnd
i qd
i kd
i vd
i aod
i tmpd
// rmsnorm reads the whole row to form its sum of squares, so it can never
// write back into its own input: another block would already have
// overwritten the row under it. gemma's Q/K and post-block norms
// therefore need somewhere to land.
i qn2d
i kn2d
i tn2d
i gd
i ud
i scored
i logitsd
* u logits_host
}
@ _lm_geti ( Vec i ) v i k → i@ string_eq_raw String a s raw → b: ~ i __lm_last_type -1Publishes the uploaded tensor's ggml type (0 when it was host-dequantised to f32), so the caller records which matvec kernel the weight needs.
: ~ b __lm_alloc_failed FSet when any device allocation comes back null (out of device memory). Checked once at the end of llm_open_st: a silent failure here used to surface as an all-zero forward pass — the model "ran" and produced token 0 forever — instead of an error naming the cause.
: i LM_CHUNK 64Positions per prefill chunk. Bounds the batched activation scratch (chunk × n_ff floats) while keeping the launches amortised.
@ llm_open s path i want_ctx → !*Llm String@ llm_open_st s path s weights i want_ctx → !*Llm StringSame model, weights from a safetensors file instead of the GGUF's own tensors. The GGUF still supplies the hyperparameters and the tokenizer — safetensors carries neither (they live in config.json and tokenizer.json next to it) — so this is a weight SOURCE, not a second model format.
It exists to be checked: run one model through both containers and the logits must agree. If the safetensors reader got a dtype, an offset or a row order wrong, that check fails here rather than silently in whatever depends on it next.
@ llm_close * Llm m → v@ llm_tok * Llm m → *Tok@ llm_n_vocab * Llm m → i@ llm_n_ctx * Llm m → i@ llm_logit * Llm m i idx → f@ llm_eval * Llm m i token i pos → vOne decode step: token at position pos (0-based; caller feeds positions sequentially). Logits for the NEXT token land in logits_host. One decode step: token at position pos.
@ llm_eval_n * Llm m ( Vec i ) ids i first i count i pos0 → vEvaluate count tokens — ids[first .. first+count) — starting at absolute position pos0. The matmuls run over the whole batch; attention runs per position over the cache this same batch already filled (its causal window differs per position). Logits are produced for the LAST position only, which is all a prefill or a decode step needs.
@ llm_eval_win * Llm m ( Vec i ) ids i first i count i pos0 i kvlen b want_logits → vDiffusion window evaluation (llada2): the batch is a block window at positions pos0.., every query attends the SAME fixed cache range [0, kvlen) — bidirectional inside the window, all earlier blocks visible — and logits come back for EVERY position when wanted (the denoise step samples all of them). The window's K/V rows land in the cache at pos0.., overwriting the previous step's; earlier (frozen) blocks keep theirs, which is exactly the block-diffusion mask.
@ llm_logit_at * Llm m i row i idx → fLogit idx of window row row after an all-positions eval (llm_eval_win with logits). Row 0 = the window's first position.
@ llm_mask_id * Llm m → i@ llm_is_diffusion * Llm m → b@ llm_chunk → iThe largest chunk a single llm_eval_n call may take.
@ llm_prefill * Llm m ( Vec i ) ids → vPrefill: run the whole prompt through the model in chunks, leaving the KV cache filled and the logits set for the LAST prompt token — the distribution the first generated token is drawn from.
packages/nurllama/src/chat.nu — chat templates.
Turns a message list into the exact prompt string a chat-tuned model was trained on. The template is chosen from the model's own GGUF metadata: tokenizer.chat_template is matched against the shapes we know, and the architecture/BOS conventions fill the rest — so a model carries its own dialect and the caller never guesses.
( chat_style_of g ) → i CHAT_* below ( chat_render style msgs ) → String prompt for generation ( chat_stop_matches style piece ) → b stop-token text guard
Styles: CHAT_LLAMA2 [INST] <<SYS>>…<</SYS>> user [/INST] assistant </s> CHAT_LLAMA3 <|start_header_id|>role<|end_header_id|>…<|eot_id|> CHAT_CHATML <|im_start|>role\n…<|im_end|> (qwen, many finetunes) CHAT_PHI3 <|user|>\n…<|end|>\n<|assistant|>\n (phi-3/3.5/4) CHAT_GEMMA <start_of_turn>user\n…<end_of_turn> (gemma; the assistant's role is spelled "model", and gemma has NO system role — the system prompt folds into the first user turn, which is what its own template does) CHAT_PLAIN no template — the messages are concatenated; a base (non-chat) model gets its raw prompt, which is the honest thing to do rather than inventing turns.
A ChatMsg's role is one of system / user / assistant.
: i CHAT_PLAIN 0: i CHAT_LLAMA2 1: i CHAT_LLAMA3 2: i CHAT_CHATML 3: i CHAT_GEMMA 4: i CHAT_PHI3 5: i CHAT_BAILING 6: ChatMsg: ChatMsg {
String role
String content
}
@ chat_msg s role s content → ChatMsg@ chat_msg_free ChatMsg m → v@ chat_msgs_free ( Vec ChatMsg ) v → v@ chat_style_of * Gguf g → iPick the style from the model's own chat template (substring match on the marker tokens each dialect must contain), falling back to PLAIN when the model ships no template — a base model, honestly served.
@ chat_render i style ( Vec ChatMsg ) msgs → String@ chat_stop_matches i style s piece → bSome chat models emit their turn terminator as ordinary text rather than the EOS id (a finetune whose eos_token_id was left at the base value). Treat the dialect's terminator as a stop sequence too.
packages/nurllama/src/diffuse.nu — the block-diffusion decode loop (LLaDA2.x "JointThreshold + editing").
( dz_generate m prompt gen_len block_len threshold edit_thr max_post temp rng ) → ( Vec i )
The sequence is a template of gen_len mask tokens after the prompt, processed one block_len window at a time. Each denoise step evaluates the whole active window bidirectionally (llm_eval_win) and samples every position at once:
threshold arecommitted (all of them; if none clears it, the single most confident one is) — the parallel-decode half
when the model now prefers a different token above edit_thr — the LLaDA2.1 editing half
edits, or after max_post post-resolution passes
Completed blocks freeze: one final KV-only evaluation writes the block's settled content into the cache, and later windows attend to it without re-evaluating — the reference implementation recomputes the whole prefix every step; the cache makes each step O(window) instead of O(prefix).
Generation stops early when a completed block leaves no masks and an EOS has appeared (eos_early_stop), and the returned ids are cut BEFORE the first EOS — callers get exactly the text tokens.
& libm @ exp f x → f@ _dz_geti ( Vec i ) v i k → i@ _dz_getf ( Vec f ) v i k → f@ dz_generate * Llm m ( Vec i ) prompt i gen_len0 i block_len0 f threshold f edit_thr i max_post f temp Rng rng → ( Vec i )packages/nurllama/src/kernels.nu — the LLM decode ops as GPU kernels.
CUDA-C sources compiled once through packages/gpu (NVRTC on the CUDA backend; the SAME source runs on the CPU/OpenMP backend, so every kernel below deliberately avoids shared / syncthreads — each thread computes its output independently, exactly like the onnx op set). Decode-step shapes (one token at a time) keep the naive forms honest: the matvecs dominate and each row is one thread's serial dot product.
Layout contracts (ggml row-major, ne0 fastest): weight W[out,in] → W[rin + c], matvec y[r] = Σc W[rin+c]·x[c] K/V cache → cache[t(n_kv·hd) + khhd + d] q/k activations → x[h*hd + d]
: LlmKernels: LlmKernels {
GpuKernel matvec
GpuKernel mv_q4_0
GpuKernel mv_q8_0
GpuKernel mv_q5_0
GpuKernel mv_q5_1
GpuKernel mv_q4_k
GpuKernel mv_q5_k
GpuKernel mv_q6_k
GpuKernel mv_f16
GpuKernel w_f32
GpuKernel w_q4_0
GpuKernel w_q8_0
GpuKernel w_q4_k
GpuKernel w_q6_k
GpuKernel w_q5_0
GpuKernel w_q5_1
GpuKernel w_q5_k
b warp
GpuKernel rmsnorm
GpuKernel rope
GpuKernel rope_neox
GpuKernel attn
GpuKernel attn_sc
GpuKernel attn_out
GpuKernel silumul
GpuKernel gelumul
GpuKernel scale
GpuKernel addv
GpuKernel addrow
GpuKernel copyat
GpuKernel axpy
b ok
}
@ lk_build Gpu g → LlmKernels@ lk_free LlmKernels ks → v@ lk_matvec LlmKernels ks i wd i xd i yd i rows i cols i batch → v@ lk_matvec_q LlmKernels ks i gt i wd i xd i yd i rows i cols i batch → bQuantised matvec dispatch: gt is the ggml tensor type of W. Returns F when the type has no device kernel (the caller then falls back to the f32 path — correctness first, always).
@ lk_rmsnorm LlmKernels ks i xd i wd i yd i n f eps i batch → v@ lk_rope LlmKernels ks i xd i nh i hd i rd i pos f base i style i batch → vstyle: 0 = NORM (llama), 1 = NEOX (qwen2)
@ lk_attn LlmKernels ks i qd i kcd i vcd i outd i scd i nh i nkv i hd i npos0 i batch i maxpos f qscale i window i causal → vnpos0 = the causal window of the FIRST batch element (element b sees npos0 + b rows). maxpos strides the per-(batch,head) score scratch. Two-pass attention: scores, then softmax + value-weighted sum.
@ lk_axpy LlmKernels ks i yd i xd f a i n → vy ← y + a·x — the routed-expert accumulator (each selected expert's down-projection lands weighted onto the residual stream).
@ lk_gelumul LlmKernels ks i gd i ud i n → vGeGLU: g ← gelu(g) * u, in place. Same shape as lk_silumul.
@ lk_scale LlmKernels ks i xd f s i n → vx ← x * s, in place.
@ lk_silumul LlmKernels ks i gd i ud i n → v@ lk_addv LlmKernels ks i ad i bd i n → v@ lk_addrow LlmKernels ks i ad i vd i n i batch → va[b][i] += v[i] for every row of the batch
@ lk_copyat LlmKernels ks i dstd i srcd i off i n → vpackages/nurllama/src/api.nu — an ollama-compatible HTTP API.
Speaks enough of ollama's wire protocol that existing clients work unchanged:
POST /api/generate { model, prompt, stream?, options? } POST /api/chat { model, messages[{role,content}], stream?, options? } → NDJSON: one JSON object per token while generating, then a final object with done:true (+ token counts). stream:false returns a single aggregated object. GET /api/tags the local store, as ollama's model list POST /api/show { model } → the model's metadata GET / "nurllama is running"
Streaming rides the http package's stream hook (http_app_stream): the handler owns the TcpConn and writes chunked NDJSON itself, so a token reaches the client the moment it is decoded.
One model is resident at a time (LRU of one): a request naming a different model swaps it in. Generation is serialised — the decode loop owns the device — which is honest for a single-GPU host; the server is single-worker so that serialisation is structural rather than a lock.
: ~ i g_api_llm 0The resident model + the store root, reachable from the handlers. Globals may only carry scalar/s literals, so the two strings are plain owned C-strings (strdup'd in, nurl_free'd out) rather than String structs.
: ~ i g_api_style 0: ~ s g_api_name ``: ~ s g_api_root ``@ api_set_root String root → v@ api_serve String root s host i port → iServe on host:port. Single worker: the decode loop owns the device, so serialisation is structural rather than a lock.
packages/nurllama/src/tokenizer.nu — the GGUF vocabulary loader.
The tokenizer ENGINE now lives in packages/tokenizer: SentencePiece merging, byte-level BPE, inline special tokens, decoding — none of which has anything to do with the container the vocabulary arrived in.
What is left here is the part that IS about GGUF: pulling the pieces, scores, token types and merge rules out of tokenizer.ggml.* metadata and handing them to tok_build. whisper does the same from vocab.json + merges.txt, with the same engine behind it (deps/tokenizer/src/hf.nu).
( tok_new gguf ) → !*Tok String
Everything else — tok_encode / tok_decode / tok_piece / tok_free / tok_bos / tok_eos / tok_unk / tok_n_vocab — comes from the package.
@ tok_new * Gguf g → !*Tok Stringnurllama — run language models locally.
nurllama pull <src> [--name N] download into ~/.nurllama (hf.co/ORG/REPO/FILE.gguf or a URL; resumes a broken pull) nurllama list · rm <name> · verify <name> the local model store nurllama serve [--host H] [--port N] ollama-compatible HTTP API nurllama chat <model|name> interactive chat (model's own template from GGUF) nurllama run <model|name> <prompt> generate (stream to stdout) -n N (default 64) · --temp F (0 = greedy, default 0.8) --topk N · --topp F · --seed N · --ctx N diffusion models add: --block N · --threshold F · --edit-threshold F · --post-steps N (temp defaults to 0) nurllama convert <hf-dir> <out.gguf> HF checkpoint → GGUF --type q8_0 (default) | f16 | bf16 | f32 nurllama logits <model.gguf> <prompt> final-position logits, one per line (verification tap) nurllama dlogits / dgen diffusion verification taps (window logits / denoise ids) nurllama tokenize <model.gguf> <text> encode → token ids nurllama detok <model.gguf> <id> [id …] decode ids → bytes nurllama vocab <model.gguf> [n] dump the first n pieces nurllama selftest synthetic SPM + BPE vocab round-trips, bit-exact
Everything is loaded from the model file alone (tokenizer.ggml. + llama. metadata + tensors via packages/gguf); compute runs on packages/gpu — CUDA when present, NURL_GPU=cpu for the host backend.
: NlCnt: NlCnt {
i pass
i fail
}
@ main → ipackages/nurllama/src/sample.nu — host-side sampling over the logits.
( sample_greedy m ) → i argmax ( sample_next m rng temp topk topp ) → i softmax(T) → top-k → top-p → draw
temp ≤ 0 collapses to greedy. top-k ≤ 0 disables the k filter; topp ≥ 1 disables the nucleus filter. Deterministic under a seeded Rng.
@ sample_greedy * Llm m → i& libm @ exp f x → fhost-side softmax needs exp — bind libm directly, the stdlib idiom
@ sample_next * Llm m Rng rng f temp i topk f topp → ipackages/nurllama/src/store.nu — the local model store (~/.nurllama).
Content-addressed, cas-shaped, but streaming end to end — a blob is never fully resident:
<root>/blobs/sha256-<64hex> the model files, named by content <root>/manifests/<name> one small JSON per model name
A manifest records { name, url, digest, size } — the name is the user-facing handle (nurllama run <name>), the digest is the proof. Two names may point at the same blob (dedup by content); rm only deletes a blob once no manifest references it. verify re-hashes the blob through the incremental sha256 and refuses drift.
( nl_store_root ) → String $NURLLAMA_HOME | ~/.nurllama ( nl_resolve root nameOrPath ) → ?String path to a model file ( nl_store_add root name url hex size ) → !v String ( nl_store_list root ) → v prints the table ( nl_store_rm root name ) → !v String ( nl_store_verify root name ) → !v String ( nl_blob_path root hex ) ( nl_manifest_path root name ) ( nl_hash_file path ) → !String String streaming sha256 hex
@ nl_store_root → String@ nl_blob_path String root s hex → String@ nl_manifest_path String root s name → String@ nl_store_init String root → !v String@ nl_hash_file s path → !String StringStreaming sha256 of a file — constant memory whatever the size.
@ nl_store_add String root s name s url s hex i size → !v String@ nl_resolve String root s arg → ?StringResolve a model argument: an existing file path wins; otherwise the store's manifest name → blob path. None = unknown.
@ nl_store_list String root → v@ nl_store_rm String root s name → !v String@ nl_store_verify String root s name → !v StringRe-hash the blob and compare with the manifest digest — the store's integrity loop (mirrors cas_get's refuse-on-mismatch, streaming).