pull.nu

packages/nurllama/src/pull.nu — streaming model download with resume.

The whole transfer is constant-memory: HTTP chunks flow straight to disk through file_write_chunk while the incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<name>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and a manifest records name → digest.

( nl_resolve_url src ) → String hf.co/ORG/REPO/FILE shorthand → the HF resolve URL; http(s):// passes through ( nl_default_name src ) → String basename minus .gguf ( nl_pull root src name ) → !v String

API

@ nl_resolve_url s src → String

hf.co/ORG/REPO/path/to/file.gguf → https://huggingface.co/ORG/REPO/resolve/main/path/to/file.gguf

@ nl_default_name s src → String

last path segment, ".gguf" stripped — the default store name

@ nl_pull String root s src s name_override → !v String

---

model.nu

packages/nurllama/src/model.nu — the llama-architecture forward pass.

Loads a GGUF model onto the device (CUDA, or the CPU/OpenMP backend via NURL_GPU=cpu — same kernels either way) and runs autoregressive decode with a device-resident KV cache:

( llm_open path n_ctx ) → !*Llm String ( llm_eval m token pos ) → v one decode step; logits land in the reused host buffer ( llm_logit m i ) → f read logit i of the last eval ( llm_n_vocab m ) ( llm_tok m ) the model's tokenizer rides along ( llm_close m )

Weights stay QUANTISED on the device: a matvec whose weight type has a device kernel (F32/F16/Q4_0/Q8_0/Q4_K/Q6_K) uploads the GGUF bytes verbatim and decodes each block inside the matmul, so a Q4_K model needs ~7× less device memory than its f32 expansion (and the memory-bound matvec reads ~7× fewer bytes). Any other type falls back to host dequantisation through gguf_dequant — correctness first, always. Activations, KV cache and logits are f32.

The token embedding is dequantised host-side once (its rows upload one at a time), which costs host RAM but no device memory. Prefill runs the decode step per prompt token — batched later.

Architecture: llama-family (RMSNorm → GQA attention with NORM-style RoPE → SwiGLU FFN, residuals around both), read from the llama.* GGUF keys. Tied-embedding models (no output.weight) reuse token_embd as the output projection.

API

: Llm

: Llm {
    Gpu g
    LlmKernels ks
    * Tok tok
    i n_embd
    i n_layer
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    i n_ctx
    i rope_dim
    f eps
    f rope_base
    i gg
    i embd_idx
    ( Vec u ) embd_host
    ( Vec i ) wdptr
    ( Vec i ) wbytes
    ( Vec i ) tq_attn_norm
    ( Vec i ) tq_wq
    ( Vec i ) tq_wk
    ( Vec i ) tq_wv
    ( Vec i ) tq_wo
    ( Vec i ) tq_ffn_norm
    ( Vec i ) tq_gate
    ( Vec i ) tq_down
    ( Vec i ) tq_up
    i tq_output
    ( Vec i ) attn_norm
    ( Vec i ) wq
    ( Vec i ) wk
    ( Vec i ) wv
    ( Vec i ) wo
    ( Vec i ) ffn_norm
    ( Vec i ) w_gate
    ( Vec i ) w_down
    ( Vec i ) w_up
    i out_norm
    i w_output
    ( Vec i ) kcache
    ( Vec i ) vcache
    i xd
    i xnd
    i qd
    i kd
    i vd
    i aod
    i tmpd
    i gd
    i ud
    i scored
    i logitsd
    * u logits_host
}

@ string_eq_raw String a s raw → b

: ~ i __lm_last_type -1

Publishes the uploaded tensor's ggml type (0 when it was host-dequantised to f32), so the caller records which matvec kernel the weight needs.

@ llm_open s path i want_ctx → !*Llm String

@ llm_close * Llm m → v

@ llm_tok * Llm m → *Tok

@ llm_n_vocab * Llm m → i

@ llm_n_ctx * Llm m → i

@ llm_logit * Llm m i idx → f

@ llm_eval * Llm m i token i pos → v

One decode step: token at position pos (0-based; caller feeds positions sequentially). Logits for the NEXT token land in logits_host.

---

chat.nu

packages/nurllama/src/chat.nu — chat templates.

Turns a message list into the exact prompt string a chat-tuned model was trained on. The template is chosen from the model's own GGUF metadata: tokenizer.chat_template is matched against the shapes we know, and the architecture/BOS conventions fill the rest — so a model carries its own dialect and the caller never guesses.

( chat_style_of g ) → i CHAT_* below ( chat_render style msgs ) → String prompt for generation ( chat_stop_matches style piece ) → b stop-token text guard

Styles: CHAT_LLAMA2 [INST] <<SYS>>…<</SYS>> user [/INST] assistant </s> CHAT_LLAMA3 <|start_header_id|>role<|end_header_id|>…<|eot_id|> CHAT_CHATML <|im_start|>role\n…<|im_end|> (qwen, many finetunes) CHAT_PLAIN no template — the messages are concatenated; a base (non-chat) model gets its raw prompt, which is the honest thing to do rather than inventing turns.

A ChatMsg's role is one of system / user / assistant.

API

: i CHAT_PLAIN 0

: i CHAT_LLAMA2 1

: i CHAT_LLAMA3 2

: i CHAT_CHATML 3

: ChatMsg

: ChatMsg {
    String role
    String content
}

@ chat_msg s role s content → ChatMsg

@ chat_msg_free ChatMsg m → v

@ chat_msgs_free ( Vec ChatMsg ) v → v

@ chat_style_of * Gguf g → i

Pick the style from the model's own chat template (substring match on the marker tokens each dialect must contain), falling back to PLAIN when the model ships no template — a base model, honestly served.

@ chat_render i style ( Vec ChatMsg ) msgs → String

@ chat_stop_matches i style s piece → b

Some chat models emit their turn terminator as ordinary text rather than the EOS id (a finetune whose eos_token_id was left at the base value). Treat the dialect's terminator as a stop sequence too.

---

kernels.nu

packages/nurllama/src/kernels.nu — the LLM decode ops as GPU kernels.

CUDA-C sources compiled once through packages/gpu (NVRTC on the CUDA backend; the SAME source runs on the CPU/OpenMP backend, so every kernel below deliberately avoids shared / syncthreads — each thread computes its output independently, exactly like the onnx op set). Decode-step shapes (one token at a time) keep the naive forms honest: the matvecs dominate and each row is one thread's serial dot product.

Layout contracts (ggml row-major, ne0 fastest): weight W[out,in] → W[rin + c], matvec y[r] = Σc W[rin+c]·x[c] K/V cache → cache[t(n_kv·hd) + khhd + d] q/k activations → x[h*hd + d]

API

: LlmKernels

: LlmKernels {
    GpuKernel matvec
    GpuKernel mv_q4_0
    GpuKernel mv_q8_0
    GpuKernel mv_q5_0
    GpuKernel mv_q5_1
    GpuKernel mv_q4_k
    GpuKernel mv_q5_k
    GpuKernel mv_q6_k
    GpuKernel mv_f16
    GpuKernel rmsnorm
    GpuKernel rope
    GpuKernel attn
    GpuKernel silumul
    GpuKernel addv
    GpuKernel copyat
    b ok
}

@ lk_build Gpu g → LlmKernels

@ lk_free LlmKernels ks → v

@ lk_matvec LlmKernels ks i wd i xd i yd i rows i cols → v

@ lk_matvec_q LlmKernels ks i gt i wd i xd i yd i rows i cols → b

Quantised matvec dispatch: gt is the ggml tensor type of W. Returns F when the type has no device kernel (the caller then falls back to the f32 path — correctness first, always).

@ lk_rmsnorm LlmKernels ks i xd i wd i yd i n f eps → v

@ lk_rope LlmKernels ks i xd i nh i hd i rd i pos f base → v

@ lk_attn LlmKernels ks i qd i kcd i vcd i outd i scd i nh i nkv i hd i npos → v

@ lk_silumul LlmKernels ks i gd i ud i n → v

@ lk_addv LlmKernels ks i ad i bd i n → v

@ lk_copyat LlmKernels ks i dstd i srcd i off i n → v

---

api.nu

packages/nurllama/src/api.nu — an ollama-compatible HTTP API.

Speaks enough of ollama's wire protocol that existing clients work unchanged:

POST /api/generate { model, prompt, stream?, options? } POST /api/chat { model, messages[{role,content}], stream?, options? } → NDJSON: one JSON object per token while generating, then a final object with done:true (+ token counts). stream:false returns a single aggregated object. GET /api/tags the local store, as ollama's model list POST /api/show { model } → the model's metadata GET / "nurllama is running"

Streaming rides the http package's stream hook (http_app_stream): the handler owns the TcpConn and writes chunked NDJSON itself, so a token reaches the client the moment it is decoded.

One model is resident at a time (LRU of one): a request naming a different model swaps it in. Generation is serialised — the decode loop owns the device — which is honest for a single-GPU host; the server is single-worker so that serialisation is structural rather than a lock.

API

: ~ i g_api_llm 0

The resident model + the store root, reachable from the handlers. Globals may only carry scalar/s literals, so the two strings are plain owned C-strings (strdup'd in, nurl_free'd out) rather than String structs.

: ~ i g_api_style 0

: ~ s g_api_name ``

: ~ s g_api_root ``

@ api_set_root String root → v

@ api_serve String root s host i port → i

Serve on host:port. Single worker: the decode loop owns the device, so serialisation is structural rather than a lock.

---

tokenizer.nu

packages/nurllama/src/tokenizer.nu — LLM tokenizer from a GGUF vocabulary.

Everything comes out of the model file itself (tokenizer.ggml.* metadata read through packages/gguf) — no external vocab files, no exporter step:

( tok_new g ) → !Tok String from an open Gguf ( tok_free t ) ( tok_encode t text add_special ) → ( Vec i ) token ids ( tok_piece t id ) → ( Vec u ) one token's raw bytes ( tok_decode t ids ) → ( Vec u ) ids → bytes (skips control tokens) ( tok_n_vocab t ) ( tok_bos t ) ( tok_eos t ) ( tok_unk t )

Two tokenizer families, selected by tokenizer.ggml.model:

SPM ("llama") — SentencePiece-style bigram merging: split the space-escaped text ( → ▁ U+2581, optional leading space) into UTF-8 characters, then repeatedly merge the adjacent pair whose concatenation is a vocab piece with the HIGHEST score (ties → leftmost), exactly llama.cpp's llm_tokenizer_spm selection order. Characters that never reach a vocab piece fall back to the <0xXX> BYTE tokens, then to UNK.

BPE ("gpt2") — byte-level BPE: GPT-2's byte→unicode remap, greedy regex-style pre-split (contractions / letters / digits / punct / whitespace), then lowest-merge-rank pairing driven by tokenizer.ggml.merges. NOTE v1: the pre-split implements the GPT-2 pattern with \p{L} approximated as "ASCII letters + any cp ≥ 0x80" — per-model tokenizer.ggml.pre variants (qwen2 digit splitting etc.) are a follow-up.

The Tok owns copies of every piece (the source Gguf can be closed after tok_new). The piece→id map borrows the buffers of pieces — stable because Vec growth moves the String structs, never the heap buffers they point to.

API

: i TT_NORMAL 1

token_type values (tokenizer.ggml.token_type, llama.cpp llama_token_type)

: i TT_UNKNOWN 2

: i TT_CONTROL 3

: i TT_USER 4

: i TT_UNUSED 5

: i TT_BYTE 6

: i TOK_SPM 0

: i TOK_BPE 1

: Tok

: Tok {
    i mode
    ( Vec String ) pieces
    ( Vec f ) scores
    ( Vec i ) ttype
    ( HashMap s i ) lookup
    ( HashMap s i ) ranks
    ( Vec String ) rankkeys
    ( Vec String ) byte_enc
    ( Vec i ) byte_id
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
}

@ tok_new * Gguf g → !*Tok String

Build a tokenizer from an open GGUF model. Copies the vocabulary — the Gguf may be closed afterwards.

@ tok_free * Tok t → v

@ tok_n_vocab * Tok t → i

@ tok_bos * Tok t → i

@ tok_eos * Tok t → i

@ tok_unk * Tok t → i

@ tok_encode * Tok t s text b add_special → ( Vec i )

@ tok_piece * Tok t i id → ( Vec u )

Raw bytes of one token: SPM BYTE pieces become their byte, ▁ becomes a space, control tokens become nothing; BPE pieces run the inverse byte remap.

@ tok_decode * Tok t ( Vec i ) ids → ( Vec u )

---

main.nu

nurllama — run language models locally.

nurllama pull <src> [--name N] download into ~/.nurllama (hf.co/ORG/REPO/FILE.gguf or a URL; resumes a broken pull) nurllama list · rm <name> · verify <name> the local model store nurllama serve [--host H] [--port N] ollama-compatible HTTP API nurllama chat <model|name> interactive chat (model's own template from GGUF) nurllama run <model|name> <prompt> generate (stream to stdout) -n N (default 64) · --temp F (0 = greedy, default 0.8) --topk N · --topp F · --seed N · --ctx N nurllama logits <model.gguf> <prompt> final-position logits, one per line (verification tap) nurllama tokenize <model.gguf> <text> encode → token ids nurllama detok <model.gguf> <id> [id …] decode ids → bytes nurllama vocab <model.gguf> [n] dump the first n pieces nurllama selftest synthetic SPM + BPE vocab round-trips, bit-exact

Everything is loaded from the model file alone (tokenizer.ggml. + llama. metadata + tensors via packages/gguf); compute runs on packages/gpu — CUDA when present, NURL_GPU=cpu for the host backend.

API

: NlCnt

: NlCnt {
    i pass
    i fail
}

@ main → i

---

sample.nu

packages/nurllama/src/sample.nu — host-side sampling over the logits.

( sample_greedy m ) → i argmax ( sample_next m rng temp topk topp ) → i softmax(T) → top-k → top-p → draw

temp ≤ 0 collapses to greedy. top-k ≤ 0 disables the k filter; topp ≥ 1 disables the nucleus filter. Deterministic under a seeded Rng.

API

@ sample_greedy * Llm m → i

& libm @ exp f x → f

host-side softmax needs exp — bind libm directly, the stdlib idiom

@ sample_next * Llm m Rng rng f temp i topk f topp → i

---

store.nu

packages/nurllama/src/store.nu — the local model store (~/.nurllama).

Content-addressed, cas-shaped, but streaming end to end — a blob is never fully resident:

<root>/blobs/sha256-<64hex> the model files, named by content <root>/manifests/<name> one small JSON per model name

A manifest records { name, url, digest, size } — the name is the user-facing handle (nurllama run <name>), the digest is the proof. Two names may point at the same blob (dedup by content); rm only deletes a blob once no manifest references it. verify re-hashes the blob through the incremental sha256 and refuses drift.

( nl_store_root ) → String $NURLLAMA_HOME | ~/.nurllama ( nl_resolve root nameOrPath ) → ?String path to a model file ( nl_store_add root name url hex size ) → !v String ( nl_store_list root ) → v prints the table ( nl_store_rm root name ) → !v String ( nl_store_verify root name ) → !v String ( nl_blob_path root hex ) ( nl_manifest_path root name ) ( nl_hash_file path ) → !String String streaming sha256 hex

API

@ nl_store_root → String

@ nl_blob_path String root s hex → String

@ nl_manifest_path String root s name → String

@ nl_store_init String root → !v String

@ nl_hash_file s path → !String String

Streaming sha256 of a file — constant memory whatever the size.

@ nl_store_add String root s name s url s hex i size → !v String

@ nl_resolve String root s arg → ?String

Resolve a model argument: an existing file path wins; otherwise the store's manifest name → blob path. None = unknown.

@ nl_store_list String root → v

@ nl_store_rm String root s name → !v String

@ nl_store_verify String root s name → !v String

Re-hash the blob and compare with the manifest digest — the store's integrity loop (mirrors cas_get's refuse-on-mismatch, streaming).