pull.nu

packages/nurllama/src/pull.nu — streaming model download with resume.

The whole transfer is constant-memory: HTTP chunks flow straight to disk through file_write_chunk while the incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<name>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and a manifest records name → digest.

( nl_resolve_url src ) → String hf.co/ORG/REPO/FILE shorthand → the HF resolve URL; http(s):// passes through ( nl_default_name src ) → String basename minus .gguf ( nl_pull root src name ) → !v String

API

@ nl_resolve_url s src → String

hf.co/ORG/REPO/path/to/file.gguf → https://huggingface.co/ORG/REPO/resolve/main/path/to/file.gguf

@ nl_default_name s src → String

last path segment, ".gguf" stripped — the default store name

@ nl_pull String root s src s name_override → !v String

---

model.nu

packages/nurllama/src/model.nu — the llama-architecture forward pass.

Loads a GGUF model onto the device (CUDA, or the CPU/OpenMP backend via NURL_GPU=cpu — same kernels either way) and runs autoregressive decode with a device-resident KV cache:

( llm_open path n_ctx ) → !*Llm String ( llm_eval m token pos ) → v one decode step; logits land in the reused host buffer ( llm_logit m i ) → f read logit i of the last eval ( llm_n_vocab m ) ( llm_tok m ) the model's tokenizer rides along ( llm_close m )

Weights stay QUANTISED on the device: a matvec whose weight type has a device kernel (F32/F16/Q4_0/Q8_0/Q4_K/Q6_K) uploads the GGUF bytes verbatim and decodes each block inside the matmul, so a Q4_K model needs ~7× less device memory than its f32 expansion (and the memory-bound matvec reads ~7× fewer bytes). Any other type falls back to host dequantisation through gguf_dequant — correctness first, always. Activations, KV cache and logits are f32.

The token embedding is dequantised host-side once (its rows upload one at a time), which costs host RAM but no device memory. The prompt is processed in CHUNKS: the matmuls run over up to LM_CHUNK positions at once, so a 300-token prompt costs a few hundred kernel launches instead of a few tens of thousands, and each weight is read once per chunk instead of once per token. Attention still runs per position (it is not the bottleneck, and the causal window differs per position anyway) — over a cache the same chunk already filled.

Architectures: the llama shape — RMSNorm → GQA attention with NORM-style RoPE → SwiGLU FFN, residuals around both — which llama and qwen2 both are. Hyperparameters are read under the model's own key prefix (llama.* / qwen2.*), and qwen2's per-layer Q/K/V biases are applied when present (llama models simply have none). Tied-embedding models (no output.weight) reuse token_embd as the output projection.

API

: Llm

: Llm {
    Gpu g
    LlmKernels ks
    * Tok tok
    i n_embd
    i n_layer
    i n_head
    i n_kv
    i head_dim
    i n_ff
    i n_vocab
    i n_ctx
    i rope_dim
    i rope_style
    f eps
    f rope_base
    i gg
    i embd_idx
    ( Vec u ) embd_host
    ( Vec i ) wdptr
    ( Vec i ) wbytes
    ( Vec i ) tq_attn_norm
    ( Vec i ) tq_wq
    ( Vec i ) tq_wk
    ( Vec i ) tq_wv
    ( Vec i ) bq
    ( Vec i ) bk
    ( Vec i ) bv
    ( Vec i ) tq_wo
    ( Vec i ) tq_ffn_norm
    ( Vec i ) tq_gate
    ( Vec i ) tq_down
    ( Vec i ) tq_up
    i tq_output
    ( Vec i ) attn_norm
    ( Vec i ) wq
    ( Vec i ) wk
    ( Vec i ) wv
    ( Vec i ) wo
    ( Vec i ) ffn_norm
    ( Vec i ) w_gate
    ( Vec i ) w_down
    ( Vec i ) w_up
    i out_norm
    i w_output
    ( Vec i ) kcache
    ( Vec i ) vcache
    i xd
    i xnd
    i qd
    i kd
    i vd
    i aod
    i tmpd
    i gd
    i ud
    i scored
    i logitsd
    * u logits_host
}

@ string_eq_raw String a s raw → b

: ~ i __lm_last_type -1

Publishes the uploaded tensor's ggml type (0 when it was host-dequantised to f32), so the caller records which matvec kernel the weight needs.

: i LM_CHUNK 64

Positions per prefill chunk. Bounds the batched activation scratch (chunk × n_ff floats) while keeping the launches amortised.

@ llm_open s path i want_ctx → !*Llm String

@ llm_close * Llm m → v

@ llm_tok * Llm m → *Tok

@ llm_n_vocab * Llm m → i

@ llm_n_ctx * Llm m → i

@ llm_logit * Llm m i idx → f

@ llm_eval * Llm m i token i pos → v

One decode step: token at position pos (0-based; caller feeds positions sequentially). Logits for the NEXT token land in logits_host. One decode step: token at position pos.

@ llm_eval_n * Llm m ( Vec i ) ids i first i count i pos0 → v

Evaluate count tokens — ids[first .. first+count) — starting at absolute position pos0. The matmuls run over the whole batch; attention runs per position over the cache this same batch already filled (its causal window differs per position). Logits are produced for the LAST position only, which is all a prefill or a decode step needs.

@ llm_chunk → i

The largest chunk a single llm_eval_n call may take.

@ llm_prefill * Llm m ( Vec i ) ids → v

Prefill: run the whole prompt through the model in chunks, leaving the KV cache filled and the logits set for the LAST prompt token — the distribution the first generated token is drawn from.

---

chat.nu

packages/nurllama/src/chat.nu — chat templates.

Turns a message list into the exact prompt string a chat-tuned model was trained on. The template is chosen from the model's own GGUF metadata: tokenizer.chat_template is matched against the shapes we know, and the architecture/BOS conventions fill the rest — so a model carries its own dialect and the caller never guesses.

( chat_style_of g ) → i CHAT_* below ( chat_render style msgs ) → String prompt for generation ( chat_stop_matches style piece ) → b stop-token text guard

Styles: CHAT_LLAMA2 [INST] <<SYS>>…<</SYS>> user [/INST] assistant </s> CHAT_LLAMA3 <|start_header_id|>role<|end_header_id|>…<|eot_id|> CHAT_CHATML <|im_start|>role\n…<|im_end|> (qwen, many finetunes) CHAT_PLAIN no template — the messages are concatenated; a base (non-chat) model gets its raw prompt, which is the honest thing to do rather than inventing turns.

A ChatMsg's role is one of system / user / assistant.

API

: i CHAT_PLAIN 0

: i CHAT_LLAMA2 1

: i CHAT_LLAMA3 2

: i CHAT_CHATML 3

: ChatMsg

: ChatMsg {
    String role
    String content
}

@ chat_msg s role s content → ChatMsg

@ chat_msg_free ChatMsg m → v

@ chat_msgs_free ( Vec ChatMsg ) v → v

@ chat_style_of * Gguf g → i

Pick the style from the model's own chat template (substring match on the marker tokens each dialect must contain), falling back to PLAIN when the model ships no template — a base model, honestly served.

@ chat_render i style ( Vec ChatMsg ) msgs → String

@ chat_stop_matches i style s piece → b

Some chat models emit their turn terminator as ordinary text rather than the EOS id (a finetune whose eos_token_id was left at the base value). Treat the dialect's terminator as a stop sequence too.

---

kernels.nu

packages/nurllama/src/kernels.nu — the LLM decode ops as GPU kernels.

CUDA-C sources compiled once through packages/gpu (NVRTC on the CUDA backend; the SAME source runs on the CPU/OpenMP backend, so every kernel below deliberately avoids shared / syncthreads — each thread computes its output independently, exactly like the onnx op set). Decode-step shapes (one token at a time) keep the naive forms honest: the matvecs dominate and each row is one thread's serial dot product.

Layout contracts (ggml row-major, ne0 fastest): weight W[out,in] → W[rin + c], matvec y[r] = Σc W[rin+c]·x[c] K/V cache → cache[t(n_kv·hd) + khhd + d] q/k activations → x[h*hd + d]

API

: LlmKernels

: LlmKernels {
    GpuKernel matvec
    GpuKernel mv_q4_0
    GpuKernel mv_q8_0
    GpuKernel mv_q5_0
    GpuKernel mv_q5_1
    GpuKernel mv_q4_k
    GpuKernel mv_q5_k
    GpuKernel mv_q6_k
    GpuKernel mv_f16
    GpuKernel w_f32
    GpuKernel w_q4_0
    GpuKernel w_q8_0
    GpuKernel w_q4_k
    GpuKernel w_q6_k
    b warp
    GpuKernel rmsnorm
    GpuKernel rope
    GpuKernel rope_neox
    GpuKernel attn
    GpuKernel attn_sc
    GpuKernel attn_out
    GpuKernel silumul
    GpuKernel addv
    GpuKernel addrow
    GpuKernel copyat
    b ok
}

@ lk_build Gpu g → LlmKernels

@ lk_free LlmKernels ks → v

@ lk_matvec LlmKernels ks i wd i xd i yd i rows i cols i batch → v

@ lk_matvec_q LlmKernels ks i gt i wd i xd i yd i rows i cols i batch → b

Quantised matvec dispatch: gt is the ggml tensor type of W. Returns F when the type has no device kernel (the caller then falls back to the f32 path — correctness first, always).

@ lk_rmsnorm LlmKernels ks i xd i wd i yd i n f eps i batch → v

@ lk_rope LlmKernels ks i xd i nh i hd i rd i pos f base i style i batch → v

style: 0 = NORM (llama), 1 = NEOX (qwen2)

@ lk_attn LlmKernels ks i qd i kcd i vcd i outd i scd i nh i nkv i hd i npos0 i batch i maxpos → v

npos0 = the causal window of the FIRST batch element (element b sees npos0 + b rows). maxpos strides the per-(batch,head) score scratch. Two-pass attention: scores, then softmax + value-weighted sum.

@ lk_silumul LlmKernels ks i gd i ud i n → v

@ lk_addv LlmKernels ks i ad i bd i n → v

@ lk_addrow LlmKernels ks i ad i vd i n i batch → v

a[b][i] += v[i] for every row of the batch

@ lk_copyat LlmKernels ks i dstd i srcd i off i n → v

---

api.nu

packages/nurllama/src/api.nu — an ollama-compatible HTTP API.

Speaks enough of ollama's wire protocol that existing clients work unchanged:

POST /api/generate { model, prompt, stream?, options? } POST /api/chat { model, messages[{role,content}], stream?, options? } → NDJSON: one JSON object per token while generating, then a final object with done:true (+ token counts). stream:false returns a single aggregated object. GET /api/tags the local store, as ollama's model list POST /api/show { model } → the model's metadata GET / "nurllama is running"

Streaming rides the http package's stream hook (http_app_stream): the handler owns the TcpConn and writes chunked NDJSON itself, so a token reaches the client the moment it is decoded.

One model is resident at a time (LRU of one): a request naming a different model swaps it in. Generation is serialised — the decode loop owns the device — which is honest for a single-GPU host; the server is single-worker so that serialisation is structural rather than a lock.

API

: ~ i g_api_llm 0

The resident model + the store root, reachable from the handlers. Globals may only carry scalar/s literals, so the two strings are plain owned C-strings (strdup'd in, nurl_free'd out) rather than String structs.

: ~ i g_api_style 0

: ~ s g_api_name ``

: ~ s g_api_root ``

@ api_set_root String root → v

@ api_serve String root s host i port → i

Serve on host:port. Single worker: the decode loop owns the device, so serialisation is structural rather than a lock.

---

tokenizer.nu

packages/nurllama/src/tokenizer.nu — LLM tokenizer from a GGUF vocabulary.

Everything comes out of the model file itself (tokenizer.ggml.* metadata read through packages/gguf) — no external vocab files, no exporter step:

( tok_new g ) → !Tok String from an open Gguf ( tok_free t ) ( tok_encode t text add_special ) → ( Vec i ) token ids ( tok_piece t id ) → ( Vec u ) one token's raw bytes ( tok_decode t ids ) → ( Vec u ) ids → bytes (skips control tokens) ( tok_n_vocab t ) ( tok_bos t ) ( tok_eos t ) ( tok_unk t )

Two tokenizer families, selected by tokenizer.ggml.model:

SPM ("llama") — SentencePiece-style bigram merging: split the space-escaped text ( → ▁ U+2581, optional leading space) into UTF-8 characters, then repeatedly merge the adjacent pair whose concatenation is a vocab piece with the HIGHEST score (ties → leftmost), exactly llama.cpp's llm_tokenizer_spm selection order. Characters that never reach a vocab piece fall back to the <0xXX> BYTE tokens, then to UNK.

BPE ("gpt2") — byte-level BPE: GPT-2's byte→unicode remap, a regex-shaped pre-split, then lowest-merge-rank pairing driven by tokenizer.ggml.merges. The pre-split VARIANT comes from the model's own tokenizer.ggml.pre:

default / gpt2 ?\p{L}+ · ?\p{N}+ (digits GROUPED) · ?[^\s\p{L}\p{N}]+ · contractions · whitespace qwen2 \p{N} (ONE digit per token) · a letter run may absorb one leading non-letter/non-digit character · case-insensitive contractions · trailing-newline punctuation runs

\p{L} is approximated as "ASCII letter or any codepoint ≥ 0x80", which is exact for every script these vocabularies actually contain.

The Tok owns copies of every piece (the source Gguf can be closed after tok_new). The piece→id map borrows the buffers of pieces — stable because Vec growth moves the String structs, never the heap buffers they point to.

API

: i TT_NORMAL 1

token_type values (tokenizer.ggml.token_type, llama.cpp llama_token_type)

: i TT_UNKNOWN 2

: i TT_CONTROL 3

: i TT_USER 4

: i TT_UNUSED 5

: i TT_BYTE 6

: i TOK_SPM 0

: i TOK_BPE 1

: i PRE_DEFAULT 0

BPE pre-tokenizer variants (tokenizer.ggml.pre)

: i PRE_QWEN2 1

: Tok

: Tok {
    i mode
    ( Vec String ) pieces
    ( Vec f ) scores
    ( Vec i ) ttype
    ( HashMap s i ) lookup
    ( HashMap s i ) ranks
    ( Vec String ) rankkeys
    ( Vec String ) byte_enc
    ( Vec i ) byte_id
    i pre
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
}

@ tok_new * Gguf g → !*Tok String

Build a tokenizer from an open GGUF model. Copies the vocabulary — the Gguf may be closed afterwards.

@ tok_free * Tok t → v

@ tok_n_vocab * Tok t → i

@ tok_bos * Tok t → i

@ tok_eos * Tok t → i

@ tok_unk * Tok t → i

@ tok_encode * Tok t s text b add_special → ( Vec i )

@ tok_piece * Tok t i id → ( Vec u )

Raw bytes of one token: SPM BYTE pieces become their byte, ▁ becomes a space, control tokens become nothing; BPE pieces run the inverse byte remap.

@ tok_decode * Tok t ( Vec i ) ids → ( Vec u )

---

main.nu

nurllama — run language models locally.

nurllama pull <src> [--name N] download into ~/.nurllama (hf.co/ORG/REPO/FILE.gguf or a URL; resumes a broken pull) nurllama list · rm <name> · verify <name> the local model store nurllama serve [--host H] [--port N] ollama-compatible HTTP API nurllama chat <model|name> interactive chat (model's own template from GGUF) nurllama run <model|name> <prompt> generate (stream to stdout) -n N (default 64) · --temp F (0 = greedy, default 0.8) --topk N · --topp F · --seed N · --ctx N nurllama logits <model.gguf> <prompt> final-position logits, one per line (verification tap) nurllama tokenize <model.gguf> <text> encode → token ids nurllama detok <model.gguf> <id> [id …] decode ids → bytes nurllama vocab <model.gguf> [n] dump the first n pieces nurllama selftest synthetic SPM + BPE vocab round-trips, bit-exact

Everything is loaded from the model file alone (tokenizer.ggml. + llama. metadata + tensors via packages/gguf); compute runs on packages/gpu — CUDA when present, NURL_GPU=cpu for the host backend.

API

: NlCnt

: NlCnt {
    i pass
    i fail
}

@ main → i

---

sample.nu

packages/nurllama/src/sample.nu — host-side sampling over the logits.

( sample_greedy m ) → i argmax ( sample_next m rng temp topk topp ) → i softmax(T) → top-k → top-p → draw

temp ≤ 0 collapses to greedy. top-k ≤ 0 disables the k filter; topp ≥ 1 disables the nucleus filter. Deterministic under a seeded Rng.

API

@ sample_greedy * Llm m → i

& libm @ exp f x → f

host-side softmax needs exp — bind libm directly, the stdlib idiom

@ sample_next * Llm m Rng rng f temp i topk f topp → i

---

store.nu

packages/nurllama/src/store.nu — the local model store (~/.nurllama).

Content-addressed, cas-shaped, but streaming end to end — a blob is never fully resident:

<root>/blobs/sha256-<64hex> the model files, named by content <root>/manifests/<name> one small JSON per model name

A manifest records { name, url, digest, size } — the name is the user-facing handle (nurllama run <name>), the digest is the proof. Two names may point at the same blob (dedup by content); rm only deletes a blob once no manifest references it. verify re-hashes the blob through the incremental sha256 and refuses drift.

( nl_store_root ) → String $NURLLAMA_HOME | ~/.nurllama ( nl_resolve root nameOrPath ) → ?String path to a model file ( nl_store_add root name url hex size ) → !v String ( nl_store_list root ) → v prints the table ( nl_store_rm root name ) → !v String ( nl_store_verify root name ) → !v String ( nl_blob_path root hex ) ( nl_manifest_path root name ) ( nl_hash_file path ) → !String String streaming sha256 hex

API

@ nl_store_root → String

@ nl_blob_path String root s hex → String

@ nl_manifest_path String root s name → String

@ nl_store_init String root → !v String

@ nl_hash_file s path → !String String

Streaming sha256 of a file — constant memory whatever the size.

@ nl_store_add String root s name s url s hex i size → !v String

@ nl_resolve String root s arg → ?String

Resolve a model argument: an existing file path wins; otherwise the store's manifest name → blob path. None = unknown.

@ nl_store_list String root → v

@ nl_store_rm String root s name → !v String

@ nl_store_verify String root s name → !v String

Re-hash the blob and compare with the manifest digest — the store's integrity loop (mirrors cas_get's refuse-on-mismatch, streaming).