main.nu

packages/tokenizer/src/main.nu — the CLI.

tokenizer encode <tokenizer.json> <text> tokenizer decode <tokenizer.json> <id> [id …] tokenizer vocab <tokenizer.json> [n]

One file: tokenizer.json is what every HF checkpoint ships, and it is the only place that says which pieces are SPECIAL. (vocab.json + merges.txt + added_tokens.json + special_tokens_map.json are four files with four conventions that disagree with each other — see src/hf.nu.) A GGUF's vocabulary is loaded by whoever has the GGUF; nurllama does.

API

@ main → i

---

hf.nu

packages/tokenizer/src/hf.nu — load a vocabulary the way Hugging Face ships one: vocab.json (piece → id) and merges.txt (BPE rules, best first).

This is the loader whisper needs. A GGUF carries its vocabulary in the model file's metadata; an HF checkpoint keeps it in two files next to the weights, and nothing else about the tokenizer changes — which is why the engine takes PARTS and does not know about either.

( tok_from_hf vocab_path merges_path added_path spec ) → !Tok String

added_path (added_tokens.json / a special-tokens file) may be empty. Every piece it names is marked CONTROL, which is what makes <|startoftranscript|> come out as one id instead of eleven — see the special-token parsing in tokenizer.nu.

merges.txt is the fragile part, and it is fragile in a specific way: its first line is a version comment (#version: 0.2) that is NOT a merge rule, and taking it as rank 0 would make the most-preferred merge in the whole vocabulary a rule that does not exist. It is skipped, and any other line that is not exactly two space-separated pieces is skipped with it.

API

@ tok_from_hf s vocab_path s merges_path s added_path TokSpec spec → !Tok String

The whole loader: vocab.json + merges.txt (+ an optional added-tokens file).

@ tok_from_tokenizer_json s path TokSpec spec → !Tok String

---

unigram.nu

packages/tokenizer/src/unigram.nu — Hugging Face Unigram tokenizer (tokenizer.json), the XLM-RoBERTa / BGE / sentencepiece-Unigram family.

The engine in tokenizer.nu covers llama-style SentencePiece (greedy bigram merging) and byte-level BPE. Unigram is a THIRD model: pieces carry log probabilities and a sentence is segmented by Viterbi — the highest-total-score path over the whole string — which greedy merging does not reproduce. This file implements the full tokenizer.json pipeline for model.type == "Unigram":

1. added-token extraction (specials matched literally on RAW text)
2. normalizer: sentencepiece "Precompiled" charsmap — a darts-clone

double-array trie of replacement rules (NFKC-style) — followed by the Replace rule (collapse space runs), as in the file

1. Metaspace pre-tokenization: prepend ▁, every ' ' → ▁ (U+2581)
2. Unigram Viterbi over piece scores; unknown characters take

unk_score = min_score − 10 and consecutive unks FUSE into one

1. TemplateProcessing: <s> … </s> (ids read from the file)

( uni_load path ) → !Unigram String ( uni_encode u text add_special ( Vec i ) out ) → b ( uni_free u ) → v early release (optional) plus uni_n_vocab / uni_bos / uni_eos / uni_unk / uni_piece

A Unigram is a handle: every copy is the same tokenizer, and the last owner releases it. Nothing here is released by hand.

Verified token-for-token against Hugging Face tokenizers on a multilingual corpus (see tests/unigram_test.sh).

API

: UnigramImpl

: UnigramImpl {
    ( Vec String ) pieces
    ( Vec f ) scores
    ( HashMap s i ) lookup
    // fast piece lookup for the Viterbi inner loop: open-addressed,
    // power-of-2 table with the FNV-1a hash STORED per slot, probing
    // against raw text bytes — no substring copy, no re-hash, and a
    // memcmp only when the 64-bit hashes already agree. `lookup` stays
    // for the cold callers (added-token names).
    //
    // Hash and id are INTERLEAVED in one array — slot s is ft[2s] and
    // ft[2s+1] — not two parallel Vecs. At 250k pieces the table is 8 MB,
    // so every probe is a cache miss, and two arrays make it two misses
    // for what fits in one 64-byte line. ft[2s+1] holds id+1; 0 is an
    // empty slot.
    ( Vec i ) ft
    i ft_mask
    i unk_id
    f unk_score
    i max_piece  // longest piece, in bytes
    // Precompiled charsmap: darts double-array units + replacement pool
    ( Vec u ) trie
    ( Vec u ) pool
    b has_norm
    b collapse_spaces  // the Replace " {2,}" → " " rule
    b add_prefix  // Metaspace prepend ▁
    i bos
    i eos
    ( Vec String ) added  // added-token literals, longest first
    ( Vec i ) added_ids
    ( Vec i ) added_lstrip  // 1 = whitespace BEFORE the token is consumed
    ( Vec i ) added_rstrip  // 1 = whitespace AFTER the token is consumed
}

: Unigram { ( Arc UnigramImpl ) a }

A Unigram is a handle on its state in an Arc (stdlib/std/arc.nu): every copy is the same tokenizer, and the last owner releases it.

@ Unigram_share Unigram h → Unigram

@ Unigram_drop sink Unigram h → v

: i UNI_FNV_OFF -3750763034362895579

FNV-1a 64-bit — extendable one byte at a time, which is what lets the Viterbi loop keep ONE rolling hash per start position instead of re-hashing every substring from scratch.

: i UNI_FNV_PRIME 1099511628211

@ uni_load s path → !Unigram String

@ uni_free sink Unigram u → v

Let go of u now rather than at the end of its owner's scope.

@ uni_n_vocab Unigram u__h → i

@ uni_bos Unigram u__h → i

@ uni_eos Unigram u__h → i

@ uni_unk Unigram u__h → i

@ uni_piece Unigram u__h i id → s

Borrowed piece text ("" out of range).

@ uni_encode Unigram u__h s text b add_special ( Vec i ) out → b

Encode text. add_special wraps the result in the file's template (<s> … </s>). Returns F only when the tokenizer is unusable.

---

tokenizer.nu

packages/tokenizer/src/tokenizer.nu — the tokenizer engine.

This is nurllama's tokenizer, lifted out of it: SentencePiece bigram merging for llama-family vocabularies, byte-level BPE for gpt2-family ones, inline special-token parsing, and decoding — the parts that have nothing to do with where the vocabulary came from.

It is deliberately loader-agnostic. tok_build takes the PARTS — pieces, scores, token types, merges, and the handful of flags — so a vocabulary can arrive from a GGUF's metadata (nurllama), from Hugging Face's vocab.json + merges.txt (whisper, see src/hf.nu), or from anywhere else, without this file gaining a dependency on any container format.

( tok_build spec pieces scores types merges ) → !Tok String (takes the four vectors) ( tok_encode t text add_special ) → ( Vec i ) ( tok_decode t ids ) → ( Vec u ) ( tok_piece t id ) → ( Vec u ) ( tok_piece_str t id ) → s piece text (borrowed) ( tok_free t ) → v early release (optional)

A Tok is a handle: every copy is the same tokenizer, and the last owner releases it. Nothing here is released by hand.

The engine's behaviour is pinned by nurllama's existing parity tests, which compare it token-for-token against independent python SentencePiece and BPE implementations on real models — the extraction is faithful iff those still pass.

API

: i TT_NORMAL 1

: i TT_UNKNOWN 2

: i TT_CONTROL 3

: i TT_USER 4

: i TT_UNUSED 5

: i TT_BYTE 6

: i TOK_SPM 0

: i TOK_BPE 1

: i PRE_DEFAULT 0

BPE pre-tokenizer variants (tokenizer.ggml.pre)

: i PRE_QWEN2 1

: TokImpl

: TokImpl {
    i mode
    ( Vec String ) pieces
    ( Vec f ) scores
    ( Vec i ) ttype
    ( HashMap String i ) lookup  // its own copy of every piece
    ( HashMap String i ) ranks
    ( Vec String ) rankkeys
    ( Vec String ) byte_enc
    ( Vec i ) byte_id
    i pre
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
    // Inline special tokens (CONTROL / USER_DEFINED pieces such as
    // `<start_of_turn>` or `<|im_start|>`). A chat template puts these in the
    // TEXT, and they must come out as their own single id — tokenised as
    // ordinary text they shred into a dozen pieces and the model answers a
    // different question than the one asked. `sp_first` is a 256-entry table
    // of the bytes any special piece can start with, so ordinary text pays
    // one array lookup per byte rather than a scan of the whole list.
    ( Vec i ) sp_ids
    ( Vec i ) sp_first
}

: Tok { ( Arc TokImpl ) a }

A Tok is a handle on its state in an Arc (stdlib/std/arc.nu): every copy is the same tokenizer, and the last owner releases it.

@ Tok_share Tok h → Tok

@ Tok_drop sink Tok h → v

@ tok_empty → Tok

A tokenizer with no vocabulary: what an engine record holds before its model's tokenizer is loaded (it encodes nothing and decodes nothing).

@ _tk_geti ( Vec i ) v i k i def → i

@ _tk_piece_data TokImpl t i k → s

Borrowed data pointer of piece #k ("" when out of range).

@ _tk_build_byte_enc → ( Vec String )

: TokSpec

: TokSpec {
    i mode  // TOK_SPM | TOK_BPE
    i pre  // PRE_DEFAULT | PRE_QWEN2
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
}

Everything a vocabulary needs to say about itself, whatever file it came in.

@ tok_build TokSpec spec sink ( Vec String ) pieces sink ( Vec f ) scores sink ( Vec i ) types sink ( Vec String ) merges → !Tok String

Build a tokenizer from its PARTS. Takes the four vectors — they become the Tok's.

pieces the vocabulary, in id order scores SPM merge scores (empty for BPE) types token types (TT_NORMAL/CONTROL/USER/BYTE/…); empty = all NORMAL merges BPE merge rules, "A B", in rank order (empty for SPM)

@ tok_free sink Tok t → v

Let go of t now rather than at the end of its owner's scope.

@ tok_n_vocab Tok t__h → i

@ tok_bos Tok t__h → i

@ tok_eos Tok t__h → i

@ tok_unk Tok t__h → i

@ tok_encode Tok t__h s text b add_special → ( Vec i )

@ tok_piece Tok t__h i id → ( Vec u )

Raw bytes of one token: SPM BYTE pieces become their byte, ▁ becomes a space, control tokens become nothing; BPE pieces run the inverse byte remap.

@ tok_piece_str Tok t__h i id → s

The text of piece id as stored in the vocabulary (borrowed: valid while the Tok is; "" out of range).

@ tok_decode Tok t__h ( Vec i ) ids → ( Vec u )