hf.nu

packages/tokenizer/src/hf.nu — load a vocabulary the way Hugging Face ships one: vocab.json (piece → id) and merges.txt (BPE rules, best first).

This is the loader whisper needs. A GGUF carries its vocabulary in the model file's metadata; an HF checkpoint keeps it in two files next to the weights, and nothing else about the tokenizer changes — which is why the engine takes PARTS and does not know about either.

( tok_from_hf vocab_path merges_path added_path spec ) → !*Tok String

added_path (added_tokens.json / a special-tokens file) may be empty. Every piece it names is marked CONTROL, which is what makes <|startoftranscript|> come out as one id instead of eleven — see the special-token parsing in tokenizer.nu.

merges.txt is the fragile part, and it is fragile in a specific way: its first line is a version comment (#version: 0.2) that is NOT a merge rule, and taking it as rank 0 would make the most-preferred merge in the whole vocabulary a rule that does not exist. It is skipped, and any other line that is not exactly two space-separated pieces is skipped with it.

API

@ tok_from_hf s vocab_path s merges_path s added_path TokSpec spec → !*Tok String

The whole loader: vocab.json + merges.txt (+ an optional added-tokens file).

@ tok_from_tokenizer_json s path TokSpec spec → !*Tok String

---

tokenizer.nu

packages/tokenizer/src/tokenizer.nu — the tokenizer engine.

This is nurllama's tokenizer, lifted out of it: SentencePiece bigram merging for llama-family vocabularies, byte-level BPE for gpt2-family ones, inline special-token parsing, and decoding — the parts that have nothing to do with where the vocabulary came from.

It is deliberately loader-agnostic. tok_build takes the PARTS — pieces, scores, token types, merges, and the handful of flags — so a vocabulary can arrive from a GGUF's metadata (nurllama), from Hugging Face's vocab.json + merges.txt (whisper, see src/hf.nu), or from anywhere else, without this file gaining a dependency on any container format.

( tok_build spec pieces scores types merges ) → !*Tok String (takes ownership) ( tok_encode t text add_special ) → ( Vec i ) ( tok_decode t ids ) → ( Vec u ) ( tok_piece t id ) → ( Vec u ) ( tok_free t ) → v

The engine's behaviour is pinned by nurllama's existing parity tests, which compare it token-for-token against independent python SentencePiece and BPE implementations on real models — the extraction is faithful iff those still pass.

API

: i TT_NORMAL 1

: i TT_UNKNOWN 2

: i TT_CONTROL 3

: i TT_USER 4

: i TT_UNUSED 5

: i TT_BYTE 6

: i TOK_SPM 0

: i TOK_BPE 1

: i PRE_DEFAULT 0

BPE pre-tokenizer variants (tokenizer.ggml.pre)

: i PRE_QWEN2 1

: Tok

: Tok {
    i mode
    ( Vec String ) pieces
    ( Vec f ) scores
    ( Vec i ) ttype
    ( HashMap s i ) lookup
    ( HashMap s i ) ranks
    ( Vec String ) rankkeys
    ( Vec String ) byte_enc
    ( Vec i ) byte_id
    i pre
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
    // Inline special tokens (CONTROL / USER_DEFINED pieces such as
    // `<start_of_turn>` or `<|im_start|>`). A chat template puts these in the
    // TEXT, and they must come out as their own single id — tokenised as
    // ordinary text they shred into a dozen pieces and the model answers a
    // different question than the one asked. `sp_first` is a 256-entry table
    // of the bytes any special piece can start with, so ordinary text pays
    // one array lookup per byte rather than a scan of the whole list.
    ( Vec i ) sp_ids
    ( Vec i ) sp_first
}

@ _tk_geti ( Vec i ) v i k i def → i

@ _tk_piece_data * Tok t i k → s

Borrowed data pointer of piece #k ("" when out of range).

@ _tk_build_byte_enc → ( Vec String )

: TokSpec

: TokSpec {
    i mode  // TOK_SPM | TOK_BPE
    i pre  // PRE_DEFAULT | PRE_QWEN2
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
}

Everything a vocabulary needs to say about itself, whatever file it came in.

@ tok_build TokSpec spec ( Vec String ) pieces ( Vec f ) scores ( Vec i ) types ( Vec String ) merges → !*Tok String

Build a tokenizer from its PARTS. TAKES OWNERSHIP of the four vectors — they become the Tok's, and tok_free releases them.

pieces the vocabulary, in id order scores SPM merge scores (empty for BPE) types token types (TT_NORMAL/CONTROL/USER/BYTE/…); empty = all NORMAL merges BPE merge rules, "A B", in rank order (empty for SPM)

@ tok_free * Tok t → v

@ tok_n_vocab * Tok t → i

@ tok_bos * Tok t → i

@ tok_eos * Tok t → i

@ tok_unk * Tok t → i

@ tok_encode * Tok t s text b add_special → ( Vec i )

@ tok_piece * Tok t i id → ( Vec u )

Raw bytes of one token: SPM BYTE pieces become their byte, ▁ becomes a space, control tokens become nothing; BPE pieces run the inverse byte remap.

@ tok_decode * Tok t ( Vec i ) ids → ( Vec u )

---

main.nu

packages/tokenizer/src/main.nu — the CLI.

tokenizer encode <tokenizer.json> <text> tokenizer decode <tokenizer.json> <id> [id …] tokenizer vocab <tokenizer.json> [n]

One file: tokenizer.json is what every HF checkpoint ships, and it is the only place that says which pieces are SPECIAL. (vocab.json + merges.txt + added_tokens.json + special_tokens_map.json are four files with four conventions that disagree with each other — see src/hf.nu.) A GGUF's vocabulary is loaded by whoever has the GGUF; nurllama does.

API

@ main → i

---

unigram.nu

packages/tokenizer/src/unigram.nu — Hugging Face Unigram tokenizer (tokenizer.json), the XLM-RoBERTa / BGE / sentencepiece-Unigram family.

The engine in tokenizer.nu covers llama-style SentencePiece (greedy bigram merging) and byte-level BPE. Unigram is a THIRD model: pieces carry log probabilities and a sentence is segmented by Viterbi — the highest-total-score path over the whole string — which greedy merging does not reproduce. This file implements the full tokenizer.json pipeline for model.type == "Unigram":

1. added-token extraction (specials matched literally on RAW text)
2. normalizer: sentencepiece "Precompiled" charsmap — a darts-clone

double-array trie of replacement rules (NFKC-style) — followed by the Replace rule (collapse space runs), as in the file

1. Metaspace pre-tokenization: prepend ▁, every ' ' → ▁ (U+2581)
2. Unigram Viterbi over piece scores; unknown characters take

unk_score = min_score − 10 and consecutive unks FUSE into one

1. TemplateProcessing: <s> … </s> (ids read from the file)

( uni_load path ) → !*Unigram String ( uni_encode u text add_special ( Vec i ) out ) → b ( uni_free u ) → v plus uni_n_vocab / uni_bos / uni_eos / uni_unk / uni_piece

Verified token-for-token against Hugging Face tokenizers on a multilingual corpus (see tests/unigram_test.sh).

API

: Unigram

: Unigram {
    ( Vec String ) pieces
    ( Vec f ) scores
    ( HashMap s i ) lookup
    i unk_id
    f unk_score
    i max_piece  // longest piece, in bytes
    // Precompiled charsmap: darts double-array units + replacement pool
    ( Vec u ) trie
    ( Vec u ) pool
    b has_norm
    b collapse_spaces  // the Replace " {2,}" → " " rule
    b add_prefix  // Metaspace prepend ▁
    i bos
    i eos
    ( Vec String ) added  // added-token literals, longest first
    ( Vec i ) added_ids
    ( Vec i ) added_lstrip  // 1 = whitespace BEFORE the token is consumed
    ( Vec i ) added_rstrip  // 1 = whitespace AFTER the token is consumed
}

@ uni_load s path → !*Unigram String

@ uni_free * Unigram u → v

@ uni_n_vocab * Unigram u → i

@ uni_bos * Unigram u → i

@ uni_eos * Unigram u → i

@ uni_unk * Unigram u → i

@ uni_piece * Unigram u i id → s

Borrowed piece text ("" out of range).

@ uni_encode * Unigram u s text b add_special ( Vec i ) out → b

Encode text. add_special wraps the result in the file's template (<s> … </s>). Returns F only when the tokenizer is unusable.