hf.nu

packages/tokenizer/src/hf.nu — load a vocabulary the way Hugging Face ships one: vocab.json (piece → id) and merges.txt (BPE rules, best first).

This is the loader whisper needs. A GGUF carries its vocabulary in the model file's metadata; an HF checkpoint keeps it in two files next to the weights, and nothing else about the tokenizer changes — which is why the engine takes PARTS and does not know about either.

( tok_from_hf vocab_path merges_path added_path spec ) → !*Tok String

added_path (added_tokens.json / a special-tokens file) may be empty. Every piece it names is marked CONTROL, which is what makes <|startoftranscript|> come out as one id instead of eleven — see the special-token parsing in tokenizer.nu.

merges.txt is the fragile part, and it is fragile in a specific way: its first line is a version comment (#version: 0.2) that is NOT a merge rule, and taking it as rank 0 would make the most-preferred merge in the whole vocabulary a rule that does not exist. It is skipped, and any other line that is not exactly two space-separated pieces is skipped with it.

API

@ tok_from_hf s vocab_path s merges_path s added_path TokSpec spec → !*Tok String

The whole loader: vocab.json + merges.txt (+ an optional added-tokens file).

@ tok_from_tokenizer_json s path TokSpec spec → !*Tok String

---

tokenizer.nu

packages/tokenizer/src/tokenizer.nu — the tokenizer engine.

This is nurllama's tokenizer, lifted out of it: SentencePiece bigram merging for llama-family vocabularies, byte-level BPE for gpt2-family ones, inline special-token parsing, and decoding — the parts that have nothing to do with where the vocabulary came from.

It is deliberately loader-agnostic. tok_build takes the PARTS — pieces, scores, token types, merges, and the handful of flags — so a vocabulary can arrive from a GGUF's metadata (nurllama), from Hugging Face's vocab.json + merges.txt (whisper, see src/hf.nu), or from anywhere else, without this file gaining a dependency on any container format.

( tok_build spec pieces scores types merges ) → !*Tok String (takes ownership) ( tok_encode t text add_special ) → ( Vec i ) ( tok_decode t ids ) → ( Vec u ) ( tok_piece t id ) → ( Vec u ) ( tok_free t ) → v

The engine's behaviour is pinned by nurllama's existing parity tests, which compare it token-for-token against independent python SentencePiece and BPE implementations on real models — the extraction is faithful iff those still pass.

API

: i TT_NORMAL 1

: i TT_UNKNOWN 2

: i TT_CONTROL 3

: i TT_USER 4

: i TT_UNUSED 5

: i TT_BYTE 6

: i TOK_SPM 0

: i TOK_BPE 1

: i PRE_DEFAULT 0

BPE pre-tokenizer variants (tokenizer.ggml.pre)

: i PRE_QWEN2 1

: Tok

: Tok {
    i mode
    ( Vec String ) pieces
    ( Vec f ) scores
    ( Vec i ) ttype
    ( HashMap s i ) lookup
    ( HashMap s i ) ranks
    ( Vec String ) rankkeys
    ( Vec String ) byte_enc
    ( Vec i ) byte_id
    i pre
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
    // Inline special tokens (CONTROL / USER_DEFINED pieces such as
    // `<start_of_turn>` or `<|im_start|>`). A chat template puts these in the
    // TEXT, and they must come out as their own single id — tokenised as
    // ordinary text they shred into a dozen pieces and the model answers a
    // different question than the one asked. `sp_first` is a 256-entry table
    // of the bytes any special piece can start with, so ordinary text pays
    // one array lookup per byte rather than a scan of the whole list.
    ( Vec i ) sp_ids
    ( Vec i ) sp_first
}

: TokSpec

: TokSpec {
    i mode  // TOK_SPM | TOK_BPE
    i pre  // PRE_DEFAULT | PRE_QWEN2
    i bos
    i eos
    i unk
    b add_bos
    b add_eos
    b add_space_prefix
}

Everything a vocabulary needs to say about itself, whatever file it came in.

@ tok_build TokSpec spec ( Vec String ) pieces ( Vec f ) scores ( Vec i ) types ( Vec String ) merges → !*Tok String

Build a tokenizer from its PARTS. TAKES OWNERSHIP of the four vectors — they become the Tok's, and tok_free releases them.

pieces the vocabulary, in id order scores SPM merge scores (empty for BPE) types token types (TT_NORMAL/CONTROL/USER/BYTE/…); empty = all NORMAL merges BPE merge rules, "A B", in rank order (empty for SPM)

@ tok_free * Tok t → v

@ tok_n_vocab * Tok t → i

@ tok_bos * Tok t → i

@ tok_eos * Tok t → i

@ tok_unk * Tok t → i

@ tok_encode * Tok t s text b add_special → ( Vec i )

@ tok_piece * Tok t i id → ( Vec u )

Raw bytes of one token: SPM BYTE pieces become their byte, ▁ becomes a space, control tokens become nothing; BPE pieces run the inverse byte remap.

@ tok_decode * Tok t ( Vec i ) ids → ( Vec u )

---

main.nu

packages/tokenizer/src/main.nu — the CLI.

tokenizer encode <tokenizer.json> <text> tokenizer decode <tokenizer.json> <id> [id …] tokenizer vocab <tokenizer.json> [n]

One file: tokenizer.json is what every HF checkpoint ships, and it is the only place that says which pieces are SPECIAL. (vocab.json + merges.txt + added_tokens.json + special_tokens_map.json are four files with four conventions that disagree with each other — see src/hf.nu.) A GGUF's vocabulary is loaded by whoever has the GGUF; nurllama does.

API

@ main → i