model.nu

packages/whisper/src/model.nu — the encoder.

Whisper's encoder turns a 30-second log-mel spectrogram (3000 frames × 80 mels) into 1500 vectors the decoder attends to. The shape, read out of the checkpoint's own tensor names rather than remembered:

conv1 (k=3, s=1, pad=1) 80 → d_model then GELU conv2 (k=3, s=2, pad=1) d_model → d_model then GELU 3000 → 1500 frames

• embed_positions (a stored tensor, not computed sinusoids)

N × [ LN → self-attn (NON-CAUSAL) → +residual LN → fc1 → GELU → fc2 → +residual ] LN

The corner that bites: k_proj HAS NO BIAS. q, v and out_proj all have one; k does not. Whisper is not alone in this — it is inherited from the original implementation — and a loader that adds a zero bias where none exists is fine, while one that expects a bias tensor and finds none will happily read whatever is next in the file.

API

: Whisper

: Whisper {
    Gpu g
    WhKernels ks
    b w_half  // matrix weights live on the device as raw f16 halves
    i st  // *St, the safetensors file (HF checkpoint) — 0 in ggml mode
    i gg  // *Gg, whisper.cpp's legacy ggml container — 0 in HF mode
    i n_mels
    i d_model
    i n_head
    i head_dim
    i d_head  // the DECODER's head count — the same in every
    i d_head_dim  // whisper so far, but the config states both
    i n_enc_layer
    i n_dec_layer
    i n_ctx_enc  // 1500 encoder positions
    i n_vocab
    // encoder weights (device pointers)
    i conv1_w
    i conv1_b
    i conv2_w
    i conv2_b
    i enc_pos
    ( Vec i ) e_ln1_w
    ( Vec i ) e_ln1_b
    ( Vec i ) e_wq
    ( Vec i ) e_bq
    ( Vec i ) e_wk  // no bias — whisper's k_proj has none
    ( Vec i ) e_wv
    ( Vec i ) e_bv
    ( Vec i ) e_wo
    ( Vec i ) e_bo
    ( Vec i ) e_ln2_w
    ( Vec i ) e_ln2_b
    ( Vec i ) e_fc1_w
    ( Vec i ) e_fc1_b
    ( Vec i ) e_fc2_w
    ( Vec i ) e_fc2_b
    i e_lnf_w
    i e_lnf_b
    // decoder weights
    i tok_embd  // also the output projection — whisper ties them
    i dec_pos
    ( Vec i ) d_ln1_w
    ( Vec i ) d_ln1_b
    ( Vec i ) d_wq
    ( Vec i ) d_bq
    ( Vec i ) d_wk  // no bias, like the encoder's
    ( Vec i ) d_wv
    ( Vec i ) d_bv
    ( Vec i ) d_wo
    ( Vec i ) d_bo
    ( Vec i ) d_lnx_w  // cross-attention's norm
    ( Vec i ) d_lnx_b
    ( Vec i ) x_wq
    ( Vec i ) x_bq
    ( Vec i ) x_wk  // no bias
    ( Vec i ) x_wv
    ( Vec i ) x_bv
    ( Vec i ) x_wo
    ( Vec i ) x_bo
    ( Vec i ) d_ln2_w
    ( Vec i ) d_ln2_b
    ( Vec i ) d_fc1_w
    ( Vec i ) d_fc1_b
    ( Vec i ) d_fc2_w
    ( Vec i ) d_fc2_b
    i d_lnf_w
    i d_lnf_b
    // decoder state
    ( Vec i ) kcache  // per layer: n_ctx_dec × d_model
    ( Vec i ) vcache
    ( Vec i ) xk  // per layer: the encoder's K, computed ONCE
    ( Vec i ) xv
    i n_ctx_dec
    i dx_d  // one token's activation
    i dxn_d
    i dq_d
    i dk_d
    i dv_d
    i dao_d
    i dtmp_d
    i dff_d
    i dsc_d  // scores: n_head × max(n_ctx_dec, n_ctx_enc)
    i logits_d
    * u logits_host
    // scratch
    i mel_d  // 3000 × n_mels
    i c1_d  // 3000 × d_model
    i c2_d  // 1500 × d_model
    i xn_d
    i q_d
    i k_d
    i v_d
    i ao_d
    i tmp_d
    i ff_d  // 1500 × 4·d_model
    i sc_d  // n_head × 1500 × 1500 scores
    i enc_out  // 1500 × d_model — what the decoder attends to
    ( Vec i ) bufs  // every device allocation, for the free
    ( Vec i ) bufsz
}

@ wh_open s config_path s weights_path → !*Whisper String

@ wh_open_ggml s path → !*Whisper String

Open a whisper.cpp ggml container: hyperparameters and tokenizer live in the same file as the weights, so the only argument is the file.

@ wh_close * Whisper w → v

@ wh_encode * Whisper w ( Vec f ) mel → v

Run the encoder over a 30-second log-mel spectrogram: mel is 3000 × n_mels, row-major (a frame is contiguous) — exactly what packages/audio produces. Leaves the 1500 encoder states on the device (wh_enc_out).

@ wh_enc_out * Whisper w → ( Vec f )

The encoder states, back on the host: 1500 × d_model, row-major.

@ wh_prepare_cross * Whisper w → v

Cross-attention's K and V come from the ENCODER, and the encoder does not change while a clip is being transcribed. So they are computed ONCE here, after wh_encode — not once per generated token, which would redo 1500×d_model of work for every word.

@ wh_decode_step * Whisper w i tok i pos → v

One decoder step: token tok at position pos, with everything before it already in the KV cache. Leaves the logits on the device.

The self-attention needs no causal mask: there is exactly ONE query, at the end, and the cache holds only what came before it. Masking is what you do when you process several positions at once.

@ wk_addv_row * Whisper w i pos → v

x += embed_positions[pos]. The row lives inside a matrix, so it is added by pointing at the row rather than by a broadcast.

@ wh_logits * Whisper w → ( Vec f )

The logits of the last step, on the host.

@ wh_argmax ( Vec f ) v → i

argmax — greedy decoding is what whisper does by default, and what the reference implementations are compared at.

---

serve.nu

packages/whisper/src/serve.nu — whisper as a service.

whisper serve <model-dir> --addr 0.0.0.0:6543 [--lang en] [--vad] [--timestamps]

The point of a server is WHEN the model loads. The CLI pays the whole bill per invocation — read 1.5 GB of safetensors, convert f16 to f32 on the device, compile the kernels — and only then hears the audio. Here all of that happens once, before the port opens; a request pays for its own audio and nothing else.

The HTTP surface is whisper.cpp's, so its clients work unchanged:

POST /inference multipart/form-data with a file field (WAV), plus optional language and response_format (json|text) fields → {"text":"..."} or the bare text. A raw WAV body (no multipart) is also accepted — curl --data-binary @clip.wav is not a browser form, and there is no reason to make it pretend to be one. GET /health {"status":"ok", model shape, request count} GET / the same (a browser poking the port should learn something, not get a 404)

Per-request vad and timestamps fields override the server-side flags, so one server can serve both a subtitle pipeline and a bare-text one.

Handlers are top-level functions reading module globals, not closures over locals — the same idiom as nurllama's api.nu, and for the same reason: closure environments are manual in NURL, and a server's handlers live for the process.

API

: ~ i g_srv_w 0 // *Whisper, as an address (0 = not serving)

: ~ i g_srv_t 0 // *Tok

: ~ i g_srv_reqs 0

: ~ s g_srv_lang ``

: ~ b g_srv_vad F

: ~ b g_srv_ts F

: ~ i g_srv_max 0

: ~ f g_srv_nospeech 0.6

: ~ s g_srv_token // empty = open server (loopback default)

: ~ i g_ws_f32 0 // 1 = client sends float32 frames, else PCM16

: ~ s g_ws_lang // per-connection language override ( = server default)

@ wh_serve s dir s host i port s lang i maxtok b use_vad b with_ts s cert s key s token → i

Load the model, open the port, serve until stopped.

---

ggml.nu

packages/whisper/src/ggml.nu — whisper.cpp's LEGACY ggml container.

The ggml-*.bin files ggerganov ships are how whisper models actually move between machines, and one is fully self-contained: hyperparameters, the tokenizer's vocabulary and every tensor in a single file — no config.json, no tokenizer.json. This reads that container so whisper transcribe ggml-tiny.bin clip.wav just works.

It is a PARSER, not a runtime: the model still runs on this package's own kernels. (The original goal said "not ggml like whisper.cpp" — that was about the ggml LIBRARY; the file format is just bytes, and hostile-input rules apply: every length is checked against the mapping before use.)

Layout (little-endian, no alignment padding):

u32 magic 'ggml' 0x67676d6c 11 × i32 hparams n_vocab, n_audio_ctx, n_audio_state, n_audio_head, n_audio_layer, n_text_ctx, n_text_state, n_text_head, n_text_layer, n_mels, ftype i32 n_mel, i32 n_fft, f32[n_mel*n_fft] mel filters (unused here — ours are verified against HF's) i32 n_vocab_in_file, then per word: i32 len, bytes (RAW bytes, not the HF byte-encoding) tensors until EOF: i32 n_dims, i32 name_len, i32 ttype, i32 dims[n_dims], name bytes, data

Two translations happen at load so the REST of the pipeline stays exactly the HF pipeline:

• tensor names: whisper.cpp keeps OpenAI's names (encoder.blocks.0.attn.

query.weight); the model layer asks in HF names (model.encoder.layers. 0.self_attn.q_proj.weight). gg_find translates.

• the tokenizer: the file carries only the 50 257 base words, as raw

bytes. The special tokens have no text at all — whisper.cpp derives their ids POSITIONALLY. We synthesize the HF spellings (<|startoftranscript|>, <|fi|>, <|0.00|> …) at those positions and byte-encode the base words, so name-based control-token lookup, the timestamp rules and tok_decode work unchanged.

API

: Gg

: Gg {
    ( Vec u ) data  // the whole file, owned by this handle
    i nbytes
    i n_vocab
    i n_audio_ctx
    i n_audio_state
    i n_audio_head
    i n_audio_layer
    i n_text_ctx
    i n_text_state
    i n_text_head
    i n_text_layer
    i n_mels
    ( Vec String ) vocab  // the raw base words, in id order
    ( Vec String ) tnames  // tensor names (ggml spelling)
    ( Vec i ) ttypes  // 0 = f32, 1 = f16
    ( Vec i ) tnelems
    ( Vec i ) toffs  // absolute byte offset of the tensor's data
}

@ gg_close * Gg g → v

@ gg_open s path → !*Gg String

@ gg_map_name s hf → String

Translate an HF tensor name to whisper.cpp's OpenAI spelling.

@ gg_find * Gg g s hf → i

Find a tensor by its HF name. -1 when absent.

@ gg_ttype * Gg g i ti → i

@ gg_nelems * Gg g i ti → i

@ gg_ptr * Gg g i ti → *u

@ gg_build_tok * Gg g → !*Tok String

Build the tokenizer: base words byte-encoded, specials synthesized at whisper.cpp's positional ids. Returns via tok_build so decode, control matching and the timestamp rules all behave exactly as with tokenizer.json.

---

kernels.nu

packages/whisper/src/kernels.nu — the encoder/decoder kernels.

CUDA-C sources compiled by packages/gpu, which runs the SAME text on the CUDA backend (via NVRTC) and on the CPU/OpenMP backend (via a compat shim). No shared, no __syncthreads: whatever runs on the GPU runs identically on the host, and the test suite requires the two to agree.

Whisper is not the llama shape, and every kernel here is a place it differs:

• LAYERNORM, not RMSNorm — it subtracts the mean and it has a BIAS.
• GELU with the ERROR FUNCTION, not the tanh approximation. HF's

ACT2FN["gelu"] is the exact one; the tanh form is a different function and using it would be a quiet, small, everywhere-wrong bias.

• CONV1D: two of them in front of the encoder (stride 1 then stride 2),

which is what turns 3000 spectrogram frames into 1500 encoder positions.

• NON-CAUSAL attention. Everything in this ecosystem so far has masked the

future; the whisper encoder looks at the whole 30 seconds at once, and the decoder's cross-attention looks at all of it too.

Weights are f32 (safetensors), so there is no quantised matvec zoo here — one warp-per-row matvec, coalesced the way nurllama's is.

API

: WhKernels

: WhKernels {
    GpuKernel matvec  // y = W x (+ bias), one thread per row — portable
    GpuKernel matvec_w  // the same, one WARP per row (CUDA only)
    GpuKernel matvec_t  // one warp per row × EIGHT positions (CUDA only)
    GpuKernel matmul  // 64x64 tiles through shared memory (CUDA only)
    b warp
    GpuKernel layernorm  // one thread per row — portable
    GpuKernel layernorm_b  // one block per row, shared reduction (CUDA)
    GpuKernel gelu
    GpuKernel conv1d
    GpuKernel attn_f  // fused attention, online softmax (CUDA, hd = 64)
    GpuKernel attn_sc  // scores = q·k / sqrt(hd), NO causal mask
    GpuKernel attn_out  // softmax(scores) · v
    GpuKernel addv
    GpuKernel addrow
    GpuKernel scale
    GpuKernel cvt_f16  // f16/bf16 weights → f32, ON THE DEVICE
    GpuKernel cvt_bf16
    GpuKernel getrow  // one row of the embedding table → the activation
    GpuKernel setrow  // write a row INTO the KV cache
    b ok
}

@ wk_build Gpu g → WhKernels

@ wk_free WhKernels ks → v

@ wk_matvec WhKernels ks i wd i xd i bd i yd i rows i cols i batch i wt → v

y[batch][rows] = W[rows][cols] · x[batch][cols] + bias[rows]. bd may be 0. wt = 1 when W's elements are raw f16 halves (the checkpoint's own precision, kept on the device); 0 for f32. The kernels widen at the point of use — exactly, denormals included — so the arithmetic is f32 either way and the result is bit-identical; what changes is that the weight bytes on the device (and through the memory bus of a memory-bound matvec) are half.

@ wk_layernorm WhKernels ks i xd i wd i bd i yd i n f eps i batch → v

@ wk_gelu WhKernels ks i xd i n → v

@ wk_conv1d WhKernels ks i xd i wd i bd i yd i t_in i c_in i c_out i stride i pad → v

@ wk_attn WhKernels ks i qd i kd i vd i scd i outd i nh i hd i nq i nkey f qscale i causal → v

Attention. causal 0 = every query sees every key (the encoder, and the decoder's cross-attention); 1 = a query sees only keys at or before it (the decoder's self-attention).

@ wk_addv WhKernels ks i ad i bd i n → v

@ wk_addrow WhKernels ks i ad i vd i n i batch → v

@ wk_scale WhKernels ks i xd f s i n → v

@ wk_getrow WhKernels ks i table i yd i row i n i wt → v

@ wk_setrow WhKernels ks i cache i vd i row i n → v

@ wk_cvt WhKernels ks i srcd i dstd i n b half → v

half = 1 for f16, 0 for bf16.

---

main.nu

packages/whisper/src/main.nu — the CLI (encoder stage).

whisper transcribe <model-dir> <audio.wav> [--lang en] [--max N] whisper encode <config.json> <model.safetensors> <audio.wav> -o enc.f32

transcribe is the whole thing: WAV → log-mel → encoder → decoder → text. encode stops after the encoder and writes its 1500 × d_model states, which is what the test suite compares against HF's WhisperModel.encoder.

A model directory is what Hugging Face ships: config.json, model.safetensors and tokenizer.json, side by side.

API

@ _wh_path s dir s name → String

<dir>/<name>

@ wh_run * Whisper w * Tok t ( Vec f ) at16_in s lang i maxtok b use_vad b with_ts f nospeech ( Vec u ) out → b

Audio longer than 30 seconds is transcribed in 30-second WINDOWS: the encoder sees exactly 30 s (that is the length it was trained on and the length its positional embedding has), so a longer clip is split, and each window is decoded from a fresh prompt with its own KV cache. The transcription core, from samples to text: VAD (optional), the 30-second window loop, decode. TAKES OWNERSHIP of at16_in (16 kHz mono) — the VAD path replaces the buffer wholesale, so the caller's handle is dead either way, and this function frees whichever buffer survives.

This is the seam the server stands on: the CLI opens the model, runs this once and exits; whisper serve opens the model ONCE and runs this per request — the 1.5 GB read, the f16→f32 conversion and the kernel compile all happen before the first request instead of inside every one.

@ main → i