model.nu

packages/whisper/src/model.nu — the encoder.

Whisper's encoder turns a 30-second log-mel spectrogram (3000 frames × 80 mels) into 1500 vectors the decoder attends to. The shape, read out of the checkpoint's own tensor names rather than remembered:

conv1 (k=3, s=1, pad=1) 80 → d_model then GELU conv2 (k=3, s=2, pad=1) d_model → d_model then GELU 3000 → 1500 frames

• embed_positions (a stored tensor, not computed sinusoids)

N × [ LN → self-attn (NON-CAUSAL) → +residual LN → fc1 → GELU → fc2 → +residual ] LN

The corner that bites: k_proj HAS NO BIAS. q, v and out_proj all have one; k does not. Whisper is not alone in this — it is inherited from the original implementation — and a loader that adds a zero bias where none exists is fine, while one that expects a bias tensor and finds none will happily read whatever is next in the file.

API

: WhisperImpl

: WhisperImpl {
    Gpu g
    WhKernels ks
    b w_half  // matrix weights live on the device as raw f16 halves
    St st  // the safetensors file (HF checkpoint) — st_none in ggml mode
    Gg gg  // whisper.cpp's legacy ggml container — the null handle in HF mode
    // the load's upload queue: every tensor is allocated as it is met and
    // sent in ONE streamed batch (gpu_upload_batch) once the last one is
    // known; f16/bf16 tensors that must be widened wait in cvt_* for their
    // bytes to land, then a kernel widens each and its raw copy is freed
    ( Vec GpuCopy ) up_q
    // the weight arena: device memory is carved from 128 MB chunks (bufs
    // holds the chunks), so a model is ~40 allocations rather than 1,300 —
    // and ~40 frees when it is closed, which is what an idle unload pays
    i arena_cur  // the chunk being carved (0 = none yet)
    i arena_off
    i arena_cap
    // the raw arena: f16/bf16 sources waiting to be widened live here until
    // the widen has run, then the chunks go back (cvt_raw lists them)
    i raw_cur
    i raw_off
    i raw_cap
    ( Vec i ) cvt_raw  // raw chunks on the device (freed after the widen)
    ( Vec i ) cvt_rawsz
    ( Vec GpuBuffer ) cvt_rawo  // their owners: clearing this frees them
    ( Vec i ) cvt_src  // per tensor: where its halves landed
    ( Vec i ) cvt_dst  // the f32 buffer the model will use
    ( Vec i ) cvt_n  // element count
    ( Vec i ) cvt_f16  // 1 = f16 source, 0 = bf16
    i n_mels
    i d_model
    i n_head
    i head_dim
    i d_head  // the DECODER's head count — the same in every
    i d_head_dim  // whisper so far, but the config states both
    i n_enc_layer
    i n_dec_layer
    i n_ctx_enc  // 1500 encoder positions
    i n_vocab
    // encoder weights (device pointers)
    i conv1_w
    i conv1_b
    i conv2_w
    i conv2_b
    i enc_pos
    ( Vec i ) e_ln1_w
    ( Vec i ) e_ln1_b
    ( Vec i ) e_wq
    ( Vec i ) e_bq
    ( Vec i ) e_wk  // no bias — whisper's k_proj has none
    ( Vec i ) e_wv
    ( Vec i ) e_bv
    ( Vec i ) e_wo
    ( Vec i ) e_bo
    ( Vec i ) e_ln2_w
    ( Vec i ) e_ln2_b
    ( Vec i ) e_fc1_w
    ( Vec i ) e_fc1_b
    ( Vec i ) e_fc2_w
    ( Vec i ) e_fc2_b
    i e_lnf_w
    i e_lnf_b
    // decoder weights
    i tok_embd  // also the output projection — whisper ties them
    i dec_pos
    ( Vec i ) d_ln1_w
    ( Vec i ) d_ln1_b
    ( Vec i ) d_wq
    ( Vec i ) d_bq
    ( Vec i ) d_wk  // no bias, like the encoder's
    ( Vec i ) d_wv
    ( Vec i ) d_bv
    ( Vec i ) d_wo
    ( Vec i ) d_bo
    ( Vec i ) d_lnx_w  // cross-attention's norm
    ( Vec i ) d_lnx_b
    ( Vec i ) x_wq
    ( Vec i ) x_bq
    ( Vec i ) x_wk  // no bias
    ( Vec i ) x_wv
    ( Vec i ) x_bv
    ( Vec i ) x_wo
    ( Vec i ) x_bo
    ( Vec i ) d_ln2_w
    ( Vec i ) d_ln2_b
    ( Vec i ) d_fc1_w
    ( Vec i ) d_fc1_b
    ( Vec i ) d_fc2_w
    ( Vec i ) d_fc2_b
    i d_lnf_w
    i d_lnf_b
    // decoder state
    ( Vec i ) kcache  // per layer: n_ctx_dec × d_model
    ( Vec i ) vcache
    ( Vec i ) xk  // per layer: the encoder's K, computed ONCE
    ( Vec i ) xv
    i n_ctx_dec
    i dx_d  // one token's activation
    i dxn_d
    i dq_d
    i dk_d
    i dv_d
    i dao_d
    i dtmp_d
    i dff_d
    i dsc_d  // decoder attention scratch: the composed path's score row, or
    // the one-query fused path's per-chunk partials
    i logits_d
    i argmax_d
    b oom  // a device allocation came back empty — see __wh_scratch
    GpuHost logits_host
    // scratch
    i mel_d  // 3000 × n_mels
    i c1_d  // 3000 × d_model
    i c2_d  // 1500 × d_model
    i xn_d
    i q_d
    i k_d
    i v_d
    i ao_d
    i tmp_d
    i ff_d  // 1500 × 4·d_model
    i sc_d  // n_head × 1500 × 1500 scores
    i enc_out  // 1500 × d_model — what the decoder attends to
    ( Vec i ) bufs  // every device allocation
    ( Vec i ) bufsz
    ( Vec GpuBuffer ) bufo  // their owners: released with the model
}

: Whisper { s ctl }

A Whisper is a handle on its state in an rcbox (stdlib/core/rcbox.nu): every copy is the same model, and the last owner releases it — every device allocation, the kernels, the context, the weight file. wh_close is an optional early release.

@ Whisper_share Whisper h → Whisper

@ Whisper_drop sink Whisper h → v

@ wh_gpu Whisper h → Gpu

The model's shape and device, for callers that size inputs or bind threads.

@ wh_n_mels Whisper h → i

@ wh_d_model Whisper h → i

@ wh_n_ctx_enc Whisper h → i

@ wh_n_enc_layer Whisper h → i

@ wh_n_dec_layer Whisper h → i

@ wh_gg Whisper h → Gg

the ggml container the model came from (the null handle in HF mode)

@ wh_open s config_path s weights_path → !Whisper String

@ wh_open_ggml s path → !Whisper String

Open a whisper.cpp ggml container: hyperparameters and tokenizer live in the same file as the weights, so the only argument is the file.

@ wh_close sink Whisper w → v

Early release (optional): the last owner gives back every device allocation, the kernels, the context and the weight file.

@ wh_encode Whisper w__h ( Vec f ) mel → v

Run the encoder over a 30-second log-mel spectrogram: mel is 3000 × n_mels, row-major (a frame is contiguous) — exactly what packages/audio produces. Leaves the 1500 encoder states on the device (wh_enc_out).

@ wh_enc_out Whisper w__h → ( Vec f )

The encoder states, back on the host: 1500 × d_model, row-major.

@ wh_prepare_cross Whisper w__h → v

Cross-attention's K and V come from the ENCODER, and the encoder does not change while a clip is being transcribed. So they are computed ONCE here, after wh_encode — not once per generated token, which would redo 1500×d_model of work for every word.

@ wh_decode_step Whisper w__h i tok i pos → v

One decoder step: token tok at position pos, with everything before it already in the KV cache. Leaves the logits on the device.

The self-attention needs no causal mask: there is exactly ONE query, at the end, and the cache holds only what came before it. Masking is what you do when you process several positions at once.

@ wk_addv_row Whisper w__h i pos → v

x += embed_positions[pos]. The row lives inside a matrix, so it is added by pointing at the row rather than by a broadcast.

@ wh_logits Whisper w__h → ( Vec f )

The logits of the last step, on the host.

@ wh_argmax_dev Whisper w__h → i

The greedy pick, without fetching the logits: the reduction runs on the device and one integer comes back. Ties go to the lower index in the kernel exactly as they do in wh_argmax below, so the token stream is the same one.

@ wh_argmax ( Vec f ) v → i

argmax — greedy decoding is what whisper does by default, and what the reference implementations are compared at.

---

serve.nu

packages/whisper/src/serve.nu — whisper as a service.

whisper serve <model-dir> --addr 0.0.0.0:6543 [--lang en] [--vad] [--timestamps]

The point of a server is WHEN the model loads. The CLI pays the whole bill per invocation — read 1.5 GB of safetensors, convert f16 to f32 on the device, compile the kernels — and only then hears the audio. Here all of that happens once, before the port opens; a request pays for its own audio and nothing else.

The HTTP surface is whisper.cpp's, so its clients work unchanged:

POST /inference multipart/form-data with a file field (WAV), plus optional language and response_format (json|text) fields → {"text":"..."} or the bare text. A raw WAV body (no multipart) is also accepted — curl --data-binary @clip.wav is not a browser form, and there is no reason to make it pretend to be one. GET /health {"status":"ok", model shape, request count} GET / the same (a browser poking the port should learn something, not get a 404)

Per-request vad and timestamps fields override the server-side flags, so one server can serve both a subtitle pipeline and a bare-text one.

Handlers are top-level functions reading module globals, not closures over locals — the same idiom as nurllama's api.nu, and for the same reason: closure environments are manual in NURL, and a server's handlers live for the process.

API

: ~ i g_srv_w 0 // the loaded Whisper's ctl word, OWNED by the server (0 = none loaded)

: ~ i g_srv_t 0 // the Tok's box, lent by __wh_serve_run's caller

: ~ i g_srv_reqs 0

: ~ s g_srv_lang ``

: ~ b g_srv_vad F

: ~ b g_srv_ts F

: ~ i g_srv_max 0

: ~ f g_srv_nospeech 0.6

: ~ s g_srv_token // empty = open server (loopback default)

: ~ s g_srv_dir // the model argument, kept for the reloads

── the model as a lease: loaded on demand, released when idle ──────

--unload-after N makes the model a thing the server HOLDS rather than IS. A reaper thread watches the clock; N seconds after the last request ends, with nothing in flight, it closes the model — device buffers, kernels, the CUDA context, the tokenizer stays (a few MB) — and the next request reloads it before it runs. What the server keeps while idle is the port and about a hundred megabytes of process; what it gives back is everything the model was (large-v3: 4.8 GB of device memory).

Who may touch g_srv_w: a request holds g_srv_busy from acquire to release, a WebSocket stream for the life of the connection, and the reaper only ever closes a model nobody holds. One mutex over the three counters; nothing waits on it for longer than a load. It sits in a WhSync block the server allocates once and keeps for the process (the reaper outlives any one scope, as the handlers do), reached through one module global.

: ~ i g_srv_unload_ms 0 // 0 = the model stays for the process's life

: ~ i g_srv_idle_since 0 // monotonic_ns when the last holder let go

: ~ i g_srv_busy 0 // requests + streams holding the model right now

: ~ i g_srv_sync 0 // *WhSync as an address (0 = never served)

: ~ i g_srv_loads 0 // (re)loads so far — 1 is the load before the port opened

: ~ i g_srv_unloads 0

: ~ i g_srv_load_ms 0 // the last load's wall time

: WhSync

: WhSync {
    Mutex m  // guards g_srv_w, g_srv_busy, the idle clock and the load counters
}

: ~ i g_ws_f32 0 // 1 = client sends float32 frames, else PCM16

: ~ s g_ws_lang // per-connection language override ( = server default)

@ wh_serve s dir s host i port s lang i maxtok b use_vad b with_ts s cert s key s token i unload_s → i

Load the model, open the port, serve until stopped.

---

ggml.nu

packages/whisper/src/ggml.nu — whisper.cpp's LEGACY ggml container.

The ggml-*.bin files ggerganov ships are how whisper models actually move between machines, and one is fully self-contained: hyperparameters, the tokenizer's vocabulary and every tensor in a single file — no config.json, no tokenizer.json. This reads that container so whisper transcribe ggml-tiny.bin clip.wav just works.

It is a PARSER, not a runtime: the model still runs on this package's own kernels. (The original goal said "not ggml like whisper.cpp" — that was about the ggml LIBRARY; the file format is just bytes, and hostile-input rules apply: every length is checked against the mapping before use.)

Layout (little-endian, no alignment padding):

u32 magic 'ggml' 0x67676d6c 11 × i32 hparams n_vocab, n_audio_ctx, n_audio_state, n_audio_head, n_audio_layer, n_text_ctx, n_text_state, n_text_head, n_text_layer, n_mels, ftype i32 n_mel, i32 n_fft, f32[n_mel*n_fft] mel filters (unused here — ours are verified against HF's) i32 n_vocab_in_file, then per word: i32 len, bytes (RAW bytes, not the HF byte-encoding) tensors until EOF: i32 n_dims, i32 name_len, i32 ttype, i32 dims[n_dims], name bytes, data

Two translations happen at load so the REST of the pipeline stays exactly the HF pipeline:

• tensor names: whisper.cpp keeps OpenAI's names (encoder.blocks.0.attn.

query.weight); the model layer asks in HF names (model.encoder.layers. 0.self_attn.q_proj.weight). gg_find translates.

• the tokenizer: the file carries only the 50 257 base words, as raw

bytes. The special tokens have no text at all — whisper.cpp derives their ids POSITIONALLY. We synthesize the HF spellings (<|startoftranscript|>, <|fi|>, <|0.00|> …) at those positions and byte-encode the base words, so name-based control-token lookup, the timestamp rules and tok_decode work unchanged.

API

: GgImpl

: GgImpl {
    // The whole file, as bytes: an mmap where the platform has one (the
    // page cache IS the copy, and a model that is already cached maps in
    // microseconds), the read fallback's Vec elsewhere. `gg_release_data`
    // gives it back once the weights are on the device — nothing reads
    // the file after that, and 3 GB of a large model is not a thing to
    // keep around for the vocabulary's sake.
    * u map
    i nbytes
    b from_mmap
    ( Vec u ) data  // the read fallback's storage — empty when mapped
    i n_vocab
    i n_audio_ctx
    i n_audio_state
    i n_audio_head
    i n_audio_layer
    i n_text_ctx
    i n_text_state
    i n_text_head
    i n_text_layer
    i n_mels
    ( Vec String ) vocab  // the raw base words, in id order
    ( Vec String ) tnames  // tensor names (ggml spelling)
    ( Vec i ) ttypes  // 0 = f32, 1 = f16
    ( Vec i ) tnelems
    ( Vec i ) toffs  // absolute byte offset of the tensor's data
}

% Drop GgImpl

% Drop GgImpl { @ drop GgImpl x → v {
        ? & . x from_mmap != # i . x map 0 { : i32 _u ( munmap . x map . x nbytes ) } {}
    } }

The mapping is raw: the last owner unmaps it (gg_release_data may have done so already). The Vecs are dropped by the compiler after this.

: Gg { s ctl }

A Gg is a handle on its state in an rcbox (stdlib/core/rcbox.nu): every copy is the same container, and the last owner releases it — the mapping (or the read buffer) and the parsed tables. gg_close is an optional early release.

@ Gg_share Gg h → Gg

@ Gg_drop sink Gg h → v

@ _Gg_ptr Gg h → *GgImpl

@ gg_none → Gg

The handle that holds no container (a model loaded from safetensors).

@ gg_is_open Gg h → b

@ gg_release_data Gg g__h → v

Give the file back — the mapping or the read buffer — and keep the parsed metadata (vocabulary, tensor table). After this gg_ptr must not be called; the loader calls it exactly once, right after the last upload.

@ gg_close sink Gg g → v

Early release (optional): the last owner gives everything back.

@ gg_open s path → !Gg String

@ gg_map_name s hf → String

Translate an HF tensor name to whisper.cpp's OpenAI spelling.

@ gg_find Gg g__h s hf → i

Find a tensor by its HF name. -1 when absent.

@ gg_ttype Gg g__h i ti → i

@ gg_nelems Gg g__h i ti → i

@ gg_ptr Gg g__h i ti → *u

@ gg_build_tok Gg g__h → !Tok String

Build the tokenizer: base words byte-encoded, specials synthesized at whisper.cpp's positional ids. Returns via tok_build so decode, control matching and the timestamp rules all behave exactly as with tokenizer.json.

---

kernels.nu

packages/whisper/src/kernels.nu — the encoder/decoder kernels.

CUDA-C sources compiled by packages/gpu, which runs the SAME text on the CUDA backend (via NVRTC) and on the CPU/OpenMP backend (via a compat shim). No shared, no __syncthreads: whatever runs on the GPU runs identically on the host, and the test suite requires the two to agree.

Whisper is not the llama shape, and every kernel here is a place it differs:

• LAYERNORM, not RMSNorm — it subtracts the mean and it has a BIAS.
• GELU with the ERROR FUNCTION, not the tanh approximation. HF's

ACT2FN["gelu"] is the exact one; the tanh form is a different function and using it would be a quiet, small, everywhere-wrong bias.

• CONV1D: two of them in front of the encoder (stride 1 then stride 2),

which is what turns 3000 spectrogram frames into 1500 encoder positions.

• NON-CAUSAL attention. Everything in this ecosystem so far has masked the

future; the whisper encoder looks at the whole 30 seconds at once, and the decoder's cross-attention looks at all of it too.

Weights are f32 (safetensors), so there is no quantised matvec zoo here — one warp-per-row matvec, coalesced the way nurllama's is.

API

: WhKernels

: WhKernels {
    GpuKernel matvec  // y = W x (+ bias), one thread per row — portable
    GpuKernel matvec_w  // the same, one WARP per row (CUDA only)
    GpuKernel matvec_t  // one warp per row × EIGHT positions (CUDA only)
    GpuKernel matmul  // 64x64 tiles through shared memory (CUDA only)
    b warp
    GpuKernel layernorm  // one thread per row — portable
    GpuKernel layernorm_b  // one block per row, shared reduction (CUDA)
    GpuKernel gelu
    GpuKernel conv1d
    GpuKernel attn_f  // fused attention, online softmax (CUDA, hd = 64)
    GpuKernel attn_d  // the same, shaped for ONE query (CUDA, hd = 64)
    GpuKernel attn_dm  // and the merge of its per-chunk partials
    GpuKernel attn_sc  // scores = q·k / sqrt(hd), NO causal mask
    GpuKernel attn_out  // softmax(scores) · v
    GpuKernel addv
    GpuKernel addrow
    GpuKernel scale
    GpuKernel cvt_f16  // f16/bf16 weights → f32, ON THE DEVICE
    GpuKernel cvt_bf16
    GpuKernel getrow  // one row of the embedding table → the activation
    GpuKernel setrow  // write a row INTO the KV cache
    GpuKernel argmaxk  // greedy decoding's argmax, on the device
    b ok
}

@ wk_build Gpu g → WhKernels

@ wk_matvec WhKernels ks i wd i xd i bd i yd i rows i cols i batch i wt → v

y[batch][rows] = W[rows][cols] · x[batch][cols] + bias[rows]. bd may be 0. wt = 1 when W's elements are raw f16 halves (the checkpoint's own precision, kept on the device); 0 for f32. The kernels widen at the point of use — exactly, denormals included — so the arithmetic is f32 either way and the result is bit-identical; what changes is that the weight bytes on the device (and through the memory bus of a memory-bound matvec) are half.

@ wk_layernorm WhKernels ks i xd i wd i bd i yd i n f eps i batch → v

@ wk_gelu WhKernels ks i xd i n → v

@ wk_conv1d WhKernels ks i xd i wd i bd i yd i t_in i c_in i c_out i stride i pad → v

@ wk_attn WhKernels ks i qd i kd i vd i scd i outd i nh i hd i nq i nkey f qscale i causal → v

Attention. causal 0 = every query sees every key (the encoder, and the decoder's cross-attention); 1 = a query sees only keys at or before it (the decoder's self-attention).

@ wk_addv WhKernels ks i ad i bd i n → v

@ wk_addrow WhKernels ks i ad i vd i n i batch → v

@ wk_scale WhKernels ks i xd f s i n → v

@ wk_getrow WhKernels ks i table i yd i row i n i wt → v

@ wk_setrow WhKernels ks i cache i vd i row i n → v

@ wk_argmax WhKernels ks i xd i n i outd → v

half = 1 for f16, 0 for bf16. The index of the largest logit, written as one int to outd.

@ wk_cvt WhKernels ks i srcd i dstd i n b half → v

---

run.nu

packages/whisper/src/run.nu — the shared middle of the pipeline: what transcribe and serve both do once a model and a tokenizer are open.

whis_ggml / whpath which container a model argument names wh_run 16 kHz mono float samples → the words (with --vad's condensed timeline mapped back, and whisper's own timestamp tokens when asked)

The CLI (main.nu) and the server (serve.nu) import this rather than each other: every source module compiles on its own, which is what the publish gate checks.

API

@ _wh_is_ggml s model → b

Is model a whisper.cpp ggml container? Decided by the file's own first bytes ('lmgg' on disk), not the extension — a renamed file still works and a directory never opens as a file.

@ _wh_path s dir s name → String

<dir>/<name>

@ wh_wav16 sink Wav aw → ( Vec f )

Audio longer than 30 seconds is transcribed in 30-second WINDOWS: the encoder sees exactly 30 s (that is the length it was trained on and the length its positional embedding has), so a longer clip is split, and each window is decoded from a fresh prompt with its own KV cache. The transcription core, from samples to text: VAD (optional), the 30-second window loop, decode. TAKES OWNERSHIP of at16_in (16 kHz mono) — the VAD path replaces the buffer wholesale, so the caller's handle is dead either way, and this function frees whichever buffer survives.

This is the seam the server stands on: the CLI opens the model, runs this once and exits; whisper serve opens the model ONCE and runs this per request — the 1.5 GB read, the f16→f32 conversion and the kernel compile all happen before the first request instead of inside every one. A WAV as the 16 kHz mono samples the model hears. Takes the WAV: it and its mono mix are gone when this returns, so a long recording is not held three times over while the model runs (an hour at 16 kHz is 460 MB a copy).

@ wh_run Whisper w Tok t sink ( Vec f ) at16_in s lang i maxtok b use_vad b with_ts f nospeech ( Vec u ) out → b

---

main.nu

packages/whisper/src/main.nu — the CLI (encoder stage).

whisper transcribe <model> <audio.wav> [--lang en] [--max N] whisper encode <config.json> <model.safetensors> <audio.wav> -o enc.f32

transcribe is the whole thing: WAV → log-mel → encoder → decoder → text. encode stops after the encoder and writes its 1500 × d_model states, which is what the test suite compares against HF's WhisperModel.encoder.

<model> is a local directory (config.json, model.safetensors, tokenizer.json side by side), a whisper.cpp ggml file, OR a Hugging Face ref (e.g. openai/whisper-base) — a ref is fetched into ~/.nurl/models via the hub package and the downloaded path is used.

API

@ main → i