model.nu

packages/whisper/src/model.nu — the encoder.

Whisper's encoder turns a 30-second log-mel spectrogram (3000 frames × 80 mels) into 1500 vectors the decoder attends to. The shape, read out of the checkpoint's own tensor names rather than remembered:

conv1 (k=3, s=1, pad=1) 80 → d_model then GELU conv2 (k=3, s=2, pad=1) d_model → d_model then GELU 3000 → 1500 frames

• embed_positions (a stored tensor, not computed sinusoids)

N × [ LN → self-attn (NON-CAUSAL) → +residual LN → fc1 → GELU → fc2 → +residual ] LN

The corner that bites: k_proj HAS NO BIAS. q, v and out_proj all have one; k does not. Whisper is not alone in this — it is inherited from the original implementation — and a loader that adds a zero bias where none exists is fine, while one that expects a bias tensor and finds none will happily read whatever is next in the file.

API

: Whisper

: Whisper {
    Gpu g
    WhKernels ks
    i st  // *St, the safetensors file
    i n_mels
    i d_model
    i n_head
    i head_dim
    i d_head  // the DECODER's head count — the same in every
    i d_head_dim  // whisper so far, but the config states both
    i n_enc_layer
    i n_dec_layer
    i n_ctx_enc  // 1500 encoder positions
    i n_vocab
    // encoder weights (device pointers)
    i conv1_w
    i conv1_b
    i conv2_w
    i conv2_b
    i enc_pos
    ( Vec i ) e_ln1_w
    ( Vec i ) e_ln1_b
    ( Vec i ) e_wq
    ( Vec i ) e_bq
    ( Vec i ) e_wk  // no bias — whisper's k_proj has none
    ( Vec i ) e_wv
    ( Vec i ) e_bv
    ( Vec i ) e_wo
    ( Vec i ) e_bo
    ( Vec i ) e_ln2_w
    ( Vec i ) e_ln2_b
    ( Vec i ) e_fc1_w
    ( Vec i ) e_fc1_b
    ( Vec i ) e_fc2_w
    ( Vec i ) e_fc2_b
    i e_lnf_w
    i e_lnf_b
    // decoder weights
    i tok_embd  // also the output projection — whisper ties them
    i dec_pos
    ( Vec i ) d_ln1_w
    ( Vec i ) d_ln1_b
    ( Vec i ) d_wq
    ( Vec i ) d_bq
    ( Vec i ) d_wk  // no bias, like the encoder's
    ( Vec i ) d_wv
    ( Vec i ) d_bv
    ( Vec i ) d_wo
    ( Vec i ) d_bo
    ( Vec i ) d_lnx_w  // cross-attention's norm
    ( Vec i ) d_lnx_b
    ( Vec i ) x_wq
    ( Vec i ) x_bq
    ( Vec i ) x_wk  // no bias
    ( Vec i ) x_wv
    ( Vec i ) x_bv
    ( Vec i ) x_wo
    ( Vec i ) x_bo
    ( Vec i ) d_ln2_w
    ( Vec i ) d_ln2_b
    ( Vec i ) d_fc1_w
    ( Vec i ) d_fc1_b
    ( Vec i ) d_fc2_w
    ( Vec i ) d_fc2_b
    i d_lnf_w
    i d_lnf_b
    // decoder state
    ( Vec i ) kcache  // per layer: n_ctx_dec × d_model
    ( Vec i ) vcache
    ( Vec i ) xk  // per layer: the encoder's K, computed ONCE
    ( Vec i ) xv
    i n_ctx_dec
    i dx_d  // one token's activation
    i dxn_d
    i dq_d
    i dk_d
    i dv_d
    i dao_d
    i dtmp_d
    i dff_d
    i dsc_d  // scores: n_head × max(n_ctx_dec, n_ctx_enc)
    i logits_d
    * u logits_host
    // scratch
    i mel_d  // 3000 × n_mels
    i c1_d  // 3000 × d_model
    i c2_d  // 1500 × d_model
    i xn_d
    i q_d
    i k_d
    i v_d
    i ao_d
    i tmp_d
    i ff_d  // 1500 × 4·d_model
    i sc_d  // n_head × 1500 × 1500 scores
    i enc_out  // 1500 × d_model — what the decoder attends to
    ( Vec i ) bufs  // every device allocation, for the free
    ( Vec i ) bufsz
}

@ wh_open s config_path s weights_path → !*Whisper String

@ wh_close * Whisper w → v

@ wh_encode * Whisper w ( Vec f ) mel → v

Run the encoder over a 30-second log-mel spectrogram: mel is 3000 × n_mels, row-major (a frame is contiguous) — exactly what packages/audio produces. Leaves the 1500 encoder states on the device (wh_enc_out).

@ wh_enc_out * Whisper w → ( Vec f )

The encoder states, back on the host: 1500 × d_model, row-major.

@ wh_prepare_cross * Whisper w → v

Cross-attention's K and V come from the ENCODER, and the encoder does not change while a clip is being transcribed. So they are computed ONCE here, after wh_encode — not once per generated token, which would redo 1500×d_model of work for every word.

@ wh_decode_step * Whisper w i tok i pos → v

One decoder step: token tok at position pos, with everything before it already in the KV cache. Leaves the logits on the device.

The self-attention needs no causal mask: there is exactly ONE query, at the end, and the cache holds only what came before it. Masking is what you do when you process several positions at once.

@ wk_addv_row * Whisper w i pos → v

x += embed_positions[pos]. The row lives inside a matrix, so it is added by pointing at the row rather than by a broadcast.

@ wh_logits * Whisper w → ( Vec f )

The logits of the last step, on the host.

@ wh_argmax ( Vec f ) v → i

argmax — greedy decoding is what whisper does by default, and what the reference implementations are compared at.

---

serve.nu

packages/whisper/src/serve.nu — whisper as a service.

whisper serve <model-dir> --addr 0.0.0.0:6543 [--lang en] [--vad] [--timestamps]

The point of a server is WHEN the model loads. The CLI pays the whole bill per invocation — read 1.5 GB of safetensors, convert f16 to f32 on the device, compile the kernels — and only then hears the audio. Here all of that happens once, before the port opens; a request pays for its own audio and nothing else.

The HTTP surface is whisper.cpp's, so its clients work unchanged:

POST /inference multipart/form-data with a file field (WAV), plus optional language and response_format (json|text) fields → {"text":"..."} or the bare text. A raw WAV body (no multipart) is also accepted — curl --data-binary @clip.wav is not a browser form, and there is no reason to make it pretend to be one. GET /health {"status":"ok", model shape, request count} GET / the same (a browser poking the port should learn something, not get a 404)

Per-request vad and timestamps fields override the server-side flags, so one server can serve both a subtitle pipeline and a bare-text one.

Handlers are top-level functions reading module globals, not closures over locals — the same idiom as nurllama's api.nu, and for the same reason: closure environments are manual in NURL, and a server's handlers live for the process.

API

: ~ i g_srv_w 0 // *Whisper, as an address (0 = not serving)

: ~ i g_srv_t 0 // *Tok

: ~ i g_srv_reqs 0

: ~ s g_srv_lang ``

: ~ b g_srv_vad F

: ~ b g_srv_ts F

: ~ i g_srv_max 0

: ~ i g_ws_f32 0 // 1 = client sends float32 frames, else PCM16

: ~ s g_ws_lang // per-connection language override ( = server default)

@ wh_serve s dir s host i port s lang i maxtok b use_vad b with_ts → i

Load the model, open the port, serve until stopped.

---

kernels.nu

packages/whisper/src/kernels.nu — the encoder/decoder kernels.

CUDA-C sources compiled by packages/gpu, which runs the SAME text on the CUDA backend (via NVRTC) and on the CPU/OpenMP backend (via a compat shim). No shared, no __syncthreads: whatever runs on the GPU runs identically on the host, and the test suite requires the two to agree.

Whisper is not the llama shape, and every kernel here is a place it differs:

• LAYERNORM, not RMSNorm — it subtracts the mean and it has a BIAS.
• GELU with the ERROR FUNCTION, not the tanh approximation. HF's

ACT2FN["gelu"] is the exact one; the tanh form is a different function and using it would be a quiet, small, everywhere-wrong bias.

• CONV1D: two of them in front of the encoder (stride 1 then stride 2),

which is what turns 3000 spectrogram frames into 1500 encoder positions.

• NON-CAUSAL attention. Everything in this ecosystem so far has masked the

future; the whisper encoder looks at the whole 30 seconds at once, and the decoder's cross-attention looks at all of it too.

Weights are f32 (safetensors), so there is no quantised matvec zoo here — one warp-per-row matvec, coalesced the way nurllama's is.

API

: WhKernels

: WhKernels {
    GpuKernel matvec  // y = W x (+ bias), one thread per row — portable
    GpuKernel matvec_w  // the same, one WARP per row (CUDA only)
    GpuKernel matvec_t  // one warp per row × EIGHT positions (CUDA only)
    GpuKernel matmul  // 64x64 tiles through shared memory (CUDA only)
    b warp
    GpuKernel layernorm  // one thread per row — portable
    GpuKernel layernorm_b  // one block per row, shared reduction (CUDA)
    GpuKernel gelu
    GpuKernel conv1d
    GpuKernel attn_f  // fused attention, online softmax (CUDA, hd = 64)
    GpuKernel attn_sc  // scores = q·k / sqrt(hd), NO causal mask
    GpuKernel attn_out  // softmax(scores) · v
    GpuKernel addv
    GpuKernel addrow
    GpuKernel scale
    GpuKernel cvt_f16  // f16/bf16 weights → f32, ON THE DEVICE
    GpuKernel cvt_bf16
    GpuKernel getrow  // one row of the embedding table → the activation
    GpuKernel setrow  // write a row INTO the KV cache
    b ok
}

@ wk_build Gpu g → WhKernels

@ wk_free WhKernels ks → v

@ wk_matvec WhKernels ks i wd i xd i bd i yd i rows i cols i batch → v

y[batch][rows] = W[rows][cols] · x[batch][cols] + bias[rows]. bd may be 0.

@ wk_layernorm WhKernels ks i xd i wd i bd i yd i n f eps i batch → v

@ wk_gelu WhKernels ks i xd i n → v

@ wk_conv1d WhKernels ks i xd i wd i bd i yd i t_in i c_in i c_out i stride i pad → v

@ wk_attn WhKernels ks i qd i kd i vd i scd i outd i nh i hd i nq i nkey f qscale i causal → v

Attention. causal 0 = every query sees every key (the encoder, and the decoder's cross-attention); 1 = a query sees only keys at or before it (the decoder's self-attention).

@ wk_addv WhKernels ks i ad i bd i n → v

@ wk_addrow WhKernels ks i ad i vd i n i batch → v

@ wk_scale WhKernels ks i xd f s i n → v

@ wk_getrow WhKernels ks i table i yd i row i n → v

@ wk_setrow WhKernels ks i cache i vd i row i n → v

@ wk_cvt WhKernels ks i srcd i dstd i n b half → v

half = 1 for f16, 0 for bf16.

---

main.nu

packages/whisper/src/main.nu — the CLI (encoder stage).

whisper transcribe <model-dir> <audio.wav> [--lang en] [--max N] whisper encode <config.json> <model.safetensors> <audio.wav> -o enc.f32

transcribe is the whole thing: WAV → log-mel → encoder → decoder → text. encode stops after the encoder and writes its 1500 × d_model states, which is what the test suite compares against HF's WhisperModel.encoder.

A model directory is what Hugging Face ships: config.json, model.safetensors and tokenizer.json, side by side.

API

@ wh_run * Whisper w * Tok t ( Vec f ) at16_in s lang i maxtok b use_vad b with_ts ( Vec u ) out → b

Audio longer than 30 seconds is transcribed in 30-second WINDOWS: the encoder sees exactly 30 s (that is the length it was trained on and the length its positional embedding has), so a longer clip is split, and each window is decoded from a fresh prompt with its own KV cache. The transcription core, from samples to text: VAD (optional), the 30-second window loop, decode. TAKES OWNERSHIP of at16_in (16 kHz mono) — the VAD path replaces the buffer wholesale, so the caller's handle is dead either way, and this function frees whichever buffer survives.

This is the seam the server stands on: the CLI opens the model, runs this once and exits; whisper serve opens the model ONCE and runs this per request — the 1.5 GB read, the f16→f32 conversion and the kernel compile all happen before the first request instead of inside every one.

@ main → i