mel.nu

packages/audio/src/mel.nu — STFT and the mel filter bank.

Every constant here was read out of transformers' audio_utils.py and feature_extraction_whisper.py, not remembered. The details that look like details are the ones that decide whether a speech model hears anything:

• the Hann window is PERIODIC — hanning(N+1) with the last sample dropped,

i.e. 0.5·(1 − cos(2πn/N)). The symmetric window (…/(N−1)) is a different window, and every frame would be subtly wrong.

• the signal is REFLECT-padded by n_fft/2 on both sides before framing

("center"), so frame k is centred on sample k·hop, not started there.

• the mel scale is SLANEY, not HTK: linear below 1 kHz (3f/200), log above.

HTK's 2595·log10(1+f/700) is the other convention, and it gives different filters.

• the filters are area-normalised, Slaney-style: 2/(f[k+2] − f[k]).
• whisper drops the LAST STFT frame, then floors the log at (max − 8) and

maps it to (x + 4)/4.

( mel_filters n_fft n_mels rate f_min f_max ) → ( Vec f ) (n_bins × n_mels) ( stft_power x plan window hop ) → ( Vec f ) frames × n_bins ( log_mel_whisper x n_fft hop n_mels rate ) → ( Vec f ) frames × n_mels

Data is row-major and flat: a spectrogram is one ( Vec f ), not a vec of vecs. It goes to a GPU next, and that is the layout a GPU wants.

API

: f MEL_PI 3.14159265358979323846

@ hz_to_mel_slaney f hz → f

Slaney mel: linear at 3f/200 below 1 kHz, logarithmic above.

@ mel_to_hz_slaney f mel → f

@ mel_filters i n_fft i n_mels i rate f f_min f f_max → ( Vec f )

The triangular filter bank, row-major (n_bins rows × n_mels columns) — the same orientation transformers uses, so a mel value is a dot product down a column.

@ hann_periodic i n → ( Vec f )

Periodic Hann: 0.5·(1 − cos(2πn/N)). NOT the symmetric one.

@ reflect_pad ( Vec f ) x i pad → ( Vec f )

Reflect-pad by pad samples on both sides: x[pad], x[pad-1], …, x[1] in front (the sample itself is not repeated) and the mirror image at the end. This is what "center=True" means, and it is why frame k is CENTRED on sample k·hop rather than starting there.

@ stft_power * FftPlan p ( Vec f ) x ( Vec f ) window i n_fft i hop → ( Vec f )

|STFT|², row-major frames × (n_fft/2+1). x is already padded; the caller owns the plan (an STFT runs the same length thousands of times).

@ log_mel_whisper ( Vec f ) x i n_fft i hop i n_mels i rate → ( Vec f )

The whisper log-mel spectrogram: exactly what WhisperFeatureExtractor produces, in the same layout (frames × n_mels, row-major).

@ pad_or_trim ( Vec f ) x i n → ( Vec f )

Pad with zeros (or trim) to exactly n samples — whisper always feeds its encoder 30 s, whatever the clip is.

---

main.nu

packages/audio/src/main.nu — the CLI.

audio info <file.wav> rate, channels, bits, duration audio mel <file.wav> -o mel.f32 whisper log-mel (80 × 3000), raw f32 LE audio resample <in.wav> <out.wav> --rate 16000 audio tone <out.wav> --rate 16000 a known signal, for tests audio vad <file.wav> where the speech is (and how much is not)

API

@ main → i

---

wav.nu

packages/audio/src/wav.nu — RIFF/WAVE, read and write.

The chunk layout is the whole format, and it is also the whole attack surface: every chunk announces its own size, and a file is free to lie about it. So the reader walks the chunks with a bounded cursor — a size that runs past the end of the file, a data chunk longer than the bytes that follow, a zero or absurd sample rate, a channel count of 0, a bit depth nobody has ever used: all of them are a clean error, none of them are an allocation sized by what the header claimed.

( wav_read path ) → !Wav String f32 samples, interleaved ( wav_read_mono_16k path ) → !( Vec f ) String what speech models want ( wav_write path samples rate ch ) → !v String 16-bit PCM ( wav_free w ) → v

Samples come back as f32 in [-1, 1], whatever the file stored them as: 8-bit unsigned, 16/24/32-bit signed PCM, or 32/64-bit IEEE float.

API

: i WAV_PCM 1

: i WAV_FLOAT 3

: i WAV_EXTENSIBLE 65534

: Wav

: Wav {
    ( Vec f ) samples  // interleaved
    i channels
    i rate
    i bits  // as stored in the file (informational)
}

@ wav_free Wav w → v

: WCur

: WCur {
    ( Vec u ) d
    i n
    i off
    b fail
}

A bounded cursor over the file bytes. Every read checks the budget against what is LEFT, never by forming off + k (which a hostile size would wrap).

@ wav_read s path → !Wav String

@ wav_parse ( Vec u ) data → !Wav String

The same parser, from bytes already in hand — an HTTP server that just received a WAV upload has no path to give. Same hostile-input treatment: the bytes are a stranger's whichever way they arrived.

@ wav_mono Wav w → ( Vec f )

Average the channels. (Speech models want one channel; picking the left one throws away half of a stereo recording's information.)

@ wav_write s path ( Vec f ) samples i rate i channels → !v String

16-bit PCM, which every tool on earth reads.

---

resample.nu

packages/audio/src/resample.nu — sample-rate conversion.

Linear interpolation is the tempting one-liner, and it is wrong in a way that matters here: it is a lousy low-pass filter, so downsampling with it folds everything above the new Nyquist frequency back into the speech band as aliasing — exactly the frequencies a speech model is listening to.

So: windowed-sinc (Kaiser-ish, a Hann-windowed sinc), the standard honest answer. The kernel is band-limited to the LOWER of the two Nyquist frequencies, which is what makes downsampling safe.

( resample x from_rate to_rate ) → ( Vec f )

API

: f RS_PI 3.14159265358979323846

: i RS_ZEROS 16

Zeros are half the taps of a windowed sinc — 16 gives a stopband deep enough that the aliasing is far below the noise floor of any real recording.

@ resample ( Vec f ) x i from_rate i to_rate → ( Vec f )

---

vad.nu

packages/audio/src/vad.nu — voice activity detection.

Finding where the speech IS, so the parts that are not speech never reach a model. An hour of recording is mostly not speech, and a speech model does not get cheaper by running over silence — it costs exactly the same.

( vad_segments samples rate opts ) → ( Vec VadSeg ) [start, end) in samples ( vad_default_opts ) → VadOpts

This is an ENERGY detector with an adaptive noise floor, not a neural one. Being clear about that matters, because it decides where it fails: it hears "something loud enough, often enough" rather than "a human voice". It handles clean speech with a steady noise floor — a recording, a call, a meeting — and it will call a slammed door speech. faster-whisper uses Silero, a small neural VAD, for exactly that reason; this is the honest version of what can be done without a second model, and the seam is here for one.

What it is NOT is a fixed threshold. A fixed dB threshold works on the file it was tuned on and nowhere else. The floor here is the 10th percentile of the frame energies — whatever the quiet part of THIS recording sounds like — and speech is what stands a margin above it.

API

: VadSeg

: VadSeg {
    i start  // first sample
    i end  // one past the last
}

: VadOpts

: VadOpts {
    f margin_db  // how far above the noise floor speech has to be
    i min_speech_ms  // a burst shorter than this is not speech
    i min_silence_ms  // a gap shorter than this does not split a segment
    i pad_ms  // keep this much either side — speech starts before it
}  // gets loud, and a word's tail is quiet

@ vad_default_opts → VadOpts

Defaults that work on speech: 6 dB over the floor, 250 ms of it, gaps under 400 ms bridged (that is a pause between words, not between sentences), and 200 ms of padding kept either side so no consonant is clipped off.

@ vad_segments ( Vec f ) x i rate VadOpts o → ( Vec VadSeg )

@ vad_voiced_samples ( Vec VadSeg ) segs → i

How much of the audio the segments actually cover — what a caller saves.

: VadRun

: VadRun {
    i cond
    i orig
    i len
}

A run of the condensed audio and where it came from: condensed sample cond … cond+len is original sample orig … orig+len. What the condensation THREW AWAY is exactly what is not covered by any run — and it is why the runs exist at all: a model transcribing the condensed audio reports times in the CONDENSED timeline, and a caller who wants to say "this was said at 3:12 of the recording" has to walk the map back.

@ vad_extract_runs ( Vec f ) x ( Vec VadSeg ) segs i max_gap ( Vec VadRun ) runs → ( Vec f )

The audio with the silence taken out: the segments, with at most max_gap samples of the real audio kept between consecutive ones, and one VadRun appended to runs for every contiguous stretch that survives.

The cap is not a detail. Glue two segments together with NOTHING between them and a listener — a speech model included — hears one utterance where there were two: whisper, handed two sentences back to back with no pause, will run them together and get the second one's words wrong. Keeping a fraction of a second of the actual room gives it the boundary it needs, and capping that fraction is what keeps the silence from being the cost again. max_gap = 0 concatenates.

@ vad_extract ( Vec f ) x ( Vec VadSeg ) segs i max_gap → ( Vec f )

@ vad_map_sample ( Vec VadRun ) runs i s → i

A condensed sample position, mapped back to the original recording. Positions past the end of a run but before the next (which cannot arise from a model reading the condensed audio, but can arise from a rounded-up timestamp) clamp to the end of the nearer run.

: VadStream

: VadStream {
    ( Vec f ) buf  // unconsumed samples; buf[0] is absolute sample `base`
    i base
    i nframe  // frames processed so far (absolute)
    ( Vec f ) e  // trailing frame energies (compacted periodically)
    i e0  // absolute frame index of e[0]
    i rate
    i win
    i hop
    f margin
    i min_speech  // frames
    i min_sil  // frames
    i pad  // samples
    i force  // samples: force-close a run at this length
    i run_start  // absolute frame, -1 = none
    i last_voiced
    f floor
    i floor_at  // frame the floor was last recomputed at
    i seg_start  // pending closed segment (absolute samples), -1 = none
    i seg_end
}

── Streaming VAD ───────────────────────────────────────────────────

The batch detector above reads the whole recording and takes its noise floor from the whole recording's percentile. A LIVE stream has no whole recording — the floor has to come from what has been heard so far, and it has to keep adapting, because the room changes: someone turns a fan on at minute ten and a frozen floor calls the fan speech forever.

: *VadStream st ( vad_stream_new 16000 ( vad_default_opts ) ) ( vad_stream_push st samples ) feed audio as it arrives ~ ( vad_stream_poll st ) { a segment CLOSED — : VadSeg g ( vad_stream_seg st ) where it sits (absolute samples) : ( Vec f ) x ( vad_stream_take st ) its audio (pads included) ...transcribe x... } ( vad_stream_flush st ) end of stream: close an open run

Same physics as the batch detector — 30 ms window / 10 ms hop, speech is what stands margin_db above the floor for min_speech_ms, gaps under min_silence_ms are bridged, pad_ms kept either side — with three streaming realities on top:

• the floor is the 10th percentile of the TRAILING minute of frame

energies, recomputed every second — adaptive in time, not only per file

• the first 300 ms are never speech: a floor estimated from three frames

would be noise calling noise speech

• a run that reaches 28 s closes FORCIBLY: whisper's window is 30 s, and

"the segment ends when the speaker pauses" is not a bound — a lecture has no obligation to pause

Memory is bounded by construction: silence is dropped as it is heard (nothing before the would-be pad of a future run is ever needed), so an hour of quiet room costs a minute of energies and a fraction of a second of samples — not an hour of either.

@ vad_stream_new i rate VadOpts o → *VadStream

@ vad_stream_free * VadStream st → v

@ vad_stream_push * VadStream st ( Vec f ) x → v

Feed samples. Frames are processed up to the data (or up to a pending segment — one is handed over at a time, and processing resumes after vad_stream_take).

@ vad_stream_poll * VadStream st → b

Is a closed segment waiting?

@ vad_stream_seg * VadStream st → VadSeg

Where the pending segment sits, in absolute samples since stream start.

@ vad_stream_take * VadStream st → ( Vec f )

The pending segment's audio. Clears the slot, releases what came before it, and resumes frame processing.

@ vad_stream_flush * VadStream st → b

End of stream: close an open run (if it was ever long enough to be speech). T = a segment is now pending.