mel.nu

packages/audio/src/mel.nu — STFT and the mel filter bank.

Every constant here was read out of transformers' audio_utils.py and feature_extraction_whisper.py, not remembered. The details that look like details are the ones that decide whether a speech model hears anything:

• the Hann window is PERIODIC — hanning(N+1) with the last sample dropped,

i.e. 0.5·(1 − cos(2πn/N)). The symmetric window (…/(N−1)) is a different window, and every frame would be subtly wrong.

• the signal is REFLECT-padded by n_fft/2 on both sides before framing

("center"), so frame k is centred on sample k·hop, not started there.

• the mel scale is SLANEY, not HTK: linear below 1 kHz (3f/200), log above.

HTK's 2595·log10(1+f/700) is the other convention, and it gives different filters.

• the filters are area-normalised, Slaney-style: 2/(f[k+2] − f[k]).
• whisper drops the LAST STFT frame, then floors the log at (max − 8) and

maps it to (x + 4)/4.

( mel_filters n_fft n_mels rate f_min f_max ) → ( Vec f ) (n_bins × n_mels) ( stft_power x plan window hop ) → ( Vec f ) frames × n_bins ( log_mel_whisper x n_fft hop n_mels rate ) → ( Vec f ) frames × n_mels

Data is row-major and flat: a spectrogram is one ( Vec f ), not a vec of vecs. It goes to a GPU next, and that is the layout a GPU wants.

API

: f MEL_PI 3.14159265358979323846

@ hz_to_mel_slaney f hz → f

Slaney mel: linear at 3f/200 below 1 kHz, logarithmic above.

@ mel_to_hz_slaney f mel → f

@ mel_filters i n_fft i n_mels i rate f f_min f f_max → ( Vec f )

The triangular filter bank, row-major (n_bins rows × n_mels columns) — the same orientation transformers uses, so a mel value is a dot product down a column.

@ hann_periodic i n → ( Vec f )

Periodic Hann: 0.5·(1 − cos(2πn/N)). NOT the symmetric one.

@ reflect_pad ( Vec f ) x i pad → ( Vec f )

Reflect-pad by pad samples on both sides: x[pad], x[pad-1], …, x[1] in front (the sample itself is not repeated) and the mirror image at the end. This is what "center=True" means, and it is why frame k is CENTRED on sample k·hop rather than starting there.

@ stft_power * FftPlan p ( Vec f ) x ( Vec f ) window i n_fft i hop → ( Vec f )

|STFT|², row-major frames × (n_fft/2+1). x is already padded; the caller owns the plan (an STFT runs the same length thousands of times).

@ log_mel_whisper ( Vec f ) x i n_fft i hop i n_mels i rate → ( Vec f )

The whisper log-mel spectrogram: exactly what WhisperFeatureExtractor produces, in the same layout (frames × n_mels, row-major).

@ pad_or_trim ( Vec f ) x i n → ( Vec f )

Pad with zeros (or trim) to exactly n samples — whisper always feeds its encoder 30 s, whatever the clip is.

---

main.nu

packages/audio/src/main.nu — the CLI.

audio info <file.wav> rate, channels, bits, duration audio mel <file.wav> -o mel.f32 whisper log-mel (80 × 3000), raw f32 LE audio resample <in.wav> <out.wav> --rate 16000 audio tone <out.wav> --rate 16000 a known signal, for tests audio vad <file.wav> where the speech is (and how much is not)

API

@ main → i

---

wav.nu

packages/audio/src/wav.nu — RIFF/WAVE, read and write.

The chunk layout is the whole format, and it is also the whole attack surface: every chunk announces its own size, and a file is free to lie about it. So the reader walks the chunks with a bounded cursor — a size that runs past the end of the file, a data chunk longer than the bytes that follow, a zero or absurd sample rate, a channel count of 0, a bit depth nobody has ever used: all of them are a clean error, none of them are an allocation sized by what the header claimed.

( wav_read path ) → !Wav String f32 samples, interleaved ( wav_read_mono_16k path ) → !( Vec f ) String what speech models want ( wav_write path samples rate ch ) → !v String 16-bit PCM ( wav_free w ) → v

Samples come back as f32 in [-1, 1], whatever the file stored them as: 8-bit unsigned, 16/24/32-bit signed PCM, or 32/64-bit IEEE float.

API

: i WAV_PCM 1

: i WAV_FLOAT 3

: i WAV_EXTENSIBLE 65534

: Wav

: Wav {
    ( Vec f ) samples  // interleaved
    i channels
    i rate
    i bits  // as stored in the file (informational)
}

@ wav_free Wav w → v

: WCur

: WCur {
    ( Vec u ) d
    i n
    i off
    b fail
}

A bounded cursor over the file bytes. Every read checks the budget against what is LEFT, never by forming off + k (which a hostile size would wrap).

@ wav_read s path → !Wav String

@ wav_mono Wav w → ( Vec f )

Average the channels. (Speech models want one channel; picking the left one throws away half of a stereo recording's information.)

@ wav_write s path ( Vec f ) samples i rate i channels → !v String

16-bit PCM, which every tool on earth reads.

---

resample.nu

packages/audio/src/resample.nu — sample-rate conversion.

Linear interpolation is the tempting one-liner, and it is wrong in a way that matters here: it is a lousy low-pass filter, so downsampling with it folds everything above the new Nyquist frequency back into the speech band as aliasing — exactly the frequencies a speech model is listening to.

So: windowed-sinc (Kaiser-ish, a Hann-windowed sinc), the standard honest answer. The kernel is band-limited to the LOWER of the two Nyquist frequencies, which is what makes downsampling safe.

( resample x from_rate to_rate ) → ( Vec f )

API

: f RS_PI 3.14159265358979323846

: i RS_ZEROS 16

Zeros are half the taps of a windowed sinc — 16 gives a stopband deep enough that the aliasing is far below the noise floor of any real recording.

@ resample ( Vec f ) x i from_rate i to_rate → ( Vec f )

---

vad.nu

packages/audio/src/vad.nu — voice activity detection.

Finding where the speech IS, so the parts that are not speech never reach a model. An hour of recording is mostly not speech, and a speech model does not get cheaper by running over silence — it costs exactly the same.

( vad_segments samples rate opts ) → ( Vec VadSeg ) [start, end) in samples ( vad_default_opts ) → VadOpts

This is an ENERGY detector with an adaptive noise floor, not a neural one. Being clear about that matters, because it decides where it fails: it hears "something loud enough, often enough" rather than "a human voice". It handles clean speech with a steady noise floor — a recording, a call, a meeting — and it will call a slammed door speech. faster-whisper uses Silero, a small neural VAD, for exactly that reason; this is the honest version of what can be done without a second model, and the seam is here for one.

What it is NOT is a fixed threshold. A fixed dB threshold works on the file it was tuned on and nowhere else. The floor here is the 10th percentile of the frame energies — whatever the quiet part of THIS recording sounds like — and speech is what stands a margin above it.

API

: VadSeg

: VadSeg {
    i start  // first sample
    i end  // one past the last
}

: VadOpts

: VadOpts {
    f margin_db  // how far above the noise floor speech has to be
    i min_speech_ms  // a burst shorter than this is not speech
    i min_silence_ms  // a gap shorter than this does not split a segment
    i pad_ms  // keep this much either side — speech starts before it
}  // gets loud, and a word's tail is quiet

@ vad_default_opts → VadOpts

Defaults that work on speech: 6 dB over the floor, 250 ms of it, gaps under 400 ms bridged (that is a pause between words, not between sentences), and 200 ms of padding kept either side so no consonant is clipped off.

@ vad_segments ( Vec f ) x i rate VadOpts o → ( Vec VadSeg )

@ vad_voiced_samples ( Vec VadSeg ) segs → i

How much of the audio the segments actually cover — what a caller saves.

: VadRun

: VadRun {
    i cond
    i orig
    i len
}

A run of the condensed audio and where it came from: condensed sample cond … cond+len is original sample orig … orig+len. What the condensation THREW AWAY is exactly what is not covered by any run — and it is why the runs exist at all: a model transcribing the condensed audio reports times in the CONDENSED timeline, and a caller who wants to say "this was said at 3:12 of the recording" has to walk the map back.

@ vad_extract_runs ( Vec f ) x ( Vec VadSeg ) segs i max_gap ( Vec VadRun ) runs → ( Vec f )

The audio with the silence taken out: the segments, with at most max_gap samples of the real audio kept between consecutive ones, and one VadRun appended to runs for every contiguous stretch that survives.

The cap is not a detail. Glue two segments together with NOTHING between them and a listener — a speech model included — hears one utterance where there were two: whisper, handed two sentences back to back with no pause, will run them together and get the second one's words wrong. Keeping a fraction of a second of the actual room gives it the boundary it needs, and capping that fraction is what keeps the silence from being the cost again. max_gap = 0 concatenates.

@ vad_extract ( Vec f ) x ( Vec VadSeg ) segs i max_gap → ( Vec f )

@ vad_map_sample ( Vec VadRun ) runs i s → i

A condensed sample position, mapped back to the original recording. Positions past the end of a run but before the next (which cannot arise from a model reading the condensed audio, but can arise from a rounded-up timestamp) clamp to the end of the nearer run.