NURL registrynurl-lang.org →packages/audio/src/mel.nu — STFT and the mel filter bank.
Every constant here was read out of transformers' audio_utils.py and feature_extraction_whisper.py, not remembered. The details that look like details are the ones that decide whether a speech model hears anything:
i.e. 0.5·(1 − cos(2πn/N)). The symmetric window (…/(N−1)) is a different window, and every frame would be subtly wrong.
("center"), so frame k is centred on sample k·hop, not started there.
HTK's 2595·log10(1+f/700) is the other convention, and it gives different filters.
maps it to (x + 4)/4.
( mel_filters n_fft n_mels rate f_min f_max ) → ( Vec f ) (n_bins × n_mels) ( stft_power x plan window hop ) → ( Vec f ) frames × n_bins ( log_mel_whisper x n_fft hop n_mels rate ) → ( Vec f ) frames × n_mels ( log_mel_whisper_padded x n_total n_fft hop n_mels rate ) → the same of x zero-padded to n_total, computed only where x is
Data is row-major and flat: a spectrogram is one ( Vec f ), not a vec of vecs. It goes to a GPU next, and that is the layout a GPU wants.
: f MEL_PI 3.14159265358979323846@ hz_to_mel_slaney f hz → fSlaney mel: linear at 3f/200 below 1 kHz, logarithmic above.
@ mel_to_hz_slaney f mel → f@ mel_filters i n_fft i n_mels i rate f f_min f f_max → ( Vec f )The triangular filter bank, row-major (n_bins rows × n_mels columns) — the same orientation transformers uses, so a mel value is a dot product down a column.
@ hann_periodic i n → ( Vec f )Periodic Hann: 0.5·(1 − cos(2πn/N)). NOT the symmetric one.
@ reflect_pad ( Vec f ) x i pad → ( Vec f )Reflect-pad by pad samples on both sides: x[pad], x[pad-1], …, x[1] in front (the sample itself is not repeated) and the mirror image at the end. This is what "center=True" means, and it is why frame k is CENTRED on sample k·hop rather than starting there.
@ stft_power FftPlan p ( Vec f ) x ( Vec f ) window i n_fft i hop → ( Vec f )|STFT|², row-major frames × (n_fft/2+1). x is already padded; the caller owns the plan (an STFT runs the same length thousands of times).
@ log_mel_whisper ( Vec f ) x i n_fft i hop i n_mels i rate → ( Vec f )The whisper log-mel spectrogram: exactly what WhisperFeatureExtractor produces, in the same layout (frames × n_mels, row-major).
@ log_mel_whisper_raw ( Vec f ) x i n_fft i hop i n_mels i rate i limit → ( Vec f )The log10 mel energies before whisper's clamp and scaling — at most limit frames of them (-1 = all), frames × n_mels.
@ log_mel_whisper_padded ( Vec f ) x i n_total i n_fft i hop i n_mels i rate → ( Vec f )log_mel_whisper of x zero-padded to n_total samples, without paying for the padding. Whisper always hears 30 s, and a two-second answer is 28 s of zeros whose frames are all the same number: a frame of zeros has power 0, its log is the 1e-10 floor, and only the max − 8 clamp (taken over the whole spectrogram) decides what that becomes. So the frames that see a sample of x are computed, the rest are the floor, and the clamp runs over all of them — the same floats as padding first and computing every frame, at the cost of the speech rather than of 30 s (2 s of audio: ~10 ms instead of ~120). x longer than n_total is trimmed, as pad_or_trim does.
@ pad_or_trim ( Vec f ) x i n → ( Vec f )Pad with zeros (or trim) to exactly n samples — whisper always feeds its encoder 30 s, whatever the clip is.
@ hz_to_mel_htk f hz → f@ mel_to_hz_htk f mel → f@ mel_filters_htk i n_fft i n_mels i rate f f_min f f_max → ( Vec f )torchaudio's melscale_fbanks(..., norm=None, mel_scale="htk"), row-major n_bins × n_mels — the same orientation mel_filters uses.
@ log_mel_vocos ( Vec f ) x i n_fft i hop i n_mels i rate → ( Vec f )log(clamp(mel(|STFT|), 1e-5)) — frames × n_mels, row-major. This is what an F5-TTS checkpoint reads and what a vocos vocoder writes back out.
packages/audio/src/main.nu — the CLI.
audio info <file.wav> rate, channels, bits, duration audio mel <file.wav> -o mel.f32 whisper log-mel (80 × 3000), raw f32 LE audio resample <in.wav> <out.wav> --rate 16000 audio tone <out.wav> --rate 16000 a known signal, for tests audio vad <file.wav> where the speech is (and how much is not) audio mp3 <in.wav> <out.mp3> --bitrate 128 MPEG-1/2 Layer III
@ main → ipackages/audio/src/istft.nu — the inverse short-time Fourier transform.
A neural vocoder of the vocos family does not predict a waveform. It predicts, per frame and per frequency bin, a magnitude and a phase — and the waveform is what comes back out of those. That step is this file.
Overlap-add is the easy half. The half that decides whether the result is the signal or a comb filter is the WINDOW ENVELOPE: each output sample is covered by several overlapping windows, by a different total weight depending on where it falls relative to the hop, and dividing by the overlap-added w² is what undoes that. Skip it and a steady tone comes back amplitude-modulated at the frame rate — audible as roughness, not as an obvious bug.
( istft_center re im n_fft hop frames ) → ( Vec f )
re and im are frames × (n_fft/2+1), row-major — the layout stft_power writes, and the layout a vocoder's head produces. "center" means the analysis was centred (torch's center=True), so n_fft/2 samples are trimmed from each end and the result is (frames−1)·hop samples long.
@ istft_center ( Vec f ) re ( Vec f ) im i n_fft i hop i frames → ( Vec f )packages/audio/src/mp3tab.nu — the constants MPEG-1/2 Layer III is made of.
None of this is a choice. Every number here is fixed by ISO/IEC 11172-3 and 13818-3, and every encoder and every decoder in the world carries the same ones: the Huffman code books of Table B.7, the scalefactor band boundaries of Table B.8 (MPEG-1) and B.2 (MPEG-2 and 2.5), and the 512-tap analysis window of Annex C. They live in their own file because they are data, not code, and because mp3.nu reads better without 400 lines of digits in the middle of it.
The Huffman books are stored as two flat arrays — one of code words, one of their lengths in bits — with a per-book starting offset, because the books are of wildly different sizes (4 entries for book 1, 256 for book 24) and a jagged array of 34 separate allocations would be slower to reach into and no clearer to read. A pair (x, y) is at off + x * ylen + y.
Books 16..23 all share one code table and differ only in linbits, the width of the escape field that carries values above 14; books 24..31 do the same with a second table. That sharing is in the standard, not a trick here.
@ _mp3_zeros i n → ( Vec i )@ _mp3_fzeros i n → ( Vec f )@ _mp3_pl8 ( Slice i ) p i at i a i b i c i d i e i f i g i h → iEight at a time: the tables below are long, and one call per value would make them longer still.
@ _mp3_pf8 ( Slice f ) p i at f v0 f v1 f v2 f v3 f v4 f v5 f v6 f v7 → i@ _mp3_hcode → ( Vec i )@ _mp3_hlen → ( Vec i )Code lengths in bits, in the same order as mp3hcode.
@ _mp3_hxlen → ( Vec i )Per-table x extent (max x index + 1).
@ _mp3_hylen → ( Vec i )Per-table y extent; the pair (x,y) indexes at x*ylen + y.
@ _mp3_hlinbits → ( Vec i )Escape-value width in bits, 0 for the tables without one.
@ _mp3_hlinmax → ( Vec i )Largest value an escape of that width can carry.
@ _mp3_hoff → ( Vec i )Where each table's data starts in the two flat arrays.
@ _mp3_sfb_index → ( Vec i )ISO Table B.8 (MPEG-1) and B.2 (MPEG-2/2.5): where each of the 21 scalefactor bands starts, per samplerate index, 23 entries each.
@ _mp3_enwindow → ( Vec f )The analysis window C[i] of ISO 11172-3 Annex C, in the order and with the signs the folding below expects. 512 coefficients, |C| <= 0.035781.
packages/audio/src/resample.nu — sample-rate conversion.
Linear interpolation is the tempting one-liner, and it is wrong in a way that matters here: it is a lousy low-pass filter, so downsampling with it folds everything above the new Nyquist frequency back into the speech band as aliasing — exactly the frequencies a speech model is listening to.
So: windowed-sinc (Kaiser-ish, a Hann-windowed sinc), the standard honest answer. The kernel is band-limited to the LOWER of the two Nyquist frequencies, which is what makes downsampling safe.
( resample x from_rate to_rate ) → ( Vec f )
: f RS_PI 3.14159265358979323846: i RS_ZEROS 16Zeros are half the taps of a windowed sinc — 16 gives a stopband deep enough that the aliasing is far below the noise floor of any real recording.
@ resample ( Vec f ) x i from_rate i to_rate → ( Vec f )packages/audio/src/vad.nu — voice activity detection.
Finding where the speech IS, so the parts that are not speech never reach a model. An hour of recording is mostly not speech, and a speech model does not get cheaper by running over silence — it costs exactly the same.
( vad_segments samples rate opts ) → ( Vec VadSeg ) [start, end) in samples ( vad_default_opts ) → VadOpts
This is an ENERGY detector with an adaptive noise floor, not a neural one. Being clear about that matters, because it decides where it fails: it hears "something loud enough, often enough" rather than "a human voice". It handles clean speech with a steady noise floor — a recording, a call, a meeting — and it will call a slammed door speech. faster-whisper uses Silero, a small neural VAD, for exactly that reason; this is the honest version of what can be done without a second model, and the seam is here for one.
What it is NOT is a fixed threshold. A fixed dB threshold works on the file it was tuned on and nowhere else. The floor here is the 10th percentile of the frame energies — whatever the quiet part of THIS recording sounds like — and speech is what stands a margin above it.
: VadSeg: VadSeg {
i start // first sample
i end // one past the last
}: VadOpts: VadOpts {
f margin_db // how far above the noise floor speech has to be
i min_speech_ms // a burst shorter than this is not speech
i min_silence_ms // a gap shorter than this does not split a segment
i pad_ms // keep this much either side — speech starts before it
} // gets loud, and a word's tail is quiet@ vad_default_opts → VadOptsDefaults that work on speech: 6 dB over the floor, 250 ms of it, gaps under 400 ms bridged (that is a pause between words, not between sentences), and 200 ms of padding kept either side so no consonant is clipped off.
@ vad_segments ( Vec f ) x i rate VadOpts o → ( Vec VadSeg )@ vad_voiced_samples ( Vec VadSeg ) segs → iHow much of the audio the segments actually cover — what a caller saves.
: VadRun: VadRun {
i cond
i orig
i len
}A run of the condensed audio and where it came from: condensed sample cond … cond+len is original sample orig … orig+len. What the condensation THREW AWAY is exactly what is not covered by any run — and it is why the runs exist at all: a model transcribing the condensed audio reports times in the CONDENSED timeline, and a caller who wants to say "this was said at 3:12 of the recording" has to walk the map back.
@ vad_extract_runs ( Vec f ) x ( Vec VadSeg ) segs i max_gap ( Vec VadRun ) runs → ( Vec f )The audio with the silence taken out: the segments, with at most max_gap samples of the real audio kept between consecutive ones, and one VadRun appended to runs for every contiguous stretch that survives.
The cap is not a detail. Glue two segments together with NOTHING between them and a listener — a speech model included — hears one utterance where there were two: whisper, handed two sentences back to back with no pause, will run them together and get the second one's words wrong. Keeping a fraction of a second of the actual room gives it the boundary it needs, and capping that fraction is what keeps the silence from being the cost again. max_gap = 0 concatenates.
@ vad_extract ( Vec f ) x ( Vec VadSeg ) segs i max_gap → ( Vec f )@ vad_map_sample ( Vec VadRun ) runs i s → iA condensed sample position, mapped back to the original recording. Positions past the end of a run but before the next (which cannot arise from a model reading the condensed audio, but can arise from a rounded-up timestamp) clamp to the end of the nearer run.
: VadStreamImpl: VadStreamImpl {
( Vec f ) buf // unconsumed samples; buf[0] is absolute sample `base`
i base
i nframe // frames processed so far (absolute)
( Vec f ) e // trailing frame energies (compacted periodically)
i e0 // absolute frame index of e[0]
i rate
i win
i hop
f margin
i min_speech // frames
i min_sil // frames
i pad // samples
i force // samples: force-close a run at this length
i run_start // absolute frame, -1 = none
i last_voiced
f floor
i floor_at // frame the floor was last recomputed at
i seg_start // pending closed segment (absolute samples), -1 = none
i seg_end
}── Streaming VAD ───────────────────────────────────────────────────
The batch detector above reads the whole recording and takes its noise floor from the whole recording's percentile. A LIVE stream has no whole recording — the floor has to come from what has been heard so far, and it has to keep adapting, because the room changes: someone turns a fan on at minute ten and a frozen floor calls the fan speech forever.
: VadStream st ( vad_stream_new 16000 ( vad_default_opts ) ) ( vad_stream_push st samples ) feed audio as it arrives ~ ( vad_stream_poll st ) { a segment CLOSED — : VadSeg g ( vad_stream_seg st ) where it sits (absolute samples) : ( Vec f ) x ( vad_stream_take st ) its audio (pads included) ...transcribe x... } ( vad_stream_flush st ) end of stream: close an open run
A VadStream is a handle: every copy is the same stream, and the last owner releases it (vad_stream_free is an optional early release).
Same physics as the batch detector — 30 ms window / 10 ms hop, speech is what stands margin_db above the floor for min_speech_ms, gaps under min_silence_ms are bridged, pad_ms kept either side — with three streaming realities on top:
energies, recomputed every second — adaptive in time, not only per file
would be noise calling noise speech
"the segment ends when the speaker pauses" is not a bound — a lecture has no obligation to pause
Memory is bounded by construction: silence is dropped as it is heard (nothing before the would-be pad of a future run is ever needed), so an hour of quiet room costs a minute of energies and a fraction of a second of samples — not an hour of either.
: VadStream { ( Locked VadStreamImpl ) l }A VadStream is a handle on its state in a Locked (stdlib/std/thread.nu): every copy is the same stream — one thread may push while another polls and takes — and the last owner releases it. Each call checks the state out for its length.
@ VadStream_share VadStream h → VadStream@ VadStream_drop sink VadStream h → v@ vad_stream_new i rate VadOpts o → VadStream@ vad_stream_free sink VadStream st → vLet go of st now rather than at the end of its owner's scope.
@ vad_stream_push VadStream st__h ( Vec f ) x → vFeed samples. Frames are processed up to the data (or up to a pending segment — one is handed over at a time, and processing resumes after vad_stream_take).
@ vad_stream_poll VadStream st__h → bIs a closed segment waiting?
@ vad_stream_seg VadStream st__h → VadSegWhere the pending segment sits, in absolute samples since stream start.
@ vad_stream_take VadStream st__h → ( Vec f )The pending segment's audio. Clears the slot, releases what came before it, and resumes frame processing.
@ vad_stream_flush VadStream st__h → bEnd of stream: close an open run (if it was ever long enough to be speech). T = a segment is now pending.
packages/audio/src/mp3.nu — MPEG-1/2/2.5 Layer III, encoding side.
An MP3 encoder in pure NURL, because a program that can only hand back WAV is a program that cannot answer output_format: "mp3", and shelling out to ffmpeg is not an answer — it moves the dependency, it does not remove it.
( mp3_encode samples rate channels bitrate ) → !( Vec u ) String
samples are f32 in [-1, 1], interleaved when channels is 2. rate must be one of the nine the format defines — 32/44.1/48 kHz (MPEG-1), 16/22.05/24 kHz (MPEG-2) or 8/11.025/12 kHz (MPEG-2.5) — and bitrate one the chosen version allows. Anything else is an error naming what was wrong, never a silently resampled guess.
WHAT IS AND IS NOT HERE. This is the full Layer III bitstream: the polyphase analysis filterbank, the 18-point MDCT with its alias-reduction butterflies, the quantiser's step-size search, Huffman coding over the big-values regions and the count1 quadruples, and a frame layout a decoder reads without knowing where it came from. What is NOT here is a psychoacoustic model. There is no masking threshold, so there are no scalefactors — every band gets the same step size, chosen only to fill the frame. That is the shape the fixed-point encoder shine proved is enough for speech and it is the honest description of the quality: the bits go where the signal is loud, not where the ear is deaf. For speech at 128 kbps the difference is not audible; for music at 64 it would be.
Long blocks only (block_type 0). Window switching exists to stop pre-echo on transients, and choosing when to switch is a psychoacoustic decision, so it belongs with the model that is not here rather than guessed at.
THE SCALE. Every value below is an ordinary f64, but the arithmetic mirrors a fixed-point encoder's exactly: a sample of 1.0 is what a decoder will call full scale, and the quantiser's 2^(-step/4) lands on the same integers. The three factors of 0.5 in the filterbank and the MDCT are what make that true — they are not a normalisation choice, they are the >>32 of the fixed-point original written out.
: i MP3_GRAN 576: i MP3_HAN 512: f MP3_PI 3.14159265358979323846: i MP3_SI_PART23 0One granule's side information. Each (granule, channel) owns ten of these slots in a flat array; the fields that only matter inside the quantiser's loop live on the encoder itself as c_*.
: i MP3_SI_BIGV 1: i MP3_SI_CNT1 2: i MP3_SI_GGAIN 3: i MP3_SI_TS0 4: i MP3_SI_TS1 5: i MP3_SI_TS2 6: i MP3_SI_R0 7: i MP3_SI_R1 8: i MP3_SI_C1T 9: i MP3_SI_N 10: Mp3: Mp3 {
i channels
i rate
i sri // samplerate index 0..8
i version // 3 = MPEG-1, 2 = MPEG-2, 0 = MPEG-2.5
i gpf // granules per frame: 2 for MPEG-1, 1 otherwise
i bri // bitrate index as it goes in the header
i sideinfo_len // bits
i whole_slots
f frac_slots
f slot_lag
i padding
i mean_bits
i resv_size
i resv_drain
// tables, built once
( Vec f ) enw // 512 analysis window coefficients
( Vec f ) fl // 32 x 64 filterbank
( Vec f ) cosl // 18 x 36 windowed MDCT basis
( Vec f ) ca // 8 alias-reduction coefficients
( Vec f ) cs // and their companions
( Vec i ) sfb // 9 x 23 scalefactor band starts
( Vec i ) hcode
( Vec i ) hlen
( Vec i ) hxlen
( Vec i ) hylen
( Vec i ) hlinbits
( Vec i ) hlinmax
( Vec i ) hoff
( Vec i ) sdv0 // region0 band count per big-values width
( Vec i ) sdv1 // region1 band count
( Vec i ) idx34 // x^(3/4) for x = 0..9999
// running state
( Vec f ) xbuf // channels x 512, the filterbank's memory
( Vec i ) xoff // where each channel's ring cursor sits
( Vec f ) sb // channels x (gpf+1) x 18 x 32 subband samples
( Vec f ) xr // channels x gpf x 576 spectral values
( Vec i ) ix // the same, quantised
( Vec i ) si // gpf x channels x 10
( Vec f ) yw // 64 scratch
( Vec f ) mdin // 36 scratch
// the granule being worked on
i c_bigv
i c_cnt1
i c_ts0
i c_ts1
i c_ts2
i c_r0
i c_r1
i c_c1t
i c_a1
i c_a2
i c_a3
i c_step
// output
( Vec u ) out
i cache
i cbits
}The encoder's state: a plain value on mp3_encode's stack, which every stage borrows inout; the compiler drops its tables when the encode returns, and the finished stream moves out.
@ mp3_encode ( Vec f ) samples i rate i channels i bitrate → !( Vec u ) StringSamples in, an MP3 file out. samples are f32 in [-1, 1] and interleaved when channels is 2; nothing is resampled and nothing is normalised, so what goes in is what a decoder gets back.
packages/audio/src/wav.nu — RIFF/WAVE, read and write.
The chunk layout is the whole format, and it is also the whole attack surface: every chunk announces its own size, and a file is free to lie about it. So the reader walks the chunks with a bounded cursor — a size that runs past the end of the file, a data chunk longer than the bytes that follow, a zero or absurd sample rate, a channel count of 0, a bit depth nobody has ever used: all of them are a clean error, none of them are an allocation sized by what the header claimed.
( wav_read path ) → !Wav String f32 samples, interleaved ( wav_read_mono_16k path ) → !( Vec f ) String what speech models want ( wav_write path samples rate ch ) → !v String 16-bit PCM ( wav_free w ) → v
Samples come back as f32 in [-1, 1], whatever the file stored them as: 8-bit unsigned, 16/24/32-bit signed PCM, or 32/64-bit IEEE float.
: i WAV_PCM 1: i WAV_FLOAT 3: i WAV_EXTENSIBLE 65534: Wav: Wav {
( Vec f ) samples // interleaved
i channels
i rate
i bits // as stored in the file (informational)
}@ wav_free sink Wav w → vLet go of w now rather than at the end of its owner's scope. A Wav is a plain value (its samples are a Vec): nothing needs releasing by hand.
: WCur: WCur {
( Vec u ) d
i n
i off
b fail
}A bounded cursor over the file bytes. Every read checks the budget against what is LEFT, never by forming off + k (which a hostile size would wrap).
@ wav_read s path → !Wav String@ wav_parse ( Vec u ) data → !Wav StringThe same parser, from bytes already in hand — an HTTP server that just received a WAV upload has no path to give. Same hostile-input treatment: the bytes are a stranger's whichever way they arrived.
@ wav_mono Wav w → ( Vec f )Average the channels. (Speech models want one channel; picking the left one throws away half of a stereo recording's information.)
@ wav_encode ( Vec f ) samples i rate i channels → ( Vec u )16-bit PCM, which every tool on earth reads. The same file, as BYTES. An HTTP server that has just synthesised audio has no path to write it to and no reason to invent one — it has a socket.
@ wav_write s path ( Vec f ) samples i rate i channels → !v String