selftest.nu

packages/safetensor/src/selftest.nu — the parser meets files that lie.

The container has exactly one interesting property: every tensor's bytes are addressed by offsets the FILE supplies. So the whole test is: craft headers whose offsets are wrong in every way a header can be wrong, and require a clean error each time — never a crash, never a read outside the mapping, never an allocation sized by a number we were handed.

A round-trip check comes first, so the negative cases are known to be rejected for the RIGHT reason: the same bytes with a truthful header parse.

API

: ~ i __st_pass 0

: ~ i __st_fail 0

@ st_selftest → i

---

safetensor.nu

packages/safetensor/src/safetensor.nu — the safetensors container, parsed as HOSTILE input.

The format is deliberately small:

u64 LE header_len header_len bytes of JSON { "name": {dtype, shape, data_offsets:[a,b]}, … } the tensor bytes (a and b are offsets INTO this region)

which means a file that lies about its offsets is the whole attack surface. So: header_len is checked against the real file size before the JSON is even looked at, every tensor's [a,b) is checked to lie inside the data region, b − a must equal what dtype × shape actually needs, and the element count is accumulated with an overflow check rather than multiplied and hoped for. Nothing is allocated on the strength of a number the file supplied.

mmap-backed and lazy, like gguf: the header is parsed up front, tensor bytes are read straight out of the mapping (stdlib/std/mmap.nu), so inspecting a multi-GB model costs no RAM. Every read is a checked slice of it.

( st_open path ) → !St String ( st_parse_bytes data ) → !St String (keeps data) ( st_n_tensors s ) → i ( st_tensors s ) → ( Vec StTensor ) (borrowed) ( st_find_tensor s name ) → i (-1 = absent) ( st_tensor_bytes s t ) → ( Slice u ) the tensor's bytes, a view that lives no longer than s ( st_tensor_ptr s t ) → *u raw: their address, for a device upload that keeps s alive ( st_dequant s idx ) → !( Vec u ) String — f32 bytes ( st_dequant_range s idx first count ) → !( Vec u ) String ( st_close s ) → v early release (optional) an empty slot is a ?St

An St is a handle: every copy is the same open file, and the last owner unmaps it (or lets go of the buffer it was parsed from). Nothing here is released by hand.

API

: i ST_F64 0

── dtypes ────────────────────────────────────────────────────────── Our own codes; the file spells them as strings.

: i ST_F32 1

: i ST_F16 2

: i ST_BF16 3

: i ST_I64 4

: i ST_I32 5

: i ST_I16 6

: i ST_I8 7

: i ST_U8 8

: i ST_BOOL 9

: i ST_U16 10

: i ST_U32 11

: i ST_U64 12

@ st_dtype_of s name → i

@ st_dtype_name i t → s

@ st_dtype_size i t → i

Bytes per element. Every dtype is fixed-width — there is no block quantisation in safetensors, which is why a row range is always exact.

: StTensor

: StTensor {
    String name
    i dtype
    i nd
    i d0
    i d1
    i d2
    i d3
    i nelems
    i offset  // absolute offset into the FILE
    i nbytes
}

: StImpl

: StImpl {
    MappedFile file  // the bytes: a mapping, or a buffer read whole
    ( Vec StTensor ) tensors
    i data_off  // absolute offset of the tensor-data region
    i data_size
}

: St { ( Arc StImpl ) a }

An St is a handle on its state in an Arc (stdlib/std/arc.nu), frozen once parsed: every copy is the same open file, and the last owner lets go of the mapping.

@ St_share St h → St

@ St_drop sink St h → v

@ st_open s path → !St String

mmap-backed where the platform has it (POSIX), a whole-file read where it does not (wasm, win32) — correct everywhere, lazy where it matters.

@ st_parse_bytes sink ( Vec u ) data → !St String

Parse an in-memory image. The St keeps data (its tensors are read out of it).

@ st_close sink St s → v

Let go of s now rather than at the end of its owner's scope; the last owner unmaps the file.

@ st_n_tensors St h → i

@ st_tensors St h → ( Vec StTensor )

The tensor table (borrowed: valid while the St is).

@ st_data_size St h → i

@ st_find_tensor St h s name → i

@ st_tensor_bytes St h StTensor t → ( Slice u )

The tensor's bytes, straight out of the mapping: a checked view that lives no longer than h (the borrow checker holds it to that). A tensor of another St that does not fit this one panics rather than reading past it.

@ _st_u32 ( Slice u ) P i o → i

@ st_dequant_range St h i idx i first i count → !( Vec u ) String

Elements [first, first+count) of tensor idx, as f32 BYTES (4 per element) — the same shape of result gguf_dequant returns, so a caller can upload it to the device without knowing which container it came from.

@ st_dequant St h i idx → !( Vec u ) String

---

write.nu

packages/safetensor/src/write.nu — the safetensors WRITER (the round-trip partner of safetensor.nu's reader). The format is small:

u64 LE header_len header_len bytes of JSON { "name":{dtype,shape,data_offsets:[a,b]}, … } the tensor bytes (a,b are offsets INTO this region)

A writer accumulates tensors (each appends its bytes and one header entry with contiguous, sorted-by-insertion data_offsets), then stw_finish emits the whole file: u64 header length, the JSON, then the concatenated data. The offsets are contiguous and gap-free by construction, which is what the reference safetensors library requires; st_open reads it back exactly.

Typed adds convert f64 HOST values to the on-disk dtype: F64/F32 exact, F16/BF16 via floatbits' encoders (the same ones gguf uses); I64 from a host i vector. stw_add_raw takes pre-encoded bytes for any other dtype.

( stw_new ) → StWriter ( stw_add_f32 w name shape v ) → v (v: Vec f, host f64) ( stw_add_f64 w name shape v ) → v ( stw_add_f16 w name shape v ) → v ( stw_add_bf16 w name shape v ) → v ( stw_add_i64 w name shape v ) → v (v: Vec i) ( stw_add_raw w name dtype shape b ) → v (b: Vec u, LE bytes) ( stw_finish w ) → ( Vec u ) — the whole file ( stw_write w path ) → !v String ( stw_free w ) → v early release (optional)

A StWriter is a handle: every copy is the same writer, and the last owner releases it. Nothing here is released by hand. It stays on the thread that made it (an Rc): adds change it in place.

API

: StwChunk

: StwChunk {
    ( Vec u ) b
}

: StWriterImpl

: StWriterImpl {
    String hdr  // building JSON, starts "{"
    ( Vec StwChunk ) chunks  // tensor bytes, one chunk per add, in order
    i dlen  // total data bytes so far (the next tensor's begin offset)
    b first  // no comma before the first entry
}

The data region lives as ONE CHUNK PER TENSOR, not one concatenated vector: concatenation re-copies every byte, and growing one giant vec holds old + new backing stores at each doubling — for a 4B model's 16 GB merge those two costs were the OOM killer, three runs in a row. A chunk is the typed add's own freshly built byte vec, moved in whole; offsets are tracked in dlen, and stw_write streams the chunks out in order. The on-disk bytes are identical to the old writer's.

: StWriter { ( Rc StWriterImpl ) r }

A StWriter is a handle on its state in an Rc (stdlib/std/rc.nu): every copy is the same writer, and the last owner releases it. An add checks the state out (rc_take), changes it and puts it back.

@ StWriter_share StWriter h → StWriter

@ StWriter_drop sink StWriter h → v

@ stw_new → StWriter

@ stw_add_raw StWriter w s name i dtype ( Vec i ) shape ( Vec u ) bytes → v

Any dtype: bytes is the tensor's raw little-endian payload (borrowed: copied into a fresh chunk, the caller keeps its own vec).

@ stw_add_f32 StWriter w s name ( Vec i ) shape ( Vec f ) v → v

F32: host f64 rounded to float32 on write.

@ stw_add_f64 StWriter w s name ( Vec i ) shape ( Vec f ) v → v

F64: host f64 exact.

@ stw_add_f16 StWriter w s name ( Vec i ) shape ( Vec f ) v → v

F16: host f64 → IEEE half (round-to-nearest-even via floatbits).

@ stw_add_bf16 StWriter w s name ( Vec i ) shape ( Vec f ) v → v

BF16: host f64 → bfloat16 (truncated top 16 bits of the f32).

@ stw_add_i64 StWriter w s name ( Vec i ) shape ( Vec i ) v → v

I64 from a host i vector.

@ stw_finish StWriter h → ( Vec u )

The whole file as one byte vector (small files / tests; a big file wants stw_write, which streams the chunks and never concatenates).

@ stw_write StWriter h s path → !v String

Write the file WITHOUT stw_finish's concatenation: the prefix (u64 + JSON header) goes first, then the data region is appended straight from the writer's own buffer. stw_finish materializes prefix + data as one new vector — a second full copy of every tensor byte, which for a 4B model's 16 GB merge briefly doubles the writer to 32 GB and is the difference between finishing and the OOM killer. Zero extra copies here; the on-disk bytes are identical.

@ stw_free sink StWriter w → v

Let go of w now rather than at the end of its owner's scope.

---

main.nu

packages/safetensor/src/main.nu — the CLI.

safetensor info <file> tensor count, data size safetensor tensors <file> every tensor: name, dtype, shape, offset safetensor verify <file> parse + re-check every extent safetensor stats <file> <name> count/min/max/mean of one tensor safetensor export <file> <name> -o out.f32 raw f32 LE safetensor selftest crafted files: every lie must be caught

API

@ main → i