selftest.nu

packages/safetensor/src/selftest.nu — the parser meets files that lie.

The container has exactly one interesting property: every tensor's bytes are addressed by offsets the FILE supplies. So the whole test is: craft headers whose offsets are wrong in every way a header can be wrong, and require a clean error each time — never a crash, never a read outside the mapping, never an allocation sized by a number we were handed.

A round-trip check comes first, so the negative cases are known to be rejected for the RIGHT reason: the same bytes with a truthful header parse.

API

: ~ i __st_pass 0

: ~ i __st_fail 0

@ st_selftest → i

---

safetensor.nu

packages/safetensor/src/safetensor.nu — the safetensors container, parsed as HOSTILE input.

The format is deliberately small:

u64 LE header_len header_len bytes of JSON { "name": {dtype, shape, data_offsets:[a,b]}, … } the tensor bytes (a and b are offsets INTO this region)

which means a file that lies about its offsets is the whole attack surface. So: header_len is checked against the real file size before the JSON is even looked at, every tensor's [a,b) is checked to lie inside the data region, b − a must equal what dtype × shape actually needs, and the element count is accumulated with an overflow check rather than multiplied and hoped for. Nothing is allocated on the strength of a number the file supplied.

mmap-backed and lazy, like gguf: the header is parsed up front, tensor bytes are addressed straight out of the mapping, so inspecting a multi-GB model costs no RAM.

( st_open path ) → !St String ( st_n_tensors s ) → i ( st_find_tensor s name ) → i (-1 = absent) ( st_tensor_ptr s t ) → u (into the mapping) ( st_dequant s idx ) → !( Vec u ) String — f32 bytes ( st_dequant_range s idx first count ) → !( Vec u ) String ( st_close s ) → v

API

: i ST_F64 0

── dtypes ────────────────────────────────────────────────────────── Our own codes; the file spells them as strings.

: i ST_F32 1

: i ST_F16 2

: i ST_BF16 3

: i ST_I64 4

: i ST_I32 5

: i ST_I16 6

: i ST_I8 7

: i ST_U8 8

: i ST_BOOL 9

: i ST_U16 10

: i ST_U32 11

: i ST_U64 12

@ st_dtype_of s name → i

@ st_dtype_name i t → s

@ st_dtype_size i t → i

Bytes per element. Every dtype is fixed-width — there is no block quantisation in safetensors, which is why a row range is always exact.

: StTensor

: StTensor {
    String name
    i dtype
    i nd
    i d0
    i d1
    i d2
    i d3
    i nelems
    i offset  // absolute offset into the FILE
    i nbytes
}

: St

: St {
    * u map
    i map_size
    b from_mmap
    ( Vec u ) buf
    ( Vec StTensor ) tensors
    i data_off  // absolute offset of the tensor-data region
    i data_size
}

@ st_open s path → !*St String

mmap-backed where the platform has it (POSIX), a whole-file read where it does not (wasm, win32) — correct everywhere, lazy where it matters.

@ st_parse_bytes ( Vec u ) data → !*St String

Parse an in-memory image. BORROWS data — the caller keeps the vec alive until st_close and frees it afterwards.

@ st_close * St s → v

@ st_n_tensors * St s → i

@ st_data_size * St s → i

@ st_find_tensor * St s s name → i

@ st_tensor_ptr * St s StTensor t → *u

The tensor's bytes, straight out of the mapping. Borrowed: valid until st_close.

@ st_dequant_range * St s i idx i first i count → !( Vec u ) String

Elements [first, first+count) of tensor idx, as f32 BYTES (4 per element) — the same shape of result gguf_dequant returns, so a caller can upload it to the device without knowing which container it came from.

@ st_dequant * St s i idx → !( Vec u ) String

---

main.nu

packages/safetensor/src/main.nu — the CLI.

safetensor info <file> tensor count, data size safetensor tensors <file> every tensor: name, dtype, shape, offset safetensor verify <file> parse + re-check every extent safetensor stats <file> <name> count/min/max/mean of one tensor safetensor export <file> <name> -o out.f32 raw f32 LE safetensor selftest crafted files: every lie must be caught

API

@ main → i