selftest.nu

packages/safetensor/src/selftest.nu — the parser meets files that lie.

The container has exactly one interesting property: every tensor's bytes are addressed by offsets the FILE supplies. So the whole test is: craft headers whose offsets are wrong in every way a header can be wrong, and require a clean error each time — never a crash, never a read outside the mapping, never an allocation sized by a number we were handed.

A round-trip check comes first, so the negative cases are known to be rejected for the RIGHT reason: the same bytes with a truthful header parse.

API

: ~ i __st_pass 0

: ~ i __st_fail 0

@ st_selftest → i

---

safetensor.nu

packages/safetensor/src/safetensor.nu — the safetensors container, parsed as HOSTILE input.

The format is deliberately small:

u64 LE header_len header_len bytes of JSON { "name": {dtype, shape, data_offsets:[a,b]}, … } the tensor bytes (a and b are offsets INTO this region)

which means a file that lies about its offsets is the whole attack surface. So: header_len is checked against the real file size before the JSON is even looked at, every tensor's [a,b) is checked to lie inside the data region, b − a must equal what dtype × shape actually needs, and the element count is accumulated with an overflow check rather than multiplied and hoped for. Nothing is allocated on the strength of a number the file supplied.

mmap-backed and lazy, like gguf: the header is parsed up front, tensor bytes are addressed straight out of the mapping, so inspecting a multi-GB model costs no RAM.

( st_open path ) → !St String ( st_n_tensors s ) → i ( st_find_tensor s name ) → i (-1 = absent) ( st_tensor_ptr s t ) → u (into the mapping) ( st_dequant s idx ) → !( Vec u ) String — f32 bytes ( st_dequant_range s idx first count ) → !( Vec u ) String ( st_close s ) → v

API

: i ST_F64 0

── dtypes ────────────────────────────────────────────────────────── Our own codes; the file spells them as strings.

: i ST_F32 1

: i ST_F16 2

: i ST_BF16 3

: i ST_I64 4

: i ST_I32 5

: i ST_I16 6

: i ST_I8 7

: i ST_U8 8

: i ST_BOOL 9

: i ST_U16 10

: i ST_U32 11

: i ST_U64 12

@ st_dtype_of s name → i

@ st_dtype_name i t → s

@ st_dtype_size i t → i

Bytes per element. Every dtype is fixed-width — there is no block quantisation in safetensors, which is why a row range is always exact.

: StTensor

: StTensor {
    String name
    i dtype
    i nd
    i d0
    i d1
    i d2
    i d3
    i nelems
    i offset  // absolute offset into the FILE
    i nbytes
}

: St

: St {
    * u map
    i map_size
    b from_mmap
    ( Vec u ) buf
    ( Vec StTensor ) tensors
    i data_off  // absolute offset of the tensor-data region
    i data_size
}

@ st_open s path → !*St String

mmap-backed where the platform has it (POSIX), a whole-file read where it does not (wasm, win32) — correct everywhere, lazy where it matters.

@ st_parse_bytes ( Vec u ) data → !*St String

Parse an in-memory image. BORROWS data — the caller keeps the vec alive until st_close and frees it afterwards.

@ st_close * St s → v

@ st_n_tensors * St s → i

@ st_data_size * St s → i

@ st_find_tensor * St s s name → i

@ st_tensor_ptr * St s StTensor t → *u

The tensor's bytes, straight out of the mapping. Borrowed: valid until st_close.

@ _st_u32 * u P i o → i

@ st_dequant_range * St s i idx i first i count → !( Vec u ) String

Elements [first, first+count) of tensor idx, as f32 BYTES (4 per element) — the same shape of result gguf_dequant returns, so a caller can upload it to the device without knowing which container it came from.

@ st_dequant * St s i idx → !( Vec u ) String

---

write.nu

packages/safetensor/src/write.nu — the safetensors WRITER (the round-trip partner of safetensor.nu's reader). The format is small:

u64 LE header_len header_len bytes of JSON { "name":{dtype,shape,data_offsets:[a,b]}, … } the tensor bytes (a,b are offsets INTO this region)

A writer accumulates tensors (each appends its bytes and one header entry with contiguous, sorted-by-insertion data_offsets), then stw_finish emits the whole file: u64 header length, the JSON, then the concatenated data. The offsets are contiguous and gap-free by construction, which is what the reference safetensors library requires; st_open reads it back exactly.

Typed adds convert f64 HOST values to the on-disk dtype: F64/F32 exact, F16/BF16 via floatbits' encoders (the same ones gguf uses); I64 from a host i vector. stw_add_raw takes pre-encoded bytes for any other dtype.

( stw_new ) → *StWriter ( stw_add_f32 w name shape v ) → v (v: Vec f, host f64) ( stw_add_f64 w name shape v ) → v ( stw_add_f16 w name shape v ) → v ( stw_add_bf16 w name shape v ) → v ( stw_add_i64 w name shape v ) → v (v: Vec i) ( stw_add_raw w name dtype shape b ) → v (b: Vec u, LE bytes) ( stw_finish w ) → ( Vec u ) — the whole file ( stw_write w path ) → !v String ( stw_free w ) → v

API

: StWriter

: StWriter {
    String hdr  // building JSON, starts "{"
    ( Vec u ) data  // concatenated tensor bytes
    b first  // no comma before the first entry
}

@ stw_new → *StWriter

@ stw_add_raw * StWriter w s name i dtype ( Vec i ) shape ( Vec u ) bytes → v

Any dtype: bytes is the tensor's raw little-endian payload.

@ stw_add_f32 * StWriter w s name ( Vec i ) shape ( Vec f ) v → v

F32: host f64 rounded to float32 on write.

@ stw_add_f64 * StWriter w s name ( Vec i ) shape ( Vec f ) v → v

F64: host f64 exact.

@ stw_add_f16 * StWriter w s name ( Vec i ) shape ( Vec f ) v → v

F16: host f64 → IEEE half (round-to-nearest-even via floatbits).

@ stw_add_bf16 * StWriter w s name ( Vec i ) shape ( Vec f ) v → v

BF16: host f64 → bfloat16 (truncated top 16 bits of the f32).

@ stw_add_i64 * StWriter w s name ( Vec i ) shape ( Vec i ) v → v

I64 from a host i vector.

@ stw_finish * StWriter w → ( Vec u )

The whole file as one byte vector.

@ stw_write * StWriter w s path → !v String

@ stw_free * StWriter w → v

---

main.nu

packages/safetensor/src/main.nu — the CLI.

safetensor info <file> tensor count, data size safetensor tensors <file> every tensor: name, dtype, shape, offset safetensor verify <file> parse + re-check every extent safetensor stats <file> <name> count/min/max/mean of one tensor safetensor export <file> <name> -o out.f32 raw f32 LE safetensor selftest crafted files: every lie must be caught

API

@ main → i