selftest.nu

packages/safetensor/src/selftest.nu — the parser meets files that lie.

The container has exactly one interesting property: every tensor's bytes are addressed by offsets the FILE supplies. So the whole test is: craft headers whose offsets are wrong in every way a header can be wrong, and require a clean error each time — never a crash, never a read outside the mapping, never an allocation sized by a number we were handed.

A round-trip check comes first, so the negative cases are known to be rejected for the RIGHT reason: the same bytes with a truthful header parse.

API

: ~ i __st_pass 0

: ~ i __st_fail 0

---

safetensor.nu

packages/safetensor/src/safetensor.nu — the safetensors container, parsed as HOSTILE input.

The format is deliberately small:

u64 LE header_len header_len bytes of JSON { "name": {dtype, shape, data_offsets:[a,b]}, … } the tensor bytes (a and b are offsets INTO this region)

which means a file that lies about its offsets is the whole attack surface. So: header_len is checked against the real file size before the JSON is even looked at, every tensor's [a,b) is checked to lie inside the data region, b − a must equal what dtype × shape actually needs, and the element count is accumulated with an overflow check rather than multiplied and hoped for. Nothing is allocated on the strength of a number the file supplied.

mmap-backed and lazy, like gguf: the header is parsed up front, tensor bytes are addressed straight out of the mapping, so inspecting a multi-GB model costs no RAM.

( st_open path ) → !St String ( st_parse_bytes data ) → !St String (keeps data) ( st_n_tensors s ) → i ( st_tensors s ) → ( Vec StTensor ) (borrowed) ( st_find_tensor s name ) → i (-1 = absent) ( st_tensor_ptr s t ) → *u (into the mapping) ( st_dequant s idx ) → !( Vec u ) String — f32 bytes ( st_dequant_range s idx first count ) → !( Vec u ) String ( st_close s ) → v early release (optional) ( st_none ) / ( st_is_open s ) → St / b an empty slot, and the test for one

An St is a handle: every copy is the same open file, and the last owner unmaps it (or lets go of the buffer it was parsed from). Nothing here is released by hand.

API

: i ST_F64 0

── dtypes ────────────────────────────────────────────────────────── Our own codes; the file spells them as strings.

: i ST_F32 1

: i ST_F16 2

: i ST_BF16 3

: i ST_I64 4

: i ST_I32 5

: i ST_I16 6

: i ST_I8 7

: i ST_U8 8

: i ST_BOOL 9

: i ST_U16 10

: i ST_U32 11

: i ST_U64 12

@ st_dtype_of s name → i

@ st_dtype_name i t → s

@ st_dtype_size i t → i

Bytes per element. Every dtype is fixed-width — there is no block quantisation in safetensors, which is why a row range is always exact.

: StTensor

: StTensor {
    String name
    i dtype
    i nd
    i d0
    i d1
    i d2
    i d3
    i nelems
    i offset  // absolute offset into the FILE
    i nbytes
}

: StImpl

: StImpl {
    * u map
    i map_size
    b from_mmap
    ( Vec u ) buf  // the bytes `map` points into, when they are not a mapping
    ( Vec StTensor ) tensors
    i data_off  // absolute offset of the tensor-data region
    i data_size
}

% Drop StImpl

% Drop StImpl {
    unsafe @ drop StImpl s → v {
        ? . s from_mmap { : i32 _u ( munmap . s map . s map_size ) } {}
    }
}

The mapping is the raw resource: its last owner unmaps it, as st_close did. The tensor table and the buffer go with the drop glue.

: St { s ctl }

An St is a handle on its state in an rcbox (stdlib/core/rcbox.nu): every copy is the same open file, and the last owner releases it.

@ St_drop sink St h → v

@ st_none → St

An St that holds no file — for a slot that may be empty (a model whose checkpoint is some other format). st_is_open tells the two apart.

@ st_is_open St s → b

@ st_close sink St s → v

Let go of s now rather than at the end of its owner's scope; the last owner unmaps the file.

---

write.nu

packages/safetensor/src/write.nu — the safetensors WRITER (the round-trip partner of safetensor.nu's reader). The format is small:

u64 LE header_len header_len bytes of JSON { "name":{dtype,shape,data_offsets:[a,b]}, … } the tensor bytes (a,b are offsets INTO this region)

A writer accumulates tensors (each appends its bytes and one header entry with contiguous, sorted-by-insertion data_offsets), then stw_finish emits the whole file: u64 header length, the JSON, then the concatenated data. The offsets are contiguous and gap-free by construction, which is what the reference safetensors library requires; st_open reads it back exactly.

Typed adds convert f64 HOST values to the on-disk dtype: F64/F32 exact, F16/BF16 via floatbits' encoders (the same ones gguf uses); I64 from a host i vector. stw_add_raw takes pre-encoded bytes for any other dtype.

( stw_new ) → StWriter ( stw_add_f32 w name shape v ) → v (v: Vec f, host f64) ( stw_add_f64 w name shape v ) → v ( stw_add_f16 w name shape v ) → v ( stw_add_bf16 w name shape v ) → v ( stw_add_i64 w name shape v ) → v (v: Vec i) ( stw_add_raw w name dtype shape b ) → v (b: Vec u, LE bytes) ( stw_finish w ) → ( Vec u ) — the whole file ( stw_write w path ) → !v String ( stw_free w ) → v early release (optional)

A StWriter is a handle: every copy is the same writer, and the last owner releases it. Nothing here is released by hand.

API

: StwChunk

: StwChunk {
    ( Vec u ) b
}

: StWriterImpl

: StWriterImpl {
    String hdr  // building JSON, starts "{"
    ( Vec StwChunk ) chunks  // tensor bytes, one chunk per add, in order
    i dlen  // total data bytes so far (the next tensor's begin offset)
    b first  // no comma before the first entry
}

The data region lives as ONE CHUNK PER TENSOR, not one concatenated vector: concatenation re-copies every byte, and growing one giant vec holds old + new backing stores at each doubling — for a 4B model's 16 GB merge those two costs were the OOM killer, three runs in a row. A chunk is the typed add's own freshly built byte vec, moved in whole; offsets are tracked in dlen, and stw_write streams the chunks out in order. The on-disk bytes are identical to the old writer's.

: StWriter { s ctl }

A StWriter is a handle on its state in an rcbox (stdlib/core/rcbox.nu): every copy is the same writer, and the last owner releases it.

@ StWriter_drop sink StWriter h → v

@ stw_free sink StWriter w → v

Let go of w now rather than at the end of its owner's scope. Let go of w now rather than at the end of its owner's scope.

---

main.nu

packages/safetensor/src/main.nu — the CLI.

safetensor info <file> tensor count, data size safetensor tensors <file> every tensor: name, dtype, shape, offset safetensor verify <file> parse + re-check every extent safetensor stats <file> <name> count/min/max/mean of one tensor safetensor export <file> <name> -o out.f32 raw f32 LE safetensor selftest crafted files: every lie must be caught

API

@ main → i