torchpt.nu

packages/torchpt/src/torchpt.nu — read PyTorch .pt / .pth checkpoints in pure NURL.

A modern torch.save file is a ZIP archive:

<name>/data.pkl the object graph, pickled <name>/data/<key> one flat storage per key, raw little-endian <name>/version etc.

Tensors in the pickle are _rebuild_tensor_v2(storage, offset, size, stride, …) calls whose storage is a persistent id naming one of the data/<key> members. So reading a checkpoint is: parse the zip, parse the pickle as DATA (never executing it — see src/pickle.nu), then map each tensor onto a byte range of the mapping.

mmap-backed and lazy, like safetensor and gguf: only the pickle is read up front, so listing the tensors of a 4.6 GB checkpoint costs megabytes, not gigabytes. Tensor bytes are addressed straight out of the mapping.

Entries are ZIP-stored (torch never deflates tensor data), which is what makes in-place addressing possible; a deflated storage is reported rather than silently mis-read.

( pt_open path ) → !Pt String ( pt_close p ) → v ( pt_n_tensors p ) → i ( pt_name p idx ) → s BORROWED ( pt_find p name ) → i -1 when absent ( pt_dtype p idx ) → i PKS_ ( pt_ndim p idx ) → i ( pt_dim p idx j ) → i ( pt_nelems p idx ) → i ( pt_nbytes p idx ) → i ( pt_is_contiguous p idx ) → b ( pt_tensor_ptr p idx ) → *u into the mapping ( pt_dequant p idx ) → !( Vec u ) String f32 LE bytes ( pt_dequant_range p idx first count ) → !( Vec u ) String ( pt_read_f64 p idx first count dst ) → b into an f64 buffer

Names are the dotted path from the pickle root: a plain state_dict gives blocks.0.attn.qkv.weight, while a training checkpoint saved as {"model": sd, "epoch": n} gives model.blocks.0.…. Nothing is stripped — what the file says is what you get.

API

: i PT_MAX_DIMS 8

Up to 8 dimensions is more than any real checkpoint tensor uses; the limit keeps the per-tensor record a fixed size.

: PtTensor

: PtTensor {
    String name
    i dtype  // PKS_* storage class
    i ndim
    i nelems
    i data_off  // absolute FILE offset of element 0
    i nbytes  // logical size: nelems x element size
    i contiguous  // 1 = row-major, 0 = a view we can only describe
    s shape  // 16 slots: dims 0..7, then strides 8..15
}

: Pt

: Pt {
    * u map
    i map_size
    b from_mmap
    ( Vec u ) buf
    ( Vec PtTensor ) tensors
}

@ pt_n_tensors * Pt p → i

@ pt_name * Pt p i idx → s

@ pt_dtype * Pt p i idx → i

@ pt_ndim * Pt p i idx → i

@ pt_dim * Pt p i idx i j → i

@ pt_stride * Pt p i idx i j → i

@ pt_nelems * Pt p i idx → i

@ pt_nbytes * Pt p i idx → i

@ pt_is_contiguous * Pt p i idx → b

@ pt_offset * Pt p i idx → i

@ pt_tensor_ptr * Pt p i idx → *u

Raw bytes of a tensor, addressed in the mapping. Valid until pt_close.

@ pt_find * Pt p s name → i

@ pt_shape_str * Pt p i idx → String

Shape rendered as "2x3x4" — for CLI listings and error messages.

@ pt_open s path → !*Pt String

@ pt_close * Pt p → v

@ pt_dequant_range * Pt p i idx i first i count → !( Vec u ) String

@ pt_dequant * Pt p i idx → !( Vec u ) String

@ pt_read_f64 * Pt p i idx i first i count * f dst → b

Read count elements starting at first straight into a caller's f64 buffer — the shape the tensor package wants, without a byte round-trip.

---

main.nu

packages/torchpt/src/main.nu — the torchpt CLI.

torchpt info <file.pt> summary: tensors, dtypes, bytes torchpt tensors <file.pt> [prefix] one line per tensor torchpt stats <file.pt> <name> min / max / mean of one tensor torchpt head <file.pt> <name> [n] first n values torchpt dump <file.pt> every tensor, every value — the oracle-diff format

API

@ usage → i

@ starts_with s hay s pre → b

@ cmd_info * Pt p → i

@ cmd_tensors * Pt p s prefix → i

@ cmd_stats * Pt p s name → i

@ cmd_head * Pt p s name i count → i

@ cmd_dump * Pt p → i

"<name> <dtype> <shape> <v0> <v1> …" per tensor. Values print through nurl_str_float, which round-trips — so a diff against torch is a diff of numbers, not of formatting.

@ main → i

---

pickle.nu

packages/torchpt/src/pickle.nu — a pickle protocol 0–5 reader, restricted to data.

Unpickling is famously arbitrary code execution: the format is a stack machine whose REDUCE opcode means "call this callable". This reader never calls anything. GLOBAL pushes the name of a callable, and REDUCE consults a fixed table of shapes it knows how to build as data — torch._utils._rebuild_tensor_v2, collections.OrderedDict and a couple of siblings. Everything else becomes an opaque PK_OTHER node. A hostile pickle can therefore make this reader produce a confusing tree, and nothing else.

The result is a value tree in flat arrays (no per-node allocation):

( pk_parse u p i n ) → !Pk String ( pk_free k ) → v ( pk_root k ) → i root node id ( pk_kind k id ) → i PK_* ( pk_int k id ) → i PK_INT / PK_BOOL value ( pk_float k id ) → f PK_FLOAT value ( pk_str k id ) → s PK_STR text (BORROWED) ( pk_len k id ) → i items (seq) or pairs (dict) ( pk_item k id j ) → i j-th item of a tuple/list ( pk_key k id j ) → i j-th key of a dict ( pk_val k id j ) → i j-th value of a dict ( pk_dict_get k id s key ) → i value node, -1 when absent

Tensors reduce to PK_TENSOR nodes; their fields come out through ( pk_tensor_storage k id ) → i storage id ( pk_tensor_offset k id ) → i element offset into the storage ( pk_tensor_ndim k id ) → i ( pk_tensor_dim k id j ) → i ( pk_tensor_stride k id j ) → i and a storage id resolves through ( pk_storage_dtype k sid ) → i PKS_* storage class ( pk_storage_key k sid ) → s the data/<key> filename (BORROWED) ( pk_storage_numel k sid ) → i

Everything is bounded before it is believed: a length read from the stream is checked against the bytes that remain, memo ids and stack depths are checked before use, and the opcode loop has a step budget proportional to the input, so a crafted pickle cannot spin, over-read or allocate on a number it merely claims.

API

: i PK_NONE 0

── node kinds ──────────────────────────────────────────────────────

: i PK_BOOL 1

: i PK_INT 2

: i PK_FLOAT 3

: i PK_STR 4

: i PK_BYTES 5

: i PK_TUPLE 6

: i PK_LIST 7

: i PK_DICT 8

: i PK_TENSOR 9

: i PK_GLOBAL 10

: i PK_PERSID 11

: i PK_OTHER 12

: i PK_MARK 13 // internal: stack sentinel, never reachable from the root

@ pk_kind_name i kd → s

: i PKS_UNKNOWN 0

── torch storage classes ───────────────────────────────────────────

: i PKS_F64 1

: i PKS_F32 2

: i PKS_F16 3

: i PKS_BF16 4

: i PKS_I64 5

: i PKS_I32 6

: i PKS_I16 7

: i PKS_I8 8

: i PKS_U8 9

: i PKS_BOOL 10

@ pk_storage_class s name → i

@ pk_storage_name i c → s

@ pk_storage_esize i c → i

Bytes per element of a storage class; 0 when unknown.

: Pk

: Pk {
    ( Vec i ) kind  // per node
    ( Vec i ) va  // per node: int / str-idx / kids-start / tensor-idx / storage-idx
    ( Vec i ) vb  // per node: item or pair count, or byte length
    ( Vec i ) kids  // child node ids, addressed by (va, vb)
    ( Vec String ) strs
    ( Vec u ) blob  // BYTES payloads, addressed by (va, vb)
    ( Vec i ) stack
    ( Vec i ) memo
    ( Vec i ) stor  // 3 per storage: class, key str-idx, numel
    ( Vec i ) tens  // 4 per tensor: storage id, elem offset, ndim, dims-start
    ( Vec i ) dims  // ndim dims then ndim strides, per tensor
    ( Vec i ) root  // slot 0: root node id
    ( Vec String ) errs
}

── the tree, in flat arrays ────────────────────────────────────────

Every field is a Vec, never a scalar: a NURL struct passed by value copies its scalar fields, so a counter living in the struct would silently not mutate. Vec handles are shared, so pushes are seen by the caller — which is why even the root id and the error live in one-element Vecs.

@ pk_new → *Pk

@ pk_free * Pk k → v

@ pk_root * Pk k → i

@ pk_n_nodes * Pk k → i

@ pk_kind * Pk k i id → i

@ pk_int * Pk k i id → i

@ pk_float * Pk k i id → f

@ pk_str * Pk k i id → s

@ pk_len * Pk k i id → i

Items in a tuple/list, pairs in a dict, bytes in a byte string.

@ pk_item * Pk k i id i j → i

@ pk_key * Pk k i id i j → i

@ pk_val * Pk k i id i j → i

@ pk_dict_get * Pk k i id s key → i

@ pk_tensor_storage * Pk k i id → i

@ pk_tensor_offset * Pk k i id → i

@ pk_tensor_ndim * Pk k i id → i

@ pk_tensor_dim * Pk k i id i j → i

@ pk_tensor_stride * Pk k i id i j → i

@ pk_n_storages * Pk k → i

@ pk_storage_dtype * Pk k i sid → i

@ pk_storage_key * Pk k i sid → s

@ pk_storage_numel * Pk k i sid → i

@ pk_error * Pk k → s

@ pk_str_of_global * Pk k i id → s

A GLOBAL's dotted name. Kept separate from pk_str so PK_STR stays the only kind that reads as text to callers.

@ pk_parse * u p i n → !*Pk String