torchpt.nu

packages/torchpt/src/torchpt.nu — read PyTorch .pt / .pth checkpoints in pure NURL.

A modern torch.save file is a ZIP archive:

<name>/data.pkl the object graph, pickled <name>/data/<key> one flat storage per key, raw little-endian <name>/version etc.

Tensors in the pickle are _rebuild_tensor_v2(storage, offset, size, stride, …) calls whose storage is a persistent id naming one of the data/<key> members. So reading a checkpoint is: parse the zip, parse the pickle as DATA (never executing it — see src/pickle.nu), then map each tensor onto a byte range of the mapping.

mmap-backed and lazy, like safetensor and gguf: only the pickle is read up front, so listing the tensors of a 4.6 GB checkpoint costs megabytes, not gigabytes. Tensor bytes are read straight out of the mapping (stdlib/std/mmap.nu), through checked views of it.

Entries are ZIP-stored (torch never deflates tensor data), which is what makes in-place addressing possible; a deflated storage is reported rather than silently mis-read.

( pt_open path ) → !Pt String ( pt_close p ) → v early release (optional) an empty slot is a ?Pt ( pt_n_tensors p ) → i ( pt_name p idx ) → s BORROWED ( pt_find p name ) → i -1 when absent ( pt_dtype p idx ) → i PKS_ ( pt_ndim p idx ) → i ( pt_dim p idx j ) → i ( pt_nelems p idx ) → i ( pt_nbytes p idx ) → i ( pt_is_contiguous p idx ) → b ( pt_tensor_bytes p idx ) → ( Slice u ) its nbytes, out of the mapping — a view held to p ( pt_tensor_ptr p idx ) → u raw: their address ( pt_dequant p idx ) → !( Vec u ) String f32 LE bytes ( pt_dequant_range p idx first count ) → !( Vec u ) String ( pt_read_f64_into p idx first count dst ) → b into a ( Slice f ) ( pt_read_f64 p idx first count dst ) → b raw: into a *f

Names are the dotted path from the pickle root: a plain state_dict gives blocks.0.attn.qkv.weight, while a training checkpoint saved as {"model": sd, "epoch": n} gives model.blocks.0.…. Nothing is stripped — what the file says is what you get.

A Pt is a handle: every copy is the same open checkpoint, and the last owner unmaps it. Nothing here is released by hand.

API

: i PT_MAX_DIMS 8

Up to 8 dimensions is more than any real checkpoint tensor uses; the limit keeps the per-tensor record a fixed size.

: PtTensor

: PtTensor {
    String name
    i dtype  // PKS_* storage class
    i ndim
    i nelems
    i data_off  // absolute FILE offset of element 0
    i nbytes  // logical size: nelems x element size
    i contiguous  // 1 = row-major, 0 = a view we can only describe
    i shape_at  // where its 16 slots start in the Pt's `dims`: dims 0..7, then strides 8..15
}

: PtImpl

: PtImpl {
    MappedFile file  // the checkpoint's bytes: a mapping, or the file read whole
    ( Vec PtTensor ) tensors
    ( Vec i ) dims  // 16 slots per tensor (PtTensor.shape_at): one array, not one per tensor
}

: Pt { ( Arc PtImpl ) a }

A Pt is a handle on its state in an Arc (stdlib/std/arc.nu), frozen once parsed: every copy is the same open checkpoint, and the last owner lets go of the mapping.

@ Pt_share Pt h → Pt

@ Pt_drop sink Pt h → v

@ _pt_geti ( Vec i ) v i k → i

@ pt_n_tensors Pt p__h → i

@ pt_name Pt p__h i idx → s

@ pt_dtype Pt p__h i idx → i

@ pt_ndim Pt p__h i idx → i

@ pt_dim Pt p__h i idx i j → i

@ pt_stride Pt p__h i idx i j → i

@ pt_nelems Pt p__h i idx → i

@ pt_nbytes Pt p__h i idx → i

@ pt_is_contiguous Pt p__h i idx → b

@ pt_offset Pt p__h i idx → i

@ pt_tensor_bytes Pt p__h i idx → ( Slice u )

A tensor's bytes — its nbytes from element 0 — straight out of the mapping: a checked view that lives no longer than p__h. Empty for an index out of range.

@ pt_find Pt p__h s name → i

@ pt_shape_str Pt p__h i idx → String

Shape rendered as "2x3x4" — for CLI listings and error messages.

@ pt_open s path → !Pt String

mmap-backed where the platform has it (POSIX), a whole-file read where it does not (wasm, win32).

@ pt_close sink Pt p → v

Let go of p now rather than at the end of its owner's scope; the last owner unmaps the checkpoint.

@ pt_dequant_range Pt p__h i idx i first i count → !( Vec u ) String

@ pt_dequant Pt p__h i idx → !( Vec u ) String

@ pt_read_f64_into Pt p__h i idx i first i count ( Slice f ) dst → b

Read count elements starting at first straight into dst (which must hold them) — the shape the tensor package wants, without a byte round-trip.

---

main.nu

packages/torchpt/src/main.nu — the torchpt CLI.

torchpt info <file.pt> summary: tensors, dtypes, bytes torchpt tensors <file.pt> [prefix] one line per tensor torchpt stats <file.pt> <name> min / max / mean of one tensor torchpt head <file.pt> <name> [n] first n values torchpt dump <file.pt> every tensor, every value — the oracle-diff format

API

@ usage → i

@ starts_with s hay s pre → b

@ cmd_info Pt p → i

@ cmd_tensors Pt p s prefix → i

@ cmd_stats Pt p s name → i

@ cmd_head Pt p s name i count → i

@ cmd_dump Pt p → i

"<name> <dtype> <shape> <v0> <v1> …" per tensor. Values print through nurl_str_float, which round-trips — so a diff against torch is a diff of numbers, not of formatting.

@ main → i

---

pickle.nu

packages/torchpt/src/pickle.nu — a pickle protocol 0–5 reader, restricted to data.

Unpickling is famously arbitrary code execution: the format is a stack machine whose REDUCE opcode means "call this callable". This reader never calls anything. GLOBAL pushes the name of a callable, and REDUCE consults a fixed table of shapes it knows how to build as data — torch._utils._rebuild_tensor_v2, collections.OrderedDict and a couple of siblings. Everything else becomes an opaque PK_OTHER node. A hostile pickle can therefore make this reader produce a confusing tree, and nothing else.

The result is a value tree in flat arrays (no per-node allocation):

( pk_parse p ) → !Pk String (p: the pickle's bytes) ( pk_free k ) → v early release (optional) ( pk_root k ) → i root node id ( __pk_kind k id ) → i PK_* ( pk_int k id ) → i PK_INT / PK_BOOL value ( pk_float k id ) → f PK_FLOAT value ( pk_str k id ) → s PK_STR text (BORROWED) ( __pk_len k id ) → i items (seq) or pairs (dict) ( pk_item k id j ) → i j-th item of a tuple/list ( pk_key k id j ) → i j-th key of a dict ( pk_val k id j ) → i j-th value of a dict ( pk_dict_get k id s key ) → i value node, -1 when absent

Tensors reduce to PK_TENSOR nodes; their fields come out through ( pk_tensor_storage k id ) → i storage id ( pk_tensor_offset k id ) → i element offset into the storage ( pk_tensor_ndim k id ) → i ( pk_tensor_dim k id j ) → i ( pk_tensor_stride k id j ) → i and a storage id resolves through ( pk_storage_dtype k sid ) → i PKS_* storage class ( pk_storage_key k sid ) → s the data/<key> filename (BORROWED) ( pk_storage_numel k sid ) → i

Everything is bounded before it is believed: a length read from the stream is checked against the bytes that remain, memo ids and stack depths are checked before use, and the opcode loop has a step budget proportional to the input, so a crafted pickle cannot spin, over-read or allocate on a number it merely claims.

A Pk is a handle: every copy is the same tree, and the last owner releases it. Nothing here is released by hand.

API

: i PK_NONE 0

── node kinds ──────────────────────────────────────────────────────

: i PK_BOOL 1

: i PK_INT 2

: i PK_FLOAT 3

: i PK_STR 4

: i PK_BYTES 5

: i PK_TUPLE 6

: i PK_LIST 7

: i PK_DICT 8

: i PK_TENSOR 9

: i PK_GLOBAL 10

: i PK_PERSID 11

: i PK_OTHER 12

: i PK_MARK 13 // internal: stack sentinel, never reachable from the root

@ pk_kind_name i kd → s

: i PKS_UNKNOWN 0

── torch storage classes ───────────────────────────────────────────

: i PKS_F64 1

: i PKS_F32 2

: i PKS_F16 3

: i PKS_BF16 4

: i PKS_I64 5

: i PKS_I32 6

: i PKS_I16 7

: i PKS_I8 8

: i PKS_U8 9

: i PKS_BOOL 10

@ pk_storage_class s name → i

@ pk_storage_name i c → s

@ pk_storage_esize i c → i

Bytes per element of a storage class; 0 when unknown.

: PkImpl

: PkImpl {
    ( Vec i ) kind  // per node
    ( Vec i ) va  // per node: int / str-idx / kids-start / tensor-idx / storage-idx
    ( Vec i ) vb  // per node: item or pair count, or byte length
    ( Vec i ) kids  // child node ids, addressed by (va, vb)
    ( Vec String ) strs
    ( Vec u ) blob  // BYTES payloads, addressed by (va, vb)
    ( Vec i ) stack
    ( Vec i ) memo
    ( Vec i ) stor  // 3 per storage: class, key str-idx, numel
    ( Vec i ) tens  // 4 per tensor: storage id, elem offset, ndim, dims-start
    ( Vec i ) dims  // ndim dims then ndim strides, per tensor
    ( Vec i ) root  // slot 0: root node id
    ( Vec String ) errs
}

── the tree, in flat arrays ────────────────────────────────────────

Every field is a Vec, never a scalar: a NURL struct passed by value copies its scalar fields, so a counter living in the struct would silently not mutate. Vec handles are shared, so pushes are seen by the caller — which is why even the root id and the error live in one-element Vecs.

: Pk { ( Arc PkImpl ) a }

A Pk is a handle on its state in an Arc (stdlib/std/arc.nu): every copy is the same tree, and the last owner releases it.

@ Pk_share Pk h → Pk

@ Pk_drop sink Pk h → v

@ pk_new → Pk

@ pk_free sink Pk k → v

Let go of k now rather than at the end of its owner's scope.

@ pk_root Pk h → i

@ pk_n_nodes Pk h → i

@ pk_kind Pk h i id → i

@ pk_int Pk h i id → i

@ pk_float Pk h i id → f

@ pk_str Pk h i id → s

@ pk_len Pk h i id → i

Items in a tuple/list, pairs in a dict, bytes in a byte string.

@ pk_item Pk h i id i j → i

@ pk_key Pk h i id i j → i

@ pk_val Pk h i id i j → i

@ pk_dict_get Pk h i id s key → i

@ pk_tensor_storage Pk h i id → i

@ pk_tensor_offset Pk h i id → i

@ pk_tensor_ndim Pk h i id → i

@ pk_tensor_dim Pk h i id i j → i

@ pk_tensor_stride Pk h i id i j → i

@ pk_n_storages Pk h → i

@ pk_storage_dtype Pk h i sid → i

@ pk_storage_key Pk h i sid → s

@ pk_storage_numel Pk h i sid → i

@ pk_error Pk h → s

@ pk_str_of_global Pk h i id → s

A GLOBAL's dotted name. Kept separate from pk_str so PK_STR stays the only kind that reads as text to callers.

@ pk_parse ( Slice u ) p → !Pk String