data.nu

data.nu — a training DataLoader: batching + seeded shuffle + epochs + drop-last + worker sharding, over an in-memory dataset OR one streamed off disk.

A dataset is n examples, each d f64 features and l f64 labels (l may be 0). In memory it is two flat vectors (x: n·d, y: n·l). On disk it is the .ndf format — a magic + n/d/l header, then n rows of (d+l) little-endian f64 (x then y) — read with random-access preads so a corpus larger than RAM never loads whole.

A DataLoader holds a permutation of its shard's example indices; dl_next gathers the next batch rows (in memory: a copy; streaming: one pread per row) into caller-provided vectors and returns the row count (0 = exhausted). dl_reset reshuffles for the next epoch. Sharding partitions [0,n) into contiguous, exactly-once slices — shard s of N gets [s·n/N, (s+1)·n/N).

( data_new x y n d l ) → DataSet (takes ownership of x,y) ( dl_new ds batch drop_last seed ) → DataLoader ( dl_new_shard ds batch drop_last seed nshards shard ) → DataLoader ( dl_next dl bx by ) → i (rows; 0 = end) ( dl_reset dl seed ) → v (next epoch) ( dl_num_batches dl ) → i ( data_save_ndf path ds ) → !v String ( ndf_open path ) → !NdfStream String ( dl_stream st batch drop_last seed ) → DataLoader ( dl_stream_shard st batch drop_last seed nshards shard ) → DataLoader

DataSet, NdfStream and DataLoader are handles: every copy is the same object, and the last owner releases it — an NdfStream closes its file. A loader holds a share of its source, so the source outlives every loader over it. data_free / ndf_close / dl_free are optional early releases.

API

: DataSetImpl

: DataSetImpl {
    ( Vec f ) x  // n·d features, row-major
    ( Vec f ) y  // n·l labels (empty when l == 0)
    i n
    i d
    i l
}

: DataSet { s ctl }

A DataSet is a handle: every copy (each loader over it holds one) is the same data, and the last owner releases it.

@ DataSet_drop sink DataSet h → v

@ data_free sink DataSet ds → v

Let go of ds now rather than at the end of its owner's scope.

: i __NDF_HDR 28

── .ndf on-disk format ──────────────────────────────────────────────── bytes: 'N' 'D' 'F' '1' | u64 n | u64 d | u64 l | rows((d+l) f64 LE)…

: NdfStreamImpl

: NdfStreamImpl {
    File f
    i n
    i d
    i l
    i data_off
}

% Drop NdfStreamImpl { @ drop NdfStreamImpl st → v { ( file_close . st f ) } }

The open file is the stream's one raw resource: its last owner closes it.

: NdfStream { s ctl }

@ NdfStream_drop sink NdfStream h → v

@ ndf_close sink NdfStream st → v

Let go of st now rather than at the end of its owner's scope (the last owner closes the file).

: DataLoaderImpl

: DataLoaderImpl {
    DataSet ds  // in-memory source (a null handle when streaming)
    NdfStream st  // streaming source (a null handle when in-memory)
    i n  // examples in this shard
    i d
    i l
    i base  // absolute index of this shard's first example
    i batch
    b drop_last
    b shuffle
    ( Vec i ) idx  // permutation of [base, base+n)
    i pos
    i last_rows
}

: DataLoader { s ctl }

A DataLoader is a handle: every copy is the same loader (one cursor through the epoch), and the last owner releases it — and with it its share of the dataset or stream it reads.

@ DataLoader_drop sink DataLoader h → v

@ dl_free sink DataLoader dl → v

Let go of dl now rather than at the end of its owner's scope.