data.nu

data.nu — a training DataLoader: batching + seeded shuffle + epochs + drop-last + worker sharding, over an in-memory dataset OR one streamed off disk.

A dataset is n examples, each d f64 features and l f64 labels (l may be 0). In memory it is two flat vectors (x: n·d, y: n·l). On disk it is the .ndf format — a magic + n/d/l header, then n rows of (d+l) little-endian f64 (x then y) — read with random-access preads so a corpus larger than RAM never loads whole.

A DataLoader holds a permutation of its shard's example indices; dl_next gathers the next batch rows (in memory: a copy; streaming: one pread per row) into caller-provided vectors and returns the row count (0 = exhausted). dl_reset reshuffles for the next epoch. Sharding partitions [0,n) into contiguous, exactly-once slices — shard s of N gets [s·n/N, (s+1)·n/N).

( data_new x y n d l ) → DataSet (takes ownership of x,y) ( dl_new ds batch drop_last seed ) → DataLoader ( dl_new_shard ds batch drop_last seed nshards shard ) → DataLoader ( dl_next dl bx by ) → i (rows; 0 = end) ( dl_reset dl seed ) → v (next epoch) ( dl_num_batches dl ) → i ( data_save_ndf path ds ) → !v String ( ndf_open path ) → !NdfStream String ( dl_stream st batch drop_last seed ) → DataLoader ( dl_stream_shard st batch drop_last seed nshards shard ) → DataLoader

DataSet, NdfStream and DataLoader are handles: every copy is the same object, and the last owner releases it — an NdfStream closes its file. A loader holds a share of its source, so the source outlives every loader over it. data_free / ndf_close / dl_free are optional early releases.

API

: DataSetImpl

: DataSetImpl {
    ( Vec f ) x  // n·d features, row-major
    ( Vec f ) y  // n·l labels (empty when l == 0)
    i n
    i d
    i l
}

: DataSet { ( Arc DataSetImpl ) a }

A DataSet is a handle: every copy (each loader over it holds one) is the same data, and the last owner releases it.

@ DataSet_share DataSet h → DataSet

@ DataSet_drop sink DataSet h → v

@ data_new sink ( Vec f ) x sink ( Vec f ) y i n i d i l → DataSet

Take ownership of x (n·d) and y (n·l).

@ data_free sink DataSet ds → v

Let go of ds now rather than at the end of its owner's scope.

@ data_n DataSet ds__h → i

@ data_d DataSet ds__h → i

@ data_l DataSet ds__h → i

: i __NDF_HDR 28

── .ndf on-disk format ──────────────────────────────────────────────── bytes: 'N' 'D' 'F' '1' | u64 n | u64 d | u64 l | rows((d+l) f64 LE)…

: NdfStreamImpl

: NdfStreamImpl {
    // Read at offsets (pread): a DataLoader's copies on several threads
    // read rows of one stream at once, and a stream with one position
    // (seek, then read) would move it under another thread's read. The
    // last owner of the handle closes the file.
    PFile f
    i n
    i d
    i l
    i data_off
}

: NdfStream { ( Arc NdfStreamImpl ) a }

@ NdfStream_share NdfStream h → NdfStream

@ NdfStream_drop sink NdfStream h → v

@ data_save_ndf s path DataSet ds__h → !v String

@ ndf_open s path → !NdfStream String

@ ndf_close sink NdfStream st → v

Let go of st now rather than at the end of its owner's scope (the last owner closes the file).

@ ndf_n NdfStream st__h → i

@ ndf_d NdfStream st__h → i

@ ndf_l NdfStream st__h → i

@ ndf_read_row NdfStream st__h i idx ( Vec f ) x_out ( Vec f ) y_out → b

Read example idx: append its d features to x_out and l labels to y_out.

: DataLoaderImpl

: DataLoaderImpl {
    ? DataSet ds  // in-memory source (F when streaming)
    ? NdfStream st  // streaming source (F when in-memory)
    i n  // examples in this shard
    i d
    i l
    i base  // absolute index of this shard's first example
    i batch
    b drop_last
    b shuffle
    ( Vec i ) idx  // permutation of [base, base+n)
    i pos
    i last_rows
}

: DataLoader { ( Locked DataLoaderImpl ) st }

A DataLoader is a handle: every copy is the same loader (one cursor through the epoch), and the last owner releases it — and with it its share of the dataset or stream it reads. The cursor is in a Locked (stdlib/std/thread.nu): a step checks it out and back in, so copies on several threads take batches in turn.

@ DataLoader_share DataLoader h → DataLoader

@ DataLoader_drop sink DataLoader h → v

@ dl_new DataSet ds__h i batch b drop_last i seed → DataLoader

Full in-memory dataset (one shard). seed <= 0 disables shuffling.

@ dl_new_shard DataSet ds__h i batch b drop_last i seed i nshards i shard → DataLoader

@ dl_stream NdfStream st__h i batch b drop_last i seed → DataLoader

Streaming dataset (one shard).

@ dl_stream_shard NdfStream st__h i batch b drop_last i seed i nshards i shard → DataLoader

@ dl_reset DataLoader dl__h i seed → v

Reshuffle for the next epoch and rewind. The permutation is a pure function of seed — idx is reset to the ordered shard range [base, base+n) BEFORE the shuffle, so the same seed always yields the same order regardless of the loader's prior state.

@ dl_num_batches DataLoader dl__h → i

@ dl_next DataLoader dl__h ( Vec f ) bx ( Vec f ) by → i

Emit the next batch into bx (rows·d) and by (rows·l), overwriting them. Returns the row count (0 at end-of-epoch; the last batch may be partial unless drop_last).

@ dl_last_rows DataLoader dl__h → i

@ dl_free sink DataLoader dl → v

Let go of dl now rather than at the end of its owner's scope.