data.nu

data.nu — a training DataLoader: batching + seeded shuffle + epochs + drop-last + worker sharding, over an in-memory dataset OR one streamed off disk.

A dataset is n examples, each d f64 features and l f64 labels (l may be 0). In memory it is two flat vectors (x: n·d, y: n·l). On disk it is the .ndf format — a magic + n/d/l header, then n rows of (d+l) little-endian f64 (x then y) — read with random-access preads so a corpus larger than RAM never loads whole.

A DataLoader holds a permutation of its shard's example indices; dl_next gathers the next batch rows (in memory: a copy; streaming: one pread per row) into caller-provided vectors and returns the row count (0 = exhausted). dl_reset reshuffles for the next epoch. Sharding partitions [0,n) into contiguous, exactly-once slices — shard s of N gets [s·n/N, (s+1)·n/N).

( data_new x y n d l ) → DataSet (takes ownership of x,y) ( dl_new ds batch drop_last seed ) → DataLoader ( dl_new_shard ds batch drop_last seed nshards shard ) → DataLoader ( dl_next dl bx by ) → i (rows; 0 = end) ( dl_reset dl seed ) → v (next epoch) ( dl_num_batches dl ) → i ( data_save_ndf path ds ) → !v String ( ndf_open path ) → !NdfStream String ( dl_stream st batch drop_last seed ) → DataLoader ( dl_stream_shard st batch drop_last seed nshards shard ) → DataLoader

API

: DataSet

: DataSet {
    ( Vec f ) x  // n·d features, row-major
    ( Vec f ) y  // n·l labels (empty when l == 0)
    i n
    i d
    i l
}

@ data_new ( Vec f ) x ( Vec f ) y i n i d i l → *DataSet

Take ownership of x (n·d) and y (n·l); data_free releases both.

@ data_free * DataSet ds → v

@ data_n * DataSet ds → i

@ data_d * DataSet ds → i

@ data_l * DataSet ds → i

: i __NDF_HDR 28

── .ndf on-disk format ──────────────────────────────────────────────── bytes: 'N' 'D' 'F' '1' | u64 n | u64 d | u64 l | rows((d+l) f64 LE)…

: NdfStream

: NdfStream {
    File f
    i n
    i d
    i l
    i data_off
}

@ data_save_ndf s path * DataSet ds → !v String

@ ndf_open s path → !*NdfStream String

@ ndf_close * NdfStream st → v

@ ndf_n * NdfStream st → i

@ ndf_d * NdfStream st → i

@ ndf_l * NdfStream st → i

@ ndf_read_row * NdfStream st i idx ( Vec f ) x_out ( Vec f ) y_out → b

Read example idx: append its d features to x_out and l labels to y_out.

: DataLoader

: DataLoader {
    * DataSet ds  // in-memory source (0 when streaming)
    * NdfStream st  // streaming source (0 when in-memory)
    i n  // examples in this shard
    i d
    i l
    i base  // absolute index of this shard's first example
    i batch
    b drop_last
    b shuffle
    ( Vec i ) idx  // permutation of [base, base+n)
    i pos
    i last_rows
}

@ dl_new * DataSet ds i batch b drop_last i seed → *DataLoader

Full in-memory dataset (one shard). seed <= 0 disables shuffling.

@ dl_new_shard * DataSet ds i batch b drop_last i seed i nshards i shard → *DataLoader

@ dl_stream * NdfStream st i batch b drop_last i seed → *DataLoader

Streaming dataset (one shard).

@ dl_stream_shard * NdfStream st i batch b drop_last i seed i nshards i shard → *DataLoader

@ dl_reset * DataLoader dl i seed → v

Reshuffle for the next epoch and rewind. The permutation is a pure function of seed — idx is reset to the ordered shard range [base, base+n) BEFORE the shuffle, so the same seed always yields the same order regardless of the loader's prior state.

@ dl_num_batches * DataLoader dl → i

@ dl_next * DataLoader dl ( Vec f ) bx ( Vec f ) by → i

Emit the next batch into bx (rows·d) and by (rows·l), overwriting them. Returns the row count (0 at end-of-epoch; the last batch may be partial unless drop_last).

@ dl_last_rows * DataLoader dl → i

@ dl_free * DataLoader dl → v