mlp.nu

mlp — a trainable multi-layer perceptron in pure NURL.

Dense layers with ReLU hidden activations and a linear output, trained with minibatch Adam on squared loss — the same recipe as sklearn's MLPRegressor, and deliberately faithful to it:

• Glorot uniform init: W, b ~ U(−√(6/(fan_in+fan_out)), +√(…)),

drawn from a seedable PRNG (std/rng) — a fixed seed gives the same network on every platform.

• squared loss with L2: grad = (AᵀΔ + α·W) / B per batch of B.
• Adam with bias correction: lr_t = lr·√(1−β2ᵗ)/(1−β1ᵗ).
• batch_size = min(200, n) — sklearn's "auto".
• early stopping: a validation split (val_frac) is held out once,

the epoch loop stops after n_no_change epochs without the validation MSE improving by > tol, and the BEST weights seen are restored (not the last). With early_stop off the same patience applies to the training loss, mirroring sklearn. (sklearn scores R² on the validation set; for a fixed split maximising R² is minimising MSE, so the criterion is the same up to tol's scale.)

An autoencoder is just ( mlp_train m X n d X d cfg ) — input as target.

Layout: one flat weight buffer + one flat bias buffer for the whole network (struct-of-arrays, like iforest's node arena). Layer l maps sizes[l] → sizes[l+1]; its weights are row-major n_out×n_in at w_off[l], its biases at b_off[l]. Hot loops read through raw pointers (vec_data) and write with vec_set — the fft.nu idiom.

Persistence: mlp_save/mlp_load round-trip through JSON with every f64 stored as its bit pattern (i64) — exact, platform-independent restore (nurl_str_float renders ~6 significant digits, so decimal text would silently perturb a trained model).

API

: Mlp

: Mlp {
    i n_layers  // weight layers = len(sizes) − 1
    ( Vec i ) sizes  // neuron counts, input first, output last
    ( Vec i ) w_off  // offset of layer l's weights in `w`
    ( Vec i ) b_off  // offset of layer l's biases in `b`
    ( Vec i ) a_off  // offset of layer l's activations in a scratch
    i n_w  // total weight count
    i n_b  // total bias count
    i n_a  // total activation count (all layers, one sample)
    ( Vec f ) w
    ( Vec f ) b
    // Adam state, same shapes as w / b
    ( Vec f ) mw
    ( Vec f ) vw
    ( Vec f ) mb
    ( Vec f ) vb
}

: MlpCfg

: MlpCfg {
    f lr  // Adam initial learning rate (sklearn: 0.001)
    f alpha  // L2 penalty (sklearn: 0.0001)
    i max_iter  // epochs (sklearn: 200; sklearn AE recipes use 500)
    i batch  // minibatch size; 0 = min(200, n) ("auto")
    b early_stop  // hold out a validation split and stop on plateau
    f val_frac  // validation fraction (sklearn: 0.1)
    i n_no_change  // patience in epochs (sklearn: 10)
    f tol  // minimum improvement (sklearn: 1e-4)
    i seed  // PRNG seed (init + shuffles + the val split)
    b verbose  // print per-epoch losses
}

: MlpTrain

: MlpTrain {
    i n_iter  // epochs actually run
    f loss  // final training MSE (plain mean squared error)
    f best_val  // best validation MSE (early_stop) or best train MSE
    b stopped_early  // the patience tripped before max_iter
}

What mlp_train reports back.

: MinMax

: MinMax {
    i n_cols
    ( Vec f ) lo
    ( Vec f ) hi
}

Min–max scaler: x' = (x − lo) / (hi − lo), zero-range columns → 0.

@ _mlp_fget ( Vec f ) v i idx → f

@ _mlp_iget ( Vec i ) v i idx → i

@ mlp_new ( Vec i ) sizes i seed → Mlp

A fresh network with Glorot-uniform weights. sizes lists every layer width, input first (e.g. [8 64 32 64 8] for the classic autoencoder); the caller keeps ownership of sizes (it is copied).

@ mlp_cfg_default → MlpCfg

@ mlp_free Mlp m → v

@ mlp_predict Mlp m ( Vec f ) x → ( Vec f )

Predict one row (a ( Vec f ) of the input width) → an OWNED output vec.

@ mlp_mse Mlp m ( Vec f ) X i n i d ( Vec f ) Y → f

Mean squared error of the network over rows X (n×d row-major) against targets Y (n×dout row-major): mean over every output element.

@ mlp_train Mlp m ( Vec f ) X i n i d ( Vec f ) Y i dout MlpCfg cfg → MlpTrain

Train on X (n×d, row-major) against Y (n×dout). For an autoencoder pass X for Y and d for dout. Returns the training report; the model is updated in place (best-validation weights when early_stop is on).

: MlpFit

: MlpFit {
    Mlp model
    MlpTrain report
}

@ mlp_fit ( Vec i ) sizes ( Vec f ) X i n i d ( Vec f ) Y i dout MlpCfg cfg i restarts → MlpFit

@ mlp_save Mlp m → String

Serialise the trained network (weights + biases; Adam state is NOT persisted — a loaded model predicts, or fine-tunes from a fresh optimiser). Returns an OWNED String of JSON.

@ mlp_load s src → ?Mlp

Rebuild a network from mlp_save output. F on malformed input.

@ minmax_fit ( Vec f ) X i n i d → MinMax

@ minmax_apply MinMax mm ( Vec f ) X i n → v

Scale rows in place: x' = (x − lo) / (hi − lo); zero-range columns → 0.

@ minmax_save MinMax mm → String

@ minmax_load s src → ?MinMax

@ minmax_free MinMax mm → v

---

gradfit.nu

mlp/gradfit.nu — the sklearn recipe over the grad autograd engine.

mlp_train_grad / mlp_fit_grad run the SAME recipe as mlp_train / mlp_fit — the same seeded PRNG call order (Glorot init, the one full shuffle, the per-epoch train-region shuffles), the same validation split, minibatch bounds, early-stopping/patience/best-weights logic and restart selection — but the forward pass, the backward pass and Adam come from grad: the backward is DERIVED from the graph, not hand-written.

The per-batch loss reproduces the hand path's gradient exactly in form:

L = (Σ e² + α · Σ‖W‖²) / (2·B) ⇒ dL/dW = (Σ δ·a + α·W) / B — precisely mlp_train's Adam input (sklearn's _backprop, L2 on weights only, batch divide after).

Results are recipe-equal, not bit-equal, to the hand path (batched matmuls accumulate in a different order than per-row loops); the sklearn oracle passes against BOTH engines. The hand path stays the default until grad's GPU backward (M5) can replace anomaly's aegpu mirror wholesale — then the duplication goes, with no window where any downstream guarantee breaks.

The trained model lands back in the ordinary Mlp struct (weights transposed between mlp's [fout,fin] rows and the tape's [fin,fout] matmul layout), so mlp_predict / mlp_save / mlp_load work unchanged. Adam moments stay in the optimizer; the model's mw/vw/mb/vb remain zero (mlp_save never persisted them).

API

@ _gf_wparam * GTape tp Mlp m i l → GVar

Layer l's weights as a tape parameter in matmul layout: tape[ifout+j] = m.w[wo + jfin + i] (mlp stores [fout,fin] rows).

@ _gf_bparam * GTape tp Mlp m i l → GVar

@ _gf_rows_const * GTape tp ( Vec f ) X i d ( Vec i ) idx i from i to → GVar

Rows idx[from..to) of the row-major table X (n×d) as a [count,d] const.

@ _gf_forward * GTape tp GVar xb ( Vec GVar ) ws ( Vec GVar ) bs i nl → GVar

The network forward on the tape: relu hidden layers, linear output.

@ _gf_write_back Mlp m * GTape tp ( Vec GVar ) ws ( Vec GVar ) bs i nl → v

Copy the (tape-layout) trained values back into the Mlp struct.

@ _gf_snapshot * GTape tp ( Vec GVar ) ws ( Vec GVar ) bs i nl ( Vec f ) dst → v

Snapshot every parameter's current tape value into one flat buffer (tape layout, params in ws-then-bs order) — the best-weights store.

@ _gf_restore * GTape tp ( Vec GVar ) ws ( Vec GVar ) bs i nl ( Vec f ) src → v

Restore a snapshot into the live tape parameters.

@ mlp_train_grad Mlp m ( Vec f ) X i n i d ( Vec f ) Y i dout MlpCfg cfg → MlpTrain

mlp_train's recipe, grad engine. Updates m in place, returns the report.

@ mlp_fit_grad ( Vec i ) sizes ( Vec f ) X i n i d ( Vec f ) Y i dout MlpCfg cfg i restarts → MlpFit

mlp_fit's restart selection, grad engine.