service.nu

anomaly/service.nu — HTTP/JSON service (milestone M6).

Thin: parse JSON → call the library → serialise. The routes and the request/response shapes mirror the Python reference service so existing dashboards keep working:

POST /detect/<model> ingest + verdict (202 warming) POST /detect_only/<model> score only, no state change GET|POST /force_train/<model> retrain now POST /detect_anomalies batch-score a CSV file GET /models/dynamic list models + metadata GET /models/dynamic/<model>/metadata metadata GET /models/dynamic/<model>/data recent points (?limit=N|all) POST /models/dynamic/<model>/reset drop data+forests, keep name DELETE|GET /delete_model/<model> delete entirely PUT /api/dynamic/<model>/schedule retraining schedule POST /api/dynamic/<model>/finetune recalibrate margins

Model names must match ^[a-zA-Z0-9_]+$ (400 otherwise, same message as the reference). Divergence from the reference: /detect_anomalies scores the CSV with a self-trained stateless forest (the reference's separate "static model" family doesn't exist here); passing model_name is a 400.

When a web root is configured (anomaly_service_set_webroot), the router also serves the self-contained dashboard pages from disk:

GET / | /modelmanager.html | /modeltrainer.html | /visualize.html | /anomalies.html

With no web root these routes 404 and the service is API-only.

The router is exposed separately from the socket (anomaly_service_router

• router_handle) so every route is testable without networking.

API

: ~ s g_an_root .``

Store root used by every handler; set once before serving.

@ anomaly_service_set_root s root → v

: ~ s g_an_webroot ``

Directory holding the dashboard HTML (modelmanager.html, etc). Empty = static serving disabled (API-only). Set once before serving.

@ anomaly_service_set_webroot s root → v

@ anomaly_service_router → Router

@ anomaly_serve s host i port → i

Serve until the listener errors. Returns a process exit code.

---

dynamic.nu

anomaly/dynamic.nu — dynamic streaming models (milestones M4 + M5).

The headline feature: a named model that is created on first use, ingests one raw JSON point at a time, and trains itself once enough history has accumulated. Each model keeps:

• a bounded ring of raw points (max_points, default 150 000) persisted

as data.jsonl — raw records, so retrains learn new categories;

• schedule-driven retraining: whenever the lifetime point count

(n_seen, monotonic — unaffected by ring eviction) reaches the next training mark, every enabled version retrains; the mark then advances by schedule.below_max (default 50) or, once the ring is full, schedule.at_max (default 1000);

• one forest per enabled version (M5): each version trains on its own

time window of the ring (window_min minutes back from "now", or the last window_pts points), falling back to the most recent min_points rows when the window is too thin;

• one shared scaler, fit over the full ring at each train.

Verdicts follow the reference service: a point is anomalous if ANY enabled version flags it (decision_function <= -margin); the reported score is the most severe (lowest) decision_function. Before min_points points (default 50) the model is warming up: ready = false, no verdict.

Every public entry point has an _at variant taking now in unix seconds — the injectable clock that makes window filtering and stamped timestamps reproducible in tests. The plain variants read the wall clock.

API

: VerVerdict

: VerVerdict {
    String vvname
    b anomaly
    f score
    f margin
}

One version's slice of a verdict.

: Verdict

: Verdict {
    b ready
    b anomaly
    f score
    ( Vec VerVerdict ) versions
}

The aggregate verdict for one point (SPEC §5.4).

: Model

: Model {
    Store store
    String mname
    *Meta meta
    ( Vec String ) lines
    ( Vec i ) times
    ( Vec VerModel ) forests
    Scaler sc
    i next_train_at
    i min_points
    i max_points
}

A live dynamic model. Obtain with model_open, release with model_free.

@ verdict_free Verdict vd → v

@ model_open_at Store st s name i now → *Model

Open (or create) the named model in the store: load metadata, the point ring, and every trained version's forest. Create-on-first-use: a missing model is initialised with fresh metadata and persisted immediately.

@ model_open Store st s name → *Model

@ model_free *Model mo → v

@ model_set_limits *Model mo i min_pts i max_pts → v

Test hook: shrink the warm-up / ring limits so eviction and scheduling are exercisable without 150 000 points.

@ model_metadata *Model mo → *Meta

@ model_n_points *Model mo → i

@ model_is_trained *Model mo → b

@ model_force_train_at *Model mo i now → i

Retrain every enabled version from the ring, as of now. Re-encodes the raw ring (so new categories/columns learned since the last train enter the feature order), refreshes the authoritative feature order, refits the shared scaler over the full ring, then trains each version on its window. Returns the number of ring points used (0 = not enough data, no change).

@ model_force_train *Model mo → i

@ model_ingest_at *Model mo Json raw i now → !Verdict String

Add one raw point: encode (learning), stamp timestamp, append to the ring (evicting the oldest at capacity), persist, retrain if the schedule says so, then score it. Errors (bad numeric / timestamp values) leave the model completely untouched.

@ model_ingest *Model mo Json raw → !Verdict String

@ model_detect_only *Model mo Json raw → !Verdict String

Score without ingesting: no metadata learning, no ring append, no retrain, no disk writes. Unknown columns/categories project to zeros.

: FtVer

: FtVer {
    String ftname
    f min_score
    f old_margin
    f new_margin
}

One version's fine-tune outcome.

: FineTuneReport

: FineTuneReport {
    ( Vec FtVer ) items
}

@ finetune_free FineTuneReport rep → v

@ model_finetune *Model mo → FineTuneReport

Recalibrate every trained version's decision margin against the observed ring: margin becomes 95% of the magnitude of the most-negative decision_function over the ring, so the most anomalous point seen so far lands just inside the anomaly band. (This is the documented INTENT of the reference's finetune — its own accumulator never updates due to an inverted comparison against -inf, so it never adjusts anything; we implement what the code meant, in place, per SPEC §5.2.) Margins are persisted and take effect immediately. Returns per-version details.

@ model_reset *Model mo → v

Drop all data and trained forests but keep the model's name, schedule and version configs. Learned columns/categories/features/scaler reset.

@ model_delete Store st s name → b

Delete a model from the store entirely (the Model handle, if any, should be freed separately with model_free).

@ model_set_margin *Model mo s vname f margin → b

Set one version's decision margin in the metadata (persisted, effective immediately at scoring — no retrain needed). Returns F for an unknown version name.

@ model_set_schedule *Model mo i below_max i at_max → v

Update the retraining schedule (persisted immediately).

---

model.nu

anomaly/model.nu — the per-point scoring core (milestone M2).

Wraps the iforest package's forest behind the M1 preprocessing layer and the scikit-learn decision conventions the reference service uses:

score_samples(x) = -iforest_score(x) (more negative = worse) decision_function(x) = score_samples(x) - offset_ offset_ = -0.5 for contamination "auto" = percentile(score_samples of the training set, 100 * contamination) otherwise predict(x) == -1 ⇔ decision_function(x) < 0 version verdict : decision_function(x) <= -decision_margin

A VerModel is one trained forest + its offset and margin — the unit that M4/M5 instantiate per time window. Training and the bulk/batch scoring paths (GPU-accelerated when available) live in src/score.nu; this file is the small, dependency-light core they build on.

API

: i ANOM_SEED 42

The reference's fixed random_state: a given training set always yields a byte-identical forest, hence identical scores, on every platform.

: VerModel

: VerModel {
    String vname
    IForest forest
    f offset
    f margin
    i n_cols
    b trained
}

One trained model version: forest + decision offset + margin.

@ anom_decision VerModel vm ( Vec f ) scaled_point → f

decision_function of an already-standardised point.

@ anom_decision_row VerModel vm ( Vec f ) scaled i r → f

decision_function of row r of an already-standardised matrix.

@ anom_is_anomaly VerModel vm f df → b

The per-version verdict convention: below (or at) the margin ⇒ anomaly.

@ anom_vermodel_free VerModel vm → v

---

csvdata.nu

anomaly/csvdata.nu — numeric CSV → row-major matrix (batch inputs).

Same forgiving shape as the iforest CLI parser: split lines, split on the delimiter, keep numeric fields; the first data row fixes the column count, later rows with a different numeric-field count are skipped. With has_header, the first line's names are captured (for batch reports that echo column values by name) instead of being parsed as data.

API

: AnomCsv

: AnomCsv {
    ( Vec f ) data
    i rows
    i cols
    ( Vec String ) headers
}

Parsed matrix: data is row-major rows*cols; headers is empty when the input had no header row.

@ anom_csv_free AnomCsv ds → v

@ anom_parse_csv s input s delim b has_header → AnomCsv

---

prep.nu

anomaly/prep.nu — feature preprocessing, standardisation, model metadata.

This is milestone M1 of the anomaly package (see SPEC.md): the pure, I/O-free foundation everything else builds on.

• A model's metadata (*Meta) records what it has learned about the

shape of its input: which columns exist, their types (numeric / categorical / timestamp), the categories seen so far, the authoritative feature order, and the scaler parameters. It round-trips through JSON.

• anomaly_preprocess turns one raw JSON record into named numeric

features, updating the metadata as new columns / categories appear: numeric passthrough, categorical → deterministic one-hot (categories kept sorted), ISO-8601 timestamp → hour/day/month/weekday.

• anomaly_project pins a named feature set onto the model's frozen

feature order: missing features become 0, unknown extras are dropped. This is the feature-order stability rule that keeps one-hot columns aligned across retrains.

• Scaler is the StandardScaler analogue: per-feature zero-mean /

unit-variance, with zero-variance features passing through unscaled.

Column types are detected once, on first sight, then frozen in metadata: a value that parses as a number is numeric, else a value that parses as ISO-8601 is a timestamp, else it is categorical (mirrors the Python reference's float()/fromisoformat()/str fallback chain).

API

: i COL_NUMERIC 0

: i COL_CATEGORICAL 1

: i COL_TIMESTAMP 2

: i ANOM_MIN_POINTS 50

Reference defaults (carried over verbatim from the Python service).

: i ANOM_MAX_POINTS 150000

: i ANOM_SCHED_BELOW 50

: i ANOM_SCHED_AT_MAX 1000

: VerCfg

: VerCfg {
    String vname
    i window_min
    i window_pts
    i n_estimators
    i max_samples
    f contamination
    f decision_margin
    b enabled
}

Config of one time-window model version. window_min filters training data to the last N minutes (0 = no time filter); window_pts caps it to the last N points (0 = no cap; used by timevector). contamination < 0 means "auto" (offset pinned at -0.5, the sklearn convention).

: Meta

: Meta {
    String name
    String created
    ( Vec String ) cols
    ( Vec i ) kinds
    ( Vec ( Vec String ) ) cats
    ( Vec String ) feats
    ( Vec f ) sc_mean
    ( Vec f ) sc_std
    i sched_below
    i sched_at_max
    i n_seen
    i last_trained
    ( Vec VerCfg ) versions
}

Everything a model knows about its input shape. Heap-allocated (*Meta) so counters and handles can be updated through any reference. cols / kinds / cats are parallel: cats[i] is the sorted category list of column i (empty unless categorical). feats is the authoritative feature order once non-empty (snapshotted at each train by meta_refresh_feats); until then the model is "unfrozen" and the order is derived on demand.

: EncPoint

: EncPoint {
    ( Vec String ) names
    ( Vec f ) vals
}

One preprocessed record: parallel (feature name, value) pairs in encounter order. Project onto a feature order with anomaly_project.

: Scaler

: Scaler {
    ( Vec f ) mean
    ( Vec f ) inv_std
}

Per-feature standardisation: y = (x - mean) * inv_std. Zero-variance features store inv_std = 1 so they pass through centred but unscaled.

@ meta_default_versions → ( Vec VerCfg )

@ meta_new s name s created → *Meta

@ meta_free *Meta m → v

@ meta_is_frozen *Meta m → b

A model is "frozen" once it has an authoritative feature order (set at first train). Before that, feature order is derived from the metadata.

@ meta_derived_feats *Meta m → ( Vec String )

The feature order implied by the current metadata: columns in first-seen order, each expanded canonically (categoricals over their sorted categories). Deterministic for a given metadata state. Owned result.

@ meta_refresh_feats *Meta m → v

Snapshot the derived feature order as authoritative (called at train time). From now on scoring projects onto exactly this vector.

@ enc_free EncPoint p → v

@ enc_find EncPoint p s name → i

Index of feature name in the encoded point, or -1.

@ anomaly_preprocess *Meta m Json raw → !EncPoint String

Encode one raw JSON record into named numeric features, updating the metadata as new columns / categories appear. The reserved key timestamp is the point's own clock and is never a feature. Numeric parse failure and bad timestamps are hard errors (owned message).

@ anomaly_preprocess_ro *Meta m Json raw → !EncPoint String

Read-only encode: never touches the metadata. Unknown columns are skipped, unseen categories one-hot to all-zeros — exactly what the frozen-feature projection would do with them anyway. For detect-only paths that must not mutate model state.

@ anomaly_project EncPoint p ( Vec String ) feats → ( Vec f )

Project an encoded point onto an authoritative feature order: missing features become 0, features not in feats are dropped. Owned result.

@ scaler_fit ( Vec f ) data i n_rows i n_cols → Scaler

Fit per-feature mean and 1/std over a row-major matrix (population variance, like sklearn's StandardScaler). Zero-variance features get inv_std = 1 so they centre but never divide by ~0.

@ scaler_apply Scaler sc ( Vec f ) point → v

Standardise one point in place: x → (x - mean) * inv_std.

@ scaler_apply_matrix Scaler sc ( Vec f ) data i n_rows i n_cols → v

Standardise a whole row-major matrix in place.

@ scaler_free Scaler sc → v

@ meta_set_scaler *Meta m Scaler sc → v

Persist a fitted scaler into metadata (stored as mean + std; zero variance is stored as std = 1, matching its inv_std = 1).

@ meta_scaler *Meta m → Scaler

Rebuild a usable Scaler from persisted metadata. Owned result.

@ meta_to_json *Meta m → Json

Serialise metadata to an owned Json object (fixed field order, so the same metadata always stringifies identically).

@ meta_from_json Json j → ?*Meta

Parse metadata back from JSON. None on malformed shape (missing/mistyped required fields); the partially-built Meta is freed on failure.

@ meta_to_json_str *Meta m → String

Convenience: metadata → compact JSON text (owned).

@ meta_from_json_str s src → ?*Meta

Convenience: JSON text → metadata; None on parse or shape errors.

---

main.nu

anomaly — streaming anomaly detection from the command line.

anomaly detect <model> key=val ... ingest one point → verdict JSON anomaly score <model> key=val ... score only (no state change) anomaly batch [-f FILE] [-H] score a CSV (index<TAB>score) anomaly train <model> force a retrain now anomaly reset <model> drop data+forests, keep the name anomaly rm <model> delete the model entirely anomaly ls list models in the store anomaly info <model> dump model metadata (pretty JSON) anomaly serve [--addr HOST:PORT] run the HTTP/JSON service + dashboard [--webroot DIR] (dashboard HTML dir; auto-located)

Values in key=val are auto-typed: numbers are numeric features, ISO-8601 strings are timestamps, anything else is categorical (one-hot). Verdicts print as one JSON object per invocation. The store lives under --store DIR (default $ANOMALY_HOME, else ~/.anomaly).

API

@ pline s x → v

@ main → i

---

score.nu

anomaly/score.nu — bulk scoring, version training, and the stateless batch path (milestones M2 + M7's GPU acceleration).

Bulk scoring routes through the gpu package when profitable:

• gpu_open picks CUDA when a device is present, else the gpu

package's CPU backend (the same kernel compiled by the host C++ compiler and run under OpenMP). If neither is available — no GPU, no C++ compiler — scoring silently stays on the pure-NURL loop, so a bare machine behaves exactly like pre-GPU builds.

• Output is BIT-IDENTICAL across all three paths, by construction: the

kernel only walks the trees and accumulates f64 path lengths in the same order as the pure loop, using per-leaf c(size) values precomputed on the host by the very same iforest_avg_path; the nonlinear finish (2^(-avg/c), the offset subtraction) runs in NURL either way. IEEE-754 f64 adds in a fixed order give the same bits on every backend — so backend choice can never change a verdict.

ANOMALY_GPU=0 disables the accelerator outright; NURL_GPU=cpu (a gpu package knob) forces its CPU backend on a CUDA machine. Batches under ANOM_GPU_MIN_ROWS rows skip the accelerator — upload latency would beat the win, and bit-equality makes the switch invisible.

API

: i ANOM_GPU_MIN_ROWS 128

Below this many rows the pure loop wins on latency. Bit-equality of the paths makes the threshold unobservable in results.

: BatchReport

: BatchReport {
    i total_rows
    i anomaly_count
    f anomaly_percentage
    ( Vec i ) anomaly_indices
    b has_anomalies
    ( Vec f ) scores
}

Batch verdict over a matrix (see SPEC §6): anomaly ⇔ predict == -1, i.e. decision_function < 0 (margin 0), matching the reference batch path.

: ~ i g_ag_state 0

: ~ i g_ag_ord 0

: ~ i g_ag_dev 0

: ~ i g_ag_ctx 0

: ~ i g_ag_kmod 0

: ~ i g_ag_kfun 0

@ anom_gpu_close → v

Release the device + kernel (tests call this so leak checkers see a closed shop; a long-running service just keeps the singleton).

@ anom_gpu_engine → s

Which engine bulk scoring would use right now: cuda, cpu (the gpu package's host backend) or none (pure NURL loop).

@ anom_scores_cpu VerModel vm ( Vec f ) scaled i n_rows i n_cols → ( Vec f )

Pure-NURL reference path: iforest_score per row.

@ anom_scores_gpu VerModel vm ( Vec f ) scaled i n_rows i n_cols → ?( Vec f )

Accelerated path. None when the accelerator is unavailable or any step fails (caller falls back; failures after a good probe are reported).

@ anom_scores VerModel vm ( Vec f ) scaled i n_rows i n_cols → ( Vec f )

Raw iforest scores for every row of an already-standardised matrix, through the best available engine. Bit-identical on all engines.

@ anom_decisions VerModel vm ( Vec f ) scaled i n_rows i n_cols → ( Vec f )

decision_function for every row: -score - offset, same expression as anom_decision.

@ anom_train_version ( Vec f ) scaled i n_rows i n_cols VerCfg cfg → VerModel

Train one version's forest over an ALREADY-STANDARDISED row-major matrix. cfg.max_samples is clamped to the row count; contamination < 0 = "auto" (offset pinned at -0.5), else offset_ is the 100*c percentile of the training set's score_samples — so predict == -1 flags ~c of training.

@ anomaly_batch ( Vec f ) data i n_rows i n_cols VerCfg cfg → BatchReport

Fit a scaler over the raw matrix, train one version on the standardised copy, then score every row. Anomaly ⇔ decision_function <= -cfg.margin (margin 0 reproduces the reference's predict == -1 batch rule).

@ anomaly_report_free BatchReport rep → v

---

store.nu

anomaly/store.nu — persistence (milestone M3).

On-disk layout, one directory per model under the store root:

<root>/<name>/ metadata.json M1 metadata (feature order, scaler, versions…) data.jsonl raw ingested points, one JSON record per line (raw records — not projected vectors — so a retrain can pick up new categories/columns) version_<v>.forest one binary forest blob per trained version

The forest blob ("ANOMFOR1") is a little-endian dump of the iforest node arena plus the version's decision offset/margin. Loading re-validates every structural invariant (lengths, index ranges, node-count caps), so a corrupt or truncated blob comes back as None — never undefined behaviour.

Writes are atomic: serialise to <file>.tmp, then rename(2) over the final name, so a crash mid-write can't leave a half-written model.

API

: i ANOM_BLOB_MAX_NODES 200000000

Refuse to load blobs claiming more than this many arena nodes / trees / name bytes — bounds untrusted counts before any allocation.

: i ANOM_BLOB_MAX_TREES 100000

: i ANOM_BLOB_MAX_NAME 4096

: BlobRd

: BlobRd {
    ( Vec u ) buf
    i pos
    b ok
}

@ vermodel_to_bytes VerModel vm → ( Vec u )

Serialise one trained version (forest + offset + margin) to owned bytes.

@ vermodel_from_bytes ( Vec u ) buf → ?VerModel

Parse a forest blob. None on any structural violation: bad magic, short buffer, absurd counts, mismatched array lengths, out-of-range indices.

: Store

: Store {
    String root
}

@ store_open s root → Store

@ store_free Store st → v

@ store_exists Store st s name → b

A model exists iff its metadata file does.

@ store_list Store st → ( Vec String )

Sorted names of every model in the store (dirs with a metadata.json).

@ store_save_meta Store st s name *Meta m → b

Persist metadata (creates the model directory as needed).

@ store_load_meta Store st s name → ?*Meta

@ store_save_forest Store st s name VerModel vm → b

Persist one trained version's forest blob.

@ store_load_forest Store st s name s vname → ?VerModel

Load one version's forest; None if absent or corrupt. The blob's embedded version name must match the requested one — a renamed or content-tampered file is treated as corrupt, not trusted.

@ store_delete_forest Store st s name s vname → v

Remove a trained version's blob (used by reset). Missing file is fine.

@ store_delete Store st s name → b

Delete a model entirely (directory and everything in it).

@ store_append_point Store st s name s line → b

Append one line (the record's compact JSON, no newline).

@ store_write_points Store st s name ( Vec String ) lines → b

Rewrite the whole log (ring eviction / reset).

@ store_load_points Store st s name → ( Vec String )

Load the log as owned lines (empty vec if the file doesn't exist).