service.nu

anomaly/service.nu — HTTP/JSON service (milestone M6).

Thin: parse JSON → call the library → serialise. The routes and the request/response shapes mirror the Python reference service so existing dashboards keep working:

POST /detect/<model> ingest + verdict (202 warming) POST /detect_only/<model> score only, no state change GET|POST /force_train/<model> retrain now POST /detect_anomalies batch-score a CSV file GET /models/dynamic list models + metadata GET /models/dynamic/<model>/metadata metadata (+ autoencoder state) PUT /models/dynamic/<model>/metadata edit schedule / version configs GET /models/dynamic/<model>/data recent points (?limit=N&from=&to=&last=&fields=) GET /models/dynamic/<model>/anomalies scored ring (cached, filtered) GET /models/dynamic/<model>/calibration alert rates vs margins POST|GET /models/dynamic/<model>/labels a reader's word on a stored point POST /models/dynamic/<model>/import a CSV/JSON/JSONL file of history POST /models/dynamic/<model>/fork a new model trained on a slice of this one's history POST /models/dynamic/<model>/reset drop data+forests, keep name DELETE|GET /delete_model/<model> delete entirely PUT /api/dynamic/<model>/schedule retraining schedule POST /api/dynamic/<model>/finetune set margins from a target alert rate POST /train/autoencoder/<model> train the autoencoder version POST /mcp the same API for a language model (src/mcp.nu) GET /.well-known/oauth-protected-resource[/mcp] where /mcp callers get a token

Model names must match ^[a-zA-Z0-9_]+$ (400 otherwise, same message as the reference). Divergence from the reference: /detect_anomalies scores the CSV with a self-trained stateless forest (the reference's separate "static model" family doesn't exist here); passing model_name is a 400.

When a web root is configured (anomaly_service_set_webroot), the router also serves the self-contained dashboard pages from disk:

GET / | /modelmanager.html | /modeltrainer.html | /visualize.html | /anomalies.html | /admin.html | /oauth/callback | /auth.js | /favicon.svg

With no web root these routes 404 and the service is API-only.

The router is exposed separately from the socket (anomaly_service_router

• router_handle) so every route is testable without networking.

API

: ~ s g_an_root .``

Store root used by every handler; set once before serving.

@ anomaly_service_set_root s root → v

: ~ s g_an_webroot ``

Directory holding the dashboard HTML (modelmanager.html, etc). Empty = static serving disabled (API-only). Set once before serving.

@ anomaly_service_set_webroot s root → v

: i AZ_GATE_OK 0

: i AZ_GATE_UNAUTH 401

: i AZ_GATE_FORBID 403

: Gate

: Gate {
    b allowed
    i status
    Principal who
    b creating  // the named model does not exist yet; this call would make it
}

@ _an_flat_alert_json * Model mo → Json

The autoencoder version's own state, which lives outside the metadata (autoencoder.json, not meta.json) and so has no place in meta_to_json. enabled is read back from the metadata: disabling the version mutes the verdict but keeps the trained net. The flatline guard as a reader needs it: the reference the last fit wrote for each numeric column, named rather than positional, and what the margin therefore asks of that column — the run of identical readings that trips it. A column with no reference is listed apart with the reason, so "the guard is on" never has to mean "every column is watched". What the flatline's margin asks of each column, in rows and — when the ring has a step to read it by — in minutes.

@ _an_flat_json * Meta mm → Json

: ~ i g_an_lock 0

The service lock. The server runs a small worker pool so that a handler which WAITS — analyze, for its job — can let go of the service while it does; every other handler holds the lock for its whole run, so the store, the GPU singleton and the authorization layer see one request at a time, exactly as under the single-threaded server. 0 = no pool (tests drive the router directly), and the release/acquire pair is a no-op.

: i AN_ANALYZE_WAIT_DEFAULT 10

POST /api/analyze — the body is the file. Query: format (csv, json, jsonl, fmi; empty = detect), time, tz, calendar, clock as for an import, name to label the result, and wait — how many seconds to hold the call for the result (10 by default, 60 at most, 0 = return the task at once). A result that is not ready in time answers 202 with the task to ask again for.

: i AN_ANALYZE_WAIT_MAX 60

: ~ b g_an_sources_on T

Whether the scheduler thread is started with the server.

@ anomaly_service_set_sources b on → v

@ anomaly_service_router → Router

@ anomaly_serve s host i port → i

Serve until the listener errors. Returns a process exit code. Serve the routes over the http package's App facade: it owns the bind + keep-alive loop, a graceful SIGINT/SIGTERM shutdown, and turns a handler panic into a 500 (rather than dropping the connection). The router itself is still built by anomaly_service_router, so every route stays drivable without a socket in the test suite.

---

dynamic.nu

anomaly/dynamic.nu — dynamic streaming models (milestones M4 + M5).

The headline feature: a named model that is created on first use, ingests one raw JSON point at a time, and trains itself once enough history has accumulated. Each model keeps:

• a bounded ring of raw points (max_points, default 150 000) persisted

as data.jsonl — raw records, so retrains learn new categories;

• schedule-driven retraining: whenever the lifetime point count

(n_seen, monotonic — unaffected by ring eviction) reaches the next training mark, every enabled version retrains; the mark then advances by schedule.below_max (default 50) or, once the ring is full, schedule.at_max (default 1000);

• one forest per enabled version (M5): each version trains on its own

time window of the ring (window_min minutes back from "now", or the last window_pts points), falling back to the most recent min_points rows when the window is too thin;

• one shared scaler, fit over the full ring at each train.

Verdicts follow the reference service: a point is anomalous if ANY enabled version flags it (decision_function <= -margin); the reported score is the most severe (lowest) decision_function. Before min_points points (default 50) the model is warming up: ready = false, no verdict.

Every public entry point has an _at variant taking now in unix seconds — the injectable clock that makes window filtering and stamped timestamps reproducible in tests. The plain variants read the wall clock.

API

: VerVerdict

: VerVerdict {
    String vvname
    b anomaly
    f score
    f margin
    f cfg_margin
    i vv_feat  // the feature this verdict is about (range_guard); -1 = none
}

One version's slice of a verdict. margin is the EFFECTIVE absolute band the score was compared against (the invariant score <= -margin ⇒ anomaly holds for every version); cfg_margin is the number stored in the metadata. They differ only for the autoencoder, whose configured margin is relative to its reconstruction threshold — see ANOM_AE_MARGIN.

: FlatOut

: FlatOut {
    b ready  // F when the ring is shorter than the window or nothing is fitted
    f worst  // the largest fraction over the watched features
    i feat  // the feature that gave it; -1 when none is watched
}

What the flatline guard saw (see __an_flat_judge).

: Verdict

: Verdict {
    b ready
    b anomaly
    f score  // the most severe version's decision value (own units)
    f severity  // that version's severity: the aggregate, unit-free
    ( Vec VerVerdict ) versions
}

The aggregate verdict for one point (SPEC §5.4).

@ anom_severity f df f margin → f

The one unit-free number every version shares (SPEC §5.4): how far past its own alert line a decision value sits, in margins — 1.0 exactly on the line, 2.0 twice as far, negative comfortably normal. A margin of 0 gives 1.0 when flagged and 0.0 otherwise. The aggregate score is the score of the version that is most severe by this measure; a plain minimum over decision values would let a forest's ~1e-1 always outrank an autoencoder's ~1e-4 and hide the joint model's alarm.

: Model

: Model {
    Store store
    String mname
    * Meta meta
    ( Vec String ) lines
    ( Vec i ) times
    ( Vec VerModel ) forests
    Scaler sc
    AeModel ae
    b ae_stale  // the net names features the current encoding no longer makes
    * FcModel fc  // the forecast version (src/forecast.nu); untrained handle when none
    i next_train_at
    i min_points
    i max_points
}

A live dynamic model. Obtain with model_open, release with model_free.

: ~ s g_an_actor library``

Who is acting, and how, for the audit log: the service sets the actor from the request's principal, the CLI and the scheduler name themselves; the action is set by the operation that changes margins (finetune, autotune, edit) around its calls to model_set_margin.

: ~ s g_an_action set_margin``

: i ANOM_ACTOR_MAX 200

: f ANOM_FT_RATE 0.01

The share of a window a margin aims to flag when the service, rather than a person, sets it: a file import's first calibration, a data source's first run, the forecast version's first fit, finetune with no rate given.

@ anomaly_set_actor s who → v

The actor is copied into a buffer allocated once for the process (the caller's string lives only as long as its request; a long name is cut). The action is always a literal and is kept as given.

@ _an_set_action s what → v

@ model_audit * Model mo i limit → Json

The newest limit audit entries of a model (all when ≤ 0).

@ verdict_free Verdict vd → v

@ _an_line_ts s line → i

Ingest timestamp of a stored data.jsonl line (0 if unparsable).

@ model_open_at Store st s name i now → *Model

@ model_open Store st s name → *Model

@ model_set_version_window * Model mo s vname i wsize i sstep → b

Set a version's sliding-window geometry (timevector). Takes effect at the NEXT retrain — detect derives the live window from the trained forest's width, so a config change can never desync scoring.

@ model_last_ts * Model mo → i

The newest stored stamp, or 0 on an empty ring.

@ model_first_ts * Model mo → i

The oldest stored stamp, 0 for an empty ring.

@ model_window_bounds * Model mo i from_ts i to_ts → ( Vec i )

A window as a reader must be able to read it: an unbounded end is the newest stored point and an unbounded start the oldest, never a null that says "no end" where the answer is "the end of the data". Both bounds are the ones the rows were actually taken between.

@ model_next_tick * Model mo → i

The tick the NEXT point gets on the count clock: one past the newest.

@ model_now * Model mo → i

"Now" for this model: the wall clock, or on the count clock the newest tick — the moment the last point arrived is the only present it has.

@ model_free * Model mo → v

@ model_set_limits * Model mo i min_pts i max_pts → v

Test hook: shrink the warm-up / ring limits so eviction and scheduling are exercisable without 150 000 points.

@ model_metadata * Model mo → *Meta

@ model_n_points * Model mo → i

@ model_is_trained * Model mo → b

: Hist

: Hist {
    i base
    i n
    i nfeat
    ( Vec f ) x  // n × nfeat, standardised; a row that failed to parse is zeros
    ( Vec i ) ok  // 1 when the row parsed and encoded
    i ae_nfeat
    ( Vec f ) ae_x  // n × ae_nfeat, raw projection onto the autoencoder's own feature order
    ( Vec ( Vec f ) ) dfs  // per forest index: its decision for every row
    ( Vec ( Vec i ) ) dfs_ok  // per forest index: 1 where that decision stands (a timevector forest has none for a row whose window is not all here)
    i fc_nw
    ( Vec f ) fc_z  // n × fc_nw: the forecast version's z per watched feature, by replay (NaN = not judged)
    ( Vec i ) fc_ok  // 1 where the row has forecast z-scores
}

── Encoded history ─────────────────────────────────────────────────── A scan, a calibration and a fine-tune judge every stored row, and each row's verdict needs the row itself, the W−1 rows before it (the timevector window) and every forest's decision for it. Encoding a stored line costs a JSON parse and a name lookup per feature, so ring rows [base, base+n) are encoded ONCE here, and each forest scores the whole matrix in one call — the accelerated path — instead of a row at a time. The numbers are the per-row path's exactly (anom_decisions is bit-identical to anom_decision, and the window is the same projection of the same rows); only the work is shared.

@ _an_ensure_fc_cfg * Model mo → v

Ensure a forecast VerCfg exists (off by default): a model from before the version gains it at its next retrain.

@ model_forecast * Model mo → *FcModel

The forecast version's handle, its j-th model, and its states caught up with the ring (for a forecast from the newest row).

@ model_step * Model mo → i

The ring's step: the median gap between consecutive stored times, in seconds (1 on a count clock; 0 when there are too few rows to say).

@ anomaly_season_of i step → i

The seasonal period, in rows, a step implies: the day for a step up to twelve hours (144 rows at ten minutes, 24 at an hour), the week for a daily step, none otherwise (or on a count clock, whose step says nothing about time).

@ model_forecast_ensure_at * Model mo i now → String

A forecast asked of a model whose forecast version is not trained: fit it now, the season from the ring's step when the version has none set, and switch it on. "" when the version is ready (already, or now); otherwise why not (the model has not trained; no feature to forecast).

: f ANOM_Z80 1.2815515655446004

: f ANOM_Z95 1.959963984540054

@ _an_fc_season_json * FcModel fc Json o → v

The next h values of every watched feature, from the states caught up with the ring, as the API answers them: means, standard errors, the 80 % and 95 % intervals, the time of each step (the newest stored time plus the ring's step, or the row number on a count clock), and each feature's fitted model. The season, said so a reader cannot mistake what is modelled for what was offered: the period in rows, and how many of the fitted models use it. season: 144 beside ARIMA(0,1,0) with s = 0 is not a daily rhythm being watched — it is the number the search was given and every feature declined.

@ model_forecast_json * Model mo i h → Json

@ model_forecast_from_json * Model mo i h i origin → Json

The forecast as it would have been made from ring row origin (inclusive): a copy of every model replayed from ANOM_FC_BURN rows before it, then the next h steps — the same shape as model_forecast_json, with the origin's row and time, so a reader can put the forecast beside what followed. The live states are untouched.

@ model_forecast_backtest * Model mo i h i n → Json

How good the forecasts are, measured: a rolling-origin backtest over the ring's last n origins. A copy of every model is replayed from ANOM_FC_BURN rows before the first origin; at each origin the h-step forecast is made from the state as it then stood and compared with the rows that followed. Per feature and per step: the mean absolute error, the mean absolute percentage error (rows with a reading near zero left out), the 95 % interval's coverage, and the same error for the two forecasts anyone can make without a model — the last value carried forward, and the value one season earlier — with the skill against each (1 − MAE/MAE_baseline: 0 is no better, 1 is perfect, negative is worse). Gaps are skipped.

@ model_forecast_model * Model mo i j → *ArimaModel

@ model_forecast_sync * Model mo → v

@ model_train_forecast * Model mo → String

Train the forecast version now, on the ring as it stands, and switch it on. The model must have trained once (a frozen feature order). Returns the error text ("" = success).

@ model_train_forecast_at * Model mo i now → String

@ model_force_train_at * Model mo i now → i

Retrain every enabled version from the ring, as of now. Re-encodes the raw ring (so new categories/columns learned since the last train enter the feature order), refreshes the authoritative feature order, refits the shared scaler over the full ring, then trains each version on its window. Returns the number of ring points used (0 = not enough data, no change).

@ an_ae_stale * Meta mm AeModel ae → b

The autoencoder's place in the retrain schedule. Its threshold is the p95 reconstruction error of the data it was trained on, and sensor data drifts: a net trained on one week's weather reconstructs the next week's worse across the board, so a never-retrained autoencoder ends up flagging everything, whatever the margin. Opting in (schedule.autoencoder) retrains it with the same hidden layout and pre-filter rate every time the forests retrain. Margins are never touched. Requires an existing trained net: the first training stays an explicit choice, because it fixes the layout. A trained autoencoder is stale when its frozen feature order names a feature the model's current encoding no longer produces — the calendar features changed shape in 0.14.0, and a cycle the training span has not seen twice is left out (see __an_cycle_seen). Projecting a point onto such an order fills the lost features with 0, which for a value that was never 0 in training is a reconstruction error thousands of times the threshold, on every point. A stale net does not score; the next forest retrain replaces it whether or not the schedule says so.

@ model_force_train * Model mo → i

@ _an_flat_window * Meta mm → i

The configured window, in rows.

@ _an_flat_ref_len * Meta mm i j → f

The run length a feature's reference asks for: twice what its training rows reached, never under ANOM_FLAT_MIN_RUN. −1 for a feature that is not watched. The margin is read against THIS, so one number (0.9) means the same thing on a column quantised to whole degrees and on a smooth one beside it.

@ _an_flat_need * Meta mm → i

How far back the guard must look: the collapse window, and enough rows for the longest reference run to be reached (so no watched column has a bar its run can never touch). Bounded by the same cap the fit used to decide what is watchable at all.

@ model_train_autoencoder * Model mo ( Vec i ) hidden f contamination → String

Train the autoencoder from the ring: encode + project the raw points (the same pass model_force_train_at runs, minus the standardising scaler — the AE recipe MinMax-scales after anomaly filtering), then hand the matrix to ae_train_matrix. hidden empty → the layout the trained net already has, so a retrain with no layout given is a retrain and not a silent return to the default; with no net yet, 64-32-64. Explicit by default; with schedule.autoencoder on, every forest retrain repeats it with the same layout and pre-filter (see __an_retrain_ae). Returns the error text ("" = success).

@ model_train_autoencoder_at * Model mo ( Vec i ) hidden f contamination i now → String

@ model_ingest_at * Model mo Json raw i now → !Verdict String

Add one raw point: encode (learning), stamp timestamp, append to the ring (evicting the oldest at capacity), persist, retrain if the schedule says so, then score it. Errors (bad numeric / timestamp values) leave the model completely untouched.

@ model_ingest * Model mo Json raw → !Verdict String

@ model_detect_only * Model mo Json raw → !Verdict String

Score without ingesting: no metadata learning, no ring append, no retrain, no disk writes. Unknown columns/categories project to zeros; a column the trained model knows and the point leaves out is an error, named — a question about a point must carry the whole point.

: ImpRow

: ImpRow {
    i ir_ts
    String ir_line
}

: ImportReport

: ImportReport {
    i accepted
    i rejected
    i stored  // points in the ring afterwards
    b trained
    String err  // non-empty ⇒ nothing was imported
    ( Vec String ) notes
}

@ import_report_free ImportReport r → v

@ model_import_at * Model mo ( Vec Json ) recs i now → ImportReport

Import recs into the model. Records keep their own timestamp when they carry one; the rest are placed at now. On the count clock a record's own stamp is ignored: the file's order is its time, and the rows take the ticks after the newest stored point, one each. Returns what happened.

@ model_import * Model mo ( Vec Json ) recs → ImportReport

: CalVer

: CalVer {
    String cvname
    f cur_margin  // the configured margin (own units)
    i n  // rows this version had a verdict for
    i flagged  // rows the current margin flags
    f worst  // the most negative decision value
    f median
    ( Vec f ) dfs  // ascending
}

One version's decision values over the calibration window.

: CalReport

: CalReport {
    ( Vec CalVer ) items
    i from_ts  // resolved window (0 = unbounded)
    i to_ts
    i n_rows  // rows scored
    i agg_flagged  // rows some enabled version flagged at the current margins
    i excluded  // rows in the window left out: labelled false positives
}

@ cal_free CalReport rep → v

@ round_sig f x i digits → f

Round to digits significant decimal digits, through an exact integer mantissa and a power of ten so the result is the double nearest to the short decimal (the same double the literal "0.106" parses to — division by an exactly representable power of ten is correctly rounded), and therefore prints as that short decimal.

@ round_sig_dir f x i digits i dir → f

dir picks the rounding: 0 nearest, -1 toward zero, +1 away from zero (on the magnitude; the sign is restored afterwards).

@ cal_flagged_at CalVer cv f margin → i

Rows the version would flag at margin: count of dfs <= -margin.

@ cal_margin_for_rate CalVer cv f rate → f

The margin at which a fraction rate of the window is flagged. The request is k = round(rate·n) rows; the margins a sorted list of decision values can supply are its gaps, so when the k-th most negative value is one of a run of TIES (a stuck sensor, a categorical feed, a forest that gives one path length to whole days of identical points) the request falls inside the run and only its two edges are achievable: everything before the run, or the run entire. The closer edge to k wins — a run of 221 equal values at k = 10 is answered with the 9 rows before it, not with 231 — and on a tie between the edges the run is flagged, so the request is at least met. rate 0 asks for a margin just above the worst point. The chosen margin is then rounded to the FEWEST significant digits (2 to 6) that still flag the chosen count, give or take a tenth of it — a margin is a setting a person reads and retypes, and "0.13" is one where the data allows it, "0.1284" where the decision values are packed too densely for fewer digits. Never negative: a negative margin would flag points the forest itself calls normal, and a rate the data cannot supply is answered by the honest count next to the margin, not by a margin below zero.

@ model_calibrate * Model mo i from_ts i to_ts → CalReport

Score every ring row in [from_ts, to_ts] (0 = unbounded) through the live verdict path and collect each version's decision values. Rows are scored as of their own ring position, exactly as the scan does, so a timevector window never sees the future.

: i ANOM_CAL_WINDOW 86400

The window fine-tune and calibration default to: the newest 24 hours of stored data, anchored on the newest stored point rather than the clock, so a model whose feed stopped still calibrates on its last day.

@ model_window_from_last * Model mo i to_ts i last → i

Resolve (from, to, last) the way the HTTP layer spells it: last seconds back from to, or from the newest stored point when to is unbounded.

: FtVer

: FtVer {
    String ftname
    f old_margin
    f new_margin
    i n  // rows in the window with a verdict
    i before  // flagged at old_margin
    i after  // flagged at new_margin
    f worst
    b applied  // F on a dry run, or when the version was filtered out
    i ft_from  // the window's lower bound this version was tuned over
    i ft_n_rows  // ring rows inside that window
    String warning  // why the rate was not met, or why the margin was left alone; "" when all is well
}

One version's fine-tune outcome.

: FineTuneReport

: FineTuneReport {
    ( Vec FtVer ) items
    f rate
    i from_ts
    i to_ts
    i n_rows
    b applied
    i excluded  // labelled false positives left out of the window
}

@ model_autotune_why * Model mo f rate → s

The first calibration of a model that arrived as a whole — a file imported, a source's first run — and has never had its margins set: the forests are trained but the margins are the defaults, which on a weather feed at ten-minute steps flag a third of the ring. Fine-tune once, to rate of the ring, and remember it (tuned_at), so the runs that follow leave the margins to the person: a calibration repeated on every run would fold the real anomalies into the rate. A ring too small for the rate to flag even one row (82 rows at 1 %) is left alone too — a margin set to flag nothing would flag nothing until the data left its range — and stays untuned, so the run that brings enough rows calibrates. Returns whether it tuned (F when already tuned, untrained, rate ≤ 0, or the ring too small for the rate). model_autotune_why gives the same answer in words: "calibrated: false" with nothing beside it left a reader to guess between four different situations, one of which (already tuned) is the normal case and none of which is an error.

@ model_autotune_at * Model mo f rate i now → b

@ _an_fc_autotune * Model mo → v

The forecast version's first margin, measured rather than assumed (see model_train_forecast_at). Lives here because it needs FineTuneReport, which is declared with the fine-tune machinery below.

@ finetune_free FineTuneReport rep → v

@ model_finetune_at * Model mo f rate i from_ts i to_ts b apply ( Vec String ) only → FineTuneReport

Set every enabled, trained version's margin so that a fraction rate of the window [from_ts, to_ts] is flagged, or only report what would change when apply is F. only (empty = all) restricts which versions are written; the rest are still reported, unapplied, so a dry run and a partial apply show the same picture. Margins are written in the version's own units, rounded to three significant digits, and take effect at the next detect. The flatline guard is left out: its margin is a fraction with a fixed meaning (SPEC §5.4), and a stuck sensor is not a 1 % property of a window — set it with the version editor.

@ model_version_from * Model mo s vname → i

The lower bound of a version's OWN window, anchored on the newest stored point: window_min minutes back for a forest version, window_size points back for timevector, and the whole ring (0) for the autoencoder, whose training set is the whole ring too.

@ model_finetune_own * Model mo f rate b apply ( Vec String ) only → FineTuneReport

Fine-tune every version over ITS OWN window — the one it trains on — so short_term's margin answers for the last three hours and seasonal's for the last ninety days, each at the same rate. One calibration per version; the report's window is the widest of them.

@ model_seq_base * Model mo → i

@ model_label_point * Model mo i index s label s by s note i at → i

Record what a reader said about the row at index. Returns the row's sequence number, -1 for an index outside the ring, -2 for a label that is not one of ANOM_LABEL_*. Verdicts do not change, so the epoch does not move.

@ model_labels * Model mo → ( Vec Label )

The labels in force (store_load_labels), evicted rows included.

@ model_label_map * Model mo ( Vec Label ) labels → ( Vec i )

Per ring position, the index into labels of its label, -1 for none.

@ _an_label_is ( Vec Label ) labels i li s what → b

Does label li of labels (as model_label_map hands it out) say what?

@ model_last_span * Model mo i last → i

A last as a caller says it — seconds on a time clock, points on a count clock — as the span model_window_from_last takes. N points back from the newest is N ticks INCLUDING it, so the span is one short of N whole ticks.

@ model_default_last * Model mo → i

The default window: a day, or its worth of points (1440) on a count clock.

@ model_finetune * Model mo → FineTuneReport

: WholeTrain

: WholeTrain {
    Json margins  // version name → decision margin, as fine-tune set it
    Json notes  // strings: what could not be done, and why
}

@ whole_train_free WholeTrain w → v

@ model_train_whole * Model mo f rate ( Vec i ) hidden → WholeTrain

The windows are opened for this one training only: the version configuration the model keeps is the one it was given (its own, or the source's when it is a fork), so a fork that goes on receiving points retrains its short_term over three hours like its source, not over everything it has ever seen. The autoencoder takes hidden as its layout (empty = the 64-16-64 default) and rate as its pre-filter.

: ScoredPt

: ScoredPt {
    i sp_idx  // ring index
    i sp_ts  // ingest timestamp (unix seconds)
    f sp_score  // aggregate decision_function (the most severe version)
    f sp_severity  // that version's severity — comparable across rows
    b sp_anomaly
    i sp_present  // bitmask over ScanOut.vnames: versions that had a verdict
    i sp_flagged  // bitmask over ScanOut.vnames: versions that flagged it
}

One scored ring point.

: ScanOut

: ScanOut {
    ( Vec ScoredPt ) pts
    ( Vec String ) vnames
    i epoch
    i total  // ring size
    i considered  // rows inside the requested time window
    i hits  // verdicts answered from the cache
    i misses  // verdicts computed this call
    i anomalies  // anomalous rows among `considered`
}

@ scan_agreed ScoredPt r i minvotes → b

Does a scored row count as an anomaly when minvotes versions have to agree? The bit count of the flagged mask is the vote.

: ScanRun

: ScanRun {
    i run  // 1-based, in ring order
    i first_k  // positions in ScanOut.pts
    i last_k
    i rows
    i worst_k  // the row with the highest severity
    f worst_sev
    i flagged  // union of the rows' flagged masks
}

: ScanRuns

: ScanRuns {
    ( Vec ScanRun ) runs
    ( Vec i ) run_of  // per position in ScanOut.pts: its run, 0 = none
}

@ scan_runs ScanOut so i minvotes → ScanRuns

@ scan_runs_free ScanRuns sr → v

@ scan_free ScanOut so → v

@ model_row_is_anomaly * Model mo i index → i

Does the model flag the stored row at index? Read from the cached ring scan, so a second question about the same model costs nothing. -1 when there is no such row.

@ model_scan_versions * Model mo → ( Vec String )

The canonical version order a scan's bitmasks are indexed by: every enabled version that could produce a verdict, in metadata order. It is stable within an epoch by construction — anything that adds, removes or toggles a version bumps the epoch — and it is written into the cache so a mismatch is caught rather than silently misread as different versions.

@ model_scan_at * Model mo i from_ts i to_ts i limit b force → ScanOut

Score every ring point whose timestamp falls in [from_ts, to_ts] (0 = unbounded on either side), newest-last, capped at limit rows (<= 0 = no cap) taken from the END of the window. force recomputes even when the cache is warm — the escape hatch for verifying the cache itself, never needed for correctness.

@ model_scan * Model mo i from_ts i to_ts i limit b force → ScanOut

@ model_point_json * Model mo i at → ?Json

The raw stored record at a ring index, parsed (None when out of range or no longer parsable).

: AeContrib

: AeContrib {
    String ac_name
    f ac_err
    f ac_share
    f ac_value
    f ac_expected
}

One feature's share of an autoencoder reconstruction error, with the value the point carried and the one the autoencoder expected of it given the other features (raw units — the MinMax undone).

@ ae_contrib_free ( Vec AeContrib ) xs → v

@ model_ae_contrib * Model mo Json raw i topk → ( Vec AeContrib )

The topk features carrying the most of a point's reconstruction error, largest first, with each one's share of the total. This is the answer the forests structurally cannot give: the autoencoder's per-feature error is how badly that feature failed to be predictable FROM THE OTHERS, so the top entries name the broken relationship rather than the extreme value. Empty when the model has no trained autoencoder.

@ model_reset * Model mo → v

Drop all data and trained forests but keep the model's name, schedule and version configs. Learned columns/categories/features/scaler reset.

@ model_delete Store st s name → b

Delete a model from the store entirely (the Model handle, if any, should be freed separately with model_free).

@ _an_audit_margin * Model mo s vname f before f after → v

One line of the audit trail: a version's alert line moved, by whom and how. Every path that changes a margin goes through here — model_set_margin (fine-tune, autotune, the CLI, a source's first train) and the metadata patch (edit), which sets margins straight into the VerCfg and for two releases moved them without a word in the log the tool promises.

@ model_set_margin * Model mo s vname f margin → b

Set one version's decision margin in the metadata (persisted, effective immediately at scoring — no retrain needed). Returns F for an unknown version name.

@ model_set_schedule * Model mo i below_max i at_max → v

Update the retraining schedule (persisted immediately).

@ model_set_version_enabled * Model mo s vname b on → b

Turn one version on or off (persisted immediately). Disabling drops the version's forest, so its verdict is gone from the very next detect and re-enabling it costs a retrain. The autoencoder version is only muted: its net carries its OWN frozen feature order, so it stays valid across retrains and is far too expensive to throw away on a checkbox. Returns F for an unknown version name.

@ meta_editable_fields → Json

Apply an editable-metadata patch:

{ "alias": "boiler room", "schedule": { "below_max": N, "at_max": N, "autoencoder": bool }, "max_data_points": N, "versions": { "<name>": { <any VerCfg field> }, ... }, "replace_versions": bool }

All top-level keys are optional but at least one must be present. The learned parts of the metadata (columns, categories, feature order, scaler) are never taken from the client — see prep.nu. Lowering max_data_points below the current fill evicts the oldest points and rewrites the log before returning, so the cap holds at once instead of converging one point per ingest. Per-version fields split two ways: enabled and decision_margin bite at the very next detect, the geometry and forest-size fields at the next retrain. Returns "" on success, else the reason (400-worthy). The top-level keys model_apply_meta_patch accepts, published with every metadata response so a client does not have to keep its own copy of the list. The dashboard kept one, and it went stale the moment max_data_points became editable: the field was patchable through the API and invisible in the JSON editor that exists to reach it. One list, named here beside the code that reads the patch, is what stops that from happening again — the editor is generated from it.

Deliberately NOT part of meta_to_json: that is also the on-disk meta.json writer, and a service-shaped descriptor has no business in the stored file.

@ _an_patch_adjustments * Meta mm Json vers ( Vec String ) notes → v

What the patch asked for and what the config actually holds, per version and field. _an_vercfg_sane refuses impossible combinations — a seasonal window with a zero step, a forest of no trees, a flatline window of one row — and it used to do that in silence, so a caller who sent step_size: 0 read back a 1 and had nothing to blame but the nearest flag it had also sent. Every field the patch names and the config did not keep is reported back beside the change.

@ _an_note_int ( Vec String ) notes s vn s field Json vo i kept → v

@ _an_note_f ( Vec String ) notes s vn s field Json vo f kept → v

@ model_apply_meta_patch * Model mo Json patch → String

@ model_apply_meta_patch_notes * Model mo Json patch ( Vec String ) notes → String

---

imptime.nu

anomaly/imptime.nu — where a file keeps its clock, and how to read it.

A stream stamps every point at the moment it arrives. A file has no such moment: its clock is in the data — a timestamp column, an ISO string under some other name, a Unix number, or a date spread over Vuosi, Kuukausi, Päivä and Aika the way a weather-service export writes it. This file reads all of those, best effort, and says which one it read, so the person importing can confirm or correct the guess before a single row lands.

Two operations:

import_inspect parsed rows → JSON: every column with what its values look like, and a proposal for the time — one column, a set of part columns, or nothing. import_time_apply rows + a plan → every row gets timestamp (Unix seconds) from the plan; the source columns are dropped so a year or a clock does not become a feature; and, on request, a time ISO string is left behind for the calendar features (hour, day, month, weekday) the preprocessing layer derives.

Naive stamps — a clock without an offset, which is what a database TIMESTAMP WITHOUT TIME ZONE, a spreadsheet and a weather export all write — are read in the timezone the caller names, defaulting to the server's local one: the file came from the place the service runs.

API

: i ANOM_TZ_LOCAL -1000000

The caller's "use the server's local zone" sentinel for a tz offset.

: i ANOM_INSPECT_ROWS 5000

Rows looked at when describing a file. A million-row file is described by its first five thousand; the plan then applies to all of them.

: i STAMP_NONE 0

What one cell turned out to be.

: i STAMP_DATETIME 1 // a date and a time of day

: i STAMP_DATE 2 // a date alone (midnight)

: i STAMP_UNIX 3 // a Unix number — seconds, ms, µs or ns by magnitude

: i STAMP_CLOCK 4 // a time of day alone, HH:MM[:SS]

: ImpStamp

: ImpStamp {
    i kind
    i secs
    b zoned
}

secs is Unix seconds when zoned (the text carried an offset, or was a Unix number); otherwise the civil fields read as if they were UTC, for the caller to shift into the zone it was told. For STAMP_CLOCK it is seconds after midnight.

: __ItNum { i val i len }

@ imp_stamp_of_text s raw → ImpStamp

One cell of text as a stamp. Formats, in the order they are tried:

2026-08-29T00:10:00Z, 2026-08-29 00:10:00+03:00, 2026-08-29 00:10, 2026-08-29 ISO 8601 / RFC 3339, SQL TIMESTAMP with or without a zone, date alone 2026/08/29 00:10 the same with slashes 29.8.2026 00:10, 29.8.2026 day first — dots are European 29/08/2026, 08/29/2026 slashes: day first unless the second number cannot be a month 20260829, 20260829T001000 compact 1756422600, 1756422600000 Unix seconds / ms / µs / ns Sat, 29 Aug 2026 00:10:00 +0300 RFC 2822 and HTTP dates 00:10, 00:10:00 a time of day alone (STAMP_CLOCK)

@ imp_stamp_of Json v → ImpStamp

One JSON cell as a stamp: a number is Unix, a string is text.

@ imp_instant_of_text s raw i tz → i

One instant from one piece of text, as a Unix second: an ISO 8601 date or date-time (a bare one is read in tz, ANOM_TZ_LOCAL for the server's zone), a Unix number in seconds, milliseconds, microseconds or nanoseconds (told apart by magnitude), or a compact 20260901[120000]. 0 when the text is not a moment — a time of day alone is not one.

@ imp_span_of_text s raw → i

A span from text: "90s", "15m", "24h", "7d", "2w", or a bare number of seconds. 0 when unreadable or not positive.

@ imp_tz_of Json spec → i

A tz spelling: "local", "utc", "Z", "+03:00", "+0300", "+03", or a number of seconds east. Unreadable → local.

: i ROLE_STAMP 0

Does name (normalised) mean one of the roles? 2 = a strong name nobody would use for anything else, 1 = a short one that might be "minimum" or "seconds" or a coordinate, 0 = no.

: i ROLE_YEAR 1

: i ROLE_MONTH 2

: i ROLE_DAY 3

: i ROLE_HOUR 4

: i ROLE_MINUTE 5

: i ROLE_SECOND 6

: i ROLE_DATE 7

: i ROLE_CLOCK 8

: ImpCol

: ImpCol {
    String cname
    String norm
    i filled
    i n_num
    i n_text
    i n_dt  // text that reads as a date and time
    i n_date  // text that reads as a date alone
    i n_clock  // text that reads as a time of day alone
    i n_unix  // numbers that could be Unix stamps
    i n_year  // numbers 1900..2100
    i n_month  // numbers 1..12
    i n_day  // numbers 1..31
    i n_hour  // numbers 0..23
    i n_minsec  // numbers 0..59
    String sample
}

@ import_time_propose ( Vec ImpCol ) cols → Json

The plan: { mode: column|parts|none, column?, parts?: {year, month, day, hour, minute, second, date, clock}, confidence: high|guess|none, reason }. Every named column exists and fits its role.

: ImpTimeResult

: ImpTimeResult {
    i stamped
    i failed  // rows the plan could not read a time from — dropped
    String first_fail  // what the first of them looked like
}

@ imp_time_result_free ImpTimeResult r → v

@ import_time_apply ( Vec Json ) rows Json plan b calendar i tz → ImpTimeResult

Rewrite rows in place under plan. With mode none nothing changes. A row whose time cannot be read is removed: a history point with no place in history is not a point this model can hold.

: ImpPlan

: ImpPlan {
    Json plan
    String err
}

Turn a request ({ mode: auto|none|column|parts, column?, parts? }) into a concrete plan against these columns. auto becomes the proposal; a named column that does not exist is an error (non-empty err).

@ imp_plan_free ImpPlan p → v

@ import_time_plan ( Vec ImpCol ) cols Json spec → ImpPlan

@ import_inspect ( Vec Json ) rows Json spec i tz → Json

{ columns: [{ name, kind, filled, sample }], time: <plan + sample>, hints: { role: [names] } }. time.sample is the first row's time under the plan, as ISO in the zone, so a person can check the guess against the file with their own eyes.

---

config.nu

anomaly/config.nu — the configuration file.

Everything the service can be told is settable three ways, and they layer in one fixed order:

command-line flag > environment variable > config file > default

The file is the persistent baseline a deployment writes once; the environment is what a container or a unit file overrides for one run; a flag is what a person types to override both. Nothing here reads the environment or the command line — main.nu owns that layering, so the precedence lives in one place instead of being re-decided per setting.

The file is TOML, because that is already the project's format:

[auth] enabled = true issuer = "https://login.microsoftonline.com/<tenant>/v2.0" client_id = "<application (client) id>" audience = "api://<application (client) id>" # optional open_ingest = true

[service] addr = "0.0.0.0:8811" webroot = "/usr/share/anomaly/static" public_url = "https://anomaly.example.com" # behind a proxy

Deliberately NOT settable here: the store directory. The file is looked for inside the store, so a store key would be a file relocating the directory it was just found in — a loop with no honest answer. It stays --store / $ANOMALY_HOME.

API

: AnomalyConfig

: AnomalyConfig {
    b loaded
    String cpath  // where it came from; empty when nothing was loaded
    String cerr  // parse failure text; empty when fine
    TomlValue root
}

@ config_empty → AnomalyConfig

@ config_free AnomalyConfig c → v

@ config_load s path → AnomalyConfig

Read and parse path. A file that does not exist is not an error — the common case is having none — but a file that exists and does not parse IS one: silently ignoring a config file someone wrote is how a service comes up unconfigured and nobody can see why.

@ config_loaded AnomalyConfig c → b

@ config_path AnomalyConfig c → s

@ config_error AnomalyConfig c → s

@ config_has AnomalyConfig c s key → b

Is the dotted key present at all? The difference between "absent" and "set to empty" is what lets a lower layer show through.

@ config_str AnomalyConfig c s key s dflt → String

@ config_bool AnomalyConfig c s key b dflt → b

@ config_str_list AnomalyConfig c s key s dflt → String

A string array, joined with commas — the form a global can hold. An absent key, a non-array, or an array with a non-string in it all yield the default, so a mistyped list demotes to it rather than to a half-read one.

@ config_find s explicit s store → String

Where the configuration file is looked for, in order:

1. explicit — --config FILE, or $ANOMALY_CONFIG
2. <store>/anomaly.toml
3. /etc/anomaly/anomaly.toml

An explicit path is returned whether or not it exists, so a typo in --config surfaces as "that file does not parse / is not there" rather than as the service quietly falling back to a different file.

---

model.nu

anomaly/model.nu — the per-point scoring core (milestone M2).

Wraps the iforest package's forest behind the M1 preprocessing layer and the scikit-learn decision conventions the reference service uses:

score_samples(x) = -iforest_score(x) (more negative = worse) decision_function(x) = score_samples(x) - offset_ offset_ = -0.5 for contamination "auto" = percentile(score_samples of the training set, 100 * contamination) otherwise predict(x) == -1 ⇔ decision_function(x) < 0 version verdict : decision_function(x) <= -decision_margin

A VerModel is one trained forest + its offset and margin — the unit that M4/M5 instantiate per time window. Training and the bulk/batch scoring paths (GPU-accelerated when available) live in src/score.nu; this file is the small, dependency-light core they build on.

API

: i ANOM_SEED 42

The reference's fixed random_state: a given training set always yields a byte-identical forest, hence identical scores, on every platform.

: VerModel

: VerModel {
    String vname
    IForest forest
    f offset
    f margin
    i n_cols
    b trained
}

One trained model version: forest + decision offset + margin.

@ anom_decision VerModel vm ( Vec f ) scaled_point → f

decision_function of an already-standardised point.

@ anom_decision_row VerModel vm ( Vec f ) scaled i r → f

decision_function of row r of an already-standardised matrix.

@ anom_is_anomaly VerModel vm f df → b

The per-version verdict convention: below (or at) the margin ⇒ anomaly.

@ anom_vermodel_free VerModel vm → v

---

orgfiles.nu

src/orgfiles.nu — an organisation's storage folder

Every organisation has a folder of files beside its database:

<root>/orgs/<org>/files/<name>

Analyses drop their results there; the API lists and serves them to the organisation's members. A file can also be handed out as a link that carries its own permission — an HMAC over (org, name, expiry) under a secret the service generates once — so a result too large to return inline can be fetched by whoever holds the link, with no sign-in. The link names one file for a bounded time and nothing else; the secret never leaves the store.

File names are the public identifier, so they are kept to one safe alphabet: letters, digits, ., _, -, not starting with a dot, at most 128 bytes. That rules out path tricks and URL escaping alike.

API

: i OF_NAME_MAX 128

: i OF_LINK_TTL_DEFAULT 604800 // a week

: i OF_LINK_TTL_MAX 2592000 // 30 days

: OfState

: OfState {
    String root
    String secret
}

What the module keeps for the life of the process: the store root (the models' directory — the organisation folders live under <root>/orgs, beside the organisation databases) and the link-signing secret. Both are OWNED copies in one heap block a global points at, so a caller may free the root it passed, and the block stays reachable — a global that held only a data pointer would leave the String header unreachable, which is a leak in every accounting.

: ~ i g_of_state 0

@ orgfiles_set_root s root → v

@ orgfiles_dir s org → String

@ orgfiles_tasks_dir s org → String

@ orgfiles_sources_dir s org → String

Where an organisation keeps its configured data sources (src/sources.nu).

@ orgfiles_root → s

The store root itself, for a module that needs the models beside the organisation folders.

@ orgfiles_orgs → ( Vec String )

Every organisation that has a folder under <root>/orgs — the names of the directories there, whatever they hold.

@ orgfiles_name_ok s name → b

One safe alphabet for file names: [A-Za-z0-9.-], no leading dot, at most OFNAME_MAX bytes.

@ orgfiles_safe_name s raw → String

A caller-chosen label, made into a safe name: every byte outside the alphabet becomes _, a leading dot too, and the result is cut to fit. Empty in, empty out — the caller picks a default.

@ orgfiles_path s org s name → String

: OrgFile

: OrgFile {
    String name
    i size
    i mtime
}

@ orgfiles_free ( Vec OrgFile ) xs → v

@ orgfiles_list s org → ( Vec OrgFile )

Every file in the folder that carries a valid name, by name. Anything else in the directory — a temp file mid-write, a stray dotfile — is not a file the API ever handed out and is not listed.

@ orgfiles_stat s org s name → OrgFile

The stat of one named file, or size -1 when there is no such file.

@ orgfiles_delete s org s name → b

@ orgfiles_write s org s name ( Vec u ) data → b

Write a file into the folder atomically (tmp + rename), so a reader never sees a half-written result.

@ orgfiles_sign s org s name i exp → String

@ orgfiles_verify s org s name i exp s sig i now → b

@ orgfiles_link s org s name i exp → String

The link's path + query, relative to the service root.

@ orgfiles_content_type s name → s

Content type from the extension: the results are JSON, a caller may also have dropped CSV there; the rest is bytes.

---

importer.nu

anomaly/importer.nu — a file of history becomes a stream of points.

The service is built around one point at a time arriving from a producer. Importing is the other direction: a file that already holds the history, turned into the same records the ingest path takes, so nothing downstream has to know where a point came from.

Three shapes, because these are the three a data file actually arrives in:

csv a header row naming the columns, one row per point. Cells that parse as numbers become numbers; everything else stays a string, which the preprocessing layer already knows what to do with (categories, ISO-8601 timestamps → calendar features). json an array of objects, or an object with the array under data, points or rows — the three spellings an export tool picks. jsonl one object per line. What this service's own /data route emits, so a model can be moved by exporting and importing it.

auto sniffs: a body whose first non-space character is [ or { is JSON or JSONL, anything else is CSV. Detection is offered because a browser upload rarely knows its own format, and is never mandatory — naming the format explicitly always wins.

A row that cannot be read does not stop the import: it is counted and the first few are described. A file of ten thousand rows with one bad line is a file with one bad line, not a failed import.

API

: i ANOM_IMPORT_MAX_BYTES 67108864

A file bigger than this is refused before it is parsed. Generous enough for years of minute-resolution history, small enough that a mistaken upload cannot exhaust memory.

: i ANOM_IMPORT_MAX_ROWS 1000000

: i ANOM_IMPORT_MAX_NOTES 5

How many bad rows are described rather than merely counted.

: ImportParse

: ImportParse {
    ( Vec Json ) rows
    i skipped
    ( Vec String ) notes  // why the first few were skipped
    String err  // non-empty ⇒ nothing was read at all
    String format  // what it turned out to be
}

@ import_parse_free ImportParse ip → v

@ import_sniff s text → String

Which of the three this body is. [ starts a JSON array; { starts either a lone object or the first line of JSONL, and which one is settled by whether a later line also starts an object.

@ import_parse s text s format → ImportParse

---

wfs.nu

anomaly/wfs.nu — an OGC WFS 2.0 service as a source of data points.

A weather service, a hydrology office, a radiation network: what they publish is a WFS endpoint with STORED QUERIES — named, parameterised requests (fmi::observations::weather::simple) that answer with a feature collection. The ::simple family, and anything shaped like it, answers in long form: one member per (location, time, parameter, value). Pivoting those on (location, time) gives one record per moment with a column per parameter — exactly the JSON point the ingest path takes, with the observation's own clock on it.

A GeoServer or MapServer — a city's open data, a road authority's, another weather service's — has no stored queries worth the name but publishes FEATURE TYPES, and a GetFeature by type name answers in wide form: one feature per member with a property per element and a geometry. Those pivot too: one record per feature, the properties as numbers or text, the geometry's first coordinate as lat/lon, and the clock read from whichever property looks like a date.

The operations, none of which touches the network:

wfs_catalog GetCapabilities or DescribeStoredQueries XML → JSON: every feature type (kind: "type") or stored query (kind: "stored") with title, abstract and, for a stored query, its parameters — for a person to pick from. wfs_pivot long-form GetFeature XML (a stored query's answer) → records: one JSON object per (location, time) with time (ISO 8601), timestamp (Unix seconds), lat, lon and a number per parameter. NaN — the service's spelling of "no reading" — is dropped, not learned. wfs_pivot_wide wide-form GetFeature XML (a feature type's answer) → records: one per feature, every simple property a number or a text, gml_id, lat/lon, and the clock from a chosen or detected date property — or the fetch time, when the features carry none. wfs_url_* the request URLs, every value percent-encoded.

and one that does: wfs_fetch, a GET with a deadline and a body cap.

Namespace prefixes are whatever the server chose (BsWfs:, wfs:, none at all), so every tag is matched on its local name.

API

: i WFS_TIMEOUT_MS 60000

A fetch that takes longer than this has hung; an answer bigger than this is not a time window a source should ask for at once.

: i WFS_BODY_MAX 67108864

: i WFS_COUNT_DEFAULT 1000

Features per GetFeature on a feature type, unless the source says.

: i WFS_COUNT_MAX 100000

: i WFS_TEXT_MAX 200

A text property longer than this is a description, not a category.

: s WFS_BY_ID_QUERY urn:ogc:def:query:OGC-WFS::GetFeatureById``

The one stored query every GeoServer lists, which is not a source.

@ wfs_base_url s url → String

The endpoint without its query string: what a person pastes is often the GetCapabilities link, and the base is what every request builds on.

@ wfs_url_ok s url → b

http or https, nothing else: the service fetches this URL on a schedule, and a scheme it does not speak is a mistake worth refusing at configuration time rather than at three in the morning.

@ wfs_url_catalog s base → String

@ wfs_url_capabilities s base → String

@ wfs_caps_has_stored s xml → b

Does a capabilities document offer stored queries at all?

@ wfs_url_feature s base s query Json params i start i end → String

GetFeature for one stored query over [start, end] (Unix seconds). The caller's parameters go in as given, except the two the window owns — a saved starttime would pin every fetch to the same hour.

@ wfs_url_type s base s typename Json params → String

GetFeature on a feature type: at most count features (params may say count), in WGS 84 latitude-first unless params name an srsName, and every other parameter — bbox, cql_filter, sortBy, filter — passed through as given. No time window: a feature type is fetched whole and the pivot reads the clock from the features.

@ wfs_catalog s xml → Json

DescribeStoredQueries, ListStoredQueries or GetCapabilities → {queries: [...]}, each with kind "stored" or "type"; {error: ...} when the document is none of those.

: WfsPivot

: WfsPivot {
    ( Vec Json ) rows  // one object per (location, time), in order of first appearance
    ( Vec String ) columns  // lat, lon, then every parameter in order of first appearance
    i members  // feature members read
    i missing  // values that were NaN or empty, left out of their row
    String err  // non-empty ⇒ nothing was read
}

@ wfs_pivot_free WfsPivot p → v

@ _wfs_pivot_err s msg → WfsPivot

@ _wfs_col_add ( Vec String ) cols s name → v

: __WfsPos { b ok f lat f lon }

"60.17523 24.94459" → (lat, lon); F when it is not two numbers.

@ wfs_pivot s xml → WfsPivot

A GetFeature answer, pivoted. err names what went wrong when nothing could be read: a service exception, a document of another shape.

: s WFS_CLOCK_NONE none``

No property is the clock: every feature is stamped with the fetch time.

@ _wfs_wide_clock Json row s time_field → b

The clock of a wide record: time_field when named, else the first text property that reads as a date or a date-time — unless the field is WFS_CLOCK_NONE, then nothing is. Sets timestamp and an ISO time (for the calendar features); T when found.

@ wfs_pivot_wide s xml s time_field i now → WfsPivot

A wide-form GetFeature answer, pivoted: one record per feature. time_field names the property that is the clock ("" = detect); a feature with no clock is stamped now — a snapshot, the moment it was fetched. gml_id is the feature's identity, for a category.

@ wfs_fetch s url → !String String

GET url; Ok(body) on a 2xx, Err(why) otherwise — the status and the first line of the body, which is where a WFS puts its exception text.

---

mcp.nu

anomaly/mcp.nu — the service as an MCP server, mounted at /mcp.

A language model talks to the same service the dashboard does, with the same credential, and gets exactly what that credential may do: the tool list is computed per caller (a viewer never sees ingest_point), and every tool runs as an in-process HTTP request through the service's own router, so the API's authorisation gates are the only gates — there is no second rule book to drift.

What the model sees is shaped for a context window rather than for a chart: timestamps are ISO-8601 UTC, floats are rounded, a listing is a summary plus the newest rows, and every reply names what it left out.

POST /mcp JSON-RPC (Streamable HTTP transport) GET /.well-known/oauth-protected-resource[/mcp] where to get a token (RFC 9728)

Sign-in: the same OAuth (Entra) tokens the dashboard sends, or an API key in Authorization: Bearer / X-API-Key. With authorisation off (simple mode) every caller is the administrator, as everywhere else.

The server object is static (built once, no per-request state); the caller is a context Json the dispatch carries into gates and handlers.

API

: s ANOMALY_VERSION 0.31.0``

One version for the CLI banner and the MCP handshake.

: McpWiring

: McpWiring {
    Router router
    b has_router
    String public_url
    b has_server
    McpServer server
}

The router the tools call back into (a shallow copy of the service router, whose route table is complete by the time it is attached), the static server, and the public origin for the resource-metadata URLs.

: ~ i g_mcp_wiring 0

@ an_mcp_attach_router Router r → v

Called by anomaly_service_router once every route is registered. The copy shares the route vector, so the router must not grow afterwards.

@ an_mcp_set_public_url s url → v

[service] public_url — the origin clients reach the service at, when it sits behind a proxy that rewrites Host. Empty: derived per request from Host / X-Forwarded-*.

: ApiOut

: ApiOut {
    i status
    Json body  // the parsed body, or JSON null when it was not JSON
}

: i MCP_SPAN_ALL -2

A span: seconds as a number, or "90s" / "15m" / "24h" / "7d" / "2w"; "all" (also "max", "*") is the whole ring, the word the REST routes know — one window vocabulary for every tool. 0 when absent, MCP_SPAN_ALL for the whole ring, -1 when unreadable.

: i MCP_CATS_WHOLE 12

A categorical column's levels, fit for a context window: a short list whole, a long one (a time-of-day text, an id) as its count and a sample — 144 dummies listed twice tell a reader nothing the count does not.

: i MCP_CATS_SAMPLE 6

: i MCP_FEATS_WHOLE 40

: i MCP_EVENTS_SHOWN 10

Events shown in an anomaly_summary; the count is always there.

: FeatShare

: FeatShare {
    String name
    f share
    i n
}

Per-feature attribution totals across the flagged rows.

: s __MCP_SOURCE_FIELDS name kind url query mode params features categorical time_field model interval_minutes history_hours calendar enabled method headers body path allow_future finetune_rate``

── Tools: data sources (src/sources.nu) ─────────────────────────────

The record's fields as the API takes them, copied from the arguments as given — the API validates and answers with the reason on a 400.

@ an_mcp_metadata_response HttpRequest req → HttpResponse

GET /.well-known/oauth-protected-resource[/mcp] — RFC 9728: which authorization server issues tokens for /mcp, and which scope to ask for. The same issuer and audience the dashboard uses. 404 in simple mode: there is no sign-in to point at.

@ an_mcp_handle HttpRequest req → HttpResponse

The /mcp endpoint. Called inside the service lock, like every handler.

---

csvdata.nu

anomaly/csvdata.nu — numeric CSV → row-major matrix (batch inputs).

Same forgiving shape as the iforest CLI parser: split lines, split on the delimiter, keep numeric fields; the first data row fixes the column count, later rows with a different numeric-field count are skipped. With has_header, the first line's names are captured (for batch reports that echo column values by name) instead of being parsed as data.

API

: AnomCsv

: AnomCsv {
    ( Vec f ) data
    i rows
    i cols
    ( Vec String ) headers
}

Parsed matrix: data is row-major rows*cols; headers is empty when the input had no header row.

@ anom_csv_free AnomCsv ds → v

@ anom_parse_csv s input s delim b has_header → AnomCsv

---

analyze.nu

src/analyze.nu — one-shot analysis of a file

POST /api/analyze takes a file and answers with its anomalies: the file becomes a throwaway model, trained as carefully as the data allows — every forest version, the 64-16-64 autoencoder over the forest- filtered points, every margin set to a 1 % alert rate — scanned once, and thrown away. What remains is the result file in the organisation's folder (src/orgfiles.nu) and a status record.

The work runs in a CHILD PROCESS, anomaly analyze-job <task dir>, not in the service: training holds the GPU singleton and the random state, both of which the live detectors use, and a file that takes the trainer down must not take the service with it. The service and the job meet only in the task directory:

<root>/orgs/<org>/tasks/<id>/params.json what to do (service → job) /input the file as posted /status.json queued | running | done | failed /store/ the temporary model, gone when done

The service watches status.json; the job writes it (atomically) at each transition, and the thread that ran the child marks it failed when the process died without saying so.

API

: f ANA_TARGET_RATE 0.01

: i ANA_MIN_ROWS 10

: i ANA_INLINE_MAX 10000 // anomalies returned in the response body; beyond it, a link

: f ANA_STANDOUT 3.0 // separation from which the flagged rows stand apart from the file

: AnaSep { f sep i rows }

Whether the flagged rows stand apart from the file or are merely its tail. The margins are the file's own 1 % quantile, so about 1 % of ANY file is flagged and severity says ~1 for all of them — a lone spike sets the very margin it is measured against. What tells a fault from a tail is a STEP in the sorted scores: a block of rows far out, and then the rest of the file well below it. Read on range_guard alone, whose score is a linear magnitude (−max |z|, in standard deviations). A forest's decision function saturates, and the autoencoder's reconstruction error is heavy-tailed even on Gaussian data (the net fits the bulk), so their ratios say nothing about the file.

For each block size j the statistic is |score of the j-th worst row| / |score of the 2j-th worst|, and the answer is the largest of those over j — with the j that produced it, which is how many rows stand apart. A file's tail thins gradually and every ratio stays near 1 (the worst of 1500 Gaussian draws is ~3.3 σ, the 30th ~2.3 σ); a real fault has one j where the ratio jumps.

Fixing j at the target rate — the one ratio this measured until 0.32.0 — could only see a fault SMALLER than 1 % of the file. A stuck sensor or a collapsed reading lasting 25 rows in 1500 put fault rows on BOTH sides of that single ratio, which then read ~1, and the file was reported as having nothing that stands out while a fifth of a column's range had gone missing. The maximum over j has no such blind spot: the step is found wherever it is.

@ _ana_separation * Model mo f rate → AnaSep

@ _ana_reading i nanom AnaSep sp → String

One sentence on what the flagged rows are worth: the margins are set from the file itself, so the count alone says nothing.

@ analyze_task_dir s org s id → String

@ analyze_status_read s dir → ?Json

@ analyze_status_write s dir Json st → b

@ _ana_jstr Json o s key → String

@ _ana_jint Json o s key i dflt → i

@ _ana_jbool Json o s key → b

@ analyze_state s dir → String

@ analyze_state_final s state → b

@ analyze_task_create s org Json params ( Vec u ) input → String

Create the task: a fresh id, its directory, the input and the parameters written, status queued. Returns the id ("" = could not create).

@ analyze_task_list s org → Json

Every task of the organisation, newest first: the status records, each with its id.

@ analyze_task_delete s org s id → b

: ~ s g_ana_exe ``

The service's own binary: the job is the same program in another process. The binary that runs jobs: this one, normally. A test binary is not anomaly — re-spawning it would run the whole suite again, in the same store — so tests point this at something harmless.

@ analyze_set_exe s exe → v

@ analyze_exe → String

@ _ana_mark_crashed s dir i code s detail → v

The child died without writing a final status: say so. A status the job wrote itself stands.

@ analyze_spawn s org s id → b

Start the job for a task: a thread runs the child process to its end and records a crash. The thread shares nothing with the service but the task directory. Returns F when no thread could be started.

@ analyze_run s dir → i

---

aegpu.nu

anomaly/aegpu.nu — GPU-accelerated autoencoder training (bit-exact).

The autoencoder trains through mlp_fit — minibatch Adam over a small dense net — and that training is where /train/autoencoder spends its time (~20 s for 20 k rows; the bulk-MSE pass after it is milliseconds). This module runs THE SAME training on the GPU when one is present, with the package's standing guarantee intact: backend choice can never change a result. The GPU path is bit-for-bit identical to mlp_fit, so a model trained on a CUDA machine equals the CPU-trained model exactly — same weights, same threshold, same verdicts. GPU is pure speed.

How bit-exactness is achieved (and pinned by tests/aegpu_parity_smoke.sh):

• Control flow stays on the host and MIRRORS mlp_train exactly: the same

seeded PRNG call order (init, the one full shuffle, the per-epoch train -region shuffles), the same validation split, minibatch bounds, early- stopping/patience/best-weights logic, and the same Adam scalar bookkeeping (lr_t via host pow/sqrt, passed to the kernel as data).

• The kernels reproduce the CPU's floating-point ROUNDING AND ORDER:

every accumulation chain uses explicit __dadd_rn/__dmul_rn/__dsub_rn (round-to-nearest, never fused — NVRTC's default fmad contraction would change results), dot products run serially in the CPU's k-order inside one thread, and per-weight gradient sums walk the batch rows in the CPU's row order. Division and sqrt are IEEE-correctly-rounded on both sides. Scalars that need libm pow are computed on the HOST and shipped to the kernel, so no transcendental runs on the device.

• Loss/criterion sums come back as per-element differences and are

folded on the host in the same flat order as the CPU loop.

Weights, Adam state and the data live on the device for the whole run; per minibatch the device does forward → backprop → gradient → Adam in four launches and only the output-layer diffs (batch×d f64s) come back.

Selection: engine cuda (score.nu's probe) → GPU; anything else — no device, ANOMALY_GPU=0, any mid-setup failure — falls back to the pure mlp_fit, which (being bit-identical) is a pace change, never a behaviour change. The gpu package's host-C++ backend is deliberately NOT used for training: native mlp_fit already is the optimized CPU path.

API

@ _aeg_eval_chunk → i

Rows per eval chunk (validation pass): bounds the acts buffer.

@ _aeg_kernel_src → s

── the kernels ─────────────────────────────────────────────────────── meta (i64): [0]=n_layers [1]=n_a [2]=d [3]=dout, then sizes(nl+1), w_off(nl), b_off(nl), a_off(nl+1). One block per row for fwd/bwd/diff (threads stride the neurons, __syncthreads between layers); one thread per parameter for grad/adam.

: AeGpu

: AeGpu {
    b ok
    Gpu g
    GpuKernel kfwd
    GpuKernel kbwd
    GpuKernel kgrad
    GpuKernel kadam
    GpuKernel kdiff
    GpuBuffer bmeta
    GpuBuffer bx
    GpuBuffer bidx
    GpuBuffer bw
    GpuBuffer bb
    GpuBuffer bmw
    GpuBuffer bvw
    GpuBuffer bmb
    GpuBuffer bvb
    GpuBuffer bgw
    GpuBuffer bgb
    GpuBuffer bacts
    GpuBuffer bdeltas
    GpuBuffer bodt  // train outdiff: bsz × dout
    GpuBuffer bode  // eval outdiff: eval_chunk × dout
}

── device context ──────────────────────────────────────────────────── Everything one aegpu_fit run holds on the device. All buffers are f64 except idx / meta (i64). X doubles as Y (autoencoder: target == input).

@ _aeg_free AeGpu cx → v

@ _aeg_meta Mlp m i d i dout → ( Vec i )

The meta buffer content for a freshly built net.

@ _aeg_open Mlp m i n i d i dout i bsz → AeGpu

Open the context: buffers + the five kernels, on score.nu's device singleton. Any failure → ok=F (caller falls back to mlp_fit).

@ _aeg_train AeGpu cx Mlp m ( Vec f ) X i n i d i dout MlpCfg cfg * u fail → MlpTrain

── the mirrored trainer ────────────────────────────────────────────── Exactly mlp_train, with the four per-minibatch compute steps and the validation forward on the device. Returns MlpTrain; the model's w/b, Adam state and t are updated in place, exactly as mlp_train leaves them. On ANY device error sets *fail and returns immediately (the caller discards everything and reruns on the CPU).

@ aegpu_fit ( Vec i ) sizes ( Vec f ) X i n i d MlpCfg cfg i restarts → MlpFit

── the public entry: mlp_fit, GPU when a CUDA device is present ───── Bit-identical to ( mlp_fit sizes X n d X d cfg restarts ) — the autoencoder case (target == input). Falls back to mlp_fit whenever the engine is not cuda or any device step fails.

---

httpsrc.nu

anomaly/httpsrc.nu — an HTTP endpoint answering JSON as a source of points.

Most of what a service publishes is not a WFS: it is a URL that answers JSON — a REST API polled with a key in a header, a device's status page, a queue's metrics. This module fetches one such URL (any method, any headers, a body if asked) and turns the answer into the same records the WFS pivots make (src/wfs.nu), so everything downstream — the chosen columns, the categorical ones, the clock, the import — is the same code.

http_fetch method + URL + headers + body → the answer's text http_pivot JSON text + a path + a clock → records

Records are found at path (dotted, indexes allowed: data.items, stations.0.values; "" = the whole answer): an array gives one record per element, an object gives one record — a snapshot. A record is flattened: a nested object's keys are prefixed with the parent's (current_temperature), numbers and booleans are numbers, short strings are text, arrays and nulls are left out. The clock is read as for a feature type: a named property, the first that reads as a date, or none — then the fetch time.

API

: i HTTP_TEXT_MAX 200

: i HTTP_DEPTH_MAX 6

@ http_pivot s text s path s time_field i now → WfsPivot

JSON text → records at path. err says what was wrong when nothing could be read.

@ http_fetch s method s url Json headers s body → !String String

One request; Ok(body text) on a 2xx, Err(why) otherwise — the status and the first line of the body, or the transport's word.

---

forecast.nu

anomaly/forecast.nu — the forecast version: a seasonal ARIMA model per numeric feature (packages/arima), judging every point against what the feature's own recent past said it would be.

The forests see a point as a whole and the guards see one reading at a time; neither sees ORDER beyond the timevector's short window. A feature that follows a rhythm — a temperature with its day, a load with its week — is normal at a value that is ordinary for the feature and wrong for the moment, and only a model of the sequence can say so. This version fits one SARIMA per numeric feature at every retrain (arima_auto, the stepwise order search, with the season the version's window_size gives), keeps each model's Kalman state current point by point, and reports, per point, how many standard errors of its forecast the reading landed from the forecast: the decision value is −max|z| over the features, so the margin reads as a sigma count like the range guard's, and the version names the feature.

State. The models' filtered states are persisted (forecast.json) with the absolute sequence number of the next ring row to absorb; the service opens a model per request, so a request catches the states up from the ring's stored rows before judging — a few rows, parsed on the spot — and writes the file every ANOM_FC_SAVE_EVERY rows absorbed. A row's reading that is missing is a gap to the model (arima_update with NaN): the forecast moves on, nothing is learned. A scan over stored rows replays a copy of each model from ANOM_FC_BURN rows before the window (a restart, then the rows), so the verdicts of a scan are those of a stream that began there; the live stream's are the same to the precision the burn-in leaves.

Training. The fit window is the version's own (window_points rows back, default ANOM_FC_WINDOW; window_minutes too), the features fit on the machine's threads, and a feature is watched when it is a declared numeric column with enough distinct, present readings to fit. Gaps inside the fit window are bridged linearly for the fit only.

API

: i ANOM_FC_SAVE_EVERY 32

Rows absorbed between two writes of forecast.json.

: i ANOM_FC_BURN 500

Rows a scan's replay runs before the window it reports on.

: i ANOM_FC_MIN_FIT 30

Fewer present readings than this in the fit window: not watched.

: i ANOM_FC_SARIMA_MAX 168

A season up to this many rows is a SARIMA polynomial (the week at an hour's step); a longer one — the day at a minute's — is Fourier terms with ANOM_FC_HARMONICS harmonics, which cost O(K) a row where the polynomial's state would be the season squared (see arima_fit_harmonic).

: i ANOM_FC_HARMONICS 4

: i ANOM_FC_WEEK_MIN 3

The second season: the week, seven of the first, as Fourier terms when the fit window covers at least this many weeks.

: i ANOM_FC_HOLDOUT_MIN 40

Model selection: every candidate form is fitted on the window's first part and judged on a holdout of its last fifth (at least ANOM_FC_HOLDOUT_MIN rows, at most ANOM_FC_HOLDOUT_MAX) by the mean absolute error of its forecasts up to ANOM_FC_SELECT_H steps ahead from every holdout row; the best form is refitted on the whole window. The error of carrying the last value forward, on the same rows and steps, is kept beside it as the skill.

: i ANOM_FC_HOLDOUT_MAX 600

: i ANOM_FC_SELECT_H 12

: f ANOM_FC_SELECT_MARGIN 0.05

The candidates are tried simplest first, and a later, richer form replaces the best only when its holdout error is smaller by this share: a near-unit-root ARIMA mimics a drift over twelve steps to within a percent and then runs away, and the tie should go to the form that says what the series does.

: f ANOM_FC_DIFFUSE 1000.0

A forecast variance past this many σ² is the diffuse start still speaking: no verdict.

: f ANOM_FC_SE_FLOOR 0.02

The verdict's scale has a floor: a reading is judged against the larger of the forecast's standard error and this share of the feature's own spread (1.4826 × the median absolute deviation of the fit window). A feature the model reproduces almost exactly — a calendar sine fed as data, a smoothed reading — has a standard error near zero, and without the floor a deviation invisible on the feature's own scale is a thousand sigma.

: f ANOM_FC_DETERMINISTIC 0.000001

A feature whose holdout error is below this share of its spread is deterministic — a signal the model can write down — and is not watched: there is nothing in it to be surprised by.

: FcModel

: FcModel {
    b trained
    ( Vec String ) feats
    ( Vec i ) cols
    ( Vec i ) models
    i nw
    i pos
    i seq
    i season
    i trained_at
    i trained_on
    i unsaved  // rows absorbed since the file was last written
    i origin_seq  // the absolute sequence number of the fit window's first row: the regressors' t = 0
    ( Vec String ) sel  // per watched feature: the form the holdout chose ("arima", "sarima", "fourier4", "fourier4+week", …)
    ( Vec f ) sel_mae  // its holdout error
    ( Vec f ) sel_naive  // the naive forecast's on the same rows and steps
    ( Vec String ) skipped  // numeric features not watched, each "name: why"
    ( Vec f ) scale  // per watched feature: its spread in the fit window (the verdict's floor is a share of it)
}

The trained version. models holds *ArimaModel per watched feature; feats their names and cols their index in the metadata's feature order, both frozen at training. pos is the ring row the states have absorbed up to (exclusive) in the open model; seq the same as an absolute sequence number, which is what the file carries.

@ fc_new → *FcModel

@ _fc_model_at * FcModel fc i j → *ArimaModel

@ _fc_geti ( Vec i ) v i k → i

@ _fc_getf ( Vec f ) v i k → f

@ fc_clear * FcModel fc → v

Drop the trained models; the handle stays, untrained.

@ fc_free * FcModel fc → v

@ fc_project EncPoint p ( Vec String ) feats → ( Vec f )

The watched features' readings of an encoded point, NaN where the point has none (a projection fills a missing feature with 0, which for a forecast would be a reading).

: FcJob

: FcJob {
    ( Vec f ) y
    i season  // the season in rows (0 = none)
    i out  // *ArimaModel, 0 until fitted
    String sel  // the form chosen
    f sel_mae
    f sel_naive
    f spread
}

One feature's fit, as a job for a worker thread: the series (gaps bridged), the season, the slot for the model and what the selection found.

: FcCand

: FcCand {
    i sarima
    ( Vec i ) periods
    i k
    String name
    b fixed  // ARIMA(0,1,0): the last value carried forward, as a candidate
    b trend  // a linear drift among the regressors
}

One form a feature's series can take: a SARIMA season (0 = none), Fourier periods in rows (empty = none) with k harmonics, a name.

@ fc_candidates i season i n → ( Vec FcCand )

The forms worth trying for a season of season rows over a fit window of n: the plain ARIMA always; the seasonal polynomial when the season fits a state (≤ ANOM_FC_SARIMA_MAX); Fourier terms of the season with two, four and six harmonics; and, when the window holds ANOM_FC_WEEK_MIN weeks, the week as Fourier terms beside the best daily form of each kind.

: FcScore

: FcScore {
    f mae
    f naive
}

The holdout error of one form: fitted on rows [0, nfit), then from every row of [nfit − 1, n − 1) the forecast of the next steps (up to h) against the rows that followed, the row then absorbed. Returns the model's mean absolute error; the naive forecast's — the origin's value carried forward — over the same comparisons goes to naive.

: FcLane

: FcLane {
    ( Vec i ) jobs
    i lane
    i stride
}

: FcSeries

: FcSeries {
    ( Vec f ) y
    i present
    b distinct
    i kind
    f spread
}

One feature's fit series over rows [from, n) of the hist: the present readings with the gaps between them bridged linearly (a gap at either end takes the nearest reading); how many readings were real, and whether they were not all one value. kind: 0 a reading worth forecasting; 1 binary (two values at most); 2 a counter — a non-decreasing count or a "seconds since" that rises by a fixed step and resets, which no ARIMA should be asked to follow. spread is 1.4826 × the median absolute deviation of the present readings (their standard deviation when that is 0).

: s FC_KIND_BINARY binary (two values at most)``

: s FC_KIND_COUNTER counter (rises by a fixed step and resets)``

@ fc_train * FcModel fc * Meta mm ( Vec EncPoint ) encs i from i season i now i base_seq → i

Train from the encoded ring: encs are every ring row in order, from the first row of the fit window, season the period in rows (0 = none), now the clock. Every declared numeric feature of the frozen order with ANOM_FC_MIN_FIT present, not-all-equal readings in the window gets a model; the models are then filtered over the whole ring so their states stand at its end. Returns the number of features watched.

@ fc_absorb * FcModel fc ( Vec f ) raw → v

Absorb one row of readings (the watched features' values, NaN = gap).

@ fc_evict * FcModel fc → v

The ring evicted its oldest row.

: FcOut

: FcOut {
    b ready  // at least one feature judged
    f worst  // the largest |z|
    i feat  // its index in the metadata's feature order; -1 = none
    ( Vec f ) z  // per watched feature; NaN where not judged
}

What the version saw of a point.

@ fc_out_free FcOut o → v

@ fc_judge * FcModel fc ( Vec f ) raw b absorb → FcOut

Judge a row of readings against the models' current forecasts. With absorb the readings are then absorbed (the live ingest: the row is the ring's newest); without, the states stay (detect_only).

: FcForecast

: FcForecast {
    ( Vec String ) feats
    ( Vec ( Vec f ) ) mean
    ( Vec ( Vec f ) ) se
}

The next h forecasts of every watched feature, from the states as they stand: means and standard errors, feature-major.

@ fc_forecast_free FcForecast o → v

@ fc_forecast * FcModel fc i h → FcForecast

@ fc_replay_begin * FcModel fc i seq → ( Vec i )

A copy of every model, restarted: the stream begins again at the row whose absolute sequence number is seq (the regressors' phase follows from it).

@ fc_replay_step * FcModel fc ( Vec i ) copies ( Vec f ) raw ( Vec f ) z → v

@ fc_replay_end ( Vec i ) copies → v

: FcPick

: FcPick {
    f worst
    i feat
}

The verdict a row of z-scores gives: the largest |z| and the feature (its index in the metadata's order; -1 when no feature was judged).

@ fc_worst ( Vec f ) z ( Vec i ) cols → FcPick

: s FC_FORMAT anomaly-forecast-1``

@ fc_to_json_str * FcModel fc → String

@ fc_from_json_str s src → ?*FcModel

None on a malformed document; a document whose models do not parse is untrained.

@ fc_seasonal_count * FcModel fc → i

How many of the fitted models actually use the season — the seasonal polynomial or Fourier terms of it. season is what the search was OFFERED; a stream whose features all chose the plain ARIMA reports a season it does not model, and a reader who takes the number at face value believes a daily rhythm is being watched when nothing of the kind is happening.

@ fc_selected_of * FcModel fc i j → s

The form a feature's fit chose ("naive", "arima", "fourier4", …).

@ fc_info_json * FcModel fc → Json

What a reader sees of the version (the metadata response's block).

---

prep.nu

anomaly/prep.nu — feature preprocessing, standardisation, model metadata.

This is milestone M1 of the anomaly package (see SPEC.md): the pure, I/O-free foundation everything else builds on.

• A model's metadata (*Meta) records what it has learned about the

shape of its input: which columns exist, their types (numeric / categorical / timestamp), the categories seen so far, the authoritative feature order, and the scaler parameters. It round-trips through JSON.

• anomaly_preprocess turns one raw JSON record into named numeric

features, updating the metadata as new columns / categories appear: numeric passthrough, categorical → deterministic one-hot (categories kept sorted), ISO-8601 timestamp → hour/day/month/weekday.

• anomaly_project pins a named feature set onto the model's frozen

feature order: missing features become NaN (absent; the scaler standardises them to 0, the mean), unknown extras are dropped. This is the feature-order stability rule that keeps one-hot columns aligned across retrains.

• Scaler is the StandardScaler analogue: per-feature zero-mean /

unit-variance, with zero-variance features passing through unscaled.

Column types are detected once, on first sight, then frozen in metadata: a value that parses as a number is numeric, else a value that parses as ISO-8601 is a timestamp, else it is categorical (mirrors the Python reference's float()/fromisoformat()/str fallback chain).

API

: i COL_NUMERIC 0

: i COL_CATEGORICAL 1

: i COL_TIMESTAMP 2

: i ANOM_MIN_POINTS 50

Reference defaults (carried over verbatim from the Python service).

: i ANOM_MAX_POINTS 150000

: i ANOM_SCHED_BELOW 50

: i ANOM_SCHED_AT_MAX 1000

: VerCfg

: VerCfg {
    String vname
    i window_min
    i window_pts
    i window_size  // sliding-window LENGTH in points (timevector; 0 = plain)
    i step_size  // sliding-window step during training (timevector)
    i n_estimators
    i max_samples
    f contamination
    f decision_margin
    b enabled
}

Config of one time-window model version. window_min filters training data to the last N minutes (0 = no time filter); window_pts caps it to the last N points (0 = no cap; used by timevector). contamination < 0 means "auto" (offset pinned at -0.5, the sklearn convention).

: Meta

: Meta {
    String name
    String created
    String alias  // human-readable nickname; empty = go by `name`
    ( Vec String ) cols
    ( Vec i ) kinds
    ( Vec ( Vec String ) ) cats
    ( Vec String ) feats
    ( Vec f ) sc_mean
    ( Vec f ) sc_std
    i sched_below
    i sched_at_max
    b sched_ae  // retrain the autoencoder whenever the forests retrain
    b count_clock  // points are numbered, not timed: see ANOM_TICK
    i n_seen
    i n_stored  // rows in the ring right now (≤ max_points); n_seen minus the evicted
    i last_trained  // n_seen at the last train (a point count, not a time)
    i trained_time  // wall clock of the last train, unix seconds; 0 = never
    i tuned_at  // wall clock of the first margin calibration, unix seconds; 0 = never (see model_autotune_at)
    i max_points
    i score_epoch
    i feat_enc  // the calendar-feature encoding the stored feature order uses
    i train_span  // seconds the last train's rows covered; 0 = unknown, every cycle kept
    ( Vec f ) flat_run  // flatline reference per feature: the run length its training runs recur at (see ANOM_FLAT_RUN_Q; -1 = not watched)
    ( Vec f ) flat_sd  // flatline reference per feature: the quiet-window std of training (see ANOM_FLAT_SD_Q)
    ( Vec f ) absurd_n  // readings left out of the last fit per feature (see anomaly_mask_absurd)
    ( Vec VerCfg ) versions
}

Everything a model knows about its input shape. Heap-allocated (*Meta) so counters and handles can be updated through any reference. cols / kinds / cats are parallel: cats[i] is the sorted category list of column i (empty unless categorical). feats is the authoritative feature order once non-empty (snapshotted at each train by meta_refresh_feats); until then the model is "unfrozen" and the order is derived on demand.

: i ANOM_FEAT_ENC 2

── Calendar features ─────────────────────────────────────────────────

A timestamp column becomes calendar features so the forests can learn that 03:00 on a Sunday is not 15:00 on a Tuesday. Two things matter:

1. The clock is the one the stamp was WRITTEN in. "2026-08-01T00:00:00

+03:00" is midnight to whoever sent it; folding the offset away and reading the UTC fields would call it 21:00 the day before, and a model fed local-offset stamps would learn a day shifted by the offset — with a step every DST change. A stamp without an offset is taken as written.

1. Cyclic quantities are encoded as (sin, cos) pairs, so 23:00 sits

next to 00:00 and Sunday next to Monday. A linear hour lets a forest cut the day at midnight, where nothing changes.

ANOM_FEAT_ENC numbers this scheme. A model trained under an older one keeps encoding its points the old way (its frozen feature order names the old features) until its next retrain, which re-encodes the ring under the current scheme; metadata reports retrain_required until then. Encoding 1 is the original: UTC fields, linear hour / day-of-month / month / weekday.

: EncPoint

: EncPoint {
    ( Vec String ) names
    ( Vec f ) vals
}

One preprocessed record: parallel (feature name, value) pairs in encounter order. Project onto a feature order with anomaly_project.

: Scaler

: Scaler {
    ( Vec f ) mean
    ( Vec f ) inv_std
}

Per-feature standardisation: y = (x - mean) * inv_std. Zero-variance features store inv_std = 1 so they pass through centred but unscaled.

: s ANOM_GUARD_NAME range_guard``

The range guard (SPEC §5.4): not a forest but the univariate check the forests cannot make — a point whose one feature sits further from its training mean than decision_margin standard deviations is an anomaly on that alone, whatever the joint picture. Its decision value is -max|z| over the standardised features, so the margin reads as a sigma count; ANOM_GUARD_SIGMA is the default line. It has no window or trees: the shared scaler every retrain refits is all it needs.

: f ANOM_GUARD_SIGMA 4.0

@ _an_is_guard_name s vname → b

@ _an_vc_guard → VerCfg

: s ANOM_FLAT_NAME flatline``

The flatline guard (SPEC §5.4): the stuck sensor is the commonest single fault in a sensor stream and structurally invisible to a point scorer — every row of a flat stretch is, on its own, an ordinary reading; the signal is that nothing moves. This version measures two things per numeric column against what a retrain learned from the ring:

run how long the run of identical values ending at this row is, as a fraction of the column's OWN reference run — twice the length its runs reach in training (flat_run, the ANOM_FLAT_RUN_Q row-weighted quantile of run length), or ANOM_FLAT_MIN_RUN rows, whichever is longer. collapse how far the window's standard deviation has fallen below the stream's own quiet windows (the ANOM_FLAT_SD_Q-th quantile of the training window stds over window_size rows).

Both are fractions; the decision value is minus the larger, over the columns, so the margin reads as a fraction: 0.9 flags a column that has repeated one value for 90 % of its own reference run, or a window ten times flatter than the stream's quietest periods.

The reference is PER FEATURE and it decides: a temperature quantised to whole degrees, sampled every minute, legitimately repeats for half an hour, and its reference says so; a smooth flow meter's reference is the floor, so a genuine freeze of ANOM_FLAT_MIN_RUN rows trips it while the coarse column beside it in the same bundle stays quiet. Scoring the run against the model-wide window instead — what this guard did until 0.32.0 — made one number mean two things: a column whose reference run passed half the window could never reach the margin at all (the run is counted over the window, so its fraction was capped below 1), and every other column needed 0.9 × window identical rows before it counted.

The reference run must describe what the column DOES, not the longest thing that ever happened to it — otherwise one freeze inside the training ring teaches the guard that freezing is normal, and the guard immunises itself against the very fault it exists to catch. Two rules together make it a description of habit:

• every maximal run of L identical values is entered into the sample

min(L, ANOM_FLAT_RUN_CAP × n) times, so a run counts for MORE the longer it is — a rain gauge's long dry stretches are its normal — but no SINGLE stretch, however long, can weigh more than a hundredth of the ring;

• the reference is the ANOM_FLAT_RUN_Q quantile of that sample.

The two together say: a column's reference is set by behaviour that RECURS. A sensor that legitimately sits still for half an hour does it again and again, and every one of those runs votes; a fault does it once, and one stretch cannot outvote the rest of the ring no matter how long it lasts. A column whose reference run is longer than the guard can look back (ANOM_FLAT_TAIL_MAX) is left unwatched rather than watched with a bar it can never reach.

: i ANOM_FLAT_WINDOW 60

: f ANOM_FLAT_MARGIN 0.9

: f ANOM_FLAT_SD_Q 0.05

: f ANOM_FLAT_RUN_Q 0.9

: f ANOM_FLAT_RUN_CAP 0.01

: i ANOM_FLAT_MIN_RUN 20

: i ANOM_FLAT_TAIL_MAX 600

@ _an_is_flat_name s vname → b

@ _an_vc_flat → VerCfg

: s ANOM_FC_NAME forecast``

The forecast version (src/forecast.nu): a seasonal ARIMA per numeric feature, judging each reading against the forecast its own past made. Its decision value is −max|z| over the features — the reading's distance from the forecast in the forecast's standard errors — so the margin reads as a sigma count like the range guard's, and it names the feature. window_points / window_minutes are its fit window, window_size the seasonal period in rows (0 = none). Off until a reader turns it on: a model per feature is a cost a stream should choose.

: f ANOM_FC_SIGMA 4.0

: i ANOM_FC_WINDOW 2000

@ _an_is_fc_name s vname → b

@ _an_vc_fc → VerCfg

@ _an_forestless_name s vname → b

The versions that are not forests: the autoencoder, the two guards and the forecast have a VerCfg (margin, enabled) but no forest blob to load, train or drop.

@ meta_default_versions → ( Vec VerCfg )

@ _an_vercfg_free VerCfg vc → v

@ meta_clone_versions * Meta m → ( Vec VerCfg )

An owned copy of a model's version configuration — what a fork takes from its source.

: i ANOM_ALIAS_MAX 120

An alias is a label, so it is bounded by what a human will read rather than by anything structural. Long enough for a sentence, short enough that it cannot be used to bloat every metadata response.

: i ANOM_TICK 60

── The count clock ───────────────────────────────────────────────────

Data without a clock — a file of readings nobody dated, a sequence of measurements where only the order matters — is a model whose time is its point count. Such a model runs on the COUNT clock: every point is stamped one tick after the previous one, ticks are ANOM_TICK seconds apart, and nothing in it ever reads the wall clock. One tick is one minute on purpose: every version window is written in minutes, so on the count clock "180 minutes" reads as "the last 180 points", and every window, scan and calibration falls out of the same code unchanged. The number a point shows is its ordinal, timestamp ÷ ANOM_TICK.

@ meta_new s name s created → *Meta

@ meta_free * Meta m → v

@ meta_is_frozen * Meta m → b

A model is "frozen" once it has an authoritative feature order (set at first train). Before that, feature order is derived from the metadata.

@ meta_has_timestamp * Meta m → b

Whether any column is a timestamp — the only kind whose encoding has changed between ANOM_FEAT_ENC schemes.

@ meta_retrain_required * Meta m → b

A trained model whose frozen feature order was built under an older calendar encoding: it still scores, the old way, but its next retrain changes what it learns.

@ meta_declare_column * Meta m s name i kind → b

Declare a column's kind before its first value is seen — the one way to make a numeric-looking column categorical: a coordinate or a station code that should be an identity, not a magnitude. A column the model already knows keeps its kind (its encoding is settled); returns T only when the declaration took.

@ meta_derived_feats * Meta m → ( Vec String )

The feature order implied by the current metadata: columns in first-seen order, each expanded canonically (categoricals over their sorted categories). Deterministic for a given metadata state. Owned result.

@ meta_numeric_feat_mask * Meta m → ( Vec i )

Which features are a numeric column as it came in — 1 per feature, 0 for a one-hot level or a calendar feature. The flatline guard watches only these: a category that does not change and a month that does not change are not sensors.

@ meta_refresh_feats * Meta m → v

Snapshot the derived feature order as authoritative (called at train time). From now on scoring projects onto exactly this vector.

@ enc_free EncPoint p → v

@ enc_find EncPoint p s name → i

Index of feature name in the encoded point, or -1.

@ _an_iso_offset s stamp → i

The zone offset an ISO-8601 stamp ends in, in seconds: "+03:00" → 10800, "-05:30" → -19800, "Z" or no designator → 0. The stamp has already passed time_parse_iso, so the tail is well-formed.

@ anomaly_preprocess * Meta m Json raw → !EncPoint String

Encode one raw JSON record into named numeric features, updating the metadata as new columns / categories appear. The reserved key timestamp is the point's own clock and is never a feature. Numeric parse failure and bad timestamps are hard errors (owned message).

@ anomaly_missing_cols * Meta m Json raw → ( Vec String )

The model's columns a point does not carry — absent, or carried as null. A trained model expects every column it learned: a numeric one the point leaves out would otherwise encode as 0, which after standardisation is however many standard deviations 0 is from the column's mean, and the range guard would then blame a value nobody sent. Owned; empty when the point is complete.

@ anomaly_null_cols * Meta m Json raw ( Vec String ) missing → ( Vec String )

Of the columns anomaly_missing_cols named, the ones the point DID carry — as null. A JSON null is not a reading, so it counts as absent; but "Missing columns: TW" over a point that says "TW": null reads as the service not seeing what the caller plainly sent, and the caller looks for a transport bug instead of for the null.

@ anomaly_preprocess_ro * Meta m Json raw → !EncPoint String

Read-only encode: never touches the metadata. Unknown columns are skipped, unseen categories one-hot to all-zeros — exactly what the frozen-feature projection would do with them anyway. For detect-only paths that must not mutate model state.

@ anomaly_project EncPoint p ( Vec String ) feats → ( Vec f )

Project an encoded point onto an authoritative feature order: features not in feats are dropped, and a feature the point does not carry is NaN — "absent", which the scaler turns into 0 (the training mean, the one value that says nothing) and which the forecast reads as a gap. It used to be 0 in RAW units, which after standardisation was however many standard deviations zero lies from the column's mean: a sensor that skipped a tick read as a reading of nothing, the range guard blamed it, and every forest saw a point nobody sent. Owned result.

: f ANOM_ABSURD_SIGMAS 1000.0

── Absurd readings ───────────────────────────────────────────────────

A reading that cannot be a measurement of the same quantity — a sensor that answered 1e200, a unit conversion that multiplied by a googol — must not set anybody's boundaries. It is still stored, still scored and still flagged; it is flagged HARDER, because the range guard now sees it against a scale it did not move. It simply takes no part in FITTING.

Why it has to be left out rather than merely survived. The fitted scale is a mean and a standard deviation, and one reading D robust sigmas out inflates the std to about D/√n. Every real reading then standardises to z ≈ √n/D, so a few hundred sigmas is where the feature stops being watched at all — and it stays unwatched until the reading leaves the ring, which at a minute's step and the 150 000-point default is fourteen weeks. The forests fare no better: a split is drawn uniformly between a column's min and max, so one absurd reading makes nearly every split of that column useless. The autoencoder's MinMax and the forecast's ARIMA go the same way.

"Absurd" is measured against the feature's OWN robust statistics — the median and 1.4826·MAD, neither of which one reading can move — over a bounded, evenly spaced sample, so the cost does not grow with the ring. A cell past ANOM_ABSURD_SIGMAS of those becomes NaN in the TRAINING MATRIX only; the stored point keeps its value. Everything downstream already knows what NaN means there (0.29.0): the scaler leaves it out of the fit, the standardiser reads it as the mean, the autoencoder fills it with the midpoint of its column's range, and the forecast reads it as a gap in the series.

The threshold is not an anomaly threshold and must not be read as one. A hundred sigmas is a spectacular anomaly and belongs in the fit; a thousand is past where the fit survives at all, for any ring size this service supports.

: i ANOM_ABSURD_SAMPLE 2000

: i ANOM_ABSURD_MIN_SAMPLE 8

@ anomaly_mask_absurd ( Vec f ) data i n_rows i n_cols ( Vec i ) counts → i

Replace every absurd cell of a row-major training matrix with NaN, in place. One entry per column is pushed onto counts. Returns the total number of cells masked. A column with too few readings to say, or with no spread at all, is left exactly as it is: not being able to tell is not a licence to erase.

@ _an_finite f x → b

A finite float: neither NaN nor an infinity.

: f ANOM_FLOAT_MAX 1.7976931348623157e308

The largest finite double, for the one std that would overflow it.

: f ANOM_Z_CAP 1000000.0

The standardised value furthest from the mean a point can carry: a million standard deviations. Past that a reading is not more anomalous, only more likely to be a broken sensor, and the cap keeps every z-score — and every JSON score computed from one — a finite number: a reading of 1e308 against a std of 1e-3 would otherwise standardise to infinity and serialise as null.

@ scaler_fit ( Vec f ) data i n_rows i n_cols → Scaler

Fit per-feature mean and 1/std over a row-major matrix (population variance, like sklearn's StandardScaler). Zero-variance features get inv_std = 1 so they centre but never divide by ~0. A NaN cell is an absent reading (anomaly_project) and is left out of its column's statistics; a column with no readings at all gets mean 0, inv_std 1.

The arithmetic cannot overflow on finite input. The mean is a sum of v/m, not a sum divided by m; the deviations are taken as halves (v/2 − mu/2 is always finite) and squared only after being scaled by the largest of them, so a single reading of 1e200 gives a std of about 1e199 and not the infinity that once turned the persisted scaler into JSON nulls the model could no longer be opened from.

@ scaler_apply Scaler sc ( Vec f ) point → v

Standardise one point in place: x → (x - mean) * inv_std.

@ scaler_apply_matrix Scaler sc ( Vec f ) data i n_rows i n_cols → v

Standardise a whole row-major matrix in place.

@ scaler_free Scaler sc → v

@ meta_set_scaler * Meta m Scaler sc → v

Persist a fitted scaler into metadata (stored as mean + std; zero variance is stored as std = 1, matching its inv_std = 1).

@ meta_scaler * Meta m → Scaler

Rebuild a usable Scaler from persisted metadata. Owned result.

@ _an_jarr_of_floats ( Vec f ) xs → Json

( Vec f ) → JSON array of numbers.

@ _an_jarr_of_strs ( Vec String ) xs → Json

( Vec String ) → JSON array of strings.

@ meta_to_json * Meta m → Json

Serialise metadata to an owned Json object (fixed field order, so the same metadata always stringifies identically).

@ _an_jint Json o s key i dflt → i

Integer field of a JSON object, or dflt when absent/mistyped.

@ _an_vercfg_of_json s vname Json vo → VerCfg

One version config out of its JSON object.

@ meta_from_json Json j → ?*Meta

Parse metadata back from JSON. None on malformed shape (missing/mistyped required fields); the partially-built Meta is freed on failure.

@ meta_to_json_str * Meta m → String

Convenience: metadata → compact JSON text (owned).

@ meta_from_json_str s src → ?*Meta

Convenience: JSON text → metadata; None on parse or shape errors.

@ meta_find_version * Meta m s vname → i

Index of the version named vname, or -1.

@ meta_version_enabled * Meta m s vname b dflt → b

Is version vname enabled? dflt when there is no such version.

@ meta_version_margin * Meta m s vname f dflt → f

Version vname's decision margin; dflt when there is no such version. The live margin of a version always comes from the CURRENT metadata, not from its forest blob — so margin changes (fine-tune, a config PUT) take effect immediately, without a retrain. The blob's stored margin is only the fallback for versions no longer present in the metadata.

@ meta_bump_epoch * Meta m → v

Bump the scoring epoch: the token every cached verdict is stamped with. Anything that can change what a stored point scores — a retrain, a new autoencoder, a margin edit, a version toggled on or off, a reset — bumps it, and every cache entry carrying an older epoch is stale by construction. One counter beats trying to reason about which caches a given edit could have invalidated.

@ _an_vercfg_sane VerCfg vc → VerCfg

Clamp a config into the range the trainer can actually honour, so a hand-written JSON patch can never produce a version that fails to train (or trains something nonsensical). contamination < 0 means "auto".

@ _an_vercfg_patch VerCfg vc Json vo → VerCfg

Patch one version config from a JSON object. Every field is optional: what the object omits keeps its current value, so the dashboard can PUT {"enabled": false} without restating the whole config.

@ _an_unknown_keys Json o s allowed s prefix → String

The keys a JSON object carries that are not in allowed (a space-separated list), as "<prefix>.<key>" joined with ", "; empty when every key is known. A patch is a statement of intent, and a key the service does not read — a typo, a field from a later version, a setting that lives somewhere else — used to vanish without a word while the rest of the patch went through.

: s ANOM_VERCFG_FIELDS enabled decision_margin window_minutes window_points window_size step_size n_estimators max_samples contamination``

The fields a version config accepts in a patch, one string for the checker and the tool descriptions alike.

@ meta_versions_patch_check * Meta m Json vers b replace → String

Check a versions patch before applying it: every value an object, every field a VerCfg field. Returns the reason to refuse, or "". A versions patch, checked before anything is applied: every key must name a version the model has, and every field inside it must be one a VerCfg carries. A name the model does not have is a typo far more often than a new version — {"autoenocder": {"enabled": false}} used to answer success and leave the real autoencoder on — so it is refused here, with the names that do exist. Creating a version is still possible and still one flag away: replace_versions makes the object the WHOLE list, which is how the dashboard's JSON editor adds and removes them, and there the names it does not know are the point.

@ meta_apply_versions_json * Meta m Json vers b replace → i

Apply a versions JSON object (the shape meta_to_json emits) to the metadata. Each key names a version and its value is a PARTIAL config; a key naming no existing version ADDS one, with _an_vercfg_of_json's defaults filling the gaps. With replace, the resulting list is exactly the keys given — versions the object omits are dropped, which is how the advanced JSON editor deletes one. Returns the version count afterwards, or -1 when vers is not a JSON object.

Creation is gated ABOVE this, in meta_versions_patch_check: a partial patch may only edit versions the model has, and only a whole-list patch (replace_versions) may name one it does not. This function is the mechanism; the rule about who may use it lives with the patch.

---

main.nu

anomaly — streaming anomaly detection from the command line.

anomaly detect <model> key=val ... ingest one point → verdict JSON anomaly score <model> key=val ... score only (no state change) anomaly batch [-f FILE] [-H] score a CSV (index<TAB>score) anomaly train <model> force a retrain now anomaly train-ae <model> train the autoencoder version anomaly train-fc <model> [--season S] train the forecast version (SARIMA per feature) anomaly forecast <model> [--horizon H] the next H values per watched feature anomaly backtest <model> [--horizon H] [--points N] forecast accuracy vs naive baselines anomaly calibrate <model> [--last S] alert rate at the current margins, margin for each standard rate anomaly finetune <model> [--rate R] set margins so R of the window is [--last S|all|own] [-n] flagged (default 1 % of 24 h; own = each version its period) anomaly reset <model> drop data+forests, keep the name anomaly rm <model> delete the model entirely anomaly ls list models in the store anomaly info <model> dump model metadata (pretty JSON) anomaly serve [--addr HOST:PORT] run the HTTP/JSON service + dashboard [--webroot DIR] (dashboard HTML dir; auto-located)

Values in key=val are auto-typed: numbers are numeric features, ISO-8601 strings are timestamps, anything else is categorical (one-hot). Verdicts print as one JSON object per invocation. The store lives under --store DIR (default $ANOMALY_HOME, else ~/.anomaly).

The command line is assembled with the cli package: one Cli, a flag set, and a handler per subcommand — routing, typed flags with the $ANOMALY_HOME / $ANOMALY_WEBROOT fallbacks, --help / --version and exit codes are the framework's job.

API

@ pline s x → v

: s ANOM_DEFAULT_ADDR 127.0.0.1:8811``

The bind address a bare anomaly serve uses. Named so the flag's default and the "was --addr actually given" test cannot drift apart.

@ main → i

---

autoenc.nu

autoenc.nu — the autoencoder model version.

The reference recipe (model_training.py train_autoencoder_model), in pure NURL over the mlp package:

1. a TEMPORARY Isolation Forest (contamination 10 %) is trained on the

standardised ring and every point it flags is dropped — the autoencoder must learn only NORMAL behaviour, and unlabeled rings contain the very anomalies we want it to spot later;

1. the surviving rows are MinMax-scaled and an MLP autoencoder

(default 64-32-64, ReLU, Adam, early stopping — sklearn MLPRegressor semantics via mlp_fit with deterministic restarts) learns to reconstruct them;

1. the detection threshold is the 95th percentile (nearest-rank) of

the training reconstruction errors.

Detection scores a point as threshold − mse: negative ⇒ anomaly — the same decision_function orientation as the forest versions, so the service's "any version flags it" aggregation needs no special case.

The autoencoder is NEVER part of the automatic retrain schedule: it is trained explicitly (CLI anomaly train-ae, HTTP POST /train/autoencoder/<model>) and stays enabled until the model is reset — mirroring the reference, where a heavier, deliberately-trained version coexists with the self-training forests.

API

: AeModel

: AeModel {
    Mlp net
    MinMax mm
    ( Vec String ) feats  // feature order frozen at AE train time
    f threshold  // p95 of the training reconstruction errors
    i trained_on  // normal rows the net was fitted on
    i filtered  // rows the pre-filter dropped as anomalous
    f prefilter  // the pre-filter's contamination rate actually used
    i trained_at  // unix seconds (0 = unknown, pre-0.10 file)
    b trained
}

@ ae_empty → AeModel

A structurally-valid untrained placeholder (freeable, never scored).

@ ae_free AeModel ae → v

: AeTrainOut

: AeTrainOut {
    AeModel ae
    String err
}

Train from the RAW (unscaled) projected matrix raw (n×d, row-major) with its feature order feats (borrowed; copied into the result). hidden lists the hidden layer widths (empty → 64-32-64); contamination is the pre-filter rate (<=0 → 0.10); min_rows gates both the input and the post-filter survivor count. On failure the returned AeModel has trained=F and err (owned) says why.

: f AE_NORM_CAP 1000000.0

The normalised value furthest from the training range a point can carry into the net: a reading a million ranges out is not more anomalous than one a thousand out, and the cap keeps the error — and the decision the service serialises — a finite number.

@ ae_train_matrix ( Vec f ) raw0 i n i d ( Vec String ) feats ( Vec i ) hidden f contamination i min_rows → AeTrainOut

@ ae_hidden AeModel ae → ( Vec i )

The hidden layer widths the net was built with (sizes minus the input and output layers) — what a retrain needs to rebuild the same shape.

@ ae_mse AeModel ae ( Vec f ) raw_point → f

Reconstruction MSE of one RAW projected point (the AE's own feature order and MinMax are applied here — independent of the forests' standardising scaler).

: f ANOM_AE_MARGIN 0.05

── The decision margin, and why it is RELATIVE ───────────────────────

Every forest version's decision_margin is an absolute offset on a decision_function whose scale is fixed by construction: sklearn's convention puts normal points near 0 and anomalies below it, so 0.06 means the same thing for every model. The autoencoder's score is threshold − mse, and MSE has no such fixed scale — it is the mean squared error of MinMax-scaled features, which lands wherever the data puts it (1e-3 for one model, 2e-4 for another).

The Python reference (model_training.py) nevertheless applies its shared absolute-margin rule to the autoencoder too: the branch computes is_anomaly = mse > reconstruction_threshold, and thirty lines later, at the same indentation as the branch itself, is_anomaly = bool(score <= -decision_margin) overwrites it unconditionally — with the autoencoder default margin of 0.05. Against a threshold of ~5e-4 that demands a reconstruction error a HUNDRED times the p95 of the training errors, which mutes the one version that models the joint distribution. The reference ships that version disabled by default, so the muting was never noticed.

We keep the tunable knob and put it on the only scale that travels: decision_margin for the autoencoder is a FRACTION of the model's own reconstruction threshold. The stored default 0.05 now reads "flag at 5 % above the p95 training error" — the documented intent — instead of "flag at p95 + 0.05", and the same number means the same thing on every model. See SPEC §5.5.

@ anom_ae_margin AeModel ae f rel → f

The effective ABSOLUTE margin for rel, so the verdict rule stays the one every version shares: decision_function <= -margin ⇒ anomaly. Substituting ae_decision gives mse >= threshold * (1 + rel).

@ anom_ae_rel_margin AeModel ae f abs_margin → f

Turn an effective absolute margin back into the relative one stored in the metadata (the inverse of anom_ae_margin; 0 when untrained).

@ ae_feature_errors AeModel ae ( Vec f ) raw_point → ( Vec f )

Per-feature squared reconstruction error of one RAW projected point, in the AE's own feature order (length = ae.mm.n_cols). This is the attribution the forests cannot give: the autoencoder's error is the amount by which each feature failed to be predictable from the others, so the largest entries name the RELATIONSHIP that broke, not merely the value that was extreme. ae_mse is the mean of this vector; it keeps its own loop because it is on the per-point scoring path and this one allocates.

@ ae_reconstruct AeModel ae ( Vec f ) raw_point → ( Vec f )

The autoencoder's reconstruction of one RAW projected point, back in raw units (the MinMax undone), in the AE's feature order. For a flagged point this is what each feature "should" have been given the others — the value the broken relationship expected. A feature whose training range was a single value reconstructs to that value.

@ ae_decision AeModel ae ( Vec f ) raw_point → f

decision_function orientation: threshold − mse (negative ⇒ anomaly).

@ ae_to_json_str AeModel ae → String

@ ae_from_json_str s src → ?AeModel

@ ae_dummy_net → Mlp

Fallback constructors for the load path's option unwrap (never used on the success path; both arms above guarantee have_net/have_mm).

@ ae_dummy_mm → MinMax

---

authz.nu

anomaly/authz.nu — who is asking, and which models they may touch.

The service was single-user by construction: every route reached every model in one flat store. This file adds the three things that turns into a shared service — an identity, a tenant, and an owner — without moving a single stored model.

identity An OIDC bearer token, verified by the oauth package against the provider's own JWKS. The token names a subject (sub) and, on Entra, a tenant (tid). tenant One SQLite database per organisation, at <store>/orgs/<org>.db. The org is implicit in the file, so no query in here carries an org column and no query can forget one. owner A row in that database binding a model name to a subject. A model with no row is UNOWNED: legacy data that predates authentication, visible to admins so somebody can claim it, never silently absorbed into whoever logged in first.

Three roles. admin manages the whole organisation's models and its users and keys; viewer reads every one of its models and may change none of them — except the scratch models named llm_…, which every member may build and destroy (az_is_scratch_model); ingest is for machines that send points (keys only, never a person). A model belongs to the organisation, so every member SEES the same set; the roles differ in what they may DO. The first subject to authenticate from an organisation becomes its admin — there is nobody else who could have granted it, and an org whose only user is a viewer would be permanently unadministrable.

API keys are for machines that cannot do an interactive login (the Node-RED flows feeding these models). A key carries the identity of the user who created it and a role of its own, admin or ingest. Keys are stored as a SHA-256 of the secret; the plaintext exists once, in the response that creates it.

EVERYTHING HERE IS OFF BY DEFAULT. With ANOMALY_AUTH unset the resolver hands every request an authenticated admin principal and the service behaves exactly as it did before this file existed. That is deliberate: a deployment upgrades the binary first and turns on authentication when its identity provider is configured, not at the same instant.

API

: ~ s g_az_root .``

: i AZ_MODE_SIMPLE 0

Two modes, and they are different products.

simple No authentication at all. Anyone who opens the page sees every model, and what the API collects lands in one public organisation. This is what the service was before sign-in existed, kept as a mode rather than as a fallback so that a deployment which wants it says so. oidc Signed in, multi-tenant. A model belongs to an ORGANISATION, and nothing is created or collected without a credential that names one.

: i AZ_MODE_OIDC 1

: s AZ_PUBLIC_ORG public``

The organisation everything belongs to in simple mode. A real organisation with a real database, so the two modes share one shape and a deployment can turn sign-in on later without moving data.

: ~ i g_az_mode 0 // AZ_MODE_SIMPLE; a global initialiser must be a literal

: ~ b g_az_open_ingest F

: AzStrings

: AzStrings {
    String s_issuer
    String s_client_id
    String s_audience
    String s_allowed
    String s_last_err  // why the last token verification failed
    String s_owner  // the owner tenant: config-only, the trust anchor
}

The configured strings, owned. A global can hold a s (a borrowed char*) but not a String, so the Strings behind these live in one heap block whose address is the global; reconfiguring frees the previous block. The obvious alternative — hand the globals ( string_data owned ) and never free the owned String — is a leak that a long-running service would never notice and a test that reconfigures would report every time.

: ~ i g_az_strs 0

@ g_az_issuer → s

@ g_az_client_id → s

@ g_az_audience → s

@ g_az_allowed → s

@ g_az_owner → s

The OWNER TENANT: the organisation whose admins administer the service itself — approving other tenants, managing any organisation's users. It is set in the configuration file and nowhere else. A tenant that could grant itself that from the dashboard would not be an anchor.

@ anomaly_authz_set_owner_tenant s tid → v

@ anomaly_authz_last_error → s

Why the last presented token was refused. A 401 with no reason is what turns a one-line configuration mistake — the wrong audience, a clock an hour out, a tenant nobody listed — into an afternoon. The service is single-threaded, so one slot is the whole story.

: ~ i g_az_prov_addr 0

The identity provider, discovered lazily and then reused: it owns the JWKS cache, and re-fetching a key set per request would turn every authenticated call into two network round trips. The service runs single-threaded (http_app_listen with no worker pool), which is the condition *OidcProvider documents for going unlocked.

Held as an address because a global cannot carry an option of a pointer; 0 means "not discovered yet". Discovery happens once and the provider then lives for the process, so nothing here frees it.

: ~ b g_az_multi F

── Multi-tenant ──────────────────────────────────────────────────────

A single-tenant application has one issuer, and a token either carries it or is refused. A MULTI-tenant one does not: every organisation signs its users' tokens with its own issuer, and the provider says so — Entra's discovery document at the multi-tenant authority literally publishes

"issuer": "https://login.microsoftonline.com/{tenantid}/v2.0"

which is a template, not a URL. So there is nothing to pin, and the oauth package's discovery refuses it: it cross-checks the document's own issuer against the one asked for, correctly, and a template never matches.

What is checked instead is what the provider itself documents: the token's iss must be that template with the token's OWN tid substituted. Both claims are inside the signature, so an attacker cannot move a token between tenants; what the check enforces is that a token claiming tenant X was issued by tenant X's issuer, which is the part a fixed string would have got wrong.

The trade is that ANY organisation can then sign in and have an organisation created for it. allowed_tenants is the list that says which may; empty means any, which is what "multi-tenant" asks for and should be a deliberate answer rather than a default nobody saw.

: ~ s g_az_iss_tmpl ``

@ anomaly_authz_set_root s root → v

@ anomaly_authz_configure b on b open_ingest s issuer s client_id s audience → v

on gates the whole thing; open_ingest keeps /detect and /detect_only reachable without credentials while a fleet of already-deployed data producers is migrated onto keys. It is a migration setting: with it on, anyone who can reach the port can write points into any model.

@ anomaly_authz_configure_tenancy b multi s allowed → v

Multi-tenant acceptance. allowed is a comma-separated tenant list; empty admits every organisation the provider will sign for.

@ anomaly_authz_multi_tenant → b

@ anomaly_authz_allowed_tenants → s

@ anomaly_authz_apply AnomalyConfig cfg → b

Resolve the settings from the configuration file and the environment, in that order — the file is the persistent baseline, the environment is what one run overrides. (A command-line flag beats both, but nothing here reads argv: main.nu owns that layer.) Returns T when authentication ended up enabled.

[auth] enabled / issuer / client_id / audience / open_ingest ANOMALY_AUTH ANOMALY_OIDC_ISSUER ANOMALY_OIDC_CLIENT_ID ANOMALY_OIDC_AUDIENCE ANOMALY_OPEN_INGEST

audience defaults to api://<client_id> — the app-id URI an OIDC provider hands out for an API the client registered for itself, which is what nearly every deployment wants and none should have to restate.

@ anomaly_authz_requested AnomalyConfig cfg → b

True when authentication was ASKED for — by file or environment — so a caller can tell "off because nobody asked" from "off because it was asked for without an issuer or a client id".

@ anomaly_authz_enabled → b

@ anomaly_authz_mode → i

@ anomaly_authz_simple → b

@ anomaly_authz_owner_tenant → s

@ anomaly_authz_open_ingest → b

@ anomaly_authz_issuer → s

@ anomaly_authz_client_id → s

@ anomaly_authz_audience → s

: s AZ_ROLE_ADMIN admin``

: s AZ_ROLE_VIEWER viewer``

: s AZ_ROLE_INGEST ingest``

A key is a credential, not a person, so it does not carry a person's role. A viewer is somebody who reads; a machine that reads is pointless and a machine that can delete a model it feeds is a hazard. The one thing a producer needs is to send points, so that is its own capability.

: Principal

: Principal {
    b authed
    b via_key  // arrived with an API key rather than a browser token
    String org
    String sub
    String email
    String pname
    String role
    String key_id  // empty unless via_key
}

@ principal_free Principal p → v

@ principal_anon → Principal

@ principal_public_admin → Principal

The principal every request gets in simple mode: an admin of the shared public organisation. A real organisation with a real database, so the two modes share one shape and turning sign-in on later does not have to move anything.

@ principal_local_admin → Principal

@ principal_is_admin Principal p → b

@ principal_is_owner_admin Principal p → b

An admin of the owner tenant: the one principal that reaches across organisations. Everything it can do, an ordinary admin can do inside its own organisation and nowhere else.

@ principal_may_ingest Principal p → b

May this credential send points to a model the organisation already has?

An admin may, because an admin may do anything here. A KEY may if it was issued to — that is what ingest names, and it is the whole point of having a separate capability: a production key should be able to feed a model without being able to delete it.

A viewer may not. A viewer reads what the models have collected and decided; sending points is not reading.

@ principal_json Principal p → Json

@ az_home_org → String

@ az_is_home_org s org → b

@ az_model_in_public s name → b

Is name held by the public organisation — the bucket for points that arrived without a credential naming an owner?

@ az_model_release_public s name → b

Release a model from the public organisation, so it can be adopted into a real one. Ownerless data is not the public organisation's property; it is data nobody has claimed yet, and public is where it waits.

@ az_db_open s org → !Database SqliteErr

Open (creating if absent) the organisation's database with its schema applied. The caller owns the handle; it closes at the caller's scope end.

: s AZ_TENANT_PENDING pending``

: s AZ_TENANT_ALLOWED allowed``

: s AZ_TENANT_BLOCKED blocked``

@ az_root_open → !Database SqliteErr

@ az_tenant_state Database db s tid → String

The recorded state of tid, or "" when it has never been seen.

@ az_tenant_note Database db s tid s label i now → String

Record that tid knocked. First time: pending. Afterwards the state is whatever was decided, and a repeat visit must not undo a decision.

@ az_tenant_set_state Database db s tid s state s by i now → b

@ az_tenant_forget Database db s tid → v

@ az_tenants_json Database db → Json

@ az_tenant_admitted s tid i now → b

May this tenant use the service? The owner tenant always may — it is the anchor the rest is decided from, and locking it out would leave nobody who could unlock anything. Everything else is a recorded decision, and a tenant nobody has decided on is recorded as pending and refused.

@ az_seed_allowed s csv i now → v

Seed the registry from the configuration file. Only ever ADDS: a deployment that lists tenants in its config gets them admitted without a dashboard, and a decision made in the dashboard is never undone by a restart.

@ az_user_count Database db → i

@ az_user_role Database db s sub → String

The role recorded for sub, or "" when the organisation has never seen them.

@ az_user_touch Database db s sub s email s name i now → String

Record that sub was here, and return the role they hold. The FIRST subject an organisation ever sees becomes its admin: there is nobody else who could grant it, and an organisation whose every user is a viewer can never appoint one.

@ az_user_set_role Database db s sub s role → b

@ az_admin_count Database db → i

@ az_users_json Database db → Json

@ az_model_owner Database db s name → String

The owner of name, or "" when the model has no row — UNOWNED, which is what every model created before this file existed looks like.

@ az_model_claim Database db s name s sub i now b force → b

Claim name for sub. force reassigns a model that already has an owner (admin territory); without it an owned model is left alone and the call reports F.

@ az_model_forget Database db s name → v

@ az_org_model_names Database db → ( Vec String )

Every model name this organisation claims, owned by anyone in it. This is the admin's whole world: the store is flat and global, so "every model" is every model of every ORGANISATION, and an admin of one tenant must never be shown another's.

@ az_all_owned_json Database db → Json

Every model the organisation has a row for, owned by anyone. An admin listing needs this only to know which stored models are still unclaimed.

@ az_may_see Database db Principal p s name → b

May this principal see the model at all?

A model belongs to an ORGANISATION, not to a person. Everyone in the organisation sees the same models; what differs is what they may do to them. That is the shape a shared service actually has — a colleague leaving must not take a production model with them — and it is why models has no owner column any more, only the database it sits in.

@ az_model_in_org Database db s name → b

Does this organisation hold this model?

Row EXISTENCE, not a non-empty creator: the two are different questions and conflating them means forgetting a person also forgets which organisation their models belonged to. owner_sub survives only as an audit trail of who first sent a point, and is blanked when they leave.

@ az_may_write Database db Principal p s name → b

May this principal CHANGE it? Only an admin. A viewer sees the organisation's models and the data they have collected — that is what a viewer is for — but retraining rewrites forests, a margin edit changes every verdict, and a reset destroys history. None of that is viewing.

The one exception is the organisation's SCRATCH namespace: a model named llm_… is an experiment, and an agent — or the viewer driving it — may build, tune and discard its own experiments without an administrator's signature. Production models keep the full rule. This is the law both the HTTP API and the MCP surface enforce, so the two never disagree.

: s AZ_LLM_PREFIX llm_``

Models whose name starts with this prefix are scratch models: every member of the organisation may create, retrain, tune and delete them.

@ az_is_scratch_model s name → b

Is this a scratch model by name? The bare prefix is not a name.

@ az_user_delete Database db s sub → b

@ az_org_models_before_delete Database db → ( Vec String )

Every model name this organisation holds, so a caller deleting the organisation knows what to remove from the store.

@ az_org_drop s org → b

Delete the organisation's database outright. The models themselves live in the shared store and are the caller's to remove — this only forgets who they belonged to, and doing both in one place would put a model deletion inside a function whose name says "database".

: s AZ_KEY_PREFIX anok_``

: KeyIssue

: KeyIssue {
    String key_id
    String secret  // the plaintext, which exists only here and in the response
}

@ key_issue_free KeyIssue k → v

@ az_key_create Database db s sub s label s role i now → KeyIssue

: String token ( string_from AZ_KEY_PREFIX )

The token the caller presents: prefix, id, secret.

@ az_key_revoke Database db s id i now → b

Revoke one of the organisation's keys. No owner check: the key is the organisation's, and whoever happened to press the button that created it has no more claim on it than any other admin.

@ az_keys_json Database db → Json

The keys sub may see: their own, or every one in the organisation when all is set (an admin listing). Secrets are never in here. The organisation's keys. There is no per-person view of them, because there is no per-person ownership: a key is the ORGANISATION's credential, and only its admins have any business seeing that one exists.

: KeyParts { b ok String kp_id String kp_secret }

Split "anok_<id>_<secret>" into its two halves. F when the token is not of that shape, which is also how the resolver tells an API key from a JWT.

@ key_parts_free KeyParts k → v

@ az_key_principal Database db s org KeyParts kp i now → Principal

Resolve a presented key against one organisation's database. The principal comes back unauthenticated when the id is unknown, the secret does not match, the key is revoked, or its owner has since been removed.

@ _az_org_ids → ( Vec String )

Every organisation database in the store, by org id. An API key names no organisation — the id alone would have to be globally unique to do that without a second index — so a presented key is tried against each. There is one database per tenant and a key arrives a few times a minute, so the scan is cheaper than a second source of truth that could disagree.

@ authz_principal_at HttpRequest req i now → Principal

The one entry point the service uses: turn a request into a principal. With authentication off this is an admin of the local organisation, which is what keeps every existing deployment working unchanged.

@ authz_principal HttpRequest req → Principal

---

score.nu

anomaly/score.nu — bulk scoring, version training, and the stateless batch path (milestones M2 + M7's GPU acceleration).

Bulk scoring routes through the gpu package when profitable:

• gpu_open picks CUDA when a device is present, else the gpu

package's CPU backend (the same kernel compiled by the host C++ compiler and run under OpenMP). If neither is available — no GPU, no C++ compiler — scoring silently stays on the pure-NURL loop, so a bare machine behaves exactly like pre-GPU builds.

• Output is BIT-IDENTICAL across all three paths, by construction: the

kernel only walks the trees and accumulates f64 path lengths in the same order as the pure loop, using per-leaf c(size) values precomputed on the host by the very same iforest_avg_path; the nonlinear finish (2^(-avg/c), the offset subtraction) runs in NURL either way. IEEE-754 f64 adds in a fixed order give the same bits on every backend — so backend choice can never change a verdict.

ANOMALY_GPU=0 disables the accelerator outright; NURL_GPU=cpu (a gpu package knob) forces its CPU backend on a CUDA machine. Batches under ANOM_GPU_MIN_ROWS rows skip the accelerator — upload latency would beat the win, and bit-equality makes the switch invisible.

API

: i ANOM_GPU_MIN_ROWS 128

Below this many rows the pure loop wins on latency. Bit-equality of the paths makes the threshold unobservable in results.

: BatchReport

: BatchReport {
    i total_rows
    i anomaly_count
    f anomaly_percentage
    ( Vec i ) anomaly_indices
    b has_anomalies
    ( Vec f ) scores
}

Batch verdict over a matrix (see SPEC §6): anomaly ⇔ predict == -1, i.e. decision_function < 0 (margin 0), matching the reference batch path.

: ~ i g_ag_state 0

: ~ i g_ag_kit 0

@ anom_gpu_kit → *GpuKit

Public view of the kit for sibling modules (aegpu.nu shares the device — __-prefixed functions are file-scoped). Only valid after anom_gpu_engine reported cuda or cpu.

@ anom_gpu_close → v

Release the device + kernel cache (tests call this so leak checkers see a closed shop; a long-running service just keeps the singleton).

@ anom_gpu_engine → s

Which engine bulk scoring would use right now: cuda, cpu (the gpu package's host backend) or none (pure NURL loop).

@ anom_scores_cpu VerModel vm ( Vec f ) scaled i n_rows i n_cols → ( Vec f )

Pure-NURL reference path: iforest_score per row.

@ anom_scores_gpu VerModel vm ( Vec f ) scaled i n_rows i n_cols → ?( Vec f )

Accelerated path. None when the accelerator is unavailable or any step fails (caller falls back; failures after a good probe are reported).

@ anom_scores VerModel vm ( Vec f ) scaled i n_rows i n_cols → ( Vec f )

Raw iforest scores for every row of an already-standardised matrix, through the best available engine. Bit-identical on all engines.

@ anom_decisions VerModel vm ( Vec f ) scaled i n_rows i n_cols → ( Vec f )

decision_function for every row: -score - offset, same expression as anom_decision.

@ _an_percentile ( Vec f ) sorted f q → f

q in [0,1] over an ASCENDING-sorted vector.

@ anom_train_version ( Vec f ) scaled i n_rows i n_cols VerCfg cfg → VerModel

Train one version's forest over an ALREADY-STANDARDISED row-major matrix. cfg.max_samples is clamped to the row count; contamination < 0 = "auto" (offset pinned at -0.5), else offset_ is the 100*c percentile of the training set's score_samples — so predict == -1 flags ~c of training.

@ anomaly_batch ( Vec f ) data i n_rows i n_cols VerCfg cfg → BatchReport

Fit a scaler over the raw matrix, train one version on the standardised copy, then score every row. Anomaly ⇔ decision_function <= -cfg.margin (margin 0 reproduces the reference's predict == -1 batch rule).

@ anomaly_report_free BatchReport rep → v

---

sources.nu

anomaly/sources.nu — data sources: configured once, fetched on a schedule.

A model fed by a producer gets its points pushed. A model fed from a public service — a weather office's WFS — has to go and get them, and somebody has to say from where, which columns, into which model and how often. That somebody is an organisation's administrator, and what they say is a SOURCE:

<root>/orgs/<org>/sources/<id>.json

{ "id", "name", "kind": "wfs" | "http", "url", "query", "params": {…}, "mode": "stored" | "type", WFS: a stored query, or a feature type "method", "headers": {…}, "body", "path", HTTP: the request, and where the records are "features": ["t2m", "ws_10min"], the columns kept (empty = all) "categorical": ["lat", "lon"], columns stored as text → one-hot "time_field": "", a feature type's clock ("" = detect) "model", "interval_minutes", "history_hours", "calendar", "enabled", "created_by", "created_at", "updated_at", "first_time", "last_time", the span of observation time fetched so far "last_run", "last_status", "last_error", "last_rows", "runs", "total_rows" }

A stored query is asked for the window the source has not seen yet — (last_time, now], a day per request. A feature type is fetched whole each run (at most count features): its features carry their own clock in a date property, or none — then every run stores a snapshot stamped with the fetch time, and the model learns how the snapshots drift. Either way the answer is pivoted into points (src/wfs.nu), the chosen columns kept — a categorical one written as text, so a coordinate or a station name becomes a one-hot identity and anomalies are judged per place — and imported with their own timestamps (model_import), so a run is the same act as importing a file of history and the model learns from it the same way. The first run reaches back history_hours; a backfill reaches further back, to before first_time, so no window is fetched twice and no point lands twice.

The scheduler is one thread that wakes every few seconds, runs what is due, and holds the service lock only while it touches the store: the network wait happens with the lock released, so a slow service does not stall a single live detection.

Windows are chunked to a day per request — a year of ten-minute observations is not one answer, and a service that caps a request's span answers a day.

API

: i SRC_ID_LEN 12

: i SRC_NAME_MAX 80

: i SRC_QUERY_MAX 200

: i SRC_PARAMS_MAX 40

: i SRC_FEATURES_MAX 200

: i SRC_INTERVAL_DEFAULT 10 // minutes

: i SRC_INTERVAL_MAX 10080 // a week

: i SRC_HISTORY_DEFAULT 168 // hours, the first run: a week, so a daily rhythm is seen seven times

: f SRC_FINETUNE_DEFAULT 0.01 // the share of the ring the first train's calibration flags

: i SRC_HISTORY_MAX 8760 // a year

: i SRC_CHUNK_SECS 86400 // one request covers at most a day

: i SRC_TICK_MS 15000 // the scheduler's wake-up

: s SRC_KIND_WFS wfs``

: s SRC_KIND_HTTP http``

: i SRC_HEADERS_MAX 20

: i SRC_BODY_MAX 65536

: s SRC_MASK ••••••••``

What the API shows in place of a header's value: a key is an admin's secret, and the record is readable by every member. Sent back as a value it means "keep what is stored".

: s SRC_MODE_STORED stored``

: s SRC_MODE_TYPE type``

: i SRC_FAR_FUTURE 315360000

A feature type's forward window has no upper edge: a forecast's rows lie in the future and still belong to this run.

@ _src_jint Json o s key i dflt → i

@ source_id_ok s id → b

A source id is what rand_hex_str makes: lowercase hex, SRC_ID_LEN long.

@ source_path s org s id → String

@ source_load s org s id → ?Json

@ source_save s org Json src → b

Written whole to a temp name and renamed over the old: a crash mid-write leaves the previous record, not half of the new one.

@ source_delete s org s id → b

@ sources_list s org → ( Vec Json )

Every source of the organisation, oldest first.

@ sources_free ( Vec Json ) xs → v

@ source_apply Json src Json body → String

Apply body (what a caller sent) to src (a record, fresh or loaded), field by field, refusing the first thing that is wrong. Fields the body leaves out keep what the record has. Returns "" when it is good.

@ source_is_http Json src → b

@ source_create s org Json body s by i now → !Json String

Create a source from body; Ok(the saved record) or Err(why).

@ source_update s org s id Json body i now → !Json String

: SrcWindow { i start i end }

@ source_is_type Json src → b

The span a run fetches. Forward: from just after the newest observation seen to now, or history_hours back on the first run. Backfill: from hours back to just before the oldest observation seen — never over ground already covered. An empty window has end < start. Fetched whole on every run, its clock in the records: a feature type, and an HTTP source alike.

@ source_window Json src i now b backfill i hours → SrcWindow

: SrcProject

: SrcProject {
    ( Vec Json ) points
    i outside  // rows whose timestamp fell outside the window
    i empty  // rows with none of the chosen features
    i unstamped  // rows whose time could not be read
    i oldest
    i newest
}

@ source_fetch Json src i start i end → !( Vec Json ) String

GET the window in day-sized chunks; every chunk's rows into one list. Err(why) on the first failed chunk — a partial answer is not imported, because the source's span would then claim ground it does not hold.

: ~ i g_src_running 0

Sources mid-run, as "<org>/<id>", so a manual run and the scheduler never fetch the same source at once. Touched only under the service lock.

@ source_is_running s org s id → b

@ source_step_of ( Vec Json ) points → i

The step of a run's points: the median gap between their timestamps in seconds, 0 when there are too few to say.

@ _src_geti ( Vec i ) v i k → i

@ source_season_of i step → i

The seasonal period, in rows, a step implies: the day for a step up to twelve hours (144 rows at ten minutes, 24 at an hour), the week for a daily step, none otherwise.

@ source_run_rows s org s id ! ( Vec Json ) String fr SrcWindow w b backfill i now → Json

What a run does once the answer is in hand: the record re-read (it may have been edited while the fetch ran), the rows projected onto the window and the chosen features, the points imported, the span and the statistics updated, the record saved. Public so a body obtained some other way — a test's fixture, a file — takes the same path. rows are consumed. The caller holds the service lock.

@ source_run s org s id b backfill i hours i now ( @ v ) unlock ( @ v ) lock → Json

One run of one source: the window, the fetch with the service lock released, then source_run_rows with it held. The answer is what the API returns and what the log line says.

backfill with hours reaches back before the oldest observation the source has seen; otherwise the run is forward.

: SrcRef { String org String id }

@ sources_due i now → ( Vec SrcRef )

Every enabled source whose interval has passed since its last run.

@ sources_tick i now ( @ v ) unlock ( @ v ) lock → i

Run everything due. Called with the service lock held; the lock is let go for each fetch. Returns how many sources ran.

@ sources_start_scheduler ( @ v ) unlock ( @ v ) lock → b

The scheduler thread: wake, take the lock, run what is due, let go. Detached; it lives as long as the service.

---

store.nu

anomaly/store.nu — persistence (milestone M3).

A model lives in its organisation's SQLite database, <root>/orgs/<org>.db — the same file that holds the organisation's members, roles and API keys. See "The organisation's database" below for the tables, for what the flat directory-per-model store it replaced could not do, and for how this behaves when several threads use one store.

The forest blob ("ANOMFOR1") is a little-endian dump of the iforest node arena plus the version's decision offset/margin. Loading re-validates every structural invariant (lengths, index ranges, node-count caps), so a corrupt or truncated blob comes back as None — never undefined behaviour.

API

: i ANOM_BLOB_MAX_NODES 200000000

Refuse to load blobs claiming more than this many arena nodes / trees / name bytes — bounds untrusted counts before any allocation.

: i ANOM_BLOB_MAX_TREES 100000

: i ANOM_BLOB_MAX_NAME 4096

: BlobRd

: BlobRd {
    ( Vec u ) buf
    i pos
    b ok
}

@ vermodel_to_bytes VerModel vm → ( Vec u )

Serialise one trained version (forest + offset + margin) to owned bytes.

@ vermodel_from_bytes ( Vec u ) buf → ?VerModel

Parse a forest blob. None on any structural violation: bad magic, short buffer, absurd counts, mismatched array lengths, out-of-range indices.

: i ANOM_SC_UNSCORED 0

── Scored-verdict cache ──────────────────────────────────────────────

Re-scoring a stored ring is pure recomputation: the same point against the same forests yields the same verdict every time. The dashboard's anomaly scan used to pay for that recomputation on every visit, one HTTP round trip AND one full model load per point, which is what made a 5000-point scan a minutes-long progress bar.

The cache is a ring-aligned array of verdicts stamped with the model's score_epoch (prep.nu). Anything that can change a verdict — a retrain, a new autoencoder, a margin edit, a version toggled, a reset — bumps the epoch, and the whole cache is stale by construction; there is no per-entry invalidation rule to get wrong.

Alignment survives ring eviction because rows are keyed on the LIFETIME point counter, not the ring index: base_seen is the lifetime index of row 0, so a ring row j of a ring of length L at counter S sits at cache index S - L + j - base_seen. Rows outside [0, nrows) are simply misses.

"ANOMSCR2" | u64 epoch | u64 base_seen | u64 nver | nver × (u64 len, bytes) | u64 nrows | nrows × (f64 score, f64 severity, u64 state, u64 present, u64 flagged)

state is 0 for a row never scored under this epoch, 1 for a scored verdict and 2 for "the model was not ready for this point". present and flagged are bitmasks over vnames, so a version that produced no verdict (a timevector window longer than the ring prefix) stays distinguishable from one that produced a clean verdict.

: i ANOM_SC_SCORED 1

: i ANOM_SC_NOT_READY 2

: i ANOM_SC_MAX_ROWS 100000000

Refuse a cache claiming more rows/versions than a model could hold.

: i ANOM_SC_MAX_VERS 64

: ScoreCache

: ScoreCache {
    i epoch
    i base_seen
    ( Vec String ) vnames
    ( Vec i ) state
    ( Vec f ) score
    ( Vec f ) severity
    ( Vec i ) present
    ( Vec i ) flagged
}

@ scorecache_new i epoch i base_seen → ScoreCache

@ scorecache_free ScoreCache c → v

@ scorecache_rows ScoreCache c → i

@ scorecache_resize ScoreCache c i n → v

Grow to n rows, every new row unscored.

@ scorecache_set ScoreCache c i at i state f score f severity i present i flagged → v

@ scorecache_vnames_match ScoreCache c ( Vec String ) live → b

Do the cached version names still match the live ones, in order?

: s ANOM_LABEL_FP false_positive``

: s ANOM_LABEL_OK confirmed``

: s ANOM_LABEL_NONE none``

: Label

: Label {
    i seq  // lifetime sequence number of the point
    i ts  // the point's own timestamp
    String label  // ANOM_LABEL_*
    String by  // who said so (a principal's name, an API key's id, or empty)
    i at  // when (unix seconds)
    String note
}

@ label_free Label l → v

@ labels_free ( Vec Label ) ls → v

@ label_known s name → b

Is s a label a reader may give?

@ label_to_json Label l → Json

: s ANOM_ORG_DEFAULT public``

: s ANOM_KIND_AE ae``

: s ANOM_KIND_FC fc``

: s ANOM_KIND_SCORES scores``

: Store

: Store {
    String root
    String org
    b ok
}

@ store_open_org s root s org → Store

Open the organisation's store: make the directory, put the database's journal into WAL and ensure this module's tables, once. Every operation after this opens its own connection for the length of the operation.

@ store_open s root → Store

The organisation a store with no sign-in belongs to. Simple mode, the CLI and the analysis sandbox all collect into public, which is a real organisation with a real database — turning sign-in on later moves nothing.

@ store_free Store st → v

@ store_org Store st → s

@ store_exists Store st s name → b

A model exists iff this organisation's database has its metadata row.

@ store_list Store st → ( Vec String )

Every model in this organisation, by name, sorted.

@ store_save_meta Store st s name * Meta m → b

@ store_load_meta Store st s name → ?*Meta

@ store_quarantine_meta Store st s name i now → String

Set metadata that would not parse aside, with the time it happened, so a model can be reopened without destroying the evidence. Returns where it went (empty when there was nothing to move).

: String where ( string_from models_meta_corrupt ( )

@ store_save_forest Store st s name VerModel vm → b

@ store_load_forest Store st s name s vname → ?VerModel

Load one version's forest; None if absent or corrupt. The blob's embedded version name must match the requested one — a moved or content-tampered blob is treated as corrupt, not trusted.

@ store_delete_forest Store st s name s vname → v

@ store_save_ae Store st s name AeModel ae → b

@ store_load_ae Store st s name → ?AeModel

@ store_save_fc Store st s name * FcModel fc → b

@ store_load_fc Store st s name → ?*FcModel

@ store_delete_fc Store st s name → v

@ store_delete Store st s name → b

Everything the model is, in one transaction: either the model is gone or it is untouched. The rows the file backend removed as a directory.

@ store_save_scores Store st s name ScoreCache c → b

@ store_load_scores Store st s name → ?ScoreCache

Load the cache; None when absent, truncated or structurally impossible. A rejected cache costs a rescan, never a wrong verdict.

@ store_delete_scores Store st s name → v

@ store_commit_point Store st s name i seq s line i evict_before * Meta m → b

One ingested point, as one transaction: the row, the eviction it may cause, and the metadata whose counter says how many points there are. A crash between them would leave the ring and n_seen disagreeing, and another thread must never read the ring half-updated. evict_before is the lifetime number of the oldest row to keep (0 evicts nothing).

@ store_evict_points Store st s name i from_seq → b

Drop every point older than from_seq.

@ store_write_points Store st s name ( Vec String ) lines i base_seq → b

Replace the whole ring: lines become the points from base_seq on. Used where the ring is rebuilt rather than extended — a reset, a file of history imported, a cap lowered below the fill. One transaction.

@ store_load_points Store st s name → ( Vec String )

The ring in order, oldest first. Owned lines.

@ store_load_points_tail Store st s name i n → ( Vec String )

The newest n points, oldest first — what scoring one point actually needs, without reading a ring that may hold a hundred thousand rows.

@ store_points_count Store st s name → i

How many points the model holds.

@ store_append_label Store st s name Label l → b

@ store_load_labels Store st s name → ( Vec Label )

The labels in force. Ascending by seq is not promised — a reader wanting the ring order joins on seq (model_label_map).

@ store_delete_labels Store st s name → v

@ store_append_audit Store st s name Json ent → b

@ store_load_audit Store st s name i limit → Json

The newest limit entries, oldest first (all when limit <= 0). Taken by the index rather than by reading the whole log and dropping the front.

@ store_migrate_dir Store st s root s name i now → b

One flat model directory into this organisation's database.

@ store_migrate_flat s root i now → i

Every flat model directory left under root, moved into the database of the organisation that owns it. Returns how many moved. Called once at startup; when there is nothing to move it costs one directory read.

@ store_move_model Store src Store dst s name i now → b

Everything name is, from src into dst. Refuses when dst already has that name — two models of one name in one organisation is exactly what the organisation-as-database rule exists to prevent. The source keeps its rows if any part of the write fails.