main.nu

yoloe-demo — live in-browser YOLOE segmentation, served by pure NURL.

The browser opens one page, grabs the webcam, and POSTs JPEG frames to this server; the server runs YOLOE (open-vocabulary detection + instance segmentation) on the GPU through the onnx package, composites the segmentation masks onto the frame, and answers with the masked JPEG plus a JSON detection list. The page draws boxes/labels on top and loops — live segmented video, end to end, with no Python, no OpenCV and no inference engine anywhere in the stack:

browser webcam → JPEG POST → http package (pure-NURL HTTP/TLS server) → image package (JPEG decode) → onnx package (every layer a CUDA-C kernel via NVRTC) → yoloe decode + mask branch → JPEG encode → browser

yoloe-demo --model yoloe-v8s-seg.onnx --classes classes.txt [--port 8090] [--host 0.0.0.0] [--gpu 0] [--tls] [--page views/index.html]

--tls generates a self-signed P-256 certificate at startup (std/x509_gen, also pure NURL) so the camera works from other machines on the LAN too — browsers only expose getUserMedia on secure origins (localhost is exempt).

Routes: GET / the demo page (rendered once at startup via the template package with the class list + device name) POST /detect body = JPEG/PNG frame; query: conf=0.25 masks=1 on=0,2,5 → { ms, n, dets:[{n,s,c,k,x,y,w,h}], img:"data:..jpeg" } GET /health liveness probe

API

: DemoState

: DemoState {
    i eng  // *Engine as int
    OGraph graph
    ( Vec String ) names
    i nc
    String device
    String model_path
    String index_html  // pre-rendered demo page
    // runtime prompting (kmax > 0 → the promptable K-slot export is loaded):
    i kmax  // prompt slots baked into the detector graph (0 = baked vocab)
    OGraph tgraph  // MobileCLIP text encoder (pure NURL on the GPU)
    Tokenizer tk  // CLIP BPE tokenizer
    ( Vec u ) tpe  // kmax×512 f32 text embeddings, zero-padded past nc
    ( Vec u ) model_bytes  // raw .onnx, served to the wasm client mode
    b wasm_mode  // web/yoloe_detect.wasm present → offer in-browser compute
}

: ~ i g_st 0

@ yd_name_at ( Vec String ) names i k → s

@ yd_read_names s path → ( Vec String )

classes.txt: one prompt per line (must match the model export)

@ yd_shape4 i a i b i c i d → ( Vec i )

@ yd_pal_r i k → i

8-colour instance palette (matches the page's JS palette by index)

@ yd_pal_g i k → i

@ yd_pal_b i k → i

@ yd_pal i ch → ( Vec i )

@ yd_params_free ( Vec UrlParam ) ps → v

@ yd_query_f ( Vec UrlParam ) ps s key f dflt → f

value of key as float, or dflt when absent/garbage

@ yd_query_b ( Vec UrlParam ) ps s key b dflt → b

flag key (0/1), dflt when absent

@ yd_enabled ( Vec UrlParam ) ps i nc i ncap → ( Vec i )

per-class enable flags from on=0,2,5; no on param → all ACTIVE classes enabled. ncap is the decode width (kmax in prompt mode) — ids in [nc, ncap) are inert pad slots and stay off either way.

@ yd_encode_prompt * DemoState st s text i slot → b

Tokenize one prompt, run the text encoder on the GPU, and write the 512-float embedding into tpe slot slot. The tokens ride as int64 [1,77]; the n1 text-encoder export ends in an L2 normalize, so the slot lands in exactly the space the contrastive head was traced with.

@ yd_decode * u o i na i nc f thresh ( Vec i ) flags → ( Vec Detection )

Per-anchor best ENABLED class. yolo_decode argmaxes over every class and filters afterwards — so a disabled seed class ('dog' 0.88) would shadow an enabled custom prompt of the same object ('a dog' 0.63) and the anchor vanished. Restricting the argmax to enabled classes makes toggles behave the way the chips read.

@ yd_detect * DemoState st Image im f conf b want_masks ( Vec i ) flags → Json

@ h_index HttpRequest req Params p → HttpResponse

@ h_wasm_model HttpRequest req Params p → HttpResponse

GET /wasm/model — the detector .onnx for the in-browser wasm engine.

@ h_tpe HttpRequest req Params p → HttpResponse

GET /tpe — the CURRENT vocabulary embeddings (nc × 512 f32, raw LE) with the class names in X-Classes. The wasm client seeds from this and re-fetches after /prompt so both engines share one vocabulary.

@ h_health HttpRequest req Params p → HttpResponse

@ h_detect HttpRequest req Params p → HttpResponse

@ h_prompt HttpRequest req Params p → HttpResponse

POST /prompt — body is the prompt text. Encodes it through the MobileCLIP text encoder ON the GPU and appends it to the vocabulary; the next /detect already sees it. JSON out: { id, name, n }.

@ wd_probe s path → ( Vec u )

1-byte marker vec when path exists and is readable ([] otherwise).

@ yd_load_graph s path * b okcell → OGraph

@ yd_render_index s page_path * DemoState st → b

render views/index.html once with the static context

@ main → i

---

wasm_detect.nu

wasm_detect.nu — YOLOE inference as a wasm32-wasi module: the SAME pure-NURL onnx runtime the server runs on the GPU, compiled for the browser and executed on the gpu package's STATIC backend (precompiled kernels_static.wasm.o linked in — no NVRTC, no dlopen, no compiler).

This is a WASI command module driven by host imports (env.*): the embedder (a web worker — see web/worker.js — or the node test harness) supplies blobs and frames, the module loops forever inside start. Blocking hostframe is legal because the module runs in a worker thread (Atomics.wait), never on the UI thread.

host_blob_size(kind) → bytes 0 = model.onnx, 1 = seed tpe f32 host_blob_read(kind, dst) → 0/-1 embedder memcpys the blob in host_frame(rgb, ctl) → cmd 0 = frame ready, 2 = add prompt, anything else = exit host_result(dets, nd, ms) nd detections ready; the masked frame is in-place in rgb host_status(code, extra) 1 model-parsed, 2 engine-ready, -1 fatal (extra = detail)

ctl is an i32/f32 slab (4-byte slots): [0] conf ‰ (i32, e.g. 250 = 0.25) [1] masks 0/1 (i32) [2] enabled-class bitmask (i32, bit k = class k) [4] add-prompt slot index (i32) [16..527] add-prompt embedding (f32) dets out: 6 f32 per detection [cls, score, x, y, w, h], max 64.

Frame geometry is fixed 640×480 RGB (the embedder letterboxes the camera into that, exactly like the host-mode page does).

API

& c @ host_blob_size i kind → i

& c @ host_blob_read i kind *u dst → i

& c @ host_frame *u rgb *u ctl → i

& c @ host_result *u dets i nd i ms → v

& c @ host_status i code i extra → v

& c @ host_use_webgpu → i

1 → run the WebGPU backend, else the static CPU kernels

: i WD_W 640

: i WD_H 480

: i WD_K 32

: i WD_MAXDET 64

@ wd_shape4 i a i b i c i d → ( Vec i )

@ wd_pal i ch i k → i

8-colour instance palette (matches the page's JS palette by index)

@ wd_decode * u o i na i nc f thresh i mask → ( Vec Detection )

Per-anchor best ENABLED class (bit k of mask = class k enabled) — same semantics as the server's yd_decode.

@ wd_load_blob i kind → ( Vec u )

Load blob kind into a fresh Vec u (embedder memcpys, we own it).

@ main → i