decode.nu

packages/yoloe/src/decode.nu — turn YOLOE's output0 into boxes.

output0 is [1, 4+nc+nm, na] (na = 8400 anchors over 3 scales): the first 4 channels are the box (cx,cy,w,h, already decoded to pixels in the letterboxed frame), the next nc are per-class scores (already sigmoid), the rest are segmentation mask coefficients (ignored here). For each anchor we take the best class above a threshold, then non-max suppress.

API

: Detection { i cls f score f cx f cy f w f h i ai }

A detection in letterboxed-pixel centre/size coordinates. ai is the anchor index it came from — needed to read the 32 mask coefficients (output0 channels 4+nc..4+nc+nm at that anchor) for segmentation.

@ yolo_decode GpuHost out__h i na i nc f thresh → ( Vec Detection )

Decode the best detections above thresh. nc = number of classes, na = anchors, no = total channels (4 + nc + masks). An output too small for that many (a class list longer than the model's) decodes to none.

@ yolo_nms ( Vec Detection ) dets f iou_thresh → ( Vec Detection )

Greedy non-max suppression (per class), keeping highest-score boxes.

---

mask.nu

packages/yoloe/src/mask.nu — instance-segmentation mask decode + overlay.

YOLOE-seg has two outputs: output0 [1, 4+nc+nm, na] (boxes, class scores, and nm=32 mask coefficients per anchor) and output1 [1, nm, MH, MW] — the mask "prototypes" (MH=MW=160, a quarter of the 640 input). The mask for a detection is a linear combination of the prototypes weighted by that anchor's 32 coefficients, then thresholded:

logit[y,x] = Σ_m coeff[m] · proto[m,y,x] (160×160) mask = sigmoid(logit) > 0.5 ⟺ logit > 0

cropped to the box and upsampled to image resolution (bilinear) — exactly ops.process_mask(..., upsample=True) in the reference. We precompute the 160×160 logit map per detection, then bilinear-sample it as we paint the box region of the original image, blending a translucent colour where the mask is set. This matches onnxruntime's masks (the proto tensor itself is bit-checked against onnxruntime in tests/seg_fwd.nu).

API

@ mask_nm → i

Mask prototype geometry: 32 coefficients, 160×160 prototypes at stride 4.

@ mask_dim → i

@ mask_stride → i

@ mask_coeffs GpuHost out i na i nc i ai → ( Vec f )

Read a detection's nm mask coefficients from output0 (channels 4+nc .. 4+nc+nm at anchor ai). Each is the f32 value, exactly.

@ mask_logits GpuHost proto ( Vec f ) coeff i MH i MW → ( Vec f )

Build the 160×160 mask logit map for one detection: logit[y,x] = Σ_m coeff[m]·proto[m,y,x]. proto is laid out [nm, MH, MW] row-major. Each pixel sums its terms in m order, as one loop per pixel did, but a prototype plane at a time: the inner loop runs over contiguous windows and vectorises. Each logit is rounded to f32, as the map was always stored. A proto smaller than that is no mask: every logit 0.

@ mask_sample ( Slice f ) L i MH i MW f fy f fx → f

Bilinear sample of an MH×MW grid at fractional (fy,fx); out-of-range coordinates clamp to the edge (align_corners=False convention).

@ mask_blend Image im i x i y i r i gg i bb i alpha → v

Blend colour (r,g,b) into a pixel at alpha/256 (alpha 0..256).

@ mask_overlay Image im ( Vec f ) L__v Letterbox lb i S i x0 i y0 i ow i oh i r i gg i bb i alpha → i

Overlay one detection's mask onto the original image. The box is given in ORIGINAL-image pixels (x0,y0,ow,oh); for each pixel we map back through the letterbox to the 160×160 mask grid and threshold the bilinearly-sampled logit at 0 (sigmoid > 0.5). S is the model input side (640).

---

display.nu

packages/yoloe/src/display.nu — show an Image live in the terminal.

Renders a packed-RGB Image as 24-bit-colour half-blocks: each character cell is the upper-half-block glyph ▀ whose FOREGROUND colour is one image pixel and BACKGROUND colour the pixel just below it — so one cell carries two vertical pixels, giving a square-ish picture in any truecolor terminal (gnome-terminal, kitty, xterm, … and over SSH). No X11, no SDL, no library.

The frame is scaled to fit the terminal (stdlib's term_width / term_height) preserving aspect, built into one String, and printed after homing the cursor so a continuous capture loop overwrites in place like a video.

API

@ term_enter → v

Clear the screen and hide the cursor (call once before a live loop).

@ term_leave → v

Show the cursor again (call when the live loop ends).

@ img_show Image im → v

Render im to the terminal as truecolor half-blocks, scaled to fit.

---

image.nu

packages/yoloe/src/image.nu — YOLO-facing adapter over packages/image.

The codecs and raster ops live in the image package (PNG / baseline + progressive JPEG / PPM decode+encode, rectangle drawing — see deps/image); this file keeps only what is YOLO-specific: letterbox preprocessing (aspect-preserving nearest resize + grey-114 pad, matching ultralytics), packing to the [0,1]-normalised NCHW float tensor the model consumes, edge-clamped reads for the mask overlay, and load/save-by-extension. The Image type is the image package's (width/height/channels/data).

API

: Letterbox { Image img f scale i padx i pady }

Letterbox result: the square image plus the transform back to original pixels (orig = (lb - pad) / scale).

@ img_w Image im → i

@ img_h Image im → i

@ img_get Image im i x i y i ch → i

Edge-clamped read (the mask overlay and letterbox sample with clamping).

@ img_set Image im i x i y i ch i val → v

Saturating store (image_set masks to the low byte; overlay math must clamp).

@ img_blank i w i h i fill → Image

An RGB image filled with a constant grey level.

@ img_load s path → ?Image

Load any supported format (PNG / baseline+progressive JPEG / PPM) as 3-channel RGB. On None, ( image_error ) says why.

@ img_save s path Image im → b

Save by extension: .png / .jpg / .jpeg (quality 90) / anything else = PPM.

@ letterbox Image im i S → Letterbox

── letterbox to S×S (aspect-preserving, grey 114 pad) ────────────

@ img_to_nchw_norm Image im → GpuHost

Pack to NCHW float tensor normalised to [0,1]: a host buffer of 3HW f32, the input rt_run_host takes.

@ img_draw_rect Image im i x0 i y0 i x1 i y1 i r i gg i bb → v

Draw a thickness-3 rectangle outline in the given colour.

---

prompt.nu

packages/yoloe/src/prompt.nu — RUNTIME-promptable open-vocabulary detection.

yoloe-prompt <model.onnx> <tpe.f32> <classes.txt> <img> [out]

Unlike src/main.nu (whose vocabulary is baked into the model at export time), this takes the prompt text embeddings as a runtime input: the model has two inputs, the image and tpe [1,K,512] (raw MobileCLIP text features for the K prompt words). Swap the tpe file + classes file to detect a different set of objects with the same model — no re-export.

Produce the model + tpe with tools/export_promptable.py and tools/gen_tpe.py.

API

& c @ nurl_peek_f32 *u base i idx → f

@ p s m → v

@ pf f x → v

@ pn i n → v

@ load_f32 s path → ( Vec u )

A raw little-endian f32 file as its bytes (empty when unreadable).

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ shape3 i a i b i c → ( Vec i )

@ main → i

---

v4l2.nu

packages/yoloe/src/v4l2.nu — webcam capture in pure NURL via Video4Linux2.

Opens /dev/videoN, negotiates a YUYV capture format, sets up memory-mapped streaming buffers, and hands back decoded RGB frames — no ffmpeg, no OpenCV, just open/ioctl/mmap on the kernel's V4L2 ABI through the FFI. The ioctl request codes and struct field offsets below are the stable x86_64 <linux/videodev2.h> layout (verified against the system headers):

v4l2_format 208 B type@0 pix.width@8 height@12 pixelformat@16 field@20 v4l2_requestbuffers 20 B count@0 type@4 memory@8 v4l2_buffer 88 B index@0 type@4 bytesused@8 memory@60 m.offset@64 length@72

Frames arrive as YUYV (4:2:2, 2 bytes/pixel) and are converted to packed RGB with the BT.601 integer transform. One Camera owns the fd plus the mmap'd ring of capture buffers: it is a handle, every copy is the same stream, and its last owner stops it, unmaps the ring and closes the fd (cam_close does that now — an optional early release).

API

& c @ ioctl i32 fd i req ... → i32

ioctl as C declares it — int ioctl(int, unsigned long, ...). stdlib/std/term.nu declares the same symbol, and one linker symbol has one ABI: stating it with i where C says int is a different function, which the compiler now rejects rather than miscompiling.

& c @ nurl_peek_i32 *u base i idx → i32

& c @ nurl_poke_i32 *u base i idx i32 val → v

: Camera { i fd i w i h i nbuf ( Vec i ) bufptr ( Vec i ) buflen i ok }

A streaming webcam: fd, frame geometry, and the mmap'd buffer ring. One holder owns it; its drop stops the stream, unmaps the ring and closes the fd.

% Drop Camera { @ drop Camera c → v { ( __cam_release c ) } }

@ cam_ok Camera h → b

@ cam_w Camera h → i

@ cam_h Camera h → i

@ cam_close sink Camera c → v

Release the camera now (optional — its drop does it at the end of its holder's scope anyway).

---

bpe.nu

packages/yoloe/src/bpe.nu — the CLIP byte-pair-encoding tokenizer in pure NURL. This is the word → token-id half of the MobileCLIP text path; the other half (token-ids → 512-d features) is the text-encoder ONNX graph already run by the onnx runtime (see tests/text_fwd.nu).

Faithful port of open_clip's SimpleTokenizer (the tokenizer MobileCLIP's ClipTokenizer wraps as get_tokenizer("ViT-B-16")):

• bytes_to_unicode() byte→symbol table (reversible byte-level coding),
• a 49408-entry encoder: 256 base symbols + 256 word-end (</w>) variants
• 48894 merge tokens + <start_of_text>/<end_of_text>,
• bpe_ranks from assets/clip_merges.txt (the decompressed

bpe_simple_vocab_16e6.txt, lines 1..48894),

• the BPE merge loop (lowest-rank bigram first, merge all occurrences),
• [sot] + ids + [eot] padded/truncated to the context length (77).

Word splitting reproduces CLIP's regex 'r"'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+" for the common case (ASCII / Latin-1 letters, digits, punctuation, English contractions). A letter is a-z / Latin-1 letters / any codepoint

= U+0100; a digit is 0-9. Exotic symbol-vs-letter classification of rare

BMP punctuation may differ from full Unicode \p{L}/\p{N}, which does not arise for object-detection class names.

API

: i BPE_INF 1000000000

: Tokenizer

: Tokenizer {
    ( Vec String ) byte_enc  // byte value 0..255 → its symbol string
    ( HashMap s i ) enc  // token string → id
    ( HashMap s i ) ranks  // "a b" merge line → rank
    ( Vec String ) arena  // owns enc/ranks key strings
    i sot
    i eot
}

A loaded tokenizer. The hash maps key on s (raw char*) pointers, so the backing strings must outlive the maps: arena owns every key string that is not already owned by byte_enc.

@ _ig ( Vec i ) v i k → i

── small Vec<i> helpers ─────────────────────────────────────────

@ tokenizer_none → Tokenizer

The tokenizer that encodes nothing (sot=-1): a placeholder for a struct field that holds one only when free-text prompting is on.

@ tokenizer_load s merges_path → Tokenizer

── load ───────────────────────────────────────────────────────── Build the tokenizer from the merges asset. Returns a Tokenizer whose maps are empty (sot=-1) on read failure.

@ tokenizer_load_builtin → Tokenizer

Build the tokenizer from the merge table compiled into the binary (src/clip_merges_data.nu, generated by tools/embed_merges.py). An installed tool has no package tree on disk, so this is the default; tokenizer_load stays for custom vocabularies via --merges.

@ bpe_encode Tokenizer tk s text → ( Vec i )

── encode / tokenize ──────────────────────────────────────────── text → list of token ids (no sot/eot, no padding).

@ bpe_tokenize Tokenizer tk s text i ctx → ( Vec i )

text → padded context: [sot] + ids + [eot], zero-padded/truncated to ctx. Returns a Vec<i> of length ctx.

---

main.nu

packages/yoloe/src/main.nu — the yoloe command: promptable open-vocabulary object detection AND instance segmentation, pure NURL on the GPU, with a live webcam mode that draws straight to the terminal. Single binary, three sub-commands, parsed by the cli package (see yoloe --help):

yoloe detect --model M --classes C --image IMG [--out OUT] yoloe seg --model M --classes C --image IMG [--out OUT] yoloe cam --model M --classes C [--device DEV] [--frames N] [--out DIR] [--no-show] [--boxes]

cam shows the segmented feed live in the terminal (truecolor half-blocks); with no --frames it runs until Ctrl-C, and --out also saves the frames.

API

@ p s m → v

@ pf f x → v

@ pn i n → v

@ read_names s path → ( Vec String )

Read the prompt vocabulary from a text file (one class per line); the line count is the class count nc. Must match the names the model was exported with (tools/export.py) — swap both to detect a different vocabulary.

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ pal_r i k → i

@ pal_g i k → i

@ pal_b i k → i

@ frame_path s dir i n → String

frame path: <dir>/frameNNNNN.ppm

@ process_frame Image im OGraph g Engine e ( Vec String ) names i nc b want_boxes b want_masks b verbose → i

Run the network on one image, drawing boxes (and, when want_masks, the per-object segmentation mask) onto im. When verbose, prints one line per detection. Returns the detection count.

@ load_model s path → ?OGraph

@ run_still b want_boxes b want_masks String mp String np String ip String op i gpu → i

── still-image modes (detect / seg) ──────────────────────────────

@ run_cam String mp String np String dev i nframes String od i mode b want_boxes b want_masks i gpu → i

── webcam mode (cam): live terminal display and/or save frames ───

@ main → i

---

clip_merges_data.nu

GENERATED by tools/embed_merges.py from assets/clip_merges.txt — do not edit. The CLIP BPE merge table (48893 lines) in ~32 KB line-aligned chunks, embedded so installed binaries need no asset file on disk (see tokenizer_load_builtin in bpe.nu). Chunked because released nurlc (≤ v0.11.1) stack-overflows on string literals past ~48 KB.

API

@ clip_merges_chunks → i

@ clip_merges_chunk i k → s

---

window.nu

packages/yoloe/src/window.nu — a real GUI window for the live preview, in pure NURL via Xlib (libX11) bound directly — no SDL, no GTK, no toolkit.

Opens an X11 window the size of the camera frame and blits each segmented RGB frame at full resolution with XPutImage (so unlike the terminal view there's no down-scaling). Polls for a keypress / window-close so the loop can exit cleanly. The toolchain auto-links -lX11 only when @XOpenDisplay appears (build.sh writes the runtime.X11 sentinel when libX11 is found), so a headless build is unaffected.

API

& X11 @ XOpenDisplay *u name → *u

& X11 @ XDefaultScreen *u dpy → i

& X11 @ XRootWindow *u dpy i screen → i

& X11 @ XDefaultVisual *u dpy i screen → *u

& X11 @ XDefaultDepth *u dpy i screen → i

& X11 @ XCreateSimpleWindow *u dpy i parent i x i y i w i h i bw i border i bg → i

& X11 @ XStoreName *u dpy i win s name → i

& X11 @ XSelectInput *u dpy i win i mask → i

& X11 @ XMapWindow *u dpy i win → i

& X11 @ XCreateGC *u dpy i drawable i valuemask *u values → *u

& X11 @ XCreateImage *u dpy *u visual i depth i format i offset *u data i width i height i pad i bpl → *u

& X11 @ XPutImage *u dpy i d *u gc *u image i sx i sy i dx i dy i w i h → i

& X11 @ XFlush *u dpy → i

& X11 @ XPending *u dpy → i

& X11 @ XNextEvent *u dpy *u event → i

& X11 @ XInternAtom *u dpy s name i only → i

& X11 @ XSetWMProtocols *u dpy i win *u protocols i count → i

& X11 @ XCloseDisplay *u dpy → i

& X11 @ XDestroyImage *u image → i

& X11 @ XFreeGC *u dpy *u gc → i

& c @ nurl_poke_i32 *u base i idx i32 val → v

& c @ nurl_peek_i32 *u base i idx → i32

: XWin { i dpy i win i gc i img ( Vec u ) data i w i h i ok }

Pointers (Display, GC, XImage) carried as i64; w/h the frame size. The pixel buffer the XImage points at is the window's own Vec. A window, owned by one holder; its drop closes the display connection.

% Drop XWin { @ drop XWin x → v { ( __xwin_release . x dpy . x gc . x img ) } }

@ xwin_none i w i h → XWin

The window that never opened (ok=0): w×h is still the frame size.

@ xwin_ok XWin h → b

@ xwin_close sink XWin x → v

Close the window now (optional — its drop does it anyway).