decode.nu

packages/yoloe/src/decode.nu — turn YOLOE's output0 into boxes.

output0 is [1, 4+nc+nm, na] (na = 8400 anchors over 3 scales): the first 4 channels are the box (cx,cy,w,h, already decoded to pixels in the letterboxed frame), the next nc are per-class scores (already sigmoid), the rest are segmentation mask coefficients (ignored here). For each anchor we take the best class above a threshold, then non-max suppress.

API

: Detection { i cls f score f cx f cy f w f h i ai }

A detection in letterboxed-pixel centre/size coordinates. ai is the anchor index it came from — needed to read the 32 mask coefficients (output0 channels 4+nc..4+nc+nm at that anchor) for segmentation.

& c @ nurl_peek_f32 *u base i idx → f

@ yolo_decode *u out i na i nc f thresh → ( Vec Detection )

Decode the best detections above thresh. nc = number of classes, na = anchors, no = total channels (4 + nc + masks).

@ yolo_nms ( Vec Detection ) dets f iou_thresh → ( Vec Detection )

Greedy non-max suppression (per class), keeping highest-score boxes.

---

mask.nu

packages/yoloe/src/mask.nu — instance-segmentation mask decode + overlay.

YOLOE-seg has two outputs: output0 [1, 4+nc+nm, na] (boxes, class scores, and nm=32 mask coefficients per anchor) and output1 [1, nm, MH, MW] — the mask "prototypes" (MH=MW=160, a quarter of the 640 input). The mask for a detection is a linear combination of the prototypes weighted by that anchor's 32 coefficients, then thresholded:

logit[y,x] = Σ_m coeff[m] · proto[m,y,x] (160×160) mask = sigmoid(logit) > 0.5 ⟺ logit > 0

cropped to the box and upsampled to image resolution (bilinear) — exactly ops.process_mask(..., upsample=True) in the reference. We precompute the 160×160 logit map per detection, then bilinear-sample it as we paint the box region of the original image, blending a translucent colour where the mask is set. This matches onnxruntime's masks (the proto tensor itself is bit-checked against onnxruntime in tests/seg_fwd.nu).

API

& c @ nurl_peek_f32 *u base i idx → f

& c @ nurl_poke_f32 *u base i idx f val → v

@ mask_nm → i

Mask prototype geometry: 32 coefficients, 160×160 prototypes at stride 4.

@ mask_dim → i

@ mask_stride → i

@ mask_coeffs *u out i na i nc i ai → *u

Read a detection's nm mask coefficients from output0 (channels 4+nc .. 4+nc+nm at anchor ai) into a fresh host buffer (caller frees).

@ mask_logits *u proto *u coeff i MH i MW → *u

Build the 160×160 mask logit map for one detection: logit[y,x] = Σ_m coeff[m]·proto[m,y,x]. proto is laid out [nm, MH, MW] row-major. Returns a fresh host f32 buffer of MH*MW (caller frees).

@ mask_sample *u L i MH i MW f fy f fx → f

Bilinear sample of an MH×MW grid at fractional (fy,fx); out-of-range coordinates clamp to the edge (align_corners=False convention).

@ mask_blend Image im i x i y i r i gg i bb i alpha → v

Blend colour (r,g,b) into a pixel at alpha/256 (alpha 0..256).

@ mask_overlay Image im *u L Letterbox lb i S i x0 i y0 i ow i oh i r i gg i bb i alpha → i

Overlay one detection's mask onto the original image. The box is given in ORIGINAL-image pixels (x0,y0,ow,oh); for each pixel we map back through the letterbox to the 160×160 mask grid and threshold the bilinearly-sampled logit at 0 (sigmoid > 0.5). S is the model input side (640).

---

display.nu

packages/yoloe/src/display.nu — show an Image live in the terminal.

Renders a packed-RGB Image as 24-bit-colour half-blocks: each character cell is the upper-half-block glyph ▀ whose FOREGROUND colour is one image pixel and BACKGROUND colour the pixel just below it — so one cell carries two vertical pixels, giving a square-ish picture in any truecolor terminal (gnome-terminal, kitty, xterm, … and over SSH). No X11, no SDL, no library.

The frame is scaled to fit the terminal (size via TIOCGWINSZ) preserving aspect, built into one String, and printed after homing the cursor so a continuous capture loop overwrites in place like a video.

API

& c @ ioctl i fd i req *u argp → i

& c @ nurl_poke_i32 *u base i idx i val → v

& c @ nurl_peek_i32 *u base i idx → i

@ term_enter → v

Clear the screen and hide the cursor (call once before a live loop).

@ term_leave → v

Show the cursor again (call when the live loop ends).

@ img_show Image im → v

Render im to the terminal as truecolor half-blocks, scaled to fit.

---

image.nu

packages/yoloe/src/image.nu — image I/O + YOLO letterbox preprocessing.

Reads/writes binary PPM (P6), letterboxes to the model's square input (aspect-preserving resize + grey pad), packs to an NCHW float tensor normalised to [0,1], and draws labelled boxes. An Image keeps the raw file bytes plus a pixel-data offset, so reading copies nothing; a fresh image (letterbox/blank) owns a new byte buffer.

API

& c @ nurl_poke_f32 *u base i idx f val → v

: Image { i w i h ( Vec u ) buf i off }

: Letterbox { Image img f scale i padx i pady }

Letterbox result: the square image plus the transform back to original pixels (orig = (lb - pad) / scale).

@ img_w Image im → i

@ img_h Image im → i

@ img_get Image im i x i y i ch → i

@ img_set Image im i x i y i ch i val → v

@ ppm_read s path → ?Image

@ img_blank i w i h i fill → Image

@ ppm_write s path Image im → b

@ letterbox Image im i S → Letterbox

── letterbox to S×S (aspect-preserving, grey 114 pad) ────────────

@ img_to_nchw_norm Image im → *u

Pack to NCHW float tensor normalised to [0,1].

@ img_draw_rect Image im i x0 i y0 i x1 i y1 i r i gg i bb → v

---

prompt.nu

packages/yoloe/src/prompt.nu — RUNTIME-promptable open-vocabulary detection.

yoloe-prompt <model.onnx> <tpe.f32> <classes.txt> <image.ppm> [out.ppm]

Unlike src/main.nu (whose vocabulary is baked into the model at export time), this takes the prompt text embeddings as a runtime input: the model has two inputs, the image and tpe [1,K,512] (raw MobileCLIP text features for the K prompt words). Swap the tpe file + classes file to detect a different set of objects with the same model — no re-export.

Produce the model + tpe with tools/export_promptable.py and tools/gen_tpe.py.

API

& c @ nurl_peek_f32 *u base i idx → f

@ p s m → v

@ pf f x → v

@ pn i n → v

@ load_f32 s path *u pcell → *u

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ shape3 i a i b i c → ( Vec i )

@ main → i

---

v4l2.nu

packages/yoloe/src/v4l2.nu — webcam capture in pure NURL via Video4Linux2.

Opens /dev/videoN, negotiates a YUYV capture format, sets up memory-mapped streaming buffers, and hands back decoded RGB frames — no ffmpeg, no OpenCV, just open/ioctl/mmap on the kernel's V4L2 ABI through the FFI. The ioctl request codes and struct field offsets below are the stable x86_64 <linux/videodev2.h> layout (verified against the system headers):

v4l2_format 208 B type@0 pix.width@8 height@12 pixelformat@16 field@20 v4l2_requestbuffers 20 B count@0 type@4 memory@8 v4l2_buffer 88 B index@0 type@4 bytesused@8 memory@60 m.offset@64 length@72

Frames arrive as YUYV (4:2:2, 2 bytes/pixel) and are converted to packed RGB with the BT.601 integer transform. One Camera owns the fd plus the mmap'd ring of capture buffers.

API

& c @ open s path i flags → i

& c @ close i fd → i

& c @ ioctl i fd i req *u argp → i

& c @ mmap *u addr i length i prot i flags i fd i offset → *u

& c @ munmap *u addr i length → i

& c @ nurl_peek_i32 *u base i idx → i

& c @ nurl_poke_i32 *u base i idx i val → v

: Camera { i fd i w i h i nbuf ( Vec i ) bufptr ( Vec i ) buflen i ok }

A streaming webcam: fd, frame geometry, and the mmap'd buffer ring.

@ cam_open s path i w i h i nbuf → Camera

Open a webcam and start YUYV streaming at w×h with nbuf ring buffers.

@ cam_ok Camera c → b

@ cam_w Camera c → i

@ cam_h Camera c → i

@ cam_grab Camera c ( Vec u ) rgb → b

Grab one frame: dequeue a filled buffer, convert YUYV→RGB into rgb (packed, 3 bytes/pixel, wh3 bytes), and requeue. Returns T on success.

@ cam_close Camera c → v

---

bpe.nu

packages/yoloe/src/bpe.nu — the CLIP byte-pair-encoding tokenizer in pure NURL. This is the word → token-id half of the MobileCLIP text path; the other half (token-ids → 512-d features) is the text-encoder ONNX graph already run by the onnx runtime (see tests/text_fwd.nu).

Faithful port of open_clip's SimpleTokenizer (the tokenizer MobileCLIP's ClipTokenizer wraps as get_tokenizer("ViT-B-16")):

• bytes_to_unicode() byte→symbol table (reversible byte-level coding),
• a 49408-entry encoder: 256 base symbols + 256 word-end (</w>) variants
• 48894 merge tokens + <start_of_text>/<end_of_text>,
• bpe_ranks from assets/clip_merges.txt (the decompressed

bpe_simple_vocab_16e6.txt, lines 1..48894),

• the BPE merge loop (lowest-rank bigram first, merge all occurrences),
• [sot] + ids + [eot] padded/truncated to the context length (77).

Word splitting reproduces CLIP's regex 'r"'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+" for the common case (ASCII / Latin-1 letters, digits, punctuation, English contractions). A letter is a-z / Latin-1 letters / any codepoint

= U+0100; a digit is 0-9. Exotic symbol-vs-letter classification of rare

BMP punctuation may differ from full Unicode \p{L}/\p{N}, which does not arise for object-detection class names.

API

: i BPE_INF 1000000000

: Tokenizer

: Tokenizer {
    ( Vec String ) byte_enc      // byte value 0..255 → its symbol string
    ( HashMap s i ) enc          // token string → id
    ( HashMap s i ) ranks        // "a b" merge line → rank
    ( Vec String ) arena         // owns enc/ranks key strings
    i sot
    i eot
}

A loaded tokenizer. The hash maps key on s (raw char*) pointers, so the backing strings must outlive the maps: arena owns every key string that is not already owned by byte_enc.

@ tokenizer_load s merges_path → Tokenizer

── load ───────────────────────────────────────────────────────── Build the tokenizer from the merges asset. Returns a Tokenizer whose maps are empty (sot=-1) on read failure.

@ bpe_encode Tokenizer tk s text → ( Vec i )

── encode / tokenize ──────────────────────────────────────────── text → list of token ids (no sot/eot, no padding).

@ bpe_tokenize Tokenizer tk s text i ctx → ( Vec i )

text → padded context: [sot] + ids + [eot], zero-padded/truncated to ctx. Returns a Vec<i> of length ctx.

---

main.nu

packages/yoloe/src/main.nu — the yoloe command: promptable open-vocabulary object detection AND instance segmentation, pure NURL on the GPU, with a live webcam mode that draws straight to the terminal. Single binary, three flag-driven sub-commands (see yoloe help):

yoloe detect --model M --classes C --image IMG [--out OUT] yoloe seg --model M --classes C --image IMG [--out OUT] yoloe cam --model M --classes C [--device DEV] [--frames N] [--out DIR] [--no-show] [--boxes]

cam shows the segmented feed live in the terminal (truecolor half-blocks); with no --frames it runs until Ctrl-C, and --out also saves the frames.

API

@ p s m → v

@ pf f x → v

@ pn i n → v

@ flag_set ( Vec String ) av s name → b

@ flag_str ( Vec String ) av s name → String

@ flag_int ( Vec String ) av s name i dflt → i

@ read_names s path → ( Vec String )

Read the prompt vocabulary from a text file (one class per line); the line count is the class count nc. Must match the names the model was exported with (tools/export.py) — swap both to detect a different vocabulary.

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ pal_r i k → i

@ pal_g i k → i

@ pal_b i k → i

@ frame_path s dir i n → String

frame path: <dir>/frameNNNNN.ppm

@ process_frame Image im OGraph g *Engine e ( Vec String ) names i nc b want_masks b verbose → i

Run the network on one image, drawing boxes (and, when want_masks, the per-object segmentation mask) onto im. When verbose, prints one line per detection. Returns the detection count.

@ load_model s path *b okcell → OGraph

@ run_still b want_masks String mp String np String ip String op → i

── still-image modes (detect / seg) ──────────────────────────────

@ run_cam String mp String np String dev i nframes String od b show b want_masks → i

── webcam mode (cam): live terminal display and/or save frames ───

@ usage → v

@ main → i