decode.nu

packages/yoloe/src/decode.nu — turn YOLOE's output0 into boxes.

output0 is [1, 4+nc+nm, na] (na = 8400 anchors over 3 scales): the first 4 channels are the box (cx,cy,w,h, already decoded to pixels in the letterboxed frame), the next nc are per-class scores (already sigmoid), the rest are segmentation mask coefficients (ignored here). For each anchor we take the best class above a threshold, then non-max suppress.

API

: Detection { i cls f score f cx f cy f w f h i ai }

A detection in letterboxed-pixel centre/size coordinates. ai is the anchor index it came from — needed to read the 32 mask coefficients (output0 channels 4+nc..4+nc+nm at that anchor) for segmentation.

& c @ nurl_peek_f32 *u base i idx → f

@ yolo_decode *u out i na i nc f thresh → ( Vec Detection )

Decode the best detections above thresh. nc = number of classes, na = anchors, no = total channels (4 + nc + masks).

@ yolo_nms ( Vec Detection ) dets f iou_thresh → ( Vec Detection )

Greedy non-max suppression (per class), keeping highest-score boxes.

---

mask.nu

packages/yoloe/src/mask.nu — instance-segmentation mask decode + overlay.

YOLOE-seg has two outputs: output0 [1, 4+nc+nm, na] (boxes, class scores, and nm=32 mask coefficients per anchor) and output1 [1, nm, MH, MW] — the mask "prototypes" (MH=MW=160, a quarter of the 640 input). The mask for a detection is a linear combination of the prototypes weighted by that anchor's 32 coefficients, then thresholded:

logit[y,x] = Σ_m coeff[m] · proto[m,y,x] (160×160) mask = sigmoid(logit) > 0.5 ⟺ logit > 0

cropped to the box and upsampled to image resolution (bilinear) — exactly ops.process_mask(..., upsample=True) in the reference. We precompute the 160×160 logit map per detection, then bilinear-sample it as we paint the box region of the original image, blending a translucent colour where the mask is set. This matches onnxruntime's masks (the proto tensor itself is bit-checked against onnxruntime in tests/seg_fwd.nu).

API

& c @ nurl_peek_f32 *u base i idx → f

& c @ nurl_poke_f32 *u base i idx f val → v

@ mask_nm → i

Mask prototype geometry: 32 coefficients, 160×160 prototypes at stride 4.

@ mask_dim → i

@ mask_stride → i

@ mask_coeffs *u out i na i nc i ai → *u

Read a detection's nm mask coefficients from output0 (channels 4+nc .. 4+nc+nm at anchor ai) into a fresh host buffer (caller frees).

@ mask_logits *u proto *u coeff i MH i MW → *u

Build the 160×160 mask logit map for one detection: logit[y,x] = Σ_m coeff[m]·proto[m,y,x]. proto is laid out [nm, MH, MW] row-major. Returns a fresh host f32 buffer of MH*MW (caller frees).

@ mask_sample *u L i MH i MW f fy f fx → f

Bilinear sample of an MH×MW grid at fractional (fy,fx); out-of-range coordinates clamp to the edge (align_corners=False convention).

@ mask_blend Image im i x i y i r i gg i bb i alpha → v

Blend colour (r,g,b) into a pixel at alpha/256 (alpha 0..256).

@ mask_overlay Image im *u L Letterbox lb i S i x0 i y0 i ow i oh i r i gg i bb i alpha → i

Overlay one detection's mask onto the original image. The box is given in ORIGINAL-image pixels (x0,y0,ow,oh); for each pixel we map back through the letterbox to the 160×160 mask grid and threshold the bilinearly-sampled logit at 0 (sigmoid > 0.5). S is the model input side (640).

---

image.nu

packages/yoloe/src/image.nu — image I/O + YOLO letterbox preprocessing.

Reads/writes binary PPM (P6), letterboxes to the model's square input (aspect-preserving resize + grey pad), packs to an NCHW float tensor normalised to [0,1], and draws labelled boxes. An Image keeps the raw file bytes plus a pixel-data offset, so reading copies nothing; a fresh image (letterbox/blank) owns a new byte buffer.

API

& c @ nurl_poke_f32 *u base i idx f val → v

: Image { i w i h ( Vec u ) buf i off }

: Letterbox { Image img f scale i padx i pady }

Letterbox result: the square image plus the transform back to original pixels (orig = (lb - pad) / scale).

@ img_w Image im → i

@ img_h Image im → i

@ img_get Image im i x i y i ch → i

@ img_set Image im i x i y i ch i val → v

@ ppm_read s path → ?Image

@ img_blank i w i h i fill → Image

@ ppm_write s path Image im → b

@ letterbox Image im i S → Letterbox

── letterbox to S×S (aspect-preserving, grey 114 pad) ────────────

@ img_to_nchw_norm Image im → *u

Pack to NCHW float tensor normalised to [0,1].

@ img_draw_rect Image im i x0 i y0 i x1 i y1 i r i gg i bb → v

---

prompt.nu

packages/yoloe/src/prompt.nu — RUNTIME-promptable open-vocabulary detection.

yoloe-prompt <model.onnx> <tpe.f32> <classes.txt> <image.ppm> [out.ppm]

Unlike src/main.nu (whose vocabulary is baked into the model at export time), this takes the prompt text embeddings as a runtime input: the model has two inputs, the image and tpe [1,K,512] (raw MobileCLIP text features for the K prompt words). Swap the tpe file + classes file to detect a different set of objects with the same model — no re-export.

Produce the model + tpe with tools/export_promptable.py and tools/gen_tpe.py.

API

& c @ nurl_peek_f32 *u base i idx → f

@ p s m → v

@ pf f x → v

@ pn i n → v

@ load_f32 s path *u pcell → *u

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ shape3 i a i b i c → ( Vec i )

@ main → i

---

segcam.nu

packages/yoloe/src/segcam.nu — LIVE instance segmentation from a webcam.

yoloe-segcam <model.onnx> <classes.txt> <out_dir> [nframes] [device]

Captures frames straight off a V4L2 webcam (pure NURL — v4l2.nu, no ffmpeg/OpenCV), runs the YOLOE-seg network on the GPU, and writes each frame to <out_dir>/frameNNNNN.ppm with every detected object's mask painted over it. One GPU engine and one camera stream are reused across all frames (kernels compiled once, buffers mmap'd once). Reassemble to a video with e.g. ffmpeg -framerate 10 -i out/frame%05d.ppm seg.mp4

This is the segmentation counterpart of src/main.nu / src/seg.nu, driven by a real camera instead of a still image.

API

@ p s m → v

@ pf f x → v

@ pn i n → v

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ pal_r i k → i

@ pal_g i k → i

@ pal_b i k → i

@ frame_path s dir i n → String

frame path: <dir>/frameNNNNN.ppm

@ seg_frame Image im OGraph g *Engine e ( Vec String ) names i nc → i

Run the seg network on one already-letterboxed-able image, painting masks and boxes onto im. Returns the number of detections.

@ main → i

---

v4l2.nu

packages/yoloe/src/v4l2.nu — webcam capture in pure NURL via Video4Linux2.

Opens /dev/videoN, negotiates a YUYV capture format, sets up memory-mapped streaming buffers, and hands back decoded RGB frames — no ffmpeg, no OpenCV, just open/ioctl/mmap on the kernel's V4L2 ABI through the FFI. The ioctl request codes and struct field offsets below are the stable x86_64 <linux/videodev2.h> layout (verified against the system headers):

v4l2_format 208 B type@0 pix.width@8 height@12 pixelformat@16 field@20 v4l2_requestbuffers 20 B count@0 type@4 memory@8 v4l2_buffer 88 B index@0 type@4 bytesused@8 memory@60 m.offset@64 length@72

Frames arrive as YUYV (4:2:2, 2 bytes/pixel) and are converted to packed RGB with the BT.601 integer transform. One Camera owns the fd plus the mmap'd ring of capture buffers.

API

& c @ open s path i flags → i

& c @ close i fd → i

& c @ ioctl i fd i req *u argp → i

& c @ mmap *u addr i length i prot i flags i fd i offset → *u

& c @ munmap *u addr i length → i

& c @ nurl_peek_i32 *u base i idx → i

& c @ nurl_poke_i32 *u base i idx i val → v

: Camera { i fd i w i h i nbuf ( Vec i ) bufptr ( Vec i ) buflen i ok }

A streaming webcam: fd, frame geometry, and the mmap'd buffer ring.

@ cam_open s path i w i h i nbuf → Camera

Open a webcam and start YUYV streaming at w×h with nbuf ring buffers.

@ cam_ok Camera c → b

@ cam_w Camera c → i

@ cam_h Camera c → i

@ cam_grab Camera c ( Vec u ) rgb → b

Grab one frame: dequeue a filled buffer, convert YUYV→RGB into rgb (packed, 3 bytes/pixel, wh3 bytes), and requeue. Returns T on success.

@ cam_close Camera c → v

---

bpe.nu

packages/yoloe/src/bpe.nu — the CLIP byte-pair-encoding tokenizer in pure NURL. This is the word → token-id half of the MobileCLIP text path; the other half (token-ids → 512-d features) is the text-encoder ONNX graph already run by the onnx runtime (see tests/text_fwd.nu).

Faithful port of open_clip's SimpleTokenizer (the tokenizer MobileCLIP's ClipTokenizer wraps as get_tokenizer("ViT-B-16")):

• bytes_to_unicode() byte→symbol table (reversible byte-level coding),
• a 49408-entry encoder: 256 base symbols + 256 word-end (</w>) variants
• 48894 merge tokens + <start_of_text>/<end_of_text>,
• bpe_ranks from assets/clip_merges.txt (the decompressed

bpe_simple_vocab_16e6.txt, lines 1..48894),

• the BPE merge loop (lowest-rank bigram first, merge all occurrences),
• [sot] + ids + [eot] padded/truncated to the context length (77).

Word splitting reproduces CLIP's regex 'r"'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+" for the common case (ASCII / Latin-1 letters, digits, punctuation, English contractions). A letter is a-z / Latin-1 letters / any codepoint

= U+0100; a digit is 0-9. Exotic symbol-vs-letter classification of rare

BMP punctuation may differ from full Unicode \p{L}/\p{N}, which does not arise for object-detection class names.

API

: i BPE_INF 1000000000

: Tokenizer

: Tokenizer {
    ( Vec String ) byte_enc      // byte value 0..255 → its symbol string
    ( HashMap s i ) enc          // token string → id
    ( HashMap s i ) ranks        // "a b" merge line → rank
    ( Vec String ) arena         // owns enc/ranks key strings
    i sot
    i eot
}

A loaded tokenizer. The hash maps key on s (raw char*) pointers, so the backing strings must outlive the maps: arena owns every key string that is not already owned by byte_enc.

@ tokenizer_load s merges_path → Tokenizer

── load ───────────────────────────────────────────────────────── Build the tokenizer from the merges asset. Returns a Tokenizer whose maps are empty (sot=-1) on read failure.

@ bpe_encode Tokenizer tk s text → ( Vec i )

── encode / tokenize ──────────────────────────────────────────── text → list of token ids (no sot/eot, no padding).

@ bpe_tokenize Tokenizer tk s text i ctx → ( Vec i )

text → padded context: [sot] + ids + [eot], zero-padded/truncated to ctx. Returns a Vec<i> of length ctx.

---

main.nu

packages/yoloe/src/main.nu — promptable open-vocabulary detection.

yoloe <model.onnx> <classes.txt> <image.ppm> [out.ppm]

Runs an exported YOLOE detector on the GPU through the onnx package, decodes the boxes, suppresses overlaps, prints the labelled detections, and writes an annotated image. The class vocabulary is read from a text file (one prompt per line) — it must match the names the model was exported with (tools/export.py), so swapping both detects a different vocabulary. The detector itself is vocabulary-agnostic.

Build: nurlpkg install then NURL_STDLIB=<repo> ../../nurl.sh src/main.nu

API

& c @ nurl_peek_f32 *u base i idx → f

@ p s m → v

@ pf f x → v

@ pn i n → v

@ read_names s path → ( Vec String )

Read the prompt vocabulary from a text file (one class per line); the number of lines is the class count nc. The file must match the names the model was exported with — swap both to detect a different vocabulary.

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ main → i

---

seg.nu

packages/yoloe/src/seg.nu — promptable open-vocabulary instance SEGMENTATION.

yoloe-seg <model.onnx> <classes.txt> <image.ppm> [out.ppm]

Like src/main.nu but it also paints each detection's segmentation MASK. The model is the standard YOLOE-seg export (tools/export.py): two outputs, output0 [1, 4+nc+32, 8400] (boxes + class scores + 32 mask coefficients) and output1 [1, 32, 160, 160] (the mask prototypes). For each surviving detection we combine the prototypes by its coefficients, threshold, and blend a translucent colour over the masked pixels (mask.nu).

API

@ p s m → v

@ pf f x → v

@ pn i n → v

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ pal_r i k → i

A small palette so distinct instances get distinct mask colours.

@ pal_g i k → i

@ pal_b i k → i

@ main → i