decode.nu

packages/yoloe/src/decode.nu — turn YOLOE's output0 into boxes.

output0 is [1, 4+nc+nm, na] (na = 8400 anchors over 3 scales): the first 4 channels are the box (cx,cy,w,h, already decoded to pixels in the letterboxed frame), the next nc are per-class scores (already sigmoid), the rest are segmentation mask coefficients (ignored here). For each anchor we take the best class above a threshold, then non-max suppress.

API

: Detection { i cls f score f cx f cy f w f h i ai }

A detection in letterboxed-pixel centre/size coordinates. ai is the anchor index it came from — needed to read the 32 mask coefficients (output0 channels 4+nc..4+nc+nm at that anchor) for segmentation.

& c @ nurl_peek_f32 *u base i idx → f

@ yolo_decode *u out i na i nc f thresh → ( Vec Detection )

Decode the best detections above thresh. nc = number of classes, na = anchors, no = total channels (4 + nc + masks).

@ yolo_nms ( Vec Detection ) dets f iou_thresh → ( Vec Detection )

Greedy non-max suppression (per class), keeping highest-score boxes.

---

mask.nu

packages/yoloe/src/mask.nu — instance-segmentation mask decode + overlay.

YOLOE-seg has two outputs: output0 [1, 4+nc+nm, na] (boxes, class scores, and nm=32 mask coefficients per anchor) and output1 [1, nm, MH, MW] — the mask "prototypes" (MH=MW=160, a quarter of the 640 input). The mask for a detection is a linear combination of the prototypes weighted by that anchor's 32 coefficients, then thresholded:

logit[y,x] = Σ_m coeff[m] · proto[m,y,x] (160×160) mask = sigmoid(logit) > 0.5 ⟺ logit > 0

cropped to the box and upsampled to image resolution (bilinear) — exactly ops.process_mask(..., upsample=True) in the reference. We precompute the 160×160 logit map per detection, then bilinear-sample it as we paint the box region of the original image, blending a translucent colour where the mask is set. This matches onnxruntime's masks (the proto tensor itself is bit-checked against onnxruntime in tests/seg_fwd.nu).

API

& c @ nurl_peek_f32 *u base i idx → f

& c @ nurl_poke_f32 *u base i idx f val → v

@ mask_nm → i

Mask prototype geometry: 32 coefficients, 160×160 prototypes at stride 4.

@ mask_dim → i

@ mask_stride → i

@ mask_coeffs *u out i na i nc i ai → *u

Read a detection's nm mask coefficients from output0 (channels 4+nc .. 4+nc+nm at anchor ai) into a fresh host buffer (caller frees).

@ mask_logits *u proto *u coeff i MH i MW → *u

Build the 160×160 mask logit map for one detection: logit[y,x] = Σ_m coeff[m]·proto[m,y,x]. proto is laid out [nm, MH, MW] row-major. Returns a fresh host f32 buffer of MH*MW (caller frees).

@ mask_sample *u L i MH i MW f fy f fx → f

Bilinear sample of an MH×MW grid at fractional (fy,fx); out-of-range coordinates clamp to the edge (align_corners=False convention).

@ mask_blend Image im i x i y i r i gg i bb i alpha → v

Blend colour (r,g,b) into a pixel at alpha/256 (alpha 0..256).

@ mask_overlay Image im *u L Letterbox lb i S i x0 i y0 i ow i oh i r i gg i bb i alpha → i

Overlay one detection's mask onto the original image. The box is given in ORIGINAL-image pixels (x0,y0,ow,oh); for each pixel we map back through the letterbox to the 160×160 mask grid and threshold the bilinearly-sampled logit at 0 (sigmoid > 0.5). S is the model input side (640).

---

image.nu

packages/yoloe/src/image.nu — image I/O + YOLO letterbox preprocessing.

Reads/writes binary PPM (P6), letterboxes to the model's square input (aspect-preserving resize + grey pad), packs to an NCHW float tensor normalised to [0,1], and draws labelled boxes. An Image keeps the raw file bytes plus a pixel-data offset, so reading copies nothing; a fresh image (letterbox/blank) owns a new byte buffer.

API

& c @ nurl_poke_f32 *u base i idx f val → v

: Image { i w i h ( Vec u ) buf i off }

: Letterbox { Image img f scale i padx i pady }

Letterbox result: the square image plus the transform back to original pixels (orig = (lb - pad) / scale).

@ img_w Image im → i

@ img_h Image im → i

@ img_get Image im i x i y i ch → i

@ img_set Image im i x i y i ch i val → v

@ ppm_read s path → ?Image

@ img_blank i w i h i fill → Image

@ ppm_write s path Image im → b

@ letterbox Image im i S → Letterbox

── letterbox to S×S (aspect-preserving, grey 114 pad) ────────────

@ img_to_nchw_norm Image im → *u

Pack to NCHW float tensor normalised to [0,1].

@ img_draw_rect Image im i x0 i y0 i x1 i y1 i r i gg i bb → v

---

prompt.nu

packages/yoloe/src/prompt.nu — RUNTIME-promptable open-vocabulary detection.

yoloe-prompt <model.onnx> <tpe.f32> <classes.txt> <image.ppm> [out.ppm]

Unlike src/main.nu (whose vocabulary is baked into the model at export time), this takes the prompt text embeddings as a runtime input: the model has two inputs, the image and tpe [1,K,512] (raw MobileCLIP text features for the K prompt words). Swap the tpe file + classes file to detect a different set of objects with the same model — no re-export.

Produce the model + tpe with tools/export_promptable.py and tools/gen_tpe.py.

API

& c @ nurl_peek_f32 *u base i idx → f

@ p s m → v

@ pf f x → v

@ pn i n → v

@ load_f32 s path *u pcell → *u

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ shape3 i a i b i c → ( Vec i )

@ main → i

---

v4l2.nu

packages/yoloe/src/v4l2.nu — webcam capture in pure NURL via Video4Linux2.

Opens /dev/videoN, negotiates a YUYV capture format, sets up memory-mapped streaming buffers, and hands back decoded RGB frames — no ffmpeg, no OpenCV, just open/ioctl/mmap on the kernel's V4L2 ABI through the FFI. The ioctl request codes and struct field offsets below are the stable x86_64 <linux/videodev2.h> layout (verified against the system headers):

v4l2_format 208 B type@0 pix.width@8 height@12 pixelformat@16 field@20 v4l2_requestbuffers 20 B count@0 type@4 memory@8 v4l2_buffer 88 B index@0 type@4 bytesused@8 memory@60 m.offset@64 length@72

Frames arrive as YUYV (4:2:2, 2 bytes/pixel) and are converted to packed RGB with the BT.601 integer transform. One Camera owns the fd plus the mmap'd ring of capture buffers.

API

& c @ open s path i flags → i

& c @ close i fd → i

& c @ ioctl i fd i req *u argp → i

& c @ mmap *u addr i length i prot i flags i fd i offset → *u

& c @ munmap *u addr i length → i

& c @ nurl_peek_i32 *u base i idx → i

& c @ nurl_poke_i32 *u base i idx i val → v

: Camera { i fd i w i h i nbuf ( Vec i ) bufptr ( Vec i ) buflen i ok }

A streaming webcam: fd, frame geometry, and the mmap'd buffer ring.

@ cam_open s path i w i h i nbuf → Camera

Open a webcam and start YUYV streaming at w×h with nbuf ring buffers.

@ cam_ok Camera c → b

@ cam_w Camera c → i

@ cam_h Camera c → i

@ cam_grab Camera c ( Vec u ) rgb → b

Grab one frame: dequeue a filled buffer, convert YUYV→RGB into rgb (packed, 3 bytes/pixel, wh3 bytes), and requeue. Returns T on success.

@ cam_close Camera c → v

---

bpe.nu

packages/yoloe/src/bpe.nu — the CLIP byte-pair-encoding tokenizer in pure NURL. This is the word → token-id half of the MobileCLIP text path; the other half (token-ids → 512-d features) is the text-encoder ONNX graph already run by the onnx runtime (see tests/text_fwd.nu).

Faithful port of open_clip's SimpleTokenizer (the tokenizer MobileCLIP's ClipTokenizer wraps as get_tokenizer("ViT-B-16")):

• bytes_to_unicode() byte→symbol table (reversible byte-level coding),
• a 49408-entry encoder: 256 base symbols + 256 word-end (</w>) variants
• 48894 merge tokens + <start_of_text>/<end_of_text>,
• bpe_ranks from assets/clip_merges.txt (the decompressed

bpe_simple_vocab_16e6.txt, lines 1..48894),

• the BPE merge loop (lowest-rank bigram first, merge all occurrences),
• [sot] + ids + [eot] padded/truncated to the context length (77).

Word splitting reproduces CLIP's regex 'r"'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+" for the common case (ASCII / Latin-1 letters, digits, punctuation, English contractions). A letter is a-z / Latin-1 letters / any codepoint

= U+0100; a digit is 0-9. Exotic symbol-vs-letter classification of rare

BMP punctuation may differ from full Unicode \p{L}/\p{N}, which does not arise for object-detection class names.

API

: i BPE_INF 1000000000

: Tokenizer

: Tokenizer {
    ( Vec String ) byte_enc      // byte value 0..255 → its symbol string
    ( HashMap s i ) enc          // token string → id
    ( HashMap s i ) ranks        // "a b" merge line → rank
    ( Vec String ) arena         // owns enc/ranks key strings
    i sot
    i eot
}

A loaded tokenizer. The hash maps key on s (raw char*) pointers, so the backing strings must outlive the maps: arena owns every key string that is not already owned by byte_enc.

@ tokenizer_load s merges_path → Tokenizer

── load ───────────────────────────────────────────────────────── Build the tokenizer from the merges asset. Returns a Tokenizer whose maps are empty (sot=-1) on read failure.

@ bpe_encode Tokenizer tk s text → ( Vec i )

── encode / tokenize ──────────────────────────────────────────── text → list of token ids (no sot/eot, no padding).

@ bpe_tokenize Tokenizer tk s text i ctx → ( Vec i )

text → padded context: [sot] + ids + [eot], zero-padded/truncated to ctx. Returns a Vec<i> of length ctx.

---

main.nu

packages/yoloe/src/main.nu — the yoloe command: promptable open-vocabulary object detection AND instance segmentation, pure NURL on the GPU, with a live webcam mode. This is the single binary nurlpkg install yoloe drops on your PATH; it dispatches three sub-commands:

yoloe detect <model.onnx> <classes.txt> <image.ppm> [out.ppm] yoloe seg <model.onnx> <classes.txt> <image.ppm> [out.ppm] yoloe cam <model.onnx> <classes.txt> <out_dir> [nframes] [device]

detect draws boxes; seg adds a per-object segmentation mask; cam streams frames off a V4L2 webcam (pure NURL — no ffmpeg/OpenCV) and writes each segmented frame to <out_dir>/frameNNNNN.ppm. See yoloe help.

API

@ p s m → v

@ pf f x → v

@ pn i n → v

@ read_names s path → ( Vec String )

Read the prompt vocabulary from a text file (one class per line); the line count is the class count nc. Must match the names the model was exported with (tools/export.py) — swap both to detect a different vocabulary.

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ pal_r i k → i

@ pal_g i k → i

@ pal_b i k → i

@ frame_path s dir i n → String

frame path: <dir>/frameNNNNN.ppm

@ process_frame Image im OGraph g *Engine e ( Vec String ) names i nc b want_masks b verbose → i

Run the network on one image, drawing boxes (and, when want_masks, the per-object segmentation mask) onto im. When verbose, prints one line per detection. Returns the detection count.

@ load_model s path *b okcell → OGraph

@ run_still b want_masks String mp String np String ip String op → i

── still-image modes (detect / seg) ──────────────────────────────

@ run_cam String mp String np String od i nframes String dev → i

── webcam mode (cam) ─────────────────────────────────────────────

@ usage → v

@ main → i