NURLNURL registrynurl-lang.org →

← yoloe

yoloe 0.6.3 API

decode.nu

packages/yoloe/src/decode.nu — turn YOLOE's output0 into boxes.

output0 is [1, 4+nc+nm, na] (na = 8400 anchors over 3 scales): the first 4 channels are the box (cx,cy,w,h, already decoded to pixels in the letterboxed frame), the next nc are per-class scores (already sigmoid), the rest are segmentation mask coefficients (ignored here). For each anchor we take the best class above a threshold, then non-max suppress.

API

: Detection { i cls f score f cx f cy f w f h i ai }

A detection in letterboxed-pixel centre/size coordinates. ai is the anchor index it came from — needed to read the 32 mask coefficients (output0 channels 4+nc..4+nc+nm at that anchor) for segmentation.

& c @ nurl_peek_f32 *u base i idx → f

@ yolo_decode *u out i na i nc f thresh → ( Vec Detection )

Decode the best detections above thresh. nc = number of classes, na = anchors, no = total channels (4 + nc + masks).

@ yolo_nms ( Vec Detection ) dets f iou_thresh → ( Vec Detection )

Greedy non-max suppression (per class), keeping highest-score boxes.

mask.nu

packages/yoloe/src/mask.nu — instance-segmentation mask decode + overlay.

YOLOE-seg has two outputs: output0 [1, 4+nc+nm, na] (boxes, class scores, and nm=32 mask coefficients per anchor) and output1 [1, nm, MH, MW] — the mask "prototypes" (MH=MW=160, a quarter of the 640 input). The mask for a detection is a linear combination of the prototypes weighted by that anchor's 32 coefficients, then thresholded:

logit[y,x] = Σ_m coeff[m] · proto[m,y,x] (160×160) mask = sigmoid(logit) > 0.5 ⟺ logit > 0

cropped to the box and upsampled to image resolution (bilinear) — exactly ops.process_mask(..., upsample=True) in the reference. We precompute the 160×160 logit map per detection, then bilinear-sample it as we paint the box region of the original image, blending a translucent colour where the mask is set. This matches onnxruntime's masks (the proto tensor itself is bit-checked against onnxruntime in tests/seg_fwd.nu).

API

& c @ nurl_peek_f32 *u base i idx → f

& c @ nurl_poke_f32 *u base i idx f val → v

@ mask_nm → i

Mask prototype geometry: 32 coefficients, 160×160 prototypes at stride 4.

@ mask_dim → i

@ mask_stride → i

@ mask_coeffs *u out i na i nc i ai → *u

Read a detection's nm mask coefficients from output0 (channels 4+nc .. 4+nc+nm at anchor ai) into a fresh host buffer (caller frees).

@ mask_logits *u proto *u coeff i MH i MW → *u

Build the 160×160 mask logit map for one detection: logit[y,x] = Σ_m coeff[m]·proto[m,y,x]. proto is laid out [nm, MH, MW] row-major. Returns a fresh host f32 buffer of MH*MW (caller frees).

@ mask_sample *u L i MH i MW f fy f fx → f

Bilinear sample of an MH×MW grid at fractional (fy,fx); out-of-range coordinates clamp to the edge (align_corners=False convention).

@ mask_blend Image im i x i y i r i gg i bb i alpha → v

Blend colour (r,g,b) into a pixel at alpha/256 (alpha 0..256).

@ mask_overlay Image im *u L Letterbox lb i S i x0 i y0 i ow i oh i r i gg i bb i alpha → i

Overlay one detection's mask onto the original image. The box is given in ORIGINAL-image pixels (x0,y0,ow,oh); for each pixel we map back through the letterbox to the 160×160 mask grid and threshold the bilinearly-sampled logit at 0 (sigmoid > 0.5). S is the model input side (640).

display.nu

packages/yoloe/src/display.nu — show an Image live in the terminal.

Renders a packed-RGB Image as 24-bit-colour half-blocks: each character cell is the upper-half-block glyph ▀ whose FOREGROUND colour is one image pixel and BACKGROUND colour the pixel just below it — so one cell carries two vertical pixels, giving a square-ish picture in any truecolor terminal (gnome-terminal, kitty, xterm, … and over SSH). No X11, no SDL, no library.

The frame is scaled to fit the terminal (size via TIOCGWINSZ) preserving aspect, built into one String, and printed after homing the cursor so a continuous capture loop overwrites in place like a video.

API

& c @ ioctl i fd i req *u argp → i

& c @ nurl_poke_i32 *u base i idx i32 val → v

& c @ nurl_peek_i32 *u base i idx → i32

@ term_enter → v

Clear the screen and hide the cursor (call once before a live loop).

@ term_leave → v

Show the cursor again (call when the live loop ends).

@ img_show Image im → v

Render im to the terminal as truecolor half-blocks, scaled to fit.

image.nu

packages/yoloe/src/image.nu — YOLO-facing adapter over packages/image.

The codecs and raster ops live in the image package (PNG / baseline + progressive JPEG / PPM decode+encode, rectangle drawing — see deps/image); this file keeps only what is YOLO-specific: letterbox preprocessing (aspect-preserving nearest resize + grey-114 pad, matching ultralytics), packing to the [0,1]-normalised NCHW float tensor the model consumes, edge-clamped reads for the mask overlay, and load/save-by-extension. The Image type is the image package's (width/height/channels/data).

API

& c @ nurl_poke_f32 *u base i idx f val → v

: Letterbox { Image img f scale i padx i pady }

Letterbox result: the square image plus the transform back to original pixels (orig = (lb - pad) / scale).

@ img_w Image im → i

@ img_h Image im → i

@ img_get Image im i x i y i ch → i

Edge-clamped read (the mask overlay and letterbox sample with clamping).

@ img_set Image im i x i y i ch i val → v

Saturating store (image_set masks to the low byte; overlay math must clamp).

@ img_blank i w i h i fill → Image

An RGB image filled with a constant grey level.

@ img_load s path → ?Image

Load any supported format (PNG / baseline+progressive JPEG / PPM) as 3-channel RGB. On None, ( image_error ) says why.

@ img_save s path Image im → b

Save by extension: .png / .jpg / .jpeg (quality 90) / anything else = PPM.

@ letterbox Image im i S → Letterbox

── letterbox to S×S (aspect-preserving, grey 114 pad) ────────────

@ img_to_nchw_norm Image im → *u

Pack to NCHW float tensor normalised to [0,1].

@ img_draw_rect Image im i x0 i y0 i x1 i y1 i r i gg i bb → v

Draw a thickness-3 rectangle outline in the given colour.

prompt.nu

packages/yoloe/src/prompt.nu — RUNTIME-promptable open-vocabulary detection.

yoloe-prompt <model.onnx> <tpe.f32> <classes.txt> <img> [out]

Unlike src/main.nu (whose vocabulary is baked into the model at export time), this takes the prompt text embeddings as a runtime input: the model has two inputs, the image and tpe [1,K,512] (raw MobileCLIP text features for the K prompt words). Swap the tpe file + classes file to detect a different set of objects with the same model — no re-export.

Produce the model + tpe with tools/export_promptable.py and tools/gen_tpe.py.

API

& c @ nurl_peek_f32 *u base i idx → f

@ p s m → v

@ pf f x → v

@ pn i n → v

@ load_f32 s path * u pcell → *u

@ read_names s path → ( Vec String )

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ shape3 i a i b i c → ( Vec i )

@ main → i

v4l2.nu

packages/yoloe/src/v4l2.nu — webcam capture in pure NURL via Video4Linux2.

Opens /dev/videoN, negotiates a YUYV capture format, sets up memory-mapped streaming buffers, and hands back decoded RGB frames — no ffmpeg, no OpenCV, just open/ioctl/mmap on the kernel's V4L2 ABI through the FFI. The ioctl request codes and struct field offsets below are the stable x86_64 <linux/videodev2.h> layout (verified against the system headers):

v4l2_format 208 B type@0 pix.width@8 height@12 pixelformat@16 field@20 v4l2_requestbuffers 20 B count@0 type@4 memory@8 v4l2_buffer 88 B index@0 type@4 bytesused@8 memory@60 m.offset@64 length@72

Frames arrive as YUYV (4:2:2, 2 bytes/pixel) and are converted to packed RGB with the BT.601 integer transform. One Camera owns the fd plus the mmap'd ring of capture buffers.

API

& c @ open s path i flags → i

& c @ close i fd → i

& c @ ioctl i fd i req *u argp → i

& c @ mmap *u addr i length i prot i flags i fd i offset → *u

& c @ munmap *u addr i length → i

& c @ nurl_peek_i32 *u base i idx → i32

& c @ nurl_poke_i32 *u base i idx i32 val → v

: Camera { i fd i w i h i nbuf ( Vec i ) bufptr ( Vec i ) buflen i ok }

A streaming webcam: fd, frame geometry, and the mmap'd buffer ring.

@ cam_open s path i w i h i nbuf → Camera

Open a webcam and start YUYV streaming at w×h with nbuf ring buffers.

@ cam_ok Camera c → b

@ cam_w Camera c → i

@ cam_h Camera c → i

@ cam_grab Camera c ( Vec u ) rgb → b

Grab one frame: dequeue a filled buffer, convert YUYV→RGB into rgb (packed, 3 bytes/pixel, wh3 bytes), and requeue. Returns T on success.

@ cam_close Camera c → v

bpe.nu

packages/yoloe/src/bpe.nu — the CLIP byte-pair-encoding tokenizer in pure NURL. This is the word → token-id half of the MobileCLIP text path; the other half (token-ids → 512-d features) is the text-encoder ONNX graph already run by the onnx runtime (see tests/text_fwd.nu).

Faithful port of open_clip's SimpleTokenizer (the tokenizer MobileCLIP's ClipTokenizer wraps as get_tokenizer("ViT-B-16")):

bpe_simple_vocab_16e6.txt, lines 1..48894),

Word splitting reproduces CLIP's regex 'r"'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+" for the common case (ASCII / Latin-1 letters, digits, punctuation, English contractions). A letter is a-z / Latin-1 letters / any codepoint

= U+0100; a digit is 0-9. Exotic symbol-vs-letter classification of rare

BMP punctuation may differ from full Unicode \p{L}/\p{N}, which does not arise for object-detection class names.

API

: i BPE_INF 1000000000

: Tokenizer

: Tokenizer {
    ( Vec String ) byte_enc  // byte value 0..255 → its symbol string
    ( HashMap s i ) enc  // token string → id
    ( HashMap s i ) ranks  // "a b" merge line → rank
    ( Vec String ) arena  // owns enc/ranks key strings
    i sot
    i eot
}

A loaded tokenizer. The hash maps key on s (raw char*) pointers, so the backing strings must outlive the maps: arena owns every key string that is not already owned by byte_enc.

@ tokenizer_load s merges_path → Tokenizer

── load ───────────────────────────────────────────────────────── Build the tokenizer from the merges asset. Returns a Tokenizer whose maps are empty (sot=-1) on read failure.

@ tokenizer_load_builtin → Tokenizer

Build the tokenizer from the merge table compiled into the binary (src/clip_merges_data.nu, generated by tools/embed_merges.py). An installed tool has no package tree on disk, so this is the default; tokenizer_load stays for custom vocabularies via --merges.

@ bpe_encode Tokenizer tk s text → ( Vec i )

── encode / tokenize ──────────────────────────────────────────── text → list of token ids (no sot/eot, no padding).

@ bpe_tokenize Tokenizer tk s text i ctx → ( Vec i )

text → padded context: [sot] + ids + [eot], zero-padded/truncated to ctx. Returns a Vec<i> of length ctx.

main.nu

packages/yoloe/src/main.nu — the yoloe command: promptable open-vocabulary object detection AND instance segmentation, pure NURL on the GPU, with a live webcam mode that draws straight to the terminal. Single binary, three sub-commands, parsed by the cli package (see yoloe --help):

yoloe detect --model M --classes C --image IMG [--out OUT] yoloe seg --model M --classes C --image IMG [--out OUT] yoloe cam --model M --classes C [--device DEV] [--frames N] [--out DIR] [--no-show] [--boxes]

cam shows the segmented feed live in the terminal (truecolor half-blocks); with no --frames it runs until Ctrl-C, and --out also saves the frames.

API

@ p s m → v

@ pf f x → v

@ pn i n → v

@ read_names s path → ( Vec String )

Read the prompt vocabulary from a text file (one class per line); the line count is the class count nc. Must match the names the model was exported with (tools/export.py) — swap both to detect a different vocabulary.

@ name_at ( Vec String ) names i i → s

@ shape4 i a i b i c i d → ( Vec i )

@ pal_r i k → i

@ pal_g i k → i

@ pal_b i k → i

@ frame_path s dir i n → String

frame path: <dir>/frameNNNNN.ppm

@ process_frame Image im OGraph g * Engine e ( Vec String ) names i nc b want_boxes b want_masks b verbose → i

Run the network on one image, drawing boxes (and, when want_masks, the per-object segmentation mask) onto im. When verbose, prints one line per detection. Returns the detection count.

@ load_model s path * b okcell → OGraph

@ run_still b want_boxes b want_masks String mp String np String ip String op i gpu → i

── still-image modes (detect / seg) ──────────────────────────────

@ run_cam String mp String np String dev i nframes String od i mode b want_boxes b want_masks i gpu → i

── webcam mode (cam): live terminal display and/or save frames ───

@ main → i

clip_merges_data.nu

GENERATED by tools/embed_merges.py from assets/clip_merges.txt — do not edit. The CLIP BPE merge table (48893 lines) in ~32 KB line-aligned chunks, embedded so installed binaries need no asset file on disk (see tokenizer_load_builtin in bpe.nu). Chunked because released nurlc (≤ v0.11.1) stack-overflows on string literals past ~48 KB.

API

@ clip_merges_chunks → i

@ clip_merges_chunk i k → s

window.nu

packages/yoloe/src/window.nu — a real GUI window for the live preview, in pure NURL via Xlib (libX11) bound directly — no SDL, no GTK, no toolkit.

Opens an X11 window the size of the camera frame and blits each segmented RGB frame at full resolution with XPutImage (so unlike the terminal view there's no down-scaling). Polls for a keypress / window-close so the loop can exit cleanly. The toolchain auto-links -lX11 only when @XOpenDisplay appears (build.sh writes the runtime.X11 sentinel when libX11 is found), so a headless build is unaffected.

API

& X11 @ XOpenDisplay *u name → *u

& X11 @ XDefaultScreen *u dpy → i

& X11 @ XRootWindow *u dpy i screen → i

& X11 @ XDefaultVisual *u dpy i screen → *u

& X11 @ XDefaultDepth *u dpy i screen → i

& X11 @ XCreateSimpleWindow *u dpy i parent i x i y i w i h i bw i border i bg → i

& X11 @ XStoreName *u dpy i win s name → i

& X11 @ XSelectInput *u dpy i win i mask → i

& X11 @ XMapWindow *u dpy i win → i

& X11 @ XCreateGC *u dpy i drawable i valuemask *u values → *u

& X11 @ XCreateImage *u dpy *u visual i depth i format i offset *u data i width i height i pad i bpl → *u

& X11 @ XPutImage *u dpy i d *u gc *u image i sx i sy i dx i dy i w i h → i

& X11 @ XFlush *u dpy → i

& X11 @ XPending *u dpy → i

& X11 @ XNextEvent *u dpy *u event → i

& X11 @ XInternAtom *u dpy s name i only → i

& X11 @ XSetWMProtocols *u dpy i win *u protocols i count → i

& X11 @ XCloseDisplay *u dpy → i

& c @ nurl_poke_i32 *u base i idx i32 val → v

& c @ nurl_peek_i32 *u base i idx → i32

: XWin { i dpy i win i gc i img i data i w i h i ok }

Pointers (Display, GC, XImage, data) carried as i64; w/h the frame size.

@ xwin_ok XWin x → b

@ xwin_open i w i h s title → XWin

Open a window of w×h titled title. ok=0 if no X display (run headless).

@ xwin_show XWin x Image im → v

Blit one RGB Image into the window. The frame must match the window size.

@ xwin_should_close XWin x → b

Drain pending events; return T if the user asked to close (key, click, or the window-manager close button).

@ xwin_close XWin x → v