dino.nu

packages/lingbot-map/src/dino.nu — the DINOv2 ViT-L/14 feature extractor the aggregator uses as its patch embedding.

It is a whole 24-block ViT, run frozen, and its output — the patch tokens after the final norm — is what the aggregator's own 48 blocks then work on. The reference reaches it as patch_embed(images)["x_norm_patchtokens"].

Token order inside DINOv2, which the port has to reproduce exactly because the position embedding is added in the middle of building it:

[cls] + patches ← pos_embed added HERE [cls] + [reg×4] + patches ← register tokens inserted AFTER

so the registers get no position embedding at all, and the patch tokens the aggregator wants are everything from index 5 on.

The position embedding ships for a 37×37 grid; a 518×294 frame is 37×21, so it is resampled every time — with torch's bicubic+antialias kernel, which is NOT the one preprocessing uses (see src/interp.nu).

( dn_load w kit ) → Dino ( dn_free d ) → v ( dn_forward kit d ws img h w gh gw tok ) → b img: [3, H, W] f32 device, ALREADY ImageNet-normalised out: [gh*gw, 1024] f32 device — x_norm_patchtokens

API

: i DN_DIM 1024

: i DN_HEADS 16

: i DN_HIDDEN 4096

: i DN_DEPTH 24

: i DN_PATCH 14

: i DN_REG 4

: i DN_GRID 37 // the grid pos_embed ships for: 37×37 + 1 cls = 1370

: f DN_EPS 0.000001

DinoVisionTransformer builds its blocks with partial(LayerNorm, eps=1e-6). The aggregator's own blocks do not pass a norm_layer at all and so get nn.LayerNorm's default, 1e-5 — see AG_EPS.

: Dino

: Dino {
    ( Vec LmBlk ) blocks
    GkBuf proj_w  // [1024, 3·14·14] — Conv2d weight, flattened
    GkBuf proj_b
    GkBuf normg
    GkBuf normb
    // cls, the four register tokens and the position grid stay on the
    // HOST: they are a few kB, they are combined per frame (pos_embed is
    // added to cls and to the patches but NOT to the registers), and the
    // position grid has to be resampled by a host routine anyway.
    ( Vec f ) cls
    ( Vec f ) reg
    ( Vec f ) pos  // [1370, 1024]
}

@ dn_free Dino d → v

@ dn_load * Lw w * GpuKit kit → Dino

@ _dn_host * Lw w s name → ( Vec f )

A tensor read into a host vector, sized from the checkpoint.

@ dn_tokens i gh i gw → i

Number of tokens DINOv2 carries for a gh×gw patch grid: cls + 4 registers + patches.

@ dn_pos_for Dino d i gh i gw → ( Vec f )

pos_embed resampled to this frame's grid, as a host vector laid out [1 + gh*gw, 1024]: the cls row unchanged, then the interpolated grid.

The reference reads pos_embed[:, 0] for cls and interpolates pos_embed[:, 1:] reshaped to 37×37 — with the ROW axis first, so a 518-wide 294-high frame resamples 37×37 → 21×37, not 37×21.

@ dn_forward * GpuKit kit Dino d LmWs ws * f img i h i w i gh i gw GkBuf tok → b

Run the frozen DINOv2 trunk over one already-normalised frame.

img is [3, H, W] on the HOST (planar, ImageNet-normalised), and the patch layout work — im2col, the cls/register head, the position grid — happens there too: it is pure addressing over a couple of megabytes, and doing it on the host keeps three device kernels from existing. Everything after the projection is on the device.

tok must hold dn_tokens(gh, gw) × 1024 f32 and comes back holding the FULL token array; the patch tokens the aggregator wants are the last gh·gw rows, i.e. lm_view tok (5·1024) (gh·gw·1024).

---

geom.nu

packages/lingbot-map/src/geom.nu — camera geometry for LingBot-Map.

The model predicts, per frame, a 9-vector "pose encoding":

[0:3] T translation, world→camera [3:7] quat rotation, world→camera, XYZW (scalar LAST) [7] fov_h [8] fov_w

Everything downstream — the extrinsics, the intrinsics, unprojecting a depth map into world space — falls out of that. This module is the arithmetic, host-side in f64: it runs once per frame over 9 numbers and a H*W depth map, so there is nothing here worth a kernel.

Conventions, kept exactly as the reference has them because they are the kind of thing that silently mirrors a reconstruction:

• OpenCV camera axes: x right, y down, z forward.
• Extrinsics are world→camera ("camera from world"), [R|t], 3x4.
• The quaternion is scalar-LAST (i, j, k, r), not scalar-first.
• The principal point is assumed to be the image centre.

( quat_to_mat qx qy qz qw out ) → v out = 9 doubles, row-major ( mat_to_quat m out ) → v out = 4 doubles, XYZW ( pose_enc_to_extri pe out ) → v out = 12 doubles (3x4) ( pose_enc_to_intri pe H W out ) → v out = 9 doubles (3x3) ( se3_inverse m4 out ) → v 4x4 → 4x4 ( unproject px py depth Kinv c2w out ) → v one pixel → world xyz

API

@ quat_to_mat f qi f qj f qk f qr * f out → v

── quaternion (XYZW, scalar last) → 3x3 rotation, row-major ────────

The reference divides by the quaternion's own squared norm rather than normalising first, so a non-unit quaternion still produces a proper rotation. Predicted quaternions are never exactly unit, and the two spellings differ in the last bits — this one matches.

@ mat_to_quat * f m * f out → v

3x3 rotation (row-major) → quaternion XYZW.

Four candidate quaternions are formed, one per component taken as the pivot, and the one with the largest |component| is kept — the standard guard against the cancellation that ruins the naive trace formula near a 180-degree rotation.

@ pose_enc_to_extri * f pe * f out → v

Extrinsics [R|t], world→camera, 3x4 row-major (12 doubles).

@ pose_enc_to_intri * f pe i H i W * f out → v

Intrinsics from the two field-of-view angles, 3x3 row-major. The principal point is the image centre — the model is only ever fed centre-cropped frames, and the reference assumes the same.

@ intri_inverse * f k * f out → v

Inverse of an intrinsics matrix of the shape above (fx, fy, cx, cy) — upper triangular, so the closed form is exact and needs no pivoting.

@ se3_inverse * f m * f out → v

Inverse of a 4x4 SE3 (rigid) matrix: [R|t] → [Rᵀ | −Rᵀt]. Closed form, not a general solve — the whole point is that a rigid transform's inverse costs a transpose and a matrix-vector product.

@ pose_enc_to_c2w * f pe * f out → v

Camera-to-world 4x4 from a pose encoding: build [R|t] world→camera, lift to 4x4, invert.

@ unproject f px f py f depth * f kinv * f c2w * f out → v

One pixel of a depth map → a world-space point.

camera ray = K⁻¹ · (px, py, 1) camera point = ray · depth (NOT ray normalised — the model's depth is along z, not range) world point = c2w · (camera point, 1)

---

block.nu

packages/lingbot-map/src/block.nu — the transformer block, 72 of which make up this model (24 DINOv2 + 24 frame + 24 global).

x = x + ls1 · attn(norm1(x)) x = x + ls2 · mlp(norm2(x))

with pre-norm LayerNorm, LayerScale on both residual branches, exact (erf) GELU in the MLP, and attention that layer-normalises q and k per head before rotating them — qk_norm=True, which is not the DINOv2 default and is easy to drop on the floor when porting.

Host f64, one frame at a time. This is the reference the device path has to agree with, and the shape everything downstream is written against; the matmuls inside are the only parts that move to kernels.

( bk_layernorm x rows cols g b eps out ) ( bk_gelu x n out ) ( bk_linear x rows k w bias n out ) out = x·Wᵀ + b, W is [n, k] ( bk_attention x n dim heads qkv_w qkv_b qn_g qn_b kn_g kn_b proj_w proj_b rows cols cos sin out scratch ) ( bk_block ... ) the whole thing

API

: f BK_EPS 0.000001 // norm1 / norm2: DINOv2's partial(LayerNorm, eps=1e-6)

: f BK_QK_EPS 0.00001

q_norm / k_norm use a DIFFERENT eps, and not on purpose: Block builds its Attention without passing norm_layer, so the qk norms fall back to nn.LayerNorm's default 1e-5 while norm1/norm2 get DINOv2's 1e-6. The reference is what it is; matching it means carrying both constants.

@ bk_layernorm * f x i rows i cols * f g * f b f eps * f out → v

out[r, :] = (x[r, :] − mean) / sqrt(var + eps) · g + b, biased variance.

@ bk_gelu * f x i n * f out → v

Exact GELU: x · Φ(x) = 0.5x(1 + erf(x/√2)). torch's nn.GELU default is the exact form, NOT the tanh approximation — they differ in the fourth decimal, which is plenty to move a pose.

@ bk_linear * f x i rows i k * f weight * f bias i n * f out → v

out[rows, n] = x[rows, k] · weightᵀ + bias, weight laid out [n, k] (torch's nn.Linear order). bias may be a null pointer for no bias.

@ bk_softmax_rows * f x i rows i cols → v

Softmax over the last axis, max-subtracted.

@ bk_attention * f x i n i dim i heads * f qkv_w * f qkv_b

Multi-head self-attention with per-head qk LayerNorm and 2-D RoPE.

scratch must hold at least 3·n·dim + n·n doubles: q, k, v laid out [heads, n, head_dim] and then one attention matrix.

@ bk_block * f x i n i dim i heads i hidden

One full transformer block.

x ← x + ls1 · attn(norm1(x)) x ← x + ls2 · mlp(norm2(x))

x is [n, dim] and is updated in place. scratch needs 3·n·dim + n·n + n·dim + n·hidden doubles.

---

weights.nu

packages/lingbot-map/src/weights.nu — the checkpoint, as the model wants to see it.

torchpt gives an mmap and a name→(dtype, shape, bytes) table. What the model wants is "give me aggregator.frame_blocks.7.attn.qkv.weight, and fail loudly if it is not 3072×1024". This is that, and nothing more: no device buffers, no transposition, no caching — those belong with whoever is building a layer, which knows what layout it needs.

Names are checked, not assumed. A checkpoint whose layer count or hidden size differs from what the caller expects produces a message naming the tensor and both shapes, at load time, instead of a wrong answer several hundred matmuls later.

( lw_open path ) → !*Lw String ( lw_close w ) → v ( lw_has w name ) → b ( lw_index w name ) → i -1 when absent ( lw_dim w name axis ) → i ( lw_nelems w name ) → i ( lw_read w name dst n ) → b f64 into a caller buffer ( lw_require w name d0 d1 d2 d3 ) → b shape check; −1 = any ( lw_error w ) → s first failure, "" if none

lw_require accumulates: call it for every tensor a module needs, then read lw_error once. That way a mismatched checkpoint reports the first thing that is wrong rather than the first thing that is read.

API

: Lw

: Lw {
    * Pt pt
    ( Vec String ) errs
}

@ lw_open s path → !*Lw String

@ lw_close * Lw w → v

@ lw_n_tensors * Lw w → i

@ lw_index * Lw w s name → i

@ lw_has * Lw w s name → b

@ lw_ndim * Lw w s name → i

@ lw_dim * Lw w s name i axis → i

@ lw_nelems * Lw w s name → i

@ lw_error * Lw w → s

@ lw_ok * Lw w → b

@ lw_read * Lw w s name * f dst i n → b

Read a whole tensor into a caller-owned f64 buffer of at least n elements. Records a failure and returns F if the tensor is absent, the wrong size, or unreadable.

@ lw_require * Lw w s name i d0 i d1 i d2 i d3 → b

Assert a tensor's presence and shape. Pass −1 for an axis that may be anything, and for axes beyond the tensor's rank. Records the first failure; returns whether THIS check passed.

@ lw_count_indexed * Lw w s prefix s suffix → i

How many <prefix>N<suffix> tensors the checkpoint holds, counting up from 0 until one is missing — the layer count, read off the file rather than hard-coded.

---

preproc.nu

packages/lingbot-map/src/preproc.nu — frames in, model input out.

The reference's load_and_preprocess_images(mode="crop", image_size=518, patch_size=14), which is what demo.py runs:

1. decode, drop any alpha onto white, force RGB
2. resize to width = 518, height = round(h · 518/w / 14) · 14

— BICUBIC, and specifically PIL's bicubic (a = −0.5), which is a different kernel from torch's (a = −0.75). The model was trained on frames that went through PIL, so PIL is what has to be matched.

1. centre-crop the height back to 518 if the resize overshot
2. scale to [0, 1]

ImageNet mean/std normalisation is NOT done here: the aggregator applies it internally, and doing it twice is the kind of mistake that produces a plausible-looking but wrong reconstruction.

Output is one f32 plane-major (CHW) buffer per frame, C = 3.

( pp_load path size patch ) → !Frame String ( pp_free fr ) → v ( pp_width fr ) ( pp_height fr ) → i ( pp_data fr ) → f CHW, [0,1], borrowed

EXIF orientation is NOT applied — see the note at pp_load.

API

: Frame

: Frame {
    i width
    i height
    ( Vec f ) data  // CHW, 3 planes, values in [0, 1]
}

@ pp_width * Frame fr → i

@ pp_height * Frame fr → i

@ pp_data * Frame fr → *f

@ pp_free * Frame fr → v

@ pp_fit Image im i size i patch → Image

Resize + centre-crop one decoded image to the model's input geometry. Returns a NEW image; the caller frees both.

@ pp_load s path i size i patch → !*Frame String

Decode a frame file and put it in model-input form.

EXIF orientation is deliberately not applied. The reference calls ImageOps.exif_transpose, which matters for phone and Sony stills stored landscape with Orientation=6/8 — but the image package does not surface EXIF, and silently ignoring a rotation is better than silently applying the wrong one. Video-derived frames, which is what streaming reconstruction is actually fed, carry no EXIF. Applying it belongs with EXIF support in image, not with a guess here.

---

camhead.nu

packages/lingbot-map/src/camhead.nu — the camera head.

This is where a pose comes out. It reads ONE token per frame — the camera token, row 0 of the last tapped aggregator layer — and refines a 9-vector prediction over four passes:

pred ← 0 repeat 4: cond = embed_pose(pred) 9 → 2048 shift, scale, gate = Linear(SiLU(cond)) 2048 → 3·2048 x = gate · (adaLN(tokens)·(1+scale) + shift) + tokens x = trunk(x) 4 blocks pred = pred + pose_branch(trunk_norm(x)) 2048 → 1024 → 9

A DiT-style adaptive-LayerNorm conditioning loop: each pass sees its own previous answer, so the trunk is asked "what is wrong with this pose" rather than "what is the pose".

The 9-vector is [T(3) | quat(4) | fov(2)], and only the last two get an activation — ReLU, because a field of view cannot be negative. Translation and quaternion pass through linear, which is why the quaternion is never unit and why src/geom.nu divides by its own norm.

The trunk is dim 2048 with 16 heads, so head_dim is 128 and its 3-D rope splits 40/44/44 — NOT the 20/22/22 the aggregator's 64-dim heads use. Camera tokens sit at (frame, 0, 0): one token per frame, so the only position that varies is time.

( ch_load w kit ) → CamHead ( ch_free c ) → v ( ch_forward kit c ws camtok fidx out ) → b camtok: [1, 2048] f32 device — the camera token of the last layer out: [9] f32 device — the activated pose encoding

API

: i CH_DIM 2048

: i CH_HEADS 16

: i CH_HIDDEN 8192 // mlp_ratio 4

: i CH_DEPTH 4

: i CH_ITERS 4

: i CH_POSE 9

: i CH_BRANCH 1024 // pose_branch hidden = dim/2

: i CH_MAXPOS 1024

: i CH_T 40

head_dim 128, split 40/44/44 → 20 + 22 + 22 = 64 complex frequencies

: i CH_H 44

: i CH_W 44

: f CH_EPS 0.00001

token_norm / trunk_norm / the trunk's own norms are plain nn.LayerNorm (1e-5); adaln_norm is explicitly eps=1e-6 AND has no affine parameters.

: f CH_ADALN_EPS 0.000001

: CamHead

: CamHead {
    ( Vec LmBlk ) trunk
    GkBuf tokng GkBuf tokngb
    GkBuf trnkg GkBuf trnkb
    GkBuf embw GkBuf embb
    GkBuf modw GkBuf modb
    GkBuf pb1w GkBuf pb1b
    GkBuf pb2w GkBuf pb2b
    GkBuf empty  // [9], the learned "no prediction yet" pose
    GkBuf ones  // [2048] of 1.0 — adaLN has no affine, and
    GkBuf zeros  // [2048] of 0.0 — gkd_layernorm always applies one
    GkBuf cos3 GkBuf sin3
}

@ ch_free CamHead c → v

@ ch_load * Lw w * GpuKit kit → CamHead

: ChWs

: ChWs {
    GkBuf tok  // [1, 2048] normalised camera token
    GkBuf cond  // [1, 2048] embed_pose output
    GkBuf mod  // [1, 6144] shift | scale | gate
    GkBuf norm  // [1, 2048]
    GkBuf x  // [1, 2048]
    GkBuf hid  // [1, 1024]
    GkBuf pred  // [9] running prediction
    GkBuf delta  // [9]
    GkBuf fr GkBuf rw GkBuf cl
    LmWs blk
}

Scratch for one frame's four refinement passes. Everything here is [1, ...] — the camera head sees exactly one token per frame.

@ ch_ws_new * GpuKit kit → ChWs

@ ch_ws_free ChWs w → v

@ ch_forward * GpuKit kit CamHead c ChWs ws GkBuf camtok i fidx GkBuf out → b

One frame's pose. camtok is the camera token (row 0) of the last tapped aggregator layer, [1, 2048]; out receives the activated 9-vector.

---

aggregator.nu

packages/lingbot-map/src/aggregator.nu — the Geometric Context Transformer's aggregator.

Per frame: the frozen DINOv2 trunk produces patch tokens, six special tokens are prepended, and then 24 FRAME blocks and 24 GLOBAL blocks alternate — frame attention within the frame, global attention across frames through a KV cache. Four of those 24 pairs ([4, 11, 17, 23]) are tapped and their frame/global outputs concatenated, giving the heads four [P, 2048] feature maps.

The six special tokens, in order:

0 camera 1 … 4 register 5 scale

so patch_start_idx is 6 and the patches begin at row 6. Each of the three parameters ships as TWO slots: slot 0 for the anchor frames and slot 1 for every frame after — camera and register switch after frame 0, the scale token after nscale frames. That is what makes frame 0 the anchor the rest of the trajectory is expressed against.

The two attentions rotate differently — frame blocks by the 2-D patch grid, global blocks by the 3-D (frame, row, column) position. See src/rope.nu; they are not interchangeable.

( ag_load w kit ) → Agg ( ag_free a ) → v ( ag_ntokens gh gw ) → i ( ag_kv_alloc kit a maxtok ) once, before the first frame ( ag_forward_one kit a ws dtok tok img h w gh gw fidx nscale kvscale kvwindow stopat taps out ) → b out: [4, P, 2048] f32 device — the four tapped layers

API

: i AG_DIM 1024

: i AG_HEADS 16

: i AG_HIDDEN 4096

: i AG_DEPTH 24

: i AG_SPECIAL 6

: i AG_REG 4

: i AG_MAXPOS3 1024 // WanRotaryPosEmbed's max_seq_len

: f AG_EPS 0.00001

The aggregator's blocks are built WITHOUT a norm_layer argument, so they get nn.LayerNorm's default epsilon — not DINOv2's 1e-6.

: Agg

: Agg {
    Dino dino
    ( Vec LmBlk ) fb
    ( Vec LmBlk ) gb
    ( Vec f ) camtok  // [2, 1024]
    ( Vec f ) regtok  // [2, 4, 1024]
    ( Vec f ) scltok  // [2, 1024]
    GkBuf cos3
    GkBuf sin3
    // One KV cache per GLOBAL block. The `used` field in each is ignored
    // — all 24 advance together, so the count is the frame's, passed in
    // per call rather than stored 24 times and kept in step by hand.
    ( Vec LmKv ) kv
}

@ ag_ntokens i gh i gw → i

@ ag_tap * i taps i i0 → i

Which of the four tapped layers, if any, block group i0 is. taps is a 4-element array; the model uses [4, 11, 17, 23], and a test can point it at early groups to see how far a divergence has travelled by then.

@ ag_default_taps * i out → v

The model's own tap points.

@ ag_free Agg a → v

@ ag_load * Lw w * GpuKit kit → Agg

: i AG_KV_SCALE 8

The reference's KV-cache eviction policy, as sizes.

It keeps the first AG_KV_SCALE frames forever, keeps the last AG_KV_WINDOW frames INCLUDING the current one, and from every frame it drops it preserves the six special-token rows.

"Including the current one" is the whole subtlety, and it is not visible in the eviction function itself. attention.py:239-244 concatenates the frame into the cache and only THEN calls applykv_cache_eviction_causal, so num_cached already counts it: the trigger is f >= AG_KV_SCALE + AG_KV_WINDOW and the live set is scale frames plus f-AG_KV_WINDOW+1 .. f. Reading the function alone gives a window one frame too wide that starts one frame too late, which is wrong from the very first eviction and by 2.7e-2.

: i AG_KV_WINDOW 64

@ ag_kv_rows i nframes i p i kvscale i kvwindow → i

Rows a cache needs for a sequence of nframes frames of p tokens. Short sequences never evict and just need every row; long ones need the two full-frame regions plus six rows per evicted frame.

@ ag_kv_woff i fidx i p i kvscale i kvwindow → i

Where frame fidx writes its rows, and how many rows are live once it has. Before the ring wraps these are the obvious ones — append at the end, everything so far is live. After it wraps the frame overwrites the oldest ring slot, and the live region is the two full-frame regions plus the preserved specials that trail them.

@ ag_kv_evicted i fidx i kvscale i kvwindow → i

How many frames have been evicted once frame fidx is in place.

@ ag_kv_nvalid i fidx i p i kvscale i kvwindow → i

@ ag_kv_alloc * GpuKit kit Agg a i nframes i p i kvscale i kvwindow → v

Allocate the 24 global-block caches for a sequence of nframes frames of p tokens. Call once, before the first frame.

@ ag_kv_evict * GpuKit kit Agg a i fidx i p i kvscale i kvwindow → b

Preserve the special-token rows of the frame that frame fidx is about to overwrite, moving them past the ring where they stay live. Runs over all 24 caches before any of them is written, because the frame passes through them one after another and the rows have to be out of the way before the first write, not the last.

@ ag_special_rows Agg a i fidx i nscale → ( Vec f )

The six special-token rows for frame fidx, as a host vector.

@ ag_pos3 i gh i gw i fidx * i fr * i rw * i cl → v

Grid coordinates for the 3-D rope: special token j at (f, j, j), patch (py, px) at (f, 6+py, 6+px) — the specials run down a diagonal the patch grid never reaches.

@ ag_pos2 i gh i gw * i rows * i cols → v

Grid coordinates for the 2-D rope: specials at 0, patches offset by 1.

@ ag_setup_rope2 * GpuKit kit LmWs ws i hd i maxpos → b

Fill the workspace's 2-D rope tables. lm_ws_new only ALLOCATES them — leaving them to the caller was a bug worth the comment: uninitialised device memory made the very first block produce NaN, and a NaN 72 blocks deep says nothing about where it came from.

@ ag_forward_one * GpuKit kit Agg a LmWs ws GkBuf dtok GkBuf tok

One frame through the whole aggregator.

out is [4, P, 2048]: for each tapped layer, the frame block's output in columns 0..1023 and the global block's in 1024..2047 — the order the reference concatenates them in.

Where this frame's rows go in the caches, and how much of them is live, follows from fidx alone — see ag_kv_woff / ag_kv_nvalid.

---

load.nu

packages/lingbot-map/src/load.nu — checkpoint → device buffers.

src/weights.nu finds a tensor and checks its shape; this puts it on the device. Nothing is transposed: torch stores a Linear weight as [out, in] and gkd_gemm reads it with transb=1, so the bytes go across unchanged.

f32 the whole way. The checkpoint is f32, the device buffers are f32, and the only f64 in between is the host staging vector — one tensor at a time, so peak extra memory is the largest single weight (an MLP's 4096x1024, 32 MB) rather than the model.

( lmw_upload w kit name ) → GkBuf empty on failure ( lmw_block w kit prefix qk eps dim hidden ) → LmBlk ( lmw_ok w ) → b nothing failed yet

A missing tensor is recorded in the Lw error list rather than thrown, so a whole model's worth of loading reports the first thing wrong instead of the first thing touched.

API

@ lmw_upload * Lw w * GpuKit kit s name → GkBuf

One tensor onto the device. The staging vector is written through its own data pointer — pt_read_f64 fills a raw span — so the elements are copied once, not twice.

@ lmw_upload_t * Lw w * GpuKit kit s name i rows i cols → GkBuf

Upload a [rows, cols] tensor TRANSPOSED, as [cols, rows].

A torch Linear weight is [out, in], which gkd_gemm reads with transb=1. On the CPU backend that is the slow path: the register-tiled kernel needs B as [K, N] and only runs for transb=0. Transposing at CALL time to reach it is a measured LOSS — a full memory round trip over a cold weight used once. Transposing HERE costs one pass, once per process, and every frame afterwards takes the tiled path.

@ lmw_block * Lw w * GpuKit kit s prefix b qk f eps i dim i hidden → LmBlk

Every parameter of one transformer block. qk selects whether the attention has q_norm/k_norm — DINOv2's own 24 blocks do not, the aggregator's 48 do — and eps is norm1/norm2's, which is 1e-6 for DINOv2's blocks and 1e-5 for the aggregator's. The four Linear weights are uploaded TRANSPOSED, [in, out] rather than the checkpoint's [out, in], so lm_block_forward can call gkd_gemm with transb=0 and get the register-tiled CPU kernel. dim and hidden are the block's widths; qkv is 3*dim wide.

@ lmw_prefix s base i idx → String

<base>.<idx>. — the prefix of an indexed block.

---

devblock.nu

packages/lingbot-map/src/devblock.nu — the transformer block on the device, in f32.

src/block.nu is the same block in host f64 and is verified against the reference to 4.4e-15; this is the one that actually runs the model, and the host version is what it is checked against. Keeping both is the point: a device pipeline of ~15 kernel launches per block has a lot of places for a stride to be wrong, and "compare against a slow version that is known correct" catches all of them at once.

Everything is composed from gpukit's gkd_* ops — gemm, layernorm, bmm, softmax, permute, broadcast-elementwise — so the same code runs on CUDA and on the CPU backend. The one kernel that had to be written here is 2-D RoPE, which is specific to how this model indexes a patch grid.

Weight layouts are torch's, unchanged: a Linear weight is [out, in] and goes into gkd_gemm with transb=0: the four Linear weights are uploaded TRANSPOSED (see lmw_block), which is what lets the CPU backend take its register-tiled kernel.

( lm_ws_new kit n dim heads hidden maxkv maxpos ) → LmWs ( lm_ws_free ws ) → v ( lm_blk_free w ) → v ( lm_block_forward kit w ws rp x n dim heads hidden ) → b x updated

rp says which rotation the attention applies. The three blocks in this model each want a different one, so it is a parameter rather than a flag: DINOv2's own blocks rotate nothing (they add a position embedding at the input), the aggregator's FRAME blocks use the 2-D grid rope, and its GLOBAL blocks use the 3-D (frame, row, column) rope. See src/rope.nu for why those two are not interchangeable.

API

: LmBlk

: LmBlk {
    GkBuf n1g GkBuf n1b
    GkBuf qkvw GkBuf qkvb
    GkBuf qng GkBuf qnb
    GkBuf kng GkBuf knb
    GkBuf pw GkBuf pb
    GkBuf ls1
    GkBuf n2g GkBuf n2b
    GkBuf f1w GkBuf f1b
    GkBuf f2w GkBuf f2b
    GkBuf ls2
    // norm1 / norm2 epsilon. NOT a constant across this model: DINOv2's
    // own blocks are built with partial(LayerNorm, eps=1e-6), while the
    // aggregator's frame and global blocks are built without a
    // norm_layer at all and so get nn.LayerNorm's default 1e-5. The
    // difference is invisible until a row's variance is small, and then
    // it is worth ~1e-3 in the output.
    f eps
}

One block's parameters, resident on the device. A norm gain/bias pair with dptr 0 means "absent": DINOv2's own blocks have no qk-norm, the aggregator's frame and global blocks do.

@ lm_blk_free LmBlk w → v

: i LM_ROPE_NONE 0

Which rotation an attention applies, and the tables it needs.

LM_ROPE_NONE nothing LM_ROPE_2D pa = row, pb = column LM_ROPE_3D pa = frame, pb = row, pc = column

: i LM_ROPE_2D 1

: i LM_ROPE_3D 2

: LmRope

: LmRope {
    i mode
    GkBuf pa
    GkBuf pb
    GkBuf pc
    GkBuf cosb
    GkBuf sinb
}

@ lm_rope_none → LmRope

: LmWs

: LmWs {
    GkBuf norm
    GkBuf qkv
    GkBuf qkvp
    GkBuf kt
    GkBuf kpack
    GkBuf vpack
    GkBuf att
    GkBuf ctx
    GkBuf ctxp
    GkBuf branch
    GkBuf hid
    GkBuf scal
    GkBuf rows
    GkBuf cols
    GkBuf cosb
    GkBuf sinb
    i maxkv
}

Scratch, sized once for the largest frame the model will see. maxkv is the widest key/value run attention will face — the token count for self-attention, the whole cache for the streaming global blocks.

@ lm_ws_new * GpuKit kit i n i dim i heads i hidden i maxkv i maxpos → LmWs

@ lm_ws_free LmWs ws → v

@ lm_rope2d * GpuKit kit GkBuf x GkBuf rows GkBuf cols GkBuf cosb GkBuf sinb

── 2-D RoPE, on device ─────────────────────────────────────────────

The one kernel this file adds. x is [heads, n, dim] and is rotated in place: the first half of each head's feature vector by the token's ROW coordinate, the second by its column. One thread per (head, token, axis, frequency) — each owns a disjoint pair of elements, so the in-place read-then-write is safe without a barrier.

@ lm_rope3d * GpuKit kit GkBuf x GkBuf fr GkBuf rw GkBuf cl GkBuf cosb GkBuf sinb

3-D RoPE on device. Unlike the 2-D one this rotates INTERLEAVED pairs — (x0,x1), (x2,x3), … — and the 64-dim head is split 20 / 22 / 22 across the frame, row and column axes rather than in half. One thread per (head, token, frequency); each owns one pair.

@ lm_rope_apply * GpuKit kit LmRope rp GkBuf x i heads i n i hd → b

Apply whichever rotation rp selects to a [heads, n, hd] tensor.

@ _lm_i2 i a i b → ( Vec i )

@ _lm_i3 i a i b i c → ( Vec i )

@ _lm_i4 i a i b i c i d → ( Vec i )

@ lm_view GkBuf b i off i len → GkBuf

A view of b's elements [off, off+len) as its own GkBuf. The device pointer is byte-addressed, so a sub-range is just an offset — no copy, and nothing to free (the parent owns the allocation).

: LmKv

: LmKv {
    GkBuf k
    GkBuf v
    i maxkv
    // Where THIS frame's rows go, and how many rows of the cache are
    // live once they are there. They used to be one number, because a
    // frame's rows always went on the end. With sliding-window eviction
    // a frame can be written into the middle of a ring while the live
    // region also covers the preserved special tokens past it, so the
    // write offset and the live count are genuinely different things.
    i woff
    i nvalid
}

A KV cache for one global block: k and v as [heads, maxkv, hd]. A cache whose buffers are absent means plain self-attention — the frame blocks, and the very first frame.

@ lm_kv_none → LmKv

@ lm_kv_new * GpuKit kit i heads i maxkv i hd → LmKv

@ lm_kv_free LmKv c → v

@ lm_block_forward * GpuKit kit LmBlk w LmWs ws LmRope rp LmKv kv GkBuf x

Transformer block over n tokens; x is [n, dim], updated in place.

With a cache, this frame's keys and values are written at row kv.woff and attention runs over the first kv.nvalid rows — the streaming global blocks. Without one it is plain self-attention. PLACEMENT IS THE CALLER'S BOOKKEEPING: both numbers are read here and not written, because one frame passes through 24 different caches and where it goes belongs to the frame, not to any one of them.

---

dpthead.nu

packages/lingbot-map/src/dpthead.nu — the DPT depth head.

This is where a depth map comes out. It takes the aggregator's four tapped layers — [P, 2048] each, patch tokens only — reads them back as 2-D feature maps at four different resolutions, and fuses them coarse to fine into a full-resolution depth and confidence pair.

For a 37x21 patch grid (a 518x294 frame) the four scales are:

tap 0 → project 2048→256 → ConvTranspose 4x4 s4 → 84x148 tap 1 → project 2048→512 → ConvTranspose 2x2 s2 → 42x74 tap 2 → project 2048→1024 → identity → 21x37 tap 3 → project 2048→1024 → Conv 3x3 s2 p1 → 11x19

The four projections do NOT share a channel count and the four resize layers are four different operations — that asymmetry is the whole point of a DPT, and it is the first thing a port gets wrong.

Then layerN_rn brings all four to 256 channels (3x3, NO bias), and four fusion blocks walk back up:

out = refine4(l4) → size of l3 out = refine3(out, l3) → size of l2 out = refine2(out, l2) → size of l1 out = refine1(out, l1) → x2 out = output_conv1(out) → 128 channels out = bilinear to H x W out = out + pos_embed out = output_conv2(out) → 2 channels

refinenet4 has no resConfUnit1 — it has nothing to fuse with.

Output activations: depth = exp(channel 0), confidence = 1 + exp( channel 1). The depth is exponentiated, so the network predicts log depth and the result is positive by construction.

( dp_load w kit ) → Dpt ( dp_free d ) → v ( dp_forward kit d taps gh gw h w trace depth conf ) → b taps: [4, P, 2048] f32 device — the aggregator's output depth: [hw] f32 device conf: [hw] f32 device

API

: i DP_IN 2048

: i DP_FEAT 256

: i DP_SPECIAL 6 // patch tokens start after the six special tokens

: i DP_PATCH 14

: f DP_POS_RATIO 0.1

: f DP_OMEGA0 100.0

: f DP_EPS 0.00001

norm is a plain nn.LayerNorm over the 2048-wide tokens.

: DpConv { GkBuf w GkBuf b i hasb }

A conv with an optional bias, as a pair of device buffers.

: DpRcu { DpConv c1 DpConv c2 }

One residual conv unit: relu → 3x3 → relu → 3x3, plus the input.

: DpFuse { DpRcu u1 DpRcu u2 DpConv outc i has1 }

A fusion block. has1 is false only for refinenet4, which has nothing to fuse with and therefore no first unit.

: Dpt

: Dpt {
    GkBuf normg GkBuf normb
    ( Vec DpConv ) projects  // 4, channel counts differ
    ( Vec DpConv ) resizes  // index 0,1 transpose; 2 unused; 3 strided
    ( Vec DpConv ) rns  // 4, 3x3 no bias, → 256
    ( Vec DpFuse ) fuse  // refinenet1..4 in that order
    DpConv oc1
    DpConv oc2a
    DpConv oc2b
}

@ dp_load * Lw lw * GpuKit kit → Dpt

@ dp_free Dpt d → v

@ _dp_rcu_fwd * GpuKit kit DpRcu r GkBuf x GkBuf t1 GkBuf t2 i ch i h i w → b

relu → conv3x3 → relu → conv3x3, added back to the RELU'd input. Runs in place on x; t1 and t2 are scratch of the same size.

The reference builds this unit with nn.ReLU(inplace=True), so its first activation overwrites the tensor it was handed and the residual added at the end is relu(x), not x. That is not a detail: the input here averages −473 and peaks at 974, and clipping it is what takes the block's output down to a peak of 0.14. Adding back the original x leaves the whole negative bulk in place and the depth map is wrong by three orders of magnitude, with nothing failing along the way.

The caller's buffer is left relu'd, exactly as the reference leaves its own. Both the port and the reference are done with it by then.

@ dp_pos_embed * GpuKit kit GkBuf x i ch i h i w f aspect → b

The sinusoidal UV position embedding the head adds after each project and again after the final upsample.

The grid spans a unit-diagonal rectangle of the frame's aspect ratio, so the embedding is the same physical field regardless of resolution — which is why it can be added at two different sizes and mean the same thing.

@ _dp_fuse_fwd * GpuKit kit DpFuse f GkBuf out GkBuf skip GkBuf up

One fusion step. out is the coarser path (or the only input, for refinenet4) at h x w; skip is the same-resolution feature to fuse in; the result lands in dst at oh x ow. All buffers are 256-channel.

dst is separate from out because the block CHANGES resolution — the upsample happens between resConfUnit2 and out_conv, so the 1x1 cannot write back over its own input. ch is 256 for every block the real head builds, but taking it as an argument rather than reading DP_FEAT lets a unit test run this at a size a human can read.

: i DP_STRIDE 9973

Print one stage in tests/agg_oracle.py's dump format. Only called when trace is on — the head is ~150 s per frame, so bisecting it by re-running with one more print each time is not an option.

@ dp_forward * GpuKit kit Dpt d GkBuf taps i gh i gw i h i w i trace

The whole head, for one frame.

taps is the aggregator's [4, P, 2048]; depth and conf receive h*w values each. Every intermediate is allocated here rather than in a workspace struct: the shapes depend on the frame's patch grid, this runs once per frame against ~100 s of transformer, and a wrong hand-computed scratch size fails closed in a way that is tedious to chase.

---

interp.nu

packages/lingbot-map/src/interp.nu — torch-compatible bicubic resampling of a float feature grid.

DINOv2 ships one position embedding for a 37x37 patch grid, and a frame of 518x294 is 37x21 patches — so every forward pass begins by resampling pos_embed to the frame's grid. The reference does it with

F.interpolate(..., mode="bicubic", antialias=True, size=(h0, w0))

and that is NOT the same resampler as the one preprocessing uses:

kernel a arithmetic PIL (preprocessing) −0.5 8-bit, fixed point, clamped torch antialias=True −0.5 float, unclamped ← this file torch antialias=False −0.75 float, border-replicated

The two −0.5 rows are the same kernel by design: torch's antialias path was written to reproduce PIL. The −0.75 row is torch's ORDINARY bicubic, which is what every reference to "torch bicubic" means and which must NOT be used here. Getting that wrong is a silent few-percent error everywhere, not a crash.

This is torch's _upsample_bicubic2d_aa for the size-given, align_corners=false case: scale = in/out per axis, support = 2·scale when shrinking and 2 when growing, coefficients normalised over the clipped window, two separable passes.

( interp_bicubic_aa src sw sh planes dw dh dst ) → v src, dst: planar f64, planes planes of swsh / dwdh

API

@ interp_bicubic_aa * f src i sw i sh i planes i dw i dh * f dst → v

Resample planes planar grids from sw×sh to dw×dh.

Separable, horizontal first, exactly as torch does it — resampling both axes at once would give different sums.

---

rope.nu

packages/lingbot-map/src/rope.nu — 2-D rotary position embedding.

The aggregator's attention rotates q and k by the token's position in the patch GRID, not its index in the sequence: the head dimension is split in half, the first half rotated by the row coordinate and the second by the column. That is what makes a token's relationship to its neighbour above the same as to its neighbour to the left.

Following the reference (RotaryPositionEmbedding2D, frequency 100):

half = D / 2 features per spatial axis inv_f = 100 ^ −(2i / half) i = 0 … half/2 − 1 angles = pos · inv_f, concatenated with itself → half values out = x · cos(angles) + rotate(x) · sin(angles) rotate = (x₁, x₂) ↦ (−x₂, x₁) split at half/2

Two details that are easy to get wrong and silent when wrong:

• the rotation splits each HALF at half/2, not the full D at D/2;
• positions carries (row, col) in that order, and row drives the

first half. Swapping them transposes the model's idea of the image without changing any shape.

Special tokens (camera / register / scale) sit at position 0 and the patches start at 1 — see the caller; this module just applies what it is given.

( rope2d_tables half maxpos cos sin ) → v tables, [maxpos, half] ( rope2d_apply x heads n dim rows cols cos sin ) → v in place

API

: f ROPE_FREQ 100.0

@ rope2d_tables i half i maxpos * f cos_t * f sin_t → v

cos/sin tables for one spatial axis: [maxpos, half] each. half is the per-axis feature count (D/2) and must be even.

maxpos only has to exceed the largest COORDINATE, not the token count — entry p depends on p alone. The reference sizes its table by the sequence length, which is simply a generous upper bound (783 tokens for a 37x21 grid whose largest coordinate is 37).

@ rope2d_apply * f x i heads i n i dim * i rows * i cols * f cos_t * f sin_t → v

Rotate x in place. x is [heads, n, dim] contiguous; rows and cols are the per-token grid coordinates (length n); the tables come from rope2d_tables with half = dim/2.

: f ROPE3_THETA 10000.0

: i R3_T 20 // head-dim slice for the frame axis

: i R3_H 22 // … for the row axis

: i R3_W 22 // … for the column axis

@ rope3d_nt → i

Half-widths, i.e. how many complex frequencies each axis contributes.

@ rope3d_nh → i

@ rope3d_nw → i

@ rope3d_width → i

@ rope3d_tables_fhw i td i hd i wd i maxpos * f cos_t * f sin_t → v

[maxpos, (t+h+w)/2] cos and sin, laid out t | h | w.

The split is a parameter because this model uses TWO of them: the aggregator's global blocks have head_dim 64 and split 20/22/22, the camera head's trunk has head_dim 128 and splits 40/44/44. Same kernel, same theta, different geometry.

@ rope3d_tables i maxpos * f cos_t * f sin_t → v

The aggregator's split: 20/22/22 over a 64-dim head.

@ rope3d_apply_fhw * f x i heads i n i dim i nt i nh * i fr * i rw * i cl

Rotate x [heads, n, dim] in place. fr / rw / cl hold each token's (frame, row, column) position. dim must be 2·width (64).

@ rope3d_apply * f x i heads i n i dim * i fr * i rw * i cl * f cos_t * f sin_t → v

---

main.nu

lingbot-map — streaming 3-D reconstruction from a sequence of frames, in pure NURL.

lingbot-map --model <checkpoint.pt> [options] <frame.png> ...

Each frame is preprocessed, run through the DINOv2 trunk and the streaming aggregator (which keeps a KV cache across frames), and then through two heads: the camera head, which gives a pose, and the DPT head, which gives a depth map and a per-pixel confidence. Depth plus pose plus intrinsics unprojects to world-space points, which is what the point cloud is.

Output is an ASCII PLY, which every viewer reads. The vertex count is not known until the last frame is done, so the header is written with a fixed-width placeholder and patched at the end rather than holding the whole cloud in memory.

API

: i LM_SIZE 518

: i LM_PATCH 14

: f LM_CONF 2.0

The reference's own default for what to draw: confidence is 1+exp(x), so 1.0 is "no information" and anything above ~2 is a real surface.

: i LM_COUNT_WIDTH 12

: Opts

: Opts {
    s model
    s out
    f conf
    i pixstride
    i maxframes
    i verbose
    ( Vec String ) frames
    i bad
}

: Ply { File f i n }

@ main → i

---

patchembed.nu

packages/lingbot-map/src/patchembed.nu — the model's entry point.

DINOv2's patch embedding is Conv2d(3, 1024, kernel=14, stride=14). Kernel and stride are equal, so the patches do not overlap and the convolution is exactly a matmul: lay each 14x14x3 patch out as one row of 588 values and multiply by the reshaped weight.

img [3, H, W] ──im2col──▶ P x 588 ──▶ P x 1024 (+ bias)

with P = (H/14) x (W/14) patches in ROW-MAJOR order, and 588 = 3 x 14 x 14 laid out channel-major then row then column — the order torch's Conv2d weight already has, so the weight is used as-is with no transposition beyond the one the matmul wants.

im2col is written out rather than fused into the multiply because the multiply is the part that belongs on a device kernel later, and the layout is the part that has to be right first.

( pe_patches h w patch ) → i P ( pe_im2col img c h w patch cols ) → v P x (c·patch²) ( pe_project cols p k weight bias out ) → v P x k · k x n → P x n

API

@ pe_grid_h i h i patch → i

@ pe_grid_w i w i patch → i

@ pe_patches i h i w i patch → i

@ pe_im2col * f img i c i h i w i patch * f cols → v

Planar image [c, h, w] → one row per patch, c·patch·patch wide. Row order is row-major over the patch grid; within a row the order is channel, then kernel row, then kernel column — Conv2d's weight layout.

@ pe_project * f cols i p i k * f weight * f bias i n * f out → v

out[P, n] = cols[P, k] · weightᵀ[k, n] + bias[n].

weight is torch's Conv2d weight flattened, [n, k] — output channel major — so this reads it along a row per output channel, which is a dot product against the patch row. Swapped for a device GEMM once the weights live on the device; the loop is the reference the GEMM has to agree with.