dino.nu

packages/map-anything/src/dino.nu — the DINOv2 ViT-giant/14 image encoder, the first 24 of the hub model's 40 blocks.

MapAnything constructs dinov2_vitg14 through torch.hub, keeps the first 24 blocks (keep_first_n_layers), REPLACES the final norm with Identity (norm_returned_features=False), and deletes the mask token; there are no register tokens. So the token order is simply

[cls] + patches ← pos_embed (cls row + resampled grid) added to ALL tokens

and the output is the raw block-24 features: patch tokens for the dense path, the cls token as the per-view "register" the aggregator carries alongside them.

The position embedding ships for a 37×37 grid; a 518×392 frame is 37×28, so it is resampled — with torch's PLAIN bicubic (a = −0.75, no antialias) and DINOv2's scale_factor = (out+0.1)/37 kludge, which is what src/interp.nu implements. A square 518×518 frame skips the resample entirely (the reference early-outs when npatch == N and w == h), which the ==37 fast path here reproduces.

( dn_load w kit ) → Dino ( dn_free d ) → v ( dn_tokens gh gw ) → i 1 + gh·gw ( dn_forward kit d ws img h w gh gw tok ) → b img: [3, H, W] f32 HOST planar, ALREADY ImageNet-normalised tok: [1 + gh·gw, 1536] f32 device — cls row first, then patches

API

: i DN_DIM 1536

: i DN_HEADS 24

: i DN_SWH 4096 // SwiGLU hidden: w12 emits 8192, w3 consumes 4096

: i DN_DEPTH 24

: i DN_PATCH 14

: i DN_GRID 37 // the grid pos_embed ships for: 37×37 + 1 cls = 1370

: f DN_EPS 0.000001

: Dino

: Dino {
    ( Vec MaBlk ) blocks
    GkBuf proj_w  // [1536, 3·14·14] — Conv2d weight, flattened
    GkBuf proj_b
    // cls and the position grid stay on the HOST: they are combined per
    // frame and the grid is resampled by a host routine anyway.
    ( Vec f ) cls
    ( Vec f ) pos  // [1370, 1536]
    // The resampled grid for the geometry this run uses, cached — every
    // view of a set asks for the same one. `Vec` is a handle to one
    // control block, so a Dino passed by value shares this.
    ( Vec i ) poskey
    ( Vec f ) poscache
}

@ dn_free Dino d → v

@ dn_load * Lw w * GpuKit kit → Dino

@ _dn_host * Lw w s name → ( Vec f )

A tensor read into a host vector, sized from the checkpoint.

@ dn_tokens i gh i gw → i

@ dn_pos_cached Dino d i gh i gw → ( Vec f )

pos_embed resampled to this frame's grid, as a host vector laid out [1 + gh·gw, 1536]: the cls row unchanged, then the grid. Cached; the returned Vec is BORROWED — it is the Dino's, do not free it.

@ dn_pos_for Dino d i gh i gw → ( Vec f )

@ dn_forward * GpuKit kit Dino d MaWs ws * f img i h i w i gh i gw GkBuf tok → b

Run the frozen trunk over one already-normalised frame.

img is [3, H, W] on the HOST; im2col and the cls/pos assembly happen there (pure addressing over a couple of megabytes), everything after the projection is on the device.

tok must hold dn_tokens(gh, gw) × 1536 f32 and comes back holding the full token array: row 0 is the cls token (the aggregator's per-view register), rows 1.. are the patch tokens. NO final norm — the model replaces it with Identity.

---

geom.nu

packages/map-anything/src/geom.nu — output geometry and masks, host f64.

The model's scene representation is raydirs+depth+pose: per pixel a unit ray direction and a depth along it, per view a cam2world pose as translation + unit quaternion in (x, y, z, w) — SCALAR-LAST, the reference's own convention — and one global metric scale. World points are

p_world = scale · (R(q) · (dir · depth) + t)

(OpenCV axes: +x right, +y down, +z forward; view 0 is the world frame, so its pose is near-identity but still predicted.)

The masks reproduce inference.py's postprocessing exactly:

• non-ambiguous: sigmoid(logit) > 0.5, i.e. logit > 0
• optional confidence percentile (the reference's

apply_confidence_mask, default off)

• edge mask: a pixel is dropped when BOTH the depth-edge test

(3×3 max-pool difference, relative tol 0.03 on z-depth) AND the normals-edge test (3×3 angle spread over 5°, computed from the world points, then max-pooled) flag it — flying pixels at depth discontinuities, killed the way MoGe kills them.

( gm_quat_to_mat qx qy qz qw R ) → v R: f, 9 ( gm_world_points dirs depth pose scale n out ) → v ( gm_nonambig logits n mask ) → v mask: u8 0/1 ( gm_conf_percentile conf n pct mask ) → v AND into mask ( gm_edge_mask pts depthz h w mask ) → v AND into mask

API

@ gm_quat_to_mat f x f y f z f w * f r → v

(x, y, z, w) → row-major 3×3, the reference's quaternion_to_rotation_matrix.

@ gm_world_points * f dirs * f depth * f pose f scale i n * f out * f depthz → v

World points for one view. dirs is [3, n] planar (the DPT head's layout), depth is [n], pose is the head's 7 floats [tx ty tz qx qy qz qw], out is [n, 3] interleaved — the layout the PLY writer wants. Also emits the z-depth (camera-frame z · scale) into depthz, which the edge mask needs.

@ gm_nonambig * f logits i n * u mask → v

sigmoid(x) > 0.5 ⇔ x > 0

@ gm_conf_percentile * f conf i n f pct * u mask → v

AND a "confidence above the pct-th percentile" test into mask. torch.quantile with linear interpolation, over ALL pixels (the reference quantiles the un-masked confidence map).

: f __GM_INF 1000000000000000000000000000000.0 // inf sentinel; real depths are ~1e0..1e4

@ gm_edge_mask * f pts * f depthz i h i w * u mask → v

The reference's edge mask: drop pixels where the depth-edge AND the normals-edge tests both fire. pts is [n, 3] world points (already masked-invalid entries are excluded via mask), depthz [n], and mask is updated in place.

@ gm_sim3_fit * f xs * f ys i n * f out → b

Fit local→global: xs, ys are [n, 3] interleaved. Fails (F) below 3 pairs or on a degenerate spread.

@ gm_sim3_apply * f pts i n * f xf → v

p ← s·R·p + t, in place over [n, 3] interleaved points.

---

heads.nu

packages/map-anything/src/heads.nu — the pose head and the metric scale head.

Both run on the info_sharing FINAL features: the pose head on one view's patch tokens, the scale head on the global scale token. Every layer in both is pointwise over tokens (1×1 convs and Linears), so on the device they are plain GEMMs over token-major features — no [C, H, W] layout ever needs to exist.

Pose head (Reloc3r/MaRePo shape): proj 1536→784 → 2× ResConvBlock (x = relu(c1(res)); x = relu(c2(x)); x = relu(c3(x)); res = res + x — the skip is Identity at equal widths) → mean over tokens → Linear 784→784 + ReLU, twice → fc_t → 3 and fc_rot → 4, output [t | quat], quat normalised by ‖q‖ clipped to 1e-8. The translation is metric-scaled by the caller.

Scale head: Linear 1536→196 → 2× (Linear 196→196 + ReLU) → Linear 196→1 → exp, clipped to ≥ 1e-8.

( ph_load w kit ) → PoseH ( ph_free p ) → v ( ph_forward kit p fin voff np out ) → b out: host *f, 7 values ( sh_load w kit ) → ScaleH ( sh_free s ) → v ( sh_forward kit s fin row ) → f the metric scale, ≤0 on error

API

: i PH_DIM 1536

: i PH_HID 784 // 4 · 14²

: PhLin { GkBuf w GkBuf b }

: PoseH

: PoseH {
    PhLin proj
    ( Vec PhLin ) res  // 2 blocks × [c1, c2, c3]
    PhLin mlp0
    PhLin mlp2
    PhLin fct
    PhLin fcr
}

@ ph_load * Lw lw * GpuKit kit → PoseH

@ ph_free PoseH p → v

@ ph_forward * GpuKit kit PoseH p GkBuf fin i voff i np * f out → b

One view's pose from the final features. out receives 7 host floats: [tx ty tz | qw? qx? ...] — exactly fc_t then fc_rot, the quaternion normalised; which convention the four are in is the GEOMETRY's business (src/geom.nu), not this head's.

: i SH_HID 196

: ScaleH

: ScaleH {
    PhLin proj
    PhLin m0
    PhLin m1
    PhLin outp
}

@ sh_load * Lw lw * GpuKit kit → ScaleH

@ sh_free ScaleH s → v

@ sh_forward * GpuKit kit ScaleH s GkBuf fin i row → f

The metric scale from the final scale-token feature (row row of the final sequence). Returns exp(x) clipped to ≥ 1e-8, or -1 on error.

---

weights.nu

packages/map-anything/src/weights.nu — the checkpoint, as the model wants to see it.

Same surface as lingbot-map's weights.nu, but the container is safetensors (facebook/map-anything-apache ships a single model.safetensors, all F32), so safetensor provides the mmap and the name→(dtype, shape, bytes) table. What the model wants is "give me info_sharing.model.layers.7.attn.qkv.weight, and fail loudly if it is not 4608×1536". This is that, and nothing more: no device buffers, no transposition, no caching — those belong with whoever is building a layer, which knows what layout it needs.

Names are checked, not assumed. A checkpoint whose layer count or hidden size differs from what the caller expects produces a message naming the tensor and both shapes, at load time, instead of a wrong answer several hundred matmuls later.

( lw_open path ) → !Lw String ( lw_close w ) → v ( lw_has w name ) → b ( lw_index w name ) → i -1 when absent ( lw_dim w name axis ) → i ( lw_nelems w name ) → i ( lw_read w name dst n ) → b f64 into a caller buffer ( lw_f32_ptr w name n ) → u zero-copy when already f32 ( lw_require w name d0 d1 d2 d3 ) → b shape check; −1 = any ( lw_error w ) → s first failure, "" if none

lw_require accumulates: call it for every tensor a module needs, then read lw_error once. That way a mismatched checkpoint reports the first thing that is wrong rather than the first thing that is read.

API

: Lw

: Lw {
    * St st
    ( Vec String ) errs
}

@ lw_open s path → !*Lw String

@ lw_close * Lw w → v

@ lw_n_tensors * Lw w → i

@ lw_index * Lw w s name → i

@ lw_has * Lw w s name → b

@ lw_ndim * Lw w s name → i

@ lw_dim * Lw w s name i axis → i

@ lw_nelems * Lw w s name → i

@ lw_error * Lw w → s

@ lw_ok * Lw w → b

@ lw_f32_ptr * Lw w s name i n → *u

The tensor's own bytes inside the mapping, when they are already contiguous float32 — which is the layout a GK_F32 device buffer wants, so it can be uploaded with no conversion at all. safetensors tensors are contiguous by construction, so only the dtype and length are checked. Returns 0 when the tensor is absent, a different dtype or a different length, and the caller falls back to the converting read.

@ lw_read * Lw w s name * f dst i n → b

Read a whole tensor into a caller-owned f64 buffer of at least n elements. Records a failure and returns F if the tensor is absent, the wrong size, or unreadable. Every dtype widens through f32 (the container's dequant path), which is exact for this checkpoint — the file is F32 throughout.

@ lw_require * Lw w s name i d0 i d1 i d2 i d3 → b

Assert a tensor's presence and shape. Pass −1 for an axis that may be anything, and for axes beyond the tensor's rank. Records the first failure; returns whether THIS check passed.

@ lw_count_indexed * Lw w s prefix s suffix → i

How many <prefix>N<suffix> tensors the checkpoint holds, counting up from 0 until one is missing — the layer count, read off the file rather than hard-coded.

---

preproc.nu

packages/map-anything/src/preproc.nu — images in, model input out.

The reference's load_images(resize_mode="fixed_mapping", resolution_set=518, patch_size=14, norm_type="dinov2"), which is what demo_images_only_inference.py runs:

1. decode every image, force RGB (PIL .convert("RGB") — alpha is

DROPPED, not composited; that is what the reference does)

1. average the images' aspect ratios W/H and pick ONE shared target

size for the whole set from a fixed table of ten 14-divisible resolutions (518×518 … 518×168 and the portrait transposes)

1. per image: scale = max(tw/W, th/H) + 1e-8, resize to

(floor(W·scale), floor(H·scale)) — LANCZOS when shrinking, BICUBIC when growing, and specifically PIL's kernels, which the image package matches byte for byte

1. centre-crop to exactly (tw, th)
2. scale to [0, 1]

ImageNet mean/std normalisation is NOT done here — it is applied at the encoder input, and the un-normalised [0,1] pixels are also what the point cloud's colours come from.

Output is one f32 plane-major (CHW) buffer per image, C = 3.

( pp_open path ) → !Image String decoded, forced RGB ( pp_pick_target avg_aspect ) → i packed (w<<16)|h ( pp_fit im tw th ) → !Frame String ( pp_free fr ) → v ( pp_width fr ) ( pp_height fr ) → i ( pp_data fr ) → f CHW, [0,1], borrowed

EXIF orientation is NOT applied — the reference calls exif_transpose, but the image package does not surface EXIF, and silently ignoring a rotation is better than silently applying the wrong one. Applying it belongs with EXIF support in image.

API

: Frame

: Frame {
    i width
    i height
    ( Vec f ) data  // CHW, 3 planes, values in [0, 1]
}

@ pp_width * Frame fr → i

@ pp_height * Frame fr → i

@ pp_data * Frame fr → *f

@ pp_free * Frame fr → v

: i __PP_N_RES 10

The reference's RESOLUTION_MAPPINGS[518]: aspect-ratio key → (w, h), all divisible by 14. Keys ascending, exactly the floats the table spells, because ties in "closest" must break the way Python's min() breaks them (first, i.e. smallest key, wins).

@ pp_pick_target f ar → i

The shared target size for a set whose average aspect ratio is ar, packed (w << 16) | h so the pair survives a single return value.

@ pp_target_w i packed → i

@ pp_target_h i packed → i

@ pp_open s path → !Image String

Decode one image and force RGB the way PIL's .convert("RGB") does: grey expands, alpha is dropped. image_convert does exactly that.

@ pp_fit Image im i tw i th → !*Frame String

Resize + centre-crop one decoded RGB image to the shared target size, and hand back the model-input planes. The image is BORROWED.

scale = max(tw/w, th/h) + 1e-8; the intermediate size floors, the kernel is LANCZOS going down and BICUBIC going up, the crop offsets floor — each of those is the reference's own arithmetic, not an approximation of it.

---

load.nu

packages/map-anything/src/load.nu — checkpoint → device buffers.

src/weights.nu finds a tensor and checks its shape; this puts it on the device. The four Linear weights of every block are uploaded TRANSPOSED — torch stores [out, in], the device gets [in, out] — so ma_block_forward runs every gkd_gemm with transb=0 and the CPU backend takes its register-tiled kernel.

f32 the whole way: the checkpoint is f32 and contiguous, so the fast path is a straight copy out of the mapping plus (for the Linears) one permute kernel on the device.

( maw_upload w kit name ) → GkBuf empty on failure ( maw_upload_t w kit name rows cols ) → GkBuf transposed ( maw_block w kit prefix eps dim swh ) → MaBlk ( maw_prefix base idx ) → String "<base>.<idx>."

A missing tensor is recorded in the Lw error list rather than thrown, so a whole model's worth of loading reports the first thing wrong instead of the first thing touched.

API

@ maw_upload * Lw w * GpuKit kit s name → GkBuf

One tensor onto the device, layout unchanged.

@ maw_upload_t * Lw w * GpuKit kit s name i rows i cols → GkBuf

Upload a [rows, cols] tensor TRANSPOSED, as [cols, rows] — straight out of the mapping, permuted on the device.

@ maw_block * Lw w * GpuKit kit s prefix f eps i dim i swh → MaBlk

Every parameter of one transformer block: the fourteen tensors under <prefix> (norm1, qkv, proj, ls1, norm2, w12, w3, ls2). Both stacks in this model share this exact layout; only eps could ever differ, and in this checkpoint both use 1e-6 (encoder blocks are built with partial(LayerNorm, eps=1e-6), the info_sharing blocks likewise).

@ maw_prefix s base i idx → String

<base>.<idx>. — the prefix of an indexed block.

---

devblock.nu

packages/map-anything/src/devblock.nu — the transformer block on the device, in f32.

MapAnything is built from ONE block shape, used 40 times: prenorm LayerNorm (eps 1e-6), biased qkv, SDPA attention, biased proj, LayerScale, then a SwiGLU MLP (w12 → chunk 2 → silu(x1)·x2 → w3), LayerScale again. The DINOv2-giant encoder's 24 blocks and the alternating-attention info_sharing's 16 blocks differ only in their weights and in which tokens they are run over — no RoPE, no qk-norm, no KV cache anywhere, which is why this file is so much smaller than lingbot-map's devblock.nu, its direct ancestor.

Everything is composed from gpukit's gkd_* ops — gemm, layernorm, bmm, softmax, permute, slice, broadcast-elementwise — so the same code runs on CUDA and on the CPU backend.

Weight layouts are torch's, transposed at load time: a Linear weight is [out, in] and is uploaded [in, out] (see load.nu), so every gkd_gemm here runs transb=0 and the CPU backend takes its register-tiled kernel.

( ma_ws_new kit n dim heads swh ) → MaWs ( ma_ws_free ws ) → v ( ma_blk_free w ) → v ( ma_block_forward kit w ws x n dim heads swh ) → b x updated

API

: MaBlk

: MaBlk {
    GkBuf n1g GkBuf n1b
    GkBuf qkvw GkBuf qkvb
    GkBuf pw GkBuf pb
    GkBuf ls1
    GkBuf n2g GkBuf n2b
    GkBuf w12w GkBuf w12b  // [dim, 2·swh] on device (transposed)
    GkBuf w3w GkBuf w3b  // [swh, dim] on device (transposed)
    GkBuf ls2
    f eps
}

One block's parameters, resident on the device.

@ ma_blk_free MaBlk w → v

: MaWs

: MaWs {
    GkBuf norm
    GkBuf qkv
    GkBuf qkvp
    GkBuf kt
    GkBuf att
    GkBuf ctx
    GkBuf ctxp
    GkBuf branch
    GkBuf hid  // [n, 2·swh]
    GkBuf sw1  // [n, swh]
    GkBuf sw2
    GkBuf scal
    i maxn
}

Scratch, sized once for the largest token run the model will see.

@ ma_ws_new * GpuKit kit i n i dim i heads i swh → MaWs

@ ma_ws_free MaWs ws → v

@ _ma_i2 i a i b → ( Vec i )

@ _ma_i3 i a i b i c → ( Vec i )

@ _ma_i4 i a i b i c i d → ( Vec i )

@ ma_view GkBuf b i off i len → GkBuf

A view of b's elements [off, off+len) as its own GkBuf. The device pointer is byte-addressed, so a sub-range is just an offset — no copy, and nothing to free (the parent owns the allocation).

@ ma_block_forward * GpuKit kit MaBlk w MaWs ws GkBuf x

Transformer block over n tokens; x is [n, dim], updated in place. swh is the SwiGLU hidden width (4096 for every block in this model; w12 produces 2·swh and w3 consumes swh).

---

dpthead.nu

packages/map-anything/src/dpthead.nu — the DPT dense head.

This is where ray directions, depth, confidence and the ambiguity mask come out. It takes the four tapped 1536-wide feature sets — the fused encoder features and the info_sharing taps at 7, 11 and 15 — reads one view's patch tokens back as 2-D maps, and fuses them coarse to fine into six full-resolution channels.

For a 37×28 patch grid (a 518×392 frame) the four scales are:

hook 0 → 1×1 to 96 → ConvTranspose 4×4 s4 → 112×148 hook 1 → 1×1 to 192 → ConvTranspose 2×2 s2 → 56×74 hook 2 → 1×1 to 384 → identity → 28×37 hook 3 → 1×1 to 768 → Conv 3×3 s2 p1 → 14×19

then a bias-less 3×3 brings each to 256 channels, and four fusion blocks walk back up, each doubling with bilinear align_corners=True. refinenet4 has no resConfUnit1 (nothing to fuse with) and its output is CROPPED to hook 2's size — its input rounded up on odd grids, and the reference slices the excess row/column off.

Unlike the VGGT-family head in lingbot-map: NO sinusoidal position embedding anywhere, the ReLUs are NOT inplace (the residual adds the ORIGINAL input back, not relu(x) — the opposite trap this time), and the regressor interpolates straight to (H, W) with align_corners=True before its final convs.

Output channels → adaptors (uniception's, exactly):

0..2 ray directions x / max(‖x‖, 1e-8) (unit sphere) 3 depth-along-ray exp(x) 4 confidence 1 + exp(x) 5 mask sigmoid(x); the logits are kept too

( dp_load w kit ) → Dpt ( dp_free d ) → v ( dp_forward kit d h0 h1 h2 h3 voff gh gw h w rays depth conf mask ) → b h0..h3: sequence buffers [·, 1536]; this view's patch tokens are rows voff .. voff+gh·gw rays: [3, h, w] — depth/conf/mask: [h*w] each, device f32

API

: i DP_IN 1536

: i DP_FEAT 256

: DpConv { GkBuf w GkBuf b i hasb }

A conv with an optional bias, as a pair of device buffers.

: DpRcu { DpConv c1 DpConv c2 }

One residual conv unit: relu → 3×3 → relu → 3×3, plus the input.

: DpFuse { DpRcu u1 DpRcu u2 DpConv outc i has1 }

A fusion block. has1 is false only for refinenet4.

: Dpt

: Dpt {
    ( Vec DpConv ) projects  // input_process.i.0.0 — 1×1 to 96/192/384/768
    ( Vec DpConv ) resizes  // input_process.i.0.1 — convT/convT/none/conv-s2
    ( Vec DpConv ) rns  // input_process.i.1 — 3×3 no bias → 256
    ( Vec DpFuse ) fuse  // refinenet1..4 in that order
    DpConv oc1  // dense_head.1.conv1
    DpConv oc2a  // dense_head.1.conv2.0
    DpConv oc2b  // dense_head.1.conv2.2
}

@ dp_load * Lw lw * GpuKit kit → Dpt

@ dp_free Dpt d → v

@ _dp_rcu_fwd * GpuKit kit DpRcu r GkBuf x GkBuf t1 GkBuf t2 i ch i h i w → b

relu → conv3x3 → relu → conv3x3, added back to the ORIGINAL input — the reference builds these units with nn.ReLU(inplace=False), the opposite of the VGGT-family head, where the inplace ReLU makes the residual relu(x). Runs in place on x (x += branch(x)); t1/t2 are scratch of the same size.

@ _dp_fuse_fwd * GpuKit kit DpFuse f GkBuf out GkBuf skip GkBuf up

One fusion step: out (+ rcu1(skip)) → rcu2 → bilinear ×2 (align_corners=True) → 1×1 out_conv. dst is at (2h, 2w).

@ dp_forward * GpuKit kit Dpt d GkBuf h0 GkBuf h1 GkBuf h2 GkBuf h3

The whole head, for one view.

---

interp.nu

packages/map-anything/src/interp.nu — torch's upsample_bicubic2d, WITHOUT antialias, which is what DINOv2's interpolate_pos_encoding calls for the torch-hub models (interpolate_antialias=False, interpolate_offset=0.1).

This is a DIFFERENT kernel from both of the other two bicubics in this codebase, and mixing them up is a silent-wrongness trap:

• Pillow / image package: a = −0.5, support scaled when

shrinking (antialiased)

• torch upsamplebicubic2d_aa a = −0.5 antialiased (what

(lingbot-map's interp.nu): lingbot-map's pos_embed wants)

• THIS: a = −0.75, four fixed taps, never

antialiased, torch's plain interpolate(mode="bicubic")

The historical-kludge offset matters too: DINOv2 passes scale_factor=(gh+0.1)/37 rather than an output size, and torch then maps output row o to source coordinate (o+0.5)/scale_factor − 0.5 — with the +0.1 folded in, that is NOT the same grid as size-based interpolation. The caller passes the reciprocal (rscale = 37/(gh+0.1)) so this file has no opinion about where the scale came from.

( interp_bicubic_torch pin ih iw c oh ow rsy rsx pout ) → v pin planar [c, ih, iw] f64, pout planar [c, oh, ow] f64 src_y = rsy·(oy+0.5) − 0.5, src_x likewise; taps edge-clamped.

API

@ interp_bicubic_torch * f pin i ih i iw i c i oh i ow f rsy f rsx * f pout → v

---

sky.nu

packages/map-anything/src/sky.nu — optional sky masking, the way the LingBot-Map demo does it (--mask_sky): JianyuanWang's skyseg.onnx (a U²-Net) scores sky per pixel, and a pixel survives only where the upsampled score sits at the map's own minimum.

The reference arithmetic, reproduced exactly:

1. the preprocessed view → 320×320 (bilinear, u8)
2. ImageNet-normalise, NCHW f32 → the ONNX graph (first output)
3. min–max normalise the 320×320 score map, quantise to u8

(truncating, as numpy's astype does)

1. bilinear-resize the u8 map to the view size
2. non-sky ⇔ resized value == 0 — the reference's

"1 − clip(u8, 0, 1) > 0.1" collapses to exactly this, and it is no accident: the U²-Net's sigmoid saturates, so ~96% of a real outdoor frame sits at the exact minimum

MapAnything's own non-ambiguous mask already catches much of the sky; this is the belt to that suspender, for outdoor scenes where the depth head hallucinates a far wall instead.

( sky_open path ) → Sky (ok=F on any failure) ( sky_free s ) → v ( sky_mask s chw w h mask ) → b AND non-sky into mask

API

& c @ nurl_poke_f32 *u base i idx f val → v

& c @ nurl_peek_f32 *u base i idx → f

: i SKY_N 320

: Sky

: Sky {
    OGraph g
    * Engine e
    b ok
}

@ sky_open s path → Sky

@ sky_free Sky s → v

@ sky_mask Sky s * f chw i w i h * u mask → b

AND "not sky" into mask for one view. chw is the fitted frame's [3, h, w] planar [0,1] host buffer (pp_data), mask h·w bytes.

---

main.nu

packages/map-anything/src/main.nu — the CLI.

map-anything photos/ images → cloud.ply map-anything walk.avi video (MJPEG-AVI in pure NURL, other codecs via ffmpeg) map-anything photos/ --view open the browser viewer after map-anything view cloud.ply just view an existing cloud

The whole set of views runs through the model AT ONCE — MapAnything is multi-view by construction (global attention over every view plus a scale token), not streaming — so memory grows with the number of views and --max-views / --stride are the levers.

Reference: https://github.com/facebookresearch/map-anything Checkpoint: facebook/map-anything-apache (fetched via hub on first use, cached under ~/.nurl/models).

API

: s MA_DEFAULT_REF facebook/map-anything-apache``

: Opts

: Opts {
    s model
    s out
    s video
    i view
    i port
    i ascii
    i verbose
    i maxviews
    i stride
    i fps
    i pstride
    f confpct
    i maskedges
    i masksky
    i window
    i overlap
    i domask
    ( Vec String ) frames
    i bad  // 0 run, 1 error, 2 help, 3 view-subcommand
    s viewfile
    s vhost
    i vtls
}

@ main → i

---

patchembed.nu

packages/map-anything/src/patchembed.nu — the model's entry point.

DINOv2's patch embedding is Conv2d(3, 1536, kernel=14, stride=14). Kernel and stride are equal, so the patches do not overlap and the convolution is exactly a matmul: lay each 14x14x3 patch out as one row of 588 values and multiply by the reshaped weight.

img [3, H, W] ──im2col──▶ P x 588 ──▶ P x 1536 (+ bias)

with P = (H/14) x (W/14) patches in ROW-MAJOR order, and 588 = 3 x 14 x 14 laid out channel-major then row then column — the order torch's Conv2d weight already has, so the weight is used as-is with no transposition beyond the one the matmul wants.

im2col is written out rather than fused into the multiply because the multiply is the part that belongs on a device kernel later, and the layout is the part that has to be right first.

( pe_patches h w patch ) → i P ( pe_im2col img c h w patch cols ) → v P x (c·patch²) ( pe_project cols p k weight bias out ) → v P x k · k x n → P x n

API

@ pe_grid_h i h i patch → i

@ pe_grid_w i w i patch → i

@ pe_patches i h i w i patch → i

@ pe_im2col * f img i c i h i w i patch * f cols → v

Planar image [c, h, w] → one row per patch, c·patch·patch wide. Row order is row-major over the patch grid; within a row the order is channel, then kernel row, then kernel column — Conv2d's weight layout.

@ pe_project * f cols i p i k * f weight * f bias i n * f out → v

out[P, n] = cols[P, k] · weightᵀ[k, n] + bias[n].

weight is torch's Conv2d weight flattened, [n, k] — output channel major — so this reads it along a row per output channel, which is a dot product against the patch row. Swapped for a device GEMM once the weights live on the device; the loop is the reference the GEMM has to agree with.

---

infoshare.nu

packages/map-anything/src/infoshare.nu — the multi-view alternating- attention transformer (uniception's MultiViewAlternatingAttention- TransformerIFR), the piece that makes MapAnything multi-view.

The whole scene lives in ONE token sequence:

[ view0: patches, cls ][ view1: patches, cls ] … [ scale ]

(per view the patches come FIRST, the encoder's cls register is appended after them; the learned scale token sits at the very end of the sequence). The sinusoid view_pos_table row is added to every view-0 token once, at input — that is how the reference view is distinguished. Then 16 blocks alternate:

even GLOBAL attention over the whole sequence, scale included odd FRAME attention within each view (patches + cls); the scale token sits the layer out

With batch size 1 the reference's (N,V·T,C) ⇄ (N·V,T,C) reshapes are exactly "run the block over each view's contiguous slice", so the frame layers here are per-view calls into the same ma_block_forward and nothing is ever moved.

Features entering this module are the encoder's patches through the FUSION LayerNorm (model.py applies fusion_norm_layer even when no geometric inputs exist — images-only fuses nothing but still norms); the cls registers do NOT pass through it. Intermediate features are captured after blocks 7 and 11 and, like the final output, go through the module's own final norm (norm_intermediate=True).

( is_load w kit ) → InfoShare ( is_free ish ) → v ( is_place kit ish X v tpv tok np ) → b one view into the seq ( is_finish_input kit ish X nv np ) → b fusion LN + viewpos + scale ( is_forward kit ish ws X n i7 i11 fin ) → b 16 blocks + normed taps

X is [nv·(np+1) + 1, 1536] on the device; i7/i11/fin are same-sized buffers that receive the NORMED sequence at each tap (X itself stays un-normed, as the blocks want it).

API

: i IS_DIM 1536

: i IS_HEADS 24

: i IS_SWH 4096

: i IS_DEPTH 16

: i IS_TAP1 7

: i IS_TAP2 11

: f IS_EPS 0.000001

uniception builds its blocks with partial(LayerNorm, eps=1e-6) — the same eps as the encoder's, unlike lingbot-map where the two differed.

: InfoShare

: InfoShare {
    ( Vec MaBlk ) blocks
    GkBuf normg GkBuf normb  // the final norm, also used at the taps
    GkBuf fng GkBuf fnb  // fusion_norm_layer
    GkBuf viewpos  // [1536] — view_pos_table row 0
    ( Vec f ) scaletok  // [1536] host — placed once per run
}

@ is_free InfoShare ish → v

@ is_load * Lw w * GpuKit kit → InfoShare

@ is_tokens i nv i np → i

Sequence length for nv views of np patches each: per view np patches

• 1 cls register, plus the global scale token.

@ is_place * GpuKit kit GkBuf x i v i tpv GkBuf tok i np → b

Place one view's encoder output into the sequence: the dino token buffer has cls at row 0 and patches at rows 1.., the sequence wants patches first and cls after them.

@ is_fuse_input * GpuKit kit InfoShare ish GkBuf x i nv i np → b

After every view is placed: fusion-norm the patch rows (NOT the cls registers). This is the state the DPT head's hook 0 wants — snapshot AFTER this and BEFORE is_mark_input, which is what the reference feeds it (the view positional encoding is info_sharing-internal).

@ is_mark_input * GpuKit kit InfoShare ish GkBuf x i nv i np → b

Add view_pos_table to every view-0 token and write the scale token into the last row.

@ is_finish_input * GpuKit kit InfoShare ish GkBuf x i nv i np → b

@ is_forward * GpuKit kit InfoShare ish MaWs ws GkBuf x i nv i np

The 16 blocks. n is the full sequence length is_tokens(nv, np); after blocks 7 and 11 the sequence is normed into i7/i11, after block 15 into fin. X itself is left un-normed.