pull.nu

packages/hub/src/pull.nu — the transfer engine: stream a URL into a content-addressed blob, constant-memory and resumable.

HTTP chunks flow straight to disk through file_write_chunk while an incremental sha256 consumes the same bytes and a tty-gated progress bar narrates. An interrupted pull leaves blobs/<staging>.part; the next attempt sends Range: bytes=<size>- — a 206 re-hashes the existing part (streamed) and appends, a 200 (server without Range) restarts cleanly. The finished file is renamed to its content address (blobs/sha256-<hex>) and the hex is returned.

When expected is a 64-hex sha256 (Hugging Face publishes one for every LFS file) two things change: an already-cached blob is returned without touching the network, and a completed download whose bytes do not hash to expected is discarded as corrupt rather than stored.

( hub_fetch_blob root url staging expected ) → !String String (hex)

API

@ hub_fetch_blob String root s url s staging s expected → !String String

---

hub.nu

packages/hub/src/hub.nu — the front door.

( hub_file ref ) → !String String download one file, return its path ( hub_dir ref ) → !String String download a whole repo, return the dir ( hub_ls ) → v list what is cached ( hub_rm handle ) → !v String forget a model, GC unshared blobs ( hub_verify handle ) → !v String re-hash every blob against its sha

hub_dir materialises a model directory the Hugging Face way: each file is a symlink under models/<repo>/snapshots/<rev>/ pointing at its content-addressed blob, so an existing loader opens a real directory and two revisions that share a file share the bytes on disk.

API

@ hub_dir s ref → !String String

@ hub_file s ref → !String String

@ hub_ls → v

@ hub_get s ref → !String String

The one call a consumer wants: turn a model argument into a usable local path, fetching it if need be. An argument that already names an existing file or directory is passed straight through (so a local model keeps working); otherwise it is a Hugging Face ref — a single file (a URL, or a ref with a subpath) resolves through hub_file, a bare org/repo through hub_dir. embed serve BAAI/bge-m3 and embed serve ./my-model both just work.

@ hub_path s ref → ?String

Resolve a ref to its local path WITHOUT fetching — the blob path for a cached file, the snapshot dir for a cached repo. None when not cached.

@ hub_verify s ref → !v String

verify/rm accept a ref and canonicalize it to a handle, so the same ref that was pulled (org/repo) matches its stored handle (org/repo@main).

@ hub_rm s ref → !v String

---

hf.nu

packages/hub/src/hf.nu — Hugging Face refs and the repo file listing.

A ref is one of: https://… | http://… a direct URL, passed through org/repo a whole repository (rev = main) org/repo/path/to/file.gguf one file inside a repository org/repo@revision[/path…] pinned to a branch/tag/commit hf.co/org/repo[/path…] the hf.co shorthand (prefix stripped)

hub_hf_tree asks GET /api/models/<repo>/tree/<rev>?recursive=1 for the file list and reads each file's size and — for LFS files, which is where the weights live — the sha256 Hugging Face publishes as lfs.oid. Small non-LFS files (config.json, tokenizer.json) carry only a git blob oid (a sha1), so they get no external anchor; hub sha256s them locally for the content address. The listing follows the Link: rel="next" pagination so a repository with hundreds of files is never silently truncated.

$HF_ENDPOINT overrides https://huggingface.co (mirrors).

API

: HubRef

: HubRef {
    b is_url
    String url
    String repo
    String rev
    String subpath
}

: HubFile

: HubFile {
    String path
    String sha
    i size
    b lfs
}

@ hub_ref_free HubRef r → v

@ hub_file_free HubFile f → v

@ hub_hf_endpoint → String

@ hub_ref_parse s ref → HubRef

Parse a ref into its parts. Never fails — a malformed ref simply yields an empty repo, which the caller reports.

@ hub_resolve_url HubRef r → String

https://<endpoint>/<repo>/resolve/<rev>/<subpath>

@ hub_hf_tree HubRef r → !( Vec HubFile ) String

List every file in <repo>@<rev>, following pagination.

---

main.nu

packages/hub/src/main.nu — the hub CLI.

hub pull <ref> fetch a file or a whole repo (auto) hub file <ref> fetch one file, print its local path hub dir <ref> fetch a whole repo, print its directory hub ls list what is cached hub path <ref> print the cached path (no fetch), else exit 1 hub verify <handle> re-hash every blob against its sha256 hub rm <handle> forget a model, GC unshared blobs

A ref is org/repo, org/repo/path/file.gguf, org/repo@rev[/path], hf.co/… , or an http(s):// URL. The cache is $NURL_MODELS or ~/.nurl/models.

API

@ main → i

---

store.nu

packages/hub/src/store.nu — the shared, content-addressed model cache (~/.nurl/models). Shaped like Hugging Face's own hub cache:

<root>/blobs/sha256-<64hex> every file, named by content <root>/models/<org>--<repo>/snapshots/<rev>/ a symlink farm = a real "model directory" <root>/manifests/<handle> one small JSON per handle

A manifest is deliberately uniform across the single-file and the whole-repo case — a lone GGUF is just a repo with one file:

{ name, kind: "file"|"repo", source, rev, size, files: [ { path, sha, size, lfs } ] }

The sha of a file IS its blob address, so two handles that share a file share the bytes; rm drops a blob only once no manifest still references it, and verify re-hashes every blob against its recorded sha and refuses drift. The symlink farm lives in hub.nu; this file owns the blob layer and the manifest.

( hub_store_root ) → String $NURL_MODELS | $HUB_HOME | ~/.nurl/models ( hub_store_init root ) → !v String ( hub_blob_path root hex ) → String blobs/sha256-<hex> ( hub_manifest_path root name ) → String ( hub_hash_file path ) → !String String streaming sha256 ( hub_store_verify root name ) → !v String

API

@ hub_store_root → String

@ _hub_safe_name s name → String

Handle names live on the filesystem: keep [A-Za-z0-9.-], map the rest (the slash of an org/repo name included) to ''.

@ hub_blob_path String root s hex → String

@ hub_manifest_path String root s name → String

@ hub_snapshot_dir String root s repo s rev → String

The materialised model directory for a repo handle: models/<safe-repo>/snapshots/<rev>

@ hub_store_init String root → !v String

@ hub_hash_file s path → !String String

Streaming sha256 of a file — constant memory whatever the size.

@ hub_manifest_read String root s name → ?Json

Read a manifest into its parsed Json (owned) — None when absent/bad.

@ hub_manifest_exists String root s name → b

Does the manifest exist on disk?

@ _hub_manifest_shas Json j ( Vec String ) out → v

Collect every blob sha referenced by a manifest into out (a Vec of owned hex Strings). j is the parsed manifest (borrow, not freed).

@ _hub_blob_shared String root s hex s except_name → b

True when any manifest OTHER than except_name references blob hex.

@ hub_store_verify String root s name → !v String

Re-hash every blob a handle references and compare to the recorded sha — the cache's integrity loop, streaming and constant-memory.