Introduction

zerus makes and hosts offline mirrors of crates.io. A mirror can hold only the crates of your projects, or a larger set of crates for general use.

The usual workflow is:

1. On a computer with internet access, download the crates with zerus mirror.
2. Copy the mirror to the offline network. Record each transfer with a manifest, so the next transfer carries only the new crates.
3. On the offline network, generate the index with zerus update-index.
4. Serve the mirror with zerus serve, and point cargo at it.

Installation

Build from the published source on crates.io:

$ cargo install zerus --locked

Or download a binary from GitHub releases.

For a container image, see Serve With Docker.

Download Crates

Use zerus mirror to download the .crate files for the dependencies of your projects.

$ zerus mirror new-mirror ../deku/Cargo.toml ../adsb_deku/Cargo.toml

To mirror a crate by name, use --crate. A name alone gives the latest version, and name@version gives that version. --crate implies --get-feature-gated.

$ zerus mirror new-mirror --crate reqwest --crate serde@1.0.210

All dependencies, for all features

Use --get-feature-gated to download all the dependencies of each crate, recursively, for all features. The mirror then has the crates that your projects need if you enable more features later.

$ zerus mirror new-mirror --get-feature-gated ../deku/Cargo.toml

Note

This expansion includes the dev and build dependencies of each crate, and ignores feature gates. Thus the result is much larger than what cargo metadata resolves. Even a small crate such as cfg-if expands to thousands of crates. To mirror only the resolved dependencies of a project, give its Cargo.toml and do not use --crate.

The top crates

To add the top 100 crates of the Rust Playground:

$ git clone https://github.com/rust-lang/rust-playground
$ zerus mirror new-mirror rust-playground/top-crates/Cargo.toml

Transfer to an Offline Network

Copy the mirror directory to your proxy or offline network.

Repeat transfers

Do not carry crates that a previous transfer carried. Record each transfer with generate-manifest, and use cull to remove the crates of previous transfers before the next one.

$ zerus mirror new-mirror Cargo.toml                                # download all crates
$ zerus cull new-mirror manifests/*.txt                             # remove the crates of previous transfers
$ zerus generate-manifest new-mirror -o manifests/$(date +%F).txt   # record this transfer

Copy new-mirror to the offline network, and keep manifests/ on this side.

cull takes one or more manifest files, and removes all the crates in them. To see what it removes, and not remove it, use --dry-run.

On the offline network, merge the new crates into the mirror, then generate the index again.

To see the manifests in a browser, see Web UI.

Generate the Index

On the offline network, use update-index to generate a registry index from the .crate files.

$ zerus update-index new-mirror --dl-url http://[IP]

--dl-url is a base URL. zerus adds /crates/{prefix}/{crate}/{version}/{crate}-{version}.crate to it. Give the server address, not the full download template.

Run update-index again each time you add crates to the mirror.

Serve the Mirror

Use zerus serve to serve the registry. It serves a sparse index, the crate downloads, a search API, and a web UI.

$ zerus serve new-mirror --bind 0.0.0.0:8080

To log the requests:

$ RUST_LOG=tower_http=debug zerus serve new-mirror

Serve with Docker

Each release pushes an image that runs zerus serve. Mount the mirror at /mirror.

$ docker run -p 8080:8080 -v "$PWD/new-mirror:/mirror:ro" ghcr.io/wcampbell0x2a/zerus

To change the command, add arguments. For example, to see the previous manifests:

$ docker run -p 8080:8080 -v "$PWD/new-mirror:/mirror:ro" ghcr.io/wcampbell0x2a/zerus \
      serve /mirror --bind 0.0.0.0:8080 --manifests /mirror/manifests

Use latest, or a version tag such as ghcr.io/wcampbell0x2a/zerus:0.16.0.

Web UI

zerus serve shows a web UI at the root path. Set --manifests to the directory of the manifest files from generate-manifest, to see the crates of each transfer.

$ zerus serve new-mirror --manifests manifests/

The manifests page shows each manifest, with the number of its crates that are in the mirror and the number that cull removed. The un-manifested row has the crates that are in the mirror and in no manifest, which are the crates for the next transfer.

A manifest page shows its crates, with the time that zerus mirror wrote each one. Click a column header to sort, and click it again to reverse the sort.

The un-manifested page shows the crates for the next transfer.

Search finds a crate in all the manifests and in the mirror, and shows if cull removed it. Press s to go to the search box.

Scans

The server scans the mirror and the manifests in the background when it starts, and keeps the result in memory. Thus the pages stay fast when the mirror is on a slow disk or a network disk. Cargo can use the registry while the first scan runs.

The header of each page shows the age of its data. When a page loads and its data is older than 10 minutes, the server scans again in the background. The page shows the old data until the scan is done. A server that nobody looks at does no scans. To scan immediately, click Refresh in the header. The page loads again when the scan is done. Only one scan runs at a time.

To set the age, use --rescan-after, for example 30s or 1h. Use 0 so that only Refresh starts a scan:

$ zerus serve new-mirror --manifests manifests/ --rescan-after 1h

Try it

just demo makes a demo mirror with five transfers in target/demo, and serves it at http://127.0.0.1:3000. To use a different port, use just demo 8000. The screenshots on this page come from this demo.

Build With the Mirror

Add this to .cargo/config.toml. Replace [IP] with the server address.

[source.zerus]
registry = "sparse+http://[IP]/crates.io-index/"

[source.crates-io]
replace-with = "zerus"

[registries.zerus]
index = "sparse+http://[IP]/crates.io-index/"

Search

With the registries entry, you can search the mirror:

$ cargo search --registry zerus serde

The server reads the index for the search when it starts, and keeps it in memory. Each run of update-index writes a marker file, crates.io-index/.zerus-updated. Thus the next search reads the index again and finds the new crates. If the index has no marker, for example from an older zerus, the search reads the index again when it is older than --rescan-after.

Development

All patches and merge requests are welcome!

$ just build              # build for x86_64-unknown-linux-musl, with cross
$ just test               # run all the tests
$ just demo               # serve a demo mirror at http://127.0.0.1:3000
$ just book               # build this book, with new screenshots

just book-screenshots takes the screenshots of the web UI with headless Firefox. The images are not in git. The book workflow makes them again each time it builds the book.