Air-Gapped Package and Image Pipeline

Goal

Nothing in the lab has internet access, so every package, container image, and binary needs a safe, repeatable way in. The end goal is an internal APT mirror so VMs can install packages normally without manual copying.

How it works today

Staging workstation. An internet-connected machine downloads packages and images, which are then copied into the lab over SCP or rsync.
Container-based downloads. To fetch packages that match a target’s exact OS release, I run a throwaway container of that release (Debian for the Proxmox nodes, Ubuntu for the VMs) and download only what’s needed.
Offline Docker images. Images are saved to tar files, transferred, and loaded on the target.

The APT mirror

My first attempt used apt-mirror, but it keeps every historical package version. A full mirror was projected at around 1.5 TB. I switched to debmirror current-snapshot runs, added a dedicated 400 GB mirror disk, and serve the result with nginx. The open question is scope: fitting the large “universe” component within that disk.

Problems I hit

A load that lied. docker load returned exit code 0 while its output reported layer extraction errors. I now check the output, not just the exit code.
Two storage roots. Docker and containerd each have their own data directory. Moving one and not the other filled the root disk.
Wrong-distro packages are dangerous. Installing Ubuntu packages on a Debian host caused a serious repair job. The rule now: Proxmox nodes only get Debian packages.

What I learned

Air-gapped operations force you to understand dependencies instead of letting a package manager hide them.
“Success” signals from tools aren’t always trustworthy. Verify the result.
Mirroring “everything” is rarely the right scope. Mirror what you actually use.

Next steps

Finish the mirror rebuild and point every VM at it