All projects
Completed

Repairing a Hypervisor with Corrupted Core Libraries

Tracing a six-month-old mistake that had downgraded about 35 core libraries on a Proxmox host, and restoring them surgically.

Stack
Debian (trixie)dpkgsnapshot.debian.orgWSLDocker

Part of FSJ-LAB.

The problem

While preparing the cluster for high availability, I found that fslab, my original Proxmox host, had a broken package system. Core libraries were the wrong versions, and libc6 was half-installed.

Root cause

The package log told the story. Six months earlier, Ubuntu SNMP packages had been force-installed on this Debian-based host. That one forced install pulled in Ubuntu builds of roughly 35 core Debian libraries and quietly downgraded them. The host kept running, but it was one update away from failing.

Choosing a fix

There were two options: reinstall the node, or repair it in place. I chose the surgical repair so I could keep the node’s configuration and continue the HA work.

The repair

  1. Built the list. The package log recorded the exact Debian version each library had been replaced from.
  2. Found the old versions. Debian’s main mirror only carries current versions, so I used snapshot.debian.org, which archives every version Debian has ever published. Because I didn’t know which dates held each version, I pointed a throwaway Debian container at several snapshot dates at once.
  3. Restored them. I installed all 37 affected packages at their original versions.
  4. Cleaned up. I purged the Ubuntu-only leftovers, after first confirming they shared no files with Debian packages.
  5. Verified. The node rebooted cleanly.

What I learned

  • The package manager’s log is a forensic record. Read it before guessing.
  • --force options don’t make problems go away. They postpone them.
  • Never install packages built for a different distribution on a hypervisor.
  • A careful repair can beat a rebuild when you understand exactly what changed.