Troubleshooting
Real problems from FSJ-LAB, written up the way I'd document an incident at work: what broke, how I narrowed it down, the root cause, the fix, and what I'd do differently. Screenshots are sanitized and lab addresses are replaced with device roles.
LibreNMS Pinned at 100% CPU — MariaDB Stuck on Daylight Time
A hard-coded UTC offset in MariaDB put the database clock an hour ahead after DST ended, so LibreNMS treated every device as overdue and polled non-stop.
LibreNMSMariaDBProxmox
Resolvedhigh severityOxidized Crash-Looping — Stale API Token and a Config Migration
Oxidized restarted 279+ times because LibreNMS rejected its API token; fixing it surfaced two more config issues on the way to Git-versioned backups.
OxidizedLibreNMSGit
Resolvedmedium severity