LibreNMS Pinned at 100% CPU — MariaDB Stuck on Daylight Time

Summary

The VM running LibreNMS (monitoring-core-1, 2 vCPU / 4 GB) sat at ~100% CPU with a load average of ~10. The cause wasn’t load or a bad device — MariaDB was configured with a fixed -04:00 offset (EDT). When daylight saving time ended on November 1, the database clock stayed an hour ahead of the OS and PHP. Every device’s last_polled timestamp looked an hour old, so the LibreNMS dispatcher re-polled all 13 devices the instant each poll finished.

Result after the fix: load average dropped from ~10 → 0.3, and polling returned to normal 5-minute bursts.

Symptoms

Proxmox CPU graph for the VM flat at ~100% for over an hour.
uptime inside the VM: load average: 9.68, 10.04, 10.01 on 2 vCPUs.
13 lnms device:poll processes running at once, and the same device polled twice simultaneously.
validate.php reported everything [OK] — no obvious misconfiguration.

Proxmox summary for monitoring-core-1 showing CPU usage flat at roughly 100% of 2 vCPUs for the entire hour — Proxmox: the LibreNMS VM pinned at ~100% across both vCPUs for the whole hour.

Investigation

I worked from the outside in, ruling out one layer at a time.

1. Host or guest?

On the Proxmox host, the VM’s kvm process was at 200% (both vCPUs saturated) while the host load was only ~2.7 on 12 threads. The host wasn’t contended — the load was real and inside the guest.

2. Which process?

Inside the guest, ps --sort=-%cpu showed a wall of php /opt/librenms/lnms device:poll processes plus MariaDB. Oxidized, Bacula and everything else were idle.

3. Is a device just slow?

SELECT device_id, hostname, os, last_polled_timetaken
FROM devices ORDER BY last_polled_timetaken DESC;

The slowest device (a Catalyst switch) took ~24 s; most took ~5 s. Total work was ~85 seconds per 5-minute cycle — nowhere near enough to peg the CPU. Something was polling far more often than every 5 minutes.

4. Is something polling twice?

No legacy cron poller (/etc/cron.d/librenms didn’t exist), exactly one librenms-service.py dispatcher, and only the normal scheduler timer. But the dispatcher log showed the smoking gun:

18:56:54  Completed poller run for 4 in 6.10s
18:56:54  Polling device 4
18:57:01  Completed poller run for 4 in 6.42s
18:57:01  Polling device 4

Each device was re-queued immediately after finishing. The dispatcher believed every device was overdue.

5. Why does every device look overdue?

SELECT NOW() AS db_now, UTC_TIMESTAMP() AS db_utc;
SELECT device_id, TIMESTAMPDIFF(SECOND, last_polled, NOW()) AS age_s FROM devices;

Value	Reading
OS `date`	18:58:49 EST
MariaDB `UTC_TIMESTAMP()`	23:58:49 (correct)
MariaDB `NOW()`	19:58:49 — one hour ahead
`age_s` for every device	~3605 seconds

Devices polled seconds earlier looked an hour stale.

Terminal output: MariaDB NOW() reads 19:58 while the system clock reads 18:58 EST, and every device shows age_s of about 3605 seconds — The smoking gun: database NOW() is an hour ahead of the OS clock, so every last_polled looks ~3600 s old.

SELECT @@global.time_zone, @@system_time_zone;
-- -04:00 | EST

/etc/mysql/mariadb.conf.d/50-server.cnf:20: default_time_zone = '-04:00'

Root Cause

MariaDB had a hard-coded UTC offset (-04:00) instead of following the system timezone. A fixed offset doesn’t change for daylight saving time, so when EDT ended the database clock drifted an hour ahead of the OS.

PHP writes last_polled in the correct local time, but the Python dispatcher compares it against MariaDB’s NOW() — so every device permanently looked ~3600 seconds overdue against a 300-second polling interval.

Why validate.php missed it: LibreNMS’s PHP layer sets its own session timezone on each DB connection, so PHP-vs-MySQL time checks matched. The Python dispatcher doesn’t, so it inherited the wrong global setting.

Fix

# 1. Correct the running server immediately (no restart needed)
sudo mysql -e "SET GLOBAL time_zone = 'SYSTEM';"

# 2. Persist it: back up and comment out the fixed offset
sudo cp /etc/mysql/mariadb.conf.d/50-server.cnf{,.bak.$(date +%F)}
sudo sed -i -E "s|^(\s*default[-_]time[-_]zone\s*=.*)|# \1|" /etc/mysql/mariadb.conf.d/50-server.cnf

# 3. Prove it survives a restart, then reconnect the dispatcher
sudo systemctl restart mariadb
sudo mysql -e "SELECT NOW(), @@global.time_zone;"; date
sudo systemctl restart librenms

I also lowered the dispatcher’s concurrency from the default 24 to 4 workers — a better fit for 2 vCPUs so polls don’t fight each other during each 5-minute burst:

sudo -u librenms lnms config:set service_poller_workers 4

Verification

Check	Before	After
MariaDB `time_zone`	`-04:00`	`SYSTEM` (persists across restart)
`NOW()` vs `date`	+1 hour	identical
Device `age_s`	~3605 s	0–300 s
Concurrent polls	13, constantly	≤ 4, in 5-minute bursts
Load average (2 vCPU)	~10	0.3
Proxmox CPU graph	flat ~100%	short spikes every 5 min

Lessons Learned

Never hard-code a UTC offset in a database. Use SYSTEM or a named zone (America/New_York) so DST is handled automatically. A fixed offset is a time bomb that goes off twice a year.
“All checks pass” isn’t proof. The validator tested the PHP path; the broken path was the Python dispatcher.
Do the math on the workload. ~85 s of polling per 5 minutes can’t explain 100% CPU — that mismatch pointed at frequency, not cost.
Compare timestamps from every layer. OS, database NOW(), and database UTC side by side exposed the problem in one query.
Isolate layer by layer: host → guest → process → workload → scheduling → time. Each step ruled something out before moving on.