Oxidized Crash-Looping — Stale API Token and a Config Migration

Summary

Oxidized (network config backup) on monitoring-core-1 was enabled in LibreNMS but never actually running — systemd had restarted it 279+ times. The root cause was a stale LibreNMS API token. Fixing it exposed two follow-on issues: a deprecated rest: setting whose replacement needed a different format, and a half-migrated output block that sent Git commits to the wrong location.

Result: Oxidized now pulls its device list from LibreNMS, backs up both Catalyst switches and the FortiGate, and stores version history in Git that LibreNMS displays on each device’s Config tab.

Issue 1 — Crash on “Loading nodes”

Oxidized::Nodes -- Loading nodes
Oxidized::CLI -- Oxidized crashed, crashfile written in ~/.config/oxidized/crash
no implicit conversion of String into Integer

The backtrace pointed into source/http.rb → transform_json → string_navigate_object — Oxidized was trying to read hostname and os from each device in the API response.

Diagnosis: that Ruby TypeError means the code expected an array of devices but got a single JSON object. Calling the API directly confirmed it:

curl -s -H "X-Auth-Token: <old token>" http://127.0.0.1/api/v0/oxidized
# {"message":"Unauthenticated."}

curl -s -H "X-Auth-Token: <valid token>" http://127.0.0.1/api/v0/oxidized
# [{"hostname":"core-sw","os":"ios",...}, ...]

Root cause: the token in the Oxidized config was no longer valid. LibreNMS returned an error object, and Oxidized crashed trying to iterate it as a device list.

Fix: replaced the token in ~/.config/oxidized/config. Oxidized loaded 3 nodes and backed up all of them.

Issue 2 — Removing a deprecated setting caused a new crash

Oxidized logged a warning that the top-level rest: 127.0.0.1:8888 setting is deprecated in favour of extensions.oxidized-web. Removing it caused a new crash:

from .../puma/binder.rb:344:in `add_tcp_listener'
getaddrinfo: Name or service not known

Root cause: the new extension block expects the address and port as separate keys. With listen: 127.0.0.1:8888, Puma tried to resolve the literal string 127.0.0.1:8888 as a hostname.

Fix:

extensions:
  oxidized-web:
    load: true
    listen: 127.0.0.1
    port: 8888

Issue 3 — Git commits landing in the wrong place

After switching output from file to git, commits were being made — but not where expected:

git --git-dir=/opt/oxidized/configs.git log --oneline
# fatal: not a git repository

Root cause: the output block was half-migrated — default: git was set, but the sub-section was still named file: with a directory: key. Git output reads git: → repo:, found neither, and silently fell back to its default path (~/.config/oxidized/oxidized.git).

Fix: moved the existing repo (keeping its history) and corrected the block:

output:
  default: git
  git:
    user: Oxidized
    email: oxidized@monitoring-core-1
    repo: /opt/oxidized/configs.git

Verification

• systemctl status oxidized → active (running), no restarts.
• curl http://127.0.0.1:8888/nodes.json → all 3 devices, "status":"success".
• git --git-dir=/opt/oxidized/configs.git log --oneline → one commit per device.
• LibreNMS device Config tab shows the Oxidized copy with version history.

Lessons Learned

• Read the error type, not just the message. A Ruby TypeError while parsing a list usually means the response wasn’t a list — often an error object. curl the source directly to see what the app actually received.
• A deprecation fix is a migration, not a deletion. The replacement setting had a different format.
• Silent fallbacks hide misconfiguration. Oxidized didn’t error on the wrong keys — it quietly used a default. Always verify where data actually lands.
• One change at a time, verified. Each fix exposed the next issue cleanly because only one thing changed per step.
• Keep secrets out of write-ups. API tokens and device credentials are redacted here and rotated after troubleshooting.