All projects
In progress

GPU Passthrough and a Local AI Stack

An RTX 3070 passed through to a VM, running private LLMs and a RAG assistant that knows my lab, with no internet.

Stack
Proxmox VFIONVIDIA driver 570 / CUDA 12.8DockerNVIDIA Container ToolkitOllamaOpen WebUInomic-embed-text
Hardware
  • NVIDIA RTX 3070 8 GB (on fslab-2)

Part of FSJ-LAB.

Goal

Run large language models entirely on my own hardware, then make them useful by teaching an assistant about my lab, all inside an air-gapped network. It’s a small-scale version of what organizations are doing with private, on-prem AI.

What I built

  • GPU passthrough. The RTX 3070 is passed through with VFIO to a dedicated VM, compute-apps-1, running the NVIDIA 570 driver and CUDA 12.8.
  • Container runtime. Docker with the NVIDIA Container Toolkit. The OS lives on a small disk, while models and images live on a separate 400 GB data disk.
  • Inference. Ollama serves models like Llama 3.1 8B on the GPU, with Open WebUI as the chat interface.
  • A lab assistant. An Open WebUI knowledge base of my lab documentation, embedded with nomic-embed-text, so the model answers questions about my environment using retrieval-augmented generation (RAG) rather than fine-tuning.

Problems I hit

  • Everything had to come in offline. Images were saved, transferred, and loaded by hand (see the air-gap pipeline).
  • Disk exhaustion on an earlier build. The root filesystem filled up because Docker and containerd each keep their own storage root. Moving only Docker’s wasn’t enough.
  • “Too many open files.” Embedding large documents hit the default file-descriptor limit. Raising nofile on both containers fixed it.
  • Performance tuning. Keeping models loaded longer between requests and enabling flash attention made the assistant noticeably more responsive on 8 GB of VRAM.

What I learned

  • GPU passthrough ties a VM to one physical host, so this VM can never be part of an HA failover group. That’s an architecture trade-off, not a bug.
  • RAG is the practical way to make a model “know” your environment. Fine-tuning is rarely needed.
  • In an air-gapped lab, even the embedding model has to be planned for in advance.

Next steps

  • Have n8n feed the knowledge base automatically from Proxmox, LibreNMS, and network configs, with secrets scrubbed first
  • Move retrieval storage into Supabase with pgvector
  • Upgrade to an RTX 3090 (24 GB) for larger models