Lennard V. John

Tag: terraform

  • Fixing Proxmox Terraform Deletes with curl + jq

    When I started automating my homelab using Terraform + Proxmox, everything worked great… until it didn’t.

    The problem

    In a push-based deployment (via GitHub Actions), I hit a blocker: the Proxmox Terraform provider cannot delete a running VM or container.

    That means:

    1. Terraform tries to destroy a resource
    2. Proxmox rejects it
    3. Pipeline fails ❌

    So the fix is simple in theory: check if the VM is running → stop it → then delete it.

    But Terraform doesn’t handle this natively — so I built a small workaround using curl and jq.

    What you need

    • curl — for API calls
    • jq — for parsing JSON

    Proxmox API access

    You’ll need a Proxmox API token. The relevant docs:

    Set your environment variables (Windows example):

    setx TF_VAR_proxmox_api_url "https://YOUR-IP:8006/api2/json"
setx TF_VAR_proxmox_api_token_id "user@pam!token"
setx TF_VAR_proxmox_api_token_secret "your-secret"

    Quick environment variable reference (Windows)

    # Temporary
set MY_VAR=HelloWorld
echo %MY_VAR%

# Persistent
setx MY_VAR "HelloWorld"

# System-wide
setx /m MY_VAR "HelloWorld"

    The fix

    We query the VM status using the Proxmox API:

    curl -sk -H "Authorization: PVEAPIToken=USER!TOKEN=SECRET" \
  https://IP:8006/api2/json/nodes/NODE/qemu/VMID/status/current

    Then extract the state with jq:

    jq -r '.data.status'

    Putting it together

    STATUS=$(curl -sk -H "$PVE_AUTH" \
  "${PVE_API}/nodes/${PVE_NODE}/qemu/${VMID}/status/current" \
  | jq -r '.data.status')

if [ "$STATUS" = "running" ]; then
  echo "Stopping VM $VMID..."
  curl -sk -X POST -H "$PVE_AUTH" \
    "${PVE_API}/nodes/${PVE_NODE}/qemu/${VMID}/status/stop"
fi

    Now Terraform can safely destroy the VM after this runs.

  • I Built a Production Platform… Just to Write a Blog

    I just wanted to write a blog.

    That’s it. No scaling requirements. No users. No revenue. Just a place to write.

    So naturally, I built a 3-node Kubernetes cluster on bare metal, automated everything with Terraform and Ansible, added GitOps, monitoring, alerting, and exposed it to the internet using a zero-trust Cloudflare tunnel.

    Completely normal.

    Why over-engineer something so simple?

    Because the goal wasn’t the blog. The goal was to learn how real platforms are built.

    Not tutorials. Not “hello world” deployments. The actual systems that sit behind production environments. So I treated this like a real platform: fully automated, reproducible, observable, secure by design.

    The one command that does everything

    docker compose up

    That single command:

    • Provisions infrastructure (Terraform → Proxmox)
    • Configures the cluster (Ansible → Kubernetes)
    • Deploys platform services (Ingress, monitoring, GitOps)
    • Deploys applications (WordPress + MariaDB)
    • Sets up public access (Cloudflare Tunnel)

    About 25 minutes later… the entire platform is live. No manual steps. No SSH after bootstrap.

    How it’s actually built

    Git → CI/CD → Terraform → VMs → Ansible → Kubernetes → Argo CD → Apps

    Infrastructure (Terraform)

    • Creates 3 VMs on Proxmox (1 control plane, 2 workers)
    • Configures Cloudflare tunnel + DNS

    Configuration (Ansible)

    • Bootstraps Kubernetes using kubeadm
    • Installs core services (Ingress, storage, networking)
    • Deploys monitoring and applications

    GitOps (Argo CD)

    • Watches the kubernetes/ directory
    • Automatically syncs changes to the cluster

    After the initial build, I don’t SSH into anything anymore. Everything is managed through Git.

    GitOps in action

    This is my favourite part. I change a file in Git → push → and Argo CD updates the cluster automatically. No kubectl. No manual deploys. Just Git.

    That shift — from “run commands” to “declare desired state” — completely changes how you think about systems.

    Zero trust (no open ports)

    Nothing in my home network is exposed directly. Instead, everything goes through a Cloudflare Tunnel:

    User → Cloudflare Edge → Tunnel → Kubernetes Ingress → App
    • No port forwarding
    • No public IP exposure
    • HTTPS handled at the edge

    It’s simple, secure, and surprisingly powerful.

    Observability (because things WILL break)

    I added Prometheus for metrics, Grafana for dashboards, and AlertManager for email alerts. If a pod crashes, a node goes down, or memory spikes — I get notified.

    Because a platform you can’t observe… is a platform you don’t understand.

    What went wrong (the important part)

    This didn’t work the first time. Some of the issues I hit:

    • Terraform not returning VM IPs (guest agent problems)
    • Ansible running before infrastructure was ready
    • Kubernetes probes killing WordPress before it stabilised
    • Infinite HTTPS redirect loops behind Cloudflare
    • PVC-related deployment deadlocks

    Most of these came down to one thing: I assumed things would be ready… when they weren’t.

    Biggest lesson

    Stop designing systems that depend on timing. Start designing systems that converge.

    Instead of “wait 30 seconds and hope”, do: retry until ready, check actual state, design for failure. That shift alone made everything more reliable.

    Reproducibility (this is the real win)

    The entire platform can be destroyed, rebuilt, and moved using the same codebase. That means no snowflake servers, no hidden config, and no “it only works on my machine”.

    What this project actually demonstrates

    This isn’t about WordPress. It demonstrates:

    • Infrastructure as Code (Terraform)
    • Configuration as Code (Ansible)
    • GitOps (Argo CD)
    • Zero Trust Networking (Cloudflare)
    • Observability-first design

    In other words: how modern platforms are actually built.

    What’s next

    • Move this to a Talos-based cluster
    • Add multi-environment support (dev/staging/prod)
    • Implement proper secrets management (SOPS or Vault)
    • Explore multi-cluster deployments

    Final thoughts

    Yes — this is overkill for a blog. But that’s kind of the point. If you can build something like this for a simple project… you can build it for something that actually matters.

    If you want to check it out: github.com/Lennardj/homelab-blog

    I’m Lennard — currently transitioning into DevOps / Platform Engineering. If you’re on the same journey, feel free to connect.