Lennard V. John

Kubernetes Platform

A three-node cluster built with kubeadm on bare metal — and the engineering decisions, trade-offs and failures behind it.

This is the technical companion to the Homelab Platform. That page covers what was built and why. This one covers how it was engineered, what it cost, and where it is still weak.

Why kubeadm, not managed Kubernetes

A managed control plane hides exactly the parts that break. Building with kubeadm means owning the container network, the load balancer, storage provisioning, certificate issuance and ingress yourself — and understanding each one when it fails at 11pm.

The cluster is one control plane and two workers, bootstrapped entirely from automation rather than by hand, so it can be destroyed and rebuilt from the same code.

Networking on bare metal

There is no cloud provider here to hand out an external IP, which changes what you have to build.

  • Calico — CNI with NetworkPolicy genuinely enforced. Policies without a compatible CNI are silently ignored, which is a dangerous way to think you are protected.
  • MetalLB — assigns external IPs from a LAN pool so type: LoadBalancer means something outside a cloud.
  • ingress-nginx — HTTP routing and TLS termination for every service behind one entry point.

Certificates when there is no inbound path

All traffic arrives through a Cloudflare Tunnel, so there is no inbound HTTP route for Let’s Encrypt’s HTTP-01 challenge to validate against. It would fail every time.

cert-manager proves domain ownership over DNS-01 instead, writing a TXT record through the Cloudflare API. No inbound HTTP required. The constraint dictated the design rather than the other way round — which is usually the sign you have understood the constraint.

Default-deny networking, and how it fails

The database accepts connections from exactly three workloads: the web application, the backup job, and the administrative CLI. Every other pod in the cluster is denied, including anything an attacker manages to land in.

The operational hazard is worth stating precisely, because it has caught me twice: a missing policy entry does not produce a connection error. Calico drops the packets and the client hangs until it times out. There is no refusal, no log line at the client, and the symptom surfaces nowhere near the cause.

The rule that came out of it: any new workload that talks to a protected service needs its policy considered before it is deployed, not after it mysteriously hangs.

Storage, and what ReadWriteOnce actually guarantees

This is the most commonly misread part of the Kubernetes storage model, and it nearly cost me a database.

ReadWriteOnce does not mean “one pod”. It means one node. Multiple pods scheduled onto that same node can all mount the volume simultaneously.

A routine memory increase triggered a rolling update on the database Deployment. The new pod did not fail to schedule — it scheduled successfully onto the same node, mounted the live data directory, and started a second database process against files already open by the running instance.

[ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
[Note]  InnoDB: Check that you do not already have another mariadbd process
        using the same InnoDB data or log files.
[ERROR] Failed to initialize plugins.
[ERROR] Aborting

The storage engine’s file lock was the only thing standing between that and corruption. Kubernetes would happily have run two database processes against one data directory; the database refused. Knowing which layer is actually protecting your data matters when reasoning about what “it didn’t break” proves.

The fix is one line — strategy: Recreate — but the lesson is larger. The wrong strategy had been sitting in the manifest since day one, harmless, because nothing had ever triggered a pod replacement. Configuration that is never exercised is untested configuration, and the day it is exercised is rarely convenient.

The correct primitive for a single-writer workload is a StatefulSet, which provides ordered, one-at-a-time replacement by design rather than by remembering to set a field.

Where GitOps stops

The monitoring stack disappeared from its namespace. Argo CD faithfully rebuilt the namespace and the ingress — both declared in Git — and could rebuild nothing else, because Helm-managed resources are rendered at install time and never stored in Git. The ingress pointed at a service that no longer existed, and every request returned 503.

This is a real blind spot in hybrid GitOps and Helm setups, and the mitigations are all trade-offs: let Argo CD own the Helm install, pre-render charts into committed manifests, or accept that Helm-managed applications need a separate recovery path.

The general principle applies well beyond Helm: GitOps protects what is in Git, and nothing else. Application data, uploaded media and database contents are outside that boundary and need their own answer.

Trusting a reverse proxy

The admin panel returned a 302 pointing at itself — an unconditional redirect loop — while the homepage served fine. Same host, same ingress, same pod.

Cloudflare terminates TLS at the edge and the tunnel forwards plain HTTP to ingress-nginx, so the application saw an HTTP request for a site configured as HTTPS and redirected — forever. The obvious fix, trusting the X-Forwarded-Proto header, did not work: ingress-nginx discards inbound forwarded headers by default and regenerates them from the actual connection.

That default is a security measure, not a bug. X-Forwarded-* headers are client-controlled and must not be trusted unless a proxy you control sits in front. Writing code that depends on a header without verifying the proxy chain forwards it is an assumption, not a design.

Fixed at the application layer rather than by enabling forwarded headers cluster-wide — a one-line change beats one that silently alters header handling for every other service in the cluster. Match the fix’s blast radius to the problem’s.

Day-2 operations

  • Resource sizing from measurement. Limits were raised against observed usage and real node headroom, not guesswork — and the application-level limits (PHP’s own memory ceiling) had to move with the container limits, or the extra headroom would never have been used.
  • Monitoring that is actually read. Prometheus, Grafana and Alertmanager, tuned down after learning first-hand what alert fatigue does to your attention. An alert nobody reads is worse than no alert.
  • Backups with a verified restore. Nightly encrypted snapshots to off-cluster storage, client-side encrypted so the backup host never holds readable data. The restore has been performed and checked against live row counts — and the drill is designed never to write to the live database, so it is safe to repeat.
  • Probes matched to reality. TCP rather than HTTP readiness for an application that returns 500 during first-run setup, because an HTTP probe would block it from ever becoming ready.

Known limitations

Stating these matters more than hiding them. Each is a deliberate trade-off with a known cost:

  • Single control plane. No etcd quorum and no HA. Losing that node loses the cluster’s API until it is rebuilt. Acceptable for this workload; unacceptable for anything with an availability target.
  • local-path enforces no quota. The volume request is decorative — it binds to a directory on the node’s disk with no limit, so a runaway upload can fill the node’s root filesystem and take the node with it.
  • Node-local storage, no replication. A disk failure on the wrong node loses that volume. Backups cover the data; the outage is still real.
  • Backups are off-cluster but not off-site. They land on the host running the VMs, which covers node failure and accidental deletion but not the loss of that machine.
  • Secrets live in environment files. Fine for one operator; it does not scale to a team or an audit trail. Migrating to a secrets manager with an external secrets operator is the intended path.

Full write-ups of every failure — symptom, diagnostic commands, wrong turns and root cause — are published in Lab Notes. The source for the platform is on GitHub.