Homelab Platform — Terraform + k3s + Flux on Forgejo

Learning project: provision VMs on the T630 with OpenTofu/Terraform, bootstrap a k3s cluster on them, and manage everything after that point through GitOps (Flux) synced from a Forgejo repo, with Forgejo Actions handling CI. The T630 keeps running its other self-hosted services throughout — this installs as ordinary KVM/libvirt packages next to them, sized deliberately small (3 VMs, 2 vCPU/2GB RAM each), not a hypervisor OS taking over the box.

A four-stage tutorial, each stage building on the last:

• docs/01-bootstrap.md — one-time host setup: KVM/libvirt, the k8s user, pushing this repo to Forgejo.
• docs/02-k3s.md — build one VM by hand, get a real k3s cluster running on it. No Terraform yet.
• docs/03-flux.md — bootstrap Flux against that cluster, deploy every app in apps/ through it (Helm-managed and plain-manifest alike), wire up the hello-app CI/image-automation loop.
• docs/04-tofu.md — graduate from the one hand-built VM to a Terraform/OpenTofu-provisioned 3-node cluster, and point the same Flux config at it. The harder stage — its HCL is schema-verified against the pinned provider version but not yet proven with a real tofu apply (see "Where this track actually stands" near the top of that doc) — worth finishing properly, but don't block on it; stages 2-3 already get you a complete, working GitOps loop without it.

The loop

OpenTofu (terraform/)
   -> provisions VMs on the T630 via KVM/libvirt (qemu:///system)
   -> cloud-init installs k3s (1 server + 2 agents)
   -> VMs live on an isolated private network (10.20.30.0/24), reachable
      only from the T630 itself

Forgejo repo (this repo)
   -> clusters/homelab/  = Flux config (what Flux itself watches)
   -> apps/               = workloads Flux deploys
   -> .forgejo/workflows/ = CI (terraform plan, image builds)

Flux (running in the cluster)
   -> watches this repo
   -> applies clusters/homelab/** and apps/** to k3s

Caddy (Podman, existing) is the public front door for boglabob.com, but only
for the low-stakes apps — Headlamp and the k3s API server stay off it:
   git.boglabob.com     -> Forgejo (existing)
   podinfo.boglabob.com -> Traefik (k3s ingress) -> podinfo
   hello.boglabob.com   -> Traefik (k3s ingress) -> hello-app
   [Headlamp]            -> kubectl port-forward only, never a public hostname
   [k3s API, port 6443]  -> reachable only via the T630 (SSH tunnel or run
                             kubectl there directly), no proxy, no public port
   see docs/Caddyfile.example, docs/03-flux.md step 5 (Headlamp), and
   docs/04-tofu.md step 5 (LAN/remote kubectl access)

Why it's structured this way

Each layer below only knows about the one directly beneath it — that's deliberate, not incidental, and it's what makes the pieces independently replaceable:

• Terraform provisions VMs, nothing more. It doesn't install k3s itself or configure Flux. Its job ends at "three VMs exist, on this network, with this cloud-init data attached." That's what makes docs/02-k3s.md possible at all — a single virt-install VM can stand in for the whole Terraform layer, because everything above it only cares that some VM with a k3s user and a working kubeconfig exists, not how it got there — which is exactly why Terraform is stage 4, not stage 1: everything above it gets learned and proven out on one hand-built VM first.
• cloud-init installs k3s, nothing about Flux. It runs get.k3s.io and stages a kubeconfig — full stop. Flux isn't baked into the image or the cloud-init data because flux bootstrap needs to commit into this repo (the generated clusters/homelab/flux-system/ manifests), which only makes sense as a deliberate, one-time, human-run command against a cluster that already exists and can reach Forgejo — not something to automate blindly on every VM boot.
• flux bootstrap is a one-time bridge, not a recurring step. It's the only command in this whole project that goes API-server-side and repo-side in one shot: it installs the controllers, and it writes clusters/homelab/flux-system/ back into git. After that one run, the cluster is entirely git-driven — rerunning flux bootstrap against the same repo is idempotent (safe if you need to recover a cluster, or to point the same config at a new cluster — see docs/04-tofu.md step 4), but nothing after that first run ever needs a human to run kubectl apply again.
• clusters/homelab/apps.yaml fans out on purpose, instead of one giant Kustomization pointing at apps/. Each app gets its own Flux Kustomization object (see clusters/homelab/apps.yaml) with its own interval and its own prune: true. That means a bad manifest in apps/hello-app can't block apps/podinfo from reconciling — each app's sync loop is independent, and you can watch/debug them individually with flux get kustomization <name> instead of one opaque blob.
• Helm apps (podinfo, headlamp) vs. plain manifests (hello-app). podinfo and headlamp are third-party charts — Flux's HelmRepository + HelmRelease pair is what lets Flux track and upgrade an upstream chart without this repo vendoring it. hello-app is this project's own code, so it's plain Kubernetes YAML (Deployment/Service/Ingress) — there's no chart to track, and plain manifests are simpler to diff/read for something this small.
• Image automation exists only for hello-app, not podinfo/headlamp. ImageRepository/ImagePolicy/ImageUpdateAutomation (apps/hello-app/image-automation.yaml) close the loop from "CI pushed a new image tag" to "the cluster is running it" — that's only relevant for an app whose image this repo's own CI builds. podinfo/headlamp pin a chart version range instead (>=6.0.0, >=0.40.0) and get upgraded by bumping that constraint by hand, since there's no CI producing new tags for them to watch.

Directory layout

• terraform/ — OpenTofu config that provisions the k3s VMs via KVM/libvirt directly on the T630 (no separate hypervisor OS, no VM template step — the cloud image is pulled straight from its URL).
• terraform/cloud-init/ — cloud-init templates that install k3s server/agent on first boot.
• clusters/homelab/ — Flux's own config for this cluster (populated by flux bootstrap, see docs/03-flux.md step 2).
• apps/podinfo/ — first GitOps app: a HelmRelease for the standard Flux demo app (podinfo). No CI needed — good for validating the Flux sync loop works.
• apps/hello-app/ — capstone app, and the public showcase piece at hello.boglabob.com: a one-page site (source + Dockerfile) built by Forgejo Actions, pushed to Forgejo's container registry, deployed via a Flux ImagePolicy/ImageUpdateAutomation so new pushes to main roll out automatically. The page itself shows the live commit SHA and deploy timestamp injected by CI, as proof the pipeline is really running rather than a static screenshot.
• apps/headlamp/ — a Kubernetes UI (replacing the now-unmaintained Kubernetes Dashboard), with a cluster-admin token (apps/headlamp/rbac.yaml). No ingress — access is via kubectl port-forward only (see docs/03-flux.md step 5).
• .forgejo/workflows/ — CI: terraform validate/plan on PRs, build+push hello-app image on merge to main, via kaniko under a rootless Podman runner (no docker.sock, no sudo — see docs/03-flux.md step 6).
• docs/Caddyfile.example — the reverse-proxy blocks for podinfo/hello-app to add to your existing Caddy (Podman) config.

Prerequisites (see docs/01-bootstrap.md onward for the full walkthrough)

• qemu-kvm/libvirt installed on the T630 (alongside its existing services)
• Forgejo instance reachable, with this repo pushed to it and Actions enabled
• kubectl, flux, tofu CLI — run on the T630 itself as the unprivileged k8s user (simplest, since the VMs' network is only reachable from there)

Milestones

1. KVM/libvirt installed on the T630, k8s user created (docs/01-bootstrap.md)
2. One hand-built VM, k3s running on it, kubectl talking to it (docs/02-k3s.md)
3. flux bootstrap against this Forgejo repo — podinfo/headlamp/hello-app deploy with no manual kubectl apply (docs/03-flux.md)
4. Wire .forgejo/workflows/build-hello-app.yml — push a code change to apps/hello-app/src, watch CI build → registry → Flux auto-deploy (docs/03-flux.md step 7)
5. tofu apply — 3 VMs come up, k3s cluster forms, same Flux config re-bootstrapped against it (docs/04-tofu.md)
6. Stretch: sealed-secrets/SOPS, kube-prometheus-stack, cert-manager