Stage 4: Graduating to Terraform/OpenTofu

Stages 2-3 got you a real, working cluster with Flux managing every app in this repo — on one hand-built VM. This stage replaces that VM with a proper 3-node cluster provisioned by Terraform/OpenTofu via the dmacvicar/libvirt provider, then points the exact same Flux config at it. Nothing in clusters/homelab/ or apps/ changes — that's the point: this repo's GitOps state was never tied to how the cluster under it got built.

You'll need docs/01-bootstrap.md steps 3-4: the k3s_token secret (generated but unused until now) and the k8s-readonly token.

Where this track actually stands

Earlier notes on this project flagged this whole track as "blocked" on a provider schema mismatch between the 0.8.x and 0.9.x lines of dmacvicar/libvirt, without pinning down exactly what broke. That's worth re-examining rather than taking on faith, both because it matters for whether you should trust main.tf and because the process of checking it is itself a reasonable thing to learn from — so here's what's actually been confirmed, and what hasn't, as of this pass:

Confirmed: this isn't really a "0.8.x vs 0.9.x" ambiguity at all. terraform/versions.tf pins ~> 0.8. Checking the provider's own release history: v0.9.0 (Nov 2025) was an intentional, permanent fork to a fully regenerated schema that maps 1:1 onto libvirt's XML — the maintainer's own release notes say so explicitly, and describe keeping 0.8.x alive in parallel specifically for people who don't want that rewrite. So ~> 0.8 doesn't risk drifting onto 0.9.x schema by accident; it's a deliberate, stable choice, not an unresolved question.

Confirmed: every resource in main.tf matches the real 0.8.x schema. Checked directly against this provider's own docs at git tag v0.8.3 (website/docs/r/{pool,volume,cloudinit,domain,network}.html.markdown in dmacvicar/terraform-provider-libvirt) — not from memory, and not from whatever an LLM's training data assumes a "libvirt provider" looks like, which is the trap the project's own earlier notes were rightly worried about. Every attribute main.tf uses lines up: libvirt_pool's type = "dir"; libvirt_volume's base_volume_id/size/format; libvirt_cloudinit_disk's user_data/network_config; libvirt_domain's disk { volume_id }, network_interface { network_id, wait_for_lease }, and console { type, target_type, target_port }; libvirt_network's mode/addresses/dhcp { enabled }/dns { enabled }. None of it uses 0.9.x-only shapes (nested create.content.url, backing_store, capacity instead of size, etc.).

Not yet confirmed: whether it actually applies. Static schema-matching isn't the same as a real tofu apply succeeding — that needs the T630's actual libvirt socket, which nothing has exercised end-to-end yet. If you're picking this stage up, this is the real remaining unknown, and a reasonable order to close it:

Free, no-VM checkpoint first: .forgejo/workflows/terraform.yml already runs tofu init -backend=false + tofu validate on every push to terraform/** — that's a real, automated check of exactly the schema question above, running today. Check its latest result in Forgejo Actions before doing anything else; if it's failing, the error message will point at a specific resource/attribute far faster than re-deriving the whole schema by hand.
tofu init on the T630 (step 2 below) and check terraform/.terraform.lock.hcl afterwards — confirm it actually resolved a 0.8.x version, not something unexpected.
tofu plan, then tofu apply, and if any single resource fails, treat that resource in isolation: re-check its specific arguments against website/docs/r/<resource>.html.markdown at whatever version .terraform.lock.hcl actually resolved (not main — the docs move with the schema, and main may already reflect a newer 0.8.x patch or even post-fork changes), rather than guessing at a fix. That's a more targeted version of the same check already done above for the whole file.
Once a full tofu apply succeeds once, the remaining steps below are what actually plug the result into the rest of this project.

Sizing is deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total) so this stays a demo rather than competing with whatever else is already running on the T630.

1. Install OpenTofu

Installing the package needs sudo, so that part is you (maq), not k8s. Installing system-wide (/usr/local/bin) means k8s can just use tofu afterwards with no further root involvement:

# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh

2. Provision the VMs with OpenTofu

Everything from here on is k8s again (sudo -iu k8s), no sudo involved — clone using the k8s-readonly token from docs/01-bootstrap.md step 4. k8s has no keyring (it's headless, no desktop session), so this uses git credential-store — a plaintext file, chmod 600'd, holding only the read-only token:

# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials

cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from docs/01-bootstrap.md step 2), k3s_token (from step 3). Defaults for
# network/sizing are fine to start.

tofu init
tofu plan
tofu apply

For any later change to terraform/: edit and push as maq as usual, then cd ~/k3s && git pull as k8s before re-running tofu plan/apply.

This brings up k3s-server-1, k3s-agent-1, k3s-agent-2 on the k3s-homelab libvirt network (10.20.30.0/24 by default — isolated from anything else already using libvirt on this box, including the default network stage 2's manual VM used). Cloud-init installs k3s on each on first boot — give it ~2 minutes after apply finishes.

3. Get kubectl talking to the new cluster

mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab

export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes   # expect 3 Ready nodes

This works as-is because you're running it on the T630, which can reach the 10.20.30.0/24 network directly. To also use kubectl from your own laptop, either scp this kubeconfig over and open an SSH tunnel first (ssh -L 6443:10.20.30.11:6443 k8s@t630, then point the kubeconfig's server: at https://127.0.0.1:6443), or just SSH into the T630 as k8s whenever you need kubectl — simplest by far for a project this size.

4. Re-bootstrap Flux against the new cluster

This is the exact same flux bootstrap git command from docs/03-flux.md step 2, just pointed at ~/.kube/config-homelab instead of ~/.kube/config-manual — and it needs to actually run again, not be skipped. The new 3-node cluster has its own fresh etcd/SQLite; nothing about the manual VM's cluster state carries over to it, flux-system namespace included. What does carry over is this repo:

flux check --pre --kubeconfig ~/.kube/config-homelab

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-homelab

Because clusters/homelab/flux-system/ already holds the exact manifests stage 3's bootstrap generated, this run doesn't need to commit anything new back to the repo — it just applies that already-correct config to a cluster that doesn't have it yet. That's the whole point of GitOps having been the deploy mechanism all along: the desired state was never tied to the specific cluster instance, so pointing the same bootstrap command at a new kubeconfig reproduces it exactly. Confirm:

flux get kustomizations --watch
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods

5. kubectl access from elsewhere on the LAN, or remotely

Best practice for the Kubernetes API server is the same principle as Headlamp (docs/03-flux.md step 5): never put 6443 on the public internet if you can avoid it, because a leaked credential there is a full cluster compromise.

Node IPs (10.20.30.0/24 by default) live on the private libvirt network from step 2 — only the T630 itself can reach them directly, which is actually a nice side effect: even the rest of your LAN can't touch the API server without going through the T630 first. Two ways to do that:

SSH into the T630 as k8s and run kubectl there directly (same as step 3) — simplest, and what this whole project assumes by default.
Tunnel from another machine (your laptop, or a phone via Termux, etc.):
```
ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
```
then point a local kubeconfig's server: at https://127.0.0.1:6443 (copy the kubeconfig from step 3 and edit that one field). The cert validates because k8s-api.boglabob.com is in the server's TLS SAN list (terraform/variables.tf's k8s_api_hostname) — add it to /etc/hosts as 127.0.0.1 k8s-api.boglabob.com on whatever machine you're tunneling from and use that as the server: host instead of the raw IP, so the hostname in the URL matches a name the cert actually covers. (This SAN entry is new in stage 4 — the manual VM's cloud-init in docs/02-k3s.md never set one, since nothing needed LAN-wide access to it.)
From outside your home network entirely: Tailscale or WireGuard on the T630, then the SSH tunnel above over the Tailscale/WireGuard link instead of the open internet. Reasonable next stretch goal once the core loop is working — don't port-forward 22 or 6443 on your router for this.

The admin-user bearer token (docs/03-flux.md step 5) also works for kubectl over the same tunnel, if you'd rather not manage the client-cert kubeconfig.

6. Point Caddy and Headlamp at the new cluster

Same steps as docs/03-flux.md step 8 (Caddy) and step 5 (Headlamp) — the mechanism is identical, only the node IP(s) changed. Caddy now has three node IPs to pick from instead of one, so use docs/Caddyfile.example's reverse_proxy blocks (which list all three) rather than the single-IP version from stage 3:

tofu output node_ips

podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile

Headlamp's port-forward command is unchanged (kubectl just needs KUBECONFIG pointed at ~/.kube/config-homelab now).

7. Tearing down the manual VM

Now that the real 3-node cluster is up, remove stage 2's throwaway one. scripts/cleanup-manual-vm.sh runs the commands below (with a confirmation prompt first, since it's destructive) — worth reading through once so you know what it's doing before you run it unattended:

virsh -c qemu:///system destroy k3s-manual              # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage   # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2   # base image isn't
                                                                            # attached to the VM
                                                                            # directly, needs its
                                                                            # own delete
rm ~/.kube/config-manual
rm -rf ~/vms

Nothing else needs cleaning up — Flux's own state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, both tokens, and the Forgejo Actions runner registration are all cluster-independent and already carried over unchanged in step 4 above.

Stretch goals, roughly in order

Remote access: Tailscale or WireGuard on the k3s server node, for kubectl/Headlamp access from outside the LAN without opening anything publicly (step 5).
TLS: cert-manager + a ClusterIssuer for Let's Encrypt (DNS-01 if boglabob.com isn't publicly reachable on 80/443).
Secrets in Git: sops + age, or sealed-secrets, so the K3S_TOKEN/API tokens above don't need to live only in Forgejo's secret store.
Monitoring: kube-prometheus-stack via Helm, deployed the same way as podinfo (HelmRepository + HelmRelease under apps/).
HA: add a second k3s server node and switch from SQLite to embedded etcd (--cluster-init on the first server, --server join on the second).

13 KiB Raw Blame History