# Stage 4: Graduating to Terraform/OpenTofu Stages 2-3 got you a real, working cluster with Flux managing every app in this repo — on one hand-built VM. This stage replaces that VM with a proper 3-node cluster provisioned by Terraform/OpenTofu via the `dmacvicar/libvirt` provider, then points the exact same Flux config at it. Nothing in `clusters/homelab/` or `apps/` changes — that's the point: this repo's GitOps state was never tied to *how* the cluster under it got built. You'll need `docs/01-bootstrap.md` steps 3-4: the `k3s_token` secret (generated but unused until now) and the `k8s-readonly` token. ## Where this track actually stands Earlier notes on this project flagged this whole track as "blocked" on a provider schema mismatch between the 0.8.x and 0.9.x lines of `dmacvicar/libvirt`, without pinning down exactly what broke. That's worth re-examining rather than taking on faith, both because it matters for whether you should trust `main.tf` and because the process of checking it is itself a reasonable thing to learn from — so here's what's actually been confirmed, and what hasn't, as of this pass: **Confirmed: this isn't really a "0.8.x vs 0.9.x" ambiguity at all.** `terraform/versions.tf` pins `~> 0.8`. Checking the provider's own release history: v0.9.0 (Nov 2025) was an intentional, permanent fork to a fully regenerated schema that maps 1:1 onto libvirt's XML — the maintainer's own release notes say so explicitly, and describe keeping 0.8.x alive in parallel specifically for people who don't want that rewrite. So `~> 0.8` doesn't risk drifting onto 0.9.x schema by accident; it's a deliberate, stable choice, not an unresolved question. **Confirmed: every resource in `main.tf` matches the real 0.8.x schema.** Checked directly against this provider's own docs at git tag `v0.8.3` (`website/docs/r/{pool,volume,cloudinit,domain,network}.html.markdown` in `dmacvicar/terraform-provider-libvirt`) — not from memory, and not from whatever an LLM's training data assumes a "libvirt provider" looks like, which is the trap the project's own earlier notes were rightly worried about. Every attribute `main.tf` uses lines up: `libvirt_pool`'s `type = "dir"`; `libvirt_volume`'s `base_volume_id`/`size`/`format`; `libvirt_cloudinit_disk`'s `user_data`/`network_config`; `libvirt_domain`'s `disk { volume_id }`, `network_interface { network_id, wait_for_lease }`, and `console { type, target_type, target_port }`; `libvirt_network`'s `mode`/`addresses`/`dhcp { enabled }`/`dns { enabled }`. None of it uses 0.9.x-only shapes (nested `create.content.url`, `backing_store`, `capacity` instead of `size`, etc.). **Not yet confirmed: whether it actually applies.** Static schema-matching isn't the same as a real `tofu apply` succeeding — that needs the T630's actual libvirt socket, which nothing has exercised end-to-end yet. If you're picking this stage up, this is the real remaining unknown, and a reasonable order to close it: 1. **Free, no-VM checkpoint first**: `.forgejo/workflows/terraform.yml` already runs `tofu init -backend=false` + `tofu validate` on every push to `terraform/**` — that's a real, automated check of exactly the schema question above, running today. Check its latest result in Forgejo Actions before doing anything else; if it's failing, the error message will point at a specific resource/attribute far faster than re-deriving the whole schema by hand. 2. **`tofu init` on the T630** (step 2 below) and check `terraform/.terraform.lock.hcl` afterwards — confirm it actually resolved a `0.8.x` version, not something unexpected. 3. **`tofu plan`, then `tofu apply`**, and if any single resource fails, treat that resource in isolation: re-check its specific arguments against `website/docs/r/.html.markdown` at whatever version `.terraform.lock.hcl` actually resolved (not `main` — the docs move with the schema, and `main` may already reflect a newer 0.8.x patch or even post-fork changes), rather than guessing at a fix. That's a more targeted version of the same check already done above for the whole file. 4. Once a full `tofu apply` succeeds once, the remaining steps below are what actually plug the result into the rest of this project. Sizing is deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total) so this stays a demo rather than competing with whatever else is already running on the T630. --- ## 1. Install OpenTofu Installing the package needs sudo, so that part is you (`maq`), not `k8s`. Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu` afterwards with no further root involvement: ```sh # as maq (has sudo) sudo apt install -y unzip curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh ``` ## 2. Provision the VMs with OpenTofu Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved — clone using the `k8s-readonly` token from `docs/01-bootstrap.md` step 4. `k8s` has no keyring (it's headless, no desktop session), so this uses `git credential-store` — a plaintext file, `chmod 600`'d, holding only the read-only token: ```sh # as k8s git config --global credential.helper store git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s # prompts for username (anything) and password (paste the k8s-readonly # token) once; stores it in ~/.git-credentials for next time chmod 600 ~/.git-credentials cd ~/k3s/terraform cp terraform.tfvars.example terraform.tfvars # edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub # from docs/01-bootstrap.md step 2), k3s_token (from step 3). Defaults for # network/sizing are fine to start. tofu init tofu plan tofu apply ``` For any later change to `terraform/`: edit and push as `maq` as usual, then `cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`. This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the `k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from anything else already using libvirt on this box, including the `default` network stage 2's manual VM used). Cloud-init installs k3s on each on first boot — give it ~2 minutes after `apply` finishes. ## 3. Get kubectl talking to the new cluster ```sh mkdir -p ~/.kube ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \ | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab export KUBECONFIG=~/.kube/config-homelab kubectl get nodes # expect 3 Ready nodes ``` This works as-is because you're running it on the T630, which can reach the `10.20.30.0/24` network directly. To also use kubectl from your own laptop, either `scp` this kubeconfig over and open an SSH tunnel first (`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's `server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s` whenever you need kubectl — simplest by far for a project this size. ## 4. Re-bootstrap Flux against the new cluster This is the exact same `flux bootstrap git` command from `docs/03-flux.md` step 2, just pointed at `~/.kube/config-homelab` instead of `~/.kube/config-manual` — and it needs to actually run again, not be skipped. The new 3-node cluster has its own fresh etcd/SQLite; nothing about the manual VM's cluster state carries over to it, `flux-system` namespace included. What *does* carry over is this repo: ```sh flux check --pre --kubeconfig ~/.kube/config-homelab flux bootstrap git \ --url=https://git.boglabob.com/codegit/cloud-demo \ --branch=main \ --path=clusters/homelab \ --username=codegit \ --password= \ --token-auth \ --kubeconfig ~/.kube/config-homelab ``` Because `clusters/homelab/flux-system/` already holds the exact manifests stage 3's bootstrap generated, this run doesn't need to commit anything new back to the repo — it just applies that already-correct config to a cluster that doesn't have it yet. That's the whole point of GitOps having been the deploy mechanism all along: the desired state was never tied to the specific cluster instance, so pointing the same bootstrap command at a new kubeconfig reproduces it exactly. Confirm: ```sh flux get kustomizations --watch kubectl -n podinfo get pods kubectl -n hello-app get pods kubectl -n headlamp get pods ``` ## 5. kubectl access from elsewhere on the LAN, or remotely Best practice for the Kubernetes API server is the same principle as Headlamp (`docs/03-flux.md` step 5): never put 6443 on the public internet if you can avoid it, because a leaked credential there is a full cluster compromise. Node IPs (`10.20.30.0/24` by default) live on the private libvirt network from step 2 — only the T630 itself can reach them directly, which is actually a nice side effect: even the rest of your LAN can't touch the API server without going through the T630 first. Two ways to do that: - **SSH into the T630 as `k8s`** and run kubectl there directly (same as step 3) — simplest, and what this whole project assumes by default. - **Tunnel from another machine** (your laptop, or a phone via Termux, etc.): ```sh ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630 ``` then point a local kubeconfig's `server:` at `https://127.0.0.1:6443` (copy the kubeconfig from step 3 and edit that one field). The cert validates because `k8s-api.boglabob.com` is in the server's TLS SAN list (`terraform/variables.tf`'s `k8s_api_hostname`) — add it to `/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine you're tunneling from and use that as the `server:` host instead of the raw IP, so the hostname in the URL matches a name the cert actually covers. (This SAN entry is new in stage 4 — the manual VM's cloud-init in `docs/02-k3s.md` never set one, since nothing needed LAN-wide access to it.) - **From outside your home network entirely**: Tailscale or WireGuard on the T630, then the SSH tunnel above over the Tailscale/WireGuard link instead of the open internet. Reasonable next stretch goal once the core loop is working — don't port-forward 22 or 6443 on your router for this. The `admin-user` bearer token (`docs/03-flux.md` step 5) also works for kubectl over the same tunnel, if you'd rather not manage the client-cert kubeconfig. ## 6. Point Caddy and Headlamp at the new cluster Same steps as `docs/03-flux.md` step 8 (Caddy) and step 5 (Headlamp) — the mechanism is identical, only the node IP(s) changed. Caddy now has three node IPs to pick from instead of one, so use `docs/Caddyfile.example`'s `reverse_proxy` blocks (which list all three) rather than the single-IP version from stage 3: ```sh tofu output node_ips ``` ```sh podman exec caddy reload --config /etc/caddy/Caddyfile ``` Headlamp's port-forward command is unchanged (`kubectl` just needs `KUBECONFIG` pointed at `~/.kube/config-homelab` now). ## 7. Tearing down the manual VM Now that the real 3-node cluster is up, remove stage 2's throwaway one. `scripts/cleanup-manual-vm.sh` runs the commands below (with a confirmation prompt first, since it's destructive) — worth reading through once so you know what it's doing before you run it unattended: ```sh virsh -c qemu:///system destroy k3s-manual # stop it virsh -c qemu:///system undefine k3s-manual --remove-all-storage # VM + overlay disk virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2 # base image isn't # attached to the VM # directly, needs its # own delete rm ~/.kube/config-manual rm -rf ~/vms ``` Nothing else needs cleaning up — Flux's own state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, both tokens, and the Forgejo Actions runner registration are all cluster-independent and already carried over unchanged in step 4 above. ## Stretch goals, roughly in order - **Remote access**: Tailscale or WireGuard on the k3s server node, for kubectl/Headlamp access from outside the LAN without opening anything publicly (step 5). - **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if `boglabob.com` isn't publicly reachable on 80/443). - **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the `K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret store. - **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as podinfo (HelmRepository + HelmRelease under `apps/`). - **HA**: add a second k3s server node and switch from SQLite to embedded etcd (`--cluster-init` on the first server, `--server` join on the second).