cloud-demo/docs/04-tofu.md

Stage 4: Graduating to Terraform/OpenTofu

Stages 2-3 got you a real, working cluster with Flux managing every app in this repo — on one hand-built VM. This stage replaces that VM with a proper 3-node cluster provisioned by Terraform/OpenTofu via the dmacvicar/libvirt provider, then points the exact same Flux config at it. Nothing in clusters/homelab/ or apps/ changes — that's the point: this repo's GitOps state was never tied to how the cluster under it got built.

You'll need docs/01-bootstrap.md steps 3-4: the k3s_token secret (generated but unused until now) and the k8s-readonly token.

Where this track actually stands

Earlier notes on this project flagged this whole track as "blocked" on a provider schema mismatch between the 0.8.x and 0.9.x lines of dmacvicar/libvirt, without pinning down exactly what broke. That's worth re-examining rather than taking on faith, both because it matters for whether you should trust main.tf and because the process of checking it is itself a reasonable thing to learn from — so here's what's actually been confirmed, and what hasn't, as of this pass:

Confirmed: this isn't really a "0.8.x vs 0.9.x" ambiguity at all. terraform/versions.tf pins ~> 0.8. Checking the provider's own release history: v0.9.0 (Nov 2025) was an intentional, permanent fork to a fully regenerated schema that maps 1:1 onto libvirt's XML — the maintainer's own release notes say so explicitly, and describe keeping 0.8.x alive in parallel specifically for people who don't want that rewrite. So ~> 0.8 doesn't risk drifting onto 0.9.x schema by accident; it's a deliberate, stable choice, not an unresolved question.

Confirmed: every resource in main.tf matches the real 0.8.x schema. Checked directly against this provider's own docs at git tag v0.8.3 (website/docs/r/{pool,volume,cloudinit,domain,network}.html.markdown in dmacvicar/terraform-provider-libvirt) — not from memory, and not from whatever an LLM's training data assumes a "libvirt provider" looks like, which is the trap the project's own earlier notes were rightly worried about. Every attribute main.tf uses lines up: libvirt_pool's type = "dir"; libvirt_volume's base_volume_id/size/format; libvirt_cloudinit_disk's user_data/network_config; libvirt_domain's disk { volume_id }, network_interface { network_id, wait_for_lease }, and console { type, target_type, target_port }; libvirt_network's mode/addresses/dhcp { enabled }/dns { enabled }. None of it uses 0.9.x-only shapes (nested create.content.url, backing_store, capacity instead of size, etc.).

Not yet confirmed: whether it actually applies. Static schema-matching isn't the same as a real tofu apply succeeding — that needs the T630's actual libvirt socket, which nothing has exercised end-to-end yet. If you're picking this stage up, this is the real remaining unknown, and a reasonable order to close it:

1. Free, no-VM checkpoint first: .forgejo/workflows/terraform.yml already runs tofu init -backend=false + tofu validate on every push to terraform/** — that's a real, automated check of exactly the schema question above, running today. Check its latest result in Forgejo Actions before doing anything else; if it's failing, the error message will point at a specific resource/attribute far faster than re-deriving the whole schema by hand.
2. tofu init on the T630 (step 2 below) and check terraform/.terraform.lock.hcl afterwards — confirm it actually resolved a 0.8.x version, not something unexpected.
3. tofu plan, then tofu apply, and if any single resource fails, treat that resource in isolation: re-check its specific arguments against website/docs/r/<resource>.html.markdown at whatever version .terraform.lock.hcl actually resolved (not main — the docs move with the schema, and main may already reflect a newer 0.8.x patch or even post-fork changes), rather than guessing at a fix. That's a more targeted version of the same check already done above for the whole file.
4. Once a full tofu apply succeeds once, the remaining steps below are what actually plug the result into the rest of this project.

Sizing is deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total) so this stays a demo rather than competing with whatever else is already running on the T630.

---

1. Install OpenTofu

Installing the package needs sudo, so that part is you (maq), not k8s. Installing system-wide (/usr/local/bin) means k8s can just use tofu afterwards with no further root involvement:

# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh

2. Provision the VMs with OpenTofu

Everything from here on is k8s again (sudo -iu k8s), no sudo involved — clone using the k8s-readonly token from docs/01-bootstrap.md step 4. k8s has no keyring (it's headless, no desktop session), so this uses git credential-store — a plaintext file, chmod 600'd, holding only the read-only token:

# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials

cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from docs/01-bootstrap.md step 2), k3s_token (from step 3). Defaults for
# network/sizing are fine to start.

tofu init
tofu plan
tofu apply

For any later change to terraform/: edit and push as maq as usual, then cd ~/k3s && git pull as k8s before re-running tofu plan/apply.

This brings up k3s-server-1, k3s-agent-1, k3s-agent-2 on the k3s-homelab libvirt network (10.20.30.0/24 by default — isolated from anything else already using libvirt on this box, including the default network stage 2's manual VM used). Cloud-init installs k3s on each on first boot — give it ~2 minutes after apply finishes.

3. Get kubectl talking to the new cluster

mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab

export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes   # expect 3 Ready nodes

This works as-is because you're running it on the T630, which can reach the 10.20.30.0/24 network directly. To also use kubectl from your own laptop, either scp this kubeconfig over and open an SSH tunnel first (ssh -L 6443:10.20.30.11:6443 k8s@t630, then point the kubeconfig's server: at https://127.0.0.1:6443), or just SSH into the T630 as k8s whenever you need kubectl — simplest by far for a project this size.

4. Re-bootstrap Flux against the new cluster

This is the exact same flux bootstrap git command from docs/03-flux.md step 2, just pointed at ~/.kube/config-homelab instead of ~/.kube/config-manual — and it needs to actually run again, not be skipped. The new 3-node cluster has its own fresh etcd/SQLite; nothing about the manual VM's cluster state carries over to it, flux-system namespace included. What does carry over is this repo:

flux check --pre --kubeconfig ~/.kube/config-homelab

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-homelab

Because clusters/homelab/flux-system/ already holds the exact manifests stage 3's bootstrap generated, this run doesn't need to commit anything new back to the repo — it just applies that already-correct config to a cluster that doesn't have it yet. That's the whole point of GitOps having been the deploy mechanism all along: the desired state was never tied to the specific cluster instance, so pointing the same bootstrap command at a new kubeconfig reproduces it exactly. Confirm:

flux get kustomizations --watch
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods

5. kubectl access from elsewhere on the LAN, or remotely

Best practice for the Kubernetes API server is the same principle as Headlamp (docs/03-flux.md step 5): never put 6443 on the public internet if you can avoid it, because a leaked credential there is a full cluster compromise.

Node IPs (10.20.30.0/24 by default) live on the private libvirt network from step 2 — only the T630 itself can reach them directly, which is actually a nice side effect: even the rest of your LAN can't touch the API server without going through the T630 first. Two ways to do that:

• SSH into the T630 as k8s and run kubectl there directly (same as step 3) — simplest, and what this whole project assumes by default.
• Tunnel from another machine (your laptop, or a phone via Termux, etc.):

  ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
  

  then point a local kubeconfig's server: at https://127.0.0.1:6443 (copy the kubeconfig from step 3 and edit that one field). The cert validates because k8s-api.boglabob.com is in the server's TLS SAN list (terraform/variables.tf's k8s_api_hostname) — add it to /etc/hosts as 127.0.0.1 k8s-api.boglabob.com on whatever machine you're tunneling from and use that as the server: host instead of the raw IP, so the hostname in the URL matches a name the cert actually covers. (This SAN entry is new in stage 4 — the manual VM's cloud-init in docs/02-k3s.md never set one, since nothing needed LAN-wide access to it.)
• From outside your home network entirely: Tailscale or WireGuard on the T630, then the SSH tunnel above over the Tailscale/WireGuard link instead of the open internet. Reasonable next stretch goal once the core loop is working — don't port-forward 22 or 6443 on your router for this.

The admin-user bearer token (docs/03-flux.md step 5) also works for kubectl over the same tunnel, if you'd rather not manage the client-cert kubeconfig.

6. Point Caddy and Headlamp at the new cluster

Same steps as docs/03-flux.md step 8 (Caddy) and step 5 (Headlamp) — the mechanism is identical, only the node IP(s) changed. Caddy now has three node IPs to pick from instead of one, so use docs/Caddyfile.example's reverse_proxy blocks (which list all three) rather than the single-IP version from stage 3:

tofu output node_ips

podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile

Headlamp's port-forward command is unchanged (kubectl just needs KUBECONFIG pointed at ~/.kube/config-homelab now).

7. Tearing down the manual VM

Now that the real 3-node cluster is up, remove stage 2's throwaway one:

virsh -c qemu:///system destroy k3s-manual              # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage   # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2   # base image isn't
                                                                            # attached to the VM
                                                                            # directly, needs its
                                                                            # own delete
rm ~/.kube/config-manual
rm -rf ~/vms

Nothing else needs cleaning up — Flux's own state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, both tokens, and the Forgejo Actions runner registration are all cluster-independent and already carried over unchanged in step 4 above.

Stretch goals, roughly in order

• Remote access: Tailscale or WireGuard on the k3s server node, for kubectl/Headlamp access from outside the LAN without opening anything publicly (step 5).
• TLS: cert-manager + a ClusterIssuer for Let's Encrypt (DNS-01 if boglabob.com isn't publicly reachable on 80/443).
• Secrets in Git: sops + age, or sealed-secrets, so the K3S_TOKEN/API tokens above don't need to live only in Forgejo's secret store.
• Monitoring: kube-prometheus-stack via Helm, deployed the same way as podinfo (HelmRepository + HelmRelease under apps/).
• HA: add a second k3s server node and switch from SQLite to embedded etcd (--cluster-init on the first server, --server join on the second).