cloud-demo/docs/04-tofu.md

# Stage 4: Graduating to Terraform/OpenTofu

Stages 2-3 got you a real, working cluster with Flux managing every app in
this repo — on one hand-built VM. This stage replaces that VM with a
proper 3-node cluster provisioned by Terraform/OpenTofu via the
`dmacvicar/libvirt` provider, then points the exact same Flux config at
it. Nothing in `clusters/homelab/` or `apps/` changes — that's the point:
this repo's GitOps state was never tied to *how* the cluster under it got
built.

You'll need `docs/01-bootstrap.md` steps 3-4: the `k3s_token` secret
(generated but unused until now) and the `k8s-readonly` token.

## Where this track actually stands

Earlier notes on this project flagged this whole track as "blocked" on a
provider schema mismatch between the 0.8.x and 0.9.x lines of
`dmacvicar/libvirt`, without pinning down exactly what broke. That's worth
re-examining rather than taking on faith, both because it matters for
whether you should trust `main.tf` and because the process of checking it
is itself a reasonable thing to learn from — so here's what's actually
been confirmed, and what hasn't, as of this pass:

**Confirmed: this isn't really a "0.8.x vs 0.9.x" ambiguity at all.**
`terraform/versions.tf` pins `~> 0.8`. Checking the provider's own release
history: v0.9.0 (Nov 2025) was an intentional, permanent fork to a fully
regenerated schema that maps 1:1 onto libvirt's XML — the maintainer's own
release notes say so explicitly, and describe keeping 0.8.x alive in
parallel specifically for people who don't want that rewrite. So `~> 0.8`
doesn't risk drifting onto 0.9.x schema by accident; it's a deliberate,
stable choice, not an unresolved question.

**Confirmed: every resource in `main.tf` matches the real 0.8.x schema.**
Checked directly against this provider's own docs at git tag `v0.8.3`
(`website/docs/r/{pool,volume,cloudinit,domain,network}.html.markdown` in
`dmacvicar/terraform-provider-libvirt`) — not from memory, and not from
whatever an LLM's training data assumes a "libvirt provider" looks like,
which is the trap the project's own earlier notes were rightly worried
about. Every attribute `main.tf` uses lines up: `libvirt_pool`'s
`type = "dir"`; `libvirt_volume`'s `base_volume_id`/`size`/`format`;
`libvirt_cloudinit_disk`'s `user_data`/`network_config`;
`libvirt_domain`'s `disk { volume_id }`, `network_interface { network_id,
wait_for_lease }`, and `console { type, target_type, target_port }`;
`libvirt_network`'s `mode`/`addresses`/`dhcp { enabled }`/`dns { enabled }`.
None of it uses 0.9.x-only shapes (nested `create.content.url`,
`backing_store`, `capacity` instead of `size`, etc.).

**Not yet confirmed: whether it actually applies.** Static schema-matching
isn't the same as a real `tofu apply` succeeding — that needs the T630's
actual libvirt socket, which nothing has exercised end-to-end yet. If
you're picking this stage up, this is the real remaining unknown, and a
reasonable order to close it:

1. **Free, no-VM checkpoint first**: `.forgejo/workflows/terraform.yml`
   already runs `tofu init -backend=false` + `tofu validate` on every push
   to `terraform/**` — that's a real, automated check of exactly the
   schema question above, running today. Check its latest result in
   Forgejo Actions before doing anything else; if it's failing, the error
   message will point at a specific resource/attribute far faster than
   re-deriving the whole schema by hand.
2. **`tofu init` on the T630** (step 2 below) and check
   `terraform/.terraform.lock.hcl` afterwards — confirm it actually
   resolved a `0.8.x` version, not something unexpected.
3. **`tofu plan`, then `tofu apply`**, and if any single resource fails,
   treat that resource in isolation: re-check its specific arguments
   against `website/docs/r/<resource>.html.markdown` at whatever version
   `.terraform.lock.hcl` actually resolved (not `main` — the docs move
   with the schema, and `main` may already reflect a newer 0.8.x patch or
   even post-fork changes), rather than guessing at a fix. That's a more
   targeted version of the same check already done above for the whole
   file.
4. Once a full `tofu apply` succeeds once, the remaining steps below are
   what actually plug the result into the rest of this project.

Sizing is deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB
total) so this stays a demo rather than competing with whatever else is
already running on the T630.

---

## 1. Install OpenTofu

Installing the package needs sudo, so that part is you (`maq`), not `k8s`.
Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu`
afterwards with no further root involvement:

```sh
# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh
```

## 2. Provision the VMs with OpenTofu

Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved —
clone using the `k8s-readonly` token from `docs/01-bootstrap.md` step 4.
`k8s` has no keyring (it's headless, no desktop session), so this uses
`git credential-store` — a plaintext file, `chmod 600`'d, holding only the
read-only token:

```sh
# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials

cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from docs/01-bootstrap.md step 2), k3s_token (from step 3). Defaults for
# network/sizing are fine to start.

tofu init
tofu plan
tofu apply
```

For any later change to `terraform/`: edit and push as `maq` as usual, then
`cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`.

This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the
`k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from
anything else already using libvirt on this box, including the `default`
network stage 2's manual VM used). Cloud-init installs k3s on each on
first boot — give it ~2 minutes after `apply` finishes.

## 3. Get kubectl talking to the new cluster

```sh
mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab

export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes   # expect 3 Ready nodes
```

This works as-is because you're running it on the T630, which can reach the
`10.20.30.0/24` network directly. To also use kubectl from your own laptop,
either `scp` this kubeconfig over and open an SSH tunnel first
(`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's
`server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s`
whenever you need kubectl — simplest by far for a project this size.

## 4. Re-bootstrap Flux against the new cluster

This is the exact same `flux bootstrap git` command from `docs/03-flux.md`
step 2, just pointed at `~/.kube/config-homelab` instead of
`~/.kube/config-manual` — and it needs to actually run again, not be
skipped. The new 3-node cluster has its own fresh etcd/SQLite; nothing
about the manual VM's cluster state carries over to it, `flux-system`
namespace included. What *does* carry over is this repo:

```sh
flux check --pre --kubeconfig ~/.kube/config-homelab

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-homelab
```

Because `clusters/homelab/flux-system/` already holds the exact manifests
stage 3's bootstrap generated, this run doesn't need to commit anything
new back to the repo — it just applies that already-correct config to a
cluster that doesn't have it yet. That's the whole point of GitOps having
been the deploy mechanism all along: the desired state was never tied to
the specific cluster instance, so pointing the same bootstrap command at a
new kubeconfig reproduces it exactly. Confirm:

```sh
flux get kustomizations --watch
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods
```

## 5. kubectl access from elsewhere on the LAN, or remotely

Best practice for the Kubernetes API server is the same principle as
Headlamp (`docs/03-flux.md` step 5): never put 6443 on the public
internet if you can avoid it, because a leaked credential there is a full
cluster compromise.

Node IPs (`10.20.30.0/24` by default) live on the private libvirt network
from step 2 — only the T630 itself can reach them directly, which is
actually a nice side effect: even the rest of your LAN can't touch the API
server without going through the T630 first. Two ways to do that:

- **SSH into the T630 as `k8s`** and run kubectl there directly (same as
  step 3) — simplest, and what this whole project assumes by default.
- **Tunnel from another machine** (your laptop, or a phone via Termux, etc.):
  ```sh
  ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
  ```
  then point a local kubeconfig's `server:` at `https://127.0.0.1:6443`
  (copy the kubeconfig from step 3 and edit that one field). The cert
  validates because `k8s-api.boglabob.com` is in the server's TLS SAN list
  (`terraform/variables.tf`'s `k8s_api_hostname`) — add it to
  `/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine
  you're tunneling from and use that as the `server:` host instead of the
  raw IP, so the hostname in the URL matches a name the cert actually
  covers. (This SAN entry is new in stage 4 — the manual VM's cloud-init
  in `docs/02-k3s.md` never set one, since nothing needed LAN-wide access
  to it.)
- **From outside your home network entirely**: Tailscale or WireGuard on
  the T630, then the SSH tunnel above over the Tailscale/WireGuard link
  instead of the open internet. Reasonable next stretch goal once the core
  loop is working — don't port-forward 22 or 6443 on your router for this.

The `admin-user` bearer token (`docs/03-flux.md` step 5) also works for
kubectl over the same tunnel, if you'd rather not manage the client-cert
kubeconfig.

## 6. Point Caddy and Headlamp at the new cluster

Same steps as `docs/03-flux.md` step 8 (Caddy) and step 5 (Headlamp) — the
mechanism is identical, only the node IP(s) changed. Caddy now has three
node IPs to pick from instead of one, so use `docs/Caddyfile.example`'s
`reverse_proxy` blocks (which list all three) rather than the single-IP
version from stage 3:

```sh
tofu output node_ips
```

```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```

Headlamp's port-forward command is unchanged (`kubectl` just needs
`KUBECONFIG` pointed at `~/.kube/config-homelab` now).

## 7. Tearing down the manual VM

Now that the real 3-node cluster is up, remove stage 2's throwaway one:

```sh
virsh -c qemu:///system destroy k3s-manual              # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage   # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2   # base image isn't
                                                                            # attached to the VM
                                                                            # directly, needs its
                                                                            # own delete
rm ~/.kube/config-manual
rm -rf ~/vms
```

Nothing else needs cleaning up — Flux's own state lived entirely inside
that VM's cluster and goes away with it. The Forgejo repo, both tokens,
and the Forgejo Actions runner registration are all cluster-independent
and already carried over unchanged in step 4 above.

## Stretch goals, roughly in order

- **Remote access**: Tailscale or WireGuard on the k3s server node, for
  kubectl/Headlamp access from outside the LAN without opening anything
  publicly (step 5).
- **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if
  `boglabob.com` isn't publicly reachable on 80/443).
- **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the
  `K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret
  store.
- **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as
  podinfo (HelmRepository + HelmRelease under `apps/`).
- **HA**: add a second k3s server node and switch from SQLite to embedded
  etcd (`--cluster-init` on the first server, `--server` join on the second).