Updated project to improve guidance

README.md

@@ -7,12 +7,21 @@ keeps running its other self-hosted services throughout — this installs as
 ordinary KVM/libvirt packages next to them, sized deliberately small (3 VMs,
 2 vCPU/2GB RAM each), not a hypervisor OS taking over the box.
 
-Two setup paths, sharing everything past the cluster itself:
-- **`docs/QUICKSTART.md`** — one manually-created VM (`virt-install`, no
-  Terraform), fastest way to a real cluster to learn Flux/GitOps on.
-- **`docs/SETUP.md`** — the full path, Terraform/OpenTofu provisioning all
-  3 VMs. Currently the harder, unfinished track (provider schema issues) —
-  worth doing properly, but don't block on it.
+A four-stage tutorial, each stage building on the last:
+- **`docs/01-bootstrap.md`** — one-time host setup: KVM/libvirt, the `k8s`
+  user, pushing this repo to Forgejo.
+- **`docs/02-k3s.md`** — build one VM by hand, get a real k3s cluster
+  running on it. No Terraform yet.
+- **`docs/03-flux.md`** — bootstrap Flux against that cluster, deploy
+  every app in `apps/` through it (Helm-managed and plain-manifest alike),
+  wire up the hello-app CI/image-automation loop.
+- **`docs/04-tofu.md`** — graduate from the one hand-built VM to a
+  Terraform/OpenTofu-provisioned 3-node cluster, and point the same Flux
+  config at it. The harder stage — its HCL is schema-verified against the
+  pinned provider version but not yet proven with a real `tofu apply` (see
+  "Where this track actually stands" near the top of that doc) — worth
+  finishing properly, but don't block on it; stages 2-3 already get you a
+  complete, working GitOps loop without it.
 
 ## The loop
 
@@ -33,16 +42,70 @@ Flux (running in the cluster)
    -> applies clusters/homelab/** and apps/** to k3s
 
 Caddy (Podman, existing) is the public front door for boglabob.com, but only
-for the low-stakes apps — the Dashboard and the k3s API server stay off it:
+for the low-stakes apps — Headlamp and the k3s API server stay off it:
    git.boglabob.com     -> Forgejo (existing)
    podinfo.boglabob.com -> Traefik (k3s ingress) -> podinfo
    hello.boglabob.com   -> Traefik (k3s ingress) -> hello-app
-   [Dashboard]           -> kubectl port-forward only, never a public hostname
+   [Headlamp]            -> kubectl port-forward only, never a public hostname
    [k3s API, port 6443]  -> reachable only via the T630 (SSH tunnel or run
                              kubectl there directly), no proxy, no public port
-   see docs/Caddyfile.example and docs/SETUP.md steps 12-13
+   see docs/Caddyfile.example, docs/03-flux.md step 5 (Headlamp), and
+   docs/04-tofu.md step 5 (LAN/remote kubectl access)
 ```
 
+## Why it's structured this way
+
+Each layer below only knows about the one directly beneath it — that's
+deliberate, not incidental, and it's what makes the pieces independently
+replaceable:
+
+- **Terraform provisions VMs, nothing more.** It doesn't install k3s itself
+  or configure Flux. Its job ends at "three VMs exist, on this network, with
+  this cloud-init data attached." That's what makes `docs/02-k3s.md`
+  possible at all — a single `virt-install` VM can stand in for the whole
+  Terraform layer, because everything above it only cares that *some* VM
+  with a `k3s` user and a working kubeconfig exists, not how it got there —
+  which is exactly why Terraform is stage 4, not stage 1: everything above
+  it gets learned and proven out on one hand-built VM first.
+- **cloud-init installs k3s, nothing about Flux.** It runs `get.k3s.io` and
+  stages a kubeconfig — full stop. Flux isn't baked into the image or the
+  cloud-init data because `flux bootstrap` needs to *commit into this repo*
+  (the generated `clusters/homelab/flux-system/` manifests), which only
+  makes sense as a deliberate, one-time, human-run command against a
+  cluster that already exists and can reach Forgejo — not something to
+  automate blindly on every VM boot.
+- **`flux bootstrap` is a one-time bridge, not a recurring step.** It's the
+  only command in this whole project that goes API-server-side *and*
+  repo-side in one shot: it installs the controllers, and it writes
+  `clusters/homelab/flux-system/` back into git. After that one run, the
+  cluster is entirely git-driven — rerunning `flux bootstrap` against the
+  same repo is idempotent (safe if you need to recover a cluster, or to
+  point the same config at a new cluster — see `docs/04-tofu.md` step 4),
+  but nothing after that first run ever needs a human to run
+  `kubectl apply` again.
+- **`clusters/homelab/apps.yaml` fans out on purpose, instead of one giant
+  Kustomization pointing at `apps/`.** Each app gets its own Flux
+  `Kustomization` object (see `clusters/homelab/apps.yaml`) with its own
+  `interval` and its own `prune: true`. That means a bad manifest in
+  `apps/hello-app` can't block `apps/podinfo` from reconciling — each app's
+  sync loop is independent, and you can watch/debug them individually with
+  `flux get kustomization <name>` instead of one opaque blob.
+- **Helm apps (podinfo, headlamp) vs. plain manifests (hello-app).**
+  podinfo and headlamp are third-party charts — Flux's `HelmRepository` +
+  `HelmRelease` pair is what lets Flux track and upgrade an upstream chart
+  without this repo vendoring it. `hello-app` is *this* project's own code,
+  so it's plain Kubernetes YAML (`Deployment`/`Service`/`Ingress`) — there's
+  no chart to track, and plain manifests are simpler to diff/read for
+  something this small.
+- **Image automation exists only for `hello-app`, not podinfo/headlamp.**
+  `ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`
+  (`apps/hello-app/image-automation.yaml`) close the loop from "CI pushed a
+  new image tag" to "the cluster is running it" — that's only relevant for
+  an app whose image *this repo's own CI* builds. podinfo/headlamp pin a
+  chart version range instead (`>=6.0.0`, `>=0.40.0`) and get upgraded by
+  bumping that constraint by hand, since there's no CI producing new tags
+  for them to watch.
+
 ## Directory layout
 
 - `terraform/` — OpenTofu config that provisions the k3s VMs via KVM/libvirt
@@ -51,7 +114,7 @@ for the low-stakes apps — the Dashboard and the k3s API server stay off it:
 - `terraform/cloud-init/` — cloud-init templates that install k3s server/agent
   on first boot.
 - `clusters/homelab/` — Flux's own config for this cluster (populated by
-  `flux bootstrap`, see docs/SETUP.md step 7).
+  `flux bootstrap`, see docs/03-flux.md step 2).
 - `apps/podinfo/` — first GitOps app: a HelmRelease for the standard Flux demo
   app (podinfo). No CI needed — good for validating the Flux sync loop works.
 - `apps/hello-app/` — capstone app, and the public showcase piece at
@@ -61,28 +124,32 @@ for the low-stakes apps — the Dashboard and the k3s API server stay off it:
   automatically. The page itself shows the live commit SHA and deploy
   timestamp injected by CI, as proof the pipeline is really running rather
   than a static screenshot.
-- `apps/kubernetes-dashboard/` — the official Kubernetes Dashboard, with a
-  cluster-admin token (`apps/kubernetes-dashboard/rbac.yaml`). No ingress —
-  access is via `kubectl port-forward` only (see docs/SETUP.md step 12).
+- `apps/headlamp/` — a Kubernetes UI (replacing the now-unmaintained
+  Kubernetes Dashboard), with a cluster-admin token
+  (`apps/headlamp/rbac.yaml`). No ingress — access is via `kubectl
+  port-forward` only (see docs/03-flux.md step 5).
 - `.forgejo/workflows/` — CI: terraform validate/plan on PRs, build+push
   hello-app image on merge to main, via kaniko under a rootless Podman
-  runner (no docker.sock, no sudo — see docs/SETUP.md step 9).
+  runner (no docker.sock, no sudo — see docs/03-flux.md step 6).
 - `docs/Caddyfile.example` — the reverse-proxy blocks for podinfo/hello-app
   to add to your existing Caddy (Podman) config.
 
-## Prerequisites (see docs/SETUP.md for the full walkthrough)
+## Prerequisites (see docs/01-bootstrap.md onward for the full walkthrough)
 
 - `qemu-kvm`/`libvirt` installed on the T630 (alongside its existing services)
 - Forgejo instance reachable, with this repo pushed to it and Actions enabled
-- `tofu`, `kubectl`, `flux` CLI — run on the T630 itself as the unprivileged
+- `kubectl`, `flux`, `tofu` CLI — run on the T630 itself as the unprivileged
   `k8s` user (simplest, since the VMs' network is only reachable from there)
 
 ## Milestones
 
-1. KVM/libvirt installed on the T630, `k8s` user created
-2. `terraform apply` — 3 VMs come up, k3s cluster forms
-3. `flux bootstrap` against this Forgejo repo
-4. Push `apps/podinfo` — watch Flux deploy it with no manual `kubectl apply`
-5. Wire `.forgejo/workflows/build-hello-app.yml` — push a code change to
+1. KVM/libvirt installed on the T630, `k8s` user created (`docs/01-bootstrap.md`)
+2. One hand-built VM, k3s running on it, kubectl talking to it (`docs/02-k3s.md`)
+3. `flux bootstrap` against this Forgejo repo — podinfo/headlamp/hello-app
+   deploy with no manual `kubectl apply` (`docs/03-flux.md`)
+4. Wire `.forgejo/workflows/build-hello-app.yml` — push a code change to
    `apps/hello-app/src`, watch CI build → registry → Flux auto-deploy
+   (`docs/03-flux.md` step 7)
+5. `tofu apply` — 3 VMs come up, k3s cluster forms, same Flux config
+   re-bootstrapped against it (`docs/04-tofu.md`)
 6. Stretch: sealed-secrets/SOPS, kube-prometheus-stack, cert-manager