# Homelab Platform — Terraform + k3s + Flux on Forgejo

Learning project: provision VMs on the T630 with OpenTofu/Terraform, bootstrap a
k3s cluster on them, and manage everything after that point through GitOps
(Flux) synced from a Forgejo repo, with Forgejo Actions handling CI. The T630
keeps running its other self-hosted services throughout — this installs as
ordinary KVM/libvirt packages next to them, sized deliberately small (3 VMs,
2 vCPU/2GB RAM each), not a hypervisor OS taking over the box.

A four-stage tutorial, each stage building on the last:
- **`docs/01-bootstrap.md`** — one-time host setup: KVM/libvirt, the `k8s`
  user, pushing this repo to Forgejo.
- **`docs/02-k3s.md`** — build one VM by hand, get a real k3s cluster
  running on it. No Terraform yet.
- **`docs/03-flux.md`** — bootstrap Flux against that cluster, deploy
  every app in `apps/` through it (Helm-managed and plain-manifest alike),
  wire up the hello-app CI/image-automation loop.
- **`docs/04-tofu.md`** — graduate from the one hand-built VM to a
  Terraform/OpenTofu-provisioned 3-node cluster, and point the same Flux
  config at it. The harder stage — its HCL is schema-verified against the
  pinned provider version but not yet proven with a real `tofu apply` (see
  "Where this track actually stands" near the top of that doc) — worth
  finishing properly, but don't block on it; stages 2-3 already get you a
  complete, working GitOps loop without it.

## The loop

```
OpenTofu (terraform/)
   -> provisions VMs on the T630 via KVM/libvirt (qemu:///system)
   -> cloud-init installs k3s (1 server + 2 agents)
   -> VMs live on an isolated private network (10.20.30.0/24), reachable
      only from the T630 itself

Forgejo repo (this repo)
   -> clusters/homelab/  = Flux config (what Flux itself watches)
   -> apps/               = workloads Flux deploys
   -> .forgejo/workflows/ = CI (terraform plan, image builds)

Flux (running in the cluster)
   -> watches this repo
   -> applies clusters/homelab/** and apps/** to k3s

Caddy (Podman, existing) is the public front door for boglabob.com, but only
for the low-stakes apps — Headlamp and the k3s API server stay off it:
   git.boglabob.com     -> Forgejo (existing)
   podinfo.boglabob.com -> Traefik (k3s ingress) -> podinfo
   hello.boglabob.com   -> Traefik (k3s ingress) -> hello-app
   [Headlamp]            -> kubectl port-forward only, never a public hostname
   [k3s API, port 6443]  -> reachable only via the T630 (SSH tunnel or run
                             kubectl there directly), no proxy, no public port
   see docs/Caddyfile.example, docs/03-flux.md step 5 (Headlamp), and
   docs/04-tofu.md step 5 (LAN/remote kubectl access)
```

## Why it's structured this way

Each layer below only knows about the one directly beneath it — that's
deliberate, not incidental, and it's what makes the pieces independently
replaceable:

- **Terraform provisions VMs, nothing more.** It doesn't install k3s itself
  or configure Flux. Its job ends at "three VMs exist, on this network, with
  this cloud-init data attached." That's what makes `docs/02-k3s.md`
  possible at all — a single `virt-install` VM can stand in for the whole
  Terraform layer, because everything above it only cares that *some* VM
  with a `k3s` user and a working kubeconfig exists, not how it got there —
  which is exactly why Terraform is stage 4, not stage 1: everything above
  it gets learned and proven out on one hand-built VM first.
- **cloud-init installs k3s, nothing about Flux.** It runs `get.k3s.io` and
  stages a kubeconfig — full stop. Flux isn't baked into the image or the
  cloud-init data because `flux bootstrap` needs to *commit into this repo*
  (the generated `clusters/homelab/flux-system/` manifests), which only
  makes sense as a deliberate, one-time, human-run command against a
  cluster that already exists and can reach Forgejo — not something to
  automate blindly on every VM boot.
- **`flux bootstrap` is a one-time bridge, not a recurring step.** It's the
  only command in this whole project that goes API-server-side *and*
  repo-side in one shot: it installs the controllers, and it writes
  `clusters/homelab/flux-system/` back into git. After that one run, the
  cluster is entirely git-driven — rerunning `flux bootstrap` against the
  same repo is idempotent (safe if you need to recover a cluster, or to
  point the same config at a new cluster — see `docs/04-tofu.md` step 4),
  but nothing after that first run ever needs a human to run
  `kubectl apply` again.
- **`clusters/homelab/apps.yaml` fans out on purpose, instead of one giant
  Kustomization pointing at `apps/`.** Each app gets its own Flux
  `Kustomization` object (see `clusters/homelab/apps.yaml`) with its own
  `interval` and its own `prune: true`. That means a bad manifest in
  `apps/hello-app` can't block `apps/podinfo` from reconciling — each app's
  sync loop is independent, and you can watch/debug them individually with
  `flux get kustomization <name>` instead of one opaque blob.
- **Helm apps (podinfo, headlamp) vs. plain manifests (hello-app).**
  podinfo and headlamp are third-party charts — Flux's `HelmRepository` +
  `HelmRelease` pair is what lets Flux track and upgrade an upstream chart
  without this repo vendoring it. `hello-app` is *this* project's own code,
  so it's plain Kubernetes YAML (`Deployment`/`Service`/`Ingress`) — there's
  no chart to track, and plain manifests are simpler to diff/read for
  something this small.
- **Image automation exists only for `hello-app`, not podinfo/headlamp.**
  `ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`
  (`apps/hello-app/image-automation.yaml`) close the loop from "CI pushed a
  new image tag" to "the cluster is running it" — that's only relevant for
  an app whose image *this repo's own CI* builds. podinfo/headlamp pin a
  chart version range instead (`>=6.0.0`, `>=0.40.0`) and get upgraded by
  bumping that constraint by hand, since there's no CI producing new tags
  for them to watch.

## Directory layout

- `terraform/` — OpenTofu config that provisions the k3s VMs via KVM/libvirt
  directly on the T630 (no separate hypervisor OS, no VM template step —
  the cloud image is pulled straight from its URL).
- `terraform/cloud-init/` — cloud-init templates that install k3s server/agent
  on first boot.
- `clusters/homelab/` — Flux's own config for this cluster (populated by
  `flux bootstrap`, see docs/03-flux.md step 2).
- `apps/podinfo/` — first GitOps app: a HelmRelease for the standard Flux demo
  app (podinfo). No CI needed — good for validating the Flux sync loop works.
- `apps/hello-app/` — capstone app, and the public showcase piece at
  `hello.boglabob.com`: a one-page site (source + Dockerfile) built by
  Forgejo Actions, pushed to Forgejo's container registry, deployed via a
  Flux `ImagePolicy`/`ImageUpdateAutomation` so new pushes to main roll out
  automatically. The page itself shows the live commit SHA and deploy
  timestamp injected by CI, as proof the pipeline is really running rather
  than a static screenshot.
- `apps/headlamp/` — a Kubernetes UI (replacing the now-unmaintained
  Kubernetes Dashboard), with a cluster-admin token
  (`apps/headlamp/rbac.yaml`). No ingress — access is via `kubectl
  port-forward` only (see docs/03-flux.md step 5).
- `.forgejo/workflows/` — CI: terraform validate/plan on PRs, build+push
  hello-app image on merge to main, via kaniko under a rootless Podman
  runner (no docker.sock, no sudo — see docs/03-flux.md step 6).
- `docs/Caddyfile.example` — the reverse-proxy blocks for podinfo/hello-app
  to add to your existing Caddy (Podman) config.

## Prerequisites (see docs/01-bootstrap.md onward for the full walkthrough)

- `qemu-kvm`/`libvirt` installed on the T630 (alongside its existing services)
- Forgejo instance reachable, with this repo pushed to it and Actions enabled
- `kubectl`, `flux`, `tofu` CLI — run on the T630 itself as the unprivileged
  `k8s` user (simplest, since the VMs' network is only reachable from there)

## Milestones

1. KVM/libvirt installed on the T630, `k8s` user created (`docs/01-bootstrap.md`)
2. One hand-built VM, k3s running on it, kubectl talking to it (`docs/02-k3s.md`)
3. `flux bootstrap` against this Forgejo repo — podinfo/headlamp/hello-app
   deploy with no manual `kubectl apply` (`docs/03-flux.md`)
4. Wire `.forgejo/workflows/build-hello-app.yml` — push a code change to
   `apps/hello-app/src`, watch CI build → registry → Flux auto-deploy
   (`docs/03-flux.md` step 7)
5. `tofu apply` — 3 VMs come up, k3s cluster forms, same Flux config
   re-bootstrapped against it (`docs/04-tofu.md`)
6. Stretch: sealed-secrets/SOPS, kube-prometheus-stack, cert-manager