155 lines
8.6 KiB
Markdown
155 lines
8.6 KiB
Markdown
# Homelab Platform — Terraform + k3s + Flux on Forgejo
|
|
|
|
Learning project: provision VMs on the T630 with OpenTofu/Terraform, bootstrap a
|
|
k3s cluster on them, and manage everything after that point through GitOps
|
|
(Flux) synced from a Forgejo repo, with Forgejo Actions handling CI. The T630
|
|
keeps running its other self-hosted services throughout — this installs as
|
|
ordinary KVM/libvirt packages next to them, sized deliberately small (3 VMs,
|
|
2 vCPU/2GB RAM each), not a hypervisor OS taking over the box.
|
|
|
|
A four-stage tutorial, each stage building on the last:
|
|
- **`docs/01-bootstrap.md`** — one-time host setup: KVM/libvirt, the `k8s`
|
|
user, pushing this repo to Forgejo.
|
|
- **`docs/02-k3s.md`** — build one VM by hand, get a real k3s cluster
|
|
running on it. No Terraform yet.
|
|
- **`docs/03-flux.md`** — bootstrap Flux against that cluster, deploy
|
|
every app in `apps/` through it (Helm-managed and plain-manifest alike),
|
|
wire up the hello-app CI/image-automation loop.
|
|
- **`docs/04-tofu.md`** — graduate from the one hand-built VM to a
|
|
Terraform/OpenTofu-provisioned 3-node cluster, and point the same Flux
|
|
config at it. The harder stage — its HCL is schema-verified against the
|
|
pinned provider version but not yet proven with a real `tofu apply` (see
|
|
"Where this track actually stands" near the top of that doc) — worth
|
|
finishing properly, but don't block on it; stages 2-3 already get you a
|
|
complete, working GitOps loop without it.
|
|
|
|
## The loop
|
|
|
|
```
|
|
OpenTofu (terraform/)
|
|
-> provisions VMs on the T630 via KVM/libvirt (qemu:///system)
|
|
-> cloud-init installs k3s (1 server + 2 agents)
|
|
-> VMs live on an isolated private network (10.20.30.0/24), reachable
|
|
only from the T630 itself
|
|
|
|
Forgejo repo (this repo)
|
|
-> clusters/homelab/ = Flux config (what Flux itself watches)
|
|
-> apps/ = workloads Flux deploys
|
|
-> .forgejo/workflows/ = CI (terraform plan, image builds)
|
|
|
|
Flux (running in the cluster)
|
|
-> watches this repo
|
|
-> applies clusters/homelab/** and apps/** to k3s
|
|
|
|
Caddy (Podman, existing) is the public front door for boglabob.com, but only
|
|
for the low-stakes apps — Headlamp and the k3s API server stay off it:
|
|
git.boglabob.com -> Forgejo (existing)
|
|
podinfo.boglabob.com -> Traefik (k3s ingress) -> podinfo
|
|
hello.boglabob.com -> Traefik (k3s ingress) -> hello-app
|
|
[Headlamp] -> kubectl port-forward only, never a public hostname
|
|
[k3s API, port 6443] -> reachable only via the T630 (SSH tunnel or run
|
|
kubectl there directly), no proxy, no public port
|
|
see docs/Caddyfile.example, docs/03-flux.md step 5 (Headlamp), and
|
|
docs/04-tofu.md step 5 (LAN/remote kubectl access)
|
|
```
|
|
|
|
## Why it's structured this way
|
|
|
|
Each layer below only knows about the one directly beneath it — that's
|
|
deliberate, not incidental, and it's what makes the pieces independently
|
|
replaceable:
|
|
|
|
- **Terraform provisions VMs, nothing more.** It doesn't install k3s itself
|
|
or configure Flux. Its job ends at "three VMs exist, on this network, with
|
|
this cloud-init data attached." That's what makes `docs/02-k3s.md`
|
|
possible at all — a single `virt-install` VM can stand in for the whole
|
|
Terraform layer, because everything above it only cares that *some* VM
|
|
with a `k3s` user and a working kubeconfig exists, not how it got there —
|
|
which is exactly why Terraform is stage 4, not stage 1: everything above
|
|
it gets learned and proven out on one hand-built VM first.
|
|
- **cloud-init installs k3s, nothing about Flux.** It runs `get.k3s.io` and
|
|
stages a kubeconfig — full stop. Flux isn't baked into the image or the
|
|
cloud-init data because `flux bootstrap` needs to *commit into this repo*
|
|
(the generated `clusters/homelab/flux-system/` manifests), which only
|
|
makes sense as a deliberate, one-time, human-run command against a
|
|
cluster that already exists and can reach Forgejo — not something to
|
|
automate blindly on every VM boot.
|
|
- **`flux bootstrap` is a one-time bridge, not a recurring step.** It's the
|
|
only command in this whole project that goes API-server-side *and*
|
|
repo-side in one shot: it installs the controllers, and it writes
|
|
`clusters/homelab/flux-system/` back into git. After that one run, the
|
|
cluster is entirely git-driven — rerunning `flux bootstrap` against the
|
|
same repo is idempotent (safe if you need to recover a cluster, or to
|
|
point the same config at a new cluster — see `docs/04-tofu.md` step 4),
|
|
but nothing after that first run ever needs a human to run
|
|
`kubectl apply` again.
|
|
- **`clusters/homelab/apps.yaml` fans out on purpose, instead of one giant
|
|
Kustomization pointing at `apps/`.** Each app gets its own Flux
|
|
`Kustomization` object (see `clusters/homelab/apps.yaml`) with its own
|
|
`interval` and its own `prune: true`. That means a bad manifest in
|
|
`apps/hello-app` can't block `apps/podinfo` from reconciling — each app's
|
|
sync loop is independent, and you can watch/debug them individually with
|
|
`flux get kustomization <name>` instead of one opaque blob.
|
|
- **Helm apps (podinfo, headlamp) vs. plain manifests (hello-app).**
|
|
podinfo and headlamp are third-party charts — Flux's `HelmRepository` +
|
|
`HelmRelease` pair is what lets Flux track and upgrade an upstream chart
|
|
without this repo vendoring it. `hello-app` is *this* project's own code,
|
|
so it's plain Kubernetes YAML (`Deployment`/`Service`/`Ingress`) — there's
|
|
no chart to track, and plain manifests are simpler to diff/read for
|
|
something this small.
|
|
- **Image automation exists only for `hello-app`, not podinfo/headlamp.**
|
|
`ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`
|
|
(`apps/hello-app/image-automation.yaml`) close the loop from "CI pushed a
|
|
new image tag" to "the cluster is running it" — that's only relevant for
|
|
an app whose image *this repo's own CI* builds. podinfo/headlamp pin a
|
|
chart version range instead (`>=6.0.0`, `>=0.40.0`) and get upgraded by
|
|
bumping that constraint by hand, since there's no CI producing new tags
|
|
for them to watch.
|
|
|
|
## Directory layout
|
|
|
|
- `terraform/` — OpenTofu config that provisions the k3s VMs via KVM/libvirt
|
|
directly on the T630 (no separate hypervisor OS, no VM template step —
|
|
the cloud image is pulled straight from its URL).
|
|
- `terraform/cloud-init/` — cloud-init templates that install k3s server/agent
|
|
on first boot.
|
|
- `clusters/homelab/` — Flux's own config for this cluster (populated by
|
|
`flux bootstrap`, see docs/03-flux.md step 2).
|
|
- `apps/podinfo/` — first GitOps app: a HelmRelease for the standard Flux demo
|
|
app (podinfo). No CI needed — good for validating the Flux sync loop works.
|
|
- `apps/hello-app/` — capstone app, and the public showcase piece at
|
|
`hello.boglabob.com`: a one-page site (source + Dockerfile) built by
|
|
Forgejo Actions, pushed to Forgejo's container registry, deployed via a
|
|
Flux `ImagePolicy`/`ImageUpdateAutomation` so new pushes to main roll out
|
|
automatically. The page itself shows the live commit SHA and deploy
|
|
timestamp injected by CI, as proof the pipeline is really running rather
|
|
than a static screenshot.
|
|
- `apps/headlamp/` — a Kubernetes UI (replacing the now-unmaintained
|
|
Kubernetes Dashboard), with a cluster-admin token
|
|
(`apps/headlamp/rbac.yaml`). No ingress — access is via `kubectl
|
|
port-forward` only (see docs/03-flux.md step 5).
|
|
- `.forgejo/workflows/` — CI: terraform validate/plan on PRs, build+push
|
|
hello-app image on merge to main, via kaniko under a rootless Podman
|
|
runner (no docker.sock, no sudo — see docs/03-flux.md step 6).
|
|
- `docs/Caddyfile.example` — the reverse-proxy blocks for podinfo/hello-app
|
|
to add to your existing Caddy (Podman) config.
|
|
|
|
## Prerequisites (see docs/01-bootstrap.md onward for the full walkthrough)
|
|
|
|
- `qemu-kvm`/`libvirt` installed on the T630 (alongside its existing services)
|
|
- Forgejo instance reachable, with this repo pushed to it and Actions enabled
|
|
- `kubectl`, `flux`, `tofu` CLI — run on the T630 itself as the unprivileged
|
|
`k8s` user (simplest, since the VMs' network is only reachable from there)
|
|
|
|
## Milestones
|
|
|
|
1. KVM/libvirt installed on the T630, `k8s` user created (`docs/01-bootstrap.md`)
|
|
2. One hand-built VM, k3s running on it, kubectl talking to it (`docs/02-k3s.md`)
|
|
3. `flux bootstrap` against this Forgejo repo — podinfo/headlamp/hello-app
|
|
deploy with no manual `kubectl apply` (`docs/03-flux.md`)
|
|
4. Wire `.forgejo/workflows/build-hello-app.yml` — push a code change to
|
|
`apps/hello-app/src`, watch CI build → registry → Flux auto-deploy
|
|
(`docs/03-flux.md` step 7)
|
|
5. `tofu apply` — 3 VMs come up, k3s cluster forms, same Flux config
|
|
re-bootstrapped against it (`docs/04-tofu.md`)
|
|
6. Stretch: sealed-secrets/SOPS, kube-prometheus-stack, cert-manager
|