88 lines
4.5 KiB
Markdown
88 lines
4.5 KiB
Markdown
# Homelab Platform — Terraform + k3s + Flux on Forgejo
|
|
|
|
Learning project: provision VMs on the T630 with OpenTofu/Terraform, bootstrap a
|
|
k3s cluster on them, and manage everything after that point through GitOps
|
|
(Flux) synced from a Forgejo repo, with Forgejo Actions handling CI. The T630
|
|
keeps running its other self-hosted services throughout — this installs as
|
|
ordinary KVM/libvirt packages next to them, sized deliberately small (3 VMs,
|
|
2 vCPU/2GB RAM each), not a hypervisor OS taking over the box.
|
|
|
|
Two setup paths, sharing everything past the cluster itself:
|
|
- **`docs/QUICKSTART.md`** — one manually-created VM (`virt-install`, no
|
|
Terraform), fastest way to a real cluster to learn Flux/GitOps on.
|
|
- **`docs/SETUP.md`** — the full path, Terraform/OpenTofu provisioning all
|
|
3 VMs. Currently the harder, unfinished track (provider schema issues) —
|
|
worth doing properly, but don't block on it.
|
|
|
|
## The loop
|
|
|
|
```
|
|
OpenTofu (terraform/)
|
|
-> provisions VMs on the T630 via KVM/libvirt (qemu:///system)
|
|
-> cloud-init installs k3s (1 server + 2 agents)
|
|
-> VMs live on an isolated private network (10.20.30.0/24), reachable
|
|
only from the T630 itself
|
|
|
|
Forgejo repo (this repo)
|
|
-> clusters/homelab/ = Flux config (what Flux itself watches)
|
|
-> apps/ = workloads Flux deploys
|
|
-> .forgejo/workflows/ = CI (terraform plan, image builds)
|
|
|
|
Flux (running in the cluster)
|
|
-> watches this repo
|
|
-> applies clusters/homelab/** and apps/** to k3s
|
|
|
|
Caddy (Podman, existing) is the public front door for boglabob.com, but only
|
|
for the low-stakes apps — the Dashboard and the k3s API server stay off it:
|
|
git.boglabob.com -> Forgejo (existing)
|
|
podinfo.boglabob.com -> Traefik (k3s ingress) -> podinfo
|
|
hello.boglabob.com -> Traefik (k3s ingress) -> hello-app
|
|
[Dashboard] -> kubectl port-forward only, never a public hostname
|
|
[k3s API, port 6443] -> reachable only via the T630 (SSH tunnel or run
|
|
kubectl there directly), no proxy, no public port
|
|
see docs/Caddyfile.example and docs/SETUP.md steps 12-13
|
|
```
|
|
|
|
## Directory layout
|
|
|
|
- `terraform/` — OpenTofu config that provisions the k3s VMs via KVM/libvirt
|
|
directly on the T630 (no separate hypervisor OS, no VM template step —
|
|
the cloud image is pulled straight from its URL).
|
|
- `terraform/cloud-init/` — cloud-init templates that install k3s server/agent
|
|
on first boot.
|
|
- `clusters/homelab/` — Flux's own config for this cluster (populated by
|
|
`flux bootstrap`, see docs/SETUP.md step 7).
|
|
- `apps/podinfo/` — first GitOps app: a HelmRelease for the standard Flux demo
|
|
app (podinfo). No CI needed — good for validating the Flux sync loop works.
|
|
- `apps/hello-app/` — capstone app, and the public showcase piece at
|
|
`hello.boglabob.com`: a one-page site (source + Dockerfile) built by
|
|
Forgejo Actions, pushed to Forgejo's container registry, deployed via a
|
|
Flux `ImagePolicy`/`ImageUpdateAutomation` so new pushes to main roll out
|
|
automatically. The page itself shows the live commit SHA and deploy
|
|
timestamp injected by CI, as proof the pipeline is really running rather
|
|
than a static screenshot.
|
|
- `apps/kubernetes-dashboard/` — the official Kubernetes Dashboard, with a
|
|
cluster-admin token (`apps/kubernetes-dashboard/rbac.yaml`). No ingress —
|
|
access is via `kubectl port-forward` only (see docs/SETUP.md step 12).
|
|
- `.forgejo/workflows/` — CI: terraform validate/plan on PRs, build+push
|
|
hello-app image on merge to main, via kaniko under a rootless Podman
|
|
runner (no docker.sock, no sudo — see docs/SETUP.md step 9).
|
|
- `docs/Caddyfile.example` — the reverse-proxy blocks for podinfo/hello-app
|
|
to add to your existing Caddy (Podman) config.
|
|
|
|
## Prerequisites (see docs/SETUP.md for the full walkthrough)
|
|
|
|
- `qemu-kvm`/`libvirt` installed on the T630 (alongside its existing services)
|
|
- Forgejo instance reachable, with this repo pushed to it and Actions enabled
|
|
- `tofu`, `kubectl`, `flux` CLI — run on the T630 itself as the unprivileged
|
|
`k8s` user (simplest, since the VMs' network is only reachable from there)
|
|
|
|
## Milestones
|
|
|
|
1. KVM/libvirt installed on the T630, `k8s` user created
|
|
2. `terraform apply` — 3 VMs come up, k3s cluster forms
|
|
3. `flux bootstrap` against this Forgejo repo
|
|
4. Push `apps/podinfo` — watch Flux deploy it with no manual `kubectl apply`
|
|
5. Wire `.forgejo/workflows/build-hello-app.yml` — push a code change to
|
|
`apps/hello-app/src`, watch CI build → registry → Flux auto-deploy
|
|
6. Stretch: sealed-secrets/SOPS, kube-prometheus-stack, cert-manager
|