CodeGit/cloud-demo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Updated project to improve guidance
Some checks failed
terraform / validate (push) Failing after 37s
Details

Browse source
This commit is contained in:
CodeGit 2026-09-03 19:06:27 +01:00
parent 63008b3ab9
commit 4ea6d8b9e5
26 changed files with 1091 additions and 695 deletions
Download patch file Download diff file Expand all files Collapse all files

132
docs/01-bootstrap.md Normal file
View file

@ -0,0 +1,132 @@
# Stage 1: Bootstrap
This project is a four-stage tutorial, each stage building on the last:
1. **`docs/01-bootstrap.md`** (this doc) — one-time host setup: KVM/libvirt,
an unprivileged user to run everything as, and pushing this repo to
Forgejo. Nothing here is specific to k3s, Flux, or Terraform — every
later stage depends on it.
2. **`docs/02-k3s.md`** — build one VM by hand and get a real k3s cluster
running on it. No Flux yet, no Terraform — just a working cluster and a
kubeconfig that talks to it.
3. **`docs/03-flux.md`** — bootstrap Flux against that cluster and deploy
this repo's apps through it, including what Helm chart management looks
like under GitOps (podinfo, headlamp) versus plain manifests
(hello-app), and the CI loop that builds and auto-deploys hello-app.
4. **`docs/04-tofu.md`** — graduate from the one hand-built VM to a proper
3-node cluster provisioned by Terraform/OpenTofu, and point the same
Flux config at it.
Each stage says exactly which earlier steps it actually depends on, rather
than assuming you need everything done up front — stage 2, for instance,
only needs steps 1-2 below.
Assumes: the T630 is an existing Debian box already running other
self-hosted services — this project installs alongside those as ordinary
packages (`qemu-kvm`/`libvirt`), not a hypervisor OS replacing Debian.
Forgejo is already running and reachable at `https://git.boglabob.com`, and
you can point DNS records under `boglabob.com` at hosts on your network
(directly, or via whatever reverse proxy/tunnel already gets
`git.boglabob.com` there).
---
## 1. Install KVM/libvirt on the T630
Ordinary packages, no reboot into an installer, nothing else on the box is
touched:
```sh
# on the T630
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst
# confirm hardware virtualization is available (T630's Xeons support it)
sudo kvm-ok
```
## 2. Create the unprivileged 'k8s' user
One dedicated, no-sudo user for everything this project touches: driving
`virsh`/`kubectl`/`flux`/`tofu` against libvirt, and running the Forgejo
Actions runner later (`docs/03-flux.md` step 6). It needs group membership
to talk to libvirt — that's a one-time root action; nothing it does
afterwards needs `sudo`.
```sh
sudo useradd -m -s /bin/bash k8s # one-time, needs root to create the user itself
sudo usermod -aG libvirt,kvm k8s
sudo loginctl enable-linger k8s # lets its services keep running after logout
# as k8s, from here on (sudo -iu, not su -, since k8s has no password set):
sudo -iu k8s
ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/id_ed25519 # only needed if you'll SSH in as k8s day-to-day
virsh -c qemu:///system list --all # sanity check: should run with no permission error, no sudo
```
Do the rest of this project logged in as `k8s` on the T630 itself (`ssh
k8s@t630`) — VM IPs live on a private libvirt network that's only directly
reachable from the T630, so this is the simplest place to run
`kubectl`/`flux`/`tofu` from. (If you'd rather drive Terraform from your
own workstation instead once you reach stage 4, see the `libvirt_uri`
comment in `terraform/terraform.tfvars.example` — you'll then need an SSH
tunnel for kubectl/flux to reach node IPs.)
## 3. Generate the secret k3s needs
```sh
openssl rand -hex 32 # -> k3s_token
```
This is the shared token agent nodes use to join a k3s server — irrelevant
for stage 2's single-node VM (a lone server needs no one to join it), but
generate it now while you're doing one-time setup; stage 4's
Terraform-provisioned multi-node cluster is what actually uses it.
## 4. Push this repo to Forgejo
Doing this before any cluster exists (rather than after) means `k8s` can
get the repo with a plain `git clone` later, instead of needing a one-off
copy handed to it — and any future change to this repo just needs a
`git pull` on the T630. Repo/owner used throughout this project:
`codegit/cloud-demo` (already baked into `apps/hello-app/deployment.yaml`
and `image-automation.yaml`'s image references — no placeholder-swapping
needed).
1. On Forgejo (`https://git.boglabob.com`), as `codegit`: **+ → New
Repository** → name `cloud-demo`. Leave it empty — don't initialize with
a README/`.gitignore`/license, since this repo already has its own.
Visibility (public/private) is your call; either works, since access for
`k8s`/Flux/CI goes through the tokens below regardless.
2. Locally, wherever you're editing this repo (`maq`):
```sh
git init # if not already
git add .
git commit -m "initial scaffold"
git remote add origin https://git.boglabob.com/codegit/cloud-demo.git
git push -u origin main
```
3. Generate two access tokens (`Settings → Applications → Generate New
Token`), scoped as narrowly as Forgejo's token UI allows to repository
read/write:
- **`k8s-readonly`** — read-only. Used only for `k8s`'s own manual
`git clone`/`pull` on the T630 — never leaves that box, isn't used by
anything automated. Not needed until `docs/04-tofu.md` (that's the
first stage that clones this repo onto the T630 rather than editing
it from your workstation).
- **`flux-write`** — read/write. Used once, as a `flux bootstrap`
argument (`docs/03-flux.md` step 2); Flux stores it as a Kubernetes
Secret inside the cluster from then on (`ImageUpdateAutomation`'s
commits back, in the hello-app section of that stage, reuse that same
in-cluster Secret) — it's never written to `k8s`'s filesystem at all.
Using HTTPS tokens instead of `k8s`'s SSH key (`~/.ssh/id_ed25519`, from
step 2) sidesteps an open question: Forgejo's git-SSH port isn't
reachable from this desktop through your router (see the SSH
troubleshooting earlier in this project's history), and whether it's
reachable from `k8s` on the T630 itself was never actually confirmed
either. HTTPS (443, via Caddy) is already proven to work, so both
tokens use that instead. Copy both token values now — Forgejo only
shows them once.
Next: `docs/02-k3s.md` — you only need steps 1-2 above to start it.

282
docs/QUICKSTART.md → docs/02-k3s.md
View file

@ -1,24 +1,17 @@
# Quickstart: manual cluster (no Terraform)
# Stage 2: A real k3s cluster, built by hand
A real k3s cluster, built by hand, so Flux/GitOps can be learned right away
instead of waiting on the Terraform/libvirt provider work in `docs/SETUP.md`
to get sorted (that provider did a breaking rewrite between 0.8.x and
0.9.x, and the HCL needs writing against the real schema, not memory).
Stage 1 (`docs/01-bootstrap.md`) got KVM/libvirt installed and the `k8s`
user created — that's all this stage needs (its steps 1-2; skip 3-4 for
now, they're not needed until stage 3). This stage builds one throwaway VM
directly with `virt-install` — no Terraform involved at all — and installs
k3s on it, so stage 3 has a real cluster to bootstrap Flux/GitOps against.
Terraform doesn't show up until stage 4, once you already understand what
it's automating.
This is one throwaway VM, created directly with `virt-install` — no
Terraform involved at all. Everything here explains the *why*, not just
the *what*: read each section before running its commands, and where a
config file is being built (the cloud-init data, the Caddy block), treat
the snippets as pieces to assemble into your own file, not something to
paste wholesale.
Shares steps 1–4 of `docs/SETUP.md` as prerequisites:
- Step 1: KVM/libvirt packages installed on the T630.
- Step 2: the unprivileged `k8s` user exists, in the `libvirt`/`kvm` groups,
with an SSH keypair at `~/.ssh/id_ed25519`.
- Step 3: skip — no join token needed here (see "one node is enough" below).
- Step 4: `cloud-demo` pushed to Forgejo, with a `k8s-readonly` and a
`flux-write` token generated.
Everything here explains the *why*, not just the *what*: read each section
before running its commands, and where a config file is being built (the
cloud-init data, the Caddy block later on), treat the snippets as pieces to
assemble into your own file, not something to paste wholesale.
Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).
@ -30,10 +23,10 @@ A k3s **server** node runs the control plane (API server, scheduler,
etcd/SQLite) *and* schedules ordinary workloads onto itself unless you
explicitly disable that. So a single server, with no agents, is already a
complete, working cluster — nothing here needs a join token or a second
VM. The 3-node design in `docs/SETUP.md` exists to mirror a more
VM. The 3-node design in `docs/04-tofu.md` exists to mirror a more
realistic multi-node cluster once Terraform can build it repeatedly and
disposably; for learning Flux and GitOps, that extra shape doesn't buy
you anything yet.
disposably; for learning k3s, Helm, and Flux/GitOps, that extra shape
doesn't buy you anything yet.
## Two libvirt connections, and why it matters
@ -43,8 +36,8 @@ perspective there are (at least) two separate connections:
- `qemu:///system` — the shared, host-wide instance. VMs here can use
privileged networking (bridges, NAT with DHCP), and management access is
gated by group membership (`libvirt`/`kvm`) checked via polkit — which is
exactly what step 2's `usermod -aG libvirt,kvm k8s` set up. No sudo
needed for any command below; that group membership *is* the
exactly what bootstrap step 2's `usermod -aG libvirt,kvm k8s` set up. No
sudo needed for any command below; that group membership *is* the
authorization.
- `qemu:///session` — a private, per-user instance with no special
privileges, and critically, its own separate storage pools and networks
@ -76,6 +69,8 @@ the simplest case). The key property: creating or writing a volume inside
a pool goes through libvirtd's API, not through `k8s`'s own filesystem
permissions — so it's libvirtd (already running with the right privileges)
that handles ownership correctly, regardless of what user asked for it.
(Stage 4's Terraform config hits this exact same problem and solves it the
same way — see `terraform/main.tf`'s `libvirt_pool` resource.)
Check what pools/networks already exist:
@ -134,7 +129,10 @@ VM built from the same base. Instead, create a **copy-on-write overlay**:
a second volume that starts out empty and only stores the *differences*
from its backing volume. Reads that haven't been changed transparently
fall through to the base image; writes go into the overlay. This is the
same relationship a Docker image layer has to its base layer.
same relationship a Docker image layer has to its base layer. (Same
relationship as `terraform/main.tf`'s `libvirt_volume.base` /
`libvirt_volume.node` pair in stage 4 — one shared base, one overlay per
node.)
```sh
virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
@ -152,15 +150,16 @@ first boot, driven by data supplied externally rather than baked into the
image. `virt-install`'s `--cloud-init` flag builds a small ISO (the
"NoCloud" datasource) containing that data and attaches it to the VM;
cloud-init, already installed in the image, detects it automatically at
boot.
boot. (`terraform/main.tf`'s `libvirt_cloudinit_disk` resource in stage 4
is the same mechanism, just built by Terraform instead of by hand.)
Two separate pieces of data go in, and they answer different questions:
- **`user-data`** — *what should exist on this machine*: users, packages,
commands to run. Written as `#cloud-config` YAML.
- **`network-config`** — *how should this machine's network be set up*.
This quickstart skips it entirely and lets the VM get an address via
DHCP from the `default` network instead (the Terraform track, by
This stage skips it entirely and lets the VM get an address via DHCP
from the `default` network instead (stage 4's Terraform config, by
contrast, uses this for static IPs, since it manages its own isolated
network).
@ -340,227 +339,6 @@ export KUBECONFIG=~/.kube/config-manual
kubectl get nodes # expect 1 Ready node
```
## Flux: what's actually happening, mechanism by mechanism
"Flux" isn't one program watching your repo by magic — it's a handful of
ordinary Kubernetes controllers (just Deployments, like anything else
you'd run on the cluster), each one understanding a couple of **Custom
Resource Definitions** — CRDs extend the Kubernetes API with new object
kinds, the same way `Deployment` or `Service` are built-in kinds.
`GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing
about any of this is special-cased outside the normal Kubernetes API —
it's the exact same "define an object, a controller notices it and acts"
loop that runs the whole rest of Kubernetes.
Install the CLI the same no-sudo, direct-binary-release way as `kubectl`:
```sh
FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version
```
Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this
uses the generic git bootstrap — over HTTPS with the `flux-write` token
from step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably
reachable from either this desktop or the T630 — see the earlier
troubleshooting in this project's history):
```sh
flux check --pre --kubeconfig ~/.kube/config-manual
flux bootstrap git \
--url=https://git.boglabob.com/codegit/cloud-demo \
--branch=main \
--path=clusters/homelab \
--username=codegit \
--password=<FLUX_WRITE_TOKEN> \
--token-auth \
--kubeconfig ~/.kube/config-manual
```
That one command did five genuinely separate things. Go look at each —
these commands work against the cluster you already bootstrapped:
**1. It installed the controllers** — plain Kubernetes Deployments, no
different in kind from anything else running on the cluster:
```sh
kubectl -n flux-system get deployments
```
`source-controller`, `kustomize-controller`, `helm-controller`,
`notification-controller`. Alongside them, it registered the CRDs those
controllers understand:
```sh
kubectl get crds | grep fluxcd
```
**2. It created a `GitRepository` object** — this is the whole "what repo
am I watching" declaration, and nothing more. `source-controller`'s job is
entirely mechanical: every `interval` (default 1m), do a real `git fetch`
against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that
tree into a `.tar.gz`, and record its location in `.status.artifact`.
That's the entire job — it doesn't know or care what's *in* the repo.
```sh
kubectl -n flux-system get gitrepository flux-system -o yaml
```
Look at `.status.conditions` and `.status.artifact` — that's the result of
an actual git fetch that already happened, not a static config.
**3. It created a `Kustomization` object** (confusingly, the same name as
the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. —
related but not identical). `kustomize-controller`'s job, on its own
interval: fetch the artifact `source-controller` produced, run the real
`kustomize` tool against `.spec.path` inside it (the exact same tool a
plain `kustomize build apps/podinfo` would run locally against those
`kustomization.yaml` files), and apply the resulting objects via the
Kubernetes API — the automated equivalent of you running
`kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer,
forever. `prune: true` (set on all the `Kustomization` objects in this
repo) means it also *deletes* anything it previously created that's no
longer present in the current git state — that's what makes it
self-healing rather than just "apply once."
```sh
kubectl -n flux-system get kustomization
```
**This is also why `apps/podinfo`, `apps/hello-app`, and
`apps/kubernetes-dashboard` started deploying without you ever running
`kubectl apply` on them.** Bootstrap's own `Kustomization` watches
`clusters/homelab` with `prune: true`. `clusters/homelab/apps.yaml`
(already sitting in the repo, hand-written earlier in this project) itself
just *defines more `Kustomization` objects*, one per app — so the first
one picks it up as part of its own normal reconcile, creates those three
child `Kustomization`s, and each of *those* then does its own
fetch-and-apply against its own app directory. It's the same mechanism
recursing, not a special case.
**4. It stored your credential as a Kubernetes Secret** — `--password`
here is the `flux-write` token, not an account password. Bootstrap
base64-wraps it into a Secret that matches what the `GitRepository`'s
`.spec.secretRef` points at:
```sh
kubectl -n flux-system get secret flux-system -o yaml
```
The `password` field is base64 (`| base64 -d` to read it) — this is
exactly the same credential `git clone https://user:token@host/repo` would
use, just read by `source-controller` on every fetch instead of typed by
you once. It's never written to `k8s`'s own filesystem.
**5. It committed that config back into the repo itself** — `git pull` in
your desktop clone and look at `clusters/homelab/flux-system/`. The
`GitRepository`/`Kustomization` objects you just inspected live in the
cluster *because* those exact YAML files are committed there — so
rebuilding this cluster from scratch would mean running `flux bootstrap`
again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and
landing in the identical state. The fact that Flux watches this repo is
itself declared *in* this repo.
If you want to watch a reconcile happen live rather than just inspect the
end state:
```sh
kubectl -n flux-system logs deploy/source-controller -f
```
then in another terminal, make any commit and push it — you'll see the
next fetch pick it up within the interval.
## Verifying the GitOps loop actually works
```sh
flux get kustomizations --watch
```
Watch until `podinfo`, `hello-app`, and `kubernetes-dashboard` all show
`Ready: True`, then confirm pods actually landed:
```sh
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n kubernetes-dashboard get pods
```
If that's all healthy, the entire chain — Forgejo repo → Flux → this
cluster — is working end to end with nothing manually `kubectl apply`'d.
## Exposing podinfo/hello-app through Caddy
k3s's bundled ingress controller (Traefik) is already listening on this
node's own IP, port 80, routing by the `Host:` header from each app's
`Ingress` resource (already defined in `apps/podinfo` and
`apps/hello-app`). Caddy just needs to forward matching requests there.
Two DNS records, as CNAMEs pointed at `git.boglabob.com` rather than
duplicating its IP directly — CNAME means "this name is an alias for that
one," so there's one place (that record) to update if the underlying IP
ever changes, instead of several:
```
podinfo.boglabob.com CNAME git.boglabob.com
hello.boglabob.com CNAME git.boglabob.com
```
Then in Caddy's own config, using the VM's pinned IP from earlier:
```
podinfo.boglabob.com {
reverse_proxy http://<VM_IP>:80
}
hello.boglabob.com {
reverse_proxy http://<VM_IP>:80
}
```
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
The Dashboard and the k3s API server are deliberately **not** here —
`docs/SETUP.md` steps 12–13 cover why (both are cluster-admin-capable, and
exposing either publicly is the exact pattern behind real breaches like
Tesla's 2018 incident) and how to reach them instead (`kubectl
port-forward`, and LAN/tunnel-only kubectl access). Both apply to this
cluster exactly as written there.
## Registering the Forgejo Actions runner
`docs/SETUP.md` step 9 applies as written — it's about the `k8s` user and
rootless Podman, not about which cluster exists. Worth understanding
before running it: the runner would normally get root-equivalent power
over its host via a mounted `docker.sock`; instead it runs as `k8s` itself
using rootless Podman's own API socket, and `build-hello-app.yml` builds
images with `kaniko` (no daemon, no elevated privileges needed at all) —
so nothing in that pipeline ever touches `sudo`.
## Tearing this down
Once the Terraform track is ready and `tofu apply` brings up the real
3-node cluster, remove this one:
```sh
virsh -c qemu:///system destroy k3s-manual # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2 # base image isn't
# attached to the VM
# directly, needs its
# own delete
rm ~/.kube/config-manual
rm -rf ~/vms
```
Nothing else needs cleaning up — Flux's own state lived entirely inside
that VM's cluster and goes away with it. The Forgejo repo, both tokens,
and the Forgejo Actions runner registration are all cluster-independent
and carry over unchanged; point `flux bootstrap` at the new cluster's
kubeconfig once it exists.
A real, working single-node k3s cluster. Next: `docs/03-flux.md` — you'll
need bootstrap steps 3-4 (the `k3s_token` secret isn't used until stage 4,
but the Forgejo tokens are needed starting now) before continuing.

408
docs/03-flux.md Normal file
View file

@ -0,0 +1,408 @@
# Stage 3: Flux, Helm, and this repo's apps
By the end of stage 2 (`docs/02-k3s.md`) you have a working k3s cluster and
a kubeconfig at `~/.kube/config-manual`. This stage also needs bootstrap
steps 3-4 from `docs/01-bootstrap.md` — the Forgejo repo pushed, and the
`flux-write` token generated (the `k3s_token` secret still isn't used until
stage 4).
This stage bootstraps Flux against that cluster, and through it deploys
every app in `apps/`: two managed via Helm (podinfo, headlamp — this is
also where Helm itself gets explained, since Flux is the only way this
project ever touches Helm) and one via plain manifests plus a CI pipeline
(hello-app).
Everything below runs as `k8s` on the T630, with
`export KUBECONFIG=~/.kube/config-manual` already set from stage 2.
---
## 1. Install the Flux CLI
Same no-sudo, direct-binary-release pattern as `kubectl` in stage 2
(`brew` assumes Homebrew, which isn't a given on a bare Debian box):
```sh
FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version
```
## 2. Bootstrap Flux against Forgejo
Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this
uses the generic git bootstrap — over HTTPS with the `flux-write` token
from `docs/01-bootstrap.md` step 4, not SSH (Forgejo's git-SSH port turned
out not to be reliably reachable from either this desktop or the T630 —
see the earlier troubleshooting in this project's history):
```sh
flux check --pre --kubeconfig ~/.kube/config-manual
flux bootstrap git \
--url=https://git.boglabob.com/codegit/cloud-demo \
--branch=main \
--path=clusters/homelab \
--username=codegit \
--password=<FLUX_WRITE_TOKEN> \
--token-auth \
--kubeconfig ~/.kube/config-manual
```
`--password` here is the `flux-write` token, not an actual account
password. Flux stores it as a Kubernetes Secret in the `flux-system`
namespace once bootstrap completes — that Secret is what
`ImageUpdateAutomation` (step 7, below) reuses to push commits back, not
anything held by `k8s` itself. Clear this command from `k8s`'s shell
history afterwards (or prefix it with a space first, if
`HISTCONTROL=ignorespace` is set) since the token was passed as a plain
argument.
This populates `clusters/homelab/flux-system/` and, because
`clusters/homelab/apps.yaml` already declares `Kustomization` objects for
`apps/podinfo`, `apps/hello-app`, and `apps/headlamp`, all three start
reconciling immediately.
### What that one command actually did, mechanism by mechanism
"Flux" isn't one program watching your repo by magic — it's a handful of
ordinary Kubernetes controllers (just Deployments, like anything else
you'd run on the cluster), each one understanding a couple of **Custom
Resource Definitions** — CRDs extend the Kubernetes API with new object
kinds, the same way `Deployment` or `Service` are built-in kinds.
`GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing
about any of this is special-cased outside the normal Kubernetes API —
it's the exact same "define an object, a controller notices it and acts"
loop that runs the whole rest of Kubernetes.
That one `flux bootstrap` command did five genuinely separate things. Go
look at each — these commands work against the cluster you already
bootstrapped:
**1. It installed the controllers** — plain Kubernetes Deployments, no
different in kind from anything else running on the cluster:
```sh
kubectl -n flux-system get deployments
```
`source-controller`, `kustomize-controller`, `helm-controller`,
`notification-controller`. Alongside them, it registered the CRDs those
controllers understand:
```sh
kubectl get crds | grep fluxcd
```
**2. It created a `GitRepository` object** — this is the whole "what repo
am I watching" declaration, and nothing more. `source-controller`'s job is
entirely mechanical: every `interval` (default 1m), do a real `git fetch`
against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that
tree into a `.tar.gz`, and record its location in `.status.artifact`.
That's the entire job — it doesn't know or care what's *in* the repo.
```sh
kubectl -n flux-system get gitrepository flux-system -o yaml
```
Look at `.status.conditions` and `.status.artifact` — that's the result of
an actual git fetch that already happened, not a static config.
**3. It created a `Kustomization` object** (confusingly, the same name as
the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. —
related but not identical). `kustomize-controller`'s job, on its own
interval: fetch the artifact `source-controller` produced, run the real
`kustomize` tool against `.spec.path` inside it (the exact same tool a
plain `kustomize build apps/podinfo` would run locally against those
`kustomization.yaml` files), and apply the resulting objects via the
Kubernetes API — the automated equivalent of you running
`kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer,
forever. `prune: true` (set on all the `Kustomization` objects in this
repo) means it also *deletes* anything it previously created that's no
longer present in the current git state — that's what makes it
self-healing rather than just "apply once."
```sh
kubectl -n flux-system get kustomization
```
**This is also why `apps/podinfo`, `apps/hello-app`, and `apps/headlamp`
started deploying without you ever running `kubectl apply` on them.**
Bootstrap's own `Kustomization` watches `clusters/homelab` with
`prune: true`. `clusters/homelab/apps.yaml` (already sitting in the repo)
itself just *defines more `Kustomization` objects*, one per app — so the
first one picks it up as part of its own normal reconcile, creates those
three child `Kustomization`s, and each of *those* then does its own
fetch-and-apply against its own app directory. It's the same mechanism
recursing, not a special case.
**4. It stored your credential as a Kubernetes Secret** — `--password`
here is the `flux-write` token, not an account password. Bootstrap
base64-wraps it into a Secret that matches what the `GitRepository`'s
`.spec.secretRef` points at:
```sh
kubectl -n flux-system get secret flux-system -o yaml
```
The `password` field is base64 (`| base64 -d` to read it) — this is
exactly the same credential `git clone https://user:token@host/repo` would
use, just read by `source-controller` on every fetch instead of typed by
you once. It's never written to `k8s`'s own filesystem.
**5. It committed that config back into the repo itself** — `git pull` in
your desktop clone and look at `clusters/homelab/flux-system/`. The
`GitRepository`/`Kustomization` objects you just inspected live in the
cluster *because* those exact YAML files are committed there — so
rebuilding this cluster from scratch would mean running `flux bootstrap`
again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and
landing in the identical state. The fact that Flux watches this repo is
itself declared *in* this repo. (Stage 4 leans on exactly this property —
re-bootstrapping against a brand new cluster reproduces the same deployed
state with no manual replay.)
If you want to watch a reconcile happen live rather than just inspect the
end state:
```sh
kubectl -n flux-system logs deploy/source-controller -f
```
then in another terminal, make any commit and push it — you'll see the
next fetch pick it up within the interval.
## 3. Verify the GitOps loop actually works
```sh
flux get kustomizations --watch
```
Watch until `podinfo`, `hello-app`, and `headlamp` all show `Ready: True`,
then confirm pods actually landed:
```sh
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods
```
If that's all healthy, the entire chain — Forgejo repo → Flux → this
cluster — is working end to end with nothing manually `kubectl apply`'d.
## 4. podinfo: Flux's first Helm-managed app
podinfo is deployed first deliberately: it needs nothing this project
builds itself — no CI, no custom image, just a public chart — so a green
`flux get kustomization podinfo` proves the whole
source-controller → helm-controller → cluster loop works before hello-app
adds a CI dependency on top of it, and before headlamp adds a
cluster-admin RBAC concern on top of *that*.
A **Helm chart** is a packaged bundle of Kubernetes YAML templates plus a
`values.yaml` of defaults — installing one produces the same kind of
`Deployment`/`Service`/etc. objects you'd otherwise hand-write, just
parameterized and versioned. Used directly (outside this project, with
nothing GitOps involved), that's a two-step, one-shot, imperative
process:
```sh
helm repo add podinfo https://stefanprodan.github.io/podinfo # register the chart index
helm install podinfo podinfo/podinfo --set replicaCount=1 # render + apply, once
```
Nothing here uses the `helm` CLI at all — Flux replaces both of those
steps with two declarative objects that `helm-controller` reconciles
continuously instead of once:
- **`apps/podinfo/helmrepository.yaml`** (a `HelmRepository`) is the
declarative form of `helm repo add` — it just tells `source-controller`
where the chart index lives and how often to refresh it (`interval`).
On its own it deploys nothing.
- **`apps/podinfo/helmrelease.yaml`** (a `HelmRelease`) is the declarative
form of `helm install`/`helm upgrade` — its `values:` block is exactly
what would otherwise be `-f values.yaml`/`--set` flags on the CLI, and
its `chart.spec.version` range (`>=6.0.0`) is what keeps it current:
`helm-controller` re-installs whenever a new matching chart version
shows up in the repository, on its own `interval`, with no one running
`helm upgrade` by hand.
Both files already have comments walking through their specific fields —
worth reading now that you know what problem they're solving.
## 5. headlamp: a cluster-admin UI, accessed on demand
`apps/headlamp/` is deployed the same Helm-via-Flux way as podinfo (see
its `helmrepository.yaml`/`helmrelease.yaml`), plus one more piece:
`apps/headlamp/rbac.yaml` creates a `ServiceAccount` bound to the
`cluster-admin` `ClusterRole`, with a long-lived token Secret. Headlamp
grants whatever its logged-in identity can do — with that token, that's
full cluster-admin. (This project used the official Kubernetes Dashboard
originally; it's since been swapped for Headlamp, which the Dashboard
project itself now points people toward, but the security posture is
identical.)
Publicly exposing a cluster-admin UI (even behind a login page) is the
exact pattern behind real breaches (Tesla, 2018: an internet-reachable,
unauthenticated Dashboard). So: no ingress, no standing hostname — only a
port-forward you open when you need it and close when you don't:
```sh
kubectl -n headlamp port-forward svc/headlamp 8443:80
```
Headlamp's chart doesn't terminate TLS on its own Service (that's left to
whatever fronts it in a real deployment — here, nothing does, since this
never leaves `localhost`), so open plain `http://localhost:8443`, not
`https://`. Log in by pasting the token:
```sh
kubectl -n headlamp get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d
```
## 6. Register the Forgejo Actions runner
hello-app (next) is the only app in this repo whose image *this project's
own CI* builds — that needs a self-hosted Forgejo Actions runner; there's
no shared runner pool. The runner normally gets root-equivalent power over
its host by mounting `/var/run/docker.sock` (anyone who can push a
workflow file effectively gets root there). Instead: it runs as the same
unprivileged `k8s` user from `docs/01-bootstrap.md` step 2, using rootless
Podman's own socket instead of Docker's — no root anywhere in this
pipeline. `.forgejo/workflows/build-hello-app.yml` already builds images
with `kaniko`, which needs no daemon and no elevated privileges at all.
The runner's job containers (kaniko, opentofu once stage 4 wires that
workflow in too) never get the libvirt socket or `k8s`'s home directory
mounted in — only the Podman socket, needed to launch those job containers
in the first place — so a compromised workflow can spawn containers as
`k8s`, but can't directly touch the VMs or Terraform state. Worth knowing
given the runner lives on the same box/user as the cluster's own
infrastructure; fine for a demo-sized project, but if this ever handles
anything sensitive, move the runner to its own user or VM so a breakout
doesn't share a blast radius with the cluster.
1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket:
```sh
systemctl --user enable --now podman.socket
echo $XDG_RUNTIME_DIR # note this path, e.g. /run/user/1001
```
2. Instance admin: `Site Administration → Actions → Runners`, confirm
Actions is enabled.
3. Repo: `Settings → Actions → Runners → Create new runner`, copy the
registration token.
4. Register and run the runner as a rootless Podman container, pointed at
the Podman socket from step 1 instead of docker.sock:
```sh
# still as k8s
podman volume create forgejo-runner-data
podman run -d --name forgejo-runner --restart unless-stopped \
-e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
-v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
-v forgejo-runner-data:/data \
code.forgejo.org/forgejo/runner:6 \
forgejo-runner register --no-interactive \
--instance https://git.boglabob.com \
--token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
```
The registered runner picks up both workflows in `.forgejo/workflows/` —
`terraform.yml`'s `container:` image (stage 4) and
`build-hello-app.yml`'s kaniko image (next) are both launched through
that same rootless Podman socket.
5. Repo `Settings → Secrets and Variables → Actions`, add:
- Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications`
on your Forgejo user, scope `package:write`) used to push images.
- Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org.
If the `hello-app` package ends up private (Forgejo package visibility
follows repo visibility by default), create the cluster-side pull secret
and uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`:
```sh
kubectl -n hello-app create secret docker-registry forgejo-registry \
--docker-server=git.boglabob.com \
--docker-username=<FORGEJO_USER> \
--docker-password=<FORGEJO_TOKEN>
```
## 7. hello-app: plain manifests, CI, and image automation
Unlike podinfo/headlamp, `apps/hello-app/` isn't a chart — it's *this*
project's own code (source + Dockerfile under `apps/hello-app/src/`), so
there's no upstream chart to track and plain
`Deployment`/`Service`/`Ingress` manifests are simpler to read for
something this small (see `apps/hello-app/deployment.yaml`'s comments).
What it does need, that podinfo/headlamp don't, is a way to notice when CI
publishes a new image and roll it out — that's
`apps/hello-app/image-automation.yaml`'s job
(`ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`, all commented in
that file).
Exercise the whole loop end to end:
```sh
sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push
```
Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the
page with the current commit SHA and build time — see
`apps/hello-app/src/index.html`) → pushes a new tag to
`git.boglabob.com/codegit/hello-app` → Flux's `ImageRepository` picks it up
within a minute → `ImageUpdateAutomation` commits the new tag back to
`apps/hello-app/deployment.yaml` → the `hello-app` Kustomization
reconciles → `kubectl -n hello-app get pods` shows a new pod, and
`https://hello.boglabob.com` shows the new commit SHA/badge (once step 8
below exposes it).
## 8. Expose podinfo/hello-app through Caddy
k3s's bundled ingress controller (Traefik) is already listening on this
node's own IP, port 80, routing by the `Host:` header from each app's
`Ingress` resource (already defined in `apps/podinfo` and
`apps/hello-app`). Caddy just needs to forward matching requests there —
headlamp and the k3s API server are deliberately **not** exposed this way;
step 5 above and `docs/04-tofu.md` step 5 cover why and how to reach them
instead.
Two DNS records, as CNAMEs pointed at `git.boglabob.com` rather than
duplicating its IP directly — CNAME means "this name is an alias for that
one," so there's one place (that record) to update if the underlying IP
ever changes, instead of several:
```
podinfo.boglabob.com CNAME git.boglabob.com
hello.boglabob.com CNAME git.boglabob.com
```
Then in Caddy's own config, using the VM's pinned IP from stage 2:
```
podinfo.boglabob.com {
reverse_proxy http://<VM_IP>:80
}
hello.boglabob.com {
reverse_proxy http://<VM_IP>:80
}
```
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
```sh
curl https://podinfo.boglabob.com/
curl https://hello.boglabob.com/
```
`docs/Caddyfile.example` has the multi-node version of these same blocks —
stage 4 points you back at it once there are three node IPs to
load-balance across instead of one.
Next: `docs/04-tofu.md` — everything above still works exactly as-is; that
stage replaces the one hand-built VM under it with a Terraform-provisioned
3-node cluster and points this same Flux config at it.

274
docs/04-tofu.md Normal file
View file

@ -0,0 +1,274 @@
# Stage 4: Graduating to Terraform/OpenTofu
Stages 2-3 got you a real, working cluster with Flux managing every app in
this repo — on one hand-built VM. This stage replaces that VM with a
proper 3-node cluster provisioned by Terraform/OpenTofu via the
`dmacvicar/libvirt` provider, then points the exact same Flux config at
it. Nothing in `clusters/homelab/` or `apps/` changes — that's the point:
this repo's GitOps state was never tied to *how* the cluster under it got
built.
You'll need `docs/01-bootstrap.md` steps 3-4: the `k3s_token` secret
(generated but unused until now) and the `k8s-readonly` token.
## Where this track actually stands
Earlier notes on this project flagged this whole track as "blocked" on a
provider schema mismatch between the 0.8.x and 0.9.x lines of
`dmacvicar/libvirt`, without pinning down exactly what broke. That's worth
re-examining rather than taking on faith, both because it matters for
whether you should trust `main.tf` and because the process of checking it
is itself a reasonable thing to learn from — so here's what's actually
been confirmed, and what hasn't, as of this pass:
**Confirmed: this isn't really a "0.8.x vs 0.9.x" ambiguity at all.**
`terraform/versions.tf` pins `~> 0.8`. Checking the provider's own release
history: v0.9.0 (Nov 2025) was an intentional, permanent fork to a fully
regenerated schema that maps 1:1 onto libvirt's XML — the maintainer's own
release notes say so explicitly, and describe keeping 0.8.x alive in
parallel specifically for people who don't want that rewrite. So `~> 0.8`
doesn't risk drifting onto 0.9.x schema by accident; it's a deliberate,
stable choice, not an unresolved question.
**Confirmed: every resource in `main.tf` matches the real 0.8.x schema.**
Checked directly against this provider's own docs at git tag `v0.8.3`
(`website/docs/r/{pool,volume,cloudinit,domain,network}.html.markdown` in
`dmacvicar/terraform-provider-libvirt`) — not from memory, and not from
whatever an LLM's training data assumes a "libvirt provider" looks like,
which is the trap the project's own earlier notes were rightly worried
about. Every attribute `main.tf` uses lines up: `libvirt_pool`'s
`type = "dir"`; `libvirt_volume`'s `base_volume_id`/`size`/`format`;
`libvirt_cloudinit_disk`'s `user_data`/`network_config`;
`libvirt_domain`'s `disk { volume_id }`, `network_interface { network_id,
wait_for_lease }`, and `console { type, target_type, target_port }`;
`libvirt_network`'s `mode`/`addresses`/`dhcp { enabled }`/`dns { enabled }`.
None of it uses 0.9.x-only shapes (nested `create.content.url`,
`backing_store`, `capacity` instead of `size`, etc.).
**Not yet confirmed: whether it actually applies.** Static schema-matching
isn't the same as a real `tofu apply` succeeding — that needs the T630's
actual libvirt socket, which nothing has exercised end-to-end yet. If
you're picking this stage up, this is the real remaining unknown, and a
reasonable order to close it:
1. **Free, no-VM checkpoint first**: `.forgejo/workflows/terraform.yml`
already runs `tofu init -backend=false` + `tofu validate` on every push
to `terraform/**` — that's a real, automated check of exactly the
schema question above, running today. Check its latest result in
Forgejo Actions before doing anything else; if it's failing, the error
message will point at a specific resource/attribute far faster than
re-deriving the whole schema by hand.
2. **`tofu init` on the T630** (step 2 below) and check
`terraform/.terraform.lock.hcl` afterwards — confirm it actually
resolved a `0.8.x` version, not something unexpected.
3. **`tofu plan`, then `tofu apply`**, and if any single resource fails,
treat that resource in isolation: re-check its specific arguments
against `website/docs/r/<resource>.html.markdown` at whatever version
`.terraform.lock.hcl` actually resolved (not `main` — the docs move
with the schema, and `main` may already reflect a newer 0.8.x patch or
even post-fork changes), rather than guessing at a fix. That's a more
targeted version of the same check already done above for the whole
file.
4. Once a full `tofu apply` succeeds once, the remaining steps below are
what actually plug the result into the rest of this project.
Sizing is deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB
total) so this stays a demo rather than competing with whatever else is
already running on the T630.
---
## 1. Install OpenTofu
Installing the package needs sudo, so that part is you (`maq`), not `k8s`.
Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu`
afterwards with no further root involvement:
```sh
# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh
```
## 2. Provision the VMs with OpenTofu
Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved —
clone using the `k8s-readonly` token from `docs/01-bootstrap.md` step 4.
`k8s` has no keyring (it's headless, no desktop session), so this uses
`git credential-store` — a plaintext file, `chmod 600`'d, holding only the
read-only token:
```sh
# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials
cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from docs/01-bootstrap.md step 2), k3s_token (from step 3). Defaults for
# network/sizing are fine to start.
tofu init
tofu plan
tofu apply
```
For any later change to `terraform/`: edit and push as `maq` as usual, then
`cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`.
This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the
`k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from
anything else already using libvirt on this box, including the `default`
network stage 2's manual VM used). Cloud-init installs k3s on each on
first boot — give it ~2 minutes after `apply` finishes.
## 3. Get kubectl talking to the new cluster
```sh
mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
| sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab
export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes # expect 3 Ready nodes
```
This works as-is because you're running it on the T630, which can reach the
`10.20.30.0/24` network directly. To also use kubectl from your own laptop,
either `scp` this kubeconfig over and open an SSH tunnel first
(`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's
`server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s`
whenever you need kubectl — simplest by far for a project this size.
## 4. Re-bootstrap Flux against the new cluster
This is the exact same `flux bootstrap git` command from `docs/03-flux.md`
step 2, just pointed at `~/.kube/config-homelab` instead of
`~/.kube/config-manual` — and it needs to actually run again, not be
skipped. The new 3-node cluster has its own fresh etcd/SQLite; nothing
about the manual VM's cluster state carries over to it, `flux-system`
namespace included. What *does* carry over is this repo:
```sh
flux check --pre --kubeconfig ~/.kube/config-homelab
flux bootstrap git \
--url=https://git.boglabob.com/codegit/cloud-demo \
--branch=main \
--path=clusters/homelab \
--username=codegit \
--password=<FLUX_WRITE_TOKEN> \
--token-auth \
--kubeconfig ~/.kube/config-homelab
```
Because `clusters/homelab/flux-system/` already holds the exact manifests
stage 3's bootstrap generated, this run doesn't need to commit anything
new back to the repo — it just applies that already-correct config to a
cluster that doesn't have it yet. That's the whole point of GitOps having
been the deploy mechanism all along: the desired state was never tied to
the specific cluster instance, so pointing the same bootstrap command at a
new kubeconfig reproduces it exactly. Confirm:
```sh
flux get kustomizations --watch
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods
```
## 5. kubectl access from elsewhere on the LAN, or remotely
Best practice for the Kubernetes API server is the same principle as
Headlamp (`docs/03-flux.md` step 5): never put 6443 on the public
internet if you can avoid it, because a leaked credential there is a full
cluster compromise.
Node IPs (`10.20.30.0/24` by default) live on the private libvirt network
from step 2 — only the T630 itself can reach them directly, which is
actually a nice side effect: even the rest of your LAN can't touch the API
server without going through the T630 first. Two ways to do that:
- **SSH into the T630 as `k8s`** and run kubectl there directly (same as
step 3) — simplest, and what this whole project assumes by default.
- **Tunnel from another machine** (your laptop, or a phone via Termux, etc.):
```sh
ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
```
then point a local kubeconfig's `server:` at `https://127.0.0.1:6443`
(copy the kubeconfig from step 3 and edit that one field). The cert
validates because `k8s-api.boglabob.com` is in the server's TLS SAN list
(`terraform/variables.tf`'s `k8s_api_hostname`) — add it to
`/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine
you're tunneling from and use that as the `server:` host instead of the
raw IP, so the hostname in the URL matches a name the cert actually
covers. (This SAN entry is new in stage 4 — the manual VM's cloud-init
in `docs/02-k3s.md` never set one, since nothing needed LAN-wide access
to it.)
- **From outside your home network entirely**: Tailscale or WireGuard on
the T630, then the SSH tunnel above over the Tailscale/WireGuard link
instead of the open internet. Reasonable next stretch goal once the core
loop is working — don't port-forward 22 or 6443 on your router for this.
The `admin-user` bearer token (`docs/03-flux.md` step 5) also works for
kubectl over the same tunnel, if you'd rather not manage the client-cert
kubeconfig.
## 6. Point Caddy and Headlamp at the new cluster
Same steps as `docs/03-flux.md` step 8 (Caddy) and step 5 (Headlamp) — the
mechanism is identical, only the node IP(s) changed. Caddy now has three
node IPs to pick from instead of one, so use `docs/Caddyfile.example`'s
`reverse_proxy` blocks (which list all three) rather than the single-IP
version from stage 3:
```sh
tofu output node_ips
```
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
Headlamp's port-forward command is unchanged (`kubectl` just needs
`KUBECONFIG` pointed at `~/.kube/config-homelab` now).
## 7. Tearing down the manual VM
Now that the real 3-node cluster is up, remove stage 2's throwaway one:
```sh
virsh -c qemu:///system destroy k3s-manual # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2 # base image isn't
# attached to the VM
# directly, needs its
# own delete
rm ~/.kube/config-manual
rm -rf ~/vms
```
Nothing else needs cleaning up — Flux's own state lived entirely inside
that VM's cluster and goes away with it. The Forgejo repo, both tokens,
and the Forgejo Actions runner registration are all cluster-independent
and already carried over unchanged in step 4 above.
## Stretch goals, roughly in order
- **Remote access**: Tailscale or WireGuard on the k3s server node, for
kubectl/Headlamp access from outside the LAN without opening anything
publicly (step 5).
- **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if
`boglabob.com` isn't publicly reachable on 80/443).
- **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the
`K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret
store.
- **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as
podinfo (HelmRepository + HelmRelease under `apps/`).
- **HA**: add a second k3s server node and switch from SQLite to embedded
etcd (`--cluster-init` on the first server, `--server` join on the second).

12
docs/Caddyfile.example
View file

@ -14,12 +14,12 @@
# forward the request (Host header included) to any node. Listing all three
# gives you free load-balancing/failover across nodes.
#
# Deliberately NOT here: the Kubernetes Dashboard and the k3s API server.
# Both grant cluster-admin-level control, and routing either through a
# public-facing reverse proxy is the exact pattern behind real-world
# cluster breaches (e.g. Tesla, 2018 — an exposed, unauthenticated
# Dashboard). Both stay LAN-only / on-demand instead — see docs/SETUP.md
# steps 12-13.
# Deliberately NOT here: Headlamp and the k3s API server. Both grant
# cluster-admin-level control, and routing either through a public-facing
# reverse proxy is the exact pattern behind real-world cluster breaches
# (e.g. Tesla, 2018 — an exposed, unauthenticated Dashboard). Both stay
# LAN-only / on-demand instead — see docs/03-flux.md step 5 (Headlamp) and
# docs/04-tofu.md step 5 (kubectl/API access).
podinfo.boglabob.com {
reverse_proxy http://10.20.30.11:80 http://10.20.30.12:80 http://10.20.30.13:80

401
docs/SETUP.md
View file

@ -1,401 +0,0 @@
# Setup walkthrough (full, Terraform-driven)
This is the "do it properly" path — Terraform/OpenTofu provisioning all 3
VMs via the `dmacvicar/libvirt` provider. It's also the one currently
blocked on getting that provider's HCL right (0.8.x vs 0.9.x schema — see
conversation history). If you want a real cluster to learn Flux/GitOps on
*right now* without waiting on that, see `docs/QUICKSTART.md` instead — a
single manually-created VM, no Terraform, with teardown instructions for
switching over once this track is sorted. Steps 1–4 below are shared
between both guides.
Assumes: the T630 is an existing Debian box already running other
self-hosted services — this project installs alongside those as ordinary
packages (`qemu-kvm`/`libvirt`), not a hypervisor OS replacing Debian, and
is sized deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total)
so it stays a demo rather than competing with what's already running. Forgejo
is already running and reachable at `https://git.boglabob.com`, and you can
point DNS records under `boglabob.com` at hosts on your network (directly,
or via whatever reverse proxy/tunnel already gets `git.boglabob.com` there).
---
## 1. Install KVM/libvirt on the T630
Ordinary packages, no reboot into an installer, nothing else on the box is
touched:
```sh
# on the T630
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst
# confirm hardware virtualization is available (T630's Xeons support it)
sudo kvm-ok
```
## 2. Create the unprivileged 'k8s' user
One dedicated, no-sudo user for everything this project touches: driving
Terraform/kubectl/flux against libvirt here, and running the Forgejo Actions
runner later (step 9). It needs group membership to talk to libvirt — that's
a one-time root action; nothing it does afterwards needs `sudo`.
```sh
sudo useradd -m -s /bin/bash k8s # one-time, needs root to create the user itself
sudo usermod -aG libvirt,kvm k8s
sudo loginctl enable-linger k8s # lets its services keep running after logout
# as k8s, from here on (sudo -iu, not su -, since k8s has no password set):
sudo -iu k8s
ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/id_ed25519 # only needed if you'll SSH in as k8s day-to-day
virsh -c qemu:///system list --all # sanity check: should run with no permission error, no sudo
```
Do the rest of this guide logged in as `k8s` on the T630 itself (`ssh
k8s@t630`) — node IPs (step 5) live on a private libvirt network that's only
directly reachable from the T630, so this is the simplest place to run
`tofu`/`kubectl`/`flux` from. (If you'd rather drive Terraform from your own
workstation instead, see the `libvirt_uri` comment in
`terraform/terraform.tfvars.example` — you'll then need an SSH tunnel for
kubectl/flux to reach node IPs.)
## 3. Generate the secrets Terraform needs
```sh
openssl rand -hex 32 # -> k3s_token
```
## 4. Push this repo to Forgejo
Doing this before provisioning (rather than after) means `k8s` can get the
repo with a plain `git clone` in step 5, instead of needing a one-off copy
handed to it — and any future Terraform change just needs a `git pull`.
Repo/owner used throughout this guide: `codegit/cloud-demo` (already baked
into `apps/hello-app/deployment.yaml` and `image-automation.yaml`'s image
references — no placeholder-swapping needed).
1. On Forgejo (`https://git.boglabob.com`), as `codegit`: **+ → New
Repository** → name `cloud-demo`. Leave it empty — don't initialize with
a README/`.gitignore`/license, since this repo already has its own.
Visibility (public/private) is your call; either works, since access for
`k8s`/Flux/CI goes through the tokens below regardless.
2. Locally, wherever you're editing this repo (`maq`):
```sh
git init # if not already
git add .
git commit -m "initial scaffold"
git remote add origin https://git.boglabob.com/codegit/cloud-demo.git
git push -u origin main
```
3. Generate two access tokens (`Settings → Applications → Generate New
Token`), scoped as narrowly as Forgejo's token UI allows to repository
read/write:
- **`k8s-readonly`** — read-only. Used only for `k8s`'s own manual
`git clone`/`pull` on the T630 (step 5) — never leaves that box, isn't
used by anything automated.
- **`flux-write`** — read/write. Used once, as a `flux bootstrap`
argument (step 7); Flux stores it as a Kubernetes Secret inside the
cluster from then on (`ImageUpdateAutomation`'s commits back in step 10
reuse that same in-cluster Secret) — it's never written to `k8s`'s
filesystem at all.
Using HTTPS tokens instead of `k8s`'s SSH key (`~/.ssh/id_ed25519`, from
step 2) sidesteps an open question: Forgejo's git-SSH port isn't
reachable from this desktop through your router (see the SSH
troubleshooting earlier in this conversation), and whether it's reachable
from `k8s` on the T630 itself was never actually confirmed either. HTTPS
(443, via Caddy) is already proven to work, so both tokens use that
instead. Copy both token values now — Forgejo only shows them once.
## 5. Provision the VMs with OpenTofu
Installing the package needs sudo, so that part is you (`maq`), not `k8s`.
Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu`
afterwards with no further root involvement:
```sh
# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh
```
Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved —
clone using the `k8s-readonly` token from step 4. `k8s` has no keyring (it's
headless, no desktop session), so this uses `git credential-store` — a
plaintext file, `chmod 600`'d, holding only the read-only token:
```sh
# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials
cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from step 2), k3s_token. Defaults for network/sizing are fine to start.
tofu init
tofu plan
tofu apply
```
For any later change to `terraform/`: edit and push as `maq` as usual, then
`cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`.
This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the
`k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from
anything else already using libvirt on this box). Cloud-init installs k3s on
each on first boot — give it ~2 minutes after `apply` finishes.
## 6. Get kubectl talking to the cluster
`kubectl` itself was never actually installed anywhere earlier in this
guide despite being listed as a prerequisite — install it now (as `k8s`,
no sudo needed, same pattern as the OpenTofu install):
```sh
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
```
```sh
mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
| sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab
export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes # expect 3 Ready nodes
```
This works as-is because you're running it on the T630, which can reach the
`10.20.30.0/24` network directly. To also use kubectl from your own laptop,
either `scp` this kubeconfig over and open an SSH tunnel first
(`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's
`server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s`
whenever you need kubectl — simplest by far for a project this size.
## 7. Bootstrap Flux against Forgejo
Forgejo isn't a Flux-native provider (unlike GitHub/GitLab), so use the
generic git bootstrap — over HTTPS with the `flux-write` token from step 4,
not SSH (same reachability reasoning as step 5):
Install the Flux CLI the same no-sudo, no-package-manager way as `kubectl`
(`brew` assumes Homebrew, which isn't a given on a bare Debian box):
```sh
FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version
```
```sh
flux check --pre --kubeconfig ~/.kube/config-homelab
flux bootstrap git \
--url=https://git.boglabob.com/codegit/cloud-demo \
--branch=main \
--path=clusters/homelab \
--username=codegit \
--password=<FLUX_WRITE_TOKEN> \
--token-auth \
--kubeconfig ~/.kube/config-homelab
```
`--password` here is the `flux-write` token, not an actual account
password. Flux stores it as a Kubernetes Secret in the `flux-system`
namespace once bootstrap completes — that Secret is what
`ImageUpdateAutomation` (step 10) reuses to push commits back, not anything
held by `k8s` itself. Clear this command from `k8s`'s shell history
afterwards (or prefix it with a space first, if `HISTCONTROL=ignorespace`
is set) since the token was passed as a plain argument.
This populates `clusters/homelab/flux-system/` and, because
`clusters/homelab/apps.yaml` already declares `Kustomization` objects for
`apps/podinfo` and `apps/hello-app`, both start reconciling immediately.
## 8. Verify the podinfo GitOps loop
```sh
flux get kustomizations --watch
kubectl -n podinfo get pods
```
Once it's `Ready`, point DNS at it and check in a browser (see step 11).
## 9. Enable Forgejo Actions and register a runner (rootless, no sudo)
Forgejo Actions needs a self-hosted runner — there's no shared runner pool.
The runner normally gets root-equivalent power over its host by mounting
`/var/run/docker.sock` (anyone who can push a workflow file effectively gets
root there). Instead: it runs as the same unprivileged `k8s` user from
step 2, using rootless Podman's own socket instead of Docker's — no root
anywhere in this pipeline. `build-hello-app.yml` already builds images with
kaniko, which needs no daemon and no elevated privileges at all.
The runner's job containers (kaniko, opentofu) never get the libvirt socket
or `k8s`'s home directory mounted in — only the Podman socket, needed to
launch those job containers in the first place — so a compromised workflow
can spawn containers as `k8s`, but can't directly touch the VMs or
Terraform state. Worth knowing given the runner lives on the same box/user
as the cluster's own infrastructure; fine for a demo-sized project, but if
this ever handles anything sensitive, move the runner to its own user or
VM so a breakout doesn't share a blast radius with the cluster.
1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket:
```sh
systemctl --user enable --now podman.socket
echo $XDG_RUNTIME_DIR # note this path, e.g. /run/user/1001
```
2. Instance admin: `Site Administration → Actions → Runners`, confirm
Actions is enabled.
3. Repo: `Settings → Actions → Runners → Create new runner`, copy the
registration token.
4. Register and run the runner as a rootless Podman container, pointed at
the Podman socket from step 1 instead of docker.sock:
```sh
# still as k8s
podman volume create forgejo-runner-data
podman run -d --name forgejo-runner --restart unless-stopped \
-e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
-v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
-v forgejo-runner-data:/data \
code.forgejo.org/forgejo/runner:6 \
forgejo-runner register --no-interactive \
--instance https://git.boglabob.com \
--token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
```
The registered runner picks up both workflows in `.forgejo/workflows/` —
`terraform.yml`'s `container:` image and `build-hello-app.yml`'s kaniko
image are both launched through that same rootless Podman socket.
5. Repo `Settings → Secrets and Variables → Actions`, add:
- Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications`
on your Forgejo user, scope `package:write`) used to push images.
- Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org.
If the `hello-app` package ends up private (Forgejo package visibility
follows repo visibility by default), create the cluster-side pull secret and
uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`:
```sh
kubectl -n hello-app create secret docker-registry forgejo-registry \
--docker-server=git.boglabob.com \
--docker-username=<FORGEJO_USER> \
--docker-password=<FORGEJO_TOKEN>
```
## 10. Exercise the full loop
```sh
sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push
```
Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the page
with the current commit SHA and build time — see `apps/hello-app/src/index.html`)
→ pushes a new tag to `git.boglabob.com/codegit/hello-app` → Flux's
`ImageRepository` picks it up within a minute → `ImageUpdateAutomation`
commits the new tag back to `apps/hello-app/deployment.yaml` → the
`hello-app` Kustomization reconciles → `kubectl -n hello-app get pods` shows
a new pod, and `https://hello.boglabob.com` shows the new commit SHA/badge.
## 11. Expose the apps through Caddy
Since Caddy (Podman) is already the front door for `git.boglabob.com`, route
`podinfo` and `hello-app` through it too — but not the Dashboard or the API
server; see steps 12-13 for why.
1. Add CNAME records for `podinfo.boglabob.com` and `hello.boglabob.com`
pointing at `git.boglabob.com` (matching how every other record for this
server is set up) — one source of truth for the Caddy host's IP, rather
than duplicating it across records.
2. Add the blocks from `docs/Caddyfile.example` to Caddy's config, filling
in your real node IPs (`tofu output node_ips`), and reload:
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
3. Check:
```sh
curl https://podinfo.boglabob.com/
curl https://hello.boglabob.com/
```
## 12. Access the Dashboard (LAN-only, on demand)
The Dashboard grants whatever its logged-in identity can do — with the
`admin-user` token from `apps/kubernetes-dashboard/rbac.yaml`, that's
cluster-admin. Publicly exposing that (even behind a login page) is the
exact pattern behind real breaches (Tesla, 2018: an internet-reachable,
unauthenticated Dashboard). So: no ingress, no standing hostname — only a
port-forward you open when you need it and close when you don't:
```sh
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
```
Then open `https://localhost:8443` and log in with the token:
```sh
kubectl -n kubernetes-dashboard get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d
```
## 13. kubectl access from elsewhere on the LAN, or remotely
Best practice for the Kubernetes API server is the same principle as the
Dashboard: never put 6443 on the public internet if you can avoid it,
because a leaked credential there is a full cluster compromise.
Node IPs (`10.20.30.0/24` by default) live on the private libvirt network
from step 5 — only the T630 itself can reach them directly, which is
actually a nice side effect: even the rest of your LAN can't touch the API
server without going through the T630 first. Two ways to do that:
- **SSH into the T630 as `k8s`** and run kubectl there directly (same as
step 6) — simplest, and what this whole guide assumes by default.
- **Tunnel from another machine** (your laptop, or a phone via Termux, etc.):
```sh
ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
```
then point a local kubeconfig's `server:` at `https://127.0.0.1:6443`
(copy the kubeconfig from step 6 and edit that one field). The cert
validates because `k8s-api.boglabob.com` is in the server's TLS SAN list
(`terraform/variables.tf`'s `k8s_api_hostname`) — add it to
`/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine
you're tunneling from and use that as the `server:` host instead of the
raw IP, so the hostname in the URL matches a name the cert actually
covers.
- **From outside your home network entirely**: Tailscale or WireGuard on
the T630, then the SSH tunnel above over the Tailscale/WireGuard link
instead of the open internet. Reasonable next stretch goal once the core
loop is working — don't port-forward 22 or 6443 on your router for this.
The `admin-user` bearer token (step 12) also works for kubectl over the same
tunnel, if you'd rather not manage the client-cert kubeconfig.
## Stretch goals, roughly in order
- **Remote access**: Tailscale or WireGuard on the k3s server node, for
kubectl/Dashboard access from outside the LAN without opening anything
publicly (step 13).
- **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if
`boglabob.com` isn't publicly reachable on 80/443).
- **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the
`K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret
store.
- **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as
podinfo (HelmRepository + HelmRelease under `apps/`).
- **HA**: add a second k3s server node and switch from SQLite to embedded
etcd (`--cluster-init` on the first server, `--server` join on the second).