cloud-demo/docs/03-flux.md

Stage 3: Flux, Helm, and this repo's apps

By the end of stage 2 (docs/02-k3s.md) you have a working k3s cluster and a kubeconfig at ~/.kube/config-manual. This stage also needs bootstrap steps 3-4 from docs/01-bootstrap.md — the Forgejo repo pushed, and the flux-write token generated (the k3s_token secret still isn't used until stage 4).

This stage bootstraps Flux against that cluster, and through it deploys every app in apps/: two managed via Helm (podinfo, headlamp — this is also where Helm itself gets explained, since Flux is the only way this project ever touches Helm) and one via plain manifests plus a CI pipeline (hello-app).

Everything below runs as k8s on the T630, with export KUBECONFIG=~/.kube/config-manual already set from stage 2.

---

1. Install the Flux CLI

Same no-sudo, direct-binary-release pattern as kubectl in stage 2 (brew assumes Homebrew, which isn't a given on a bare Debian box):

FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version

2. Bootstrap Flux against Forgejo

Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this uses the generic git bootstrap — over HTTPS with the flux-write token from docs/01-bootstrap.md step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably reachable from either this desktop or the T630 — see the earlier troubleshooting in this project's history):

flux bootstrap only installs its four core controllers (source-controller, kustomize-controller, helm-controller, notification-controller) unless told otherwise. hello-app's image automation (step 7 below — ImageRepository/ImagePolicy/ ImageUpdateAutomation) needs two more, image-reflector-controller and image-automation-controller, which are opt-in via --components-extra; without it, hello-app's Kustomization fails dry-run with "no matches for kind ImageRepository" since the CRD was never installed.

flux check --pre --kubeconfig ~/.kube/config-manual

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --components-extra=image-reflector-controller,image-automation-controller \
  --kubeconfig ~/.kube/config-manual

--password here is the flux-write token, not an actual account password. Flux stores it as a Kubernetes Secret in the flux-system namespace once bootstrap completes — that Secret is what ImageUpdateAutomation (step 7, below) reuses to push commits back, not anything held by k8s itself. Clear this command from k8s's shell history afterwards (or prefix it with a space first, if HISTCONTROL=ignorespace is set) since the token was passed as a plain argument.

This populates clusters/homelab/flux-system/ and, because clusters/homelab/apps.yaml already declares Kustomization objects for apps/podinfo, apps/hello-app, and apps/headlamp, all three start reconciling immediately.

What that one command actually did, mechanism by mechanism

"Flux" isn't one program watching your repo by magic — it's a handful of ordinary Kubernetes controllers (just Deployments, like anything else you'd run on the cluster), each one understanding a couple of Custom Resource Definitions — CRDs extend the Kubernetes API with new object kinds, the same way Deployment or Service are built-in kinds. GitRepository and Kustomization are two such kinds Flux adds. Nothing about any of this is special-cased outside the normal Kubernetes API — it's the exact same "define an object, a controller notices it and acts" loop that runs the whole rest of Kubernetes.

That one flux bootstrap command did five genuinely separate things. Go look at each — these commands work against the cluster you already bootstrapped:

1. It installed the controllers — plain Kubernetes Deployments, no different in kind from anything else running on the cluster:

kubectl -n flux-system get deployments

source-controller, kustomize-controller, helm-controller, notification-controller, plus image-reflector-controller and image-automation-controller from --components-extra above. Alongside them, it registered the CRDs those controllers understand:

kubectl get crds | grep fluxcd

2. It created a GitRepository object — this is the whole "what repo am I watching" declaration, and nothing more. source-controller's job is entirely mechanical: every interval (default 1m), do a real git fetch against .spec.url/.spec.ref; if the commit SHA changed, package that tree into a .tar.gz, and record its location in .status.artifact. That's the entire job — it doesn't know or care what's in the repo.

kubectl -n flux-system get gitrepository flux-system -o yaml

Look at .status.conditions and .status.artifact — that's the result of an actual git fetch that already happened, not a static config.

3. It created a Kustomization object (confusingly, the same name as the kustomization.yaml files already sitting in apps/podinfo/ etc. — related but not identical). kustomize-controller's job, on its own interval: fetch the artifact source-controller produced, run the real kustomize tool against .spec.path inside it (the exact same tool a plain kustomize build apps/podinfo would run locally against those kustomization.yaml files), and apply the resulting objects via the Kubernetes API — the automated equivalent of you running kubectl apply -f <(kustomize build apps/podinfo) yourself, on a timer, forever. prune: true (set on all the Kustomization objects in this repo) means it also deletes anything it previously created that's no longer present in the current git state — that's what makes it self-healing rather than just "apply once."

kubectl -n flux-system get kustomization

This is also why apps/podinfo, apps/hello-app, and apps/headlamp started deploying without you ever running kubectl apply on them. Bootstrap's own Kustomization watches clusters/homelab with prune: true. clusters/homelab/apps.yaml (already sitting in the repo) itself just defines more Kustomization objects, one per app — so the first one picks it up as part of its own normal reconcile, creates those three child Kustomizations, and each of those then does its own fetch-and-apply against its own app directory. It's the same mechanism recursing, not a special case.

4. It stored your credential as a Kubernetes Secret — --password here is the flux-write token, not an account password. Bootstrap base64-wraps it into a Secret that matches what the GitRepository's .spec.secretRef points at:

kubectl -n flux-system get secret flux-system -o yaml

The password field is base64 (| base64 -d to read it) — this is exactly the same credential git clone https://user:token@host/repo would use, just read by source-controller on every fetch instead of typed by you once. It's never written to k8s's own filesystem.

5. It committed that config back into the repo itself — git pull in your desktop clone and look at clusters/homelab/flux-system/. The GitRepository/Kustomization objects you just inspected live in the cluster because those exact YAML files are committed there — so rebuilding this cluster from scratch would mean running flux bootstrap again (or even just kubectl apply -f clusters/homelab/flux-system/) and landing in the identical state. The fact that Flux watches this repo is itself declared in this repo. (Stage 4 leans on exactly this property — re-bootstrapping against a brand new cluster reproduces the same deployed state with no manual replay.)

If you want to watch a reconcile happen live rather than just inspect the end state:

kubectl -n flux-system logs deploy/source-controller -f

then in another terminal, make any commit and push it — you'll see the next fetch pick it up within the interval.

3. Verify the GitOps loop actually works

flux get kustomizations --watch

Watch until podinfo, hello-app, and headlamp all show Ready: True, then confirm pods actually landed:

kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods

If that's all healthy, the entire chain — Forgejo repo → Flux → this cluster — is working end to end with nothing manually kubectl apply'd.

4. podinfo: Flux's first Helm-managed app

podinfo is deployed first deliberately: it needs nothing this project builds itself — no CI, no custom image, just a public chart — so a green flux get kustomization podinfo proves the whole source-controller → helm-controller → cluster loop works before hello-app adds a CI dependency on top of it, and before headlamp adds a cluster-admin RBAC concern on top of that.

A Helm chart is a packaged bundle of Kubernetes YAML templates plus a values.yaml of defaults — installing one produces the same kind of Deployment/Service/etc. objects you'd otherwise hand-write, just parameterized and versioned. Used directly (outside this project, with nothing GitOps involved), that's a two-step, one-shot, imperative process — for comparison only, don't actually run this (it's not needed, and helm isn't even installed anywhere in this project):

helm repo add podinfo https://stefanprodan.github.io/podinfo   # register the chart index
helm install podinfo podinfo/podinfo --set replicaCount=1      # render + apply, once

Nothing here uses the helm CLI at all — Flux replaces both of those steps with two declarative objects that helm-controller reconciles continuously instead of once:

• apps/podinfo/helmrepository.yaml (a HelmRepository) is the declarative form of helm repo add — it just tells source-controller where the chart index lives and how often to refresh it (interval). On its own it deploys nothing.
• apps/podinfo/helmrelease.yaml (a HelmRelease) is the declarative form of helm install/helm upgrade — its values: block is exactly what would otherwise be -f values.yaml/--set flags on the CLI, and its chart.spec.version range (>=6.0.0) is what keeps it current: helm-controller re-installs whenever a new matching chart version shows up in the repository, on its own interval, with no one running helm upgrade by hand.

Both files already have comments walking through their specific fields — worth reading now that you know what problem they're solving.

5. headlamp: a cluster-admin UI, accessed on demand

apps/headlamp/ is deployed the same Helm-via-Flux way as podinfo (see its helmrepository.yaml/helmrelease.yaml), plus one more piece: apps/headlamp/rbac.yaml creates a ServiceAccount bound to the cluster-admin ClusterRole, with a long-lived token Secret. Headlamp grants whatever its logged-in identity can do — with that token, that's full cluster-admin. (This project used the official Kubernetes Dashboard originally; it's since been swapped for Headlamp, which the Dashboard project itself now points people toward, but the security posture is identical.)

Publicly exposing a cluster-admin UI (even behind a login page) is the exact pattern behind real breaches (Tesla, 2018: an internet-reachable, unauthenticated Dashboard). So: no ingress, no standing hostname — only a port-forward you open when you need it and close when you don't:

kubectl -n headlamp port-forward svc/headlamp 8443:80

Headlamp's chart doesn't terminate TLS on its own Service (that's left to whatever fronts it in a real deployment — here, nothing does, since this never leaves localhost), so open plain http://localhost:8443, not https://. Log in by pasting the token:

kubectl -n headlamp get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d

6. Register the Forgejo Actions runner

hello-app (next) is the only app in this repo whose image this project's own CI builds — that needs a self-hosted Forgejo Actions runner; there's no shared runner pool. The runner normally gets root-equivalent power over its host by mounting /var/run/docker.sock (anyone who can push a workflow file effectively gets root there). Instead: it runs as the same unprivileged k8s user from docs/01-bootstrap.md step 2, using rootless Podman's own socket instead of Docker's — no root anywhere in this pipeline. .forgejo/workflows/build-hello-app.yml already builds images with kaniko, which needs no daemon and no elevated privileges at all.

The runner's job containers (kaniko, opentofu once stage 4 wires that workflow in too) never get the libvirt socket or k8s's home directory mounted in — only the Podman socket, needed to launch those job containers in the first place — so a compromised workflow can spawn containers as k8s, but can't directly touch the VMs or Terraform state. Worth knowing given the runner lives on the same box/user as the cluster's own infrastructure; fine for a demo-sized project, but if this ever handles anything sensitive, move the runner to its own user or VM so a breakout doesn't share a blast radius with the cluster.

1. As k8s (ssh k8s@t630), enable the rootless Podman API socket:

   systemctl --user enable --now podman.socket
   echo $XDG_RUNTIME_DIR   # note this path, e.g. /run/user/1001
   

2. Instance admin: Site Administration → Actions → Runners, confirm Actions is enabled. Site Administration is a top-level menu only visible to instance admin accounts (not just repo owners) — click your profile avatar (top-right of any page) and look for it in the dropdown; if it's not there, the account you're logged in as isn't an instance admin. Inside, it's a left-hand sidebar (Dashboard, Users, Organizations, Repositories, Packages, Actions, Config, Notices, Monitor, ...) — click Actions there for the runners view.
3. Repo: Settings → Actions → Runners → Create new runner, copy the registration token. This token is unrelated to FORGEJO_TOKEN in step 5 below — easy to mix up since both are "a Forgejo token," but they're different things from different pages: this one is single-purpose, scoped to registering exactly one runner, generated on this Actions → Runners page. FORGEJO_TOKEN (step 5) is a personal access token from Settings → Applications on your own account, scoped to package:write, used by CI to push images — it has nothing to do with runner registration. If you paste FORGEJO_TOKEN into step 4 below by mistake, registration fails with invalid_argument: runner registration token not found.
4. Register and run the runner as a rootless Podman container, pointed at the Podman socket from step 1 instead of docker.sock. This is two separate commands, not one — forgejo-runner register is a one-shot action that talks to Forgejo once and exits; the thing that actually stays running and picks up jobs is a separate forgejo-runner daemon process. Running only register under --restart unless-stopped (an easy mistake, since it looks like a normal long-running container command) makes it silently loop: register succeeds, the container exits, Podman restarts it, it registers again, forever — never once actually listening for a job. Both commands also need --userns=keep-id: rootless Podman's socket file is owned by k8s's own UID on the host, but a container's "root" user normally maps to a different, subuid-remapped UID under the hood — --userns=keep-id makes the container's user be k8s's actual UID instead, so it can open a socket file owned by that UID. Without it, the daemon fails with permission denied connecting to the socket.

   # still as k8s
   podman volume create forgejo-runner-data
   
   # one-shot: register, then exits (--rm, not -d)
   podman run --rm --userns=keep-id \
     -v forgejo-runner-data:/data \
     code.forgejo.org/forgejo/runner:6 \
     forgejo-runner register --no-interactive \
       --instance https://git.boglabob.com \
       --token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
   
   # persistent: the actual daemon that listens for jobs
   podman run -d --name forgejo-runner --restart unless-stopped \
     --userns=keep-id \
     -e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v forgejo-runner-data:/data \
     code.forgejo.org/forgejo/runner:6 \
     forgejo-runner daemon
   

   Confirm it's actually stable rather than looping — podman ps --filter name=forgejo-runner should show one steadily increasing uptime, not a container repeatedly restarting seconds after creation:

   podman ps --filter name=forgejo-runner
   podman logs --tail 20 forgejo-runner
   

   The registered runner picks up both workflows in .forgejo/workflows/ — terraform.yml's container: image (stage 4) and build-hello-app.yml's kaniko image (next) are both launched through that same rootless Podman socket.
5. Repo Settings → Secrets and Variables → Actions, add:
   • Secret FORGEJO_TOKEN — a personal access token (Settings → Applications on your Forgejo user, scope package:write) used to push images. See the callout on step 3 above — this is a different token from the runner registration one, despite both living under "Forgejo tokens."
   • Variable FORGEJO_USER, FORGEJO_ORG — your Forgejo username/org.

If the hello-app package ends up private (Forgejo package visibility follows repo visibility by default), create the cluster-side pull secret and uncomment the imagePullSecrets line in apps/hello-app/deployment.yaml. <FORGEJO_USER>/<FORGEJO_TOKEN> below are placeholders to replace with your actual values, not literal syntax — bash reads a bare <word> as input redirection, so pasting them unreplaced fails with a confusing syntax error near unexpected token 'newline'' instead of a helpful message:

kubectl -n hello-app create secret docker-registry forgejo-registry \
  --docker-server=git.boglabob.com \
  --docker-username=<FORGEJO_USER> \
  --docker-password=<FORGEJO_TOKEN>

7. hello-app: plain manifests, CI, and image automation

Unlike podinfo/headlamp, apps/hello-app/ isn't a chart — it's this project's own code (source + Dockerfile under apps/hello-app/src/), so there's no upstream chart to track and plain Deployment/Service/Ingress manifests are simpler to read for something this small (see apps/hello-app/deployment.yaml's comments). What it does need, that podinfo/headlamp don't, is a way to notice when CI publishes a new image and roll it out — that's apps/hello-app/image-automation.yaml's job (ImageRepository/ImagePolicy/ImageUpdateAutomation, all commented in that file).

Exercise the whole loop end to end. Unlike the rest of this stage, this runs on your desktop (the maq clone from docs/01-bootstrap.md step 4), not as k8s on the T630 — k8s's own git access (k8s-readonly) is read-only by design, so it can't push. Once pushed, Forgejo Actions picks it up on its own self-hosted runner automatically; you don't need to be on the T630 for any of this step:

sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push

Watch: build-hello-app runs in Forgejo Actions (which also stamps the page with the current commit SHA and build time — see apps/hello-app/src/index.html) → pushes a new tag to git.boglabob.com/codegit/hello-app → Flux's ImageRepository picks it up within a minute → ImageUpdateAutomation commits the new tag back to apps/hello-app/deployment.yaml → the hello-app Kustomization reconciles → kubectl -n hello-app get pods shows a new pod, and https://hello.boglabob.com shows the new commit SHA/badge (once step 8 below exposes it).

8. Expose podinfo/hello-app through Caddy

k3s's bundled ingress controller (Traefik) is already listening on this node's own IP, port 80, routing by the Host: header from each app's Ingress resource (already defined in apps/podinfo and apps/hello-app). Caddy just needs to forward matching requests there — headlamp and the k3s API server are deliberately not exposed this way; step 5 above and docs/04-tofu.md step 5 cover why and how to reach them instead.

Two DNS records, as CNAMEs pointed at git.boglabob.com rather than duplicating its IP directly — CNAME means "this name is an alias for that one," so there's one place (that record) to update if the underlying IP ever changes, instead of several:

podinfo.boglabob.com  CNAME  git.boglabob.com
hello.boglabob.com    CNAME  git.boglabob.com

Then in Caddy's own config, using the VM's pinned IP from stage 2:

podinfo.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

hello.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile

curl https://podinfo.boglabob.com/
curl https://hello.boglabob.com/

docs/Caddyfile.example has the multi-node version of these same blocks — stage 4 points you back at it once there are three node IPs to load-balance across instead of one.

Next: docs/04-tofu.md — everything above still works exactly as-is; that stage replaces the one hand-built VM under it with a Terraform-provisioned 3-node cluster and points this same Flux config at it.