# Stage 3: Flux, Helm, and this repo's apps By the end of stage 2 (`docs/02-k3s.md`) you have a working k3s cluster and a kubeconfig at `~/.kube/config-manual`. This stage also needs bootstrap steps 3-4 from `docs/01-bootstrap.md` — the Forgejo repo pushed, and the `flux-write` token generated (the `k3s_token` secret still isn't used until stage 4). This stage bootstraps Flux against that cluster, and through it deploys every app in `apps/`: two managed via Helm (podinfo, headlamp — this is also where Helm itself gets explained, since Flux is the only way this project ever touches Helm) and one via plain manifests plus a CI pipeline (hello-app). Everything below runs as `k8s` on the T630, with `export KUBECONFIG=~/.kube/config-manual` already set from stage 2. --- ## 1. Install the Flux CLI Same no-sudo, direct-binary-release pattern as `kubectl` in stage 2 (`brew` assumes Homebrew, which isn't a given on a bare Debian box): ```sh FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//') curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux rm /tmp/flux.tar.gz flux --version ``` ## 2. Bootstrap Flux against Forgejo Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this uses the generic git bootstrap — over HTTPS with the `flux-write` token from `docs/01-bootstrap.md` step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably reachable from either this desktop or the T630 — see the earlier troubleshooting in this project's history): `flux bootstrap` only installs its four core controllers (`source-controller`, `kustomize-controller`, `helm-controller`, `notification-controller`) unless told otherwise. hello-app's image automation (step 7 below — `ImageRepository`/`ImagePolicy`/ `ImageUpdateAutomation`) needs two more, `image-reflector-controller` and `image-automation-controller`, which are opt-in via `--components-extra`; without it, hello-app's `Kustomization` fails dry-run with "no matches for kind ImageRepository" since the CRD was never installed. ```sh flux check --pre --kubeconfig ~/.kube/config-manual flux bootstrap git \ --url=https://git.boglabob.com/codegit/cloud-demo \ --branch=main \ --path=clusters/homelab \ --username=codegit \ --password= \ --token-auth \ --components-extra=image-reflector-controller,image-automation-controller \ --kubeconfig ~/.kube/config-manual ``` `--password` here is the `flux-write` token, not an actual account password. Flux stores it as a Kubernetes Secret in the `flux-system` namespace once bootstrap completes — that Secret is what `ImageUpdateAutomation` (step 7, below) reuses to push commits back, not anything held by `k8s` itself. Clear this command from `k8s`'s shell history afterwards (or prefix it with a space first, if `HISTCONTROL=ignorespace` is set) since the token was passed as a plain argument. This populates `clusters/homelab/flux-system/` and, because `clusters/homelab/apps.yaml` already declares `Kustomization` objects for `apps/podinfo`, `apps/hello-app`, and `apps/headlamp`, all three start reconciling immediately. ### What that one command actually did, mechanism by mechanism "Flux" isn't one program watching your repo by magic — it's a handful of ordinary Kubernetes controllers (just Deployments, like anything else you'd run on the cluster), each one understanding a couple of **Custom Resource Definitions** — CRDs extend the Kubernetes API with new object kinds, the same way `Deployment` or `Service` are built-in kinds. `GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing about any of this is special-cased outside the normal Kubernetes API — it's the exact same "define an object, a controller notices it and acts" loop that runs the whole rest of Kubernetes. That one `flux bootstrap` command did five genuinely separate things. Go look at each — these commands work against the cluster you already bootstrapped: **1. It installed the controllers** — plain Kubernetes Deployments, no different in kind from anything else running on the cluster: ```sh kubectl -n flux-system get deployments ``` `source-controller`, `kustomize-controller`, `helm-controller`, `notification-controller`, plus `image-reflector-controller` and `image-automation-controller` from `--components-extra` above. Alongside them, it registered the CRDs those controllers understand: ```sh kubectl get crds | grep fluxcd ``` **2. It created a `GitRepository` object** — this is the whole "what repo am I watching" declaration, and nothing more. `source-controller`'s job is entirely mechanical: every `interval` (default 1m), do a real `git fetch` against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that tree into a `.tar.gz`, and record its location in `.status.artifact`. That's the entire job — it doesn't know or care what's *in* the repo. ```sh kubectl -n flux-system get gitrepository flux-system -o yaml ``` Look at `.status.conditions` and `.status.artifact` — that's the result of an actual git fetch that already happened, not a static config. **3. It created a `Kustomization` object** (confusingly, the same name as the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. — related but not identical). `kustomize-controller`'s job, on its own interval: fetch the artifact `source-controller` produced, run the real `kustomize` tool against `.spec.path` inside it (the exact same tool a plain `kustomize build apps/podinfo` would run locally against those `kustomization.yaml` files), and apply the resulting objects via the Kubernetes API — the automated equivalent of you running `kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer, forever. `prune: true` (set on all the `Kustomization` objects in this repo) means it also *deletes* anything it previously created that's no longer present in the current git state — that's what makes it self-healing rather than just "apply once." ```sh kubectl -n flux-system get kustomization ``` **This is also why `apps/podinfo`, `apps/hello-app`, and `apps/headlamp` started deploying without you ever running `kubectl apply` on them.** Bootstrap's own `Kustomization` watches `clusters/homelab` with `prune: true`. `clusters/homelab/apps.yaml` (already sitting in the repo) itself just *defines more `Kustomization` objects*, one per app — so the first one picks it up as part of its own normal reconcile, creates those three child `Kustomization`s, and each of *those* then does its own fetch-and-apply against its own app directory. It's the same mechanism recursing, not a special case. **4. It stored your credential as a Kubernetes Secret** — `--password` here is the `flux-write` token, not an account password. Bootstrap base64-wraps it into a Secret that matches what the `GitRepository`'s `.spec.secretRef` points at: ```sh kubectl -n flux-system get secret flux-system -o yaml ``` The `password` field is base64 (`| base64 -d` to read it) — this is exactly the same credential `git clone https://user:token@host/repo` would use, just read by `source-controller` on every fetch instead of typed by you once. It's never written to `k8s`'s own filesystem. **5. It committed that config back into the repo itself** — `git pull` in your desktop clone and look at `clusters/homelab/flux-system/`. The `GitRepository`/`Kustomization` objects you just inspected live in the cluster *because* those exact YAML files are committed there — so rebuilding this cluster from scratch would mean running `flux bootstrap` again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and landing in the identical state. The fact that Flux watches this repo is itself declared *in* this repo. (Stage 4 leans on exactly this property — re-bootstrapping against a brand new cluster reproduces the same deployed state with no manual replay.) If you want to watch a reconcile happen live rather than just inspect the end state: ```sh kubectl -n flux-system logs deploy/source-controller -f ``` then in another terminal, make any commit and push it — you'll see the next fetch pick it up within the interval. ## 3. Verify the GitOps loop actually works ```sh flux get kustomizations --watch ``` Watch until `podinfo`, `hello-app`, and `headlamp` all show `Ready: True`, then confirm pods actually landed: ```sh kubectl -n podinfo get pods kubectl -n hello-app get pods kubectl -n headlamp get pods ``` If that's all healthy, the entire chain — Forgejo repo → Flux → this cluster — is working end to end with nothing manually `kubectl apply`'d. ## 4. podinfo: Flux's first Helm-managed app podinfo is deployed first deliberately: it needs nothing this project builds itself — no CI, no custom image, just a public chart — so a green `flux get kustomization podinfo` proves the whole source-controller → helm-controller → cluster loop works before hello-app adds a CI dependency on top of it, and before headlamp adds a cluster-admin RBAC concern on top of *that*. A **Helm chart** is a packaged bundle of Kubernetes YAML templates plus a `values.yaml` of defaults — installing one produces the same kind of `Deployment`/`Service`/etc. objects you'd otherwise hand-write, just parameterized and versioned. Used directly (outside this project, with nothing GitOps involved), that's a two-step, one-shot, imperative process — for comparison only, **don't actually run this** (it's not needed, and `helm` isn't even installed anywhere in this project): ```sh helm repo add podinfo https://stefanprodan.github.io/podinfo # register the chart index helm install podinfo podinfo/podinfo --set replicaCount=1 # render + apply, once ``` Nothing here uses the `helm` CLI at all — Flux replaces both of those steps with two declarative objects that `helm-controller` reconciles continuously instead of once: - **`apps/podinfo/helmrepository.yaml`** (a `HelmRepository`) is the declarative form of `helm repo add` — it just tells `source-controller` where the chart index lives and how often to refresh it (`interval`). On its own it deploys nothing. - **`apps/podinfo/helmrelease.yaml`** (a `HelmRelease`) is the declarative form of `helm install`/`helm upgrade` — its `values:` block is exactly what would otherwise be `-f values.yaml`/`--set` flags on the CLI, and its `chart.spec.version` range (`>=6.0.0`) is what keeps it current: `helm-controller` re-installs whenever a new matching chart version shows up in the repository, on its own `interval`, with no one running `helm upgrade` by hand. Both files already have comments walking through their specific fields — worth reading now that you know what problem they're solving. ## 5. headlamp: a cluster-admin UI, accessed on demand `apps/headlamp/` is deployed the same Helm-via-Flux way as podinfo (see its `helmrepository.yaml`/`helmrelease.yaml`), plus one more piece: `apps/headlamp/rbac.yaml` creates a `ServiceAccount` bound to the `cluster-admin` `ClusterRole`, with a long-lived token Secret. Headlamp grants whatever its logged-in identity can do — with that token, that's full cluster-admin. (This project used the official Kubernetes Dashboard originally; it's since been swapped for Headlamp, which the Dashboard project itself now points people toward, but the security posture is identical.) Publicly exposing a cluster-admin UI (even behind a login page) is the exact pattern behind real breaches (Tesla, 2018: an internet-reachable, unauthenticated Dashboard). So: no ingress, no standing hostname — only a port-forward you open when you need it and close when you don't: ```sh kubectl -n headlamp port-forward svc/headlamp 8443:80 ``` Headlamp's chart doesn't terminate TLS on its own Service (that's left to whatever fronts it in a real deployment — here, nothing does, since this never leaves `localhost`), so open plain `http://localhost:8443`, not `https://`. Log in by pasting the token: ```sh kubectl -n headlamp get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d ``` ## 6. Register the Forgejo Actions runner hello-app (next) is the only app in this repo whose image *this project's own CI* builds — that needs a self-hosted Forgejo Actions runner; there's no shared runner pool. The runner normally gets root-equivalent power over its host by mounting `/var/run/docker.sock` (anyone who can push a workflow file effectively gets root there). Instead: it runs as the same unprivileged `k8s` user from `docs/01-bootstrap.md` step 2, using rootless Podman's own socket instead of Docker's — no root anywhere in this pipeline. `.forgejo/workflows/build-hello-app.yml` already builds images with `kaniko`, which needs no daemon and no elevated privileges at all. The runner's job containers (kaniko, opentofu once stage 4 wires that workflow in too) never get the libvirt socket or `k8s`'s home directory mounted in — only the Podman socket, needed to launch those job containers in the first place — so a compromised workflow can spawn containers as `k8s`, but can't directly touch the VMs or Terraform state. Worth knowing given the runner lives on the same box/user as the cluster's own infrastructure; fine for a demo-sized project, but if this ever handles anything sensitive, move the runner to its own user or VM so a breakout doesn't share a blast radius with the cluster. 1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket: ```sh systemctl --user enable --now podman.socket echo $XDG_RUNTIME_DIR # note this path, e.g. /run/user/1001 ``` 2. Instance admin: `Site Administration → Actions → Runners`, confirm Actions is enabled. `Site Administration` is a top-level menu only visible to instance admin accounts (not just repo owners) — click your profile avatar (top-right of any page) and look for it in the dropdown; if it's not there, the account you're logged in as isn't an instance admin. Inside, it's a left-hand sidebar (Dashboard, Users, Organizations, Repositories, Packages, Actions, Config, Notices, Monitor, ...) — click `Actions` there for the runners view. 3. Repo: `Settings → Actions → Runners → Create new runner`, copy the registration token. **This token is unrelated to `FORGEJO_TOKEN` in step 5 below** — easy to mix up since both are "a Forgejo token," but they're different things from different pages: this one is single-purpose, scoped to registering exactly one runner, generated on this `Actions → Runners` page. `FORGEJO_TOKEN` (step 5) is a personal access token from `Settings → Applications` on your own account, scoped to `package:write`, used by CI to push images — it has nothing to do with runner registration. If you paste `FORGEJO_TOKEN` into step 4 below by mistake, registration fails with `invalid_argument: runner registration token not found`. 4. Register and run the runner as a rootless Podman container, pointed at the Podman socket from step 1 instead of docker.sock. This is two separate commands, not one — `forgejo-runner register` is a one-shot action that talks to Forgejo once and exits; the thing that actually stays running and picks up jobs is a separate `forgejo-runner daemon` process. Running only `register` under `--restart unless-stopped` (an easy mistake, since it looks like a normal long-running container command) makes it silently loop: register succeeds, the container exits, Podman restarts it, it registers again, forever — never once actually listening for a job. Both commands also need `--userns=keep-id`: rootless Podman's socket file is owned by `k8s`'s own UID on the host, but a container's "root" user normally maps to a *different*, subuid-remapped UID under the hood — `--userns=keep-id` makes the container's user *be* `k8s`'s actual UID instead, so it can open a socket file owned by that UID. Without it, the daemon fails with `permission denied` connecting to the socket. ```sh # still as k8s podman volume create forgejo-runner-data # one-shot: register, then exits (--rm, not -d) podman run --rm --userns=keep-id \ -v forgejo-runner-data:/data \ code.forgejo.org/forgejo/runner:6 \ forgejo-runner register --no-interactive \ --instance https://git.boglabob.com \ --token --labels docker:docker://node:20-bookworm # persistent: the actual daemon that listens for jobs podman run -d --name forgejo-runner --restart unless-stopped \ --userns=keep-id \ -e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \ -v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \ -v forgejo-runner-data:/data \ code.forgejo.org/forgejo/runner:6 \ forgejo-runner daemon ``` Confirm it's actually stable rather than looping — `podman ps --filter name=forgejo-runner` should show one steadily increasing uptime, not a container repeatedly restarting seconds after creation: ```sh podman ps --filter name=forgejo-runner podman logs --tail 20 forgejo-runner ``` The registered runner picks up both workflows in `.forgejo/workflows/` — `terraform.yml`'s `container:` image (stage 4) and `build-hello-app.yml`'s kaniko image (next) are both launched through that same rootless Podman socket. 5. Repo `Settings → Secrets and Variables → Actions`, add: - Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications` on your Forgejo user, scope `package:write`) used to push images. See the callout on step 3 above — this is a different token from the runner registration one, despite both living under "Forgejo tokens." - Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org. If the `hello-app` package ends up private (Forgejo package visibility follows repo visibility by default), create the cluster-side pull secret and uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`. ``/`` below are placeholders to replace with your actual values, not literal syntax — bash reads a bare `` as input redirection, so pasting them unreplaced fails with a confusing `syntax error near unexpected token 'newline''` instead of a helpful message: ```sh kubectl -n hello-app create secret docker-registry forgejo-registry \ --docker-server=git.boglabob.com \ --docker-username= \ --docker-password= ``` ## 7. hello-app: plain manifests, CI, and image automation Unlike podinfo/headlamp, `apps/hello-app/` isn't a chart — it's *this* project's own code (source + Dockerfile under `apps/hello-app/src/`), so there's no upstream chart to track and plain `Deployment`/`Service`/`Ingress` manifests are simpler to read for something this small (see `apps/hello-app/deployment.yaml`'s comments). What it does need, that podinfo/headlamp don't, is a way to notice when CI publishes a new image and roll it out — that's `apps/hello-app/image-automation.yaml`'s job (`ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`, all commented in that file). One easy-to-miss requirement, worth calling out explicitly since getting it wrong fails silently: `ImageUpdateAutomation` must live in the *same namespace* as the `ImagePolicy` it targets — the controller only ever lists `ImagePolicy` objects from its own namespace, so a cross-namespace `$imagepolicy` marker matches nothing, and every reconcile just reports "repository up-to-date" with no error and no commit, no matter how correct the marker syntax, RBAC, or policy resolution otherwise are (see `image-automation.yaml`'s own comment for the exact controller source line this comes from). `GitRepository` itself is fine to reference cross-namespace, via `sourceRef.namespace`. Exercise the whole loop end to end. Unlike the rest of this stage, this runs on your **desktop** (the `maq` clone from `docs/01-bootstrap.md` step 4), not as `k8s` on the T630 — `k8s`'s own git access (`k8s-readonly`) is read-only by design, so it can't push. Once pushed, Forgejo Actions picks it up on its own self-hosted runner automatically; you don't need to be on the T630 for any of this step: ```sh sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html git add apps/hello-app/src/index.html git commit -m "test the pipeline" git push ``` Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the page with the current commit SHA and build time — see `apps/hello-app/src/index.html`) → pushes a new tag to `git.boglabob.com/codegit/hello-app` → Flux's `ImageRepository` picks it up within a minute → `ImageUpdateAutomation` commits the new tag back to `apps/hello-app/deployment.yaml` → the `hello-app` Kustomization reconciles → `kubectl -n hello-app get pods` shows a new pod, and `https://hello.boglabob.com` shows the new commit SHA/badge (once step 8 below exposes it). ## 8. Expose podinfo/hello-app through Caddy k3s's bundled ingress controller (Traefik) is already listening on this node's own IP, port 80, routing by the `Host:` header from each app's `Ingress` resource (already defined in `apps/podinfo` and `apps/hello-app`). Caddy just needs to forward matching requests there — headlamp and the k3s API server are deliberately **not** exposed this way; step 5 above and `docs/04-tofu.md` step 5 cover why and how to reach them instead. Two DNS records, as CNAMEs pointed at the bare apex `boglabob.com` — the zone's one actual A record — rather than duplicating its IP directly. This matches every other hostname on this server (`git`, `photos`, `music`, etc. are all `CNAME`s to the apex too); it's *not* about `git.boglabob.com` being some special canonical host to alias through — an earlier version of this doc got that wrong, pointing these two at `git.boglabob.com` instead of the apex, inconsistent with every other record in the zone: ``` podinfo.boglabob.com CNAME boglabob.com hello.boglabob.com CNAME boglabob.com ``` Then in Caddy's own config, using the VM's pinned IP from stage 2. Both blocks point at the exact same `` — that's not a mistake, there's only one VM right now, and it's simultaneously the control plane *and* the only place pods can run ("Why one node is enough," above). Caddy isn't picking between podinfo/hello-app here at all — it just forwards everything for both hostnames to that one node's port 80, and it's Traefik, once the request lands there, that actually reads the `Host:` header and routes to the right backend. This changes in stage 4, where there are real node IPs to load-balance across (`docs/Caddyfile.example`'s multi-node blocks): ``` podinfo.boglabob.com { reverse_proxy http://:80 } hello.boglabob.com { reverse_proxy http://:80 } ``` ```sh podman exec caddy reload --config /etc/caddy/Caddyfile ``` ```sh curl https://podinfo.boglabob.com/ curl https://hello.boglabob.com/ ``` `docs/Caddyfile.example` has the multi-node version of these same blocks — stage 4 points you back at it once there are three node IPs to load-balance across instead of one. Next: `docs/04-tofu.md` — everything above still works exactly as-is; that stage replaces the one hand-built VM under it with a Terraform-provisioned 3-node cluster and points this same Flux config at it.