cloud-demo/docs/03-flux.md

# Stage 3: Flux, Helm, and this repo's apps

By the end of stage 2 (`docs/02-k3s.md`) you have a working k3s cluster and
a kubeconfig at `~/.kube/config-manual`. This stage also needs bootstrap
steps 3-4 from `docs/01-bootstrap.md` — the Forgejo repo pushed, and the
`flux-write` token generated (the `k3s_token` secret still isn't used until
stage 4).

This stage bootstraps Flux against that cluster, and through it deploys
every app in `apps/`: two managed via Helm (podinfo, headlamp — this is
also where Helm itself gets explained, since Flux is the only way this
project ever touches Helm) and one via plain manifests plus a CI pipeline
(hello-app).

Everything below runs as `k8s` on the T630, with
`export KUBECONFIG=~/.kube/config-manual` already set from stage 2.

---

## 1. Install the Flux CLI

Same no-sudo, direct-binary-release pattern as `kubectl` in stage 2
(`brew` assumes Homebrew, which isn't a given on a bare Debian box):

```sh
FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version
```

## 2. Bootstrap Flux against Forgejo

Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this
uses the generic git bootstrap — over HTTPS with the `flux-write` token
from `docs/01-bootstrap.md` step 4, not SSH (Forgejo's git-SSH port turned
out not to be reliably reachable from either this desktop or the T630 —
see the earlier troubleshooting in this project's history):

`flux bootstrap` only installs its four core controllers
(`source-controller`, `kustomize-controller`, `helm-controller`,
`notification-controller`) unless told otherwise. hello-app's image
automation (step 7 below — `ImageRepository`/`ImagePolicy`/
`ImageUpdateAutomation`) needs two more, `image-reflector-controller` and
`image-automation-controller`, which are opt-in via `--components-extra`;
without it, hello-app's `Kustomization` fails dry-run with "no matches for
kind ImageRepository" since the CRD was never installed.

```sh
flux check --pre --kubeconfig ~/.kube/config-manual

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --components-extra=image-reflector-controller,image-automation-controller \
  --kubeconfig ~/.kube/config-manual
```

`--password` here is the `flux-write` token, not an actual account
password. Flux stores it as a Kubernetes Secret in the `flux-system`
namespace once bootstrap completes — that Secret is what
`ImageUpdateAutomation` (step 7, below) reuses to push commits back, not
anything held by `k8s` itself. Clear this command from `k8s`'s shell
history afterwards (or prefix it with a space first, if
`HISTCONTROL=ignorespace` is set) since the token was passed as a plain
argument.

This populates `clusters/homelab/flux-system/` and, because
`clusters/homelab/apps.yaml` already declares `Kustomization` objects for
`apps/podinfo`, `apps/hello-app`, and `apps/headlamp`, all three start
reconciling immediately.

### What that one command actually did, mechanism by mechanism

"Flux" isn't one program watching your repo by magic — it's a handful of
ordinary Kubernetes controllers (just Deployments, like anything else
you'd run on the cluster), each one understanding a couple of **Custom
Resource Definitions** — CRDs extend the Kubernetes API with new object
kinds, the same way `Deployment` or `Service` are built-in kinds.
`GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing
about any of this is special-cased outside the normal Kubernetes API —
it's the exact same "define an object, a controller notices it and acts"
loop that runs the whole rest of Kubernetes.

That one `flux bootstrap` command did five genuinely separate things. Go
look at each — these commands work against the cluster you already
bootstrapped:

**1. It installed the controllers** — plain Kubernetes Deployments, no
different in kind from anything else running on the cluster:

```sh
kubectl -n flux-system get deployments
```

`source-controller`, `kustomize-controller`, `helm-controller`,
`notification-controller`, plus `image-reflector-controller` and
`image-automation-controller` from `--components-extra` above. Alongside
them, it registered the CRDs those
controllers understand:

```sh
kubectl get crds | grep fluxcd
```

**2. It created a `GitRepository` object** — this is the whole "what repo
am I watching" declaration, and nothing more. `source-controller`'s job is
entirely mechanical: every `interval` (default 1m), do a real `git fetch`
against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that
tree into a `.tar.gz`, and record its location in `.status.artifact`.
That's the entire job — it doesn't know or care what's *in* the repo.

```sh
kubectl -n flux-system get gitrepository flux-system -o yaml
```

Look at `.status.conditions` and `.status.artifact` — that's the result of
an actual git fetch that already happened, not a static config.

**3. It created a `Kustomization` object** (confusingly, the same name as
the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. —
related but not identical). `kustomize-controller`'s job, on its own
interval: fetch the artifact `source-controller` produced, run the real
`kustomize` tool against `.spec.path` inside it (the exact same tool a
plain `kustomize build apps/podinfo` would run locally against those
`kustomization.yaml` files), and apply the resulting objects via the
Kubernetes API — the automated equivalent of you running
`kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer,
forever. `prune: true` (set on all the `Kustomization` objects in this
repo) means it also *deletes* anything it previously created that's no
longer present in the current git state — that's what makes it
self-healing rather than just "apply once."

```sh
kubectl -n flux-system get kustomization
```

**This is also why `apps/podinfo`, `apps/hello-app`, and `apps/headlamp`
started deploying without you ever running `kubectl apply` on them.**
Bootstrap's own `Kustomization` watches `clusters/homelab` with
`prune: true`. `clusters/homelab/apps.yaml` (already sitting in the repo)
itself just *defines more `Kustomization` objects*, one per app — so the
first one picks it up as part of its own normal reconcile, creates those
three child `Kustomization`s, and each of *those* then does its own
fetch-and-apply against its own app directory. It's the same mechanism
recursing, not a special case.

**4. It stored your credential as a Kubernetes Secret** — `--password`
here is the `flux-write` token, not an account password. Bootstrap
base64-wraps it into a Secret that matches what the `GitRepository`'s
`.spec.secretRef` points at:

```sh
kubectl -n flux-system get secret flux-system -o yaml
```

The `password` field is base64 (`| base64 -d` to read it) — this is
exactly the same credential `git clone https://user:token@host/repo` would
use, just read by `source-controller` on every fetch instead of typed by
you once. It's never written to `k8s`'s own filesystem.

**5. It committed that config back into the repo itself** — `git pull` in
your desktop clone and look at `clusters/homelab/flux-system/`. The
`GitRepository`/`Kustomization` objects you just inspected live in the
cluster *because* those exact YAML files are committed there — so
rebuilding this cluster from scratch would mean running `flux bootstrap`
again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and
landing in the identical state. The fact that Flux watches this repo is
itself declared *in* this repo. (Stage 4 leans on exactly this property —
re-bootstrapping against a brand new cluster reproduces the same deployed
state with no manual replay.)

If you want to watch a reconcile happen live rather than just inspect the
end state:

```sh
kubectl -n flux-system logs deploy/source-controller -f
```

then in another terminal, make any commit and push it — you'll see the
next fetch pick it up within the interval.

## 3. Verify the GitOps loop actually works

```sh
flux get kustomizations --watch
```

Watch until `podinfo`, `hello-app`, and `headlamp` all show `Ready: True`,
then confirm pods actually landed:

```sh
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n headlamp get pods
```

If that's all healthy, the entire chain — Forgejo repo → Flux → this
cluster — is working end to end with nothing manually `kubectl apply`'d.

## 4. podinfo: Flux's first Helm-managed app

podinfo is deployed first deliberately: it needs nothing this project
builds itself — no CI, no custom image, just a public chart — so a green
`flux get kustomization podinfo` proves the whole
source-controller → helm-controller → cluster loop works before hello-app
adds a CI dependency on top of it, and before headlamp adds a
cluster-admin RBAC concern on top of *that*.

A **Helm chart** is a packaged bundle of Kubernetes YAML templates plus a
`values.yaml` of defaults — installing one produces the same kind of
`Deployment`/`Service`/etc. objects you'd otherwise hand-write, just
parameterized and versioned. Used directly (outside this project, with
nothing GitOps involved), that's a two-step, one-shot, imperative
process — for comparison only, **don't actually run this** (it's not
needed, and `helm` isn't even installed anywhere in this project):

```sh
helm repo add podinfo https://stefanprodan.github.io/podinfo   # register the chart index
helm install podinfo podinfo/podinfo --set replicaCount=1      # render + apply, once
```

Nothing here uses the `helm` CLI at all — Flux replaces both of those
steps with two declarative objects that `helm-controller` reconciles
continuously instead of once:

- **`apps/podinfo/helmrepository.yaml`** (a `HelmRepository`) is the
  declarative form of `helm repo add` — it just tells `source-controller`
  where the chart index lives and how often to refresh it (`interval`).
  On its own it deploys nothing.
- **`apps/podinfo/helmrelease.yaml`** (a `HelmRelease`) is the declarative
  form of `helm install`/`helm upgrade` — its `values:` block is exactly
  what would otherwise be `-f values.yaml`/`--set` flags on the CLI, and
  its `chart.spec.version` range (`>=6.0.0`) is what keeps it current:
  `helm-controller` re-installs whenever a new matching chart version
  shows up in the repository, on its own `interval`, with no one running
  `helm upgrade` by hand.

Both files already have comments walking through their specific fields —
worth reading now that you know what problem they're solving.

## 5. headlamp: a cluster-admin UI, accessed on demand

`apps/headlamp/` is deployed the same Helm-via-Flux way as podinfo (see
its `helmrepository.yaml`/`helmrelease.yaml`), plus one more piece:
`apps/headlamp/rbac.yaml` creates a `ServiceAccount` bound to the
`cluster-admin` `ClusterRole`, with a long-lived token Secret. Headlamp
grants whatever its logged-in identity can do — with that token, that's
full cluster-admin. (This project used the official Kubernetes Dashboard
originally; it's since been swapped for Headlamp, which the Dashboard
project itself now points people toward, but the security posture is
identical.)

Publicly exposing a cluster-admin UI (even behind a login page) is the
exact pattern behind real breaches (Tesla, 2018: an internet-reachable,
unauthenticated Dashboard). So: no ingress, no standing hostname — only a
port-forward you open when you need it and close when you don't:

```sh
kubectl -n headlamp port-forward svc/headlamp 8443:80
```

Headlamp's chart doesn't terminate TLS on its own Service (that's left to
whatever fronts it in a real deployment — here, nothing does, since this
never leaves `localhost`), so open plain `http://localhost:8443`, not
`https://`. Log in by pasting the token:

```sh
kubectl -n headlamp get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d
```

## 6. Register the Forgejo Actions runner

hello-app (next) is the only app in this repo whose image *this project's
own CI* builds — that needs a self-hosted Forgejo Actions runner; there's
no shared runner pool. The runner normally gets root-equivalent power over
its host by mounting `/var/run/docker.sock` (anyone who can push a
workflow file effectively gets root there). Instead: it runs as the same
unprivileged `k8s` user from `docs/01-bootstrap.md` step 2, using rootless
Podman's own socket instead of Docker's — no root anywhere in this
pipeline. `.forgejo/workflows/build-hello-app.yml` already builds images
with `kaniko`, which needs no daemon and no elevated privileges at all.

The runner's job containers (kaniko, opentofu once stage 4 wires that
workflow in too) never get the libvirt socket or `k8s`'s home directory
mounted in — only the Podman socket, needed to launch those job containers
in the first place — so a compromised workflow can spawn containers as
`k8s`, but can't directly touch the VMs or Terraform state. Worth knowing
given the runner lives on the same box/user as the cluster's own
infrastructure; fine for a demo-sized project, but if this ever handles
anything sensitive, move the runner to its own user or VM so a breakout
doesn't share a blast radius with the cluster.

1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket:
   ```sh
   systemctl --user enable --now podman.socket
   echo $XDG_RUNTIME_DIR   # note this path, e.g. /run/user/1001
   ```
2. Instance admin: `Site Administration → Actions → Runners`, confirm
   Actions is enabled. `Site Administration` is a top-level menu only
   visible to instance admin accounts (not just repo owners) — click your
   profile avatar (top-right of any page) and look for it in the dropdown;
   if it's not there, the account you're logged in as isn't an instance
   admin. Inside, it's a left-hand sidebar (Dashboard, Users,
   Organizations, Repositories, Packages, Actions, Config, Notices,
   Monitor, ...) — click `Actions` there for the runners view.
3. Repo: `Settings → Actions → Runners → Create new runner`, copy the
   registration token. **This token is unrelated to `FORGEJO_TOKEN` in step
   5 below** — easy to mix up since both are "a Forgejo token," but they're
   different things from different pages: this one is single-purpose,
   scoped to registering exactly one runner, generated on this
   `Actions → Runners` page. `FORGEJO_TOKEN` (step 5) is a personal access
   token from `Settings → Applications` on your own account, scoped to
   `package:write`, used by CI to push images — it has nothing to do with
   runner registration. If you paste `FORGEJO_TOKEN` into step 4 below by
   mistake, registration fails with `invalid_argument: runner registration
   token not found`.
4. Register and run the runner as a rootless Podman container, pointed at
   the Podman socket from step 1 instead of docker.sock. This is two
   separate commands, not one — `forgejo-runner register` is a one-shot
   action that talks to Forgejo once and exits; the thing that actually
   stays running and picks up jobs is a separate `forgejo-runner daemon`
   process. Running only `register` under `--restart unless-stopped` (an
   easy mistake, since it looks like a normal long-running container
   command) makes it silently loop: register succeeds, the container exits,
   Podman restarts it, it registers again, forever — never once actually
   listening for a job. Both commands also need `--userns=keep-id`:
   rootless Podman's socket file is owned by `k8s`'s own UID on the host,
   but a container's "root" user normally maps to a *different*,
   subuid-remapped UID under the hood — `--userns=keep-id` makes the
   container's user *be* `k8s`'s actual UID instead, so it can open a
   socket file owned by that UID. Without it, the daemon fails with
   `permission denied` connecting to the socket.
   ```sh
   # still as k8s
   podman volume create forgejo-runner-data

   # one-shot: register, then exits (--rm, not -d)
   podman run --rm --userns=keep-id \
     -v forgejo-runner-data:/data \
     code.forgejo.org/forgejo/runner:6 \
     forgejo-runner register --no-interactive \
       --instance https://git.boglabob.com \
       --token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm

   # persistent: the actual daemon that listens for jobs
   podman run -d --name forgejo-runner --restart unless-stopped \
     --userns=keep-id \
     -e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v forgejo-runner-data:/data \
     code.forgejo.org/forgejo/runner:6 \
     forgejo-runner daemon
   ```
   Confirm it's actually stable rather than looping — `podman ps --filter
   name=forgejo-runner` should show one steadily increasing uptime, not a
   container repeatedly restarting seconds after creation:
   ```sh
   podman ps --filter name=forgejo-runner
   podman logs --tail 20 forgejo-runner
   ```
   The registered runner picks up both workflows in `.forgejo/workflows/` —
   `terraform.yml`'s `container:` image (stage 4) and
   `build-hello-app.yml`'s kaniko image (next) are both launched through
   that same rootless Podman socket.
5. Repo `Settings → Secrets and Variables → Actions`, add:
   - Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications`
     on your Forgejo user, scope `package:write`) used to push images. See
     the callout on step 3 above — this is a different token from the
     runner registration one, despite both living under "Forgejo tokens."
   - Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org.

If the `hello-app` package ends up private (Forgejo package visibility
follows repo visibility by default), create the cluster-side pull secret
and uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`.
`<FORGEJO_USER>`/`<FORGEJO_TOKEN>` below are placeholders to replace with
your actual values, not literal syntax — bash reads a bare `<word>` as
input redirection, so pasting them unreplaced fails with a confusing
`syntax error near unexpected token 'newline''` instead of a helpful
message:

```sh
kubectl -n hello-app create secret docker-registry forgejo-registry \
  --docker-server=git.boglabob.com \
  --docker-username=<FORGEJO_USER> \
  --docker-password=<FORGEJO_TOKEN>
```

## 7. hello-app: plain manifests, CI, and image automation

Unlike podinfo/headlamp, `apps/hello-app/` isn't a chart — it's *this*
project's own code (source + Dockerfile under `apps/hello-app/src/`), so
there's no upstream chart to track and plain
`Deployment`/`Service`/`Ingress` manifests are simpler to read for
something this small (see `apps/hello-app/deployment.yaml`'s comments).
What it does need, that podinfo/headlamp don't, is a way to notice when CI
publishes a new image and roll it out — that's
`apps/hello-app/image-automation.yaml`'s job
(`ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`, all commented in
that file). One easy-to-miss requirement, worth calling out explicitly
since getting it wrong fails silently: `ImageUpdateAutomation` must live
in the *same namespace* as the `ImagePolicy` it targets — the controller
only ever lists `ImagePolicy` objects from its own namespace, so a
cross-namespace `$imagepolicy` marker matches nothing, and every
reconcile just reports "repository up-to-date" with no error and no
commit, no matter how correct the marker syntax, RBAC, or policy
resolution otherwise are (see `image-automation.yaml`'s own comment for
the exact controller source line this comes from). `GitRepository` itself
is fine to reference cross-namespace, via `sourceRef.namespace`.

Exercise the whole loop end to end. Unlike the rest of this stage, this
runs on your **desktop** (the `maq` clone from `docs/01-bootstrap.md` step
4), not as `k8s` on the T630 — `k8s`'s own git access (`k8s-readonly`) is
read-only by design, so it can't push. Once pushed, Forgejo Actions picks
it up on its own self-hosted runner automatically; you don't need to be on
the T630 for any of this step:

```sh
sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push
```

Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the
page with the current commit SHA and build time — see
`apps/hello-app/src/index.html`) → pushes a new tag to
`git.boglabob.com/codegit/hello-app` → Flux's `ImageRepository` picks it up
within a minute → `ImageUpdateAutomation` commits the new tag back to
`apps/hello-app/deployment.yaml` → the `hello-app` Kustomization
reconciles → `kubectl -n hello-app get pods` shows a new pod, and
`https://hello.boglabob.com` shows the new commit SHA/badge (once step 8
below exposes it).

## 8. Expose podinfo/hello-app through Caddy

k3s's bundled ingress controller (Traefik) is already listening on this
node's own IP, port 80, routing by the `Host:` header from each app's
`Ingress` resource (already defined in `apps/podinfo` and
`apps/hello-app`). Caddy just needs to forward matching requests there —
headlamp and the k3s API server are deliberately **not** exposed this way;
step 5 above and `docs/04-tofu.md` step 5 cover why and how to reach them
instead.

Two DNS records, as CNAMEs pointed at the bare apex `boglabob.com` — the
zone's one actual A record — rather than duplicating its IP directly.
This matches every other hostname on this server (`git`, `photos`,
`music`, etc. are all `CNAME`s to the apex too); it's *not* about
`git.boglabob.com` being some special canonical host to alias through —
an earlier version of this doc got that wrong, pointing these two at
`git.boglabob.com` instead of the apex, inconsistent with every other
record in the zone:

```
podinfo.boglabob.com  CNAME  boglabob.com
hello.boglabob.com    CNAME  boglabob.com
```

Then in Caddy's own config, using the VM's pinned IP from stage 2. Both
blocks point at the exact same `<VM_IP>` — that's not a mistake, there's
only one VM right now, and it's simultaneously the control plane *and*
the only place pods can run ("Why one node is enough," above). Caddy
isn't picking between podinfo/hello-app here at all — it just forwards
everything for both hostnames to that one node's port 80, and it's
Traefik, once the request lands there, that actually reads the `Host:`
header and routes to the right backend. This changes in stage 4, where
there are real node IPs to load-balance across (`docs/Caddyfile.example`'s
multi-node blocks):

```
podinfo.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

hello.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}
```

```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```

```sh
curl https://podinfo.boglabob.com/
curl https://hello.boglabob.com/
```

`docs/Caddyfile.example` has the multi-node version of these same blocks —
stage 4 points you back at it once there are three node IPs to
load-balance across instead of one.

Next: `docs/04-tofu.md` — everything above still works exactly as-is; that
stage replaces the one hand-built VM under it with a Terraform-provisioned
3-node cluster and points this same Flux config at it.