cloud-demo/docs/SETUP.md

# Setup walkthrough (full, Terraform-driven)

This is the "do it properly" path — Terraform/OpenTofu provisioning all 3
VMs via the `dmacvicar/libvirt` provider. It's also the one currently
blocked on getting that provider's HCL right (0.8.x vs 0.9.x schema — see
conversation history). If you want a real cluster to learn Flux/GitOps on
*right now* without waiting on that, see `docs/QUICKSTART.md` instead — a
single manually-created VM, no Terraform, with teardown instructions for
switching over once this track is sorted. Steps 1–4 below are shared
between both guides.

Assumes: the T630 is an existing Debian box already running other
self-hosted services — this project installs alongside those as ordinary
packages (`qemu-kvm`/`libvirt`), not a hypervisor OS replacing Debian, and
is sized deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total)
so it stays a demo rather than competing with what's already running. Forgejo
is already running and reachable at `https://git.boglabob.com`, and you can
point DNS records under `boglabob.com` at hosts on your network (directly,
or via whatever reverse proxy/tunnel already gets `git.boglabob.com` there).

---

## 1. Install KVM/libvirt on the T630

Ordinary packages, no reboot into an installer, nothing else on the box is
touched:

```sh
# on the T630
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst

# confirm hardware virtualization is available (T630's Xeons support it)
sudo kvm-ok
```

## 2. Create the unprivileged 'k8s' user

One dedicated, no-sudo user for everything this project touches: driving
Terraform/kubectl/flux against libvirt here, and running the Forgejo Actions
runner later (step 9). It needs group membership to talk to libvirt — that's
a one-time root action; nothing it does afterwards needs `sudo`.

```sh
sudo useradd -m -s /bin/bash k8s          # one-time, needs root to create the user itself
sudo usermod -aG libvirt,kvm k8s
sudo loginctl enable-linger k8s           # lets its services keep running after logout

# as k8s, from here on (sudo -iu, not su -, since k8s has no password set):
sudo -iu k8s
ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/id_ed25519   # only needed if you'll SSH in as k8s day-to-day
virsh -c qemu:///system list --all        # sanity check: should run with no permission error, no sudo
```

Do the rest of this guide logged in as `k8s` on the T630 itself (`ssh
k8s@t630`) — node IPs (step 5) live on a private libvirt network that's only
directly reachable from the T630, so this is the simplest place to run
`tofu`/`kubectl`/`flux` from. (If you'd rather drive Terraform from your own
workstation instead, see the `libvirt_uri` comment in
`terraform/terraform.tfvars.example` — you'll then need an SSH tunnel for
kubectl/flux to reach node IPs.)

## 3. Generate the secrets Terraform needs

```sh
openssl rand -hex 32   # -> k3s_token
```

## 4. Push this repo to Forgejo

Doing this before provisioning (rather than after) means `k8s` can get the
repo with a plain `git clone` in step 5, instead of needing a one-off copy
handed to it — and any future Terraform change just needs a `git pull`.
Repo/owner used throughout this guide: `codegit/cloud-demo` (already baked
into `apps/hello-app/deployment.yaml` and `image-automation.yaml`'s image
references — no placeholder-swapping needed).

1. On Forgejo (`https://git.boglabob.com`), as `codegit`: **+ → New
   Repository** → name `cloud-demo`. Leave it empty — don't initialize with
   a README/`.gitignore`/license, since this repo already has its own.
   Visibility (public/private) is your call; either works, since access for
   `k8s`/Flux/CI goes through the tokens below regardless.
2. Locally, wherever you're editing this repo (`maq`):
   ```sh
   git init   # if not already
   git add .
   git commit -m "initial scaffold"
   git remote add origin https://git.boglabob.com/codegit/cloud-demo.git
   git push -u origin main
   ```
3. Generate two access tokens (`Settings → Applications → Generate New
   Token`), scoped as narrowly as Forgejo's token UI allows to repository
   read/write:
   - **`k8s-readonly`** — read-only. Used only for `k8s`'s own manual
     `git clone`/`pull` on the T630 (step 5) — never leaves that box, isn't
     used by anything automated.
   - **`flux-write`** — read/write. Used once, as a `flux bootstrap`
     argument (step 7); Flux stores it as a Kubernetes Secret inside the
     cluster from then on (`ImageUpdateAutomation`'s commits back in step 10
     reuse that same in-cluster Secret) — it's never written to `k8s`'s
     filesystem at all.

   Using HTTPS tokens instead of `k8s`'s SSH key (`~/.ssh/id_ed25519`, from
   step 2) sidesteps an open question: Forgejo's git-SSH port isn't
   reachable from this desktop through your router (see the SSH
   troubleshooting earlier in this conversation), and whether it's reachable
   from `k8s` on the T630 itself was never actually confirmed either. HTTPS
   (443, via Caddy) is already proven to work, so both tokens use that
   instead. Copy both token values now — Forgejo only shows them once.

## 5. Provision the VMs with OpenTofu

Installing the package needs sudo, so that part is you (`maq`), not `k8s`.
Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu`
afterwards with no further root involvement:

```sh
# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh
```

Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved —
clone using the `k8s-readonly` token from step 4. `k8s` has no keyring (it's
headless, no desktop session), so this uses `git credential-store` — a
plaintext file, `chmod 600`'d, holding only the read-only token:

```sh
# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials

cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from step 2), k3s_token. Defaults for network/sizing are fine to start.

tofu init
tofu plan
tofu apply
```

For any later change to `terraform/`: edit and push as `maq` as usual, then
`cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`.

This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the
`k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from
anything else already using libvirt on this box). Cloud-init installs k3s on
each on first boot — give it ~2 minutes after `apply` finishes.

## 6. Get kubectl talking to the cluster

`kubectl` itself was never actually installed anywhere earlier in this
guide despite being listed as a prerequisite — install it now (as `k8s`,
no sudo needed, same pattern as the OpenTofu install):

```sh
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
```

```sh
mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab

export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes   # expect 3 Ready nodes
```

This works as-is because you're running it on the T630, which can reach the
`10.20.30.0/24` network directly. To also use kubectl from your own laptop,
either `scp` this kubeconfig over and open an SSH tunnel first
(`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's
`server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s`
whenever you need kubectl — simplest by far for a project this size.

## 7. Bootstrap Flux against Forgejo

Forgejo isn't a Flux-native provider (unlike GitHub/GitLab), so use the
generic git bootstrap — over HTTPS with the `flux-write` token from step 4,
not SSH (same reachability reasoning as step 5):

```sh
brew install fluxcd/tap/flux
flux check --pre --kubeconfig ~/.kube/config-homelab

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-homelab
```

`--password` here is the `flux-write` token, not an actual account
password. Flux stores it as a Kubernetes Secret in the `flux-system`
namespace once bootstrap completes — that Secret is what
`ImageUpdateAutomation` (step 10) reuses to push commits back, not anything
held by `k8s` itself. Clear this command from `k8s`'s shell history
afterwards (or prefix it with a space first, if `HISTCONTROL=ignorespace`
is set) since the token was passed as a plain argument.

This populates `clusters/homelab/flux-system/` and, because
`clusters/homelab/apps.yaml` already declares `Kustomization` objects for
`apps/podinfo` and `apps/hello-app`, both start reconciling immediately.

## 8. Verify the podinfo GitOps loop

```sh
flux get kustomizations --watch
kubectl -n podinfo get pods
```

Once it's `Ready`, point DNS at it and check in a browser (see step 11).

## 9. Enable Forgejo Actions and register a runner (rootless, no sudo)

Forgejo Actions needs a self-hosted runner — there's no shared runner pool.
The runner normally gets root-equivalent power over its host by mounting
`/var/run/docker.sock` (anyone who can push a workflow file effectively gets
root there). Instead: it runs as the same unprivileged `k8s` user from
step 2, using rootless Podman's own socket instead of Docker's — no root
anywhere in this pipeline. `build-hello-app.yml` already builds images with
kaniko, which needs no daemon and no elevated privileges at all.

The runner's job containers (kaniko, opentofu) never get the libvirt socket
or `k8s`'s home directory mounted in — only the Podman socket, needed to
launch those job containers in the first place — so a compromised workflow
can spawn containers as `k8s`, but can't directly touch the VMs or
Terraform state. Worth knowing given the runner lives on the same box/user
as the cluster's own infrastructure; fine for a demo-sized project, but if
this ever handles anything sensitive, move the runner to its own user or
VM so a breakout doesn't share a blast radius with the cluster.

1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket:
   ```sh
   systemctl --user enable --now podman.socket
   echo $XDG_RUNTIME_DIR   # note this path, e.g. /run/user/1001
   ```
2. Instance admin: `Site Administration → Actions → Runners`, confirm
   Actions is enabled.
3. Repo: `Settings → Actions → Runners → Create new runner`, copy the
   registration token.
4. Register and run the runner as a rootless Podman container, pointed at
   the Podman socket from step 1 instead of docker.sock:
   ```sh
   # still as k8s
   podman volume create forgejo-runner-data
   podman run -d --name forgejo-runner --restart unless-stopped \
     -e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v forgejo-runner-data:/data \
     code.forgejo.org/forgejo/runner:6 \
     forgejo-runner register --no-interactive \
       --instance https://git.boglabob.com \
       --token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
   ```
   The registered runner picks up both workflows in `.forgejo/workflows/` —
   `terraform.yml`'s `container:` image and `build-hello-app.yml`'s kaniko
   image are both launched through that same rootless Podman socket.
5. Repo `Settings → Secrets and Variables → Actions`, add:
   - Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications`
     on your Forgejo user, scope `package:write`) used to push images.
   - Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org.

If the `hello-app` package ends up private (Forgejo package visibility
follows repo visibility by default), create the cluster-side pull secret and
uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`:

```sh
kubectl -n hello-app create secret docker-registry forgejo-registry \
  --docker-server=git.boglabob.com \
  --docker-username=<FORGEJO_USER> \
  --docker-password=<FORGEJO_TOKEN>
```

## 10. Exercise the full loop

```sh
sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push
```

Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the page
with the current commit SHA and build time — see `apps/hello-app/src/index.html`)
→ pushes a new tag to `git.boglabob.com/codegit/hello-app` → Flux's
`ImageRepository` picks it up within a minute → `ImageUpdateAutomation`
commits the new tag back to `apps/hello-app/deployment.yaml` → the
`hello-app` Kustomization reconciles → `kubectl -n hello-app get pods` shows
a new pod, and `https://hello.boglabob.com` shows the new commit SHA/badge.

## 11. Expose the apps through Caddy

Since Caddy (Podman) is already the front door for `git.boglabob.com`, route
`podinfo` and `hello-app` through it too — but not the Dashboard or the API
server; see steps 12-13 for why.

1. Add A records for `podinfo.boglabob.com` and `hello.boglabob.com`
   pointing wherever `git.boglabob.com` already points (Caddy's host).
2. Add the blocks from `docs/Caddyfile.example` to Caddy's config, filling
   in your real node IPs (`tofu output node_ips`), and reload:
   ```sh
   podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
   ```
3. Check:
   ```sh
   curl https://podinfo.boglabob.com/
   curl https://hello.boglabob.com/
   ```

## 12. Access the Dashboard (LAN-only, on demand)

The Dashboard grants whatever its logged-in identity can do — with the
`admin-user` token from `apps/kubernetes-dashboard/rbac.yaml`, that's
cluster-admin. Publicly exposing that (even behind a login page) is the
exact pattern behind real breaches (Tesla, 2018: an internet-reachable,
unauthenticated Dashboard). So: no ingress, no standing hostname — only a
port-forward you open when you need it and close when you don't:

```sh
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
```

Then open `https://localhost:8443` and log in with the token:

```sh
kubectl -n kubernetes-dashboard get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d
```

## 13. kubectl access from elsewhere on the LAN, or remotely

Best practice for the Kubernetes API server is the same principle as the
Dashboard: never put 6443 on the public internet if you can avoid it,
because a leaked credential there is a full cluster compromise.

Node IPs (`10.20.30.0/24` by default) live on the private libvirt network
from step 5 — only the T630 itself can reach them directly, which is
actually a nice side effect: even the rest of your LAN can't touch the API
server without going through the T630 first. Two ways to do that:

- **SSH into the T630 as `k8s`** and run kubectl there directly (same as
  step 6) — simplest, and what this whole guide assumes by default.
- **Tunnel from another machine** (your laptop, or a phone via Termux, etc.):
  ```sh
  ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
  ```
  then point a local kubeconfig's `server:` at `https://127.0.0.1:6443`
  (copy the kubeconfig from step 6 and edit that one field). The cert
  validates because `k8s-api.boglabob.com` is in the server's TLS SAN list
  (`terraform/variables.tf`'s `k8s_api_hostname`) — add it to
  `/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine
  you're tunneling from and use that as the `server:` host instead of the
  raw IP, so the hostname in the URL matches a name the cert actually
  covers.
- **From outside your home network entirely**: Tailscale or WireGuard on
  the T630, then the SSH tunnel above over the Tailscale/WireGuard link
  instead of the open internet. Reasonable next stretch goal once the core
  loop is working — don't port-forward 22 or 6443 on your router for this.

The `admin-user` bearer token (step 12) also works for kubectl over the same
tunnel, if you'd rather not manage the client-cert kubeconfig.

## Stretch goals, roughly in order

- **Remote access**: Tailscale or WireGuard on the k3s server node, for
  kubectl/Dashboard access from outside the LAN without opening anything
  publicly (step 13).
- **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if
  `boglabob.com` isn't publicly reachable on 80/443).
- **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the
  `K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret
  store.
- **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as
  podinfo (HelmRepository + HelmRelease under `apps/`).
- **HA**: add a second k3s server node and switch from SQLite to embedded
  etcd (`--cluster-init` on the first server, `--server` join on the second).