# Setup walkthrough (full, Terraform-driven) This is the "do it properly" path — Terraform/OpenTofu provisioning all 3 VMs via the `dmacvicar/libvirt` provider. It's also the one currently blocked on getting that provider's HCL right (0.8.x vs 0.9.x schema — see conversation history). If you want a real cluster to learn Flux/GitOps on *right now* without waiting on that, see `docs/QUICKSTART.md` instead — a single manually-created VM, no Terraform, with teardown instructions for switching over once this track is sorted. Steps 1–4 below are shared between both guides. Assumes: the T630 is an existing Debian box already running other self-hosted services — this project installs alongside those as ordinary packages (`qemu-kvm`/`libvirt`), not a hypervisor OS replacing Debian, and is sized deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total) so it stays a demo rather than competing with what's already running. Forgejo is already running and reachable at `https://git.boglabob.com`, and you can point DNS records under `boglabob.com` at hosts on your network (directly, or via whatever reverse proxy/tunnel already gets `git.boglabob.com` there). --- ## 1. Install KVM/libvirt on the T630 Ordinary packages, no reboot into an installer, nothing else on the box is touched: ```sh # on the T630 sudo apt update sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst # confirm hardware virtualization is available (T630's Xeons support it) sudo kvm-ok ``` ## 2. Create the unprivileged 'k8s' user One dedicated, no-sudo user for everything this project touches: driving Terraform/kubectl/flux against libvirt here, and running the Forgejo Actions runner later (step 9). It needs group membership to talk to libvirt — that's a one-time root action; nothing it does afterwards needs `sudo`. ```sh sudo useradd -m -s /bin/bash k8s # one-time, needs root to create the user itself sudo usermod -aG libvirt,kvm k8s sudo loginctl enable-linger k8s # lets its services keep running after logout # as k8s, from here on (sudo -iu, not su -, since k8s has no password set): sudo -iu k8s ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/id_ed25519 # only needed if you'll SSH in as k8s day-to-day virsh -c qemu:///system list --all # sanity check: should run with no permission error, no sudo ``` Do the rest of this guide logged in as `k8s` on the T630 itself (`ssh k8s@t630`) — node IPs (step 5) live on a private libvirt network that's only directly reachable from the T630, so this is the simplest place to run `tofu`/`kubectl`/`flux` from. (If you'd rather drive Terraform from your own workstation instead, see the `libvirt_uri` comment in `terraform/terraform.tfvars.example` — you'll then need an SSH tunnel for kubectl/flux to reach node IPs.) ## 3. Generate the secrets Terraform needs ```sh openssl rand -hex 32 # -> k3s_token ``` ## 4. Push this repo to Forgejo Doing this before provisioning (rather than after) means `k8s` can get the repo with a plain `git clone` in step 5, instead of needing a one-off copy handed to it — and any future Terraform change just needs a `git pull`. Repo/owner used throughout this guide: `codegit/cloud-demo` (already baked into `apps/hello-app/deployment.yaml` and `image-automation.yaml`'s image references — no placeholder-swapping needed). 1. On Forgejo (`https://git.boglabob.com`), as `codegit`: **+ → New Repository** → name `cloud-demo`. Leave it empty — don't initialize with a README/`.gitignore`/license, since this repo already has its own. Visibility (public/private) is your call; either works, since access for `k8s`/Flux/CI goes through the tokens below regardless. 2. Locally, wherever you're editing this repo (`maq`): ```sh git init # if not already git add . git commit -m "initial scaffold" git remote add origin https://git.boglabob.com/codegit/cloud-demo.git git push -u origin main ``` 3. Generate two access tokens (`Settings → Applications → Generate New Token`), scoped as narrowly as Forgejo's token UI allows to repository read/write: - **`k8s-readonly`** — read-only. Used only for `k8s`'s own manual `git clone`/`pull` on the T630 (step 5) — never leaves that box, isn't used by anything automated. - **`flux-write`** — read/write. Used once, as a `flux bootstrap` argument (step 7); Flux stores it as a Kubernetes Secret inside the cluster from then on (`ImageUpdateAutomation`'s commits back in step 10 reuse that same in-cluster Secret) — it's never written to `k8s`'s filesystem at all. Using HTTPS tokens instead of `k8s`'s SSH key (`~/.ssh/id_ed25519`, from step 2) sidesteps an open question: Forgejo's git-SSH port isn't reachable from this desktop through your router (see the SSH troubleshooting earlier in this conversation), and whether it's reachable from `k8s` on the T630 itself was never actually confirmed either. HTTPS (443, via Caddy) is already proven to work, so both tokens use that instead. Copy both token values now — Forgejo only shows them once. ## 5. Provision the VMs with OpenTofu Installing the package needs sudo, so that part is you (`maq`), not `k8s`. Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu` afterwards with no further root involvement: ```sh # as maq (has sudo) sudo apt install -y unzip curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh ``` Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved — clone using the `k8s-readonly` token from step 4. `k8s` has no keyring (it's headless, no desktop session), so this uses `git credential-store` — a plaintext file, `chmod 600`'d, holding only the read-only token: ```sh # as k8s git config --global credential.helper store git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s # prompts for username (anything) and password (paste the k8s-readonly # token) once; stores it in ~/.git-credentials for next time chmod 600 ~/.git-credentials cd ~/k3s/terraform cp terraform.tfvars.example terraform.tfvars # edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub # from step 2), k3s_token. Defaults for network/sizing are fine to start. tofu init tofu plan tofu apply ``` For any later change to `terraform/`: edit and push as `maq` as usual, then `cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`. This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the `k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from anything else already using libvirt on this box). Cloud-init installs k3s on each on first boot — give it ~2 minutes after `apply` finishes. ## 6. Get kubectl talking to the cluster `kubectl` itself was never actually installed anywhere earlier in this guide despite being listed as a prerequisite — install it now (as `k8s`, no sudo needed, same pattern as the OpenTofu install): ```sh curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl mkdir -p ~/.local/bin mv kubectl ~/.local/bin/ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc export PATH="$HOME/.local/bin:$PATH" ``` ```sh mkdir -p ~/.kube ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \ | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab export KUBECONFIG=~/.kube/config-homelab kubectl get nodes # expect 3 Ready nodes ``` This works as-is because you're running it on the T630, which can reach the `10.20.30.0/24` network directly. To also use kubectl from your own laptop, either `scp` this kubeconfig over and open an SSH tunnel first (`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's `server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s` whenever you need kubectl — simplest by far for a project this size. ## 7. Bootstrap Flux against Forgejo Forgejo isn't a Flux-native provider (unlike GitHub/GitLab), so use the generic git bootstrap — over HTTPS with the `flux-write` token from step 4, not SSH (same reachability reasoning as step 5): Install the Flux CLI the same no-sudo, no-package-manager way as `kubectl` (`brew` assumes Homebrew, which isn't a given on a bare Debian box): ```sh FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//') curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux rm /tmp/flux.tar.gz flux --version ``` ```sh flux check --pre --kubeconfig ~/.kube/config-homelab flux bootstrap git \ --url=https://git.boglabob.com/codegit/cloud-demo \ --branch=main \ --path=clusters/homelab \ --username=codegit \ --password= \ --token-auth \ --kubeconfig ~/.kube/config-homelab ``` `--password` here is the `flux-write` token, not an actual account password. Flux stores it as a Kubernetes Secret in the `flux-system` namespace once bootstrap completes — that Secret is what `ImageUpdateAutomation` (step 10) reuses to push commits back, not anything held by `k8s` itself. Clear this command from `k8s`'s shell history afterwards (or prefix it with a space first, if `HISTCONTROL=ignorespace` is set) since the token was passed as a plain argument. This populates `clusters/homelab/flux-system/` and, because `clusters/homelab/apps.yaml` already declares `Kustomization` objects for `apps/podinfo` and `apps/hello-app`, both start reconciling immediately. ## 8. Verify the podinfo GitOps loop ```sh flux get kustomizations --watch kubectl -n podinfo get pods ``` Once it's `Ready`, point DNS at it and check in a browser (see step 11). ## 9. Enable Forgejo Actions and register a runner (rootless, no sudo) Forgejo Actions needs a self-hosted runner — there's no shared runner pool. The runner normally gets root-equivalent power over its host by mounting `/var/run/docker.sock` (anyone who can push a workflow file effectively gets root there). Instead: it runs as the same unprivileged `k8s` user from step 2, using rootless Podman's own socket instead of Docker's — no root anywhere in this pipeline. `build-hello-app.yml` already builds images with kaniko, which needs no daemon and no elevated privileges at all. The runner's job containers (kaniko, opentofu) never get the libvirt socket or `k8s`'s home directory mounted in — only the Podman socket, needed to launch those job containers in the first place — so a compromised workflow can spawn containers as `k8s`, but can't directly touch the VMs or Terraform state. Worth knowing given the runner lives on the same box/user as the cluster's own infrastructure; fine for a demo-sized project, but if this ever handles anything sensitive, move the runner to its own user or VM so a breakout doesn't share a blast radius with the cluster. 1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket: ```sh systemctl --user enable --now podman.socket echo $XDG_RUNTIME_DIR # note this path, e.g. /run/user/1001 ``` 2. Instance admin: `Site Administration → Actions → Runners`, confirm Actions is enabled. 3. Repo: `Settings → Actions → Runners → Create new runner`, copy the registration token. 4. Register and run the runner as a rootless Podman container, pointed at the Podman socket from step 1 instead of docker.sock: ```sh # still as k8s podman volume create forgejo-runner-data podman run -d --name forgejo-runner --restart unless-stopped \ -e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \ -v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \ -v forgejo-runner-data:/data \ code.forgejo.org/forgejo/runner:6 \ forgejo-runner register --no-interactive \ --instance https://git.boglabob.com \ --token --labels docker:docker://node:20-bookworm ``` The registered runner picks up both workflows in `.forgejo/workflows/` — `terraform.yml`'s `container:` image and `build-hello-app.yml`'s kaniko image are both launched through that same rootless Podman socket. 5. Repo `Settings → Secrets and Variables → Actions`, add: - Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications` on your Forgejo user, scope `package:write`) used to push images. - Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org. If the `hello-app` package ends up private (Forgejo package visibility follows repo visibility by default), create the cluster-side pull secret and uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`: ```sh kubectl -n hello-app create secret docker-registry forgejo-registry \ --docker-server=git.boglabob.com \ --docker-username= \ --docker-password= ``` ## 10. Exercise the full loop ```sh sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html git add apps/hello-app/src/index.html git commit -m "test the pipeline" git push ``` Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the page with the current commit SHA and build time — see `apps/hello-app/src/index.html`) → pushes a new tag to `git.boglabob.com/codegit/hello-app` → Flux's `ImageRepository` picks it up within a minute → `ImageUpdateAutomation` commits the new tag back to `apps/hello-app/deployment.yaml` → the `hello-app` Kustomization reconciles → `kubectl -n hello-app get pods` shows a new pod, and `https://hello.boglabob.com` shows the new commit SHA/badge. ## 11. Expose the apps through Caddy Since Caddy (Podman) is already the front door for `git.boglabob.com`, route `podinfo` and `hello-app` through it too — but not the Dashboard or the API server; see steps 12-13 for why. 1. Add CNAME records for `podinfo.boglabob.com` and `hello.boglabob.com` pointing at `git.boglabob.com` (matching how every other record for this server is set up) — one source of truth for the Caddy host's IP, rather than duplicating it across records. 2. Add the blocks from `docs/Caddyfile.example` to Caddy's config, filling in your real node IPs (`tofu output node_ips`), and reload: ```sh podman exec caddy reload --config /etc/caddy/Caddyfile ``` 3. Check: ```sh curl https://podinfo.boglabob.com/ curl https://hello.boglabob.com/ ``` ## 12. Access the Dashboard (LAN-only, on demand) The Dashboard grants whatever its logged-in identity can do — with the `admin-user` token from `apps/kubernetes-dashboard/rbac.yaml`, that's cluster-admin. Publicly exposing that (even behind a login page) is the exact pattern behind real breaches (Tesla, 2018: an internet-reachable, unauthenticated Dashboard). So: no ingress, no standing hostname — only a port-forward you open when you need it and close when you don't: ```sh kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443 ``` Then open `https://localhost:8443` and log in with the token: ```sh kubectl -n kubernetes-dashboard get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d ``` ## 13. kubectl access from elsewhere on the LAN, or remotely Best practice for the Kubernetes API server is the same principle as the Dashboard: never put 6443 on the public internet if you can avoid it, because a leaked credential there is a full cluster compromise. Node IPs (`10.20.30.0/24` by default) live on the private libvirt network from step 5 — only the T630 itself can reach them directly, which is actually a nice side effect: even the rest of your LAN can't touch the API server without going through the T630 first. Two ways to do that: - **SSH into the T630 as `k8s`** and run kubectl there directly (same as step 6) — simplest, and what this whole guide assumes by default. - **Tunnel from another machine** (your laptop, or a phone via Termux, etc.): ```sh ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630 ``` then point a local kubeconfig's `server:` at `https://127.0.0.1:6443` (copy the kubeconfig from step 6 and edit that one field). The cert validates because `k8s-api.boglabob.com` is in the server's TLS SAN list (`terraform/variables.tf`'s `k8s_api_hostname`) — add it to `/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine you're tunneling from and use that as the `server:` host instead of the raw IP, so the hostname in the URL matches a name the cert actually covers. - **From outside your home network entirely**: Tailscale or WireGuard on the T630, then the SSH tunnel above over the Tailscale/WireGuard link instead of the open internet. Reasonable next stretch goal once the core loop is working — don't port-forward 22 or 6443 on your router for this. The `admin-user` bearer token (step 12) also works for kubectl over the same tunnel, if you'd rather not manage the client-cert kubeconfig. ## Stretch goals, roughly in order - **Remote access**: Tailscale or WireGuard on the k3s server node, for kubectl/Dashboard access from outside the LAN without opening anything publicly (step 13). - **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if `boglabob.com` isn't publicly reachable on 80/443). - **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the `K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret store. - **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as podinfo (HelmRepository + HelmRelease under `apps/`). - **HA**: add a second k3s server node and switch from SQLite to embedded etcd (`--cluster-init` on the first server, `--server` join on the second).