cloud-demo/docs/SETUP.md

Setup walkthrough (full, Terraform-driven)

This is the "do it properly" path — Terraform/OpenTofu provisioning all 3 VMs via the dmacvicar/libvirt provider. It's also the one currently blocked on getting that provider's HCL right (0.8.x vs 0.9.x schema — see conversation history). If you want a real cluster to learn Flux/GitOps on right now without waiting on that, see docs/QUICKSTART.md instead — a single manually-created VM, no Terraform, with teardown instructions for switching over once this track is sorted. Steps 1–4 below are shared between both guides.

Assumes: the T630 is an existing Debian box already running other self-hosted services — this project installs alongside those as ordinary packages (qemu-kvm/libvirt), not a hypervisor OS replacing Debian, and is sized deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total) so it stays a demo rather than competing with what's already running. Forgejo is already running and reachable at https://git.boglabob.com, and you can point DNS records under boglabob.com at hosts on your network (directly, or via whatever reverse proxy/tunnel already gets git.boglabob.com there).

---

1. Install KVM/libvirt on the T630

Ordinary packages, no reboot into an installer, nothing else on the box is touched:

# on the T630
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst

# confirm hardware virtualization is available (T630's Xeons support it)
sudo kvm-ok

2. Create the unprivileged 'k8s' user

One dedicated, no-sudo user for everything this project touches: driving Terraform/kubectl/flux against libvirt here, and running the Forgejo Actions runner later (step 9). It needs group membership to talk to libvirt — that's a one-time root action; nothing it does afterwards needs sudo.

sudo useradd -m -s /bin/bash k8s          # one-time, needs root to create the user itself
sudo usermod -aG libvirt,kvm k8s
sudo loginctl enable-linger k8s           # lets its services keep running after logout

# as k8s, from here on (sudo -iu, not su -, since k8s has no password set):
sudo -iu k8s
ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/id_ed25519   # only needed if you'll SSH in as k8s day-to-day
virsh -c qemu:///system list --all        # sanity check: should run with no permission error, no sudo

Do the rest of this guide logged in as k8s on the T630 itself (ssh k8s@t630) — node IPs (step 5) live on a private libvirt network that's only directly reachable from the T630, so this is the simplest place to run tofu/kubectl/flux from. (If you'd rather drive Terraform from your own workstation instead, see the libvirt_uri comment in terraform/terraform.tfvars.example — you'll then need an SSH tunnel for kubectl/flux to reach node IPs.)

3. Generate the secrets Terraform needs

openssl rand -hex 32   # -> k3s_token

4. Push this repo to Forgejo

Doing this before provisioning (rather than after) means k8s can get the repo with a plain git clone in step 5, instead of needing a one-off copy handed to it — and any future Terraform change just needs a git pull. Repo/owner used throughout this guide: codegit/cloud-demo (already baked into apps/hello-app/deployment.yaml and image-automation.yaml's image references — no placeholder-swapping needed).

1. On Forgejo (https://git.boglabob.com), as codegit: + → New Repository → name cloud-demo. Leave it empty — don't initialize with a README/.gitignore/license, since this repo already has its own. Visibility (public/private) is your call; either works, since access for k8s/Flux/CI goes through the tokens below regardless.

2. Locally, wherever you're editing this repo (maq):

   git init   # if not already
   git add .
   git commit -m "initial scaffold"
   git remote add origin https://git.boglabob.com/codegit/cloud-demo.git
   git push -u origin main
   

3. Generate two access tokens (Settings → Applications → Generate New Token), scoped as narrowly as Forgejo's token UI allows to repository read/write:

   • k8s-readonly — read-only. Used only for k8s's own manual git clone/pull on the T630 (step 5) — never leaves that box, isn't used by anything automated.
   • flux-write — read/write. Used once, as a flux bootstrap argument (step 7); Flux stores it as a Kubernetes Secret inside the cluster from then on (ImageUpdateAutomation's commits back in step 10 reuse that same in-cluster Secret) — it's never written to k8s's filesystem at all.

   Using HTTPS tokens instead of k8s's SSH key (~/.ssh/id_ed25519, from step 2) sidesteps an open question: Forgejo's git-SSH port isn't reachable from this desktop through your router (see the SSH troubleshooting earlier in this conversation), and whether it's reachable from k8s on the T630 itself was never actually confirmed either. HTTPS (443, via Caddy) is already proven to work, so both tokens use that instead. Copy both token values now — Forgejo only shows them once.

5. Provision the VMs with OpenTofu

Installing the package needs sudo, so that part is you (maq), not k8s. Installing system-wide (/usr/local/bin) means k8s can just use tofu afterwards with no further root involvement:

# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh

Everything from here on is k8s again (sudo -iu k8s), no sudo involved — clone using the k8s-readonly token from step 4. k8s has no keyring (it's headless, no desktop session), so this uses git credential-store — a plaintext file, chmod 600'd, holding only the read-only token:

# as k8s
git config --global credential.helper store
git clone https://git.boglabob.com/codegit/cloud-demo.git ~/k3s
# prompts for username (anything) and password (paste the k8s-readonly
# token) once; stores it in ~/.git-credentials for next time
chmod 600 ~/.git-credentials

cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/id_ed25519.pub
# from step 2), k3s_token. Defaults for network/sizing are fine to start.

tofu init
tofu plan
tofu apply

For any later change to terraform/: edit and push as maq as usual, then cd ~/k3s && git pull as k8s before re-running tofu plan/apply.

This brings up k3s-server-1, k3s-agent-1, k3s-agent-2 on the k3s-homelab libvirt network (10.20.30.0/24 by default — isolated from anything else already using libvirt on this box). Cloud-init installs k3s on each on first boot — give it ~2 minutes after apply finishes.

6. Get kubectl talking to the cluster

kubectl itself was never actually installed anywhere earlier in this guide despite being listed as a prerequisite — install it now (as k8s, no sudo needed, same pattern as the OpenTofu install):

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"

mkdir -p ~/.kube
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab

export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes   # expect 3 Ready nodes

This works as-is because you're running it on the T630, which can reach the 10.20.30.0/24 network directly. To also use kubectl from your own laptop, either scp this kubeconfig over and open an SSH tunnel first (ssh -L 6443:10.20.30.11:6443 k8s@t630, then point the kubeconfig's server: at https://127.0.0.1:6443), or just SSH into the T630 as k8s whenever you need kubectl — simplest by far for a project this size.

7. Bootstrap Flux against Forgejo

Forgejo isn't a Flux-native provider (unlike GitHub/GitLab), so use the generic git bootstrap — over HTTPS with the flux-write token from step 4, not SSH (same reachability reasoning as step 5):

Install the Flux CLI the same no-sudo, no-package-manager way as kubectl (brew assumes Homebrew, which isn't a given on a bare Debian box):

FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version

flux check --pre --kubeconfig ~/.kube/config-homelab

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-homelab

--password here is the flux-write token, not an actual account password. Flux stores it as a Kubernetes Secret in the flux-system namespace once bootstrap completes — that Secret is what ImageUpdateAutomation (step 10) reuses to push commits back, not anything held by k8s itself. Clear this command from k8s's shell history afterwards (or prefix it with a space first, if HISTCONTROL=ignorespace is set) since the token was passed as a plain argument.

This populates clusters/homelab/flux-system/ and, because clusters/homelab/apps.yaml already declares Kustomization objects for apps/podinfo and apps/hello-app, both start reconciling immediately.

8. Verify the podinfo GitOps loop

flux get kustomizations --watch
kubectl -n podinfo get pods

Once it's Ready, point DNS at it and check in a browser (see step 11).

9. Enable Forgejo Actions and register a runner (rootless, no sudo)

Forgejo Actions needs a self-hosted runner — there's no shared runner pool. The runner normally gets root-equivalent power over its host by mounting /var/run/docker.sock (anyone who can push a workflow file effectively gets root there). Instead: it runs as the same unprivileged k8s user from step 2, using rootless Podman's own socket instead of Docker's — no root anywhere in this pipeline. build-hello-app.yml already builds images with kaniko, which needs no daemon and no elevated privileges at all.

The runner's job containers (kaniko, opentofu) never get the libvirt socket or k8s's home directory mounted in — only the Podman socket, needed to launch those job containers in the first place — so a compromised workflow can spawn containers as k8s, but can't directly touch the VMs or Terraform state. Worth knowing given the runner lives on the same box/user as the cluster's own infrastructure; fine for a demo-sized project, but if this ever handles anything sensitive, move the runner to its own user or VM so a breakout doesn't share a blast radius with the cluster.

1. As k8s (ssh k8s@t630), enable the rootless Podman API socket:

   systemctl --user enable --now podman.socket
   echo $XDG_RUNTIME_DIR   # note this path, e.g. /run/user/1001
   

2. Instance admin: Site Administration → Actions → Runners, confirm Actions is enabled.
3. Repo: Settings → Actions → Runners → Create new runner, copy the registration token.
4. Register and run the runner as a rootless Podman container, pointed at the Podman socket from step 1 instead of docker.sock:

   # still as k8s
   podman volume create forgejo-runner-data
   podman run -d --name forgejo-runner --restart unless-stopped \
     -e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
     -v forgejo-runner-data:/data \
     code.forgejo.org/forgejo/runner:6 \
     forgejo-runner register --no-interactive \
       --instance https://git.boglabob.com \
       --token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
   

   The registered runner picks up both workflows in .forgejo/workflows/ — terraform.yml's container: image and build-hello-app.yml's kaniko image are both launched through that same rootless Podman socket.
5. Repo Settings → Secrets and Variables → Actions, add:
   • Secret FORGEJO_TOKEN — a personal access token (Settings → Applications on your Forgejo user, scope package:write) used to push images.
   • Variable FORGEJO_USER, FORGEJO_ORG — your Forgejo username/org.

If the hello-app package ends up private (Forgejo package visibility follows repo visibility by default), create the cluster-side pull secret and uncomment the imagePullSecrets line in apps/hello-app/deployment.yaml:

kubectl -n hello-app create secret docker-registry forgejo-registry \
  --docker-server=git.boglabob.com \
  --docker-username=<FORGEJO_USER> \
  --docker-password=<FORGEJO_TOKEN>

10. Exercise the full loop

sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push

Watch: build-hello-app runs in Forgejo Actions (which also stamps the page with the current commit SHA and build time — see apps/hello-app/src/index.html) → pushes a new tag to git.boglabob.com/codegit/hello-app → Flux's ImageRepository picks it up within a minute → ImageUpdateAutomation commits the new tag back to apps/hello-app/deployment.yaml → the hello-app Kustomization reconciles → kubectl -n hello-app get pods shows a new pod, and https://hello.boglabob.com shows the new commit SHA/badge.

11. Expose the apps through Caddy

Since Caddy (Podman) is already the front door for git.boglabob.com, route podinfo and hello-app through it too — but not the Dashboard or the API server; see steps 12-13 for why.

1. Add CNAME records for podinfo.boglabob.com and hello.boglabob.com pointing at git.boglabob.com (matching how every other record for this server is set up) — one source of truth for the Caddy host's IP, rather than duplicating it across records.
2. Add the blocks from docs/Caddyfile.example to Caddy's config, filling in your real node IPs (tofu output node_ips), and reload:

   podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
   

3. Check:

   curl https://podinfo.boglabob.com/
   curl https://hello.boglabob.com/
   

12. Access the Dashboard (LAN-only, on demand)

The Dashboard grants whatever its logged-in identity can do — with the admin-user token from apps/kubernetes-dashboard/rbac.yaml, that's cluster-admin. Publicly exposing that (even behind a login page) is the exact pattern behind real breaches (Tesla, 2018: an internet-reachable, unauthenticated Dashboard). So: no ingress, no standing hostname — only a port-forward you open when you need it and close when you don't:

kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443

Then open https://localhost:8443 and log in with the token:

kubectl -n kubernetes-dashboard get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d

13. kubectl access from elsewhere on the LAN, or remotely

Best practice for the Kubernetes API server is the same principle as the Dashboard: never put 6443 on the public internet if you can avoid it, because a leaked credential there is a full cluster compromise.

Node IPs (10.20.30.0/24 by default) live on the private libvirt network from step 5 — only the T630 itself can reach them directly, which is actually a nice side effect: even the rest of your LAN can't touch the API server without going through the T630 first. Two ways to do that:

• SSH into the T630 as k8s and run kubectl there directly (same as step 6) — simplest, and what this whole guide assumes by default.
• Tunnel from another machine (your laptop, or a phone via Termux, etc.):

  ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
  

  then point a local kubeconfig's server: at https://127.0.0.1:6443 (copy the kubeconfig from step 6 and edit that one field). The cert validates because k8s-api.boglabob.com is in the server's TLS SAN list (terraform/variables.tf's k8s_api_hostname) — add it to /etc/hosts as 127.0.0.1 k8s-api.boglabob.com on whatever machine you're tunneling from and use that as the server: host instead of the raw IP, so the hostname in the URL matches a name the cert actually covers.
• From outside your home network entirely: Tailscale or WireGuard on the T630, then the SSH tunnel above over the Tailscale/WireGuard link instead of the open internet. Reasonable next stretch goal once the core loop is working — don't port-forward 22 or 6443 on your router for this.

The admin-user bearer token (step 12) also works for kubectl over the same tunnel, if you'd rather not manage the client-cert kubeconfig.

Stretch goals, roughly in order

• Remote access: Tailscale or WireGuard on the k3s server node, for kubectl/Dashboard access from outside the LAN without opening anything publicly (step 13).
• TLS: cert-manager + a ClusterIssuer for Let's Encrypt (DNS-01 if boglabob.com isn't publicly reachable on 80/443).
• Secrets in Git: sops + age, or sealed-secrets, so the K3S_TOKEN/API tokens above don't need to live only in Forgejo's secret store.
• Monitoring: kube-prometheus-stack via Helm, deployed the same way as podinfo (HelmRepository + HelmRelease under apps/).
• HA: add a second k3s server node and switch from SQLite to embedded etcd (--cluster-init on the first server, --server join on the second).