CodeGit/cloud-demo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

initial scaffold
Some checks failed
terraform / validate (push) Waiting to run
Details
build-hello-app / build-and-push (push) Has been cancelled
Details

Browse source
This commit is contained in:
CodeGit 2026-08-18 20:22:42 +01:00
commit 5c2080a73b
31 changed files with 1244 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

339
docs/SETUP.md Normal file
View file

@ -0,0 +1,339 @@
# Setup walkthrough
Assumes: the T630 is an existing Debian box already running other
self-hosted services — this project installs alongside those as ordinary
packages (`qemu-kvm`/`libvirt`), not a hypervisor OS replacing Debian, and
is sized deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total)
so it stays a demo rather than competing with what's already running. Forgejo
is already running and reachable at `https://git.boglabob.com`, and you can
point DNS records under `boglabob.com` at hosts on your network (directly,
or via whatever reverse proxy/tunnel already gets `git.boglabob.com` there).
---
## 1. Install KVM/libvirt on the T630
Ordinary packages, no reboot into an installer, nothing else on the box is
touched:
```sh
# on the T630
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst
# confirm hardware virtualization is available (T630's Xeons support it)
sudo kvm-ok
```
## 2. Create the unprivileged 'k8s' user
One dedicated, no-sudo user for everything this project touches: driving
Terraform/kubectl/flux against libvirt here, and running the Forgejo Actions
runner later (step 9). It needs group membership to talk to libvirt — that's
a one-time root action; nothing it does afterwards needs `sudo`.
```sh
sudo useradd -m -s /bin/bash k8s # one-time, needs root to create the user itself
sudo usermod -aG libvirt,kvm k8s
sudo loginctl enable-linger k8s # lets its services keep running after logout
# as k8s, from here on (sudo -iu, not su -, since k8s has no password set):
sudo -iu k8s
ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/k3s_homelab # only needed if you'll SSH in as k8s day-to-day
virsh -c qemu:///system list --all # sanity check: should run with no permission error, no sudo
```
Do the rest of this guide logged in as `k8s` on the T630 itself (`ssh
k8s@t630`) — node IPs (step 5) live on a private libvirt network that's only
directly reachable from the T630, so this is the simplest place to run
`tofu`/`kubectl`/`flux` from. (If you'd rather drive Terraform from your own
workstation instead, see the `libvirt_uri` comment in
`terraform/terraform.tfvars.example` — you'll then need an SSH tunnel for
kubectl/flux to reach node IPs.)
## 3. Generate the secrets Terraform needs
```sh
openssl rand -hex 32 # -> k3s_token
```
## 4. Push this repo to Forgejo
Doing this before provisioning (rather than after) means `k8s` can get the
repo with a plain `git clone` in step 5, instead of needing a one-off copy
handed to it — and any future Terraform change just needs a `git pull`.
Repo/owner used throughout this guide: `codegit/cloud-demo` (already baked
into `apps/hello-app/deployment.yaml` and `image-automation.yaml`'s image
references — no placeholder-swapping needed).
1. On Forgejo (`https://git.boglabob.com`), as `codegit`: **+ → New
Repository** → name `cloud-demo`. Leave it empty — don't initialize with
a README/`.gitignore`/license, since this repo already has its own.
Visibility (public/private) is your call; either works, since access for
`k8s`/Flux/CI goes through the deploy key below regardless.
2. Locally, wherever you're editing this repo (`maq`):
```sh
git init # if not already
git add .
git commit -m "initial scaffold"
git remote add origin https://git.boglabob.com/codegit/cloud-demo.git
git push -u origin main
```
3. Add `k8s`'s public key (`~/.ssh/k3s_homelab.pub` from step 2, on the
T630 — `cat ~/.ssh/k3s_homelab.pub` as `k8s` if you need to grab it
again) as a Deploy Key on this repo — `Settings → Deploy Keys` — **with
write access**. One key, added once, covers `k8s`'s `git clone`/`pull`
(step 5) *and* `flux bootstrap` (step 7) *and*
`ImageUpdateAutomation`'s commits back (step 10), since all three reuse
this same keypair.
## 5. Provision the VMs with OpenTofu
Installing the package needs sudo, so that part is you (`maq`), not `k8s`.
Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu`
afterwards with no further root involvement:
```sh
# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh
```
Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved —
clone using the deploy key from step 4:
```sh
# as k8s
GIT_SSH_COMMAND="ssh -i ~/.ssh/k3s_homelab" \
git clone git@git.boglabob.com:codegit/cloud-demo.git ~/k3s
cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/k3s_homelab.pub
# from step 2), k3s_token. Defaults for network/sizing are fine to start.
tofu init
tofu plan
tofu apply
```
For any later change to `terraform/`: edit and push as `maq` as usual, then
`cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`.
This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the
`k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from
anything else already using libvirt on this box). Cloud-init installs k3s on
each on first boot — give it ~2 minutes after `apply` finishes.
## 6. Get kubectl talking to the cluster
```sh
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
| sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab
export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes # expect 3 Ready nodes
```
This works as-is because you're running it on the T630, which can reach the
`10.20.30.0/24` network directly. To also use kubectl from your own laptop,
either `scp` this kubeconfig over and open an SSH tunnel first
(`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's
`server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s`
whenever you need kubectl — simplest by far for a project this size.
## 7. Bootstrap Flux against Forgejo
Forgejo isn't a Flux-native provider (unlike GitHub/GitLab), so use the
generic git bootstrap, reusing `k8s`'s keypair — already added as a write
deploy key in step 4, so unlike a default `flux bootstrap` run, this one
won't print a new key to add:
```sh
brew install fluxcd/tap/flux
flux check --pre --kubeconfig ~/.kube/config-homelab
flux bootstrap git \
--url=ssh://git@git.boglabob.com:22/codegit/cloud-demo.git \
--branch=main \
--path=clusters/homelab \
--private-key-file=~/.ssh/k3s_homelab \
--kubeconfig ~/.kube/config-homelab
```
This populates `clusters/homelab/flux-system/` and, because
`clusters/homelab/apps.yaml` already declares `Kustomization` objects for
`apps/podinfo` and `apps/hello-app`, both start reconciling immediately.
## 8. Verify the podinfo GitOps loop
```sh
flux get kustomizations --watch
kubectl -n podinfo get pods
```
Once it's `Ready`, point DNS at it and check in a browser (see step 11).
## 9. Enable Forgejo Actions and register a runner (rootless, no sudo)
Forgejo Actions needs a self-hosted runner — there's no shared runner pool.
The runner normally gets root-equivalent power over its host by mounting
`/var/run/docker.sock` (anyone who can push a workflow file effectively gets
root there). Instead: it runs as the same unprivileged `k8s` user from
step 2, using rootless Podman's own socket instead of Docker's — no root
anywhere in this pipeline. `build-hello-app.yml` already builds images with
kaniko, which needs no daemon and no elevated privileges at all.
The runner's job containers (kaniko, opentofu) never get the libvirt socket
or `k8s`'s home directory mounted in — only the Podman socket, needed to
launch those job containers in the first place — so a compromised workflow
can spawn containers as `k8s`, but can't directly touch the VMs or
Terraform state. Worth knowing given the runner lives on the same box/user
as the cluster's own infrastructure; fine for a demo-sized project, but if
this ever handles anything sensitive, move the runner to its own user or
VM so a breakout doesn't share a blast radius with the cluster.
1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket:
```sh
systemctl --user enable --now podman.socket
echo $XDG_RUNTIME_DIR # note this path, e.g. /run/user/1001
```
2. Instance admin: `Site Administration → Actions → Runners`, confirm
Actions is enabled.
3. Repo: `Settings → Actions → Runners → Create new runner`, copy the
registration token.
4. Register and run the runner as a rootless Podman container, pointed at
the Podman socket from step 1 instead of docker.sock:
```sh
# still as k8s
podman volume create forgejo-runner-data
podman run -d --name forgejo-runner --restart unless-stopped \
-e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
-v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
-v forgejo-runner-data:/data \
code.forgejo.org/forgejo/runner:6 \
forgejo-runner register --no-interactive \
--instance https://git.boglabob.com \
--token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
```
The registered runner picks up both workflows in `.forgejo/workflows/` —
`terraform.yml`'s `container:` image and `build-hello-app.yml`'s kaniko
image are both launched through that same rootless Podman socket.
5. Repo `Settings → Secrets and Variables → Actions`, add:
- Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications`
on your Forgejo user, scope `package:write`) used to push images.
- Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org.
If the `hello-app` package ends up private (Forgejo package visibility
follows repo visibility by default), create the cluster-side pull secret and
uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`:
```sh
kubectl -n hello-app create secret docker-registry forgejo-registry \
--docker-server=git.boglabob.com \
--docker-username=<FORGEJO_USER> \
--docker-password=<FORGEJO_TOKEN>
```
## 10. Exercise the full loop
```sh
sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push
```
Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the page
with the current commit SHA and build time — see `apps/hello-app/src/index.html`)
→ pushes a new tag to `git.boglabob.com/codegit/hello-app` → Flux's
`ImageRepository` picks it up within a minute → `ImageUpdateAutomation`
commits the new tag back to `apps/hello-app/deployment.yaml` → the
`hello-app` Kustomization reconciles → `kubectl -n hello-app get pods` shows
a new pod, and `https://hello.boglabob.com` shows the new commit SHA/badge.
## 11. Expose the apps through Caddy
Since Caddy (Podman) is already the front door for `git.boglabob.com`, route
`podinfo` and `hello-app` through it too — but not the Dashboard or the API
server; see steps 12-13 for why.
1. Add A records for `podinfo.boglabob.com` and `hello.boglabob.com`
pointing wherever `git.boglabob.com` already points (Caddy's host).
2. Add the blocks from `docs/Caddyfile.example` to Caddy's config, filling
in your real node IPs (`tofu output node_ips`), and reload:
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
3. Check:
```sh
curl https://podinfo.boglabob.com/
curl https://hello.boglabob.com/
```
## 12. Access the Dashboard (LAN-only, on demand)
The Dashboard grants whatever its logged-in identity can do — with the
`admin-user` token from `apps/kubernetes-dashboard/rbac.yaml`, that's
cluster-admin. Publicly exposing that (even behind a login page) is the
exact pattern behind real breaches (Tesla, 2018: an internet-reachable,
unauthenticated Dashboard). So: no ingress, no standing hostname — only a
port-forward you open when you need it and close when you don't:
```sh
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
```
Then open `https://localhost:8443` and log in with the token:
```sh
kubectl -n kubernetes-dashboard get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d
```
## 13. kubectl access from elsewhere on the LAN, or remotely
Best practice for the Kubernetes API server is the same principle as the
Dashboard: never put 6443 on the public internet if you can avoid it,
because a leaked credential there is a full cluster compromise.
Node IPs (`10.20.30.0/24` by default) live on the private libvirt network
from step 5 — only the T630 itself can reach them directly, which is
actually a nice side effect: even the rest of your LAN can't touch the API
server without going through the T630 first. Two ways to do that:
- **SSH into the T630 as `k8s`** and run kubectl there directly (same as
step 6) — simplest, and what this whole guide assumes by default.
- **Tunnel from another machine** (your laptop, or a phone via Termux, etc.):
```sh
ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
```
then point a local kubeconfig's `server:` at `https://127.0.0.1:6443`
(copy the kubeconfig from step 6 and edit that one field). The cert
validates because `k8s-api.boglabob.com` is in the server's TLS SAN list
(`terraform/variables.tf`'s `k8s_api_hostname`) — add it to
`/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine
you're tunneling from and use that as the `server:` host instead of the
raw IP, so the hostname in the URL matches a name the cert actually
covers.
- **From outside your home network entirely**: Tailscale or WireGuard on
the T630, then the SSH tunnel above over the Tailscale/WireGuard link
instead of the open internet. Reasonable next stretch goal once the core
loop is working — don't port-forward 22 or 6443 on your router for this.
The `admin-user` bearer token (step 12) also works for kubectl over the same
tunnel, if you'd rather not manage the client-cert kubeconfig.
## Stretch goals, roughly in order
- **Remote access**: Tailscale or WireGuard on the k3s server node, for
kubectl/Dashboard access from outside the LAN without opening anything
publicly (step 13).
- **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if
`boglabob.com` isn't publicly reachable on 80/443).
- **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the
`K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret
store.
- **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as
podinfo (HelmRepository + HelmRelease under `apps/`).
- **HA**: add a second k3s server node and switch from SQLite to embedded
etcd (`--cluster-init` on the first server, `--server` join on the second).