CodeGit/cloud-demo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

initial scaffold
Some checks failed
terraform / validate (push) Waiting to run
Details
build-hello-app / build-and-push (push) Has been cancelled
Details

Browse source
This commit is contained in:
CodeGit 2026-08-18 20:22:42 +01:00
commit 5c2080a73b
31 changed files with 1244 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

40
.forgejo/workflows/build-hello-app.yml Normal file
View file

@ -0,0 +1,40 @@
name: build-hello-app
on:
push:
branches: [main]
paths:
- "apps/hello-app/src/**"
jobs:
build-and-push:
runs-on: docker
# kaniko builds the image itself with no daemon and no special host
# privileges, so the runner host only ever needs a rootless Podman
# socket to launch this container — never docker.sock, never sudo.
container:
image: gcr.io/kaniko-project/executor:debug
options: --entrypoint ""
steps:
- uses: actions/checkout@v4
- name: Write registry auth
run: |
mkdir -p /kaniko/.docker
AUTH=$(printf '%s:%s' "${{ vars.FORGEJO_USER }}" "${{ secrets.FORGEJO_TOKEN }}" | base64 -w0)
printf '{"auths":{"git.boglabob.com":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- name: Inject build info
run: |
SHORT_SHA="${GITHUB_SHA::7}"
BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
sed -i "s/__GIT_SHA__/$SHORT_SHA/; s/__BUILD_TIME__/$BUILD_TIME/" apps/hello-app/src/index.html
- name: Build and push
run: |
TAG="main-${GITHUB_SHA::7}-$(date +%s)"
/kaniko/executor \
--context="${{ github.workspace }}/apps/hello-app/src" \
--dockerfile="${{ github.workspace }}/apps/hello-app/src/Dockerfile" \
--destination="git.boglabob.com/${{ vars.FORGEJO_ORG }}/hello-app:$TAG" \
--destination="git.boglabob.com/${{ vars.FORGEJO_ORG }}/hello-app:latest"

35
.forgejo/workflows/terraform.yml Normal file
View file

@ -0,0 +1,35 @@
name: terraform
on:
pull_request:
paths:
- "terraform/**"
push:
branches: [main]
paths:
- "terraform/**"
jobs:
validate:
runs-on: docker
container:
image: ghcr.io/opentofu/opentofu:1.8
defaults:
run:
working-directory: terraform
steps:
- uses: actions/checkout@v4
- name: tofu fmt
run: tofu fmt -check -recursive
- name: tofu init
run: tofu init -backend=false
- name: tofu validate
run: tofu validate
# No `tofu plan` here: the libvirt provider needs to reach the T630's
# libvirt socket, which this ephemeral job container doesn't have
# access to. Real applies happen from the T630 itself as the `k8s`
# user (docs/SETUP.md step 5) - fmt/validate is what CI checks.

5
.gitignore vendored Normal file
View file

@ -0,0 +1,5 @@
terraform/.terraform/
terraform/terraform.tfvars
terraform/*.tfstate
terraform/*.tfstate.backup
*.env

81
README.md Normal file
View file

@ -0,0 +1,81 @@
# Homelab Platform — Terraform + k3s + Flux on Forgejo
Learning project: provision VMs on the T630 with OpenTofu/Terraform, bootstrap a
k3s cluster on them, and manage everything after that point through GitOps
(Flux) synced from a Forgejo repo, with Forgejo Actions handling CI. The T630
keeps running its other self-hosted services throughout — this installs as
ordinary KVM/libvirt packages next to them, sized deliberately small (3 VMs,
2 vCPU/2GB RAM each), not a hypervisor OS taking over the box.
## The loop
```
OpenTofu (terraform/)
-> provisions VMs on the T630 via KVM/libvirt (qemu:///system)
-> cloud-init installs k3s (1 server + 2 agents)
-> VMs live on an isolated private network (10.20.30.0/24), reachable
only from the T630 itself
Forgejo repo (this repo)
-> clusters/homelab/ = Flux config (what Flux itself watches)
-> apps/ = workloads Flux deploys
-> .forgejo/workflows/ = CI (terraform plan, image builds)
Flux (running in the cluster)
-> watches this repo
-> applies clusters/homelab/** and apps/** to k3s
Caddy (Podman, existing) is the public front door for boglabob.com, but only
for the low-stakes apps — the Dashboard and the k3s API server stay off it:
git.boglabob.com -> Forgejo (existing)
podinfo.boglabob.com -> Traefik (k3s ingress) -> podinfo
hello.boglabob.com -> Traefik (k3s ingress) -> hello-app
[Dashboard] -> kubectl port-forward only, never a public hostname
[k3s API, port 6443] -> reachable only via the T630 (SSH tunnel or run
kubectl there directly), no proxy, no public port
see docs/Caddyfile.example and docs/SETUP.md steps 12-13
```
## Directory layout
- `terraform/` — OpenTofu config that provisions the k3s VMs via KVM/libvirt
directly on the T630 (no separate hypervisor OS, no VM template step —
the cloud image is pulled straight from its URL).
- `terraform/cloud-init/` — cloud-init templates that install k3s server/agent
on first boot.
- `clusters/homelab/` — Flux's own config for this cluster (populated by
`flux bootstrap`, see docs/SETUP.md step 7).
- `apps/podinfo/` — first GitOps app: a HelmRelease for the standard Flux demo
app (podinfo). No CI needed — good for validating the Flux sync loop works.
- `apps/hello-app/` — capstone app, and the public showcase piece at
`hello.boglabob.com`: a one-page site (source + Dockerfile) built by
Forgejo Actions, pushed to Forgejo's container registry, deployed via a
Flux `ImagePolicy`/`ImageUpdateAutomation` so new pushes to main roll out
automatically. The page itself shows the live commit SHA and deploy
timestamp injected by CI, as proof the pipeline is really running rather
than a static screenshot.
- `apps/kubernetes-dashboard/` — the official Kubernetes Dashboard, with a
cluster-admin token (`apps/kubernetes-dashboard/rbac.yaml`). No ingress —
access is via `kubectl port-forward` only (see docs/SETUP.md step 12).
- `.forgejo/workflows/` — CI: terraform validate/plan on PRs, build+push
hello-app image on merge to main, via kaniko under a rootless Podman
runner (no docker.sock, no sudo — see docs/SETUP.md step 9).
- `docs/Caddyfile.example` — the reverse-proxy blocks for podinfo/hello-app
to add to your existing Caddy (Podman) config.
## Prerequisites (see docs/SETUP.md for the full walkthrough)
- `qemu-kvm`/`libvirt` installed on the T630 (alongside its existing services)
- Forgejo instance reachable, with this repo pushed to it and Actions enabled
- `tofu`, `kubectl`, `flux` CLI — run on the T630 itself as the unprivileged
`k8s` user (simplest, since the VMs' network is only reachable from there)
## Milestones
1. KVM/libvirt installed on the T630, `k8s` user created
2. `terraform apply` — 3 VMs come up, k3s cluster forms
3. `flux bootstrap` against this Forgejo repo
4. Push `apps/podinfo` — watch Flux deploy it with no manual `kubectl apply`
5. Wire `.forgejo/workflows/build-hello-app.yml` — push a code change to
`apps/hello-app/src`, watch CI build → registry → Flux auto-deploy
6. Stretch: sealed-secrets/SOPS, kube-prometheus-stack, cert-manager

56
apps/hello-app/deployment.yaml Normal file
View file

@ -0,0 +1,56 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-app
namespace: hello-app
spec:
replicas: 1
selector:
matchLabels:
app: hello-app
template:
metadata:
labels:
app: hello-app
spec:
# Uncomment if the hello-app package is set to private in Forgejo
# (see docs/SETUP.md step 9).
# imagePullSecrets:
# - name: forgejo-registry
containers:
- name: hello-app
# {"$imagepolicy": "hello-app:hello-app"}
image: git.boglabob.com/codegit/hello-app:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: hello-app
namespace: hello-app
spec:
selector:
app: hello-app
ports:
- port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-app
namespace: hello-app
spec:
ingressClassName: traefik
rules:
- host: hello.boglabob.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello-app
port:
number: 80

57
apps/hello-app/image-automation.yaml Normal file
View file

@ -0,0 +1,57 @@
# Flux image automation: watches the Forgejo registry for new hello-app
# tags, and rewrites deployment.yaml's image tag + commits back to this repo
# when one shows up. Requires clusters/homelab/flux-system to have write
# access to the Forgejo repo (flux bootstrap sets this up).
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
name: hello-app
namespace: hello-app
spec:
image: git.boglabob.com/codegit/hello-app
interval: 1m
# Uncomment + create the secret if the package is private.
# secretRef:
# name: forgejo-registry
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
name: hello-app
namespace: hello-app
spec:
imageRepositoryRef:
name: hello-app
policy:
numerical:
order: asc
filterTags:
pattern: '^main-[a-f0-9]+-(?P<ts>\d+)$'
extract: '$ts'
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: hello-app
namespace: flux-system
spec:
interval: 1m
sourceRef:
kind: GitRepository
name: flux-system
git:
checkout:
ref:
branch: main
commit:
author:
email: flux@boglabob.com
name: fluxcdbot
messageTemplate: |
chore(hello-app): auto-update image to {{range .Updated.Images}}{{println .}}{{end}}
push:
branch: main
update:
path: ./apps/hello-app
strategy: Setters

6
apps/hello-app/kustomization.yaml Normal file
View file

@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- deployment.yaml
- image-automation.yaml

4
apps/hello-app/namespace.yaml Normal file
View file

@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: hello-app

2
apps/hello-app/src/Dockerfile Normal file
View file

@ -0,0 +1,2 @@
FROM nginx:1.27-alpine
COPY index.html /usr/share/nginx/html/index.html

155
apps/hello-app/src/index.html Normal file
View file

@ -0,0 +1,155 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Homelab GitOps Platform</title>
<style>
:root {
--bg: #0b0e14;
--panel: #131722;
--border: #232838;
--text: #e6e9ef;
--muted: #8b93a7;
--accent: #5eead4;
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--bg);
color: var(--text);
font-family: -apple-system, "Segoe UI", Roboto, sans-serif;
line-height: 1.5;
}
main {
max-width: 780px;
margin: 0 auto;
padding: 4rem 1.5rem 3rem;
}
h1 {
font-size: 1.9rem;
margin: 0 0 0.4rem;
}
.tagline {
color: var(--muted);
margin: 0 0 2rem;
}
.badge-row {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
margin-bottom: 2.5rem;
}
.badge {
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 0.8rem;
background: var(--panel);
border: 1px solid var(--border);
border-radius: 999px;
padding: 0.3rem 0.8rem;
color: var(--muted);
}
.badge strong { color: var(--accent); font-weight: 600; }
section {
background: var(--panel);
border: 1px solid var(--border);
border-radius: 12px;
padding: 1.5rem 1.75rem;
margin-bottom: 1.25rem;
}
section h2 {
font-size: 0.85rem;
text-transform: uppercase;
letter-spacing: 0.06em;
color: var(--muted);
margin: 0 0 1rem;
}
.flow {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: 0.4rem;
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 0.85rem;
}
.flow .step {
background: var(--bg);
border: 1px solid var(--border);
border-radius: 6px;
padding: 0.4rem 0.7rem;
}
.flow .arrow { color: var(--muted); }
.stack {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
}
.stack span {
background: var(--bg);
border: 1px solid var(--border);
border-radius: 6px;
padding: 0.35rem 0.7rem;
font-size: 0.85rem;
}
footer {
color: var(--muted);
font-size: 0.85rem;
margin-top: 2rem;
}
footer a { color: var(--accent); }
</style>
</head>
<body>
<main>
<h1>Homelab GitOps Platform</h1>
<p class="tagline">Terraform-provisioned k3s cluster on a Dell T630, deployed and kept in sync entirely through GitOps. This page is served from it.</p>
<div class="badge-row">
<span class="badge">commit <strong>__GIT_SHA__</strong></span>
<span class="badge">deployed <strong>__BUILD_TIME__</strong> UTC</span>
<span class="badge">served by <strong>k3s</strong></span>
</div>
<section>
<h2>How this page got here</h2>
<div class="flow">
<span class="step">git push</span><span class="arrow">&rarr;</span>
<span class="step">Forgejo Actions</span><span class="arrow">&rarr;</span>
<span class="step">kaniko build</span><span class="arrow">&rarr;</span>
<span class="step">Forgejo registry</span><span class="arrow">&rarr;</span>
<span class="step">Flux image automation</span><span class="arrow">&rarr;</span>
<span class="step">Git commit</span><span class="arrow">&rarr;</span>
<span class="step">Flux reconcile</span><span class="arrow">&rarr;</span>
<span class="step">k3s rollout</span>
</div>
</section>
<section>
<h2>Stack</h2>
<div class="stack">
<span>OpenTofu / Terraform</span>
<span>KVM / libvirt</span>
<span>k3s</span>
<span>Flux CD</span>
<span>Forgejo + Forgejo Actions</span>
<span>kaniko (rootless builds)</span>
<span>Podman</span>
<span>Caddy</span>
</div>
</section>
<section>
<h2>What's not shown here</h2>
<p style="margin:0; color: var(--muted);">
The Kubernetes Dashboard and the cluster's API server are deliberately
not public — both are cluster-admin-capable and stay LAN/VPN-only.
This page and its GitOps pipeline are the intentionally public part.
</p>
</section>
<footer>
Source: <a href="https://git.boglabob.com">git.boglabob.com</a>
</footer>
</main>
</body>
</html>

22
apps/kubernetes-dashboard/helmrelease.yaml Normal file
View file

@ -0,0 +1,22 @@
# Deliberately no ingress here — the official Dashboard grants whatever the
# logged-in identity can do, and a leaked cluster-admin token to a publicly
# reachable Dashboard is a well-known real-world breach vector (see e.g. the
# 2018 Tesla incident). Access is via `kubectl port-forward` only
# (docs/SETUP.md step 12) — no standing hostname, no attack surface between
# sessions.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
interval: 10m
chart:
spec:
chart: kubernetes-dashboard
version: ">=7.0.0"
sourceRef:
kind: HelmRepository
name: kubernetes-dashboard
namespace: kubernetes-dashboard
values: {}

8
apps/kubernetes-dashboard/helmrepository.yaml Normal file
View file

@ -0,0 +1,8 @@
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
interval: 1h
url: https://kubernetes.github.io/dashboard/

7
apps/kubernetes-dashboard/kustomization.yaml Normal file
View file

@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- rbac.yaml
- helmrepository.yaml
- helmrelease.yaml

4
apps/kubernetes-dashboard/namespace.yaml Normal file
View file

@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard

32
apps/kubernetes-dashboard/rbac.yaml Normal file
View file

@ -0,0 +1,32 @@
# Cluster-admin ServiceAccount + long-lived token, used both to log in to
# the Dashboard (docs/SETUP.md step 12) and as the client identity for
# direct LAN kubectl access if you'd rather not manage the SSH-fetched
# kubeconfig from step 6. Fine for a single-user homelab; split into
# narrower-scoped accounts if more people get access later.
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
name: admin-user-token
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: admin-user
type: kubernetes.io/service-account-token

25
apps/podinfo/helmrelease.yaml Normal file
View file

@ -0,0 +1,25 @@
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: podinfo
namespace: podinfo
spec:
interval: 10m
chart:
spec:
chart: podinfo
version: ">=6.0.0"
sourceRef:
kind: HelmRepository
name: podinfo
namespace: podinfo
values:
replicaCount: 1
ingress:
enabled: true
className: traefik
hosts:
- host: podinfo.boglabob.com
paths:
- path: /
pathType: ImplementationSpecific

8
apps/podinfo/helmrepository.yaml Normal file
View file

@ -0,0 +1,8 @@
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: podinfo
namespace: podinfo
spec:
interval: 1h
url: https://stefanprodan.github.io/podinfo

6
apps/podinfo/kustomization.yaml Normal file
View file

@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- helmrepository.yaml
- helmrelease.yaml

4
apps/podinfo/namespace.yaml Normal file
View file

@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: podinfo

42
clusters/homelab/apps.yaml Normal file
View file

@ -0,0 +1,42 @@
# flux bootstrap points the flux-system Kustomization at ./clusters/homelab
# and applies everything under it (prune: true), so these Kustomization
# objects are how the rest of the repo gets pulled in.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: podinfo
namespace: flux-system
spec:
interval: 10m
path: ./apps/podinfo
prune: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: hello-app
namespace: flux-system
spec:
interval: 10m
path: ./apps/hello-app
prune: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: kubernetes-dashboard
namespace: flux-system
spec:
interval: 10m
path: ./apps/kubernetes-dashboard
prune: true
sourceRef:
kind: GitRepository
name: flux-system

3
clusters/homelab/flux-system/.gitkeep Normal file
View file

@ -0,0 +1,3 @@
This directory is populated by `flux bootstrap` (docs/SETUP.md step 5) —
it will contain gotk-components.yaml, gotk-sync.yaml, and kustomization.yaml.
Do not hand-edit those files; re-run `flux bootstrap` to change them.

30
docs/Caddyfile.example Normal file
View file

@ -0,0 +1,30 @@
# Add these blocks to the Caddyfile your Podman Caddy container already
# loads (alongside the existing git.boglabob.com block for Forgejo).
# Replace the node IPs with your actual k3s node IPs from `tofu output
# node_ips` (10.20.30.0/24 by default).
#
# These IPs are on the private libvirt network the VMs live on (see
# terraform/variables.tf's network_cidr), only directly routable from the
# T630 itself — but since Caddy also runs on the T630 (Podman), it reaches
# them the same way any other process on the host would: no extra network
# config needed here.
#
# podinfo/hello sit behind Traefik's ServiceLB, which listens on port 80 of
# every k3s node and routes by the Host header — so Caddy just needs to
# forward the request (Host header included) to any node. Listing all three
# gives you free load-balancing/failover across nodes.
#
# Deliberately NOT here: the Kubernetes Dashboard and the k3s API server.
# Both grant cluster-admin-level control, and routing either through a
# public-facing reverse proxy is the exact pattern behind real-world
# cluster breaches (e.g. Tesla, 2018 — an exposed, unauthenticated
# Dashboard). Both stay LAN-only / on-demand instead — see docs/SETUP.md
# steps 12-13.
podinfo.boglabob.com {
reverse_proxy http://10.20.30.11:80 http://10.20.30.12:80 http://10.20.30.13:80
}
hello.boglabob.com {
reverse_proxy http://10.20.30.11:80 http://10.20.30.12:80 http://10.20.30.13:80
}

339
docs/SETUP.md Normal file
View file

@ -0,0 +1,339 @@
# Setup walkthrough
Assumes: the T630 is an existing Debian box already running other
self-hosted services — this project installs alongside those as ordinary
packages (`qemu-kvm`/`libvirt`), not a hypervisor OS replacing Debian, and
is sized deliberately small (3 VMs, 2 vCPU/2GB RAM each = 6 vCPU/6GB total)
so it stays a demo rather than competing with what's already running. Forgejo
is already running and reachable at `https://git.boglabob.com`, and you can
point DNS records under `boglabob.com` at hosts on your network (directly,
or via whatever reverse proxy/tunnel already gets `git.boglabob.com` there).
---
## 1. Install KVM/libvirt on the T630
Ordinary packages, no reboot into an installer, nothing else on the box is
touched:
```sh
# on the T630
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst
# confirm hardware virtualization is available (T630's Xeons support it)
sudo kvm-ok
```
## 2. Create the unprivileged 'k8s' user
One dedicated, no-sudo user for everything this project touches: driving
Terraform/kubectl/flux against libvirt here, and running the Forgejo Actions
runner later (step 9). It needs group membership to talk to libvirt — that's
a one-time root action; nothing it does afterwards needs `sudo`.
```sh
sudo useradd -m -s /bin/bash k8s # one-time, needs root to create the user itself
sudo usermod -aG libvirt,kvm k8s
sudo loginctl enable-linger k8s # lets its services keep running after logout
# as k8s, from here on (sudo -iu, not su -, since k8s has no password set):
sudo -iu k8s
ssh-keygen -t ed25519 -C "k3s-homelab" -f ~/.ssh/k3s_homelab # only needed if you'll SSH in as k8s day-to-day
virsh -c qemu:///system list --all # sanity check: should run with no permission error, no sudo
```
Do the rest of this guide logged in as `k8s` on the T630 itself (`ssh
k8s@t630`) — node IPs (step 5) live on a private libvirt network that's only
directly reachable from the T630, so this is the simplest place to run
`tofu`/`kubectl`/`flux` from. (If you'd rather drive Terraform from your own
workstation instead, see the `libvirt_uri` comment in
`terraform/terraform.tfvars.example` — you'll then need an SSH tunnel for
kubectl/flux to reach node IPs.)
## 3. Generate the secrets Terraform needs
```sh
openssl rand -hex 32 # -> k3s_token
```
## 4. Push this repo to Forgejo
Doing this before provisioning (rather than after) means `k8s` can get the
repo with a plain `git clone` in step 5, instead of needing a one-off copy
handed to it — and any future Terraform change just needs a `git pull`.
Repo/owner used throughout this guide: `codegit/cloud-demo` (already baked
into `apps/hello-app/deployment.yaml` and `image-automation.yaml`'s image
references — no placeholder-swapping needed).
1. On Forgejo (`https://git.boglabob.com`), as `codegit`: **+ → New
Repository** → name `cloud-demo`. Leave it empty — don't initialize with
a README/`.gitignore`/license, since this repo already has its own.
Visibility (public/private) is your call; either works, since access for
`k8s`/Flux/CI goes through the deploy key below regardless.
2. Locally, wherever you're editing this repo (`maq`):
```sh
git init # if not already
git add .
git commit -m "initial scaffold"
git remote add origin https://git.boglabob.com/codegit/cloud-demo.git
git push -u origin main
```
3. Add `k8s`'s public key (`~/.ssh/k3s_homelab.pub` from step 2, on the
T630 — `cat ~/.ssh/k3s_homelab.pub` as `k8s` if you need to grab it
again) as a Deploy Key on this repo — `Settings → Deploy Keys` — **with
write access**. One key, added once, covers `k8s`'s `git clone`/`pull`
(step 5) *and* `flux bootstrap` (step 7) *and*
`ImageUpdateAutomation`'s commits back (step 10), since all three reuse
this same keypair.
## 5. Provision the VMs with OpenTofu
Installing the package needs sudo, so that part is you (`maq`), not `k8s`.
Installing system-wide (`/usr/local/bin`) means `k8s` can just use `tofu`
afterwards with no further root involvement:
```sh
# as maq (has sudo)
sudo apt install -y unzip
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
sudo sh install-opentofu.sh --install-method standalone && rm install-opentofu.sh
```
Everything from here on is `k8s` again (`sudo -iu k8s`), no sudo involved —
clone using the deploy key from step 4:
```sh
# as k8s
GIT_SSH_COMMAND="ssh -i ~/.ssh/k3s_homelab" \
git clone git@git.boglabob.com:codegit/cloud-demo.git ~/k3s
cd ~/k3s/terraform
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: ssh_public_key (contents of ~/.ssh/k3s_homelab.pub
# from step 2), k3s_token. Defaults for network/sizing are fine to start.
tofu init
tofu plan
tofu apply
```
For any later change to `terraform/`: edit and push as `maq` as usual, then
`cd ~/k3s && git pull` as `k8s` before re-running `tofu plan`/`apply`.
This brings up `k3s-server-1`, `k3s-agent-1`, `k3s-agent-2` on the
`k3s-homelab` libvirt network (`10.20.30.0/24` by default — isolated from
anything else already using libvirt on this box). Cloud-init installs k3s on
each on first boot — give it ~2 minutes after `apply` finishes.
## 6. Get kubectl talking to the cluster
```sh
ssh k3s@$(tofu output -raw server_ip) sudo cat /etc/rancher/k3s/k3s.yaml \
| sed "s/127.0.0.1/$(tofu output -raw server_ip)/" > ~/.kube/config-homelab
export KUBECONFIG=~/.kube/config-homelab
kubectl get nodes # expect 3 Ready nodes
```
This works as-is because you're running it on the T630, which can reach the
`10.20.30.0/24` network directly. To also use kubectl from your own laptop,
either `scp` this kubeconfig over and open an SSH tunnel first
(`ssh -L 6443:10.20.30.11:6443 k8s@t630`, then point the kubeconfig's
`server:` at `https://127.0.0.1:6443`), or just SSH into the T630 as `k8s`
whenever you need kubectl — simplest by far for a project this size.
## 7. Bootstrap Flux against Forgejo
Forgejo isn't a Flux-native provider (unlike GitHub/GitLab), so use the
generic git bootstrap, reusing `k8s`'s keypair — already added as a write
deploy key in step 4, so unlike a default `flux bootstrap` run, this one
won't print a new key to add:
```sh
brew install fluxcd/tap/flux
flux check --pre --kubeconfig ~/.kube/config-homelab
flux bootstrap git \
--url=ssh://git@git.boglabob.com:22/codegit/cloud-demo.git \
--branch=main \
--path=clusters/homelab \
--private-key-file=~/.ssh/k3s_homelab \
--kubeconfig ~/.kube/config-homelab
```
This populates `clusters/homelab/flux-system/` and, because
`clusters/homelab/apps.yaml` already declares `Kustomization` objects for
`apps/podinfo` and `apps/hello-app`, both start reconciling immediately.
## 8. Verify the podinfo GitOps loop
```sh
flux get kustomizations --watch
kubectl -n podinfo get pods
```
Once it's `Ready`, point DNS at it and check in a browser (see step 11).
## 9. Enable Forgejo Actions and register a runner (rootless, no sudo)
Forgejo Actions needs a self-hosted runner — there's no shared runner pool.
The runner normally gets root-equivalent power over its host by mounting
`/var/run/docker.sock` (anyone who can push a workflow file effectively gets
root there). Instead: it runs as the same unprivileged `k8s` user from
step 2, using rootless Podman's own socket instead of Docker's — no root
anywhere in this pipeline. `build-hello-app.yml` already builds images with
kaniko, which needs no daemon and no elevated privileges at all.
The runner's job containers (kaniko, opentofu) never get the libvirt socket
or `k8s`'s home directory mounted in — only the Podman socket, needed to
launch those job containers in the first place — so a compromised workflow
can spawn containers as `k8s`, but can't directly touch the VMs or
Terraform state. Worth knowing given the runner lives on the same box/user
as the cluster's own infrastructure; fine for a demo-sized project, but if
this ever handles anything sensitive, move the runner to its own user or
VM so a breakout doesn't share a blast radius with the cluster.
1. As `k8s` (`ssh k8s@t630`), enable the rootless Podman API socket:
```sh
systemctl --user enable --now podman.socket
echo $XDG_RUNTIME_DIR # note this path, e.g. /run/user/1001
```
2. Instance admin: `Site Administration → Actions → Runners`, confirm
Actions is enabled.
3. Repo: `Settings → Actions → Runners → Create new runner`, copy the
registration token.
4. Register and run the runner as a rootless Podman container, pointed at
the Podman socket from step 1 instead of docker.sock:
```sh
# still as k8s
podman volume create forgejo-runner-data
podman run -d --name forgejo-runner --restart unless-stopped \
-e DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" \
-v "$XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock" \
-v forgejo-runner-data:/data \
code.forgejo.org/forgejo/runner:6 \
forgejo-runner register --no-interactive \
--instance https://git.boglabob.com \
--token <TOKEN_FROM_STEP_3> --labels docker:docker://node:20-bookworm
```
The registered runner picks up both workflows in `.forgejo/workflows/` —
`terraform.yml`'s `container:` image and `build-hello-app.yml`'s kaniko
image are both launched through that same rootless Podman socket.
5. Repo `Settings → Secrets and Variables → Actions`, add:
- Secret `FORGEJO_TOKEN` — a personal access token (`Settings → Applications`
on your Forgejo user, scope `package:write`) used to push images.
- Variable `FORGEJO_USER`, `FORGEJO_ORG` — your Forgejo username/org.
If the `hello-app` package ends up private (Forgejo package visibility
follows repo visibility by default), create the cluster-side pull secret and
uncomment the `imagePullSecrets` line in `apps/hello-app/deployment.yaml`:
```sh
kubectl -n hello-app create secret docker-registry forgejo-registry \
--docker-server=git.boglabob.com \
--docker-username=<FORGEJO_USER> \
--docker-password=<FORGEJO_TOKEN>
```
## 10. Exercise the full loop
```sh
sed -i 's/This page is served from it\./This page is served from it — and this line proves it: edited via git push./' apps/hello-app/src/index.html
git add apps/hello-app/src/index.html
git commit -m "test the pipeline"
git push
```
Watch: `build-hello-app` runs in Forgejo Actions (which also stamps the page
with the current commit SHA and build time — see `apps/hello-app/src/index.html`)
→ pushes a new tag to `git.boglabob.com/codegit/hello-app` → Flux's
`ImageRepository` picks it up within a minute → `ImageUpdateAutomation`
commits the new tag back to `apps/hello-app/deployment.yaml` → the
`hello-app` Kustomization reconciles → `kubectl -n hello-app get pods` shows
a new pod, and `https://hello.boglabob.com` shows the new commit SHA/badge.
## 11. Expose the apps through Caddy
Since Caddy (Podman) is already the front door for `git.boglabob.com`, route
`podinfo` and `hello-app` through it too — but not the Dashboard or the API
server; see steps 12-13 for why.
1. Add A records for `podinfo.boglabob.com` and `hello.boglabob.com`
pointing wherever `git.boglabob.com` already points (Caddy's host).
2. Add the blocks from `docs/Caddyfile.example` to Caddy's config, filling
in your real node IPs (`tofu output node_ips`), and reload:
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
3. Check:
```sh
curl https://podinfo.boglabob.com/
curl https://hello.boglabob.com/
```
## 12. Access the Dashboard (LAN-only, on demand)
The Dashboard grants whatever its logged-in identity can do — with the
`admin-user` token from `apps/kubernetes-dashboard/rbac.yaml`, that's
cluster-admin. Publicly exposing that (even behind a login page) is the
exact pattern behind real breaches (Tesla, 2018: an internet-reachable,
unauthenticated Dashboard). So: no ingress, no standing hostname — only a
port-forward you open when you need it and close when you don't:
```sh
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
```
Then open `https://localhost:8443` and log in with the token:
```sh
kubectl -n kubernetes-dashboard get secret admin-user-token -o jsonpath='{.data.token}' | base64 -d
```
## 13. kubectl access from elsewhere on the LAN, or remotely
Best practice for the Kubernetes API server is the same principle as the
Dashboard: never put 6443 on the public internet if you can avoid it,
because a leaked credential there is a full cluster compromise.
Node IPs (`10.20.30.0/24` by default) live on the private libvirt network
from step 5 — only the T630 itself can reach them directly, which is
actually a nice side effect: even the rest of your LAN can't touch the API
server without going through the T630 first. Two ways to do that:
- **SSH into the T630 as `k8s`** and run kubectl there directly (same as
step 6) — simplest, and what this whole guide assumes by default.
- **Tunnel from another machine** (your laptop, or a phone via Termux, etc.):
```sh
ssh -L 6443:$(tofu output -raw server_ip):6443 k8s@t630
```
then point a local kubeconfig's `server:` at `https://127.0.0.1:6443`
(copy the kubeconfig from step 6 and edit that one field). The cert
validates because `k8s-api.boglabob.com` is in the server's TLS SAN list
(`terraform/variables.tf`'s `k8s_api_hostname`) — add it to
`/etc/hosts` as `127.0.0.1 k8s-api.boglabob.com` on whatever machine
you're tunneling from and use that as the `server:` host instead of the
raw IP, so the hostname in the URL matches a name the cert actually
covers.
- **From outside your home network entirely**: Tailscale or WireGuard on
the T630, then the SSH tunnel above over the Tailscale/WireGuard link
instead of the open internet. Reasonable next stretch goal once the core
loop is working — don't port-forward 22 or 6443 on your router for this.
The `admin-user` bearer token (step 12) also works for kubectl over the same
tunnel, if you'd rather not manage the client-cert kubeconfig.
## Stretch goals, roughly in order
- **Remote access**: Tailscale or WireGuard on the k3s server node, for
kubectl/Dashboard access from outside the LAN without opening anything
publicly (step 13).
- **TLS**: `cert-manager` + a `ClusterIssuer` for Let's Encrypt (DNS-01 if
`boglabob.com` isn't publicly reachable on 80/443).
- **Secrets in Git**: `sops` + `age`, or `sealed-secrets`, so the
`K3S_TOKEN`/API tokens above don't need to live only in Forgejo's secret
store.
- **Monitoring**: `kube-prometheus-stack` via Helm, deployed the same way as
podinfo (HelmRepository + HelmRelease under `apps/`).
- **HA**: add a second k3s server node and switch from SQLite to embedded
etcd (`--cluster-init` on the first server, `--server` join on the second).

33
terraform/cloud-init/agent.yaml.tpl Normal file
View file

@ -0,0 +1,33 @@
#cloud-config
hostname: ${hostname}
manage_etc_hosts: true
users:
- name: k3s
groups: sudo
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_authorized_keys:
- ${ssh_public_key}
package_update: true
packages:
- curl
write_files:
- path: /usr/local/bin/join-k3s.sh
permissions: '0755'
content: |
#!/bin/sh
# Server may still be booting; retry the join until it answers.
until curl -sk https://${server_ip}:6443/ping >/dev/null 2>&1; do
echo "waiting for k3s server at ${server_ip}..."
sleep 5
done
curl -sfL https://get.k3s.io | \
K3S_URL="https://${server_ip}:6443" \
K3S_TOKEN="${k3s_token}" \
sh -s - agent
runcmd:
- /usr/local/bin/join-k3s.sh

8
terraform/cloud-init/network-config.yaml.tpl Normal file
View file

@ -0,0 +1,8 @@
version: 2
ethernets:
eth0:
addresses:
- ${ip}/${prefix_length}
gateway4: ${gateway}
nameservers:
addresses: [${gateway}]

30
terraform/cloud-init/server.yaml.tpl Normal file
View file

@ -0,0 +1,30 @@
#cloud-config
hostname: ${hostname}
manage_etc_hosts: true
users:
- name: k3s
groups: sudo
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_authorized_keys:
- ${ssh_public_key}
package_update: true
packages:
- curl
write_files:
- path: /etc/rancher/k3s/config.yaml
permissions: '0600'
content: |
token: "${k3s_token}"
tls-san:
- "${hostname}"
- "${k8s_api_hostname}"
runcmd:
- curl -sfL https://get.k3s.io | sh -s - server
- mkdir -p /home/k3s/.kube
- k3s kubectl config view --raw > /home/k3s/.kube/config
- chown -R k3s:k3s /home/k3s/.kube

95
terraform/main.tf Normal file
View file

@ -0,0 +1,95 @@
locals {
server_node = [for name, n in var.nodes : n if n.role == "server"][0]
server_ip = local.server_node.ip
prefix_length = split("/", var.network_cidr)[1]
}
resource "libvirt_pool" "k3s" {
name = var.storage_pool
type = "dir"
path = var.storage_pool_path
}
resource "libvirt_volume" "base" {
name = "k3s-base.qcow2"
pool = libvirt_pool.k3s.name
source = var.base_image_url
format = "qcow2"
}
# A dedicated, isolated NAT network so this project can't collide with
# anything else already using the host's default libvirt network. DHCP is
# off — every node gets a static IP via cloud-init instead.
resource "libvirt_network" "k3s" {
name = "k3s-homelab"
mode = "nat"
domain = "k3s.local"
addresses = [var.network_cidr]
dhcp {
enabled = false
}
dns {
enabled = true
}
}
resource "libvirt_volume" "node" {
for_each = var.nodes
name = "${each.key}.qcow2"
pool = libvirt_pool.k3s.name
base_volume_id = libvirt_volume.base.id
size = each.value.disk_gb * 1024 * 1024 * 1024
format = "qcow2"
}
resource "libvirt_cloudinit_disk" "node" {
for_each = var.nodes
name = "${each.key}-cloudinit.iso"
pool = libvirt_pool.k3s.name
user_data = templatefile("${path.module}/cloud-init/${each.value.role}.yaml.tpl", {
hostname = each.key
ssh_public_key = var.ssh_public_key
k3s_token = var.k3s_token
server_ip = local.server_ip
k8s_api_hostname = var.k8s_api_hostname
})
network_config = templatefile("${path.module}/cloud-init/network-config.yaml.tpl", {
ip = each.value.ip
prefix_length = local.prefix_length
gateway = var.gateway_ip
})
}
resource "libvirt_domain" "node" {
for_each = var.nodes
name = each.key
vcpu = each.value.vcpu
memory = each.value.memory
cloudinit = libvirt_cloudinit_disk.node[each.key].id
network_interface {
network_id = libvirt_network.k3s.id
wait_for_lease = false
}
disk {
volume_id = libvirt_volume.node[each.key].id
}
# Serial console only, no display — this box doesn't need a GUI hop for
# a couple of small demo VMs.
console {
type = "pty"
target_type = "serial"
target_port = "0"
}
# Terraform brings all VMs up in parallel; the agent cloud-init script
# (cloud-init/agent.yaml.tpl) retries the join until the server's API is
# reachable, so node boot order doesn't matter.
}

8
terraform/outputs.tf Normal file
View file

@ -0,0 +1,8 @@
output "server_ip" {
description = "IP of the k3s server node — fetch its kubeconfig from here"
value = local.server_ip
}
output "node_ips" {
value = { for name, n in var.nodes : name => n.ip }
}

14
terraform/terraform.tfvars.example Normal file
View file

@ -0,0 +1,14 @@
# Copy to terraform.tfvars and fill in. terraform.tfvars is gitignored —
# never commit real secrets.
# Default (qemu:///system) assumes you're running tofu on the T630 itself
# as the 'k8s' user - see docs/SETUP.md step 2. Leave commented out unless
# you're running Terraform from a separate workstation instead.
# libvirt_uri = "qemu+ssh://k8s@t630.lan/system?keyfile=/home/you/.ssh/k3s_homelab"
ssh_public_key = "ssh-ed25519 AAAA... you@workstation"
k3s_token = "generate-with: openssl rand -hex 32"
# Defaults in variables.tf (network_cidr, gateway_ip, node IPs/sizing) are
# fine as-is for a first apply — override here only if they clash with
# something else on the T630.

68
terraform/variables.tf Normal file
View file

@ -0,0 +1,68 @@
variable "libvirt_uri" {
description = "libvirt connection URI. Default assumes tofu/kubectl/flux all run directly on the T630 as the 'k8s' user (see docs/SETUP.md step 2) - simplest option, since node IPs (network_cidr) are only directly reachable from the T630 itself. Use qemu+ssh://k8s@t630.lan/system?keyfile=... instead if you'd rather run Terraform from a separate workstation (you'll then need an SSH tunnel for kubectl/flux to reach node IPs - see step 6)."
type = string
default = "qemu:///system"
}
variable "base_image_url" {
description = "Cloud image libvirt clones for every node's disk (downloaded once, cached in the pool)"
type = string
default = "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
}
variable "storage_pool" {
description = "Name of the libvirt storage pool this project's disks live in (created if missing)"
type = string
default = "k3s-homelab"
}
variable "storage_pool_path" {
description = "Host filesystem path backing the storage pool"
type = string
default = "/var/lib/libvirt/images/k3s-homelab"
}
variable "network_cidr" {
description = "Subnet for the dedicated NAT network this project's VMs live on (isolated from any other libvirt networks already on the host)"
type = string
default = "10.20.30.0/24"
}
variable "gateway_ip" {
description = "Gateway address within network_cidr (libvirt itself, on the host)"
type = string
default = "10.20.30.1"
}
variable "ssh_public_key" {
description = "Public key injected into each VM via cloud-init for the 'k3s' admin user"
type = string
}
variable "k3s_token" {
description = "Shared cluster token agents use to join the k3s server"
type = string
sensitive = true
}
variable "k8s_api_hostname" {
description = "LAN-only hostname for the k3s API server, added to the server's TLS SAN list so client-cert kubeconfigs validate against it. Resolve it via local DNS only (never a public record) - see docs/SETUP.md step 13."
type = string
default = "k8s-api.boglabob.com"
}
variable "nodes" {
description = "k3s nodes to provision. Sizing is deliberately small (2 vCPU/2GB each = 6GB total) so this stays a demo, not a resource hog, alongside the T630's other services."
type = map(object({
role = string # "server" or "agent"
ip = string # e.g. "10.20.30.11" - must be inside network_cidr
vcpu = number
memory = number # MiB
disk_gb = number
}))
default = {
"k3s-server-1" = { role = "server", ip = "10.20.30.11", vcpu = 2, memory = 2048, disk_gb = 20 }
"k3s-agent-1" = { role = "agent", ip = "10.20.30.12", vcpu = 2, memory = 2048, disk_gb = 20 }
"k3s-agent-2" = { role = "agent", ip = "10.20.30.13", vcpu = 2, memory = 2048, disk_gb = 20 }
}
}

17
terraform/versions.tf Normal file
View file

@ -0,0 +1,17 @@
terraform {
required_version = ">= 1.6.0"
required_providers {
libvirt = {
source = "dmacvicar/libvirt"
version = "~> 0.8"
}
}
}
# qemu+ssh:// so this can be run from your workstation against the T630;
# requires the connecting user to be in the T630's `libvirt` and `kvm`
# groups (see docs/SETUP.md step 2) — no sudo needed after that.
provider "libvirt" {
uri = var.libvirt_uri
}