CodeGit/cloud-demo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: full rewrite of QUICKSTART with real explanations, not just steps

Browse source 
Previous version listed commands with light justification; this
explains the actual mechanism at each stage - qemu:///system vs
session, why pools mediate permissions, COW overlays, what cloud-init's
two data files are for, what each virt-install flag does, what a
kubeconfig actually contains, and (the deepest gap) what flux bootstrap
concretely does under the hood: the controllers/CRDs involved, what
GitRepository and Kustomization objects actually do on their reconcile
loops, why apps.yaml gets picked up automatically, and how the auth
Secret works - with kubectl/flux commands to go verify each claim
against the already-bootstrapped cluster rather than take it on faith.
This commit is contained in:
CodeGit 2026-08-19 10:32:39 +01:00
parent 71415dae21
commit 68c111b969
1 changed files with 407 additions and 117 deletions
Download patch file Download diff file Expand all files Collapse all files

524
docs/QUICKSTART.md
View file

@ -1,98 +1,209 @@
# Quickstart: manual cluster (no Terraform) # Quickstart: manual cluster (no Terraform)
Gets a real k3s cluster running today, by hand, so you can learn/practice A real k3s cluster, built by hand, so Flux/GitOps can be learned right away
Flux and GitOps immediately instead of waiting on the Terraform/libvirt instead of waiting on the Terraform/libvirt provider work in `docs/SETUP.md`
provider work in `docs/SETUP.md` to be sorted out. One throwaway VM, created to get sorted (that provider did a breaking rewrite between 0.8.x and
directly with `virt-install` rather than the `dmacvicar/libvirt` Terraform 0.9.x, and the HCL needs writing against the real schema, not memory).
provider. When the Terraform track is ready, tear this down (last section)
and provision the "real" 3-node cluster properly instead.
Shares steps 1–4 of `docs/SETUP.md` as prerequisites — do those first if you This is one throwaway VM, created directly with `virt-install` — no
haven't: Terraform involved at all. Everything here explains the *why*, not just
- Step 1: KVM/libvirt installed on the T630. the *what*: read each section before running its commands, and where a
- Step 2: the unprivileged `k8s` user exists, with `~/.ssh/id_ed25519` config file is being built (the cloud-init data, the Caddy block), treat
generated. the snippets as pieces to assemble into your own file, not something to
- Step 3: `openssl rand -hex 32` isn't needed here (no agents joining, so no paste wholesale.
cluster token) — skip it.
- Step 4: `cloud-demo` pushed to Forgejo, with the `flux-write` token Shares steps 1–4 of `docs/SETUP.md` as prerequisites:
generated. - Step 1: KVM/libvirt packages installed on the T630.
- Step 2: the unprivileged `k8s` user exists, in the `libvirt`/`kvm` groups,
with an SSH keypair at `~/.ssh/id_ed25519`.
- Step 3: skip — no join token needed here (see "one node is enough" below).
- Step 4: `cloud-demo` pushed to Forgejo, with a `k8s-readonly` and a
`flux-write` token generated.
Everything below runs as `k8s` on the T630 (`sudo -iu k8s`). Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).
## 1. Create the VM ---
No sudo needed anywhere in this guide — `k8s`'s `libvirt`/`kvm` group ## Why one node is enough
membership (step 2) is what authorizes talking to libvirt at all, via
polkit. But group membership alone doesn't solve everything: system libvirt
runs actual VMs as a separate, restricted `libvirt-qemu` user, not as
`k8s` — so a disk image sitting under `k8s`'s home directory (mode 700 by
default) would fail at boot, since that user can't read into `k8s`'s home
at all. The fix is to let libvirt manage the disk storage itself, inside
its own pool, rather than pointing at a raw path in `~` — pool operations
go through libvirtd's API, so it's libvirtd (already running with the
right privileges) that handles the file ownership, not `k8s` directly.
Every `virsh`/`virt-install` command below uses `-c qemu:///system` / A k3s **server** node runs the control plane (API server, scheduler,
`--connect qemu:///system` explicitly, rather than relying on the etcd/SQLite) *and* schedules ordinary workloads onto itself unless you
`LIBVIRT_DEFAULT_URI` env var — for a non-root user that env var is what explicitly disable that. So a single server, with no agents, is already a
picks system vs. the per-user `qemu:///session` instance, and it's easy for complete, working cluster — nothing here needs a join token or a second
it to be set in one terminal and not another, silently causing commands to VM. The 3-node design in `docs/SETUP.md` exists to mirror a more
operate on the wrong (session) libvirt instance where nothing you created realistic multi-node cluster once Terraform can build it repeatedly and
system-side is visible. Explicit `-c` on every command avoids that disposably; for learning Flux and GitOps, that extra shape doesn't buy
ambiguity entirely. you anything yet.
**Debian's `libvirt-daemon-system` does *not* auto-create a `default` ## Two libvirt connections, and why it matters
storage pool** the way some other distros' packaging does (it does
auto-create the `default` *network*, just left inactive) — check and fix libvirt isn't one daemon with one namespace of VMs — from a client's
both before doing anything else: perspective there are (at least) two separate connections:
- `qemu:///system` — the shared, host-wide instance. VMs here can use
privileged networking (bridges, NAT with DHCP), and management access is
gated by group membership (`libvirt`/`kvm`) checked via polkit — which is
exactly what step 2's `usermod -aG libvirt,kvm k8s` set up. No sudo
needed for any command below; that group membership *is* the
authorization.
- `qemu:///session` — a private, per-user instance with no special
privileges, and critically, its own separate storage pools and networks
that don't overlap with the system instance at all.
For a non-root user, `virsh`/`virt-install` **default to `session`** unless
told otherwise. This matters a lot in practice: it's easy to set
`LIBVIRT_DEFAULT_URI=qemu:///system` in one terminal, run a command in a
different terminal where it isn't set, and have that command silently
create something under `session` instead — where it's invisible to
everything else you're doing. Every command below uses `-c qemu:///system`
/ `--connect qemu:///system` explicitly for exactly this reason, rather
than relying on the environment variable.
## Storage: why a raw path in `~` doesn't work
The most natural first instinct is to put a VM's disk file somewhere in
`k8s`'s home directory and point `--disk` at it directly. That fails
non-obviously: under `qemu:///system`, the actual QEMU process backing a
VM doesn't run as `k8s` — it runs as a separate, restricted `libvirt-qemu`
user (a deliberate security boundary, so a compromised VM process has its
own limited identity rather than the identity of whoever created it).
`k8s`'s home directory defaults to mode `700` — readable only by `k8s` —
so `libvirt-qemu` can't read into it at all, and the VM fails at boot.
The fix is to let libvirt manage the storage itself, in a **pool**. A pool
is just a named, libvirt-tracked location for disk images (a directory, in
the simplest case). The key property: creating or writing a volume inside
a pool goes through libvirtd's API, not through `k8s`'s own filesystem
permissions — so it's libvirtd (already running with the right privileges)
that handles ownership correctly, regardless of what user asked for it.
Check what pools/networks already exist:
```sh ```sh
virsh -c qemu:///system pool-list --all virsh -c qemu:///system pool-list --all
virsh -c qemu:///system net-list --all virsh -c qemu:///system net-list --all
``` ```
If `pool-list` comes back empty: On this box, `net-list` showed a `default` network already defined (just
inactive), but `pool-list` came back completely empty — **Debian's
`libvirt-daemon-system` package does not auto-create a `default` storage
pool**, unlike some other distros' packaging. Both need fixing before
anything else:
```sh ```sh
# only if pool-list was empty
virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default virsh -c qemu:///system pool-autostart default
```
If `net-list` shows `default` as `inactive`: # only if net-list showed 'default' as inactive
```sh
virsh -c qemu:///system net-start default virsh -c qemu:///system net-start default
virsh -c qemu:///system net-autostart default virsh -c qemu:///system net-autostart default
``` ```
Download the cloud image somewhere `k8s` can read it — this location `pool-autostart`/`net-autostart` mean both come back up automatically after
itself doesn't need to be qemu-readable, since it's only ever read by a host reboot — without it, they'd need manually starting again every time.
`virsh` (running as `k8s`), never directly by the VM:
## Building the base image and the VM's own disk
Downloading the cloud image doesn't need to go through the pool — this
copy is only ever read by `virsh` itself (running as `k8s`), never
directly by the VM, so an ordinary temp location is fine:
```sh ```sh
curl -L -o /tmp/noble-base.img \ curl -L -o /tmp/noble-base.img \
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
``` ```
Import it into the `default` pool, then create a copy-on-write overlay on Getting it *into* the pool, though, does need to go through libvirt's API
top of it, sized up to 20G: — `vol-create-as` allocates an empty volume of a given size inside the
pool, and `vol-upload` streams a local file's bytes into it:
```sh ```sh
virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \ virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
--capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2 --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img rm /tmp/noble-base.img
```
The VM itself shouldn't boot directly off this base image — if it did,
every write the OS makes (logs, package installs, k3s's own state) would
permanently modify the one shared base file, corrupting it for any future
VM built from the same base. Instead, create a **copy-on-write overlay**:
a second volume that starts out empty and only stores the *differences*
from its backing volume. Reads that haven't been changed transparently
fall through to the base image; writes go into the overlay. This is the
same relationship a Docker image layer has to its base layer.
```sh
virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \ virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
--backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2 --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2
virsh -c qemu:///system vol-list --pool default # should list both volumes virsh -c qemu:///system vol-list --pool default # should list both volumes now
``` ```
Write the cloud-init user-data — paste in the contents of ## cloud-init: how a stock image becomes *this* VM
`~/.ssh/id_ed25519.pub` where marked:
The base image is a generic Ubuntu install — it has no idea it's about to
become a k3s node, and has no user account you could log into. **cloud-init**
is the standard mechanism cloud images use to configure themselves on
first boot, driven by data supplied externally rather than baked into the
image. `virt-install`'s `--cloud-init` flag builds a small ISO (the
"NoCloud" datasource) containing that data and attaches it to the VM;
cloud-init, already installed in the image, detects it automatically at
boot.
Two separate pieces of data go in, and they answer different questions:
- **`user-data`** — *what should exist on this machine*: users, packages,
commands to run. Written as `#cloud-config` YAML.
- **`network-config`** — *how should this machine's network be set up*.
This quickstart skips it entirely and lets the VM get an address via
DHCP from the `default` network instead (the Terraform track, by
contrast, uses this for static IPs, since it manages its own isolated
network).
Build `user-data` up piece by piece. Start with identity:
```yaml
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true
```
Then the one thing you actually need to log in and administer this box —
a user, with your public key rather than a password (cloud images have no
default password, and SSH password auth is normally disabled anyway), and
passwordless sudo so you're not stuck typing a password you never set:
```yaml
users:
- name: k3s
groups: sudo
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_authorized_keys:
- <contents of ~/.ssh/id_ed25519.pub>
```
Then the actual payload — install k3s, and stage a copy of its
auto-generated kubeconfig somewhere the `k3s` user can read (by default
it's only readable by root):
```yaml
package_update: true
packages:
- curl
runcmd:
- curl -sfL https://get.k3s.io | sh -s - server
- mkdir -p /home/k3s/.kube
- k3s kubectl config view --raw > /home/k3s/.kube/config
- chown -R k3s:k3s /home/k3s/.kube
```
Assemble those three pieces into one file:
```sh ```sh
mkdir -p ~/vms mkdir -p ~/vms
@ -121,25 +232,41 @@ runcmd:
EOF EOF
``` ```
This is a single, standalone server — no join token, no agents, no custom Unlike the disk, this file is fine sitting under `~/vms` — it's only ever
network (uses libvirt's default NAT network + DHCP). A k3s server node read client-side by `virt-install` (as `k8s`), which hands the resulting
schedules workloads on itself by default, so this alone is a complete, seed data to libvirtd over the API; `libvirt-qemu` never touches it
usable cluster. directly.
The cloud-init user-data file itself is fine living under `~/vms` — unlike ## Creating the VM
the disk, it's only ever read client-side by `virt-install` (as `k8s`),
which then hands the rendered seed data to libvirtd over the API. Note
`$HOME` rather than `~` in the `--cloud-init` argument below — `~` only
expands at the start of a shell word, not after `=` inside one, so
`user-data=~/vms/...` would be passed to virt-install literally with the
tilde still in it.
`--os-variant ubuntu22.04` below is deliberately not `ubuntu24.04` (the Each `virt-install` flag is answering a specific question:
actual image) — `osinfo-db` on Debian 12 may not recognize the newer
variant name yet. Check what yours knows about with | Flag | Answers |
`osinfo-query os | grep -i ubuntu`; the hint mostly just tunes libvirt's |---|---|
domain defaults (virtio devices, clock, etc.), so a close-enough variant | `--connect qemu:///system` | which libvirt instance (see above) |
against a cloud-init `--import` like this is harmless: | `--name` | the domain's name, used everywhere else (`virsh`, `domifaddr`, teardown) |
| `--memory` / `--vcpus` | resource allocation — kept small deliberately |
| `--disk vol=default/k3s-manual.qcow2` | use the pool-managed overlay, not a raw path |
| `--import` | boot the disk as-is rather than running an OS installer against it |
| `--os-variant` | a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed |
| `--network network=default` | attach to the NAT network from earlier |
| `--cloud-init user-data=...` | the file just built |
| `--graphics none` | no VNC/spice display — this is a headless server VM |
| `--noautoconsole` | don't attach to its console interactively after creation |
Two gotchas worth knowing before running this:
- **`$HOME`, not `~`, in the `--cloud-init` argument.** Bash only expands
`~` at the very start of a word; `user-data=~/vms/...` is *inside* a
word (after `=`), so the tilde would be passed through literally and
`virt-install` would fail looking for a file called `~`. `$HOME`
expands regardless of position.
- **`--os-variant` may need to be an older release than the actual image.**
`osinfo-db` (the database `virt-install` validates this against) can lag
behind real Ubuntu releases — if `ubuntu24.04` comes back "unknown", check
what's actually available with `osinfo-query os | grep -i ubuntu` and use
the newest one it recognizes. Since `--import` just boots the disk as-is,
this hint doesn't change what's actually installed.
```sh ```sh
virt-install \ virt-install \
@ -156,16 +283,50 @@ virt-install \
--noautoconsole --noautoconsole
``` ```
## 2. Find its IP and get kubectl talking to it ## Finding the VM and connecting
Give cloud-init ~2 minutes to finish installing k3s after the VM boots.
```sh ```sh
virsh -c qemu:///system domifaddr k3s-manual # note the IP under the default network virsh -c qemu:///system domifaddr k3s-manual
``` ```
If you haven't installed `kubectl` on the T630 yet (as `k8s`, no sudo This prints something like `192.168.122.67/24` — the `/24` is CIDR
needed): notation for the subnet mask (`255.255.255.0`), describing the *network*
this address belongs to, not part of the address itself. Use just the
plain IP (`192.168.122.67`) to actually connect.
Give cloud-init a couple of minutes after `Domain creation completed`
before it's reachable — it's installing k3s in the background.
This address comes from the `default` network's DHCP server, leased
against the VM's MAC address (shown in the same `domifaddr` output). DHCP
leases are "sticky" in practice — the VM will keep asking for and getting
the same address on renewal — but that's not the same as *guaranteed*
fixed. Pin it explicitly if you don't want to risk it changing later:
```sh
virsh -c qemu:///system net-update default add ip-dhcp-host \
"<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config
```
## kubectl: what's actually in a kubeconfig
A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate
(so your client trusts the API server's identity) plus a client
certificate and private key (so the API server trusts yours). k3s
generates one for itself pointed at `127.0.0.1`, which only works from
inside the VM — the `sed` below swaps that for the VM's real address so
it works from the T630 instead:
```sh
mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
| sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual
```
`kubectl` itself isn't installed anywhere yet. Rather than a system
package (which would need `sudo`, which `k8s` doesn't have), grab the
official binary release straight from Kubernetes' own distribution point
and drop it somewhere already on `k8s`'s `PATH`:
```sh ```sh
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
@ -174,20 +335,24 @@ mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/ mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH" export PATH="$HOME/.local/bin:$PATH"
```
```sh
mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
| sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual
export KUBECONFIG=~/.kube/config-manual export KUBECONFIG=~/.kube/config-manual
kubectl get nodes # expect 1 Ready node kubectl get nodes # expect 1 Ready node
``` ```
## 3. Bootstrap Flux against Forgejo ## Flux: what's actually happening, mechanism by mechanism
If you haven't installed the Flux CLI yet (no sudo needed): "Flux" isn't one program watching your repo by magic — it's a handful of
ordinary Kubernetes controllers (just Deployments, like anything else
you'd run on the cluster), each one understanding a couple of **Custom
Resource Definitions** — CRDs extend the Kubernetes API with new object
kinds, the same way `Deployment` or `Service` are built-in kinds.
`GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing
about any of this is special-cased outside the normal Kubernetes API —
it's the exact same "define an object, a controller notices it and acts"
loop that runs the whole rest of Kubernetes.
Install the CLI the same no-sudo, direct-binary-release way as `kubectl`:
```sh ```sh
FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//') FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
@ -197,8 +362,11 @@ rm /tmp/flux.tar.gz
flux --version flux --version
``` ```
Same repo, same token, same target path as the full guide would use — Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this
nothing about this is cluster-specific: uses the generic git bootstrap — over HTTPS with the `flux-write` token
from step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably
reachable from either this desktop or the T630 — see the earlier
troubleshooting in this project's history):
```sh ```sh
flux check --pre --kubeconfig ~/.kube/config-manual flux check --pre --kubeconfig ~/.kube/config-manual
@ -213,33 +381,136 @@ flux bootstrap git \
--kubeconfig ~/.kube/config-manual --kubeconfig ~/.kube/config-manual
``` ```
## 4. Continue with the shared steps That one command did five genuinely separate things. Go look at each —
these commands work against the cluster you already bootstrapped:
From here, `docs/SETUP.md` steps 8–10 and 12–13 apply exactly as written, **1. It installed the controllers** — plain Kubernetes Deployments, no
regardless of how the cluster was created — verifying the podinfo GitOps different in kind from anything else running on the cluster:
loop, registering the Forgejo Actions runner, the Dashboard, remote kubectl
access. Just use `~/.kube/config-manual` as the kubeconfig throughout
instead of the Terraform-provisioned one.
**Step 11 (exposing apps through Caddy) does not apply as-written**,
though — it assumes the Terraform track's static `10.20.30.x` addressing.
This VM is on libvirt's `default` network instead, using DHCP:
```sh ```sh
virsh -c qemu:///system domifaddr k3s-manual # note the IP and MAC address kubectl -n flux-system get deployments
``` ```
DHCP leases are "sticky" to a MAC address in practice, but not guaranteed `source-controller`, `kustomize-controller`, `helm-controller`,
fixed the way the Terraform track's static IPs are — pin it explicitly so `notification-controller`. Alongside them, it registered the CRDs those
it can't change later: controllers understand:
```sh ```sh
virsh -c qemu:///system net-update default add ip-dhcp-host \ kubectl get crds | grep fluxcd
"<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config
``` ```
Then in Caddy's config, use that IP directly rather than **2. It created a `GitRepository` object** — this is the whole "what repo
`docs/Caddyfile.example`'s multi-node placeholder blocks: am I watching" declaration, and nothing more. `source-controller`'s job is
entirely mechanical: every `interval` (default 1m), do a real `git fetch`
against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that
tree into a `.tar.gz`, and record its location in `.status.artifact`.
That's the entire job — it doesn't know or care what's *in* the repo.
```sh
kubectl -n flux-system get gitrepository flux-system -o yaml
```
Look at `.status.conditions` and `.status.artifact` — that's the result of
an actual git fetch that already happened, not a static config.
**3. It created a `Kustomization` object** (confusingly, the same name as
the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. —
related but not identical). `kustomize-controller`'s job, on its own
interval: fetch the artifact `source-controller` produced, run the real
`kustomize` tool against `.spec.path` inside it (the exact same tool a
plain `kustomize build apps/podinfo` would run locally against those
`kustomization.yaml` files), and apply the resulting objects via the
Kubernetes API — the automated equivalent of you running
`kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer,
forever. `prune: true` (set on all the `Kustomization` objects in this
repo) means it also *deletes* anything it previously created that's no
longer present in the current git state — that's what makes it
self-healing rather than just "apply once."
```sh
kubectl -n flux-system get kustomization
```
**This is also why `apps/podinfo`, `apps/hello-app`, and
`apps/kubernetes-dashboard` started deploying without you ever running
`kubectl apply` on them.** Bootstrap's own `Kustomization` watches
`clusters/homelab` with `prune: true`. `clusters/homelab/apps.yaml`
(already sitting in the repo, hand-written earlier in this project) itself
just *defines more `Kustomization` objects*, one per app — so the first
one picks it up as part of its own normal reconcile, creates those three
child `Kustomization`s, and each of *those* then does its own
fetch-and-apply against its own app directory. It's the same mechanism
recursing, not a special case.
**4. It stored your credential as a Kubernetes Secret** — `--password`
here is the `flux-write` token, not an account password. Bootstrap
base64-wraps it into a Secret that matches what the `GitRepository`'s
`.spec.secretRef` points at:
```sh
kubectl -n flux-system get secret flux-system -o yaml
```
The `password` field is base64 (`| base64 -d` to read it) — this is
exactly the same credential `git clone https://user:token@host/repo` would
use, just read by `source-controller` on every fetch instead of typed by
you once. It's never written to `k8s`'s own filesystem.
**5. It committed that config back into the repo itself** — `git pull` in
your desktop clone and look at `clusters/homelab/flux-system/`. The
`GitRepository`/`Kustomization` objects you just inspected live in the
cluster *because* those exact YAML files are committed there — so
rebuilding this cluster from scratch would mean running `flux bootstrap`
again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and
landing in the identical state. The fact that Flux watches this repo is
itself declared *in* this repo.
If you want to watch a reconcile happen live rather than just inspect the
end state:
```sh
kubectl -n flux-system logs deploy/source-controller -f
```
then in another terminal, make any commit and push it — you'll see the
next fetch pick it up within the interval.
## Verifying the GitOps loop actually works
```sh
flux get kustomizations --watch
```
Watch until `podinfo`, `hello-app`, and `kubernetes-dashboard` all show
`Ready: True`, then confirm pods actually landed:
```sh
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n kubernetes-dashboard get pods
```
If that's all healthy, the entire chain — Forgejo repo → Flux → this
cluster — is working end to end with nothing manually `kubectl apply`'d.
## Exposing podinfo/hello-app through Caddy
k3s's bundled ingress controller (Traefik) is already listening on this
node's own IP, port 80, routing by the `Host:` header from each app's
`Ingress` resource (already defined in `apps/podinfo` and
`apps/hello-app`). Caddy just needs to forward matching requests there.
Two DNS records, as CNAMEs pointed at `git.boglabob.com` rather than
duplicating its IP directly — CNAME means "this name is an alias for that
one," so there's one place (that record) to update if the underlying IP
ever changes, instead of several:
```
podinfo.boglabob.com CNAME git.boglabob.com
hello.boglabob.com CNAME git.boglabob.com
```
Then in Caddy's own config, using the VM's pinned IP from earlier:
``` ```
podinfo.boglabob.com { podinfo.boglabob.com {
@ -251,26 +522,45 @@ hello.boglabob.com {
} }
``` ```
```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```
The Dashboard and the k3s API server are deliberately **not** here —
`docs/SETUP.md` steps 12–13 cover why (both are cluster-admin-capable, and
exposing either publicly is the exact pattern behind real breaches like
Tesla's 2018 incident) and how to reach them instead (`kubectl
port-forward`, and LAN/tunnel-only kubectl access). Both apply to this
cluster exactly as written there.
## Registering the Forgejo Actions runner
`docs/SETUP.md` step 9 applies as written — it's about the `k8s` user and
rootless Podman, not about which cluster exists. Worth understanding
before running it: the runner would normally get root-equivalent power
over its host via a mounted `docker.sock`; instead it runs as `k8s` itself
using rootless Podman's own API socket, and `build-hello-app.yml` builds
images with `kaniko` (no daemon, no elevated privileges needed at all) —
so nothing in that pipeline ever touches `sudo`.
## Tearing this down ## Tearing this down
Once the Terraform track in `docs/SETUP.md` is ready and you `tofu apply` Once the Terraform track is ready and `tofu apply` brings up the real
the real 3-node cluster, remove this one: 3-node cluster, remove this one:
```sh ```sh
# as k8s virsh -c qemu:///system destroy k3s-manual # stop it
export LIBVIRT_DEFAULT_URI=qemu:///system virsh -c qemu:///system undefine k3s-manual --remove-all-storage # VM + overlay disk
virsh destroy k3s-manual # stop it virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2 # base image isn't
virsh undefine k3s-manual --remove-all-storage # delete VM + its overlay disk # attached to the VM
virsh vol-delete --pool default k3s-manual-base.qcow2 # the base image isn't # directly, needs its
# attached to the VM # own delete
# directly, so it
# needs its own delete
rm ~/.kube/config-manual rm ~/.kube/config-manual
rm -rf ~/vms rm -rf ~/vms
``` ```
Nothing else needs cleaning up — Flux's state lived entirely inside that Nothing else needs cleaning up — Flux's own state lived entirely inside
VM's cluster and goes away with it. The Forgejo repo, deploy tokens, and that VM's cluster and goes away with it. The Forgejo repo, both tokens,
the Forgejo Actions runner registration are all cluster-independent and and the Forgejo Actions runner registration are all cluster-independent
carry over to the real cluster unchanged; just re-run `flux bootstrap` and carry over unchanged; point `flux bootstrap` at the new cluster's
against its kubeconfig once it's up. kubeconfig once it exists.