cloud-demo/docs/QUICKSTART.md

Quickstart: manual cluster (no Terraform)

A real k3s cluster, built by hand, so Flux/GitOps can be learned right away instead of waiting on the Terraform/libvirt provider work in docs/SETUP.md to get sorted (that provider did a breaking rewrite between 0.8.x and 0.9.x, and the HCL needs writing against the real schema, not memory).

This is one throwaway VM, created directly with virt-install — no Terraform involved at all. Everything here explains the why, not just the what: read each section before running its commands, and where a config file is being built (the cloud-init data, the Caddy block), treat the snippets as pieces to assemble into your own file, not something to paste wholesale.

Shares steps 1–4 of docs/SETUP.md as prerequisites:

• Step 1: KVM/libvirt packages installed on the T630.
• Step 2: the unprivileged k8s user exists, in the libvirt/kvm groups, with an SSH keypair at ~/.ssh/id_ed25519.
• Step 3: skip — no join token needed here (see "one node is enough" below).
• Step 4: cloud-demo pushed to Forgejo, with a k8s-readonly and a flux-write token generated.

Everything below runs as k8s on the T630 (sudo -iu k8s).

---

Why one node is enough

A k3s server node runs the control plane (API server, scheduler, etcd/SQLite) and schedules ordinary workloads onto itself unless you explicitly disable that. So a single server, with no agents, is already a complete, working cluster — nothing here needs a join token or a second VM. The 3-node design in docs/SETUP.md exists to mirror a more realistic multi-node cluster once Terraform can build it repeatedly and disposably; for learning Flux and GitOps, that extra shape doesn't buy you anything yet.

Two libvirt connections, and why it matters

libvirt isn't one daemon with one namespace of VMs — from a client's perspective there are (at least) two separate connections:

• qemu:///system — the shared, host-wide instance. VMs here can use privileged networking (bridges, NAT with DHCP), and management access is gated by group membership (libvirt/kvm) checked via polkit — which is exactly what step 2's usermod -aG libvirt,kvm k8s set up. No sudo needed for any command below; that group membership is the authorization.
• qemu:///session — a private, per-user instance with no special privileges, and critically, its own separate storage pools and networks that don't overlap with the system instance at all.

For a non-root user, virsh/virt-install default to session unless told otherwise. This matters a lot in practice: it's easy to set LIBVIRT_DEFAULT_URI=qemu:///system in one terminal, run a command in a different terminal where it isn't set, and have that command silently create something under session instead — where it's invisible to everything else you're doing. Every command below uses -c qemu:///system / --connect qemu:///system explicitly for exactly this reason, rather than relying on the environment variable.

Storage: why a raw path in ~ doesn't work

The most natural first instinct is to put a VM's disk file somewhere in k8s's home directory and point --disk at it directly. That fails non-obviously: under qemu:///system, the actual QEMU process backing a VM doesn't run as k8s — it runs as a separate, restricted libvirt-qemu user (a deliberate security boundary, so a compromised VM process has its own limited identity rather than the identity of whoever created it). k8s's home directory defaults to mode 700 — readable only by k8s — so libvirt-qemu can't read into it at all, and the VM fails at boot.

The fix is to let libvirt manage the storage itself, in a pool. A pool is just a named, libvirt-tracked location for disk images (a directory, in the simplest case). The key property: creating or writing a volume inside a pool goes through libvirtd's API, not through k8s's own filesystem permissions — so it's libvirtd (already running with the right privileges) that handles ownership correctly, regardless of what user asked for it.

Check what pools/networks already exist:

virsh -c qemu:///system pool-list --all
virsh -c qemu:///system net-list --all

On this box, net-list showed a default network already defined (just inactive), but pool-list came back completely empty — Debian's libvirt-daemon-system package does not auto-create a default storage pool, unlike some other distros' packaging. Both need fixing before anything else:

# only if pool-list was empty
virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default

# only if net-list showed 'default' as inactive
virsh -c qemu:///system net-start default
virsh -c qemu:///system net-autostart default

pool-autostart/net-autostart mean both come back up automatically after a host reboot — without it, they'd need manually starting again every time.

Building the base image and the VM's own disk

Downloading the cloud image doesn't need to go through the pool — this copy is only ever read by virsh itself (running as k8s), never directly by the VM, so an ordinary temp location is fine:

curl -L -o /tmp/noble-base.img \
  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

Getting it into the pool, though, does need to go through libvirt's API — vol-create-as allocates an empty volume of a given size inside the pool, and vol-upload streams a local file's bytes into it:

virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img

The VM itself shouldn't boot directly off this base image — if it did, every write the OS makes (logs, package installs, k3s's own state) would permanently modify the one shared base file, corrupting it for any future VM built from the same base. Instead, create a copy-on-write overlay: a second volume that starts out empty and only stores the differences from its backing volume. Reads that haven't been changed transparently fall through to the base image; writes go into the overlay. This is the same relationship a Docker image layer has to its base layer.

virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2

virsh -c qemu:///system vol-list --pool default   # should list both volumes now

cloud-init: how a stock image becomes this VM

The base image is a generic Ubuntu install — it has no idea it's about to become a k3s node, and has no user account you could log into. cloud-init is the standard mechanism cloud images use to configure themselves on first boot, driven by data supplied externally rather than baked into the image. virt-install's --cloud-init flag builds a small ISO (the "NoCloud" datasource) containing that data and attaches it to the VM; cloud-init, already installed in the image, detects it automatically at boot.

Two separate pieces of data go in, and they answer different questions:

• user-data — what should exist on this machine: users, packages, commands to run. Written as #cloud-config YAML.
• network-config — how should this machine's network be set up. This quickstart skips it entirely and lets the VM get an address via DHCP from the default network instead (the Terraform track, by contrast, uses this for static IPs, since it manages its own isolated network).

Build user-data up piece by piece. Start with identity:

#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

Then the one thing you actually need to log in and administer this box — a user, with your public key rather than a password (cloud images have no default password, and SSH password auth is normally disabled anyway), and passwordless sudo so you're not stuck typing a password you never set:

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <contents of ~/.ssh/id_ed25519.pub>

Then the actual payload — install k3s, and stage a copy of its auto-generated kubeconfig somewhere the k3s user can read (by default it's only readable by root):

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube

Assemble those three pieces into one file:

mkdir -p ~/vms
cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
EOF

Unlike the disk, this file is fine sitting under ~/vms — it's only ever read client-side by virt-install (as k8s), which hands the resulting seed data to libvirtd over the API; libvirt-qemu never touches it directly.

Creating the VM

Each virt-install flag is answering a specific question:

Flag	Answers
--connect qemu:///system	which libvirt instance (see above)
--name	the domain's name, used everywhere else (virsh, domifaddr, teardown)
--memory / --vcpus	resource allocation — kept small deliberately
--disk vol=default/k3s-manual.qcow2	use the pool-managed overlay, not a raw path
--import	boot the disk as-is rather than running an OS installer against it
--os-variant	a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed
--network network=default	attach to the NAT network from earlier
--cloud-init user-data=...	the file just built
--graphics none	no VNC/spice display — this is a headless server VM
--noautoconsole	don't attach to its console interactively after creation

Two gotchas worth knowing before running this:

• $HOME, not ~, in the --cloud-init argument. Bash only expands ~ at the very start of a word; user-data=~/vms/... is inside a word (after =), so the tilde would be passed through literally and virt-install would fail looking for a file called ~. $HOME expands regardless of position.
• --os-variant may need to be an older release than the actual image. osinfo-db (the database virt-install validates this against) can lag behind real Ubuntu releases — if ubuntu24.04 comes back "unknown", check what's actually available with osinfo-query os | grep -i ubuntu and use the newest one it recognizes. Since --import just boots the disk as-is, this hint doesn't change what's actually installed.

virt-install \
  --connect qemu:///system \
  --name k3s-manual \
  --memory 2048 \
  --vcpus 2 \
  --disk vol=default/k3s-manual.qcow2 \
  --import \
  --os-variant ubuntu22.04 \
  --network network=default \
  --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \
  --graphics none \
  --noautoconsole

Finding the VM and connecting

virsh -c qemu:///system domifaddr k3s-manual

This prints something like 192.168.122.67/24 — the /24 is CIDR notation for the subnet mask (255.255.255.0), describing the network this address belongs to, not part of the address itself. Use just the plain IP (192.168.122.67) to actually connect.

Give cloud-init a couple of minutes after Domain creation completed before it's reachable — it's installing k3s in the background.

This address comes from the default network's DHCP server, leased against the VM's MAC address (shown in the same domifaddr output). DHCP leases are "sticky" in practice — the VM will keep asking for and getting the same address on renewal — but that's not the same as guaranteed fixed. Pin it explicitly if you don't want to risk it changing later:

virsh -c qemu:///system net-update default add ip-dhcp-host \
  "<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config

kubectl: what's actually in a kubeconfig

A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate (so your client trusts the API server's identity) plus a client certificate and private key (so the API server trusts yours). k3s generates one for itself pointed at 127.0.0.1, which only works from inside the VM — the sed below swaps that for the VM's real address so it works from the T630 instead:

mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual

kubectl itself isn't installed anywhere yet. Rather than a system package (which would need sudo, which k8s doesn't have), grab the official binary release straight from Kubernetes' own distribution point and drop it somewhere already on k8s's PATH:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"

export KUBECONFIG=~/.kube/config-manual
kubectl get nodes   # expect 1 Ready node

Flux: what's actually happening, mechanism by mechanism

"Flux" isn't one program watching your repo by magic — it's a handful of ordinary Kubernetes controllers (just Deployments, like anything else you'd run on the cluster), each one understanding a couple of Custom Resource Definitions — CRDs extend the Kubernetes API with new object kinds, the same way Deployment or Service are built-in kinds. GitRepository and Kustomization are two such kinds Flux adds. Nothing about any of this is special-cased outside the normal Kubernetes API — it's the exact same "define an object, a controller notices it and acts" loop that runs the whole rest of Kubernetes.

Install the CLI the same no-sudo, direct-binary-release way as kubectl:

FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version

Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this uses the generic git bootstrap — over HTTPS with the flux-write token from step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably reachable from either this desktop or the T630 — see the earlier troubleshooting in this project's history):

flux check --pre --kubeconfig ~/.kube/config-manual

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-manual

That one command did five genuinely separate things. Go look at each — these commands work against the cluster you already bootstrapped:

1. It installed the controllers — plain Kubernetes Deployments, no different in kind from anything else running on the cluster:

kubectl -n flux-system get deployments

source-controller, kustomize-controller, helm-controller, notification-controller. Alongside them, it registered the CRDs those controllers understand:

kubectl get crds | grep fluxcd

2. It created a GitRepository object — this is the whole "what repo am I watching" declaration, and nothing more. source-controller's job is entirely mechanical: every interval (default 1m), do a real git fetch against .spec.url/.spec.ref; if the commit SHA changed, package that tree into a .tar.gz, and record its location in .status.artifact. That's the entire job — it doesn't know or care what's in the repo.

kubectl -n flux-system get gitrepository flux-system -o yaml

Look at .status.conditions and .status.artifact — that's the result of an actual git fetch that already happened, not a static config.

3. It created a Kustomization object (confusingly, the same name as the kustomization.yaml files already sitting in apps/podinfo/ etc. — related but not identical). kustomize-controller's job, on its own interval: fetch the artifact source-controller produced, run the real kustomize tool against .spec.path inside it (the exact same tool a plain kustomize build apps/podinfo would run locally against those kustomization.yaml files), and apply the resulting objects via the Kubernetes API — the automated equivalent of you running kubectl apply -f <(kustomize build apps/podinfo) yourself, on a timer, forever. prune: true (set on all the Kustomization objects in this repo) means it also deletes anything it previously created that's no longer present in the current git state — that's what makes it self-healing rather than just "apply once."

kubectl -n flux-system get kustomization

This is also why apps/podinfo, apps/hello-app, and apps/kubernetes-dashboard started deploying without you ever running kubectl apply on them. Bootstrap's own Kustomization watches clusters/homelab with prune: true. clusters/homelab/apps.yaml (already sitting in the repo, hand-written earlier in this project) itself just defines more Kustomization objects, one per app — so the first one picks it up as part of its own normal reconcile, creates those three child Kustomizations, and each of those then does its own fetch-and-apply against its own app directory. It's the same mechanism recursing, not a special case.

4. It stored your credential as a Kubernetes Secret — --password here is the flux-write token, not an account password. Bootstrap base64-wraps it into a Secret that matches what the GitRepository's .spec.secretRef points at:

kubectl -n flux-system get secret flux-system -o yaml

The password field is base64 (| base64 -d to read it) — this is exactly the same credential git clone https://user:token@host/repo would use, just read by source-controller on every fetch instead of typed by you once. It's never written to k8s's own filesystem.

5. It committed that config back into the repo itself — git pull in your desktop clone and look at clusters/homelab/flux-system/. The GitRepository/Kustomization objects you just inspected live in the cluster because those exact YAML files are committed there — so rebuilding this cluster from scratch would mean running flux bootstrap again (or even just kubectl apply -f clusters/homelab/flux-system/) and landing in the identical state. The fact that Flux watches this repo is itself declared in this repo.

If you want to watch a reconcile happen live rather than just inspect the end state:

kubectl -n flux-system logs deploy/source-controller -f

then in another terminal, make any commit and push it — you'll see the next fetch pick it up within the interval.

Verifying the GitOps loop actually works

flux get kustomizations --watch

Watch until podinfo, hello-app, and kubernetes-dashboard all show Ready: True, then confirm pods actually landed:

kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n kubernetes-dashboard get pods

If that's all healthy, the entire chain — Forgejo repo → Flux → this cluster — is working end to end with nothing manually kubectl apply'd.

Exposing podinfo/hello-app through Caddy

k3s's bundled ingress controller (Traefik) is already listening on this node's own IP, port 80, routing by the Host: header from each app's Ingress resource (already defined in apps/podinfo and apps/hello-app). Caddy just needs to forward matching requests there.

Two DNS records, as CNAMEs pointed at git.boglabob.com rather than duplicating its IP directly — CNAME means "this name is an alias for that one," so there's one place (that record) to update if the underlying IP ever changes, instead of several:

podinfo.boglabob.com  CNAME  git.boglabob.com
hello.boglabob.com    CNAME  git.boglabob.com

Then in Caddy's own config, using the VM's pinned IP from earlier:

podinfo.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

hello.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile

The Dashboard and the k3s API server are deliberately not here — docs/SETUP.md steps 12–13 cover why (both are cluster-admin-capable, and exposing either publicly is the exact pattern behind real breaches like Tesla's 2018 incident) and how to reach them instead (kubectl port-forward, and LAN/tunnel-only kubectl access). Both apply to this cluster exactly as written there.

Registering the Forgejo Actions runner

docs/SETUP.md step 9 applies as written — it's about the k8s user and rootless Podman, not about which cluster exists. Worth understanding before running it: the runner would normally get root-equivalent power over its host via a mounted docker.sock; instead it runs as k8s itself using rootless Podman's own API socket, and build-hello-app.yml builds images with kaniko (no daemon, no elevated privileges needed at all) — so nothing in that pipeline ever touches sudo.

Tearing this down

Once the Terraform track is ready and tofu apply brings up the real 3-node cluster, remove this one:

virsh -c qemu:///system destroy k3s-manual              # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage   # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2   # base image isn't
                                                                            # attached to the VM
                                                                            # directly, needs its
                                                                            # own delete
rm ~/.kube/config-manual
rm -rf ~/vms

Nothing else needs cleaning up — Flux's own state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, both tokens, and the Forgejo Actions runner registration are all cluster-independent and carry over unchanged; point flux bootstrap at the new cluster's kubeconfig once it exists.