# Quickstart: manual cluster (no Terraform) A real k3s cluster, built by hand, so Flux/GitOps can be learned right away instead of waiting on the Terraform/libvirt provider work in `docs/SETUP.md` to get sorted (that provider did a breaking rewrite between 0.8.x and 0.9.x, and the HCL needs writing against the real schema, not memory). This is one throwaway VM, created directly with `virt-install` — no Terraform involved at all. Everything here explains the *why*, not just the *what*: read each section before running its commands, and where a config file is being built (the cloud-init data, the Caddy block), treat the snippets as pieces to assemble into your own file, not something to paste wholesale. Shares steps 1–4 of `docs/SETUP.md` as prerequisites: - Step 1: KVM/libvirt packages installed on the T630. - Step 2: the unprivileged `k8s` user exists, in the `libvirt`/`kvm` groups, with an SSH keypair at `~/.ssh/id_ed25519`. - Step 3: skip — no join token needed here (see "one node is enough" below). - Step 4: `cloud-demo` pushed to Forgejo, with a `k8s-readonly` and a `flux-write` token generated. Everything below runs as `k8s` on the T630 (`sudo -iu k8s`). --- ## Why one node is enough A k3s **server** node runs the control plane (API server, scheduler, etcd/SQLite) *and* schedules ordinary workloads onto itself unless you explicitly disable that. So a single server, with no agents, is already a complete, working cluster — nothing here needs a join token or a second VM. The 3-node design in `docs/SETUP.md` exists to mirror a more realistic multi-node cluster once Terraform can build it repeatedly and disposably; for learning Flux and GitOps, that extra shape doesn't buy you anything yet. ## Two libvirt connections, and why it matters libvirt isn't one daemon with one namespace of VMs — from a client's perspective there are (at least) two separate connections: - `qemu:///system` — the shared, host-wide instance. VMs here can use privileged networking (bridges, NAT with DHCP), and management access is gated by group membership (`libvirt`/`kvm`) checked via polkit — which is exactly what step 2's `usermod -aG libvirt,kvm k8s` set up. No sudo needed for any command below; that group membership *is* the authorization. - `qemu:///session` — a private, per-user instance with no special privileges, and critically, its own separate storage pools and networks that don't overlap with the system instance at all. For a non-root user, `virsh`/`virt-install` **default to `session`** unless told otherwise. This matters a lot in practice: it's easy to set `LIBVIRT_DEFAULT_URI=qemu:///system` in one terminal, run a command in a different terminal where it isn't set, and have that command silently create something under `session` instead — where it's invisible to everything else you're doing. Every command below uses `-c qemu:///system` / `--connect qemu:///system` explicitly for exactly this reason, rather than relying on the environment variable. ## Storage: why a raw path in `~` doesn't work The most natural first instinct is to put a VM's disk file somewhere in `k8s`'s home directory and point `--disk` at it directly. That fails non-obviously: under `qemu:///system`, the actual QEMU process backing a VM doesn't run as `k8s` — it runs as a separate, restricted `libvirt-qemu` user (a deliberate security boundary, so a compromised VM process has its own limited identity rather than the identity of whoever created it). `k8s`'s home directory defaults to mode `700` — readable only by `k8s` — so `libvirt-qemu` can't read into it at all, and the VM fails at boot. The fix is to let libvirt manage the storage itself, in a **pool**. A pool is just a named, libvirt-tracked location for disk images (a directory, in the simplest case). The key property: creating or writing a volume inside a pool goes through libvirtd's API, not through `k8s`'s own filesystem permissions — so it's libvirtd (already running with the right privileges) that handles ownership correctly, regardless of what user asked for it. Check what pools/networks already exist: ```sh virsh -c qemu:///system pool-list --all virsh -c qemu:///system net-list --all ``` On this box, `net-list` showed a `default` network already defined (just inactive), but `pool-list` came back completely empty — **Debian's `libvirt-daemon-system` package does not auto-create a `default` storage pool**, unlike some other distros' packaging. Both need fixing before anything else: ```sh # only if pool-list was empty virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images virsh -c qemu:///system pool-build default virsh -c qemu:///system pool-start default virsh -c qemu:///system pool-autostart default # only if net-list showed 'default' as inactive virsh -c qemu:///system net-start default virsh -c qemu:///system net-autostart default ``` `pool-autostart`/`net-autostart` mean both come back up automatically after a host reboot — without it, they'd need manually starting again every time. ## Building the base image and the VM's own disk Downloading the cloud image doesn't need to go through the pool — this copy is only ever read by `virsh` itself (running as `k8s`), never directly by the VM, so an ordinary temp location is fine: ```sh curl -L -o /tmp/noble-base.img \ https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img ``` Getting it *into* the pool, though, does need to go through libvirt's API — `vol-create-as` allocates an empty volume of a given size inside the pool, and `vol-upload` streams a local file's bytes into it: ```sh virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \ --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2 virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse rm /tmp/noble-base.img ``` The VM itself shouldn't boot directly off this base image — if it did, every write the OS makes (logs, package installs, k3s's own state) would permanently modify the one shared base file, corrupting it for any future VM built from the same base. Instead, create a **copy-on-write overlay**: a second volume that starts out empty and only stores the *differences* from its backing volume. Reads that haven't been changed transparently fall through to the base image; writes go into the overlay. This is the same relationship a Docker image layer has to its base layer. ```sh virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \ --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2 virsh -c qemu:///system vol-list --pool default # should list both volumes now ``` ## cloud-init: how a stock image becomes *this* VM The base image is a generic Ubuntu install — it has no idea it's about to become a k3s node, and has no user account you could log into. **cloud-init** is the standard mechanism cloud images use to configure themselves on first boot, driven by data supplied externally rather than baked into the image. `virt-install`'s `--cloud-init` flag builds a small ISO (the "NoCloud" datasource) containing that data and attaches it to the VM; cloud-init, already installed in the image, detects it automatically at boot. Two separate pieces of data go in, and they answer different questions: - **`user-data`** — *what should exist on this machine*: users, packages, commands to run. Written as `#cloud-config` YAML. - **`network-config`** — *how should this machine's network be set up*. This quickstart skips it entirely and lets the VM get an address via DHCP from the `default` network instead (the Terraform track, by contrast, uses this for static IPs, since it manages its own isolated network). Build `user-data` up piece by piece. Start with identity: ```yaml #cloud-config hostname: k3s-manual manage_etc_hosts: true ``` Then the one thing you actually need to log in and administer this box — a user, with your public key rather than a password (cloud images have no default password, and SSH password auth is normally disabled anyway), and passwordless sudo so you're not stuck typing a password you never set: ```yaml users: - name: k3s groups: sudo shell: /bin/bash sudo: ALL=(ALL) NOPASSWD:ALL ssh_authorized_keys: - ``` Then the actual payload — install k3s, and stage a copy of its auto-generated kubeconfig somewhere the `k3s` user can read (by default it's only readable by root): ```yaml package_update: true packages: - curl runcmd: - curl -sfL https://get.k3s.io | sh -s - server - mkdir -p /home/k3s/.kube - k3s kubectl config view --raw > /home/k3s/.kube/config - chown -R k3s:k3s /home/k3s/.kube ``` Assemble those three pieces into one file: ```sh mkdir -p ~/vms cat > ~/vms/k3s-manual-user-data.yaml <<'EOF' #cloud-config hostname: k3s-manual manage_etc_hosts: true users: - name: k3s groups: sudo shell: /bin/bash sudo: ALL=(ALL) NOPASSWD:ALL ssh_authorized_keys: - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE package_update: true packages: - curl runcmd: - curl -sfL https://get.k3s.io | sh -s - server - mkdir -p /home/k3s/.kube - k3s kubectl config view --raw > /home/k3s/.kube/config - chown -R k3s:k3s /home/k3s/.kube EOF ``` Unlike the disk, this file is fine sitting under `~/vms` — it's only ever read client-side by `virt-install` (as `k8s`), which hands the resulting seed data to libvirtd over the API; `libvirt-qemu` never touches it directly. ## Creating the VM Each `virt-install` flag is answering a specific question: | Flag | Answers | |---|---| | `--connect qemu:///system` | which libvirt instance (see above) | | `--name` | the domain's name, used everywhere else (`virsh`, `domifaddr`, teardown) | | `--memory` / `--vcpus` | resource allocation — kept small deliberately | | `--disk vol=default/k3s-manual.qcow2` | use the pool-managed overlay, not a raw path | | `--import` | boot the disk as-is rather than running an OS installer against it | | `--os-variant` | a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed | | `--network network=default` | attach to the NAT network from earlier | | `--cloud-init user-data=...` | the file just built | | `--graphics none` | no VNC/spice display — this is a headless server VM | | `--noautoconsole` | don't attach to its console interactively after creation | Two gotchas worth knowing before running this: - **`$HOME`, not `~`, in the `--cloud-init` argument.** Bash only expands `~` at the very start of a word; `user-data=~/vms/...` is *inside* a word (after `=`), so the tilde would be passed through literally and `virt-install` would fail looking for a file called `~`. `$HOME` expands regardless of position. - **`--os-variant` may need to be an older release than the actual image.** `osinfo-db` (the database `virt-install` validates this against) can lag behind real Ubuntu releases — if `ubuntu24.04` comes back "unknown", check what's actually available with `osinfo-query os | grep -i ubuntu` and use the newest one it recognizes. Since `--import` just boots the disk as-is, this hint doesn't change what's actually installed. ```sh virt-install \ --connect qemu:///system \ --name k3s-manual \ --memory 2048 \ --vcpus 2 \ --disk vol=default/k3s-manual.qcow2 \ --import \ --os-variant ubuntu22.04 \ --network network=default \ --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \ --graphics none \ --noautoconsole ``` ## Finding the VM and connecting ```sh virsh -c qemu:///system domifaddr k3s-manual ``` This prints something like `192.168.122.67/24` — the `/24` is CIDR notation for the subnet mask (`255.255.255.0`), describing the *network* this address belongs to, not part of the address itself. Use just the plain IP (`192.168.122.67`) to actually connect. Give cloud-init a couple of minutes after `Domain creation completed` before it's reachable — it's installing k3s in the background. This address comes from the `default` network's DHCP server, leased against the VM's MAC address (shown in the same `domifaddr` output). DHCP leases are "sticky" in practice — the VM will keep asking for and getting the same address on renewal — but that's not the same as *guaranteed* fixed. Pin it explicitly if you don't want to risk it changing later: ```sh virsh -c qemu:///system net-update default add ip-dhcp-host \ "" --live --config ``` ## kubectl: what's actually in a kubeconfig A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate (so your client trusts the API server's identity) plus a client certificate and private key (so the API server trusts yours). k3s generates one for itself pointed at `127.0.0.1`, which only works from inside the VM — the `sed` below swaps that for the VM's real address so it works from the T630 instead: ```sh mkdir -p ~/.kube ssh k3s@ sudo cat /etc/rancher/k3s/k3s.yaml \ | sed "s/127.0.0.1//" > ~/.kube/config-manual ``` `kubectl` itself isn't installed anywhere yet. Rather than a system package (which would need `sudo`, which `k8s` doesn't have), grab the official binary release straight from Kubernetes' own distribution point and drop it somewhere already on `k8s`'s `PATH`: ```sh curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl mkdir -p ~/.local/bin mv kubectl ~/.local/bin/ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc export PATH="$HOME/.local/bin:$PATH" export KUBECONFIG=~/.kube/config-manual kubectl get nodes # expect 1 Ready node ``` ## Flux: what's actually happening, mechanism by mechanism "Flux" isn't one program watching your repo by magic — it's a handful of ordinary Kubernetes controllers (just Deployments, like anything else you'd run on the cluster), each one understanding a couple of **Custom Resource Definitions** — CRDs extend the Kubernetes API with new object kinds, the same way `Deployment` or `Service` are built-in kinds. `GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing about any of this is special-cased outside the normal Kubernetes API — it's the exact same "define an object, a controller notices it and acts" loop that runs the whole rest of Kubernetes. Install the CLI the same no-sudo, direct-binary-release way as `kubectl`: ```sh FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//') curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux rm /tmp/flux.tar.gz flux --version ``` Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this uses the generic git bootstrap — over HTTPS with the `flux-write` token from step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably reachable from either this desktop or the T630 — see the earlier troubleshooting in this project's history): ```sh flux check --pre --kubeconfig ~/.kube/config-manual flux bootstrap git \ --url=https://git.boglabob.com/codegit/cloud-demo \ --branch=main \ --path=clusters/homelab \ --username=codegit \ --password= \ --token-auth \ --kubeconfig ~/.kube/config-manual ``` That one command did five genuinely separate things. Go look at each — these commands work against the cluster you already bootstrapped: **1. It installed the controllers** — plain Kubernetes Deployments, no different in kind from anything else running on the cluster: ```sh kubectl -n flux-system get deployments ``` `source-controller`, `kustomize-controller`, `helm-controller`, `notification-controller`. Alongside them, it registered the CRDs those controllers understand: ```sh kubectl get crds | grep fluxcd ``` **2. It created a `GitRepository` object** — this is the whole "what repo am I watching" declaration, and nothing more. `source-controller`'s job is entirely mechanical: every `interval` (default 1m), do a real `git fetch` against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that tree into a `.tar.gz`, and record its location in `.status.artifact`. That's the entire job — it doesn't know or care what's *in* the repo. ```sh kubectl -n flux-system get gitrepository flux-system -o yaml ``` Look at `.status.conditions` and `.status.artifact` — that's the result of an actual git fetch that already happened, not a static config. **3. It created a `Kustomization` object** (confusingly, the same name as the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. — related but not identical). `kustomize-controller`'s job, on its own interval: fetch the artifact `source-controller` produced, run the real `kustomize` tool against `.spec.path` inside it (the exact same tool a plain `kustomize build apps/podinfo` would run locally against those `kustomization.yaml` files), and apply the resulting objects via the Kubernetes API — the automated equivalent of you running `kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer, forever. `prune: true` (set on all the `Kustomization` objects in this repo) means it also *deletes* anything it previously created that's no longer present in the current git state — that's what makes it self-healing rather than just "apply once." ```sh kubectl -n flux-system get kustomization ``` **This is also why `apps/podinfo`, `apps/hello-app`, and `apps/kubernetes-dashboard` started deploying without you ever running `kubectl apply` on them.** Bootstrap's own `Kustomization` watches `clusters/homelab` with `prune: true`. `clusters/homelab/apps.yaml` (already sitting in the repo, hand-written earlier in this project) itself just *defines more `Kustomization` objects*, one per app — so the first one picks it up as part of its own normal reconcile, creates those three child `Kustomization`s, and each of *those* then does its own fetch-and-apply against its own app directory. It's the same mechanism recursing, not a special case. **4. It stored your credential as a Kubernetes Secret** — `--password` here is the `flux-write` token, not an account password. Bootstrap base64-wraps it into a Secret that matches what the `GitRepository`'s `.spec.secretRef` points at: ```sh kubectl -n flux-system get secret flux-system -o yaml ``` The `password` field is base64 (`| base64 -d` to read it) — this is exactly the same credential `git clone https://user:token@host/repo` would use, just read by `source-controller` on every fetch instead of typed by you once. It's never written to `k8s`'s own filesystem. **5. It committed that config back into the repo itself** — `git pull` in your desktop clone and look at `clusters/homelab/flux-system/`. The `GitRepository`/`Kustomization` objects you just inspected live in the cluster *because* those exact YAML files are committed there — so rebuilding this cluster from scratch would mean running `flux bootstrap` again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and landing in the identical state. The fact that Flux watches this repo is itself declared *in* this repo. If you want to watch a reconcile happen live rather than just inspect the end state: ```sh kubectl -n flux-system logs deploy/source-controller -f ``` then in another terminal, make any commit and push it — you'll see the next fetch pick it up within the interval. ## Verifying the GitOps loop actually works ```sh flux get kustomizations --watch ``` Watch until `podinfo`, `hello-app`, and `kubernetes-dashboard` all show `Ready: True`, then confirm pods actually landed: ```sh kubectl -n podinfo get pods kubectl -n hello-app get pods kubectl -n kubernetes-dashboard get pods ``` If that's all healthy, the entire chain — Forgejo repo → Flux → this cluster — is working end to end with nothing manually `kubectl apply`'d. ## Exposing podinfo/hello-app through Caddy k3s's bundled ingress controller (Traefik) is already listening on this node's own IP, port 80, routing by the `Host:` header from each app's `Ingress` resource (already defined in `apps/podinfo` and `apps/hello-app`). Caddy just needs to forward matching requests there. Two DNS records, as CNAMEs pointed at `git.boglabob.com` rather than duplicating its IP directly — CNAME means "this name is an alias for that one," so there's one place (that record) to update if the underlying IP ever changes, instead of several: ``` podinfo.boglabob.com CNAME git.boglabob.com hello.boglabob.com CNAME git.boglabob.com ``` Then in Caddy's own config, using the VM's pinned IP from earlier: ``` podinfo.boglabob.com { reverse_proxy http://:80 } hello.boglabob.com { reverse_proxy http://:80 } ``` ```sh podman exec caddy reload --config /etc/caddy/Caddyfile ``` The Dashboard and the k3s API server are deliberately **not** here — `docs/SETUP.md` steps 12–13 cover why (both are cluster-admin-capable, and exposing either publicly is the exact pattern behind real breaches like Tesla's 2018 incident) and how to reach them instead (`kubectl port-forward`, and LAN/tunnel-only kubectl access). Both apply to this cluster exactly as written there. ## Registering the Forgejo Actions runner `docs/SETUP.md` step 9 applies as written — it's about the `k8s` user and rootless Podman, not about which cluster exists. Worth understanding before running it: the runner would normally get root-equivalent power over its host via a mounted `docker.sock`; instead it runs as `k8s` itself using rootless Podman's own API socket, and `build-hello-app.yml` builds images with `kaniko` (no daemon, no elevated privileges needed at all) — so nothing in that pipeline ever touches `sudo`. ## Tearing this down Once the Terraform track is ready and `tofu apply` brings up the real 3-node cluster, remove this one: ```sh virsh -c qemu:///system destroy k3s-manual # stop it virsh -c qemu:///system undefine k3s-manual --remove-all-storage # VM + overlay disk virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2 # base image isn't # attached to the VM # directly, needs its # own delete rm ~/.kube/config-manual rm -rf ~/vms ``` Nothing else needs cleaning up — Flux's own state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, both tokens, and the Forgejo Actions runner registration are all cluster-independent and carry over unchanged; point `flux bootstrap` at the new cluster's kubeconfig once it exists.