Updated project to improve guidance

2026-09-03 19:06:27 +01:00 · 2026-09-03 19:06:27 +01:00 · 4ea6d8b9e5
commit 4ea6d8b9e5
parent 63008b3ab9
26 changed files with 1091 additions and 695 deletions
--- a/docs/02-k3s.md
+++ b/docs/02-k3s.md
@ -0,0 +1,344 @@
+# Stage 2: A real k3s cluster, built by hand
+
+Stage 1 (`docs/01-bootstrap.md`) got KVM/libvirt installed and the `k8s`
+user created — that's all this stage needs (its steps 1-2; skip 3-4 for
+now, they're not needed until stage 3). This stage builds one throwaway VM
+directly with `virt-install` — no Terraform involved at all — and installs
+k3s on it, so stage 3 has a real cluster to bootstrap Flux/GitOps against.
+Terraform doesn't show up until stage 4, once you already understand what
+it's automating.
+
+Everything here explains the *why*, not just the *what*: read each section
+before running its commands, and where a config file is being built (the
+cloud-init data, the Caddy block later on), treat the snippets as pieces to
+assemble into your own file, not something to paste wholesale.
+
+Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).
+
+---
+
+## Why one node is enough
+
+A k3s **server** node runs the control plane (API server, scheduler,
+etcd/SQLite) *and* schedules ordinary workloads onto itself unless you
+explicitly disable that. So a single server, with no agents, is already a
+complete, working cluster — nothing here needs a join token or a second
+VM. The 3-node design in `docs/04-tofu.md` exists to mirror a more
+realistic multi-node cluster once Terraform can build it repeatedly and
+disposably; for learning k3s, Helm, and Flux/GitOps, that extra shape
+doesn't buy you anything yet.
+
+## Two libvirt connections, and why it matters
+
+libvirt isn't one daemon with one namespace of VMs — from a client's
+perspective there are (at least) two separate connections:
+
+- `qemu:///system` — the shared, host-wide instance. VMs here can use
+  privileged networking (bridges, NAT with DHCP), and management access is
+  gated by group membership (`libvirt`/`kvm`) checked via polkit — which is
+  exactly what bootstrap step 2's `usermod -aG libvirt,kvm k8s` set up. No
+  sudo needed for any command below; that group membership *is* the
+  authorization.
+- `qemu:///session` — a private, per-user instance with no special
+  privileges, and critically, its own separate storage pools and networks
+  that don't overlap with the system instance at all.
+
+For a non-root user, `virsh`/`virt-install` **default to `session`** unless
+told otherwise. This matters a lot in practice: it's easy to set
+`LIBVIRT_DEFAULT_URI=qemu:///system` in one terminal, run a command in a
+different terminal where it isn't set, and have that command silently
+create something under `session` instead — where it's invisible to
+everything else you're doing. Every command below uses `-c qemu:///system`
+/ `--connect qemu:///system` explicitly for exactly this reason, rather
+than relying on the environment variable.
+
+## Storage: why a raw path in `~` doesn't work
+
+The most natural first instinct is to put a VM's disk file somewhere in
+`k8s`'s home directory and point `--disk` at it directly. That fails
+non-obviously: under `qemu:///system`, the actual QEMU process backing a
+VM doesn't run as `k8s` — it runs as a separate, restricted `libvirt-qemu`
+user (a deliberate security boundary, so a compromised VM process has its
+own limited identity rather than the identity of whoever created it).
+`k8s`'s home directory defaults to mode `700` — readable only by `k8s` —
+so `libvirt-qemu` can't read into it at all, and the VM fails at boot.
+
+The fix is to let libvirt manage the storage itself, in a **pool**. A pool
+is just a named, libvirt-tracked location for disk images (a directory, in
+the simplest case). The key property: creating or writing a volume inside
+a pool goes through libvirtd's API, not through `k8s`'s own filesystem
+permissions — so it's libvirtd (already running with the right privileges)
+that handles ownership correctly, regardless of what user asked for it.
+(Stage 4's Terraform config hits this exact same problem and solves it the
+same way — see `terraform/main.tf`'s `libvirt_pool` resource.)
+
+Check what pools/networks already exist:
+
+```sh
+virsh -c qemu:///system pool-list --all
+virsh -c qemu:///system net-list --all
+```
+
+On this box, `net-list` showed a `default` network already defined (just
+inactive), but `pool-list` came back completely empty — **Debian's
+`libvirt-daemon-system` package does not auto-create a `default` storage
+pool**, unlike some other distros' packaging. Both need fixing before
+anything else:
+
+```sh
+# only if pool-list was empty
+virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
+virsh -c qemu:///system pool-build default
+virsh -c qemu:///system pool-start default
+virsh -c qemu:///system pool-autostart default
+
+# only if net-list showed 'default' as inactive
+virsh -c qemu:///system net-start default
+virsh -c qemu:///system net-autostart default
+```
+
+`pool-autostart`/`net-autostart` mean both come back up automatically after
+a host reboot — without it, they'd need manually starting again every time.
+
+## Building the base image and the VM's own disk
+
+Downloading the cloud image doesn't need to go through the pool — this
+copy is only ever read by `virsh` itself (running as `k8s`), never
+directly by the VM, so an ordinary temp location is fine:
+
+```sh
+curl -L -o /tmp/noble-base.img \
+  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
+```
+
+Getting it *into* the pool, though, does need to go through libvirt's API
+— `vol-create-as` allocates an empty volume of a given size inside the
+pool, and `vol-upload` streams a local file's bytes into it:
+
+```sh
+virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
+  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
+virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
+rm /tmp/noble-base.img
+```
+
+The VM itself shouldn't boot directly off this base image — if it did,
+every write the OS makes (logs, package installs, k3s's own state) would
+permanently modify the one shared base file, corrupting it for any future
+VM built from the same base. Instead, create a **copy-on-write overlay**:
+a second volume that starts out empty and only stores the *differences*
+from its backing volume. Reads that haven't been changed transparently
+fall through to the base image; writes go into the overlay. This is the
+same relationship a Docker image layer has to its base layer. (Same
+relationship as `terraform/main.tf`'s `libvirt_volume.base` /
+`libvirt_volume.node` pair in stage 4 — one shared base, one overlay per
+node.)
+
+```sh
+virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
+  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2
+
+virsh -c qemu:///system vol-list --pool default   # should list both volumes now
+```
+
+## cloud-init: how a stock image becomes *this* VM
+
+The base image is a generic Ubuntu install — it has no idea it's about to
+become a k3s node, and has no user account you could log into. **cloud-init**
+is the standard mechanism cloud images use to configure themselves on
+first boot, driven by data supplied externally rather than baked into the
+image. `virt-install`'s `--cloud-init` flag builds a small ISO (the
+"NoCloud" datasource) containing that data and attaches it to the VM;
+cloud-init, already installed in the image, detects it automatically at
+boot. (`terraform/main.tf`'s `libvirt_cloudinit_disk` resource in stage 4
+is the same mechanism, just built by Terraform instead of by hand.)
+
+Two separate pieces of data go in, and they answer different questions:
+
+- **`user-data`** — *what should exist on this machine*: users, packages,
+  commands to run. Written as `#cloud-config` YAML.
+- **`network-config`** — *how should this machine's network be set up*.
+  This stage skips it entirely and lets the VM get an address via DHCP
+  from the `default` network instead (stage 4's Terraform config, by
+  contrast, uses this for static IPs, since it manages its own isolated
+  network).
+
+Build `user-data` up piece by piece. Start with identity:
+
+```yaml
+#cloud-config
+hostname: k3s-manual
+manage_etc_hosts: true
+```
+
+Then the one thing you actually need to log in and administer this box —
+a user, with your public key rather than a password (cloud images have no
+default password, and SSH password auth is normally disabled anyway), and
+passwordless sudo so you're not stuck typing a password you never set:
+
+```yaml
+users:
+  - name: k3s
+    groups: sudo
+    shell: /bin/bash
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    ssh_authorized_keys:
+      - <contents of ~/.ssh/id_ed25519.pub>
+```
+
+Then the actual payload — install k3s, and stage a copy of its
+auto-generated kubeconfig somewhere the `k3s` user can read (by default
+it's only readable by root):
+
+```yaml
+package_update: true
+packages:
+  - curl
+
+runcmd:
+  - curl -sfL https://get.k3s.io | sh -s - server
+  - mkdir -p /home/k3s/.kube
+  - k3s kubectl config view --raw > /home/k3s/.kube/config
+  - chown -R k3s:k3s /home/k3s/.kube
+```
+
+Assemble those three pieces into one file:
+
+```sh
+mkdir -p ~/vms
+cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
+#cloud-config
+hostname: k3s-manual
+manage_etc_hosts: true
+
+users:
+  - name: k3s
+    groups: sudo
+    shell: /bin/bash
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    ssh_authorized_keys:
+      - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE
+
+package_update: true
+packages:
+  - curl
+
+runcmd:
+  - curl -sfL https://get.k3s.io | sh -s - server
+  - mkdir -p /home/k3s/.kube
+  - k3s kubectl config view --raw > /home/k3s/.kube/config
+  - chown -R k3s:k3s /home/k3s/.kube
+EOF
+```
+
+Unlike the disk, this file is fine sitting under `~/vms` — it's only ever
+read client-side by `virt-install` (as `k8s`), which hands the resulting
+seed data to libvirtd over the API; `libvirt-qemu` never touches it
+directly.
+
+## Creating the VM
+
+Each `virt-install` flag is answering a specific question:
+
+| Flag | Answers |
+|---|---|
+| `--connect qemu:///system` | which libvirt instance (see above) |
+| `--name` | the domain's name, used everywhere else (`virsh`, `domifaddr`, teardown) |
+| `--memory` / `--vcpus` | resource allocation — kept small deliberately |
+| `--disk vol=default/k3s-manual.qcow2` | use the pool-managed overlay, not a raw path |
+| `--import` | boot the disk as-is rather than running an OS installer against it |
+| `--os-variant` | a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed |
+| `--network network=default` | attach to the NAT network from earlier |
+| `--cloud-init user-data=...` | the file just built |
+| `--graphics none` | no VNC/spice display — this is a headless server VM |
+| `--noautoconsole` | don't attach to its console interactively after creation |
+
+Two gotchas worth knowing before running this:
+
+- **`$HOME`, not `~`, in the `--cloud-init` argument.** Bash only expands
+  `~` at the very start of a word; `user-data=~/vms/...` is *inside* a
+  word (after `=`), so the tilde would be passed through literally and
+  `virt-install` would fail looking for a file called `~`. `$HOME`
+  expands regardless of position.
+- **`--os-variant` may need to be an older release than the actual image.**
+  `osinfo-db` (the database `virt-install` validates this against) can lag
+  behind real Ubuntu releases — if `ubuntu24.04` comes back "unknown", check
+  what's actually available with `osinfo-query os | grep -i ubuntu` and use
+  the newest one it recognizes. Since `--import` just boots the disk as-is,
+  this hint doesn't change what's actually installed.
+
+```sh
+virt-install \
+  --connect qemu:///system \
+  --name k3s-manual \
+  --memory 2048 \
+  --vcpus 2 \
+  --disk vol=default/k3s-manual.qcow2 \
+  --import \
+  --os-variant ubuntu22.04 \
+  --network network=default \
+  --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \
+  --graphics none \
+  --noautoconsole
+```
+
+## Finding the VM and connecting
+
+```sh
+virsh -c qemu:///system domifaddr k3s-manual
+```
+
+This prints something like `192.168.122.67/24` — the `/24` is CIDR
+notation for the subnet mask (`255.255.255.0`), describing the *network*
+this address belongs to, not part of the address itself. Use just the
+plain IP (`192.168.122.67`) to actually connect.
+
+Give cloud-init a couple of minutes after `Domain creation completed`
+before it's reachable — it's installing k3s in the background.
+
+This address comes from the `default` network's DHCP server, leased
+against the VM's MAC address (shown in the same `domifaddr` output). DHCP
+leases are "sticky" in practice — the VM will keep asking for and getting
+the same address on renewal — but that's not the same as *guaranteed*
+fixed. Pin it explicitly if you don't want to risk it changing later:
+
+```sh
+virsh -c qemu:///system net-update default add ip-dhcp-host \
+  "<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config
+```
+
+## kubectl: what's actually in a kubeconfig
+
+A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate
+(so your client trusts the API server's identity) plus a client
+certificate and private key (so the API server trusts yours). k3s
+generates one for itself pointed at `127.0.0.1`, which only works from
+inside the VM — the `sed` below swaps that for the VM's real address so
+it works from the T630 instead:
+
+```sh
+mkdir -p ~/.kube
+ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
+  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual
+```
+
+`kubectl` itself isn't installed anywhere yet. Rather than a system
+package (which would need `sudo`, which `k8s` doesn't have), grab the
+official binary release straight from Kubernetes' own distribution point
+and drop it somewhere already on `k8s`'s `PATH`:
+
+```sh
+curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
+chmod +x kubectl
+mkdir -p ~/.local/bin
+mv kubectl ~/.local/bin/
+echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
+export PATH="$HOME/.local/bin:$PATH"
+
+export KUBECONFIG=~/.kube/config-manual
+kubectl get nodes   # expect 1 Ready node
+```
+
+A real, working single-node k3s cluster. Next: `docs/03-flux.md` — you'll
+need bootstrap steps 3-4 (the `k3s_token` secret isn't used until stage 4,
+but the Forgejo tokens are needed starting now) before continuing.