cloud-demo/docs/02-k3s.md

Stage 2: A real k3s cluster, built by hand

Stage 1 (docs/01-bootstrap.md) got KVM/libvirt installed and the k8s user created — that's all this stage needs (its steps 1-2; skip 3-4 for now, they're not needed until stage 3). This stage builds one throwaway VM directly with virt-install — no Terraform involved at all — and installs k3s on it, so stage 3 has a real cluster to bootstrap Flux/GitOps against. Terraform doesn't show up until stage 4, once you already understand what it's automating.

Everything here explains the why, not just the what: read each section before running its commands, and where a config file is being built (the cloud-init data, the Caddy block later on), treat the snippets as pieces to assemble into your own file, not something to paste wholesale.

Everything below runs as k8s on the T630 (sudo -iu k8s).

---

Why one node is enough

A k3s server node runs the control plane (API server, scheduler, etcd/SQLite) and schedules ordinary workloads onto itself unless you explicitly disable that. So a single server, with no agents, is already a complete, working cluster — nothing here needs a join token or a second VM. The 3-node design in docs/04-tofu.md exists to mirror a more realistic multi-node cluster once Terraform can build it repeatedly and disposably; for learning k3s, Helm, and Flux/GitOps, that extra shape doesn't buy you anything yet.

Two libvirt connections, and why it matters

libvirt isn't one daemon with one namespace of VMs — from a client's perspective there are (at least) two separate connections:

• qemu:///system — the shared, host-wide instance. VMs here can use privileged networking (bridges, NAT with DHCP), and management access is gated by group membership (libvirt/kvm) checked via polkit — which is exactly what bootstrap step 2's usermod -aG libvirt,kvm k8s set up. No sudo needed for any command below; that group membership is the authorization.
• qemu:///session — a private, per-user instance with no special privileges, and critically, its own separate storage pools and networks that don't overlap with the system instance at all.

For a non-root user, virsh/virt-install default to session unless told otherwise. This matters a lot in practice: it's easy to set LIBVIRT_DEFAULT_URI=qemu:///system in one terminal, run a command in a different terminal where it isn't set, and have that command silently create something under session instead — where it's invisible to everything else you're doing. Every command below uses -c qemu:///system / --connect qemu:///system explicitly for exactly this reason, rather than relying on the environment variable.

Storage: why a raw path in ~ doesn't work

The most natural first instinct is to put a VM's disk file somewhere in k8s's home directory and point --disk at it directly. That fails non-obviously: under qemu:///system, the actual QEMU process backing a VM doesn't run as k8s — it runs as a separate, restricted libvirt-qemu user (a deliberate security boundary, so a compromised VM process has its own limited identity rather than the identity of whoever created it). k8s's home directory defaults to mode 700 — readable only by k8s — so libvirt-qemu can't read into it at all, and the VM fails at boot.

The fix is to let libvirt manage the storage itself, in a pool. A pool is just a named, libvirt-tracked location for disk images (a directory, in the simplest case). The key property: creating or writing a volume inside a pool goes through libvirtd's API, not through k8s's own filesystem permissions — so it's libvirtd (already running with the right privileges) that handles ownership correctly, regardless of what user asked for it. (Stage 4's Terraform config hits this exact same problem and solves it the same way — see terraform/main.tf's libvirt_pool resource.)

Check what pools/networks already exist:

virsh -c qemu:///system pool-list --all
virsh -c qemu:///system net-list --all

On this box, net-list showed a default network already defined (just inactive), but pool-list came back completely empty — Debian's libvirt-daemon-system package does not auto-create a default storage pool, unlike some other distros' packaging. Both need fixing before anything else:

# only if pool-list was empty
virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default

# only if net-list showed 'default' as inactive
virsh -c qemu:///system net-start default
virsh -c qemu:///system net-autostart default

pool-autostart/net-autostart mean both come back up automatically after a host reboot — without it, they'd need manually starting again every time.

Building the base image and the VM's own disk

Downloading the cloud image doesn't need to go through the pool — this copy is only ever read by virsh itself (running as k8s), never directly by the VM, so an ordinary temp location is fine:

curl -L -o /tmp/noble-base.img \
  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

Getting it into the pool, though, does need to go through libvirt's API — vol-create-as allocates an empty volume of a given size inside the pool, and vol-upload streams a local file's bytes into it:

virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img

The VM itself shouldn't boot directly off this base image — if it did, every write the OS makes (logs, package installs, k3s's own state) would permanently modify the one shared base file, corrupting it for any future VM built from the same base. Instead, create a copy-on-write overlay: a second volume that starts out empty and only stores the differences from its backing volume. Reads that haven't been changed transparently fall through to the base image; writes go into the overlay. This is the same relationship a Docker image layer has to its base layer. (Same relationship as terraform/main.tf's libvirt_volume.base / libvirt_volume.node pair in stage 4 — one shared base, one overlay per node.)

virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2

virsh -c qemu:///system vol-list --pool default   # should list both volumes now

cloud-init: how a stock image becomes this VM

The base image is a generic Ubuntu install — it has no idea it's about to become a k3s node, and has no user account you could log into. cloud-init is the standard mechanism cloud images use to configure themselves on first boot, driven by data supplied externally rather than baked into the image. virt-install's --cloud-init flag builds a small ISO (the "NoCloud" datasource) containing that data and attaches it to the VM; cloud-init, already installed in the image, detects it automatically at boot. (terraform/main.tf's libvirt_cloudinit_disk resource in stage 4 is the same mechanism, just built by Terraform instead of by hand.)

Two separate pieces of data go in, and they answer different questions:

• user-data — what should exist on this machine: users, packages, commands to run. Written as #cloud-config YAML.
• network-config — how should this machine's network be set up. This stage skips it entirely and lets the VM get an address via DHCP from the default network instead (stage 4's Terraform config, by contrast, uses this for static IPs, since it manages its own isolated network).

Build user-data up piece by piece. Start with identity:

#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

Then the one thing you actually need to log in and administer this box — a user, with your public key rather than a password (cloud images have no default password, and SSH password auth is normally disabled anyway), and passwordless sudo so you're not stuck typing a password you never set:

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <contents of ~/.ssh/id_ed25519.pub>

Then the actual payload — install k3s, and stage a copy of its auto-generated kubeconfig somewhere the k3s user can read (by default it's only readable by root):

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube

Assemble those three pieces into one file:

mkdir -p ~/vms
cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
EOF

Unlike the disk, this file is fine sitting under ~/vms — it's only ever read client-side by virt-install (as k8s), which hands the resulting seed data to libvirtd over the API; libvirt-qemu never touches it directly.

Creating the VM

Each virt-install flag is answering a specific question:

Flag	Answers
--connect qemu:///system	which libvirt instance (see above)
--name	the domain's name, used everywhere else (virsh, domifaddr, teardown)
--memory / --vcpus	resource allocation — kept small deliberately
--disk vol=default/k3s-manual.qcow2	use the pool-managed overlay, not a raw path
--import	boot the disk as-is rather than running an OS installer against it
--os-variant	a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed
--network network=default	attach to the NAT network from earlier
--cloud-init user-data=...	the file just built
--graphics none	no VNC/spice display — this is a headless server VM
--noautoconsole	don't attach to its console interactively after creation

Two gotchas worth knowing before running this:

• $HOME, not ~, in the --cloud-init argument. Bash only expands ~ at the very start of a word; user-data=~/vms/... is inside a word (after =), so the tilde would be passed through literally and virt-install would fail looking for a file called ~. $HOME expands regardless of position.
• --os-variant may need to be an older release than the actual image. osinfo-db (the database virt-install validates this against) can lag behind real Ubuntu releases — if ubuntu24.04 comes back "unknown", check what's actually available with osinfo-query os | grep -i ubuntu and use the newest one it recognizes. Since --import just boots the disk as-is, this hint doesn't change what's actually installed.

virt-install \
  --connect qemu:///system \
  --name k3s-manual \
  --memory 2048 \
  --vcpus 2 \
  --disk vol=default/k3s-manual.qcow2 \
  --import \
  --os-variant ubuntu22.04 \
  --network network=default \
  --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \
  --graphics none \
  --noautoconsole

Finding the VM and connecting

virsh -c qemu:///system domifaddr k3s-manual

This prints something like 192.168.122.67/24 — the /24 is CIDR notation for the subnet mask (255.255.255.0), describing the network this address belongs to, not part of the address itself. Use just the plain IP (192.168.122.67) to actually connect.

Give cloud-init a couple of minutes after Domain creation completed before it's reachable — it's installing k3s in the background.

This address comes from the default network's DHCP server, leased against the VM's MAC address (shown in the same domifaddr output). DHCP leases are "sticky" in practice — the VM will keep asking for and getting the same address on renewal — but that's not the same as guaranteed fixed. Pin it explicitly if you don't want to risk it changing later:

virsh -c qemu:///system net-update default add ip-dhcp-host \
  "<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config

kubectl: what's actually in a kubeconfig

A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate (so your client trusts the API server's identity) plus a client certificate and private key (so the API server trusts yours). k3s generates one for itself pointed at 127.0.0.1, which only works from inside the VM — the sed below swaps that for the VM's real address so it works from the T630 instead:

mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual

kubectl itself isn't installed anywhere yet. Rather than a system package (which would need sudo, which k8s doesn't have), grab the official binary release straight from Kubernetes' own distribution point and drop it somewhere already on k8s's PATH:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"

export KUBECONFIG=~/.kube/config-manual
kubectl get nodes   # expect 1 Ready node

A real, working single-node k3s cluster. Next: docs/03-flux.md — you'll need bootstrap steps 3-4 (the k3s_token secret isn't used until stage 4, but the Forgejo tokens are needed starting now) before continuing.