docs: fix QUICKSTART manual VM storage for system libvirt

Disk images under k8s's home directory aren't readable by the
libvirt-qemu user that actually runs VMs under qemu:///system. Use
libvirt's own default storage pool (vol-create-as/vol-upload) instead,
so libvirtd handles file ownership rather than k8s directly, and pin
LIBVIRT_DEFAULT_URI=qemu:///system throughout.
This commit is contained in:
CodeGit 2026-08-19 07:29:32 +01:00
parent a3cfba3d6a
commit 440cb2077f

View file

@ -21,13 +21,49 @@ Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).
## 1. Create the VM
```sh
mkdir -p ~/vms
curl -L -o ~/vms/noble-base.img \
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
No sudo needed anywhere in this guide — `k8s`'s `libvirt`/`kvm` group
membership (step 2) is what authorizes talking to libvirt at all, via
polkit. But group membership alone doesn't solve everything: system libvirt
runs actual VMs as a separate, restricted `libvirt-qemu` user, not as
`k8s` — so a disk image sitting under `k8s`'s home directory (mode 700 by
default) would fail at boot, since that user can't read into `k8s`'s home
at all. The fix is to let libvirt manage the disk storage itself, inside
its own pool, rather than pointing at a raw path in `~` — pool operations
go through libvirtd's API, so it's libvirtd (already running with the
right privileges) that handles the file ownership, not `k8s` directly.
# a writable overlay on top of the base image, sized up to 20G
qemu-img create -f qcow2 -F qcow2 -b ~/vms/noble-base.img ~/vms/k3s-manual.qcow2 20G
Make sure everything below targets the system instance — matches where
Terraform will eventually create the real VMs too:
```sh
export LIBVIRT_DEFAULT_URI=qemu:///system
```
This only lasts for the current shell — add it to `k8s`'s `~/.bashrc` too
(`echo 'export LIBVIRT_DEFAULT_URI=qemu:///system' >> ~/.bashrc`) so it's
still set in step 2 and the teardown section later, even in a fresh login.
Download the cloud image somewhere `k8s` can read it — this location
itself doesn't need to be qemu-readable, since it's only ever read by
`virsh` (running as `k8s`), never directly by the VM:
```sh
curl -L -o /tmp/noble-base.img \
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
```
Import it into libvirt's `default` storage pool (created automatically
when `libvirt-daemon-system` was installed in step 1), then create a
copy-on-write overlay on top of it, sized up to 20G:
```sh
virsh vol-create-as default k3s-manual-base.qcow2 \
--capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img
virsh vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
--backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2
```
Write the cloud-init user-data — paste in the contents of
@ -64,12 +100,16 @@ network (uses libvirt's default NAT network + DHCP). A k3s server node
schedules workloads on itself by default, so this alone is a complete,
usable cluster.
The cloud-init user-data file itself is fine living under `~/vms` — unlike
the disk, it's only ever read client-side by `virt-install` (as `k8s`),
which then hands the rendered seed data to libvirtd over the API:
```sh
virt-install \
--name k3s-manual \
--memory 2048 \
--vcpus 2 \
--disk ~/vms/k3s-manual.qcow2 \
--disk vol=default/k3s-manual.qcow2 \
--import \
--os-variant ubuntu24.04 \
--network network=default \
@ -123,8 +163,13 @@ the real 3-node cluster, remove this one:
```sh
# as k8s
export LIBVIRT_DEFAULT_URI=qemu:///system
virsh destroy k3s-manual # stop it
virsh undefine k3s-manual --remove-all-storage # delete VM + its disk
virsh undefine k3s-manual --remove-all-storage # delete VM + its overlay disk
virsh vol-delete --pool default k3s-manual-base.qcow2 # the base image isn't
# attached to the VM
# directly, so it
# needs its own delete
rm ~/.kube/config-manual
rm -rf ~/vms
```