docs: fix QUICKSTART manual VM storage for system libvirt
Disk images under k8s's home directory aren't readable by the libvirt-qemu user that actually runs VMs under qemu:///system. Use libvirt's own default storage pool (vol-create-as/vol-upload) instead, so libvirtd handles file ownership rather than k8s directly, and pin LIBVIRT_DEFAULT_URI=qemu:///system throughout.
This commit is contained in:
parent
a3cfba3d6a
commit
440cb2077f
1 changed files with 53 additions and 8 deletions
|
|
@ -21,13 +21,49 @@ Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).
|
||||||
|
|
||||||
## 1. Create the VM
|
## 1. Create the VM
|
||||||
|
|
||||||
```sh
|
No sudo needed anywhere in this guide — `k8s`'s `libvirt`/`kvm` group
|
||||||
mkdir -p ~/vms
|
membership (step 2) is what authorizes talking to libvirt at all, via
|
||||||
curl -L -o ~/vms/noble-base.img \
|
polkit. But group membership alone doesn't solve everything: system libvirt
|
||||||
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
|
runs actual VMs as a separate, restricted `libvirt-qemu` user, not as
|
||||||
|
`k8s` — so a disk image sitting under `k8s`'s home directory (mode 700 by
|
||||||
|
default) would fail at boot, since that user can't read into `k8s`'s home
|
||||||
|
at all. The fix is to let libvirt manage the disk storage itself, inside
|
||||||
|
its own pool, rather than pointing at a raw path in `~` — pool operations
|
||||||
|
go through libvirtd's API, so it's libvirtd (already running with the
|
||||||
|
right privileges) that handles the file ownership, not `k8s` directly.
|
||||||
|
|
||||||
# a writable overlay on top of the base image, sized up to 20G
|
Make sure everything below targets the system instance — matches where
|
||||||
qemu-img create -f qcow2 -F qcow2 -b ~/vms/noble-base.img ~/vms/k3s-manual.qcow2 20G
|
Terraform will eventually create the real VMs too:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||||||
|
```
|
||||||
|
|
||||||
|
This only lasts for the current shell — add it to `k8s`'s `~/.bashrc` too
|
||||||
|
(`echo 'export LIBVIRT_DEFAULT_URI=qemu:///system' >> ~/.bashrc`) so it's
|
||||||
|
still set in step 2 and the teardown section later, even in a fresh login.
|
||||||
|
|
||||||
|
Download the cloud image somewhere `k8s` can read it — this location
|
||||||
|
itself doesn't need to be qemu-readable, since it's only ever read by
|
||||||
|
`virsh` (running as `k8s`), never directly by the VM:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -L -o /tmp/noble-base.img \
|
||||||
|
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
|
||||||
|
```
|
||||||
|
|
||||||
|
Import it into libvirt's `default` storage pool (created automatically
|
||||||
|
when `libvirt-daemon-system` was installed in step 1), then create a
|
||||||
|
copy-on-write overlay on top of it, sized up to 20G:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
virsh vol-create-as default k3s-manual-base.qcow2 \
|
||||||
|
--capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
|
||||||
|
virsh vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
|
||||||
|
rm /tmp/noble-base.img
|
||||||
|
|
||||||
|
virsh vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
|
||||||
|
--backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2
|
||||||
```
|
```
|
||||||
|
|
||||||
Write the cloud-init user-data — paste in the contents of
|
Write the cloud-init user-data — paste in the contents of
|
||||||
|
|
@ -64,12 +100,16 @@ network (uses libvirt's default NAT network + DHCP). A k3s server node
|
||||||
schedules workloads on itself by default, so this alone is a complete,
|
schedules workloads on itself by default, so this alone is a complete,
|
||||||
usable cluster.
|
usable cluster.
|
||||||
|
|
||||||
|
The cloud-init user-data file itself is fine living under `~/vms` — unlike
|
||||||
|
the disk, it's only ever read client-side by `virt-install` (as `k8s`),
|
||||||
|
which then hands the rendered seed data to libvirtd over the API:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
virt-install \
|
virt-install \
|
||||||
--name k3s-manual \
|
--name k3s-manual \
|
||||||
--memory 2048 \
|
--memory 2048 \
|
||||||
--vcpus 2 \
|
--vcpus 2 \
|
||||||
--disk ~/vms/k3s-manual.qcow2 \
|
--disk vol=default/k3s-manual.qcow2 \
|
||||||
--import \
|
--import \
|
||||||
--os-variant ubuntu24.04 \
|
--os-variant ubuntu24.04 \
|
||||||
--network network=default \
|
--network network=default \
|
||||||
|
|
@ -123,8 +163,13 @@ the real 3-node cluster, remove this one:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# as k8s
|
# as k8s
|
||||||
|
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||||||
virsh destroy k3s-manual # stop it
|
virsh destroy k3s-manual # stop it
|
||||||
virsh undefine k3s-manual --remove-all-storage # delete VM + its disk
|
virsh undefine k3s-manual --remove-all-storage # delete VM + its overlay disk
|
||||||
|
virsh vol-delete --pool default k3s-manual-base.qcow2 # the base image isn't
|
||||||
|
# attached to the VM
|
||||||
|
# directly, so it
|
||||||
|
# needs its own delete
|
||||||
rm ~/.kube/config-manual
|
rm ~/.kube/config-manual
|
||||||
rm -rf ~/vms
|
rm -rf ~/vms
|
||||||
```
|
```
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue