CodeGit
440cb2077f
Disk images under k8s's home directory aren't readable by the libvirt-qemu user that actually runs VMs under qemu:///system. Use libvirt's own default storage pool (vol-create-as/vol-upload) instead, so libvirtd handles file ownership rather than k8s directly, and pin LIBVIRT_DEFAULT_URI=qemu:///system throughout.
181 lines
6.4 KiB
Markdown
181 lines
6.4 KiB
Markdown
# Quickstart: manual cluster (no Terraform)
|
||
|
||
Gets a real k3s cluster running today, by hand, so you can learn/practice
|
||
Flux and GitOps immediately instead of waiting on the Terraform/libvirt
|
||
provider work in `docs/SETUP.md` to be sorted out. One throwaway VM, created
|
||
directly with `virt-install` rather than the `dmacvicar/libvirt` Terraform
|
||
provider. When the Terraform track is ready, tear this down (last section)
|
||
and provision the "real" 3-node cluster properly instead.
|
||
|
||
Shares steps 1–4 of `docs/SETUP.md` as prerequisites — do those first if you
|
||
haven't:
|
||
- Step 1: KVM/libvirt installed on the T630.
|
||
- Step 2: the unprivileged `k8s` user exists, with `~/.ssh/k3s_homelab`
|
||
generated.
|
||
- Step 3: `openssl rand -hex 32` isn't needed here (no agents joining, so no
|
||
cluster token) — skip it.
|
||
- Step 4: `cloud-demo` pushed to Forgejo, with the `flux-write` token
|
||
generated.
|
||
|
||
Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).
|
||
|
||
## 1. Create the VM
|
||
|
||
No sudo needed anywhere in this guide — `k8s`'s `libvirt`/`kvm` group
|
||
membership (step 2) is what authorizes talking to libvirt at all, via
|
||
polkit. But group membership alone doesn't solve everything: system libvirt
|
||
runs actual VMs as a separate, restricted `libvirt-qemu` user, not as
|
||
`k8s` — so a disk image sitting under `k8s`'s home directory (mode 700 by
|
||
default) would fail at boot, since that user can't read into `k8s`'s home
|
||
at all. The fix is to let libvirt manage the disk storage itself, inside
|
||
its own pool, rather than pointing at a raw path in `~` — pool operations
|
||
go through libvirtd's API, so it's libvirtd (already running with the
|
||
right privileges) that handles the file ownership, not `k8s` directly.
|
||
|
||
Make sure everything below targets the system instance — matches where
|
||
Terraform will eventually create the real VMs too:
|
||
|
||
```sh
|
||
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||
```
|
||
|
||
This only lasts for the current shell — add it to `k8s`'s `~/.bashrc` too
|
||
(`echo 'export LIBVIRT_DEFAULT_URI=qemu:///system' >> ~/.bashrc`) so it's
|
||
still set in step 2 and the teardown section later, even in a fresh login.
|
||
|
||
Download the cloud image somewhere `k8s` can read it — this location
|
||
itself doesn't need to be qemu-readable, since it's only ever read by
|
||
`virsh` (running as `k8s`), never directly by the VM:
|
||
|
||
```sh
|
||
curl -L -o /tmp/noble-base.img \
|
||
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
|
||
```
|
||
|
||
Import it into libvirt's `default` storage pool (created automatically
|
||
when `libvirt-daemon-system` was installed in step 1), then create a
|
||
copy-on-write overlay on top of it, sized up to 20G:
|
||
|
||
```sh
|
||
virsh vol-create-as default k3s-manual-base.qcow2 \
|
||
--capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
|
||
virsh vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
|
||
rm /tmp/noble-base.img
|
||
|
||
virsh vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
|
||
--backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2
|
||
```
|
||
|
||
Write the cloud-init user-data — paste in the contents of
|
||
`~/.ssh/k3s_homelab.pub` where marked:
|
||
|
||
```sh
|
||
cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
|
||
#cloud-config
|
||
hostname: k3s-manual
|
||
manage_etc_hosts: true
|
||
|
||
users:
|
||
- name: k3s
|
||
groups: sudo
|
||
shell: /bin/bash
|
||
sudo: ALL=(ALL) NOPASSWD:ALL
|
||
ssh_authorized_keys:
|
||
- PASTE ~/.ssh/k3s_homelab.pub CONTENTS HERE
|
||
|
||
package_update: true
|
||
packages:
|
||
- curl
|
||
|
||
runcmd:
|
||
- curl -sfL https://get.k3s.io | sh -s - server
|
||
- mkdir -p /home/k3s/.kube
|
||
- k3s kubectl config view --raw > /home/k3s/.kube/config
|
||
- chown -R k3s:k3s /home/k3s/.kube
|
||
EOF
|
||
```
|
||
|
||
This is a single, standalone server — no join token, no agents, no custom
|
||
network (uses libvirt's default NAT network + DHCP). A k3s server node
|
||
schedules workloads on itself by default, so this alone is a complete,
|
||
usable cluster.
|
||
|
||
The cloud-init user-data file itself is fine living under `~/vms` — unlike
|
||
the disk, it's only ever read client-side by `virt-install` (as `k8s`),
|
||
which then hands the rendered seed data to libvirtd over the API:
|
||
|
||
```sh
|
||
virt-install \
|
||
--name k3s-manual \
|
||
--memory 2048 \
|
||
--vcpus 2 \
|
||
--disk vol=default/k3s-manual.qcow2 \
|
||
--import \
|
||
--os-variant ubuntu24.04 \
|
||
--network network=default \
|
||
--cloud-init user-data=~/vms/k3s-manual-user-data.yaml \
|
||
--graphics none \
|
||
--noautoconsole
|
||
```
|
||
|
||
## 2. Find its IP and get kubectl talking to it
|
||
|
||
Give cloud-init ~2 minutes to finish installing k3s after the VM boots.
|
||
|
||
```sh
|
||
virsh domifaddr k3s-manual # note the IP under the default network
|
||
|
||
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
|
||
| sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual
|
||
|
||
export KUBECONFIG=~/.kube/config-manual
|
||
kubectl get nodes # expect 1 Ready node
|
||
```
|
||
|
||
## 3. Bootstrap Flux against Forgejo
|
||
|
||
Same repo, same token, same target path as the full guide would use —
|
||
nothing about this is cluster-specific:
|
||
|
||
```sh
|
||
flux bootstrap git \
|
||
--url=https://git.boglabob.com/codegit/cloud-demo \
|
||
--branch=main \
|
||
--path=clusters/homelab \
|
||
--username=codegit \
|
||
--password=<FLUX_WRITE_TOKEN> \
|
||
--token-auth \
|
||
--kubeconfig ~/.kube/config-manual
|
||
```
|
||
|
||
## 4. Continue with the shared steps
|
||
|
||
From here, `docs/SETUP.md` steps 8 onward apply exactly as written,
|
||
regardless of how the cluster was created — verifying the podinfo GitOps
|
||
loop, registering the Forgejo Actions runner, exposing apps through Caddy,
|
||
the Dashboard, remote kubectl access. Just use `~/.kube/config-manual` as
|
||
the kubeconfig throughout instead of the Terraform-provisioned one.
|
||
|
||
## Tearing this down
|
||
|
||
Once the Terraform track in `docs/SETUP.md` is ready and you `tofu apply`
|
||
the real 3-node cluster, remove this one:
|
||
|
||
```sh
|
||
# as k8s
|
||
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||
virsh destroy k3s-manual # stop it
|
||
virsh undefine k3s-manual --remove-all-storage # delete VM + its overlay disk
|
||
virsh vol-delete --pool default k3s-manual-base.qcow2 # the base image isn't
|
||
# attached to the VM
|
||
# directly, so it
|
||
# needs its own delete
|
||
rm ~/.kube/config-manual
|
||
rm -rf ~/vms
|
||
```
|
||
|
||
Nothing else needs cleaning up — Flux's state lived entirely inside that
|
||
VM's cluster and goes away with it. The Forgejo repo, deploy tokens, and
|
||
the Forgejo Actions runner registration are all cluster-independent and
|
||
carry over to the real cluster unchanged; just re-run `flux bootstrap`
|
||
against its kubeconfig once it's up.
|