cloud-demo/docs/QUICKSTART.md

Quickstart: manual cluster (no Terraform)

Gets a real k3s cluster running today, by hand, so you can learn/practice Flux and GitOps immediately instead of waiting on the Terraform/libvirt provider work in docs/SETUP.md to be sorted out. One throwaway VM, created directly with virt-install rather than the dmacvicar/libvirt Terraform provider. When the Terraform track is ready, tear this down (last section) and provision the "real" 3-node cluster properly instead.

Shares steps 1–4 of docs/SETUP.md as prerequisites — do those first if you haven't:

• Step 1: KVM/libvirt installed on the T630.
• Step 2: the unprivileged k8s user exists, with ~/.ssh/k3s_homelab generated.
• Step 3: openssl rand -hex 32 isn't needed here (no agents joining, so no cluster token) — skip it.
• Step 4: cloud-demo pushed to Forgejo, with the flux-write token generated.

Everything below runs as k8s on the T630 (sudo -iu k8s).

1. Create the VM

No sudo needed anywhere in this guide — k8s's libvirt/kvm group membership (step 2) is what authorizes talking to libvirt at all, via polkit. But group membership alone doesn't solve everything: system libvirt runs actual VMs as a separate, restricted libvirt-qemu user, not as k8s — so a disk image sitting under k8s's home directory (mode 700 by default) would fail at boot, since that user can't read into k8s's home at all. The fix is to let libvirt manage the disk storage itself, inside its own pool, rather than pointing at a raw path in ~ — pool operations go through libvirtd's API, so it's libvirtd (already running with the right privileges) that handles the file ownership, not k8s directly.

Make sure everything below targets the system instance — matches where Terraform will eventually create the real VMs too:

export LIBVIRT_DEFAULT_URI=qemu:///system

This only lasts for the current shell — add it to k8s's ~/.bashrc too (echo 'export LIBVIRT_DEFAULT_URI=qemu:///system' >> ~/.bashrc) so it's still set in step 2 and the teardown section later, even in a fresh login.

Download the cloud image somewhere k8s can read it — this location itself doesn't need to be qemu-readable, since it's only ever read by virsh (running as k8s), never directly by the VM:

curl -L -o /tmp/noble-base.img \
  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

Import it into libvirt's default storage pool (created automatically when libvirt-daemon-system was installed in step 1), then create a copy-on-write overlay on top of it, sized up to 20G:

virsh vol-create-as default k3s-manual-base.qcow2 \
  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img

virsh vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2

Write the cloud-init user-data — paste in the contents of ~/.ssh/k3s_homelab.pub where marked:

cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - PASTE ~/.ssh/k3s_homelab.pub CONTENTS HERE

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
EOF

This is a single, standalone server — no join token, no agents, no custom network (uses libvirt's default NAT network + DHCP). A k3s server node schedules workloads on itself by default, so this alone is a complete, usable cluster.

The cloud-init user-data file itself is fine living under ~/vms — unlike the disk, it's only ever read client-side by virt-install (as k8s), which then hands the rendered seed data to libvirtd over the API:

virt-install \
  --name k3s-manual \
  --memory 2048 \
  --vcpus 2 \
  --disk vol=default/k3s-manual.qcow2 \
  --import \
  --os-variant ubuntu24.04 \
  --network network=default \
  --cloud-init user-data=~/vms/k3s-manual-user-data.yaml \
  --graphics none \
  --noautoconsole

2. Find its IP and get kubectl talking to it

Give cloud-init ~2 minutes to finish installing k3s after the VM boots.

virsh domifaddr k3s-manual   # note the IP under the default network

ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual

export KUBECONFIG=~/.kube/config-manual
kubectl get nodes   # expect 1 Ready node

3. Bootstrap Flux against Forgejo

Same repo, same token, same target path as the full guide would use — nothing about this is cluster-specific:

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-manual

4. Continue with the shared steps

From here, docs/SETUP.md steps 8 onward apply exactly as written, regardless of how the cluster was created — verifying the podinfo GitOps loop, registering the Forgejo Actions runner, exposing apps through Caddy, the Dashboard, remote kubectl access. Just use ~/.kube/config-manual as the kubeconfig throughout instead of the Terraform-provisioned one.

Tearing this down

Once the Terraform track in docs/SETUP.md is ready and you tofu apply the real 3-node cluster, remove this one:

# as k8s
export LIBVIRT_DEFAULT_URI=qemu:///system
virsh destroy k3s-manual              # stop it
virsh undefine k3s-manual --remove-all-storage   # delete VM + its overlay disk
virsh vol-delete --pool default k3s-manual-base.qcow2   # the base image isn't
                                                          # attached to the VM
                                                          # directly, so it
                                                          # needs its own delete
rm ~/.kube/config-manual
rm -rf ~/vms

Nothing else needs cleaning up — Flux's state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, deploy tokens, and the Forgejo Actions runner registration are all cluster-independent and carry over to the real cluster unchanged; just re-run flux bootstrap against its kubeconfig once it's up.