CodeGit/cloud-demo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
cloud-demo/docs/QUICKSTART.md

9.6 KiB
Raw Blame History

Quickstart: manual cluster (no Terraform)

Gets a real k3s cluster running today, by hand, so you can learn/practice Flux and GitOps immediately instead of waiting on the Terraform/libvirt provider work in docs/SETUP.md to be sorted out. One throwaway VM, created directly with virt-install rather than the dmacvicar/libvirt Terraform provider. When the Terraform track is ready, tear this down (last section) and provision the "real" 3-node cluster properly instead.

Shares steps 1–4 of docs/SETUP.md as prerequisites — do those first if you haven't:

  • Step 1: KVM/libvirt installed on the T630.
  • Step 2: the unprivileged k8s user exists, with ~/.ssh/id_ed25519 generated.
  • Step 3: openssl rand -hex 32 isn't needed here (no agents joining, so no cluster token) — skip it.
  • Step 4: cloud-demo pushed to Forgejo, with the flux-write token generated.

Everything below runs as k8s on the T630 (sudo -iu k8s).

1. Create the VM

No sudo needed anywhere in this guide — k8s's libvirt/kvm group membership (step 2) is what authorizes talking to libvirt at all, via polkit. But group membership alone doesn't solve everything: system libvirt runs actual VMs as a separate, restricted libvirt-qemu user, not as k8s — so a disk image sitting under k8s's home directory (mode 700 by default) would fail at boot, since that user can't read into k8s's home at all. The fix is to let libvirt manage the disk storage itself, inside its own pool, rather than pointing at a raw path in ~ — pool operations go through libvirtd's API, so it's libvirtd (already running with the right privileges) that handles the file ownership, not k8s directly.

Every virsh/virt-install command below uses -c qemu:///system / --connect qemu:///system explicitly, rather than relying on the LIBVIRT_DEFAULT_URI env var — for a non-root user that env var is what picks system vs. the per-user qemu:///session instance, and it's easy for it to be set in one terminal and not another, silently causing commands to operate on the wrong (session) libvirt instance where nothing you created system-side is visible. Explicit -c on every command avoids that ambiguity entirely.

Debian's libvirt-daemon-system does not auto-create a default storage pool the way some other distros' packaging does (it does auto-create the default network, just left inactive) — check and fix both before doing anything else:

virsh -c qemu:///system pool-list --all
virsh -c qemu:///system net-list --all

If pool-list comes back empty:

virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default

If net-list shows default as inactive:

virsh -c qemu:///system net-start default
virsh -c qemu:///system net-autostart default

Download the cloud image somewhere k8s can read it — this location itself doesn't need to be qemu-readable, since it's only ever read by virsh (running as k8s), never directly by the VM:

curl -L -o /tmp/noble-base.img \
  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

Import it into the default pool, then create a copy-on-write overlay on top of it, sized up to 20G:

virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img

virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2

virsh -c qemu:///system vol-list --pool default   # should list both volumes

Write the cloud-init user-data — paste in the contents of ~/.ssh/id_ed25519.pub where marked:

mkdir -p ~/vms
cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
EOF

This is a single, standalone server — no join token, no agents, no custom network (uses libvirt's default NAT network + DHCP). A k3s server node schedules workloads on itself by default, so this alone is a complete, usable cluster.

The cloud-init user-data file itself is fine living under ~/vms — unlike the disk, it's only ever read client-side by virt-install (as k8s), which then hands the rendered seed data to libvirtd over the API. Note $HOME rather than ~ in the --cloud-init argument below — ~ only expands at the start of a shell word, not after = inside one, so user-data=~/vms/... would be passed to virt-install literally with the tilde still in it.

--os-variant ubuntu22.04 below is deliberately not ubuntu24.04 (the actual image) — osinfo-db on Debian 12 may not recognize the newer variant name yet. Check what yours knows about with osinfo-query os | grep -i ubuntu; the hint mostly just tunes libvirt's domain defaults (virtio devices, clock, etc.), so a close-enough variant against a cloud-init --import like this is harmless:

virt-install \
  --connect qemu:///system \
  --name k3s-manual \
  --memory 2048 \
  --vcpus 2 \
  --disk vol=default/k3s-manual.qcow2 \
  --import \
  --os-variant ubuntu22.04 \
  --network network=default \
  --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \
  --graphics none \
  --noautoconsole

2. Find its IP and get kubectl talking to it

Give cloud-init ~2 minutes to finish installing k3s after the VM boots.

virsh -c qemu:///system domifaddr k3s-manual   # note the IP under the default network

If you haven't installed kubectl on the T630 yet (as k8s, no sudo needed):

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"

mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual

export KUBECONFIG=~/.kube/config-manual
kubectl get nodes   # expect 1 Ready node

3. Bootstrap Flux against Forgejo

If you haven't installed the Flux CLI yet (no sudo needed):

FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version

Same repo, same token, same target path as the full guide would use — nothing about this is cluster-specific:

flux check --pre --kubeconfig ~/.kube/config-manual

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-manual

4. Continue with the shared steps

From here, docs/SETUP.md steps 8–10 and 12–13 apply exactly as written, regardless of how the cluster was created — verifying the podinfo GitOps loop, registering the Forgejo Actions runner, the Dashboard, remote kubectl access. Just use ~/.kube/config-manual as the kubeconfig throughout instead of the Terraform-provisioned one.

Step 11 (exposing apps through Caddy) does not apply as-written, though — it assumes the Terraform track's static 10.20.30.x addressing. This VM is on libvirt's default network instead, using DHCP:

virsh -c qemu:///system domifaddr k3s-manual   # note the IP and MAC address

DHCP leases are "sticky" to a MAC address in practice, but not guaranteed fixed the way the Terraform track's static IPs are — pin it explicitly so it can't change later:

virsh -c qemu:///system net-update default add ip-dhcp-host \
  "<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config

Then in Caddy's config, use that IP directly rather than docs/Caddyfile.example's multi-node placeholder blocks:

podinfo.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

hello.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

Tearing this down

Once the Terraform track in docs/SETUP.md is ready and you tofu apply the real 3-node cluster, remove this one:

# as k8s
export LIBVIRT_DEFAULT_URI=qemu:///system
virsh destroy k3s-manual              # stop it
virsh undefine k3s-manual --remove-all-storage   # delete VM + its overlay disk
virsh vol-delete --pool default k3s-manual-base.qcow2   # the base image isn't
                                                          # attached to the VM
                                                          # directly, so it
                                                          # needs its own delete
rm ~/.kube/config-manual
rm -rf ~/vms

Nothing else needs cleaning up — Flux's state lived entirely inside that VM's cluster and goes away with it. The Forgejo repo, deploy tokens, and the Forgejo Actions runner registration are all cluster-independent and carry over to the real cluster unchanged; just re-run flux bootstrap against its kubeconfig once it's up.