cloud-demo/docs/02-k3s.md

# Stage 2: A real k3s cluster, built by hand

Stage 1 (`docs/01-bootstrap.md`) got KVM/libvirt installed and the `k8s`
user created — that's all this stage needs (its steps 1-2; skip 3-4 for
now, they're not needed until stage 3). This stage builds one throwaway VM
directly with `virt-install` — no Terraform involved at all — and installs
k3s on it, so stage 3 has a real cluster to bootstrap Flux/GitOps against.
Terraform doesn't show up until stage 4, once you already understand what
it's automating.

Everything here explains the *why*, not just the *what*: read each section
before running its commands, and where a config file is being built (the
cloud-init data, the Caddy block later on), treat the snippets as pieces to
assemble into your own file, not something to paste wholesale.

Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).

---

## Why one node is enough

A k3s **server** node runs the control plane (API server, scheduler,
etcd/SQLite) *and* schedules ordinary workloads onto itself unless you
explicitly disable that. So a single server, with no agents, is already a
complete, working cluster — nothing here needs a join token or a second
VM. The 3-node design in `docs/04-tofu.md` exists to mirror a more
realistic multi-node cluster once Terraform can build it repeatedly and
disposably; for learning k3s, Helm, and Flux/GitOps, that extra shape
doesn't buy you anything yet.

## Two libvirt connections, and why it matters

libvirt isn't one daemon with one namespace of VMs — from a client's
perspective there are (at least) two separate connections:

- `qemu:///system` — the shared, host-wide instance. VMs here can use
  privileged networking (bridges, NAT with DHCP), and management access is
  gated by group membership (`libvirt`/`kvm`) checked via polkit — which is
  exactly what bootstrap step 2's `usermod -aG libvirt,kvm k8s` set up. No
  sudo needed for any command below; that group membership *is* the
  authorization.
- `qemu:///session` — a private, per-user instance with no special
  privileges, and critically, its own separate storage pools and networks
  that don't overlap with the system instance at all.

For a non-root user, `virsh`/`virt-install` **default to `session`** unless
told otherwise. This matters a lot in practice: it's easy to set
`LIBVIRT_DEFAULT_URI=qemu:///system` in one terminal, run a command in a
different terminal where it isn't set, and have that command silently
create something under `session` instead — where it's invisible to
everything else you're doing. Every command below uses `-c qemu:///system`
/ `--connect qemu:///system` explicitly for exactly this reason, rather
than relying on the environment variable.

## Storage: why a raw path in `~` doesn't work

The most natural first instinct is to put a VM's disk file somewhere in
`k8s`'s home directory and point `--disk` at it directly. That fails
non-obviously: under `qemu:///system`, the actual QEMU process backing a
VM doesn't run as `k8s` — it runs as a separate, restricted `libvirt-qemu`
user (a deliberate security boundary, so a compromised VM process has its
own limited identity rather than the identity of whoever created it).
`k8s`'s home directory defaults to mode `700` — readable only by `k8s` —
so `libvirt-qemu` can't read into it at all, and the VM fails at boot.

The fix is to let libvirt manage the storage itself, in a **pool**. A pool
is just a named, libvirt-tracked location for disk images (a directory, in
the simplest case). The key property: creating or writing a volume inside
a pool goes through libvirtd's API, not through `k8s`'s own filesystem
permissions — so it's libvirtd (already running with the right privileges)
that handles ownership correctly, regardless of what user asked for it.
(Stage 4's Terraform config hits this exact same problem and solves it the
same way — see `terraform/main.tf`'s `libvirt_pool` resource.)

Check what pools/networks already exist:

```sh
virsh -c qemu:///system pool-list --all
virsh -c qemu:///system net-list --all
```

On this box, `net-list` showed a `default` network already defined (just
inactive), but `pool-list` came back completely empty — **Debian's
`libvirt-daemon-system` package does not auto-create a `default` storage
pool**, unlike some other distros' packaging. Both need fixing before
anything else:

```sh
# only if pool-list was empty
virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default

# only if net-list showed 'default' as inactive
virsh -c qemu:///system net-start default
virsh -c qemu:///system net-autostart default
```

`pool-autostart`/`net-autostart` mean both come back up automatically after
a host reboot — without it, they'd need manually starting again every time.

## Building the base image and the VM's own disk

Downloading the cloud image doesn't need to go through the pool — this
copy is only ever read by `virsh` itself (running as `k8s`), never
directly by the VM, so an ordinary temp location is fine:

```sh
curl -L -o /tmp/noble-base.img \
  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
```

Getting it *into* the pool, though, does need to go through libvirt's API
— `vol-create-as` allocates an empty volume of a given size inside the
pool, and `vol-upload` streams a local file's bytes into it:

```sh
virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img
```

The VM itself shouldn't boot directly off this base image — if it did,
every write the OS makes (logs, package installs, k3s's own state) would
permanently modify the one shared base file, corrupting it for any future
VM built from the same base. Instead, create a **copy-on-write overlay**:
a second volume that starts out empty and only stores the *differences*
from its backing volume. Reads that haven't been changed transparently
fall through to the base image; writes go into the overlay. This is the
same relationship a Docker image layer has to its base layer. (Same
relationship as `terraform/main.tf`'s `libvirt_volume.base` /
`libvirt_volume.node` pair in stage 4 — one shared base, one overlay per
node.)

```sh
virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2

virsh -c qemu:///system vol-list --pool default   # should list both volumes now
```

## cloud-init: how a stock image becomes *this* VM

The base image is a generic Ubuntu install — it has no idea it's about to
become a k3s node, and has no user account you could log into. **cloud-init**
is the standard mechanism cloud images use to configure themselves on
first boot, driven by data supplied externally rather than baked into the
image. `virt-install`'s `--cloud-init` flag builds a small ISO (the
"NoCloud" datasource) containing that data and attaches it to the VM;
cloud-init, already installed in the image, detects it automatically at
boot. (`terraform/main.tf`'s `libvirt_cloudinit_disk` resource in stage 4
is the same mechanism, just built by Terraform instead of by hand.)

Two separate pieces of data go in, and they answer different questions:

- **`user-data`** — *what should exist on this machine*: users, packages,
  commands to run. Written as `#cloud-config` YAML.
- **`network-config`** — *how should this machine's network be set up*.
  This stage skips it entirely and lets the VM get an address via DHCP
  from the `default` network instead (stage 4's Terraform config, by
  contrast, uses this for static IPs, since it manages its own isolated
  network).

Build `user-data` up piece by piece. Start with identity:

```yaml
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true
```

Then the one thing you actually need to log in and administer this box —
a user, with your public key rather than a password (cloud images have no
default password, and SSH password auth is normally disabled anyway), and
passwordless sudo so you're not stuck typing a password you never set:

```yaml
users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <contents of ~/.ssh/id_ed25519.pub>
```

Then the actual payload — install k3s, and stage a copy of its
auto-generated kubeconfig somewhere the `k3s` user can read (by default
it's only readable by root):

```yaml
package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
```

Assemble those three pieces into one file:

```sh
mkdir -p ~/vms
cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
EOF
```

Unlike the disk, this file is fine sitting under `~/vms` — it's only ever
read client-side by `virt-install` (as `k8s`), which hands the resulting
seed data to libvirtd over the API; `libvirt-qemu` never touches it
directly.

## Creating the VM

Each `virt-install` flag is answering a specific question:

| Flag | Answers |
|---|---|
| `--connect qemu:///system` | which libvirt instance (see above) |
| `--name` | the domain's name, used everywhere else (`virsh`, `domifaddr`, teardown) |
| `--memory` / `--vcpus` | resource allocation — kept small deliberately |
| `--disk vol=default/k3s-manual.qcow2` | use the pool-managed overlay, not a raw path |
| `--import` | boot the disk as-is rather than running an OS installer against it |
| `--os-variant` | a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed |
| `--network network=default` | attach to the NAT network from earlier |
| `--cloud-init user-data=...` | the file just built |
| `--graphics none` | no VNC/spice display — this is a headless server VM |
| `--noautoconsole` | don't attach to its console interactively after creation |

Two gotchas worth knowing before running this:

- **`$HOME`, not `~`, in the `--cloud-init` argument.** Bash only expands
  `~` at the very start of a word; `user-data=~/vms/...` is *inside* a
  word (after `=`), so the tilde would be passed through literally and
  `virt-install` would fail looking for a file called `~`. `$HOME`
  expands regardless of position.
- **`--os-variant` may need to be an older release than the actual image.**
  `osinfo-db` (the database `virt-install` validates this against) can lag
  behind real Ubuntu releases — if `ubuntu24.04` comes back "unknown", check
  what's actually available with `osinfo-query os | grep -i ubuntu` and use
  the newest one it recognizes. Since `--import` just boots the disk as-is,
  this hint doesn't change what's actually installed.

```sh
virt-install \
  --connect qemu:///system \
  --name k3s-manual \
  --memory 2048 \
  --vcpus 2 \
  --disk vol=default/k3s-manual.qcow2 \
  --import \
  --os-variant ubuntu22.04 \
  --network network=default \
  --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \
  --graphics none \
  --noautoconsole
```

## Finding the VM and connecting

```sh
virsh -c qemu:///system domifaddr k3s-manual
```

This prints something like `192.168.122.67/24` — the `/24` is CIDR
notation for the subnet mask (`255.255.255.0`), describing the *network*
this address belongs to, not part of the address itself. Use just the
plain IP (`192.168.122.67`) to actually connect.

Give cloud-init a couple of minutes after `Domain creation completed`
before it's reachable — it's installing k3s in the background.

This address comes from the `default` network's DHCP server, leased
against the VM's MAC address (shown in the same `domifaddr` output). DHCP
leases are "sticky" in practice — the VM will keep asking for and getting
the same address on renewal — but that's not the same as *guaranteed*
fixed. Pin it explicitly if you don't want to risk it changing later:

```sh
virsh -c qemu:///system net-update default add ip-dhcp-host \
  "<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config
```

## kubectl: what's actually in a kubeconfig

A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate
(so your client trusts the API server's identity) plus a client
certificate and private key (so the API server trusts yours). k3s
generates one for itself pointed at `127.0.0.1`, which only works from
inside the VM — the `sed` below swaps that for the VM's real address so
it works from the T630 instead:

```sh
mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual
```

`kubectl` itself isn't installed anywhere yet. Rather than a system
package (which would need `sudo`, which `k8s` doesn't have), grab the
official binary release straight from Kubernetes' own distribution point
and drop it somewhere already on `k8s`'s `PATH`:

```sh
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"

echo 'export KUBECONFIG=~/.kube/config-manual' >> ~/.bashrc
export KUBECONFIG=~/.kube/config-manual
kubectl get nodes   # expect 1 Ready node
```

A real, working single-node k3s cluster. Next: `docs/03-flux.md` — you'll
need bootstrap steps 3-4 (the `k3s_token` secret isn't used until stage 4,
but the Forgejo tokens are needed starting now) before continuing.