cloud-demo/docs/QUICKSTART.md

# Quickstart: manual cluster (no Terraform)

A real k3s cluster, built by hand, so Flux/GitOps can be learned right away
instead of waiting on the Terraform/libvirt provider work in `docs/SETUP.md`
to get sorted (that provider did a breaking rewrite between 0.8.x and
0.9.x, and the HCL needs writing against the real schema, not memory).

This is one throwaway VM, created directly with `virt-install` — no
Terraform involved at all. Everything here explains the *why*, not just
the *what*: read each section before running its commands, and where a
config file is being built (the cloud-init data, the Caddy block), treat
the snippets as pieces to assemble into your own file, not something to
paste wholesale.

Shares steps 1–4 of `docs/SETUP.md` as prerequisites:
- Step 1: KVM/libvirt packages installed on the T630.
- Step 2: the unprivileged `k8s` user exists, in the `libvirt`/`kvm` groups,
  with an SSH keypair at `~/.ssh/id_ed25519`.
- Step 3: skip — no join token needed here (see "one node is enough" below).
- Step 4: `cloud-demo` pushed to Forgejo, with a `k8s-readonly` and a
  `flux-write` token generated.

Everything below runs as `k8s` on the T630 (`sudo -iu k8s`).

---

## Why one node is enough

A k3s **server** node runs the control plane (API server, scheduler,
etcd/SQLite) *and* schedules ordinary workloads onto itself unless you
explicitly disable that. So a single server, with no agents, is already a
complete, working cluster — nothing here needs a join token or a second
VM. The 3-node design in `docs/SETUP.md` exists to mirror a more
realistic multi-node cluster once Terraform can build it repeatedly and
disposably; for learning Flux and GitOps, that extra shape doesn't buy
you anything yet.

## Two libvirt connections, and why it matters

libvirt isn't one daemon with one namespace of VMs — from a client's
perspective there are (at least) two separate connections:

- `qemu:///system` — the shared, host-wide instance. VMs here can use
  privileged networking (bridges, NAT with DHCP), and management access is
  gated by group membership (`libvirt`/`kvm`) checked via polkit — which is
  exactly what step 2's `usermod -aG libvirt,kvm k8s` set up. No sudo
  needed for any command below; that group membership *is* the
  authorization.
- `qemu:///session` — a private, per-user instance with no special
  privileges, and critically, its own separate storage pools and networks
  that don't overlap with the system instance at all.

For a non-root user, `virsh`/`virt-install` **default to `session`** unless
told otherwise. This matters a lot in practice: it's easy to set
`LIBVIRT_DEFAULT_URI=qemu:///system` in one terminal, run a command in a
different terminal where it isn't set, and have that command silently
create something under `session` instead — where it's invisible to
everything else you're doing. Every command below uses `-c qemu:///system`
/ `--connect qemu:///system` explicitly for exactly this reason, rather
than relying on the environment variable.

## Storage: why a raw path in `~` doesn't work

The most natural first instinct is to put a VM's disk file somewhere in
`k8s`'s home directory and point `--disk` at it directly. That fails
non-obviously: under `qemu:///system`, the actual QEMU process backing a
VM doesn't run as `k8s` — it runs as a separate, restricted `libvirt-qemu`
user (a deliberate security boundary, so a compromised VM process has its
own limited identity rather than the identity of whoever created it).
`k8s`'s home directory defaults to mode `700` — readable only by `k8s` —
so `libvirt-qemu` can't read into it at all, and the VM fails at boot.

The fix is to let libvirt manage the storage itself, in a **pool**. A pool
is just a named, libvirt-tracked location for disk images (a directory, in
the simplest case). The key property: creating or writing a volume inside
a pool goes through libvirtd's API, not through `k8s`'s own filesystem
permissions — so it's libvirtd (already running with the right privileges)
that handles ownership correctly, regardless of what user asked for it.

Check what pools/networks already exist:

```sh
virsh -c qemu:///system pool-list --all
virsh -c qemu:///system net-list --all
```

On this box, `net-list` showed a `default` network already defined (just
inactive), but `pool-list` came back completely empty — **Debian's
`libvirt-daemon-system` package does not auto-create a `default` storage
pool**, unlike some other distros' packaging. Both need fixing before
anything else:

```sh
# only if pool-list was empty
virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default

# only if net-list showed 'default' as inactive
virsh -c qemu:///system net-start default
virsh -c qemu:///system net-autostart default
```

`pool-autostart`/`net-autostart` mean both come back up automatically after
a host reboot — without it, they'd need manually starting again every time.

## Building the base image and the VM's own disk

Downloading the cloud image doesn't need to go through the pool — this
copy is only ever read by `virsh` itself (running as `k8s`), never
directly by the VM, so an ordinary temp location is fine:

```sh
curl -L -o /tmp/noble-base.img \
  https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
```

Getting it *into* the pool, though, does need to go through libvirt's API
— `vol-create-as` allocates an empty volume of a given size inside the
pool, and `vol-upload` streams a local file's bytes into it:

```sh
virsh -c qemu:///system vol-create-as default k3s-manual-base.qcow2 \
  --capacity "$(stat -c%s /tmp/noble-base.img)" --format qcow2
virsh -c qemu:///system vol-upload --pool default k3s-manual-base.qcow2 /tmp/noble-base.img --sparse
rm /tmp/noble-base.img
```

The VM itself shouldn't boot directly off this base image — if it did,
every write the OS makes (logs, package installs, k3s's own state) would
permanently modify the one shared base file, corrupting it for any future
VM built from the same base. Instead, create a **copy-on-write overlay**:
a second volume that starts out empty and only stores the *differences*
from its backing volume. Reads that haven't been changed transparently
fall through to the base image; writes go into the overlay. This is the
same relationship a Docker image layer has to its base layer.

```sh
virsh -c qemu:///system vol-create-as default k3s-manual.qcow2 20G --format qcow2 \
  --backing-vol k3s-manual-base.qcow2 --backing-vol-format qcow2

virsh -c qemu:///system vol-list --pool default   # should list both volumes now
```

## cloud-init: how a stock image becomes *this* VM

The base image is a generic Ubuntu install — it has no idea it's about to
become a k3s node, and has no user account you could log into. **cloud-init**
is the standard mechanism cloud images use to configure themselves on
first boot, driven by data supplied externally rather than baked into the
image. `virt-install`'s `--cloud-init` flag builds a small ISO (the
"NoCloud" datasource) containing that data and attaches it to the VM;
cloud-init, already installed in the image, detects it automatically at
boot.

Two separate pieces of data go in, and they answer different questions:

- **`user-data`** — *what should exist on this machine*: users, packages,
  commands to run. Written as `#cloud-config` YAML.
- **`network-config`** — *how should this machine's network be set up*.
  This quickstart skips it entirely and lets the VM get an address via
  DHCP from the `default` network instead (the Terraform track, by
  contrast, uses this for static IPs, since it manages its own isolated
  network).

Build `user-data` up piece by piece. Start with identity:

```yaml
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true
```

Then the one thing you actually need to log in and administer this box —
a user, with your public key rather than a password (cloud images have no
default password, and SSH password auth is normally disabled anyway), and
passwordless sudo so you're not stuck typing a password you never set:

```yaml
users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <contents of ~/.ssh/id_ed25519.pub>
```

Then the actual payload — install k3s, and stage a copy of its
auto-generated kubeconfig somewhere the `k3s` user can read (by default
it's only readable by root):

```yaml
package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
```

Assemble those three pieces into one file:

```sh
mkdir -p ~/vms
cat > ~/vms/k3s-manual-user-data.yaml <<'EOF'
#cloud-config
hostname: k3s-manual
manage_etc_hosts: true

users:
  - name: k3s
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - PASTE ~/.ssh/id_ed25519.pub CONTENTS HERE

package_update: true
packages:
  - curl

runcmd:
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/k3s/.kube
  - k3s kubectl config view --raw > /home/k3s/.kube/config
  - chown -R k3s:k3s /home/k3s/.kube
EOF
```

Unlike the disk, this file is fine sitting under `~/vms` — it's only ever
read client-side by `virt-install` (as `k8s`), which hands the resulting
seed data to libvirtd over the API; `libvirt-qemu` never touches it
directly.

## Creating the VM

Each `virt-install` flag is answering a specific question:

| Flag | Answers |
|---|---|
| `--connect qemu:///system` | which libvirt instance (see above) |
| `--name` | the domain's name, used everywhere else (`virsh`, `domifaddr`, teardown) |
| `--memory` / `--vcpus` | resource allocation — kept small deliberately |
| `--disk vol=default/k3s-manual.qcow2` | use the pool-managed overlay, not a raw path |
| `--import` | boot the disk as-is rather than running an OS installer against it |
| `--os-variant` | a hint for libvirt's own defaults (virtio devices, clock behavior) — not what OS actually gets installed |
| `--network network=default` | attach to the NAT network from earlier |
| `--cloud-init user-data=...` | the file just built |
| `--graphics none` | no VNC/spice display — this is a headless server VM |
| `--noautoconsole` | don't attach to its console interactively after creation |

Two gotchas worth knowing before running this:

- **`$HOME`, not `~`, in the `--cloud-init` argument.** Bash only expands
  `~` at the very start of a word; `user-data=~/vms/...` is *inside* a
  word (after `=`), so the tilde would be passed through literally and
  `virt-install` would fail looking for a file called `~`. `$HOME`
  expands regardless of position.
- **`--os-variant` may need to be an older release than the actual image.**
  `osinfo-db` (the database `virt-install` validates this against) can lag
  behind real Ubuntu releases — if `ubuntu24.04` comes back "unknown", check
  what's actually available with `osinfo-query os | grep -i ubuntu` and use
  the newest one it recognizes. Since `--import` just boots the disk as-is,
  this hint doesn't change what's actually installed.

```sh
virt-install \
  --connect qemu:///system \
  --name k3s-manual \
  --memory 2048 \
  --vcpus 2 \
  --disk vol=default/k3s-manual.qcow2 \
  --import \
  --os-variant ubuntu22.04 \
  --network network=default \
  --cloud-init user-data=$HOME/vms/k3s-manual-user-data.yaml \
  --graphics none \
  --noautoconsole
```

## Finding the VM and connecting

```sh
virsh -c qemu:///system domifaddr k3s-manual
```

This prints something like `192.168.122.67/24` — the `/24` is CIDR
notation for the subnet mask (`255.255.255.0`), describing the *network*
this address belongs to, not part of the address itself. Use just the
plain IP (`192.168.122.67`) to actually connect.

Give cloud-init a couple of minutes after `Domain creation completed`
before it's reachable — it's installing k3s in the background.

This address comes from the `default` network's DHCP server, leased
against the VM's MAC address (shown in the same `domifaddr` output). DHCP
leases are "sticky" in practice — the VM will keep asking for and getting
the same address on renewal — but that's not the same as *guaranteed*
fixed. Pin it explicitly if you don't want to risk it changing later:

```sh
virsh -c qemu:///system net-update default add ip-dhcp-host \
  "<host mac='<VM_MAC>' ip='<VM_IP>'/>" --live --config
```

## kubectl: what's actually in a kubeconfig

A kubeconfig isn't a password — it's mutual TLS: a cluster CA certificate
(so your client trusts the API server's identity) plus a client
certificate and private key (so the API server trusts yours). k3s
generates one for itself pointed at `127.0.0.1`, which only works from
inside the VM — the `sed` below swaps that for the VM's real address so
it works from the T630 instead:

```sh
mkdir -p ~/.kube
ssh k3s@<VM_IP> sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed "s/127.0.0.1/<VM_IP>/" > ~/.kube/config-manual
```

`kubectl` itself isn't installed anywhere yet. Rather than a system
package (which would need `sudo`, which `k8s` doesn't have), grab the
official binary release straight from Kubernetes' own distribution point
and drop it somewhere already on `k8s`'s `PATH`:

```sh
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"

export KUBECONFIG=~/.kube/config-manual
kubectl get nodes   # expect 1 Ready node
```

## Flux: what's actually happening, mechanism by mechanism

"Flux" isn't one program watching your repo by magic — it's a handful of
ordinary Kubernetes controllers (just Deployments, like anything else
you'd run on the cluster), each one understanding a couple of **Custom
Resource Definitions** — CRDs extend the Kubernetes API with new object
kinds, the same way `Deployment` or `Service` are built-in kinds.
`GitRepository` and `Kustomization` are two such kinds Flux adds. Nothing
about any of this is special-cased outside the normal Kubernetes API —
it's the exact same "define an object, a controller notices it and acts"
loop that runs the whole rest of Kubernetes.

Install the CLI the same no-sudo, direct-binary-release way as `kubectl`:

```sh
FLUX_VERSION=$(curl -s https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f4 | sed 's/^v//')
curl -L -o /tmp/flux.tar.gz "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz"
tar -xzf /tmp/flux.tar.gz -C ~/.local/bin flux
rm /tmp/flux.tar.gz
flux --version
```

Forgejo isn't a Flux-native provider the way GitHub/GitLab are, so this
uses the generic git bootstrap — over HTTPS with the `flux-write` token
from step 4, not SSH (Forgejo's git-SSH port turned out not to be reliably
reachable from either this desktop or the T630 — see the earlier
troubleshooting in this project's history):

```sh
flux check --pre --kubeconfig ~/.kube/config-manual

flux bootstrap git \
  --url=https://git.boglabob.com/codegit/cloud-demo \
  --branch=main \
  --path=clusters/homelab \
  --username=codegit \
  --password=<FLUX_WRITE_TOKEN> \
  --token-auth \
  --kubeconfig ~/.kube/config-manual
```

That one command did five genuinely separate things. Go look at each —
these commands work against the cluster you already bootstrapped:

**1. It installed the controllers** — plain Kubernetes Deployments, no
different in kind from anything else running on the cluster:

```sh
kubectl -n flux-system get deployments
```

`source-controller`, `kustomize-controller`, `helm-controller`,
`notification-controller`. Alongside them, it registered the CRDs those
controllers understand:

```sh
kubectl get crds | grep fluxcd
```

**2. It created a `GitRepository` object** — this is the whole "what repo
am I watching" declaration, and nothing more. `source-controller`'s job is
entirely mechanical: every `interval` (default 1m), do a real `git fetch`
against `.spec.url`/`.spec.ref`; if the commit SHA changed, package that
tree into a `.tar.gz`, and record its location in `.status.artifact`.
That's the entire job — it doesn't know or care what's *in* the repo.

```sh
kubectl -n flux-system get gitrepository flux-system -o yaml
```

Look at `.status.conditions` and `.status.artifact` — that's the result of
an actual git fetch that already happened, not a static config.

**3. It created a `Kustomization` object** (confusingly, the same name as
the `kustomization.yaml` files already sitting in `apps/podinfo/` etc. —
related but not identical). `kustomize-controller`'s job, on its own
interval: fetch the artifact `source-controller` produced, run the real
`kustomize` tool against `.spec.path` inside it (the exact same tool a
plain `kustomize build apps/podinfo` would run locally against those
`kustomization.yaml` files), and apply the resulting objects via the
Kubernetes API — the automated equivalent of you running
`kubectl apply -f <(kustomize build apps/podinfo)` yourself, on a timer,
forever. `prune: true` (set on all the `Kustomization` objects in this
repo) means it also *deletes* anything it previously created that's no
longer present in the current git state — that's what makes it
self-healing rather than just "apply once."

```sh
kubectl -n flux-system get kustomization
```

**This is also why `apps/podinfo`, `apps/hello-app`, and
`apps/kubernetes-dashboard` started deploying without you ever running
`kubectl apply` on them.** Bootstrap's own `Kustomization` watches
`clusters/homelab` with `prune: true`. `clusters/homelab/apps.yaml`
(already sitting in the repo, hand-written earlier in this project) itself
just *defines more `Kustomization` objects*, one per app — so the first
one picks it up as part of its own normal reconcile, creates those three
child `Kustomization`s, and each of *those* then does its own
fetch-and-apply against its own app directory. It's the same mechanism
recursing, not a special case.

**4. It stored your credential as a Kubernetes Secret** — `--password`
here is the `flux-write` token, not an account password. Bootstrap
base64-wraps it into a Secret that matches what the `GitRepository`'s
`.spec.secretRef` points at:

```sh
kubectl -n flux-system get secret flux-system -o yaml
```

The `password` field is base64 (`| base64 -d` to read it) — this is
exactly the same credential `git clone https://user:token@host/repo` would
use, just read by `source-controller` on every fetch instead of typed by
you once. It's never written to `k8s`'s own filesystem.

**5. It committed that config back into the repo itself** — `git pull` in
your desktop clone and look at `clusters/homelab/flux-system/`. The
`GitRepository`/`Kustomization` objects you just inspected live in the
cluster *because* those exact YAML files are committed there — so
rebuilding this cluster from scratch would mean running `flux bootstrap`
again (or even just `kubectl apply -f clusters/homelab/flux-system/`) and
landing in the identical state. The fact that Flux watches this repo is
itself declared *in* this repo.

If you want to watch a reconcile happen live rather than just inspect the
end state:

```sh
kubectl -n flux-system logs deploy/source-controller -f
```

then in another terminal, make any commit and push it — you'll see the
next fetch pick it up within the interval.

## Verifying the GitOps loop actually works

```sh
flux get kustomizations --watch
```

Watch until `podinfo`, `hello-app`, and `kubernetes-dashboard` all show
`Ready: True`, then confirm pods actually landed:

```sh
kubectl -n podinfo get pods
kubectl -n hello-app get pods
kubectl -n kubernetes-dashboard get pods
```

If that's all healthy, the entire chain — Forgejo repo → Flux → this
cluster — is working end to end with nothing manually `kubectl apply`'d.

## Exposing podinfo/hello-app through Caddy

k3s's bundled ingress controller (Traefik) is already listening on this
node's own IP, port 80, routing by the `Host:` header from each app's
`Ingress` resource (already defined in `apps/podinfo` and
`apps/hello-app`). Caddy just needs to forward matching requests there.

Two DNS records, as CNAMEs pointed at `git.boglabob.com` rather than
duplicating its IP directly — CNAME means "this name is an alias for that
one," so there's one place (that record) to update if the underlying IP
ever changes, instead of several:

```
podinfo.boglabob.com  CNAME  git.boglabob.com
hello.boglabob.com    CNAME  git.boglabob.com
```

Then in Caddy's own config, using the VM's pinned IP from earlier:

```
podinfo.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}

hello.boglabob.com {
	reverse_proxy http://<VM_IP>:80
}
```

```sh
podman exec <caddy-container> caddy reload --config /etc/caddy/Caddyfile
```

The Dashboard and the k3s API server are deliberately **not** here —
`docs/SETUP.md` steps 12–13 cover why (both are cluster-admin-capable, and
exposing either publicly is the exact pattern behind real breaches like
Tesla's 2018 incident) and how to reach them instead (`kubectl
port-forward`, and LAN/tunnel-only kubectl access). Both apply to this
cluster exactly as written there.

## Registering the Forgejo Actions runner

`docs/SETUP.md` step 9 applies as written — it's about the `k8s` user and
rootless Podman, not about which cluster exists. Worth understanding
before running it: the runner would normally get root-equivalent power
over its host via a mounted `docker.sock`; instead it runs as `k8s` itself
using rootless Podman's own API socket, and `build-hello-app.yml` builds
images with `kaniko` (no daemon, no elevated privileges needed at all) —
so nothing in that pipeline ever touches `sudo`.

## Tearing this down

Once the Terraform track is ready and `tofu apply` brings up the real
3-node cluster, remove this one:

```sh
virsh -c qemu:///system destroy k3s-manual              # stop it
virsh -c qemu:///system undefine k3s-manual --remove-all-storage   # VM + overlay disk
virsh -c qemu:///system vol-delete --pool default k3s-manual-base.qcow2   # base image isn't
                                                                            # attached to the VM
                                                                            # directly, needs its
                                                                            # own delete
rm ~/.kube/config-manual
rm -rf ~/vms
```

Nothing else needs cleaning up — Flux's own state lived entirely inside
that VM's cluster and goes away with it. The Forgejo repo, both tokens,
and the Forgejo Actions runner registration are all cluster-independent
and carry over unchanged; point `flux bootstrap` at the new cluster's
kubeconfig once it exists.