home-services/TALOS.md

TALOS Configuration

Prior and separate to this build chain, Talos needs to be configured. Here are the steps that were completed.

Reference: https://docs.siderolabs.com/talos/v1.14/platform-specific-installations/virtualized-platforms/kvm

Note that kubectl, talosctl, and terraform were installed using homebrew prior.

# Setting up VMs
mkdir -p ~/vm/talos/configs
cd ~/vm/talos/configs
vim my-talos-net.xml
sudo virsh net-define my-talos-net.xml
sudo virsh net-start my-talos-net
sudo virsh net-autostart my-talos-net
sudo mv ~/downloads/metal-arm64.iso /var/lib/libvirt/images/
sudo chown root: /var/lib/libvirt/images/metal-amd64.iso
sudo virt-install --virt-type kvm --name control-plane-node-1 --ram 8192 --vcpus 4 --disk path=control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 --cdrom metal-amd64.iso --os-variant=linux2022 --network network=my-talos-net --boot hd,cdrom --noautoconsole
sudo virt-install --virt-type kvm --name worker-node-1 --ram 8192 --vcpus 4 --disk path=control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 --cdrom metal-amd64.iso --os-variant=linux2022 --network network=my-talos-net --boot hd,cdrom --noautoconsole

Note that the VMs are booted into the ISO and terraform initiates the install and cluster build.

Upgrade Talos from 1.13 to 1.14

Talos 1.14 no longer publishes the release installer image at ghcr.io/siderolabs/installer:v1.14.0. Use the Talos Image Factory installer image with the generic amd64 KVM schematic:

export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"

export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.14.0"

This repository's schematic is defined in terraform/talos-image-schematic.yaml and includes siderolabs/iscsi-tools and siderolabs/util-linux-tools, which Longhorn requires. Rebuild the schematic if the Talos version or extension list changes, then update both talos_installer_image in Terraform and TALOS_INSTALLER_IMAGE here. To regenerate the schematic ID:

curl -fsS -X POST https://factory.talos.dev/schematics \
  -H 'Content-Type: application/yaml' \
  --data-binary @terraform/talos-image-schematic.yaml | jq -r .id

For an existing cluster, upgrade one node at a time, worker first:

talosctl upgrade \
	--endpoints 10.0.0.246 \
	--nodes 10.0.0.246 \
	--image "$TALOS_INSTALLER_IMAGE" \
	--drain=false \
	--wait

Use --drain=false for the worker because it is the only schedulable worker and Argo CD has no second worker on which to reschedule. After the worker is healthy, upgrade the single control-plane node without draining:

talosctl upgrade \
	--endpoints 10.0.0.99 \
	--nodes 10.0.0.99 \
	--image "$TALOS_INSTALLER_IMAGE" \
	--drain=false \
	--wait

Verify both nodes and their installed extensions and volumes before reconciling Terraform:

kubectl --kubeconfig "$HOME/.kube/home-services" get nodes -o wide
talosctl get extensions --endpoints 10.0.0.246 --nodes 10.0.0.246
talosctl get extensions --endpoints 10.0.0.99 --nodes 10.0.0.99
talosctl get volumestatus --endpoints 10.0.0.246 --nodes 10.0.0.246
talosctl get volumestatus --endpoints 10.0.0.99 --nodes 10.0.0.99

Then update terraform.tfvars to talos_version = "v1.14.0" if needed and run:

terraform plan
terraform apply

Do not use talosctl reset or recreate the VMs for this upgrade. The installer preserves the Talos state and Kubernetes data.

Longhorn node prerequisites

Every node providing Longhorn storage must have the Image Factory extensions above and a dedicated data disk attached as /dev/vdb. The current VMs use a 10 GiB disk for Longhorn and a separate /dev/vda install disk. Do not select or format /dev/vda.

Talos provisions the extra disk through terraform/longhorn-user-volume.yaml as a UserVolumeConfig, mounting it at /var/mnt/longhorn. After Terraform configures the nodes and before applying the Argo CD ApplicationSets, apply the volume config to each node:

talosctl patch machineconfig \
  --endpoints 10.0.0.246 --nodes 10.0.0.246 \
  --patch @terraform/longhorn-user-volume.yaml
talosctl patch machineconfig \
  --endpoints 10.0.0.99 --nodes 10.0.0.99 \
  --patch @terraform/longhorn-user-volume.yaml

Reboot nodes one at a time, worker first. For an existing cluster, the extension upgrade above also reboots each node and can be performed after applying the volume configuration. Wait for the worker to return Ready before upgrading or rebooting the control-plane node. Verify the Talos volume is available on both nodes before allowing Longhorn to use it. Longhorn's chart sets defaultDataPath to /var/mnt/longhorn.

Cilium Install

Cilium is installed using the recommended method via helm using ArgoCD.

https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium#deploy-cilium-cni