home-services/GITOPS.md

GitOps Runbook

Boundaries

Terraform owns Talos machine configuration, cluster bootstrap, and the Argo CD Helm release. Argo CD owns resources under infrastructure/ and services/ after the ApplicationSets are registered. The existing Compose stacks remain independent under compose-services/.

First bootstrap

1. Confirm Talos v1.14.0, a supported Kubernetes version, the control-plane endpoint, the VM install disk, and the nodes list. Mark exactly one control-plane node as bootstrap = true.
2. Confirm the Gitea URL is reachable from the cluster and its certificate is trusted.
3. Run Terraform from terraform/ using a local variable file that is not committed.
4. Verify the argocd namespace and Argo CD pods.
5. Verify the gitea-repository Secret has the Argo CD repository label and does not expose credentials in Git.
6. Apply bootstrap/applicationsets/ once.

Terraform generates one machine-secrets bundle, renders one machine configuration per role, applies the matching configuration to every node, and bootstraps only the selected control-plane node. For more than one control-plane node, cluster_endpoint must resolve to a stable VIP or load balancer rather than an individual node address.

Talos 1.13 to 1.14 upgrade

Talos 1.14 uses the Image Factory installer image. The old image ghcr.io/siderolabs/installer:v1.14.0 is not published and returns NotFound. For these amd64 KVM nodes, use the configured schematic:

export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba:v1.14.0"

Upgrade one node at a time, worker first:

talosctl upgrade --endpoints 10.0.0.246 --nodes 10.0.0.246 --image "$TALOS_INSTALLER_IMAGE" --wait
talosctl version --endpoints 10.0.0.246 --nodes 10.0.0.246

talosctl upgrade --endpoints 10.0.0.99 --nodes 10.0.0.99 --image "$TALOS_INSTALLER_IMAGE" --drain=false --wait
talosctl version --endpoints 10.0.0.99 --nodes 10.0.0.99
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246

Set --endpoints to the same address as --nodes for each upgrade. This prevents Talosctl from attempting request forwarding through the other node during the version-compatibility check. The control-plane upgrade uses --drain=false because this single-control-plane cluster cannot safely drain through a Kubernetes kubeconfig during the Talos transition.

After both nodes report 1.14, set talos_version = "v1.14.0" in terraform.tfvars, then run terraform plan and terraform apply to reconcile the generated machine configurations and GitOps workloads. Do not reset or recreate the VMs during an in-place upgrade.

CLI credentials and outputs

After terraform apply completes, export the sensitive Terraform outputs to local files for the Talos and Kubernetes CLI tools. Do not commit or share these files:

cd terraform

export KUBECONFIG="$HOME/.kube/home-services"
mkdir -p "$(dirname "$KUBECONFIG")"
terraform output -raw kubeconfig > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
kubectl cluster-info

export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
mkdir -p "$(dirname "$TALOSCONFIG")"
terraform output -raw talosconfig > "$TALOSCONFIG"
chmod 600 "$TALOSCONFIG"
talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version

Terraform state contains the Talos client credentials and Kubernetes client certificates. Protect the state backend and local state files. The kubeconfig and talosconfig outputs are marked sensitive, but terraform output -raw intentionally writes their contents to the terminal or redirected file.

Argo CD is installed as an internal ClusterIP service. Use the generated port-forward command to connect the Argo CD CLI:

argocd login --port-forward --port-forward-namespace argocd argocd-server
argocd app list

The initial admin password can be retrieved with:

kubectl -n argocd get secret argocd-initial-admin-secret \
	-o jsonpath='{.data.password}' | base64 -d

Useful output values include talos_nodes, argocd_namespace, argocd_server_service, argocd_cli_port_forward_command, and argocd_admin_password_command.

Talos certificate mismatch

talos_machine_secrets.cluster generates the Talos CA and client certificate used by Terraform. The nodes must have been configured from that same secrets bundle. An error such as x509: certificate signed by unknown authority means the node at the configured IP was initialized with a different Talos CA, or the Terraform state was replaced.

Do not disable TLS verification and do not regenerate talos_machine_secrets when preserving the cluster. Recover the original machine-secrets bundle and its Terraform state, then retry. If the cluster is disposable, remove the old Talos installation and Terraform state together and bootstrap the nodes again from the newly generated secrets. Verify the target IPs before either action; destroying state alone does not change the CA installed on a node.

Reconciliation order

Cilium must become healthy before workloads depending on its networking features. Cilium requires host networking, host paths, privileged containers, and additional Linux capabilities, so infrastructure/cilium/templates/namespace.yaml labels only the Cilium namespace with Pod Security privileged enforcement. On Talos, the machine configuration also removes the default Flannel CNI and kube-proxy, disables Cilium cgroup auto-mounting, uses KubePrism on localhost:7445, drops SYS_MODULE, and enables legacy host routing for DNS compatibility.

If Cilium is unhealthy, it adds node.cilium.io/agent-not-ready taints and CoreDNS may remain Pending. That leaves Argo CD unable to resolve argocd-repo-server. After the corrected Cilium configuration is committed and pushed, temporarily allow CoreDNS to schedule so Argo CD can reconcile:

export KUBECONFIG="$HOME/.kube/home-services"
kubectl taint nodes --all node.cilium.io/agent-not-ready:NoSchedule-
kubectl -n kube-system rollout status deployment/coredns

Then refresh the Cilium Application and verify the agents before relying on Argo CD again:

kubectl -n argocd annotate application infrastructure-cilium argocd.argoproj.io/refresh=hard --overwrite
kubectl -n cilium get pods -o wide
kubectl -n argocd get application infrastructure-cilium

Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.

SOPS workflow

Use an age recipient in .sops.yaml, encrypt secret manifests as *.sops.yaml, and provision the matching private key out-of-band. Do not place age private keys, plaintext values, tokens, or passwords in this repository.

Recovery

If Gitea is unavailable, existing resources remain at their last reconciled state but new changes cannot be applied. Restore repository access, inspect Argo CD Application health, and allow reconciliation to resume. For a workload rollback, revert the Git commit and let Argo CD reconcile the known-good revision.

Two-node limitation

This is not a highly available control plane with one control-plane node. Storage replica counts are intentionally set to one until a second suitable storage node and disk policy are confirmed.