home-services/GITOPS.md

GitOps Runbook

Boundaries

Terraform owns Talos machine configuration, cluster bootstrap, and the Argo CD Helm release. Argo CD owns resources under infrastructure/ and services/ after the ApplicationSets are registered. The existing Compose stacks remain independent under compose-services/.

First bootstrap

1. Confirm Talos v1.14.0, a supported Kubernetes version, the control-plane endpoint, the VM install disk, and the nodes list. Mark exactly one control-plane node as bootstrap = true.
2. Confirm the Gitea URL is reachable from the cluster and its certificate is trusted.
3. Run Terraform from terraform/ using a local variable file that is not committed.
4. Verify the argocd namespace and Argo CD pods.
5. Verify the gitea-repository Secret has the Argo CD repository label and does not expose credentials in Git.
6. Apply bootstrap/applicationsets/ once.

Terraform generates one machine-secrets bundle, renders one machine configuration per role, applies the matching configuration to every node, and bootstraps only the selected control-plane node. For more than one control-plane node, cluster_endpoint must resolve to a stable VIP or load balancer rather than an individual node address.

Talos certificate mismatch

talos_machine_secrets.cluster generates the Talos CA and client certificate used by Terraform. The nodes must have been configured from that same secrets bundle. An error such as x509: certificate signed by unknown authority means the node at the configured IP was initialized with a different Talos CA, or the Terraform state was replaced.

Do not disable TLS verification and do not regenerate talos_machine_secrets when preserving the cluster. Recover the original machine-secrets bundle and its Terraform state, then retry. If the cluster is disposable, remove the old Talos installation and Terraform state together and bootstrap the nodes again from the newly generated secrets. Verify the target IPs before either action; destroying state alone does not change the CA installed on a node.

Reconciliation order

Cilium must become healthy before workloads depending on its networking features. Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.

SOPS workflow

Use an age recipient in .sops.yaml, encrypt secret manifests as *.sops.yaml, and provision the matching private key out-of-band. Do not place age private keys, plaintext values, tokens, or passwords in this repository.

Recovery

If Gitea is unavailable, existing resources remain at their last reconciled state but new changes cannot be applied. Restore repository access, inspect Argo CD Application health, and allow reconciliation to resume. For a workload rollback, revert the Git commit and let Argo CD reconcile the known-good revision.

Two-node limitation

This is not a highly available control plane with one control-plane node. Storage replica counts are intentionally set to one until a second suitable storage node and disk policy are confirmed.