whentimeslows
e3c51bb3ae
Add the Talos extension image and dedicated user volume configuration, point Longhorn at the mounted disk, and document rebuild and rolling-upgrade steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
131 lines
8.6 KiB
Markdown
131 lines
8.6 KiB
Markdown
# GitOps Runbook
|
|
|
|
## Boundaries
|
|
|
|
Terraform owns Talos machine configuration, cluster bootstrap, and the Argo CD Helm release. Argo CD owns resources under `infrastructure/` and `services/` after the ApplicationSets are registered. The existing Compose stacks remain independent under `compose-services/`.
|
|
|
|
## First bootstrap
|
|
|
|
1. Confirm Talos `v1.14.0`, the matching `talos_installer_image` (including `iscsi-tools` and `util-linux-tools`), a supported Kubernetes version, the control-plane endpoint, the VM install disk, a dedicated `/dev/vdb` Longhorn data disk on every storage node, and the `nodes` list. Mark exactly one control-plane node as `bootstrap = true`.
|
|
2. Confirm the Gitea URL is reachable from the cluster and its certificate is trusted.
|
|
3. Run Terraform from `terraform/` using a local variable file that is not committed.
|
|
4. Verify the `argocd` namespace and Argo CD pods.
|
|
5. Verify the `gitea-repository` Secret has the Argo CD repository label and does not expose credentials in Git.
|
|
6. Apply `terraform/longhorn-user-volume.yaml` to each storage node with `talosctl patch machineconfig`, then reboot nodes one at a time (worker first) and verify they rejoin with the volume mounted at `/var/mnt/longhorn`:
|
|
|
|
```bash
|
|
talosctl patch machineconfig --endpoints 10.0.0.246 --nodes 10.0.0.246 --patch @terraform/longhorn-user-volume.yaml
|
|
talosctl patch machineconfig --endpoints 10.0.0.99 --nodes 10.0.0.99 --patch @terraform/longhorn-user-volume.yaml
|
|
talosctl reboot --endpoints 10.0.0.246 --nodes 10.0.0.246 --wait
|
|
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
|
|
talosctl reboot --endpoints 10.0.0.99 --nodes 10.0.0.99 --wait
|
|
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
|
|
```
|
|
|
|
7. Apply `bootstrap/applicationsets/` once.
|
|
|
|
Terraform generates one machine-secrets bundle, renders one machine configuration per role, applies the matching configuration to every node, and bootstraps only the selected control-plane node. For more than one control-plane node, `cluster_endpoint` must resolve to a stable VIP or load balancer rather than an individual node address.
|
|
|
|
The Talos installer image is generated from `terraform/talos-image-schematic.yaml`; keep its Image Factory schematic ID and Talos version aligned in the Terraform default and this runbook. The extra Longhorn data disk is provisioned separately using the checked-in UserVolumeConfig because it is a Talos resource, not a Kubernetes workload.
|
|
|
|
## Talos 1.13 to 1.14 upgrade
|
|
|
|
Talos 1.14 uses the Image Factory installer image. The old image `ghcr.io/siderolabs/installer:v1.14.0` is not published and returns `NotFound`. For these amd64 KVM nodes, use the configured schematic:
|
|
|
|
```bash
|
|
export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
|
|
export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.14.0"
|
|
```
|
|
|
|
Upgrade one node at a time, worker first:
|
|
|
|
```bash
|
|
talosctl upgrade --endpoints 10.0.0.246 --nodes 10.0.0.246 --image "$TALOS_INSTALLER_IMAGE" --drain=false --wait
|
|
talosctl version --endpoints 10.0.0.246 --nodes 10.0.0.246
|
|
|
|
talosctl upgrade --endpoints 10.0.0.99 --nodes 10.0.0.99 --image "$TALOS_INSTALLER_IMAGE" --drain=false --wait
|
|
talosctl version --endpoints 10.0.0.99 --nodes 10.0.0.99
|
|
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
|
|
```
|
|
|
|
Set `--endpoints` to the same address as `--nodes` for each upgrade. This prevents Talosctl from attempting request forwarding through the other node during the version-compatibility check. Both upgrades use `--drain=false`: there is only one schedulable worker for Argo CD, and the single-control-plane cluster cannot safely drain through a Kubernetes kubeconfig during the Talos transition.
|
|
|
|
After both nodes report 1.14, set `talos_version = "v1.14.0"` in `terraform.tfvars`, then run `terraform plan` and `terraform apply` to reconcile the generated machine configurations and GitOps workloads. Do not reset or recreate the VMs during an in-place upgrade.
|
|
|
|
## CLI credentials and outputs
|
|
|
|
After `terraform apply` completes, export the sensitive Terraform outputs to local files for the Talos and Kubernetes CLI tools. Do not commit or share these files:
|
|
|
|
```bash
|
|
cd terraform
|
|
|
|
export KUBECONFIG="$HOME/.kube/home-services"
|
|
mkdir -p "$(dirname "$KUBECONFIG")"
|
|
terraform output -raw kubeconfig > "$KUBECONFIG"
|
|
chmod 600 "$KUBECONFIG"
|
|
kubectl cluster-info
|
|
|
|
export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
|
|
mkdir -p "$(dirname "$TALOSCONFIG")"
|
|
terraform output -raw talosconfig > "$TALOSCONFIG"
|
|
chmod 600 "$TALOSCONFIG"
|
|
talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version
|
|
```
|
|
|
|
Terraform state contains the Talos client credentials and Kubernetes client certificates. Protect the state backend and local state files. The `kubeconfig` and `talosconfig` outputs are marked sensitive, but `terraform output -raw` intentionally writes their contents to the terminal or redirected file.
|
|
|
|
Argo CD is installed as an internal `ClusterIP` service. Use the generated port-forward command to connect the Argo CD CLI:
|
|
|
|
```bash
|
|
argocd login --port-forward --port-forward-namespace argocd argocd-server
|
|
argocd app list
|
|
```
|
|
|
|
The initial admin password can be retrieved with:
|
|
|
|
```bash
|
|
kubectl -n argocd get secret argocd-initial-admin-secret \
|
|
-o jsonpath='{.data.password}' | base64 -d
|
|
```
|
|
|
|
Useful output values include `talos_nodes`, `argocd_namespace`, `argocd_server_service`, `argocd_cli_port_forward_command`, and `argocd_admin_password_command`.
|
|
|
|
## Talos certificate mismatch
|
|
|
|
`talos_machine_secrets.cluster` generates the Talos CA and client certificate used by Terraform. The nodes must have been configured from that same secrets bundle. An error such as `x509: certificate signed by unknown authority` means the node at the configured IP was initialized with a different Talos CA, or the Terraform state was replaced.
|
|
|
|
Do not disable TLS verification and do not regenerate `talos_machine_secrets` when preserving the cluster. Recover the original machine-secrets bundle and its Terraform state, then retry. If the cluster is disposable, remove the old Talos installation and Terraform state together and bootstrap the nodes again from the newly generated secrets. Verify the target IPs before either action; destroying state alone does not change the CA installed on a node.
|
|
|
|
## Reconciliation order
|
|
|
|
Cilium must become healthy before workloads depending on its networking features. Cilium requires host networking, host paths, privileged containers, and additional Linux capabilities, so `infrastructure/cilium/templates/namespace.yaml` labels only the Cilium namespace with Pod Security `privileged` enforcement. On Talos, the machine configuration also removes the default Flannel CNI and kube-proxy, disables Cilium cgroup auto-mounting, uses KubePrism on `localhost:7445`, drops `SYS_MODULE`, and enables legacy host routing for DNS compatibility.
|
|
|
|
If Cilium is unhealthy, it adds `node.cilium.io/agent-not-ready` taints and CoreDNS may remain Pending. That leaves Argo CD unable to resolve `argocd-repo-server`. After the corrected Cilium configuration is committed and pushed, temporarily allow CoreDNS to schedule so Argo CD can reconcile:
|
|
|
|
```bash
|
|
export KUBECONFIG="$HOME/.kube/home-services"
|
|
kubectl taint nodes --all node.cilium.io/agent-not-ready:NoSchedule-
|
|
kubectl -n kube-system rollout status deployment/coredns
|
|
```
|
|
|
|
Then refresh the Cilium Application and verify the agents before relying on Argo CD again:
|
|
|
|
```bash
|
|
kubectl -n argocd annotate application infrastructure-cilium argocd.argoproj.io/refresh=hard --overwrite
|
|
kubectl -n cilium get pods -o wide
|
|
kubectl -n argocd get application infrastructure-cilium
|
|
```
|
|
|
|
Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.
|
|
|
|
## SOPS workflow
|
|
|
|
Use an age recipient in `.sops.yaml`, encrypt secret manifests as `*.sops.yaml`, and provision the matching private key out-of-band. Do not place age private keys, plaintext values, tokens, or passwords in this repository.
|
|
|
|
## Recovery
|
|
|
|
If Gitea is unavailable, existing resources remain at their last reconciled state but new changes cannot be applied. Restore repository access, inspect Argo CD Application health, and allow reconciliation to resume. For a workload rollback, revert the Git commit and let Argo CD reconcile the known-good revision.
|
|
|
|
## Two-node limitation
|
|
|
|
This is not a highly available control plane with one control-plane node. Storage replica counts are intentionally set to one until a second suitable storage node and disk policy are confirmed. |