A number of fixes to get things running
Deploy home services / deploy-nas1 (push) Has been cancelled
Deploy home services / deploy-rhel1 (push) Has been cancelled

This commit is contained in:
2026-10-03 09:43:45 -06:00
parent cb47e319df
commit 461b0889e6
10 changed files with 2035 additions and 10 deletions
+81 -1
View File
@@ -15,6 +15,68 @@ Terraform owns Talos machine configuration, cluster bootstrap, and the Argo CD H
Terraform generates one machine-secrets bundle, renders one machine configuration per role, applies the matching configuration to every node, and bootstraps only the selected control-plane node. For more than one control-plane node, `cluster_endpoint` must resolve to a stable VIP or load balancer rather than an individual node address.
## Talos 1.13 to 1.14 upgrade
Talos 1.14 uses the Image Factory installer image. The old image `ghcr.io/siderolabs/installer:v1.14.0` is not published and returns `NotFound`. For these amd64 KVM nodes, use the configured schematic:
```bash
export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba:v1.14.0"
```
Upgrade one node at a time, worker first:
```bash
talosctl upgrade --endpoints 10.0.0.246 --nodes 10.0.0.246 --image "$TALOS_INSTALLER_IMAGE" --wait
talosctl version --endpoints 10.0.0.246 --nodes 10.0.0.246
talosctl upgrade --endpoints 10.0.0.99 --nodes 10.0.0.99 --image "$TALOS_INSTALLER_IMAGE" --drain=false --wait
talosctl version --endpoints 10.0.0.99 --nodes 10.0.0.99
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
```
Set `--endpoints` to the same address as `--nodes` for each upgrade. This prevents Talosctl from attempting request forwarding through the other node during the version-compatibility check. The control-plane upgrade uses `--drain=false` because this single-control-plane cluster cannot safely drain through a Kubernetes kubeconfig during the Talos transition.
After both nodes report 1.14, set `talos_version = "v1.14.0"` in `terraform.tfvars`, then run `terraform plan` and `terraform apply` to reconcile the generated machine configurations and GitOps workloads. Do not reset or recreate the VMs during an in-place upgrade.
## CLI credentials and outputs
After `terraform apply` completes, export the sensitive Terraform outputs to local files for the Talos and Kubernetes CLI tools. Do not commit or share these files:
```bash
cd terraform
export KUBECONFIG="$HOME/.kube/home-services"
mkdir -p "$(dirname "$KUBECONFIG")"
terraform output -raw kubeconfig > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
kubectl cluster-info
export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
mkdir -p "$(dirname "$TALOSCONFIG")"
terraform output -raw talosconfig > "$TALOSCONFIG"
chmod 600 "$TALOSCONFIG"
talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version
```
Terraform state contains the Talos client credentials and Kubernetes client certificates. Protect the state backend and local state files. The `kubeconfig` and `talosconfig` outputs are marked sensitive, but `terraform output -raw` intentionally writes their contents to the terminal or redirected file.
Argo CD is installed as an internal `ClusterIP` service. Use the generated port-forward command to connect the Argo CD CLI:
```bash
argocd login --port-forward --port-forward-namespace argocd argocd-server
argocd app list
```
The initial admin password can be retrieved with:
```bash
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d
```
Useful output values include `talos_nodes`, `argocd_namespace`, `argocd_server_service`, `argocd_cli_port_forward_command`, and `argocd_admin_password_command`.
## Talos certificate mismatch
`talos_machine_secrets.cluster` generates the Talos CA and client certificate used by Terraform. The nodes must have been configured from that same secrets bundle. An error such as `x509: certificate signed by unknown authority` means the node at the configured IP was initialized with a different Talos CA, or the Terraform state was replaced.
@@ -23,7 +85,25 @@ Do not disable TLS verification and do not regenerate `talos_machine_secrets` wh
## Reconciliation order
Cilium must become healthy before workloads depending on its networking features. Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.
Cilium must become healthy before workloads depending on its networking features. Cilium requires host networking, host paths, privileged containers, and additional Linux capabilities, so `infrastructure/cilium/templates/namespace.yaml` labels only the Cilium namespace with Pod Security `privileged` enforcement. On Talos, the machine configuration also removes the default Flannel CNI and kube-proxy, disables Cilium cgroup auto-mounting, uses KubePrism on `localhost:7445`, drops `SYS_MODULE`, and enables legacy host routing for DNS compatibility.
If Cilium is unhealthy, it adds `node.cilium.io/agent-not-ready` taints and CoreDNS may remain Pending. That leaves Argo CD unable to resolve `argocd-repo-server`. After the corrected Cilium configuration is committed and pushed, temporarily allow CoreDNS to schedule so Argo CD can reconcile:
```bash
export KUBECONFIG="$HOME/.kube/home-services"
kubectl taint nodes --all node.cilium.io/agent-not-ready:NoSchedule-
kubectl -n kube-system rollout status deployment/coredns
```
Then refresh the Cilium Application and verify the agents before relying on Argo CD again:
```bash
kubectl -n argocd annotate application infrastructure-cilium argocd.argoproj.io/refresh=hard --overwrite
kubectl -n cilium get pods -o wide
kubectl -n argocd get application infrastructure-cilium
```
Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.
## SOPS workflow