A number of fixes to get things running
This commit is contained in:
@@ -15,6 +15,68 @@ Terraform owns Talos machine configuration, cluster bootstrap, and the Argo CD H
|
||||
|
||||
Terraform generates one machine-secrets bundle, renders one machine configuration per role, applies the matching configuration to every node, and bootstraps only the selected control-plane node. For more than one control-plane node, `cluster_endpoint` must resolve to a stable VIP or load balancer rather than an individual node address.
|
||||
|
||||
## Talos 1.13 to 1.14 upgrade
|
||||
|
||||
Talos 1.14 uses the Image Factory installer image. The old image `ghcr.io/siderolabs/installer:v1.14.0` is not published and returns `NotFound`. For these amd64 KVM nodes, use the configured schematic:
|
||||
|
||||
```bash
|
||||
export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
|
||||
export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba:v1.14.0"
|
||||
```
|
||||
|
||||
Upgrade one node at a time, worker first:
|
||||
|
||||
```bash
|
||||
talosctl upgrade --endpoints 10.0.0.246 --nodes 10.0.0.246 --image "$TALOS_INSTALLER_IMAGE" --wait
|
||||
talosctl version --endpoints 10.0.0.246 --nodes 10.0.0.246
|
||||
|
||||
talosctl upgrade --endpoints 10.0.0.99 --nodes 10.0.0.99 --image "$TALOS_INSTALLER_IMAGE" --drain=false --wait
|
||||
talosctl version --endpoints 10.0.0.99 --nodes 10.0.0.99
|
||||
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
|
||||
```
|
||||
|
||||
Set `--endpoints` to the same address as `--nodes` for each upgrade. This prevents Talosctl from attempting request forwarding through the other node during the version-compatibility check. The control-plane upgrade uses `--drain=false` because this single-control-plane cluster cannot safely drain through a Kubernetes kubeconfig during the Talos transition.
|
||||
|
||||
After both nodes report 1.14, set `talos_version = "v1.14.0"` in `terraform.tfvars`, then run `terraform plan` and `terraform apply` to reconcile the generated machine configurations and GitOps workloads. Do not reset or recreate the VMs during an in-place upgrade.
|
||||
|
||||
## CLI credentials and outputs
|
||||
|
||||
After `terraform apply` completes, export the sensitive Terraform outputs to local files for the Talos and Kubernetes CLI tools. Do not commit or share these files:
|
||||
|
||||
```bash
|
||||
cd terraform
|
||||
|
||||
export KUBECONFIG="$HOME/.kube/home-services"
|
||||
mkdir -p "$(dirname "$KUBECONFIG")"
|
||||
terraform output -raw kubeconfig > "$KUBECONFIG"
|
||||
chmod 600 "$KUBECONFIG"
|
||||
kubectl cluster-info
|
||||
|
||||
export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
|
||||
mkdir -p "$(dirname "$TALOSCONFIG")"
|
||||
terraform output -raw talosconfig > "$TALOSCONFIG"
|
||||
chmod 600 "$TALOSCONFIG"
|
||||
talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version
|
||||
```
|
||||
|
||||
Terraform state contains the Talos client credentials and Kubernetes client certificates. Protect the state backend and local state files. The `kubeconfig` and `talosconfig` outputs are marked sensitive, but `terraform output -raw` intentionally writes their contents to the terminal or redirected file.
|
||||
|
||||
Argo CD is installed as an internal `ClusterIP` service. Use the generated port-forward command to connect the Argo CD CLI:
|
||||
|
||||
```bash
|
||||
argocd login --port-forward --port-forward-namespace argocd argocd-server
|
||||
argocd app list
|
||||
```
|
||||
|
||||
The initial admin password can be retrieved with:
|
||||
|
||||
```bash
|
||||
kubectl -n argocd get secret argocd-initial-admin-secret \
|
||||
-o jsonpath='{.data.password}' | base64 -d
|
||||
```
|
||||
|
||||
Useful output values include `talos_nodes`, `argocd_namespace`, `argocd_server_service`, `argocd_cli_port_forward_command`, and `argocd_admin_password_command`.
|
||||
|
||||
## Talos certificate mismatch
|
||||
|
||||
`talos_machine_secrets.cluster` generates the Talos CA and client certificate used by Terraform. The nodes must have been configured from that same secrets bundle. An error such as `x509: certificate signed by unknown authority` means the node at the configured IP was initialized with a different Talos CA, or the Terraform state was replaced.
|
||||
@@ -23,7 +85,25 @@ Do not disable TLS verification and do not regenerate `talos_machine_secrets` wh
|
||||
|
||||
## Reconciliation order
|
||||
|
||||
Cilium must become healthy before workloads depending on its networking features. Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.
|
||||
Cilium must become healthy before workloads depending on its networking features. Cilium requires host networking, host paths, privileged containers, and additional Linux capabilities, so `infrastructure/cilium/templates/namespace.yaml` labels only the Cilium namespace with Pod Security `privileged` enforcement. On Talos, the machine configuration also removes the default Flannel CNI and kube-proxy, disables Cilium cgroup auto-mounting, uses KubePrism on `localhost:7445`, drops `SYS_MODULE`, and enables legacy host routing for DNS compatibility.
|
||||
|
||||
If Cilium is unhealthy, it adds `node.cilium.io/agent-not-ready` taints and CoreDNS may remain Pending. That leaves Argo CD unable to resolve `argocd-repo-server`. After the corrected Cilium configuration is committed and pushed, temporarily allow CoreDNS to schedule so Argo CD can reconcile:
|
||||
|
||||
```bash
|
||||
export KUBECONFIG="$HOME/.kube/home-services"
|
||||
kubectl taint nodes --all node.cilium.io/agent-not-ready:NoSchedule-
|
||||
kubectl -n kube-system rollout status deployment/coredns
|
||||
```
|
||||
|
||||
Then refresh the Cilium Application and verify the agents before relying on Argo CD again:
|
||||
|
||||
```bash
|
||||
kubectl -n argocd annotate application infrastructure-cilium argocd.argoproj.io/refresh=hard --overwrite
|
||||
kubectl -n cilium get pods -o wide
|
||||
kubectl -n argocd get application infrastructure-cilium
|
||||
```
|
||||
|
||||
Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.
|
||||
|
||||
## SOPS workflow
|
||||
|
||||
|
||||
Reference in New Issue
Block a user