A number of fixes to get things running

2026-10-03 09:43:45 -06:00
parent cb47e319df
commit 461b0889e6
10 changed files with 2035 additions and 10 deletions
@@ -0,0 +1,116 @@
+# Copilot instructions for home-services
+
+## Repository overview
+
+This repository manages a home lab through two delivery paths:
+
+- Host services via Docker/Podman Compose under `compose-services/`
+- A Talos Kubernetes cluster bootstrapped with Terraform and reconciled by Argo CD
+
+The core directories are:
+
+- `compose-services/` — host-specific compose stacks (`media_nas1`, `media_rhel1`)
+- `bootstrap/applicationsets/` — Argo CD ApplicationSets for infrastructure and app workloads
+- `infrastructure/` — cluster-level Helm apps such as Cilium and Longhorn
+- `services/` — application Helm charts such as Vaultwarden
+- `terraform/` — Talos bootstrap, cluster provisioning, and Argo CD installation
+- `.gitea/workflows/deploy.yaml` — deployment automation for the host stacks
+
+## Build, test, and validation commands
+
+There is no repo-wide lint step in this project. Validation is a combination of the Python unit tests for `composectl.py` and Terraform validation for the bootstrap path.
+
+Run the unit tests:
+
+```bash
+python3 -m unittest tests.test_composectl
+```
+
+Run a single test by name:
+
+```bash
+python3 -m unittest tests.test_composectl.ComposeCtlTests.test_build_compose_command_uses_selected_runtime
+```
+
+Inspect the CLI and supported actions:
+
+```bash
+python3 composectl.py --help
+```
+
+Dry-run a stack refresh before changing live workloads:
+
+```bash
+python3 composectl.py --dry-run --stack media_nas1 refresh
+python3 composectl.py --dry-run --stack media_rhel1 refresh
+```
+
+Terraform bootstrap validation (from `terraform/`):
+
+```bash
+cd terraform
+terraform init
+terraform fmt -check -diff
+terraform validate
+```
+
+## Architecture and operational flow
+
+### 1) Compose deployment path
+
+`composectl.py` is the entry point for runtime-agnostic stack operations. It detects Docker or Podman at runtime and runs the same `docker compose`/`podman compose` flow against a stack under `compose-services/<stack>/compose.yaml`.
+
+The default maintenance flow for a stack is:
+
+1. `compose config`
+2. `pull`
+3. `up -d --remove-orphans`
+4. optional `image prune -f`
+
+This is the path used by the Gitea workflow in `.gitea/workflows/deploy.yaml` for `media_nas1` and `media_rhel1`.
+
+### 2) Talos + GitOps path
+
+`terraform/main.tf` bootstraps the Talos cluster and installs Argo CD. It creates:
+
+- Talos machine secrets and client configuration
+- per-role machine configs for control-plane and worker nodes
+- bootstrap of the selected control-plane node
+- a kubeconfig for the cluster
+- a Helm release for Argo CD
+- the `gitea-repository` secret used by Argo CD to pull from this repo
+
+Once the cluster is ready, Argo CD discovers the application manifests via `bootstrap/applicationsets/`, which fan out into `infrastructure/*` and `services/*` chart definitions.
+
+### 3) Secret handling and environment boundaries
+
+Sensitive values are intentionally not committed to the repo. The project expects:
+
+- `.sops.yaml` as the repository secret policy
+- age-based decryption for encrypted secrets
+- local-only Terraform `.tfvars` files and environment variables such as `TF_VAR_gitea_username` / `TF_VAR_gitea_password`
+- non-committed kubeconfig and talosconfig outputs saved to local paths
+
+## MCP server
+
+VS Code loads the portable workspace MCP configuration from `.mcp.json`. It runs the pinned HashiCorp Terraform MCP Server image through Podman and exposes only read-only public Registry lookups for providers and modules. Podman must be installed; the image is pulled on first use. The configuration does not provide HCP Terraform credentials or enable workspace operations.
+
+## Conventions specific to this repo
+
+- Treat `compose-services/<host>/compose.yaml` as the source of truth for host workloads; keep stack naming aligned with the folder name.
+- Prefer `composectl.py --dry-run` before any live deployment or refresh.
+- Keep the runtime abstraction intact: avoid writing Docker-only or Podman-only commands into repo scripts when `composectl.py` can handle both.
+- Use the existing stack naming pattern: `media_nas1` maps to `media_nas1_services`, and similarly for `media_rhel1`.
+- Keep Terraform inputs site-specific and local; `terraform.tfvars.example` is a template, not a checked-in secret file.
+- Do not commit credentials, tokens, kubeconfigs, or local state artifacts. If a file is sensitive, keep it outside the repo or encrypted under the SOPS policy.
+- The Gitea workflow is keyed to the `main` branch and the first Friday of the month schedule; do not change that deployment cadence without understanding the automated rollback behavior.
+- When editing cluster bootstrapping or Argo CD setup, keep the GitOps boundary clear: Terraform provisions the platform, while chart directories under `infrastructure/` and `services/` hold the workloads.
+
+## Practical guidance for Copilot sessions
+
+When making changes here, favor the repository's two-layer model:
+
+- Compose changes belong in `compose-services/*/compose.yaml` or `composectl.py`
+- Cluster or app delivery changes belong in `terraform/`, `bootstrap/applicationsets/`, `infrastructure/`, or `services/`
+
+Before proposing deploy commands, check whether the change targets the host Compose path or the Kubernetes/GitOps path. The repository intentionally keeps those flows separate.
@@ -0,0 +1,14 @@
+{
+  "mcpServers": {
+    "terraform-registry": {
+      "command": "podman",
+      "args": [
+        "run",
+        "-i",
+        "--rm",
+        "docker.io/hashicorp/terraform-mcp-server:1.3.0",
+        "--tools=search_providers,get_provider_details,get_latest_provider_version,get_provider_capabilities,search_modules,get_module_details,get_latest_module_version"
+      ]
+    }
+  }
+}
@@ -15,6 +15,68 @@ Terraform owns Talos machine configuration, cluster bootstrap, and the Argo CD H

 Terraform generates one machine-secrets bundle, renders one machine configuration per role, applies the matching configuration to every node, and bootstraps only the selected control-plane node. For more than one control-plane node, `cluster_endpoint` must resolve to a stable VIP or load balancer rather than an individual node address.

+## Talos 1.13 to 1.14 upgrade
+
+Talos 1.14 uses the Image Factory installer image. The old image `ghcr.io/siderolabs/installer:v1.14.0` is not published and returns `NotFound`. For these amd64 KVM nodes, use the configured schematic:
+
+```bash
+export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
+export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba:v1.14.0"
+```
+
+Upgrade one node at a time, worker first:
+
+```bash
+talosctl upgrade --endpoints 10.0.0.246 --nodes 10.0.0.246 --image "$TALOS_INSTALLER_IMAGE" --wait
+talosctl version --endpoints 10.0.0.246 --nodes 10.0.0.246
+
+talosctl upgrade --endpoints 10.0.0.99 --nodes 10.0.0.99 --image "$TALOS_INSTALLER_IMAGE" --drain=false --wait
+talosctl version --endpoints 10.0.0.99 --nodes 10.0.0.99
+talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
+```
+
+Set `--endpoints` to the same address as `--nodes` for each upgrade. This prevents Talosctl from attempting request forwarding through the other node during the version-compatibility check. The control-plane upgrade uses `--drain=false` because this single-control-plane cluster cannot safely drain through a Kubernetes kubeconfig during the Talos transition.
+
+After both nodes report 1.14, set `talos_version = "v1.14.0"` in `terraform.tfvars`, then run `terraform plan` and `terraform apply` to reconcile the generated machine configurations and GitOps workloads. Do not reset or recreate the VMs during an in-place upgrade.
+
+## CLI credentials and outputs
+
+After `terraform apply` completes, export the sensitive Terraform outputs to local files for the Talos and Kubernetes CLI tools. Do not commit or share these files:
+
+```bash
+cd terraform
+
+export KUBECONFIG="$HOME/.kube/home-services"
+mkdir -p "$(dirname "$KUBECONFIG")"
+terraform output -raw kubeconfig > "$KUBECONFIG"
+chmod 600 "$KUBECONFIG"
+kubectl cluster-info
+
+export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
+mkdir -p "$(dirname "$TALOSCONFIG")"
+terraform output -raw talosconfig > "$TALOSCONFIG"
+chmod 600 "$TALOSCONFIG"
+talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version
+```
+
+Terraform state contains the Talos client credentials and Kubernetes client certificates. Protect the state backend and local state files. The `kubeconfig` and `talosconfig` outputs are marked sensitive, but `terraform output -raw` intentionally writes their contents to the terminal or redirected file.
+
+Argo CD is installed as an internal `ClusterIP` service. Use the generated port-forward command to connect the Argo CD CLI:
+
+```bash
+argocd login --port-forward --port-forward-namespace argocd argocd-server
+argocd app list
+```
+
+The initial admin password can be retrieved with:
+
+```bash
+kubectl -n argocd get secret argocd-initial-admin-secret \
+	-o jsonpath='{.data.password}' | base64 -d
+```
+
+Useful output values include `talos_nodes`, `argocd_namespace`, `argocd_server_service`, `argocd_cli_port_forward_command`, and `argocd_admin_password_command`.
+
 ## Talos certificate mismatch

 `talos_machine_secrets.cluster` generates the Talos CA and client certificate used by Terraform. The nodes must have been configured from that same secrets bundle. An error such as `x509: certificate signed by unknown authority` means the node at the configured IP was initialized with a different Talos CA, or the Terraform state was replaced.
@@ -23,7 +85,25 @@ Do not disable TLS verification and do not regenerate `talos_machine_secrets` wh

 ## Reconciliation order

-Cilium must become healthy before workloads depending on its networking features. Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.
+Cilium must become healthy before workloads depending on its networking features. Cilium requires host networking, host paths, privileged containers, and additional Linux capabilities, so `infrastructure/cilium/templates/namespace.yaml` labels only the Cilium namespace with Pod Security `privileged` enforcement. On Talos, the machine configuration also removes the default Flannel CNI and kube-proxy, disables Cilium cgroup auto-mounting, uses KubePrism on `localhost:7445`, drops `SYS_MODULE`, and enables legacy host routing for DNS compatibility.
+
+If Cilium is unhealthy, it adds `node.cilium.io/agent-not-ready` taints and CoreDNS may remain Pending. That leaves Argo CD unable to resolve `argocd-repo-server`. After the corrected Cilium configuration is committed and pushed, temporarily allow CoreDNS to schedule so Argo CD can reconcile:
+
+```bash
+export KUBECONFIG="$HOME/.kube/home-services"
+kubectl taint nodes --all node.cilium.io/agent-not-ready:NoSchedule-
+kubectl -n kube-system rollout status deployment/coredns
+```
+
+Then refresh the Cilium Application and verify the agents before relying on Argo CD again:
+
+```bash
+kubectl -n argocd annotate application infrastructure-cilium argocd.argoproj.io/refresh=hard --overwrite
+kubectl -n cilium get pods -o wide
+kubectl -n argocd get application infrastructure-cilium
+```
+
+Longhorn must have a usable disk before PVC-backed workloads are considered ready. ApplicationSets use automated sync, pruning, self-healing, and namespace creation, so deleting a chart directory intentionally removes its generated Application resources.

 ## SOPS workflow

@@ -147,8 +147,31 @@ The Kubernetes path accepts a `nodes` list containing any number of control-plan
  ```

  Set `KUBECONFIG` only after `terraform apply` has completed and the `kubeconfig` output exists. This file is a cluster credential and must not be committed or shared.
-7. Apply both ApplicationSets with `kubectl apply -f bootstrap/applicationsets/`.
-8. Confirm Argo CD discovers `infrastructure/cilium`, `infrastructure/longhorn`, and `services/vaultwarden`.
+7. Export the Talos client configuration for `talosctl`:
+
+  ```bash
+  export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
+  mkdir -p "$(dirname "$TALOSCONFIG")"
+  terraform output -raw talosconfig > "$TALOSCONFIG"
+  chmod 600 "$TALOSCONFIG"
+  talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version
+  ```
+
+8. Apply both ApplicationSets with `kubectl apply -f bootstrap/applicationsets/`.
+9. Connect the Argo CD CLI through the internal service:
+
+  ```bash
+  argocd login --port-forward --port-forward-namespace argocd argocd-server
+  argocd app list
+  ```
+
+  Retrieve the initial admin password with:
+
+  ```bash
+  kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d
+  ```
+
+10. Confirm Argo CD discovers `infrastructure/cilium`, `infrastructure/longhorn`, and `services/vaultwarden`.

 Talos uses its default CNI during bootstrap so Argo CD can start. Cilium is then reconciled as the first infrastructure application. Confirm the LAN interface and address pool before enabling production LoadBalancer services. Longhorn also requires a validated disk or mount on the storage node.

@@ -23,3 +23,50 @@ sudo virt-install --virt-type kvm --name worker-node-1 --ram 4096 --vcpus 4 --di
 ```

 Note that the VMs are booted into the ISO and terraform initiates the install and cluster build.
+
+## Upgrade Talos from 1.13 to 1.14
+
+Talos 1.14 no longer publishes the release installer image at `ghcr.io/siderolabs/installer:v1.14.0`. Use the Talos Image Factory installer image with the generic amd64 KVM schematic:
+
+```bash
+export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
+
+export TALOS_INSTALLER_IMAGE="factory.talos.dev/metal-installer/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba:v1.14.0"
+```
+
+Upgrade the worker first:
+
+```bash
+talosctl upgrade \
+	--endpoints 10.0.0.246 \
+	--nodes 10.0.0.246 \
+	--image "$TALOS_INSTALLER_IMAGE" \
+	--wait
+```
+
+After the worker is healthy, upgrade the control-plane node:
+
+```bash
+talosctl upgrade \
+	--endpoints 10.0.0.99 \
+	--nodes 10.0.0.99 \
+	--image "$TALOS_INSTALLER_IMAGE" \
+	--wait
+```
+
+Verify both nodes before reconciling Terraform:
+
+```bash
+talosctl version --endpoints 10.0.0.246 --nodes 10.0.0.246
+talosctl version --endpoints 10.0.0.99 --nodes 10.0.0.99
+talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
+```
+
+Then update `terraform.tfvars` to `talos_version = "v1.14.0"` if needed and run:
+
+```bash
+terraform plan
+terraform apply
+```
+
+Do not use `talosctl reset` or recreate the VMs for this upgrade. The installer preserves the Talos state and Kubernetes data.
@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cilium
+  labels:
+    pod-security.kubernetes.io/enforce: privileged
+    pod-security.kubernetes.io/audit: privileged
+    pod-security.kubernetes.io/warn: privileged
@@ -1,12 +1,42 @@
 # Confirm the interface and address pool before enabling L2 announcements.
 cilium:
  kubeProxyReplacement: true
-  k8sServiceHost: 10.0.0.99
-  k8sServicePort: 6443
+  k8sServiceHost: localhost
+  k8sServicePort: 7445

  ipam:
    mode: kubernetes

+  bpf:
+    hostLegacyRouting: true
+
+  cgroup:
+    autoMount:
+      enabled: false
+    hostRoot: /sys/fs/cgroup
+
+  extraConfig:
+    direct-routing-device: enp1s0
+
+  securityContext:
+    capabilities:
+      ciliumAgent:
+        - CHOWN
+        - KILL
+        - NET_ADMIN
+        - NET_RAW
+        - IPC_LOCK
+        - SYS_ADMIN
+        - SYS_RESOURCE
+        - DAC_OVERRIDE
+        - FOWNER
+        - SETGID
+        - SETUID
+      cleanCiliumState:
+        - NET_ADMIN
+        - SYS_ADMIN
+        - SYS_RESOURCE
+
  l2announcements:
    enabled: true

@@ -16,9 +46,9 @@ cilium:
  operator:
    replicas: 1

-# Replace with the LAN interface on both Talos nodes.
+  # The LAN interface is enp1s0 on both Talos nodes.
  devices:
-    - "<lan-interface>"
+    - enp1s0

 # Configure a CiliumLoadBalancerIPPool and L2AnnouncementPolicy after
 # confirming the unused address range in the cluster LAN.
@@ -33,6 +33,13 @@ resource "talos_machine_secrets" "cluster" {
  talos_version = var.talos_version
 }

+data "talos_client_configuration" "cluster" {
+  cluster_name         = var.cluster_name
+  client_configuration = talos_machine_secrets.cluster.client_configuration
+  endpoints            = [for node in var.nodes : node.ip]
+  nodes                = [for node in var.nodes : node.ip]
+}
+
 data "talos_machine_configuration" "role" {
  for_each = toset(["controlplane", "worker"])

@@ -50,6 +57,16 @@ data "talos_machine_configuration" "role" {
          disk = var.install_disk
        }
      }
+      cluster = {
+        network = {
+          cni = {
+            name = "none"
+          }
+        }
+        proxy = {
+          disabled = true
+        }
+      }
    })
  ]
 }
@@ -4,12 +4,37 @@ output "kubeconfig" {
  sensitive   = true
 }

-output "talos_client_configuration" {
-  description = "Talos client configuration generated for this cluster."
-  value       = talos_machine_secrets.cluster.client_configuration
+output "talosconfig" {
+  description = "Raw talosconfig for use with talosctl."
+  value       = data.talos_client_configuration.cluster.talos_config
  sensitive   = true
 }

+output "talos_nodes" {
+  description = "Talos node addresses configured by this module."
+  value       = [for node in var.nodes : node.ip]
+}
+
+output "argocd_namespace" {
+  description = "Namespace containing Argo CD."
+  value       = helm_release.argocd.namespace
+}
+
+output "argocd_server_service" {
+  description = "Internal Argo CD server service address."
+  value       = "argocd-server.${helm_release.argocd.namespace}.svc.cluster.local:443"
+}
+
+output "argocd_cli_port_forward_command" {
+  description = "Command to connect argocd CLI to the internal Argo CD server."
+  value       = "argocd login --port-forward --port-forward-namespace ${helm_release.argocd.namespace} argocd-server"
+}
+
+output "argocd_admin_password_command" {
+  description = "Command to retrieve the initial Argo CD admin password."
+  value       = "kubectl -n ${helm_release.argocd.namespace} get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d"
+}
+
 output "argocd_repository_secret_name" {
  description = "Argo CD repository credential Secret name."
  value       = kubernetes_secret_v1.gitea.metadata[0].name