home-services/README.md

Home Services

This repository manages home-lab services through two delivery paths: existing host Compose stacks and a Talos Kubernetes cluster reconciled by Argo CD.

flowchart LR
  developer[Developer workstation]
  repo[Gitea repository<br/>whentimeslows/home-services]

  subgraph compose[Compose delivery path]
    actions[Gitea Actions]
    nas[Self-hosted runner: nas1]
    rhel[Self-hosted runner: rhel1]
    nasStack[compose-services/media_nas1<br/>Docker Compose]
    rhelStack[compose-services/media_rhel1<br/>Podman Compose]
    actions --> nas --> nasStack
    actions --> rhel --> rhelStack
  end

  subgraph bootstrap[Terraform bootstrap path]
    terraform[Terraform]
    secrets[Talos machine secrets]
    talosConfig[Role-specific Talos machine configuration]
    terraform --> secrets
    terraform --> talosConfig
  end

  subgraph cluster[Talos Kubernetes cluster]
    cp[Control-plane nodes]
    workers[Worker nodes]
    argocd[Argo CD]
    appsets[ApplicationSets]
    apps[Generated Applications]
    workloads[Cluster workloads]
    cp --> argocd
    workers --> workloads
    argocd --> appsets --> apps --> workloads
  end

  subgraph gitops[GitOps repository directories]
    appsetFiles[bootstrap/applicationsets]
    infra[infrastructure/*<br/>Cilium and Longhorn charts]
    services[services/*<br/>Vaultwarden and other charts]
    appsetFiles -. registers .-> appsets
    infra --> apps
    services --> apps
  end

  developer -->|pushes code| repo
  repo -->|workflow checkout| actions
  repo -->|Terraform source and chart source| terraform
  terraform -->|configures and bootstraps| cluster
  repo -->|Git polling or webhook| argocd
  developer -->|kubectl handoff| appsetFiles

The Compose path deploys host services through Gitea Actions. The GitOps path uses Terraform for Talos day-zero bootstrap, then Argo CD pulls chart definitions from Gitea and continuously reconciles Kubernetes resources. The bootstrap/applicationsets/ manifests are the initial handoff between Terraform-installed Argo CD and the GitOps repository.

Repository layout

compose-services/
  media_nas1/
    compose.yaml
  media_rhel1/
    compose.yaml
terraform/
bootstrap/applicationsets/
infrastructure/
services/
  vaultwarden/
composectl.py
update_containers.sh
.gitea/
  workflows/
    deploy.yaml

• compose-services/ contains host-specific Compose stacks managed by Gitea Actions.
• infrastructure/ and services/ contain Helm charts discovered by Argo CD ApplicationSets.
• terraform/ bootstraps Talos and installs the Argo CD engine; it does not manage day-two workloads.
• composectl.py infers Compose stack names and paths from compose-services/.
• update_containers.sh is a compatibility shim that delegates to the Python tool.

Developer workstation tools

On this Linux workstation, install the repository's Kubernetes and infrastructure CLI tools with Homebrew:

brew install hashicorp/tap/terraform helm kubernetes-cli jq sops age
brew tap siderolabs/tap
brew install siderolabs/tap/talosctl@1.14
brew tap devantler-tech/tap
brew install --cask devantler-tech/tap/ksail

Terraform, Helm, kubectl, and Talosctl are used for cluster provisioning and operations; jq supports the documented command examples; SOPS and age support the repository's encrypted-secret workflow. The Talosctl formula is pinned to the 1.14 minor line used by this cluster. KSail is also configured as a workspace MCP server in the root .mcp.json; keep the Homebrew binary on PATH so VS Code can launch it.

Compose operations additionally need Docker or Podman. This workstation uses its OS-managed Podman package; install Podman through the OS package manager when setting up another Linux machine, or use brew install podman if managing that CLI with Homebrew. Homebrew versions are workstation tooling, not deployment pins: deployed versions remain defined in the repository's Compose, Helm, and Terraform configuration.

Runtime support

The CLI detects and uses whichever runtime is available:

• docker compose ...
• podman compose ...

This means the same service directories can be managed from either backend without changing the compose files.

Common commands

From the repository root:

python3 composectl.py --help
python3 composectl.py --dry-run --stack media_nas1 refresh
python3 composectl.py --dry-run --stack media_rhel1 refresh
python3 composectl.py --stack media_nas1 pull
python3 composectl.py --stack media_rhel1 status

The shell entrypoint delegates to the Python tool:

bash update_containers.sh --dry-run --stack media_rhel1 refresh

Refresh flow

The refresh flow validates and updates one stack:

1. compose config
2. pull
3. up -d --remove-orphans
4. optionally image prune -f when --prune is supplied

The up operation avoids an unnecessary down, reducing downtime during normal updates.

GitOps bootstrap

The Kubernetes path accepts a nodes list containing any number of control-plane and worker nodes. Each node has a unique name, IP address, and role; exactly one control-plane node is marked bootstrap = true. The endpoint, Kubernetes version, storage device, Cilium interface, and load-balancer address pool remain explicit inputs because they are site-specific.

1. Copy terraform/terraform.tfvars.example to a local .tfvars file and fill in the supported Kubernetes version, stable control-plane endpoint, and nodes list.
2. Create a dedicated Gitea service account with read-only access to whentimeslows/home-services, then create a read-only personal access token for that account. Do not use an administrator token or commit the token to terraform.tfvars.
3. From terraform/, export the Gitea credentials for this apply. The Terraform Kubernetes provider will create the Argo CD repository Secret named gitea-repository in the argocd namespace:

export TF_VAR_gitea_username="argocd-home-services"
export TF_VAR_gitea_password="<read-only-gitea-token>"

The token is stored in Terraform state and in the Kubernetes Secret, so protect the state backend and shell history. Use an environment variable or a protected variable-file mechanism rather than committing the token. 4. Run terraform init, terraform fmt, and terraform validate from terraform/. 5. Boot every KVM VM from the Talos ISO with its install disk and dedicated 10 GiB Longhorn data disk attached. The data disk must appear as /dev/vdb. Terraform applies the role-specific configuration to every node, installs the pinned Talos Image Factory image (including the Longhorn-required extensions) to install_disk, bootstraps the selected control-plane node, and installs Argo CD. 6. Export the kubeconfig produced by Terraform before running kubectl. Terraform cannot safely infer your shell's active kubeconfig or make a later interactive kubectl command part of the apply, but it does expose the generated config as the sensitive kubeconfig output:

export KUBECONFIG="$HOME/.kube/home-services"
mkdir -p "$(dirname "$KUBECONFIG")"
terraform output -raw kubeconfig > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
kubectl cluster-info

Set KUBECONFIG only after terraform apply has completed and the kubeconfig output exists. This file is a cluster credential and must not be committed or shared. 7. Export the Talos client configuration for talosctl:

export TALOSCONFIG="$HOME/.talos/home-services/talosconfig"
mkdir -p "$(dirname "$TALOSCONFIG")"
terraform output -raw talosconfig > "$TALOSCONFIG"
chmod 600 "$TALOSCONFIG"
talosctl --nodes "$(terraform output -json talos_nodes | jq -r 'join(",")')" version

8. Provision the dedicated Longhorn disk on each node using the checked-in Talos volume configuration. Apply it to both nodes before rebooting either node:

talosctl patch machineconfig --endpoints 10.0.0.246 --nodes 10.0.0.246 --patch @terraform/longhorn-user-volume.yaml
talosctl patch machineconfig --endpoints 10.0.0.99 --nodes 10.0.0.99 --patch @terraform/longhorn-user-volume.yaml

For a fresh bootstrap, reboot one node at a time, worker first, and wait for it to rejoin before restarting the control-plane node:

talosctl reboot --endpoints 10.0.0.246 --nodes 10.0.0.246 --wait
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246
talosctl reboot --endpoints 10.0.0.99 --nodes 10.0.0.99 --wait
talosctl health --endpoints 10.0.0.99 --nodes 10.0.0.99,10.0.0.246

For an existing cluster, use the rolling extension upgrade procedure in TALOS.md instead of a plain reboot. The Talos volume is mounted at /var/mnt/longhorn, which is the Longhorn data path. The selector only targets /dev/vdb; verify the added disk appears there on every VM before applying it. 9. Apply both ApplicationSets with kubectl apply -f bootstrap/applicationsets/. 10. Connect the Argo CD CLI through the internal service:

argocd login --port-forward --port-forward-namespace argocd argocd-server
argocd app list

Retrieve the initial admin password with:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d

11. Confirm Argo CD discovers infrastructure/cilium, infrastructure/longhorn, and services/vaultwarden, and that the Longhorn StorageClass is available before relying on Vaultwarden persistence.

Talos uses its default CNI during bootstrap so Argo CD can start. Cilium is then reconciled as the first infrastructure application. Cilium's Envoy ingress controller is enabled in shared-load-balancer mode; Ingress resources must set spec.ingressClassName: cilium. The Cilium LoadBalancer IP pool is 10.0.0.10–10.0.0.89, announced over enp1s0 with an L2 announcement policy. Keep that range reserved from DHCP and other static assignments, and recheck the interface and pool if the node network changes. Longhorn also requires a validated disk or mount on the storage node.

The Terraform Talos provider uses one generated machine-secrets bundle for the whole node list. If the nodes were previously configured with another Talos CA, reset/reinstall them or restore the matching Terraform state before applying. Terraform state contains the generated Talos credentials and must remain private.

Terraform's Talos client certificates must match the CA already installed on the nodes. If apply reports x509: certificate signed by unknown authority, stop and verify the node IPs and recover the original machine-secrets bundle/state. Recreating Terraform secrets without rebuilding the nodes will not repair the trust relationship.

Secrets

SOPS with age is the intended secret boundary. The repository policy is in .sops.yaml; age private keys and plaintext secret files are ignored. Provision the age private key to the cluster through a protected bootstrap process, then commit only files matching the repository's encrypted-file policy. Vaultwarden expects an existing Secret named vaultwarden-secrets.

Gitea Actions deployment

The workflow in .gitea/workflows/deploy.yaml deploys each host-specific stack to its matching self-hosted runner:

• runner label nas1 deploys media_nas1 with Docker
• runner label rhel1 deploys media_rhel1 with Podman

Install and register one persistent Gitea Actions runner on each server. The runner should use a dedicated service account and should only execute trusted changes from the protected main branch.

Deployments run on:

• pushes to main
• manual workflow dispatch
• the first Friday of each month at 03:00 using a fixed UTC-6 offset (09:00 UTC)

The scheduled trigger evaluates every Friday and skips dates after the seventh day, leaving only the first Friday eligible. The host jobs always target their own stack; they do not use --all.

To roll back, dispatch the workflow from a known-good commit or revert the change on main and redeploy.