From 931581eb88a5c010ee08703b6118872f0492ae24 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 5 Apr 2022 19:28:53 +0300 Subject: [PATCH] kubeadm: add task page for cluster reconf The new task page outlines steps for reconfiguring a kubeadm cluster and persisting reconfiguration. Link the new page from the existing guides for "customizing components", "creating a cluster", and "kubeadm upgrade". Co-authored-by: Paco Xu Co-authored-by: Qiming Teng --- .../tools/kubeadm/control-plane-flags.md | 7 +- .../tools/kubeadm/create-cluster-kubeadm.md | 3 + .../kubeadm/kubeadm-reconfigure.md | 281 ++++++++++++++++++ .../kubeadm/kubeadm-upgrade.md | 7 +- 4 files changed, 295 insertions(+), 3 deletions(-) create mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure.md diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md b/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md index a4c2526572..f15315f3ad 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md @@ -24,9 +24,12 @@ you can skip the default CoreDNS deployment and deploy your own variant. For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases). {{< /note >}} - +{{< note >}} +To reconfigure a cluster that has already been created see +[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure). +{{< /note >}} -{{< feature-state for_k8s_version="v1.12" state="stable" >}} + ## Customizing the control plane with flags in `ClusterConfiguration` diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md index 8349b4a817..1dcc375eef 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md @@ -162,6 +162,9 @@ To customize control plane components, including optional IPv6 assignment to liv for control plane components and etcd server, provide extra arguments to each component as documented in [custom arguments](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/). +To reconfigure a cluster that has already been created see +[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure). + To run `kubeadm init` again, you must first [tear down the cluster](#tear-down). If you join a node with a different architecture to your cluster, make sure that your deployed DaemonSets diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure.md new file mode 100644 index 0000000000..6c63600266 --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure.md @@ -0,0 +1,281 @@ +--- +reviewers: +- sig-cluster-lifecycle +title: Reconfiguring a kubeadm cluster +content_type: task +weight: 10 +--- + + + +kubeadm does not support automated ways of reconfiguring components that +were deployed on managed nodes. One way of automating this would be +by using a custom [operator](/docs/concepts/extend-kubernetes/operator/). + +To modify the components configuration you must manually edit associated cluster +objects and files on disk. + +This guide shows the correct sequence of steps that need to be performed +to achieve kubeadm cluster reconfiguration. + +## {{% heading "prerequisites" %}} + +- You need a cluster that was deployed using kubeadm +- Have administrator credentials (`/etc/kubernetes/admin.conf`) and network connectivity +to a running kube-apiserver in the cluster from a host that has kubectl installed +- Have a text editor installed on all hosts + + + +## Reconfiguring the cluster + +kubeadm writes a set of cluster wide component configuration options in +ConfigMaps and other objects. These objects must be manually edited. The command `kubectl edit` +can be used for that. + +The `kubectl edit` command will open a text editor where you can edit and save the object directly. + +You can use the environment variables `KUBECONFIG` and `KUBE_EDITOR` to specify the location of +the kubectl consumed kubeconfig file and preferred text editor. + +For example: +``` +KUBECONFIG=/etc/kubernetes/admin.conf KUBE_EDITOR=nano kubectl edit +``` + +{{< note >}} +Upon saving any changes to these cluster objects, components running on nodes may not be +automatically updated. The steps below instruct you on how to perform that manually. +{{< /note >}} + +{{< warning >}} +Component configuration in ConfigMaps is stored as unstructured data (YAML string). +This means that validation will not be performed upon updating the contents of a ConfigMap. +You have to be careful to follow the documented API format for a particular +component configuration and avoid introducing typos and YAML indentation mistakes. +{{< /warning >}} + +### Applying cluster configuration changes + +#### Updating the `ClusterConfiguration` + +During cluster creation and upgrade, kubeadm writes its +[`ClusterConfiguration`](/docs/reference/config-api/kubeadm-config.v1beta3/) +in a ConfigMap called `kubeadm-config` in the `kube-system` namespace. + +To change a particular option in the `ClusterConfiguration` you can edit the ConfigMap with this command: + +```shell +kubectl edit cm -n kube-system kubeadm-config +``` + +The configuration is located under the `data.ClusterConfiguration` key. + +{{< note >}} +The `ClusterConfiguration` includes a variety of options that affect the configuration of individual +components such as kube-apiserver, kube-scheduler, kube-controller-manager, CoreDNS, etcd and kube-proxy. +Changes to the configuration must be reflected on node components manually. +{{< /note >}} + +#### Reflecting `ClusterConfiguration` changes on control plane nodes + +kubeadm manages the control plane components as static Pod manifests located in +the directory `/etc/kubernetes/manifests`. +Any changes to the `ClusterConfiguration` under the `apiServer`, `controllerManager`, `scheduler` or `etcd` +keys must be reflected in the associated files in the manifests directory on a control plane node. + +Such changes may include: +- `extraArgs` - requires updating the list of flags passed to a component container +- `extraMounts` - requires updated the volume mounts for a component container +- `*SANs` - requires writing new certificates with updated Subject Alternative Names. + +Before proceeding with these changes, make sure you have backed up the directory `/etc/kubernetes/`. + +To write new certificates you can use: +```shell +kubeadm init phase certs --config +``` + +To write new manifest files in `/etc/kubernetes/manifests` you can use: + +```shell +kubeadm init phase control-plane --config +``` + +The `` contents must match the updated `ClusterConfiguration`. +The `` value must be the name of the component. + +{{< note >}} +Updating a file in `/etc/kubernetes/manifests` will tell the kubelet to restart the static Pod for the corresponding component. +Try doing these changes one node at a time to leave the cluster without downtime. +{{< /note >}} + +### Applying kubelet configuration changes + +#### Updating the `KubeletConfiguration` + +During cluster creation and upgrade, kubeadm writes its +[`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/) +in a ConfigMap called `kubelet-config` in the `kube-system` namespace. + +You can edit the ConfigMap with this command: + +```shell +kubectl edit cm -n kube-system kubelet-config +``` + +The configuration is located under the `data.kubelet` key. + +#### Reflecting the kubelet changes + +To reflect the change on kubeadm nodes you must do the following: +- Log in to a kubeadm node +- Run `kubeadm upgrade node phase kubelet-config` to download the latest `kubelet-config` +ConfigMap contents into the local file `/var/lib/kubelet/config.conf` +- Edit the file `/var/lib/kubelet/kubeadm-flags.env` to apply additional configuration with +flags +- Restart the kubelet service with `systemctl restart kubelet` + +{{< note >}} +Do these changes one node at a time to allow workloads to be rescheduled properly. +{{< /note >}} + +{{< note >}} +During `kubeadm upgrade`, kubeadm downloads the `KubeletConfiguration` from the +`kubelet-config` ConfigMap and overwrite the contents of `/var/lib/kubelet/config.conf`. +This means that node local configuration must be applied either by flags in +`/var/lib/kubelet/kubeadm-flags.env` or by manually updating the contents of +`/var/lib/kubelet/config.conf` after `kubeadm upgrade`, and then restarting the kubelet. +{{< /note >}} + +### Applying kube-proxy configuration changes + +#### Updating the `KubeProxyConfiguration` + +During cluster creation and upgrade, kubeadm writes its +[`KubeProxyConfiguration`](/docs/reference/config-api/kube-proxy-config.v1alpha1/) +in a ConfigMap in the `kube-system` namespace called `kube-proxy`. + +This ConfigMap is used by the `kube-proxy` DaemonSet in the `kube-system` namespace. + +To change a particular option in the `KubeProxyConfiguration`, you can edit the ConfigMap with this command: + +```shell +kubectl edit cm -n kube-system kube-proxy +``` + +The configuration is located under the `data.config.conf` key. + +#### Reflecting the kube-proxy changes + +Once the `kube-proxy` ConfigMap is updated, you can restart all kube-proxy Pods: + +Obtain the Pod names: + +```shell +kubectl get po -n kube-system | grep kube-proxy +``` + +Delete a Pod with: + +```shell +kubectl delete po -n kube-system +``` + +New Pods that use the updated ConfigMap will be created. + +{{< note >}} +Because kubeadm deploys kube-proxy as a DaemonSet, node specific configuration is unsupported. +{{< /note >}} + +### Applying CoreDNS configuration changes + +#### Updating the CoreDNS Deployment and Service + +kubeadm deploys CoreDNS as a Deployment called `coredns` and with a Service `kube-dns`, +both in the `kube-system` namespace. + +To update any of the CoreDNS settings, you can edit the Deployment and +Service objects: + +```shell +kubectl edit deployment -n kube-system coredns +kubectl edit service -n kube-system kube-dns +``` + +#### Reflecting the CoreDNS changes + +Once the CoreDNS changes are applied you can delete the CoreDNS Pods: + +Obtain the Pod names: + +```shell +kubectl get po -n kube-system | grep coredns +``` + +Delete a Pod with: + +```shell +kubectl delete po -n kube-system +``` + +New Pods with the updated CoreDNS configuration will be created. + +{{< note >}} +kubeadm does not allow CoreDNS configuration during cluster creation and upgrade. +This means that if you execute `kubeadm upgrade apply`, your changes to the CoreDNS +objects will be lost and must be reapplied. +{{< /note >}} + +## Persisting the reconfiguration + +During the execution of `kubeadm upgrade` on a managed node, kubeadm might overwrite configuration +that was applied after the cluster was created (reconfiguration). + +### Persisting Node object reconfiguration + +kubeadm writes Labels, Taints, CRI socket and other information on the Node object for a particular +Kubernetes node. To change any of the contents of this Node object you can use: + +```shell +kubectl edit no +``` + +During `kubeadm upgrade` the contents of such a Node might get overwritten. +If you would like to persist your modifications to the Node object after upgrade, +you can prepare a [kubectl patch](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/) +and apply it to the Node object: + +```shell +kubectl patch no --patch-file +``` + +#### Persisting control plane component reconfiguration + +The main source of control plane configuration is the `ClusterConfiguration` +object stored in the cluster. To extend the static Pod manifests configuration, +[patches](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#patches) can be used. + +These patch files must remain as files on the control plane nodes to ensure that +they can be used by the `kubeadm upgrade ... --patches `. + +If reconfiguration is done to the `ClusterConfiguration` and static Pod manifests on disk, +the set of node specific patches must be updated accordingly. + +#### Persisting kubelet reconfiguration + +Any changes to the `KubeletConfiguration` stored in `/var/lib/kubelet/config.conf` will be overwritten on +`kubeadm upgrade` by downloading the contents of the cluster wide `kubelet-config` ConfigMap. +To persist kubelet node specific configuration either the file `/var/lib/kubelet/config.conf` +has to be updated manually post-upgrade or the file `/var/lib/kubelet/kubeadm-flags.env` can include flags. +The kubelet flags override the associated `KubeletConfiguration` options, but note that +some of the flags are deprecated. + +A kubelet restart will be required after changing `/var/lib/kubelet/config.conf` or +`/var/lib/kubelet/kubeadm-flags.env`. + +{{% heading "whatsnext" %}} + +- [Upgrading kubeadm clusters](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade) +- [Customizing components with the kubeadm API](/docs/setup/production-environment/tools/kubeadm/control-plane-flags) +- [Certificate management with kubeadm](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index ec24fca8e8..36e5a1a00c 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -43,7 +43,12 @@ first drain the node (or nodes) that you are upgrading. In the case of control p they could be running CoreDNS Pods or other critical workloads. For more information see [Draining nodes](/docs/tasks/administer-cluster/safely-drain-node/). - All containers are restarted after upgrade, because the container spec hash value is changed. -- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded, you can execute `systemctl status kubelet` or view the service logs with `journalctl -xeu kubelet`. +- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded, +you can execute `systemctl status kubelet` or view the service logs with `journalctl -xeu kubelet`. +- Usage of the `--config` flag of `kubeadm upgrade` with +[kubeadm configuration API types](/docs/reference/config-api/kubeadm-config.v1beta3) +with the purpose of reconfiguring the cluster is not recommended and can have unexpected results. Follow the steps in +[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure) instead.