Merge pull request #26786 from neolit123/1.21-cgroup-drivers-new-page

kubeadm: add separate page for configuring / migrating cgroup driver

content/en/docs/setup/production-environment/container-runtimes.md

@@ -48,7 +48,7 @@ Changing the settings such that your container runtime and kubelet use `systemd`
 stabilized the system. To configure this for Docker, set `native.cgroupdriver=systemd`.
 
 {{< caution >}}
-Changing the cgroup driver of a Node that has joined a cluster is strongly *not* recommended.  
+Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation.
 If the kubelet has created Pods using the semantics of one cgroup driver, changing the container
 runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox
 for such existing Pods. Restarting the kubelet may not solve such errors.
@@ -57,6 +57,11 @@ If you have automation that makes it feasible, replace the node with another usi
 configuration, or reinstall it using automation.
 {{< /caution >}}
 
+### Migrating to the `systemd` driver in kubeadm managed clusters
+
+Follow this [Migration guide](/tasks/administer-cluster/kubeadm/configure-cgroup-driver)
+if you wish to migrate to the `systemd` cgroup driver in existing kubeadm managed clusters.
+
 ## Container runtimes
 
 {{% thirdparty-content %}}

content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md

@@ -244,7 +244,7 @@ sudo mkdir -p /opt/cni/bin
 curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz
 ```
 
-Define the directory to download command files  
+Define the directory to download command files
 
 {{< note >}}
 The `DOWNLOAD_DIR` variable must be set to a writable directory.
@@ -295,33 +295,17 @@ See the [Kubeadm Troubleshooting guide](/docs/setup/production-environment/tools
 The kubelet is now restarting every few seconds, as it waits in a crashloop for
 kubeadm to tell it what to do.
 
-## Configure cgroup driver used by kubelet on control-plane node
+## Configuring a cgroup driver
 
-When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet
+Both the container runtime and the kubelet have a property called
-and set it in the `/var/lib/kubelet/config.yaml` file during runtime.
+["cgroup driver"](/docs/setup/production-environment/container-runtimes/), which is important
+for the management of cgroups on Linux machines.
 
-If you are using a different CRI, you must pass your `cgroupDriver` value to `kubeadm init`, like so:
+{{< warning >}}
-
+Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.
-```yaml
+{{< warning >}}
-apiVersion: kubelet.config.k8s.io/v1beta1
-kind: KubeletConfiguration
-cgroupDriver: <value>
-```
-
-For further details, please read [Using kubeadm init with a configuration file](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file).
-
-Please mind, that you **only** have to do that if the cgroup driver of your CRI
-is not `cgroupfs`, because that is the default value in the kubelet already.
-
-{{< note >}}
-Since `--cgroup-driver` flag has been deprecated by the kubelet, if you have that in `/var/lib/kubelet/kubeadm-flags.env`
-or `/etc/default/kubelet`(`/etc/sysconfig/kubelet` for RPMs), please remove it and use the KubeletConfiguration instead
-(stored in `/var/lib/kubelet/config.yaml` by default).
-{{< /note >}}
-
-The automatic detection of cgroup driver for other container runtimes
-like CRI-O and containerd is work in progress.
 
+See [Configuring a cgroup driver](/tasks/administer-cluster/kubeadm/configure-cgroup-driver) for more details.
 
 ## Troubleshooting
 

content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md

@@ -0,0 +1,127 @@
+---
+title: Configuring a cgroup driver
+content_type: task
+weight: 10
+---
+
+<!-- overview -->
+
+This page explains how to configure the kubelet cgroup driver to match the container
+runtime cgroup driver for kubeadm clusters.
+
+## {{% heading "prerequisites" %}}
+
+You should be familiar with the Kubernetes
+[container runtime requirements](/docs/setup/production-environment/container-runtimes).
+
+<!-- steps -->
+
+## Configuring the container runtime cgroup driver
+
+The [Container runtimes](/docs/setup/production-environment/container-runtimes) page
+explains that the `systemd` driver is recommended for kubeadm based setups instead
+of the `cgroupfs` driver, because kubeadm manages the kubelet as a systemd service.
+
+The page also provides details on how to setup a number of different container runtimes with the
+`systemd` driver by default.
+
+## Configuring the kubelet cgroup driver
+
+kubeadm allows you to pass a `KubeletConfiguration` structure during `kubeadm init`.
+This `KubeletConfiguration` can include the `cgroupDriver` field which controls the cgroup
+driver of the kubelet.
+
+{{< note >}}
+
+{{< feature-state for_k8s_version="v1.21" state="stable" >}}
+
+If the user is not setting the `cgroupDriver` field under `KubeletConfiguration`,
+`kubeadm init` will default it to `systemd`.
+{{< /note >}}
+
+A minimal example of configuring the field explicitly:
+
+```yaml
+# kubeadm-config.yaml
+kind: ClusterConfiguration
+apiVersion: kubeadm.k8s.io/v1beta2
+kubernetesVersion: v1.21
+---
+kind: KubeletConfiguration
+apiVersion: kubelet.config.k8s.io/v1beta1
+cgroupDriver: systemd
+```
+
+Such a configuration file can then be passed to the kubeadm command:
+
+```shell
+kubeadm init --config kubeadm-config.yaml
+```
+
+{{< note >}}
+Kubeadm uses the same `KubeletConfiguration` for all nodes in the cluster.
+The `KubeletConfiguration` is stored in a [ConfigMap](docs/concepts/configuration/configmap)
+object under the `kube-system` namespace.
+
+Executing the sub commands `init`, `join` and `upgrade` would result in kubeadm
+writing the `KubeletConfiguration` as a file under `/var/lib/kubelet/config.yaml`
+and passing it to the local node kubelet.
+{{< /note >}}
+
+## Using the `cgroupfs` driver
+
+As this guide explains using the `cgroupfs` driver with kubeadm is not recommended.
+
+To continue using `cgroupfs` and to prevent `kubeadm upgrade` from modifying the
+`KubeletConfiguration` cgroup driver on existing setups, you must be explicit
+about its value. This applies to a case where you do not wish future versions
+of kubeadm to apply the `systemd` driver by default.
+
+See the below section on "Modify the kubelet ConfigMap" for details on
+how to be explicit about the value.
+
+If you wish to configure a container runtime to use the `cgroupfs` driver,
+you must refer to the documentation of the container runtime of your choice.
+
+## Migrating to the `systemd` driver
+
+To change the cgroup driver of an existing kubeadm cluster to `systemd` in-place,
+a similar procedure to a kubelet upgrade is required. This must include both
+steps outlined below.
+
+{{< note >}}
+Alternatively, it is possible to replace the old nodes in the cluster with new ones
+that use the `systemd` driver. This requires executing only the first step below
+before joining the new nodes and ensuring the workloads can safely move to the new
+nodes before deleting the old nodes.
+{{< /note >}}
+
+### Modify the kubelet ConfigMap
+
+- Find the kubelet ConfigMap name using `kubectl get cm -n kube-system | grep kubelet-config`.
+- Call `kubectl edit cm kubelet-config-x.yy -n kube-system` (replace `x.yy` with
+the Kubernetes version).
+- Either modify the existing `cgroupDriver` value or add a new field that looks like this:
+
+  ```yaml
+  cgroupDriver: systemd
+  ```
+  This field must be present under the `kubelet:` section of the ConfigMap.
+
+### Update the cgroup driver on all nodes
+
+For each node in the cluster:
+
+- [Drain the node](/docs/tasks/administer-cluster/safely-drain-node) using `kubectl drain <node-name> --ignore-daemonsets`
+- Stop the kubelet using `systemctl stop kubelet`
+- Stop the container runtime
+- Modify the container runtime cgroup driver to `systemd`
+- Set `cgroupDriver: systemd` in `/var/lib/kubelet/config.yaml`
+- Start the container runtime
+- Start the kubelet using `systemctl start kubelet`
+- [Uncordon the node](/docs/tasks/administer-cluster/safely-drain-node) using `kubectl uncordon <node-name>`
+
+Execute these steps on nodes one at a time to ensure workloads
+have sufficient time to schedule on different nodes.
+
+Once the process is complete ensure that all nodes and workloads are healthy.