From 0f0e87ac53b601456581f5b4cae7bff06865697e Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Mon, 12 Oct 2020 21:32:55 +0100 Subject: [PATCH 1/3] =?UTF-8?q?Revise=20=E2=80=9CContainer=20runtimes?= =?UTF-8?q?=E2=80=9D=20getting=20started=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Reword page introduction. - Match headings to contents; before, headings were sometimes followed first by unrelated text and then, with no clear divider, text that did not relate to the heading. - Remove Fracti link (not in line with content guide). --- .../container-runtimes.md | 118 +++++++++--------- 1 file changed, 59 insertions(+), 59 deletions(-) diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index aa345587e1..e8d9a3483a 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -7,62 +7,64 @@ content_type: concept weight: 10 --- -{{< feature-state for_k8s_version="v1.6" state="stable" >}} -To run containers in Pods, Kubernetes uses a container runtime. Here are -the installation instructions for various runtimes. - +You need to install a +{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} +into each node in the cluster so that Pods can run there. This page outlines +what is involved and describes related tasks for setting up nodes. +This page lists details for using several common container runtimes with +Kubernetes, on Linux: -{{< caution >}} -A flaw was found in the way runc handled system file descriptors when running containers. -A malicious container could use this flaw to overwrite contents of the runc binary and -consequently run arbitrary commands on the container host system. - -Please refer to [CVE-2019-5736](https://access.redhat.com/security/cve/cve-2019-5736) for more -information about the issue. -{{< /caution >}} - -### Applicability +- [Docker](#docker) +- [CRI-O](#cri-o) +- [containerd](#containerd) {{< note >}} -This document is written for users installing CRI onto Linux. For other operating -systems, look for documentation specific to your platform. +For other operating systems, look for documentation specific to your platform. {{< /note >}} -### Cgroup drivers - -When systemd is chosen as the init system for a Linux distribution, the init process generates -and consumes a root control group (`cgroup`) and acts as a cgroup manager. Systemd has a tight -integration with cgroups and will allocate cgroups per process. It's possible to configure your -container runtime and the kubelet to use `cgroupfs`. Using `cgroupfs` alongside systemd means -that there will be two different cgroup managers. +## Cgroup drivers Control groups are used to constrain resources that are allocated to processes. -A single cgroup manager will simplify the view of what resources are being allocated -and will by default have a more consistent view of the available and in-use resources. When we have -two managers we end up with two views of those resources. We have seen cases in the field -where nodes that are configured to use `cgroupfs` for the kubelet and Docker, and `systemd` -for the rest of the processes running on the node becomes unstable under resource pressure. + +When [systemd](https://www.freedesktop.org/wiki/Software/systemd/) is chosen as the init +system for a Linux distribution, the init process generates and consumes a root control group +(`cgroup`) and acts as a cgroup manager. +Systemd has a tight integration with cgroups and allocates a cgroup per systemd unit. It's possible +to configure your container runtime and the kubelet to use `cgroupfs`. Using `cgroupfs` alongside +systemd means that there will be two different cgroup managers. + +A single cgroup manager simplifies the view of what resources are being allocated +and will by default have a more consistent view of the available and in-use resources. +When there are two cgroup managers on a system, you end up with two views of those resources. +In the field, people have reported cases where nodes that are configured to use `cgroupfs` +for the kubelet and Docker, but `systemd` for the rest of the processes, become unstable under +resource pressure. Changing the settings such that your container runtime and kubelet use `systemd` as the cgroup driver -stabilized the system. Please note the `native.cgroupdriver=systemd` option in the Docker setup below. +stabilized the system. To configure this for Docker, set `native.cgroupdriver=systemd`. {{< caution >}} -Changing the cgroup driver of a Node that has joined a cluster is highly unrecommended. +Changing the cgroup driver of a Node that has joined a cluster is strongly *not* recommended. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container -runtime to another cgroup driver can cause errors when trying to re-create the PodSandbox -for such existing Pods. Restarting the kubelet may not solve such errors. The recommendation -is to drain the Node from its workloads, remove it from the cluster and re-join it. +runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox +for such existing Pods. Restarting the kubelet may not solve such errors. + +If you have automation that makes it feasible, replace the node with another using the updated +configuration, or reinstall it using automation. {{< /caution >}} -## Docker +## Container runtimes -On each of your machines, install Docker. -Version 19.03.11 is recommended, but 1.13.1, 17.03, 17.06, 17.09, 18.06 and 18.09 are known to work as well. -Keep track of the latest verified Docker version in the Kubernetes release notes. +### Docker + +On each of your nodes, install Docker CE. + +The Kubernetes release notes list which versions of Docker are compatible +with that version of Kubernetes. Use the following commands to install Docker on your system: @@ -179,7 +181,7 @@ sudo systemctl restart docker {{% /tab %}} {{< /tabs >}} -If you want the docker service to start on boot, run the following command: +If you want the `docker` service to start on boot, run the following command: ```shell sudo systemctl enable docker @@ -188,9 +190,9 @@ sudo systemctl enable docker Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/) for more information. -## CRI-O +### CRI-O -This section contains the necessary steps to install `CRI-O` as CRI runtime. +This section contains the necessary steps to install CRI-O as a container runtime. Use the following commands to install CRI-O on your system: @@ -199,7 +201,7 @@ The CRI-O major and minor versions must match the Kubernetes major and minor ver For more information, see the [CRI-O compatibility matrix](https://github.com/cri-o/cri-o). {{< /note >}} -### Prerequisites +Install and configure prerequisites: ```shell sudo modprobe overlay @@ -218,9 +220,10 @@ sudo sysctl --system {{< tabs name="tab-cri-cri-o-installation" >}} {{% tab name="Debian" %}} -To install CRI-O on the following operating systems, set the environment variable $OS to the appropriate field in the following table: +To install CRI-O on the following operating systems, set the environment variable `OS` +to the appropriate value from the following table: -| Operating system | $OS | +| Operating system | `$OS` | | ---------------- | ----------------- | | Debian Unstable | `Debian_Unstable` | | Debian Testing | `Debian_Testing` | @@ -252,9 +255,9 @@ sudo apt-get install cri-o cri-o-runc {{% tab name="Ubuntu" %}} -To install on the following operating systems, set the environment variable $OS to the appropriate field in the following table: +To install on the following operating systems, set the environment variable `OS` to the appropriate field in the following table: -| Operating system | $OS | +| Operating system | `$OS` | | ---------------- | ----------------- | | Ubuntu 20.04 | `xUbuntu_20.04` | | Ubuntu 19.10 | `xUbuntu_19.10` | @@ -283,9 +286,9 @@ sudo apt-get install cri-o cri-o-runc {{% tab name="CentOS" %}} -To install on the following operating systems, set the environment variable $OS to the appropriate field in the following table: +To install on the following operating systems, set the environment variable `OS` to the appropriate field in the following table: -| Operating system | $OS | +| Operating system | `$OS` | | ---------------- | ----------------- | | Centos 8 | `CentOS_8` | | Centos 8 Stream | `CentOS_8_Stream` | @@ -316,7 +319,8 @@ sudo zypper install cri-o {{% tab name="Fedora" %}} Set `$VERSION` to the CRI-O version that matches your Kubernetes version. -For instance, if you want to install CRI-O 1.18, `VERSION=1.18` +For instance, if you want to install CRI-O 1.18, `VERSION=1.18`. + You can find available versions with: ```shell sudo dnf module list cri-o @@ -332,7 +336,7 @@ sudo dnf install cri-o {{% /tab %}} {{< /tabs >}} -### Start CRI-O +Start CRI-O: ```shell sudo systemctl daemon-reload @@ -342,13 +346,13 @@ sudo systemctl start crio Refer to the [CRI-O installation guide](https://github.com/kubernetes-sigs/cri-o#getting-started) for more information. -## Containerd +### containerd This section contains the necessary steps to use `containerd` as CRI runtime. Use the following commands to install Containerd on your system: -### Prerequisites +Install and configure prerequisites: ```shell cat <}} {{% tab name="Ubuntu 16.04" %}} @@ -470,7 +474,7 @@ Start-Service containerd {{% /tab %}} {{< /tabs >}} -### systemd +#### systemd To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc` set @@ -480,11 +484,7 @@ To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc` [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true ``` + When using kubeadm, manually configure the -[cgroup driver for kubelet](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node) - -## Other CRI runtimes: frakti - -Refer to the [Frakti QuickStart guide](https://github.com/kubernetes/frakti#quickstart) for more information. - +[cgroup driver for kubelet](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node). From 6b0165e7b59df9be8aff32fa9a884e5e0771960c Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Mon, 12 Oct 2020 21:30:57 +0100 Subject: [PATCH 2/3] Improve advice for configuring PGP keys for APT The Debian tool, APT, lets you write GPG keyrings into a file per trusted source, inside /etc/apt/trusted.gpg.d Recommend that approach, as it is easier to manage. --- .../production-environment/container-runtimes.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index e8d9a3483a..84dfc96a9f 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -81,7 +81,7 @@ sudo apt-get update && sudo apt-get install -y \ ```shell # Add Docker's official GPG key: -curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add --keyring /etc/apt/trusted.gpg.d/docker.gpg - ``` ```shell @@ -244,8 +244,8 @@ cat < /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list sudo echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list -curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key add - -curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add - +curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg +curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key add --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg - sudo apt-get update sudo apt-get install cri-o cri-o-runc @@ -387,7 +387,7 @@ sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificat ```shell ## Add Docker's official GPG key -curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add --keyring /etc/apt/trusted.gpg.d/docker.gpg - ``` ```shell From 0c9a8e8242245e7e1d211428f9a0a3461967eb2d Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Mon, 12 Oct 2020 21:37:42 +0100 Subject: [PATCH 3/3] List container runtimes in alphabetical order --- .../container-runtimes.md | 330 +++++++++--------- 1 file changed, 166 insertions(+), 164 deletions(-) diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index 84dfc96a9f..b44ad9eab0 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -18,9 +18,9 @@ what is involved and describes related tasks for setting up nodes. This page lists details for using several common container runtimes with Kubernetes, on Linux: -- [Docker](#docker) -- [CRI-O](#cri-o) - [containerd](#containerd) +- [CRI-O](#cri-o) +- [Docker](#docker) {{< note >}} For other operating systems, look for documentation specific to your platform. @@ -59,136 +59,147 @@ configuration, or reinstall it using automation. ## Container runtimes -### Docker +{{% thirdparty-content %}} -On each of your nodes, install Docker CE. +### containerd -The Kubernetes release notes list which versions of Docker are compatible -with that version of Kubernetes. +This section contains the necessary steps to use `containerd` as CRI runtime. -Use the following commands to install Docker on your system: +Use the following commands to install Containerd on your system: -{{< tabs name="tab-cri-docker-installation" >}} -{{% tab name="Ubuntu 16.04+" %}} +Install and configure prerequisites: ```shell -# (Install Docker CE) -## Set up the repository: +cat <}} +{{% tab name="Ubuntu 16.04" %}} + +```shell +# (Install containerd) +## Set up the repository ### Install packages to allow apt to use a repository over HTTPS -sudo apt-get update && sudo apt-get install -y \ - apt-transport-https ca-certificates curl software-properties-common gnupg2 +sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common ``` ```shell -# Add Docker's official GPG key: +## Add Docker's official GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add --keyring /etc/apt/trusted.gpg.d/docker.gpg - ``` ```shell -# Add the Docker apt repository: +## Add Docker apt repository. sudo add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) \ - stable" + "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" ``` ```shell -# Install Docker CE -sudo apt-get update && sudo apt-get install -y \ - containerd.io=1.2.13-2 \ - docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \ - docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) +## Install containerd +sudo apt-get update && sudo apt-get install -y containerd.io ``` ```shell -# Set up the Docker daemon -cat < /etc/containerd/config.toml ``` ```shell -sudo mkdir -p /etc/systemd/system/docker.service.d -``` - -```shell -# Restart Docker -sudo systemctl daemon-reload -sudo systemctl restart docker +# Restart containerd +sudo systemctl restart containerd ``` {{% /tab %}} {{% tab name="CentOS/RHEL 7.4+" %}} ```shell -# (Install Docker CE) +# (Install containerd) ## Set up the repository ### Install required packages sudo yum install -y yum-utils device-mapper-persistent-data lvm2 ``` ```shell -## Add the Docker repository -sudo yum-config-manager --add-repo \ - https://download.docker.com/linux/centos/docker-ce.repo +## Add docker repository +sudo yum-config-manager \ + --add-repo \ + https://download.docker.com/linux/centos/docker-ce.repo ``` ```shell -# Install Docker CE -sudo yum update -y && sudo yum install -y \ - containerd.io-1.2.13 \ - docker-ce-19.03.11 \ - docker-ce-cli-19.03.11 +## Install containerd +sudo yum update -y && sudo yum install -y containerd.io ``` ```shell -## Create /etc/docker -sudo mkdir /etc/docker +## Configure containerd +sudo mkdir -p /etc/containerd +sudo containerd config default > /etc/containerd/config.toml ``` ```shell -# Set up the Docker daemon -cat <}} -If you want the `docker` service to start on boot, run the following command: +#### systemd -```shell -sudo systemctl enable docker +To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc`, set + +``` +[plugins.cri] +systemd_cgroup = true ``` -Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/) -for more information. +When using kubeadm, manually configure the +[cgroup driver for kubelet](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node). ### CRI-O @@ -346,145 +357,136 @@ sudo systemctl start crio Refer to the [CRI-O installation guide](https://github.com/kubernetes-sigs/cri-o#getting-started) for more information. -### containerd -This section contains the necessary steps to use `containerd` as CRI runtime. -Use the following commands to install Containerd on your system: +### Docker -Install and configure prerequisites: +On each of your nodes, install Docker CE. + +The Kubernetes release notes list which versions of Docker are compatible +with that version of Kubernetes. + +Use the following commands to install Docker on your system: + +{{< tabs name="tab-cri-docker-installation" >}} +{{% tab name="Ubuntu 16.04+" %}} ```shell -cat <}} -{{% tab name="Ubuntu 16.04" %}} - -```shell -# (Install containerd) -## Set up the repository +# (Install Docker CE) +## Set up the repository: ### Install packages to allow apt to use a repository over HTTPS -sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common +sudo apt-get update && sudo apt-get install -y \ + apt-transport-https ca-certificates curl software-properties-common gnupg2 ``` ```shell -## Add Docker's official GPG key +# Add Docker's official GPG key: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add --keyring /etc/apt/trusted.gpg.d/docker.gpg - ``` ```shell -## Add Docker apt repository. +# Add the Docker apt repository: sudo add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) \ - stable" + "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" ``` ```shell -## Install containerd -sudo apt-get update && sudo apt-get install -y containerd.io +# Install Docker CE +sudo apt-get update && sudo apt-get install -y \ + containerd.io=1.2.13-2 \ + docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \ + docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) ``` ```shell -# Configure containerd -sudo mkdir -p /etc/containerd -sudo containerd config default > /etc/containerd/config.toml +# Set up the Docker daemon +cat < /etc/containerd/config.toml +## Create /etc/docker +sudo mkdir /etc/docker ``` ```shell -# Restart containerd -sudo systemctl restart containerd -``` -{{% /tab %}} -{{% tab name="Windows (PowerShell)" %}} -```powershell -# (Install containerd) -# download containerd -cmd /c curl -OL https://github.com/containerd/containerd/releases/download/v1.4.0-beta.2/containerd-1.4.0-beta.2-windows-amd64.tar.gz -cmd /c tar xvf .\containerd-1.4.0-beta.2-windows-amd64.tar.gz +# Set up the Docker daemon +cat <}} -#### systemd +If you want the `docker` service to start on boot, run the following command: -To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc` set - -``` -[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] - ... - [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] - SystemdCgroup = true +```shell +sudo systemctl enable docker ``` -When using kubeadm, manually configure the -[cgroup driver for kubelet](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node). +Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/) +for more information.