Fix several issues in "Using sysctls in a Kubernetes Cluster" (#15248)

1. Replace net.ipv4.route.min_pmtu with net.core.somaxconn in the
   example of using unsafe sysctls in containers, since the former is not
   accessible within container namespace at all.

2. Not all net.* sysctls are namespaced. Explain the correct way to
   identify the namespaced networking sysctls.
This commit is contained in:
Yang Guo
2019-07-06 04:10:35 -07:00
committed by Kubernetes Prow Robot
parent a0084c609d
commit 14b33db63c
3 changed files with 29 additions and 20 deletions
@@ -43,8 +43,8 @@ sudo sysctl -a
## Enabling Unsafe Sysctls ## Enabling Unsafe Sysctls
Sysctls are grouped into _safe_ and _unsafe_ sysctls. In addition to proper Sysctls are grouped into _safe_ and _unsafe_ sysctls. In addition to proper
namespacing a _safe_ sysctl must be properly _isolated_ between pods on the same namespacing, a _safe_ sysctl must be properly _isolated_ between pods on the
node. This means that setting a _safe_ sysctl for one pod same node. This means that setting a _safe_ sysctl for one pod
- must not have any influence on any other pod on the node - must not have any influence on any other pod on the node
- must not allow to harm the node's health - must not allow to harm the node's health
@@ -78,13 +78,13 @@ flag of the kubelet, e.g.:
```shell ```shell
kubelet --allowed-unsafe-sysctls \ kubelet --allowed-unsafe-sysctls \
'kernel.msg*,net.ipv4.route.min_pmtu' ... 'kernel.msg*,net.core.somaxconn' ...
``` ```
For {{< glossary_tooltip term_id="minikube" >}}, this can be done via the `extra-config` flag: For {{< glossary_tooltip term_id="minikube" >}}, this can be done via the `extra-config` flag:
```shell ```shell
minikube start --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.ipv4.route.min_pmtu"... minikube start --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn"...
``` ```
Only _namespaced_ sysctls can be enabled this way. Only _namespaced_ sysctls can be enabled this way.
@@ -102,7 +102,10 @@ in future versions of the Linux kernel.
- `kernel.msg*`, - `kernel.msg*`,
- `kernel.sem`, - `kernel.sem`,
- `fs.mqueue.*`, - `fs.mqueue.*`,
- `net.*`. - The parameters under `net.*` that can be set in container networking
namespace. However, there are exceptions (e.g.,
`net.netfilter.nf_conntrack_max` and `net.netfilter.nf_conntrack_expect_max`
can be set in container networking namespace but they are unnamespaced).
Sysctls with no namespace are called _node-level_ sysctls. If you need to set Sysctls with no namespace are called _node-level_ sysctls. If you need to set
them, you must manually configure them on each node's operating system, or by them, you must manually configure them on each node's operating system, or by
@@ -112,8 +115,8 @@ Use the pod securityContext to configure namespaced sysctls. The securityContext
applies to all containers in the same pod. applies to all containers in the same pod.
This example uses the pod securityContext to set a safe sysctl This example uses the pod securityContext to set a safe sysctl
`kernel.shm_rmid_forced` and two unsafe sysctls `net.ipv4.route.min_pmtu` and `kernel.shm_rmid_forced` and two unsafe sysctls `net.core.somaxconn` and
`kernel.msgmax` There is no distinction between _safe_ and _unsafe_ sysctls in `kernel.msgmax`. There is no distinction between _safe_ and _unsafe_ sysctls in
the specification. the specification.
{{< warning >}} {{< warning >}}
@@ -131,8 +134,8 @@ spec:
sysctls: sysctls:
- name: kernel.shm_rmid_forced - name: kernel.shm_rmid_forced
value: "0" value: "0"
- name: net.ipv4.route.min_pmtu - name: net.core.somaxconn
value: "552" value: "1024"
- name: kernel.msgmax - name: kernel.msgmax
value: "65536" value: "65536"
... ...
@@ -34,7 +34,10 @@ $ sudo sysctl -a
- `kernel.msg*`(内核中SystemV消息队列相关参数), - `kernel.msg*`(内核中SystemV消息队列相关参数),
- `kernel.sem`(内核中信号量参数), - `kernel.sem`(内核中信号量参数),
- `fs.mqueue.*`(内核中POSIX消息队列相关参数), - `fs.mqueue.*`(内核中POSIX消息队列相关参数),
- `net.*`(内核中网络配置项相关参数) - `net.*`(内核中网络配置项相关参数),如果它可以在容器命名空间里被更改。然而,也有一些特例
(例如,`net.netfilter.nf_conntrack_max`
`net.netfilter.nf_conntrack_expect_max`
可以在容器命名空间里被更改,但它们是非命名空间的)。
Sysctls中非命名空间级的被称为 _节点级_ ,其必须由集群管理员手动设置,要么通过节点的底层Linux分布方式(例如,通过 `/etc/sysctls.conf`),亦或在特权容器中使用Daemonset。 Sysctls中非命名空间级的被称为 _节点级_ ,其必须由集群管理员手动设置,要么通过节点的底层Linux分布方式(例如,通过 `/etc/sysctls.conf`),亦或在特权容器中使用Daemonset。
@@ -71,7 +74,7 @@ Sysctls被分为 _安全的_ 和 _不安全的_ sysctls。同一节点上的pods
sysctls。 _不安全的_ sysctls 会打上kubelet标识,在逐节点的基础上被启用,例如: sysctls。 _不安全的_ sysctls 会打上kubelet标识,在逐节点的基础上被启用,例如:
```shell ```shell
$ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.ipv4.route.min_pmtu' ... $ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.core.somaxconn' ...
``` ```
只有 _命名空间级_ sysctls 可以使用该方法启用。 只有 _命名空间级_ sysctls 可以使用该方法启用。
@@ -89,7 +92,7 @@ metadata:
name: sysctl-example name: sysctl-example
annotations: annotations:
security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1 security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1
security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.route.min_pmtu=1000,kernel.msgmax=1 2 3 security.alpha.kubernetes.io/unsafe-sysctls: net.core.somaxconn=1024,kernel.msgmax=1 2 3
spec: spec:
... ...
``` ```
@@ -136,7 +136,7 @@ flag of the kubelet, e.g.:
```shell ```shell
$ kubelet --allowed-unsafe-sysctls \ $ kubelet --allowed-unsafe-sysctls \
'kernel.msg*,net.ipv4.route.min_pmtu' ... 'kernel.msg*,net.core.somaxconn' ...
``` ```
<!-- <!--
For minikube, this can be done via the `extra-config` flag: For minikube, this can be done via the `extra-config` flag:
@@ -144,7 +144,7 @@ For minikube, this can be done via the `extra-config` flag:
如果您使用 minikube,可以通过 `extra-config` 参数来配置: 如果您使用 minikube,可以通过 `extra-config` 参数来配置:
```shell ```shell
$ minikube start --extra-config="kubelet.AllowedUnsafeSysctls=kernel.msg*,net.ipv4.route.min_pmtu"... $ minikube start --extra-config="kubelet.AllowedUnsafeSysctls=kernel.msg*,net.core.somaxconn"...
``` ```
<!-- <!--
Only _namespaced_ sysctls can be enabled this way. Only _namespaced_ sysctls can be enabled this way.
@@ -173,7 +173,10 @@ in future versions of the Linux kernel.
- `kernel.msg*`, - `kernel.msg*`,
- `kernel.sem`, - `kernel.sem`,
- `fs.mqueue.*`, - `fs.mqueue.*`,
- `net.*`. - `net.*`(内核中网络配置项相关参数),如果它可以在容器命名空间里被更改。然而,也有一些特例
(例如,`net.netfilter.nf_conntrack_max`
`net.netfilter.nf_conntrack_expect_max`
可以在容器命名空间里被更改,但它们是非命名空间的)。
<!-- <!--
Sysctls with no namespace are called _node-level_ sysctls. If you need to set Sysctls with no namespace are called _node-level_ sysctls. If you need to set
@@ -190,11 +193,11 @@ applies to all containers in the same pod.
<!-- <!--
This example uses the pod securityContext to set a safe sysctl This example uses the pod securityContext to set a safe sysctl
`kernel.shm_rmid_forced` and two unsafe sysctls `net.ipv4.route.min_pmtu` and `kernel.shm_rmid_forced` and two unsafe sysctls `net.core.somaxconn` and
`kernel.msgmax` There is no distinction between _safe_ and _unsafe_ sysctls in `kernel.msgmax` There is no distinction between _safe_ and _unsafe_ sysctls in
the specification. the specification.
---> --->
此示例中,使用 Pod SecurityContext 来对一个安全的 sysctl 参数 `kernel.shm_rmid_forced` 以及两个非安全的 sysctl 参数 `net.ipv4.route.min_pmtu``kernel.msgmax` 进行设置。在 Pod 规格中对 _安全的__非安全的_ sysctl 参数不做区分。 此示例中,使用 Pod SecurityContext 来对一个安全的 sysctl 参数 `kernel.shm_rmid_forced` 以及两个非安全的 sysctl 参数 `net.core.somaxconn``kernel.msgmax` 进行设置。在 Pod 规格中对 _安全的__非安全的_ sysctl 参数不做区分。
{{< warning >}} {{< warning >}}
<!-- <!--
@@ -214,8 +217,8 @@ spec:
sysctls: sysctls:
- name: kernel.shm_rmid_forced - name: kernel.shm_rmid_forced
value: "0" value: "0"
- name: net.ipv4.route.min_pmtu - name: net.core.somaxconn
value: "552" value: "1024"
- name: kernel.msgmax - name: kernel.msgmax
value: "65536" value: "65536"
... ...