Merge master into dev-1.22 to keep in sync.
This commit is contained in:
Victor Palade
Executable → Regular
Executable → Regular
@@ -85,7 +85,11 @@ Additionally, kubeadm informs the user if the certificate is externally managed;
|
||||
{{< /warning >}}
|
||||
|
||||
{{< note >}}
|
||||
`kubelet.conf` is not included in the list above because kubeadm configures kubelet for automatic certificate renewal.
|
||||
`kubelet.conf` is not included in the list above because kubeadm configures kubelet
|
||||
for [automatic certificate renewal](/docs/tasks/tls/certificate-rotation/)
|
||||
with rotatable certificates under `/var/lib/kubelet/pki`.
|
||||
To repair an expired kubelet client certificate see
|
||||
[Kubelet client certificate rotation fails](/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert).
|
||||
{{< /note >}}
|
||||
|
||||
{{< warning >}}
|
||||
@@ -142,7 +146,7 @@ The Kubernetes certificates normally reach their expiration date after one year.
|
||||
|
||||
## Renew certificates with the Kubernetes certificates API
|
||||
|
||||
This section provide more details about how to execute manual certificate renewal using the Kubernetes certificates API.
|
||||
This section provides more details about how to execute manual certificate renewal using the Kubernetes certificates API.
|
||||
|
||||
{{< caution >}}
|
||||
These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead.
|
||||
|
||||
@@ -51,7 +51,7 @@ The Memory Manager updates the Node Map during the startup and runtime as follow
|
||||
|
||||
This occurs once a node administrator employs `--reserved-memory` (section [Reserved memory flag](#reserved-memory-flag)). In this case, the Node Map becomes updated to reflect this reservation as illustrated in [Memory Manager KEP: Memory Maps at start-up (with examples)][5].
|
||||
|
||||
The administrator must provide `--reserved-memory` flag when `static` policy is configured.
|
||||
The administrator must provide `--reserved-memory` flag when `Static` policy is configured.
|
||||
|
||||
### Runtime
|
||||
|
||||
@@ -61,7 +61,7 @@ Important topic in the context of Memory Manager operation is the management of
|
||||
|
||||
## Memory Manager configuration
|
||||
|
||||
Other Managers should be first pre-configured (section [Pre-configuration](#pre-configuration)). Next, the Memory Manger feature should be enabled (section [Enable the Memory Manager feature](#enable-the-memory-manager-feature)) and be run with `static` policy (section [static policy](#static-policy)). Optionally, some amount of memory can be reserved for system or kubelet processes to increase node stability (section [Reserved memory flag](#reserved-memory-flag)).
|
||||
Other Managers should be first pre-configured (section [Pre-configuration](#pre-configuration)). Next, the Memory Manger feature should be enabled (section [Enable the Memory Manager feature](#enable-the-memory-manager-feature)) and be run with `Static` policy (section [Static policy](#static-policy)). Optionally, some amount of memory can be reserved for system or kubelet processes to increase node stability (section [Reserved memory flag](#reserved-memory-flag)).
|
||||
|
||||
### Policies
|
||||
|
||||
@@ -69,21 +69,21 @@ Memory Manager supports two policies. You can select a policy via a `kubelet` fl
|
||||
|
||||
Two policies can be selected:
|
||||
|
||||
* `none` (default)
|
||||
* `static`
|
||||
* `None` (default)
|
||||
* `Static`
|
||||
|
||||
#### none policy {#policy-none}
|
||||
#### None policy {#policy-none}
|
||||
|
||||
This is the default policy and does not affect the memory allocation in any way.
|
||||
It acts the same as if the Memory Manager is not present at all.
|
||||
|
||||
The `none` policy returns default topology hint. This special hint denotes that Hint Provider (Memory Manger in this case) has no preference for NUMA affinity with any resource.
|
||||
The `None` policy returns default topology hint. This special hint denotes that Hint Provider (Memory Manger in this case) has no preference for NUMA affinity with any resource.
|
||||
|
||||
#### static policy {#policy-static}
|
||||
#### Static policy {#policy-static}
|
||||
|
||||
In the case of the `Guaranteed` pod, the `static` Memory Manger policy returns topology hints relating to the set of NUMA nodes where the memory can be guaranteed, and reserves the memory through updating the internal [NodeMap][2] object.
|
||||
In the case of the `Guaranteed` pod, the `Static` Memory Manger policy returns topology hints relating to the set of NUMA nodes where the memory can be guaranteed, and reserves the memory through updating the internal [NodeMap][2] object.
|
||||
|
||||
In the case of the `BestEffort` or `Burstable` pod, the `static` Memory Manager policy sends back the default topology hint as there is no request for the guaranteed memory, and does not reserve the memory in the internal [NodeMap][2] object.
|
||||
In the case of the `BestEffort` or `Burstable` pod, the `Static` Memory Manager policy sends back the default topology hint as there is no request for the guaranteed memory, and does not reserve the memory in the internal [NodeMap][2] object.
|
||||
|
||||
### Reserved memory flag
|
||||
|
||||
@@ -100,7 +100,7 @@ The Memory Manager will not use this reserved memory for the allocation of conta
|
||||
|
||||
For example, if you have a NUMA node "NUMA0" with `10Gi` of memory available, and the `--reserved-memory` was specified to reserve `1Gi` of memory at "NUMA0", the Memory Manager assumes that only `9Gi` is available for containers.
|
||||
|
||||
You can omit this parameter, however, you should be aware that the quantity of reserved memory from all NUMA nodes should be equal to the quantity of memory specified by the [Node Allocatable feature](/docs/tasks/administer-cluster/reserve-compute-resources/). If at least one node allocatable parameter is non-zero, you will need to specify `--reserved-memory` for at least one NUMA node. In fact, `eviction-hard` threshold value is equal to `100Mi` by default, so if `static` policy is used, `--reserved-memory` is obligatory.
|
||||
You can omit this parameter, however, you should be aware that the quantity of reserved memory from all NUMA nodes should be equal to the quantity of memory specified by the [Node Allocatable feature](/docs/tasks/administer-cluster/reserve-compute-resources/). If at least one node allocatable parameter is non-zero, you will need to specify `--reserved-memory` for at least one NUMA node. In fact, `eviction-hard` threshold value is equal to `100Mi` by default, so if `Static` policy is used, `--reserved-memory` is obligatory.
|
||||
|
||||
Also, avoid the following configurations:
|
||||
1. duplicates, i.e. the same NUMA node or memory type, but with a different value;
|
||||
@@ -152,7 +152,7 @@ Here is an example of a correct configuration:
|
||||
--feature-gates=MemoryManager=true
|
||||
--kube-reserved=cpu=4,memory=4Gi
|
||||
--system-reserved=cpu=1,memory=1Gi
|
||||
--memory-manager-policy=static
|
||||
--memory-manager-policy=Static
|
||||
--reserved-memory 0:memory=3Gi --reserved-memory 1:memory=2148Mi
|
||||
```
|
||||
Let us validate the configuration above:
|
||||
@@ -163,7 +163,7 @@ Let us validate the configuration above:
|
||||
|
||||
## Placing a Pod in the Guaranteed QoS class
|
||||
|
||||
If the selected policy is anything other than `none`, the Memory Manager identifies pods that are in the `Guaranteed` QoS class. The Memory Manager provides specific topology hints to the Topology Manager for each `Guaranteed` pod. For pods in a QoS class other than `Guaranteed`, the Memory Manager provides default topology hints to the Topology Manager.
|
||||
If the selected policy is anything other than `None`, the Memory Manager identifies pods that are in the `Guaranteed` QoS class. The Memory Manager provides specific topology hints to the Topology Manager for each `Guaranteed` pod. For pods in a QoS class other than `Guaranteed`, the Memory Manager provides default topology hints to the Topology Manager.
|
||||
|
||||
The following excerpts from pod manifests assign a pod to the `Guaranteed` QoS class.
|
||||
|
||||
@@ -270,7 +270,7 @@ spec:
|
||||
Next, let us log into the node where it was deployed and examine the state file in `/var/lib/kubelet/memory_manager_state`:
|
||||
```json
|
||||
{
|
||||
"policyName":"static",
|
||||
"policyName":"Static",
|
||||
"machineState":{
|
||||
"0":{
|
||||
"numberOfAssignments":1,
|
||||
|
||||
Executable → Regular
+1
@@ -78,3 +78,4 @@ telemetry agents on the node, make sure to check with the vendor of the agent wh
|
||||
We keep the work in progress version of migration instructions for various telemetry and security agent vendors
|
||||
in [Google doc](https://docs.google.com/document/d/1ZFi4uKit63ga5sxEiZblfb-c23lFhvy6RXVPikS8wf0/edit#).
|
||||
Please contact the vendor to get up to date instructions for migrating from dockershim.
|
||||
|
||||
|
||||
@@ -17,33 +17,27 @@ itself. Unless resources are set aside for these system daemons, pods and system
|
||||
daemons compete for resources and lead to resource starvation issues on the
|
||||
node.
|
||||
|
||||
The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve
|
||||
The `kubelet` exposes a feature named 'Node Allocatable' that helps to reserve
|
||||
compute resources for system daemons. Kubernetes recommends cluster
|
||||
administrators to configure `Node Allocatable` based on their workload density
|
||||
administrators to configure 'Node Allocatable' based on their workload density
|
||||
on each node.
|
||||
|
||||
|
||||
|
||||
|
||||
## {{% heading "prerequisites" %}}
|
||||
|
||||
|
||||
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
||||
Your Kubernetes server must be at or later than version 1.17 to use
|
||||
the kubelet command line option `--reserved-cpus` to set an
|
||||
[explicitly reserved CPU list](#explicitly-reserved-cpu-list).
|
||||
|
||||
|
||||
|
||||
<!-- steps -->
|
||||
|
||||
## Node Allocatable
|
||||
|
||||
|
||||
|
||||
`Allocatable` on a Kubernetes node is defined as the amount of compute resources
|
||||
'Allocatable' on a Kubernetes node is defined as the amount of compute resources
|
||||
that are available for pods. The scheduler does not over-subscribe
|
||||
`Allocatable`. `CPU`, `memory` and `ephemeral-storage` are supported as of now.
|
||||
'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now.
|
||||
|
||||
Node Allocatable is exposed as part of `v1.Node` object in the API and as part
|
||||
of `kubectl describe node` in the CLI.
|
||||
@@ -97,8 +91,7 @@ flag.
|
||||
It is recommended that the kubernetes system daemons are placed under a top
|
||||
level control group (`runtime.slice` on systemd machines for example). Each
|
||||
system daemon should ideally run within its own child control group. Refer to
|
||||
[this
|
||||
doc](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md#recommended-cgroups-setup)
|
||||
[the design proposal](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md#recommended-cgroups-setup)
|
||||
for more details on recommended control group hierarchy.
|
||||
|
||||
Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't
|
||||
@@ -109,7 +102,6 @@ exist. Kubelet will fail if an invalid cgroup is specified.
|
||||
- **Kubelet Flag**: `--system-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]`
|
||||
- **Kubelet Flag**: `--system-reserved-cgroup=`
|
||||
|
||||
|
||||
`system-reserved` is meant to capture resource reservation for OS system daemons
|
||||
like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the
|
||||
`kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time.
|
||||
@@ -127,13 +119,14 @@ kubelet flag.
|
||||
It is recommended that the OS system daemons are placed under a top level
|
||||
control group (`system.slice` on systemd machines for example).
|
||||
|
||||
Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't
|
||||
exist. Kubelet will fail if an invalid cgroup is specified.
|
||||
Note that `kubelet` **does not** create `--system-reserved-cgroup` if it doesn't
|
||||
exist. `kubelet` will fail if an invalid cgroup is specified.
|
||||
|
||||
### Explicitly Reserved CPU List
|
||||
|
||||
{{< feature-state for_k8s_version="v1.17" state="stable" >}}
|
||||
|
||||
- **Kubelet Flag**: `--reserved-cpus=0-3`
|
||||
**Kubelet Flag**: `--reserved-cpus=0-3`
|
||||
|
||||
`reserved-cpus` is meant to define an explicit CPU set for OS system daemons and
|
||||
kubernetes system daemons. `reserved-cpus` is for systems that do not intend to
|
||||
@@ -154,14 +147,15 @@ For example: in Centos, you can do this using the tuned toolset.
|
||||
|
||||
### Eviction Thresholds
|
||||
|
||||
- **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
|
||||
**Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
|
||||
|
||||
Memory pressure at the node level leads to System OOMs which affects the entire
|
||||
node and all pods running on it. Nodes can go offline temporarily until memory
|
||||
has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
|
||||
provides [`Out of Resource`](/docs/tasks/administer-cluster/out-of-resource/) management. Evictions are
|
||||
provides [out of resource](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
|
||||
management. Evictions are
|
||||
supported for `memory` and `ephemeral-storage` only. By reserving some memory via
|
||||
`--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory
|
||||
`--eviction-hard` flag, the `kubelet` attempts to evict pods whenever memory
|
||||
availability on the node drops below the reserved value. Hypothetically, if
|
||||
system daemons did not exist on a node, pods cannot use more than `capacity -
|
||||
eviction-hard`. For this reason, resources reserved for evictions are not
|
||||
@@ -169,17 +163,17 @@ available for pods.
|
||||
|
||||
### Enforcing Node Allocatable
|
||||
|
||||
- **Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]`
|
||||
**Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]`
|
||||
|
||||
The scheduler treats `Allocatable` as the available `capacity` for pods.
|
||||
The scheduler treats 'Allocatable' as the available `capacity` for pods.
|
||||
|
||||
`kubelet` enforce `Allocatable` across pods by default. Enforcement is performed
|
||||
`kubelet` enforce 'Allocatable' across pods by default. Enforcement is performed
|
||||
by evicting pods whenever the overall usage across all pods exceeds
|
||||
`Allocatable`. More details on eviction policy can be found
|
||||
[here](/docs/tasks/administer-cluster/out-of-resource/#eviction-policy). This enforcement is controlled by
|
||||
'Allocatable'. More details on eviction policy can be found
|
||||
on the [node pressure eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
|
||||
page. This enforcement is controlled by
|
||||
specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.
|
||||
|
||||
|
||||
Optionally, `kubelet` can be made to enforce `kube-reserved` and
|
||||
`system-reserved` by specifying `kube-reserved` & `system-reserved` values in
|
||||
the same flag. Note that to enforce `kube-reserved` or `system-reserved`,
|
||||
@@ -188,10 +182,10 @@ respectively.
|
||||
|
||||
## General Guidelines
|
||||
|
||||
System daemons are expected to be treated similar to `Guaranteed` pods. System
|
||||
System daemons are expected to be treated similar to 'Guaranteed' pods. System
|
||||
daemons can burst within their bounding control groups and this behavior needs
|
||||
to be managed as part of kubernetes deployments. For example, `kubelet` should
|
||||
have its own control group and share `Kube-reserved` resources with the
|
||||
have its own control group and share `kube-reserved` resources with the
|
||||
container runtime. However, Kubelet cannot burst and use up all available Node
|
||||
resources if `kube-reserved` is enforced.
|
||||
|
||||
@@ -200,9 +194,9 @@ to critical system services being CPU starved, OOM killed, or unable
|
||||
to fork on the node. The
|
||||
recommendation is to enforce `system-reserved` only if a user has profiled their
|
||||
nodes exhaustively to come up with precise estimates and is confident in their
|
||||
ability to recover if any process in that group is oom_killed.
|
||||
ability to recover if any process in that group is oom-killed.
|
||||
|
||||
* To begin with enforce `Allocatable` on `pods`.
|
||||
* To begin with enforce 'Allocatable' on `pods`.
|
||||
* Once adequate monitoring and alerting is in place to track kube system
|
||||
daemons, attempt to enforce `kube-reserved` based on usage heuristics.
|
||||
* If absolutely necessary, enforce `system-reserved` over time.
|
||||
@@ -212,8 +206,6 @@ more features are added. Over time, kubernetes project will attempt to bring
|
||||
down utilization of node system daemons, but that is not a priority as of now.
|
||||
So expect a drop in `Allocatable` capacity in future releases.
|
||||
|
||||
|
||||
|
||||
<!-- discussion -->
|
||||
|
||||
## Example Scenario
|
||||
@@ -225,15 +217,15 @@ Here is an example to illustrate Node Allocatable computation:
|
||||
* `--system-reserved` is set to `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`
|
||||
* `--eviction-hard` is set to `memory.available<500Mi,nodefs.available<10%`
|
||||
|
||||
Under this scenario, `Allocatable` will be `14.5 CPUs`, `28.5Gi` of memory and
|
||||
Under this scenario, 'Allocatable' will be 14.5 CPUs, 28.5Gi of memory and
|
||||
`88Gi` of local storage.
|
||||
Scheduler ensures that the total memory `requests` across all pods on this node does
|
||||
not exceed `28.5Gi` and storage doesn't exceed `88Gi`.
|
||||
Kubelet evicts pods whenever the overall memory usage across pods exceeds `28.5Gi`,
|
||||
or if overall disk usage exceeds `88Gi` If all processes on the node consume as
|
||||
much CPU as they can, pods together cannot consume more than `14.5 CPUs`.
|
||||
not exceed 28.5Gi and storage doesn't exceed 88Gi.
|
||||
Kubelet evicts pods whenever the overall memory usage across pods exceeds 28.5Gi,
|
||||
or if overall disk usage exceeds 88Gi If all processes on the node consume as
|
||||
much CPU as they can, pods together cannot consume more than 14.5 CPUs.
|
||||
|
||||
If `kube-reserved` and/or `system-reserved` is not enforced and system daemons
|
||||
exceed their reservation, `kubelet` evicts pods whenever the overall node memory
|
||||
usage is higher than `31.5Gi` or `storage` is greater than `90Gi`
|
||||
usage is higher than 31.5Gi or `storage` is greater than 90Gi.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user