From f0f34f3132b169cdb253638975e4150c558d3798 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 13 Apr 2017 13:09:42 -0700 Subject: [PATCH] Move Guide topic: Reserve Compute Resources. (#3382) --- _data/tasks.yml | 1 + docs/admin/node-allocatable.md | 223 +---------------- .../reserve-compute-resources.md | 230 ++++++++++++++++++ 3 files changed, 233 insertions(+), 221 deletions(-) create mode 100644 docs/tasks/administer-cluster/reserve-compute-resources.md diff --git a/_data/tasks.yml b/_data/tasks.yml index f17c6d7909..dd09b41a3a 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -89,6 +89,7 @@ toc: section: - docs/tasks/administer-cluster/overview.md - docs/tasks/administer-cluster/assign-pods-nodes.md + - docs/tasks/administer-cluster/reserve-compute-resources.md - docs/tasks/administer-cluster/dns-horizontal-autoscaling.md - docs/tasks/administer-cluster/safely-drain-node.md - docs/tasks/administer-cluster/change-pv-reclaim-policy.md diff --git a/docs/admin/node-allocatable.md b/docs/admin/node-allocatable.md index 91cd9ecdc8..c8bd4e1b80 100644 --- a/docs/admin/node-allocatable.md +++ b/docs/admin/node-allocatable.md @@ -6,225 +6,6 @@ assignees: title: Reserving Compute Resources for System Daemons --- -* TOC -{:toc} - -Kubernetes nodes can be scheduled to `Capacity`. Pods can consume all the -available capacity on a node by default. This is an issue because nodes -typically run quite a few system daemons that power the OS and Kubernetes -itself. Unless resources are set aside for these system daemons, pods and system -daemons compete for resources and lead to resource starvation issues on the -node. - -The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve -compute resources for system daemons. Kubernetes recommends cluster -administrators to configure `Node Allocatable` based on their workload density -on each node. - -## Node Allocatable - -```text - Node Capacity ---------------------------- -| kube-reserved | -|-------------------------| -| system-reserved | -|-------------------------| -| eviction-threshold | -|-------------------------| -| | -| allocatable | -| (available for pods) | -| | -| | ---------------------------- -``` - -`Allocatable` on a Kubernetes node is defined as the amount of compute resources -that are available for pods. The scheduler does not over-subscribe -`Allocatable`. `CPU` and `memory` are supported as of now. Support for `storage` -is expected to be added in the future. - -Node Allocatable is exposed as part of `v1.Node` object in the API and as part -of `kubectl describe node` in the CLI. - -Resources can be reserved for two categories of system daemons in the `kubelet`. - -### Enabling QoS and Pod level cgroups - -To properly enforce node allocatable constraints on the node, you must -enable the new cgroup hierarchy via the `--cgroups-per-qos` flag. This flag is -enabled by default. When enabled, the `kubelet` will parent all end-user pods -under a cgroup hierarchy managed by the `kubelet`. - -### Configuring a cgroup driver - -The `kubelet` supports manipulation of the cgroup hierarchy on -the host using a cgroup driver. The driver is configured via the -`--cgroup-driver` flag. - -The supported values are the following: - -* `cgroupfs` is the default driver that performs direct manipulation of the -cgroup filesystem on the host in order to manage cgroup sandboxes. -* `systemd` is an alternative driver that manages cgroup sandboxes using -transient slices for resources that are supported by that init system. - -Depending on the configuration of the associated container runtime, -operators may have to choose a particular cgroup driver to ensure -proper system behavior. For example, if operators use the `systemd` -cgroup driver provided by the `docker` runtime, the `kubelet` must -be configured to use the `systemd` cgroup driver. - -### Kube Reserved - -- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi]` -- **Kubelet Flag**: `--kube-reserved-cgroup=` - -`kube-reserved` is meant to capture resource reservation for kubernetes system -daemons like the `kubelet`, `container runtime`, `node problem detector`, etc. -It is not meant to reserve resources for system daemons that are run as pods. -`kube-reserved` is typically a function of `pod density` on the nodes. [This -performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and -`memory` usage profiles of `kubelet` and `docker engine` at multiple levels of -pod density. [This blog -post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html) -explains how the dashboard can be interpreted to come up with a suitable -`kube-reserved` reservation. - -To optionally enforce `kube-reserved` on system daemons, specify the parent -control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet -flag. - -It is recommended that the kubernetes system daemons are placed under a top -level control group (`runtime.slice` on systemd machines for example). Each -system daemon should ideally run within its own child control group. Refer to -[this -doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#recommended-cgroups-setup) -for more details on recommended control group hierarchy. - -Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't -exist. Kubelet will fail if an invalid cgroup is specified. - -### System Reserved - -- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]` -- **Kubelet Flag**: `--system-reserved-cgroup=` - - -`system-reserved` is meant to capture resource reservation for OS system daemons -like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the -`kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time. -Reserving resources for user login sessions is also recommended (`user.slice` in -systemd world). - -To optionally enforce `system-reserved` on system daemons, specify the parent -control group for OS system daemons as the value for `--system-reserved-cgroup` -kubelet flag. - -It is recommended that the OS system daemons are placed under a top level -control group (`system.slice` on systemd machines for example). - -Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't -exist. Kubelet will fail if an invalid cgroup is specified. - -### Eviction Thresholds - -- **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]` - -Memory pressure at the node level leads to System OOMs which affects the entire -node and all pods running on it. Nodes can go offline temporarily until memory -has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet -provides [`Out of Resource`](./out-of-resource.md) management. Evictions are -supported for `memory` and `storage` only. By reserving some memory via -`--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory -availability on the node drops below the reserved value. Hypothetically, if -system daemons did not exist on a node, pods cannot use more than `capacity - -eviction-hard`. For this reason, resources reserved for evictions are not -available for pods. - -### Enforcing Node Allocatable - -- **Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]` - -The scheduler treats `Allocatable` as the available `capacity` for pods. - -`kubelet` enforce `Allocatable` across pods by default. Enforcement is performed -by evicting pods whenever the overall usage across all pods exceeds -`Allocatable`. More details on eviction policy can be found -[here](./out-of-resource.md#eviction-policy) This enforcement is controlled by -specifying `pods` value to the kubelet flag `--enforce-node-allocatable`. - - -Optionally, `kubelet` can be made to enforce `kube-reserved` and -`system-reserved` by specifying `kube-reserved` & `system-reserved` values in -the same flag. Note that to enforce `kube-reserved` or `system-reserved`, -`--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified -respectively. - -## General Guidelines - -System daemons are expected to be treated similar to `Guaranteed` pods. System -daemons can burst within their bounding control groups and this behavior needs -to be managed as part of kubernetes deployments. For example, `kubelet` should -have its own control group and share `Kube-reserved` resources with the -container runtime. However, Kubelet cannot burst and use up all available Node -resources if `kube-reserved` is enforced. - -Be extra careful while enforcing `system-reserved` reservation since it can lead -to critical system services being CPU starved or OOM killed on the node. The -recommendation is to enforce `system-reserved` only if a user has profiled their -nodes exhaustively to come up with precise estimates and is confident in their -ability to recover if any process in that group is oom_killed. - -* To begin with enforce `Allocatable` on `pods`. -* Once adequate monitoring and alerting is in place to track kube system - daemons, attempt to enforce `kube-reserved` based on usage heuristics. -* If absolutely necessary, enforce `system-reserved` over time. - -The resource requirements of kube system daemons may grow over time as more and -more features are added. Over time, kubernetes project will attempt to bring -down utilization of node system daemons, but that is not a priority as of now. -So expect a drop in `Allocatable` capacity in future releases. - -## Example Scenario - -Here is an example to illustrate Node Allocatable computation: - -* Node has `32Gi` of `memory` and `16 CPUs` -* `--kube-reserved` is set to `cpu=1,memory=2Gi` -* `--system-reserved` is set to `cpu=500m,memory=1Gi` -* `--eviction-hard` is set to `memory.available<500Mi` - -Under this scenario, `Allocatable` will be `14.5 CPUs` & `28.5Gi` of memory. -Scheduler ensures that the total `requests` across all pods on this node does -not exceed `28.5Gi`. Kubelet evicts pods whenever the overall memory usage -exceeds across pods exceed `28.5Gi`. If all processes on the node consume as -much CPU as they can, pods together cannot consume more than `14.5 CPUs`. - -If `kube-reserved` and/or `system-reserved` is not enforced and system daemons -exceed their reservation, `kubelet` evicts pods whenever the overall node memory -usage is higher than `31.5Gi`. - -## Feature Availability - -As of Kubernetes version 1.2, it has been possible to **optionally** specify -`kube-reserved` and `system-reserved` reservations. The scheduler switched to -using `Allocatable` instead of `Capacity` when available in the same release. - -As of Kubernetes version 1.6, `eviction-thresholds` are being considered by -computing `Allocatable`. To revert to the old behavior set -`--experimental-allocatable-ignore-eviction` kubelet flag to `true`. - -As of Kubernetes version 1.6, `kubelet` enforces `Allocatable` on pods using -control groups. To revert to the old behavior unset `--enforce-node-allocatable` -kubelet flag. Note that unless `--kube-reserved`, or `--system-reserved` or -`--eviction-hard` flags have non-default values, `Allocatable` enforcement does -not affect existing deployments. - -As of Kubernetes version 1.6, `kubelet` launches pods in their own cgroup -sandbox in a dedicated part of the cgroup hierarchy it manages. Operators are -required to drain their nodes prior to upgrade of the `kubelet` from prior -versions in order to ensure pods and their associated containers are launched in -the proper part of the cgroup hierarchy. +{% include user-guide-content-moved.md %} +[Reserving Compute Resources for System Daemons](/docs/tasks/administer-cluster/reserve-compute-resources/) diff --git a/docs/tasks/administer-cluster/reserve-compute-resources.md b/docs/tasks/administer-cluster/reserve-compute-resources.md new file mode 100644 index 0000000000..91cd9ecdc8 --- /dev/null +++ b/docs/tasks/administer-cluster/reserve-compute-resources.md @@ -0,0 +1,230 @@ +--- +assignees: +- vishh +- derekwaynecarr +- dashpole +title: Reserving Compute Resources for System Daemons +--- + +* TOC +{:toc} + +Kubernetes nodes can be scheduled to `Capacity`. Pods can consume all the +available capacity on a node by default. This is an issue because nodes +typically run quite a few system daemons that power the OS and Kubernetes +itself. Unless resources are set aside for these system daemons, pods and system +daemons compete for resources and lead to resource starvation issues on the +node. + +The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve +compute resources for system daemons. Kubernetes recommends cluster +administrators to configure `Node Allocatable` based on their workload density +on each node. + +## Node Allocatable + +```text + Node Capacity +--------------------------- +| kube-reserved | +|-------------------------| +| system-reserved | +|-------------------------| +| eviction-threshold | +|-------------------------| +| | +| allocatable | +| (available for pods) | +| | +| | +--------------------------- +``` + +`Allocatable` on a Kubernetes node is defined as the amount of compute resources +that are available for pods. The scheduler does not over-subscribe +`Allocatable`. `CPU` and `memory` are supported as of now. Support for `storage` +is expected to be added in the future. + +Node Allocatable is exposed as part of `v1.Node` object in the API and as part +of `kubectl describe node` in the CLI. + +Resources can be reserved for two categories of system daemons in the `kubelet`. + +### Enabling QoS and Pod level cgroups + +To properly enforce node allocatable constraints on the node, you must +enable the new cgroup hierarchy via the `--cgroups-per-qos` flag. This flag is +enabled by default. When enabled, the `kubelet` will parent all end-user pods +under a cgroup hierarchy managed by the `kubelet`. + +### Configuring a cgroup driver + +The `kubelet` supports manipulation of the cgroup hierarchy on +the host using a cgroup driver. The driver is configured via the +`--cgroup-driver` flag. + +The supported values are the following: + +* `cgroupfs` is the default driver that performs direct manipulation of the +cgroup filesystem on the host in order to manage cgroup sandboxes. +* `systemd` is an alternative driver that manages cgroup sandboxes using +transient slices for resources that are supported by that init system. + +Depending on the configuration of the associated container runtime, +operators may have to choose a particular cgroup driver to ensure +proper system behavior. For example, if operators use the `systemd` +cgroup driver provided by the `docker` runtime, the `kubelet` must +be configured to use the `systemd` cgroup driver. + +### Kube Reserved + +- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi]` +- **Kubelet Flag**: `--kube-reserved-cgroup=` + +`kube-reserved` is meant to capture resource reservation for kubernetes system +daemons like the `kubelet`, `container runtime`, `node problem detector`, etc. +It is not meant to reserve resources for system daemons that are run as pods. +`kube-reserved` is typically a function of `pod density` on the nodes. [This +performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and +`memory` usage profiles of `kubelet` and `docker engine` at multiple levels of +pod density. [This blog +post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html) +explains how the dashboard can be interpreted to come up with a suitable +`kube-reserved` reservation. + +To optionally enforce `kube-reserved` on system daemons, specify the parent +control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet +flag. + +It is recommended that the kubernetes system daemons are placed under a top +level control group (`runtime.slice` on systemd machines for example). Each +system daemon should ideally run within its own child control group. Refer to +[this +doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#recommended-cgroups-setup) +for more details on recommended control group hierarchy. + +Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't +exist. Kubelet will fail if an invalid cgroup is specified. + +### System Reserved + +- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]` +- **Kubelet Flag**: `--system-reserved-cgroup=` + + +`system-reserved` is meant to capture resource reservation for OS system daemons +like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the +`kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time. +Reserving resources for user login sessions is also recommended (`user.slice` in +systemd world). + +To optionally enforce `system-reserved` on system daemons, specify the parent +control group for OS system daemons as the value for `--system-reserved-cgroup` +kubelet flag. + +It is recommended that the OS system daemons are placed under a top level +control group (`system.slice` on systemd machines for example). + +Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't +exist. Kubelet will fail if an invalid cgroup is specified. + +### Eviction Thresholds + +- **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]` + +Memory pressure at the node level leads to System OOMs which affects the entire +node and all pods running on it. Nodes can go offline temporarily until memory +has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet +provides [`Out of Resource`](./out-of-resource.md) management. Evictions are +supported for `memory` and `storage` only. By reserving some memory via +`--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory +availability on the node drops below the reserved value. Hypothetically, if +system daemons did not exist on a node, pods cannot use more than `capacity - +eviction-hard`. For this reason, resources reserved for evictions are not +available for pods. + +### Enforcing Node Allocatable + +- **Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]` + +The scheduler treats `Allocatable` as the available `capacity` for pods. + +`kubelet` enforce `Allocatable` across pods by default. Enforcement is performed +by evicting pods whenever the overall usage across all pods exceeds +`Allocatable`. More details on eviction policy can be found +[here](./out-of-resource.md#eviction-policy) This enforcement is controlled by +specifying `pods` value to the kubelet flag `--enforce-node-allocatable`. + + +Optionally, `kubelet` can be made to enforce `kube-reserved` and +`system-reserved` by specifying `kube-reserved` & `system-reserved` values in +the same flag. Note that to enforce `kube-reserved` or `system-reserved`, +`--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified +respectively. + +## General Guidelines + +System daemons are expected to be treated similar to `Guaranteed` pods. System +daemons can burst within their bounding control groups and this behavior needs +to be managed as part of kubernetes deployments. For example, `kubelet` should +have its own control group and share `Kube-reserved` resources with the +container runtime. However, Kubelet cannot burst and use up all available Node +resources if `kube-reserved` is enforced. + +Be extra careful while enforcing `system-reserved` reservation since it can lead +to critical system services being CPU starved or OOM killed on the node. The +recommendation is to enforce `system-reserved` only if a user has profiled their +nodes exhaustively to come up with precise estimates and is confident in their +ability to recover if any process in that group is oom_killed. + +* To begin with enforce `Allocatable` on `pods`. +* Once adequate monitoring and alerting is in place to track kube system + daemons, attempt to enforce `kube-reserved` based on usage heuristics. +* If absolutely necessary, enforce `system-reserved` over time. + +The resource requirements of kube system daemons may grow over time as more and +more features are added. Over time, kubernetes project will attempt to bring +down utilization of node system daemons, but that is not a priority as of now. +So expect a drop in `Allocatable` capacity in future releases. + +## Example Scenario + +Here is an example to illustrate Node Allocatable computation: + +* Node has `32Gi` of `memory` and `16 CPUs` +* `--kube-reserved` is set to `cpu=1,memory=2Gi` +* `--system-reserved` is set to `cpu=500m,memory=1Gi` +* `--eviction-hard` is set to `memory.available<500Mi` + +Under this scenario, `Allocatable` will be `14.5 CPUs` & `28.5Gi` of memory. +Scheduler ensures that the total `requests` across all pods on this node does +not exceed `28.5Gi`. Kubelet evicts pods whenever the overall memory usage +exceeds across pods exceed `28.5Gi`. If all processes on the node consume as +much CPU as they can, pods together cannot consume more than `14.5 CPUs`. + +If `kube-reserved` and/or `system-reserved` is not enforced and system daemons +exceed their reservation, `kubelet` evicts pods whenever the overall node memory +usage is higher than `31.5Gi`. + +## Feature Availability + +As of Kubernetes version 1.2, it has been possible to **optionally** specify +`kube-reserved` and `system-reserved` reservations. The scheduler switched to +using `Allocatable` instead of `Capacity` when available in the same release. + +As of Kubernetes version 1.6, `eviction-thresholds` are being considered by +computing `Allocatable`. To revert to the old behavior set +`--experimental-allocatable-ignore-eviction` kubelet flag to `true`. + +As of Kubernetes version 1.6, `kubelet` enforces `Allocatable` on pods using +control groups. To revert to the old behavior unset `--enforce-node-allocatable` +kubelet flag. Note that unless `--kube-reserved`, or `--system-reserved` or +`--eviction-hard` flags have non-default values, `Allocatable` enforcement does +not affect existing deployments. + +As of Kubernetes version 1.6, `kubelet` launches pods in their own cgroup +sandbox in a dedicated part of the cgroup hierarchy it manages. Operators are +required to drain their nodes prior to upgrade of the `kubelet` from prior +versions in order to ensure pods and their associated containers are launched in +the proper part of the cgroup hierarchy. +