Move Guide topic: Reserve Compute Resources. (#3382)

2017-04-13 13:09:42 -07:00
parent bf2640e993
commit f0f34f3132
3 changed files with 233 additions and 221 deletions
@@ -89,6 +89,7 @@ toc:
  section:
  - docs/tasks/administer-cluster/overview.md
  - docs/tasks/administer-cluster/assign-pods-nodes.md
  - docs/tasks/administer-cluster/reserve-compute-resources.md
  - docs/tasks/administer-cluster/dns-horizontal-autoscaling.md
  - docs/tasks/administer-cluster/safely-drain-node.md
  - docs/tasks/administer-cluster/change-pv-reclaim-policy.md
@@ -6,225 +6,6 @@ assignees:
 title: Reserving Compute Resources for System Daemons
 ---
-* TOC
+{% include user-guide-content-moved.md %}
 {:toc}
 Kubernetes nodes can be scheduled to `Capacity`. Pods can consume all the
 available capacity on a node by default. This is an issue because nodes
 typically run quite a few system daemons that power the OS and Kubernetes
 itself. Unless resources are set aside for these system daemons, pods and system
 daemons compete for resources and lead to resource starvation issues on the
 node.
 The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve
 compute resources for system daemons. Kubernetes recommends cluster
 administrators to configure `Node Allocatable` based on their workload density
 on each node.
 ## Node Allocatable
 ```text
      Node Capacity
 ---------------------------
 |     kube-reserved       |
 |-------------------------|
 |     system-reserved     |
 |-------------------------|
 |    eviction-threshold   |
 |-------------------------|
 |                         |
 |      allocatable        |
 |   (available for pods)  |
 |                         |
 |                         |
 ---------------------------
 ```
 `Allocatable` on a Kubernetes node is defined as the amount of compute resources
 that are available for pods. The scheduler does not over-subscribe
 `Allocatable`. `CPU` and `memory` are supported as of now. Support for `storage`
 is expected to be added in the future.
 Node Allocatable is exposed as part of `v1.Node` object in the API and as part
 of `kubectl describe node` in the CLI.
 Resources can be reserved for two categories of system daemons in the `kubelet`.
 ### Enabling QoS and Pod level cgroups
 To properly enforce node allocatable constraints on the node, you must
 enable the new cgroup hierarchy via the `--cgroups-per-qos` flag.  This flag is
 enabled by default.  When enabled, the `kubelet` will parent all end-user pods
 under a cgroup hierarchy managed by the `kubelet`.
 ### Configuring a cgroup driver
 The `kubelet` supports manipulation of the cgroup hierarchy on
 the host using a cgroup driver. The driver is configured via the
 `--cgroup-driver` flag.
 The supported values are the following:
 * `cgroupfs` is the default driver that performs direct manipulation of the
 cgroup filesystem on the host in order to manage cgroup sandboxes.
 * `systemd` is an alternative driver that manages cgroup sandboxes using
 transient slices for resources that are supported by that init system.
 Depending on the configuration of the associated container runtime,
 operators may have to choose a particular cgroup driver to ensure
 proper system behavior.  For example, if operators use the `systemd`
 cgroup driver provided by the `docker` runtime, the `kubelet` must
 be configured to use the `systemd` cgroup driver.
 ### Kube Reserved
 - **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi]`
 - **Kubelet Flag**: `--kube-reserved-cgroup=`
 `kube-reserved` is meant to capture resource reservation for kubernetes system
 daemons like the `kubelet`, `container runtime`, `node problem detector`, etc.
 It is not meant to reserve resources for system daemons that are run as pods.
 `kube-reserved` is typically a function of `pod density` on the nodes. [This
 performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and
 `memory` usage profiles of `kubelet` and `docker engine` at multiple levels of
 pod density. [This blog
 post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html)
 explains how the dashboard can be interpreted to come up with a suitable
 `kube-reserved` reservation.
 To optionally enforce `kube-reserved` on system daemons, specify the parent
 control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet
 flag.
 It is recommended that the kubernetes system daemons are placed under a top
 level control group (`runtime.slice` on systemd machines for example). Each
 system daemon should ideally run within its own child control group. Refer to
 [this
 doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#recommended-cgroups-setup)
 for more details on recommended control group hierarchy.
 Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't
 exist. Kubelet will fail if an invalid cgroup is specified.
 ### System Reserved
 - **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]`
 - **Kubelet Flag**: `--system-reserved-cgroup=`
 `system-reserved` is meant to capture resource reservation for OS system daemons
 like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the
 `kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time.
 Reserving resources for user login sessions is also recommended (`user.slice` in
 systemd world).
 To optionally enforce `system-reserved` on system daemons, specify the parent
 control group for OS system daemons as the value for `--system-reserved-cgroup`
 kubelet flag.
 It is recommended that the OS system daemons are placed under a top level
 control group (`system.slice` on systemd machines for example).
 Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't
 exist. Kubelet will fail if an invalid cgroup is specified.
 ### Eviction Thresholds
 - **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
 Memory pressure at the node level leads to System OOMs which affects the entire
 node and all pods running on it. Nodes can go offline temporarily until memory
 has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
 provides [`Out of Resource`](./out-of-resource.md) management. Evictions are
 supported for `memory` and `storage` only. By reserving some memory via
 `--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory
 availability on the node drops below the reserved value. Hypothetically, if
 system daemons did not exist on a node, pods cannot use more than `capacity -
 eviction-hard`. For this reason, resources reserved for evictions are not
 available for pods.
 ### Enforcing Node Allocatable
 - **Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]`
 The scheduler treats `Allocatable` as the available `capacity` for pods.
 `kubelet` enforce `Allocatable` across pods by default. Enforcement is performed
 by evicting pods whenever the overall usage across all pods exceeds
 `Allocatable`. More details on eviction policy can be found
 [here](./out-of-resource.md#eviction-policy) This enforcement is controlled by
 specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.
 Optionally, `kubelet` can be made to enforce `kube-reserved` and
 `system-reserved` by specifying `kube-reserved` & `system-reserved` values in
 the same flag. Note that to enforce `kube-reserved` or `system-reserved`,
 `--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified
 respectively.
 ## General Guidelines
 System daemons are expected to be treated similar to `Guaranteed` pods. System
 daemons can burst within their bounding control groups and this behavior needs
 to be managed as part of kubernetes deployments. For example, `kubelet` should
 have its own control group and share `Kube-reserved` resources with the
 container runtime. However, Kubelet cannot burst and use up all available Node
 resources if `kube-reserved` is enforced.
 Be extra careful while enforcing `system-reserved` reservation since it can lead
 to critical system services being CPU starved or OOM killed on the node. The
 recommendation is to enforce `system-reserved` only if a user has profiled their
 nodes exhaustively to come up with precise estimates and is confident in their
 ability to recover if any process in that group is oom_killed.
 * To begin with enforce `Allocatable` on `pods`.
 * Once adequate monitoring and alerting is in place to track kube system
  daemons, attempt to enforce `kube-reserved` based on usage heuristics.
 * If absolutely necessary, enforce `system-reserved` over time.
 The resource requirements of kube system daemons may grow over time as more and
 more features are added. Over time, kubernetes project will attempt to bring
 down utilization of node system daemons, but that is not a priority as of now.
 So expect a drop in `Allocatable` capacity in future releases.
 ## Example Scenario
 Here is an example to illustrate Node Allocatable computation:
 * Node has `32Gi` of `memory` and `16 CPUs`
 * `--kube-reserved` is set to `cpu=1,memory=2Gi`
 * `--system-reserved` is set to `cpu=500m,memory=1Gi`
 * `--eviction-hard` is set to `memory.available<500Mi`
 Under this scenario, `Allocatable` will be `14.5 CPUs` & `28.5Gi` of memory.
 Scheduler ensures that the total `requests` across all pods on this node does
 not exceed `28.5Gi`. Kubelet evicts pods whenever the overall memory usage
 exceeds across pods exceed `28.5Gi`. If all processes on the node consume as
 much CPU as they can, pods together cannot consume more than `14.5 CPUs`.
 If `kube-reserved` and/or `system-reserved` is not enforced and system daemons
 exceed their reservation, `kubelet` evicts pods whenever the overall node memory
 usage is higher than `31.5Gi`.
 ## Feature Availability
 As of Kubernetes version 1.2, it has been possible to **optionally** specify
 `kube-reserved` and `system-reserved` reservations. The scheduler switched to
 using `Allocatable` instead of `Capacity` when available in the same release.
 As of Kubernetes version 1.6, `eviction-thresholds` are being considered by
 computing `Allocatable`. To revert to the old behavior set
 `--experimental-allocatable-ignore-eviction` kubelet flag to `true`.
 As of Kubernetes version 1.6, `kubelet` enforces `Allocatable` on pods using
 control groups. To revert to the old behavior unset `--enforce-node-allocatable`
 kubelet flag. Note that unless `--kube-reserved`, or `--system-reserved` or
 `--eviction-hard` flags have non-default values, `Allocatable` enforcement does
 not affect existing deployments.
 As of Kubernetes version 1.6, `kubelet` launches pods in their own cgroup
 sandbox in a dedicated part of the cgroup hierarchy it manages.  Operators are
 required to drain their nodes prior to upgrade of the `kubelet` from prior
 versions in order to ensure pods and their associated containers are launched in
 the proper part of the cgroup hierarchy.
 [Reserving Compute Resources for System Daemons](/docs/tasks/administer-cluster/reserve-compute-resources/)
@@ -0,0 +1,230 @@
 ---
 assignees:
 - vishh
 - derekwaynecarr
 - dashpole
 title: Reserving Compute Resources for System Daemons
 ---
 * TOC
 {:toc}
 Kubernetes nodes can be scheduled to `Capacity`. Pods can consume all the
 available capacity on a node by default. This is an issue because nodes
 typically run quite a few system daemons that power the OS and Kubernetes
 itself. Unless resources are set aside for these system daemons, pods and system
 daemons compete for resources and lead to resource starvation issues on the
 node.
 The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve
 compute resources for system daemons. Kubernetes recommends cluster
 administrators to configure `Node Allocatable` based on their workload density
 on each node.
 ## Node Allocatable
 ```text
      Node Capacity
 ---------------------------
 |     kube-reserved       |
 |-------------------------|
 |     system-reserved     |
 |-------------------------|
 |    eviction-threshold   |
 |-------------------------|
 |                         |
 |      allocatable        |
 |   (available for pods)  |
 |                         |
 |                         |
 ---------------------------
 ```
 `Allocatable` on a Kubernetes node is defined as the amount of compute resources
 that are available for pods. The scheduler does not over-subscribe
 `Allocatable`. `CPU` and `memory` are supported as of now. Support for `storage`
 is expected to be added in the future.
 Node Allocatable is exposed as part of `v1.Node` object in the API and as part
 of `kubectl describe node` in the CLI.
 Resources can be reserved for two categories of system daemons in the `kubelet`.
 ### Enabling QoS and Pod level cgroups
 To properly enforce node allocatable constraints on the node, you must
 enable the new cgroup hierarchy via the `--cgroups-per-qos` flag.  This flag is
 enabled by default.  When enabled, the `kubelet` will parent all end-user pods
 under a cgroup hierarchy managed by the `kubelet`.
 ### Configuring a cgroup driver
 The `kubelet` supports manipulation of the cgroup hierarchy on
 the host using a cgroup driver. The driver is configured via the
 `--cgroup-driver` flag.
 The supported values are the following:
 * `cgroupfs` is the default driver that performs direct manipulation of the
 cgroup filesystem on the host in order to manage cgroup sandboxes.
 * `systemd` is an alternative driver that manages cgroup sandboxes using
 transient slices for resources that are supported by that init system.
 Depending on the configuration of the associated container runtime,
 operators may have to choose a particular cgroup driver to ensure
 proper system behavior.  For example, if operators use the `systemd`
 cgroup driver provided by the `docker` runtime, the `kubelet` must
 be configured to use the `systemd` cgroup driver.
 ### Kube Reserved
 - **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi]`
 - **Kubelet Flag**: `--kube-reserved-cgroup=`
 `kube-reserved` is meant to capture resource reservation for kubernetes system
 daemons like the `kubelet`, `container runtime`, `node problem detector`, etc.
 It is not meant to reserve resources for system daemons that are run as pods.
 `kube-reserved` is typically a function of `pod density` on the nodes. [This
 performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and
 `memory` usage profiles of `kubelet` and `docker engine` at multiple levels of
 pod density. [This blog
 post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html)
 explains how the dashboard can be interpreted to come up with a suitable
 `kube-reserved` reservation.
 To optionally enforce `kube-reserved` on system daemons, specify the parent
 control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet
 flag.
 It is recommended that the kubernetes system daemons are placed under a top
 level control group (`runtime.slice` on systemd machines for example). Each
 system daemon should ideally run within its own child control group. Refer to
 [this
 doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#recommended-cgroups-setup)
 for more details on recommended control group hierarchy.
 Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't
 exist. Kubelet will fail if an invalid cgroup is specified.
 ### System Reserved
 - **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]`
 - **Kubelet Flag**: `--system-reserved-cgroup=`
 `system-reserved` is meant to capture resource reservation for OS system daemons
 like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the
 `kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time.
 Reserving resources for user login sessions is also recommended (`user.slice` in
 systemd world).
 To optionally enforce `system-reserved` on system daemons, specify the parent
 control group for OS system daemons as the value for `--system-reserved-cgroup`
 kubelet flag.
 It is recommended that the OS system daemons are placed under a top level
 control group (`system.slice` on systemd machines for example).
 Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't
 exist. Kubelet will fail if an invalid cgroup is specified.
 ### Eviction Thresholds
 - **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
 Memory pressure at the node level leads to System OOMs which affects the entire
 node and all pods running on it. Nodes can go offline temporarily until memory
 has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
 provides [`Out of Resource`](./out-of-resource.md) management. Evictions are
 supported for `memory` and `storage` only. By reserving some memory via
 `--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory
 availability on the node drops below the reserved value. Hypothetically, if
 system daemons did not exist on a node, pods cannot use more than `capacity -
 eviction-hard`. For this reason, resources reserved for evictions are not
 available for pods.
 ### Enforcing Node Allocatable
 - **Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]`
 The scheduler treats `Allocatable` as the available `capacity` for pods.
 `kubelet` enforce `Allocatable` across pods by default. Enforcement is performed
 by evicting pods whenever the overall usage across all pods exceeds
 `Allocatable`. More details on eviction policy can be found
 [here](./out-of-resource.md#eviction-policy) This enforcement is controlled by
 specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.
 Optionally, `kubelet` can be made to enforce `kube-reserved` and
 `system-reserved` by specifying `kube-reserved` & `system-reserved` values in
 the same flag. Note that to enforce `kube-reserved` or `system-reserved`,
 `--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified
 respectively.
 ## General Guidelines
 System daemons are expected to be treated similar to `Guaranteed` pods. System
 daemons can burst within their bounding control groups and this behavior needs
 to be managed as part of kubernetes deployments. For example, `kubelet` should
 have its own control group and share `Kube-reserved` resources with the
 container runtime. However, Kubelet cannot burst and use up all available Node
 resources if `kube-reserved` is enforced.
 Be extra careful while enforcing `system-reserved` reservation since it can lead
 to critical system services being CPU starved or OOM killed on the node. The
 recommendation is to enforce `system-reserved` only if a user has profiled their
 nodes exhaustively to come up with precise estimates and is confident in their
 ability to recover if any process in that group is oom_killed.
 * To begin with enforce `Allocatable` on `pods`.
 * Once adequate monitoring and alerting is in place to track kube system
  daemons, attempt to enforce `kube-reserved` based on usage heuristics.
 * If absolutely necessary, enforce `system-reserved` over time.
 The resource requirements of kube system daemons may grow over time as more and
 more features are added. Over time, kubernetes project will attempt to bring
 down utilization of node system daemons, but that is not a priority as of now.
 So expect a drop in `Allocatable` capacity in future releases.
 ## Example Scenario
 Here is an example to illustrate Node Allocatable computation:
 * Node has `32Gi` of `memory` and `16 CPUs`
 * `--kube-reserved` is set to `cpu=1,memory=2Gi`
 * `--system-reserved` is set to `cpu=500m,memory=1Gi`
 * `--eviction-hard` is set to `memory.available<500Mi`
 Under this scenario, `Allocatable` will be `14.5 CPUs` & `28.5Gi` of memory.
 Scheduler ensures that the total `requests` across all pods on this node does
 not exceed `28.5Gi`. Kubelet evicts pods whenever the overall memory usage
 exceeds across pods exceed `28.5Gi`. If all processes on the node consume as
 much CPU as they can, pods together cannot consume more than `14.5 CPUs`.
 If `kube-reserved` and/or `system-reserved` is not enforced and system daemons
 exceed their reservation, `kubelet` evicts pods whenever the overall node memory
 usage is higher than `31.5Gi`.
 ## Feature Availability
 As of Kubernetes version 1.2, it has been possible to **optionally** specify
 `kube-reserved` and `system-reserved` reservations. The scheduler switched to
 using `Allocatable` instead of `Capacity` when available in the same release.
 As of Kubernetes version 1.6, `eviction-thresholds` are being considered by
 computing `Allocatable`. To revert to the old behavior set
 `--experimental-allocatable-ignore-eviction` kubelet flag to `true`.
 As of Kubernetes version 1.6, `kubelet` enforces `Allocatable` on pods using
 control groups. To revert to the old behavior unset `--enforce-node-allocatable`
 kubelet flag. Note that unless `--kube-reserved`, or `--system-reserved` or
 `--eviction-hard` flags have non-default values, `Allocatable` enforcement does
 not affect existing deployments.
 As of Kubernetes version 1.6, `kubelet` launches pods in their own cgroup
 sandbox in a dedicated part of the cgroup hierarchy it manages.  Operators are
 required to drain their nodes prior to upgrade of the `kubelet` from prior
 versions in order to ensure pods and their associated containers are launched in
 the proper part of the cgroup hierarchy.