From 0ecd5254d9f577d4b0afa5b945f5916825fac8e1 Mon Sep 17 00:00:00 2001 From: Vishnu kannan Date: Tue, 28 Feb 2017 16:12:04 -0800 Subject: [PATCH] adding docs for node allocatable Signed-off-by: Vishnu kannan --- _data/guides.yml | 1 + docs/admin/node-allocatable.md | 144 +++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100644 docs/admin/node-allocatable.md diff --git a/_data/guides.yml b/_data/guides.yml index 9d102e99d6..2feb9a20f4 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -172,6 +172,7 @@ toc: - docs/admin/cluster-management.md - docs/admin/kubeadm.md - docs/admin/addons.md + - docs/admin/node-allocatable.md - docs/admin/audit.md - docs/admin/ha-master-gce.md - docs/admin/namespaces/index.md diff --git a/docs/admin/node-allocatable.md b/docs/admin/node-allocatable.md new file mode 100644 index 0000000000..f3277ef794 --- /dev/null +++ b/docs/admin/node-allocatable.md @@ -0,0 +1,144 @@ +--- +assignees: +-vishh +-derekwaynecarr +-dashpole +title: Reserving Compute Resources for System Daemons +--- + +* TOC +{:toc} + +Kubernetes nodes can be scheduled to `capacity`. +Pods can consume all the available capacity on a node by default. +This is an issue because nodes typically run quite a few system daemons that power the OS and Kubernetes itself. +Unless resources are set aside for these system daemons, pods and system daemons will compete for resources and lead to resource starvation issues on the node. +The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve compute resources for system daemons. +Kubernetes recommends cluster administrators to configure `Node Allocatable` based on their workload density on each node. + +## Node Allocatable + + Node Capacity +--------------------------- +| kube-reserved | +|-------------------------| +| system-reserved | +|-------------------------| +| eviction-threshold | +|-------------------------| +| | +| allocatable | +| (available | +| for pods) | +| | +| | +--------------------------- + +`Allocatable` on a Kubernetes node is defined as the amount of compute resources that are available for pods. +The scheduler does not over subscribe `allocatable`. +`CPU` and `memory` are supported as of now. +Support for `storage` will be added in the future. + +Node Allocatable is exposed as part of `v1.Node` object in the API and as part of `kubectl describe node` in the CLI. + +Resources can be reserved for two categories of system daemons in the `kubelet`. + +### Kube Reserved + +**Kubelet Flag**: `--kube-reserved=[cpu=100mi][,][memory=100Mi]` +**Kubelet Flag**: `--kube-reserved-cgroup=`/runtime.slice` + +`kube-reserved` is meant to capture resource reservation for kubernetes system daemons like the `kubelet`, `container runtime`, `node problem detector`, etc. +It is not meant to reserve resources for system daemons that are run as pods. +`kube-reserved` is typically a function of `pod density` on the nodes. +[This performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and `memory` usage profiles of `kubelet` and `docker engine` at multiple levels of pod density. +[This blog post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html) explains how the dashboard can be interpreted to come up with a suitable `kube-reserved` reservation. + +It is recommended that the kubernetes system daemons are placed under a top level control group (`system.slice` on systemd machines for example). +Each system daemon should ideally run within its own child control group. +Refer to [this doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#recommended-cgroups-setup) for more details on recommended control group hierarchy. + +To optionally enforce `kube-reserved` on system daemons, specify the parent control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet flag. + +### System Reserved + +**Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]` +**Kubelet Flag**: `--system-reserved-cgroup=`/system.slice` + + +`system-reserved` is meant to capture resource reservation for OS system daemons like `sshd`, `udev`, etc. +`system-reserved` should reserve `memory` for the `kernel` too since `kernel` memory is not accounted to pods (yet) in Kubernetes. +Reserving resources for user login sessions is also recommended (`user.slice` in systemd world). + +To optionally enforce `system-reserved` on system daemons, specify the parent control group for OS system daemons as the value for `--system-reserved-cgroup` kubelet flag. + +### Eviction Thresholds + +**Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]` + +Memory pressure at the node level leads to System OOMs which affects the entire node and all pods running on it. +Nodes can go offline temporarily until memory has been reclaimed. +To avoid (or reduce the probabilty) system OOMs kubelet provides [`Out of Resource`](./out-of-resource.md) management. +Evictions are supported for `memory` and `storage` only. +By reserving some memory via `--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory availability on the node drops below the reserved value. +Hypothetically, if system daemons did not exist on a node, pods cannot use more than `capacity - eviction-hard`. +For this reason, resources reserved for evictions will not be available for pods. + +### Enforcing Node Allocatable + +**Kubelet Flag**: `--enforce-node-allocatable=[pods][,][system-reserved][,][kube-reserved]` + +The scheduler will treat `Allocatable` as the available `capacity` for pods. + +`kubelet` will enforce `Allocatable` across pods by default. +This enforcement is controlled by specifying `pods` value to the kubelet flag `--enforce-node-allocatable`. + +Optionally, `kubelet` can be made to enforce `kube-reserved` and `system-reserved` by specifying `kube-reserved` & `system-reserved` values in the same flag. +Note that to enforce `kube-reserved` or `system-reserved`, `--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified respectively. + +## General Guidelines + +System daemons are expected to be treated similar to `Guaranteed` pods. +System daemons can burst within their bounding control groups and this behavior needs to be managed as part of kubernetes deployments. +For example, `kubelet` should have its own control group and share `Kube-reserved` resources with the container runtime. +However, Kubelet cannot burst and use up all available Node resources if `kube-reserved` is enforced. + +Be extra careful while enforcing `system-reserved` reservation since it can lead to critical system services being CPU starved or OOM killed on the node. +The recommendation is to enforce `system-reserved` only if a user has profiled their nodes exhaustively to come up with precise estimates. + +* To begin with enforce `Allocatable` on `pods`. +* Once adequate monitoring and alerting is in place to track kube system daemons, attempt to enforce `kube-reserved` based on usage heuristics. +* If aboslutely necessary, enforce `system-reserved` over time. + +The resource requirements of kube system daemons will grow over time as more and more features are added. +Over time, kubernetes will attempt to bring down utilization of node system daemons, but that is not a priority as of now. +So expect a drop in `Allocatable` capacity in future releases. + +## Example Scenario + +Here is an example to illustrate Node Allocatable computation: + +* Node has `32Gi` of `memory` and `16 CPUs` +* `--kube-reserved` is set to `cpu=1,memory=2Gi` +* `--system-reserved` is set to `cpu=500m,memory=1Gi` +* ``--eviction-hard` is set to `memory.available<500Mi` + +Under this scenario, `Allocatable` will be `14.5 CPUs` & `28.5Gi` of memory. +Scheduler will ensure that the total `requests` across all pods on this node does not exceed `28.5Gi`. +Kubelet will evict pods whenever the overall memory usage exceeds across pods exceed `28.5Gi`. +If all processes on the node consume as much CPU as they can, pods together cannot consume more than `14.5 CPUs`. + +If `kube-reserved` and/or `system-reserved` is not enforced and system daemons exceed their reservation, `kubelet` will evict pods whenever the overall node memory usage is higher than `31.5Gi`. + +## Feature Availability + +Since `v1.2`, it has been possible to **optionally** specify `kube-reserved` and `system-reserved` reservations. +The scheduler switched to using `Allocatable` instead of `Capacity` when available in the same release. + +Since `v1.6`, `eviction-thresholds` are being considered by computing `Allocatable`. +To revert to the old behavior set `--experimental-allocatable-ignore-eviction` kubelet flag to `true`. + +Since `v1.6`, `kubelet` will enforce `Allocatable` on pods using control groups. +To revert to the old behavior unset `--enforce-node-allocatable` kubelet flag. +Note that unless `--kube-reserved`, or `--system-reserved` or `--eviction-hard` flags have non-default values, `Allocatable` enforcement will not affect existing deployments. +