From 5b4ef96268b98e7392c7e062c183207c249ba030 Mon Sep 17 00:00:00 2001 From: Seth Jennings Date: Tue, 15 Aug 2017 11:14:01 -0500 Subject: [PATCH] add workload placement docs --- _data/tasks.yml | 1 + .../cpu-management-policies.md | 145 ++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/tasks/administer-cluster/cpu-management-policies.md diff --git a/_data/tasks.yml b/_data/tasks.yml index ddbd0dcb2a..7c96d4e271 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -140,6 +140,7 @@ toc: - docs/tasks/administer-cluster/cpu-memory-limit.md - docs/tasks/administer-cluster/out-of-resource.md - docs/tasks/administer-cluster/reserve-compute-resources.md + - docs/tasks/administer-cluster/cpu-management-policies.md - docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md - docs/tasks/administer-cluster/declare-network-policy.md - title: Install Network Policy Provider diff --git a/docs/tasks/administer-cluster/cpu-management-policies.md b/docs/tasks/administer-cluster/cpu-management-policies.md new file mode 100644 index 0000000000..3faf7de653 --- /dev/null +++ b/docs/tasks/administer-cluster/cpu-management-policies.md @@ -0,0 +1,145 @@ +--- +title: Control CPU Management Policies on the Node +--- + +* TOC +{:toc} + +Kubernetes keeps many aspects of how pods execute on nodes abstracted +from the user. This is by design.  However, some workloads require +stronger guarantees in terms of latency and/or performance in order to operate +acceptably.  The kubelet provides methods to enable more complex workload +placement policies while keeping the abstraction free from explicit placement +directives. + +## CPU Management Policies + +By default, the kubelet uses [CFS quota](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler) +to enforce pod CPU limits.  When the node runs many CPU bound pods, +the workload can move to different CPU cores depending on +whether the pod is throttled and which CPU cores are available at +scheduling time.  Many workloads are not sensitive to this migration and thus +work fine without any intervention. + +However, in workloads where CPU cache affinity and scheduling latency +significantly affect workload performance, the kubelet allows alternative CPU +management policies to determine some placement preferences on the node. + +Enable these management policies with the `--cpu-manager-policy` kubelet +option.  There are two supported policies: + +* `none`: the default, which represents the existing scheduling behavior +* `static`: allows pods with certain resource characteristics to be + granted increased CPU affinity and exclusivity on the node. + +### None policy + +The `none` policy explicitly enables the existing default CPU +affinity scheme, providing no affinity beyond what the OS scheduler does +automatically.  Limits on CPU usage for +[Guaranteed pods](/docs/tasks/configure-pod-container/quality-service-pod) +are enforced using CFS quota. + +### Static policy + +The `static` policy allows containers in `Guaranteed` pods with integer CPU +`requests` access to exclusive CPUs on the node.  This exclusivity is enforced +using the [cpuset cgroup controller](https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt). + +**Note:** System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods. +{: .note} + +This policy manages a shared pool of CPUs that initially contains all CPUs in the +node minus any reservations by the kubelet `--kube-reserved` or +`--system-reserved` options. CPUs reserved by these options are taken, in +integer quantity, from the initial shared pool in ascending order by physical +core ID.  This shared pool is the set of CPUs on which any containers in +`BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional +cpu `requests` also run on CPUs in the shared pool. Only containers that are +both part of a `Guaranteed` pod and have integer CPU `requests` are assigned +exclusive CPUs. + +**Note:** When reserving CPU with `--kube-reserved` or `--system-reserved` options, it is advised to use *integer* CPU quantities. +{: .note} + +As `Guaranteed` pods whose containers fit the requirements for being statically +assigned are scheduled to the node, CPUs are removed from the shared pool and +placed in the cpuset for the container.  CFS quota is not used to bound +the CPU usage of these containers as their usage is bound by the scheduling domain +itself. In others words, the number of CPUs in the container cpuset is equal to the integer +CPU `limit` specified in the pod spec.  This static assignment increases CPU +affinity and decreases context switches due to throttling for the CPU bound +workload. + +In the event that the shared pool is depleted the kubelet takes two actions: + +* Evict all pods that include a container that does not specify a `cpu` + quantity in `requests` as those pods now have no CPUs on which to run. +* Set a `NodeCPUPressure` node condition to `true` in the node status. When + this condition is true, the scheduler will not assign any pod to the node + that has a container which lacks a `cpu` quantity in `requests`. + +Consider the containers in the following pod specs: + +```yaml +spec: + containers: + - name: nginx + image: nginx +``` + +This pod runs in the `BestEffort` QoS class because no resource `requests` or +`limits` are specified. It is evicted if shared pool is depleted. It runs +in the shared pool. + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + requests: + memory: "100Mi" +``` + +This pod runs in the `Burstable` QoS class because resource `requests` do not +equal `limits` and the `cpu` quantity is not specified. It is +evicted if shared pool is depleted. It runs in the shared pool. + + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" + requests: + memory: "100Mi" + cpu: "1" +``` + +This pod runs in the `Burstable` QoS class because resource `requests` do not +equal `limits`. The non-zero `cpu` quantity in `requests` prevents the +shared pool from depleting. It runs in the shared pool. + + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" +``` + +This pod runs in the `Guaranteed` QoS class because only `limits` are specified +and `requests` are set equal to `limits` when not explicitly specified. The +`nginx` container is granted 2 exclusive CPUs. +