From 6700656cb40acd15760d611f7ba4a9e8f0bbcd5f Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 10 May 2022 11:01:05 +0100 Subject: [PATCH 1/7] Move Pod Topology Spread Constraints into scheduling These constraints apply specifically to the mechanism for placing Pods onto nodes (that is, scheduling). --- .../blog/_posts/2020-05-05-introducing-podtopologyspread.md | 2 +- .../blog/_posts/2020-09-30-writing-crl-scheduler/index.md | 2 +- content/en/docs/concepts/scheduling-eviction/_index.md | 1 + .../en/docs/concepts/scheduling-eviction/kube-scheduler.md | 2 +- .../topology-spread-constraints.md} | 0 content/en/docs/concepts/workloads/pods/_index.md | 2 +- content/en/docs/contribute/style/diagram-guide.md | 2 +- .../reference/command-line-tools-reference/feature-gates.md | 6 +++--- content/en/docs/reference/scheduling/config.md | 2 +- content/en/docs/setup/best-practices/multiple-zones.md | 2 +- static/_redirects | 1 + 11 files changed, 12 insertions(+), 10 deletions(-) rename content/en/docs/concepts/{workloads/pods/pod-topology-spread-constraints.md => scheduling-eviction/topology-spread-constraints.md} (100%) diff --git a/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md b/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md index dfed13ca43..eab08ed299 100644 --- a/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md +++ b/content/en/blog/_posts/2020-05-05-introducing-podtopologyspread.md @@ -67,7 +67,7 @@ Let's see an example of a cluster to understand this API. As the feature name "PodTopologySpread" implies, the basic usage of this feature is to run your workload with an absolute even manner (maxSkew=1), or relatively even manner (maxSkew>=2). See the [official -document](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) +document](/docs/concepts/scheduling-eviction/topology-spread-constraints/) for more details. In addition to this basic usage, there are some advanced usage examples that diff --git a/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md b/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md index 0fab185f98..dc4b97db2a 100644 --- a/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md +++ b/content/en/blog/_posts/2020-09-30-writing-crl-scheduler/index.md @@ -70,7 +70,7 @@ To correct the latter issue, we now employ a "hunt and peck" approach to removin ### 1. Upgrade to kubernetes 1.18 and make use of Pod Topology Spread Constraints While this seems like it could have been the perfect solution, at the time of writing Kubernetes 1.18 was unavailable on the two most common managed Kubernetes services in public cloud, EKS and GKE. -Furthermore, [pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) were still a [beta feature in 1.18](https://v1-18.docs.kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) which meant that it [wasn't guaranteed to be available in managed clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#kubernetes_feature_choices) even when v1.18 became available. +Furthermore, [pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) were still a beta feature in 1.18 which meant that it [wasn't guaranteed to be available in managed clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#kubernetes_feature_choices) even when v1.18 became available. The entire endeavour was concerningly reminiscent of checking [caniuse.com](https://caniuse.com/) when Internet Explorer 8 was still around. ### 2. Deploy a statefulset _per zone_. diff --git a/content/en/docs/concepts/scheduling-eviction/_index.md b/content/en/docs/concepts/scheduling-eviction/_index.md index 21e9371f03..fd1c0bbf00 100644 --- a/content/en/docs/concepts/scheduling-eviction/_index.md +++ b/content/en/docs/concepts/scheduling-eviction/_index.md @@ -23,6 +23,7 @@ of terminating one or more Pods on Nodes. * [Kubernetes Scheduler](/docs/concepts/scheduling-eviction/kube-scheduler/) * [Assigning Pods to Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/) * [Pod Overhead](/docs/concepts/scheduling-eviction/pod-overhead/) +* [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) * [Taints and Tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/) * [Scheduling Framework](/docs/concepts/scheduling-eviction/scheduling-framework) * [Scheduler Performance Tuning](/docs/concepts/scheduling-eviction/scheduler-perf-tuning/) diff --git a/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md b/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md index df688ded9a..c27013f6b7 100644 --- a/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md +++ b/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md @@ -83,7 +83,7 @@ of the scheduler: ## {{% heading "whatsnext" %}} * Read about [scheduler performance tuning](/docs/concepts/scheduling-eviction/scheduler-perf-tuning/) -* Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) +* Read about [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) * Read the [reference documentation](/docs/reference/command-line-tools-reference/kube-scheduler/) for kube-scheduler * Read the [kube-scheduler config (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) reference * Learn about [configuring multiple schedulers](/docs/tasks/extend-kubernetes/configure-multiple-schedulers/) diff --git a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md similarity index 100% rename from content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md rename to content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md diff --git a/content/en/docs/concepts/workloads/pods/_index.md b/content/en/docs/concepts/workloads/pods/_index.md index e7b5b4dc86..77994f754f 100644 --- a/content/en/docs/concepts/workloads/pods/_index.md +++ b/content/en/docs/concepts/workloads/pods/_index.md @@ -320,12 +320,12 @@ in the Pod Lifecycle documentation. * Learn about the [lifecycle of a Pod](/docs/concepts/workloads/pods/pod-lifecycle/). * Learn about [RuntimeClass](/docs/concepts/containers/runtime-class/) and how you can use it to configure different Pods with different container runtime configurations. -* Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). * Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how you can use it to manage application availability during disruptions. * Pod is a top-level resource in the Kubernetes REST API. The {{< api-reference page="workload-resources/pod-v1" >}} object definition describes the object in detail. * [The Distributed System Toolkit: Patterns for Composite Containers](/blog/2015/06/the-distributed-system-toolkit-patterns/) explains common layouts for Pods with more than one container. +* Read about [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) To understand the context for why Kubernetes wraps a common Pod API in other resources (such as {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} or {{< glossary_tooltip text="Deployments" term_id="deployment" >}}), you can read about the prior art, including: diff --git a/content/en/docs/contribute/style/diagram-guide.md b/content/en/docs/contribute/style/diagram-guide.md index b31c2e190e..68a750e2b2 100644 --- a/content/en/docs/contribute/style/diagram-guide.md +++ b/content/en/docs/contribute/style/diagram-guide.md @@ -438,7 +438,7 @@ Note that the live editor doesn't recognize Hugo shortcodes. ### Example 1 - Pod topology spread constraints Figure 6 shows the diagram appearing in the -[Pod topology pread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/#node-labels) +[Pod topology pread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/#node-labels) page. {{< mermaid >}} diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 9d1e67b3c0..d2d740c193 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -808,7 +808,7 @@ Each feature gate is designed for enabling/disabling a specific feature: availability during update per node. See [Perform a Rolling Update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/). - `DefaultPodTopologySpread`: Enables the use of `PodTopologySpread` scheduling plugin to do - [default spreading](/docs/concepts/workloads/pods/pod-topology-spread-constraints/#internal-default-constraints). + [default spreading](/docs/concepts/scheduling-eviction/topology-spread-constraints/#internal-default-constraints). - `DelegateFSGroupToCSIDriver`: If supported by the CSI driver, delegates the role of applying `fsGroup` from a Pod's `securityContext` to the driver by passing `fsGroup` through the NodeStageVolume and NodePublishVolume CSI calls. @@ -854,7 +854,7 @@ Each feature gate is designed for enabling/disabling a specific feature: {{< glossary_tooltip text="ephemeral containers" term_id="ephemeral-container" >}} to running pods. - `EvenPodsSpread`: Enable pods to be scheduled evenly across topology domains. See - [Pod Topology Spread Constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). + [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/). - `ExecProbeTimeout`: Ensure kubelet respects exec probe timeouts. This feature gate exists in case any of your existing workloads depend on a now-corrected fault where Kubernetes ignored exec probe timeouts. See @@ -995,7 +995,7 @@ Each feature gate is designed for enabling/disabling a specific feature: - `MemoryQoS`: Enable memory protection and usage throttle on pod / container using cgroup v2 memory controller. - `MinDomainsInPodTopologySpread`: Enable `minDomains` in Pod - [topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). + [topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/). - `MixedProtocolLBService`: Enable using different protocols in the same `LoadBalancer` type Service instance. - `MountContainers`: Enable using utility containers on host as the volume mounter. diff --git a/content/en/docs/reference/scheduling/config.md b/content/en/docs/reference/scheduling/config.md index 502ffcb61e..0911058ad4 100644 --- a/content/en/docs/reference/scheduling/config.md +++ b/content/en/docs/reference/scheduling/config.md @@ -123,7 +123,7 @@ extension points: and [node affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity). Extension points: `filter`, `score`. - `PodTopologySpread`: Implements - [Pod topology spread](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). + [Pod topology spread](/docs/concepts/scheduling-eviction/topology-spread-constraints/). Extension points: `preFilter`, `filter`, `preScore`, `score`. - `NodeUnschedulable`: Filters out nodes that have `.spec.unschedulable` set to true. diff --git a/content/en/docs/setup/best-practices/multiple-zones.md b/content/en/docs/setup/best-practices/multiple-zones.md index 8f51a3bd06..3bce166937 100644 --- a/content/en/docs/setup/best-practices/multiple-zones.md +++ b/content/en/docs/setup/best-practices/multiple-zones.md @@ -63,7 +63,7 @@ These labels can include If your cluster spans multiple zones or regions, you can use node labels in conjunction with -[Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) +[Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) to control how Pods are spread across your cluster among fault domains: regions, zones, and even specific nodes. These hints enable the diff --git a/static/_redirects b/static/_redirects index 6e274595a8..4680bce31a 100644 --- a/static/_redirects +++ b/static/_redirects @@ -158,6 +158,7 @@ /docs/concepts/workloads/controllers/statefulset.md /docs/concepts/workloads/controllers/statefulset/ 301! /docs/concepts/workloads/pods/pod/ /docs/concepts/workloads/pods/ 301 /docs/concepts/workloads/pods/pod-overview/ /docs/concepts/workloads/pods/ 301 +/docs/concepts/workloads/pods/pod-topology-spread-constraints/ /docs/concepts/scheduling-eviction/topology-spread-constraints/ 301 /docs/concepts/workloads/pods/init-containers/Kubernetes/ /docs/concepts/workloads/pods/init-containers/ 301 /docs/concepts/policy/pod-security-policy/ /docs/concepts/security/pod-security-policy/ 301 From 3225a082dfbd5a69c1bb2c271e57d861b2f403ec Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 10 May 2022 11:07:55 +0100 Subject: [PATCH 2/7] Mention topology spread constraints appropriately The concept Assigning Pods to Nodes should mention Pod topology spread constraints. This commit ensures that it does. --- .../concepts/scheduling-eviction/assign-pod-node.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md index db9f1d900d..8b458cd38b 100644 --- a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md +++ b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md @@ -29,6 +29,7 @@ specific Pods: * [nodeSelector](#nodeselector) field matching against [node labels](#built-in-node-labels) * [Affinity and anti-affinity](#affinity-and-anti-affinity) * [nodeName](#nodename) field + * [Pod topology spread constraints](#pod-topology-spread-constraints) ## Node labels {#built-in-node-labels} @@ -468,6 +469,16 @@ spec: The above Pod will only run on the node `kube-01`. +## Pod topology spread constraints + +You can use _topology spread constraints_ to control how {{< glossary_tooltip text="Pods" term_id="Pod" >}} +are spread across your cluster among failure-domains such as regions, zones, nodes, or among any other +topology domains that you define. You might do this to improve performance, expected availability, or +overall utilization. + +Read [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) +to learn more about how these work. + ## {{% heading "whatsnext" %}} * Read more about [taints and tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/) . From 311cdc386af62eb3362371566f00b7ec1e7beba6 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 10 May 2022 11:02:19 +0100 Subject: [PATCH 3/7] Reword topic Assigning Pods to Nodes - Rewording - Tidying Co-authored-by: Abdullah Gharaibeh <40361897+ahg-g@users.noreply.github.com> --- .../scheduling-eviction/assign-pod-node.md | 38 +++++++++++-------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md index 8b458cd38b..370d1a419c 100644 --- a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md +++ b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md @@ -11,20 +11,22 @@ weight: 20 -You can constrain a {{< glossary_tooltip text="Pod" term_id="pod" >}} so that it can only run on particular set of -{{< glossary_tooltip text="node(s)" term_id="node" >}}. +You can constrain a {{< glossary_tooltip text="Pod" term_id="pod" >}} so that it is +_restricted_ to run on particular {{< glossary_tooltip text="node(s)" term_id="node" >}}, +or to _prefer_ to run on particular nodes. There are several ways to do this and the recommended approaches all use [label selectors](/docs/concepts/overview/working-with-objects/labels/) to facilitate the selection. -Generally such constraints are unnecessary, as the scheduler will automatically do a reasonable placement +Often, you do not need to set any such constraints; the +{{< glossary_tooltip text="scheduler" term_id="kube-scheduler" >}} will automatically do a reasonable placement (for example, spreading your Pods across nodes so as not place Pods on a node with insufficient free resources). However, there are some circumstances where you may want to control which node -the Pod deploys to, for example, to ensure that a Pod ends up on a node with an SSD attached to it, or to co-locate Pods from two different -services that communicate a lot into the same availability zone. +the Pod deploys to, for example, to ensure that a Pod ends up on a node with an SSD attached to it, +or to co-locate Pods from two different services that communicate a lot into the same availability zone. You can use any of the following methods to choose where Kubernetes schedules -specific Pods: +specific Pods: * [nodeSelector](#nodeselector) field matching against [node labels](#built-in-node-labels) * [Affinity and anti-affinity](#affinity-and-anti-affinity) @@ -338,13 +340,15 @@ null `namespaceSelector` matches the namespace of the Pod where the rule is defi Inter-pod affinity and anti-affinity can be even more useful when they are used with higher level collections such as ReplicaSets, StatefulSets, Deployments, etc. These rules allow you to configure that a set of workloads should -be co-located in the same defined topology, eg., the same node. +be co-located in the same defined topology; for example, preferring to place two related +Pods onto the same node. -Take, for example, a three-node cluster running a web application with an -in-memory cache like redis. You could use inter-pod affinity and anti-affinity -to co-locate the web servers with the cache as much as possible. +For example: imagine a three-node cluster. You use the cluster to run a web application +and also an in-memory cache (such as Redis). For this example, also assume that latency between +the web application and the memory cache should be as low as is practical. You could use inter-pod +affinity and anti-affinity to co-locate the web servers with the cache as much as possible. -In the following example Deployment for the redis cache, the replicas get the label `app=store`. The +In the following example Deployment for the Redis cache, the replicas get the label `app=store`. The `podAntiAffinity` rule tells the scheduler to avoid placing multiple replicas with the `app=store` label on a single node. This creates each cache in a separate node. @@ -379,10 +383,10 @@ spec: image: redis:3.2-alpine ``` -The following Deployment for the web servers creates replicas with the label `app=web-store`. The -Pod affinity rule tells the scheduler to place each replica on a node that has a -Pod with the label `app=store`. The Pod anti-affinity rule tells the scheduler -to avoid placing multiple `app=web-store` servers on a single node. +The following example Deployment for the web servers creates replicas with the label `app=web-store`. +The Pod affinity rule tells the scheduler to place each replica on a node that has a Pod +with the label `app=store`. The Pod anti-affinity rule tells the scheduler never to place +multiple `app=web-store` servers on a single node. ```yaml apiVersion: apps/v1 @@ -431,6 +435,10 @@ where each web server is co-located with a cache, on three separate nodes. | *webserver-1* | *webserver-2* | *webserver-3* | | *cache-1* | *cache-2* | *cache-3* | +The overall effect is that each cache instance is likely to be accessed by a single client, that +is running on the same node. This approach aims to minimize both skew (imbalanced load) and latency. + +You might have other reasons to use Pod anti-affinity. See the [ZooKeeper tutorial](/docs/tutorials/stateful-application/zookeeper/#tolerating-node-failure) for an example of a StatefulSet configured with anti-affinity for high availability, using the same technique as this example. From 829dee0940fc31bc9e44f45aafd640534657db76 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 10 May 2022 11:14:16 +0100 Subject: [PATCH 4/7] Wrap text for Pod Topology Spread Constraints Wrapping helps localization teams pick up and work with changes. --- .../topology-spread-constraints.md | 112 ++++++++++++++---- 1 file changed, 86 insertions(+), 26 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md index 7c10da0acb..965f0d23f9 100644 --- a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md +++ b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md @@ -7,7 +7,11 @@ weight: 40 -You can use _topology spread constraints_ to control how {{< glossary_tooltip text="Pods" term_id="Pod" >}} are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization. +You can use _topology spread constraints_ to control how +{{< glossary_tooltip text="Pods" term_id="Pod" >}} are spread across your cluster +among failure-domains such as regions, zones, nodes, and other user-defined topology +domains. This can help to achieve high availability as well as efficient resource +utilization. @@ -16,7 +20,9 @@ You can use _topology spread constraints_ to control how {{< glossary_tooltip te ### Node Labels -Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in. For example, a Node might have labels: `node=node1,zone=us-east-1a,region=us-east-1` +Topology spread constraints rely on node labels to identify the topology +domain(s) that each Node is in. For example, a Node might have labels: +`node=node1,zone=us-east-1a,region=us-east-1` Suppose you have a 4-node cluster with the following labels: @@ -48,7 +54,9 @@ graph TB class zoneA,zoneB cluster; {{< /mermaid >}} -Instead of manually applying labels, you can also reuse the [well-known labels](/docs/reference/labels-annotations-taints/) that are created and populated automatically on most clusters. +Instead of manually applying labels, you can also reuse the +[well-known labels](/docs/reference/labels-annotations-taints/) that are created and populated +automatically on most clusters. ## Spread Constraints for Pods @@ -70,7 +78,9 @@ spec: labelSelector: ``` -You can define one or multiple `topologySpreadConstraint` to instruct the kube-scheduler how to place each incoming Pod in relation to the existing Pods across your cluster. The fields are: +You can define one or multiple `topologySpreadConstraint` to instruct the +kube-scheduler how to place each incoming Pod in relation to the existing Pods across +your cluster. The fields are: - **maxSkew** describes the degree to which Pods may be unevenly distributed. It must be greater than zero. Its semantics differs according to the value of `whenUnsatisfiable`: @@ -104,15 +114,24 @@ You can define one or multiple `topologySpreadConstraint` to instruct the kube-s in order to use it. {{< /note >}} -- **topologyKey** is the key of node labels. If two Nodes are labelled with this key and have identical values for that label, the scheduler treats both Nodes as being in the same topology. The scheduler tries to place a balanced number of Pods into each topology domain. +- **topologyKey** is the key of node labels. If two Nodes are labelled with this key + and have identical values for that label, the scheduler treats both Nodes as being + in the same topology. The scheduler tries to place a balanced number of Pods into + each topology domain. - **whenUnsatisfiable** indicates how to deal with a Pod if it doesn't satisfy the spread constraint: - `DoNotSchedule` (default) tells the scheduler not to schedule it. - `ScheduleAnyway` tells the scheduler to still schedule it while prioritizing nodes that minimize the skew. -- **labelSelector** is used to find matching Pods. Pods that match this label selector are counted to determine the number of Pods in their corresponding topology domain. See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) for more details. +- **labelSelector** is used to find matching Pods. Pods + that match this label selector are counted to determine the + number of Pods in their corresponding topology domain. + See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) + for more details. -When a Pod defines more than one `topologySpreadConstraint`, those constraints are ANDed: The kube-scheduler looks for a node for the incoming Pod that satisfies all the constraints. +When a Pod defines more than one `topologySpreadConstraint`, those constraints are +ANDed: The kube-scheduler looks for a node for the incoming Pod that satisfies all +the constraints. You can read more about this field by running `kubectl explain Pod.spec.topologySpreadConstraints`. @@ -142,9 +161,14 @@ If we want an incoming Pod to be evenly spread with existing Pods across zones, {{< codenew file="pods/topology-spread-constraints/one-constraint.yaml" >}} -`topologyKey: zone` implies the even distribution will only be applied to the nodes which have label pair "zone:<any value>" present. `whenUnsatisfiable: DoNotSchedule` tells the scheduler to let it stay pending if the incoming Pod can't satisfy the constraint. +`topologyKey: zone` implies the even distribution will only be applied to the +nodes which have label pair "zone:<any value>" present. `whenUnsatisfiable: +DoNotSchedule` tells the scheduler to let it stay pending if the incoming Pod can't +satisfy the constraint. -If the scheduler placed this incoming Pod into "zoneA", the Pods distribution would become [3, 1], hence the actual skew is 2 (3 - 1) - which violates `maxSkew: 1`. In this example, the incoming Pod can only be placed into "zoneB": +If the scheduler placed this incoming Pod into "zoneA", the Pods distribution would +become [3, 1], hence the actual skew is 2 (3 - 1) - which violates `maxSkew: 1`. In +this example, the incoming Pod can only be placed into "zoneB": {{}} graph BT @@ -189,13 +213,21 @@ graph BT You can tweak the Pod spec to meet various kinds of requirements: -- Change `maxSkew` to a bigger value like "2" so that the incoming Pod can be placed into "zoneA" as well. -- Change `topologyKey` to "node" so as to distribute the Pods evenly across nodes instead of zones. In the above example, if `maxSkew` remains "1", the incoming Pod can only be placed onto "node4". -- Change `whenUnsatisfiable: DoNotSchedule` to `whenUnsatisfiable: ScheduleAnyway` to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs are satisfied). However, it's preferred to be placed onto the topology domain which has fewer matching Pods. (Be aware that this preferability is jointly normalized with other internal scheduling priorities like resource usage ratio, etc.) +- Change `maxSkew` to a bigger value like "2" so that the incoming Pod can be placed + into "zoneA" as well. +- Change `topologyKey` to "node" so as to distribute the Pods evenly across nodes + instead of zones. In the above example, if `maxSkew` remains "1", the incoming + Pod can only be placed onto "node4". +- Change `whenUnsatisfiable: DoNotSchedule` to `whenUnsatisfiable: ScheduleAnyway` + to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs + are satisfied). However, it's preferred to be placed into the topology domain which + has fewer matching Pods. (Be aware that this preferability is jointly normalized + with other internal scheduling priorities like resource usage ratio, etc.) ### Example: Multiple TopologySpreadConstraints -This builds upon the previous example. Suppose you have a 4-node cluster where 3 Pods labeled `foo:bar` are located in node1, node2 and node3 respectively: +This builds upon the previous example. Suppose you have a 4-node cluster where 3 +Pods labeled `foo:bar` are located in node1, node2 and node3 respectively: {{}} graph BT @@ -220,7 +252,10 @@ You can use 2 TopologySpreadConstraints to control the Pods spreading on both zo {{< codenew file="pods/topology-spread-constraints/two-constraints.yaml" >}} -In this case, to match the first constraint, the incoming Pod can only be placed into "zoneB"; while in terms of the second constraint, the incoming Pod can only be placed onto "node4". Then the results of 2 constraints are ANDed, so the only viable option is to place on "node4". +In this case, to match the first constraint, the incoming Pod can only be placed into +"zoneB"; while in terms of the second constraint, the incoming Pod can only be placed +onto "node4". Then the results of 2 constraints are ANDed, so the only viable option +is to place on "node4". Multiple constraints can lead to conflicts. Suppose you have a 3-node cluster across 2 zones: @@ -243,13 +278,18 @@ graph BT class zoneA,zoneB cluster; {{< /mermaid >}} -If you apply "two-constraints.yaml" to this cluster, you will notice "mypod" stays in `Pending` state. This is because: to satisfy the first constraint, "mypod" can only placed into "zoneB"; while in terms of the second constraint, "mypod" can only be placed onto "node2". Then a joint result of "zoneB" and "node2" returns nothing. +If you apply "two-constraints.yaml" to this cluster, you will notice "mypod" stays in +`Pending` state. This is because: to satisfy the first constraint, "mypod" can only placed +into "zoneB"; while in terms of the second constraint, "mypod" can only be placed onto +"node2". Then a joint result of "zoneB" and "node2" returns nothing. -To overcome this situation, you can either increase the `maxSkew` or modify one of the constraints to use `whenUnsatisfiable: ScheduleAnyway`. +To overcome this situation, you can either increase the `maxSkew` or modify one of +the constraints to use `whenUnsatisfiable: ScheduleAnyway`. ### Interaction With Node Affinity and Node Selectors -The scheduler will skip the non-matching nodes from the skew calculations if the incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined. +The scheduler will skip the non-matching nodes from the skew calculations if the +incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined. ### Example: TopologySpreadConstraints with NodeAffinity @@ -287,11 +327,17 @@ class n5 k8s; class zoneC cluster; {{< /mermaid >}} -and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed into "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected. +and you know that "zoneC" must be excluded. In this case, you can compose the yaml +as below, so that "mypod" will be placed into "zoneB" instead of "zoneC". +Similarly `spec.nodeSelector` is also respected. {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} -The scheduler doesn't have prior knowledge of all the zones or other topology domains that a cluster has. They are determined from the existing nodes in the cluster. This could lead to a problem in autoscaled clusters, when a node pool (or node group) is scaled to zero nodes and the user is expecting them to scale up, because, in this case, those topology domains won't be considered until there is at least one node in them. +The scheduler doesn't have prior knowledge of all the zones or other topology domains +that a cluster has. They are determined from the existing nodes in the cluster. This +could lead to a problem in autoscaled clusters, when a node pool (or node group) is +scaled to zero nodes and the user is expecting them to scale up, because, in this case, +those topology domains won't be considered until there is at least one node in them. ### Other Noticeable Semantics @@ -301,10 +347,21 @@ There are some implicit conventions worth noting here: - The scheduler will bypass the nodes without `topologySpreadConstraints[*].topologyKey` present. This implies that: - 1. the Pods located on those nodes do not impact `maxSkew` calculation - in the above example, suppose "node1" does not have label "zone", then the 2 Pods will be disregarded, hence the incoming Pod will be scheduled into "zoneA". - 2. the incoming Pod has no chances to be scheduled onto such nodes - in the above example, suppose a "node5" carrying label `{zone-typo: zoneC}` joins the cluster, it will be bypassed due to the absence of label key "zone". + 1. the Pods located on those nodes do not impact `maxSkew` calculation - in the + above example, suppose "node1" does not have label "zone", then the 2 Pods will + be disregarded, hence the incoming Pod will be scheduled into "zoneA". + 2. the incoming Pod has no chances to be scheduled onto such nodes - + in the above example, suppose a "node5" carrying label `{zone-typo: zoneC}` + joins the cluster, it will be bypassed due to the absence of label key "zone". -- Be aware of what will happen if the incoming Pod's `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the above example, if we remove the incoming Pod's labels, it can still be placed into "zoneB" since the constraints are still satisfied. However, after the placement, the degree of imbalance of the cluster remains unchanged - it's still zoneA having 2 Pods which hold label {foo:bar}, and zoneB having 1 Pod which holds label {foo:bar}. So if this is not what you expect, we recommend the workload's `topologySpreadConstraints[*].labelSelector` to match its own labels. +- Be aware of what will happen if the incomingPod's + `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the + above example, if we remove the incoming Pod's labels, it can still be placed into + "zoneB" since the constraints are still satisfied. However, after the placement, + the degree of imbalance of the cluster remains unchanged - it's still zoneA + having 2 Pods which hold label {foo:bar}, and zoneB having 1 Pod which holds + label {foo:bar}. So if this is not what you expect, we recommend the workload's + `topologySpreadConstraints[*].labelSelector` to match its own labels. ### Cluster-level default constraints @@ -405,15 +462,18 @@ scheduled - more packed or more scattered. For finer control, you can specify topology spread constraints to distribute Pods across different topology domains - to achieve either high availability or cost-saving. This can also help on rolling update workloads and scaling out -replicas smoothly. See +replicas smoothly. +See [Motivation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/895-pod-topology-spread#motivation) for more details. ## Known Limitations -- There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution. -You can use [Descheduler](https://github.com/kubernetes-sigs/descheduler) to rebalance the Pods distribution. -- Pods matched on tainted nodes are respected. See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921) +- There's no guarantee that the constraints remain satisfied when Pods are removed. For + example, scaling down a Deployment may result in imbalanced Pods distribution. + You can use [Descheduler](https://github.com/kubernetes-sigs/descheduler) to rebalance the Pods distribution. +- Pods matched on tainted nodes are respected. + See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921). ## {{% heading "whatsnext" %}} From 72a070e619d31db66ad092814e84256c85494ac3 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 10 May 2022 12:54:59 +0100 Subject: [PATCH 5/7] Improve Pod Topology Spread Constraints concept MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Adjust heading levels - Link to API reference for Pod - Clarify examples - Add introductory text - Split two combined examples - Explain that Pods in a group should set the same topology spread constraints - Write headings in sentence case - Avoid using “we” --- .../topology-spread-constraints.md | 393 +++++++++++------- 1 file changed, 239 insertions(+), 154 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md index 965f0d23f9..1c1a33c5ed 100644 --- a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md +++ b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md @@ -13,16 +13,141 @@ among failure-domains such as regions, zones, nodes, and other user-defined topo domains. This can help to achieve high availability as well as efficient resource utilization. +You can set [cluster-level constraints](#cluster-level-default-constraints) as a default, +or configure topology spread constraints for individual workloads. -## Prerequisites +## Motivation -### Node Labels +Imagine that you have a cluster of up to twenty nodes, and you want to run a +{{< glossary_tooltip text="workload" term_id="workload" >}} +that automatically scales how many replicas it uses. There could be as few as +two Pods or as many as fifteen. +When there are only two Pods, you'd prefer not to have both of those Pods run on the +same node: you would run the risk that a single node failure takes your workload +offline. + +In addition to this basic usage, there are some advanced usage examples that +enable your workloads to benefit on high availability and cluster utilization. + +As you scale up and run more Pods, a different concern becomes important. Imagine +that you have three nodes running five Pods each. The nodes have enough capacity +to run that many replicas; however, the clients that interact with this workload +are split across three different datacenters (or infrastructure zones). Now you +have less concern about a single node failure, but you notice that latency is +higher than you'd like, and you are paying for network costs associated with +sending network traffic between the different zones. + +You decide that under normal operation you'd prefer to have a similar number of replicas +[scheduled](/docs/concepts/scheduling-eviction/) into each infrastructure zone, +and you'd like the cluster to self-heal in the case that there is a problem. + +Pod topology spread constraints offer you a declarative way to configure that. + + +## `topologySpreadConstraints` field + +The Pod API includes a field, `spec.topologySpreadConstraints`. Here is an example: + +```yaml +--- +apiVersion: v1 +kind: Pod +metadata: + name: example-pod +spec: + # Configure a topology spread constraint + topologySpreadConstraints: + - maxSkew: + minDomains: # optional; alpha since v1.24 + topologyKey: + whenUnsatisfiable: + labelSelector: + ### other Pod fields go here +``` + +You can read more about this field by running `kubectl explain Pod.spec.topologySpreadConstraints`. + +### Spread constraint definition + +You can define one or multiple `topologySpreadConstraints` entries to instruct the +kube-scheduler how to place each incoming Pod in relation to the existing Pods across +your cluster. Those fields are: + +- **maxSkew** describes the degree to which Pods may be unevenly distributed. You must + specify this field and the number must be greater than zero. Its semantics differ + according to the value of `whenUnsatisfiable`: + + - if you select `whenUnsatisfiable: DoNotSchedule`, then `maxSkew` defines the + maximum permitted difference between the number of matching pods in the target + topology and the _global minimum_ + (the minimum number of pods that match the label selector in a topology domain). + For example, if you have 3 zones with 2, 4 and 5 matching pods respectively, + then the global minimum is 2 and `maxSkew` is compared relative to that number. + - if you select `whenUnsatisfiable: ScheduleAnyway`, the scheduler gives higher + precedence to topologies that would help reduce the skew. + +- **minDomains** indicates a minimum number of eligible domains. This field is optional. + A domain is a particular instance of a topology. An eligible domain is a domain whose + nodes match the node selector. + + {{< note >}} + The `minDomains` field is an alpha field added in 1.24. You have to enable the + `MinDomainsInPodToplogySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) + in order to use it. + {{< /note >}} + + - The value of `minDomains` must be greater than 0, when specified. + You can only specify `minDomains` in conjunction with `whenUnsatisfiable: DoNotSchedule`. + - When the number of eligible domains with match topology keys is less than `minDomains`, + Pod topology spread treats global minimum as 0, and then the calculation of `skew` is performed. + The global minimum is the minimum number of matching Pods in an eligible domain, + or zero if the number of eligible domains is less than `minDomains`. + - When the number of eligible domains with matching topology keys equals or is greater than + `minDomains`, this value has no effect on scheduling. + - If you do not specify `minDomains`, the constraint behaves as if `minDomains` is 1. + +- **topologyKey** is the key of [node labels](#node-labels). If two Nodes are labelled + with this key and have identical values for that label, the scheduler treats both + Nodes as being in the same topology. The scheduler tries to place a balanced number + of Pods into each topology domain. + +- **whenUnsatisfiable** indicates how to deal with a Pod if it doesn't satisfy the spread constraint: + - `DoNotSchedule` (default) tells the scheduler not to schedule it. + - `ScheduleAnyway` tells the scheduler to still schedule it while prioritizing nodes that minimize the skew. + +- **labelSelector** is used to find matching Pods. Pods + that match this label selector are counted to determine the + number of Pods in their corresponding topology domain. + See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) + for more details. + +When a Pod defines more than one `topologySpreadConstraint`, those constraints are +combined using a logical AND operation: the kube-scheduler looks for a node for the incoming Pod +that satisfies all the configured constraints. + +### Node labels Topology spread constraints rely on node labels to identify the topology -domain(s) that each Node is in. For example, a Node might have labels: -`node=node1,zone=us-east-1a,region=us-east-1` +domain(s) that each {{< glossary_tooltip text="node" term_id="node" >}} is in. +For example, a node might have labels: +```yaml + region: us-east-1 + zone: us-east-1a +``` + +{{< note >}} +For brevity, this example doesn't use the +[well-known](/docs/reference/labels-annotations-taints/) label keys +`topology.kubernetes.io/zone` and `topology.kubernetes.io/region`. However, +those registered label keys are nonetheless recommended rather than the private +(unqualified) label keys `region` and `zone` that are used here. + +You can't make a reliable assumption about the meaning of a private label key +between different contexts. +{{< /note >}} + Suppose you have a 4-node cluster with the following labels: @@ -54,90 +179,27 @@ graph TB class zoneA,zoneB cluster; {{< /mermaid >}} -Instead of manually applying labels, you can also reuse the -[well-known labels](/docs/reference/labels-annotations-taints/) that are created and populated -automatically on most clusters. +## Consistency -## Spread Constraints for Pods +You should set the same Pod topology spread constraints on all pods in a group. -### API +Usually, if you are using a workload controller such as a Deployment, the pod template +takes care of this for you. If you mix different spread constraints then Kubernetes +follows the API definition of the field; however, the behavior is more likely to become +confusing and troubleshooting is less straightforward. -The API field `pod.spec.topologySpreadConstraints` is defined as below: +You need a mechanism to ensure that all the nodes in a topology domain (such as a +cloud provider region) are labelled consistently. +To avoid you needing to manually label nodes, most clusters automatically +populate well-known labels such as `topology.kubernetes.io/hostname`. Check whether +your cluster supports this. -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: mypod -spec: - topologySpreadConstraints: - - maxSkew: - minDomains: - topologyKey: - whenUnsatisfiable: - labelSelector: -``` +## Topology spread constraint examples -You can define one or multiple `topologySpreadConstraint` to instruct the -kube-scheduler how to place each incoming Pod in relation to the existing Pods across -your cluster. The fields are: +### Example: one topology spread constraint {#example-one-topologyspreadconstraint} -- **maxSkew** describes the degree to which Pods may be unevenly distributed. - It must be greater than zero. Its semantics differs according to the value of `whenUnsatisfiable`: - - - when `whenUnsatisfiable` equals to "DoNotSchedule", `maxSkew` is the maximum - permitted difference between the number of matching pods in the target - topology and the global minimum - (the minimum number of pods that match the label selector in a topology domain. - For example, if you have 3 zones with 0, 2 and 3 matching pods respectively, - The global minimum is 0). - - when `whenUnsatisfiable` equals to "ScheduleAnyway", scheduler gives higher - precedence to topologies that would help reduce the skew. - -- **minDomains** indicates a minimum number of eligible domains. - A domain is a particular instance of a topology. An eligible domain is a domain whose - nodes match the node selector. - - - The value of `minDomains` must be greater than 0, when specified. - - When the number of eligible domains with match topology keys is less than `minDomains`, - Pod topology spread treats "global minimum" as 0, and then the calculation of `skew` is performed. - The "global minimum" is the minimum number of matching Pods in an eligible domain, - or zero if the number of eligible domains is less than `minDomains`. - - When the number of eligible domains with matching topology keys equals or is greater than - `minDomains`, this value has no effect on scheduling. - - When `minDomains` is nil, the constraint behaves as if `minDomains` is 1. - - When `minDomains` is not nil, the value of `whenUnsatisfiable` must be "`DoNotSchedule`". - - {{< note >}} - The `minDomains` field is an alpha field added in 1.24. You have to enable the - `MinDomainsInPodToplogySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) - in order to use it. - {{< /note >}} - -- **topologyKey** is the key of node labels. If two Nodes are labelled with this key - and have identical values for that label, the scheduler treats both Nodes as being - in the same topology. The scheduler tries to place a balanced number of Pods into - each topology domain. - -- **whenUnsatisfiable** indicates how to deal with a Pod if it doesn't satisfy the spread constraint: - - `DoNotSchedule` (default) tells the scheduler not to schedule it. - - `ScheduleAnyway` tells the scheduler to still schedule it while prioritizing nodes that minimize the skew. - -- **labelSelector** is used to find matching Pods. Pods - that match this label selector are counted to determine the - number of Pods in their corresponding topology domain. - See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) - for more details. - -When a Pod defines more than one `topologySpreadConstraint`, those constraints are -ANDed: The kube-scheduler looks for a node for the incoming Pod that satisfies all -the constraints. - -You can read more about this field by running `kubectl explain Pod.spec.topologySpreadConstraints`. - -### Example: One TopologySpreadConstraint - -Suppose you have a 4-node cluster where 3 Pods labeled `foo:bar` are located in node1, node2 and node3 respectively: +Suppose you have a 4-node cluster where 3 Pods labelled `foo: bar` are located in +node1, node2 and node3 respectively: {{}} graph BT @@ -157,18 +219,20 @@ graph BT class zoneA,zoneB cluster; {{< /mermaid >}} -If we want an incoming Pod to be evenly spread with existing Pods across zones, the spec can be given as: +If you want an incoming Pod to be evenly spread with existing Pods across zones, you +can use a manifest similar to: {{< codenew file="pods/topology-spread-constraints/one-constraint.yaml" >}} -`topologyKey: zone` implies the even distribution will only be applied to the -nodes which have label pair "zone:<any value>" present. `whenUnsatisfiable: -DoNotSchedule` tells the scheduler to let it stay pending if the incoming Pod can't -satisfy the constraint. +From that manifest, `topologyKey: zone` implies the even distribution will only be applied +to nodes that are labelled `zone: ` (nodes that don't have a `zone` label +are skipped). The field `whenUnsatisfiable: DoNotSchedule` tells the scheduler to let the +incoming Pod stay pending if the scheduler can't find a way to satisfy the constraint. -If the scheduler placed this incoming Pod into "zoneA", the Pods distribution would -become [3, 1], hence the actual skew is 2 (3 - 1) - which violates `maxSkew: 1`. In -this example, the incoming Pod can only be placed into "zoneB": +If the scheduler placed this incoming Pod into zone `A`, the distribution of Pods would +become `[3, 1]`. That means the actual skew is then 2 (calculated as `3 - 1`), which +violates `maxSkew: 1`. To satisfy the constraints and context for this example, the +incoming Pod can only be placed onto a node in zone `B`: {{}} graph BT @@ -213,21 +277,21 @@ graph BT You can tweak the Pod spec to meet various kinds of requirements: -- Change `maxSkew` to a bigger value like "2" so that the incoming Pod can be placed - into "zoneA" as well. -- Change `topologyKey` to "node" so as to distribute the Pods evenly across nodes - instead of zones. In the above example, if `maxSkew` remains "1", the incoming - Pod can only be placed onto "node4". +- Change `maxSkew` to a bigger value - such as `2` - so that the incoming Pod can + be placed into zone `A` as well. +- Change `topologyKey` to `node` so as to distribute the Pods evenly across nodes + instead of zones. In the above example, if `maxSkew` remains `1`, the incoming + Pod can only be placed onto the node `node4`. - Change `whenUnsatisfiable: DoNotSchedule` to `whenUnsatisfiable: ScheduleAnyway` to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs are satisfied). However, it's preferred to be placed into the topology domain which - has fewer matching Pods. (Be aware that this preferability is jointly normalized - with other internal scheduling priorities like resource usage ratio, etc.) + has fewer matching Pods. (Be aware that this preference is jointly normalized + with other internal scheduling priorities such as resource usage ratio). -### Example: Multiple TopologySpreadConstraints +### Example: multiple topology spread constraints {#example-multiple-topologyspreadconstraints} This builds upon the previous example. Suppose you have a 4-node cluster where 3 -Pods labeled `foo:bar` are located in node1, node2 and node3 respectively: +existing Pods labeled `foo: bar` are located on node1, node2 and node3 respectively: {{}} graph BT @@ -248,14 +312,17 @@ graph BT class zoneA,zoneB cluster; {{< /mermaid >}} -You can use 2 TopologySpreadConstraints to control the Pods spreading on both zone and node: +You can combine two topology spread constraints to control the spread of Pods both +by node and by zone: {{< codenew file="pods/topology-spread-constraints/two-constraints.yaml" >}} -In this case, to match the first constraint, the incoming Pod can only be placed into -"zoneB"; while in terms of the second constraint, the incoming Pod can only be placed -onto "node4". Then the results of 2 constraints are ANDed, so the only viable option -is to place on "node4". +In this case, to match the first constraint, the incoming Pod can only be placed onto +nodes in zone `B`; while in terms of the second constraint, the incoming Pod can only be +scheduled to the node `node4`. The scheduler only considers options that satisfy all +defined constraints, so the only valid placement is onto node `node4`. + +### Example: conflicting topology spread constraints {#example-conflicting-topologyspreadconstraints} Multiple constraints can lead to conflicts. Suppose you have a 3-node cluster across 2 zones: @@ -278,22 +345,28 @@ graph BT class zoneA,zoneB cluster; {{< /mermaid >}} -If you apply "two-constraints.yaml" to this cluster, you will notice "mypod" stays in -`Pending` state. This is because: to satisfy the first constraint, "mypod" can only placed -into "zoneB"; while in terms of the second constraint, "mypod" can only be placed onto -"node2". Then a joint result of "zoneB" and "node2" returns nothing. +If you were to apply +[`two-constraints.yaml`](https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/topology-spread-constraints/two-constraints.yaml) +(the manifest from the previous example) +to **this** cluster, you would see that the Pod `mypod` stays in the `Pending` state. +This happens because: to satisfy the first constraint, the Pod `mypod` can only +be placed into zone `B`; while in terms of the second constraint, the Pod `mypod` +can only schedule to node `node2`. The intersection of the two constraints returns +an empty set, and the scheduler cannot place the Pod. -To overcome this situation, you can either increase the `maxSkew` or modify one of -the constraints to use `whenUnsatisfiable: ScheduleAnyway`. +To overcome this situation, you can either increase the value of `maxSkew` or modify +one of the constraints to use `whenUnsatisfiable: ScheduleAnyway`. Depending on +circumstances, you might also decide to delete an existing Pod manually - for example, +if you are troubleshooting why a bug-fix rollout is not making progress. -### Interaction With Node Affinity and Node Selectors +#### Interaction with node affinity and node selectors The scheduler will skip the non-matching nodes from the skew calculations if the incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined. -### Example: TopologySpreadConstraints with NodeAffinity +### Example: topology spread constraints with node affinity {#example-topologyspreadconstraints-with-nodeaffinity} -Suppose you have a 5-node cluster ranging from zoneA to zoneC: +Suppose you have a 5-node cluster ranging across zones A to C: {{}} graph BT @@ -327,9 +400,9 @@ class n5 k8s; class zoneC cluster; {{< /mermaid >}} -and you know that "zoneC" must be excluded. In this case, you can compose the yaml -as below, so that "mypod" will be placed into "zoneB" instead of "zoneC". -Similarly `spec.nodeSelector` is also respected. +and you know that zone `C` must be excluded. In this case, you can compose a manifest +as below, so that Pod `mypod` will be placed into zone `B` instead of zone `C`. +Similarly, Kubernetes also respects `spec.nodeSelector`. {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} @@ -339,43 +412,45 @@ could lead to a problem in autoscaled clusters, when a node pool (or node group) scaled to zero nodes and the user is expecting them to scale up, because, in this case, those topology domains won't be considered until there is at least one node in them. -### Other Noticeable Semantics +## Implicit conventions There are some implicit conventions worth noting here: - Only the Pods holding the same namespace as the incoming Pod can be matching candidates. -- The scheduler will bypass the nodes without `topologySpreadConstraints[*].topologyKey` present. This implies that: +- The scheduler bypasses any nodes that don't have any `topologySpreadConstraints[*].topologyKey` + present. This implies that: - 1. the Pods located on those nodes do not impact `maxSkew` calculation - in the - above example, suppose "node1" does not have label "zone", then the 2 Pods will - be disregarded, hence the incoming Pod will be scheduled into "zoneA". - 2. the incoming Pod has no chances to be scheduled onto such nodes - - in the above example, suppose a "node5" carrying label `{zone-typo: zoneC}` - joins the cluster, it will be bypassed due to the absence of label key "zone". + 1. any Pods located on those bypassed nodes do not impact `maxSkew` calculation - in the + above example, suppose the node `node1` does not have a label "zone", then the 2 Pods will + be disregarded, hence the incoming Pod will be scheduled into zone `A`. + 2. the incoming Pod has no chances to be scheduled onto this kind of nodes - + in the above example, suppose a node `node5` has the **mistyped** label `zone-typo: zoneC` + (and no `zone` label set). After node `node5` joins the cluster, it will be bypassed and + Pods for this workload aren't scheduled there. -- Be aware of what will happen if the incomingPod's +- Be aware of what will happen if the incoming Pod's `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the - above example, if we remove the incoming Pod's labels, it can still be placed into - "zoneB" since the constraints are still satisfied. However, after the placement, - the degree of imbalance of the cluster remains unchanged - it's still zoneA - having 2 Pods which hold label {foo:bar}, and zoneB having 1 Pod which holds - label {foo:bar}. So if this is not what you expect, we recommend the workload's - `topologySpreadConstraints[*].labelSelector` to match its own labels. + above example, if you remove the incoming Pod's labels, it can still be placed onto + nodes in zone `B`, since the constraints are still satisfied. However, after that + placement, the degree of imbalance of the cluster remains unchanged - it's still zone `A` + having 2 Pods labelled as `foo: bar`, and zone `B` having 1 Pod labelled as + `foo: bar`. If this is not what you expect, update the workload's + `topologySpreadConstraints[*].labelSelector` to match the labels in the pod template. -### Cluster-level default constraints +## Cluster-level default constraints It is possible to set default topology spread constraints for a cluster. Default topology spread constraints are applied to a Pod if, and only if: - It doesn't define any constraints in its `.spec.topologySpreadConstraints`. -- It belongs to a service, replication controller, replica set or stateful set. +- It belongs to a Service, ReplicaSet, StatefulSet or ReplicationController. -Default constraints can be set as part of the `PodTopologySpread` plugin args -in a [scheduling profile](/docs/reference/scheduling/config/#profiles). +Default constraints can be set as part of the `PodTopologySpread` plugin +arguments in a [scheduling profile](/docs/reference/scheduling/config/#profiles). The constraints are specified with the same [API above](#api), except that -`labelSelector` must be empty. The selectors are calculated from the services, -replication controllers, replica sets or stateful sets that the Pod belongs to. +`labelSelector` must be empty. The selectors are calculated from the Services, +ReplicaSets, StatefulSets or ReplicationControllers that the Pod belongs to. An example configuration might look like follows: @@ -396,12 +471,12 @@ profiles: ``` {{< note >}} -[`SelectorSpread` plugin](/docs/reference/scheduling/config/#scheduling-plugins) -is disabled by default. It's recommended to use `PodTopologySpread` to achieve similar -behavior. +The [`SelectorSpread` plugin](/docs/reference/scheduling/config/#scheduling-plugins) +is disabled by default. The Kubernetes project recommends using `PodTopologySpread` +to achieve similar behavior. {{< /note >}} -#### Built-in default constraints {#internal-default-constraints} +### Built-in default constraints {#internal-default-constraints} {{< feature-state for_k8s_version="v1.24" state="stable" >}} @@ -449,33 +524,43 @@ profiles: defaultingType: List ``` -## Comparison with PodAffinity/PodAntiAffinity +## Comparison with podAffinity and podAntiAffinity {#comparison-with-podaffinity-podantiaffinity} -In Kubernetes, directives related to "Affinity" control how Pods are -scheduled - more packed or more scattered. +In Kubernetes, [inter-Pod affinity and anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity) +control how Pods are scheduled in relation to one another - either more packed +or more scattered. -- For `PodAffinity`, you can try to pack any number of Pods into qualifying +`podAffinity` +: attracts Pods; you can try to pack any number of Pods into qualifying topology domain(s) -- For `PodAntiAffinity`, only one Pod can be scheduled into a - single topology domain. +`podAntiAffinity` +: repels Pods. If you set this to `requiredDuringSchedulingIgnoredDuringExecution` mode then + only a single Pod can be scheduled into a single topology domain; if you choose + `preferredDuringSchedulingIgnoredDuringExecution` then you lose the ability to enforce the + constraint. For finer control, you can specify topology spread constraints to distribute Pods across different topology domains - to achieve either high availability or cost-saving. This can also help on rolling update workloads and scaling out replicas smoothly. -See -[Motivation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/895-pod-topology-spread#motivation) -for more details. -## Known Limitations +For more context, see the +[Motivation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/895-pod-topology-spread#motivation) +section of the enhancement proposal about Pod topology spread constraints. + +## Known limitations - There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution. - You can use [Descheduler](https://github.com/kubernetes-sigs/descheduler) to rebalance the Pods distribution. + + You can use a tool such as the [Descheduler](https://github.com/kubernetes-sigs/descheduler) + to rebalance the Pods distribution. - Pods matched on tainted nodes are respected. See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921). ## {{% heading "whatsnext" %}} -- [Blog: Introducing PodTopologySpread](/blog/2020/05/introducing-podtopologyspread/) - explains `maxSkew` in details, as well as bringing up some advanced usage examples. +- The blog article [Introducing PodTopologySpread](/blog/2020/05/introducing-podtopologyspread/) + explains `maxSkew` in some detail, as well as covering some advanced usage examples. +- Read the [scheduling](/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling) section of + the API reference for Pod. From bfff661ac0024fee950c9c9d9be991257a36ffae Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 10 May 2022 13:00:19 +0100 Subject: [PATCH 6/7] Clarify known limitation of Pod topology spread constraints The limitation is more around cluster autoscaling; nonetheless it seems to belong under Known limitations. --- .../topology-spread-constraints.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md index 1c1a33c5ed..77f4d1ea55 100644 --- a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md +++ b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md @@ -406,12 +406,6 @@ Similarly, Kubernetes also respects `spec.nodeSelector`. {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} -The scheduler doesn't have prior knowledge of all the zones or other topology domains -that a cluster has. They are determined from the existing nodes in the cluster. This -could lead to a problem in autoscaled clusters, when a node pool (or node group) is -scaled to zero nodes and the user is expecting them to scale up, because, in this case, -those topology domains won't be considered until there is at least one node in them. - ## Implicit conventions There are some implicit conventions worth noting here: @@ -557,6 +551,16 @@ section of the enhancement proposal about Pod topology spread constraints. to rebalance the Pods distribution. - Pods matched on tainted nodes are respected. See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921). +- The scheduler doesn't have prior knowledge of all the zones or other topology + domains that a cluster has. They are determined from the existing nodes in the + cluster. This could lead to a problem in autoscaled clusters, when a node pool (or + node group) is scaled to zero nodes, and you're expecting the cluster to scale up, + because, in this case, those topology domains won't be considered until there is + at least one node in them. + You can work around this by using an cluster autoscaling tool that is aware of + Pod topology spread constraints and is also aware of the overall set of topology + domains. + ## {{% heading "whatsnext" %}} From ed58f048b9b6fa3ac83f7ecc1bf3f8cb8275f624 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Wed, 13 Jul 2022 00:40:22 +0100 Subject: [PATCH 7/7] Fix typo --- content/en/docs/contribute/style/diagram-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/contribute/style/diagram-guide.md b/content/en/docs/contribute/style/diagram-guide.md index 68a750e2b2..ac3a4fd529 100644 --- a/content/en/docs/contribute/style/diagram-guide.md +++ b/content/en/docs/contribute/style/diagram-guide.md @@ -438,7 +438,7 @@ Note that the live editor doesn't recognize Hugo shortcodes. ### Example 1 - Pod topology spread constraints Figure 6 shows the diagram appearing in the -[Pod topology pread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/#node-labels) +[Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/#node-labels) page. {{< mermaid >}}