Add scheduler concept guide (#14637)
* Add scheduler concept page * Rename scheduling overview * Fix non-ASCII colon symbols * Reword scheduler concept page * Move scheduler performance tuning into scheduling * Signpost from overview to kube-scheduler, etc * Add whatsnext section to scheduler concept * Restructure scheduling concept Now there's a concept page for scheduling, some of the details in the performance tuning page can have a better home. * Omit link to (unwritten) scheduling extensions page * Drop deprecated / superseded filtering rules
This commit is contained in:
Tim Bannister
committed by
Kubernetes Prow Robot
Kubernetes Prow Robot
parent
6b0711ae01
commit
d550d112af
@@ -0,0 +1,5 @@
|
||||
---
|
||||
title: "Scheduling"
|
||||
weight: 90
|
||||
---
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
---
|
||||
title: Kubernetes Scheduler
|
||||
content_template: templates/concept
|
||||
weight: 60
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
In Kubernetes, _scheduling_ refers to making sure that {{< glossary_tooltip text="Pods" term_id="pod" >}}
|
||||
are matched to {{< glossary_tooltip text="Nodes" term_id="node" >}} so that
|
||||
{{< glossary_tooltip term_id="kubelet" >}} can run them.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Scheduling overview {#scheduling}
|
||||
|
||||
A scheduler watches for newly created Pods that have no Node assigned. For
|
||||
every Pod that the scheduler discovers, the scheduler becomes responsible
|
||||
for finding the best Node for that Pod to run on. The scheduler reaches
|
||||
this placement decision taking into account the scheduling principles
|
||||
described below.
|
||||
|
||||
If you want to understand why Pods are placed onto a particular Node,
|
||||
or if you're planning to implement a custom scheduler yourself, this
|
||||
page will help you learn about scheduling.
|
||||
|
||||
## kube-scheduler
|
||||
|
||||
[kube-scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/)
|
||||
is the default scheduler for Kubernetes and runs as part of the
|
||||
{{< glossary_tooltip text="control plane" term_id="control-plane" >}}.
|
||||
kube-scheduler is designed so that, if you want and need to, you can
|
||||
write your own scheduling component and use that instead.
|
||||
|
||||
For every newly created pods or other unscheduled pods, kube-scheduler
|
||||
selects a optimal node for them to run on. However, every container in
|
||||
pods has different requirements for resources and every pod also has
|
||||
different requirements. Therefore, existing nodes need to be filtered
|
||||
according to the specific scheduling requirements.
|
||||
|
||||
In a cluster, Nodes that meet the scheduling requirements for a Pod
|
||||
are called _feasible_ nodes. If none of the nodes are suitable, the pod
|
||||
remains unscheduled until the scheduler is able to place it.
|
||||
|
||||
The scheduler finds feasible Nodes for a Pod and then runs a set of
|
||||
functions to score the feasible Nodes and picks a Node with the highest
|
||||
score among the feasible ones to run the Pod. The scheduler then notifies
|
||||
the API server about this decision in a process called _binding_.
|
||||
|
||||
Factors that need taken into account for scheduling decisions include
|
||||
individual and collective resource requirements, hardware / software /
|
||||
policy constraints, affinity and anti-affinity specifications, data
|
||||
locality, inter-workload interference, and so on.
|
||||
|
||||
## Scheduling with kube-scheduler {#kube-scheduler-implementation}
|
||||
|
||||
kube-scheduler selects a node for the pod in a 2-step operation:
|
||||
|
||||
1. Filtering
|
||||
|
||||
2. Scoring
|
||||
|
||||
|
||||
The _filtering_ step finds the set of Nodes where it's feasible to
|
||||
schedule the Pod. For example, the PodFitsResources filter checks whether a
|
||||
candidate Node has enough available resource to meet a Pod's specific
|
||||
resource requests. After this step, the node list contains any suitable
|
||||
Nodes; often, there will be more than one. If the list is empty, that
|
||||
Pod isn't (yet) schedulable.
|
||||
|
||||
In the _scoring_ step, the scheduler ranks the remaining nodes to choose
|
||||
the most suitable Pod placement. The scheduler assigns a score to each Node
|
||||
that survived filtering, basing this score on the active scoring rules.
|
||||
|
||||
Finally, kube-scheduler assigns the Pod to the Node with the highest ranking.
|
||||
If there is more than one node with equal scores, kube-scheduler selects
|
||||
one of these at random.
|
||||
|
||||
|
||||
### Default policies
|
||||
|
||||
kube-scheduler has a default set of scheduling policies.
|
||||
|
||||
### Filtering
|
||||
|
||||
- `PodFitsHostPorts`: Checks if a Node has free ports (the network protocol kind)
|
||||
for the Pod ports the the Pod is requesting.
|
||||
|
||||
- `PodFitsHost`: Checks if a Pod specifies a specific Node by it hostname.
|
||||
|
||||
- `PodFitsResources`: Checks if the Node has free resources (eg, CPU and Memory)
|
||||
to meet the requirement of the Pod.
|
||||
|
||||
- `PodMatchNodeSelector`: Checks if a Pod's Node {{< glossary_tooltip term_id="selector" >}}
|
||||
matches the Node's {{< glossary_tooltip text="label(s)" term_id="label" >}}.
|
||||
|
||||
- `NoVolumeZoneConflict`: Evaluate if the {{< glossary_tooltip text="Volumes" term_id="volume" >}}
|
||||
that a Pod requests are available on the Node, given the failure zone restrictions for
|
||||
that storage.
|
||||
|
||||
- `NoDiskConflict`: Evaluates if a Pod can fit on a Node due to the volumes it requests,
|
||||
and those that are already mounted.
|
||||
|
||||
- `MaxCSIVolumeCount`: Decides how many {{< glossary_tooltip text="CSI" term_id="csi" >}}
|
||||
volumes should be attached, and whether that's over a configured limit.
|
||||
|
||||
- `CheckNodeMemoryPressure`: If a Node is reporting memory pressure, and there's no
|
||||
configured exception, the Pod won't be scheduled there.
|
||||
|
||||
- `CheckNodePIDPressure`: If a Node is reporting that process IDs are scarce, and
|
||||
there's no configured exception, the Pod won't be scheduled there.
|
||||
|
||||
- `CheckNodeDiskPressure`: If a Node is reporting storage pressure (a filesystem that
|
||||
is full or nearly full), and there's no configured exception, the Pod won't be
|
||||
scheduled there.
|
||||
|
||||
- `CheckNodeCondition`: Nodes can report that they have a completely full filesystem,
|
||||
that networking isn't available or that kubelet is otherwise not ready to run Pods.
|
||||
If such a condition is set for a Node, and there's no configured exception, the Pod
|
||||
won't be scheduled there.
|
||||
|
||||
- `PodToleratesNodeTaints`: checks if a Pod's {{< glossary_tooltip text="tolerations" term_id="toleration" >}}
|
||||
can tolerate the Node's {{< glossary_tooltip text="taints" term_id="taint" >}}.
|
||||
|
||||
- `CheckVolumeBinding`: Evaluates if a Pod can fit due to the volumes it requests.
|
||||
This applies for both bound and unbound
|
||||
{{< glossary_tooltip text="PVCs" term_id="persistent-volume-claim" >}}
|
||||
|
||||
### Scoring
|
||||
|
||||
- `SelectorSpreadPriority`: Spreads Pods across hosts, considering Pods that
|
||||
belonging to the same {{< glossary_tooltip text="Service" term_id="service" >}},
|
||||
{{< glossary_tooltip term_id="statefulset" >}} or
|
||||
{{< glossary_tooltip term_id="replica-set" >}}.
|
||||
|
||||
- `InterPodAffinityPriority`: Computes a sum by iterating through the elements
|
||||
of weightedPodAffinityTerm and adding “weight” to the sum if the corresponding
|
||||
PodAffinityTerm is satisfied for that node; the node(s) with the highest sum
|
||||
are the most preferred.
|
||||
|
||||
- `LeastRequestedPriority`: Favors nodes with fewer requested resources. In other
|
||||
words, the more Pods that are placed on a Node, and the more resources those
|
||||
Pods use, the lower the ranking this policy will give.
|
||||
|
||||
- `MostRequestedPriority`: Favors nodes with most requested resources. This policy
|
||||
will fit the scheduled Pods onto the smallest number of Nodes needed to run your
|
||||
overall set of workloads.
|
||||
|
||||
- `RequestedToCapacityRatioPriority`: Creates a requestedToCapacity based ResourceAllocationPriority using default resource scoring function shape.
|
||||
|
||||
- `BalancedResourceAllocation`: Favors nodes with balanced resource usage.
|
||||
|
||||
- `NodePreferAvoidPodsPriority`: Priorities nodes according to the node annotation
|
||||
`scheduler.alpha.kubernetes.io/preferAvoidPods`. You can use this to hint that
|
||||
two different Pods shouldn't run on the same Node.
|
||||
|
||||
- `NodeAffinityPriority`: Prioritizes nodes according to node affinity scheduling
|
||||
preferences indicated in PreferredDuringSchedulingIgnoredDuringExecution.
|
||||
You can read more about this in [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/)
|
||||
|
||||
- `TaintTolerationPriority`: Prepares the priority list for all the nodes, based on
|
||||
the number of intolerable taints on the node. This policy adjusts a node's rank
|
||||
taking that list into account.
|
||||
|
||||
- `ImageLocalityPriority`: Favors nodes that already have the
|
||||
{{< glossary_tooltip text="container images" term_id="image" >}} for that
|
||||
Pod cached locally.
|
||||
|
||||
- `ServiceSpreadingPriority`: For a given Service, this policy aims to make sure that
|
||||
the Pods for the Service run on different nodes. It favouring scheduling onto nodes
|
||||
that don't have Pods for the service already assigned there. The overall outcome is
|
||||
that the Service becomes more resilient to a single Node failure.
|
||||
|
||||
- `CalculateAntiAffinityPriorityMap`: This policy helps implement
|
||||
[pod anti-affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity).
|
||||
|
||||
- `EqualPriorityMap`: Gives an equal weight of one to all nodes.
|
||||
|
||||
{{% /capture %}}
|
||||
{{% capture whatsnext %}}
|
||||
* Read about [scheduler performance tuning](/docs/concepts/scheduling/scheduler-perf-tuning/)
|
||||
* Read the [reference documentation](/docs/reference/command-line-tools-reference/kube-scheduler/) for kube-scheduler
|
||||
* Learn about [configuring multiple schedulers](https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/)
|
||||
{{% /capture %}}
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
reviewers:
|
||||
- bsalamat
|
||||
title: Scheduler Performance Tuning
|
||||
content_template: templates/concept
|
||||
weight: 70
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="1.14" state="beta" >}}
|
||||
|
||||
[kube-scheduler](/docs/concepts/scheduling/kube-scheduler/#kube-scheduler)
|
||||
is the Kubernetes default scheduler. It is responsible for placement of Pods
|
||||
on Nodes in a cluster.
|
||||
|
||||
Nodes in a cluster that meet the scheduling requirements of a Pod are
|
||||
called _feasible_ Nodes for the Pod. The scheduler finds feasible Nodes
|
||||
for a Pod and then runs a set of functions to score the feasible Nodes,
|
||||
picking a Node with the highest score among the feasible ones to run
|
||||
the Pod. The scheduler then notifies the API server about this decision
|
||||
in a process called _Binding_.
|
||||
|
||||
This page explains performance tuning optimizations that are relevant for
|
||||
large Kubernetes clusters.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Percentage of Nodes to Score
|
||||
|
||||
Before Kubernetes 1.12, Kube-scheduler used to check the feasibility of all
|
||||
nodes in a cluster and then scored the feasible ones. Kubernetes 1.12 added a
|
||||
new feature that allows the scheduler to stop looking for more feasible nodes
|
||||
once it finds a certain number of them. This improves the scheduler's
|
||||
performance in large clusters. The number is specified as a percentage of the
|
||||
cluster size. The percentage can be controlled by a configuration option called
|
||||
`percentageOfNodesToScore`. The range should be between 1 and 100. Larger values
|
||||
are considered as 100%. Zero is equivalent to not providing the config option.
|
||||
Kubernetes 1.14 has logic to find the percentage of nodes to score based on the
|
||||
size of the cluster if it is not specified in the configuration. It uses a
|
||||
linear formula which yields 50% for a 100-node cluster. The formula yields 10%
|
||||
for a 5000-node cluster. The lower bound for the automatic value is 5%. In other
|
||||
words, the scheduler always scores at least 5% of the cluster no matter how
|
||||
large the cluster is, unless the user provides the config option with a value
|
||||
smaller than 5.
|
||||
|
||||
Below is an example configuration that sets `percentageOfNodesToScore` to 50%.
|
||||
|
||||
```yaml
|
||||
apiVersion: kubescheduler.config.k8s.io/v1alpha1
|
||||
kind: KubeSchedulerConfiguration
|
||||
algorithmSource:
|
||||
provider: DefaultProvider
|
||||
|
||||
...
|
||||
|
||||
percentageOfNodesToScore: 50
|
||||
```
|
||||
|
||||
{{< note >}} In clusters with less than 50 feasible nodes, the scheduler still
|
||||
checks all the nodes, simply because there are not enough feasible nodes to stop
|
||||
the scheduler's search early. {{< /note >}}
|
||||
|
||||
**To disable this feature**, you can set `percentageOfNodesToScore` to 100.
|
||||
|
||||
### Tuning percentageOfNodesToScore
|
||||
|
||||
`percentageOfNodesToScore` must be a value between 1 and 100 with the default
|
||||
value being calculated based on the cluster size. There is also a hardcoded
|
||||
minimum value of 50 nodes. This means that changing
|
||||
this option to lower values in clusters with several hundred nodes will not have
|
||||
much impact on the number of feasible nodes that the scheduler tries to find.
|
||||
This is intentional as this option is unlikely to improve performance noticeably
|
||||
in smaller clusters. In large clusters with over a 1000 nodes setting this value
|
||||
to lower numbers may show a noticeable performance improvement.
|
||||
|
||||
An important note to consider when setting this value is that when a smaller
|
||||
number of nodes in a cluster are checked for feasibility, some nodes are not
|
||||
sent to be scored for a given Pod. As a result, a Node which could possibly
|
||||
score a higher value for running the given Pod might not even be passed to the
|
||||
scoring phase. This would result in a less than ideal placement of the Pod. For
|
||||
this reason, the value should not be set to very low percentages. A general rule
|
||||
of thumb is to never set the value to anything lower than 10. Lower values
|
||||
should be used only when the scheduler's throughput is critical for your
|
||||
application and the score of nodes is not important. In other words, you prefer
|
||||
to run the Pod on any Node as long as it is feasible.
|
||||
|
||||
If your cluster has several hundred Nodes or fewer, we do not recommend lowering
|
||||
the default value of this configuration option. It is unlikely to improve the
|
||||
scheduler's performance significantly.
|
||||
|
||||
### How the scheduler iterates over Nodes
|
||||
|
||||
This section is intended for those who want to understand the internal details
|
||||
of this feature.
|
||||
|
||||
In order to give all the Nodes in a cluster a fair chance of being considered
|
||||
for running Pods, the scheduler iterates over the nodes in a round robin
|
||||
fashion. You can imagine that Nodes are in an array. The scheduler starts from
|
||||
the start of the array and checks feasibility of the nodes until it finds enough
|
||||
Nodes as specified by `percentageOfNodesToScore`. For the next Pod, the
|
||||
scheduler continues from the point in the Node array that it stopped at when
|
||||
checking feasibility of Nodes for the previous Pod.
|
||||
|
||||
If Nodes are in multiple zones, the scheduler iterates over Nodes in various
|
||||
zones to ensure that Nodes from different zones are considered in the
|
||||
feasibility checks. As an example, consider six nodes in two zones:
|
||||
|
||||
```
|
||||
Zone 1: Node 1, Node 2, Node 3, Node 4
|
||||
Zone 2: Node 5, Node 6
|
||||
```
|
||||
|
||||
The Scheduler evaluates feasibility of the nodes in this order:
|
||||
|
||||
```
|
||||
Node 1, Node 5, Node 2, Node 6, Node 3, Node 4
|
||||
```
|
||||
|
||||
After going over all the Nodes, it goes back to Node 1.
|
||||
|
||||
{{% /capture %}}
|
||||
Reference in New Issue
Block a user