Add user docs for pod priority and preemption (#5328)
* Add user docs for pod priority and preemption * Update pod-priority-preemption.md * More updates
This commit is contained in:
Bobby (Babak) Salamat
committed by
Steve Perry
Steve Perry
parent
2a7a878d0e
commit
a30edd4c00
@@ -61,6 +61,7 @@ toc:
|
||||
- docs/concepts/configuration/taint-and-toleration.md
|
||||
- docs/concepts/configuration/secret.md
|
||||
- docs/concepts/configuration/organize-cluster-access-kubeconfig.md
|
||||
- docs/concepts/configuration/pod-priority-preemption.md
|
||||
|
||||
- title: Services, Load Balancing, and Networking
|
||||
section:
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
approvers:
|
||||
- davidopp
|
||||
- wojtek-t
|
||||
title: Pod Priority and Preemption
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
{% include feature-state-alpha.md %}
|
||||
|
||||
[Pods](/docs/user-guide/pods) in Kubernetes 1.8 and later can have priority. Priority
|
||||
indicates the importance of a Pod relative to other Pods. When a Pod cannot be scheduled,
|
||||
the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the
|
||||
pending Pod possible. In a future Kubernetes release, priority will also affect
|
||||
out-of-resource eviction ordering on the Node.
|
||||
|
||||
**Note:** Preemption does not respect PodDisruptionBudget; see
|
||||
[the limitations section](#poddisruptionbudget-is-not-supported) for more details.
|
||||
{: .note}
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture body %}
|
||||
|
||||
## How to use priority and preemption
|
||||
To use priority and preemption in Kubernetes 1.8, follow these steps:
|
||||
|
||||
1. Enable the feature.
|
||||
|
||||
1. Add one or more PriorityClasses.
|
||||
|
||||
1. Create Pods with `PriorityClassName` set to one of the added PriorityClasses.
|
||||
Of course you do not need to create the Pods directly; normally you would add
|
||||
`PriorityClassName` to the Pod template of a collection object like a Deployment.
|
||||
|
||||
The following sections provide more information about these steps.
|
||||
|
||||
## Enabling priority and preemption
|
||||
|
||||
Pod priority and preemption is disabled by default in Kubernetes 1.8.
|
||||
To enable the feature, set this command-line flag for the API server
|
||||
and the scheduler:
|
||||
|
||||
```
|
||||
--feature-gates=PodPriority=true
|
||||
```
|
||||
|
||||
Also set this flag for API server:
|
||||
|
||||
|
||||
```
|
||||
--runtime-config=scheduling.k8s.io/v1alpha1=true
|
||||
```
|
||||
|
||||
After the feature is enabled, you can create [PriorityClasses](#priorityclass)
|
||||
and create Pods with [`PriorityClassName`](#pod-priority) set.
|
||||
|
||||
If you try the feature and then decide to disable it, you must remove the PodPriority
|
||||
command-line flag or set it to false, and then restart the API server and
|
||||
scheduler. After the feature is disabled, the existing Pods keep their priority
|
||||
fields, but preemption is disabled, and priority fields are ignored, and you
|
||||
cannot set PriorityClassName in new Pods.
|
||||
|
||||
## PriorityClass
|
||||
|
||||
A PriorityClass is a non-namespaced object that defines a mapping from a priority
|
||||
class name to the integer value of the priority. The name is specified in the `name`
|
||||
field of the PriorityClass object's metadata. The value is specified in the required
|
||||
`value` field. The higher the value, the higher the priority.
|
||||
|
||||
A PriorityClass object can have any 32-bit integer value smaller than or equal to
|
||||
1 billion. Larger numbers are reserved for critical system Pods that should not
|
||||
normally be preempted or evicted. A cluster admin should create one PriorityClass
|
||||
object for each such mapping that they want.
|
||||
|
||||
PriorityClass also has two optional fields: `globalDefault` and `description`.
|
||||
The `globalDefault` field indicates that the value of this PriorityClass should
|
||||
be used for Pods without a `PriorityClassName`. Only one PriorityClass with
|
||||
`globalDefault` set to true can exist in the system. If there is no PriorityClass
|
||||
with `globalDefault` set, the priority of Pods with no `PriorityClassName` is zero.
|
||||
|
||||
The `description` field is an arbitrary string. It is meant to tell users of
|
||||
the cluster when they should use this PriorityClass.
|
||||
|
||||
**Note 1**: If you upgrade your existing cluster and enable this feature, the priority
|
||||
of your existing Pods will be considered to be zero.
|
||||
{: .note}
|
||||
|
||||
**Note 2**: Addition of a PriorityClass with `globalDefault` set to true does not
|
||||
change the priorities of existing Pods. The value of such a PriorityClass is used only
|
||||
for Pods created after the PriorityClass is added.
|
||||
{: .note}
|
||||
|
||||
**Note 3**: If you delete a PriorityClass, existing Pods that use the name of the
|
||||
deleted priority class remain unchanged, but you are not able to create more Pods
|
||||
that use the name of the deleted PriorityClass.
|
||||
{: .note}
|
||||
|
||||
### Example PriorityClass
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: high-priority
|
||||
value: 1000000
|
||||
globalDefault: false
|
||||
description: "This priority class should be used for XYZ service pods only."
|
||||
```
|
||||
|
||||
## Pod priority
|
||||
|
||||
After you have one or more PriorityClasses, you can create Pods that specify one
|
||||
of those PriorityClass names in their specifications. The priority admission
|
||||
controller uses the `priorityClassName` field and populates the integer value
|
||||
of the priority. If the priority class is not found, the Pod is rejected.
|
||||
|
||||
The following YAML is an example of a Pod configuration that uses the PriorityClass
|
||||
created in the preceding example. The priority admission controller checks the
|
||||
specification and resolves the priority of the Pod to 1000000.
|
||||
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: nginx
|
||||
labels:
|
||||
env: test
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx
|
||||
imagePullPolicy: IfNotPresent
|
||||
priorityClassName: high-priority
|
||||
```
|
||||
|
||||
## Preemption
|
||||
|
||||
When Pods are created, they go to a queue and wait to be scheduled. The scheduler
|
||||
picks a Pod from the queue and tries to schedule it on a Node. If no Node is found
|
||||
that satisfies all the specified requirements of the Pod, preemption logic is triggered
|
||||
for the pending Pod. Let's call the pending pod P. Preemption logic tries to find a Node
|
||||
where removal of one or more Pods with lower priority than P would enable P to be scheduled
|
||||
on that Node. If such a Node is found, one or more lower priority Pods get
|
||||
deleted from the Node. After the Pods are gone, P can be scheduled on the Node.
|
||||
|
||||
### Limitations of preemption (alpha version)
|
||||
|
||||
#### Starvation of preempting Pod
|
||||
|
||||
When Pods are preempted, the victims get their
|
||||
[graceful termination period](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods).
|
||||
They have that much time to finish their work and exit. If they don't, they are
|
||||
killed. This graceful termination period creates a time gap between the point
|
||||
that the scheduler preempts Pods and the time when the pending Pod (P) can be
|
||||
scheduled on the Node (N). In the meantime, the scheduler keeps scheduling other
|
||||
pending Pods. As victims exit or get terminated, the scheduler tries to schedule
|
||||
Pods in the pending queue, and one or more of them may be considered and
|
||||
scheduled to N before the scheduler considers scheduling P on N. In such a case,
|
||||
it is likely that when all the victims exit, Pod P won't fit on Node N anymore.
|
||||
So, scheduler will have to preempt other Pods on Node N or another Node so that
|
||||
P can be scheduled. This scenario might be repeated again for the second and
|
||||
subsequent rounds of preemption, and P might not get scheduled for a while.
|
||||
This scenario can cause problems in various clusters, but is particularly
|
||||
problematic in clusters with a high Pod creation rate.
|
||||
|
||||
We will address this problem in the beta version of Pod preemption. The solution
|
||||
we plan to implement is
|
||||
[provided here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-preemption.md#preemption-mechanics).
|
||||
|
||||
#### PodDisruptionBudget is not supported
|
||||
|
||||
A [Pod Disruption Budget (PDB)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
|
||||
allows application owners to limit the number Pods of a replicated application that
|
||||
are down simultaneously from voluntary disruptions. However, the alpha version of
|
||||
preemption does not respect PDB when choosing preemption victims.
|
||||
We plan to add PDB support in beta, but even in beta, respecting PDB will be best
|
||||
effort. The Scheduler will try to find victims whose PDB won't be violated by preemption,
|
||||
but if no such victims are found, preemption will still happen, and lower priority Pods
|
||||
will be removed despite their PDBs being violated.
|
||||
|
||||
#### Inter-Pod affinity on lower-priority Pods
|
||||
|
||||
In version 1.8, a Node is considered for preemption only when
|
||||
the answer to this question is yes: "If all the Pods with lower priority than
|
||||
the pending Pod are removed from the Node, can the pending pod be scheduled on
|
||||
the Node?"
|
||||
|
||||
**Note:** Preemption does not necessarily remove all lower-priority Pods. If the
|
||||
pending pod can be scheduled by removing fewer than all lower-priority Pods, then
|
||||
only a portion of the lower-priority Pods are removed. Even so, the answer to the
|
||||
preceding question must be yes. If the answer is no, the Node is not considered
|
||||
for preemption.
|
||||
{: .note}
|
||||
|
||||
If a pending Pod has inter-pod affinity to one or more of the lower-priority Pods
|
||||
on the Node, the inter-Pod affinity rule cannot be satisfied in the absence of those
|
||||
lower-priority Pods. In this case, the scheduler does not preempt any Pods on the
|
||||
Node. Instead, it looks for another Node. The scheduler might find a suitable Node
|
||||
or it might not. There is no guarantee that the pending Pod can be scheduled.
|
||||
|
||||
We might address this issue in future versions, but we don't have a clear plan yet.
|
||||
We will not consider it a blocker for Beta or GA. Part
|
||||
of the reason is that finding the set of lower-priority Pods that satisfy all
|
||||
inter-Pod affinity rules is computationally expensive, and adds substantial
|
||||
complexity to the preemption logic. Besides, even if preemption keeps the lower-priority
|
||||
Pods to satisfy inter-Pod affinity, the lower priority Pods might be preempted
|
||||
later by other Pods, which removes the benefits of having the complex logic of
|
||||
respecting inter-Pod affinity.
|
||||
|
||||
Our recommended solution for this problem is to create inter-Pod affinity only towards
|
||||
equal or higher priority pods.
|
||||
|
||||
#### Cross node preemption
|
||||
|
||||
Suppose a Node N is being considered for preemption so that a pending Pod P
|
||||
can be scheduled on N. P might become feasible on N only if a Pod on another
|
||||
Node is preempted. Here's an example:
|
||||
|
||||
* Pod P is being considered for Node N.
|
||||
* Pod Q is running on another Node in the same zone as Node N.
|
||||
* Pod P has anit-affinity with Pod Q.
|
||||
* There are no other cases of anti-affinity between Pod P and other Pods in the zone.
|
||||
* In order to schedule Pod P on Node N, Pod Q should be preempted, but scheduler
|
||||
does not perform cross-node preemption. So, Pod P will be deemed unschedulable
|
||||
on Node N.
|
||||
|
||||
If Pod Q were removed from its Node, the anti-affinity violation would be gone,
|
||||
and Pod P could possibly be scheduled on Node N.
|
||||
|
||||
We may consider adding cross Node preemption in future versions if we find an
|
||||
algorithm with reasonable performance. We cannot promise anything at this point,
|
||||
and cross Node preemption will not be considered a blocker for Beta or GA.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
* Learn more about [this](...).
|
||||
* See this [related task](...).
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/concept.md %}
|
||||
Reference in New Issue
Block a user