whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PodDisruptionBudget documentation Improvements (#4140)

Browse Source 
* Changes from #3885

Title: Update PDB documentation to explain new field
Author: foxish

* Added Placeholder Disruptions Concept Guide

New file: docs/concepts/workloads/pods/disruptions.md
Intented contents: concept for Pod Disruption Budget,
 cross reference to Eviction and Preemption docs.
Linked from: concepts > workloads > pods

* Added placeholder Configuring PDB Task

New file: docs/tasks/run-application/configure-pdb.md
Intented contents: task for writing a Pod Disruption Budget.
Linked from: tasks > configuring-applications > configure pdb.

* Add refs to the "drain a node" task.

* Refactor PDB docs.

Move the "Requesting an eviction" section from:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
-- which is going away -- to:
docs/tasks/administer-cluster/safely-drain-node.md

The move is verbatim, except for an introductory sentence.

Also added assignees.

* Refactor of PDB docs

Moved the section:
Specifying a PodDisruptionBudget
from:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
to:
docs/tasks/run-application/configure-pdb.md
because that former file is going away.
Move is verbatim.

* Explain how Eviction tools should handle failures

* Refactor PDB docs

Move text from:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
to:
docs/concepts/workloads/pods/disruptions.md

Delete the now empty:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md

Added a redirects_from section to the new doc, containing the path
of the now-deleted doc, plus all the redirects from the deleted
doc.

* Expand PDB Concept guide

Building on a little content from the old task,
greatly expanded the Disruptions concept
guide, including an abstract example.

* Update creating a pdb Task.

* Address review comments.

* Fixed for all cody-clark's review comments

* Address review comments from mml

* Address review comments from maisem

* Fix missing backtick
This commit is contained in:
Eric Tune
2017-06-26 13:54:25 -07:00
committed by Andrew Chen
parent 098b4a84ef
commit 4ff5587583
6 changed files with 578 additions and 172 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
-169
View File
@@ -1,169 +0,0 @@
---
assignees:
- davidopp
- erictune
- foxish
- kow3ns
title: Configure a Pod Disruption Budget
redirect_from:
- "/docs/admin/disruptions/"
- "/docs/admin/disruptions.html"
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
---
This guide is for anyone wishing to specify safety constraints on pods or anyone
wishing to write software (typically automation software) that respects those
constraints.
* TOC
{:toc}
## Rationale
Various cluster management operations may voluntarily evict pods. "Voluntary"
means an eviction can be safely delayed for a reasonable period of time. The
principal examples today are draining a node for maintenance or upgrade
(`kubectl drain`), and cluster autoscaling down. In the future the
[rescheduler](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/rescheduling.md)
may also perform voluntary evictions. By contrast, something like evicting pods
because a node has become unreachable or reports `NotReady`, is not "voluntary."
For voluntary evictions, it can be useful for applications to be able to limit
the number of pods that are down simultaneously. For example, a quorum-based application would
like to ensure that the number of replicas running is never brought below the
number needed for a quorum, even temporarily. Or a web front end might want to
ensure that the number of replicas serving load never falls below a certain
percentage of the total, even briefly. `PodDisruptionBudget` is an API object
that specifies the minimum number or percentage of replicas of a collection that
must be up at a time. Components that wish to evict a pod subject to disruption
budget use the `/eviction` subresource; unlike a regular pod deletion, this
operation may be rejected by the API server if the eviction would cause a
disruption budget to be violated.
## Specifying a PodDisruptionBudget
A `PodDisruptionBudget` has three fields:
* A label selector `selector` to specify the set of
pods to which it applies. This is a required field.
* `minAvailable` which is a description of the number of pods from that
set that must still be available after the eviction, i.e. even in the absence
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
* `maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description
of the number of pods from that set that can be unavailable after the eviction.
It can also be either an absolute number or a percentage.
You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`.
`maxUnavailable` can only be used to control the eviction of pods
that have an associated controller managing them. In the examples below, "desired replicas"
is the `scale` of the controller managing the pods being selected by the
`PodDisruptionBudget`.
Example 1: With a `minAvailable` of 5, evictions will be allowed as long as they leave behind
5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`.
Example 2: With a `minAvailable` of 30%, evictions will be allowed as long as at least 30%
of the number of desired replicas are healthy.
Example 3: With a `maxUnavailable` of 5, evictions will be allowed as long as there are at most 5
unhealthy replicas among the total number of desired replicas.
Example 4: With a `maxUnavailable` of 30%, evictions will be allowed as long as no more than 30%
of the desired replicas are unhealthy.
In typical usage, a single budget would be used for a collection of pods managed by
a controller—for example, the pods in a single ReplicaSet or StatefulSet.
Note that a disruption budget does not truly guarantee that the specified
number/percentage of pods will always be up. For example, a node that hosts a
pod from the collection may fail when the collection is at the minimum size
specified in the budget, thus bringing the number of available pods from the
collection below the specified size. The budget can only protect against
voluntary evictions, not all causes of unavailability.
A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the
number of replicas) may block node drains entirely. This is permitted as per the
semantics of `PodDisruptionBudget`.
You can find examples of pod disruption budgets defined below. They match pods with the label
`app: zookeeper`.
Example PDB Using maxUnavailable:
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: zookeeper
```
Example PDB Using maxUnavailable (Kubernetes 1.7 or higher):
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: zookeeper
```
For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both
specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it
automatically responds to changes in the number of replicas of the corresponding controller.
## Requesting an eviction
See the task explaining [draining nodes](/docs/tasks/administer-cluster/safely-drain-node/) for a higher level construct used to trigger evictions of pods on chosen nodes.
If you are writing infrastructure software that wants to produce these voluntary
evictions, you will need to use the eviction API. The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
"apiVersion": "policy/v1beta1",
"kind": "Eviction",
"metadata": {
"name": "quux",
"namespace": "default"
}
}
```
You can attempt an eviction using `curl`:
```bash
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
```
The API can respond in one of three ways.
1. If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
2. If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
future versions.
3. If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases.
1. There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
2. There is at least one budget. In this case, any of the three above responses may
apply.
docs/tasks/administer-cluster/safely-drain-node.md
+74 -3
View File
@@ -1,6 +1,9 @@
---
assignees:
- davidopp
- mml
- foxish
- kow3ns
title: Safely Drain a Node while Respecting Application SLOs
---
@@ -14,8 +17,12 @@ disruption SLOs you have specified using PodDisruptionBudget.
This task assumes that you have met the following prerequisites:
* You are using Kubernetes release >= 1.5.
* You have created [PodDisruptionBudget(s)](/docs/tasks/configure-pod-container/configure-pod-disruption-budget/) to express the
application-level disruption SLOs you want the system to enforce.
* Either:
1. You do not require your applications to be highly available during the
node drain, or
1. You have read about the [PodDisruptionBudget concept](/docs/concepts/workloads/pods/disruptions.md)
and [Configured PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb.md) for
applications that need them.
{% endcapture %}
@@ -81,11 +88,75 @@ that only one pod is unavailable at any given time. Any drains that
would cause the number of ready replicas to fall below the specified
budget are blocked.
## The Eviction API
If you prefer not to use [kubectl drain](/docs/user-guide/kubectl/v1.6/#drain) (such as
to avoid calling to an external command, or to get finer control over over the pod
eviction process), you can also programmatically cause evictions using the eviction API.
You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api.md#programmatic-access-to-the-api).
The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
"apiVersion": "policy/v1beta1",
"kind": "Eviction",
"metadata": {
"name": "quux",
"namespace": "default"
}
}
```
You can attempt an eviction using `curl`:
```bash
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
```
The API can respond in one of three ways.
- If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
- If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
future versions.
- If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases.
- There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
- There is at least one budget. In this case, any of the three above responses may
apply.
In some cases, an application may reach a broken state where it will never return anything
other than 429 or 500. This can happen, for example, if the replacement pod created by the
application's controller does not become ready, or if the last pod evicted has a very long
termination grace period.
In this case, there are two potential solutions:
- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation.
- After a suitably long wait, `DELETE` the pod instead of using the eviction API.
Kubernetes does not specify what the behavior should be in this case; it is up to the
application owners and cluster owners to establish an agreement on behavior in these cases.
{% endcapture %}
{% capture whatsnext %}
*TODO: link to other docs about Stateful Set?*
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application//configure-pdb.md).
{% endcapture %}
{% include templates/task.md %}
docs/tasks/run-application/configure-pdb.md
+225
View File
@@ -0,0 +1,225 @@
---
title: Specifying a Disruption Budget for your Application
---
{% capture overview %}
This page shows how to limit the number of concurrent disruptions
that your application experiences, allowing for higher availability
while permitting the cluster administrator to manage the clusters
nodes.
{% endcapture %}
{% capture prerequisites %}
* You are the owner of an application running on a Kubernetes cluster that requires
high availability.
* You should know how to deploy [Replicated Stateless Applications](/docs/tasks/run-application/run-stateless-application-deployment.md)
and/or [Replicated Stateful Applications](/docs/tasks/run-application/run-replicated-stateful-application.md).
* You should have read about the [Pod Disruption Budget concept](/docs/tasks/run-application/configure-pdb.md).
* You should confirm with your cluster owner or service provider that they respect
Pod Disruption Budgets.
{% endcapture %}
{% capture steps %}
## Protecting an Application with a PodDisruptionBudget
1. Identify what application you want to protect with a PodDisruptionBudget (PDB).
1. Think about how your application reacts to disruptions
1. Create a PDB definition as a YAML file.
1. Create the PDB object from the YAML file.
{% endcapture %}
{% capture discussion %}
## Identify an Application to Protect
The most common use case when you want to protect an application
specified by one of the built-in Kubernetes controllers:
- Deployment
- ReplicationController
- ReplicaSet
- StatefulSet
In this case, make a note of the controller's `.spec.selector`; the same
selector goes into the PDBs `.spec.selector`.
You can also use PDBs with pods which are not controlled by one of the above
controllers, or arbitrary groups of pods, but there are some restrictions,
described in [Arbitrary Controllers and Selectors](#arbitrary-controllers-and-selectors).
## Think about how your application reacts to disruptions
Decide how many instances can be down at the same time for a short period
due to a voluntary disruption.
- Stateless frontends
- Concern: don't reduce serving capacity by more than 10%.
- Solution: use PDB with minAvailable 90% for example.
- Single-instance Stateful Application
- Concern: do not terminate this application without talking to me.
- Possible Solution 1: Do not use a PDB and tolerate occasional downtime.
- Possible Solution 2: Set PDB with maxUnavailable=0. Have an understanding
(outside of Kubernetes) that the cluster operator needs to consult you before
termination. When the cluster operator contacts you, prepare for downtime,
and then delete the PDB to indicate readiness for disruption. Recreate afterwards.
- Multiple-instance Stateful application such as Consul, ZooKeeper, or etcd
- Concern: Do not reduce number of instances below quorum, otherwise writes fail.
- Possible Solution 1: set maxUnavailable to 1 (works with varying scale of application).
- Possible Solution 2: set minAvailable to quorum-size (e.g. 3 when scale is 5). (Allows more disruptions at once).
- Restartable Batch Job:
- Concern: Job needs to complete in case of voluntary disruption
- Possible solution: Do not create a PDB. The Job controller will create a replacement pod.
## Specifying a PodDisruptionBudget
A `PodDisruptionBudget` has three fields:
* A label selector `.spec.selector` to specify the set of
pods to which it applies. This field is required.
* `.spec.minAvailable` which is a description of the number of pods from that
set that must still be available after the eviction, even in the absence
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
* `.spec.maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description
of the number of pods from that set that can be unavailable after the eviction.
It can be either an absolute number or a percentage.
You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`.
`maxUnavailable` can only be used to control the eviction of pods
that have an associated controller managing them. In the examples below, "desired replicas"
is the `scale` of the controller managing the pods being selected by the
`PodDisruptionBudget`.
Example 1: With a `minAvailable` of 5, evictions are be allowed as long as they leave behind
5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`.
Example 2: With a `minAvailable` of 30%, evictions are allowed as long as at least 30%
of the number of desired replicas are healthy.
Example 3: With a `maxUnavailable` of 5, evictions are allowed as long as there are at most 5
unhealthy replicas among the total number of desired replicas.
Example 4: With a `maxUnavailable` of 30%, evictions are allowed as long as no more than 30%
of the desired replicas are unhealthy.
In typical usage, a single budget would be used for a collection of pods managed by
a controller—for example, the pods in a single ReplicaSet or StatefulSet.
Note that a disruption budget does not truly guarantee that the specified
number/percentage of pods will always be up. For example, a node that hosts a
pod from the collection may fail when the collection is at the minimum size
specified in the budget, thus bringing the number of available pods from the
collection below the specified size. The budget can only protect against
voluntary evictions, not all causes of unavailability.
A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the
number of replicas) may block node drains entirely. This is permitted as per the
semantics of `PodDisruptionBudget`.
You can find examples of pod disruption budgets defined below. They match pods with the label
`app: zookeeper`.
Example PDB Using maxUnavailable:
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: zookeeper
```
Example PDB Using maxUnavailable (Kubernetes 1.7 or higher):
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: zookeeper
```
For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both
specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it
automatically responds to changes in the number of replicas of the corresponding controller.
# Create the PDB object
You can create the PDB object with a command like `kubectl create -f mypdb.yaml`.
You cannot update PDB objects. They must be deleted and re-created.
# Check the status of the PDB
Use kubectl to check that your PDB is created.
Assuming you don't actually have pods matching `app: zookeeper` in your namespace,
then you'll see something like this:
```shell
$ kubectl get poddisruptionbudgets
NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE
zk-pdb 2 0 7s
```
If there are matching pods (say, 3), then you would see something like this:
```shell
$ kubectl get poddisruptionbudgets
NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE
zk-pdb 2 1 7s
```
The non-zero value for `ALLOWED-DISRUPTIONS` means that the disruption controller
has seen the PDB and counted the matching PDB, and updated the status
of the PDB.
You can get more information about the status of a PDB with this command:
```shell
$ kubectl get poddisruptionbudgets zk-pdb -o yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
...
status:
currentHealthy: 3
desiredHealthy: 3
disruptedPods: null
disruptionsAllowed: 1
expectedPods: 3
observedGeneration: 1
```
# Arbitrary Controllers and Selectors
You can skip this section if you only use PDBs with the built-in
application controllers (Deployment, ReplicationController, ReplicaSet, and StatefulSet),
with the PDB selector matching the controller's selector.
You can use a PDB with pods controlled by another type of controller, by an
"operator", or bare pods, but with these restrictions:
- only `.spec.minAvailable` can be used, not `.spec.maxUnavailable`.
- only an integer value can be used with `.spec.minAvailable`, not a percentage.
You can use a selector which selects a subset or superset of the pods beloning to a built-in
controller. However, when there are multiple PDBs in a namespace, you must be careful not
to create PDBs whose selectors overlap.
{% endcapture %}
{% include templates/task.md %}