whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PodDisruptionBudget documentation Improvements (#4140)

Browse Source 
* Changes from #3885

Title: Update PDB documentation to explain new field
Author: foxish

* Added Placeholder Disruptions Concept Guide

New file: docs/concepts/workloads/pods/disruptions.md
Intented contents: concept for Pod Disruption Budget,
 cross reference to Eviction and Preemption docs.
Linked from: concepts > workloads > pods

* Added placeholder Configuring PDB Task

New file: docs/tasks/run-application/configure-pdb.md
Intented contents: task for writing a Pod Disruption Budget.
Linked from: tasks > configuring-applications > configure pdb.

* Add refs to the "drain a node" task.

* Refactor PDB docs.

Move the "Requesting an eviction" section from:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
-- which is going away -- to:
docs/tasks/administer-cluster/safely-drain-node.md

The move is verbatim, except for an introductory sentence.

Also added assignees.

* Refactor of PDB docs

Moved the section:
Specifying a PodDisruptionBudget
from:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
to:
docs/tasks/run-application/configure-pdb.md
because that former file is going away.
Move is verbatim.

* Explain how Eviction tools should handle failures

* Refactor PDB docs

Move text from:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
to:
docs/concepts/workloads/pods/disruptions.md

Delete the now empty:
docs/tasks/administer-cluster/configure-pod-disruption-budget.md

Added a redirects_from section to the new doc, containing the path
of the now-deleted doc, plus all the redirects from the deleted
doc.

* Expand PDB Concept guide

Building on a little content from the old task,
greatly expanded the Disruptions concept
guide, including an abstract example.

* Update creating a pdb Task.

* Address review comments.

* Fixed for all cody-clark's review comments

* Address review comments from mml

* Address review comments from maisem

* Fix missing backtick
This commit is contained in:
Eric Tune
2017-06-26 13:54:25 -07:00
committed by Andrew Chen
parent 098b4a84ef
commit 4ff5587583
6 changed files with 578 additions and 172 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_data/concepts.yml
+1
View File
@@ -35,6 +35,7 @@ toc:
- docs/concepts/workloads/pods/pod.md
- docs/concepts/workloads/pods/pod-lifecycle.md
- docs/concepts/workloads/pods/init-containers.md
- docs/concepts/workloads/pods/disruptions.md
- title: Controllers
section:
- docs/concepts/workloads/controllers/replicaset.md
_data/tasks.yml
+1
View File
@@ -49,6 +49,7 @@ toc:
- docs/tasks/run-application/rolling-update-replication-controller.md
- docs/tasks/run-application/horizontal-pod-autoscale.md
- docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md
- docs/tasks/run-application/configure-pdb.md
- title: Run Jobs
section:
docs/concepts/workloads/pods/disruptions.md
+277
View File
@@ -0,0 +1,277 @@
---
assignees:
- erictune
- foxish
- davidopp
title: Disruptions
redirect_from:
- "/docs/admin/disruptions/"
- "/docs/admin/disruptions.html"
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
- "/docs/tasks/administer-cluster/configure-pod-disruption-budget/"
---
{% capture overview %}
This guide is for application owners who want to build
highly availabile applications, and thus need to understand
what types of Disruptions can happen to Pods.
It is also for Cluster Administrators who want to perform automated
cluster actions, like upgrading and autoscaling clusters.
{% endcapture %}
{:toc}
{% capture body %}
## Voluntary and Involuntary Disruptions
Pods do not disappear until someone (a person or a controller) destroys them, or
there is an unavoidable hardware or system software error.
We call these unavoidable cases *involuntary disruptions* to
an applicaton. Examples are:
- a hardware failure of the physical machine backing the node
- cluster administrator deletes VM (instance) by mistake
- cloud provider or hypervisor failure makes VM dissappear
- a kernel panic
- if the node to disappears from the cluster due to cluster network partition
- eviction of a pod due to the node being [out-of-resources](/docs/tasks/administer-cluster/out-of-resource.md).
Except for the out-of-resources condition, all these conditions
should be familiar to most users; they are are not specific
to Kubernetes.
We call other cases *voluntary disruptions*. These include both
actions initiated by the application owner and those initiated by a Cluster
Administrator. Typical application owner actions include:
- deleting the deployment or other controller that manages the pod
- updating a deployment's pod template causing a restart
- directly deleting a pod (e.g. by accident)
Cluster Administrator actions include:
- [Draining a node](/docs//tasks/administer-cluster/safely-drain-node.md) for repair or upgrade.
- Draining a node from a cluster to scale the cluster down (learn about
[Cluster Autoscaling](/docs/tasks/administer-cluster/cluster-management/#cluster-autoscaler)
).
- Removing a pod from a node to permit something else to fit on that node.
These actions might be taken directly by the cluster administrator, or by automation
run by the cluster administrator, or by your cluster hosting provider.
Ask your cluster administrator or consult your cloud provider or distribution documentation
to determine if any sources of voluntary disruptions are enabled for your cluster.
If none are enabled, you can skip creating Pod Disruption Budgets.
## Dealing with Disruptions
Here are some ways to mitigate involuntary disruptions:
- Ensure your pod [requests the resources](/docs/tasks/configure-pod-container/assign-cpu-ram-container) it needs.
- Replicate your application if you need higher availability. (Learn about running replicated
[stateless](/docs/tasks/run-application/run-stateless-application-deployment.md)
and [stateful](/docs/tasks/run-application/run-replicated-stateful-application.md) applications.)
- For even higher availability when running replicated applications,
spread applications across racks (using
[anti-affinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature))
or across zones (if using a
[multi-zone cluster](/docs/admin/multiple-zones).)
The frequency of voluntary disruptions varies. On a basic Kubernetes cluster, there are
no voluntary disruptions at all. However, your cluster admnistrator or hosting provider
may run some additional services which cause voluntary disruptions. For example,
rolling out node software updates can cause voluntary updates. Also, some implementations
of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes.
You cluster adminstrator or hosting provider should have documented what level of voluntary
disruptions, if any, to expect.
Kubernetes offers features to help run highly available applications at the same
time as frequent voluntary disruptions. We call this set of features
*Disruption Budgets*.
## How Disruption Budgets Work
An Application Owner can create a `PodDisruptionBudget` object (PDB) for each application.
A PDB limits the number pods of a replicated application that are down simultaneously from
voluntary disruptions. For example, a quorum-based application would
like to ensure that the number of replicas running is never brought below the
number needed for a quorum. A web front end might want to
ensure that the number of replicas serving load never falls below a certain
percentage of the total.
Cluster managers and hosting providers should use tools which
respect Pod Disruption Budgets by calling the [Eviction API](/docs/tasks/administer-cluster/safely-drain-node/#the-eviction-api)
instead of directly deleting pods. Examples are the `kubectl drain` command
and the Kubernetes-on-GCE cluster upgrade script (`cluster/gce/upgrade.sh`).
When a cluster administrator wants to drain a node
they use the `kubectl drain` command. That tool tries to evict all
the pods on the machine. The eviction request may be temporarily rejected,
and the tool periodically retries all failed requests until all pods
are terminated, or until a configureable timeout is reached.
A PDB specifies the number of replicas that an application can tolerate having, relative to how
many it is intended to have. For example, a Deployment which has a `spec.replicas: 5` is
supposed to have 5 pods at any given time. If its PDB allows for there to be 4 at a time,
then the Eviction API will allow voluntary disruption of one, but not two pods, at a time.
The group of pods that comprise the application is specified using a label selector, the same
as the one used by the application's controller (deployment, stateful-set, etc).
The "intended" number of pods is computed from the `.spec.replicas` of the pods controller.
The controller is discovered from the pods using the `.metadata.ownerReferences` of the object.
PDBs cannot prevent [involuntary disruptions](#voluntary-and-involuntary-disruptions) from
occuring, but they do count against the budget.
Pods which are deleted or unavailable due to a rolling upgrade to an application do count
against the disruption budget, but controllers (like deployment and stateful-set)
are not limited by PDBs when doing rolling upgrades -- the handling of failures
during application updates is configured in the controller spec.
(Learn about [updating a deployment](/docs/concepts/cluster-administration/manage-deployment/#updating-your-application-without-a-service-outage).)
When a pod is evicted using the eviction API, it is gracefully terminated (see
`terminationGracePeriodSeconds` in [PodSpec](/docs/resources-reference/v1.6/#podspec-v1-core).)
## PDB Example
Consider a cluster with 3 nodes, `node-1` through `node-3`.
The cluster is running several applications. One of them has 3 replicas initially called
`pod-a`, `pod-b`, and `pod-c`. Another, unrelated pod without a PDB, called `pod-x`, is also shown.
Initially, the pods are layed out as follows:
| node-1 | node-2 | node-3 |
|:--------------------:|:-------------------:|:------------------:|
| pod-a *available* | pod-b *available* | pod-c *available* |
| pod-x *available* | | |
All 3 pods are part of an deployment, and they collectively have a PDB which requires
there be at least 2 of the 3 pods to be available at all times.
For example, assume the cluster administrator wants to reboot into a new kernel version to fix a bug in the kernel.
The cluster administrator first tries to drain `node-1` using the `kubectl drain` command.
That tool tries to evict `pod-a` and `pod-x`. This succeeds immediately.
Both pods go into the `terminating` state at the same time.
This puts the cluster in this state:
| node-1 *draining* | node-2 | node-3 |
|:--------------------:|:-------------------:|:------------------:|
| pod-a *terminating* | pod-b *available* | pod-c *available* |
| pod-x *terminating* | | |
The deployment notices that one of the pods is terminating, so it creates a replacement
called `pod-d`. Since `node-1` is cordoned, it lands on another node. Something has
also created `pod-y` as a replacement for `pod-x`.
(Note: for a StatefulSet, `pod-a`, which would be called something like `pod-1`, would need
to terminate completely before its replacement, which is also called `pod-1` but has a
different UID, could be created. Otherwise, the example applies to a StatefulSet as well.)
Now the cluster is in this state:
| node-1 *draining* | node-2 | node-3 |
|:--------------------:|:-------------------:|:------------------:|
| pod-a *terminating* | pod-b *available* | pod-c *available* |
| pod-x *terminating* | pod-d *starting* | pod-y |
At some point, the pods terminate, and the cluster look like this:
| node-1 *drained* | node-2 | node-3 |
|:--------------------:|:-------------------:|:------------------:|
| | pod-b *available* | pod-c *available* |
| | pod-d *starting* | pod-y |
At this point, if an impatient cluster administrator tries to drain `node-2` or
`node-3`, the drain command will block, because there are only 2 available
pods for the deployment, and its PDB requires at least 2. After some time
asses, `pod-d` becomes available.
The cluster state now looks like this:
| node-1 *drained* | node-2 | node-3 |
|:--------------------:|:-------------------:|:------------------:|
| | pod-b *available* | pod-c *available* |
| | pod-d *available* | pod-y |
Now, the cluster admin tries to drain `node-2`.
The drain command will try to evict the two pods in some order, say
`pod-b` first and then `pod-d`. It will succeed at evicting `pod-b`.
But, when it tries to evict `pod-d`, it will be refused because that would leave only
one pod available for the deployment.
The deployment creates a replacement for `pod-b` called `pod-e`.
However, not there are not enough resources in the cluster to schedule
`pod-e`. So, the drain then the drain will block. The cluster may end up in this
state:
| node-1 *drained* | node-2 | node-3 | *no node* |
|:--------------------:|:-------------------:|:------------------:|:------------------:|
| | pod-b *available* | pod-c *available* | pod-e *pending* |
| | pod-d *available* | pod-y | |
At this point, the cluster administrator needs to
add a node back to the cluster to proceed with the upgrade.
You can see how Kubernetes varies the rate at which disruptions
can happen, according to:
- how many replicas an application needs
- how long it takes to gracefully shutdown an instance
- how long it takes a new instance to start up
- the type of controller
- the cluster's resource capacity
## Separating Cluster Owner and Application Owner Roles
Often, it is useful to think of the Cluster Manager
and Application Owner as separate roles with limited knowlege
of each other. This separation of responsibilities
may make sense in these scenarios:
- when there are many application teams sharing a Kubernetes cluster, and
there is natural specialization of roles
- when third-party tools or services are used to automate cluster management
Pod Disrutption Budgets support this separation of roles by providing an
interface between the roles.
If you do not have such a separation of responsibilities in your organization,
you may not need to use Pod Disruption Budgets.
## How to perform Distruptive Actions your Cluster
If you are a Cluster Administrator, and you need to perform a disruptive action on all
the nodes in your cluster, such as a node or system software upgrade, here are some options:
- Accept downtime during the upgrade.
- Fail over to another complete replica cluster.
- No downtime, but may be costly both for the duplicated nodes,
and for human effort to orchestrate the switchover.
- Write disruption tolerant applications and use PDBs.
- No downtime.
- Minimal resource duplication.
- Allows more automation of cluster administration.
- Writing disruption-tolerant applications is tricky, but the work to tolerate voluntary
disruptions largely overlaps with work to support autoscaling and tolerating
involuntary disruptions.
{% endcapture %}
{% capture whatsnext %}
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application//configure-pdb.md).
* Learn more about [draining nodes](/docs/tasks/administer-cluster//safely-drain-node.md)
{% endcapture %}
{% include templates/concept.md %}
docs/tasks/administer-cluster/configure-pod-disruption-budget.md
-169
View File
@@ -1,169 +0,0 @@
---
assignees:
- davidopp
- erictune
- foxish
- kow3ns
title: Configure a Pod Disruption Budget
redirect_from:
- "/docs/admin/disruptions/"
- "/docs/admin/disruptions.html"
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
---
This guide is for anyone wishing to specify safety constraints on pods or anyone
wishing to write software (typically automation software) that respects those
constraints.
* TOC
{:toc}
## Rationale
Various cluster management operations may voluntarily evict pods. "Voluntary"
means an eviction can be safely delayed for a reasonable period of time. The
principal examples today are draining a node for maintenance or upgrade
(`kubectl drain`), and cluster autoscaling down. In the future the
[rescheduler](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/rescheduling.md)
may also perform voluntary evictions. By contrast, something like evicting pods
because a node has become unreachable or reports `NotReady`, is not "voluntary."
For voluntary evictions, it can be useful for applications to be able to limit
the number of pods that are down simultaneously. For example, a quorum-based application would
like to ensure that the number of replicas running is never brought below the
number needed for a quorum, even temporarily. Or a web front end might want to
ensure that the number of replicas serving load never falls below a certain
percentage of the total, even briefly. `PodDisruptionBudget` is an API object
that specifies the minimum number or percentage of replicas of a collection that
must be up at a time. Components that wish to evict a pod subject to disruption
budget use the `/eviction` subresource; unlike a regular pod deletion, this
operation may be rejected by the API server if the eviction would cause a
disruption budget to be violated.
## Specifying a PodDisruptionBudget
A `PodDisruptionBudget` has three fields:
* A label selector `selector` to specify the set of
pods to which it applies. This is a required field.
* `minAvailable` which is a description of the number of pods from that
set that must still be available after the eviction, i.e. even in the absence
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
* `maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description
of the number of pods from that set that can be unavailable after the eviction.
It can also be either an absolute number or a percentage.
You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`.
`maxUnavailable` can only be used to control the eviction of pods
that have an associated controller managing them. In the examples below, "desired replicas"
is the `scale` of the controller managing the pods being selected by the
`PodDisruptionBudget`.
Example 1: With a `minAvailable` of 5, evictions will be allowed as long as they leave behind
5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`.
Example 2: With a `minAvailable` of 30%, evictions will be allowed as long as at least 30%
of the number of desired replicas are healthy.
Example 3: With a `maxUnavailable` of 5, evictions will be allowed as long as there are at most 5
unhealthy replicas among the total number of desired replicas.
Example 4: With a `maxUnavailable` of 30%, evictions will be allowed as long as no more than 30%
of the desired replicas are unhealthy.
In typical usage, a single budget would be used for a collection of pods managed by
a controller—for example, the pods in a single ReplicaSet or StatefulSet.
Note that a disruption budget does not truly guarantee that the specified
number/percentage of pods will always be up. For example, a node that hosts a
pod from the collection may fail when the collection is at the minimum size
specified in the budget, thus bringing the number of available pods from the
collection below the specified size. The budget can only protect against
voluntary evictions, not all causes of unavailability.
A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the
number of replicas) may block node drains entirely. This is permitted as per the
semantics of `PodDisruptionBudget`.
You can find examples of pod disruption budgets defined below. They match pods with the label
`app: zookeeper`.
Example PDB Using maxUnavailable:
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: zookeeper
```
Example PDB Using maxUnavailable (Kubernetes 1.7 or higher):
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: zookeeper
```
For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both
specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it
automatically responds to changes in the number of replicas of the corresponding controller.
## Requesting an eviction
See the task explaining [draining nodes](/docs/tasks/administer-cluster/safely-drain-node/) for a higher level construct used to trigger evictions of pods on chosen nodes.
If you are writing infrastructure software that wants to produce these voluntary
evictions, you will need to use the eviction API. The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
"apiVersion": "policy/v1beta1",
"kind": "Eviction",
"metadata": {
"name": "quux",
"namespace": "default"
}
}
```
You can attempt an eviction using `curl`:
```bash
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
```
The API can respond in one of three ways.
1. If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
2. If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
future versions.
3. If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases.
1. There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
2. There is at least one budget. In this case, any of the three above responses may
apply.
docs/tasks/administer-cluster/safely-drain-node.md
+74 -3
View File
@@ -1,6 +1,9 @@
---
assignees:
- davidopp
- mml
- foxish
- kow3ns
title: Safely Drain a Node while Respecting Application SLOs
---
@@ -14,8 +17,12 @@ disruption SLOs you have specified using PodDisruptionBudget.
This task assumes that you have met the following prerequisites:
* You are using Kubernetes release >= 1.5.
* You have created [PodDisruptionBudget(s)](/docs/tasks/configure-pod-container/configure-pod-disruption-budget/) to express the
application-level disruption SLOs you want the system to enforce.
* Either:
1. You do not require your applications to be highly available during the
node drain, or
1. You have read about the [PodDisruptionBudget concept](/docs/concepts/workloads/pods/disruptions.md)
and [Configured PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb.md) for
applications that need them.
{% endcapture %}
@@ -81,11 +88,75 @@ that only one pod is unavailable at any given time. Any drains that
would cause the number of ready replicas to fall below the specified
budget are blocked.
## The Eviction API
If you prefer not to use [kubectl drain](/docs/user-guide/kubectl/v1.6/#drain) (such as
to avoid calling to an external command, or to get finer control over over the pod
eviction process), you can also programmatically cause evictions using the eviction API.
You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api.md#programmatic-access-to-the-api).
The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
"apiVersion": "policy/v1beta1",
"kind": "Eviction",
"metadata": {
"name": "quux",
"namespace": "default"
}
}
```
You can attempt an eviction using `curl`:
```bash
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
```
The API can respond in one of three ways.
- If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
- If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
future versions.
- If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases.
- There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
- There is at least one budget. In this case, any of the three above responses may
apply.
In some cases, an application may reach a broken state where it will never return anything
other than 429 or 500. This can happen, for example, if the replacement pod created by the
application's controller does not become ready, or if the last pod evicted has a very long
termination grace period.
In this case, there are two potential solutions:
- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation.
- After a suitably long wait, `DELETE` the pod instead of using the eviction API.
Kubernetes does not specify what the behavior should be in this case; it is up to the
application owners and cluster owners to establish an agreement on behavior in these cases.
{% endcapture %}
{% capture whatsnext %}
*TODO: link to other docs about Stateful Set?*
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application//configure-pdb.md).
{% endcapture %}
{% include templates/task.md %}
docs/tasks/run-application/configure-pdb.md
+225
View File
@@ -0,0 +1,225 @@
---
title: Specifying a Disruption Budget for your Application
---
{% capture overview %}
This page shows how to limit the number of concurrent disruptions
that your application experiences, allowing for higher availability
while permitting the cluster administrator to manage the clusters
nodes.
{% endcapture %}
{% capture prerequisites %}
* You are the owner of an application running on a Kubernetes cluster that requires
high availability.
* You should know how to deploy [Replicated Stateless Applications](/docs/tasks/run-application/run-stateless-application-deployment.md)
and/or [Replicated Stateful Applications](/docs/tasks/run-application/run-replicated-stateful-application.md).
* You should have read about the [Pod Disruption Budget concept](/docs/tasks/run-application/configure-pdb.md).
* You should confirm with your cluster owner or service provider that they respect
Pod Disruption Budgets.
{% endcapture %}
{% capture steps %}
## Protecting an Application with a PodDisruptionBudget
1. Identify what application you want to protect with a PodDisruptionBudget (PDB).
1. Think about how your application reacts to disruptions
1. Create a PDB definition as a YAML file.
1. Create the PDB object from the YAML file.
{% endcapture %}
{% capture discussion %}
## Identify an Application to Protect
The most common use case when you want to protect an application
specified by one of the built-in Kubernetes controllers:
- Deployment
- ReplicationController
- ReplicaSet
- StatefulSet
In this case, make a note of the controller's `.spec.selector`; the same
selector goes into the PDBs `.spec.selector`.
You can also use PDBs with pods which are not controlled by one of the above
controllers, or arbitrary groups of pods, but there are some restrictions,
described in [Arbitrary Controllers and Selectors](#arbitrary-controllers-and-selectors).
## Think about how your application reacts to disruptions
Decide how many instances can be down at the same time for a short period
due to a voluntary disruption.
- Stateless frontends
- Concern: don't reduce serving capacity by more than 10%.
- Solution: use PDB with minAvailable 90% for example.
- Single-instance Stateful Application
- Concern: do not terminate this application without talking to me.
- Possible Solution 1: Do not use a PDB and tolerate occasional downtime.
- Possible Solution 2: Set PDB with maxUnavailable=0. Have an understanding
(outside of Kubernetes) that the cluster operator needs to consult you before
termination. When the cluster operator contacts you, prepare for downtime,
and then delete the PDB to indicate readiness for disruption. Recreate afterwards.
- Multiple-instance Stateful application such as Consul, ZooKeeper, or etcd
- Concern: Do not reduce number of instances below quorum, otherwise writes fail.
- Possible Solution 1: set maxUnavailable to 1 (works with varying scale of application).
- Possible Solution 2: set minAvailable to quorum-size (e.g. 3 when scale is 5). (Allows more disruptions at once).
- Restartable Batch Job:
- Concern: Job needs to complete in case of voluntary disruption
- Possible solution: Do not create a PDB. The Job controller will create a replacement pod.
## Specifying a PodDisruptionBudget
A `PodDisruptionBudget` has three fields:
* A label selector `.spec.selector` to specify the set of
pods to which it applies. This field is required.
* `.spec.minAvailable` which is a description of the number of pods from that
set that must still be available after the eviction, even in the absence
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
* `.spec.maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description
of the number of pods from that set that can be unavailable after the eviction.
It can be either an absolute number or a percentage.
You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`.
`maxUnavailable` can only be used to control the eviction of pods
that have an associated controller managing them. In the examples below, "desired replicas"
is the `scale` of the controller managing the pods being selected by the
`PodDisruptionBudget`.
Example 1: With a `minAvailable` of 5, evictions are be allowed as long as they leave behind
5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`.
Example 2: With a `minAvailable` of 30%, evictions are allowed as long as at least 30%
of the number of desired replicas are healthy.
Example 3: With a `maxUnavailable` of 5, evictions are allowed as long as there are at most 5
unhealthy replicas among the total number of desired replicas.
Example 4: With a `maxUnavailable` of 30%, evictions are allowed as long as no more than 30%
of the desired replicas are unhealthy.
In typical usage, a single budget would be used for a collection of pods managed by
a controller—for example, the pods in a single ReplicaSet or StatefulSet.
Note that a disruption budget does not truly guarantee that the specified
number/percentage of pods will always be up. For example, a node that hosts a
pod from the collection may fail when the collection is at the minimum size
specified in the budget, thus bringing the number of available pods from the
collection below the specified size. The budget can only protect against
voluntary evictions, not all causes of unavailability.
A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the
number of replicas) may block node drains entirely. This is permitted as per the
semantics of `PodDisruptionBudget`.
You can find examples of pod disruption budgets defined below. They match pods with the label
`app: zookeeper`.
Example PDB Using maxUnavailable:
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: zookeeper
```
Example PDB Using maxUnavailable (Kubernetes 1.7 or higher):
```yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: zookeeper
```
For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both
specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it
automatically responds to changes in the number of replicas of the corresponding controller.
# Create the PDB object
You can create the PDB object with a command like `kubectl create -f mypdb.yaml`.
You cannot update PDB objects. They must be deleted and re-created.
# Check the status of the PDB
Use kubectl to check that your PDB is created.
Assuming you don't actually have pods matching `app: zookeeper` in your namespace,
then you'll see something like this:
```shell
$ kubectl get poddisruptionbudgets
NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE
zk-pdb 2 0 7s
```
If there are matching pods (say, 3), then you would see something like this:
```shell
$ kubectl get poddisruptionbudgets
NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE
zk-pdb 2 1 7s
```
The non-zero value for `ALLOWED-DISRUPTIONS` means that the disruption controller
has seen the PDB and counted the matching PDB, and updated the status
of the PDB.
You can get more information about the status of a PDB with this command:
```shell
$ kubectl get poddisruptionbudgets zk-pdb -o yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
...
status:
currentHealthy: 3
desiredHealthy: 3
disruptedPods: null
disruptionsAllowed: 1
expectedPods: 3
observedGeneration: 1
```
# Arbitrary Controllers and Selectors
You can skip this section if you only use PDBs with the built-in
application controllers (Deployment, ReplicationController, ReplicaSet, and StatefulSet),
with the PDB selector matching the controller's selector.
You can use a PDB with pods controlled by another type of controller, by an
"operator", or bare pods, but with these restrictions:
- only `.spec.minAvailable` can be used, not `.spec.maxUnavailable`.
- only an integer value can be used with `.spec.minAvailable`, not a percentage.
You can use a selector which selects a subset or superset of the pods beloning to a built-in
controller. However, when there are multiple PDBs in a namespace, you must be careful not
to create PDBs whose selectors overlap.
{% endcapture %}
{% include templates/task.md %}