From 4ff55875838a935d4ff204b4f24fbc261cf61e87 Mon Sep 17 00:00:00 2001 From: Eric Tune Date: Mon, 26 Jun 2017 13:54:25 -0700 Subject: [PATCH] PodDisruptionBudget documentation Improvements (#4140) * Changes from #3885 Title: Update PDB documentation to explain new field Author: foxish * Added Placeholder Disruptions Concept Guide New file: docs/concepts/workloads/pods/disruptions.md Intented contents: concept for Pod Disruption Budget, cross reference to Eviction and Preemption docs. Linked from: concepts > workloads > pods * Added placeholder Configuring PDB Task New file: docs/tasks/run-application/configure-pdb.md Intented contents: task for writing a Pod Disruption Budget. Linked from: tasks > configuring-applications > configure pdb. * Add refs to the "drain a node" task. * Refactor PDB docs. Move the "Requesting an eviction" section from: docs/tasks/administer-cluster/configure-pod-disruption-budget.md -- which is going away -- to: docs/tasks/administer-cluster/safely-drain-node.md The move is verbatim, except for an introductory sentence. Also added assignees. * Refactor of PDB docs Moved the section: Specifying a PodDisruptionBudget from: docs/tasks/administer-cluster/configure-pod-disruption-budget.md to: docs/tasks/run-application/configure-pdb.md because that former file is going away. Move is verbatim. * Explain how Eviction tools should handle failures * Refactor PDB docs Move text from: docs/tasks/administer-cluster/configure-pod-disruption-budget.md to: docs/concepts/workloads/pods/disruptions.md Delete the now empty: docs/tasks/administer-cluster/configure-pod-disruption-budget.md Added a redirects_from section to the new doc, containing the path of the now-deleted doc, plus all the redirects from the deleted doc. * Expand PDB Concept guide Building on a little content from the old task, greatly expanded the Disruptions concept guide, including an abstract example. * Update creating a pdb Task. * Address review comments. * Fixed for all cody-clark's review comments * Address review comments from mml * Address review comments from maisem * Fix missing backtick --- _data/concepts.yml | 1 + _data/tasks.yml | 1 + docs/concepts/workloads/pods/disruptions.md | 277 ++++++++++++++++++ .../configure-pod-disruption-budget.md | 169 ----------- .../administer-cluster/safely-drain-node.md | 77 ++++- docs/tasks/run-application/configure-pdb.md | 225 ++++++++++++++ 6 files changed, 578 insertions(+), 172 deletions(-) create mode 100644 docs/concepts/workloads/pods/disruptions.md delete mode 100644 docs/tasks/administer-cluster/configure-pod-disruption-budget.md create mode 100644 docs/tasks/run-application/configure-pdb.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 5d231f7f5a..c07471ca4e 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -35,6 +35,7 @@ toc: - docs/concepts/workloads/pods/pod.md - docs/concepts/workloads/pods/pod-lifecycle.md - docs/concepts/workloads/pods/init-containers.md + - docs/concepts/workloads/pods/disruptions.md - title: Controllers section: - docs/concepts/workloads/controllers/replicaset.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 34195d5a5d..802dffb2bc 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -49,6 +49,7 @@ toc: - docs/tasks/run-application/rolling-update-replication-controller.md - docs/tasks/run-application/horizontal-pod-autoscale.md - docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md + - docs/tasks/run-application/configure-pdb.md - title: Run Jobs section: diff --git a/docs/concepts/workloads/pods/disruptions.md b/docs/concepts/workloads/pods/disruptions.md new file mode 100644 index 0000000000..912acc1f7a --- /dev/null +++ b/docs/concepts/workloads/pods/disruptions.md @@ -0,0 +1,277 @@ +--- +assignees: +- erictune +- foxish +- davidopp +title: Disruptions +redirect_from: +- "/docs/admin/disruptions/" +- "/docs/admin/disruptions.html" +- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/" +- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/" +- "/docs/tasks/administer-cluster/configure-pod-disruption-budget/" +--- + +{% capture overview %} +This guide is for application owners who want to build +highly availabile applications, and thus need to understand +what types of Disruptions can happen to Pods. + +It is also for Cluster Administrators who want to perform automated +cluster actions, like upgrading and autoscaling clusters. + +{% endcapture %} + +{:toc} + +{% capture body %} + +## Voluntary and Involuntary Disruptions + +Pods do not disappear until someone (a person or a controller) destroys them, or +there is an unavoidable hardware or system software error. + +We call these unavoidable cases *involuntary disruptions* to +an applicaton. Examples are: + +- a hardware failure of the physical machine backing the node +- cluster administrator deletes VM (instance) by mistake +- cloud provider or hypervisor failure makes VM dissappear +- a kernel panic +- if the node to disappears from the cluster due to cluster network partition +- eviction of a pod due to the node being [out-of-resources](/docs/tasks/administer-cluster/out-of-resource.md). + +Except for the out-of-resources condition, all these conditions +should be familiar to most users; they are are not specific +to Kubernetes. + +We call other cases *voluntary disruptions*. These include both +actions initiated by the application owner and those initiated by a Cluster +Administrator. Typical application owner actions include: + +- deleting the deployment or other controller that manages the pod +- updating a deployment's pod template causing a restart +- directly deleting a pod (e.g. by accident) + +Cluster Administrator actions include: + +- [Draining a node](/docs//tasks/administer-cluster/safely-drain-node.md) for repair or upgrade. +- Draining a node from a cluster to scale the cluster down (learn about +[Cluster Autoscaling](/docs/tasks/administer-cluster/cluster-management/#cluster-autoscaler) +). +- Removing a pod from a node to permit something else to fit on that node. + +These actions might be taken directly by the cluster administrator, or by automation +run by the cluster administrator, or by your cluster hosting provider. + +Ask your cluster administrator or consult your cloud provider or distribution documentation +to determine if any sources of voluntary disruptions are enabled for your cluster. +If none are enabled, you can skip creating Pod Disruption Budgets. + +## Dealing with Disruptions + +Here are some ways to mitigate involuntary disruptions: + +- Ensure your pod [requests the resources](/docs/tasks/configure-pod-container/assign-cpu-ram-container) it needs. +- Replicate your application if you need higher availability. (Learn about running replicated +[stateless](/docs/tasks/run-application/run-stateless-application-deployment.md) +and [stateful](/docs/tasks/run-application/run-replicated-stateful-application.md) applications.) +- For even higher availability when running replicated applications, +spread applications across racks (using +[anti-affinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature)) +or across zones (if using a +[multi-zone cluster](/docs/admin/multiple-zones).) + +The frequency of voluntary disruptions varies. On a basic Kubernetes cluster, there are +no voluntary disruptions at all. However, your cluster admnistrator or hosting provider +may run some additional services which cause voluntary disruptions. For example, +rolling out node software updates can cause voluntary updates. Also, some implementations +of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes. +You cluster adminstrator or hosting provider should have documented what level of voluntary +disruptions, if any, to expect. + +Kubernetes offers features to help run highly available applications at the same +time as frequent voluntary disruptions. We call this set of features +*Disruption Budgets*. + + +## How Disruption Budgets Work + +An Application Owner can create a `PodDisruptionBudget` object (PDB) for each application. +A PDB limits the number pods of a replicated application that are down simultaneously from +voluntary disruptions. For example, a quorum-based application would +like to ensure that the number of replicas running is never brought below the +number needed for a quorum. A web front end might want to +ensure that the number of replicas serving load never falls below a certain +percentage of the total. + +Cluster managers and hosting providers should use tools which +respect Pod Disruption Budgets by calling the [Eviction API](/docs/tasks/administer-cluster/safely-drain-node/#the-eviction-api) +instead of directly deleting pods. Examples are the `kubectl drain` command +and the Kubernetes-on-GCE cluster upgrade script (`cluster/gce/upgrade.sh`). + +When a cluster administrator wants to drain a node +they use the `kubectl drain` command. That tool tries to evict all +the pods on the machine. The eviction request may be temporarily rejected, +and the tool periodically retries all failed requests until all pods +are terminated, or until a configureable timeout is reached. + +A PDB specifies the number of replicas that an application can tolerate having, relative to how +many it is intended to have. For example, a Deployment which has a `spec.replicas: 5` is +supposed to have 5 pods at any given time. If its PDB allows for there to be 4 at a time, +then the Eviction API will allow voluntary disruption of one, but not two pods, at a time. + +The group of pods that comprise the application is specified using a label selector, the same +as the one used by the application's controller (deployment, stateful-set, etc). + +The "intended" number of pods is computed from the `.spec.replicas` of the pods controller. +The controller is discovered from the pods using the `.metadata.ownerReferences` of the object. + +PDBs cannot prevent [involuntary disruptions](#voluntary-and-involuntary-disruptions) from +occuring, but they do count against the budget. + +Pods which are deleted or unavailable due to a rolling upgrade to an application do count +against the disruption budget, but controllers (like deployment and stateful-set) +are not limited by PDBs when doing rolling upgrades -- the handling of failures +during application updates is configured in the controller spec. +(Learn about [updating a deployment](/docs/concepts/cluster-administration/manage-deployment/#updating-your-application-without-a-service-outage).) + +When a pod is evicted using the eviction API, it is gracefully terminated (see +`terminationGracePeriodSeconds` in [PodSpec](/docs/resources-reference/v1.6/#podspec-v1-core).) + +## PDB Example + +Consider a cluster with 3 nodes, `node-1` through `node-3`. +The cluster is running several applications. One of them has 3 replicas initially called +`pod-a`, `pod-b`, and `pod-c`. Another, unrelated pod without a PDB, called `pod-x`, is also shown. +Initially, the pods are layed out as follows: + +| node-1 | node-2 | node-3 | +|:--------------------:|:-------------------:|:------------------:| +| pod-a *available* | pod-b *available* | pod-c *available* | +| pod-x *available* | | | + +All 3 pods are part of an deployment, and they collectively have a PDB which requires +there be at least 2 of the 3 pods to be available at all times. + +For example, assume the cluster administrator wants to reboot into a new kernel version to fix a bug in the kernel. +The cluster administrator first tries to drain `node-1` using the `kubectl drain` command. +That tool tries to evict `pod-a` and `pod-x`. This succeeds immediately. +Both pods go into the `terminating` state at the same time. +This puts the cluster in this state: + +| node-1 *draining* | node-2 | node-3 | +|:--------------------:|:-------------------:|:------------------:| +| pod-a *terminating* | pod-b *available* | pod-c *available* | +| pod-x *terminating* | | | + +The deployment notices that one of the pods is terminating, so it creates a replacement +called `pod-d`. Since `node-1` is cordoned, it lands on another node. Something has +also created `pod-y` as a replacement for `pod-x`. + +(Note: for a StatefulSet, `pod-a`, which would be called something like `pod-1`, would need +to terminate completely before its replacement, which is also called `pod-1` but has a +different UID, could be created. Otherwise, the example applies to a StatefulSet as well.) + +Now the cluster is in this state: + +| node-1 *draining* | node-2 | node-3 | +|:--------------------:|:-------------------:|:------------------:| +| pod-a *terminating* | pod-b *available* | pod-c *available* | +| pod-x *terminating* | pod-d *starting* | pod-y | + +At some point, the pods terminate, and the cluster look like this: + +| node-1 *drained* | node-2 | node-3 | +|:--------------------:|:-------------------:|:------------------:| +| | pod-b *available* | pod-c *available* | +| | pod-d *starting* | pod-y | + +At this point, if an impatient cluster administrator tries to drain `node-2` or +`node-3`, the drain command will block, because there are only 2 available +pods for the deployment, and its PDB requires at least 2. After some time + +asses, `pod-d` becomes available. + +The cluster state now looks like this: + +| node-1 *drained* | node-2 | node-3 | +|:--------------------:|:-------------------:|:------------------:| +| | pod-b *available* | pod-c *available* | +| | pod-d *available* | pod-y | + +Now, the cluster admin tries to drain `node-2`. +The drain command will try to evict the two pods in some order, say +`pod-b` first and then `pod-d`. It will succeed at evicting `pod-b`. +But, when it tries to evict `pod-d`, it will be refused because that would leave only +one pod available for the deployment. + +The deployment creates a replacement for `pod-b` called `pod-e`. +However, not there are not enough resources in the cluster to schedule +`pod-e`. So, the drain then the drain will block. The cluster may end up in this +state: + +| node-1 *drained* | node-2 | node-3 | *no node* | +|:--------------------:|:-------------------:|:------------------:|:------------------:| +| | pod-b *available* | pod-c *available* | pod-e *pending* | +| | pod-d *available* | pod-y | | + +At this point, the cluster administrator needs to +add a node back to the cluster to proceed with the upgrade. + +You can see how Kubernetes varies the rate at which disruptions +can happen, according to: + +- how many replicas an application needs +- how long it takes to gracefully shutdown an instance +- how long it takes a new instance to start up +- the type of controller +- the cluster's resource capacity + +## Separating Cluster Owner and Application Owner Roles + +Often, it is useful to think of the Cluster Manager +and Application Owner as separate roles with limited knowlege +of each other. This separation of responsibilities +may make sense in these scenarios: + +- when there are many application teams sharing a Kubernetes cluster, and + there is natural specialization of roles +- when third-party tools or services are used to automate cluster management + +Pod Disrutption Budgets support this separation of roles by providing an +interface between the roles. + +If you do not have such a separation of responsibilities in your organization, +you may not need to use Pod Disruption Budgets. + +## How to perform Distruptive Actions your Cluster + +If you are a Cluster Administrator, and you need to perform a disruptive action on all +the nodes in your cluster, such as a node or system software upgrade, here are some options: + +- Accept downtime during the upgrade. +- Fail over to another complete replica cluster. + - No downtime, but may be costly both for the duplicated nodes, + and for human effort to orchestrate the switchover. +- Write disruption tolerant applications and use PDBs. + - No downtime. + - Minimal resource duplication. + - Allows more automation of cluster administration. + - Writing disruption-tolerant applications is tricky, but the work to tolerate voluntary + disruptions largely overlaps with work to support autoscaling and tolerating + involuntary disruptions. + +{% endcapture %} + + +{% capture whatsnext %} + +* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application//configure-pdb.md). + +* Learn more about [draining nodes](/docs/tasks/administer-cluster//safely-drain-node.md) + +{% endcapture %} + + +{% include templates/concept.md %} diff --git a/docs/tasks/administer-cluster/configure-pod-disruption-budget.md b/docs/tasks/administer-cluster/configure-pod-disruption-budget.md deleted file mode 100644 index b9a413c78b..0000000000 --- a/docs/tasks/administer-cluster/configure-pod-disruption-budget.md +++ /dev/null @@ -1,169 +0,0 @@ ---- -assignees: -- davidopp -- erictune -- foxish -- kow3ns -title: Configure a Pod Disruption Budget -redirect_from: -- "/docs/admin/disruptions/" -- "/docs/admin/disruptions.html" -- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/" -- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/" ---- - -This guide is for anyone wishing to specify safety constraints on pods or anyone -wishing to write software (typically automation software) that respects those -constraints. - -* TOC -{:toc} - -## Rationale - -Various cluster management operations may voluntarily evict pods. "Voluntary" -means an eviction can be safely delayed for a reasonable period of time. The -principal examples today are draining a node for maintenance or upgrade -(`kubectl drain`), and cluster autoscaling down. In the future the -[rescheduler](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/rescheduling.md) -may also perform voluntary evictions. By contrast, something like evicting pods -because a node has become unreachable or reports `NotReady`, is not "voluntary." - -For voluntary evictions, it can be useful for applications to be able to limit -the number of pods that are down simultaneously. For example, a quorum-based application would -like to ensure that the number of replicas running is never brought below the -number needed for a quorum, even temporarily. Or a web front end might want to -ensure that the number of replicas serving load never falls below a certain -percentage of the total, even briefly. `PodDisruptionBudget` is an API object -that specifies the minimum number or percentage of replicas of a collection that -must be up at a time. Components that wish to evict a pod subject to disruption -budget use the `/eviction` subresource; unlike a regular pod deletion, this -operation may be rejected by the API server if the eviction would cause a -disruption budget to be violated. - -## Specifying a PodDisruptionBudget - -A `PodDisruptionBudget` has three fields: - -* A label selector `selector` to specify the set of -pods to which it applies. This is a required field. -* `minAvailable` which is a description of the number of pods from that -set that must still be available after the eviction, i.e. even in the absence -of the evicted pod. `minAvailable` can be either an absolute number or a percentage. -* `maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description -of the number of pods from that set that can be unavailable after the eviction. -It can also be either an absolute number or a percentage. - -You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`. -`maxUnavailable` can only be used to control the eviction of pods -that have an associated controller managing them. In the examples below, "desired replicas" -is the `scale` of the controller managing the pods being selected by the -`PodDisruptionBudget`. - -Example 1: With a `minAvailable` of 5, evictions will be allowed as long as they leave behind -5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`. - -Example 2: With a `minAvailable` of 30%, evictions will be allowed as long as at least 30% -of the number of desired replicas are healthy. - -Example 3: With a `maxUnavailable` of 5, evictions will be allowed as long as there are at most 5 -unhealthy replicas among the total number of desired replicas. - -Example 4: With a `maxUnavailable` of 30%, evictions will be allowed as long as no more than 30% -of the desired replicas are unhealthy. - -In typical usage, a single budget would be used for a collection of pods managed by -a controller—for example, the pods in a single ReplicaSet or StatefulSet. - -Note that a disruption budget does not truly guarantee that the specified -number/percentage of pods will always be up. For example, a node that hosts a -pod from the collection may fail when the collection is at the minimum size -specified in the budget, thus bringing the number of available pods from the -collection below the specified size. The budget can only protect against -voluntary evictions, not all causes of unavailability. - -A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the -number of replicas) may block node drains entirely. This is permitted as per the -semantics of `PodDisruptionBudget`. - -You can find examples of pod disruption budgets defined below. They match pods with the label -`app: zookeeper`. - -Example PDB Using maxUnavailable: - -```yaml -apiVersion: policy/v1beta1 -kind: PodDisruptionBudget -metadata: - name: zk-pdb -spec: - minAvailable: 2 - selector: - matchLabels: - app: zookeeper -``` - -Example PDB Using maxUnavailable (Kubernetes 1.7 or higher): - -```yaml -apiVersion: policy/v1beta1 -kind: PodDisruptionBudget -metadata: - name: zk-pdb -spec: - maxUnavailable: 1 - selector: - matchLabels: - app: zookeeper -``` - -For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both -specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it -automatically responds to changes in the number of replicas of the corresponding controller. - -## Requesting an eviction - -See the task explaining [draining nodes](/docs/tasks/administer-cluster/safely-drain-node/) for a higher level construct used to trigger evictions of pods on chosen nodes. - -If you are writing infrastructure software that wants to produce these voluntary -evictions, you will need to use the eviction API. The eviction subresource of a -pod can be thought of as a kind of policy-controlled DELETE operation on the pod -itself. To attempt an eviction (perhaps more REST-precisely, to attempt to -*create* an eviction), you POST an attempted operation. Here's an example: - -```json -{ - "apiVersion": "policy/v1beta1", - "kind": "Eviction", - "metadata": { - "name": "quux", - "namespace": "default" - } -} -``` - -You can attempt an eviction using `curl`: - -```bash -$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json -``` - -The API can respond in one of three ways. - - 1. If the eviction is granted, then the pod is deleted just as if you had sent - a `DELETE` request to the pod's URL and you get back `200 OK`. - 2. If the current state of affairs wouldn't allow an eviction by the rules set - forth in the budget, you get back `429 Too Many Requests`. This is - typically used for generic rate limiting of *any* requests, but here we mean - that this request isn't allowed *right now* but it may be allowed later. - Currently, callers do not get any `Retry-After` advice, but they may in - future versions. - 3. If there is some kind of misconfiguration, like multiple budgets pointing at - the same pod, you will get `500 Internal Server Error`. - -For a given eviction request, there are two cases. - - 1. There is no budget that matches this pod. In this case, the server always - returns `200 OK`. - 2. There is at least one budget. In this case, any of the three above responses may - apply. diff --git a/docs/tasks/administer-cluster/safely-drain-node.md b/docs/tasks/administer-cluster/safely-drain-node.md index 4ead95accd..28d460c062 100644 --- a/docs/tasks/administer-cluster/safely-drain-node.md +++ b/docs/tasks/administer-cluster/safely-drain-node.md @@ -1,6 +1,9 @@ --- assignees: - davidopp +- mml +- foxish +- kow3ns title: Safely Drain a Node while Respecting Application SLOs --- @@ -14,8 +17,12 @@ disruption SLOs you have specified using PodDisruptionBudget. This task assumes that you have met the following prerequisites: * You are using Kubernetes release >= 1.5. -* You have created [PodDisruptionBudget(s)](/docs/tasks/configure-pod-container/configure-pod-disruption-budget/) to express the -application-level disruption SLOs you want the system to enforce. +* Either: + 1. You do not require your applications to be highly available during the + node drain, or + 1. You have read about the [PodDisruptionBudget concept](/docs/concepts/workloads/pods/disruptions.md) + and [Configured PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb.md) for + applications that need them. {% endcapture %} @@ -81,11 +88,75 @@ that only one pod is unavailable at any given time. Any drains that would cause the number of ready replicas to fall below the specified budget are blocked. +## The Eviction API + +If you prefer not to use [kubectl drain](/docs/user-guide/kubectl/v1.6/#drain) (such as +to avoid calling to an external command, or to get finer control over over the pod +eviction process), you can also programmatically cause evictions using the eviction API. + +You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api.md#programmatic-access-to-the-api). + +The eviction subresource of a +pod can be thought of as a kind of policy-controlled DELETE operation on the pod +itself. To attempt an eviction (perhaps more REST-precisely, to attempt to +*create* an eviction), you POST an attempted operation. Here's an example: + +```json +{ + "apiVersion": "policy/v1beta1", + "kind": "Eviction", + "metadata": { + "name": "quux", + "namespace": "default" + } +} +``` + +You can attempt an eviction using `curl`: + +```bash +$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json +``` + +The API can respond in one of three ways. + +- If the eviction is granted, then the pod is deleted just as if you had sent + a `DELETE` request to the pod's URL and you get back `200 OK`. +- If the current state of affairs wouldn't allow an eviction by the rules set + forth in the budget, you get back `429 Too Many Requests`. This is + typically used for generic rate limiting of *any* requests, but here we mean + that this request isn't allowed *right now* but it may be allowed later. + Currently, callers do not get any `Retry-After` advice, but they may in + future versions. +- If there is some kind of misconfiguration, like multiple budgets pointing at + the same pod, you will get `500 Internal Server Error`. + +For a given eviction request, there are two cases. + +- There is no budget that matches this pod. In this case, the server always + returns `200 OK`. +- There is at least one budget. In this case, any of the three above responses may + apply. + +In some cases, an application may reach a broken state where it will never return anything +other than 429 or 500. This can happen, for example, if the replacement pod created by the +application's controller does not become ready, or if the last pod evicted has a very long +termination grace period. + +In this case, there are two potential solutions: + +- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation. +- After a suitably long wait, `DELETE` the pod instead of using the eviction API. + +Kubernetes does not specify what the behavior should be in this case; it is up to the +application owners and cluster owners to establish an agreement on behavior in these cases. {% endcapture %} {% capture whatsnext %} -*TODO: link to other docs about Stateful Set?* + +* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application//configure-pdb.md). + {% endcapture %} {% include templates/task.md %} diff --git a/docs/tasks/run-application/configure-pdb.md b/docs/tasks/run-application/configure-pdb.md new file mode 100644 index 0000000000..91f7d6c795 --- /dev/null +++ b/docs/tasks/run-application/configure-pdb.md @@ -0,0 +1,225 @@ +--- +title: Specifying a Disruption Budget for your Application +--- + +{% capture overview %} + +This page shows how to limit the number of concurrent disruptions +that your application experiences, allowing for higher availability +while permitting the cluster administrator to manage the clusters +nodes. + +{% endcapture %} + +{% capture prerequisites %} +* You are the owner of an application running on a Kubernetes cluster that requires + high availability. +* You should know how to deploy [Replicated Stateless Applications](/docs/tasks/run-application/run-stateless-application-deployment.md) + and/or [Replicated Stateful Applications](/docs/tasks/run-application/run-replicated-stateful-application.md). +* You should have read about the [Pod Disruption Budget concept](/docs/tasks/run-application/configure-pdb.md). +* You should confirm with your cluster owner or service provider that they respect + Pod Disruption Budgets. +{% endcapture %} + +{% capture steps %} + +## Protecting an Application with a PodDisruptionBudget + +1. Identify what application you want to protect with a PodDisruptionBudget (PDB). +1. Think about how your application reacts to disruptions +1. Create a PDB definition as a YAML file. +1. Create the PDB object from the YAML file. + +{% endcapture %} + +{% capture discussion %} + +## Identify an Application to Protect + +The most common use case when you want to protect an application +specified by one of the built-in Kubernetes controllers: + +- Deployment +- ReplicationController +- ReplicaSet +- StatefulSet + +In this case, make a note of the controller's `.spec.selector`; the same +selector goes into the PDBs `.spec.selector`. + +You can also use PDBs with pods which are not controlled by one of the above +controllers, or arbitrary groups of pods, but there are some restrictions, +described in [Arbitrary Controllers and Selectors](#arbitrary-controllers-and-selectors). + + +## Think about how your application reacts to disruptions + +Decide how many instances can be down at the same time for a short period +due to a voluntary disruption. + +- Stateless frontends + - Concern: don't reduce serving capacity by more than 10%. + - Solution: use PDB with minAvailable 90% for example. +- Single-instance Stateful Application + - Concern: do not terminate this application without talking to me. + - Possible Solution 1: Do not use a PDB and tolerate occasional downtime. + - Possible Solution 2: Set PDB with maxUnavailable=0. Have an understanding + (outside of Kubernetes) that the cluster operator needs to consult you before + termination. When the cluster operator contacts you, prepare for downtime, + and then delete the PDB to indicate readiness for disruption. Recreate afterwards. +- Multiple-instance Stateful application such as Consul, ZooKeeper, or etcd + - Concern: Do not reduce number of instances below quorum, otherwise writes fail. + - Possible Solution 1: set maxUnavailable to 1 (works with varying scale of application). + - Possible Solution 2: set minAvailable to quorum-size (e.g. 3 when scale is 5). (Allows more disruptions at once). +- Restartable Batch Job: + - Concern: Job needs to complete in case of voluntary disruption + - Possible solution: Do not create a PDB. The Job controller will create a replacement pod. + +## Specifying a PodDisruptionBudget + +A `PodDisruptionBudget` has three fields: + +* A label selector `.spec.selector` to specify the set of +pods to which it applies. This field is required. +* `.spec.minAvailable` which is a description of the number of pods from that +set that must still be available after the eviction, even in the absence +of the evicted pod. `minAvailable` can be either an absolute number or a percentage. +* `.spec.maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description +of the number of pods from that set that can be unavailable after the eviction. +It can be either an absolute number or a percentage. + +You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`. +`maxUnavailable` can only be used to control the eviction of pods +that have an associated controller managing them. In the examples below, "desired replicas" +is the `scale` of the controller managing the pods being selected by the +`PodDisruptionBudget`. + +Example 1: With a `minAvailable` of 5, evictions are be allowed as long as they leave behind +5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`. + +Example 2: With a `minAvailable` of 30%, evictions are allowed as long as at least 30% +of the number of desired replicas are healthy. + +Example 3: With a `maxUnavailable` of 5, evictions are allowed as long as there are at most 5 +unhealthy replicas among the total number of desired replicas. + +Example 4: With a `maxUnavailable` of 30%, evictions are allowed as long as no more than 30% +of the desired replicas are unhealthy. + +In typical usage, a single budget would be used for a collection of pods managed by +a controller—for example, the pods in a single ReplicaSet or StatefulSet. + +Note that a disruption budget does not truly guarantee that the specified +number/percentage of pods will always be up. For example, a node that hosts a +pod from the collection may fail when the collection is at the minimum size +specified in the budget, thus bringing the number of available pods from the +collection below the specified size. The budget can only protect against +voluntary evictions, not all causes of unavailability. + +A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the +number of replicas) may block node drains entirely. This is permitted as per the +semantics of `PodDisruptionBudget`. + +You can find examples of pod disruption budgets defined below. They match pods with the label +`app: zookeeper`. + +Example PDB Using maxUnavailable: + +```yaml +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: zk-pdb +spec: + minAvailable: 2 + selector: + matchLabels: + app: zookeeper +``` + +Example PDB Using maxUnavailable (Kubernetes 1.7 or higher): + +```yaml +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: zk-pdb +spec: + maxUnavailable: 1 + selector: + matchLabels: + app: zookeeper +``` + +For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both +specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it +automatically responds to changes in the number of replicas of the corresponding controller. + +# Create the PDB object + +You can create the PDB object with a command like `kubectl create -f mypdb.yaml`. + +You cannot update PDB objects. They must be deleted and re-created. + +# Check the status of the PDB + +Use kubectl to check that your PDB is created. + +Assuming you don't actually have pods matching `app: zookeeper` in your namespace, +then you'll see something like this: + +```shell +$ kubectl get poddisruptionbudgets +NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE +zk-pdb 2 0 7s +``` + +If there are matching pods (say, 3), then you would see something like this: + +```shell +$ kubectl get poddisruptionbudgets +NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE +zk-pdb 2 1 7s +``` + +The non-zero value for `ALLOWED-DISRUPTIONS` means that the disruption controller +has seen the PDB and counted the matching PDB, and updated the status +of the PDB. + +You can get more information about the status of a PDB with this command: + +```shell +$ kubectl get poddisruptionbudgets zk-pdb -o yaml +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: zk-pdb +... +status: + currentHealthy: 3 + desiredHealthy: 3 + disruptedPods: null + disruptionsAllowed: 1 + expectedPods: 3 + observedGeneration: 1 +``` + +# Arbitrary Controllers and Selectors + +You can skip this section if you only use PDBs with the built-in +application controllers (Deployment, ReplicationController, ReplicaSet, and StatefulSet), +with the PDB selector matching the controller's selector. + +You can use a PDB with pods controlled by another type of controller, by an +"operator", or bare pods, but with these restrictions: + +- only `.spec.minAvailable` can be used, not `.spec.maxUnavailable`. +- only an integer value can be used with `.spec.minAvailable`, not a percentage. + +You can use a selector which selects a subset or superset of the pods beloning to a built-in +controller. However, when there are multiple PDBs in a namespace, you must be careful not +to create PDBs whose selectors overlap. + +{% endcapture %} + +{% include templates/task.md %}