* 'master' of https://github.com/kubernetes/kubernetes.github.io: (62 commits)
  Move Guide topic: Admin Guide
  Move Guide topic: Resource Monitoring. (#2895)
  Move Guide topic: Service. (#2891)
  Move Guide topic: Connecting Apps with Services. (#2885)
  Move Guide topic: Federation Service Discovery. (#2884)
  Move Guide topic: Config Provider Firewalls. (#2883)
  Move Guide topic: Kubeconfig File
  Move Accessing Clusters topic to Concepts. (#2875)
  Move Guide topic: Sharing Clusters
  Move Guide topic: Prereqs
  Move Garbage Collection topic. (#2874)
  Move PetSets topic. (#2873)
  Move Guide topic: Pod Templates (#2872)
  Move StatefulSets topic. (#2869)
  Rename /docs/tasks/job/work-queue-1/
  Move Guide topic: Fine Parallel Processing using a Work Queue (#2870)
  Move Guide topic: Coarse Parallel Processing Using a Work Queue
  Move Init Containers topic. (#2866)
  Move Guide topic: Parallel Processing using Expansions (#2867)
  Move Pod overview. (#2865)
  ...

# Conflicts:
#	docs/api.md
#	docs/user-guide/jobs.md
This commit is contained in:
Andrew Chen
2017-03-17 17:33:48 -07:00
169 changed files with 9841 additions and 8993 deletions
@@ -0,0 +1,366 @@
---
assignees:
- derekwaynecarr
- janetkuo
title: Applying Resource Quotas and Limits
---
This example demonstrates a typical setup to control for resource usage in a namespace.
It demonstrates using the following resources:
* [Namespace](/docs/admin/namespaces)
* [Resource Quota](/docs/admin/resourcequota/)
* [Limit Range](/docs/admin/limitrange/)
This example assumes you have a functional Kubernetes setup.
## Scenario
The cluster-admin is operating a cluster on behalf of a user population and the cluster-admin
wants to control the amount of resources that can be consumed in a particular namespace to promote
fair sharing of the cluster and control cost.
The cluster-admin has the following goals:
* Limit the amount of compute resource for running pods
* Limit the number of persistent volume claims to control access to storage
* Limit the number of load balancers to control cost
* Prevent the use of node ports to preserve scarce resources
* Provide default compute resource requests to enable better scheduling decisions
## Step 1: Create a namespace
This example will work in a custom namespace to demonstrate the concepts involved.
Let's create a new namespace called quota-example:
```shell
$ kubectl create -f docs/admin/resourcequota/namespace.yaml
namespace "quota-example" created
$ kubectl get namespaces
NAME STATUS AGE
default Active 2m
kube-system Active 2m
quota-example Active 39s
```
## Step 2: Apply an object-count quota to the namespace
The cluster-admin wants to control the following resources:
* persistent volume claims
* load balancers
* node ports
Let's create a simple quota that controls object counts for those resource types in this namespace.
```shell
$ kubectl create -f docs/admin/resourcequota/object-counts.yaml --namespace=quota-example
resourcequota "object-counts" created
```
The quota system will observe that a quota has been created, and will calculate consumption
in the namespace in response. This should happen quickly.
Let's describe the quota to see what is currently being consumed in this namespace:
```shell
$ kubectl describe quota object-counts --namespace=quota-example
Name: object-counts
Namespace: quota-example
Resource Used Hard
-------- ---- ----
persistentvolumeclaims 0 2
services.loadbalancers 0 2
services.nodeports 0 0
```
The quota system will now prevent users from creating more than the specified amount for each resource.
## Step 3: Apply a compute-resource quota to the namespace
To limit the amount of compute resource that can be consumed in this namespace,
let's create a quota that tracks compute resources.
```shell
$ kubectl create -f docs/admin/resourcequota/compute-resources.yaml --namespace=quota-example
resourcequota "compute-resources" created
```
Let's describe the quota to see what is currently being consumed in this namespace:
```shell
$ kubectl describe quota compute-resources --namespace=quota-example
Name: compute-resources
Namespace: quota-example
Resource Used Hard
-------- ---- ----
limits.cpu 0 2
limits.memory 0 2Gi
pods 0 4
requests.cpu 0 1
requests.memory 0 1Gi
```
The quota system will now prevent the namespace from having more than 4 non-terminal pods. In
addition, it will enforce that each container in a pod makes a `request` and defines a `limit` for
`cpu` and `memory`.
## Step 4: Applying default resource requests and limits
Pod authors rarely specify resource requests and limits for their pods.
Since we applied a quota to our project, let's see what happens when an end-user creates a pod that has unbounded
cpu and memory by creating an nginx container.
To demonstrate, lets create a deployment that runs nginx:
```shell
$ kubectl run nginx --image=nginx --replicas=1 --namespace=quota-example
deployment "nginx" created
```
Now let's look at the pods that were created.
```shell
$ kubectl get pods --namespace=quota-example
```
What happened? I have no pods! Let's describe the deployment to get a view of what is happening.
```shell
$ kubectl describe deployment nginx --namespace=quota-example
Name: nginx
Namespace: quota-example
CreationTimestamp: Mon, 06 Jun 2016 16:11:37 -0400
Labels: run=nginx
Selector: run=nginx
Replicas: 0 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
OldReplicaSets: <none>
NewReplicaSet: nginx-3137573019 (0/1 replicas created)
...
```
A deployment created a corresponding replica set and attempted to size it to create a single pod.
Let's look at the replica set to get more detail.
```shell
$ kubectl describe rs nginx-3137573019 --namespace=quota-example
Name: nginx-3137573019
Namespace: quota-example
Image(s): nginx
Selector: pod-template-hash=3137573019,run=nginx
Labels: pod-template-hash=3137573019
run=nginx
Replicas: 0 current / 1 desired
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
4m 7s 11 {replicaset-controller } Warning FailedCreate Error creating: pods "nginx-3137573019-" is forbidden: Failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
```
The Kubernetes API server is rejecting the replica set requests to create a pod because our pods
do not specify `requests` or `limits` for `cpu` and `memory`.
So let's set some default values for the amount of `cpu` and `memory` a pod can consume:
```shell
$ kubectl create -f docs/admin/resourcequota/limits.yaml --namespace=quota-example
limitrange "limits" created
$ kubectl describe limits limits --namespace=quota-example
Name: limits
Namespace: quota-example
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory - - 256Mi 512Mi -
Container cpu - - 100m 200m -
```
If the Kubernetes API server observes a request to create a pod in this namespace, and the containers
in that pod do not make any compute resource requests, a default request and default limit will be applied
as part of admission control.
In this example, each pod created will have compute resources equivalent to the following:
```shell
$ kubectl run nginx \
--image=nginx \
--replicas=1 \
--requests=cpu=100m,memory=256Mi \
--limits=cpu=200m,memory=512Mi \
--namespace=quota-example
```
Now that we have applied default compute resources for our namespace, our replica set should be able to create
its pods.
```shell
$ kubectl get pods --namespace=quota-example
NAME READY STATUS RESTARTS AGE
nginx-3137573019-fvrig 1/1 Running 0 6m
```
And if we print out our quota usage in the namespace:
```shell
$ kubectl describe quota --namespace=quota-example
Name: compute-resources
Namespace: quota-example
Resource Used Hard
-------- ---- ----
limits.cpu 200m 2
limits.memory 512Mi 2Gi
pods 1 4
requests.cpu 100m 1
requests.memory 256Mi 1Gi
Name: object-counts
Namespace: quota-example
Resource Used Hard
-------- ---- ----
persistentvolumeclaims 0 2
services.loadbalancers 0 2
services.nodeports 0 0
```
As you can see, the pod that was created is consuming explicit amounts of compute resources, and the usage is being
tracked by Kubernetes properly.
## Step 5: Advanced quota scopes
Let's imagine you did not want to specify default compute resource consumption in your namespace.
Instead, you want to let users run a specific number of `BestEffort` pods in their namespace to take
advantage of slack compute resources, and then require that users make an explicit resource request for
pods that require a higher quality of service.
Let's create a new namespace with two quotas to demonstrate this behavior:
```shell
$ kubectl create namespace quota-scopes
namespace "quota-scopes" created
$ kubectl create -f docs/admin/resourcequota/best-effort.yaml --namespace=quota-scopes
resourcequota "best-effort" created
$ kubectl create -f docs/admin/resourcequota/not-best-effort.yaml --namespace=quota-scopes
resourcequota "not-best-effort" created
$ kubectl describe quota --namespace=quota-scopes
Name: best-effort
Namespace: quota-scopes
Scopes: BestEffort
* Matches all pods that have best effort quality of service.
Resource Used Hard
-------- ---- ----
pods 0 10
Name: not-best-effort
Namespace: quota-scopes
Scopes: NotBestEffort
* Matches all pods that do not have best effort quality of service.
Resource Used Hard
-------- ---- ----
limits.cpu 0 2
limits.memory 0 2Gi
pods 0 4
requests.cpu 0 1
requests.memory 0 1Gi
```
In this scenario, a pod that makes no compute resource requests will be tracked by the `best-effort` quota.
A pod that does make compute resource requests will be tracked by the `not-best-effort` quota.
Let's demonstrate this by creating two deployments:
```shell
$ kubectl run best-effort-nginx --image=nginx --replicas=8 --namespace=quota-scopes
deployment "best-effort-nginx" created
$ kubectl run not-best-effort-nginx \
--image=nginx \
--replicas=2 \
--requests=cpu=100m,memory=256Mi \
--limits=cpu=200m,memory=512Mi \
--namespace=quota-scopes
deployment "not-best-effort-nginx" created
```
Even though no default limits were specified, the `best-effort-nginx` deployment will create
all 8 pods. This is because it is tracked by the `best-effort` quota, and the `not-best-effort`
quota will just ignore it. The `not-best-effort` quota will track the `not-best-effort-nginx`
deployment since it creates pods with `Burstable` quality of service.
Let's list the pods in the namespace:
```shell
$ kubectl get pods --namespace=quota-scopes
NAME READY STATUS RESTARTS AGE
best-effort-nginx-3488455095-2qb41 1/1 Running 0 51s
best-effort-nginx-3488455095-3go7n 1/1 Running 0 51s
best-effort-nginx-3488455095-9o2xg 1/1 Running 0 51s
best-effort-nginx-3488455095-eyg40 1/1 Running 0 51s
best-effort-nginx-3488455095-gcs3v 1/1 Running 0 51s
best-effort-nginx-3488455095-rq8p1 1/1 Running 0 51s
best-effort-nginx-3488455095-udhhd 1/1 Running 0 51s
best-effort-nginx-3488455095-zmk12 1/1 Running 0 51s
not-best-effort-nginx-2204666826-7sl61 1/1 Running 0 23s
not-best-effort-nginx-2204666826-ke746 1/1 Running 0 23s
```
As you can see, all 10 pods have been allowed to be created.
Let's describe current quota usage in the namespace:
```shell
$ kubectl describe quota --namespace=quota-scopes
Name: best-effort
Namespace: quota-scopes
Scopes: BestEffort
* Matches all pods that have best effort quality of service.
Resource Used Hard
-------- ---- ----
pods 8 10
Name: not-best-effort
Namespace: quota-scopes
Scopes: NotBestEffort
* Matches all pods that do not have best effort quality of service.
Resource Used Hard
-------- ---- ----
limits.cpu 400m 2
limits.memory 1Gi 2Gi
pods 2 4
requests.cpu 200m 1
requests.memory 512Mi 1Gi
```
As you can see, the `best-effort` quota has tracked the usage for the 8 pods we created in
the `best-effort-nginx` deployment, and the `not-best-effort` quota has tracked the usage for
the 2 pods we created in the `not-best-effort-nginx` quota.
Scopes provide a mechanism to subdivide the set of resources that are tracked by
any quota document to allow greater flexibility in how operators deploy and track resource
consumption.
In addition to `BestEffort` and `NotBestEffort` scopes, there are scopes to restrict
long-running versus time-bound pods. The `Terminating` scope will match any pod
where `spec.activeDeadlineSeconds is not nil`. The `NotTerminating` scope will match any pod
where `spec.activeDeadlineSeconds is nil`. These scopes allow you to quota pods based on their
anticipated permanence on a node in your cluster.
## Summary
Actions that consume node resources for cpu and memory can be subject to hard quota limits defined by the namespace quota.
Any action that consumes those resources can be tweaked, or can pick up namespace level defaults to meet your end goal.
Quota can be apportioned based on quality of service and anticipated permanence on a node in your cluster.
@@ -0,0 +1,95 @@
---
assignees:
- davidopp
title: Configuring a Pod Disruption Budget
---
This guide is for anyone wishing to specify safety constraints on pods or anyone
wishing to write software (typically automation software) that respects those
constraints.
* TOC
{:toc}
## Rationale
Various cluster management operations may voluntarily evict pods. "Voluntary"
means an eviction can be safely delayed for a reasonable period of time. The
principal examples today are draining a node for maintenance or upgrade
(`kubectl drain`), and cluster autoscaling down. In the future the
[rescheduler](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/rescheduling.md)
may also perform voluntary evictions. By contrast, something like evicting pods
because a node has become unreachable or reports `NotReady`, is not "voluntary."
For voluntary evictions, it can be useful for applications to be able to limit
the number of pods that are down simultaneously. For example, a quorum-based application would
like to ensure that the number of replicas running is never brought below the
number needed for a quorum, even temporarily. Or a web front end might want to
ensure that the number of replicas serving load never falls below a certain
percentage of the total, even briefly. `PodDisruptionBudget` is an API object
that specifies the minimum number or percentage of replicas of a collection that
must be up at a time. Components that wish to evict a pod subject to disruption
budget use the `/eviction` subresource; unlike a regular pod deletion, this
operation may be rejected by the API server if the eviction would cause a
disruption budget to be violated.
## Specifying a PodDisruptionBudget
A `PodDisruptionBudget` has two components: a label selector `selector` to specify the set of
pods to which it applies, and `minAvailable` which is a description of the number of pods from that
set that must still be available after the eviction, i.e. even in the absence
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
So for example, 100% means no voluntary evictions from the set are permitted. In
typical usage, a single budget would be used for a collection of pods managed by
a controller—for example, the pods in a single ReplicaSet.
Note that a disruption budget does not truly guarantee that the specified
number/percentage of pods will always be up. For example, a node that hosts a
pod from the collection may fail when the collection is at the minimum size
specified in the budget, thus bringing the number of available pods from the
collection below the specified size. The budget can only protect against
voluntary evictions, not all causes of unavailability.
## Requesting an eviction
If you are writing infrastructure software that wants to produce these voluntary
evictions, you will need to use the eviction API. The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
"apiVersion": "policy/v1beta1",
"kind": "Eviction",
"metadata": {
"name": "quux",
"namespace": "default"
}
}
```
You can attempt an eviction using `curl`:
```bash
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
```
The API can respond in one of three ways.
1. If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
2. If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
future versions.
3. If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases.
1. There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
2. There is at least one budget. In this case, any of the three above responses may
apply.
@@ -0,0 +1,214 @@
---
assignees:
- derekwaynecarr
- janetkuo
title: Setting Pod CPU and Memory Limits
---
By default, pods run with unbounded CPU and memory limits. This means that any pod in the
system will be able to consume as much CPU and memory on the node that executes the pod.
Users may want to impose restrictions on the amount of resources a single pod in the system may consume
for a variety of reasons.
For example:
1. Each node in the cluster has 2GB of memory. The cluster operator does not want to accept pods
that require more than 2GB of memory since no node in the cluster can support the requirement. To prevent a
pod from being permanently unscheduled to a node, the operator instead chooses to reject pods that exceed 2GB
of memory as part of admission control.
2. A cluster is shared by two communities in an organization that runs production and development workloads
respectively. Production workloads may consume up to 8GB of memory, but development workloads may consume up
to 512MB of memory. The cluster operator creates a separate namespace for each workload, and applies limits to
each namespace.
3. Users may create a pod which consumes resources just below the capacity of a machine. The left over space
may be too small to be useful, but big enough for the waste to be costly over the entire cluster. As a result,
the cluster operator may want to set limits that a pod must consume at least 20% of the memory and CPU of their
average node size in order to provide for more uniform scheduling and limit waste.
This example demonstrates how limits can be applied to a Kubernetes [namespace](/docs/admin/namespaces/walkthrough/) to control
min/max resource limits per pod. In addition, this example demonstrates how you can
apply default resource limits to pods in the absence of an end-user specified value.
See [LimitRange design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/)
## Step 0: Prerequisites
This example requires a running Kubernetes cluster. See the [Getting Started guides](/docs/getting-started-guides/) for how to get started.
Change to the `<kubernetes>` directory if you're not already there.
## Step 1: Create a namespace
This example will work in a custom namespace to demonstrate the concepts involved.
Let's create a new namespace called limit-example:
```shell
$ kubectl create namespace limit-example
namespace "limit-example" created
```
Note that `kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands:
```shell
$ kubectl get namespaces
NAME STATUS AGE
default Active 51s
limit-example Active 45s
```
## Step 2: Apply a limit to the namespace
Let's create a simple limit in our namespace.
```shell
$ kubectl create -f docs/admin/limitrange/limits.yaml --namespace=limit-example
limitrange "mylimits" created
```
Let's describe the limits that we have imposed in our namespace.
```shell
$ kubectl describe limits mylimits --namespace=limit-example
Name: mylimits
Namespace: limit-example
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod cpu 200m 2 - - -
Pod memory 6Mi 1Gi - - -
Container cpu 100m 2 200m 300m -
Container memory 3Mi 1Gi 100Mi 200Mi -
```
In this scenario, we have said the following:
1. If a max constraint is specified for a resource (2 CPU and 1Gi memory in this case), then a limit
must be specified for that resource across all containers. Failure to specify a limit will result in
a validation error when attempting to create the pod. Note that a default value of limit is set by
*default* in file `limits.yaml` (300m CPU and 200Mi memory).
2. If a min constraint is specified for a resource (100m CPU and 3Mi memory in this case), then a
request must be specified for that resource across all containers. Failure to specify a request will
result in a validation error when attempting to create the pod. Note that a default value of request is
set by *defaultRequest* in file `limits.yaml` (200m CPU and 100Mi memory).
3. For any pod, the sum of all containers memory requests must be >= 6Mi and the sum of all containers
memory limits must be <= 1Gi; the sum of all containers CPU requests must be >= 200m and the sum of all
containers CPU limits must be <= 2.
## Step 3: Enforcing limits at point of creation
The limits enumerated in a namespace are only enforced when a pod is created or updated in
the cluster. If you change the limits to a different value range, it does not affect pods that
were previously created in a namespace.
If a resource (CPU or memory) is being restricted by a limit, the user will get an error at time
of creation explaining why.
Let's first spin up a [Deployment](/docs/user-guide/deployments) that creates a single container Pod to demonstrate
how default values are applied to each pod.
```shell
$ kubectl run nginx --image=nginx --replicas=1 --namespace=limit-example
deployment "nginx" created
```
Note that `kubectl run` creates a Deployment named "nginx" on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead.
If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/user-guide/kubectl/kubectl_run/) for more details.
The Deployment manages 1 replica of single container Pod. Let's take a look at the Pod it manages. First, find the name of the Pod:
```shell
$ kubectl get pods --namespace=limit-example
NAME READY STATUS RESTARTS AGE
nginx-2040093540-s8vzu 1/1 Running 0 11s
```
Let's print this Pod with yaml output format (using `-o yaml` flag), and then `grep` the `resources` field. Note that your pod name will be different.
```shell
$ kubectl get pods nginx-2040093540-s8vzu --namespace=limit-example -o yaml | grep resources -C 8
resourceVersion: "57"
selfLink: /api/v1/namespaces/limit-example/pods/nginx-2040093540-ivimu
uid: 67b20741-f53b-11e5-b066-64510658e388
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
resources:
limits:
cpu: 300m
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi
terminationMessagePath: /dev/termination-log
volumeMounts:
```
Note that our nginx container has picked up the namespace default CPU and memory resource *limits* and *requests*.
Let's create a pod that exceeds our allowed limits by having it have a container that requests 3 CPU cores.
```shell
$ kubectl create -f docs/admin/limitrange/invalid-pod.yaml --namespace=limit-example
Error from server: error when creating "docs/admin/limitrange/invalid-pod.yaml": Pod "invalid-pod" is forbidden: [Maximum cpu usage per Pod is 2, but limit is 3., Maximum cpu usage per Container is 2, but limit is 3.]
```
Let's create a pod that falls within the allowed limit boundaries.
```shell
$ kubectl create -f docs/admin/limitrange/valid-pod.yaml --namespace=limit-example
pod "valid-pod" created
```
Now look at the Pod's resources field:
```shell
$ kubectl get pods valid-pod --namespace=limit-example -o yaml | grep -C 6 resources
uid: 3b1bfd7a-f53c-11e5-b066-64510658e388
spec:
containers:
- image: gcr.io/google_containers/serve_hostname
imagePullPolicy: Always
name: kubernetes-serve-hostname
resources:
limits:
cpu: "1"
memory: 512Mi
requests:
cpu: "1"
memory: 512Mi
```
Note that this pod specifies explicit resource *limits* and *requests* so it did not pick up the namespace
default values.
Note: The *limits* for CPU resource are enforced in the default Kubernetes setup on the physical node
that runs the container unless the administrator deploys the kubelet with the following flag:
```shell
$ kubelet --help
Usage of kubelet
....
--cpu-cfs-quota[=true]: Enable CPU CFS quota enforcement for containers that specify CPU limits
$ kubelet --cpu-cfs-quota=false ...
```
## Step 4: Cleanup
To remove the resources used by this example, you can just delete the limit-example namespace.
```shell
$ kubectl delete namespace limit-example
namespace "limit-example" deleted
$ kubectl get namespaces
NAME STATUS AGE
default Active 12m
```
## Summary
Cluster operators that want to restrict the amount of resources a single container or pod may consume
are able to define allowable ranges per Kubernetes namespace. In the absence of any explicit assignments,
the Kubernetes system is able to apply default resource *limits* and *requests* if desired in order to
constrain the amount of resource a pod consumes on a node.