From 57abef3e780b52a87420b1843e2c30b4cf6c143f Mon Sep 17 00:00:00 2001 From: derekwaynecarr Date: Thu, 23 Jun 2016 10:48:05 -0400 Subject: [PATCH] Update 1.3 documentation for quota --- docs/admin/resourcequota/index.md | 189 ++++++++++++++++++++---------- 1 file changed, 126 insertions(+), 63 deletions(-) diff --git a/docs/admin/resourcequota/index.md b/docs/admin/resourcequota/index.md index 40559f058e..59be97e38e 100644 --- a/docs/admin/resourcequota/index.md +++ b/docs/admin/resourcequota/index.md @@ -4,29 +4,26 @@ When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources. -Resource quotas are a tool for administrators to address this concern. Resource quotas -work like this: +Resource quotas are a tool for administrators to address this concern. + +A resource quota, defined by a `ResourceQuota` object, provides constraints that limit +aggregate resource consumption per namespace. It can limit the quantity of objects that can +be created in a namespace by type, as well as the total amount of compute resources that may +be consumed by resources in that project. + +Resource quotas work like this: - Different teams work in different namespaces. Currently this is voluntary, but support for making this mandatory via ACLs is planned. -- The administrator creates a Resource Quota for each namespace. -- Users put compute resource requests on their pods. The sum of all resource requests across - all pods in the same namespace must not exceed any hard resource limit in any Resource Quota - document for the namespace. Note that we used to verify Resource Quota by taking the sum of - resource limits of the pods, but this was altered to use resource requests. Backwards compatibility - for those pods previously created is preserved because pods that only specify a resource limit have - their resource requests defaulted to match their defined limits. The user is only charged for the - resources they request in the Resource Quota versus their limits because the request is the minimum - amount of resource guaranteed by the cluster during scheduling. For more information on over commit, - see [compute-resources](/docs/user-guide/compute-resources). -- If creating a pod would cause the namespace to exceed any of the limits specified in the - the Resource Quota for that namespace, then the request will fail with HTTP status - code `403 FORBIDDEN`. -- If quota is enabled in a namespace and the user does not specify *requests* on the pod for each - of the resources for which quota is enabled, then the POST of the pod will fail with HTTP - status code `403 FORBIDDEN`. Hint: Use the LimitRange admission controller to force default - values of *limits* (then resource *requests* would be equal to *limits* by default, see - [admission controller](/docs/admin/admission-controllers)) before the quota is checked to avoid this problem. +- The administrator creates one or more Resource Quota objects for each namespace. +- Users create resources (pods, services, etc.) in the namespace, and the quota system + tracks usage to ensure it does not exceed hard resource limits defined in a Resource Quota. +- If creating or updating a resource violates a quota constraint, the request will fail with HTTP + status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated. +- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify + requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use + the LimitRange admission controller to force defaults for pods that make no compute resource requirements. + See the [walkthrough](/docs/admin/resourcequota/walkthrough.md) for an example to avoid this problem. Examples of policies that could be created using namespaces and quotas are: @@ -38,7 +35,7 @@ Examples of policies that could be created using namespaces and quotas are: In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, there may be contention for resources. This is handled on a first-come-first-served basis. -Neither contention nor changes to quota will affect already-running pods. +Neither contention nor changes to quota will affect already created resources. ## Enabling Resource Quota @@ -57,11 +54,12 @@ in a namespace can be limited. The following compute resource types are support | ResourceName | Description | | ------------ | ----------- | -| cpu | Total cpu requests of containers | -| memory | Total memory requests of containers - -For example, `cpu` quota sums up the `resources.requests.cpu` fields of every -container of every pod in the namespace, and enforces a maximum on that sum. +| cpu | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | +| limits.cpu | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. | +| limits.memory | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. | +| memory | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | +| requests.cpu | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | +| requests.memory | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | ## Object Count Quota @@ -70,12 +68,15 @@ are supported: | ResourceName | Description | | ------------ | ----------- | -| pods | Total number of pods | -| services | Total number of services | -| replicationcontrollers | Total number of replication controllers | -| resourcequotas | Total number of [resource quotas](/docs/admin/admission-controllers/#resourcequota) | -| secrets | Total number of secrets | -| persistentvolumeclaims | Total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) | +| configmaps | The total number of config maps that can exist in the namespace. | +| persistentvolumeclaims | The total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | +| pods | The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `status.phase in (Failed, Succeeded)` is true. | +| replicationcontrollers | The total number of replication controllers that can exist in the namespace. | +| resourcequotas | The total number of [resource quotas](/docs/admin/admission-controllers/#resourcequota) that can exist in the namespace. | +| services | The total number of services that can exist in the namespace. | +| services.loadbalancers | The total number of services of type load balancer that can exist in the namespace. | +| services.nodeports | The total number of services of type node port that can exist in the namespace. | +| secrets | The total number of secrets that can exist in the namespace. | For example, `pods` quota counts and enforces a maximum on the number of `pods` created in a single namespace. @@ -84,45 +85,107 @@ You might want to set a pods quota on a namespace to avoid the case where a user creates many small pods and exhausts the cluster's supply of Pod IPs. +## Quota Scopes + +Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches +the intersection of enumerated scopes. + +When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. +Resources specified on the quota outside of the allowed set results in a validation error. + +| Scope | Description | +| ----- | ----------- | +| Terminating | Match pods where `spec.activeDeadlineSeconds >= 0` | +| NotTerminating | Match pods where `spec.activeDeadlineSeconds is nil` | +| BestEffort | Match pods that have best effort quality of service. | +| NotBestEffort | Match pods that do not have best effort quality of service. | + +The `BestEffort` scope restricts a quota to tracking the following resources: +* pods + +The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota to tracking the following resources: +* cpu +* limits.cpu +* limits.memory +* memory +* pods +* requests.cpu +* requests.memory + +## Requests vs Limits + +When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. +The quota can be configured to quota either value. + +If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming +container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`, +then it requires that every incoming container specifies an explict limit for those resources. + ## Viewing and Setting Quotas Kubectl supports creating, updating, and viewing quotas: ```shell -$ kubectl namespace myspace -$ cat < quota.json -{ - "apiVersion": "v1", - "kind": "ResourceQuota", - "metadata": { - "name": "quota" - }, - "spec": { - "hard": { - "memory": "1Gi", - "cpu": "20", - "pods": "10", - "services": "5", - "replicationcontrollers":"20", - "resourcequotas":"1" - } - } -} +$ kubectl create namespace myspace + +$ cat < compute-resources.yaml +apiVersion: v1 +kind: ResourceQuota +metadata: + name: compute-resources +spec: + hard: + pods: "4" + requests.cpu: "1" + requests.memory: 1Gi + limits.cpu: "2" + limits.memory: 2Gi EOF -$ kubectl create -f ./quota.json -$ kubectl get quota -NAME -quota -$ kubectl describe quota quota -Name: quota +$ kubectl create -f ./compute-resources.yaml --namespace=myspace + +$ cat < object-counts.yaml +apiVersion: v1 +kind: ResourceQuota +metadata: + name: object-counts +spec: + hard: + configmaps: "10" + persistentvolumeclaims: "4" + replicationcontrollers: "20" + secrets: "10" + services: "10" + services.loadbalancers: "2" +EOF +$ kubectl create -f ./object-counts.yaml --namespace=myspace + +$ kubectl get quota --namespace=myspace +NAME AGE +compute-resources 30s +object-counts 32s + +$ kubectl describe quota compute-resources --namespace=myspace +Name: compute-resources +Namespace: myspace +Resource Used Hard +-------- ---- ---- +limits.cpu 0 2 +limits.memory 0 2Gi +pods 0 4 +requests.cpu 0 1 +requests.memory 0 1Gi + +$ kubectl describe quota object-counts --namespace=myspace +Name: object-counts +Namespace: myspace Resource Used Hard -------- ---- ---- -cpu 0m 20 -memory 0 1Gi -pods 5 10 -replicationcontrollers 5 20 -resourcequotas 1 1 -services 3 5 +configmaps 0 10 +persistentvolumeclaims 0 4 +replicationcontrollers 0 20 +secrets 1 10 +services 0 10 +services.loadbalancers 0 2 ``` ## Quota and Cluster Capacity