whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #29227 from sftim/20210804_tweak_resource_limiting_troubleshooting_advice

Browse Source 
Tweak advice about managing resources for containers
This commit is contained in:
Kubernetes Prow Robot
2021-12-21 17:13:36 -08:00
committed by GitHub
parent 4c5b12c986 367faca49f
commit 2521d09aff
1 changed files with 182 additions and 141 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/concepts/configuration/manage-resources-containers.md
+182 -141
View File
@@ -1,22 +1,24 @@
--- ---
title: Managing Resources for Containers title: Resource Management for Pods and Containers
content_type: concept content_type: concept
weight: 40 weight: 40
feature: feature:
title: Automatic bin packing title: Automatic bin packing
description: > description: >
Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability. Mix critical and best-effort workloads in order to drive up utilization and save even more resources. Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability.
Mix critical and best-effort workloads in order to drive up utilization and save even more resources.
--- ---
<!-- overview --> <!-- overview -->
When you specify a {{< glossary_tooltip term_id="pod" >}}, you can optionally specify how When you specify a {{< glossary_tooltip term_id="pod" >}}, you can optionally specify how
much of each resource a {{< glossary_tooltip text="Container" term_id="container" >}} needs. much of each resource a {{< glossary_tooltip text="container" term_id="container" >}} needs.
The most common resources to specify are CPU and memory (RAM); there are others. The most common resources to specify are CPU and memory (RAM); there are others.
When you specify the resource _request_ for Containers in a Pod, the scheduler uses this When you specify the resource _request_ for containers in a Pod, the
{{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}} uses this
information to decide which node to place the Pod on. When you specify a resource _limit_ information to decide which node to place the Pod on. When you specify a resource _limit_
for a Container, the kubelet enforces those limits so that the running container is not for a container, the kubelet enforces those limits so that the running container is not
allowed to use more of that resource than the limit you set. The kubelet also reserves allowed to use more of that resource than the limit you set. The kubelet also reserves
at least the _request_ amount of that system resource specifically for that container at least the _request_ amount of that system resource specifically for that container
to use. to use.
@@ -33,7 +35,7 @@ For example, if you set a `memory` request of 256 MiB for a container, and that
a Pod scheduled to a Node with 8GiB of memory and no other Pods, then the container can try to use a Pod scheduled to a Node with 8GiB of memory and no other Pods, then the container can try to use
more RAM. more RAM.
If you set a `memory` limit of 4GiB for that Container, the kubelet (and If you set a `memory` limit of 4GiB for that container, the kubelet (and
{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}) enforce the limit. {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}) enforce the limit.
The runtime prevents the container from using more than the configured resource limit. For example: The runtime prevents the container from using more than the configured resource limit. For example:
when a process in the container tries to consume more than the allowed amount of memory, when a process in the container tries to consume more than the allowed amount of memory,
@@ -45,8 +47,8 @@ or by enforcement (the system prevents the container from ever exceeding the lim
runtimes can have different ways to implement the same restrictions. runtimes can have different ways to implement the same restrictions.
{{< note >}} {{< note >}}
If a Container specifies its own memory limit, but does not specify a memory request, Kubernetes If a container specifies its own memory limit, but does not specify a memory request, Kubernetes
automatically assigns a memory request that matches the limit. Similarly, if a Container specifies its own automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own
CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches
the limit. the limit.
{{< /note >}} {{< /note >}}
@@ -56,7 +58,7 @@ the limit.
*CPU* and *memory* are each a *resource type*. A resource type has a base unit. *CPU* and *memory* are each a *resource type*. A resource type has a base unit.
CPU represents compute processing and is specified in units of [Kubernetes CPUs](#meaning-of-cpu). CPU represents compute processing and is specified in units of [Kubernetes CPUs](#meaning-of-cpu).
Memory is specified in units of bytes. Memory is specified in units of bytes.
If you're using Kubernetes v1.14 or newer, you can specify _huge page_ resources. For Linux workloads, you can specify _huge page_ resources.
Huge pages are a Linux-specific feature where the node kernel allocates blocks of memory Huge pages are a Linux-specific feature where the node kernel allocates blocks of memory
that are much larger than the default page size. that are much larger than the default page size.
@@ -76,9 +78,10 @@ consumed. They are distinct from
[Services](/docs/concepts/services-networking/service/) are objects that can be read and modified [Services](/docs/concepts/services-networking/service/) are objects that can be read and modified
through the Kubernetes API server. through the Kubernetes API server.
## Resource requests and limits of Pod and Container ## Resource requests and limits of Pod and container
Each Container of a Pod can specify one or more of the following: For each container, you can specify resource limits and requests,
including the following:
* `spec.containers[].resources.limits.cpu` * `spec.containers[].resources.limits.cpu`
* `spec.containers[].resources.limits.memory` * `spec.containers[].resources.limits.memory`
@@ -87,49 +90,64 @@ Each Container of a Pod can specify one or more of the following:
* `spec.containers[].resources.requests.memory` * `spec.containers[].resources.requests.memory`
* `spec.containers[].resources.requests.hugepages-<size>` * `spec.containers[].resources.requests.hugepages-<size>`
Although requests and limits can only be specified on individual Containers, it Although you can only specify requests and limits for individual containers,
is convenient to talk about Pod resource requests and limits. A it is also useful to think about the overall resource requests and limits for
*Pod resource request/limit* for a particular resource type is the sum of the a Pod.
resource requests/limits of that type for each Container in the Pod. For a particular resource, a *Pod resource request/limit* is the sum of the
resource requests/limits of that type for each container in the Pod.
## Resource units in Kubernetes ## Resource units in Kubernetes
### Meaning of CPU ### CPU resource units {#meaning-of-cpu}
Limits and requests for CPU resources are measured in *cpu* units. Limits and requests for CPU resources are measured in *cpu* units.
One cpu, in Kubernetes, is equivalent to **1 vCPU/Core** for cloud providers and **1 hyperthread** on bare-metal Intel processors. In Kubernetes, 1 CPU unit is equivalent to **1 physical CPU core**,
or **1 virtual core**, depending on whether the node is a physical host
or a virtual machine running inside a physical machine.
Fractional requests are allowed. When you define a container with Fractional requests are allowed. When you define a container with
`spec.containers[].resources.requests.cpu` set to `0.5`, you are requesting half `spec.containers[].resources.requests.cpu` set to `0.5`, you are requesting half
as much CPU time compared to if you asked for `1.0` CPU. as much CPU time compared to if you asked for `1.0` CPU.
For CPU resource units, the expression `0.1` is equivalent to the For CPU resource units, the [quantity](/docs/reference/kubernetes-api/common-definitions/quantity/) expression `0.1` is equivalent to the
expression `100m`, which can be read as "one hundred millicpu". Some people say expression `100m`, which can be read as "one hundred millicpu". Some people say
"one hundred millicores", and this is understood to mean the same thing. A "one hundred millicores", and this is understood to mean the same thing.
request with a decimal point, like `0.1`, is converted to `100m` by the API, and
precision finer than `1m` is not allowed. For this reason, the form `100m` might
be preferred.
CPU is always requested as an absolute quantity, never as a relative quantity; CPU resource is always specified as an absolute amount of resource, never as a relative amount. For example,
0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine. `500m` CPU represents the roughly same amount of computing power whether that container
runs on a single-core, dual-core, or 48-core machine.
### Meaning of memory {{< note >}}
Kubernetes doesn't allow you to specify CPU resources with a precision finer than
`1m`. Because of this, it's useful to specify CPU units less than `1.0` or `1000m` using
the milliCPU form; for example, `5m` rather than `0.005`.
{{< /note >}}
### Memory resource units {#meaning-of-memory}
Limits and requests for `memory` are measured in bytes. You can express memory as Limits and requests for `memory` are measured in bytes. You can express memory as
a plain integer or as a fixed-point number using one of these suffixes: a plain integer or as a fixed-point number using one of these
E, P, T, G, M, k, m (millis). You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, [quantity](/docs/reference/kubernetes-api/common-definitions/quantity/) suffixes:
E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
Mi, Ki. For example, the following represent roughly the same value: Mi, Ki. For example, the following represent roughly the same value:
```shell ```shell
128974848, 129e6, 129M, 128974848000m, 123Mi 128974848, 129e6, 129M, 128974848000m, 123Mi
``` ```
Here's an example. Take care about case for suffixes. If you request `400m` of memory, this is a request
The following Pod has two Containers. Each Container has a request of 0.25 cpu for 0.4 bytes. Someone who types that probably meant to ask for 400 mebibytes (`400Mi`)
and 64MiB (2<sup>26</sup> bytes) of memory. Each Container has a limit of 0.5 or 400 megabytes (`400M`).
cpu and 128MiB of memory. You can say the Pod has a request of 0.5 cpu and 128
MiB of memory, and a limit of 1 cpu and 256MiB of memory. ## Container resources example {#example-1}
The following Pod has two containers. Both containers are defined with a request for
0.25 CPU
and 64MiB (2<sup>26</sup> bytes) of memory. Each container has a limit of 0.5
CPU and 128MiB of memory. You can say the Pod has a request of 0.5 CPU and 128
MiB of memory, and a limit of 1 CPU and 256MiB of memory.
```yaml ```yaml
---
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
metadata: metadata:
@@ -162,56 +180,54 @@ When you create a Pod, the Kubernetes scheduler selects a node for the Pod to
run on. Each node has a maximum capacity for each of the resource types: the run on. Each node has a maximum capacity for each of the resource types: the
amount of CPU and memory it can provide for Pods. The scheduler ensures that, amount of CPU and memory it can provide for Pods. The scheduler ensures that,
for each resource type, the sum of the resource requests of the scheduled for each resource type, the sum of the resource requests of the scheduled
Containers is less than the capacity of the node. Note that although actual memory containers is less than the capacity of the node.
Note that although actual memory
or CPU resource usage on nodes is very low, the scheduler still refuses to place or CPU resource usage on nodes is very low, the scheduler still refuses to place
a Pod on a node if the capacity check fails. This protects against a resource a Pod on a node if the capacity check fails. This protects against a resource
shortage on a node when resource usage later increases, for example, during a shortage on a node when resource usage later increases, for example, during a
daily peak in request rate. daily peak in request rate.
## How Pods with resource limits are run ## How Kubernetes applies resource requests and limits {#how-pods-with-resource-limits-are-run}
When the kubelet starts a Container of a Pod, it passes the CPU and memory limits When the kubelet starts a container as part of a Pod, the kubelet passes that container's
to the container runtime. requests and limits for memory and CPU to the container runtime.
When using Docker: On Linux, the container runtime typically configures
kernel {{< glossary_tooltip text="cgroups" term_id="cgroup" >}} that apply and enforce the
limits you defined.
- The `spec.containers[].resources.requests.cpu` is converted to its core value, - The CPU limit defines a hard ceiling on how much CPU time that the container can use.
which is potentially fractional, and multiplied by 1024. The greater of this number During each scheduling interval (time slice), the Linux kernel checks to see if this
or 2 is used as the value of the limit is exceeded; if so, the kernel waits before allowing that cgroup to resume execution.
[`--cpu-shares`](https://docs.docker.com/engine/reference/run/#cpu-share-constraint) - The CPU request typically defines a weighting. If several different containers (cgroups)
flag in the `docker run` command. want to run on a contended system, workloads with larger CPU requests are allocated more
CPU time than workloads with small requests.
- The memory request is mainly used during (Kubernetes) Pod scheduling. On a node that uses
cgroups v2, the container runtime might use the memory request as a hint to set
`memory.min` and `memory.low`.
- The memory limit defines a memory limit for that cgroup. If the container tries to
allocate more memory than this limit, the Linux kernel out-of-memory subsystem activates
and, typically, intervenes by stopping one of the processes in the container that tried
to allocate memory. If that process is the container's PID 1, and the container is marked
as restartable, Kubernetes restarts the container.
- The memory limit for the Pod or container can also apply to pages in memory backed
volumes, such as an `emptyDir`. The kubelet tracks `tmpfs` emptyDir volumes as container
memory use, rather than as local ephemeral storage.
- The `spec.containers[].resources.limits.cpu` is converted to its millicore value and If a container exceeds its memory request and the node that it runs on becomes short of
multiplied by 100. The resulting value is the total amount of CPU time in microseconds memory overall, it is likely that the Pod the container belongs to will be
that a container can use every 100ms. A container cannot use more than its share of {{< glossary_tooltip text="evicted" term_id="eviction" >}}.
CPU time during this interval.
{{< note >}} A container might or might not be allowed to exceed its CPU limit for extended periods of time.
The default quota period is 100ms. The minimum resolution of CPU quota is 1ms. However, container runtimes don't terminate Pods or containers for excessive CPU usage.
{{</ note >}}
- The `spec.containers[].resources.limits.memory` is converted to an integer, and To determine whether a container cannot be scheduled or is being killed due to resource limits,
used as the value of the see the [Troubleshooting](#troubleshooting) section.
[`--memory`](https://docs.docker.com/engine/reference/run/#/user-memory-constraints)
flag in the `docker run` command.
If a Container exceeds its memory limit, it might be terminated. If it is
restartable, the kubelet will restart it, as with any other type of runtime
failure.
If a Container exceeds its memory request, it is likely that its Pod will
be evicted whenever the node runs out of memory.
A Container might or might not be allowed to exceed its CPU limit for extended
periods of time. However, it will not be killed for excessive CPU usage.
To determine whether a Container cannot be scheduled or is being killed due to
resource limits, see the
[Troubleshooting](#troubleshooting) section.
### Monitoring compute & memory resource usage ### Monitoring compute & memory resource usage
The resource usage of a Pod is reported as part of the Pod status. The kubelet reports the resource usage of a Pod as part of the Pod
[`status`](/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status).
If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
are available in your cluster, then Pod resource usage can be retrieved either are available in your cluster, then Pod resource usage can be retrieved either
@@ -309,21 +325,26 @@ than as local ephemeral storage.
### Setting requests and limits for local ephemeral storage ### Setting requests and limits for local ephemeral storage
You can use _ephemeral-storage_ for managing local ephemeral storage. Each Container of a Pod can specify one or more of the following: You can specify `ephemeral-storage` for managing local ephemeral storage. Each
container of a Pod can specify either or both of the following:
* `spec.containers[].resources.limits.ephemeral-storage` * `spec.containers[].resources.limits.ephemeral-storage`
* `spec.containers[].resources.requests.ephemeral-storage` * `spec.containers[].resources.requests.ephemeral-storage`
Limits and requests for `ephemeral-storage` are measured in bytes. You can express storage as Limits and requests for `ephemeral-storage` are measured in byte quantities.
a plain integer or as a fixed-point number using one of these suffixes: You can express storage as a plain integer or as a fixed-point number using one of these suffixes:
E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
Mi, Ki. For example, the following represent roughly the same value: Mi, Ki. For example, the following quantities all represent roughly the same value:
```shell - `128974848`
128974848, 129e6, 129M, 123Mi - `129e6`
``` - `129M`
- `123Mi`
In the following example, the Pod has two Containers. Each Container has a request of 2GiB of local ephemeral storage. Each Container has a limit of 4GiB of local ephemeral storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage. In the following example, the Pod has two containers. Each container has a request of
2GiB of local ephemeral storage. Each container has a limit of 4GiB of local ephemeral
storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and
a limit of 8GiB of local ephemeral storage.
```yaml ```yaml
apiVersion: v1 apiVersion: v1
@@ -360,9 +381,11 @@ spec:
### How Pods with ephemeral-storage requests are scheduled ### How Pods with ephemeral-storage requests are scheduled
When you create a Pod, the Kubernetes scheduler selects a node for the Pod to When you create a Pod, the Kubernetes scheduler selects a node for the Pod to
run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods. For more information, see [Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods.
For more information, see
[Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
The scheduler ensures that the sum of the resource requests of the scheduled Containers is less than the capacity of the node. The scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node.
### Ephemeral storage consumption management {#resource-emphemeralstorage-consumption} ### Ephemeral storage consumption management {#resource-emphemeralstorage-consumption}
@@ -376,7 +399,7 @@ kubelet measures storage use in:
If a Pod is using more ephemeral storage than you allow it to, the kubelet If a Pod is using more ephemeral storage than you allow it to, the kubelet
sets an eviction signal that triggers Pod eviction. sets an eviction signal that triggers Pod eviction.
For container-level isolation, if a Container's writable layer and log For container-level isolation, if a container's writable layer and log
usage exceeds its storage limit, the kubelet marks the Pod for eviction. usage exceeds its storage limit, the kubelet marks the Pod for eviction.
For pod-level isolation the kubelet works out an overall Pod storage limit by For pod-level isolation the kubelet works out an overall Pod storage limit by
@@ -493,15 +516,19 @@ Plugin](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
for how to advertise device plugin managed resources on each node. for how to advertise device plugin managed resources on each node.
##### Other resources ##### Other resources
To advertise a new node-level extended resource, the cluster operator can To advertise a new node-level extended resource, the cluster operator can
submit a `PATCH` HTTP request to the API server to specify the available submit a `PATCH` HTTP request to the API server to specify the available
quantity in the `status.capacity` for a node in the cluster. After this quantity in the `status.capacity` for a node in the cluster. After this
operation, the node's `status.capacity` will include a new resource. The operation, the node's `status.capacity` will include a new resource. The
`status.allocatable` field is updated automatically with the new resource `status.allocatable` field is updated automatically with the new resource
asynchronously by the kubelet. Note that because the scheduler uses the node asynchronously by the kubelet.
`status.allocatable` value when evaluating Pod fitness, there may be a short
delay between patching the node capacity with a new resource and the first Pod Because the scheduler uses the node's `status.allocatable` value when
that requests the resource to be scheduled on that node. evaluating Pod fitness, the scheduler only takes account of the new value after
that asynchronous update. There may be a short delay between patching the
node capacity with a new resource and the time when the first Pod that requests
the resource can be scheduled on that node.
**Example:** **Example:**
@@ -611,27 +638,32 @@ spec:
## PID limiting ## PID limiting
Process ID (PID) limits allow for the configuration of a kubelet to limit the number of PIDs that a given Pod can consume. See [Pid Limiting](/docs/concepts/policy/pid-limiting/) for information. Process ID (PID) limits allow for the configuration of a kubelet
to limit the number of PIDs that a given Pod can consume. See
[PID Limiting](/docs/concepts/policy/pid-limiting/) for information.
## Troubleshooting ## Troubleshooting
### My Pods are pending with event message failedScheduling ### My Pods are pending with event message `FailedScheduling`
If the scheduler cannot find any node where a Pod can fit, the Pod remains If the scheduler cannot find any node where a Pod can fit, the Pod remains
unscheduled until a place can be found. An event is produced each time the unscheduled until a place can be found. An
scheduler fails to find a place for the Pod, like this: [Event](/docs/reference/kubernetes-api/cluster-resources/event-v1/) is produced
each time the scheduler fails to find a place for the Pod. You can use `kubectl`
to view the events for a Pod; for example:
```shell ```shell
kubectl describe pod frontend | grep -A 3 Events kubectl describe pod frontend | grep -A 9999999999 Events
``` ```
``` ```
Events: Events:
FirstSeen LastSeen Count From Subobject PathReason Message Type Reason Age From Message
36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others ---- ------ ---- ---- -------
Warning FailedScheduling 23s default-scheduler 0/42 nodes available: insufficient cpu
``` ```
In the preceding example, the Pod named "frontend" fails to be scheduled due to In the preceding example, the Pod named "frontend" fails to be scheduled due to
insufficient CPU resource on the node. Similar error messages can also suggest insufficient CPU resource on any node. Similar error messages can also suggest
failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod
is pending with a message of this type, there are several things to try: is pending with a message of this type, there are several things to try:
@@ -640,6 +672,9 @@ is pending with a message of this type, there are several things to try:
- Check that the Pod is not larger than all the nodes. For example, if all the - Check that the Pod is not larger than all the nodes. For example, if all the
nodes have a capacity of `cpu: 1`, then a Pod with a request of `cpu: 1.1` will nodes have a capacity of `cpu: 1`, then a Pod with a request of `cpu: 1.1` will
never be scheduled. never be scheduled.
- Check for node taints. If most of your nodes are tainted, and the new Pod does
not tolerate that taint, the scheduler only considers placements onto the
remaining nodes that don't have that taint.
You can check node capacities and amounts allocated with the You can check node capacities and amounts allocated with the
`kubectl describe nodes` command. For example: `kubectl describe nodes` command. For example:
@@ -674,31 +709,46 @@ Allocated resources:
680m (34%) 400m (20%) 920Mi (11%) 1070Mi (13%) 680m (34%) 400m (20%) 920Mi (11%) 1070Mi (13%)
``` ```
In the preceding output, you can see that if a Pod requests more than 1120m In the preceding output, you can see that if a Pod requests more than 1.120 CPUs,
CPUs or 6.23Gi of memory, it will not fit on the node. or more than 6.23Gi of memory, that Pod will not fit on the node.
By looking at the `Pods` section, you can see which Pods are taking up space on By looking at the “Pods” section, you can see which Pods are taking up space on
the node. the node.
The amount of resources available to Pods is less than the node capacity, because The amount of resources available to Pods is less than the node capacity, because
system daemons use a portion of the available resources. The `allocatable` field system daemons use a portion of the available resources. Within the Kubernetes API,
[NodeStatus](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#nodestatus-v1-core) each Node has a `.status.allocatable` field
gives the amount of resources that are available to Pods. For more information, see (see [NodeStatus](/docs/reference/kubernetes-api/cluster-resources/node-v1/#NodeStatus)
[Node Allocatable Resources](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md). for details).
The [resource quota](/docs/concepts/policy/resource-quotas/) feature can be configured The `.status.allocatable` field describes the amount of resources that are available
to limit the total amount of resources that can be consumed. If used in conjunction to Pods on that node (for example: 15 virtual CPUs and 7538 MiB of memory).
with namespaces, it can prevent one team from hogging all the resources. For more information on node allocatable resources in Kubernetes, see
[Reserve Compute Resources for System Daemons](/docs/tasks/administer-cluster/reserve-compute-resources/).
### My Container is terminated You can configure [resource quotas](/docs/concepts/policy/resource-quotas/)
to limit the total amount of resources that a namespace can consume.
Kubernetes enforces quotas for objects in particular namespace when there is a
ResourceQuota in that namespace.
For example, if you assign specific namespaces to different teams, you
can add ResourceQuotas into those namespaces. Setting resource quotas helps to
prevent one team from using so much of any resource that this over-use affects other teams.
Your Container might get terminated because it is resource-starved. To check You should also consider what access you grant to that namespace:
whether a Container is being killed because it is hitting a resource limit, call **full** write access to a namespace allows someone with that access to remove any
resource, include a configured ResourceQuota.
### My container is terminated
Your container might get terminated because it is resource-starved. To check
whether a container is being killed because it is hitting a resource limit, call
`kubectl describe pod` on the Pod of interest: `kubectl describe pod` on the Pod of interest:
```shell ```shell
kubectl describe pod simmemleak-hra99 kubectl describe pod simmemleak-hra99
``` ```
The output is similar to:
``` ```
Name: simmemleak-hra99 Name: simmemleak-hra99
Namespace: default Namespace: default
@@ -709,57 +759,48 @@ Status: Running
Reason: Reason:
Message: Message:
IP: 10.244.2.75 IP: 10.244.2.75
Replication Controllers: simmemleak (1/1 replicas created)
Containers: Containers:
simmemleak: simmemleak:
Image: saadali/simmemleak Image: saadali/simmemleak:latest
Limits: Limits:
cpu: 100m cpu: 100m
memory: 50Mi memory: 50Mi
State: Running State: Running
Started: Tue, 07 Jul 2015 12:54:41 -0700 Started: Tue, 07 Jul 2019 12:54:41 -0700
Last Termination State: Terminated Last State: Terminated
Exit Code: 1 Reason: OOMKilled
Started: Fri, 07 Jul 2015 12:54:30 -0700 Exit Code: 137
Finished: Fri, 07 Jul 2015 12:54:33 -0700 Started: Fri, 07 Jul 2019 12:54:30 -0700
Ready: False Finished: Fri, 07 Jul 2019 12:54:33 -0700
Restart Count: 5 Ready: False
Restart Count: 5
Conditions: Conditions:
Type Status Type Status
Ready False Ready False
Events: Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message Type Reason Age From Message
Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {scheduler } scheduled Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f ---- ------ ---- ---- -------
Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD pulled Pod container image "k8s.gcr.io/pause:0.8.0" already present on machine Normal Scheduled 42s default-scheduler Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f
Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD created Created with docker id 6a41280f516d Normal Pulled 41s kubelet Container image "saadali/simmemleak:latest" already present on machine
Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD started Started with docker id 6a41280f516d Normal Created 41s kubelet Created container simmemleak
Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} spec.containers{simmemleak} created Created with docker id 87348f12526a Normal Started 40s kubelet Started container simmemleak
Normal Killing 32s kubelet Killing container with id ead3fb35-5cf5-44ed-9ae1-488115be66c6: Need to kill Pod
``` ```
In the preceding example, the `Restart Count: 5` indicates that the `simmemleak` In the preceding example, the `Restart Count: 5` indicates that the `simmemleak`
Container in the Pod was terminated and restarted five times. container in the Pod was terminated and restarted five times (so far).
The `OOMKilled` reason shows that the container tried to use more memory than its limit.
You can call `kubectl get pod` with the `-o go-template=...` option to fetch the status Your next step might be to check the application code for a memory leak. If you
of previously terminated Containers: find that the application is behaving how you expect, consider setting a higher
memory limit (and possibly request) for that container.
```shell
kubectl get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-hra99
```
```
Container Name: simmemleak
LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]
```
You can see that the Container was terminated because of `reason:OOM Killed`, where `OOM` stands for Out Of Memory.
## {{% heading "whatsnext" %}} ## {{% heading "whatsnext" %}}
* Get hands-on experience [assigning Memory resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/). * Get hands-on experience [assigning Memory resources to containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/).
* Get hands-on experience [assigning CPU resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/). * Get hands-on experience [assigning CPU resources to containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/).
* For more details about the difference between requests and limits, see * Read how the API reference defines a [container](/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container)
[Resource QoS](https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md). and its [resource requirements](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#resources)
* Read the [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) API reference
* Read the [ResourceRequirements](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcerequirements-v1-core) API reference
* Read about [project quotas](https://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/xfs-quotas.html) in XFS * Read about [project quotas](https://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/xfs-quotas.html) in XFS
* Read more about the [kube-scheduler configuration reference (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) * Read more about the [kube-scheduler configuration reference (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/)