diff --git a/_data/glossary/configmap.yaml b/_data/glossary/configmap.yaml new file mode 100644 index 0000000000..b961b734b0 --- /dev/null +++ b/_data/glossary/configmap.yaml @@ -0,0 +1,13 @@ +id: configmap +name: ConfigMap +full-link: /docs/tasks/configure-pod-container/configmap/ +related: +- pod +- secret +tags: +- core-object +short-description: > + An API object used to store non-confidential data in key-value pairs. Can be consumed as environment variables, command-line arguments, or config files in a {% glossary_tooltip text="volume" term_id="volume" %}. +long-description: > + Allows you to decouple environment-specific configuration from your {% glossary_tooltip text="container images" term_id="container" %}, so that your applications are easily portable. + When storing confidential data use a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/). diff --git a/_data/glossary/daemonset.yaml b/_data/glossary/daemonset.yaml new file mode 100644 index 0000000000..09142d4fd6 --- /dev/null +++ b/_data/glossary/daemonset.yaml @@ -0,0 +1,10 @@ +id: daemonset +name: DaemonSet +full-link: /docs/concepts/workloads/controllers/daemonset +tags: +- fundamental +- workload +short-description: > + Ensures a copy of a {% glossary_tooltip term_id="pod" %} is running across a set of nodes in a {% glossary_tooltip term_id="cluster" %}. +long-description: > + Used to deploy system daemons such as log collectors and monitoring agents that typically must run on every {% glossary_tooltip term_id="node" %}. diff --git a/_data/glossary/image.yaml b/_data/glossary/image.yaml new file mode 100644 index 0000000000..d6ed55e004 --- /dev/null +++ b/_data/glossary/image.yaml @@ -0,0 +1,9 @@ +id: image +name: Image +tags: +- fundamental +short-description: > + Stored instance of a container that holds a set of software needed to run an application. +long-description: > + A way of packaging software that allows it to be stored in a container registry, pulled to a local system, and run as an application. Meta data is included in the image that can indicate what executable to run, who built it, and other information. + diff --git a/_data/glossary/job.yaml b/_data/glossary/job.yaml new file mode 100644 index 0000000000..c868a68941 --- /dev/null +++ b/_data/glossary/job.yaml @@ -0,0 +1,9 @@ +id: job +name: Job +full-link: /docs/concepts/workloads/controllers/jobs-run-to-completion +tags: +- core-object +short-description: > + A finite or batch task that runs to completion. +long-description: > + Creates one or more {% glossary_tooltip term_id="pod" %} objects and ensures that a specified number of them successfully terminate. As Pods successfully complete, the Job tracks the successful completions. diff --git a/_data/glossary/namespace.yaml b/_data/glossary/namespace.yaml new file mode 100644 index 0000000000..6661479620 --- /dev/null +++ b/_data/glossary/namespace.yaml @@ -0,0 +1,9 @@ +id: namespace +name: Namespace +full-link: /docs/concepts/overview/working-with-objects/namespaces +tags: +- fundamental +short-description: > + An abstraction used by Kubernetes to support virtual clusters on the same physical {% glossary_tooltip term_id="cluster" %}. +long-description: > + Namespaces are used to organize objects in a cluster and provide a way to divide cluster resources. Names of resources need to be unique within a namespace, but not across namespaces. diff --git a/_data/glossary/network-policy.yaml b/_data/glossary/network-policy.yaml new file mode 100644 index 0000000000..302c78f289 --- /dev/null +++ b/_data/glossary/network-policy.yaml @@ -0,0 +1,13 @@ +id: network-policy +name: Network Policy +full-link: /docs/concepts/services-networking/network-policies/ +aka: + - NetworkPolicy +tags: +- networking +- architecture +- extension +short-description: > + A specification of how groups of Pods are allowed to communicate with each other and with other network endpoints. +long-description: > + Network Policies help you declaratively configure which Pods are allowed to connect to each other, which namespaces are allowed to communicate, and more specifically which port numbers to enforce each policy on. `NetworkPolicy` resources use labels to select Pods and define rules which specify what traffic is allowed to the selected Pods. Network Policies are implemented by a supported network plugin provided by a network provider. Be aware that creating a network resource without a controller to implement it will have no effect. diff --git a/_data/glossary/node.yaml b/_data/glossary/node.yaml new file mode 100644 index 0000000000..3586dd5ca0 --- /dev/null +++ b/_data/glossary/node.yaml @@ -0,0 +1,13 @@ +id: node +name: Node +full-link: /docs/concepts/architecture/node +aka: +- Minion +related: +- deployment +tags: +- fundamental +short-description: > + A node is a worker machine in Kubernetes. +long-description: > + A worker machine may be a VM or physical machine, depending on the cluster. It has the {% glossary_tooltip text="Services" term_id="service" %} necessary to run {% glossary_tooltip text="Pods" term_id="pod" %} and is managed by the master components. The {% glossary_tooltip text="Services" term_id="service" %} on a node include Docker, kubelet and kube-proxy. diff --git a/_data/glossary/pod-security-policy.yaml b/_data/glossary/pod-security-policy.yaml new file mode 100644 index 0000000000..5c0b721b48 --- /dev/null +++ b/_data/glossary/pod-security-policy.yaml @@ -0,0 +1,19 @@ +id: pod-security-policy +name: Pod Security Policy +full-link: /docs/concepts/policy/pod-security-policy/ +related: +- pod +- container +- sidecar +- deployment +- statefulset +- security +tags: +- core-object +- fundamental +short-description: > + Enables fine-grained authorization of {% glossary_tooltip term_id="pod" %} creation and updates. +long-description: > + A cluster-level resource that controls security sensitive aspects of the Pod specification. + The `PodSecurityPolicy` objects define a set of conditions that a Pod must run with in order to be accepted into the system, as well as defaults for the related fields. + Pod Security Policy control is implemented as an optional admission controller. diff --git a/_data/glossary/replication-controller.yaml b/_data/glossary/replication-controller.yaml new file mode 100644 index 0000000000..6009b49c8b --- /dev/null +++ b/_data/glossary/replication-controller.yaml @@ -0,0 +1,9 @@ +id: replication-controller +name: Replication Controller +tags: +- fundamental +short-description: > + Kubernetes service that ensures a specific number of instances of a pod are always running. +long-description: > + Will automatically add or remove running instances of a pod, based on a set value for that pod. Allows the pod to return to the defined number of instances if pods are deleted or if too many are started by mistake. + diff --git a/_data/glossary/security-context.yaml b/_data/glossary/security-context.yaml new file mode 100644 index 0000000000..7281a7e6cd --- /dev/null +++ b/_data/glossary/security-context.yaml @@ -0,0 +1,9 @@ +id: security-context +name: Security Context +full-link: /docs/tasks/configure-pod-container/security-context/ +tags: +- security +short-description: > + The securityContext field defines privilege and access control settings for a Pod or Container, including the runtime UID and GID. +long-description: > + The securityContext field in a {% glossary_tooltip term_id="pod" %} (applying to all containers) or container is used to set the user (runAsUser) and group (fsGroup), capabilities, privilege settings, and security policies (SELinux/AppArmor/Seccomp) that container processes use. diff --git a/_data/glossary/service-account.yaml b/_data/glossary/service-account.yaml new file mode 100644 index 0000000000..2684bc5e93 --- /dev/null +++ b/_data/glossary/service-account.yaml @@ -0,0 +1,10 @@ +id: service-account +name: Service Account +full-link: /docs/tasks/configure-pod-container/configure-service-account/ +tags: +- fundamental +- core-object +short-description: > + Provides an identity for processes that run in a Pod {% glossary_tooltip text="Pods" term_id="pod" %}. +long-description: > + When processes inside Pods access the cluster, they are authenticated by the API server as a particular service account, for example, `default`. When you create a Pod, if you do not specify a service account, it is automatically assigned the default service account in the same namespace {% glossary_tooltip text="Namespace" term_id="namespace" %}. diff --git a/_data/glossary/volume.yaml b/_data/glossary/volume.yaml new file mode 100644 index 0000000000..60d0fe231e --- /dev/null +++ b/_data/glossary/volume.yaml @@ -0,0 +1,17 @@ +id: volume +name: Volume +full-link: kubernetes.io/docs/concepts/storage/volumes/ +related: +- pod +- container +- secret +tags: +- core-object +- fundamental +short-description: > + A directory containing data, accessible to the containers in a {% glossary_tooltip text="pod" term_id="pod" %}. +long-description: > + A Kubernetes volume lives as long as the {% glossary_tooltip text="pod" term_id="pod" %} that encloses it. + Consequently, a volume outlives any {% glossary_tooltip text="containers" term_id="container" %} that run within the + {% glossary_tooltip text="pod" term_id="pod" %}, and data is preserved across + {% glossary_tooltip text="container" term_id="container" %} restarts. diff --git a/cn/docs/concepts/overview/what-is-kubernetes.md b/cn/docs/concepts/overview/what-is-kubernetes.md index 61537cbefa..46bce4b2f4 100644 --- a/cn/docs/concepts/overview/what-is-kubernetes.md +++ b/cn/docs/concepts/overview/what-is-kubernetes.md @@ -70,7 +70,7 @@ Kubernetes 满足了生产中运行应用程序的许多常见的需求，例如 * [Pod](/docs/user-guide/pods/) 提供复合应用并保留一个应用一个容器的容器模型, * [挂载外部存储](/docs/user-guide/volumes/), * [Secret管理](/docs/user-guide/secrets/), -* [应用健康检查](/docs/user-guide/production-pods/#liveness-and-readiness-probes-aka-health-checks), +* [应用健康检查](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/), * [副本应用实例](/docs/user-guide/replication-controller/), * [横向自动扩缩容](/docs/user-guide/horizontal-pod-autoscaling/), * [服务发现](/docs/user-guide/connecting-applications/), diff --git a/cn/docs/concepts/services-networking/connect-applications-service.md b/cn/docs/concepts/services-networking/connect-applications-service.md index 869d0bcea6..bf548162dd 100644 --- a/cn/docs/concepts/services-networking/connect-applications-service.md +++ b/cn/docs/concepts/services-networking/connect-applications-service.md @@ -366,16 +366,7 @@ LoadBalancer Ingress: a320587ffd19711e5a37606cf4a74574-1142138393.us-east-1.el ... ``` - - ## 进一步阅读 Kubernetes 也支持联合 Service，能够跨多个集群和云提供商，为 Service 提供逐步增强的可用性、更优的容错、更好的可伸缩性。 查看 [联合 Service 用户指南](/docs/concepts/cluster-administration/federation-service-discovery/) 获取更进一步信息。 - - - -## 下一步 - -[了解更多关于 Kubernetes 的特性，有助于在生产环境中可靠地运行容器](/docs/user-guide/production-pods) - diff --git a/cn/docs/concepts/workloads/pods/init-containers.md b/cn/docs/concepts/workloads/pods/init-containers.md index e075dbbec6..08611bf61a 100644 --- a/cn/docs/concepts/workloads/pods/init-containers.md +++ b/cn/docs/concepts/workloads/pods/init-containers.md @@ -85,7 +85,7 @@ Init 容器支持应用容器的全部字段和特性，包括资源限制、数 * 克隆 Git 仓库到数据卷。 * 将配置值放到配置文件中，运行模板工具为主应用容器动态地生成配置文件。例如，在配置文件中存放 POD_IP 值，并使用 Jinja 生成主应用配置文件。 -更多详细用法示例，可以在 [StatefulSet 文档](/docs/concepts/abstractions/controllers/statefulsets/) 和 [生产环境 Pod 指南](/docs/user-guide/production-pods.md#handling-initialization) 中找到。 +更多详细用法示例，可以在 [StatefulSet 文档](/docs/concepts/abstractions/controllers/statefulsets/) 和 [Pod 初始化](/docs/tasks/configure-pod-container/configure-pod-initialization) 中找到。 diff --git a/docs/tasks/administer-cluster/out-of-resource.md b/docs/tasks/administer-cluster/out-of-resource.md index 02098595df..87e88709ed 100644 --- a/docs/tasks/administer-cluster/out-of-resource.md +++ b/docs/tasks/administer-cluster/out-of-resource.md @@ -9,27 +9,25 @@ title: Configure Out Of Resource Handling * TOC {:toc} +This page explains how to configure out of resource handling with `kubelet`. + The `kubelet` needs to preserve node stability when available compute resources -are low. - -This is especially important when dealing with incompressible resources such as -memory or disk. - -If either resource is exhausted, the node would become unstable. +are low. This is especially important when dealing with incompressible +compute resources, such as memory or disk space. If such resources are exhausted, +nodes become unstable. ## Eviction Policy -The `kubelet` can pro-actively monitor for and prevent against total starvation -of a compute resource. In those cases, the `kubelet` can pro-actively fail one -or more pods in order to reclaim the starved resource. When the `kubelet` fails -a pod, it terminates all containers in the pod, and the `PodPhase` is -transitioned to `Failed`. +The `kubelet` can proactively monitor for and prevent total starvation of a +compute resource. In those cases, the `kubelet` can reclaim the starved +resource by proactively failing one or more Pods. When the `kubelet` fails +a Pod, it terminates all of its containers and transitions its `PodPhase` to `Failed`. ### Eviction Signals -The `kubelet` can support the ability to trigger eviction decisions on the -signals described in the table below. The value of each signal is described in -the description column based on the `kubelet` summary API. +The `kubelet` supports eviction decisions based on the signals described in the following +table. The value of each signal is described in the Description column, which is based on +the `kubelet` summary API. | Eviction Signal | Description | |----------------------------|-----------------------------------------------------------------------| @@ -47,7 +45,7 @@ The value for `memory.available` is derived from the cgroupfs instead of tools like `free -m`. This is important because `free -m` does not work in a container, and if users use the [node allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) feature, out of resource decisions -are made local to the end user pod part of the cgroup hierarchy as well as the +are made local to the end user Pod part of the cgroup hierarchy as well as the root node. This [script](/docs/tasks/administer-cluster/out-of-resource/memory-available.sh) reproduces the same set of steps that the `kubelet` performs to calculate @@ -67,64 +65,64 @@ of configurations are not currently supported by the kubelet. For example, it is *not OK* to store volumes and logs in a dedicated `filesystem`. In future releases, the `kubelet` will deprecate the existing [garbage -collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/) support in favor of eviction in -response to disk pressure. +collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/) +support in favor of eviction in response to disk pressure. ### Eviction Thresholds The `kubelet` supports the ability to specify eviction thresholds that trigger the `kubelet` to reclaim resources. -Each threshold is of the following form: +Each threshold has the following form: -`` +`[eviction-signal][operator][quantity]` -* valid `eviction-signal` tokens as defined above. -* valid `operator` tokens are `<` -* valid `quantity` tokens must match the quantity representation used by Kubernetes -* an eviction threshold can be expressed as a percentage if ends with `%` token. +where: -For example, if a node has `10Gi` of memory, and the desire is to induce eviction -if available memory falls below `1Gi`, an eviction threshold can be specified as either -of the following (but not both). +* `eviction-signal` is a eviction signal token as defined in the previous table. +* `operator` is the desired relational operator, such as `<` (less than). +* `quantity` is the eviction threshhold quantity, such as `1Gi`. These tokens must +match the quantity representation used by Kubernetes. An eviction threshold can also +be expressed as a percentage using the `%` token. -* `memory.available<10%` -* `memory.available<1Gi` +For example, if a node has `10Gi` of total memory and you want trigger eviction if +the available memory falls below `1Gi`, you can define the eviction threshold as +either `memory.available<10%` or `memory.available<1Gi`. You cannot use both. #### Soft Eviction Thresholds A soft eviction threshold pairs an eviction threshold with a required -administrator specified grace period. No action is taken by the `kubelet` +administrator-specified grace period. No action is taken by the `kubelet` to reclaim resources associated with the eviction signal until that grace -period has been exceeded. If no grace period is provided, the `kubelet` will -error on startup. +period has been exceeded. If no grace period is provided, the `kubelet` +returns an error on startup. In addition, if a soft eviction threshold has been met, an operator can -specify a maximum allowed pod termination grace period to use when evicting -pods from the node. If specified, the `kubelet` will use the lesser value among +specify a maximum allowed Pod termination grace period to use when evicting +pods from the node. If specified, the `kubelet` uses the lesser value among the `pod.Spec.TerminationGracePeriodSeconds` and the max allowed grace period. -If not specified, the `kubelet` will kill pods immediately with no graceful +If not specified, the `kubelet` kills Pods immediately with no graceful termination. To configure soft eviction thresholds, the following flags are supported: * `eviction-soft` describes a set of eviction thresholds (e.g. `memory.available<1.5Gi`) that if met over a -corresponding grace period would trigger a pod eviction. +corresponding grace period would trigger a Pod eviction. * `eviction-soft-grace-period` describes a set of eviction grace periods (e.g. `memory.available=1m30s`) that -correspond to how long a soft eviction threshold must hold before triggering a pod eviction. +correspond to how long a soft eviction threshold must hold before triggering a Pod eviction. * `eviction-max-pod-grace-period` describes the maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. #### Hard Eviction Thresholds A hard eviction threshold has no grace period, and if observed, the `kubelet` -will take immediate action to reclaim the associated starved resource. If a -hard eviction threshold is met, the `kubelet` will kill the pod immediately +will take immediate action to reclaim the associated starved resource. If a +hard eviction threshold is met, the `kubelet` kills the Pod immediately with no graceful termination. To configure hard eviction thresholds, the following flag is supported: * `eviction-hard` describes a set of eviction thresholds (e.g. `memory.available<1Gi`) that if met -would trigger a pod eviction. +would trigger a Pod eviction. The `kubelet` has the following default hard eviction threshold: @@ -138,10 +136,10 @@ The `kubelet` evaluates eviction thresholds per its configured housekeeping inte ### Node Conditions -The `kubelet` will map one or more eviction signals to a corresponding node condition. +The `kubelet` maps one or more eviction signals to a corresponding node condition. If a hard eviction threshold has been met, or a soft eviction threshold has been met -independent of its associated grace period, the `kubelet` will report a condition that +independent of its associated grace period, the `kubelet` reports a condition that reflects the node is under pressure. The following node conditions are defined that correspond to the specified eviction signal. @@ -151,7 +149,7 @@ The following node conditions are defined that correspond to the specified evict | `MemoryPressure` | `memory.available` | Available memory on the node has satisfied an eviction threshold | | `DiskPressure` | `nodefs.available`, `nodefs.inodesFree`, `imagefs.available`, or `imagefs.inodesFree` | Available disk space and inodes on either the node's root filesytem or image filesystem has satisfied an eviction threshold | -The `kubelet` will continue to report node status updates at the frequency specified by +The `kubelet` continues to report node status updates at the frequency specified by `--node-status-update-frequency` which defaults to `10s`. ### Oscillation of node conditions @@ -174,85 +172,76 @@ condition back to `false`. ### Reclaiming node level resources If an eviction threshold has been met and the grace period has passed, -the `kubelet` will initiate the process of reclaiming the pressured resource +the `kubelet` initiates the process of reclaiming the pressured resource until it has observed the signal has gone below its defined threshold. -The `kubelet` attempts to reclaim node level resources prior to evicting end-user pods. If +The `kubelet` attempts to reclaim node level resources prior to evicting end-user Pods. If disk pressure is observed, the `kubelet` reclaims node level resources differently if the machine has a dedicated `imagefs` configured for the container runtime. -#### With Imagefs +#### With `imagefs` -If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: +If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting the dead Pods and their containers. -1. Delete dead pods/containers +If `imagefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting all unused images. -If `imagefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: +#### Without `imagefs` +If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space in the following order: + +1. Delete dead Pods and their containers 1. Delete all unused images -#### Without Imagefs +### Evicting end-user Pods -If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: +If the `kubelet` is unable to reclaim sufficient resource on the node, `kubelet` begins evicting Pods. -1. Delete dead pods/containers -1. Delete all unused images +The `kubelet` ranks Pods for eviction first by their quality of service, and then by the consumption +of the starved compute resource relative to the Pods' scheduling requests. -### Evicting end-user pods +As a result, `kubectl` ranks and evicts Pods in the following order: -If the `kubelet` is unable to reclaim sufficient resource on the node, -it will begin evicting pods. - -The `kubelet` ranks pods for eviction as follows: - -* by their quality of service. -* by the consumption of the starved compute resource relative to the pods scheduling request. - -As a result, pod eviction occurs in the following order: - -* `BestEffort` pods that consume the most of the starved resource are failed -first. -* `Burstable` pods that consume the greatest amount of the starved resource -relative to their request for that resource are killed first. If no pod +* `BestEffort` Pods consume the most of the starved resource are failed first. +Local disk is a `BestEffort` resource. +* `Burstable` Pods consume the greatest amount of the starved resource +relative to their request for that resource are killed first. If no Pod has exceeded its request, the strategy targets the largest consumer of the starved resource. -* `Guaranteed` pods that consume the greatest amount of the starved resource -relative to their request are killed first. If no pod has exceeded its request, -the strategy targets the largest consumer of the starved resource. - -A `Guaranteed` pod is guaranteed to never be evicted because of another pod's -resource consumption. If a system daemon (i.e. `kubelet`, `docker`, `journald`, etc.) -is consuming more resources than were reserved via `system-reserved` or `kube-reserved` allocations, -and the node only has `Guaranteed` pod(s) remaining, then the node must choose to evict a -`Guaranteed` pod in order to preserve node stability, and to limit the impact -of the unexpected consumption to other `Guaranteed` pod(s). - -Local disk is a `BestEffort` resource. If necessary, `kubelet` will evict pods one at a time to reclaim -disk when `DiskPressure` is encountered. The `kubelet` will rank pods by quality of service. If the `kubelet` -is responding to `inode` starvation, it will reclaim `inodes` by evicting pods with the lowest quality of service -first. If the `kubelet` is responding to lack of available disk, it will rank pods within a quality of service +* `Guaranteed` Pods are guaranteed only when requests and limits are specified +for all the containers and they are equal A `Guaranteed` Pod is guaranteed to +never be evicted because of another Pod's resource consumption. If a system +daemon (such as `kubelet`, `docker`, and `journald`) is consuming more resources +than were reserved via `system-reserved` or `kube-reserved` allocations, and the +node only has `Guaranteed` Pods remaining, then the node must choose to evict a +`Guaranteed` Pod in order to preserve node stability and to limit the impact +of the unexpected consumption to other `Guaranteed` Pods. + +If necessary, `kubelet` evicts Pods one at a time to reclaim disk when `DiskPressure` +is encountered. If the `kubelet` is responding to `inode` starvation, it reclaims +`inodes` by evicting Pods with the lowest quality of service first. If the `kubelet` +is responding to lack of available disk, it ranks Pods within a quality of service that consumes the largest amount of disk and kill those first. -#### With Imagefs +#### With `imagefs` -If `nodefs` is triggering evictions, `kubelet` will sort pods based on the usage on `nodefs` +If `nodefs` is triggering evictions, `kubelet` sorts Pods based on the usage on `nodefs` - local volumes + logs of all its containers. -If `imagefs` is triggering evictions, `kubelet` will sort pods based on the writable layer usage of all its containers. +If `imagefs` is triggering evictions, `kubelet` sorts Pods based on the writable layer usage of all its containers. -#### Without Imagefs +#### Without `imagefs` -If `nodefs` is triggering evictions, `kubelet` will sort pods based on their total disk usage +If `nodefs` is triggering evictions, `kubelet` sorts Pods based on their total disk usage - local volumes + logs & writable layer of all its containers. ### Minimum eviction reclaim -In certain scenarios, eviction of pods could result in reclamation of small amount of resources. This can result in +In certain scenarios, eviction of Pods could result in reclamation of small amount of resources. This can result in `kubelet` hitting eviction thresholds in repeated successions. In addition to that, eviction of resources like `disk`, is time consuming. To mitigate these issues, `kubelet` can have a per-resource `minimum-reclaim`. Whenever `kubelet` observes -resource pressure, `kubelet` will attempt to reclaim at least `minimum-reclaim` amount of resource below +resource pressure, `kubelet` attempts to reclaim at least `minimum-reclaim` amount of resource below the configured eviction threshold. For example, with the following configuration: @@ -262,31 +251,31 @@ For example, with the following configuration: --eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"` ``` -If an eviction threshold is triggered for `memory.available`, the `kubelet` will work to ensure -that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` will work -to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it will -work to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure +If an eviction threshold is triggered for `memory.available`, the `kubelet` works to ensure +that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` works +to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it +works to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure on their associated resources. The default `eviction-minimum-reclaim` is `0` for all resources. ### Scheduler -The node will report a condition when a compute resource is under pressure. The +The node reports a condition when a compute resource is under pressure. The scheduler views that condition as a signal to dissuade placing additional pods on the node. | Node Condition | Scheduler Behavior | | ---------------- | ------------------------------------------------ | -| `MemoryPressure` | No new `BestEffort` pods are scheduled to the node. | -| `DiskPressure` | No new pods are scheduled to the node. | +| `MemoryPressure` | No new `BestEffort` Pods are scheduled to the node. | +| `DiskPressure` | No new Pods are scheduled to the node. | ## Node OOM Behavior If the node experiences a system OOM (out of memory) event prior to the `kubelet` is able to reclaim memory, the node depends on the [oom_killer](https://lwn.net/Articles/391222/) to respond. -The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the pod. +The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the Pod. | Quality of Service | oom_score_adj | |----------------------------|-----------------------------------------------------------------------| @@ -294,7 +283,7 @@ The `kubelet` sets a `oom_score_adj` value for each container based on the quali | `BestEffort` | 1000 | | `Burstable` | min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999) | -If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` will calculate +If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` calculates an `oom_score` based on the percentage of memory it's using on the node, and then add the `oom_score_adj` to get an effective `oom_score` for the container, and then kills the container with the highest score. @@ -302,17 +291,19 @@ The intended behavior should be that containers with the lowest quality of servi are consuming the largest amount of memory relative to the scheduling request should be killed first in order to reclaim memory. -Unlike pod eviction, if a pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`. +Unlike Pod eviction, if a Pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`. ## Best Practices +The following sections describe best practices for out of resource handling. + ### Schedulable resources and eviction policies -Let's imagine the following scenario: +Consider the following scenario: * Node memory capacity: `10Gi` * Operator wants to reserve 10% of memory capacity for system daemons (kernel, `kubelet`, etc.) -* Operator wants to evict pods at 95% memory utilization to reduce thrashing and incidence of system OOM. +* Operator wants to evict Pods at 95% memory utilization to reduce thrashing and incidence of system OOM. To facilitate this scenario, the `kubelet` would be launched as follows: @@ -324,31 +315,30 @@ To facilitate this scenario, the `kubelet` would be launched as follows: Implicit in this configuration is the understanding that "System reserved" should include the amount of memory covered by the eviction threshold. -To reach that capacity, either some pod is using more than its request, or the system is using more than `500Mi`. +To reach that capacity, either some Pod is using more than its request, or the system is using more than `500Mi`. -This configuration will ensure that the scheduler does not place pods on a node that immediately induce memory pressure -and trigger eviction assuming those pods use less than their configured request. +This configuration ensures that the scheduler does not place Pods on a node that immediately induce memory pressure +and trigger eviction assuming those Pods use less than their configured request. ### DaemonSet -It is never desired for a `kubelet` to evict a pod that was derived from -a `DaemonSet` since the pod will immediately be recreated and rescheduled -back to the same node. +It is never desired for a `kubelet` to evict a `DaemonSet` Pod, since the Pod is +immediately recreated and rescheduled back to the same node. -At the moment, the `kubelet` has no ability to distinguish a pod created -from `DaemonSet` versus any other object. If/when that information is -available, the `kubelet` could pro-actively filter those pods from the -candidate set of pods provided to the eviction strategy. +At the moment, the `kubelet` has no ability to distinguish a Pod created +from `DaemonSet` versus any other object. If/when that information is +available, the `kubelet` could pro-actively filter those Pods from the +candidate set of Pods provided to the eviction strategy. In general, it is strongly recommended that `DaemonSet` not -create `BestEffort` pods to avoid being identified as a candidate pod -for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` pods. +create `BestEffort` Pods to avoid being identified as a candidate Pod +for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` Pods. ## Deprecation of existing feature flags to reclaim disk `kubelet` has been freeing up disk space on demand to keep the node stable. -As disk based eviction matures, the following `kubelet` flags will be marked for deprecation +As disk based eviction matures, the following `kubelet` flags are marked for deprecation in favor of the simpler configuration supported around eviction. | Existing Flag | New Flag | @@ -363,26 +353,27 @@ in favor of the simpler configuration supported around eviction. ## Known issues +The following sections describe known issues related to out of resource handling. + ### kubelet may not observe memory pressure right away -The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage +The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage increases within that window rapidly, the `kubelet` may not observe `MemoryPressure` fast enough, and the `OOMKiller` -will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this +will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this latency, and instead have the kernel tell us when a threshold has been crossed immediately. If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for -this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature +this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature to prevent system OOMs, and promote eviction of workloads so cluster state can rebalance. -### kubelet may evict more pods than needed +### kubelet may evict more Pods than needed -The pod eviction may evict more pods than needed due to stats collection timing gap. This can be mitigated by adding +The Pod eviction may evict more Pods than needed due to stats collection timing gap. This can be mitigated by adding the ability to get root container stats on an on-demand basis [(https://github.com/google/cadvisor/issues/1247)](https://github.com/google/cadvisor/issues/1247) in the future. -### How kubelet ranks pods for eviction in response to inode exhaustion +### How kubelet ranks Pods for eviction in response to inode exhaustion -At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes -inode exhaustion, it will evict pods by ranking them by quality of service. The following issue has been opened in cadvisor -to track per container inode consumption [(https://github.com/google/cadvisor/issues/1422)](https://github.com/google/cadvisor/issues/1422) which would allow us to rank pods -by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict -that pod over others. +At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes +inode exhaustion, it evicts Pods by ranking them by quality of service. The following issue has been opened in cadvisor +to track per container inode consumption [(https://github.com/google/cadvisor/issues/1422)](https://github.com/google/cadvisor/issues/1422) which would allow us to rank Pods +by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict that Pod over others.