Merge pull request #28407 from tengqm/fix-links
Fix some links in the concepts section
This commit is contained in:
@@ -159,11 +159,12 @@ You can run your own controller as a set of Pods,
|
||||
or externally to Kubernetes. What fits best will depend on what that particular
|
||||
controller does.
|
||||
|
||||
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Read about the [Kubernetes control plane](/docs/concepts/overview/components/#control-plane-components)
|
||||
* Discover some of the basic [Kubernetes objects](/docs/concepts/overview/working-with-objects/kubernetes-objects/)
|
||||
* Learn more about the [Kubernetes API](/docs/concepts/overview/kubernetes-api/)
|
||||
* If you want to write your own controller, see [Extension Patterns](/docs/concepts/extend-kubernetes/extend-cluster/#extension-patterns) in Extending Kubernetes.
|
||||
* If you want to write your own controller, see
|
||||
[Extension Patterns](/docs/concepts/extend-kubernetes/#extension-patterns)
|
||||
in Extending Kubernetes.
|
||||
|
||||
|
||||
@@ -33,8 +33,6 @@ the `--max-requests-inflight` flag without the API Priority and
|
||||
Fairness feature enabled.
|
||||
{{< /caution >}}
|
||||
|
||||
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Enabling/Disabling API Priority and Fairness
|
||||
@@ -65,6 +63,7 @@ The command-line flag `--enable-priority-and-fairness=false` will disable the
|
||||
API Priority and Fairness feature, even if other flags have enabled it.
|
||||
|
||||
## Concepts
|
||||
|
||||
There are several distinct features involved in the API Priority and Fairness
|
||||
feature. Incoming requests are classified by attributes of the request using
|
||||
_FlowSchemas_, and assigned to priority levels. Priority levels add a degree of
|
||||
@@ -75,12 +74,13 @@ each other, and allows for requests to be queued to prevent bursty traffic from
|
||||
causing failed requests when the average load is acceptably low.
|
||||
|
||||
### Priority Levels
|
||||
Without APF enabled, overall concurrency in
|
||||
the API server is limited by the `kube-apiserver` flags
|
||||
`--max-requests-inflight` and `--max-mutating-requests-inflight`. With APF
|
||||
enabled, the concurrency limits defined by these flags are summed and then the sum is divided up
|
||||
among a configurable set of _priority levels_. Each incoming request is assigned
|
||||
to a single priority level, and each priority level will only dispatch as many
|
||||
|
||||
Without APF enabled, overall concurrency in the API server is limited by the
|
||||
`kube-apiserver` flags `--max-requests-inflight` and
|
||||
`--max-mutating-requests-inflight`. With APF enabled, the concurrency limits
|
||||
defined by these flags are summed and then the sum is divided up among a
|
||||
configurable set of _priority levels_. Each incoming request is assigned to a
|
||||
single priority level, and each priority level will only dispatch as many
|
||||
concurrent requests as its configuration allows.
|
||||
|
||||
The default configuration, for example, includes separate priority levels for
|
||||
@@ -90,6 +90,7 @@ requests cannot prevent leader election or actions by the built-in controllers
|
||||
from succeeding.
|
||||
|
||||
### Queuing
|
||||
|
||||
Even within a priority level there may be a large number of distinct sources of
|
||||
traffic. In an overload situation, it is valuable to prevent one stream of
|
||||
requests from starving others (in particular, in the relatively common case of a
|
||||
@@ -114,15 +115,18 @@ independent flows will all make progress when total traffic exceeds capacity),
|
||||
tolerance for bursty traffic, and the added latency induced by queuing.
|
||||
|
||||
### Exempt requests
|
||||
|
||||
Some requests are considered sufficiently important that they are not subject to
|
||||
any of the limitations imposed by this feature. These exemptions prevent an
|
||||
improperly-configured flow control configuration from totally disabling an API
|
||||
server.
|
||||
|
||||
## Defaults
|
||||
|
||||
The Priority and Fairness feature ships with a suggested configuration that
|
||||
should suffice for experimentation; if your cluster is likely to
|
||||
experience heavy load then you should consider what configuration will work best. The suggested configuration groups requests into five priority
|
||||
experience heavy load then you should consider what configuration will work
|
||||
best. The suggested configuration groups requests into five priority
|
||||
classes:
|
||||
|
||||
* The `system` priority level is for requests from the `system:nodes` group,
|
||||
@@ -180,19 +184,18 @@ If you add the following additional FlowSchema, this exempts those
|
||||
requests from rate limiting.
|
||||
|
||||
{{< caution >}}
|
||||
|
||||
Making this change also allows any hostile party to then send
|
||||
health-check requests that match this FlowSchema, at any volume they
|
||||
like. If you have a web traffic filter or similar external security
|
||||
mechanism to protect your cluster's API server from general internet
|
||||
traffic, you can configure rules to block any health check requests
|
||||
that originate from outside your cluster.
|
||||
|
||||
{{< /caution >}}
|
||||
|
||||
{{< codenew file="priority-and-fairness/health-for-strangers.yaml" >}}
|
||||
|
||||
## Resources
|
||||
|
||||
The flow control API involves two kinds of resources.
|
||||
[PriorityLevelConfigurations](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#prioritylevelconfiguration-v1beta1-flowcontrol-apiserver-k8s-io)
|
||||
define the available isolation classes, the share of the available concurrency
|
||||
@@ -204,6 +207,7 @@ of the same API group, and it has the same Kinds with the same syntax and
|
||||
semantics.
|
||||
|
||||
### PriorityLevelConfiguration
|
||||
|
||||
A PriorityLevelConfiguration represents a single isolation class. Each
|
||||
PriorityLevelConfiguration has an independent limit on the number of outstanding
|
||||
requests, and limitations on the number of queued requests.
|
||||
@@ -217,6 +221,7 @@ server by restarting `kube-apiserver` with a different value for
|
||||
`--max-requests-inflight` (or `--max-mutating-requests-inflight`), and all
|
||||
PriorityLevelConfigurations will see their maximum allowed concurrency go up (or
|
||||
down) by the same fraction.
|
||||
|
||||
{{< caution >}}
|
||||
With the Priority and Fairness feature enabled, the total concurrency limit for
|
||||
the server is set to the sum of `--max-requests-inflight` and
|
||||
@@ -235,8 +240,8 @@ above the threshold will be queued, with the shuffle sharding and fair queuing t
|
||||
to balance progress between request flows.
|
||||
|
||||
The queuing configuration allows tuning the fair queuing algorithm for a
|
||||
priority level. Details of the algorithm can be read in the [enhancement
|
||||
proposal](#whats-next), but in short:
|
||||
priority level. Details of the algorithm can be read in the
|
||||
[enhancement proposal](#whats-next), but in short:
|
||||
|
||||
* Increasing `queues` reduces the rate of collisions between different flows, at
|
||||
the cost of increased memory usage. A value of 1 here effectively disables the
|
||||
@@ -249,15 +254,15 @@ proposal](#whats-next), but in short:
|
||||
* Changing `handSize` allows you to adjust the probability of collisions between
|
||||
different flows and the overall concurrency available to a single flow in an
|
||||
overload situation.
|
||||
{{< note >}}
|
||||
A larger `handSize` makes it less likely for two individual flows to collide
|
||||
(and therefore for one to be able to starve the other), but more likely that
|
||||
a small number of flows can dominate the apiserver. A larger `handSize` also
|
||||
potentially increases the amount of latency that a single high-traffic flow
|
||||
can cause. The maximum number of queued requests possible from a
|
||||
single flow is `handSize * queueLengthLimit`.
|
||||
{{< /note >}}
|
||||
|
||||
{{< note >}}
|
||||
A larger `handSize` makes it less likely for two individual flows to collide
|
||||
(and therefore for one to be able to starve the other), but more likely that
|
||||
a small number of flows can dominate the apiserver. A larger `handSize` also
|
||||
potentially increases the amount of latency that a single high-traffic flow
|
||||
can cause. The maximum number of queued requests possible from a
|
||||
single flow is `handSize * queueLengthLimit`.
|
||||
{{< /note >}}
|
||||
|
||||
Following is a table showing an interesting collection of shuffle
|
||||
sharding configurations, showing for each the probability that a
|
||||
@@ -319,6 +324,7 @@ considered part of a single flow. The correct choice for a given FlowSchema
|
||||
depends on the resource and your particular environment.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Every HTTP response from an API server with the priority and fairness feature
|
||||
enabled has two extra headers: `X-Kubernetes-PF-FlowSchema-UID` and
|
||||
`X-Kubernetes-PF-PriorityLevel-UID`, noting the flow schema that matched the request
|
||||
@@ -356,13 +362,14 @@ poorly-behaved workloads that may be harming system health.
|
||||
matched the request), `priority_level` (indicating the one to which
|
||||
the request was assigned), and `reason`. The `reason` label will be
|
||||
have one of the following values:
|
||||
* `queue-full`, indicating that too many requests were already
|
||||
queued,
|
||||
* `concurrency-limit`, indicating that the
|
||||
PriorityLevelConfiguration is configured to reject rather than
|
||||
queue excess requests, or
|
||||
* `time-out`, indicating that the request was still in the queue
|
||||
when its queuing time limit expired.
|
||||
|
||||
* `queue-full`, indicating that too many requests were already
|
||||
queued,
|
||||
* `concurrency-limit`, indicating that the
|
||||
PriorityLevelConfiguration is configured to reject rather than
|
||||
queue excess requests, or
|
||||
* `time-out`, indicating that the request was still in the queue
|
||||
when its queuing time limit expired.
|
||||
|
||||
* `apiserver_flowcontrol_dispatched_requests_total` is a counter
|
||||
vector (cumulative since server start) of requests that began
|
||||
@@ -430,14 +437,15 @@ poorly-behaved workloads that may be harming system health.
|
||||
sample to its histogram, reporting the length of the queue immediately
|
||||
after the request was added. Note that this produces different
|
||||
statistics than an unbiased survey would.
|
||||
{{< note >}}
|
||||
An outlier value in a histogram here means it is likely that a single flow
|
||||
(i.e., requests by one user or for one namespace, depending on
|
||||
configuration) is flooding the API server, and being throttled. By contrast,
|
||||
if one priority level's histogram shows that all queues for that priority
|
||||
level are longer than those for other priority levels, it may be appropriate
|
||||
to increase that PriorityLevelConfiguration's concurrency shares.
|
||||
{{< /note >}}
|
||||
|
||||
{{< note >}}
|
||||
An outlier value in a histogram here means it is likely that a single flow
|
||||
(i.e., requests by one user or for one namespace, depending on
|
||||
configuration) is flooding the API server, and being throttled. By contrast,
|
||||
if one priority level's histogram shows that all queues for that priority
|
||||
level are longer than those for other priority levels, it may be appropriate
|
||||
to increase that PriorityLevelConfiguration's concurrency shares.
|
||||
{{< /note >}}
|
||||
|
||||
* `apiserver_flowcontrol_request_concurrency_limit` is a gauge vector
|
||||
holding the computed concurrency limit (based on the API server's
|
||||
@@ -450,12 +458,13 @@ poorly-behaved workloads that may be harming system health.
|
||||
`priority_level` (indicating the one to which the request was
|
||||
assigned), and `execute` (indicating whether the request started
|
||||
executing).
|
||||
{{< note >}}
|
||||
Since each FlowSchema always assigns requests to a single
|
||||
PriorityLevelConfiguration, you can add the histograms for all the
|
||||
FlowSchemas for one priority level to get the effective histogram for
|
||||
requests assigned to that priority level.
|
||||
{{< /note >}}
|
||||
|
||||
{{< note >}}
|
||||
Since each FlowSchema always assigns requests to a single
|
||||
PriorityLevelConfiguration, you can add the histograms for all the
|
||||
FlowSchemas for one priority level to get the effective histogram for
|
||||
requests assigned to that priority level.
|
||||
{{< /note >}}
|
||||
|
||||
* `apiserver_flowcontrol_request_execution_seconds` is a histogram
|
||||
vector of how long requests took to actually execute, broken down by
|
||||
@@ -465,14 +474,19 @@ poorly-behaved workloads that may be harming system health.
|
||||
|
||||
### Debug endpoints
|
||||
|
||||
When you enable the API Priority and Fairness feature, the kube-apiserver serves the following additional paths at its HTTP[S] ports.
|
||||
When you enable the API Priority and Fairness feature, the `kube-apiserver`
|
||||
serves the following additional paths at its HTTP[S] ports.
|
||||
|
||||
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of
|
||||
all the priority levels and the current state of each. You can fetch like this:
|
||||
|
||||
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of all the priority levels and the current state of each. You can fetch like this:
|
||||
```shell
|
||||
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
|
||||
```
|
||||
|
||||
The output is similar to this:
|
||||
```
|
||||
|
||||
```none
|
||||
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests,
|
||||
workload-low, 0, true, false, 0, 0,
|
||||
global-default, 0, true, false, 0, 0,
|
||||
@@ -483,12 +497,16 @@ When you enable the API Priority and Fairness feature, the kube-apiserver serves
|
||||
workload-high, 0, true, false, 0, 0,
|
||||
```
|
||||
|
||||
- `/debug/api_priority_and_fairness/dump_queues` - a listing of all the queues and their current state. You can fetch like this:
|
||||
- `/debug/api_priority_and_fairness/dump_queues` - a listing of all the
|
||||
queues and their current state. You can fetch like this:
|
||||
|
||||
```shell
|
||||
kubectl get --raw /debug/api_priority_and_fairness/dump_queues
|
||||
```
|
||||
|
||||
The output is similar to this:
|
||||
```
|
||||
|
||||
```none
|
||||
PriorityLevelName, Index, PendingRequests, ExecutingRequests, VirtualStart,
|
||||
workload-high, 0, 0, 0, 0.0000,
|
||||
workload-high, 1, 0, 0, 0.0000,
|
||||
@@ -498,25 +516,33 @@ When you enable the API Priority and Fairness feature, the kube-apiserver serves
|
||||
leader-election, 15, 0, 0, 0.0000,
|
||||
```
|
||||
|
||||
- `/debug/api_priority_and_fairness/dump_requests` - a listing of all the requests that are currently waiting in a queue. You can fetch like this:
|
||||
- `/debug/api_priority_and_fairness/dump_requests` - a listing of all the requests
|
||||
that are currently waiting in a queue. You can fetch like this:
|
||||
|
||||
```shell
|
||||
kubectl get --raw /debug/api_priority_and_fairness/dump_requests
|
||||
```
|
||||
|
||||
The output is similar to this:
|
||||
```
|
||||
|
||||
```none
|
||||
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime,
|
||||
exempt, <none>, <none>, <none>, <none>, <none>,
|
||||
system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:26:57.179170694Z,
|
||||
```
|
||||
|
||||
In addition to the queued requests, the output includes one phantom line for each priority level that is exempt from limitation.
|
||||
In addition to the queued requests, the output includes one phantom line
|
||||
for each priority level that is exempt from limitation.
|
||||
|
||||
You can get a more detailed listing with a command like this:
|
||||
|
||||
```shell
|
||||
kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
|
||||
```
|
||||
|
||||
The output is similar to this:
|
||||
```
|
||||
|
||||
```none
|
||||
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, UserName, Verb, APIPath, Namespace, Name, APIVersion, Resource, SubResource,
|
||||
system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:31:03.583823404Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
|
||||
system, system-nodes, 12, 1, system:node:127.0.0.1, 2020-07-23T15:31:03.594555947Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
|
||||
@@ -528,4 +554,4 @@ When you enable the API Priority and Fairness feature, the kube-apiserver serves
|
||||
For background information on design details for API priority and fairness, see
|
||||
the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness).
|
||||
You can make suggestions and feature requests via [SIG API Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
|
||||
or the feature's [slack channel](http://kubernetes.slack.com/messages/api-priority-and-fairness).
|
||||
or the feature's [slack channel](https://kubernetes.slack.com/messages/api-priority-and-fairness).
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
---
|
||||
reviewers:
|
||||
title: Garbage collection for container images
|
||||
content_type: concept
|
||||
weight: 70
|
||||
@@ -7,12 +6,13 @@ weight: 70
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Garbage collection is a helpful function of kubelet that will clean up unused [images](/docs/concepts/containers/#container-images) and unused [containers](/docs/concepts/containers/). Kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.
|
||||
|
||||
External garbage collection tools are not recommended as these tools can potentially break the behavior of kubelet by removing containers expected to exist.
|
||||
|
||||
|
||||
Garbage collection is a helpful function of kubelet that will clean up unused
|
||||
[images](/docs/concepts/containers/#container-images) and unused
|
||||
[containers](/docs/concepts/containers/). Kubelet will perform garbage collection
|
||||
for containers every minute and garbage collection for images every five minutes.
|
||||
|
||||
External garbage collection tools are not recommended as these tools can potentially
|
||||
break the behavior of kubelet by removing containers expected to exist.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
@@ -28,10 +28,24 @@ threshold has been met.
|
||||
|
||||
## Container Collection
|
||||
|
||||
The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers every single
|
||||
pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting `MinAge` to zero and setting `MaxPerPodContainer` and `MaxContainers` respectively to less than zero.
|
||||
The policy for garbage collecting containers considers three user-defined variables.
|
||||
`MinAge` is the minimum age at which a container can be garbage collected.
|
||||
`MaxPerPodContainer` is the maximum number of dead containers every single
|
||||
pod (UID, container name) pair is allowed to have.
|
||||
`MaxContainers` is the maximum number of total dead containers.
|
||||
These variables can be individually disabled by setting `MinAge` to zero and
|
||||
setting `MaxPerPodContainer` and `MaxContainers` respectively to less than zero.
|
||||
|
||||
Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. `MaxPerPodContainer` and `MaxContainer` may potentially conflict with each other in situations where retaining the maximum number of containers per pod (`MaxPerPodContainer`) would go outside the allowable range of global dead containers (`MaxContainers`). `MaxPerPodContainer` would be adjusted in this situation: A worst case scenario would be to downgrade `MaxPerPodContainer` to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`.
|
||||
Kubelet will act on containers that are unidentified, deleted, or outside of
|
||||
the boundaries set by the previously mentioned flags. The oldest containers
|
||||
will generally be removed first. `MaxPerPodContainer` and `MaxContainer` may
|
||||
potentially conflict with each other in situations where retaining the maximum
|
||||
number of containers per pod (`MaxPerPodContainer`) would go outside the
|
||||
allowable range of global dead containers (`MaxContainers`).
|
||||
`MaxPerPodContainer` would be adjusted in this situation: A worst case
|
||||
scenario would be to downgrade `MaxPerPodContainer` to 1 and evict the oldest
|
||||
containers. Additionally, containers owned by pods that have been deleted are
|
||||
removed once they are older than `MinAge`.
|
||||
|
||||
Containers that are not managed by kubelet are not subject to container garbage collection.
|
||||
|
||||
@@ -40,18 +54,18 @@ Containers that are not managed by kubelet are not subject to container garbage
|
||||
You can adjust the following thresholds to tune image garbage collection with the following kubelet flags :
|
||||
|
||||
1. `image-gc-high-threshold`, the percent of disk usage which triggers image garbage collection.
|
||||
Default is 85%.
|
||||
Default is 85%.
|
||||
2. `image-gc-low-threshold`, the percent of disk usage to which image garbage collection attempts
|
||||
to free. Default is 80%.
|
||||
to free. Default is 80%.
|
||||
|
||||
You can customize the garbage collection policy through the following kubelet flags:
|
||||
|
||||
1. `minimum-container-ttl-duration`, minimum age for a finished container before it is
|
||||
garbage collected. Default is 0 minute, which means every finished container will be garbage collected.
|
||||
garbage collected. Default is 0 minute, which means every finished container will be garbage collected.
|
||||
2. `maximum-dead-containers-per-container`, maximum number of old instances to be retained
|
||||
per container. Default is 1.
|
||||
per container. Default is 1.
|
||||
3. `maximum-dead-containers`, maximum number of old instances of containers to retain globally.
|
||||
Default is -1, which means there is no global limit.
|
||||
Default is -1, which means there is no global limit.
|
||||
|
||||
Containers can potentially be garbage collected before their usefulness has expired. These containers
|
||||
can contain logs and other data that can be useful for troubleshooting. A sufficiently large value for
|
||||
@@ -77,10 +91,8 @@ Including:
|
||||
| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources |
|
||||
| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources |
|
||||
|
||||
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
|
||||
See [Configuring Out Of Resource Handling](/docs/tasks/administer-cluster/out-of-resource/) for more details.
|
||||
See [Configuring Out Of Resource Handling](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
|
||||
for more details.
|
||||
|
||||
|
||||
@@ -10,7 +10,8 @@ weight: 40
|
||||
|
||||
{{< feature-state for_k8s_version="v1.20" state="stable" >}}
|
||||
|
||||
Kubernetes allow you to limit the number of process IDs (PIDs) that a {{< glossary_tooltip term_id="Pod" text="Pod" >}} can use.
|
||||
Kubernetes allow you to limit the number of process IDs (PIDs) that a
|
||||
{{< glossary_tooltip term_id="Pod" text="Pod" >}} can use.
|
||||
You can also reserve a number of allocatable PIDs for each {{< glossary_tooltip term_id="node" text="node" >}}
|
||||
for use by the operating system and daemons (rather than by Pods).
|
||||
|
||||
@@ -84,7 +85,9 @@ gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
||||
Kubernetes allows you to limit the number of processes running in a Pod. You
|
||||
specify this limit at the node level, rather than configuring it as a resource
|
||||
limit for a particular Pod. Each Node can have a different PID limit.
|
||||
To configure the limit, you can specify the command line parameter `--pod-max-pids` to the kubelet, or set `PodPidsLimit` in the kubelet [configuration file](/docs/tasks/administer-cluster/kubelet-config-file/).
|
||||
To configure the limit, you can specify the command line parameter `--pod-max-pids`
|
||||
to the kubelet, or set `PodPidsLimit` in the kubelet
|
||||
[configuration file](/docs/tasks/administer-cluster/kubelet-config-file/).
|
||||
|
||||
{{< note >}}
|
||||
Before Kubernetes version 1.20, PID resource limiting for Pods required enabling
|
||||
@@ -95,9 +98,12 @@ the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
||||
## PID based eviction
|
||||
|
||||
You can configure kubelet to start terminating a Pod when it is misbehaving and consuming abnormal amount of resources.
|
||||
This feature is called eviction. You can [Configure Out of Resource Handling](/docs/tasks/administer-cluster/out-of-resource) for various eviction signals.
|
||||
This feature is called eviction. You can
|
||||
[Configure Out of Resource Handling](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
|
||||
for various eviction signals.
|
||||
Use `pid.available` eviction signal to configure the threshold for number of PIDs used by Pod.
|
||||
You can set soft and hard eviction policies. However, even with the hard eviction policy, if the number of PIDs growing very fast,
|
||||
You can set soft and hard eviction policies.
|
||||
However, even with the hard eviction policy, if the number of PIDs growing very fast,
|
||||
node can still get into unstable state by hitting the node PIDs limit.
|
||||
Eviction signal value is calculated periodically and does NOT enforce the limit.
|
||||
|
||||
@@ -112,6 +118,7 @@ when one Pod is misbehaving.
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
- Refer to the [PID Limiting enhancement document](https://github.com/kubernetes/enhancements/blob/097b4d8276bc9564e56adf72505d43ce9bc5e9e8/keps/sig-node/20190129-pid-limiting.md) for more information.
|
||||
- For historical context, read [Process ID Limiting for Stability Improvements in Kubernetes 1.14](/blog/2019/04/15/process-id-limiting-for-stability-improvements-in-kubernetes-1.14/).
|
||||
- For historical context, read
|
||||
[Process ID Limiting for Stability Improvements in Kubernetes 1.14](/blog/2019/04/15/process-id-limiting-for-stability-improvements-in-kubernetes-1.14/).
|
||||
- Read [Managing Resources for Containers](/docs/concepts/configuration/manage-resources-containers/).
|
||||
- Learn how to [Configure Out of Resource Handling](/docs/tasks/administer-cluster/out-of-resource).
|
||||
- Learn how to [Configure Out of Resource Handling](/docs/concepts/scheduling-eviction/node-pressure-eviction/).
|
||||
|
||||
@@ -57,8 +57,9 @@ Neither contention nor changes to quota will affect already created resources.
|
||||
|
||||
## Enabling Resource Quota
|
||||
|
||||
Resource Quota support is enabled by default for many Kubernetes distributions. It is
|
||||
enabled when the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} `--enable-admission-plugins=` flag has `ResourceQuota` as
|
||||
Resource Quota support is enabled by default for many Kubernetes distributions. It is
|
||||
enabled when the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}}
|
||||
`--enable-admission-plugins=` flag has `ResourceQuota` as
|
||||
one of its arguments.
|
||||
|
||||
A resource quota is enforced in a particular namespace when there is a
|
||||
@@ -66,7 +67,9 @@ ResourceQuota in that namespace.
|
||||
|
||||
## Compute Resource Quota
|
||||
|
||||
You can limit the total sum of [compute resources](/docs/concepts/configuration/manage-resources-containers/) that can be requested in a given namespace.
|
||||
You can limit the total sum of
|
||||
[compute resources](/docs/concepts/configuration/manage-resources-containers/)
|
||||
that can be requested in a given namespace.
|
||||
|
||||
The following resource types are supported:
|
||||
|
||||
@@ -125,7 +128,9 @@ In release 1.8, quota support for local ephemeral storage is added as an alpha f
|
||||
| `ephemeral-storage` | Same as `requests.ephemeral-storage`. |
|
||||
|
||||
{{< note >}}
|
||||
When using a CRI container runtime, container logs will count against the ephemeral storage quota. This can result in the unexpected eviction of pods that have exhausted their storage quotas. Refer to [Logging Architecture](/docs/concepts/cluster-administration/logging/) for details.
|
||||
When using a CRI container runtime, container logs will count against the ephemeral storage quota.
|
||||
This can result in the unexpected eviction of pods that have exhausted their storage quotas.
|
||||
Refer to [Logging Architecture](/docs/concepts/cluster-administration/logging/) for details.
|
||||
{{< /note >}}
|
||||
|
||||
## Object Count Quota
|
||||
@@ -192,7 +197,7 @@ Resources specified on the quota outside of the allowed set results in a validat
|
||||
| `NotTerminating` | Match pods where `.spec.activeDeadlineSeconds is nil` |
|
||||
| `BestEffort` | Match pods that have best effort quality of service. |
|
||||
| `NotBestEffort` | Match pods that do not have best effort quality of service. |
|
||||
| `PriorityClass` | Match pods that references the specified [priority class](/docs/concepts/configuration/pod-priority-preemption). |
|
||||
| `PriorityClass` | Match pods that references the specified [priority class](/docs/concepts/scheduling-eviction/pod-priority-preemption). |
|
||||
| `CrossNamespacePodAffinity` | Match pods that have cross-namespace pod [(anti)affinity terms](/docs/concepts/scheduling-eviction/assign-pod-node). |
|
||||
|
||||
The `BestEffort` scope restricts a quota to tracking the following resource:
|
||||
@@ -248,13 +253,14 @@ specified.
|
||||
|
||||
{{< feature-state for_k8s_version="v1.17" state="stable" >}}
|
||||
|
||||
Pods can be created at a specific [priority](/docs/concepts/configuration/pod-priority-preemption/#pod-priority).
|
||||
Pods can be created at a specific [priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/#pod-priority).
|
||||
You can control a pod's consumption of system resources based on a pod's priority, by using the `scopeSelector`
|
||||
field in the quota spec.
|
||||
|
||||
A quota is matched and consumed only if `scopeSelector` in the quota spec selects the pod.
|
||||
|
||||
When quota is scoped for priority class using `scopeSelector` field, quota object is restricted to track only following resources:
|
||||
When quota is scoped for priority class using `scopeSelector` field, quota object
|
||||
is restricted to track only following resources:
|
||||
|
||||
* `pods`
|
||||
* `cpu`
|
||||
@@ -554,7 +560,7 @@ kubectl create -f ./object-counts.yaml --namespace=myspace
|
||||
kubectl get quota --namespace=myspace
|
||||
```
|
||||
|
||||
```
|
||||
```none
|
||||
NAME AGE
|
||||
compute-resources 30s
|
||||
object-counts 32s
|
||||
@@ -564,7 +570,7 @@ object-counts 32s
|
||||
kubectl describe quota compute-resources --namespace=myspace
|
||||
```
|
||||
|
||||
```
|
||||
```none
|
||||
Name: compute-resources
|
||||
Namespace: myspace
|
||||
Resource Used Hard
|
||||
@@ -580,7 +586,7 @@ requests.nvidia.com/gpu 0 4
|
||||
kubectl describe quota object-counts --namespace=myspace
|
||||
```
|
||||
|
||||
```
|
||||
```none
|
||||
Name: object-counts
|
||||
Namespace: myspace
|
||||
Resource Used Hard
|
||||
@@ -677,10 +683,10 @@ Then, create a resource quota object in the `kube-system` namespace:
|
||||
{{< codenew file="policy/priority-class-resourcequota.yaml" >}}
|
||||
|
||||
```shell
|
||||
$ kubectl apply -f https://k8s.io/examples/policy/priority-class-resourcequota.yaml -n kube-system
|
||||
kubectl apply -f https://k8s.io/examples/policy/priority-class-resourcequota.yaml -n kube-system
|
||||
```
|
||||
|
||||
```
|
||||
```none
|
||||
resourcequota/pods-cluster-services created
|
||||
```
|
||||
|
||||
|
||||
@@ -214,7 +214,7 @@ signal below the threshold, the kubelet begins to evict end-user pods.
|
||||
The kubelet uses the following parameters to determine pod eviction order:
|
||||
|
||||
1. Whether the pod's resource usage exceeds requests
|
||||
1. [Pod Priority](/docs/concepts/configuration/pod-priority-preemption/)
|
||||
1. [Pod Priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/)
|
||||
1. The pod's resource usage relative to requests
|
||||
|
||||
As a result, kubelet ranks and evicts pods in the following order:
|
||||
|
||||
@@ -285,7 +285,7 @@ arbitrary tolerations to DaemonSets.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Read about [out of resource handling](/docs/tasks/administer-cluster/out-of-resource/) and how you can configure it
|
||||
* Read about [pod priority](/docs/concepts/configuration/pod-priority-preemption/)
|
||||
* Read about [out of resource handling](/docs/concepts/scheduling-eviction/out-of-resource/) and how you can configure it
|
||||
* Read about [pod priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/)
|
||||
|
||||
|
||||
|
||||
@@ -249,5 +249,4 @@ implementation in `kube-proxy`.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Learn about [Enabling EndpointSlices](/docs/tasks/administer-cluster/enabling-endpointslices)
|
||||
* Read [Connecting Applications with Services](/docs/concepts/services-networking/connect-applications-service/)
|
||||
|
||||
@@ -215,7 +215,7 @@ each Service port. The value of this field is mirrored by the corresponding
|
||||
Endpoints and EndpointSlice objects.
|
||||
|
||||
This field follows standard Kubernetes label syntax. Values should either be
|
||||
[IANA standard service names](http://www.iana.org/assignments/service-names) or
|
||||
[IANA standard service names](https://www.iana.org/assignments/service-names) or
|
||||
domain prefixed names such as `mycompany.com/my-custom-protocol`.
|
||||
|
||||
## Virtual IPs and service proxies
|
||||
|
||||
@@ -32,7 +32,8 @@ different flags and/or different memory and cpu requests for different hardware
|
||||
|
||||
### Create a DaemonSet
|
||||
|
||||
You can describe a DaemonSet in a YAML file. For example, the `daemonset.yaml` file below describes a DaemonSet that runs the fluentd-elasticsearch Docker image:
|
||||
You can describe a DaemonSet in a YAML file. For example, the `daemonset.yaml` file below
|
||||
describes a DaemonSet that runs the fluentd-elasticsearch Docker image:
|
||||
|
||||
{{< codenew file="controllers/daemonset.yaml" >}}
|
||||
|
||||
@@ -46,19 +47,23 @@ kubectl apply -f https://k8s.io/examples/controllers/daemonset.yaml
|
||||
|
||||
As with all other Kubernetes config, a DaemonSet needs `apiVersion`, `kind`, and `metadata` fields. For
|
||||
general information about working with config files, see
|
||||
[running stateless applications](/docs/tasks/run-application/run-stateless-application-deployment/),
|
||||
[configuring containers](/docs/tasks/), and [object management using kubectl](/docs/concepts/overview/working-with-objects/object-management/) documents.
|
||||
[running stateless applications](/docs/tasks/run-application/run-stateless-application-deployment/)
|
||||
and [object management using kubectl](/docs/concepts/overview/working-with-objects/object-management/).
|
||||
|
||||
The name of a DaemonSet object must be a valid
|
||||
[DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names).
|
||||
|
||||
A DaemonSet also needs a [`.spec`](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status) section.
|
||||
A DaemonSet also needs a
|
||||
[`.spec`](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status)
|
||||
section.
|
||||
|
||||
### Pod Template
|
||||
|
||||
The `.spec.template` is one of the required fields in `.spec`.
|
||||
|
||||
The `.spec.template` is a [pod template](/docs/concepts/workloads/pods/#pod-templates). It has exactly the same schema as a {{< glossary_tooltip text="Pod" term_id="pod" >}}, except it is nested and does not have an `apiVersion` or `kind`.
|
||||
The `.spec.template` is a [pod template](/docs/concepts/workloads/pods/#pod-templates).
|
||||
It has exactly the same schema as a {{< glossary_tooltip text="Pod" term_id="pod" >}},
|
||||
except it is nested and does not have an `apiVersion` or `kind`.
|
||||
|
||||
In addition to required fields for a Pod, a Pod template in a DaemonSet has to specify appropriate
|
||||
labels (see [pod selector](#pod-selector)).
|
||||
@@ -79,20 +84,23 @@ unintentional orphaning of Pods, and it was found to be confusing to users.
|
||||
|
||||
The `.spec.selector` is an object consisting of two fields:
|
||||
|
||||
* `matchLabels` - works the same as the `.spec.selector` of a [ReplicationController](/docs/concepts/workloads/controllers/replicationcontroller/).
|
||||
* `matchLabels` - works the same as the `.spec.selector` of a
|
||||
[ReplicationController](/docs/concepts/workloads/controllers/replicationcontroller/).
|
||||
* `matchExpressions` - allows to build more sophisticated selectors by specifying key,
|
||||
list of values and an operator that relates the key and values.
|
||||
|
||||
When the two are specified the result is ANDed.
|
||||
|
||||
If the `.spec.selector` is specified, it must match the `.spec.template.metadata.labels`. Config with these not matching will be rejected by the API.
|
||||
If the `.spec.selector` is specified, it must match the `.spec.template.metadata.labels`.
|
||||
Config with these not matching will be rejected by the API.
|
||||
|
||||
### Running Pods on select Nodes
|
||||
|
||||
If you specify a `.spec.template.spec.nodeSelector`, then the DaemonSet controller will
|
||||
create Pods on nodes which match that [node
|
||||
selector](/docs/concepts/scheduling-eviction/assign-pod-node/). Likewise if you specify a `.spec.template.spec.affinity`,
|
||||
then DaemonSet controller will create Pods on nodes which match that [node affinity](/docs/concepts/scheduling-eviction/assign-pod-node/).
|
||||
create Pods on nodes which match that [node selector](/docs/concepts/scheduling-eviction/assign-pod-node/).
|
||||
Likewise if you specify a `.spec.template.spec.affinity`,
|
||||
then DaemonSet controller will create Pods on nodes which match that
|
||||
[node affinity](/docs/concepts/scheduling-eviction/assign-pod-node/).
|
||||
If you do not specify either, then the DaemonSet controller will create Pods on all nodes.
|
||||
|
||||
## How Daemon Pods are scheduled
|
||||
@@ -106,18 +114,19 @@ node that a Pod runs on is selected by the Kubernetes scheduler. However,
|
||||
DaemonSet pods are created and scheduled by the DaemonSet controller instead.
|
||||
That introduces the following issues:
|
||||
|
||||
* Inconsistent Pod behavior: Normal Pods waiting to be scheduled are created
|
||||
and in `Pending` state, but DaemonSet pods are not created in `Pending`
|
||||
state. This is confusing to the user.
|
||||
* [Pod preemption](/docs/concepts/configuration/pod-priority-preemption/)
|
||||
is handled by default scheduler. When preemption is enabled, the DaemonSet controller
|
||||
will make scheduling decisions without considering pod priority and preemption.
|
||||
* Inconsistent Pod behavior: Normal Pods waiting to be scheduled are created
|
||||
and in `Pending` state, but DaemonSet pods are not created in `Pending`
|
||||
state. This is confusing to the user.
|
||||
* [Pod preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/)
|
||||
is handled by default scheduler. When preemption is enabled, the DaemonSet controller
|
||||
will make scheduling decisions without considering pod priority and preemption.
|
||||
|
||||
`ScheduleDaemonSetPods` allows you to schedule DaemonSets using the default
|
||||
scheduler instead of the DaemonSet controller, by adding the `NodeAffinity` term
|
||||
to the DaemonSet pods, instead of the `.spec.nodeName` term. The default
|
||||
scheduler is then used to bind the pod to the target host. If node affinity of
|
||||
the DaemonSet pod already exists, it is replaced (the original node affinity was taken into account before selecting the target host). The DaemonSet controller only
|
||||
the DaemonSet pod already exists, it is replaced (the original node affinity was
|
||||
taken into account before selecting the target host). The DaemonSet controller only
|
||||
performs these operations when creating or modifying DaemonSet pods, and no
|
||||
changes are made to the `spec.template` of the DaemonSet.
|
||||
|
||||
@@ -158,10 +167,12 @@ Some possible patterns for communicating with Pods in a DaemonSet are:
|
||||
|
||||
- **Push**: Pods in the DaemonSet are configured to send updates to another service, such
|
||||
as a stats database. They do not have clients.
|
||||
- **NodeIP and Known Port**: Pods in the DaemonSet can use a `hostPort`, so that the pods are reachable via the node IPs. Clients know the list of node IPs somehow, and know the port by convention.
|
||||
- **DNS**: Create a [headless service](/docs/concepts/services-networking/service/#headless-services) with the same pod selector,
|
||||
and then discover DaemonSets using the `endpoints` resource or retrieve multiple A records from
|
||||
DNS.
|
||||
- **NodeIP and Known Port**: Pods in the DaemonSet can use a `hostPort`, so that the pods
|
||||
are reachable via the node IPs.
|
||||
Clients know the list of node IPs somehow, and know the port by convention.
|
||||
- **DNS**: Create a [headless service](/docs/concepts/services-networking/service/#headless-services)
|
||||
with the same pod selector, and then discover DaemonSets using the `endpoints`
|
||||
resource or retrieve multiple A records from DNS.
|
||||
- **Service**: Create a service with the same Pod selector, and use the service to reach a
|
||||
daemon on a random node. (No way to reach specific node.)
|
||||
|
||||
|
||||
@@ -86,7 +86,7 @@ rolling out node software updates can cause voluntary disruptions. Also, some im
|
||||
of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes.
|
||||
Your cluster administrator or hosting provider should have documented what level of voluntary
|
||||
disruptions, if any, to expect. Certain configuration options, such as
|
||||
[using PriorityClasses](/docs/concepts/configuration/pod-priority-preemption/)
|
||||
[using PriorityClasses](/docs/concepts/scheduling-eviction/pod-priority-preemption/)
|
||||
in your pod spec can also cause voluntary (and involuntary) disruptions.
|
||||
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ You can use _topology spread constraints_ to control how {{< glossary_tooltip te
|
||||
In versions of Kubernetes before v1.18, you must enable the `EvenPodsSpread`
|
||||
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) on
|
||||
the [API server](/docs/concepts/overview/components/#kube-apiserver) and the
|
||||
[scheduler](/docs/reference/generated/kube-scheduler/) in order to use Pod
|
||||
[scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/) in order to use Pod
|
||||
topology spread constraints.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user