Merge remote-tracking branch 'upstream/main' into dev-1.23
This commit is contained in:
Jesse Butler
@@ -402,7 +402,7 @@ Graceful node shutdown is controlled with the `GracefulNodeShutdown`
|
||||
enabled by default in 1.21.
|
||||
|
||||
Note that by default, both configuration options described below,
|
||||
`ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero,
|
||||
`shutdownGracePeriod` and `shutdownGracePeriodCriticalPods` are set to zero,
|
||||
thus not activating Graceful node shutdown functionality.
|
||||
To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values.
|
||||
|
||||
@@ -412,13 +412,13 @@ During a graceful shutdown, kubelet terminates pods in two phases:
|
||||
2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) running on the node.
|
||||
|
||||
Graceful node shutdown feature is configured with two [`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options:
|
||||
* `ShutdownGracePeriod`:
|
||||
* `shutdownGracePeriod`:
|
||||
* Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
|
||||
* `ShutdownGracePeriodCriticalPods`:
|
||||
* Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `ShutdownGracePeriod`.
|
||||
* `shutdownGracePeriodCriticalPods`:
|
||||
* Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `shutdownGracePeriod`.
|
||||
|
||||
For example, if `ShutdownGracePeriod=30s`, and
|
||||
`ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
|
||||
For example, if `shutdownGracePeriod=30s`, and
|
||||
`shutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
|
||||
30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved
|
||||
for gracefully terminating normal pods, and the last 10 seconds would be
|
||||
reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
|
||||
@@ -535,6 +535,11 @@ the kubelet, and the `--fail-swap-on` command line flag or `failSwapOn`
|
||||
[configuration setting](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
|
||||
must be set to false.
|
||||
|
||||
{{< warning >}}
|
||||
When the memory swap feature is turned on, Kubernetes data such as the content
|
||||
of Secret objects that were written to tmpfs now could be swapped to disk.
|
||||
{{< /warning >}}
|
||||
|
||||
A user can also optionally configure `memorySwap.swapBehavior` in order to
|
||||
specify how a node will use swap memory. For example,
|
||||
|
||||
|
||||
@@ -45,6 +45,11 @@ This page lists some of the available add-ons and links to their respective inst
|
||||
## Infrastructure
|
||||
|
||||
* [KubeVirt](https://kubevirt.io/user-guide/#/installation/installation) is an add-on to run virtual machines on Kubernetes. Usually run on bare-metal clusters.
|
||||
* The
|
||||
[node problem detector](https://github.com/kubernetes/node-problem-detector)
|
||||
runs on Linux nodes and reports system issues as either
|
||||
[Events](/docs/reference/kubernetes-api/cluster-resources/event-v1/) or
|
||||
[Node conditions](/docs/concepts/architecture/nodes/#condition).
|
||||
|
||||
## Legacy Add-ons
|
||||
|
||||
|
||||
@@ -26,6 +26,10 @@ fair queuing technique so that, for example, a poorly-behaved
|
||||
{{< glossary_tooltip text="controller" term_id="controller" >}} need not
|
||||
starve others (even at the same priority level).
|
||||
|
||||
This feature is designed to work well with standard controllers, which
|
||||
use informers and react to failures of API requests with exponential
|
||||
back-off, and other clients that also work this way.
|
||||
|
||||
{{< caution >}}
|
||||
Requests classified as "long-running" — primarily watches — are not
|
||||
subject to the API Priority and Fairness filter. This is also true for
|
||||
@@ -102,6 +106,8 @@ name of the matching FlowSchema plus a _flow distinguisher_ — which
|
||||
is either the requesting user, the target resource's namespace, or nothing — and the
|
||||
system attempts to give approximately equal weight to requests in different
|
||||
flows of the same priority level.
|
||||
To enable distinct handling of distinct instances, controllers that have
|
||||
many instances should authenticate with distinct usernames
|
||||
|
||||
After classifying a request into a flow, the API Priority and Fairness
|
||||
feature then may assign the request to a queue. This assignment uses
|
||||
|
||||
@@ -169,49 +169,6 @@ With this toolset DANM is able to provide multiple separated network interfaces,
|
||||
network that satisfies the Kubernetes requirements. Many
|
||||
people have reported success with Flannel and Kubernetes.
|
||||
|
||||
### Google Compute Engine (GCE)
|
||||
|
||||
For the Google Compute Engine cluster configuration scripts, [advanced
|
||||
routing](https://cloud.google.com/vpc/docs/routes) is used to
|
||||
assign each VM a subnet (default is `/24` - 254 IPs). Any traffic bound for that
|
||||
subnet will be routed directly to the VM by the GCE network fabric. This is in
|
||||
addition to the "main" IP address assigned to the VM, which is NAT'ed for
|
||||
outbound internet access. A linux bridge (called `cbr0`) is configured to exist
|
||||
on that subnet, and is passed to docker's `--bridge` flag.
|
||||
|
||||
Docker is started with:
|
||||
|
||||
```shell
|
||||
DOCKER_OPTS="--bridge=cbr0 --iptables=false --ip-masq=false"
|
||||
```
|
||||
|
||||
This bridge is created by Kubelet (controlled by the `--network-plugin=kubenet`
|
||||
flag) according to the `Node`'s `.spec.podCIDR`.
|
||||
|
||||
Docker will now allocate IPs from the `cbr-cidr` block. Containers can reach
|
||||
each other and `Nodes` over the `cbr0` bridge. Those IPs are all routable
|
||||
within the GCE project network.
|
||||
|
||||
GCE itself does not know anything about these IPs, though, so it will not NAT
|
||||
them for outbound internet traffic. To achieve that an iptables rule is used
|
||||
to masquerade (aka SNAT - to make it seem as if packets came from the `Node`
|
||||
itself) traffic that is bound for IPs outside the GCE project network
|
||||
(10.0.0.0/8).
|
||||
|
||||
```shell
|
||||
iptables -t nat -A POSTROUTING ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE
|
||||
```
|
||||
|
||||
Lastly IP forwarding is enabled in the kernel (so the kernel will process
|
||||
packets for bridged containers):
|
||||
|
||||
```shell
|
||||
sysctl net.ipv4.ip_forward=1
|
||||
```
|
||||
|
||||
The result of all this is that all `Pods` can reach each other and can egress
|
||||
traffic to the internet.
|
||||
|
||||
### Jaguar
|
||||
|
||||
[Jaguar](https://gitlab.com/sdnlab/jaguar) is an open source solution for Kubernetes's network based on OpenDaylight. Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.
|
||||
|
||||
@@ -116,11 +116,11 @@ CPU is always requested as an absolute quantity, never as a relative quantity;
|
||||
|
||||
Limits and requests for `memory` are measured in bytes. You can express memory as
|
||||
a plain integer or as a fixed-point number using one of these suffixes:
|
||||
E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
|
||||
E, P, T, G, M, k, m (millis). You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
|
||||
Mi, Ki. For example, the following represent roughly the same value:
|
||||
|
||||
```shell
|
||||
128974848, 129e6, 129M, 123Mi
|
||||
128974848, 129e6, 129M, 128974848000m, 123Mi
|
||||
```
|
||||
|
||||
Here's an example.
|
||||
|
||||
@@ -57,12 +57,11 @@ Kubernetes as a project supports and maintains [AWS](https://github.com/kubernet
|
||||
|
||||
## Using multiple Ingress controllers
|
||||
|
||||
You may deploy [any number of ingress controllers](https://git.k8s.io/ingress-nginx/docs/user-guide/multiple-ingress.md#multiple-ingress-controllers)
|
||||
within a cluster. When you create an ingress, you should annotate each ingress with the appropriate
|
||||
[`ingress.class`](https://git.k8s.io/ingress-gce/docs/faq/README.md#how-do-i-run-multiple-ingress-controllers-in-the-same-cluster)
|
||||
to indicate which ingress controller should be used if more than one exists within your cluster.
|
||||
You may deploy any number of ingress controllers using [ingress class](/docs/concepts/services-networking/ingress/#ingress-class)
|
||||
within a cluster. Note the `.metadata.name` of your ingress class resource. When you create an ingress you would need that name to specify the `ingressClassName` field on your Ingress object (refer to [IngressSpec v1 reference](/docs/reference/kubernetes-api/service-resources/ingress-v1/#IngressSpec). `ingressClassName` is a replacement of the older [annotation method](/docs/concepts/services-networking/ingress/#deprecated-annotation).
|
||||
|
||||
If you do not define a class, your cloud provider may use a default ingress controller.
|
||||
If you do not specify an IngressClass for an Ingress, and your cluster has exactly one IngressClass marked as default, then Kubernetes [applies](/docs/concepts/services-networking/ingress/#default-ingress-class) the cluster's default IngressClass to the Ingress.
|
||||
You mark an IngressClass as default by setting the [`ingressclass.kubernetes.io/is-default-class` annotation](/docs/reference/labels-annotations-taints/#ingressclass-kubernetes-io-is-default-class) on that IngressClass, with the string value `"true"`.
|
||||
|
||||
Ideally, all ingress controllers should fulfill this specification, but the various ingress
|
||||
controllers operate slightly differently.
|
||||
|
||||
@@ -7,7 +7,7 @@ reviewers:
|
||||
- pohly
|
||||
title: Ephemeral Volumes
|
||||
content_type: concept
|
||||
weight: 50
|
||||
weight: 30
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
@@ -10,14 +10,13 @@ feature:
|
||||
title: Storage orchestration
|
||||
description: >
|
||||
Automatically mount the storage system of your choice, whether from local storage, a public cloud provider such as <a href="https://cloud.google.com/storage/">GCP</a> or <a href="https://aws.amazon.com/products/storage/">AWS</a>, or a network storage system such as NFS, iSCSI, Gluster, Ceph, Cinder, or Flocker.
|
||||
|
||||
content_type: concept
|
||||
weight: 20
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
This document describes the current state of _persistent volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested.
|
||||
This document describes _persistent volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
---
|
||||
reviewers:
|
||||
- sftim
|
||||
- marosset
|
||||
- jsturtevant
|
||||
- zshihang
|
||||
title: Projected Volumes
|
||||
content_type: concept
|
||||
weight: 21 # just after persistent volumes
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
This document describes the current state of _projected volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested.
|
||||
This document describes _projected volumes_ in Kubernetes. Familiarity with [volumes](/docs/concepts/storage/volumes/) is suggested.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ reviewers:
|
||||
- pohly
|
||||
title: Storage Capacity
|
||||
content_type: concept
|
||||
weight: 45
|
||||
weight: 70
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
@@ -16,7 +16,6 @@ Storage capacity is limited and may vary depending on the node on
|
||||
which a pod runs: network-attached storage might not be accessible by
|
||||
all nodes, or storage is local to a node to begin with.
|
||||
|
||||
{{< feature-state for_k8s_version="v1.19" state="alpha" >}}
|
||||
{{< feature-state for_k8s_version="v1.21" state="beta" >}}
|
||||
|
||||
This page describes how Kubernetes keeps track of storage capacity and
|
||||
|
||||
@@ -6,7 +6,7 @@ reviewers:
|
||||
- msau42
|
||||
title: CSI Volume Cloning
|
||||
content_type: concept
|
||||
weight: 30
|
||||
weight: 60
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
@@ -8,7 +8,7 @@ reviewers:
|
||||
- yuxiangqian
|
||||
title: Volume Snapshot Classes
|
||||
content_type: concept
|
||||
weight: 30
|
||||
weight: 41 # just after volume snapshots
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
@@ -8,7 +8,7 @@ reviewers:
|
||||
- yuxiangqian
|
||||
title: Volume Snapshots
|
||||
content_type: concept
|
||||
weight: 20
|
||||
weight: 40
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
@@ -32,7 +32,7 @@ The following are typical use cases for Deployments:
|
||||
* [Declare the new state of the Pods](#updating-a-deployment) by updating the PodTemplateSpec of the Deployment. A new ReplicaSet is created and the Deployment manages moving the Pods from the old ReplicaSet to the new one at a controlled rate. Each new ReplicaSet updates the revision of the Deployment.
|
||||
* [Rollback to an earlier Deployment revision](#rolling-back-a-deployment) if the current state of the Deployment is not stable. Each rollback updates the revision of the Deployment.
|
||||
* [Scale up the Deployment to facilitate more load](#scaling-a-deployment).
|
||||
* [Pause the Deployment](#pausing-and-resuming-a-deployment) to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout.
|
||||
* [Pause the rollout of a Deployment](#pausing-and-resuming-a-deployment) to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout.
|
||||
* [Use the status of the Deployment](#deployment-status) as an indicator that a rollout has stuck.
|
||||
* [Clean up older ReplicaSets](#clean-up-policy) that you don't need anymore.
|
||||
|
||||
@@ -697,9 +697,12 @@ nginx-deployment-1989198191 7 7 0 7m
|
||||
nginx-deployment-618515232 11 11 11 7m
|
||||
```
|
||||
|
||||
## Pausing and Resuming a Deployment
|
||||
## Pausing and Resuming a rollout of a Deployment {#pausing-and-resuming-a-deployment}
|
||||
|
||||
You can pause a Deployment before triggering one or more updates and then resume it. This allows you to
|
||||
When you update a Deployment, or plan to, you can pause rollouts
|
||||
for that Deployment before you trigger one or more updates. When
|
||||
you're ready to apply those changes, you resume rollouts for the
|
||||
Deployment. This approach allows you to
|
||||
apply multiple fixes in between pausing and resuming without triggering unnecessary rollouts.
|
||||
|
||||
* For example, with a Deployment that was created:
|
||||
@@ -775,10 +778,10 @@ apply multiple fixes in between pausing and resuming without triggering unnecess
|
||||
deployment.apps/nginx-deployment resource requirements updated
|
||||
```
|
||||
|
||||
The initial state of the Deployment prior to pausing it will continue its function, but new updates to
|
||||
the Deployment will not have any effect as long as the Deployment is paused.
|
||||
The initial state of the Deployment prior to pausing its rollout will continue its function, but new updates to
|
||||
the Deployment will not have any effect as long as the Deployment rollout is paused.
|
||||
|
||||
* Eventually, resume the Deployment and observe a new ReplicaSet coming up with all the new updates:
|
||||
* Eventually, resume the Deployment rollout and observe a new ReplicaSet coming up with all the new updates:
|
||||
```shell
|
||||
kubectl rollout resume deployment/nginx-deployment
|
||||
```
|
||||
@@ -912,8 +915,8 @@ example, rollback the Deployment to its previous version.
|
||||
{{< /note >}}
|
||||
|
||||
{{< note >}}
|
||||
If you pause a Deployment, Kubernetes does not check progress against your specified deadline.
|
||||
You can safely pause a Deployment in the middle of a rollout and resume without triggering
|
||||
If you pause a Deployment rollout, Kubernetes does not check progress against your specified deadline.
|
||||
You can safely pause a Deployment rollout in the middle of a rollout and resume without triggering
|
||||
the condition for exceeding the deadline.
|
||||
{{< /note >}}
|
||||
|
||||
@@ -1065,6 +1068,18 @@ allowed, which is the default if not specified.
|
||||
|
||||
`.spec.replicas` is an optional field that specifies the number of desired Pods. It defaults to 1.
|
||||
|
||||
Should you manually scale a Deployment, example via `kubectl scale deployment
|
||||
deployment --replicas=X`, and then you update that Deployment based on a manifest
|
||||
(for example: by running `kubectl apply -f deployment.yaml`),
|
||||
then applying that manifest overwrites the manual scaling that you previously did.
|
||||
|
||||
If a [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/) (or any
|
||||
similar API for horizontal scaling) is managing scaling for a Deployment, don't set `.spec.replicas`.
|
||||
|
||||
Instead, allow the Kubernetes
|
||||
{{< glossary_tooltip text="control plane" term_id="control-plane" >}} to manage the
|
||||
`.spec.replicas` field automatically.
|
||||
|
||||
### Selector
|
||||
|
||||
`.spec.selector` is a required field that specifies a [label selector](/docs/concepts/overview/working-with-objects/labels/)
|
||||
|
||||
@@ -379,6 +379,22 @@ which will verify owner references before terminating Pods. If that is not possi
|
||||
operator should verify the owner references on PVCs to ensure the expected objects are
|
||||
deleted when Pods are force-deleted.
|
||||
|
||||
### Replicas
|
||||
|
||||
`.spec.replicas` is an optional field that specifies the number of desired Pods. It defaults to 1.
|
||||
|
||||
Should you manually scale a deployment, example via `kubectl scale
|
||||
statefulset statefulset --replicas=X`, and then you update that StatefulSet
|
||||
based on a manifest (for example: by running `kubectl apply -f
|
||||
statefulset.yaml`), then applying that manifest overwrites the manual scaling
|
||||
that you previously did.
|
||||
|
||||
If a [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/)
|
||||
(or any similar API for horizontal scaling) is managing scaling for a
|
||||
Statefulset, don't set `.spec.replicas`. Instead, allow the Kubernetes
|
||||
{{<glossary_tooltip text="control plane" term_id="control-plane" >}} to manage
|
||||
the `.spec.replicas` field automatically.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Learn about [Pods](/docs/concepts/workloads/pods).
|
||||
|
||||
@@ -159,7 +159,7 @@ through which the Pod has or has not passed:
|
||||
* `PodScheduled`: the Pod has been scheduled to a node.
|
||||
* `ContainersReady`: all containers in the Pod are ready.
|
||||
* `Initialized`: all [init containers](/docs/concepts/workloads/pods/init-containers/)
|
||||
have started successfully.
|
||||
have completed successfully.
|
||||
* `Ready`: the Pod is able to serve requests and should be added to the load
|
||||
balancing pools of all matching Services.
|
||||
|
||||
|
||||
@@ -85,7 +85,7 @@ You can define one or multiple `topologySpreadConstraint` to instruct the kube-s
|
||||
It must be greater than zero. Its semantics differs according to the value of `whenUnsatisfiable`:
|
||||
- when `whenUnsatisfiable` equals to "DoNotSchedule", `maxSkew` is the maximum
|
||||
permitted difference between the number of matching pods in the target
|
||||
topology and the global minimum
|
||||
topology and the global minimum
|
||||
(the minimum number of pods that match the label selector in a topology domain. For example, if you have 3 zones with 0, 2 and 3 matching pods respectively, The global minimum is 0).
|
||||
- when `whenUnsatisfiable` equals to "ScheduleAnyway", scheduler gives higher
|
||||
precedence to topologies that would help reduce the skew.
|
||||
@@ -234,6 +234,8 @@ To overcome this situation, you can either increase the `maxSkew` or modify one
|
||||
|
||||
The scheduler will skip the non-matching nodes from the skew calculations if the incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined.
|
||||
|
||||
### Example: TopologySpreadConstraints with NodeAffinity
|
||||
|
||||
Suppose you have a 5-node cluster ranging from zoneA to zoneC:
|
||||
|
||||
{{<mermaid>}}
|
||||
@@ -349,12 +351,14 @@ Also, the legacy `SelectorSpread` plugin, which provides an equivalent behavior,
|
||||
is disabled.
|
||||
|
||||
{{< note >}}
|
||||
The `PodTopologySpread` plugin does not score the nodes that don't have
|
||||
the topology keys specified in the spreading constraints. This might result
|
||||
in a different default behavior compared to the legacy `SelectorSpread` plugin when
|
||||
using the default topology constraints.
|
||||
|
||||
If your nodes are not expected to have **both** `kubernetes.io/hostname` and
|
||||
`topology.kubernetes.io/zone` labels set, define your own constraints
|
||||
instead of using the Kubernetes defaults.
|
||||
|
||||
The `PodTopologySpread` plugin does not score the nodes that don't have
|
||||
the topology keys specified in the spreading constraints.
|
||||
{{< /note >}}
|
||||
|
||||
If you don't want to use the default Pod spreading constraints for your cluster,
|
||||
@@ -392,7 +396,7 @@ for more details.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution.
|
||||
- There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution.
|
||||
You can use [Descheduler](https://github.com/kubernetes-sigs/descheduler) to rebalance the Pods distribution.
|
||||
- Pods matched on tainted nodes are respected. See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user