[Do Not Merge] Release 1.12 (#10292)
* Update docs for fields allowed at root of CRD schema (#9973) * add plugin docs and examples (#10053) * docs update to promote TaintNodesByCondition to beta (#9626) * HPA Specificity Improvements (#8757) Updated the HPA docs to reference the `autoscaling/v2beta2` API version, and added documentation about the new fields. * adjust docs for pod ready++ (#10049) * Remove --cadvisor-port - has been deprecated since v1.10 (#10023) Change-Id: Id2a685473a243aef492a98ff450759f39e362557 * Add Documentation for Snapshot Feature (#9948) * Add documentation for snapshot feature * Update volume-snapshots.md * Add dry-run to api-concepts (#10033) * kubeadm-init: Update the offline support section (#10062) The update includes the following things (in mind with Kubernetes 1.12): - Remove the 1.8 image versions - Add the 1.10 image versions that were missing until now - Include a comment for the missing arch suffixes in 1.12 Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com> * Say bye to `DynamicProvisioningScheduling` (#10157) The mentioned feature gate is now collapsed into `VolumeScheduling`. xref: kubernetes/kubernetes#67432 * Update ResourceQuota per PriorityClass state for 1.12 (#10229) * TokenRequest and TokenRequestProjection now beta (#10161) xref: kubernetes/kubernetes#67349 * Change feature state for kms provider to beta. (#10230) KMS Provider will be graduating to beta in v1.12, reflecting this change on the website. * coredns default (#10200) * Promote ShareProcessNamespace to beta in docs (#9996) * Add CoreDNS details to DNS Debug docs (#10201) * add coredns details * address nits, add query logging section * Update docs with topology aware dynamic provisioning (#9939) * Document topology aware volume binding feature * update for readability * Update storage-classes.md * comma splice * don't abbreviate * HPA Algorithm Information Improvements (#9780) * Update HPA docs with more algorithm details The HPA docs pointed to an out-of-date document for information on the algorithm details, which users were finding confusing. This sticks a section on the algorithm in the HPA docs instead, documenting both general behavior and corner cases. * Add glossary info, HPA docs on quantities People often ask about the quantity notation when working with the metrics APIs, so this adds a glossary entry on quantities (since they're used elsewhere in the system), and a short explantation in the HPA walkthough. * Information about HPA readiness and stabilization This adds information about the new changes to HPA readiness and stabilization from kubernetes/features#591, and other minor changes that landed in Kubernetes 1.12. * Update horizontal-pod-autoscale.md * Audit 1.12 doc (#9953) * audit 1.12 document * remove legacy audit feature https://github.com/kubernetes/kubernetes/pull/65862 * update feature gate doc * MountPropagation is now GA (#10090) * RuntimeClass documentation (#10102) * RuntimeClass documentation * Update runtime-class.md * Add documentation for Scheduler performance tuning (#10048) * Add documentation for Scheduler performance tuning * Update scheduler-perf-tuning.md * TTL controller for cleaning up finished resources (#10064) * TTL controller for cleaning up finished resources * Address comments * Update ttlafterfinished.md * Bump quota configuration api version (#10217) * Incremental update from master (#10278) * fix invalid href of cloud controller manager (#10240) * fix invalid yaml format (#10238) * update storage-limits doc with Azure disk part (#10224) update storage-limits doc with Azure disk part fix comments * Update kubelet-config-file.md (#10222) Update link to KubeletConfiguration struct. * fix a trivial misspelling (#10244) * Fix cassandra-statefulset.yaml indent level (#10243) * Mention minimum etcd versions (#10208) Source: https://groups.google.com/d/msg/kubernetes-dev/jMPA4JzKiY4/HIx2ugvLBAAJ * fix 404 error (#10250) * Small verb tweak (#10190) Present participle, ftw. * Add AnchorJS logic for header links (#10155) * Add AnchorJS JavaScript * Remove existing inpage_heading logic * Remove underline from anchor tags * Use single icon and add touch visibility * Use paragraph link icon for AnchorJS * Update Sass to use code formatting in docsContent headers * Update header size coverage to H3-H6 * fix broken link in kubefed.md (#10254) * Update the version numbers for the X-Remote-Extra- and Impersonate-Extra- key fixes (#9827) The fix was cherry picked into 1.11.3, 1.10.7, and 1.9.11: https://github.com/kubernetes/kubernetes/pull/67162 https://github.com/kubernetes/kubernetes/pull/67163 https://github.com/kubernetes/kubernetes/pull/67164 * fix typo (#10168) * fix typo * addressing comments. * Update setup-ha-etcd-with-kubeadm.md * fix typos (#10252) * fix description of contribute guide (#10253) * describe truncate feature about advanced audit (#10236) * describe truncate feature about advanced audit * Update audit.md * docs update to promote ScheduleDaemonSetPods to beta (#9923) * Dynamic volume limit updates for 1.12 (#10211) * add a placeholder commit * Update docs for csi volume limits * Update storage-limits.md * Add "MayRunAs" value among other GroupStrategies (#9888) * Add CoreDNS details to the customize DNS doc (#10228) * Add CoreDNS details to the customize DNS doc Rewrite the document to include more details about CoreDNS, since it's now the default from v1.12 * Address comments * Improve doc wording * Fix link * Update dns-custom-nameservers.md * Update dns-custom-nameservers.md * Fix secrets docs in 1.12 branch (#10056) * Fix secrets docs * Update secret.md * Revert CoreDNS Docs (#10319) * Revert "Add CoreDNS details to DNS Debug docs (#10201)" This reverts commit462817a674. * Revert "Add CoreDNS details to the customize DNS doc (#10228)" This reverts commite7319eeb8c. * Revert "coredns default (#10200)" This reverts commit698e93b441. * Add CRI installation instructions page Added cri-installation page with CRI installation instructions Referenced it from kubeadm-init and install-kubeadm pages. * kubeadm: update API types documentation for 1.12 (#10283) v1alpha2 -> v1alpha3 MasterConfiguration -> [new-api-types] * TokenRequest feature documentation (#10295) * AdvancedAuditing is now GA (#10156) xref: kubernetes/kubernetes#65862 `AdvancedAuditing` feature is GA in 1.12. This PR adjusts the related docs. * update runtime-class.md (#10332) * update runtime-class.md * Update runtime-class.md * Document cross-authorizer permissions for creating RBAC roles (#10015) * Document cross-authorizer permissions for creating RBAC roles * Update rbac.md * kubeadm: update authored content for 1.12 (reference docs and cluster creation) (#10348) * kubeadm: update authored content in reference docs for 1.12 * kubeadm: add time frame in create-cluster-kubeadm for 1.12 * add AllowedProcMountTypes and ProcMountType to docs (#9911) Signed-off-by: Jess Frazelle <acidburn@microsoft.com> * kubeadm: add new command line reference (#10306) Add: - placeholder files - include place holder files - include "renew" sub command - add missing tabs for "alpha phase kubelet" * Documenting SCTP support in Kubernetes (#10279) * Documenting SCTP support in Kubernetes Service, Endpoint, NetworkPolicy and Pod * Updates based on comments on the PR * kubectl expose update with SCTP support * Updated according to comments in the PR * Revert "kubectl expose update with SCTP support" This reverts commit 0d5a1e6720a012390cf100c83e16b4a8c0782356. * TLS Bootstrap and Server Cert Rotation feature documentation (#10232) * TokenRequest feature documentation * line wrapping to make review not insane * update content for GA without major refactor * Update kubelet-tls-bootstrapping.md * Add clarifications for volume snapshots (#10296) * Update kubadm ha installation for 1.12 (#10264) * Update kubadm ha installation for 1.12 Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * update stable version Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Update stacked control plane for v1.12 (#2) * use v1alpha3 Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * more v1alpha3 (#4) * updates Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Document how to run in-tree cloud providers with kubeadm (#10357) Change-Id: Iab6b996a830503d74a6eb0c507c5f8ca7a39235b * kubeadm reference doc for release 1.12 (#10359) * Revert "Revert "Add CoreDNS details to DNS Debug docs (#10201)"" This reverts commit bb30f4d1fcd6fba2fe6190778ead99f8010033b7. * Revert "Revert "Add CoreDNS details to the customize DNS doc (#10228)"" This reverts commit bc23d45c09d7b83cac130fe22a0bd91e72435862. * Revert "Revert "coredns default (#10200)"" This reverts commit 7f4350d6ab7fc554ee53126d3875e845d2e43d1f. * add missing instruction for ha guide (#10374) Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * kubeadm - Ha upgrade updates (#10340) * Update HA upgrade docs * Adds external etcd HA upgrade guide Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * copyedit * more edits * add runasgroup in psp (#10076) * update KubeletPluginsWatcher feature gate (#10205) * generated 1.12 docs * Building Multi-arch images with Manifests (#10379) In 1.12, a variety of images used in a typical kubernetes installation have started to using manifests to better support environments with arm or ppc64le architectures. For example all images used with kubeadm by default have manifests, another would be all the tests in the conformance test suite. Here we capture the best practices for everyone to start using manifests in their own workflows. Change-Id: I5ba4c5fe55ffc9486a8251760f3352be4f2e1494 * Upgrade docs for v1.12 (#10344) * generated assets and docs * remove 1.7 * update 1.12 * update plugin documentation under docs>tasks>extend-kubectl (#10259) * update plugin documentation under docs>tasks>extend-kubectl * Update kubectl-plugins.md
This commit is contained in:
Jim Angel
@@ -76,11 +76,9 @@ the `Terminating` or `Unknown` state. In cases where Kubernetes cannot deduce fr
|
||||
permanently left a cluster, the cluster administrator may need to delete the node object by hand. Deleting the node object from
|
||||
Kubernetes causes all the Pod objects running on the node to be deleted from the apiserver, and frees up their names.
|
||||
|
||||
Version 1.8 introduced an alpha feature that automatically creates
|
||||
In version 1.12, `TaintNodesByCondition` feature is promoted to beta，so node lifecycle controller automatically creates
|
||||
[taints](/docs/concepts/configuration/taint-and-toleration/) that represent conditions.
|
||||
To enable this behavior, pass an additional feature gate flag `--feature-gates=...,TaintNodesByCondition=true`
|
||||
to the API server, controller manager, and scheduler.
|
||||
When `TaintNodesByCondition` is enabled, the scheduler ignores conditions when considering a Node; instead
|
||||
Similarly the scheduler ignores conditions when considering a Node; instead
|
||||
it looks at the Node's taints and a Pod's tolerations.
|
||||
|
||||
Now users can choose between the old scheduling model and a new, more flexible scheduling model.
|
||||
|
||||
@@ -9,7 +9,47 @@ This page explains how to manage Kubernetes running on a specific
|
||||
cloud provider.
|
||||
{{% /capture %}}
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
{{% capture body %}}
|
||||
### kubeadm
|
||||
[kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/) is a popular option for creating kubernetes clusters.
|
||||
kubeadm has configuration options to specify configuration information for cloud providers. For example a typical
|
||||
in-tree cloud provider can be configured using kubeadm as shown below:
|
||||
|
||||
```yaml
|
||||
apiVersion: kubeadm.k8s.io/v1alpha3
|
||||
kind: InitConfiguration
|
||||
nodeRegistration:
|
||||
kubeletExtraArgs:
|
||||
cloud-provider: "openstack"
|
||||
cloud-config: "/etc/kubernetes/cloud.conf"
|
||||
---
|
||||
kind: ClusterConfiguration
|
||||
apiVersion: kubeadm.k8s.io/v1alpha3
|
||||
kubernetesVersion: v1.12.0
|
||||
apiServerExtraArgs:
|
||||
cloud-provider: "openstack"
|
||||
cloud-config: "/etc/kubernetes/cloud.conf"
|
||||
apiServerExtraVolumes:
|
||||
- name: cloud
|
||||
hostPath: "/etc/kubernetes/cloud.conf"
|
||||
mountPath: "/etc/kubernetes/cloud.conf"
|
||||
controllerManagerExtraArgs:
|
||||
cloud-provider: "openstack"
|
||||
cloud-config: "/etc/kubernetes/cloud.conf"
|
||||
controllerManagerExtraVolumes:
|
||||
- name: cloud
|
||||
hostPath: "/etc/kubernetes/cloud.conf"
|
||||
mountPath: "/etc/kubernetes/cloud.conf"
|
||||
```
|
||||
|
||||
The in-tree cloud providers typically need both `--cloud-provider` and `--cloud-config` specified in the command lines
|
||||
for the [kube-apiserver](/docs/admin/kube-apiserver/), [kube-controller-manager](/docs/admin/kube-controller-manager/) and the
|
||||
[kubelet](/docs/admin/kubelet/). The contents of the file specified in `--cloud-config` for each provider is documented below as well.
|
||||
|
||||
For all external cloud providers, please follow the instructions on the individual repositories.
|
||||
|
||||
## AWS
|
||||
This section describes all the possible configurations which can
|
||||
be used when running Kubernetes on Amazon Web Services.
|
||||
|
||||
@@ -36,7 +36,7 @@ There are several different proxies you may encounter when using Kubernetes:
|
||||
1. The [kube proxy](/docs/concepts/services-networking/service/#ips-and-vips):
|
||||
|
||||
- runs on each node
|
||||
- proxies UDP and TCP
|
||||
- proxies UDP, TCP and SCTP
|
||||
- does not understand HTTP
|
||||
- provides load balancing
|
||||
- is just used to reach services
|
||||
@@ -51,7 +51,8 @@ There are several different proxies you may encounter when using Kubernetes:
|
||||
|
||||
- are provided by some cloud providers (e.g. AWS ELB, Google Cloud Load Balancer)
|
||||
- are created automatically when the Kubernetes service has type `LoadBalancer`
|
||||
- use UDP/TCP only
|
||||
- usually supports UDP/TCP only
|
||||
- SCTP support is up to the load balancer implementation of the cloud provider
|
||||
- implementation varies by cloud provider.
|
||||
|
||||
Kubernetes users will typically not need to worry about anything other than the first two types. The cluster admin
|
||||
|
||||
@@ -42,7 +42,7 @@ other pods to be evicted/not get scheduled. To resolve this issue,
|
||||
[ResourceQuota](https://kubernetes.io/docs/concepts/policy/resource-quotas/) is
|
||||
augmented to support Pod priority. An admin can create ResourceQuota for users
|
||||
at specific priority levels, preventing them from creating pods at high
|
||||
priorities. However, this feature is in alpha as of Kubernetes 1.11.
|
||||
priorities. This feature is in beta since Kubernetes 1.12.
|
||||
{{< /warning >}}
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
reviewers:
|
||||
- bsalamat
|
||||
title: Scheduler Performance Tuning
|
||||
content_template: templates/concept
|
||||
weight: 70
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="1.12" >}}
|
||||
|
||||
Kube-scheduler is the Kubernetes default scheduler. It is responsible for
|
||||
placement of Pods on Nodes in a cluster. Nodes in a cluster that meet the
|
||||
scheduling requirements of a Pod are called "feasible" Nodes for the Pod. The
|
||||
scheduler finds feasible Nodes for a Pod and then runs a set of functions to
|
||||
score the feasible Nodes and picks a Node with the highest score among the
|
||||
feasible ones to run the Pod. The scheduler then notifies the API server about this
|
||||
decision in a process called "Binding".
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Percentage of Nodes to Score
|
||||
|
||||
Before Kubernetes 1.12, Kube-scheduler used to check the feasibility of all the
|
||||
nodes in a cluster and then scored the feasible ones. Kubernetes 1.12 has a new
|
||||
feature that allows the scheduler to stop looking for more feasible nodes once
|
||||
it finds a certain number of them. This improves the scheduler's performance in
|
||||
large clusters. The number is specified as a percentage of the cluster size and
|
||||
is controlled by a configuration option called `percentageOfNodesToScore`. The
|
||||
range should be between 1 and 100. Other values are considered as 100%. The
|
||||
default value of this option is 50%. A cluster administrator can change this value by providing a
|
||||
different value in the scheduler configuration. However, it may not be necessary to change this value.
|
||||
|
||||
```yaml
|
||||
apiVersion: componentconfig/v1alpha1
|
||||
kind: KubeSchedulerConfiguration
|
||||
algorithmSource:
|
||||
provider: DefaultProvider
|
||||
|
||||
...
|
||||
|
||||
percentageOfNodesToScore: 50
|
||||
```
|
||||
|
||||
{{< note >}} **Note**: In clusters with zero or less than 50 feasible nodes, the
|
||||
scheduler still checks all the nodes, simply because there are not enough
|
||||
feasible nodes to stop the scheduler's search early. {{< /note >}}
|
||||
|
||||
**To disable this feature**, you can set `percentageOfNodesToScore` to 100.
|
||||
|
||||
### Tuning percentageOfNodesToScore
|
||||
|
||||
`percentageOfNodesToScore` must be a value between 1 and 100
|
||||
with the default value of 50. There is also a hardcoded minimum value of 50
|
||||
nodes which is applied internally. The scheduler tries to find at
|
||||
least 50 nodes regardless of the value of `percentageOfNodesToScore`. This means
|
||||
that changing this option to lower values in clusters with several hundred nodes
|
||||
will not have much impact on the number of feasible nodes that the scheduler
|
||||
tries to find. This is intentional as this option is unlikely to improve
|
||||
performance noticeably in smaller clusters. In large clusters with over a 1000
|
||||
nodes setting this value to lower numbers may show a noticeable performance
|
||||
improvement.
|
||||
|
||||
An important note to consider when setting this value is that when a smaller
|
||||
number of nodes in a cluster are checked for feasibility, some nodes are not
|
||||
sent to be scored for a given Pod. As a result, a Node which could possibly
|
||||
score a higher value for running the given Pod might not even be passed to the
|
||||
scoring phase. This would result in a less than ideal placement of the Pod. For
|
||||
this reason, the value should not be set to very low percentages. A general rule
|
||||
of thumb is to never set the value to anything lower than 30. Lower values
|
||||
should be used only when the scheduler's throughput is critical for your
|
||||
application and the score of nodes is not important. In other words, you prefer
|
||||
to run the Pod on any Node as long as it is feasible.
|
||||
|
||||
It is not recommended to lower this value from its default if your cluster has
|
||||
only several hundred Nodes. It is unlikely to improve the scheduler's
|
||||
performance significantly.
|
||||
|
||||
### How the scheduler iterates over Nodes
|
||||
|
||||
This section is intended for those who want to understand the internal details
|
||||
of this feature.
|
||||
|
||||
In order to give all the Nodes in a cluster a fair chance of being considered
|
||||
for running Pods, the scheduler iterates over the nodes in a round robin
|
||||
fashion. You can imagine that Nodes are in an array. The scheduler starts from
|
||||
the start of the array and checks feasibility of the nodes until it finds enough
|
||||
Nodes as specified by `percentageOfNodesToScore`. For the next Pod, the
|
||||
scheduler continues from the point in the Node array that it stopped at when checking
|
||||
feasibility of Nodes for the previous Pod.
|
||||
|
||||
If Nodes are in multiple zones, the scheduler iterates over Nodes in various
|
||||
zones to ensure that Nodes from different zones are considered in the
|
||||
feasibility checks. As an example, consider six nodes in two zones:
|
||||
|
||||
```
|
||||
Zone 1: Node 1, Node 2, Node 3, Node 4
|
||||
Zone 2: Node 5, Node 6
|
||||
```
|
||||
|
||||
The Scheduler evaluates feasibility of the nodes in this order:
|
||||
|
||||
```
|
||||
Node 1, Node 5, Node 2, Node 6, Node 3, Node 4
|
||||
```
|
||||
|
||||
After going over all the Nodes, it goes back to Node 1.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -343,9 +343,15 @@ files.
|
||||
|
||||
When a secret being already consumed in a volume is updated, projected keys are eventually updated as well.
|
||||
Kubelet is checking whether the mounted secret is fresh on every periodic sync.
|
||||
However, it is using its local ttl-based cache for getting the current value of the secret.
|
||||
As a result, the total delay from the moment when the secret is updated to the moment when new keys are
|
||||
projected to the pod can be as long as kubelet sync period + ttl of secrets cache in kubelet.
|
||||
However, it is using its local cache for getting the current value of the Secret.
|
||||
The type of the cache is configurable using the (`ConfigMapAndSecretChangeDetectionStrategy` field in
|
||||
[KubeletConfiguration struct](https://github.com/kubernetes/kubernetes/blob/{{< param "docsbranch" >}}/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go)).
|
||||
It can be either propagated via watch (default), ttl-based, or simply redirecting
|
||||
all requests to directly kube-apiserver.
|
||||
As a result, the total delay from the moment when the Secret is updated to the moment
|
||||
when new keys are projected to the Pod can be as long as kubelet sync period + cache
|
||||
propagation delay, where cache propagation delay depends on the chosen cache type
|
||||
(it equals to watch propagation delay, ttl of cache, or zero corespondingly).
|
||||
|
||||
{{< note >}}
|
||||
**Note:** A container using a Secret as a
|
||||
|
||||
@@ -279,9 +279,10 @@ which matches the behavior when this feature is disabled.
|
||||
|
||||
## Taint Nodes by Condition
|
||||
|
||||
Version 1.8 introduces an alpha feature that causes the node controller to create taints corresponding to
|
||||
Node conditions. When this feature is enabled (you can do this by including `TaintNodesByCondition=true` in the `--feature-gates` command line flag to the scheduler, such as
|
||||
`--feature-gates=FooBar=true,TaintNodesByCondition=true`), the scheduler does not check Node conditions; instead the scheduler checks taints. This assures that Node conditions don't affect what's scheduled onto the Node. The user can choose to ignore some of the Node's problems (represented as Node conditions) by adding appropriate Pod tolerations.
|
||||
In version 1.12, `TaintNodesByCondition` feature is promoted to beta, so node lifecycle controller automatically creates taints corresponding to
|
||||
Node conditions.
|
||||
Similarly the scheduler does not check Node conditions; instead the scheduler checks taints. This assures that Node conditions don't affect what's scheduled onto the Node. The user can choose to ignore some of the Node's problems (represented as Node conditions) by adding appropriate Pod tolerations.
|
||||
Note that `TaintNodesByCondition` only taints nodes with `NoSchedule` effect. `NoExecute` effect is controlled by `TaintBasedEviction` which is an alpha feature and disabled by default.
|
||||
|
||||
Starting in Kubernetes 1.8, the DaemonSet controller automatically adds the
|
||||
following `NoSchedule` tolerations to all daemons, to prevent DaemonSets from
|
||||
|
||||
@@ -32,6 +32,26 @@ you can do one of the following:
|
||||
|
||||
Note that you should avoid using `:latest` tag, see [Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images) for more information.
|
||||
|
||||
## Building Multi-architecture Images with Manifests
|
||||
|
||||
Docker CLI now supports the following command `docker manifest` with sub commands like `create`, `annotate` and `push`. These commands can be used to build and push the manifests. You can use `docker manifest inspect` to view the manifest.
|
||||
|
||||
Please see docker documentation here:
|
||||
https://docs.docker.com/edge/engine/reference/commandline/manifest/
|
||||
|
||||
See examples on how we use this in our build harness:
|
||||
https://cs.k8s.io/?q=docker%20manifest%20(create%7Cpush%7Cannotate)&i=nope&files=&repos=
|
||||
|
||||
These commands rely on and are implemented purely on the Docker CLI. You will need to either edit the `$HOME/.docker/config.json` and set `experimental` key to `enabled` or you can just set `DOCKER_CLI_EXPERIMENTAL` environment variable to `enabled` when you call the CLI commands.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** Please use Docker *18.06 or above*, versions below that either have bugs or do not support the experimental command line option. Example https://github.com/docker/cli/issues/1135 causes problems under containerd.
|
||||
{{< /note >}}
|
||||
|
||||
If you run into trouble with uploading stale manifests, just clean up the older manifests in `$HOME/.docker/manifests` to start fresh.
|
||||
|
||||
For Kubernetes, we have typically used images with suffix `-$(ARCH)`. For backward compatability, please generate the older images with suffixes. The idea is to generate say `pause` image which has the manifest for all the arch(es) and say `pause-amd64` which is backwards compatible for older configurations or YAML files which may have hard coded the images with suffixes.
|
||||
|
||||
## Using a Private Registry
|
||||
|
||||
Private registries may require keys to read images from them.
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
reviewers:
|
||||
- tallclair
|
||||
- dchen1107
|
||||
title: Runtime Class
|
||||
content_template: templates/concept
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
This page describes the RuntimeClass resource and runtime selection mechanism.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Runtime Class
|
||||
|
||||
RuntimeClass is an alpha feature for selecting the container runtime configuration to use to run a
|
||||
pod's containers.
|
||||
|
||||
### Set Up
|
||||
|
||||
As an early alpha feature, there are some additional setup steps that must be taken in order to use
|
||||
the RuntimeClass feature:
|
||||
|
||||
1. Enable the RuntimeClass feature gate (on apiservers & kubelets, requires version 1.12+)
|
||||
2. Install the RuntimeClass CRD
|
||||
3. Configure the CRI implementation on nodes (runtime dependent)
|
||||
4. Create the corresponding RuntimeClass resources
|
||||
|
||||
#### 1. Enable the RuntimeClass feature gate
|
||||
|
||||
See [Feature Gates](/docs/reference/command-line-tools-reference/feature-gates/) for an explanation
|
||||
of enabling feature gates. The `RuntimeClass` feature gate must be enabled on apiservers _and_
|
||||
kubelets.
|
||||
|
||||
#### 2. Install the RuntimeClass CRD
|
||||
|
||||
The RuntimeClass [CustomResourceDefinition][/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/] (CRD) can be found in the addons directory of the
|
||||
Kubernetes git repo:
|
||||
|
||||
https://github.com/kubernetes/kubernetes/tree/release-1.12/cluster/addons/runtimeclass/runtimeclass_crd.yaml
|
||||
|
||||
Install the CRD with `kubectl apply -f runtimeclass_crd.yaml`.
|
||||
|
||||
[CustomResourceDefinition][/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/]
|
||||
|
||||
#### 3. Configure the CRI implementation on nodes
|
||||
|
||||
The configurations to select between with RuntimeClass are CRI implementation dependent. See the
|
||||
corresponding documentation for your CRI implementation for how to configure. As this is an alpha
|
||||
feature, not all CRIs support multiple RuntimeClasses yet.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** RuntimeClass currently assumes a homogeneous node configuration across the cluster
|
||||
(which means that all nodes are configured the same way with respect to container runtimes). Any heterogeneity (varying configurations) must be
|
||||
managed independently of RuntimeClass through scheduling features (see [Assigning Pods to
|
||||
Nodes](/docs/concepts/configuration/assign-pod-node/)).
|
||||
{{< /note >}}
|
||||
|
||||
The configurations have a corresponding `RuntimeHandler` name, referenced by the RuntimeClass. The
|
||||
RuntimeHandler must be a valid DNS 1123 subdomain (alpha-numeric + `-` and `.` characters).
|
||||
|
||||
#### 4. Create the corresponding RuntimeClass resources
|
||||
|
||||
The configurations setup in step 3 should each have an associated `RuntimeHandler` name, which
|
||||
identifies the configuration. For each RuntimeHandler (and optionally the empty `""` handler),
|
||||
create a corresponding RuntimeClass object.
|
||||
|
||||
The RuntimeClass resource currently only has 2 significant fields: the RuntimeClass name
|
||||
(`metadata.name`) and the RuntimeHandler (`spec.runtimeHandler`). The object definition looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: node.k8s.io/v1alpha1 # RuntimeClass is defined in the node.k8s.io API group
|
||||
kind: RuntimeClass
|
||||
metadata:
|
||||
name: myclass # The name the RuntimeClass will be referenced by
|
||||
# RuntimeClass is a non-namespaced resource
|
||||
spec:
|
||||
runtimeHandler: myconfiguration # The name of the correpsonding CRI configuration
|
||||
```
|
||||
|
||||
|
||||
{{< note >}}
|
||||
|
||||
**Note:** It is recommended that RuntimeClass write operations (create/update/patch/delete) be
|
||||
restricted to the cluster administrator. This is typically the default. See [Authorization
|
||||
Overview](https://kubernetes.io/docs/reference/access-authn-authz/authorization/) for more details.
|
||||
|
||||
{{< /note >}}
|
||||
|
||||
### Usage
|
||||
|
||||
Once RuntimeClasses are configured for the cluster, using them is very simple. Specify a
|
||||
`runtimeClassName` in the Pod spec. For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: mypod
|
||||
spec:
|
||||
runtimeClassName: myclass
|
||||
# ...
|
||||
```
|
||||
|
||||
This will instruct the Kubelet to use the named RuntimeClass to run this pod. If the named
|
||||
RuntimeClass does not exist, or the CRI cannot run the corresponding handler, the pod will enter the
|
||||
`Failed` terminal [phase](/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase). Look for a
|
||||
corresponding [event](/docs/tasks/debug-application-cluster/debug-application-introspection/) for an
|
||||
error message.
|
||||
|
||||
If no `runtimeClassName` is specified, the default RuntimeHandler will be used, which is equivalent
|
||||
to the behavior when the RuntimeClass feature is disabled.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -42,6 +42,7 @@ administrator to control the following:
|
||||
| Restricting escalation to root privileges | [`allowPrivilegeEscalation`, `defaultAllowPrivilegeEscalation`](#privilege-escalation) |
|
||||
| Linux capabilities | [`defaultAddCapabilities`, `requiredDropCapabilities`, `allowedCapabilities`](#capabilities) |
|
||||
| The SELinux context of the container | [`seLinux`](#selinux) |
|
||||
| The Allowed Proc Mount types for the container | [`allowedProcMountTypes`](#allowedProcMountTypes) |
|
||||
| The AppArmor profile used by containers | [annotations](#apparmor) |
|
||||
| The seccomp profile used by containers | [annotations](#seccomp) |
|
||||
| The sysctl profile used by containers | [annotations](#sysctl) |
|
||||
@@ -421,6 +422,9 @@ The **recommended minimum set** of allowed volumes for new PSPs are:
|
||||
|
||||
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
- *MayRunAs* - Requires at least one `range` to be specified. Allows
|
||||
`FSGroups` to be left unset without providing a default. Validates against
|
||||
all ranges if `FSGroups` is set.
|
||||
- *RunAsAny* - No default provided. Allows any `fsGroup` ID to be specified.
|
||||
|
||||
**AllowedHostPaths** - This specifies a whitelist of host paths that are allowed
|
||||
@@ -487,10 +491,24 @@ image. No default provided. Setting `allowPrivilegeEscalation=false` is strongly
|
||||
recommended with this strategy.
|
||||
- *RunAsAny* - No default provided. Allows any `runAsUser` to be specified.
|
||||
|
||||
**RunAsGroup** - Controls the what primary group ID containers run as.
|
||||
|
||||
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
- *MustRunAsNonRoot* - Requires that the pod be submitted with a non-zero
|
||||
`runAsUser` or have the `USER` directive defined (using a numeric GID) in the
|
||||
image. No default provided. Setting `allowPrivilegeEscalation=false` is strongly
|
||||
recommended with this strategy.
|
||||
- *RunAsAny* - No default provided. Allows any `runAsGroup` to be specified.
|
||||
|
||||
|
||||
**SupplementalGroups** - Controls which group IDs containers add.
|
||||
|
||||
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
- *MayRunAs* - Requires at least one `range` to be specified. Allows
|
||||
`supplementalGroups` to be left unset without providing a default.
|
||||
Validates against all ranges if `supplementalGroups` is set.
|
||||
- *RunAsAny* - No default provided. Allows any `supplementalGroups` to be
|
||||
specified.
|
||||
|
||||
@@ -548,6 +566,21 @@ for the default list of capabilities when using the Docker runtime.
|
||||
- *RunAsAny* - No default provided. Allows any `seLinuxOptions` to be
|
||||
specified.
|
||||
|
||||
### AllowedProcMountTypes
|
||||
|
||||
`allowedProcMountTypes` is a whitelist of allowed ProcMountTypes.
|
||||
Empty or nil indicates that only the `DefaultProcMountType` may be used.
|
||||
|
||||
`DefaultProcMount` uses the container runtime defaults for readonly and masked
|
||||
paths for /proc. Most container runtimes mask certain paths in /proc to avoid
|
||||
accidental security exposure of special devices or information. This is denoted
|
||||
as the string `Default`.
|
||||
|
||||
The only other ProcMountType is `UnmaskedProcMount`, which bypasses the
|
||||
default masking behavior of the container runtime and ensures the newly
|
||||
created /proc the container stays in tact with no modifications. This is
|
||||
denoted as the string `Unmasked`.
|
||||
|
||||
### AppArmor
|
||||
|
||||
Controlled via annotations on the PodSecurityPolicy. Refer to the [AppArmor
|
||||
|
||||
@@ -194,7 +194,7 @@ The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota
|
||||
|
||||
### Resource Quota Per PriorityClass
|
||||
|
||||
{{< feature-state for_k8s_version="1.11" state="alpha" >}}
|
||||
{{< feature-state for_k8s_version="1.12" state="beta" >}}
|
||||
|
||||
Pods can be created at a specific [priority](/docs/concepts/configuration/pod-priority-preemption/#pod-priority).
|
||||
You can control a pod's consumption of system resources based on a pod's priority, by using the `scopeSelector`
|
||||
@@ -548,7 +548,7 @@ kind: AdmissionConfiguration
|
||||
plugins:
|
||||
- name: "ResourceQuota"
|
||||
configuration:
|
||||
apiVersion: resourcequota.admission.k8s.io/v1alpha1
|
||||
apiVersion: resourcequota.admission.k8s.io/v1beta1
|
||||
kind: Configuration
|
||||
limitedResources:
|
||||
- resource: pods
|
||||
@@ -568,10 +568,6 @@ For example:
|
||||
values: ["cluster-services"]
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
**Note:** `scopeSelector` is an alpha field and feature gate `ResourceQuotaScopeSelectors` must be enabled before using it.
|
||||
{{< /note >}}
|
||||
|
||||
See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765) and [Quota supoport for priority class design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-priority-resourcequota.md) for more information.
|
||||
|
||||
## Example
|
||||
|
||||
@@ -246,6 +246,15 @@ spec:
|
||||
|
||||
This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed ingress or egress traffic.
|
||||
|
||||
## SCTP support
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
Kubernetes supports SCTP as a `protocol` value in `NetworkPolicy` definitions as an alpha feature. To enable this feature, the cluster administrator needs to enable the `SCTPSupport` feature gate on the apiserver, for example, `“--feature-gates=SCTPSupport=true,...”`. When the feature gate is enabled, users can set the `protocol` field of a `NetworkPolicy` to `SCTP`. Kubernetes sets up the network accordingly for the SCTP associations, just like it does for TCP connections.
|
||||
|
||||
The CNI plugin has to support SCTP as `protocol` value in `NetworkPolicy`.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
@@ -84,9 +84,13 @@ deploying and evolving your `Services`. For example, you can change the port
|
||||
number that pods expose in the next version of your backend software, without
|
||||
breaking clients.
|
||||
|
||||
Kubernetes `Services` support `TCP` and `UDP` for protocols. The default
|
||||
Kubernetes `Services` support `TCP`, `UDP` and `SCTP` for protocols. The default
|
||||
is `TCP`.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** SCTP support is an alpha feature since Kubernetes 1.12
|
||||
{{< /note >}}
|
||||
|
||||
### Services without selectors
|
||||
|
||||
Services generally abstract access to Kubernetes `Pods`, but they can also
|
||||
@@ -459,6 +463,12 @@ cloud provider does not support the feature, the field will be ignored.
|
||||
public IP address resource needs to be created first, and it should be in the same resource
|
||||
group of the other automatically created resources of the cluster. For example, `MC_myResourceGroup_myAKSCluster_eastus`. Specify the assigned IP address as loadBalancerIP. Ensure that you have updated the securityGroupName in the cloud provider configuration file. For information about troubleshooting `CreatingLoadBalancerFailed` permission issues see, [Use a static IP address with the Azure Kubernetes Service (AKS) load balancer](https://docs.microsoft.com/en-us/azure/aks/static-ip) or [CreatingLoadBalancerFailed on AKS cluster with advanced networking](https://github.com/Azure/AKS/issues/357).
|
||||
|
||||
{{< note >}}
|
||||
**Note:** The support of SCTP in the cloud provider's load balancer is up to the cloud provider's
|
||||
load balancer implementation. If SCTP is not supported by the cloud provider's load balancer the
|
||||
Service creation request is accepted but the creation of the load balancer fails.
|
||||
{{< /note >}}
|
||||
|
||||
#### Internal load balancer
|
||||
In a mixed environment it is sometimes necessary to route traffic from services inside the same VPC.
|
||||
|
||||
@@ -922,6 +932,32 @@ Service is a top-level resource in the Kubernetes REST API. More details about t
|
||||
API object can be found at:
|
||||
[Service API object](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core).
|
||||
|
||||
## SCTP support
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
Kubernetes supports SCTP as a `protocol` value in `Service`, `Endpoint`, `NetworkPolicy` and `Pod` definitions as an alpha feature. To enable this feature, the cluster administrator needs to enable the `SCTPSupport` feature gate on the apiserver, for example, `“--feature-gates=SCTPSupport=true,...”`. When the feature gate is enabled, users can set the `protocol` field of a `Service`, `Endpoint`, `NetworkPolicy` and `Pod` to `SCTP`. Kubernetes sets up the network accordingly for the SCTP associations, just like it does for TCP connections.
|
||||
|
||||
### Warnings
|
||||
|
||||
#### The support of multihomed SCTP associations
|
||||
|
||||
The support of multihomed SCTP associations requires that the CNI plugin can support the assignment of multiple interfaces and IP addresses to a `Pod`.
|
||||
|
||||
NAT for multihomed SCTP assoications requires special logic in the corresponding kernel modules.
|
||||
|
||||
#### Service with type=LoadBalancer
|
||||
|
||||
A `Service` with `type` LoadBalancer and `protocol` SCTP can be created only if the cloud provider's load balancer implementation supports SCTP as a protocol. Otherwise the `Service` creation request is rejected. The current set of cloud load balancer providers (`Azure`, `AWS`, `CloudStack`, `GCE`, `OpenStack`) do not support SCTP.
|
||||
|
||||
#### Windows
|
||||
|
||||
SCTP is not supported on Windows based nodes.
|
||||
|
||||
#### Userspace kube-proxy
|
||||
|
||||
The kube-proxy does not support the management of SCTP associations when it is in userspace mode.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
@@ -124,6 +124,13 @@ Note that there can be at most one *default* storage class on a cluster, or
|
||||
a `PersistentVolumeClaim` without `storageClassName` explicitly specified cannot
|
||||
be created.
|
||||
|
||||
## Topology Awareness
|
||||
|
||||
In [Multi-Zone](/docs/setup/multiple-zones) clusters, Pods can be spread across
|
||||
Zones in a Region. Single-Zone storage backends should be provisioned in the Zones where
|
||||
Pods are scheduled. This can be accomplished by setting the [Volume Binding
|
||||
Mode](/docs/concepts/storage/storage-classes/#volume-binding-mode).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -631,6 +631,34 @@ Volume binding matrix for statically provisioned volumes:
|
||||
**Note:** Only statically provisioned volumes are supported for alpha release. Administrators should take care to consider these values when working with raw block devices.
|
||||
{{< /note >}}
|
||||
|
||||
## Volume Snapshot and Restore Volume from Snapshot Support
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
Volume snapshot feature was added to support CSI Volume Plugins only. For details, see [volume snapshots](/docs/concepts/storage/volume-snapshots/).
|
||||
|
||||
To enable support for restoring a volume from a volume snapshot data source, enable the
|
||||
`VolumeSnapshotDataSource` feature gate on the apiserver and controller-manager.
|
||||
|
||||
### Create Persistent Volume Claim from Volume Snapshot
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: restore-pvc
|
||||
spec:
|
||||
storageClassName: csi-hostpath-sc
|
||||
dataSource:
|
||||
name: new-snapshot-test
|
||||
kind: VolumeSnapshot
|
||||
apiGroup: snapshot.storage.k8s.io
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
## Writing Portable Configuration
|
||||
|
||||
If you're writing configuration templates or examples that run on a wide range of clusters
|
||||
|
||||
@@ -55,6 +55,7 @@ parameters:
|
||||
reclaimPolicy: Retain
|
||||
mountOptions:
|
||||
- debug
|
||||
volumeBindingMode: Immediate
|
||||
```
|
||||
|
||||
### Provisioner
|
||||
@@ -64,7 +65,7 @@ for provisioning PVs. This field must be specified.
|
||||
|
||||
| Volume Plugin | Internal Provisioner| Config Example |
|
||||
| :--- | :---: | :---: |
|
||||
| AWSElasticBlockStore | ✓ | [AWS](#aws) |
|
||||
| AWSElasticBlockStore | ✓ | [AWS EBS](#aws-ebs) |
|
||||
| AzureFile | ✓ | [Azure File](#azure-file) |
|
||||
| AzureDisk | ✓ | [Azure Disk](#azure-disk) |
|
||||
| CephFS | - | - |
|
||||
@@ -72,7 +73,7 @@ for provisioning PVs. This field must be specified.
|
||||
| FC | - | - |
|
||||
| Flexvolume | - | - |
|
||||
| Flocker | ✓ | - |
|
||||
| GCEPersistentDisk | ✓ | [GCE](#gce) |
|
||||
| GCEPersistentDisk | ✓ | [GCE PD](#gce-pd) |
|
||||
| Glusterfs | ✓ | [Glusterfs](#glusterfs) |
|
||||
| iSCSI | - | - |
|
||||
| Quobyte | ✓ | [Quobyte](#quobyte) |
|
||||
@@ -118,6 +119,74 @@ If the volume plugin does not support mount options but mount options are
|
||||
specified, provisioning will fail. Mount options are not validated on either
|
||||
the class or PV, so mount of the PV will simply fail if one is invalid.
|
||||
|
||||
### Volume Binding Mode
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="beta" >}}
|
||||
|
||||
**Note:** This feature requires the `VolumeScheduling` feature gate to be
|
||||
enabled.
|
||||
|
||||
The `volumeBindingMode` field controls when [volume binding and dynamic
|
||||
provisioning](/docs/concepts/storage/persistent-volumes/#provisioning) should occur.
|
||||
|
||||
By default, the `Immediate` mode indicates that volume binding and dynamic
|
||||
provisioning occurs once the PersistentVolumeClaim is created. For storage
|
||||
backends that are topology-constrained and not globally accessible from all Nodes
|
||||
in the cluster, PersistentVolumes will be bound or provisioned without knowledge of the Pod's scheduling
|
||||
requirements. This may result in unschedulable Pods.
|
||||
|
||||
A cluster administrator can address this issue by specifying the `WaitForFirstConsumer` mode which
|
||||
will delay the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.
|
||||
PersistentVolumes will be selected or provisioned conforming to the topology that is
|
||||
specified by the Pod's scheduling constraints. These include, but are not limited to, [resource
|
||||
requirements](/docs/concepts/configuration/manage-compute-resources-container),
|
||||
[node selectors](/docs/concepts/configuration/assign-pod-node/#nodeselector),
|
||||
[pod affinity and
|
||||
anti-affinity](/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity),
|
||||
and [taints and tolerations](/docs/concepts/configuration/taint-and-toleration).
|
||||
|
||||
The following plugins support `WaitForFirstConsumer` with dynamic provisioning:
|
||||
|
||||
* [AWSElasticBlockStore](#aws-ebs)
|
||||
* [GCEPersistentDisk](#gce-pd)
|
||||
* [AzureDisk](#azure-disk)
|
||||
|
||||
The following plugins support `WaitForFirstConsumer` with pre-created PersistentVolume binding:
|
||||
|
||||
* All of the above
|
||||
* [Local](#local)
|
||||
|
||||
### Allowed Topologies
|
||||
{{< feature-state for_k8s_version="v1.12" state="beta" >}}
|
||||
|
||||
**Note:** This feature requires the `VolumeScheduling` feature gate to be
|
||||
enabled.
|
||||
|
||||
When a cluster operactor specifies the `WaitForFirstConsumer` volume binding mode, it is no longer necessary
|
||||
to restrict provisioning to specific topologies in most situations. However,
|
||||
if still required, `allowedTopologies` can be specified.
|
||||
|
||||
This example demonstrates how to restrict the topology of provisioned volumes to specific
|
||||
zones and should be used as a replacement for the `zone` and `zones` parameters for the
|
||||
supported plugins.
|
||||
|
||||
```yaml
|
||||
kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: standard
|
||||
provisioner: kubernetes.io/gce-pd
|
||||
parameters:
|
||||
type: pd-standard
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
allowedTopologies:
|
||||
- matchLabelExpressions:
|
||||
- key: failure-domain.beta.kubernetes.io/zone
|
||||
values:
|
||||
- us-central1-a
|
||||
- us-central1-b
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
Storage classes have parameters that describe volumes belonging to the storage
|
||||
@@ -126,7 +195,7 @@ class. Different parameters may be accepted depending on the `provisioner`. For
|
||||
`iopsPerGB` are specific to EBS. When a parameter is omitted, some default is
|
||||
used.
|
||||
|
||||
### AWS
|
||||
### AWS EBS
|
||||
|
||||
```yaml
|
||||
kind: StorageClass
|
||||
@@ -136,7 +205,6 @@ metadata:
|
||||
provisioner: kubernetes.io/aws-ebs
|
||||
parameters:
|
||||
type: io1
|
||||
zones: us-east-1d, us-east-1c
|
||||
iopsPerGB: "10"
|
||||
fsType: ext4
|
||||
```
|
||||
@@ -144,10 +212,10 @@ parameters:
|
||||
* `type`: `io1`, `gp2`, `sc1`, `st1`. See
|
||||
[AWS docs](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
|
||||
for details. Default: `gp2`.
|
||||
* `zone`: AWS zone. If neither `zone` nor `zones` is specified, volumes are
|
||||
* `zone` (Deprecated): AWS zone. If neither `zone` nor `zones` is specified, volumes are
|
||||
generally round-robin-ed across all active zones where Kubernetes cluster
|
||||
has a node. `zone` and `zones` parameters must not be used at the same time.
|
||||
* `zones`: A comma separated list of AWS zone(s). If neither `zone` nor `zones`
|
||||
* `zones` (Deprecated): A comma separated list of AWS zone(s). If neither `zone` nor `zones`
|
||||
is specified, volumes are generally round-robin-ed across all active zones
|
||||
where Kubernetes cluster has a node. `zone` and `zones` parameters must not
|
||||
be used at the same time.
|
||||
@@ -164,7 +232,10 @@ parameters:
|
||||
encrypting the volume. If none is supplied but `encrypted` is true, a key is
|
||||
generated by AWS. See AWS docs for valid ARN value.
|
||||
|
||||
### GCE
|
||||
**Note:** `zone` and `zones` parameters are deprecated and replaced with
|
||||
[allowedTopologies](#allowed-topologies)
|
||||
|
||||
### GCE PD
|
||||
|
||||
```yaml
|
||||
kind: StorageClass
|
||||
@@ -174,15 +245,14 @@ metadata:
|
||||
provisioner: kubernetes.io/gce-pd
|
||||
parameters:
|
||||
type: pd-standard
|
||||
zones: us-central1-a, us-central1-b
|
||||
replication-type: none
|
||||
```
|
||||
|
||||
* `type`: `pd-standard` or `pd-ssd`. Default: `pd-standard`
|
||||
* `zone`: GCE zone. If neither `zone` nor `zones` is specified, volumes are
|
||||
* `zone` (Deprecated): GCE zone. If neither `zone` nor `zones` is specified, volumes are
|
||||
generally round-robin-ed across all active zones where Kubernetes cluster has
|
||||
a node. `zone` and `zones` parameters must not be used at the same time.
|
||||
* `zones`: A comma separated list of GCE zone(s). If neither `zone` nor `zones`
|
||||
* `zones` (Deprecated): A comma separated list of GCE zone(s). If neither `zone` nor `zones`
|
||||
is specified, volumes are generally round-robin-ed across all active zones
|
||||
where Kubernetes cluster has a node. `zone` and `zones` parameters must not
|
||||
be used at the same time.
|
||||
@@ -199,6 +269,9 @@ specified, Kubernetes will arbitrarily choose among the specified zones. If the
|
||||
`zones` parameter is omitted, Kubernetes will arbitrarily choose among zones
|
||||
managed by the cluster.
|
||||
|
||||
**Note:** `zone` and `zones` parameters are deprecated and replaced with
|
||||
[allowedTopologies](#allowed-topologies)
|
||||
|
||||
### Glusterfs
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -11,11 +11,11 @@ content_template: templates/concept
|
||||
{{% capture overview %}}
|
||||
|
||||
This page describes the maximum number of volumes that can be attached
|
||||
to a node for various cloud providers.
|
||||
to a Node for various cloud providers.
|
||||
|
||||
Cloud providers like Google, Amazon, and Microsoft typically have a limit on
|
||||
how many volumes can be attached to a node. It is important for Kubernetes to
|
||||
respect those limits. Otherwise, Pods scheduled on a node could get stuck
|
||||
how many volumes can be attached to a Node. It is important for Kubernetes to
|
||||
respect those limits. Otherwise, Pods scheduled on a Node could get stuck
|
||||
waiting for volumes to attach.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -25,10 +25,10 @@ waiting for volumes to attach.
|
||||
## Kubernetes default limits
|
||||
|
||||
The Kubernetes scheduler has default limits on the number of volumes
|
||||
that can be attached to a node:
|
||||
that can be attached to a Node:
|
||||
|
||||
<table>
|
||||
<tr><th>Cloud service</th><th>Maximum volumes per node</th></tr>
|
||||
<tr><th>Cloud service</th><th>Maximum volumes per Node</th></tr>
|
||||
<tr><td><a href="https://aws.amazon.com/ebs/">Amazon Elastic Block Store (EBS)</a></td><td>39</td></tr>
|
||||
<tr><td><a href="https://cloud.google.com/persistent-disk/">Google Persistent Disk</a></td><td>16</td></tr>
|
||||
<tr><td><a href="https://azure.microsoft.com/en-us/services/storage/main-disks/">Microsoft Azure Disk Storage</a></td><td>16</td></tr>
|
||||
@@ -40,28 +40,28 @@ You can change these limits by setting the value of the
|
||||
`KUBE_MAX_PD_VOLS` environment variable, and then starting the scheduler.
|
||||
|
||||
Use caution if you set a limit that is higher than the default limit. Consult
|
||||
the cloud provider's documentation to make sure that nodes can actually support
|
||||
the cloud provider's documentation to make sure that Nodes can actually support
|
||||
the limit you set.
|
||||
|
||||
The limit applies to the entire cluster, so it affects all nodes.
|
||||
The limit applies to the entire cluster, so it affects all Nodes.
|
||||
|
||||
## Dynamic volume limits
|
||||
|
||||
{{< feature-state state="alpha" for_k8s_version="v1.11" >}}
|
||||
{{< feature-state state="beta" for_k8s_version="v1.12" >}}
|
||||
|
||||
Kubernetes 1.11 introduces dynamic volume limits based on node type. This
|
||||
is an alpha feature that supports these services:
|
||||
Kubernetes 1.11 introduced support for dynamic volume limits based on Node type as an Alpha feature.
|
||||
In Kubernetes 1.12 this feature is graduating to Beta and will be enabled by default.
|
||||
|
||||
Dynamic volume limits is supported for following volume types.
|
||||
|
||||
- Amazon EBS
|
||||
- Google Persistent Disk
|
||||
- Azure Disk
|
||||
- CSI
|
||||
|
||||
To enable dynamic volume limits, set the `AttachVolumeLimit`
|
||||
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
||||
to True.
|
||||
|
||||
When the dynamic volume limits feature is enabled, Kubernetes automatically
|
||||
determines the node type and supports the appropriate number of attachable
|
||||
determines the Node type and enforces the appropriate number of attachable
|
||||
volumes for the node. For example:
|
||||
|
||||
* On
|
||||
@@ -69,11 +69,14 @@ volumes for the node. For example:
|
||||
up to 128 volumes can be attached to a node, [depending on the node
|
||||
type](https://cloud.google.com/compute/docs/disks/#pdnumberlimits).
|
||||
|
||||
* For Amazon EBS disks on M5/C5 instance types, Kubernetes allows only 25
|
||||
volumes to be attached to a node. For other instance types on
|
||||
* For Amazon EBS disks on M5,C5,R5,T3 and Z1D instance types, Kubernetes allows only 25
|
||||
volumes to be attached to a Node. For other instance types on
|
||||
<a href="https://aws.amazon.com/ec2/">Amazon Elastic Compute Cloud (EC2)</a>,
|
||||
Kubernetes allows 39 volumes to be attached to a node.
|
||||
Kubernetes allows 39 volumes to be attached to a Node.
|
||||
|
||||
* On Azure, up to 64 disks can be attached to a node, depending on the node type. For more details, refer to [Sizes for virtual machines in Azure](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes).
|
||||
|
||||
* For CSI, any driver that advertises volume attach limits via CSI specs will have those limits available as the Node's allocatable property
|
||||
and the Scheduler will not schedule Pods with volumes on any Node that is already at its capacity. Refer to the [CSI specs](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfo) for more details.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
reviewers:
|
||||
- jsafrane
|
||||
- saad-ali
|
||||
- thockin
|
||||
- msau42
|
||||
title: Volume Snapshot Classes
|
||||
content_template: templates/concept
|
||||
weight: 30
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
This document describes the concept of `VolumeSnapshotClass` in Kubernetes. Familiarity
|
||||
with [volume snapshots](/docs/concepts/storage/volume-snapshots/) and
|
||||
[storage classes](/docs/concepts/storage/storage-classes) is suggested.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Introduction
|
||||
|
||||
Just like `StorageClass` provides a way for administrators to describe the "classes"
|
||||
of storage they offer when provisioning a volume, `VolumeSnapshotClass` provides a
|
||||
way to describe the "classes" of storage when provisioning a volume snapshot.
|
||||
|
||||
## The VolumeSnapshotClass Resource
|
||||
|
||||
Each `VolumeSnapshotClass` contains the fields `snapshotter` and `parameters`,
|
||||
which are used when a `VolumeSnapshot` belonging to the class needs to be
|
||||
dynamically provisioned.
|
||||
|
||||
The name of a `VolumeSnapshotClass` object is significant, and is how users can
|
||||
request a particular class. Administrators set the name and other parameters
|
||||
of a class when first creating `VolumeSnapshotClass` objects, and the objects cannot
|
||||
be updated once they are created.
|
||||
|
||||
Administrators can specify a default `VolumeSnapshotClass` just for VolumeSnapshots
|
||||
that don't request any particular class to bind to.
|
||||
|
||||
```yaml
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshotClass
|
||||
metadata:
|
||||
name: csi-hostpath-snapclass
|
||||
snapshotter: csi-hostpath
|
||||
parameters:
|
||||
```
|
||||
|
||||
### Snapshotter
|
||||
|
||||
Volume snapshot classes have a snapshotter that determines what CSI volume plugin is
|
||||
used for provisioning VolumeSnapshots. This field must be specified.
|
||||
|
||||
## Parameters
|
||||
|
||||
Volume snapshot classes have parameters that describe volume snapshots belonging to
|
||||
the volume snapshot class. Different parameters may be accepted depending on the
|
||||
`snapshotter`.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
reviewers:
|
||||
- jsafrane
|
||||
- saad-ali
|
||||
- thockin
|
||||
- msau42
|
||||
title: Volume Snapshots
|
||||
content_template: templates/concept
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
This document describes the current state of `VolumeSnapshots` in Kubernetes. Familiarity with [persistent volumes](/docs/concepts/storage/persistent-volumes/) is suggested.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Introduction
|
||||
|
||||
Similar to how API resources `PersistentVolume` and `PersistentVolumeClaim` are used to provision volumes for users and administrators, `VolumeSnapshotContent` and `VolumeSnapshot` API resources are provided to create volume snapshots for users and administrators.
|
||||
|
||||
A `VolumeSnapshotContent` is a snapshot taken from a volume in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a PersistentVolume is a cluster resource.
|
||||
|
||||
A `VolumeSnapshot` is a request for snapshot of a volume by a user. It is similar to a PersistentVolumeClaim.
|
||||
|
||||
While `VolumeSnapshots` allow a user to consume abstract storage resources, cluster administrators
|
||||
need to be able to offer a variety of `VolumeSnapshotContents` without exposing
|
||||
users to the details of how those volume snapshots should be provisioned. For these needs
|
||||
there is the `VolumeSnapshotClass` resource.
|
||||
|
||||
Users need to be aware of the following when using this feature:
|
||||
|
||||
* API Objects `VolumeSnapshot`, `VolumeSnapshotContent`, and `VolumeSnapshotClass` are CRDs, not part of the core API.
|
||||
* `VolumeSnapshot` support is only available for CSI drivers.
|
||||
* As part of the deployment process, the Kubernetes team provides a sidecar helper container for the snapshot controller called `external-snapshotter`. It watches `VolumeSnapshot` objects and triggers `CreateSnapshot` and `DeleteSnapshot` operations against a CSI endpoint.
|
||||
* CSI drivers may or may not have implemented the volume snapshot functionality. The CSI drivers that have provided support for volume snapshot will likely use `external-snapshotter`.
|
||||
* The CSI drivers that support volume snapshot will automatically install CRDs defined for the volume snapshots.
|
||||
|
||||
## Lifecycle of a volume snapshot and volume snapshot content
|
||||
|
||||
`VolumeSnapshotContents` are resources in the cluster. `VolumeSnapshots` are requests for those resources. The interaction between `VolumeSnapshotContents` and `VolumeSnapshots` follow this lifecycle:
|
||||
|
||||
### Provisioning Volume Snapshot
|
||||
|
||||
There are two ways snapshots may be provisioned: statically or dynamically.
|
||||
|
||||
#### Static
|
||||
A cluster administrator creates a number of `VolumeSnapshotContents`. They carry the details of the real storage which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
|
||||
|
||||
#### Dynamic
|
||||
When none of the static `VolumeSnapshotContents` the administrator created matches a user's `VolumeSnapshot`,
|
||||
the cluster may try to dynamically provision a volume snapshot specially for the `VolumeSnapshot` object.
|
||||
This provisioning is based on `VolumeSnapshotClasses`: the `VolumeSnapshot` must request a
|
||||
[volume snapshot class](/docs/concepts/storage/volume-snapshot-classes/) and
|
||||
the administrator must have created and configured that class in order for dynamic
|
||||
provisioning to occur.
|
||||
|
||||
### Binding
|
||||
|
||||
A user creates, or has already created in the case of dynamic provisioning, a `VolumeSnapshot` with a specific amount of storage requested and with certain access modes. A control loop watches for new VolumeSnapshots, finds a matching VolumeSnapshotContent (if possible), and binds them together. If a VolumeSnapshotContent was dynamically provisioned for a new VolumeSnapshot, the loop will always bind that VolumeSnapshotContent to the VolumeSnapshot. Once bound, `VolumeSnapshot` binds are exclusive, regardless of how they were bound. A VolumeSnapshot to VolumeSnapshotContent binding is a one-to-one mapping.
|
||||
|
||||
VolumeSnapshots will remain unbound indefinitely if a matching VolumeSnapshotContent does not exist. VolumeSnapshots will be bound as matching VolumeSnapshotContents become available.
|
||||
|
||||
### Delete
|
||||
|
||||
Deletion removes both the `VolumeSnapshotContent` object from the Kubernetes API, as well as the associated storage asset in the external infrastructure.
|
||||
|
||||
## Volume Snapshot Contents
|
||||
|
||||
Each VolumeSnapshotContent contains a spec, which is the specification of the volume snapshot.
|
||||
|
||||
```yaml
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshotContent
|
||||
metadata:
|
||||
name: new-snapshot-content-test
|
||||
spec:
|
||||
snapshotClassName: csi-hostpath-snapclass
|
||||
source:
|
||||
name: pvc-test
|
||||
kind: PersistentVolumeClaim
|
||||
volumeSnapshotSource:
|
||||
csiVolumeSnapshotSource:
|
||||
creationTime: 1535478900692119403
|
||||
driver: csi-hostpath
|
||||
restoreSize: 10Gi
|
||||
snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002
|
||||
```
|
||||
|
||||
### Class
|
||||
|
||||
A VolumeSnapshotContent can have a class, which is specified by setting the
|
||||
`snapshotClassName` attribute to the name of a
|
||||
[VolumeSnapshotClass](/docs/concepts/storage/volume-snapshot-classes/).
|
||||
A VolumeSnapshotContent of a particular class can only be bound to VolumeSnapshots requesting
|
||||
that class. A VolumeSnapshotContent with no `snapshotClassName` has no class and can only be bound
|
||||
to VolumeSnapshots that request no particular class.
|
||||
|
||||
## VolumeSnapshots
|
||||
|
||||
Each VolumeSnapshot contains a spec and a status, which is the specification and status of the volume snapshot.
|
||||
|
||||
```yaml
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshot
|
||||
metadata:
|
||||
name: new-snapshot-test
|
||||
spec:
|
||||
snapshotClassName: csi-hostpath-snapclass
|
||||
source:
|
||||
name: pvc-test
|
||||
kind: PersistentVolumeClaim
|
||||
```
|
||||
|
||||
### Class
|
||||
|
||||
A volume snapshot can request a particular class by specifying the name of a
|
||||
[VolumeSnapshotClass](/docs/concepts/storage/volume-snapshot-classes/)
|
||||
using the attribute `snapshotClassName`.
|
||||
Only VolumeSnapshotContents of the requested class, ones with the same `snapshotClassName`
|
||||
as the VolumeSnapshot, can be bound to the VolumeSnapshot.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -656,7 +656,8 @@ All sources are required to be in the same namespace as the Pod. For more detail
|
||||
see the [all-in-one volume design document](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/node/all-in-one-volume.md).
|
||||
|
||||
The projection of service account tokens is a feature introduced in Kubernetes
|
||||
1.11. To enable this feature, you need to explicitly set the `TokenRequestProjection`
|
||||
1.11 and promoted to Beta in 1.12.
|
||||
To enable this feature on 1.11, you need to explicitly set the `TokenRequestProjection`
|
||||
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to
|
||||
True.
|
||||
|
||||
@@ -1236,16 +1237,9 @@ More details can be found [here](https://github.com/kubernetes/community/blob/ma
|
||||
|
||||
## Mount propagation
|
||||
|
||||
{{< feature-state for_k8s_version="v1.10" state="beta" >}}
|
||||
|
||||
Mount propagation allows for sharing volumes mounted by a Container to
|
||||
other Containers in the same Pod, or even to other Pods on the same node.
|
||||
|
||||
If the "`MountPropagation`" feature is disabled or a Pod does not explicitly
|
||||
specify specific mount propagation, volume mounts in the Pod's Containers are
|
||||
not propagated. That is, Containers run with `private` mount propagation as
|
||||
described in the [Linux kernel documentation](https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt).
|
||||
|
||||
Mount propagation of a volume is controlled by `mountPropagation` field in Container.volumeMounts.
|
||||
Its values are:
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ If you do not specify either, then the DaemonSet controller will create Pods on
|
||||
|
||||
## How Daemon Pods are Scheduled
|
||||
|
||||
### Scheduled by DaemonSet controller (default)
|
||||
### Scheduled by DaemonSet controller (disabled by default since 1.12)
|
||||
|
||||
Normally, the machine that a Pod runs on is selected by the Kubernetes scheduler. However, Pods
|
||||
created by the DaemonSet controller have the machine already selected (`.spec.nodeName` is specified
|
||||
@@ -115,9 +115,9 @@ when the Pod is created, so it is ignored by the scheduler). Therefore:
|
||||
bootstrap.
|
||||
|
||||
|
||||
### Scheduled by default scheduler
|
||||
### Scheduled by default scheduler (enabled by default since 1.12)
|
||||
|
||||
{{< feature-state state="alpha" for-kubernetes-version="1.11" >}}
|
||||
{{< feature-state state="beta" for-kubernetes-version="1.12" >}}
|
||||
|
||||
A DaemonSet ensures that all eligible nodes run a copy of a Pod. Normally, the
|
||||
node that a Pod runs on is selected by the Kubernetes scheduler. However,
|
||||
@@ -151,14 +151,8 @@ nodeAffinity:
|
||||
```
|
||||
|
||||
In addition, `node.kubernetes.io/unschedulable:NoSchedule` toleration is added
|
||||
automatically to DaemonSet Pods. The DaemonSet controller ignores
|
||||
`unschedulable` Nodes when scheduling DaemonSet Pods. You must enable
|
||||
`TaintNodesByCondition` to ensure that the default scheduler behaves the same
|
||||
way and schedules DaemonSet pods on `unschedulable` nodes.
|
||||
|
||||
When this feature and `TaintNodesByCondition` are enabled together, if DaemonSet
|
||||
uses the host network, you must also add the
|
||||
`node.kubernetes.io/network-unavailable:NoSchedule toleration`.
|
||||
automatically to DaemonSet Pods. The default scheduler ignores
|
||||
`unschedulable` Nodes when scheduling DaemonSet Pods.
|
||||
|
||||
|
||||
### Taints and Tolerations
|
||||
@@ -170,13 +164,12 @@ the related features.
|
||||
|
||||
| Toleration Key | Effect | Alpha Features | Version | Description |
|
||||
| ---------------------------------------- | ---------- | ------------------------------------------------------------ | ------- | ------------------------------------------------------------ |
|
||||
| `node.kubernetes.io/not-ready` | NoExecute | `TaintBasedEvictions` | 1.8+ | when `TaintBasedEvictions` is enabled,they will not be evicted when there are node problems such as a network partition. |
|
||||
| `node.kubernetes.io/unreachable` | NoExecute | `TaintBasedEvictions` | 1.8+ | when `TaintBasedEvictions` is enabled,they will not be evicted when there are node problems such as a network partition. |
|
||||
| `node.kubernetes.io/disk-pressure` | NoSchedule | `TaintNodesByCondition` | 1.8+ | |
|
||||
| `node.kubernetes.io/memory-pressure` | NoSchedule | `TaintNodesByCondition` | 1.8+ | |
|
||||
| `node.kubernetes.io/unschedulable` | NoSchedule | `ScheduleDaemonSetPods`, `TaintNodesByCondition` | 1.11+ | When ` ScheduleDaemonSetPods` is enabled, ` TaintNodesByCondition` is necessary to make sure DaemonSet pods tolerate unschedulable attributes by default scheduler. |
|
||||
| `node.kubernetes.io/network-unavailable` | NoSchedule | `ScheduleDaemonSetPods`, `TaintNodesByCondition`, hostnework | 1.11+ | When ` ScheduleDaemonSetPods` is enabled, ` TaintNodesByCondition` is necessary to make sure DaemonSet pods, who uses host network, tolerate network-unavailable attributes by default scheduler. |
|
||||
| `node.kubernetes.io/out-of-disk` | NoSchedule | `ExperimentalCriticalPodAnnotation` (critical pod only), `TaintNodesByCondition` | 1.8+ | |
|
||||
| `node.kubernetes.io/not-ready` | NoExecute | `TaintBasedEvictions` | 1.8+ | When `TaintBasedEvictions` is enabled, they will not be evicted when there are node problems such as a network partition. |
|
||||
| `node.kubernetes.io/unreachable` | NoExecute | `TaintBasedEvictions` | 1.8+ | When `TaintBasedEvictions` is enabled, they will not be evicted when there are node problems such as a network partition. |
|
||||
| `node.kubernetes.io/disk-pressure` | NoSchedule | | 1.8+ | |
|
||||
| `node.kubernetes.io/memory-pressure` | NoSchedule | | 1.8+ | |
|
||||
| `node.kubernetes.io/unschedulable` | NoSchedule | | 1.12+ | DaemonSet pods tolerate unschedulable attributes by default scheduler. |
|
||||
| `node.kubernetes.io/network-unavailable` | NoSchedule | | 1.12+ | DaemonSet pods, who uses host network, tolerate network-unavailable attributes by default scheduler. |
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -251,6 +251,59 @@ spec:
|
||||
|
||||
Note that both the Job Spec and the [Pod Template Spec](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#detailed-behavior) within the Job have an `activeDeadlineSeconds` field. Ensure that you set this field at the proper level.
|
||||
|
||||
## Clean Up Finished Jobs Automatically
|
||||
|
||||
Finished Jobs are usually no longer needed in the system. Keeping them around in
|
||||
the system will put pressure on the API server. If the Jobs are managed directly
|
||||
by a higher level controller, such as
|
||||
[CronJobs](/docs/concepts/workloads/controllers/cron-jobs/), the Jobs can be
|
||||
cleaned up by CronJobs based on the specified capacity-based cleanup policy.
|
||||
|
||||
### TTL Mechanism for Finished Jobs
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
Another way to clean up finished Jobs (either `Complete` or `Failed`)
|
||||
automatically is to use a TTL mechanism provided by a
|
||||
[TTL controller](/docs/concepts/workloads/controllers/ttlafterfinished/) for
|
||||
finished resources, by specifying the `.spec.ttlSecondsAfterFinished` field of
|
||||
the Job.
|
||||
|
||||
When the TTL controller cleans up the Job, it will delete the Job cascadingly,
|
||||
i.e. delete its dependent objects, such as Pods, together with the Job. Note
|
||||
that when the Job is deleted, its lifecycle guarantees, such as finalizers, will
|
||||
be honored.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: pi-with-ttl
|
||||
spec:
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 100
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: pi
|
||||
image: perl
|
||||
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
|
||||
restartPolicy: Never
|
||||
```
|
||||
|
||||
The Job `pi-with-ttl` will be eligible to be automatically deleted, `100`
|
||||
seconds after it finishes.
|
||||
|
||||
If the field is set to `0`, the Job will be eligible to be automatically deleted
|
||||
immediately after it finishes. If the field is unset, this Job won't be cleaned
|
||||
up by the TTL controller after it finishes.
|
||||
|
||||
Note that this TTL mechanism is alpha, with feature gate `TTLAfterFinished`. For
|
||||
more information, see the documentation for
|
||||
[TTL controller](/docs/concepts/workloads/controllers/ttlafterfinished/) for
|
||||
finished resources.
|
||||
|
||||
## Job Patterns
|
||||
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
reviewers:
|
||||
- janetkuo
|
||||
title: TTL Controller for Finished Resources
|
||||
content_template: templates/concept
|
||||
weight: 65
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
The TTL controller provides a TTL mechanism to limit the lifetime of resource
|
||||
objects that have finished execution. TTL controller only handles
|
||||
[Jobs](/docs/concepts/workloads/controllers/jobs-run-to-completion/) for
|
||||
now, and may be expanded to handle other resources that will finish execution,
|
||||
such as Pods and custom resources.
|
||||
|
||||
Alpha Disclaimer: this feature is currently alpha, and can be enabled with
|
||||
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
||||
`TTLAfterFinished`.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## TTL Controller
|
||||
|
||||
The TTL controller only supports Jobs for now. A cluster operator can use this feature to clean
|
||||
up finished Jobs (either `Complete` or `Failed`) automatically by specifying the
|
||||
`.spec.ttlSecondsAfterFinished` field of a Job, as in this
|
||||
[example](/docs/concepts/workloads/controllers/jobs-run-to-completion/#clean-up-finished-jobs-automatically).
|
||||
The TTL controller will assume that a resource is eligible to be cleaned up
|
||||
TTL seconds after the resource has finished, in other words, when the TTL has expired. When the
|
||||
TTL controller cleans up a resource, it will delete it cascadingly, i.e. delete
|
||||
its dependent objects together with it. Note that when the resource is deleted,
|
||||
its lifecycle guarantees, such as finalizers, will be honored.
|
||||
|
||||
The TTL seconds can be set at any time. Here are some examples for setting the
|
||||
`.spec.ttlSecondsAfterFinished` field of a Job:
|
||||
|
||||
* Specify this field in the resource manifest, so that a Job can be cleaned up
|
||||
automatically some time after it finishes.
|
||||
* Set this field of existing, already finished resources, to adopt this new
|
||||
feature.
|
||||
* Use a
|
||||
[mutating admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks)
|
||||
to set this field dynamically at resource creation time. Cluster administrators can
|
||||
use this to enforce a TTL policy for finished resources.
|
||||
* Use a
|
||||
[mutating admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks)
|
||||
to set this field dynamically after the resource has finished, and choose
|
||||
different TTL values based on resource status, labels, etc.
|
||||
|
||||
## Caveat
|
||||
|
||||
### Updating TTL Seconds
|
||||
|
||||
Note that the TTL period, e.g. `.spec.ttlSecondsAfterFinished` field of Jobs,
|
||||
can be modified after the resource is created or has finished. However, once the
|
||||
Job becomes eligible to be deleted (when the TTL has expired), the system won't
|
||||
guarantee that the Jobs will be kept, even if an update to extend the TTL
|
||||
returns a successful API response.
|
||||
|
||||
### Time Skew
|
||||
|
||||
Because TTL controller uses timestamps stored in the Kubernetes resources to
|
||||
determine whether the TTL has expired or not, this feature is sensitive to time
|
||||
skew in the cluster, which may cause TTL controller to clean up resource objects
|
||||
at the wrong time.
|
||||
|
||||
In Kubernetes, it's required to run NTP on all nodes
|
||||
(see [#6159](https://github.com/kubernetes/kubernetes/issues/6159#issuecomment-93844058))
|
||||
to avoid time skew. Clocks aren't always correct, but the difference should be
|
||||
very small. Please be aware of this risk when setting a non-zero TTL.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
[Clean up Jobs automatically](/docs/concepts/workloads/controllers/jobs-run-to-completion/#clean-up-finished-jobs-automatically)
|
||||
|
||||
[Design doc](https://github.com/kubernetes/community/blob/master/keps/sig-apps/0026-ttl-after-finish.md)
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -157,7 +157,7 @@ Note that the information reported as Pod status depends on the current
|
||||
|
||||
## Pod readiness gate
|
||||
|
||||
{{< feature-state for_k8s_version="v1.11" state="alpha" >}}
|
||||
{{< feature-state for_k8s_version="v1.12" state="beta" >}}
|
||||
|
||||
In order to add extensibility to Pod readiness by enabling the injection of
|
||||
extra feedbacks or signals into `PodStatus`, Kubernetes 1.11 introduced a
|
||||
@@ -203,9 +203,11 @@ when both the following statements are true:
|
||||
To facilitate this change to Pod readiness evaluation, a new Pod condition
|
||||
`ContainersReady` is introduced to capture the old Pod `Ready` condition.
|
||||
|
||||
As an alpha feature, the "Pod Ready++" feature has to be explicitly enabled by
|
||||
In K8s 1.11, as an alpha feature, the "Pod Ready++" feature has to be explicitly enabled by
|
||||
setting the `PodReadinessGates` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
||||
to True.
|
||||
to true.
|
||||
|
||||
In K8s 1.12, the feature is enabled by default.
|
||||
|
||||
## Restart policy
|
||||
|
||||
|
||||
Reference in New Issue
Block a user