Release 1.8 (#5659)
* GC now supports non-core resources * Add two examples about how to analysis audits of kube-apiserver (#4264) * Deprecate system:nodes binding * [1.8] StatefulSet `initialized` annotation is now ignored. * inits the kubeadm upgrade docs addresses kubernetes/kubernetes.github.io/issues/4689 * adds kubeadm upgrade cmd to ToC addresses kubernetes/kubernetes.github.io/issues/4689 * add workload placement docs * ScaleIO - document udpate for 1.8 * Add documentation on storageClass.mountOptions and PV.mountOptions (#5254) * Add documentation on storageClass.mountOptions and PV.mountOptions * convert notes into callouts * Add docs for CustomResource validation add info about supported fields * advanced audit beta features (#5300) * Update job workload doc with backoff failure policy (#5319) Add to the Jobs documentation how to use the new backoffLimit field that limit the number of Pod failure before considering the Job as failed. * Documented additional AWS Service annotations (#4864) * Add device plugin doc under concepts/cluster-administration. (#5261) * Add device plugin doc under concepts/cluster-administration. * Update device-plugins.md * Update device-plugins.md Add meta description. Fix typo. Change bare metal deployment to manual deployment. * Update device-plugins.md Fix typo again. * Update page.version. (#5341) * Add documentation on storageClass.reclaimPolicy (#5171) * [Advanced audit] use new herf for audit-api (#5349) This tag contains all the changes in v1beta1 version. Update it now. * Added documentation around creating the InitializerConfiguration for the persistent volume label controller in the cloud-controller-manager (#5255) * Documentation for kubectl plugins (#5294) * Documentation for kubectl plugins * Update kubectl-plugins.md * Update kubectl-plugins.md * Updated CPU manager docs to match implementation. (#5332) * Noted limitation of alpha static cpumanager. * Updated CPU manager docs to match implementation. - Removed references to CPU pressure node condition and evictions. - Added note about new --cpu-manager-reconcile-period flag. - Added note about node allocatable requirements for static policy. - Noted limitation of alpha static cpumanager. * Move cpu-manager task link to rsc mgmt section. * init containers annotation removed in 1.8 (#5390) * Add documentation for TaintNodesByCondition (#5352) * Add documentation for TaintNodesByCondition * Update nodes.md * Update taint-and-toleration.md * Update daemonset.md * Update nodes.md * Update taint-and-toleration.md * Update daemonset.md * Fix deployments (#5421) * Document extended resources and OIR deprecation. (#5399) * Document extended resources and OIR deprecation. * Updated extended resources doc per reviews. * reverts extra spacing in _data/tasks.yml * addresses `kubeadm upgrade` review comments Feedback from @chenopis, @luxas, and @steveperry-53 addressed with this commit * HugePages documentation (#5419) * Update cpu-management-policies.md (#5407) Fixed the bad link. Modified "cpu" to "CPU". Added more 'yaml' as supplement. * Update RBAC docs for v1 (#5445) * Add user docs for pod priority and preemption (#5328) * Add user docs for pod priority and preemption * Update pod-priority-preemption.md * More updates * Update docs/admin/kubeadm.md for 1.8 (#5440) - Made a couple of minor wording changes (not strictly 1.8 related). - Did some reformatting (not strictly 1.8 related). - Updated references to the default token TTL (was infinite, now 24 hours). - Documented the new `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join`. - Added references to the new `--discovery-token-ca-cert-hash` flag in all the default examples. - Added a new _Security model_ section that describes the security tradeoffs of the various discovery modes. - Documented the new `--groups` flag for `kubeadm token create`. - Added a note of caution under _Automating kubeadm_ that references the _Security model_ section. - Updated the component version table to drop 1.6 and add 1.8. - Update `_data/reference.yml` to try to get the sidebar fixed up and more consistent with `kubefed`. * Update StatefulSet Basics for 1.8 release (#5398) * addresses `kubeadm upgrade` review comments 2nd iteration review comments by @luxas * adds kubelet upgrade section to kubeadm upgrade * Fix a bulleted list on docs/admin/kubeadm.md. (#5458) I updated this doc yesterday and I was absolutely sure I fixed this, but I just saw that this commit got lost somehow. This was introduced recently in https://github.com/kubernetes/kubernetes.github.io/pull/5440. * Clarify the API to check for device plugins * Moving Flexvolume to separate out-of-tree section * addresses `kubeadm upgrade` review comments CC: @luxas * fixes kubeadm upgrade index * Update Stackdriver Logging documentation (#5495) * Re-update WordPress and MySQL PV doc to use apps/v1beta2 APIs (#5526) * Update statefulset concepts doc to use apps/v1beta2 APIs (#5420) * add document on kubectl's behavior regarding initializers (#5505) * Update docs/admin/kubeadm.md to cover self-hosting in 1.8. (#5497) This is a new beta feature in 1.8. * Update kubectl patch doc to use apps/v1beta2 APIs (#5422) * [1.8] Update "Run Applications" tasks to apps/v1beta2. (#5525) * Update replicated stateful application task for 1.8. * Update single instance stateful app task for 1.8. * Update stateless app task for 1.8. * Update kubectl patch task for 1.8. * fix the link of persistent storage (#5515) * update the admission-controllers.md index.md what-is-kubernetes.md link * fix the link of persistent storage * Add quota support for local ephemeral storage (#5493) * Add quota support for local ephemeral storage update the doc to this alpha feature * Update resource-quotas.md * Updated Deployments concepts doc (#5491) * Updated Deployments concepts doc * Addressed comments * Addressed more comments * Modify allocatable storage to ephemeral-storage (#5490) Update the doc to use ephemeral-storage instead of storage * Revamped concepts doc for ReplicaSet (#5463) * Revamped concepts doc for ReplicaSet * Minor changes to call out specific versions for selector defaulting and immutability * Addressed doc review comments * Remove petset documentations (#5395) * Update docs to use batch/v1beta1 cronjobs (#5475) * add federation job doc (#5485) * add federation job doc * Update job.md Edits for clarity and consistency * Update job.md Fixed a typo * update DaemonSet concept for 1.8 release (#5397) * update DaemonSet concept for 1.8 release * Update daemonset.md Fix typo. than -> then * Update bootstrap tokens doc for 1.8. (#5479) * Update bootstrap tokens doc for 1.8. This has some changes I missed when I was updating the main kubeadm documention: - Bootstrap tokens are now beta, not alpha (https://github.com/kubernetes/features/issues/130) - The apiserver flag to enable the authenticator changedin 1.8 (https://github.com/kubernetes/kubernetes/pull/51198) - Added `auth-extra-groups` documentaion (https://github.com/kubernetes/kubernetes/pull/50933) - Updated the _Token Management with `kubeadm`_ section to link to the main kubeadm docs, since it was just duplicated information. * Update bootstrap-tokens.md * Updated the Cassandra tutorial to use apps/v1beta2 (#5548) * add docs for AllowPrivilegeEscalation (#5448) Signed-off-by: Jess Frazelle <acidburn@microsoft.com> * Add local ephemeral storage alpha feature in managing compute resource (#5522) * Add local ephemeral storage alpha feature in managing compute resource Since 1.8, we add the local ephemeral storage alpha feature as one resource type to manage. Add this feature into the doc. * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Added documentation for Metrics Server (#5560) * authorization: improve authorization debugging docs (#5549) * Document mount propagation (#5544) * Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. (#5524) This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in https://github.com/kubernetes/kubernetes/pull/49520 and some version bumps. * Add task doc for alpha dynamic kubelet configuration (#5523) * Fix input/output of selfsubjectaccess review (#5593) * Add docs for implementing resize (#5528) * Add docs for implementing resize * Update admission-controllers.md * Added link to PVC section * minor typo fixes * Update NetworkPolicy concept guide with egress and CIDR changes (#5529) * update zookeeper tutorial for 1.8 release * add doc for hostpath type (#5503) * Federated Hpa feature doc (#5487) * Federated Hpa feature doc * Federated Hpa feature doc review fixes * Update hpa.md * Update hpa.md * update cloud controller manager docs for v1.8 * Update cronjob with defaults information (#5556) * Kubernetes 1.8 reference docs (#5632) * Kubernetes 1.8 reference docs * Kubectl reference docs for 1.8 * Update side bar with 1.8 kubectl and api ref docs links * remove petset.md * update on state of HostAlias in 1.8 with hostNetwork Pod support (#5644) * Fix cron job deletion section (#5655) * update imported docs (#5656) * Add documentation for certificate rotation. (#5639) * Link to using kubeadm page * fix the command output fix the command output * fix typo in api/resources reference: "Worloads" * Add documentation for certificate rotation. * Create TOC entry for cloud controller manager. (#5662) * Updates for new versions of API types * Followup 5655: fix link to garbage collection (#5666) * Temporarily redirect resources-reference to api-reference. (#5668) * Update config for 1.8 release. (#5661) * Update config for 1.8 release. * Address reviewer comments. * Switch references in HPA docs from alpha to beta (#5671) The HPA docs still referenced the alpha version. This switches them to talk about v2beta1, which is the appropriate version for Kubernetes 1.8 * Deprecate openstack heat (#5670) * Fix typo in pod preset conflict example Move container port definition to the correct line. * Highlight openstack-heat provider deprecation The openstack-heat provider for kube-up is being deprecated and will be removed in a future release. * Temporarily fix broken links by redirecting. (#5672) * Fix broken links. (#5675) * Fix render of code block (#5674) * Fix broken links. (#5677) * Add a small note about auto-bootstrapped CSR ClusterRoles (#5660) * Update kubeadm install doc for v1.8 (#5676) * add draft workloads api content for 1.8 (#5650) * add draft workloads api content for 1.8 * edits per review, add tables, for 1.8 workloads api doc * fix typo * Minor fixes to kubeadm 1.8 upgrade guide. (#5678) - The kubelet upgrade instructions should be done on every host, not just worker nodes. - We should just upgrade all packages, instead of calling out kubelet specifically. This will also upgrade kubectl, kubeadm, and kubernetes-cni, if installed. - Draining nodes should also ignore daemonsets, and master errors can be ignored. - Make sure that the new kubeadm download is chmoded correctly. - Add a step to run `kubeadm version` to verify after downloading. - Manually approve new kubelet CSRs if rotation is enabled (known issue). * Release 1.8 (#5680) * Fix versions for 1.8 API ref docs * Updates for 1.8 kubectl reference docs * Kubeadm /docs/admin/kubeadm.md cleanup, editing. (#5681) * Update docs/admin/kubeadm.md (mostly 1.8 related). This is Fabrizio's work, which I'm committing along with my edits (in a commit on top of this). * A few of my own edits to clarify and clean up some Markdown.
This commit is contained in:
Steve Perry
@@ -66,7 +66,27 @@ The node condition is represented as a JSON object. For example, the following r
|
||||
|
||||
If the Status of the Ready condition is "Unknown" or "False" for longer than the `pod-eviction-timeout`, an argument is passed to the [kube-controller-manager](/docs/admin/kube-controller-manager) and all of the Pods on the node are scheduled for deletion by the Node Controller. The default eviction timeout duration is **five minutes**. In some cases when the node is unreachable, the apiserver is unable to communicate with the kubelet on it. The decision to delete the pods cannot be communicated to the kubelet until it re-establishes communication with the apiserver. In the meantime, the pods which are scheduled for deletion may continue to run on the partitioned node.
|
||||
|
||||
In versions of Kubernetes prior to 1.5, the node controller would [force delete](/docs/concepts/workloads/pods/pod/#force-deletion-of-pods) these unreachable pods from the apiserver. However, in 1.5 and higher, the node controller does not force delete pods until it is confirmed that they have stopped running in the cluster. One can see these pods which may be running on an unreachable node as being in the "Terminating" or "Unknown" states. In cases where Kubernetes cannot deduce from the underlying infrastructure if a node has permanently left a cluster, the cluster administrator may need to delete the node object by hand. Deleting the node object from Kubernetes causes all the Pod objects running on it to be deleted from the apiserver, freeing up their names.
|
||||
In versions of Kubernetes prior to 1.5, the node controller would [force delete](/docs/concepts/workloads/pods/pod/#force-deletion-of-pods)
|
||||
these unreachable pods from the apiserver. However, in 1.5 and higher, the node controller does not force delete pods until it is
|
||||
confirmed that they have stopped running in the cluster. One can see these pods which may be running on an unreachable node as being in
|
||||
the "Terminating" or "Unknown" states. In cases where Kubernetes cannot deduce from the underlying infrastructure if a node has
|
||||
permanently left a cluster, the cluster administrator may need to delete the node object by hand. Deleting the node object from
|
||||
Kubernetes causes all the Pod objects running on it to be deleted from the apiserver, freeing up their names.
|
||||
|
||||
Version 1.8 introduces an alpha feature that automatically creates
|
||||
[taints](/docs/concepts/configuration/taint-and-toleration) that represent conditions.
|
||||
To enable this behavior, pass an additional feature gate flag `--feature-gates=...,TaintNodesByCondition=true`
|
||||
to the API server, controller manager, and scheduler.
|
||||
When `TaintNodesByCondition` is enabled, the scheduler ignores conditions when considering a Node; instead
|
||||
it looks at the Node's taints and a Pod's tolerations.
|
||||
|
||||
Now users can choose between the old scheduling model and a new, more flexible scheduling model.
|
||||
A Pod that does not have any tolerations gets scheduled according to the old model. But a Pod that
|
||||
tolerates the taints of a particular Node can be scheduled on that Node.
|
||||
|
||||
Note that because of small delay, usually less than one second, between time when condition is observed and a taint
|
||||
is created, it's possible that enabling this feature will slightly increase number of Pods that are successfully
|
||||
scheduled but rejected by the kubelet.
|
||||
|
||||
### Capacity
|
||||
|
||||
@@ -175,6 +195,9 @@ NodeController is responsible for adding taints corresponding to node problems l
|
||||
node unreachable or not ready. See [this documentation](/docs/concepts/configuration/taint-and-toleration)
|
||||
for details about `NoExecute` taints and the alpha feature.
|
||||
|
||||
Starting in version 1.8, the node controller can be made responsible for creating taints that represent
|
||||
Node conditions. This is an alpha feature of version 1.8.
|
||||
|
||||
### Self-Registration of Nodes
|
||||
|
||||
When the kubelet flag `--register-node` is true (the default), the kubelet will attempt to
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
approvers:
|
||||
title: Device Plugins
|
||||
description: Use the Kubernetes device plugin framework to implement plugins for GPUs, NICs, FPGAs, InfiniBand, and similar resources that require vendor-specific setup.
|
||||
---
|
||||
|
||||
{% include feature-state-alpha.md %}
|
||||
|
||||
{% capture overview %}
|
||||
Starting in version 1.8, Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/device-plugin.md)
|
||||
for vendors to advertise their resources to the kubelet without changing Kubernetes core code.
|
||||
Instead of writing custom Kubernetes code, vendors can implement a device plugin that can
|
||||
be deployed manually or as a DaemonSet. The targeted devices include GPUs,
|
||||
High-performance NICs, FPGAs, InfiniBand, and other similar computing resources
|
||||
that may require vendor specific initialization and setup.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture body %}
|
||||
|
||||
## Device plugin registration
|
||||
|
||||
The device plugins feature is gated by the `DevicePlugins` feature gate and is disabled by default.
|
||||
When the device plugins feature is enabled, the kubelet exports a `Registration` gRPC service:
|
||||
|
||||
```gRPC
|
||||
service Registration {
|
||||
rpc Register(RegisterRequest) returns (Empty) {}
|
||||
}
|
||||
```
|
||||
A device plugin can register itself with the kubelet through this gRPC service.
|
||||
During the registration, the device plugin needs to send:
|
||||
|
||||
* The name of its Unix socket.
|
||||
* The Device Plugin API version against which it was built.
|
||||
* The `ResourceName` it wants to advertise. Here `ResourceName` needs to follow the
|
||||
[extended resource naming scheme](https://github.com/kubernetes/kubernetes/pull/48922)
|
||||
as `vendor-domain/resource`.
|
||||
For example, an Nvidia GPU is advertised as `nvidia.com/gpu`.
|
||||
|
||||
Following a successful registration, the device plugin sends the kubelet the
|
||||
list of devices it manages, and the kubelet is then in charge of advertising those
|
||||
resources to the API server as part of the kubelet node status update.
|
||||
For example, after a device plugin registers `vendor-domain/foo` with the kubelet
|
||||
and reports two healthy devices on a node, the node status is updated
|
||||
to advertise 2 `vendor-domain/foo`.
|
||||
|
||||
Then, developers can request devices in a
|
||||
[Container](/docs/api-reference/{{page.version}}/#container-v1-core)
|
||||
specification by using the same process that is used for
|
||||
[opaque integer resources](/docs/tasks/configure-pod-container/opaque-integer-resource/).
|
||||
In version 1.8, extended resources are spported only as integer resources and must have
|
||||
`limit` equal to `request` in the Container specification.
|
||||
|
||||
## Device plugin implementation
|
||||
|
||||
The general workflow of a device plugin includes the following steps:
|
||||
|
||||
* Initialization. During this phase, the device plugin performs vendor specific
|
||||
initialization and setup to make sure the devices are in a ready state.
|
||||
|
||||
* The plugin starts a gRPC service, with a Unix socket under host path
|
||||
`/var/lib/kubelet/device-plugins/`, that implements the following interfaces:
|
||||
|
||||
```gRPC
|
||||
service DevicePlugin {
|
||||
// ListAndWatch returns a stream of List of Devices
|
||||
// Whenever a Device state change or a Device disapears, ListAndWatch
|
||||
// returns the new list
|
||||
rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
|
||||
|
||||
// Allocate is called during container creation so that the Device
|
||||
// Plugin can run device specific operations and instruct Kubelet
|
||||
// of the steps to make the Device available in the container
|
||||
rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
|
||||
}
|
||||
```
|
||||
|
||||
* The plugin registers itself with the kubelet through the Unix socket at host
|
||||
path `/var/lib/kubelet/device-plugins/kubelet.sock`.
|
||||
|
||||
* After successfully registering itself, the device plugin runs in serving mode, during which it keeps
|
||||
monitoring device health and reports back to the kubelet upon any device state changes.
|
||||
It is also responsible for serving `Allocate` gRPC requests. During `Allocate`, the device plugin may
|
||||
do device-specific preparation; for example, GPU cleanup or QRNG initialization.
|
||||
If the operations succeed, the device plugin returns an `AllocateResponse` that contains container
|
||||
runtime configurations for accessing the allocated devices. The kubelet passes this information
|
||||
to the container runtime.
|
||||
|
||||
A device plugin is expected to detect kubelet restarts and re-register itself with the new
|
||||
kubelet instance. In version 1.8, a new kubelet instance cleans up all the existing Unix sockets
|
||||
under `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion
|
||||
of its Unix socket and re-register itself upon such an event.
|
||||
|
||||
## Device plugin deployment
|
||||
|
||||
A device plugin can be deployed manually or as a DaemonSet. Being deployed as a DaemonSet has
|
||||
the benefit that Kubernetes can restart the device plugin if it fails.
|
||||
Otherwise, an extra mechanism is needed to recover from device plugin failures.
|
||||
The canonical directory `/var/lib/kubelet/device-plugins` requires privileged access,
|
||||
so a device plugin must run in a privileged security context.
|
||||
If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins`
|
||||
must be mounted as a
|
||||
[Volume](/docs/api-reference/{{page.version}}/#volume-v1-core)
|
||||
in the plugin's
|
||||
[PodSpec](/docs/api-reference/{{paage.version}}/#podspec-v1-core).
|
||||
|
||||
## Examples
|
||||
|
||||
For an example device plugin implementation, see
|
||||
[nvidia GPU device plugin for COS base OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/nvidia_gpu).
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/concept.md %}
|
||||
@@ -91,7 +91,9 @@ The following guides explain some of the resources in detail:
|
||||
* [DaemonSets](/docs/tasks/administer-federation/daemonset/)
|
||||
* [Deployment](/docs/tasks/administer-federation/deployment/)
|
||||
* [Events](/docs/tasks/administer-federation/events/)
|
||||
* [Hpa](/docs/tasks/administer-federation/hpa/)
|
||||
* [Ingress](/docs/tasks/administer-federation/ingress/)
|
||||
* [Jobs](/docs/tasks/administer-federation/job/)
|
||||
* [Namespaces](/docs/tasks/administer-federation/namespaces/)
|
||||
* [ReplicaSets](/docs/tasks/administer-federation/replicaset/)
|
||||
* [Secrets](/docs/tasks/administer-federation/secret/)
|
||||
|
||||
@@ -303,7 +303,67 @@ LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-0
|
||||
You can see that the Container was terminated because of `reason:OOM Killed`,
|
||||
where `OOM` stands for Out Of Memory.
|
||||
|
||||
## Opaque integer resources (Alpha feature)
|
||||
## Local ephemeral storage (alpha feature)
|
||||
|
||||
Kubernetes version 1.8 introduces a new resource, _ephemeral-storage_ for managing local ephemeral storage. In each Kubernetes node, kubelet's root directory (/var/lib/kubelet by default) and log directory (/var/log) are stored on the root partition of the node. This partition is also shared and consumed by pods via EmptyDir volumes, container logs, image layers and container writable layers.
|
||||
|
||||
This partition is “ephemeral” and applications cannot expect any performance SLAs (Disk IOPS for example) from this partition. Local ephemeral storage management only applies for the root partition; the optional partition for image layer and writable layer is out of scope.
|
||||
|
||||
**Note:** If an optional runntime partition is used, root parition will not hold any image layer or writable layers.
|
||||
{: .note}
|
||||
|
||||
### Requests and limits setting for local ephemeral storage
|
||||
Each Container of a Pod can specify one or more of the following:
|
||||
|
||||
* `spec.containers[].resources.limits.ephemeral-storage`
|
||||
* `spec.containers[].resources.requests.ephemeral-storage`
|
||||
|
||||
Limits and requests for `ephemeral-storage` are measured in bytes. You can express storage as
|
||||
a plain integer or as a fixed-point integer using one of these suffixes:
|
||||
E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
|
||||
Mi, Ki. For example, the following represent roughly the same value:
|
||||
|
||||
```shell
|
||||
128974848, 129e6, 129M, 123Mi
|
||||
```
|
||||
|
||||
For example, the following Pod has two Containers. Each Container has a request of 2GiB of local ephemeral storage. Each Container has a limit of 4GiB of local ephemeral storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of storage.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: frontend
|
||||
spec:
|
||||
containers:
|
||||
- name: db
|
||||
image: mysql
|
||||
resources:
|
||||
requests:
|
||||
ephemeral-storage: "2Gi"
|
||||
limits:
|
||||
ephemeral-storage: "4Gi"
|
||||
- name: wp
|
||||
image: wordpress
|
||||
resources:
|
||||
requests:
|
||||
ephemeral-storage: "2Gi"
|
||||
limits:
|
||||
ephemeral-storage: "4Gi"
|
||||
```
|
||||
|
||||
### How Pods with ephemeral-storage requests are scheduled
|
||||
|
||||
When you create a Pod, the Kubernetes scheduler selects a node for the Pod to
|
||||
run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods. (For more information, see ["Node Allocatable"](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) The scheduler ensures that the sum of the resource requests of the scheduled Containers is less than the capacity of the node.
|
||||
|
||||
### How Pods with ephemeral-storage limits run
|
||||
|
||||
For container-level isolation, if a Container's writable layer and logs usage exceeds its storage limit, the pod will be evicted. For pod-level isolation, if the sum of the local ephemeral storage usage from all containers and also the pod's EmptyDir volumes exceeds the limit, the pod will be evicted.
|
||||
|
||||
## Opaque integer resources (alpha feature)
|
||||
|
||||
{% include feature-state-deprecated.md %}
|
||||
|
||||
Kubernetes version 1.5 introduces Opaque integer resources. Opaque
|
||||
integer resources allow cluster operators to advertise new node-level
|
||||
@@ -313,9 +373,12 @@ Users can consume these resources in Pod specs just like CPU and memory.
|
||||
The scheduler takes care of the resource accounting so that no more than the
|
||||
available amount is simultaneously allocated to Pods.
|
||||
|
||||
**Note:** Opaque integer resources are Alpha in Kubernetes version 1.5.
|
||||
Only resource accounting is implemented; node-level isolation is still
|
||||
under active development.
|
||||
**Note:** Opaque Integer Resources will be removed in version 1.9.
|
||||
[Extended Resources](#extended-resources) are a replacement for Opaque Integer
|
||||
Resources. Users can use any domain name prefix outside of the `kubernetes.io/`
|
||||
domain instead of the previous `pod.alpha.kubernetes.io/opaque-int-resource-`
|
||||
prefix.
|
||||
{: .note}
|
||||
|
||||
Opaque integer resources are resources that begin with the prefix
|
||||
`pod.alpha.kubernetes.io/opaque-int-resource-`. The API server
|
||||
@@ -339,22 +402,9 @@ first pod that requests the resource to be scheduled on that node.
|
||||
|
||||
**Example:**
|
||||
|
||||
Here is an HTTP request that advertises five "foo" resources on node `k8s-node-1` whose master is `k8s-master`.
|
||||
|
||||
```http
|
||||
PATCH /api/v1/nodes/k8s-node-1/status HTTP/1.1
|
||||
Accept: application/json
|
||||
Content-Type: application/json-patch+json
|
||||
Host: k8s-master:8080
|
||||
|
||||
[
|
||||
{
|
||||
"op": "add",
|
||||
"path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-foo",
|
||||
"value": "5"
|
||||
}
|
||||
]
|
||||
```
|
||||
Here is an example showing how to use `curl` to form an HTTP request that
|
||||
advertises five "foo" resources on node `k8s-node-1` whose master is
|
||||
`k8s-master`.
|
||||
|
||||
```shell
|
||||
curl --header "Content-Type: application/json-patch+json" \
|
||||
@@ -395,6 +445,92 @@ spec:
|
||||
pod.alpha.kubernetes.io/opaque-int-resource-foo: 1
|
||||
```
|
||||
|
||||
## Extended Resources
|
||||
|
||||
Kubernetes version 1.8 introduces Extended Resources. Extended Resources are
|
||||
fully-qualified resource names outside the `kubernetes.io` domain. Extended
|
||||
Resources allow cluster operators to advertise new node-level resources that
|
||||
would be otherwise unknown to the system. Extended Resource quantities must be
|
||||
integers and cannot be overcommitted.
|
||||
|
||||
Users can consume Extended Resources in Pod specs just like CPU and memory.
|
||||
The scheduler takes care of the resource accounting so that no more than the
|
||||
available amount is simultaneously allocated to Pods.
|
||||
|
||||
The API server restricts quantities of Extended Resources to whole numbers.
|
||||
Examples of _valid_ quantities are `3`, `3000m` and `3Ki`. Examples of
|
||||
_invalid_ quantities are `0.5` and `1500m`.
|
||||
|
||||
**Note:** Extended Resources replace [Opaque Integer
|
||||
Resources](#opaque-integer-resources-alpha-feature). Users can use any domain
|
||||
name prefix outside of the `kubernetes.io/` domain instead of the previous
|
||||
`pod.alpha.kubernetes.io/opaque-int-resource-` prefix.
|
||||
{: .note}
|
||||
|
||||
There are two steps required to use Extended Resources. First, the
|
||||
cluster operator must advertise a per-node Extended Resource on one or more
|
||||
nodes. Second, users must request the Extended Resource in Pods.
|
||||
|
||||
To advertise a new Extended Resource, the cluster operator should
|
||||
submit a `PATCH` HTTP request to the API server to specify the available
|
||||
quantity in the `status.capacity` for a node in the cluster. After this
|
||||
operation, the node's `status.capacity` will include a new resource. The
|
||||
`status.allocatable` field is updated automatically with the new resource
|
||||
asynchronously by the kubelet. Note that because the scheduler uses the
|
||||
node `status.allocatable` value when evaluating Pod fitness, there may
|
||||
be a short delay between patching the node capacity with a new resource and the
|
||||
first pod that requests the resource to be scheduled on that node.
|
||||
|
||||
**Example:**
|
||||
|
||||
Here is an example showing how to use `curl` to form an HTTP request that
|
||||
advertises five "example.com/foo" resources on node `k8s-node-1` whose master
|
||||
is `k8s-master`.
|
||||
|
||||
```shell
|
||||
curl --header "Content-Type: application/json-patch+json" \
|
||||
--request PATCH \
|
||||
--data '[{"op": "add", "path": "/status/capacity/example.com~1foo", "value": "5"}]' \
|
||||
http://k8s-master:8080/api/v1/nodes/k8s-node-1/status
|
||||
```
|
||||
|
||||
**Note**: In the preceding request, `~1` is the encoding for the character `/`
|
||||
in the patch path. The operation path value in JSON-Patch is interpreted as a
|
||||
JSON-Pointer. For more details, see
|
||||
[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3).
|
||||
{: .note}
|
||||
|
||||
To consume an Extended Resource in a Pod, include the resource name as a key
|
||||
in the `spec.containers[].resources.requests` map.
|
||||
|
||||
**Note:** Extended resources cannot be overcommitted, so request and limit
|
||||
must be equal if both are present in a container spec.
|
||||
{: .note}
|
||||
|
||||
The Pod is scheduled only if all of the resource requests are
|
||||
satisfied, including cpu, memory and any Extended Resources. The Pod will
|
||||
remain in the `PENDING` state as long as the resource request cannot be met by
|
||||
any node.
|
||||
|
||||
**Example:**
|
||||
|
||||
The Pod below requests 2 cpus and 1 "example.com/foo" (an extended resource.)
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: my-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: my-container
|
||||
image: myimage
|
||||
resources:
|
||||
requests:
|
||||
cpu: 2
|
||||
example.com/foo: 1
|
||||
```
|
||||
|
||||
## Planned Improvements
|
||||
|
||||
Kubernetes version 1.5 only allows resource quantities to be specified on a
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
approvers:
|
||||
- davidopp
|
||||
- wojtek-t
|
||||
title: Pod Priority and Preemption
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
{% include feature-state-alpha.md %}
|
||||
|
||||
[Pods](/docs/user-guide/pods) in Kubernetes 1.8 and later can have priority. Priority
|
||||
indicates the importance of a Pod relative to other Pods. When a Pod cannot be scheduled,
|
||||
the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the
|
||||
pending Pod possible. In a future Kubernetes release, priority will also affect
|
||||
out-of-resource eviction ordering on the Node.
|
||||
|
||||
**Note:** Preemption does not respect PodDisruptionBudget; see
|
||||
[the limitations section](#poddisruptionbudget-is-not-supported) for more details.
|
||||
{: .note}
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture body %}
|
||||
|
||||
## How to use priority and preemption
|
||||
To use priority and preemption in Kubernetes 1.8, follow these steps:
|
||||
|
||||
1. Enable the feature.
|
||||
|
||||
1. Add one or more PriorityClasses.
|
||||
|
||||
1. Create Pods with `PriorityClassName` set to one of the added PriorityClasses.
|
||||
Of course you do not need to create the Pods directly; normally you would add
|
||||
`PriorityClassName` to the Pod template of a collection object like a Deployment.
|
||||
|
||||
The following sections provide more information about these steps.
|
||||
|
||||
## Enabling priority and preemption
|
||||
|
||||
Pod priority and preemption is disabled by default in Kubernetes 1.8.
|
||||
To enable the feature, set this command-line flag for the API server
|
||||
and the scheduler:
|
||||
|
||||
```
|
||||
--feature-gates=PodPriority=true
|
||||
```
|
||||
|
||||
Also set this flag for API server:
|
||||
|
||||
|
||||
```
|
||||
--runtime-config=scheduling.k8s.io/v1alpha1=true
|
||||
```
|
||||
|
||||
After the feature is enabled, you can create [PriorityClasses](#priorityclass)
|
||||
and create Pods with [`PriorityClassName`](#pod-priority) set.
|
||||
|
||||
If you try the feature and then decide to disable it, you must remove the PodPriority
|
||||
command-line flag or set it to false, and then restart the API server and
|
||||
scheduler. After the feature is disabled, the existing Pods keep their priority
|
||||
fields, but preemption is disabled, and priority fields are ignored, and you
|
||||
cannot set PriorityClassName in new Pods.
|
||||
|
||||
## PriorityClass
|
||||
|
||||
A PriorityClass is a non-namespaced object that defines a mapping from a priority
|
||||
class name to the integer value of the priority. The name is specified in the `name`
|
||||
field of the PriorityClass object's metadata. The value is specified in the required
|
||||
`value` field. The higher the value, the higher the priority.
|
||||
|
||||
A PriorityClass object can have any 32-bit integer value smaller than or equal to
|
||||
1 billion. Larger numbers are reserved for critical system Pods that should not
|
||||
normally be preempted or evicted. A cluster admin should create one PriorityClass
|
||||
object for each such mapping that they want.
|
||||
|
||||
PriorityClass also has two optional fields: `globalDefault` and `description`.
|
||||
The `globalDefault` field indicates that the value of this PriorityClass should
|
||||
be used for Pods without a `PriorityClassName`. Only one PriorityClass with
|
||||
`globalDefault` set to true can exist in the system. If there is no PriorityClass
|
||||
with `globalDefault` set, the priority of Pods with no `PriorityClassName` is zero.
|
||||
|
||||
The `description` field is an arbitrary string. It is meant to tell users of
|
||||
the cluster when they should use this PriorityClass.
|
||||
|
||||
**Note 1**: If you upgrade your existing cluster and enable this feature, the priority
|
||||
of your existing Pods will be considered to be zero.
|
||||
{: .note}
|
||||
|
||||
**Note 2**: Addition of a PriorityClass with `globalDefault` set to true does not
|
||||
change the priorities of existing Pods. The value of such a PriorityClass is used only
|
||||
for Pods created after the PriorityClass is added.
|
||||
{: .note}
|
||||
|
||||
**Note 3**: If you delete a PriorityClass, existing Pods that use the name of the
|
||||
deleted priority class remain unchanged, but you are not able to create more Pods
|
||||
that use the name of the deleted PriorityClass.
|
||||
{: .note}
|
||||
|
||||
### Example PriorityClass
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: high-priority
|
||||
value: 1000000
|
||||
globalDefault: false
|
||||
description: "This priority class should be used for XYZ service pods only."
|
||||
```
|
||||
|
||||
## Pod priority
|
||||
|
||||
After you have one or more PriorityClasses, you can create Pods that specify one
|
||||
of those PriorityClass names in their specifications. The priority admission
|
||||
controller uses the `priorityClassName` field and populates the integer value
|
||||
of the priority. If the priority class is not found, the Pod is rejected.
|
||||
|
||||
The following YAML is an example of a Pod configuration that uses the PriorityClass
|
||||
created in the preceding example. The priority admission controller checks the
|
||||
specification and resolves the priority of the Pod to 1000000.
|
||||
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: nginx
|
||||
labels:
|
||||
env: test
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx
|
||||
imagePullPolicy: IfNotPresent
|
||||
priorityClassName: high-priority
|
||||
```
|
||||
|
||||
## Preemption
|
||||
|
||||
When Pods are created, they go to a queue and wait to be scheduled. The scheduler
|
||||
picks a Pod from the queue and tries to schedule it on a Node. If no Node is found
|
||||
that satisfies all the specified requirements of the Pod, preemption logic is triggered
|
||||
for the pending Pod. Let's call the pending pod P. Preemption logic tries to find a Node
|
||||
where removal of one or more Pods with lower priority than P would enable P to be scheduled
|
||||
on that Node. If such a Node is found, one or more lower priority Pods get
|
||||
deleted from the Node. After the Pods are gone, P can be scheduled on the Node.
|
||||
|
||||
### Limitations of preemption (alpha version)
|
||||
|
||||
#### Starvation of preempting Pod
|
||||
|
||||
When Pods are preempted, the victims get their
|
||||
[graceful termination period](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods).
|
||||
They have that much time to finish their work and exit. If they don't, they are
|
||||
killed. This graceful termination period creates a time gap between the point
|
||||
that the scheduler preempts Pods and the time when the pending Pod (P) can be
|
||||
scheduled on the Node (N). In the meantime, the scheduler keeps scheduling other
|
||||
pending Pods. As victims exit or get terminated, the scheduler tries to schedule
|
||||
Pods in the pending queue, and one or more of them may be considered and
|
||||
scheduled to N before the scheduler considers scheduling P on N. In such a case,
|
||||
it is likely that when all the victims exit, Pod P won't fit on Node N anymore.
|
||||
So, scheduler will have to preempt other Pods on Node N or another Node so that
|
||||
P can be scheduled. This scenario might be repeated again for the second and
|
||||
subsequent rounds of preemption, and P might not get scheduled for a while.
|
||||
This scenario can cause problems in various clusters, but is particularly
|
||||
problematic in clusters with a high Pod creation rate.
|
||||
|
||||
We will address this problem in the beta version of Pod preemption. The solution
|
||||
we plan to implement is
|
||||
[provided here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-preemption.md#preemption-mechanics).
|
||||
|
||||
#### PodDisruptionBudget is not supported
|
||||
|
||||
A [Pod Disruption Budget (PDB)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
|
||||
allows application owners to limit the number Pods of a replicated application that
|
||||
are down simultaneously from voluntary disruptions. However, the alpha version of
|
||||
preemption does not respect PDB when choosing preemption victims.
|
||||
We plan to add PDB support in beta, but even in beta, respecting PDB will be best
|
||||
effort. The Scheduler will try to find victims whose PDB won't be violated by preemption,
|
||||
but if no such victims are found, preemption will still happen, and lower priority Pods
|
||||
will be removed despite their PDBs being violated.
|
||||
|
||||
#### Inter-Pod affinity on lower-priority Pods
|
||||
|
||||
In version 1.8, a Node is considered for preemption only when
|
||||
the answer to this question is yes: "If all the Pods with lower priority than
|
||||
the pending Pod are removed from the Node, can the pending pod be scheduled on
|
||||
the Node?"
|
||||
|
||||
**Note:** Preemption does not necessarily remove all lower-priority Pods. If the
|
||||
pending pod can be scheduled by removing fewer than all lower-priority Pods, then
|
||||
only a portion of the lower-priority Pods are removed. Even so, the answer to the
|
||||
preceding question must be yes. If the answer is no, the Node is not considered
|
||||
for preemption.
|
||||
{: .note}
|
||||
|
||||
If a pending Pod has inter-pod affinity to one or more of the lower-priority Pods
|
||||
on the Node, the inter-Pod affinity rule cannot be satisfied in the absence of those
|
||||
lower-priority Pods. In this case, the scheduler does not preempt any Pods on the
|
||||
Node. Instead, it looks for another Node. The scheduler might find a suitable Node
|
||||
or it might not. There is no guarantee that the pending Pod can be scheduled.
|
||||
|
||||
We might address this issue in future versions, but we don't have a clear plan yet.
|
||||
We will not consider it a blocker for Beta or GA. Part
|
||||
of the reason is that finding the set of lower-priority Pods that satisfy all
|
||||
inter-Pod affinity rules is computationally expensive, and adds substantial
|
||||
complexity to the preemption logic. Besides, even if preemption keeps the lower-priority
|
||||
Pods to satisfy inter-Pod affinity, the lower priority Pods might be preempted
|
||||
later by other Pods, which removes the benefits of having the complex logic of
|
||||
respecting inter-Pod affinity.
|
||||
|
||||
Our recommended solution for this problem is to create inter-Pod affinity only towards
|
||||
equal or higher priority pods.
|
||||
|
||||
#### Cross node preemption
|
||||
|
||||
Suppose a Node N is being considered for preemption so that a pending Pod P
|
||||
can be scheduled on N. P might become feasible on N only if a Pod on another
|
||||
Node is preempted. Here's an example:
|
||||
|
||||
* Pod P is being considered for Node N.
|
||||
* Pod Q is running on another Node in the same zone as Node N.
|
||||
* Pod P has anit-affinity with Pod Q.
|
||||
* There are no other cases of anti-affinity between Pod P and other Pods in the zone.
|
||||
* In order to schedule Pod P on Node N, Pod Q should be preempted, but scheduler
|
||||
does not perform cross-node preemption. So, Pod P will be deemed unschedulable
|
||||
on Node N.
|
||||
|
||||
If Pod Q were removed from its Node, the anti-affinity violation would be gone,
|
||||
and Pod P could possibly be scheduled on Node N.
|
||||
|
||||
We may consider adding cross Node preemption in future versions if we find an
|
||||
algorithm with reasonable performance. We cannot promise anything at this point,
|
||||
and cross Node preemption will not be considered a blocker for Beta or GA.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/concept.md %}
|
||||
@@ -188,7 +188,7 @@ running on the node as follows
|
||||
|
||||
The above behavior is a beta feature. In addition, Kubernetes 1.6 has alpha
|
||||
support for representing node problems. In other words, the node controller
|
||||
automatically taints a node when certain condition is true. The builtin taints
|
||||
automatically taints a node when certain condition is true. The built-in taints
|
||||
currently include:
|
||||
|
||||
* `node.alpha.kubernetes.io/notReady`: Node is not ready. This corresponds to
|
||||
@@ -249,9 +249,20 @@ admission controller](https://git.k8s.io/kubernetes/plugin/pkg/admission/default
|
||||
|
||||
* `node.alpha.kubernetes.io/unreachable`
|
||||
* `node.alpha.kubernetes.io/notReady`
|
||||
* `node.kubernetes.io/memoryPressure`
|
||||
* `node.kubernetes.io/diskPressure`
|
||||
* `node.kubernetes.io/outOfDisk` (*only for critical pods*)
|
||||
|
||||
This ensures that DaemonSet pods are never evicted due to these problems,
|
||||
which matches the behavior when this feature is disabled.
|
||||
|
||||
## Taint Nodes by Condition
|
||||
|
||||
Version 1.8 introduces an alpha feature that causes the node controller to create taints corresponding to
|
||||
Node conditions. When this feature is enabled, the scheduler does not check conditions; instead the scheduler checks taints. This assures that conditions don't affect what's scheduled onto the Node. The user can choose to ignore some of the Node's problems (represented as conditions) by adding appropriate Pod tolerations.
|
||||
|
||||
To make sure that turning on this feature doesn't break DaemonSets, starting in version 1.8, the DaemonSet controller automatically adds the following `NoSchedule` tolerations to all daemons:
|
||||
|
||||
* `node.kubernetes.io/memory-pressure`
|
||||
* `node.kubernetes.io/disk-pressure`
|
||||
* `node.kubernetes.io/out-of-disk` (*only for critical pods*)
|
||||
|
||||
The above settings ensure backward compatibility, but we understand they may not fit all user's needs, which is why
|
||||
cluster admin may choose to add arbitrary tolerations to DaemonSets.
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
apiVersion: batch/v2alpha1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: hello
|
||||
spec:
|
||||
schedule: "*/1 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: hello
|
||||
image: busybox
|
||||
args:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- date; echo Hello from the Kubernetes cluster
|
||||
restartPolicy: OnFailure
|
||||
@@ -21,22 +21,24 @@ actions that a pod can perform and what it has the ability to access. The
|
||||
run with in order to be accepted into the system. They allow an
|
||||
administrator to control the following:
|
||||
|
||||
| Control Aspect | Field Name |
|
||||
| ------------------------------------------------------------- | --------------------------------- |
|
||||
| Running of privileged containers | `privileged` |
|
||||
| Default set of capabilities that will be added to a container | `defaultAddCapabilities` |
|
||||
| Capabilities that will be dropped from a container | `requiredDropCapabilities` |
|
||||
| Capabilities a container can request to be added | `allowedCapabilities` |
|
||||
| Controlling the usage of volume types | [`volumes`](#controlling-volumes) |
|
||||
| The use of host networking | [`hostNetwork`](#host-network) |
|
||||
| The use of host ports | `hostPorts` |
|
||||
| The use of host's PID namespace | `hostPID` |
|
||||
| The use of host's IPC namespace | `hostIPC` |
|
||||
| The SELinux context of the container | [`seLinux`](#selinux) |
|
||||
| The user ID | [`runAsUser`](#runasuser) |
|
||||
| Configuring allowable supplemental groups | [`supplementalGroups`](#supplementalgroups) |
|
||||
| Allocating an FSGroup that owns the pod's volumes | [`fsGroup`](#fsgroup) |
|
||||
| Requiring the use of a read only root file system | `readOnlyRootFilesystem` |
|
||||
| Control Aspect | Field Name |
|
||||
| ---------------------------------------------------------------------- | ------------------------------------------- |
|
||||
| Running of privileged containers | `privileged` |
|
||||
| Default set of capabilities that will be added to a container | `defaultAddCapabilities` |
|
||||
| Capabilities that will be dropped from a container | `requiredDropCapabilities` |
|
||||
| Capabilities a container can request to be added | `allowedCapabilities` |
|
||||
| Controlling the usage of volume types | [`volumes`](#controlling-volumes) |
|
||||
| The use of host networking | [`hostNetwork`](#host-network) |
|
||||
| The use of host ports | `hostPorts` |
|
||||
| The use of host's PID namespace | `hostPID` |
|
||||
| The use of host's IPC namespace | `hostIPC` |
|
||||
| The SELinux context of the container | [`seLinux`](#selinux) |
|
||||
| The user ID | [`runAsUser`](#runasuser) |
|
||||
| Configuring allowable supplemental groups | [`supplementalGroups`](#supplementalgroups) |
|
||||
| Allocating an FSGroup that owns the pod's volumes | [`fsGroup`](#fsgroup) |
|
||||
| Requiring the use of a read only root file system | `readOnlyRootFilesystem` |
|
||||
| Running of a container that allow privilege escalation from its parent | [`allowPrivilegeEscalation`](#allowPrivilegeEscalation) |
|
||||
| Control whether a process can gain more privileges than its parent process | [`defaultAllowPrivilegeEscalation`](#defaultAllowPrivilegeEscalation) |
|
||||
|
||||
_Pod Security Policies_ are comprised of settings and strategies that
|
||||
control the security features a pod has access to. These settings fall
|
||||
@@ -124,7 +126,21 @@ configMap, downwardAPI, emptyDir, persistentVolumeClaim, secret, and projected.
|
||||
|
||||
### Host Network
|
||||
- *HostPorts*, default `empty`. List of `HostPortRange`, defined by `min`(inclusive) and `max`(inclusive), which define the allowed host ports.
|
||||
|
||||
|
||||
### AllowPrivilegeEscalation
|
||||
|
||||
Gates whether or not a user is allowed to set the security context of a container
|
||||
to `allowPrivilegeEscalation=true`. This field defaults to `false`.
|
||||
|
||||
### DefaultAllowPrivilegeEscalation
|
||||
|
||||
Sets the default for the security context `AllowPrivilegeEscalation` of a container.
|
||||
This bool directly controls whether the `no_new_privs` flag gets set on the
|
||||
container process. It defaults to `nil`. The default behavior of `nil`
|
||||
allows privilege escalation so as to not break setuid binaries. Setting it to `false`
|
||||
ensures that no child process of a container can gain more privileges than
|
||||
its parent.
|
||||
|
||||
## Admission
|
||||
|
||||
_Admission control_ with `PodSecurityPolicy` allows for control over the
|
||||
|
||||
@@ -84,6 +84,13 @@ define a quota as follows:
|
||||
* `gold.storageclass.storage.k8s.io/requests.storage: 500Gi`
|
||||
* `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi`
|
||||
|
||||
In release 1.8, quota support for local ephemeral storage is added as alpha feature
|
||||
|
||||
| Resource Name | Description |
|
||||
| ------------------------------- |----------------------------------------------------------- |
|
||||
| `requests.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value. |
|
||||
| `limits.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value. |
|
||||
|
||||
## Object Count Quota
|
||||
|
||||
The number of objects of a given type can be restricted. The following types
|
||||
|
||||
@@ -80,7 +80,11 @@ With the additional entries specified at the bottom.
|
||||
|
||||
## Limitations
|
||||
|
||||
As of 1.7, Pods with hostNetwork enabled will not be able to use this feature. This is because kubelet only manages the hosts file for non-hostNetwork Pods. There are ongoing discussions to change this.
|
||||
HostAlias is only supported in 1.7+.
|
||||
|
||||
HostAlias support in 1.7 is limited to non-hostNetwork Pods because kubelet only manages the hosts file for non-hostNetwork Pods.
|
||||
|
||||
In 1.8, HostAlias is supported for all Pods regardless of network configuration.
|
||||
|
||||
## Why Does Kubelet Manage the Hosts File?
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ Network policies are implemented by the network plugin, so you must be using a n
|
||||
|
||||
By default, pods are non-isolated; they accept traffic from any source.
|
||||
|
||||
Pods become isolated by having a NetworkPolicy that selects them. Once there is any NetworkPolicy in a Namespace selecting a particular pod, that pod will reject any connections that are not allowed by any NetworkPolicy. (Other pods in the Namespace that are not selected by any NetworkPolicy will continue to accept all traffic.)
|
||||
Pods become isolated by having a NetworkPolicy that selects them. Once there is any NetworkPolicy in a namespace selecting a particular pod, that pod will reject any connections that are not allowed by any NetworkPolicy. (Other pods in the namespace that are not selected by any NetworkPolicy will continue to accept all traffic.)
|
||||
|
||||
## The `NetworkPolicy` Resource
|
||||
|
||||
@@ -39,6 +39,9 @@ spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
role: db
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- ipBlock:
|
||||
@@ -54,6 +57,13 @@ spec:
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.0.0.0/24
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5978
|
||||
```
|
||||
|
||||
*POSTing this to the API server will have no effect unless your chosen networking solution supports network policy.*
|
||||
@@ -64,7 +74,11 @@ __spec__: `NetworkPolicy` [spec](https://git.k8s.io/community/contributors/devel
|
||||
|
||||
__podSelector__: Each `NetworkPolicy` includes a `podSelector` which selects the grouping of pods to which the policy applies. Since `NetworkPolicy` currently only supports defining `ingress` rules, this `podSelector` essentially defines the "destination pods" for the policy. The example policy selects pods with the label "role=db". An empty `podSelector` selects all pods in the namespace.
|
||||
|
||||
__ingress__: Each `NetworkPolicy` includes a list of whitelist `ingress` rules. Each rule allows traffic which matches both the `from` and `ports` sections. The example policy contains a single rule, which matches traffic on a single port, from either of two sources, the first specified via a `namespaceSelector` and the second specified via a `podSelector`.
|
||||
__policyTypes__: Each `NetworkPolicy` includes a `policyTypes` list which may include either `Ingress`, `Egress`, or both. The `policyTypes` field indicates whether or not the given policy applies to ingress traffic to selected pod, egress traffic from selected pods, or both. If no `policyTypes` are specified on a NetworkPolicy then by default `Ingress` will always be set and `Egress` will be set if the NetworkPolicy has any egress rules.
|
||||
|
||||
__ingress__: Each `NetworkPolicy` may include a list of whitelist `ingress` rules. Each rule allows traffic which matches both the `from` and `ports` sections. The example policy contains a single rule, which matches traffic on a single port, from either of two sources, the first specified via a `namespaceSelector` and the second specified via a `podSelector`.
|
||||
|
||||
__egress__: Each `NetworkPolicy` may include a list of whitelist `egress` rules. Each rule allows traffic which matches both the `to` and `ports` sections. The example policy contains a single rule, which matches traffic on a single port to any destination in `10.0.0.0/24`.
|
||||
|
||||
__ipBlock__: `ipBlock` describes a particular CIDR that is allowed to
|
||||
the pods matched by a NetworkPolicySpec's podSelector. The `except` entry
|
||||
@@ -73,15 +87,21 @@ values will be rejected if they are outside the CIDR range.
|
||||
|
||||
So, the example NetworkPolicy:
|
||||
|
||||
1. isolates "role=db" pods in the "default" namespace (if they weren't already isolated)
|
||||
1. isolates "role=db" pods in the "default" namespace for both ingress and egress traffic (if they weren't already isolated)
|
||||
2. allows connections to TCP port 6379 of "role=db" pods in the "default" namespace from any pod in the "default" namespace with the label "role=frontend"
|
||||
3. allows connections to TCP port 6379 of "role=db" pods in the "default" namespace from any pod in a namespace with the label "project=myproject"
|
||||
3. allows connections from any pod in the "default" namespace with the label "role=db" to CIDR 10.0.0.0/24 on TCP port 5978
|
||||
|
||||
See the [NetworkPolicy getting started guide](/docs/getting-started-guides/network-policy/walkthrough) for further examples.
|
||||
|
||||
## Default policies
|
||||
|
||||
You can create a "default" isolation policy for a Namespace by creating a NetworkPolicy that selects all pods but does not allow any traffic:
|
||||
By default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to and from pods in that namespace. The following examples let you change the default behavior
|
||||
in that namespace.
|
||||
|
||||
### Default deny all ingress traffic
|
||||
|
||||
You can create a "default" isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any ingress traffic to those pods.
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
@@ -90,11 +110,15 @@ metadata:
|
||||
name: default-deny
|
||||
spec:
|
||||
podSelector:
|
||||
policyTypes:
|
||||
- Ingress
|
||||
```
|
||||
|
||||
This ensures that even pods that aren't selected by any other NetworkPolicy will still be isolated.
|
||||
This ensures that even pods that aren't selected by any other NetworkPolicy will still be isolated. This policy does not change the default egress isolation behavior.
|
||||
|
||||
Alternatively, if you want to allow all traffic for all pods in a Namespace (even if policies are added that cause some pods to be treated as "isolated"), you can create a policy that explicitly allows all traffic:
|
||||
### Default allow all ingress traffic
|
||||
|
||||
If you want to allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as "isolated"), you can create a policy that explicitly allows all traffic in that namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
@@ -107,6 +131,57 @@ spec:
|
||||
- {}
|
||||
```
|
||||
|
||||
### Default deny all egress traffic.
|
||||
|
||||
You can create a "default" egress isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any egress traffic from those pods.
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
spec:
|
||||
podSelector:
|
||||
policyTypes:
|
||||
- Egress
|
||||
```
|
||||
|
||||
This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed egress traffic. This policy does not
|
||||
change the default ingress isolation behavior.
|
||||
|
||||
### Default allow all egress traffic
|
||||
|
||||
If you want to allow all traffic from all pods in a namespace (even if policies are added that cause some pods to be treated as "isolated"), you can create a policy that explicitly allows all egress traffic in that namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: allow-all
|
||||
spec:
|
||||
podSelector:
|
||||
egress:
|
||||
- {}
|
||||
```
|
||||
|
||||
### Default deny all ingress and all egress traffic
|
||||
|
||||
You can create a "default" policy for a namespace which prevents all ingress AND egress traffic by creating the following NetworkPolicy in that namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
spec:
|
||||
podSelector:
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
```
|
||||
|
||||
This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed ingress or egress traffic.
|
||||
|
||||
## What's next?
|
||||
|
||||
- See the [Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/)
|
||||
|
||||
@@ -470,9 +470,9 @@ metadata:
|
||||
{% capture azure %}
|
||||
```yaml
|
||||
[...]
|
||||
metadata:
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
annotations:
|
||||
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
|
||||
[...]
|
||||
```
|
||||
@@ -516,40 +516,37 @@ ELB at the other end of its connection) when forwarding requests.
|
||||
TCP and SSL will select layer 4 proxying: the ELB will forward traffic without
|
||||
modifying the headers.
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: (comma-separated-port-names)
|
||||
```
|
||||
|
||||
The third annotation indicates which port(s) should accept SSL traffic on the ELB. If a certificate is set on the
|
||||
Service, the default is to configure all ports on the ELB to use SSL. This is not typically desired for HTTP servers.
|
||||
|
||||
A complete HTTP Service supporting SSL might look like:
|
||||
In a mixed-use environment where some ports are secured and others are left unencrypted,
|
||||
the following annotations may be used:
|
||||
|
||||
```yaml
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
|
||||
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012
|
||||
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
ports:
|
||||
- port: 80
|
||||
name: http
|
||||
targetPort: 8080
|
||||
- port: 443
|
||||
name: https
|
||||
targetPort: 8080
|
||||
selector:
|
||||
app: MyApp
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
|
||||
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443,8443"
|
||||
```
|
||||
|
||||
In the above example, if the service contained three ports, `80`, `443`, and
|
||||
`8443`, then `443` and `8443` would use the SSL certificate, but `80` would just
|
||||
be proxied HTTP.
|
||||
|
||||
#### PROXY protocol support on AWS
|
||||
|
||||
To enable [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt)
|
||||
support for clusters running on AWS, you can use the following service
|
||||
annotation:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
|
||||
```
|
||||
|
||||
Since version 1.3.0 the use of this annotation applies to all ports proxied by the ELB
|
||||
and cannot be configured otherwise.
|
||||
|
||||
### External IPs
|
||||
|
||||
If there are external IPs that route to one or more cluster nodes, Kubernetes services can be exposed on those
|
||||
|
||||
@@ -107,7 +107,40 @@ However, the particular path specified in the custom recycler pod template in th
|
||||
|
||||
#### Deleting
|
||||
|
||||
For volume plugins that support the Delete reclaim policy, deletion removes both the `PersistentVolume` object from Kubernetes, as well as deleting the associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume. Volumes that were dynamically provisioned are always deleted. If that is not desired, currently, the only option is to edit or patch the PV after it is created. See [Change the Reclaim Policy of a PersistentVolume](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/).
|
||||
For volume plugins that support the Delete reclaim policy, deletion removes both the `PersistentVolume` object from Kubernetes, as well as deleting the associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume. Volumes that were dynamically provisioned inherit the [reclaim policy of their `StorageClass`](#reclaim-policy-1), which defaults to Delete. The administrator should configure the `StorageClass` according to users' expectations, otherwise the PV must be edited or patched after it is created. See [Change the Reclaim Policy of a PersistentVolume](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/).
|
||||
|
||||
|
||||
### Expanding Persistent Volumes Claims
|
||||
|
||||
With Kubernetes 1.8, we have added Alpha support for expanding persistent volumes. The current Alpha support was designed to only support volume types
|
||||
that don't need file system resizing (Currently only glusterfs).
|
||||
|
||||
Administrator can allow expanding persistent volume claims by setting `ExpandPersistentVolumes` feature gate to true. Administrator
|
||||
should also enable [`PersistentVolumeClaimResize` admission plugin](/docs/admin/admission-controllers/#persistentvolumeclaimresize)
|
||||
to perform additional validations of volumes that can be resized.
|
||||
|
||||
Once `PersistentVolumeClaimResize` admission plug-in has been turned on, resizing will only be allowed for storage classes
|
||||
whose `allowVolumeExpansion` field is set to true.
|
||||
|
||||
``` yaml
|
||||
kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: gluster-vol-default
|
||||
provisioner: kubernetes.io/glusterfs
|
||||
parameters:
|
||||
resturl: "http://192.168.10.100:8080"
|
||||
restuser: ""
|
||||
secretNamespace: ""
|
||||
secretName: ""
|
||||
allowVolumeExpansion: true
|
||||
```
|
||||
|
||||
Once both feature gate and aforementioned admission plug-in are turned on, an user can request larger volume for their `PersistentVolumeClaim`
|
||||
by simply editing the claim and requesting bigger size. This in turn will trigger expansion of volume that is backing underlying `PersistentVolume`.
|
||||
|
||||
Under no circustances a new `PersistentVolume` gets created to satisfy the claim. Kubernetes will attempt to resize existing volume to satisfy the claim.
|
||||
|
||||
|
||||
## Types of Persistent Volumes
|
||||
|
||||
@@ -139,20 +172,23 @@ For volume plugins that support the Delete reclaim policy, deletion removes both
|
||||
Each PV contains a spec and status, which is the specification and status of the volume.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: pv0003
|
||||
spec:
|
||||
capacity:
|
||||
storage: 5Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Recycle
|
||||
storageClassName: slow
|
||||
nfs:
|
||||
path: /tmp
|
||||
server: 172.17.0.2
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: pv0003
|
||||
spec:
|
||||
capacity:
|
||||
storage: 5Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Recycle
|
||||
storageClassName: slow
|
||||
mountOptions:
|
||||
- hard
|
||||
- nfsvers=4.1
|
||||
nfs:
|
||||
path: /tmp
|
||||
server: 172.17.0.2
|
||||
```
|
||||
|
||||
### Capacity
|
||||
@@ -225,47 +261,14 @@ Current reclaim policies are:
|
||||
|
||||
Currently, only NFS and HostPath support recycling. AWS EBS, GCE PD, Azure Disk, and Cinder volumes support deletion.
|
||||
|
||||
### Phase
|
||||
|
||||
A volume will be in one of the following phases:
|
||||
|
||||
* Available -- a free resource that is not yet bound to a claim
|
||||
* Bound -- the volume is bound to a claim
|
||||
* Released -- the claim has been deleted, but the resource is not yet reclaimed by the cluster
|
||||
* Failed -- the volume has failed its automatic reclamation
|
||||
|
||||
The CLI will show the name of the PVC bound to the PV.
|
||||
|
||||
### Mount Options
|
||||
|
||||
A Kubernetes administrator can specify additional mount options for when a Persistent Volume is being mounted on a node.
|
||||
A Kubernetes administrator can specify additional mount options for when a Persistent Volume is mounted on a node.
|
||||
|
||||
You can specify a mount option by using the annotation `volume.beta.kubernetes.io/mount-options` on
|
||||
your Persistent Volume.
|
||||
**Note:** Not all Persistent volume types support mount options.
|
||||
{: .note}
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: "v1"
|
||||
kind: "PersistentVolume"
|
||||
metadata:
|
||||
name: gce-disk-1
|
||||
annotations:
|
||||
volume.beta.kubernetes.io/mount-options: "discard"
|
||||
spec:
|
||||
capacity:
|
||||
storage: "10Gi"
|
||||
accessModes:
|
||||
- "ReadWriteOnce"
|
||||
gcePersistentDisk:
|
||||
fsType: "ext4"
|
||||
pdName: "gce-disk-1"
|
||||
```
|
||||
|
||||
A mount option is a string which will be cumulatively joined and used while mounting volume to the disk.
|
||||
|
||||
Note that not all Persistent volume types support mount options. In Kubernetes version 1.6, the following
|
||||
volume types support mount options.
|
||||
The following volume types support mount options:
|
||||
|
||||
* GCEPersistentDisk
|
||||
* AWSElasticBlockStore
|
||||
@@ -281,6 +284,22 @@ volume types support mount options.
|
||||
* Quobyte Volumes
|
||||
* VMware Photon
|
||||
|
||||
Mount options are not validated, so mount will simply fail if one is invalid.
|
||||
|
||||
In the past, the annotation `volume.beta.kubernetes.io/mount-options` was used instead
|
||||
of the `mountOptions` attribute. This annotation is still working, however
|
||||
it will become fully deprecated in a future Kubernetes release.
|
||||
|
||||
### Phase
|
||||
|
||||
A volume will be in one of the following phases:
|
||||
|
||||
* Available -- a free resource that is not yet bound to a claim
|
||||
* Bound -- the volume is bound to a claim
|
||||
* Released -- the claim has been deleted, but the resource is not yet reclaimed by the cluster
|
||||
* Failed -- the volume has failed its automatic reclamation
|
||||
|
||||
The CLI will show the name of the PVC bound to the PV.
|
||||
|
||||
## PersistentVolumeClaims
|
||||
|
||||
@@ -355,8 +374,10 @@ to Kubernetes cluster by addon manager during installation.
|
||||
|
||||
When a PVC specifies a `selector` in addition to requesting a `StorageClass`,
|
||||
the requirements are ANDed together: only a PV of the requested class and with
|
||||
the requested labels may be bound to the PVC. Note that currently, a PVC with a
|
||||
non-empty `selector` can't have a PV dynamically provisioned for it.
|
||||
the requested labels may be bound to the PVC.
|
||||
|
||||
**Note:** Currently, a PVC with a non-empty `selector` can't have a PV dynamically provisioned for it.
|
||||
{: .note}
|
||||
|
||||
In the past, the annotation `volume.beta.kubernetes.io/storage-class` was used instead
|
||||
of `storageClassName` attribute. This annotation is still working, however
|
||||
@@ -390,9 +411,9 @@ spec:
|
||||
|
||||
## StorageClasses
|
||||
|
||||
Each `StorageClass` contains the fields `provisioner` and `parameters`, which
|
||||
are used when a `PersistentVolume` belonging to the class needs to be
|
||||
dynamically provisioned.
|
||||
Each `StorageClass` contains the fields `provisioner`, `parameters`, and
|
||||
`reclaimPolicy`, which are used when a `PersistentVolume` belonging to the
|
||||
class needs to be dynamically provisioned.
|
||||
|
||||
The name of a `StorageClass` object is significant, and is how users can
|
||||
request a particular class. Administrators set the name and other parameters
|
||||
@@ -412,6 +433,9 @@ metadata:
|
||||
provisioner: kubernetes.io/aws-ebs
|
||||
parameters:
|
||||
type: gp2
|
||||
reclaimPolicy: Retain
|
||||
mountOptions:
|
||||
- debug
|
||||
```
|
||||
|
||||
### Provisioner
|
||||
@@ -455,13 +479,22 @@ There are also cases when 3rd party storage vendors provide their own external
|
||||
provisioner.
|
||||
|
||||
### Reclaim Policy
|
||||
Persistent Volumes that are dynamically created by a storage class will have a reclaim
|
||||
policy of `delete`. If that is not desired, the only current option is to edit the
|
||||
PV after it is created.
|
||||
Persistent Volumes that are dynamically created by a storage class will have the
|
||||
reclaim policy specified in the `reclaimPolicy` field of the class, which can be
|
||||
either `Delete` or `Retain`. If no `reclaimPolicy` is specified when a
|
||||
`StorageClass` object is created, it will default to `Delete`.
|
||||
|
||||
Persistent Volumes that are created manually and managed via a storage class will have
|
||||
whatever reclaim policy they were assigned at creation.
|
||||
|
||||
### Mount Options
|
||||
Persistent Volumes that are dynamically created by a storage class will have the
|
||||
mount options specified in the `mountOptions` field of the class.
|
||||
|
||||
If the volume plugin does not support mount options but mount options are
|
||||
specified, provisioning will fail. Mount options are not validated on neither
|
||||
the class nor PV, so mount of the PV will simply fail if one is invalid.
|
||||
|
||||
### Parameters
|
||||
Storage classes have parameters that describe volumes belonging to the storage
|
||||
class. Different parameters may be accepted depending on the `provisioner`. For
|
||||
@@ -795,8 +828,8 @@ provisioner: kubernetes.io/scaleio
|
||||
parameters:
|
||||
gateway: https://192.168.99.200:443/api
|
||||
system: scaleio
|
||||
protectionDomain: default
|
||||
storagePool: default
|
||||
protectionDomain: pd0
|
||||
storagePool: sp1
|
||||
storageMode: ThinProvisionned
|
||||
secretRef: sio-secret
|
||||
readOnly: false
|
||||
@@ -806,12 +839,12 @@ parameters:
|
||||
* `provisioner`: attribute is set to `kubernetes.io/scaleio`
|
||||
* `gateway`: address to a ScaleIO API gateway (required)
|
||||
* `system`: the name of the ScaleIO system (required)
|
||||
* `protectionDomain`: the name of the ScaleIO protection domain
|
||||
* `storagePool`: the name of the volume storage pool
|
||||
* `protectionDomain`: the name of the ScaleIO protection domain (required)
|
||||
* `storagePool`: the name of the volume storage pool (required)
|
||||
* `storageMode`: the storage provision mode: `ThinProvisionned` (default) or `ThickProvisionned`
|
||||
* `secretRef`: reference to a configured Secret object (required, see detail below)
|
||||
* `readOnly`: specifies the access mode to the mounted volume
|
||||
* `fsType`: the file system to use for the volume
|
||||
* `secretRef`: reference to a configured Secret object (required)
|
||||
* `readOnly`: specifies the access mode to the mounted volume (default false)
|
||||
* `fsType`: the file system to use for the volume (default ext4)
|
||||
|
||||
The ScaleIO Kubernetes volume plugin requires a configured Secret object.
|
||||
The secret must be created with type `kubernetes.io/scaleio` and use the same namespace value as that of the PVC where it is referenced
|
||||
|
||||
@@ -141,7 +141,7 @@ spec:
|
||||
### hostPath
|
||||
|
||||
A `hostPath` volume mounts a file or directory from the host node's filesystem
|
||||
into your pod. This is not something that most Pods will need, but it offers a
|
||||
into your pod. This is not something that most Pods will need, but it offers a
|
||||
powerful escape hatch for some applications.
|
||||
|
||||
For example, some uses for a `hostPath` are:
|
||||
@@ -149,6 +149,24 @@ For example, some uses for a `hostPath` are:
|
||||
* running a container that needs access to Docker internals; use a `hostPath`
|
||||
of `/var/lib/docker`
|
||||
* running cAdvisor in a container; use a `hostPath` of `/dev/cgroups`
|
||||
* allowing a pod to specify whether a given hostPath should exist prior to the
|
||||
pod running, whether it should be created, and what it should exist as
|
||||
|
||||
In addition to the required `path` property, user can optionally specify a `type` for a `hostPath` volume.
|
||||
|
||||
The supported values for field `type` are:
|
||||
|
||||
|
||||
| Value | Behavior |
|
||||
|:------|:---------|
|
||||
| | Empty string (default) is for backward compatibility, which means that no checks will be performed before mounting the hostPath volume. |
|
||||
| `DirectoryOrCreate` | If nothing exists at the given path, an empty directory will be created there as needed with permission set to 0755, having the same group and ownership with Kubelet. |
|
||||
| `Directory` | A directory must exist at the given path |
|
||||
| `FileOrCreate` | If nothing exists at the given path, an empty file will be created there as needed with permission set to 0644, having the same group and ownership with Kubelet. |
|
||||
| `File` | A file must exist at the given path |
|
||||
| `Socket` | A UNIX socket must exist at the given path |
|
||||
| `CharDevice` | A character device must exist at the given path |
|
||||
| `BlockDevice` | A block device must exist at the given path |
|
||||
|
||||
Watch out when using this type of volume, because:
|
||||
|
||||
@@ -156,7 +174,7 @@ Watch out when using this type of volume, because:
|
||||
behave differently on different nodes due to different files on the nodes
|
||||
* when Kubernetes adds resource-aware scheduling, as is planned, it will not be
|
||||
able to account for resources used by a `hostPath`
|
||||
* the directories created on the underlying hosts are only writable by root. You
|
||||
* the files or directories created on the underlying hosts are only writable by root. You
|
||||
either need to run your process as root in a
|
||||
[privileged container](/docs/user-guide/security-context) or modify the file
|
||||
permissions on the host to be able to write to a `hostPath` volume
|
||||
@@ -180,6 +198,8 @@ spec:
|
||||
hostPath:
|
||||
# directory location on host
|
||||
path: /data
|
||||
# this field is optional
|
||||
type: Directory
|
||||
```
|
||||
|
||||
### gcePersistentDisk
|
||||
@@ -549,14 +569,6 @@ with ConfigMap naming.
|
||||
volume source. However, as illustrated above, you can explicitly set the `mode`
|
||||
for each individual projection.
|
||||
|
||||
### FlexVolume
|
||||
|
||||
A `FlexVolume` enables users to mount vendor volumes into a pod. It expects vendor
|
||||
drivers are installed in the volume plugin path on each kubelet node. This is
|
||||
an alpha feature and may change in future.
|
||||
|
||||
More details are in [here](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/volumes/flexvolume/README.md).
|
||||
|
||||
### AzureFileVolume
|
||||
|
||||
A `AzureFileVolume` is used to mount a Microsoft Azure File Volume (SMB 2.1 and 3.0)
|
||||
@@ -699,6 +711,8 @@ spec:
|
||||
scaleIO:
|
||||
gateway: https://localhost:443/api
|
||||
system: scaleio
|
||||
protectionDomain: sd0
|
||||
storagePool: sp1
|
||||
volumeName: vol-0
|
||||
secretRef:
|
||||
name: sio-secret
|
||||
@@ -847,6 +861,67 @@ request a certain amount of space using a [resource](/docs/user-guide/compute-re
|
||||
specification, and to select the type of media to use, for clusters that have
|
||||
several media types.
|
||||
|
||||
## Out-of-Tree Volume Plugins
|
||||
In addition to the previously listed volume types, storage vendors may create
|
||||
custom plugins without adding it to the Kubernetes repository. This can be
|
||||
achieved by using the `FlexVolume` plugin.
|
||||
|
||||
`FlexVolume` enables users to mount vendor volumes into a pod. The vendor plugin
|
||||
is implemented using a driver, an executable supporting a list of volume commands
|
||||
defined by the `FlexVolume` API. Drivers must be installed in a pre-defined
|
||||
volume plugin path on each node. This is an alpha feature and may change in future.
|
||||
|
||||
More details can be found [here](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md).
|
||||
|
||||
|
||||
## Mount propagation
|
||||
|
||||
**Note:** Mount propagation is an alpha feature in Kubernetes 1.8 and may be
|
||||
redesigned or even removed in future releases.
|
||||
{: .note}
|
||||
|
||||
Mount propagation allows for sharing volumes mounted by a Container to
|
||||
other Containers in the same Pod, or even to other Pods on the same node.
|
||||
|
||||
If the MountPropagation feature is disabled, volume mounts in pods are not propagated.
|
||||
That is, Containers run with `private` mount propagation as described in the
|
||||
[Linux kernel documentation](https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt).
|
||||
|
||||
To enable this feature, specify `MountPropagation=true` in the
|
||||
`--feature-gates` command line option. When enabled, the `volumeMounts` field
|
||||
of a Container has a new `mountPropagation` subfield. Its values are:
|
||||
|
||||
* `HostToContainer` - This volume mount will receive all subsequent mounts
|
||||
that are mounted to this volume or any of its subdirectories. This is
|
||||
the default mode when the MountPropagation feature is enabled.
|
||||
|
||||
In other words, if the host mounts anything inside the volume mount, the
|
||||
Container will see it mounted there.
|
||||
|
||||
Similarly, if any pod with `Bidirectional` mount propagation to the same
|
||||
volume mounts anything there, the Container with `HostToContainer` mount
|
||||
propagation will see it.
|
||||
|
||||
This mode is equal to `rslave` mount propagation as described in the
|
||||
[Linux kernel documentation](https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt)
|
||||
|
||||
* `Bidirectional` - This volume mount behaves the same the `HostToContainer` mount.
|
||||
In addition, all volume mounts created by the Container will be propagated
|
||||
back to the host and to all Containers of all Pods that use the same volume.
|
||||
|
||||
A typical use case for this mode is a Pod with a Flex volume driver or
|
||||
a Pod that needs to mount something on the host using a HostPath volume.
|
||||
|
||||
This mode is equal to `rshared` mount propagation as described in the
|
||||
[Linux kernel documentation](https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt)
|
||||
|
||||
**Caution:** `Bidirectional` mount propagation can be dangerous. It can damage
|
||||
the host operating system and therefore it is allowed only in privileged
|
||||
Containers. Familiarity with Linux kernel behavior is strongly recommended.
|
||||
In addition, any volume mounts created by Containers in Pods must be destroyed
|
||||
(unmounted) by the Containers on termination.
|
||||
{: .caution}
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
|
||||
@@ -22,8 +22,10 @@ on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) forma
|
||||
**Note:** The question mark (`?`) in the schedule has the same meaning as an asterisk `*`,
|
||||
that is, it stands for any of available value for a given field.
|
||||
|
||||
**Note:** ScheduledJob resource was introduced in Kubernetes version 1.4, but starting
|
||||
from version 1.5 its current name is CronJob.
|
||||
**Note:** CronJob resource in `batch/v2alpha1` API group has been deprecated starting
|
||||
from cluster version 1.8. You should switch to using `batch/v1beta1`, instead, which is
|
||||
enabled by default in the API server. Further in this document, we will be using
|
||||
`batch/v1beta1` in all the examples.
|
||||
|
||||
A typical use case is:
|
||||
|
||||
@@ -32,8 +34,8 @@ A typical use case is:
|
||||
|
||||
### Prerequisites
|
||||
|
||||
You need a working Kubernetes cluster at version >= 1.4 (for ScheduledJob), >= 1.5 (for CronJob),
|
||||
with batch/v2alpha1 API turned on by passing `--runtime-config=batch/v2alpha1=true` while bringing up
|
||||
You need a working Kubernetes cluster at version >= 1.8 (for CronJob). For previous versions of cluster (< 1.8)
|
||||
you need to explicitly enable `batch/v2alpha1` API by passing `--runtime-config=batch/v2alpha1=true` to
|
||||
the API server (see [Turn on or off an API version for your cluster](/docs/admin/cluster-management/#turn-on-or-off-an-api-version-for-your-cluster)
|
||||
for more).
|
||||
|
||||
@@ -111,25 +113,8 @@ $ kubectl delete cronjob hello
|
||||
cronjob "hello" deleted
|
||||
```
|
||||
|
||||
This stops new jobs from being created. However, running jobs won't be stopped, and no jobs or their pods will
|
||||
be deleted. To clean up those jobs and pods, you need to list all jobs created by the cron job, and delete them all:
|
||||
|
||||
```shell
|
||||
$ kubectl get jobs
|
||||
NAME DESIRED SUCCESSFUL AGE
|
||||
hello-1201907962 1 1 11m
|
||||
hello-1202039034 1 1 8m
|
||||
...
|
||||
|
||||
$ kubectl delete jobs hello-1201907962 hello-1202039034 ...
|
||||
job "hello-1201907962" deleted
|
||||
job "hello-1202039034" deleted
|
||||
...
|
||||
```
|
||||
|
||||
Once the jobs are deleted, the pods created by them are deleted as well. Note that all jobs created by cron
|
||||
job "hello" will be prefixed "hello-". You can delete them at once with `kubectl delete jobs --all`, if you want to
|
||||
delete all jobs in the current namespace (not just the ones created by "hello").
|
||||
This stops new jobs from being created and removes all the jobs and pods created by this cronjob.
|
||||
You can read more about it in [garbage collection section](/docs/concepts/workloads/controllers/garbage-collection/).
|
||||
|
||||
## Cron Job Limitations
|
||||
|
||||
@@ -187,6 +172,7 @@ apply to already started executions. Defaults to false.
|
||||
|
||||
### Jobs History Limits
|
||||
|
||||
The `.spec.successfulJobsHistoryLimit` and `.spec.failedJobsHistoryLimit` fields are optional. These fields specify how many completed and failed jobs should be kept.
|
||||
|
||||
By default, there are no limits, and all successful and failed jobs are kept. However, jobs can pile up quickly when running a cron job, and setting these fields is recommended. Setting a limit to `0` corresponds to keeping none of the corresponding kind of jobs after they finish.
|
||||
The `.spec.successfulJobsHistoryLimit` and `.spec.failedJobsHistoryLimit` fields are optional.
|
||||
These fields specify how many completed and failed jobs should be kept. By default, they are
|
||||
set to 3 and 1 respectively. Setting a limit to `0` corresponds to keeping none of the corresponding
|
||||
kind of jobs after they finish.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
apiVersion: batch/v2alpha1
|
||||
apiVersion: batch/v1beta1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: hello
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
---
|
||||
approvers:
|
||||
- enisoc
|
||||
- erictune
|
||||
- foxish
|
||||
- janetkuo
|
||||
- kow3ns
|
||||
title: Daemon Sets
|
||||
---
|
||||
|
||||
@@ -9,9 +13,9 @@ title: Daemon Sets
|
||||
|
||||
## What is a DaemonSet?
|
||||
|
||||
A _DaemonSet_ ensures that all (or some) nodes run a copy of a pod. As nodes are added to the
|
||||
cluster, pods are added to them. As nodes are removed from the cluster, those pods are garbage
|
||||
collected. Deleting a DaemonSet will clean up the pods it created.
|
||||
A _DaemonSet_ ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the
|
||||
cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage
|
||||
collected. Deleting a DaemonSet will clean up the Pods it created.
|
||||
|
||||
Some typical uses of a DaemonSet are:
|
||||
|
||||
@@ -47,20 +51,26 @@ A DaemonSet also needs a [`.spec`](https://git.k8s.io/community/contributors/dev
|
||||
|
||||
### Pod Template
|
||||
|
||||
The `.spec.template` is the only required field of the `.spec`.
|
||||
The `.spec.template` is one of the required fields in `.spec`.
|
||||
|
||||
The `.spec.template` is a [pod template](/docs/concepts/workloads/pods/pod-overview/#pod-templates). It has exactly the same schema as a [pod](/docs/concepts/workloads/pods/pod/), except it is nested and does not have an `apiVersion` or `kind`.
|
||||
The `.spec.template` is a [pod template](/docs/concepts/workloads/pods/pod-overview/#pod-templates). It has exactly the same schema as a [Pod](/docs/concepts/workloads/pods/pod/), except it is nested and does not have an `apiVersion` or `kind`.
|
||||
|
||||
In addition to required fields for a pod, a pod template in a DaemonSet has to specify appropriate
|
||||
In addition to required fields for a Pod, a Pod template in a DaemonSet has to specify appropriate
|
||||
labels (see [pod selector](#pod-selector)).
|
||||
|
||||
A pod template in a DaemonSet must have a [`RestartPolicy`](/docs/user-guide/pod-states)
|
||||
A Pod Template in a DaemonSet must have a [`RestartPolicy`](/docs/user-guide/pod-states)
|
||||
equal to `Always`, or be unspecified, which defaults to `Always`.
|
||||
|
||||
### Pod Selector
|
||||
|
||||
The `.spec.selector` field is a pod selector. It works the same as the `.spec.selector` of
|
||||
a [Job](/docs/concepts/jobs/run-to-completion-finite-workloads/) or other new resources.
|
||||
a [Job](/docs/concepts/jobs/run-to-completion-finite-workloads/).
|
||||
|
||||
As of Kubernetes 1.8, you must specify a pod selector that matches the labels of the
|
||||
`.spec.template`. The pod selector will no longer be defaulted when left empty. Selector
|
||||
defaulting was not compatible with `kubectl apply`. Also, once a DaemonSet is created,
|
||||
its `spec.selector` can not be mutated. Mutating the pod selector can lead to the
|
||||
unintentional orphaning of Pods, and it was found to be confusing to users.
|
||||
|
||||
The `spec.selector` is an object consisting of two fields:
|
||||
|
||||
@@ -73,53 +83,58 @@ When the two are specified the result is ANDed.
|
||||
If the `.spec.selector` is specified, it must match the `.spec.template.metadata.labels`. If not
|
||||
specified, they are defaulted to be equal. Config with these not matching will be rejected by the API.
|
||||
|
||||
Also you should not normally create any pods whose labels match this selector, either directly, via
|
||||
another DaemonSet, or via other controller such as ReplicationController. Otherwise, the DaemonSet
|
||||
controller will think that those pods were created by it. Kubernetes will not stop you from doing
|
||||
this. One case where you might want to do this is manually create a pod with a different value on
|
||||
Also you should not normally create any Pods whose labels match this selector, either directly, via
|
||||
another DaemonSet, or via other controller such as ReplicaSet. Otherwise, the DaemonSet
|
||||
controller will think that those Pods were created by it. Kubernetes will not stop you from doing
|
||||
this. One case where you might want to do this is manually create a Pod with a different value on
|
||||
a node for testing.
|
||||
|
||||
If you attempt to create a DaemonSet such that
|
||||
|
||||
### Running Pods on Only Some Nodes
|
||||
|
||||
If you specify a `.spec.template.spec.nodeSelector`, then the DaemonSet controller will
|
||||
create pods on nodes which match that [node
|
||||
create Pods on nodes which match that [node
|
||||
selector](/docs/concepts/configuration/assign-pod-node/). Likewise if you specify a `.spec.template.spec.affinity`,
|
||||
then DaemonSet controller will create pods on nodes which match that [node affinity](/docs/concepts/configuration/assign-pod-node/).
|
||||
If you do not specify either, then the DaemonSet controller will create pods on all nodes.
|
||||
then DaemonSet controller will create Pods on nodes which match that [node affinity](/docs/concepts/configuration/assign-pod-node/).
|
||||
If you do not specify either, then the DaemonSet controller will create Pods on all nodes.
|
||||
|
||||
## How Daemon Pods are Scheduled
|
||||
|
||||
Normally, the machine that a pod runs on is selected by the Kubernetes scheduler. However, pods
|
||||
created by the Daemon controller have the machine already selected (`.spec.nodeName` is specified
|
||||
when the pod is created, so it is ignored by the scheduler). Therefore:
|
||||
Normally, the machine that a Pod runs on is selected by the Kubernetes scheduler. However, Pods
|
||||
created by the DaemonSet controller have the machine already selected (`.spec.nodeName` is specified
|
||||
when the Pod is created, so it is ignored by the scheduler). Therefore:
|
||||
|
||||
- The [`unschedulable`](/docs/admin/node/#manual-node-administration) field of a node is not respected
|
||||
by the DaemonSet controller.
|
||||
- DaemonSet controller can make pods even when the scheduler has not been started, which can help cluster
|
||||
- The DaemonSet controller can make Pods even when the scheduler has not been started, which can help cluster
|
||||
bootstrap.
|
||||
|
||||
Daemon pods do respect [taints and tolerations](/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature),
|
||||
Daemon Pods do respect [taints and tolerations](/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature),
|
||||
but they are created with `NoExecute` tolerations for the following taints with no `tolerationSeconds`:
|
||||
|
||||
- `node.alpha.kubernetes.io/notReady`
|
||||
- `node.alpha.kubernetes.io/unreachable`
|
||||
- `node.alpha.kubernetes.io/memoryPressure`
|
||||
- `node.alpha.kubernetes.io/diskPressure`
|
||||
|
||||
When the support to critical pods is enabled and the pods in a DaemonSet are
|
||||
labelled as critical, the Daemon pods are created with an additional
|
||||
`NoExecute` toleration for the `node.alpha.kubernetes.io/outOfDisk` taint with
|
||||
no `tolerationSeconds`.
|
||||
|
||||
This ensures that when the `TaintBasedEvictions` alpha feature is enabled,
|
||||
they will not be evicted when there are node problems such as a network partition. (When the
|
||||
`TaintBasedEvictions` feature is not enabled, they are also not evicted in these scenarios, but
|
||||
due to hard-coded behavior of the NodeController rather than due to tolerations).
|
||||
|
||||
They also tolerate following `NoSchedule` taints:
|
||||
|
||||
- `node.kubernetes.io/memory-pressure`
|
||||
- `node.kubernetes.io/disk-pressure`
|
||||
|
||||
When the support to critical pods is enabled and the pods in a DaemonSet are
|
||||
labelled as critical, the Daemon pods are created with an additional
|
||||
`NoSchedule` toleration for the `node.kubernetes.io/out-of-disk` taint.
|
||||
|
||||
Note that all above `NoSchedule` taints above are created only in version 1.8 or later if the alpha feature `TaintNodesByCondition` is enabled.
|
||||
|
||||
## Communicating with Daemon Pods
|
||||
|
||||
Some possible patterns for communicating with pods in a DaemonSet are:
|
||||
Some possible patterns for communicating with Pods in a DaemonSet are:
|
||||
|
||||
- **Push**: Pods in the DaemonSet are configured to send updates to another service, such
|
||||
as a stats database. They do not have clients.
|
||||
@@ -127,24 +142,24 @@ Some possible patterns for communicating with pods in a DaemonSet are:
|
||||
- **DNS**: Create a [headless service](/docs/concepts/services-networking/service/#headless-services) with the same pod selector,
|
||||
and then discover DaemonSets using the `endpoints` resource or retrieve multiple A records from
|
||||
DNS.
|
||||
- **Service**: Create a service with the same pod selector, and use the service to reach a
|
||||
- **Service**: Create a service with the same Pod selector, and use the service to reach a
|
||||
daemon on a random node. (No way to reach specific node.)
|
||||
|
||||
## Updating a DaemonSet
|
||||
|
||||
If node labels are changed, the DaemonSet will promptly add pods to newly matching nodes and delete
|
||||
pods from newly not-matching nodes.
|
||||
If node labels are changed, the DaemonSet will promptly add Pods to newly matching nodes and delete
|
||||
Pods from newly not-matching nodes.
|
||||
|
||||
You can modify the pods that a DaemonSet creates. However, pods do not allow all
|
||||
You can modify the Pods that a DaemonSet creates. However, Pods do not allow all
|
||||
fields to be updated. Also, the DaemonSet controller will use the original template the next
|
||||
time a node (even with the same name) is created.
|
||||
|
||||
|
||||
You can delete a DaemonSet. If you specify `--cascade=false` with `kubectl`, then the pods
|
||||
You can delete a DaemonSet. If you specify `--cascade=false` with `kubectl`, then the Pods
|
||||
will be left on the nodes. You can then create a new DaemonSet with a different template.
|
||||
the new DaemonSet with the different template will recognize all the existing pods as having
|
||||
matching labels. It will not modify or delete them despite a mismatch in the pod template.
|
||||
You will need to force new pod creation by deleting the pod or deleting the node.
|
||||
The new DaemonSet with the different template will recognize all the existing Pods as having
|
||||
matching labels. It will not modify or delete them despite a mismatch in the Pod template.
|
||||
You will need to force new Pod creation by deleting the Pod or deleting the node.
|
||||
|
||||
In Kubernetes version 1.6 and later, you can [perform a rolling update](/docs/tasks/manage-daemon/update-daemon-set/) on a DaemonSet.
|
||||
|
||||
@@ -159,35 +174,35 @@ It is certainly possible to run daemon processes by directly starting them on a
|
||||
running such processes via a DaemonSet:
|
||||
|
||||
- Ability to monitor and manage logs for daemons in the same way as applications.
|
||||
- Same config language and tools (e.g. pod templates, `kubectl`) for daemons and applications.
|
||||
- Same config language and tools (e.g. Pod templates, `kubectl`) for daemons and applications.
|
||||
- Future versions of Kubernetes will likely support integration between DaemonSet-created
|
||||
pods and node upgrade workflows.
|
||||
Pods and node upgrade workflows.
|
||||
- Running daemons in containers with resource limits increases isolation between daemons from app
|
||||
containers. However, this can also be accomplished by running the daemons in a container but not in a pod
|
||||
containers. However, this can also be accomplished by running the daemons in a container but not in a Pod
|
||||
(e.g. start directly via Docker).
|
||||
|
||||
### Bare Pods
|
||||
|
||||
It is possible to create pods directly which specify a particular node to run on. However,
|
||||
a DaemonSet replaces pods that are deleted or terminated for any reason, such as in the case of
|
||||
It is possible to create Pods directly which specify a particular node to run on. However,
|
||||
a DaemonSet replaces Pods that are deleted or terminated for any reason, such as in the case of
|
||||
node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, you should
|
||||
use a DaemonSet rather than creating individual pods.
|
||||
use a DaemonSet rather than creating individual Pods.
|
||||
|
||||
### Static Pods
|
||||
|
||||
It is possible to create pods by writing a file to a certain directory watched by Kubelet. These
|
||||
It is possible to create Pods by writing a file to a certain directory watched by Kubelet. These
|
||||
are called [static pods](/docs/concepts/cluster-administration/static-pod/).
|
||||
Unlike DaemonSet, static pods cannot be managed with kubectl
|
||||
or other Kubernetes API clients. Static pods do not depend on the apiserver, making them useful
|
||||
in cluster bootstrapping cases. Also, static pods may be deprecated in the future.
|
||||
Unlike DaemonSet, static Pods cannot be managed with kubectl
|
||||
or other Kubernetes API clients. Static Pods do not depend on the apiserver, making them useful
|
||||
in cluster bootstrapping cases. Also, static Pods may be deprecated in the future.
|
||||
|
||||
### Replication Controller
|
||||
### Deployments
|
||||
|
||||
DaemonSet are similar to [Replication Controllers](/docs/user-guide/replication-controller) in that
|
||||
they both create pods, and those pods have processes which are not expected to terminate (e.g. web servers,
|
||||
DaemonSets are similar to [Deployments](/docs/concepts/workloads/controllers/deployment.md) in that
|
||||
they both create Pods, and those Pods have processes which are not expected to terminate (e.g. web servers,
|
||||
storage servers).
|
||||
|
||||
Use a replication controller for stateless services, like frontends, where scaling up and down the
|
||||
Use a Deployment for stateless services, like frontends, where scaling up and down the
|
||||
number of replicas and rolling out updates are more important than controlling exactly which host
|
||||
the pod runs on. Use a Daemon Controller when it is important that a copy of a pod always run on
|
||||
all or certain hosts, and when it needs to start before other pods.
|
||||
the Pod runs on. Use a DaemonSet when it is important that a copy of a Pod always run on
|
||||
all or certain hosts, and when it needs to start before other Pods.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
apiVersion: apps/v1beta2
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: fluentd-elasticsearch
|
||||
@@ -6,6 +6,9 @@ metadata:
|
||||
labels:
|
||||
k8s-app: fluentd-logging
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
name: fluentd-elasticsearch
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
@@ -33,4 +36,4 @@ spec:
|
||||
path: /var/log
|
||||
- name: varlibdockercontainers
|
||||
hostPath:
|
||||
path: /var/lib/docker/containers
|
||||
path: /var/lib/docker/containers
|
||||
|
||||
@@ -288,6 +288,9 @@ It is generally discouraged to make label selector updates and it is suggested t
|
||||
In any case, if you need to perform a label selector update, exercise great caution and make sure you have grasped
|
||||
all of the implications.
|
||||
|
||||
**Note:** In API version `apps/v1beta2`, a Deployment's label selector is immutable after it gets created.
|
||||
{: .note}
|
||||
|
||||
* Selector additions require the pod template labels in the Deployment spec to be updated with the new label too,
|
||||
otherwise a validation error is returned. This change is a non-overlapping one, meaning that the new selector does
|
||||
not select ReplicaSets and Pods created with the old selector, resulting in orphaning all old ReplicaSets and
|
||||
@@ -850,9 +853,9 @@ allowed, which is the default if not specified.
|
||||
`.spec.selector` is an optional field that specifies a [label selector](/docs/concepts/overview/working-with-objects/labels/)
|
||||
for the Pods targeted by this deployment.
|
||||
|
||||
If specified, `.spec.selector` must match `.spec.template.metadata.labels`, or it will be rejected by
|
||||
the API. If `.spec.selector` is unspecified, `.spec.selector.matchLabels` defaults to
|
||||
`.spec.template.metadata.labels`.
|
||||
`.spec.selector` must match `.spec.template.metadata.labels`, or it will be rejected by the API.
|
||||
|
||||
In API version `apps/v1beta2`, `.spec.selector` and `.metadata.labels` no longer default to `.spec.template.metadata.labels` if not set. So they must be set explicitly. Also note that `.spec.selector` is immutable after creation of the Deployment in `apps/v1beta2`.
|
||||
|
||||
A Deployment may terminate Pods whose labels match the selector if their template is different
|
||||
from `.spec.template` or if the total number of such Pods exceeds `.spec.replicas`. It brings up new
|
||||
@@ -926,20 +929,7 @@ a Pod is considered ready, see [Container Probes](/docs/concepts/workloads/pods/
|
||||
|
||||
### Rollback To
|
||||
|
||||
`.spec.rollbackTo` is an optional field with the configuration the Deployment
|
||||
should roll back to. Setting this field triggers a rollback, and this field will
|
||||
be cleared by the server after a rollback is done.
|
||||
|
||||
Because this field will be cleared by the server, it should not be used
|
||||
declaratively. For example, you should not perform `kubectl apply` with a
|
||||
manifest with `.spec.rollbackTo` field set.
|
||||
|
||||
#### Revision
|
||||
|
||||
`.spec.rollbackTo.revision` is an optional field specifying the revision to roll
|
||||
back to. Setting to 0 means rolling back to the last revision in history;
|
||||
otherwise, means rolling back to the specified revision. This defaults to 0 when
|
||||
[`spec.rollbackTo`](#rollback-to) is set.
|
||||
Field `.spec.rollbackTo` has been deprecated in API versions `extensions/v1beta1` and `apps/v1beta1`, and is no longer supported in API version `apps/v1beta2`. Instead, `kubectl rollout undo` as introduced in [Rolling Back to a Previous Revision](#rolling-back-to-a-previous-revision) should be used.
|
||||
|
||||
### Revision History Limit
|
||||
|
||||
|
||||
@@ -1,20 +1,14 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
apiVersion: apps/v1beta2 # for versions before 1.6.0 use extensions/v1beta1
|
||||
kind: ReplicaSet
|
||||
metadata:
|
||||
name: frontend
|
||||
# these labels can be applied automatically
|
||||
# from the labels in the pod template if not set
|
||||
# labels:
|
||||
# app: guestbook
|
||||
# tier: frontend
|
||||
labels:
|
||||
app: guestbook
|
||||
tier: frontend
|
||||
spec:
|
||||
# this replicas value is default
|
||||
# modify it according to your case
|
||||
replicas: 3
|
||||
# selector can be applied automatically
|
||||
# from the labels in the pod template if not set,
|
||||
# but we are specifying the selector here to
|
||||
# demonstrate its usage.
|
||||
selector:
|
||||
matchLabels:
|
||||
tier: frontend
|
||||
|
||||
@@ -155,11 +155,8 @@ is not used, only the ReplicaSets will be deleted, and the Pods will be orphaned
|
||||
See [kubeadm/#149](https://github.com/kubernetes/kubeadm/issues/149#issuecomment-284766613) for more information.
|
||||
|
||||
## Known issues
|
||||
* As of 1.7, garbage collection does not yet support
|
||||
[custom resources](/docs/concepts/api-extension/custom-resources/),
|
||||
such as those added through CustomResourceDefinition or aggregated API servers.
|
||||
|
||||
[Other known issues](https://github.com/kubernetes/kubernetes/issues/26120)
|
||||
Tracked at [#26120](https://github.com/kubernetes/kubernetes/issues/26120)
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
@@ -12,4 +12,5 @@ spec:
|
||||
image: perl
|
||||
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
|
||||
restartPolicy: Never
|
||||
backoffLimit: 4
|
||||
|
||||
|
||||
@@ -196,6 +196,12 @@ sometimes be started twice.
|
||||
If you do specify `.spec.parallelism` and `.spec.completions` both greater than 1, then there may be
|
||||
multiple pods running at once. Therefore, your pods must also be tolerant of concurrency.
|
||||
|
||||
### Pod Backoff failure policy
|
||||
|
||||
There are situations where you want to fail a Job after some amount of retries due to a logical error in configuration etc.
|
||||
To do so set `.spec.template.spec.backoffLimit` to specify the number of retries before considering a Job as failed.
|
||||
The back-off limit is set by default to 6. Failed Pods associated with the Job are recreated by the Job controller with an exponential back-off delay (10s, 20s, 40s ...) capped at six minutes, The back-off limit is reset if no new failed Pods appear before the Job's next status check.
|
||||
|
||||
## Job Termination and Cleanup
|
||||
|
||||
When a Job completes, no more Pods are created, but the Pods are not deleted either. Since they are terminated,
|
||||
@@ -230,6 +236,7 @@ spec:
|
||||
image: perl
|
||||
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
|
||||
restartPolicy: Never
|
||||
backoffLimit: 5
|
||||
```
|
||||
|
||||
Note that both the Job Spec and the Pod Template Spec within the Job have a field with the same name.
|
||||
|
||||
@@ -1,9 +1,14 @@
|
||||
apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
|
||||
apiVersion: apps/v1beta2 # for versions before 1.7.0 use apps/v1beta1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: nginx-deployment
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nginx
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
|
||||
@@ -1,441 +0,0 @@
|
||||
---
|
||||
approvers:
|
||||
- bprashanth
|
||||
- enisoc
|
||||
- erictune
|
||||
- foxish
|
||||
- janetkuo
|
||||
- kow3ns
|
||||
- smarterclayton
|
||||
title: PetSets
|
||||
---
|
||||
|
||||
__Warning:__ Starting in Kubernetes version 1.5, PetSet has been renamed to [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets). To use (or continue to use) PetSet in Kubernetes 1.5, you _must_ [migrate](/docs/tasks/manage-stateful-set/upgrade-pet-set-to-stateful-set/) your existing PetSets to StatefulSets. For information on working with StatefulSet, see the tutorial on [how to run replicated stateful applications](/docs/tasks/run-application/run-replicated-stateful-application/).
|
||||
|
||||
__This document has been deprecated__, but can still apply if you're using
|
||||
Kubernetes version 1.4 or earlier.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
__Terminology__
|
||||
|
||||
Throughout this doc you will see a few terms that are sometimes used interchangeably elsewhere, that might cause confusion. This section attempts to clarify them.
|
||||
|
||||
* Node: A single virtual or physical machine in a Kubernetes cluster.
|
||||
* Cluster: A group of nodes in a single failure domain, unless mentioned otherwise.
|
||||
* Persistent Volume Claim (PVC): A request for storage, typically a [persistent volume](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/).
|
||||
* Host name: The hostname attached to the UTS namespace of the pod, i.e. the output of `hostname` in the pod.
|
||||
* DNS/Domain name: A *cluster local* domain name resolvable using standard methods (e.g.: [gethostbyname](http://linux.die.net/man/3/gethostbyname)).
|
||||
* Ordinality: the property of being "ordinal", or occupying a position in a sequence.
|
||||
* Pet: a single member of a PetSet; more generally, a stateful application.
|
||||
* Peer: a process running a server, capable of communicating with other such processes.
|
||||
|
||||
__Prerequisites__
|
||||
|
||||
This doc assumes familiarity with the following Kubernetes concepts:
|
||||
|
||||
* [Pods](/docs/user-guide/pods/single-container/)
|
||||
* [Cluster DNS](/docs/concepts/services-networking/dns-pod-service/)
|
||||
* [Headless Services](/docs/concepts/services-networking/service/#headless-services)
|
||||
* [Persistent Volumes](/docs/concepts/storage/persistent-volumes/)
|
||||
* [Persistent Volume Provisioning](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/persistent-volume-provisioning/README.md)
|
||||
|
||||
You need a working Kubernetes cluster at version >= 1.3, with a healthy DNS [cluster addon](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/README.md) at version >= 15. You cannot use PetSet on a hosted Kubernetes provider that has disabled `alpha` resources.
|
||||
|
||||
## What is a PetSet?
|
||||
|
||||
In Kubernetes, most pod management abstractions group them into disposable units of work that compose a micro service. Replication controllers for example, are designed with a weak guarantee - that there should be N replicas of a particular pod template. The pods are treated as stateless units, if one of them is unhealthy or superseded by a newer version, the system just disposes it.
|
||||
|
||||
```
|
||||
foo.default.svc.cluster.local
|
||||
|service|
|
||||
/ \
|
||||
| pod-asdf | | pod-zxcv |
|
||||
```
|
||||
|
||||
A PetSet, in contrast, is a group of stateful pods that require a stronger notion of identity. The document refers to these as "clustered applications".
|
||||
|
||||
```
|
||||
*.foo.default.svc.cluster.local
|
||||
| mysql-0 | <-> | mysql-1 |
|
||||
[pv 0] [pv 1]
|
||||
```
|
||||
|
||||
The co-ordinated deployment of clustered applications is notoriously hard. They require stronger notions of identity and membership, which they use in opaque internal protocols, and are especially prone to race conditions and deadlock. Traditionally administrators have deployed these applications by leveraging nodes as stable, long-lived entities with persistent storage and static ips.
|
||||
|
||||
The goal of PetSet is to decouple this dependency by assigning identities to individual instances of an application that are not anchored to the underlying physical infrastructure. For the rest of this document we will refer to these entities as "Pets". Our use of this term is predated by the "Pets vs Cattle" analogy.
|
||||
|
||||
__Relationship between Pets and Pods__: PetSet requires there be {0..N-1} Pets. Each Pet has a deterministic name - PetSetName-Ordinal, and a unique identity. Each Pet has at most one pod, and each PetSet has at most one Pet with a given identity.
|
||||
|
||||
## When to use PetSet?
|
||||
|
||||
A PetSet ensures that a specified number of "pets" with unique identities are running at any given time. The identity of a Pet is comprised of:
|
||||
|
||||
* a stable hostname, available in DNS
|
||||
* an ordinal index
|
||||
* stable storage: linked to the ordinal & hostname
|
||||
|
||||
These properties are useful in deploying stateful applications. However most stateful applications are also clustered, meaning they form groups with strict membership requirements that rely on stored state. PetSet also helps with the 2 most common problems encountered managing such clustered applications:
|
||||
|
||||
* discovery of peers for quorum
|
||||
* startup/teardown ordering
|
||||
|
||||
Only use PetSet if your application requires some or all of these properties. Managing pods as stateless replicas is vastly easier.
|
||||
|
||||
Example workloads for PetSet:
|
||||
|
||||
* Databases like MySQL or PostgreSQL that require a single instance attached to an NFS persistent volume at any time
|
||||
* Clustered software like Zookeeper, Etcd, or Elasticsearch that require stable membership.
|
||||
|
||||
## Alpha limitations
|
||||
|
||||
Before you start deploying applications as PetSets, there are a few limitations you should understand.
|
||||
|
||||
* PetSet is an *alpha* resource, not available in any Kubernetes release prior to 1.3.
|
||||
* As with all alpha/beta resources, it can be disabled through the `--runtime-config` option passed to the apiserver, and in fact most likely will be disabled on hosted offerings of Kubernetes.
|
||||
* The only updatable field on a PetSet is `replicas`.
|
||||
* The storage for a given pet must either be provisioned by a [persistent volume provisioner](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/persistent-volume-provisioning/README.md) based on the requested `storage class`, or pre-provisioned by an admin. Note that persistent volume provisioning is also currently in alpha.
|
||||
* Deleting and/or scaling a PetSet down will *not* delete the volumes associated with the PetSet. This is done to ensure safety first, your data is more valuable than an auto purge of all related PetSet resources. **Deleting the Persistent Volume Claims will result in a deletion of the associated volumes**.
|
||||
* All PetSets currently require a "governing service", or a Service responsible for the network identity of the pets. The user is responsible for this Service.
|
||||
* Updating an existing PetSet is currently a manual process, meaning you either need to deploy a new PetSet with the new image version, or orphan Pets one by one, update their image, and join them back to the cluster.
|
||||
|
||||
## Example PetSet
|
||||
|
||||
We'll create a basic PetSet to demonstrate how Pets are assigned unique and "sticky" identities.
|
||||
|
||||
{% include code.html language="yaml" file="petset.yaml" ghlink="/docs/concepts/workloads/controllers/petset.yaml" %}
|
||||
|
||||
Saving this config into `petset.yaml` and submitting it to a Kubernetes cluster should create the defined PetSet and Pets it manages:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f petset.yaml
|
||||
service "nginx" created
|
||||
petset "web" created
|
||||
```
|
||||
|
||||
## Pet Identity
|
||||
|
||||
The identity of a Pet sticks to it, regardless of which node it's (re) scheduled on. We can examine the identity of the pets we just created.
|
||||
|
||||
### Ordinal index
|
||||
|
||||
you should see 2 pods with predictable names formatted thus: `$(petset name)-$(ordinal index assigned by petset controller)`
|
||||
|
||||
```shell
|
||||
$ kubectl get po
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
web-0 1/1 Running 0 10m
|
||||
web-1 1/1 Running 0 10m
|
||||
```
|
||||
|
||||
### Stable storage
|
||||
|
||||
2 persistent volumes, one per pod. This is auto created by the PetSet based on the `volumeClaimTemplate` field
|
||||
|
||||
```shell
|
||||
$ kubectl get pv
|
||||
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
|
||||
pvc-90234946-3717-11e6-a46e-42010af00002 1Gi RWO Bound default/www-web-0 11m
|
||||
pvc-902733c2-3717-11e6-a46e-42010af00002 1Gi RWO Bound default/www-web-1 11m
|
||||
```
|
||||
|
||||
### Network identity
|
||||
|
||||
The network identity has 2 parts. First, we created a headless Service that controls the domain within which we create Pets. The domain managed by this Service takes the form: `$(service name).$(namespace).svc.cluster.local`, where "cluster.local" is the [cluster domain](/docs/concepts/services-networking/dns-pod-service/). As each pet is created, it gets a matching DNS subdomain, taking the form: `$(petname).$(governing service domain)`, where the governing service is defined by the `serviceName` field on the PetSet.
|
||||
|
||||
Here are some examples of choices for Cluster Domain, Service name, PetSet name, and how that affects the DNS names for the Pets and the hostnames in the Pet's pods:
|
||||
|
||||
Cluster Domain | Service (ns/name) | PetSet (ns/name) | PetSet Domain | Pet DNS | Pet Hostname |
|
||||
-------------- | ----------------- | ----------------- | -------------- | ------- | ------------ |
|
||||
cluster.local | default/nginx | default/web | nginx.default.svc.cluster.local | web-{0..N-1}.nginx.default.svc.cluster.local | web-{0..N-1} |
|
||||
cluster.local | foo/nginx | foo/web | nginx.foo.svc.cluster.local | web-{0..N-1}.nginx.foo.svc.cluster.local | web-{0..N-1} |
|
||||
kube.local | foo/nginx | foo/web | nginx.foo.svc.kube.local | web-{0..N-1}.nginx.foo.svc.kube.local | web-{0..N-1} |
|
||||
|
||||
Note that Cluster Domain will be set to `cluster.local` unless [otherwise configured](https://github.com/kubernetes/kubernetes/blob/master/examples/cluster-dns/README.md).
|
||||
|
||||
Let's verify our assertion with a simple test.
|
||||
|
||||
```shell
|
||||
$ kubectl get svc
|
||||
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
nginx None <none> 80/TCP 12m
|
||||
...
|
||||
```
|
||||
|
||||
First, the PetSet provides a stable hostname:
|
||||
|
||||
```shell
|
||||
$ for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname'; done
|
||||
web-0
|
||||
web-1
|
||||
```
|
||||
|
||||
And the hostname is linked to the in-cluster DNS address:
|
||||
|
||||
```shell
|
||||
$ kubectl run -i --tty --image busybox dns-test --restart=Never /bin/sh
|
||||
dns-test # nslookup web-0.nginx
|
||||
Server: 10.0.0.10
|
||||
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
|
||||
|
||||
Name: web-0.nginx
|
||||
Address 1: 10.180.3.5
|
||||
|
||||
dns-test # nslookup web-1.nginx
|
||||
Server: 10.0.0.10
|
||||
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
|
||||
|
||||
Name: web-1.nginx
|
||||
Address 1: 10.180.0.9
|
||||
```
|
||||
|
||||
The containers are running nginx webservers, which by default will look for an index.html file in `/usr/share/nginx/html/index.html`. That directory is backed by a `PersistentVolume` created by the PetSet. So let's write our hostname there:
|
||||
|
||||
```shell
|
||||
$ for i in 0 1; do
|
||||
kubectl exec web-$i -- sh -c 'echo $(hostname) > /usr/share/nginx/html/index.html';
|
||||
done
|
||||
```
|
||||
|
||||
And verify each webserver serves its own hostname:
|
||||
|
||||
```shell
|
||||
$ for i in 0 1; do kubectl exec -it web-$i -- curl localhost; done
|
||||
web-0
|
||||
web-1
|
||||
```
|
||||
|
||||
Now delete all pods in the petset:
|
||||
|
||||
```shell
|
||||
$ kubectl delete po -l app=nginx
|
||||
pod "web-0" deleted
|
||||
pod "web-1" deleted
|
||||
```
|
||||
|
||||
Wait for them to come back up, and try to retrieve the previously written hostname through the DNS name of the peer. They match, because the storage, DNS name, and hostname stick to the Pet no matter where it gets scheduled:
|
||||
|
||||
```shell
|
||||
$ kubectl exec -it web-1 -- curl web-0.nginx
|
||||
web-0
|
||||
$ kubectl exec -it web-0 -- curl web-1.nginx
|
||||
web-1
|
||||
```
|
||||
|
||||
## Peer discovery
|
||||
|
||||
A pet can piece together its own identity:
|
||||
|
||||
1. Use the [downward api](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) to find its pod name
|
||||
2. Run `hostname` to find its DNS name
|
||||
3. Run `mount` or `df` to find its volumes (usually this is unnecessary)
|
||||
|
||||
It's not necessary to "discover" the governing Service of a PetSet, since it's known at creation time you can simply pass it down through an [environment variable](/docs/user-guide/environment-guide).
|
||||
|
||||
Usually pets also need to find their peers. In the previous nginx example, we just used `kubectl` to get the names of existing pods, and as humans, we could tell which ones belonged to a given PetSet. Another way to find peers is by contacting the API server, just like `kubectl`, but that has several disadvantages (you end up implementing a Kubernetes specific init system that runs as pid 1 in your application container).
|
||||
|
||||
PetSet gives you a way to discover your peers using DNS records. To illustrate this we can use the previous example (note: one usually doesn't `apt-get` in a container).
|
||||
|
||||
```shell
|
||||
$ kubectl exec -it web-0 /bin/sh
|
||||
web-0 # apt-get update && apt-get install -y dnsutils
|
||||
...
|
||||
|
||||
web-0 # nslookup -type=srv nginx.default
|
||||
Server: 10.0.0.10
|
||||
Address: 10.0.0.10#53
|
||||
|
||||
nginx.default.svc.cluster.local service = 10 50 0 web-1.ub.default.svc.cluster.local.
|
||||
nginx.default.svc.cluster.local service = 10 50 0 web-0.ub.default.svc.cluster.local.
|
||||
```
|
||||
|
||||
## Updating a PetSet
|
||||
|
||||
You cannot update any field of the PetSet except `spec.replicas` and the `containers` in the podTemplate. Updating `spec.replicas` will scale the PetSet, updating `containers` will not have any effect till a Pet is deleted, at which time it is recreated with the modified podTemplate.
|
||||
|
||||
## Scaling a PetSet
|
||||
|
||||
You can scale a PetSet by updating the "replicas" field. Note however that the controller will only:
|
||||
|
||||
1. Create one pet at a time, in order from {0..N-1}, and wait till each one is in [Running and Ready](/docs/user-guide/pod-states) before creating the next
|
||||
2. Delete one pet at a time, in reverse order from {N-1..0}, and wait till each one is completely shutdown (past its [terminationGracePeriodSeconds](/docs/concepts/workloads/pods/pod/#termination-of-pods) before deleting the next
|
||||
|
||||
```shell
|
||||
$ kubectl get po
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
web-0 1/1 Running 0 30s
|
||||
web-1 1/1 Running 0 36s
|
||||
|
||||
$ kubectl patch petset web -p '{"spec":{"replicas":3}}'
|
||||
petset "web" patched
|
||||
|
||||
$ kubectl get po
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
web-0 1/1 Running 0 40s
|
||||
web-1 1/1 Running 0 46s
|
||||
web-2 1/1 Running 0 8s
|
||||
```
|
||||
|
||||
You can also use the `kubectl scale` command:
|
||||
|
||||
```shell
|
||||
$ kubectl get petset
|
||||
NAME DESIRED CURRENT AGE
|
||||
web 3 3 24m
|
||||
|
||||
$ kubectl scale petset web --replicas=5
|
||||
petset "web" scaled
|
||||
|
||||
$ kubectl get po --watch-only
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
web-0 1/1 Running 0 10m
|
||||
web-1 1/1 Running 0 27m
|
||||
web-2 1/1 Running 0 10m
|
||||
web-3 1/1 Running 0 3m
|
||||
web-4 0/1 ContainerCreating 0 48s
|
||||
|
||||
$ kubectl get petset web
|
||||
NAME DESIRED CURRENT AGE
|
||||
web 5 5 30m
|
||||
```
|
||||
|
||||
Note however, that scaling up to N and back down to M *will not* delete the volumes of the M-N pets, as described in the section on [deletion](#deleting-a-petset), i.e. scaling back up to M creates new pets that use the same volumes. To see this in action, scale the PetSet back down to 3:
|
||||
|
||||
```shell
|
||||
$ kubectl get po --watch-only
|
||||
web-4 1/1 Terminating 0 4m
|
||||
web-4 1/1 Terminating 0 4m
|
||||
web-3 1/1 Terminating 0 6m
|
||||
web-3 1/1 Terminating 0 6m
|
||||
```
|
||||
|
||||
Note that we still have 5 pvcs:
|
||||
|
||||
```shell
|
||||
$ kubectl get pvc
|
||||
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
|
||||
www-web-0 Bound pvc-42ca5cef-8113-11e6-82f6-42010af00002 1Gi RWO 32m
|
||||
www-web-1 Bound pvc-42de30af-8113-11e6-82f6-42010af00002 1Gi RWO 32m
|
||||
www-web-2 Bound pvc-ba416413-8115-11e6-82f6-42010af00002 1Gi RWO 14m
|
||||
www-web-3 Bound pvc-ba45f19c-8115-11e6-82f6-42010af00002 1Gi RWO 14m
|
||||
www-web-4 Bound pvc-ba47674a-8115-11e6-82f6-42010af00002 1Gi RWO 14m
|
||||
```
|
||||
|
||||
This allows you to upgrade the image of a petset and have it come back up with the same data, as described in the next section.
|
||||
|
||||
## Image upgrades
|
||||
|
||||
PetSet currently *does not* support automated image upgrade as noted in the section on [limitations](#alpha-limitations), however you can update the `image` field of any container in the podTemplate and delete Pets one by one, the PetSet controller will recreate it with the new image.
|
||||
|
||||
Edit the image on the PetSet to `gcr.io/google_containers/nginx-slim:0.7` and delete 1 Pet:
|
||||
|
||||
```shell{% raw %}
|
||||
$ for p in 0 1 2; do kubectl get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
|
||||
$ kubectl delete po web-0
|
||||
pod "web-0" deleted
|
||||
|
||||
$ for p in 0 1 2; do kubectl get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
{% endraw %}```
|
||||
|
||||
Delete the remaining 2:
|
||||
|
||||
```shell
|
||||
$ kubectl delete po web-1 web-2
|
||||
pod "web-1" deleted
|
||||
pod "web-2" deleted
|
||||
```
|
||||
|
||||
Wait till the PetSet is stable and check the images:
|
||||
|
||||
```shell{% raw %}
|
||||
$ for p in 0 1 2; do kubectl get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
{% endraw %}```
|
||||
|
||||
## Deleting a PetSet
|
||||
|
||||
Deleting a PetSet through kubectl will scale it down to 0, thereby deleting all the Pets. If you wish to delete just the PetSet and not the Pets, use `--cascade=false`:
|
||||
|
||||
```shell
|
||||
$ kubectl delete -f petset.yaml --cascade=false
|
||||
petset "web" deleted
|
||||
|
||||
$ kubectl get po -l app=nginx
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
web-0 1/1 Running 0 21h
|
||||
web-1 1/1 Running 0 21h
|
||||
|
||||
$ kubectl delete po -l app=nginx
|
||||
pod "web-0" deleted
|
||||
pod "web-1" deleted
|
||||
```
|
||||
|
||||
Deleting the pods will *not* delete the volumes. Until we finalize the recycle policy for these volumes they will have to get cleaned up by an admin. This is to ensure that you have the chance to copy data off the volume before deleting it. Simply deleting the PVC after the pods have left the [terminating state](/docs/concepts/workloads/pods/pod/#termination-of-pods) should trigger deletion of the backing Persistent Volumes.
|
||||
|
||||
**Note: you will lose all your data once the PVC is deleted, do this with caution.**
|
||||
|
||||
```shell
|
||||
$ kubectl get po -l app=nginx
|
||||
$ kubectl get pvc -l app=nginx
|
||||
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
|
||||
www-web-0 Bound pvc-62d271cd-3822-11e6-b1b7-42010af00002 0 21h
|
||||
www-web-1 Bound pvc-62d6750e-3822-11e6-b1b7-42010af00002 0 21h
|
||||
|
||||
$ kubectl delete pvc -l app=nginx
|
||||
$ kubectl get pv
|
||||
```
|
||||
|
||||
If you simply want to clean everything:
|
||||
|
||||
```shell{% raw %}
|
||||
$ grace=$(kubectl get po web-0 --template '{{.spec.terminationGracePeriodSeconds}}')
|
||||
$ kubectl delete petset,po -l app=nginx
|
||||
$ sleep $grace
|
||||
$ kubectl delete pvc -l app=nginx
|
||||
{% endraw %}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
You might have noticed an `annotations` field in all the PetSets shown above.
|
||||
|
||||
```yaml
|
||||
annotations:
|
||||
pod.alpha.kubernetes.io/initialized: "true"
|
||||
```
|
||||
|
||||
This field is a debugging hook. It pauses any scale up/down operations on the entire PetSet. If you'd like to pause a petset after each pet, set it to `false` in the template, wait for each pet to come up, verify it has initialized correctly, and then set it to `true` using `kubectl edit` on the pet (setting it to `false` on *any pet* is enough to pause the PetSet). If you don't need it, create the PetSet with it set to `true` as shown. This is surprisingly useful in debugging bootstrapping race conditions.
|
||||
|
||||
## Future Work
|
||||
|
||||
There are a LOT of planned improvements since PetSet is still in alpha.
|
||||
|
||||
* Data gravity and local storage
|
||||
* Richer notification events
|
||||
* Public network identities
|
||||
* WAN cluster deployments (multi-AZ/region/cloud provider)
|
||||
* Image and node upgrades
|
||||
|
||||
This list goes on, if you have examples, ideas or thoughts, please contribute.
|
||||
|
||||
## Alternatives
|
||||
|
||||
Deploying one RC of size 1/Service per pod is a popular alternative, as is simply deploying a DaemonSet that utilizes the identity of a Node.
|
||||
|
||||
## Next steps
|
||||
|
||||
* Learn about [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets/),
|
||||
the replacement for PetSet introduced in Kubernetes version 1.5.
|
||||
* [Migrate your existing PetSets to StatefulSets](/docs/tasks/manage-stateful-set/upgrade-pet-set-to-stateful-set/)
|
||||
when upgrading to Kubernetes version 1.5 or higher.
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
# A headless service to create DNS records
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: nginx
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
ports:
|
||||
- port: 80
|
||||
name: web
|
||||
# *.nginx.default.svc.cluster.local
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: nginx
|
||||
---
|
||||
apiVersion: apps/v1alpha1
|
||||
kind: PetSet
|
||||
metadata:
|
||||
name: web
|
||||
spec:
|
||||
serviceName: "nginx"
|
||||
replicas: 2
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx
|
||||
annotations:
|
||||
pod.alpha.kubernetes.io/initialized: "true"
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 0
|
||||
containers:
|
||||
- name: nginx
|
||||
image: gcr.io/google_containers/nginx-slim:0.8
|
||||
ports:
|
||||
- containerPort: 80
|
||||
name: web
|
||||
volumeMounts:
|
||||
- name: www
|
||||
mountPath: /usr/share/nginx/html
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: www
|
||||
annotations:
|
||||
volume.alpha.kubernetes.io/storage-class: anything
|
||||
spec:
|
||||
accessModes: [ "ReadWriteOnce" ]
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
|
||||
@@ -17,7 +17,6 @@ whereas a Replication Controller only supports equality-based selector requireme
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
{% capture body %}
|
||||
|
||||
## How to use a ReplicaSet
|
||||
@@ -52,20 +51,21 @@ use a Deployment instead, and define your application in the spec section.
|
||||
|
||||
{% include code.html language="yaml" file="frontend.yaml" ghlink="/docs/concepts/workloads/controllers/frontend.yaml" %}
|
||||
|
||||
Saving this config into `frontend.yaml` and submitting it to a Kubernetes cluster should
|
||||
Saving this manifest into `frontend.yaml` and submitting it to a Kubernetes cluster should
|
||||
create the defined ReplicaSet and the pods that it manages.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f frontend.yaml
|
||||
replicaset "frontend" created
|
||||
$ kubectl describe rs/frontend
|
||||
Name: frontend
|
||||
Namespace: default
|
||||
Selector: tier=frontend,tier in (frontend)
|
||||
Labels: app=guestbook,tier=frontend
|
||||
Annotations: <none>
|
||||
Replicas: 3 current / 3 desired
|
||||
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
|
||||
Name: frontend
|
||||
Namespace: default
|
||||
Selector: tier=frontend,tier in (frontend)
|
||||
Labels: app=guestbook
|
||||
tier=frontend
|
||||
Annotations: <none>
|
||||
Replicas: 3 current / 3 desired
|
||||
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
|
||||
Pod Template:
|
||||
Labels: app=guestbook
|
||||
tier=frontend
|
||||
@@ -93,7 +93,98 @@ frontend-dnjpy 1/1 Running 0 1m
|
||||
frontend-qhloh 1/1 Running 0 1m
|
||||
```
|
||||
|
||||
## ReplicaSet as an Horizontal Pod Autoscaler target
|
||||
## Writing a ReplicaSet Spec
|
||||
|
||||
As with all other Kubernetes API objects, a ReplicaSet needs the `apiVersion`, `kind`, and `metadata` fields. For
|
||||
general information about working with manifests, see [here](/docs/user-guide/simple-yaml/),
|
||||
[here](/docs/user-guide/configuring-containers/), and [here](/docs/concepts/tools/kubectl/object-management-overview/).
|
||||
|
||||
A ReplicaSet also needs a [`.spec` section](https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status).
|
||||
|
||||
### Pod Template
|
||||
|
||||
The `.spec.template` is the only required field of the `.spec`. The `.spec.template` is a
|
||||
[pod template](/docs/concepts/workloads/pods/pod-overview/#pod-templates). It has exactly the same schema as a
|
||||
[pod](/docs/concepts/workloads/pods/pod/), except that it is nested and does not have an `apiVersion` or `kind`.
|
||||
|
||||
In addition to required fields of a pod, a pod template in a ReplicaSet must specify appropriate
|
||||
labels and an appropriate restart policy.
|
||||
|
||||
For labels, make sure to not overlap with other controllers. For more information, see [pod selector](#pod-selector).
|
||||
|
||||
For [restart policy](/docs/concepts/workloads/pods/pod-lifecycle/), the only allowed value for `.spec.template.spec.restartPolicy` is `Always`, which is the default.
|
||||
|
||||
For local container restarts, ReplicaSet delegates to an agent on the node,
|
||||
for example the [Kubelet](/docs/admin/kubelet/) or Docker.
|
||||
|
||||
### Pod Selector
|
||||
|
||||
The `.spec.selector` field is a [label selector](/docs/user-guide/labels/#label-selectors). A ReplicaSet
|
||||
manages all the pods with labels that match the selector. It does not distinguish
|
||||
between pods that it created or deleted and pods that another person or process created or
|
||||
deleted. This allows the ReplicaSet to be replaced without affecting the running pods.
|
||||
|
||||
The `.spec.template.metadata.labels` must match the `.spec.selector`, or it will
|
||||
be rejected by the API.
|
||||
|
||||
In Kubernetes 1.8 the API version `apps/v1beta2` on the ReplicaSet kind is the current version and is enabled by default. The API version `extensions/v1beta1` is deprecated. In API version `apps/v1beta2`, `.spec.selector` and `.metadata.labels` no longer default to `.spec.template.metadata.labels` if not set. So they must be set explicitly. Also note that `.spec.selector` is immutable after creation starting in API version `apps/v1beta2`.
|
||||
|
||||
Also you should not normally create any pods whose labels match this selector, either directly, with
|
||||
another ReplicaSet, or with another controller such as a Deployment. If you do so, the ReplicaSet thinks that it
|
||||
created the other pods. Kubernetes does not stop you from doing this.
|
||||
|
||||
If you do end up with multiple controllers that have overlapping selectors, you
|
||||
will have to manage the deletion yourself.
|
||||
|
||||
### Labels on a ReplicaSet
|
||||
|
||||
The ReplicaSet can itself have labels (`.metadata.labels`). Typically, you
|
||||
would set these the same as the `.spec.template.metadata.labels`. However, they are allowed to be
|
||||
different, and the `.metadata.labels` do not affect the behavior of the ReplicaSet.
|
||||
|
||||
### Replicas
|
||||
|
||||
You can specify how many pods should run concurrently by setting `.spec.replicas`. The number running at any time may be higher
|
||||
or lower, such as if the replicas were just increased or decreased, or if a pod is gracefully
|
||||
shut down, and a replacement starts early.
|
||||
|
||||
If you do not specify `.spec.replicas`, then it defaults to 1.
|
||||
|
||||
## Working with ReplicaSets
|
||||
|
||||
### Deleting a ReplicaSet and its Pods
|
||||
|
||||
To delete a ReplicaSet and all its pods, use [`kubectl
|
||||
delete`](/docs/user-guide/kubectl/{{page.version}}/#delete). Kubectl will scale the ReplicaSet to zero and wait
|
||||
for it to delete each pod before deleting the ReplicaSet itself. If this kubectl command is interrupted, it can
|
||||
be restarted.
|
||||
|
||||
When using the REST API or go client library, you need to do the steps explicitly (scale replicas to
|
||||
0, wait for pod deletions, then delete the ReplicaSet).
|
||||
|
||||
### Deleting just a ReplicaSet
|
||||
|
||||
You can delete a ReplicaSet without affecting any of its pods, using [`kubectl delete`](/docs/user-guide/kubectl/{{page.version}}/#delete) with the `--cascade=false` option.
|
||||
|
||||
When using the REST API or go client library, simply delete the ReplicaSet object.
|
||||
|
||||
Once the original is deleted, you can create a new ReplicaSet to replace it. As long
|
||||
as the old and new `.spec.selector` are the same, then the new one will adopt the old pods.
|
||||
However, it will not make any effort to make existing pods match a new, different pod template.
|
||||
To update pods to a new spec in a controlled way, use a [rolling update](#rolling-updates).
|
||||
|
||||
### Isolating pods from a ReplicaSet
|
||||
|
||||
Pods may be removed from a ReplicaSet's target set by changing their labels. This technique may be used to remove pods
|
||||
from service for debugging, data recovery, etc. Pods that are removed in this way will be replaced automatically (
|
||||
assuming that the number of replicas is not also changed).
|
||||
|
||||
### Scaling a ReplicaSet
|
||||
|
||||
A ReplicaSet can be easily scaled up or down by simply updating the `.spec.replicas` field. The ReplicaSet controller
|
||||
ensures that that a desired number of pods with a matching label selector are available and operational.
|
||||
|
||||
### ReplicaSet as an Horizontal Pod Autoscaler Target
|
||||
|
||||
A ReplicaSet can also be a target for
|
||||
[Horizontal Pod Autoscalers (HPA)](/docs/tasks/run-application/horizontal-pod-autoscale/). That is,
|
||||
@@ -102,8 +193,7 @@ the ReplicaSet we created in the previous example.
|
||||
|
||||
{% include code.html language="yaml" file="hpa-rs.yaml" ghlink="/docs/concepts/workloads/controllers/hpa-rs.yaml" %}
|
||||
|
||||
|
||||
Saving this config into `hpa-rs.yaml` and submitting it to a Kubernetes cluster should
|
||||
Saving this manifest into `hpa-rs.yaml` and submitting it to a Kubernetes cluster should
|
||||
create the defined HPA that autoscales the target ReplicaSet depending on the CPU usage
|
||||
of the replicated pods.
|
||||
|
||||
@@ -118,6 +208,31 @@ Alternatively, you can use the `kubectl autoscale` command to accomplish the sam
|
||||
kubectl autoscale rs frontend
|
||||
```
|
||||
|
||||
## Alternatives to ReplicaSet
|
||||
|
||||
### Deployment (Recommended)
|
||||
|
||||
[`Deployment`](/docs/concepts/workloads/controllers/deployment/) is a higher-level API object that updates its underlying ReplicaSets and their Pods
|
||||
in a similar fashion as `kubectl rolling-update`. Deployments are recommended if you want this rolling update functionality,
|
||||
because unlike `kubectl rolling-update`, they are declarative, server-side, and have additional features. For more information on running a stateless
|
||||
application using a Deployment, please read [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/).
|
||||
|
||||
### Bare Pods
|
||||
|
||||
Unlike the case where a user directly created pods, a ReplicaSet replaces pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your application requires only a single pod. Think of it similarly to a process supervisor, only it supervises multiple pods across multiple nodes instead of individual processes on a single node. A ReplicaSet delegates local container restarts to some agent on the node (for example, Kubelet or Docker).
|
||||
|
||||
### Job
|
||||
|
||||
Use a [`Job`](/docs/concepts/jobs/run-to-completion-finite-workloads/) instead of a ReplicaSet for pods that are expected to terminate on their own
|
||||
(that is, batch jobs).
|
||||
|
||||
### DaemonSet
|
||||
|
||||
Use a [`DaemonSet`](/docs/concepts/workloads/controllers/daemonset/) instead of a ReplicaSet for pods that provide a
|
||||
machine-level function, such as machine monitoring or machine logging. These pods have a lifetime that is tied
|
||||
to a machine lifetime: the pod needs to be running on the machine before other pods start, and are
|
||||
safe to terminate when the machine is otherwise ready to be rebooted/shutdown.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/concept.md %}
|
||||
|
||||
@@ -10,10 +10,8 @@ title: StatefulSets
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
**StatefulSets are a beta feature in 1.7. This feature replaces the
|
||||
PetSets feature from 1.4. Users of PetSets are referred to the 1.5
|
||||
[Upgrade Guide](/docs/tasks/run-application/upgrade-pet-set-to-stateful-set/)
|
||||
for further information on how to upgrade existing PetSets to StatefulSets.**
|
||||
**StatefulSet is the workload API object used to manage stateful applications.
|
||||
StatefulSets are beta in 1.8.**
|
||||
|
||||
{% include templates/glossary/snippet.md term="statefulset" length="long" %}
|
||||
{% endcapture %}
|
||||
@@ -51,8 +49,7 @@ The example below demonstrates the components of a StatefulSet.
|
||||
|
||||
* A Headless Service, named nginx, is used to control the network domain.
|
||||
* The StatefulSet, named web, has a Spec that indicates that 3 replicas of the nginx container will be launched in unique Pods.
|
||||
* The volumeClaimTemplates will provide stable storage using [PersistentVolumes](/docs/concepts/storage/volumes/) provisioned by a
|
||||
PersistentVolume Provisioner.
|
||||
* The volumeClaimTemplates will provide stable storage using [PersistentVolumes](/docs/concepts/storage/volumes/) provisioned by a PersistentVolume Provisioner.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -69,17 +66,20 @@ spec:
|
||||
selector:
|
||||
app: nginx
|
||||
---
|
||||
apiVersion: apps/v1beta1
|
||||
apiVersion: apps/v1beta2
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: web
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nginx # has to match .spec.template.metadata.labels
|
||||
serviceName: "nginx"
|
||||
replicas: 3
|
||||
replicas: 3 # by default is 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx
|
||||
app: nginx # has to match .spec.selector.matchLabels
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 10
|
||||
containers:
|
||||
@@ -102,6 +102,9 @@ spec:
|
||||
storage: 1Gi
|
||||
```
|
||||
|
||||
## Pod Selector
|
||||
You must set the `spec.selector` field of a StatefulSet to match the labels of its `.spec.template.metadata.labels`. Prior to Kubernetes 1.8, the `spec.selector` field was defaulted when omitted. In 1.8 and later versions, failing to specify a matching Pod Selector will result in a validation error during StatefulSet creation.
|
||||
|
||||
## Pod Identity
|
||||
StatefulSet Pods have a unique identity that is comprised of an ordinal, a
|
||||
stable network identity, and stable storage. The identity sticks to the Pod,
|
||||
|
||||
@@ -14,7 +14,8 @@ scripts not present in an app image.
|
||||
|
||||
This feature has exited beta in 1.6. Init Containers can be specified in the PodSpec
|
||||
alongside the app `containers` array. The beta annotation value will still be respected
|
||||
and overrides the PodSpec field value.
|
||||
and overrides the PodSpec field value, however, they are deprecated in 1.6 and 1.7.
|
||||
In 1.8, the annotations are no longer supported and must be converted to the PodSpec field.
|
||||
|
||||
{% capture body %}
|
||||
## Understanding Init Containers
|
||||
@@ -123,7 +124,7 @@ spec:
|
||||
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
|
||||
```
|
||||
|
||||
There is a new syntax in Kubernetes 1.6, although the old annotation syntax still works. We have moved the declaration of init containers to `spec`:
|
||||
There is a new syntax in Kubernetes 1.6, although the old annotation syntax still works for 1.6 and 1.7. The new syntax must be used for 1.8 or greater. We have moved the declaration of init containers to `spec`:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -146,7 +147,7 @@ spec:
|
||||
command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']
|
||||
```
|
||||
|
||||
1.5 syntax still works on 1.6, but we recommend using 1.6 syntax. In Kubernetes 1.6, Init Containers were made a field in the API. The beta annotation is still respected but will be deprecated in future releases.
|
||||
1.5 syntax still works on 1.6, but we recommend using 1.6 syntax. In Kubernetes 1.6, Init Containers were made a field in the API. The beta annotation is still respected in 1.6 and 1.7, but is not supported in 1.8 or greater.
|
||||
|
||||
Yaml file below outlines the `mydb` and `myservice` services:
|
||||
|
||||
@@ -311,6 +312,10 @@ into alpha and beta annotations so that Kubelets version 1.3.0 or greater can ex
|
||||
Init Containers, and so that a version 1.6 apiserver can safely be rolled back to version
|
||||
1.5.x without losing Init Container functionality for existing created pods.
|
||||
|
||||
In Apiserver and Kubelet versions 1.8.0 or greater, support for the alpha and beta annotations
|
||||
is removed, requiring a conversion from the deprecated annotations to the
|
||||
`spec.initContainers` field.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
|
||||
@@ -150,7 +150,7 @@ Pod is exposed as a primitive in order to facilitate:
|
||||
* clean composition of Kubelet-level functionality with cluster-level functionality — Kubelet is effectively the "pod controller"
|
||||
* high-availability applications, which will expect pods to be replaced in advance of their termination and certainly in advance of deletion, such as in the case of planned evictions, image prefetching, or live pod migration [#3949](http://issue.k8s.io/3949)
|
||||
|
||||
There is new first-class support for stateful pods with the [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets/) controller (currently in beta). The feature was alpha in 1.4 and was called [PetSet](/docs/concepts/workloads/controllers/petset/). For prior versions of Kubernetes, best practice for having stateful pods is to create a replication controller with `replicas` equal to `1` and a corresponding service, see [this MySQL deployment example](/docs/tutorials/stateful-application/run-stateful-application/).
|
||||
There is new first-class support for stateful pods with the [StatefulSet](/docs/concepts/workloads/controllers/statefulset.md) controller (currently in beta). The feature was alpha in 1.4 and was called PetSet. For prior versions of Kubernetes, best practice for having stateful pods is to create a replication controller with `replicas` equal to `1` and a corresponding service, see [this MySQL deployment example](/docs/tutorials/stateful-application/run-stateful-application/).
|
||||
|
||||
## Termination of Pods
|
||||
|
||||
|
||||
Reference in New Issue
Block a user