Release docs for Kubernetes 1.11 (#9171)
* Seperate priority and preemption (#8144) * Doc about PID pressure condition. (#8211) * Doc about PID pressure condition. Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com> * "so" -> "too" * Update version selector for 1.11 * StorageObjectInUseProtection is GA (#8291) * Feature gate: StorageObjectInUseProtection is GA Update feature gate reference for 1.11 * Trivial commit to re-trigger Netlify * CRIContainerLogRotation is Beta in 1.11 (#8665) * Seperate priority and preemption (#8144) * CRIContainerLogRotation is Beta in 1.11 xref: kubernetes/kubernetes#64046 * Bring StorageObjectInUseProtection feature to GA (#8159) * StorageObjectInUseProtection is GA (#8291) * Feature gate: StorageObjectInUseProtection is GA Update feature gate reference for 1.11 * Trivial commit to re-trigger Netlify * Bring StorageObjectInUseProtection feature to GA StorageObjectInUseProtection is Beta in K8s 1.10. It's brought to GA in K8s 1.11. * Fixed typo and added feature state tags. * Remove KUBE_API_VERSIONS doc (#8292) The support to the KUBER_API_VERSIONS environment variable is completely dropped (no deprecation). This PR removes the related doc in release-1.11. xref: kubernetes/kubernetes#63165 * Remove InitialResources from admission controllers (#8293) The feature (was experimental) is dropped in 1.11. xref: kubernetes/kubernetes#58784 * Remove docs related to in-tree support to GPU (#8294) * Remove docs related to in-tree support to GPU The in-tree support to GPU is completely removed in release 1.11. This PR removes the related docs in release-1.11 branch. xref: kubernetes/kubernetes#61498 * Update content updated by PR to Hugo syntax Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Update the doc about extra volume in kubeadm config (#8453) Signed-off-by: Xianglin Gao <xianglin.gxl@alibaba-inc.com> * Update CRD Subresources for 1.11 (#8519) * coredns: update notes in administer-cluster/coredns.md (#8697) CoreDNS is installed by default in 1.11. Add notes on how to install kube-dns instead. Update notes about CoreDNS->CoreDNS upgrades as in 1.11 the Corefile is retained. Add example on upgrading from kube-dns to CoreDNS. * kubeadm-alpha: CoreDNS related changes (#8727) Update note about CoreDNS feature gate. This change also updates a tab as a kubeadm sub-command will change. It looks for a new generated file: generated/kubeadm_alpha_phase_addon_coredns.md instead of: generated/kubeadm_alpha_phase_addon_kube-dns.md * Update cloud controller manager docs to beta 1.11 (#8756) * Update cloud controller manager docs to beta 1.11 * Use Hugo shortcode for feature state * kubeadm-upgrade: include new command `kubeadm upgrade diff` (#8617) Also: - Include note that this was added in 1.11. - Modify the note about upgrade guidance. * independent: update CoreDNS mentions for kubeadm (#8753) Give CoreDNS instead of kube-dns examples in: - docs/setup/independent/create-cluster-kubeadm.md - docs/setup/independent/troubleshooting-kubeadm.md * update 1.11 --server-print info (#8870) * update 1.11 --server-print info * Copyedit * Mark ExpandPersistentVolumes feature to beta (#8778) * Update version selector for 1.11 * Mark ExpandPersistentVolumes Beta xref: kubernetes/kubernetes#64288 * fix shortcode, add placeholder files to fix deploy failures (#8874) * declare ipvs ga (#8850) * kubeadm: update info about CoreDNS in kubeadm-init.md (#8728) Add info to install kube-dns instead of CoreDNS, as CoreDNS is the default DNS server in 1.11. Add notes that kubeadm config images can be used to list and pull the required images in 1.11. * kubeadm: update implementation-details.md about CoreDNS (#8829) - Replace examples from kube-dns to CoreDNS - Add notes about the CoreDNS feature gate status in 1.11 - Add note that the service name for CoreDNS is also called `kube-dns` * Update block device support for 1.11 (#8895) * Update block device support for 1.11 * Copyedits * Fix typo 'fiber channel' (#8957) Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * kubeadm-upgrade: add the 'node [config]' sub-command (#8960) - Add includes for the generated pages - Include placeholder generated pages * kubeadm-init: update the example for the MasterConfiguration (#8958) - include godocs link for MasterConfiguration - include example MasterConfiguration - add note that `kubeadm config print-default` can be used * kubeadm-config: include new commands (#8862) Add notes and includes for these new commands in 1.11: - kubeadm config print-default - kubeadm config migrate - kubeadm config images list - kubeadm config images pull Include placeholder generated files for the above. * administer-cluster/coredns: include more changes (#8985) It was requested that for this page a couple of methods should be outlined: - manual installation for CoreDNS explained at the Kubernetes section of the GitHub project for CoreDNS - installation and upgrade via kubeadm Make the above changes and also add a section "About CoreDNS". This commit also lowercases a section title. * Update CRD subresources doc for 1.11 (#8918) * Add docs for volume expansion and online resizing (#8896) * Add docs for volume expansion going beta * Copyedit * Address feedback * Update exec plugin docs with TLS credentials (#8826) * Update exec plugin docs with TLS credentials kubernetes/kubernetes#61803 implements TLS client credential support for 1.11. * Copyedit * More copyedits for clarification * Additional copyedit * Change token->credential * NodeRestriction admission prevents kubelet taint removal (#8911) * dns-custom-namerserver: break down the page into mutliple sections (#8900) * dns-custom-namerserver: break down the page into mutliple sections This page is currently about kube-dns and is a bit outdated. Introduce the heading `# Customizing kube-dns`. Introduce a separate section about CoreDNS. * Copyedits, fix headings for customizing DNS Hey Lubomir, I coypedited pretty heavily because this workflow is so much easier for docs and because I'm trying to help improve everything touching kubeadm as much as possible. But there's one outstanding issue wrt headings and intro content: you can't add a heading 1 to a topic to do what you wanted to do. The page title in the front matter is rendered as a heading 1 and everything else has to start at heading 2. (We still need to doc this better in the docs contributing content, I know.) Instead, I think we need to rewrite the top-of-page intro content to explain better the relationship between kube-dns and CoreDNS. I'm happy to write something, but I thought I'd push this commit first so you can see what I'm doing. Hope it's all clear -- ping here or on Slack with any questions ~ Jennifer * Interim fix for talking about CoreDNS * Fix CoreDNS details * PSP readOnly hostPath (#8898) * Add documentation for crictl (#8880) * Add documentation for crictl * Copyedit Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Final copyedit * VolumeSubpathEnvExpansion alpha feature (#8835) * Note that Heapster is deprecated (#8827) * Note that Heapster is deprecated This notes that Heapster is deprecated, and migrates the relevant docs to talk about metrics-server or other solutions by default. * Copyedits and improvements Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Address feedback * fix shortcode to troubleshoot deploy (#9057) * update dynamic kubelet config docs for v1.11 (#8766) * update dynamic kubelet config docs for v1.11 * Substantial copyedit * Address feedback * Reference doc for kubeadm (release-1.11) (#9044) * Reference doc for kubeadm (release-1.11) * fix shortcode to troubleshoot deploy (#9057) * Reference doc for kube-components (release-1.11) (#9045) * Reference doc for kube-components (release-1.11) * Update cloud-controller-manager.md * fix shortcode to troubleshoot deploy (#9057) * Documentation on lowercasing kubeadm init apiserver SANs (#9059) * Documentation on lowercasing kubeadm init apiserver SANs * fix shortcode to troubleshoot deploy (#9057) * Clarification in dynamic Kubelet config doc (#9061) * Promote sysctls to Beta (#8804) * Promote sysctls to Beta * Copyedits Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Review comments * Address feedback * More feedback * kubectl reference docs for 1.11 (#9080) * Update Kubernetes API 1.11 ref docs (#8977) * Update v1alpha1 to v1beta1. * Adjust left nav for 1.11 ref docs. * Trim list of old ref docs. * Update Federation API ref docs for 1.11. (#9064) * Update Federation API ref docs for 1.11. * Add titles. * Update definitions.html * CRD versioning Public Documentation (#8834) * CRD versioning Public Documentation * Copyedit Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Address feedback * More rewrites * Address feedback * Update main CRD page in light of versioning * Reorg CRD docs * Further reorg * Tweak title * CSI documentation update for raw block volume support (#8927) * CSI documetation update for raw block volume support * minor edits for "CSI raw block volume support" Some small grammar and style nits. * minor CSIBlockVolume edits * Update kubectl component ref page for 1.11. (#9094) * Update kubectl component ref page for 1.11. * Add title. Replace stevepe with username. * crd versioning doc: fix nits (#9142) * Update `DynamicKubeletConfig` feature to beta (#9110) xref: kubernetes/kubernetes#64275 * Documentation for dynamic volume limits based on node type (#8871) * add cos for storage limits * Update docs specific for aws and gce * fix some minor things * Update storage-limits.md * Add k8s version to feature-state shortcode * The Doc update for ScheduleDaemonSetPods (#8842) Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com> * Update docs related to PersistentVolumeLabel admission control (#9109) The said admission controller is disabled by default in 1.11 (kubernetes/kubernetes#64326) and scheduled to be removed in future release. * client exec auth: updates for 1.11 (#9154) * Updates HA kubeadm docs (#9066) * Updates HA kubeadm docs Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * kubeadm HA - Add stacked control plane steps * ssh instructions and some typos in the bash scripts Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Fix typos and copypasta errors * Fix rebase issues * Integrate more changes Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * copyedits, layout and formatting fixes * final copyedits * Adds a sanity check for load balancer connection Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * formatting fixes, copyedits * fix typos, formatting * Document the Pod Ready++ feature (#9180) Closes: #9107 Xref: kubernetes/kubernetes#64057 * Mention 'KubeletPluginsWatcher' feature (#9177) * Mention 'KubeletPluginsWatcher' feature This feature is more developers oriented than users oriented, so simply mention it in the feature gate should be fine. In future, when the design doc is migrated from Google doc to the kubernetes/community repo, we can add links to it for users who want to dig deeper. Closes: #9108 Xref: kubernetes/kubernetes#63328, kubernetes/kubernetes#64605 * Copyedit * Amend dynamic volume list docs (#9181) The dynamic volume list feature has been documented but the feature gate related was not there yet. Closes: #9105 * Document for service account projection (#9182) This adds docs for the service account projection feature. Xref: kubernetes/kubernetes#63819, kubernetes/community#1973 Closes: #9102 * Update pod priority and preemption user docs (#9172) * Update pod priority and preemption user docs * Copyedit * Documentation on setting node name with Kubeadm (#8925) * Documentation on setting node name with Kubeadm * copyedit * Add kubeadm upgrade docs for 1.11 (#9089) * Add kubeadm upgrade docs for 1.11 * Initial docs review feedback * Add 1-11 to outline * Fix formatting on tab blocks * Move file to correct location * Add `kubeadm upgrade node config` step * Overzealous ediffing * copyedit, fix lists and headings * clarify --force flag for fixing bad state * Get TOML ready for 1.11 release * Blog post for 1.11 release (#9254) * Blog post for 1.11 release * Update 2018-06-26-kubernetes-1.11-release-announcement.md * Update 2018-06-26-kubernetes-1.11-release-announcement.md * Update 2018-06-26-kubernetes-1.11-release-announcement.md
This commit is contained in:
Misty Linville
committed by
k8s-ci-robot
k8s-ci-robot
parent
e091e29a5d
commit
5f30e7d4c3
@@ -5,6 +5,7 @@ reviewers:
|
||||
- cheftako
|
||||
- chenopis
|
||||
content_template: templates/task
|
||||
weight: 10
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
title: "Use Custom Resources"
|
||||
weight: 10
|
||||
---
|
||||
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
---
|
||||
title: Versions of CustomResourceDefinitions
|
||||
reviewers:
|
||||
- mbohlool
|
||||
- sttts
|
||||
- liggitt
|
||||
content_template: templates/task
|
||||
weight: 30
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page explains how to add versioning information to
|
||||
[CustomResourceDefinitions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions), to indicate the stability
|
||||
level of your CustomResourceDefinitions. It also describes how to upgrade an
|
||||
object from one version to another.
|
||||
|
||||
{{< note >}}
|
||||
**Note**: All specified versions must use the same schema. The is no schema
|
||||
conversion between versions.
|
||||
{{< /note >}}
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
||||
|
||||
* Make sure your Kubernetes cluster has a master version of 1.11.0 or higher.
|
||||
|
||||
* Read about [custom resources](/docs/concepts/api-extension/custom-resources/).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Overview
|
||||
|
||||
The CustomResourceDefinition API supports a `versions` field that you can use to
|
||||
support multiple versions of custom resources that you have developed, and
|
||||
indicate the stability of a given custom resource. All versions must currently
|
||||
use the same schema, so if you need to add a field, you must add it to all
|
||||
versions.
|
||||
|
||||
{{< note >}}
|
||||
Earlier iterations included a `version` field instead of `versions`. The
|
||||
`version` field is deprecated and optional, but if it is not empty, it must
|
||||
match the first item in the `versions` field.
|
||||
{{< /note >}}
|
||||
|
||||
## Specify multiple versions
|
||||
|
||||
This example shows a CustomResourceDefinition with two versions. The comments in
|
||||
the YAML provide more context.
|
||||
|
||||
```yaml
|
||||
apiVersion: apiextensions.k8s.io/v1beta1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
# name must match the spec fields below, and be in the form: <plural>.<group>
|
||||
name: crontabs.example.com
|
||||
spec:
|
||||
# group name to use for REST API: /apis/<group>/<version>
|
||||
group: example.com
|
||||
# list of versions supported by this CustomResourceDefinition
|
||||
versions:
|
||||
- name: v1beta1
|
||||
# Each version can be enabled/disabled by Served flag.
|
||||
served: true
|
||||
# One and only one version must be marked as the storage version.
|
||||
storage: true
|
||||
- name: v1
|
||||
served: true
|
||||
storage: false
|
||||
# either Namespaced or Cluster
|
||||
scope: Namespaced
|
||||
names:
|
||||
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
|
||||
plural: crontabs
|
||||
# singular name to be used as an alias on the CLI and for display
|
||||
singular: crontab
|
||||
# kind is normally the CamelCased singular type. Your resource manifests use this.
|
||||
kind: CronTab
|
||||
# shortNames allow shorter string to match your resource on the CLI
|
||||
shortNames:
|
||||
- ct
|
||||
```
|
||||
|
||||
You can save the CustomResourceDefinition in a YAML file, then use
|
||||
`kubectl create` to create it.
|
||||
|
||||
```shell
|
||||
kubectl create -f my-versioned-crontab.yaml
|
||||
```
|
||||
|
||||
After creation, the API server starts to serve each enabled version at an HTTP
|
||||
REST endpoint. In the above example, the API versions are available at
|
||||
`/apis/example.com/v1beta1` and `/apis/example.com/v1`.
|
||||
|
||||
### Version priority
|
||||
|
||||
Regardless of the order in which versions are defined in a
|
||||
CustomResourceDefinition, the version with the highest priority is used by
|
||||
kubectl as the default version to access objects. The priority is determined
|
||||
by parsing the _name_ field to determine the version number, the stability
|
||||
(GA, Beta, or Alpha), and the sequence within that stability level.
|
||||
|
||||
The algorithm used for sorting the versions is designed to sort versions in the
|
||||
same way that the Kubernetes project sorts Kubernetes versions. Versions start with a
|
||||
`v` followed by a number, an optional `beta` or `alpha` designation, and
|
||||
optional additional numeric versioning information. Broadly, a version string might look
|
||||
like `v2` or `v2beta1`. Versions are sorted using the following algorithm:
|
||||
|
||||
- Entries that follow Kubernetes version patterns are sorted before those that
|
||||
do not.
|
||||
- For entries that follow Kubernetes version patterns, the numeric portions of
|
||||
the version string is sorted largest to smallest.
|
||||
- If the strings `beta` or `alpha` follow the first numeric portion, they sorted
|
||||
in that order, after the equivalent string without the `beta` or `alpha`
|
||||
suffix (which is presumed to be the GA version).
|
||||
- If another number follows the `beta`, or `alpha`, those numbers are also
|
||||
sorted from largest to smallest.
|
||||
- Strings that don't fit the above format are sorted alphabetically and the
|
||||
numeric portions are not treated specially. Notice that in the example below,
|
||||
`foo1` is sorted above `foo10`. This is different from the sorting of the
|
||||
numeric portion of entries that do follow the Kubernetes version patterns.
|
||||
|
||||
This might make sense if you look at the following sorted version list:
|
||||
|
||||
```none
|
||||
- v10
|
||||
- v2
|
||||
- v1
|
||||
- v11beta2
|
||||
- v10beta3
|
||||
- v3beta1
|
||||
- v12alpha1
|
||||
- v11alpha2
|
||||
- foo1
|
||||
- foo10
|
||||
```
|
||||
|
||||
For the example in [Specify multiple versions](#specify-multiple-versions), the
|
||||
version sort order is `v1`, followed by `v1beta1`. This causes the kubectl
|
||||
command to use `v1` as the default version unless the provided object specifies
|
||||
the version.
|
||||
|
||||
## Writing, reading, and updating versioned CustomResourceDefinition objects
|
||||
|
||||
When an object is written, it is persisted at the version designated as the
|
||||
storage version at the time of the write. If the storage version changes,
|
||||
existing objects are never converted automatically. However, newly-created
|
||||
or updated objects are written at the new storage version. It is possible for an
|
||||
object to have been written at a version that is no longer served.
|
||||
|
||||
When you read an object, you specify the version as part of the path. If you
|
||||
specify a version that is different from the object's persisted version,
|
||||
Kubernetes returns the object to you at the version you requested, but the
|
||||
persisted object is neither changed on disk, nor converted in any way
|
||||
(other than changing the `apiVersion` string) while serving the request.
|
||||
You can request an object at any version that is currently served.
|
||||
|
||||
If you update an existing object, it is rewritten at the version that is
|
||||
currently the storage version. This is the only way that objects can change from
|
||||
one version to another.
|
||||
|
||||
To illustrate this, consider the following hypothetical series of events:
|
||||
|
||||
1. The storage version is `v1beta1`. You create an object. It is persisted in
|
||||
storage at version `v1beta1`
|
||||
2. You add version `v1` to your CustomResourceDefinition and designate it as
|
||||
the storage version.
|
||||
3. You read your object at version `v1beta1`, then you read the object again at
|
||||
version `v1`. Both returned objects are identical except for the apiVersion
|
||||
field.
|
||||
4. You create a new object. It is persisted in storage at version `v1`. You now
|
||||
have two objects, one of which is at `v1beta1`, and the other of which is at
|
||||
`v1`.
|
||||
5. You update the first object. It is now persisted at version `v1` since that
|
||||
is the current storage version.
|
||||
|
||||
### Previous storage versions
|
||||
|
||||
The API server records each version which has ever been marked as the storage
|
||||
version in the status field `storedVersions`. Objects may have been persisted
|
||||
at any version that has ever been designated as a storage version. No objects
|
||||
can exist in storage at a version that has never been a storage version.
|
||||
|
||||
## Upgrade existing objects to a new stored version
|
||||
|
||||
When deprecating versions and dropping support, devise a storage upgrade
|
||||
procedure. The following is an example procedure to upgrade from `v1beta1`
|
||||
to `v1`.
|
||||
|
||||
1. Set `v1` as the storage in the CustomResourceDefinition file and apply it
|
||||
using kubectl. The `storedVersions` is now `v1beta1, v1`.
|
||||
2. Write an upgrade procedure to list all existing objects and write them with
|
||||
the same content. This forces the backend to write objects in the current
|
||||
storage version, which is `v1`.
|
||||
3. Update the CustomResourceDefinition `Status` by removing `v1beta1` from
|
||||
`storedVersions` field.
|
||||
|
||||
{{% /capture %}}
|
||||
+41
-19
@@ -4,11 +4,12 @@ reviewers:
|
||||
- deads2k
|
||||
- enisoc
|
||||
content_template: templates/task
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page shows how to install a
|
||||
[custom resource](/docs/concepts/api-extension/custom-resources/)
|
||||
[custom resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
|
||||
into the Kubernetes API by creating a
|
||||
[CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
|
||||
{{% /capture %}}
|
||||
@@ -27,10 +28,11 @@ into the Kubernetes API by creating a
|
||||
## Create a CustomResourceDefinition
|
||||
|
||||
When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server
|
||||
reacts by creating a new RESTful resource path, either namespaced or cluster-scoped,
|
||||
as specified in the CRD's `scope` field. As with existing built-in objects, deleting a
|
||||
namespace deletes all custom objects in that namespace.
|
||||
CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.
|
||||
creates a new RESTful resource path for each version you specify. The CRD can be
|
||||
either namespaced or cluster-scoped, as specified in the CRD's `scope` field. As
|
||||
with existing built-in objects, deleting a namespace deletes all custom objects
|
||||
in that namespace. CustomResourceDefinitions themselves are non-namespaced and
|
||||
are available to all namespaces.
|
||||
|
||||
For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`:
|
||||
|
||||
@@ -43,8 +45,13 @@ metadata:
|
||||
spec:
|
||||
# group name to use for REST API: /apis/<group>/<version>
|
||||
group: stable.example.com
|
||||
# version name to use for REST API: /apis/<group>/<version>
|
||||
version: v1
|
||||
# list of versions supported by this CustomResourceDefinition
|
||||
versions:
|
||||
- name: v1
|
||||
# Each version can be enabled/disabled by Served flag.
|
||||
served: true
|
||||
# One and only one version must be marked as the storage version.
|
||||
storage: true
|
||||
# either Namespaced or Cluster
|
||||
scope: Namespaced
|
||||
names:
|
||||
@@ -75,7 +82,7 @@ This endpoint URL can then be used to create and manage custom objects.
|
||||
The `kind` of these objects will be `CronTab` from the spec of the
|
||||
CustomResourceDefinition object you created above.
|
||||
|
||||
Please note that it might take a few seconds for the endpoint to be created.
|
||||
It might take a few seconds for the endpoint to be created.
|
||||
You can watch the `Established` condition of your CustomResourceDefinition
|
||||
to be true or watch the discovery information of the API server for your
|
||||
resource to show up.
|
||||
@@ -120,9 +127,8 @@ NAME AGE
|
||||
my-new-cron-object 6s
|
||||
```
|
||||
|
||||
Note that resource names are not case-sensitive when using kubectl,
|
||||
and you can use either the singular or plural forms defined in the CRD,
|
||||
as well as any short names.
|
||||
Resource names are not case-sensitive when using kubectl, and you can use either
|
||||
the singular or plural forms defined in the CRD, as well as any short names.
|
||||
|
||||
You can also view the raw YAML data:
|
||||
|
||||
@@ -173,6 +179,12 @@ Error from server (NotFound): Unable to list "crontabs": the server could not fi
|
||||
|
||||
If you later recreate the same CustomResourceDefinition, it will start out empty.
|
||||
|
||||
## Serving multiple versions of a CRD
|
||||
|
||||
See [Custom resource definition versioning](custom-resource-definition-versioning)
|
||||
for more information about serving multiple versions of your
|
||||
CustomResourceDefinition and migrating your objects from one version to another.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture discussion %}}
|
||||
@@ -245,6 +257,10 @@ metadata:
|
||||
name: crontabs.stable.example.com
|
||||
spec:
|
||||
group: stable.example.com
|
||||
versions:
|
||||
- name: v1
|
||||
served: true
|
||||
storage: true
|
||||
version: v1
|
||||
scope: Namespaced
|
||||
names:
|
||||
@@ -333,18 +349,15 @@ crontab "my-new-cron-object" created
|
||||
### Subresources
|
||||
|
||||
Custom resources support `/status` and `/scale` subresources.
|
||||
This feature is __alpha__ in v1.10 and may change in backward incompatible ways.
|
||||
This feature is __beta__ in v1.11 and enabled by default.
|
||||
|
||||
Enable this feature using the `CustomResourceSubresources` feature gate on
|
||||
You can disable this feature using the `CustomResourceSubresources` feature gate on
|
||||
the [kube-apiserver](/docs/admin/kube-apiserver):
|
||||
|
||||
```
|
||||
--feature-gates=CustomResourceSubresources=true
|
||||
--feature-gates=CustomResourceSubresources=false
|
||||
```
|
||||
|
||||
When the `CustomResourceSubresources` feature gate is enabled, only the `properties` construct
|
||||
is allowed in the root schema for custom resource validation.
|
||||
|
||||
The status and scale subresources can be optionally enabled by
|
||||
defining them in the CustomResourceDefinition.
|
||||
|
||||
@@ -357,6 +370,7 @@ When the status subresource is enabled, the `/status` subresource for the custom
|
||||
- `PUT` requests to the `/status` subresource only validate the status stanza of the custom resource.
|
||||
- `PUT`/`POST`/`PATCH` requests to the custom resource ignore changes to the status stanza.
|
||||
- Any changes to the spec stanza increments the value at `.metadata.generation`.
|
||||
- `properties`, `required` and `description` are the only constructs allowed in the root of the CRD OpenAPI validation schema.
|
||||
|
||||
#### Scale subresource
|
||||
|
||||
@@ -398,7 +412,10 @@ metadata:
|
||||
name: crontabs.stable.example.com
|
||||
spec:
|
||||
group: stable.example.com
|
||||
version: v1
|
||||
versions:
|
||||
- name: v1
|
||||
served: true
|
||||
storage: true
|
||||
scope: Namespaced
|
||||
names:
|
||||
plural: crontabs
|
||||
@@ -489,7 +506,10 @@ metadata:
|
||||
name: crontabs.stable.example.com
|
||||
spec:
|
||||
group: stable.example.com
|
||||
version: v1
|
||||
versions:
|
||||
- name: v1
|
||||
served: true
|
||||
storage: true
|
||||
scope: Namespaced
|
||||
names:
|
||||
plural: crontabs
|
||||
@@ -546,6 +566,8 @@ crontabs/my-new-cron-object 3s
|
||||
{{% capture whatsnext %}}
|
||||
* Learn how to [Migrate a ThirdPartyResource to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/).
|
||||
* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions-k8s-io).
|
||||
* Serve [multiple versions](custom-resource-definitions-versioning) of a
|
||||
CustomResourceDefinition
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
+6
-2
@@ -4,6 +4,7 @@ reviewers:
|
||||
- enisoc
|
||||
- deads2k
|
||||
content_template: templates/task
|
||||
weight: 50
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -71,7 +72,10 @@ you **on a best-effort basis**.
|
||||
spec:
|
||||
scope: Namespaced
|
||||
group: stable.example.com
|
||||
version: v1
|
||||
versions:
|
||||
- name: v1
|
||||
served: true
|
||||
storage: true
|
||||
names:
|
||||
kind: CronTab
|
||||
plural: crontabs
|
||||
@@ -163,7 +167,7 @@ you **on a best-effort basis**.
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
* Learn more about [custom resources](/docs/concepts/api-extension/custom-resources/).
|
||||
* Learn more about [using CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
|
||||
* Learn more about [using CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/).
|
||||
* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
title: Use an HTTP Proxy to Access the Kubernetes API
|
||||
content_template: templates/task
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -5,6 +5,7 @@ reviewers:
|
||||
- cheftako
|
||||
- chenopis
|
||||
content_template: templates/task
|
||||
weight: 15
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -206,9 +206,6 @@ The objects that are stored to disk for a cluster's internal representation of t
|
||||
When the supported API changes, these objects may need to be rewritten in the newer API. Failure to do this will eventually result in resources that are no longer decodable or usable
|
||||
by the Kubernetes API server.
|
||||
|
||||
`KUBE_API_VERSIONS` environment variable for the `kube-apiserver` binary which controls the API versions that are supported in the cluster. The first version in the list is used as the cluster's storage version. Hence, to set a specific version as the storage version, bring it to the front of list of versions in the value of `KUBE_API_VERSIONS`. You need to restart the `kube-apiserver` binary
|
||||
for changes to this variable to take effect.
|
||||
|
||||
### Switching your config files to a new API version
|
||||
|
||||
You can use `kubectl convert` command to convert config files between different API versions.
|
||||
@@ -217,6 +214,4 @@ You can use `kubectl convert` command to convert config files between different
|
||||
kubectl convert -f pod.yaml --output-version v1
|
||||
```
|
||||
|
||||
For more options, please refer to the usage of [kubectl convert](/docs/reference/generated/kubectl/kubectl-commands/#convert) command.
|
||||
|
||||
{{% /capture %}}
|
||||
For more options, please refer to the usage of [kubectl convert](/docs/user-guide/kubectl/{{page.version}}/#convert) command.
|
||||
|
||||
@@ -6,11 +6,8 @@ min-kubernetes-server-version: v1.9
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{< feature-state state="beta" >}}
|
||||
|
||||
{{% capture overview %}}
|
||||
This page describes how to enable CoreDNS instead of kube-dns for service
|
||||
discovery.
|
||||
This page describes the CoreDNS upgrade process and how to install kube-dns instead of CoreDNS.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
@@ -19,41 +16,47 @@ discovery.
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## About CoreDNS
|
||||
|
||||
[CoreDNS](https://coredns.io) is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS.
|
||||
Like Kubernetes, the CoreDNS project is hosted by the [CNCF.](http://www.cncf.io)
|
||||
|
||||
You can use CoreDNS instead of kube-dns in your cluster by replacing kube-dns in an existing
|
||||
deployment, or by using tools like kubeadm that will deploy and upgrade the cluster for you.
|
||||
|
||||
## Installing CoreDNS
|
||||
|
||||
For manual deployment or replacement of kube-dns, see the documentation at the
|
||||
[CoreDNS GitHub project.](https://github.com/coredns/deployment/tree/master/kubernetes)
|
||||
|
||||
## Installing CoreDNS with kubeadm
|
||||
|
||||
In Kubernetes 1.9, [CoreDNS](https://coredns.io) is available as an alpha feature, and
|
||||
in Kubernetes 1.10 it is available as a beta feature. In either case, you may install
|
||||
it during cluster creation by setting the `CoreDNS` feature gate to `true` during `kubeadm init`:
|
||||
|
||||
In Kubernetes 1.11, CoreDNS has graduated to General Availability (GA)
|
||||
and is installed by default. To install kube-dns instead, set the `CoreDNS` feature gate
|
||||
value to `false`:
|
||||
```
|
||||
kubeadm init --feature-gates=CoreDNS=true
|
||||
kubeadm init --feature-gates=CoreDNS=false
|
||||
```
|
||||
|
||||
This installs CoreDNS instead of kube-dns.
|
||||
## Upgrading an existing cluster with kubeadm
|
||||
|
||||
## Using a custom CoreDNS image repository with kubeadm
|
||||
|
||||
To use a custom image repository for the CoreDNS image, e.g. one located in your own Docker registry,
|
||||
you can execute the following command after kubeadm has deployed the CoreDNS manifest:
|
||||
|
||||
```shell
|
||||
kubectl set image -n kube-system deploy/coredns coredns=prefix.example.com/coredns/coredns:1.0.6
|
||||
```
|
||||
|
||||
## Upgrading an Existing Cluster with kubeadm
|
||||
|
||||
In Kubernetes 1.10, you can also move to CoreDNS when you use `kubeadm` to upgrade
|
||||
In Kubernetes version 1.10 and later, you can also move to CoreDNS when you use `kubeadm` to upgrade
|
||||
a cluster that is using `kube-dns`. In this case, `kubeadm` will generate the CoreDNS configuration
|
||||
("Corefile") based upon the `kube-dns` ConfigMap, preserving configurations for federation,
|
||||
stub domains, and upstream name server.
|
||||
|
||||
Note that if you are running CoreDNS in your cluster already, prior to upgrade, your existing Corefile will be
|
||||
**overwritten** by the one created during upgrade. **You should save your existing ConfigMap
|
||||
if you have customized it.** You may re-apply your customizations after the new ConfigMap is
|
||||
up and running.
|
||||
If you are moving from kube-dns to CoreDNS, make sure to set the `CoreDNS` feature gate to `true`
|
||||
during an upgrade. For example, here is what a `v1.11.0` upgrade would look like:
|
||||
```
|
||||
kubeadm upgrade apply v1.11.0 --feature-gates=CoreDNS=true
|
||||
```
|
||||
|
||||
This process will be modified for the GA release of this feature, such that an existing
|
||||
Corefile will not be overwritten.
|
||||
In versions prior to 1.11 the Corefile will be **overwritten** by the one created during upgrade.
|
||||
**You should save your existing ConfigMap if you have customized it.** You may re-apply your
|
||||
customizations after the new ConfigMap is up and running.
|
||||
|
||||
If you are running CoreDNS in Kubernetes version 1.11 and later, during upgrade,
|
||||
your existing Corefile will be retained.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -9,6 +9,11 @@ content_template: templates/concept
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.11" state="beta" >}}
|
||||
In upcoming releases, Cloud Controller Manager will
|
||||
be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers
|
||||
can develop their features independently from the core Kubernetes release cycles.**
|
||||
|
||||
{{< feature-state for_k8s_version="1.8" state="alpha" >}}
|
||||
|
||||
Before going into how to build your own cloud controller manager, some background on how it works under the hood is helpful. The cloud controller manager is code from `kube-controller-manager` utilizing Go interfaces to allow implementations from any cloud to be plugged in. Most of the scaffolding and generic controller implementations will be in core, but it will always exec out to the cloud interfaces it is provided, so long as the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L29-L50) is satisfied.
|
||||
|
||||
@@ -7,56 +7,59 @@ content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page provides hints on configuring DNS Pod and guidance on customizing the
|
||||
DNS resolution process.
|
||||
This page explains how to configure your DNS Pod and customize the
|
||||
DNS resolution process. In Kubernetes version 1.11 and later, CoreDNS is at GA
|
||||
and is installed by default with kubeadm. See [Configuring CoreDNS](#config-coredns)
|
||||
and [Using CoreDNS for Service Discovery](/docs/tasks/administer-cluster/coredns/).
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
||||
* Kubernetes version 1.6 and above.
|
||||
* The cluster must be configured to use the `kube-dns` addon.
|
||||
* Kubernetes version 1.6 or later. To work with CoreDNS, version 1.9 or later.
|
||||
* The appropriate add-on: kube-dns or CoreDNS. To install with kubeadm,
|
||||
see [the kubeadm reference documentation](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-addon).
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Introduction
|
||||
|
||||
Starting from Kubernetes v1.3, DNS is a built-in service launched automatically
|
||||
DNS is a built-in Kubernetes service launched automatically
|
||||
using the addon manager
|
||||
[cluster add-on](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/README.md).
|
||||
|
||||
The running Kubernetes DNS pod holds 3 containers:
|
||||
The running DNS Pod holds 3 containers:
|
||||
|
||||
- "`kubedns`": The `kubedns` process watches the Kubernetes master for changes
|
||||
- "`kubedns`": watches the Kubernetes master for changes
|
||||
in Services and Endpoints, and maintains in-memory lookup structures to serve
|
||||
DNS requests.
|
||||
- "`dnsmasq`": The `dnsmasq` container adds DNS caching to improve performance.
|
||||
- "`sidecar`": The `sidecar` container provides a single health check endpoint
|
||||
while performing dual healthchecks (for `dnsmasq` and `kubedns`).
|
||||
- "`dnsmasq`": adds DNS caching to improve performance.
|
||||
- "`sidecar`": provides a single health check endpoint
|
||||
to perform healthchecks for `dnsmasq` and `kubedns`.
|
||||
|
||||
The DNS pod is exposed as a Kubernetes Service with a static IP. Once assigned
|
||||
the kubelet passes DNS configured using the `--cluster-dns=<dns-service-ip>`
|
||||
flag to each container.
|
||||
The DNS Pod is exposed as a Kubernetes Service with a static IP.
|
||||
The kubelet passes DNS to each container with the `--cluster-dns=<dns-service-ip>`
|
||||
flag.
|
||||
|
||||
DNS names also need domains. The local domain is configurable in the kubelet
|
||||
using the flag `--cluster-domain=<default-local-domain>`.
|
||||
DNS names also need domains. You configure the local domain in the kubelet
|
||||
with the flag `--cluster-domain=<default-local-domain>`.
|
||||
|
||||
The Kubernetes cluster DNS server is based off the
|
||||
The Kubernetes cluster DNS server is based on the
|
||||
[SkyDNS](https://github.com/skynetservices/skydns) library. It supports forward
|
||||
lookups (A records), service lookups (SRV records) and reverse IP address
|
||||
lookups (A records), service lookups (SRV records), and reverse IP address
|
||||
lookups (PTR records).
|
||||
|
||||
## Inheriting DNS from the node
|
||||
|
||||
When running a pod, kubelet will prepend the cluster DNS server and search
|
||||
paths to the node's own DNS settings. If the node is able to resolve DNS names
|
||||
specific to the larger environment, pods should be able to, also.
|
||||
See [Known issues](#known-issues) below for a caveat.
|
||||
When running a Pod, kubelet prepends the cluster DNS server and searches
|
||||
paths to the node's DNS settings. If the node is able to resolve DNS names
|
||||
specific to the larger environment, Pods should also be able to resolve.
|
||||
But see [Known issues](#known-issues).
|
||||
|
||||
If you don't want this, or if you want a different DNS config for pods, you can
|
||||
use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will
|
||||
not inherit DNS. Setting it to a valid file path means that kubelet will use
|
||||
this file instead of `/etc/resolv.conf` for DNS inheritance.
|
||||
use the kubelet's `--resolv-conf` flag. Set this flag to "" to prevent Pods from
|
||||
inheriting DNS. Set it to a valid file path to specify a file other than
|
||||
`/etc/resolv.conf` for DNS inheritance.
|
||||
|
||||
## Configure stub-domain and upstream DNS servers
|
||||
|
||||
@@ -64,7 +67,7 @@ Cluster administrators can specify custom stub domains and upstream nameservers
|
||||
by providing a ConfigMap for kube-dns (`kube-system:kube-dns`).
|
||||
|
||||
For example, the following ConfigMap sets up a DNS configuration with a single stub domain and two
|
||||
upstream nameservers.
|
||||
upstream nameservers:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -79,11 +82,11 @@ data:
|
||||
["8.8.8.8", "8.8.4.4"]
|
||||
```
|
||||
|
||||
As specified, DNS requests with the “.acme.local” suffix
|
||||
DNS requests with the “.acme.local” suffix
|
||||
are forwarded to a DNS listening at 1.2.3.4. Google Public DNS
|
||||
serves the upstream queries.
|
||||
|
||||
The table below describes how queries with certain domain names would map to
|
||||
The table below describes how queries with certain domain names map to
|
||||
their destination DNS servers:
|
||||
|
||||
| Domain name | Server answering the query |
|
||||
@@ -99,9 +102,9 @@ details about the configuration option format.
|
||||
|
||||
{{% capture discussion %}}
|
||||
|
||||
### Impacts on Pods
|
||||
### Effects on Pods
|
||||
|
||||
Custom upstream nameservers and stub domains won't impact Pods that have their
|
||||
Custom upstream nameservers and stub domains do not affect Pods with a
|
||||
`dnsPolicy` set to "`Default`" or "`None`".
|
||||
|
||||
If a Pod's `dnsPolicy` is set to "`ClusterFirst`", its name resolution is
|
||||
@@ -113,23 +116,23 @@ cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream
|
||||
nameserver inherited from the node.
|
||||
|
||||
**With custom configurations**: If stub domains and upstream DNS servers are
|
||||
configured (as in the [previous example](#configuring-stub-domain-and-upstream-dns-servers)),
|
||||
DNS queries will be routed according to the following flow:
|
||||
configured,
|
||||
DNS queries are routed according to the following flow:
|
||||
|
||||
1. The query is first sent to the DNS caching layer in kube-dns.
|
||||
|
||||
1. From the caching layer, the suffix of the request is examined and then
|
||||
forwarded to the appropriate DNS, based on the following cases:
|
||||
|
||||
* *Names with the cluster suffix* (e.g.".cluster.local"):
|
||||
* *Names with the cluster suffix*, for example ".cluster.local":
|
||||
The request is sent to kube-dns.
|
||||
|
||||
* *Names with the stub domain suffix* (e.g. ".acme.local"):
|
||||
The request is sent to the configured custom DNS resolver (e.g. listening at 1.2.3.4).
|
||||
* *Names with the stub domain suffix*, for example ".acme.local":
|
||||
The request is sent to the configured custom DNS resolver, listening for example at 1.2.3.4.
|
||||
|
||||
* *Names without a matching suffix* (e.g."widget.com"):
|
||||
The request is forwarded to the upstream DNS
|
||||
(e.g. Google public DNS servers at 8.8.8.8 and 8.8.4.4).
|
||||
* *Names without a matching suffix*, for example "widget.com":
|
||||
The request is forwarded to the upstream DNS,
|
||||
for example Google public DNS servers at 8.8.8.8 and 8.8.4.4.
|
||||
|
||||
|
||||
|
||||
@@ -139,17 +142,17 @@ Options for the kube-dns `kube-system:kube-dns` ConfigMap:
|
||||
|
||||
| Field | Format | Description |
|
||||
| ----- | ------ | ----------- |
|
||||
| `stubDomains` (optional) | A JSON map using a DNS suffix key (e.g. “acme.local”) and a value consisting of a JSON array of DNS IPs. | The target nameserver may itself be a Kubernetes service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
|
||||
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | Note: If specified, then the values specified replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
|
||||
| `stubDomains` (optional) | A JSON map using a DNS suffix key such as “acme.local”, and a value consisting of a JSON array of DNS IPs. | The target nameserver can itself be a Kubernetes Service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
|
||||
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | If specified, the values replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
|
||||
|
||||
### Examples
|
||||
|
||||
#### Example: Stub domain
|
||||
|
||||
In this example, the user has a Consul DNS service discovery system that they wish to
|
||||
In this example, the user has a Consul DNS service discovery system they want to
|
||||
integrate with kube-dns. The consul domain server is located at 10.150.0.1, and
|
||||
all consul names have the suffix “.consul.local”. To configure Kubernetes, the
|
||||
cluster administrator simply creates a ConfigMap object as shown below.
|
||||
all consul names have the suffix `.consul.local`. To configure Kubernetes, the
|
||||
cluster administrator creates the following ConfigMap:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -162,7 +165,7 @@ data:
|
||||
{"consul.local": ["10.150.0.1"]}
|
||||
```
|
||||
|
||||
Note that the cluster administrator did not wish to override the node’s
|
||||
Note that the cluster administrator does not want to override the node’s
|
||||
upstream nameservers, so they did not specify the optional
|
||||
`upstreamNameservers` field.
|
||||
|
||||
@@ -170,8 +173,8 @@ upstream nameservers, so they did not specify the optional
|
||||
|
||||
In this example the cluster administrator wants to explicitly force all
|
||||
non-cluster DNS lookups to go through their own nameserver at 172.16.0.1.
|
||||
Again, this is easy to accomplish; they just need to create a ConfigMap with the
|
||||
`upstreamNameservers` field specifying the desired nameserver.
|
||||
In this case, they create a ConfigMap with the
|
||||
`upstreamNameservers` field specifying the desired nameserver:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -184,9 +187,97 @@ data:
|
||||
["172.16.0.1"]
|
||||
```
|
||||
|
||||
## What's next
|
||||
- [Debugging DNS Resolution](/docs/tasks/administer-cluster/dns-debugging-resolution/).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
## Configuring CoreDNS {config-coredns}
|
||||
|
||||
You can configure [CoreDNS](https://coredns.io/) as a service discovery.
|
||||
|
||||
CoreDNS is available as an option in Kubernetes starting with version 1.9.
|
||||
It is currently a [GA feature](https://github.com/kubernetes/community/blob/master/keps/sig-network/0010-20180314-coredns-GA-proposal.md) and is on course to be [the default](https://github.com/kubernetes/community/blob/master/keps/sig-network/0012-20180518-coredns-default-proposal.md), replacing kube-dns.
|
||||
|
||||
|
||||
## CoreDNS ConfigMap options
|
||||
|
||||
CoreDNS chains plugins and can be configured by maintaining a Corefile with the ConfigMap. CoreDNS supports all the functionalities and more that is provided by kube-dns.
|
||||
A ConfigMap created for kube-dns to support `StubDomains`and `upstreamnameserver` translates to the `proxy` plugin in CoreDNS.
|
||||
Similarly, the `Federation` plugin translates to the `federation` plugin in CoreDNS.
|
||||
|
||||
### Example
|
||||
|
||||
This example ConfigMap for kube0dns specifies federations, stubdomains and upstreamnameservers:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
data:
|
||||
federations: |
|
||||
{"foo" : "foo.feddomain.com"}
|
||||
stubDomains: |
|
||||
{"abc.com" : ["1.2.3.4"], "my.cluster.local" : ["2.3.4.5"]}
|
||||
upstreamNameservers: |
|
||||
["8.8.8.8", "8.8.4.4"]
|
||||
kind: ConfigMap
|
||||
```
|
||||
|
||||
The equivalent configuration in CoreDNS creates a Corefile:
|
||||
|
||||
* For federations:
|
||||
```yaml
|
||||
federation cluster.local {
|
||||
foo foo.feddomain.com
|
||||
}
|
||||
```
|
||||
|
||||
* For stubDomains:
|
||||
```yaml
|
||||
abc.com:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 1.2.3.4
|
||||
}
|
||||
my.cluster.local:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 2.3.4.5
|
||||
}
|
||||
```
|
||||
|
||||
The complete Corefile with the default plugins:
|
||||
|
||||
```yaml
|
||||
.:53 {
|
||||
errors
|
||||
health
|
||||
kubernetes cluster.local in-addr.arpa ip6.arpa {
|
||||
upstream 8.8.8.8 8.8.4.4
|
||||
pods insecure
|
||||
fallthrough in-addr.arpa ip6.arpa
|
||||
}
|
||||
federation cluster.local {
|
||||
foo foo.feddomain.com
|
||||
}
|
||||
prometheus :9153
|
||||
proxy . 8.8.8.8 8.8.4.4
|
||||
cache 30
|
||||
}
|
||||
abc.com:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 1.2.3.4
|
||||
}
|
||||
my.cluster.local:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 2.3.4.5
|
||||
}
|
||||
```
|
||||
|
||||
In Kubernetes version 1.10 and later, kubeadm supports automatic translation of the CoreDNS ConfigMap from the kube-dns ConfigMap.
|
||||
|
||||
## Migration to CoreDNS
|
||||
|
||||
A number of tools support the installation of CoreDNS instead of kube-dns.
|
||||
To migrate from kube-dns to CoreDNS, [a detailed blog](https://coredns.io/2018/05/21/migration-from-kube-dns-to-coredns/) is available to help users adapt CoreDNS in place of kube-dns.
|
||||
|
||||
## What's next
|
||||
- [Debugging DNS Resolution](/docs/tasks/administer-cluster/dns-debugging-resolution/).
|
||||
|
||||
@@ -83,7 +83,6 @@ The output shows that the Node has a capacity of 4 dongles:
|
||||
|
||||
```
|
||||
"capacity": {
|
||||
"alpha.kubernetes.io/nvidia-gpu": "0",
|
||||
"cpu": "2",
|
||||
"memory": "2049008Ki",
|
||||
"example.com/dongle": "4",
|
||||
@@ -99,7 +98,6 @@ Once again, the output shows the dongle resource:
|
||||
|
||||
```yaml
|
||||
Capacity:
|
||||
alpha.kubernetes.io/nvidia-gpu: 0
|
||||
cpu: 2
|
||||
memory: 2049008Ki
|
||||
example.com/dongle: 4
|
||||
@@ -205,6 +203,3 @@ kubectl describe node <your-node-name> | grep dongle
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ content_template: templates/concept
|
||||
|
||||
In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a master machine
|
||||
there are a number of add-ons which, for various reasons, must run on a regular cluster node (rather than the Kubernetes master).
|
||||
Some of these add-ons are critical to a fully functional cluster, such as Heapster, DNS, and UI.
|
||||
Some of these add-ons are critical to a fully functional cluster, such as metrics-server, DNS, and UI.
|
||||
A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade)
|
||||
and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space
|
||||
vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason).
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
---
|
||||
reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Upgrading kubeadm clusters from v1.10 to v1.11
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
This page explains how to upgrade a Kubernetes cluster created with `kubeadm` from version 1.10.x to version 1.11.x, and from version 1.11.x to 1.11.y, where `y > x`.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- You need to have a `kubeadm` Kubernetes cluster running version 1.10.0 or later. Swap must be disabled. The cluster should use a static control plane and etcd pods.
|
||||
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md) carefully.
|
||||
- Make sure to back up any important components, such as app-level state stored in a database. `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice.
|
||||
|
||||
### Additional information
|
||||
|
||||
- All containers are restarted after upgrade, because the container spec hash value is changed.
|
||||
- You can upgrade only froom one minor version to the next minor version. That is, you cannot skip versions when you upgrade. For example, you can upgrade only from 1.10 to 1.11, not from 1.9 to 1.11.
|
||||
- The default DNS provider in version 1.11 is [CoreDNS](https://coredns.io/) rather than [kube-dns](https://github.com/kubernetes/dns).
|
||||
To keep `kube-dns`, pass `--feature-flags=CoreDNS=false` to `kubeadm upgrade apply`.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Upgrade the control plane
|
||||
|
||||
1. On your master node, run the following (as root:
|
||||
|
||||
export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
|
||||
export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
|
||||
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
|
||||
chmod a+rx /usr/bin/kubeadm
|
||||
|
||||
Note that upgrading the `kubeadm` package on your system prior to upgrading the control plane causes a failed upgrade. Even though `kubeadm` ships in the Kubernetes repositories, it's important to install it manually. The kubeadm team is working on fixing this limitation.
|
||||
|
||||
1. Verify that the download works and has the expected version:
|
||||
|
||||
```shell
|
||||
kubeadm version
|
||||
```
|
||||
|
||||
1. On the master node, run:
|
||||
|
||||
```shell
|
||||
kubeadm upgrade plan
|
||||
```
|
||||
|
||||
You should see output similar to this:
|
||||
|
||||
<!-- TODO: copy-paste actual output once new version is stable -->
|
||||
|
||||
```shell
|
||||
[preflight] Running pre-flight checks.
|
||||
[upgrade] Making sure the cluster is healthy:
|
||||
[upgrade/config] Making sure the configuration is correct:
|
||||
[upgrade/config] Reading configuration from the cluster...
|
||||
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
|
||||
I0618 20:32:32.950358 15307 feature_gate.go:230] feature gates: &{map[]}
|
||||
[upgrade] Fetching available versions to upgrade to
|
||||
[upgrade/versions] Cluster version: v1.10.4
|
||||
[upgrade/versions] kubeadm version: v1.11.0-beta.2.78+e0b33dbc2bde88
|
||||
|
||||
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
|
||||
COMPONENT CURRENT AVAILABLE
|
||||
Kubelet 1 x v1.10.4 v1.11.0
|
||||
|
||||
Upgrade to the latest version in the v1.10 series:
|
||||
|
||||
COMPONENT CURRENT AVAILABLE
|
||||
API Server v1.10.4 v1.11.0
|
||||
Controller Manager v1.10.4 v1.11.0
|
||||
Scheduler v1.10.4 v1.11.0
|
||||
Kube Proxy v1.10.4 v1.11.0
|
||||
CoreDNS 1.1.3
|
||||
Kube DNS 1.14.8
|
||||
Etcd 3.1.12 3.2.18
|
||||
|
||||
You can now apply the upgrade by executing the following command:
|
||||
|
||||
kubeadm upgrade apply v1.11.0
|
||||
|
||||
Note: Before you can perform this upgrade, you have to update kubeadm to v1.11.0.
|
||||
|
||||
_____________________________________________________________________
|
||||
```
|
||||
|
||||
This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to.
|
||||
|
||||
1. Choose a version to upgrade to, and run the appropriate command. For example:
|
||||
|
||||
```shell
|
||||
kubeadm upgrade apply v1.11.0
|
||||
```
|
||||
|
||||
If you currently use `kube-dns` and wish to continue doing so, add `--feature-flags=CoreDNS=false`.
|
||||
|
||||
You should see output similar to this:
|
||||
|
||||
<!-- TODO: output from stable -->
|
||||
|
||||
```shell
|
||||
[preflight] Running pre-flight checks.
|
||||
[upgrade] Making sure the cluster is healthy:
|
||||
[upgrade/config] Making sure the configuration is correct:
|
||||
[upgrade/config] Reading configuration from the cluster...
|
||||
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
|
||||
I0614 20:56:08.320369 30918 feature_gate.go:230] feature gates: &{map[]}
|
||||
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
|
||||
[upgrade/version] You have chosen to change the cluster version to "v1.11.0-beta.2.78+e0b33dbc2bde88"
|
||||
[upgrade/versions] Cluster version: v1.10.4
|
||||
[upgrade/versions] kubeadm version: v1.11.0-beta.2.78+e0b33dbc2bde88
|
||||
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
|
||||
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
|
||||
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.11.0-beta.2.78+e0b33dbc2bde88"...
|
||||
Static pod: kube-apiserver-ip-172-31-85-18 hash: 7a329408b21bc0c44d7b3b78ff8187bf
|
||||
Static pod: kube-controller-manager-ip-172-31-85-18 hash: 24fd3157627c7567b687968967c6a5e8
|
||||
Static pod: kube-scheduler-ip-172-31-85-18 hash: 5179266fb24d4c1834814c4f69486371
|
||||
Static pod: etcd-ip-172-31-85-18 hash: 9dfc197f444be11fcc70ab1467b030b8
|
||||
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/etcd.yaml"
|
||||
[certificates] Using the existing etcd/ca certificate and key.
|
||||
[certificates] Using the existing etcd/server certificate and key.
|
||||
[certificates] Using the existing etcd/peer certificate and key.
|
||||
[certificates] Using the existing etcd/healthcheck-client certificate and key.
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/etcd.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: etcd-ip-172-31-85-18 hash: 9dfc197f444be11fcc70ab1467b030b8
|
||||
< snip >
|
||||
[apiclient] Found 1 Pods for label selector component=etcd
|
||||
[upgrade/staticpods] Component "etcd" upgraded successfully!
|
||||
[upgrade/etcd] Waiting for etcd to become available
|
||||
[util/etcd] Waiting 0s for initial delay
|
||||
[util/etcd] Attempting to see if all cluster endpoints are available 1/10
|
||||
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939"
|
||||
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/kube-apiserver.yaml"
|
||||
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/kube-controller-manager.yaml"
|
||||
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/kube-scheduler.yaml"
|
||||
[certificates] Using the existing etcd/ca certificate and key.
|
||||
[certificates] Using the existing apiserver-etcd-client certificate and key.
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/kube-apiserver.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: kube-apiserver-ip-172-31-85-18 hash: 7a329408b21bc0c44d7b3b78ff8187bf
|
||||
< snip >
|
||||
[apiclient] Found 1 Pods for label selector component=kube-apiserver
|
||||
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/kube-controller-manager.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: kube-controller-manager-ip-172-31-85-18 hash: 24fd3157627c7567b687968967c6a5e8
|
||||
Static pod: kube-controller-manager-ip-172-31-85-18 hash: 63992ff14733dcb9dcfa6ac0a3b8031a
|
||||
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
|
||||
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/kube-scheduler.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: kube-scheduler-ip-172-31-85-18 hash: 5179266fb24d4c1834814c4f69486371
|
||||
Static pod: kube-scheduler-ip-172-31-85-18 hash: 831e4b9425f758e572392976311e56d9
|
||||
[apiclient] Found 1 Pods for label selector component=kube-scheduler
|
||||
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
|
||||
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
|
||||
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
|
||||
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
|
||||
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
|
||||
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ip-172-31-85-18" as an annotation
|
||||
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
|
||||
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
|
||||
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
|
||||
[addons] Applied essential addon: CoreDNS
|
||||
[addons] Applied essential addon: kube-proxy
|
||||
|
||||
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.11.0-beta.2.78+e0b33dbc2bde88". Enjoy!
|
||||
|
||||
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
|
||||
```
|
||||
|
||||
1. Manually upgrade your Software Defined Network (SDN).
|
||||
|
||||
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow.
|
||||
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
|
||||
find your CNI provider and see whther additional upgrade steps are required.
|
||||
|
||||
## Upgrade master and node packages
|
||||
|
||||
1. Prepare each host for maintenance, marking it unschedulable and evicting the workload:
|
||||
|
||||
```shell
|
||||
kubectl drain $HOST --ignore-daemonsets
|
||||
```
|
||||
|
||||
On the master host, you must add `--ignore-daemonsets`:
|
||||
|
||||
```shell
|
||||
kubectl drain ip-172-31-85-18
|
||||
node "ip-172-31-85-18" cordoned
|
||||
error: unable to drain node "ip-172-31-85-18", aborting command...
|
||||
|
||||
There are pending nodes to be drained:
|
||||
ip-172-31-85-18
|
||||
error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-5798d, kube-proxy-thjp9
|
||||
```
|
||||
|
||||
```
|
||||
kubectl drain ip-172-31-85-18 --ignore-daemonsets
|
||||
node "ip-172-31-85-18" already cordoned
|
||||
WARNING: Ignoring DaemonSet-managed pods: calico-node-5798d, kube-proxy-thjp9
|
||||
node "ip-172-31-85-18" drained
|
||||
```
|
||||
|
||||
1. Upgrade the Kubernetes package version on each `$HOST` node by running the Linux package manager for your distribution:
|
||||
|
||||
{{< tabs name="k8s_install" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
apt-get update
|
||||
apt-get upgrade -y kubelet kubeadm
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
yum upgrade -y kubelet kubedam
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
## Upgrade kubelet on each node
|
||||
|
||||
1. On each node except the master node, upgrade the kubelet config:
|
||||
|
||||
```shell
|
||||
sudo kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)
|
||||
```
|
||||
|
||||
1. Restart the kubectl process:
|
||||
|
||||
```shell
|
||||
sudo systemctl restart kubelet
|
||||
```
|
||||
|
||||
1. Verify that the new version of the `kubelet` is running on the host:
|
||||
|
||||
```shell
|
||||
systemctl status kubelet
|
||||
```
|
||||
|
||||
1. Bring the host back online by marking it schedulable:
|
||||
|
||||
```shell
|
||||
kubectl uncordon $HOST
|
||||
```
|
||||
|
||||
1. After the kubelet is upgraded on all hosts, verify that all nodes are available again by running the following command from anywhere -- for example, from outside the cluster:
|
||||
|
||||
```shell
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
The `STATUS` column should show `Ready` for all your hosts, and the version number should be updated.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
## Recovering from a failure state
|
||||
|
||||
If `kubeadm upgrade` fails and does not roll back, for example because of an unexpected shutdown during execution,
|
||||
you can run `kubeadm upgrade` again. This command is idempotent and eventually makes sure that the actual state is the desired state you declare.
|
||||
|
||||
To recover from a bad state, you can also run `kubeadm upgrade --force` without changing the version that your cluster is running.
|
||||
|
||||
## How it works
|
||||
|
||||
`kubeadm upgrade apply` does the following:
|
||||
|
||||
- Checks that your cluster is in an upgradeable state:
|
||||
- The API server is reachable,
|
||||
- All nodes are in the `Ready` state
|
||||
- The control plane is healthy
|
||||
- Enforces the version skew policies.
|
||||
- Makes sure the control plane images are available or available to pull to the machine.
|
||||
- Upgrades the control plane components or rollbacks if any of them fails to come up.
|
||||
- Applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
|
||||
- Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days.
|
||||
@@ -1,145 +0,0 @@
|
||||
---
|
||||
reviewers:
|
||||
- msau42
|
||||
- jsafrane
|
||||
title: Persistent Volume Claim Protection
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state for_k8s_version="v1.9" state="alpha" >}}
|
||||
|
||||
As of Kubernetes 1.9, persistent volume claims (PVCs) that are in active use by a pod can be protected from pre-mature removal.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- A v1.9 or higher Kubernetes must be installed.
|
||||
- As PVC Protection is a Kubernetes v1.9 alpha feature it must be enabled:
|
||||
1. [Admission controller](/docs/admin/admission-controllers/) must be started with the [PVC Protection plugin](/docs/admin/admission-controllers/#persistent-volume-claim-protection-alpha).
|
||||
2. All Kubernetes components must be started with the `PVCProtection` alpha features enabled.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## PVC Protection Verification
|
||||
|
||||
The example below uses a GCE PD `StorageClass`, however, similar steps can be performed for any volume type.
|
||||
|
||||
Create a `StorageClass` for convenient storage provisioning:
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: slow
|
||||
provisioner: kubernetes.io/gce-pd
|
||||
parameters:
|
||||
type: pd-standard
|
||||
```
|
||||
|
||||
There are two scenarios: a PVC deleted by a user is either in active use or not in active use by a pod.
|
||||
|
||||
### Scenario 1: The PVC is not in active use by a pod
|
||||
|
||||
- Create a PVC:
|
||||
|
||||
```yaml
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: slzc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: slow
|
||||
resources:
|
||||
requests:
|
||||
storage: 3.7Gi
|
||||
```
|
||||
|
||||
- Check that the PVC has the finalizer `kubernetes.io/pvc-protection` set:
|
||||
```shell
|
||||
$ kubectl describe pvc slzc
|
||||
Name: slzc
|
||||
Namespace: default
|
||||
StorageClass: slow
|
||||
Status: Bound
|
||||
Volume: pvc-bee8c30a-d6a3-11e7-9af0-42010a800002
|
||||
Labels: <none>
|
||||
Annotations: pv.kubernetes.io/bind-completed=yes
|
||||
pv.kubernetes.io/bound-by-controller=yes
|
||||
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
|
||||
Finalizers: [kubernetes.io/pvc-protection]
|
||||
Capacity: 4Gi
|
||||
Access Modes: RWO
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal ProvisioningSucceeded 2m persistentvolume-controller Successfully provisioned volume pvc-bee8c30a-d6a3-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
|
||||
```
|
||||
|
||||
- Delete the PVC and check that the PVC (not in active use by a pod) was removed successfully.
|
||||
|
||||
### Scenario 2: The PVC is in active use by a pod
|
||||
|
||||
- Again, create the same PVC.
|
||||
- Create a pod that uses the PVC:
|
||||
|
||||
```yaml
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: app1
|
||||
spec:
|
||||
containers:
|
||||
- name: test-pod
|
||||
image: k8s.gcr.io/busybox:1.24
|
||||
command:
|
||||
- "/bin/sh"
|
||||
args:
|
||||
- "-c"
|
||||
- "date > /mnt/app1.txt; sleep 60 && exit 0 || exit 1"
|
||||
volumeMounts:
|
||||
- name: path-pvc
|
||||
mountPath: "/mnt"
|
||||
restartPolicy: "Never"
|
||||
volumes:
|
||||
- name: path-pvc
|
||||
persistentVolumeClaim:
|
||||
claimName: slzc
|
||||
```
|
||||
|
||||
- Wait until the pod status is `Running`, i.e. the PVC becomes in active use.
|
||||
- Delete the PVC that is now in active use by a pod and verify that the PVC is not removed but its status is `Terminating`:
|
||||
|
||||
```shell
|
||||
Name: slzc
|
||||
Namespace: default
|
||||
StorageClass: slow
|
||||
Status: Terminating (since Fri, 01 Dec 2017 14:47:55 +0000)
|
||||
Volume: pvc-803a1f4d-d6a6-11e7-9af0-42010a800002
|
||||
Labels: <none>
|
||||
Annotations: pv.kubernetes.io/bind-completed=yes
|
||||
pv.kubernetes.io/bound-by-controller=yes
|
||||
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
|
||||
Finalizers: [kubernetes.io/pvc-protection]
|
||||
Capacity: 4Gi
|
||||
Access Modes: RWO
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal ProvisioningSucceeded 52s persistentvolume-controller Successfully provisioned volume pvc-803a1f4d-d6a6-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
|
||||
```
|
||||
- Wait until the pod status is `Terminated` (either delete the pod or wait until it finishes). Afterwards, check that the PVC is removed.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture discussion %}}
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -7,31 +7,28 @@ content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state state="alpha" >}}
|
||||
As of Kubernetes 1.8, the new
|
||||
{{< feature-state for_k8s_version="v1.11" state="beta" >}}
|
||||
|
||||
[Dynamic Kubelet Configuration](https://github.com/kubernetes/features/issues/281)
|
||||
feature is available in alpha. This allows you to change the configuration of
|
||||
Kubelets in a live Kubernetes cluster via first-class Kubernetes concepts.
|
||||
Specifically, this feature allows you to configure individual Nodes' Kubelets
|
||||
via ConfigMaps.
|
||||
allows you to change the configuration of each Kubelet in a live Kubernetes
|
||||
cluster by deploying a ConfigMap and configuring each Node to use it.
|
||||
|
||||
{{< warning >}}
|
||||
**Warning:** All Kubelet configuration parameters may be changed dynamically,
|
||||
but not all parameters are safe to change dynamically. This feature is intended
|
||||
for system experts who have a strong understanding of how configuration changes
|
||||
will affect behavior. No documentation currently exists which plainly lists
|
||||
"safe to change" fields, but we plan to add it before this feature graduates
|
||||
from alpha.
|
||||
**Warning:** All Kubelet configuration parameters can be changed dynamically,
|
||||
but this is unsafe for some parameters. Before deciding to change a parameter
|
||||
dynamically, you need a strong understanding of how that change will affect your
|
||||
cluster's behavior. Always carefully test configuration changes on a small set
|
||||
of nodes before rolling them out cluster-wide. Advice on configuring specific
|
||||
fields is available in the inline `KubeletConfiguration`
|
||||
[type documentation](https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go).
|
||||
{{< /warning >}}
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
- A live Kubernetes cluster with both Master and Node at v1.8 or higher must be
|
||||
running, with the `DynamicKubeletConfig` feature gate enabled and the Kubelet's
|
||||
`--dynamic-config-dir` flag set to a writable directory on the Node.
|
||||
This flag must be set to enable Dynamic Kubelet Configuration.
|
||||
- The kubectl command-line tool must be also v1.8 or higher, and must be
|
||||
configured to communicate with the cluster.
|
||||
- Kubernetes v1.11 or higher on both the Master and the Nodes
|
||||
- kubectl v1.11 or higher, configured to communicate with the cluster
|
||||
- The Kubelet's `--dynamic-config-dir` flag must be set to a writable
|
||||
directory on the Node.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
@@ -50,20 +47,19 @@ Kubelet's configuration.
|
||||
Each Kubelet watches a configuration reference on its respective Node object.
|
||||
When this reference changes, the Kubelet downloads the new configuration,
|
||||
updates a local reference to refer to the file, and exits.
|
||||
For the feature to work correctly, you must be running a process manager
|
||||
(like systemd) which will restart the Kubelet when it exits. When the Kubelet is
|
||||
restarted, it will begin using the new configuration.
|
||||
For the feature to work correctly, you must be running an OS-level service
|
||||
manager (such as systemd), which will restart the Kubelet if it exits. When the
|
||||
Kubelet is restarted, it will begin using the new configuration.
|
||||
|
||||
The new configuration completely overrides configuration provided by `--config`,
|
||||
and is overridden by command-line flags. Unspecified values in the new configuration
|
||||
will receive default values appropriate to the configuration version
|
||||
(e.g. `kubelet.config.k8s.io/v1beta1`), unless overridden by flags.
|
||||
|
||||
The status of the Node's Kubelet configuration is reported via the `KubeletConfigOK`
|
||||
condition in the Node status. Once you have updated a Node to use the new
|
||||
ConfigMap, you can observe this condition to confirm that the Node is using the
|
||||
intended configuration. A table describing the possible conditions can be found
|
||||
at the end of this article.
|
||||
The status of the Node's Kubelet configuration is reported via
|
||||
`Node.Spec.Status.Config`. Once you have updated a Node to use the new
|
||||
ConfigMap, you can observe this status to confirm that the Node is using the
|
||||
intended configuration.
|
||||
|
||||
This document describes editing Nodes using `kubectl edit`.
|
||||
There are other ways to modify a Node's spec, including `kubectl patch`, for
|
||||
@@ -72,16 +68,19 @@ example, which facilitate scripted workflows.
|
||||
This document only describes a single Node consuming each ConfigMap. Keep in
|
||||
mind that it is also valid for multiple Nodes to consume the same ConfigMap.
|
||||
|
||||
### Node Authorizer Workarounds
|
||||
{{< warning >}}
|
||||
**Warning:** While it is *possible* to change the configuration by
|
||||
updating the ConfigMap in-place, this causes all Kubelets configured with
|
||||
that ConfigMap to update simultaneously. It is much safer to treat ConfigMaps
|
||||
as immutable by convention, aided by `kubectl`'s `--append-hash` option,
|
||||
and incrementally roll out updates to `Node.Spec.ConfigSource`.
|
||||
{{< /warning >}}
|
||||
|
||||
The Node Authorizer does not yet pay attention to which ConfigMaps are assigned
|
||||
to which Nodes. If you currently use the Node authorizer, your Kubelets will not
|
||||
be automatically granted permission to download their respective ConfigMaps.
|
||||
### Automatic RBAC rules for Node Authorizer
|
||||
|
||||
The temporary workaround used in this document is to manually create the RBAC
|
||||
Roles and RoleBindings for each ConfigMap. The Node Authorizer will be extended
|
||||
before the Dynamic Kubelet Configuration feature graduates from alpha, so doing
|
||||
this in production should never be necessary.
|
||||
Previously, you were required to manually create RBAC rules
|
||||
to allow Nodes to access their assigned ConfigMaps. The Node Authorizer now
|
||||
automatically configures these rules.
|
||||
|
||||
### Generating a file that contains the current configuration
|
||||
|
||||
@@ -92,54 +91,63 @@ and debug issues. The compromise, however, is that you must start with knowledge
|
||||
of the existing configuration to ensure that you only change the fields you
|
||||
intend to change.
|
||||
|
||||
In the future, the Kubelet will be bootstrapped from a file on disk
|
||||
Ideally, the Kubelet would be bootstrapped from a file on disk
|
||||
and you could edit this file (which could also be version-controlled),
|
||||
to create the first Kubelet ConfigMap
|
||||
(see [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)),
|
||||
and you will simply edit a copy of this file (which, as a best practice, should
|
||||
live in version control) while creating the first Kubelet ConfigMap. Today,
|
||||
however, the Kubelet is still bootstrapped with command-line flags. Fortunately,
|
||||
there is a dirty trick you can use to generate a config file containing a Node's
|
||||
current configuration. The trick involves accessing the Kubelet server's `configz`
|
||||
endpoint via the kubectl proxy. This endpoint, in its current implementation, is
|
||||
intended to be used only as a debugging aid, which is part of why this is a
|
||||
dirty trick. The endpoint may be improved in the future, but until then
|
||||
it should not be relied on for production scenarios.
|
||||
This trick also requires the `jq` command to be installed on your machine,
|
||||
for unpacking and editing the JSON response from the endpoint.
|
||||
Currently, the Kubelet is bootstrapped with **a combination of this file and command-line flags**
|
||||
that can override the configuration in the file.
|
||||
As a workaround, you can generate a config file containing a Node's current
|
||||
configuration by accessing the Kubelet server's `configz` endpoint via the
|
||||
kubectl proxy. This endpoint, in its current implementation, is intended to be
|
||||
used only as a debugging aid. Do not rely on the behavior of this endpoint for
|
||||
production scenarios. The examples below use the `jq` command to streamline
|
||||
working with JSON. To follow the tasks as written, you need to have `jq`
|
||||
installed, but you can adapt the tasks if you prefer to extract the
|
||||
`kubeletconfig` subobject manually.
|
||||
|
||||
Do the following to generate the file:
|
||||
#### Generate the configuration file
|
||||
|
||||
1. Pick a Node to reconfigure. We will refer to this Node's name as NODE_NAME.
|
||||
2. Start the kubectl proxy in the background with `kubectl proxy --port=8001 &`
|
||||
3. Run the following command to download and unpack the configuration from the
|
||||
configz endpoint:
|
||||
1. Choose a Node to reconfigure. In this example, the name of this Node is
|
||||
referred to as `NODE_NAME`.
|
||||
2. Start the kubectl proxy in the background using the following command:
|
||||
```bash
|
||||
kubectl proxy --port=8001 &
|
||||
```
|
||||
3. Run the following command to download and unpack the configuration from the
|
||||
`configz` endpoint. The command is long, so be careful when copying and
|
||||
pasting. **If you use zsh**, note that common zsh configurations add backslashes
|
||||
to escape the opening and closing curly braces around the variable name in the URL.
|
||||
For example: `${NODE_NAME}` will be rewritten as `$\{NODE_NAME\}` during the paste.
|
||||
You must remove the backslashes before running the command, or the command will fail.
|
||||
|
||||
```
|
||||
$ export NODE_NAME=the-name-of-the-node-you-are-reconfiguring
|
||||
$ curl -sSL http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
|
||||
```
|
||||
```bash
|
||||
NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
|
||||
```
|
||||
|
||||
Note that we have to manually add the `kind` and `apiVersion` to the downloaded
|
||||
object, as these are not reported by the configz endpoint. This is one of the
|
||||
limitations of the endpoint.
|
||||
{{< note >}}
|
||||
You need to manually add the `kind` and `apiVersion` to the downloaded
|
||||
object, because they are not reported by the `configz` endpoint.
|
||||
{{< /note >}}
|
||||
|
||||
### Edit the configuration file
|
||||
#### Edit the configuration file
|
||||
|
||||
Using your editor of choice, change one of the parameters in the
|
||||
`kubelet_configz_${NODE_NAME}` file from the previous step. A QPS parameter,
|
||||
`eventRecordQPS` for example, is a good candidate.
|
||||
Using a text editor, change one of the parameters in the
|
||||
file generated by the previous procedure. For example, you
|
||||
might edit the QPS parameter `eventRecordQPS`.
|
||||
|
||||
### Push the configuration file to the control plane
|
||||
#### Push the configuration file to the control plane
|
||||
|
||||
Push the edited configuration file to the control plane with the
|
||||
following command:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
|
||||
```bash
|
||||
kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
|
||||
```
|
||||
|
||||
You should see a response similar to:
|
||||
This is an example of a valid response:
|
||||
|
||||
```
|
||||
```none
|
||||
apiVersion: v1
|
||||
data:
|
||||
kubelet: |
|
||||
@@ -154,304 +162,204 @@ metadata:
|
||||
uid: 946d785e-998a-11e7-a8dd-42010a800006
|
||||
```
|
||||
|
||||
Note that the configuration data must appear under the ConfigMap's
|
||||
`kubelet` key.
|
||||
|
||||
We create the ConfigMap in the `kube-system` namespace, which is appropriate
|
||||
because this ConfigMap configures a Kubernetes system component - the Kubelet.
|
||||
The ConfigMap is created in the `kube-system` namespace because this
|
||||
ConfigMap configures a Kubelet, which is Kubernetes system component.
|
||||
|
||||
The `--append-hash` option appends a short checksum of the ConfigMap contents
|
||||
to the name. This is convenient for an edit->push workflow, as it will
|
||||
automatically, yet deterministically, generate new names for new ConfigMaps.
|
||||
to the name. This is convenient for an edit-then-push workflow, because it
|
||||
automatically, yet deterministically, generates new names for new ConfigMaps.
|
||||
The name that includes this generated hash is referred to as `CONFIG_MAP_NAME`
|
||||
in the following examples.
|
||||
|
||||
We use the `-o yaml` output format so that the name, namespace, and uid are all
|
||||
reported following creation. We will need these in the next step. We will refer
|
||||
to the name as CONFIG_MAP_NAME and the uid as CONFIG_MAP_UID.
|
||||
|
||||
### Authorize your Node to read the new ConfigMap
|
||||
|
||||
Now that you've created a new ConfigMap, you need to authorize your node to
|
||||
read it. First, create a Role for your new ConfigMap with the
|
||||
following commands:
|
||||
|
||||
```
|
||||
$ export CONFIG_MAP_NAME=name-from-previous-output
|
||||
$ kubectl -n kube-system create role ${CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${CONFIG_MAP_NAME}
|
||||
```
|
||||
|
||||
Next, create a RoleBinding to associate your Node with the new Role:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system create rolebinding ${CONFIG_MAP_NAME}-reader --role=${CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
|
||||
```
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
|
||||
### Set the Node to use the new configuration
|
||||
#### Set the Node to use the new configuration
|
||||
|
||||
Edit the Node's reference to point to the new ConfigMap with the
|
||||
following command:
|
||||
|
||||
```
|
||||
```bash
|
||||
kubectl edit node ${NODE_NAME}
|
||||
```
|
||||
|
||||
Once in your editor, add the following YAML under `spec`:
|
||||
In your text editor, add the following YAML under `spec`:
|
||||
|
||||
```
|
||||
```yaml
|
||||
configSource:
|
||||
configMapRef:
|
||||
configMap:
|
||||
name: CONFIG_MAP_NAME
|
||||
namespace: kube-system
|
||||
uid: CONFIG_MAP_UID
|
||||
kubeletConfigKey: kubelet
|
||||
```
|
||||
|
||||
Be sure to specify all three of `name`, `namespace`, and `uid`.
|
||||
You must specify all three of `name`, `namespace`, and `kubeletConfigKey`.
|
||||
The `kubeletConfigKey` parameter shows the Kubelet which key of the ConfigMap
|
||||
contains its config.
|
||||
|
||||
### Observe that the Node begins using the new configuration
|
||||
#### Observe that the Node begins using the new configuration
|
||||
|
||||
Retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and look for the
|
||||
`KubeletConfigOK` condition in `status.conditions`. You should see the message
|
||||
`Using current (UID: CONFIG_MAP_UID)` when the Kubelet starts using the new
|
||||
configuration.
|
||||
Retrieve the Node using the `kubectl get node ${NODE_NAME} -o yaml` command and inspect
|
||||
`Node.Status.Config`. The config sources corresponding to the `active`,
|
||||
`assigned`, and `lastKnownGood` configurations are reported in the status.
|
||||
|
||||
For convenience, you can use the following command (using `jq`) to filter down
|
||||
to the `KubeletConfigOK` condition:
|
||||
- The `active` configuration is the version the Kubelet is currently running with.
|
||||
- The `assigned` configuration is the latest version the Kubelet has resolved based on
|
||||
`Node.Spec.ConfigSource`.
|
||||
- The `lastKnownGood` configuration is the version the
|
||||
Kubelet will fall back to if an invalid config is assigned in `Node.Spec.ConfigSource`.
|
||||
|
||||
The`lastKnownGood` configuration might not be present if it is set to its default value,
|
||||
the local config deployed with the node. The status will update `lastKnownGood` to
|
||||
match a valid `assigned` config after the Kubelet becomes comfortable with the config.
|
||||
The details of how the Kubelet determines a config should become the `lastKnownGood` are
|
||||
not guaranteed by the API, but is currently implemented as a 10-minute grace period.
|
||||
|
||||
You can use the following command (using `jq`) to filter down
|
||||
to the config status:
|
||||
|
||||
```bash
|
||||
kubectl get no ${NODE_NAME} -o json | jq '.status.config'
|
||||
```
|
||||
$ kubectl get no ${NODE_NAME} -o json | jq '.status.conditions|map(select(.type=="KubeletConfigOK"))'
|
||||
[
|
||||
{
|
||||
"lastHeartbeatTime": "2017-09-20T18:08:29Z",
|
||||
"lastTransitionTime": "2017-09-20T18:08:17Z",
|
||||
"message": "using current: /api/v1/namespaces/kube-system/configmaps/my-node-config-gkt4c2m4b2",
|
||||
"reason": "passing all checks",
|
||||
"status": "True",
|
||||
"type": "KubeletConfigOK"
|
||||
|
||||
The following is an example response:
|
||||
|
||||
```json
|
||||
{
|
||||
"active": {
|
||||
"configMap": {
|
||||
"kubeletConfigKey": "kubelet",
|
||||
"name": "my-node-config-9mbkccg2cc",
|
||||
"namespace": "kube-system",
|
||||
"resourceVersion": "1326",
|
||||
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
|
||||
}
|
||||
},
|
||||
"assigned": {
|
||||
"configMap": {
|
||||
"kubeletConfigKey": "kubelet",
|
||||
"name": "my-node-config-9mbkccg2cc",
|
||||
"namespace": "kube-system",
|
||||
"resourceVersion": "1326",
|
||||
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
|
||||
}
|
||||
},
|
||||
"lastKnownGood": {
|
||||
"configMap": {
|
||||
"kubeletConfigKey": "kubelet",
|
||||
"name": "my-node-config-9mbkccg2cc",
|
||||
"namespace": "kube-system",
|
||||
"resourceVersion": "1326",
|
||||
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
If something goes wrong, you may see one of several different error conditions,
|
||||
detailed in the table of KubeletConfigOK conditions, below. When this happens, you
|
||||
should check the Kubelet's log for more details.
|
||||
|
||||
### Edit the configuration file again
|
||||
|
||||
To change the configuration again, we simply repeat the above workflow.
|
||||
Try editing the `kubelet` file, changing the previously changed parameter to a
|
||||
new value.
|
||||
|
||||
### Push the newly edited configuration to the control plane
|
||||
|
||||
Push the new configuration to the control plane in a new ConfigMap with the
|
||||
following command:
|
||||
}
|
||||
|
||||
```
|
||||
$ kubectl create configmap my-node-config --namespace=kube-system --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
|
||||
```
|
||||
|
||||
This new ConfigMap will get a new name, as we have changed the contents.
|
||||
We will refer to the new name as NEW_CONFIG_MAP_NAME and the new uid
|
||||
as NEW_CONFIG_MAP_UID.
|
||||
If an error occurs, the Kubelet reports it in the `Node.Status.Config.Error`
|
||||
structure. Possible errors are listed in
|
||||
[Understanding Node.Status.Config.Error messages](#understanding-node-status-config-error-messages).
|
||||
You can search for the identical text in the Kubelet log for additional details
|
||||
and context about the error.
|
||||
|
||||
### Authorize your Node to read the new ConfigMap
|
||||
#### Make more changes
|
||||
|
||||
Now that you've created a new ConfigMap, you need to authorize your node to
|
||||
read it. First, create a Role for your new ConfigMap with the
|
||||
following commands:
|
||||
Follow the workflow above to make more changes and push them again. Each time
|
||||
you push a ConfigMap with new contents, the --append-hash kubectl option creates
|
||||
the ConfigMap with a new name. The safest rollout strategy is to first create a
|
||||
new ConfigMap, and then update the Node to use the new ConfigMap.
|
||||
|
||||
```
|
||||
$ export NEW_CONFIG_MAP_NAME=name-from-previous-output
|
||||
$ kubectl -n kube-system create role ${NEW_CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${NEW_CONFIG_MAP_NAME}
|
||||
```
|
||||
#### Reset the Node to use its local default configuration
|
||||
|
||||
Next, create a RoleBinding to associate your Node with the new Role:
|
||||
To reset the Node to use the configuration it was provisioned with, edit the
|
||||
Node using `kubectl edit node ${NODE_NAME}` and remove the
|
||||
`Node.Spec.ConfigSource` field.
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system create rolebinding ${NEW_CONFIG_MAP_NAME}-reader --role=${NEW_CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
|
||||
```
|
||||
#### Observe that the Node is using its local default configuration
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
|
||||
### Configure the Node to use the new configuration
|
||||
|
||||
Once more, edit the Node's `spec.configSource` with
|
||||
`kubectl edit node ${NODE_NAME}`. Your new `spec.configSource` should look like
|
||||
the following, with `name` and `uid` substituted as necessary:
|
||||
|
||||
```
|
||||
configSource:
|
||||
configMapRef:
|
||||
name: ${NEW_CONFIG_MAP_NAME}
|
||||
namespace: kube-system
|
||||
uid: ${NEW_CONFIG_MAP_UID}
|
||||
```
|
||||
|
||||
### Observe that the Kubelet is using the new configuration
|
||||
|
||||
Once more, retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and
|
||||
look for the `KubeletConfigOK` condition in `status.conditions`. You should see the message
|
||||
`using current: /api/v1/namespaces/kube-system/configmaps/${NEW_CONFIG_MAP_NAME}` when the Kubelet starts using the
|
||||
new configuration.
|
||||
|
||||
### Deauthorize your Node from reading the old ConfigMap
|
||||
|
||||
Once you know your Node is using the new configuration and are confident that
|
||||
the new configuration has not caused any problems, it is a good idea to
|
||||
deauthorize the node from reading the old ConfigMap. Run the following
|
||||
commands to remove the RoleBinding and Role:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete rolebinding ${CONFIG_MAP_NAME}-reader
|
||||
$ kubectl -n kube-system delete role ${CONFIG_MAP_NAME}-reader
|
||||
```
|
||||
|
||||
Note that this does not necessarily prevent the Node from reverting to the old
|
||||
configuration, as it may locally cache the old ConfigMap for an indefinite
|
||||
period of time.
|
||||
|
||||
You may optionally also choose to remove the old ConfigMap:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete configmap ${CONFIG_MAP_NAME}
|
||||
```
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
|
||||
### Reset the Node to use its local default configuration
|
||||
|
||||
Finally, if you wish to reset the Node to use the configuration it was
|
||||
provisioned with, simply edit the Node with `kubectl edit node ${NODE_NAME}` and
|
||||
remove the `spec.configSource` subfield.
|
||||
|
||||
### Observe that the Node is using its local default configuration
|
||||
|
||||
After removing this subfield, you should eventually observe that the KubeletConfigOK
|
||||
condition's message reverts to `using current: local`.
|
||||
|
||||
### Deauthorize your Node from reading the old ConfigMap
|
||||
|
||||
Once you know your Node is using the default configuration again, it is a good
|
||||
idea to deauthorize the node from reading the old ConfigMap. Run the following
|
||||
commands to remove the RoleBinding and Role:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete rolebinding ${NEW_CONFIG_MAP_NAME}-reader
|
||||
$ kubectl -n kube-system delete role ${NEW_CONFIG_MAP_NAME}-reader
|
||||
```
|
||||
|
||||
Note that this does not necessarily prevent the Node from reverting to the old
|
||||
ConfigMap, as it may locally cache the old ConfigMap for an indefinite
|
||||
period of time.
|
||||
|
||||
You may optionally also choose to remove the old ConfigMap:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete configmap ${NEW_CONFIG_MAP_NAME}
|
||||
```
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
After removing this subfield, `Node.Status.Config` eventually becomes
|
||||
empty, since all config sources have been reset to `nil`, which indicates that
|
||||
the local default config is `assigned`, `active`, and `lastKnownGood`, and no
|
||||
error is reported.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture discussion %}}
|
||||
## Kubectl Patch Example
|
||||
As mentioned above, there are many ways to change a Node's configSource.
|
||||
Here is an example command that uses `kubectl patch`:
|
||||
|
||||
```
|
||||
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMapRef\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"uid\":\"${CONFIG_MAP_UID}\"}}}}"
|
||||
You can change a Node's configSource using several different mechanisms.
|
||||
This example uses `kubectl patch`:
|
||||
|
||||
```bash
|
||||
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
|
||||
```
|
||||
|
||||
## Understanding KubeletConfigOK Conditions
|
||||
## Understanding how the Kubelet checkpoints config
|
||||
|
||||
The following table describes several of the `KubeletConfigOK` Node conditions you
|
||||
might encounter in a cluster that has Dynamic Kubelet Config enabled. If you
|
||||
observe a condition with `status=False`, you should check the Kubelet log for
|
||||
more error details by searching for the message or reason text.
|
||||
When a new config is assigned to the Node, the Kubelet downloads and unpacks the
|
||||
config payload as a set of files on the local disk. The Kubelet also records metadata
|
||||
that locally tracks the assigned and last-known-good config sources, so that the
|
||||
Kubelet knows which config to use across restarts, even if the API server becomes
|
||||
unavailable. After checkpointing a config and the relevant metadata, the Kubelet
|
||||
exits if it detects that the assigned config has changed. When the Kubelet is
|
||||
restarted by the OS-level service manager (such as `systemd`), it reads the new
|
||||
metadata and uses the new config.
|
||||
|
||||
The recorded metadata is fully resolved, meaning that it contains all necessary
|
||||
information to choose a specific config version - typically a `UID` and `ResourceVersion`.
|
||||
This is in contrast to `Node.Spec.ConfigSource`, where the intended config is declared
|
||||
via the idempotent `namespace/name` that identifies the target ConfigMap; the Kubelet
|
||||
tries to use the latest version of this ConfigMap.
|
||||
|
||||
When you are debugging problems on a node, you can inspect the Kubelet's config
|
||||
metadata and checkpoints. The structure of the Kubelet's checkpointing directory is:
|
||||
|
||||
```none
|
||||
- --dynamic-config-dir (root for managing dynamic config)
|
||||
| - meta
|
||||
| - assigned (encoded kubeletconfig/v1beta1.SerializedNodeConfigSource object, indicating the assigned config)
|
||||
| - last-known-good (encoded kubeletconfig/v1beta1.SerializedNodeConfigSource object, indicating the last-known-good config)
|
||||
| - checkpoints
|
||||
| - uid1 (dir for versions of object identified by uid1)
|
||||
| - resourceVersion1 (dir for unpacked files from resourceVersion1 of object with uid1)
|
||||
| - ...
|
||||
| - ...
|
||||
```
|
||||
|
||||
## Understanding Node.Status.Config.Error messages
|
||||
|
||||
The following table describes error messages that can occur
|
||||
when using Dynamic Kubelet Config. You can search for the identical text
|
||||
in the Kubelet log for additional details and context about the error.
|
||||
|
||||
<table>
|
||||
|
||||
|
||||
<table align="left">
|
||||
<tr>
|
||||
<th>Possible Messages</th>
|
||||
<th>Possible Reasons</th>
|
||||
<th>Status</th>
|
||||
<th>Error Message</th>
|
||||
<th>Possible Causes</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using current: local</p></td>
|
||||
<td><p>when the config source is nil, the Kubelet uses its local config</p></td>
|
||||
<td><p>True</p></td>
|
||||
<td><p>failed to load config, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet likely could not parse the downloaded config payload, or encountered a filesystem error attempting to load the payload from disk.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</p></td>
|
||||
<td><p>passing all checks</p></td>
|
||||
<td><p>True</p></td>
|
||||
<td><p>failed to validate config, see Kubelet log for details</p></td>
|
||||
<td><p>The configuration in the payload, combined with any command-line flag overrides, and the sum of feature gates from flags, the config file, and the remote payload, was determined to be invalid by the Kubelet.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using last-known-good: local</p></td>
|
||||
<td>
|
||||
<ul>
|
||||
<li>failed to load current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to parse current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to validate current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
</ul>
|
||||
</td>
|
||||
<td><p>False</p></td>
|
||||
<td><p>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</p></td>
|
||||
<td><p>Since Node.Spec.ConfigSource is validated by the API server to contain at least one non-nil subfield, this likely means that the Kubelet is older than the API server and does not recognize a newer source type.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using last-known-good: /api/v1/namespaces/${LAST_KNOWN_GOOD_CONFIG_MAP_NAMESPACE}/configmaps/${LAST_KNOWN_GOOD_CONFIG_MAP_NAME}</p></td>
|
||||
<td>
|
||||
<ul>
|
||||
<li>failed to load current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to parse current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to validate current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
</ul>
|
||||
</td>
|
||||
<td><p>False</p></td>
|
||||
<td><p>failed to sync: failed to download config, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet could not download the config. It is possible that Node.Spec.ConfigSource could not be resolved to a concrete API object, or that network errors disrupted the download attempt. The Kubelet will retry the download when in this error state.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<p>
|
||||
The reasons in the next column could potentially appear for any of
|
||||
the above messages.
|
||||
</p>
|
||||
<p>
|
||||
This condition indicates that the Kubelet is having trouble
|
||||
reconciling `spec.configSource`, and thus no change to the in-use
|
||||
configuration has occurred.
|
||||
</p>
|
||||
<p>
|
||||
The "failed to sync" reasons are specific to the failure that
|
||||
occurred, and the next column does not necessarily contain all
|
||||
possible failure reasons.
|
||||
</p>
|
||||
</td>
|
||||
<td>
|
||||
<p>failed to sync, reason:</p>
|
||||
<ul>
|
||||
<li>failed to read Node from informer object cache</li>
|
||||
<li>failed to reset to local config</li>
|
||||
<li>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</li>
|
||||
<li>invalid ObjectReference, all of UID, Name, and Namespace must be specified</li>
|
||||
<li>invalid ConfigSource.ConfigMapRef.UID: ${UID} does not match ${API_PATH}.UID: ${UID_OF_CONFIG_MAP_AT_API_PATH}</li>
|
||||
<li>failed to determine whether object ${API_PATH} with UID ${UID} was already checkpointed</li>
|
||||
<li>failed to download ConfigMap with name ${NAME} from namespace ${NAMESPACE}</li>
|
||||
<li>failed to save config checkpoint for object ${API_PATH} with UID ${UID}</li>
|
||||
<li>failed to set current config checkpoint to local config</li>
|
||||
<li>failed to set current config checkpoint to object ${API_PATH} with UID ${UID}</li>
|
||||
</ul>
|
||||
</td>
|
||||
<td><p>False</p></td>
|
||||
<td><p>failed to sync: internal failure, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet encountered some internal problem and failed to update its config as a result. Examples include filesystem errors and reading objects from the internal informer cache.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>internal failure, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet encountered some internal problem while manipulating config, outside of the configuration sync loop.</p></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ content_template: templates/concept
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state state="alpha" >}}
|
||||
{{< feature-state state="beta" >}}
|
||||
|
||||
Kubernetes v1.6 introduced a new binary called `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops. These cloud-specific control loops were originally in the `kube-controller-manager`. Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud vendors to evolve independently from the core Kubernetes code.
|
||||
|
||||
@@ -37,7 +37,8 @@ Successfully running cloud-controller-manager requires some changes to your clus
|
||||
|
||||
* `kube-apiserver` and `kube-controller-manager` MUST NOT specify the `--cloud-provider` flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.
|
||||
* `kubelet` must run with `--cloud-provider=external`. This is to ensure that the kubelet is aware that it must be initialized by the cloud controller manager before it is scheduled any work.
|
||||
* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller since the cloud controller manager takes over labeling persistent volumes. To prevent the PersistentVolumeLabel admission plugin from running in `kube-apiserver`, include the `PersistentVolumeLabel` as a listed value in the `--disable-admission-plugins` flag.
|
||||
* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller
|
||||
since the cloud controller manager takes over labeling persistent volumes.
|
||||
* For the `cloud-controller-manager` to label persistent volumes, initializers will need to be enabled and an InitializerConifguration needs to be added to the system. Follow [these instructions](/docs/admin/extensible-admission-controllers.md#enable-initializers-alpha-feature) to enable initializers. Use the following YAML to create the InitializerConfiguration:
|
||||
|
||||
{{< code file="persistent-volume-label-initializer-config.yaml" >}}
|
||||
@@ -53,7 +54,7 @@ As of v1.8, cloud controller manager can implement:
|
||||
* node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
|
||||
* service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
|
||||
* route controller - responsible for setting up network routes on your cloud
|
||||
* [PersistentVolumeLabel Admission Controller](/docs/admin/admission-controllers#persistentvolumelabel) - responsible for labeling persistent volumes on your cloud - ensure that the persistent volume label admission plugin is not enabled on your kube-apiserver.
|
||||
* persistent valume labels controller - responsible for setting the zone and region labels on PersistentVolumes created in GCP and AWS clouds.
|
||||
* any other features you would like to implement if you are running an out-of-tree provider.
|
||||
|
||||
|
||||
|
||||
@@ -7,15 +7,16 @@ content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state for_k8s_version="v1.10" state="beta" >}}
|
||||
|
||||
Persistent volume claims (PVCs) that are in active use by a pod and persistent volumes (PVs) that are bound to PVCs can be protected from pre-mature removal.
|
||||
Persistent volume claims (PVCs) that are in active use by a pod and persistent volumes (PVs) that are bound to PVCs can be protected from premature removal.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- The Storage Object in Use Protection feature is enabled in a version of Kubernetes in which it is supported.
|
||||
The Storage Object in Use Protection feature is enabled in one of the below Kubernetes versions:
|
||||
- {% assign for_k8s_version = "1.10" %} {% include feature-state-beta.md %}
|
||||
- {% assign for_k8s_version = "1.11" %} {% include feature-state-stable.md %}
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -312,5 +313,3 @@ No resources found.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
---
|
||||
title: Using Sysctls in a Kubernetes Cluster
|
||||
title: Using sysctls in a Kubernetes Cluster
|
||||
reviewers:
|
||||
- sttts
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state for_k8s_version="v1.11" state="beta" >}}
|
||||
|
||||
This document describes how sysctls are used within a Kubernetes cluster.
|
||||
This document describes how to configure and use kernel parameters within a
|
||||
Kubernetes cluster using the sysctl interface.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -74,7 +76,7 @@ application tuning. _Unsafe_ sysctls are enabled on a node-by-node basis with a
|
||||
flag of the kubelet, e.g.:
|
||||
|
||||
```shell
|
||||
$ kubelet --experimental-allowed-unsafe-sysctls \
|
||||
$ kubelet --allowed-unsafe-sysctls \
|
||||
'kernel.msg*,net.ipv4.route.min_pmtu' ...
|
||||
```
|
||||
|
||||
@@ -89,10 +91,11 @@ Only _namespaced_ sysctls can be enabled this way.
|
||||
## Setting Sysctls for a Pod
|
||||
|
||||
A number of sysctls are _namespaced_ in today's Linux kernels. This means that
|
||||
they can be set independently for each pod on a node. Being namespaced is a
|
||||
requirement for sysctls to be accessible in a pod context within Kubernetes.
|
||||
they can be set independently for each pod on a node. Only namespaced sysctls
|
||||
are configurable via the pod securityContext within Kubernetes.
|
||||
|
||||
The following sysctls are known to be _namespaced_:
|
||||
The following sysctls are known to be namespaced. This list could change
|
||||
in future versions of the Linux kernel.
|
||||
|
||||
- `kernel.shm*`,
|
||||
- `kernel.msg*`,
|
||||
@@ -100,25 +103,37 @@ The following sysctls are known to be _namespaced_:
|
||||
- `fs.mqueue.*`,
|
||||
- `net.*`.
|
||||
|
||||
Sysctls which are not namespaced are called _node-level_ and must be set
|
||||
manually by the cluster admin, either by means of the underlying Linux
|
||||
distribution of the nodes (e.g. via `/etc/sysctls.conf`) or using a DaemonSet
|
||||
with privileged containers.
|
||||
Sysctls with no namespace are called _node-level_ sysctls. If you need to set
|
||||
them, you must manually configure them on each node's operating system, or by
|
||||
using a DaemonSet with privileged containers.
|
||||
|
||||
The sysctl feature is an alpha API. Therefore, sysctls are set using annotations
|
||||
on pods. They apply to all containers in the same pod.
|
||||
Use the pod securityContext to configure namespaced sysctls. The securityContext
|
||||
applies to all containers in the same pod.
|
||||
|
||||
Here is an example, with different annotations for _safe_ and _unsafe_ sysctls:
|
||||
This example uses the pod securityContext to set a safe sysctl
|
||||
`kernel.shm_rmid_forced` and two unsafe sysctls `net.ipv4.route.min_pmtu` and
|
||||
`kernel.msgmax` There is no distinction between _safe_ and _unsafe_ sysctls in
|
||||
the specification.
|
||||
|
||||
{{< warning >}}
|
||||
Only modify sysctl parameters after you understand their effects, to avoid
|
||||
destabilizing your operating system.
|
||||
{{< /warning >}}
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: sysctl-example
|
||||
annotations:
|
||||
security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1
|
||||
security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.route.min_pmtu=1000,kernel.msgmax=1 2 3
|
||||
spec:
|
||||
securityContext:
|
||||
sysctls:
|
||||
- name: kernel.shm_rmid_forced
|
||||
value: "0"
|
||||
- name: net.ipv4.route.min_pmtu
|
||||
value: "552"
|
||||
- name: kernel.msgmax
|
||||
value: "65536"
|
||||
...
|
||||
```
|
||||
{{% /capture %}}
|
||||
@@ -143,27 +158,52 @@ is recommended to use
|
||||
[taints on nodes](/docs/concepts/configuration/taint-and-toleration/)
|
||||
to schedule those pods onto the right nodes.
|
||||
|
||||
## PodSecurityPolicy Annotations
|
||||
## PodSecurityPolicy
|
||||
|
||||
The use of sysctl in pods can be controlled via annotation on the PodSecurityPolicy.
|
||||
You can further control which sysctls can be set in pods by specifying lists of
|
||||
sysctls or sysctl patterns in the `forbiddenSysctls` and/or
|
||||
`allowedUnsafeSysctls` fields of the PodSecurityPolicy. A sysctl pattern ends
|
||||
with a `*` character, such as `kernel.*`. A `*` character on its own matches
|
||||
all sysctls.
|
||||
|
||||
Sysctl annotation represents a whitelist of allowed safe and unsafe sysctls
|
||||
in a pod spec. It's a comma-separated list of plain sysctl names or sysctl patterns
|
||||
(which end in `*`). The string `*` matches all sysctls.
|
||||
By default, all safe sysctls are allowed.
|
||||
|
||||
Here is an example, it authorizes binding user creating pod with corresponding sysctls.
|
||||
Both `forbiddenSysctls` and `allowedUnsafeSysctls` are lists of plain sysctl names
|
||||
or sysctl patterns (which end with `*`). The string `*` matches all sysctls.
|
||||
|
||||
The `forbiddenSysctls` field excludes specific sysctls. You can forbid a
|
||||
combination of safe and unsafe sysctls in the list. To forbid setting any
|
||||
sysctls, use `*` on its own.
|
||||
|
||||
If you specify any unsafe sysctl in the `allowedUnsafeSysctls` field and it is
|
||||
not present in the `forbiddenSysctls` field, that sysctl can be used in Pods
|
||||
using this PodSecurityPolicy. To allow all unsafe sysctls in the
|
||||
PodSecurityPolicy to be set, use `*` on its own.
|
||||
|
||||
Do not configure these two fields such that there is overlap, meaning that a
|
||||
given sysctl is both allowed and forbidden.
|
||||
|
||||
{{< warning >}}
|
||||
**Warning**: If you whitelist unsafe sysctls via the `allowedUnsafeSysctls` field
|
||||
in a PodSecurityPolicy, any pod using such a sysctl will fail to start
|
||||
if the sysctl is not whitelisted via the `--allowed-unsafe-sysctls` kubelet
|
||||
flag as well on that node.
|
||||
{{< /warning >}}
|
||||
|
||||
This example allows unsafe sysctls prefixed with `kernel.msg` to be set and
|
||||
disallows setting of the `kernel.shm_rmid_forced` sysctl.
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: sysctl-psp
|
||||
annotations:
|
||||
security.alpha.kubernetes.io/sysctls: 'net.ipv4.route.*,kernel.msg*'
|
||||
spec:
|
||||
allowedUnsafeSysctls:
|
||||
- kernel.msg*
|
||||
forbiddenSysctls:
|
||||
- kernel.shm_rmid_forced
|
||||
...
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -20,28 +20,32 @@ but is not allowed to use more CPU than its limit.
|
||||
|
||||
Each node in your cluster must have at least 1 cpu.
|
||||
|
||||
A few of the steps on this page require that the
|
||||
[Heapster](https://github.com/kubernetes/heapster) service is running
|
||||
in your cluster. But if you don't have Heapster running, you can do most
|
||||
of the steps, and it won't be a problem if you skip the Heapster steps.
|
||||
A few of the steps on this page require you to run the
|
||||
[metrics-server](https://github.com/kubernetes-incubator/metrics-server)
|
||||
service in your cluster. If you don't have metrics-server
|
||||
running, you can skip those steps.
|
||||
|
||||
If you are running minikube, run the following command to enable heapster:
|
||||
If you are running minikube, run the following command to enable
|
||||
metrics-server:
|
||||
|
||||
```shell
|
||||
minikube addons enable heapster
|
||||
minikube addons enable metrics-server
|
||||
```
|
||||
|
||||
To see whether the Heapster service is running, enter this command:
|
||||
To see whether metrics-server (or another provider of the resource metrics
|
||||
API, `metrics.k8s.io`) is running, enter this command:
|
||||
|
||||
```shell
|
||||
kubectl get services --namespace=kube-system
|
||||
kubectl get apiservices
|
||||
```
|
||||
|
||||
If the heapster service is running, it shows in the output:
|
||||
If the resource metrics API is available, the output will include a
|
||||
reference to `metrics.k8s.io`.
|
||||
|
||||
|
||||
```shell
|
||||
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
kube-system heapster 10.11.240.9 <none> 80/TCP 6d
|
||||
NAME
|
||||
v1beta1.metrics.k8s.io
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -101,26 +105,18 @@ resources:
|
||||
cpu: 500m
|
||||
```
|
||||
|
||||
Start a proxy so that you can call the heapster service:
|
||||
Use `kubectl top` to fetch the metrics for the pod:
|
||||
|
||||
```shell
|
||||
kubectl proxy
|
||||
```
|
||||
|
||||
In another command window, get the CPU usage rate from the heapster service:
|
||||
|
||||
```
|
||||
curl http://localhost:8001/api/v1/namespaces/kube-system/services/heapster/proxy/api/v1/model/namespaces/cpu-example/pods/cpu-demo/metrics/cpu/usage_rate
|
||||
kubectl top pod memory-demo
|
||||
```
|
||||
|
||||
The output shows that the Pod is using 974 millicpu, which is just a bit less than
|
||||
the limit of 1 cpu specified in the Pod's configuration file.
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2017-06-22T18:48:00Z",
|
||||
"value": 974
|
||||
}
|
||||
```
|
||||
NAME CPU(cores) MEMORY(bytes)
|
||||
memory-demo 794m <something>
|
||||
```
|
||||
|
||||
Recall that by setting `-cpu "2"`, you configured the Container to attempt to use 2 cpus.
|
||||
|
||||
@@ -19,28 +19,33 @@ but is not allowed to use more memory than its limit.
|
||||
|
||||
Each node in your cluster must have at least 300 MiB of memory.
|
||||
|
||||
A few of the steps on this page require that the
|
||||
[Heapster](https://github.com/kubernetes/heapster) service is running
|
||||
in your cluster. But if you don't have Heapster running, you can do most
|
||||
of the steps, and it won't be a problem if you skip the Heapster steps.
|
||||
A few of the steps on this page require you to run the
|
||||
[metrics-server](https://github.com/kubernetes-incubator/metrics-server)
|
||||
service in your cluster. If you don't have metrics-server
|
||||
+running, you can skip those steps.
|
||||
|
||||
If you are running minikube, run the following command to enable heapster:
|
||||
If you are running minikube, run the following command to enable
|
||||
metrics-server:
|
||||
|
||||
```shell
|
||||
minikube addons enable heapster
|
||||
minikube addons enable metrics-server
|
||||
```
|
||||
|
||||
To see whether the Heapster service is running, enter this command:
|
||||
To see whether metrics-server (or another provider of the resource metrics
|
||||
API, `metrics.k8s.io`) is running, enter this command:
|
||||
|
||||
```shell
|
||||
kubectl get services --namespace=kube-system
|
||||
kubectl get apiservices
|
||||
```
|
||||
|
||||
If the Heapster service is running, it shows in the output:
|
||||
If the resource metrics API is available, the output will include a
|
||||
reference to `metrics.k8s.io`.
|
||||
|
||||
|
||||
|
||||
```shell
|
||||
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
kube-system heapster 10.11.240.9 <none> 80/TCP 6d
|
||||
NAME
|
||||
v1beta1.metrics.k8s.io
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -103,29 +108,20 @@ resources:
|
||||
...
|
||||
```
|
||||
|
||||
Start a proxy so that you can call the Heapster service:
|
||||
Use `kubectl top` to fetch the metrics for the pod:
|
||||
|
||||
```shell
|
||||
kubectl proxy
|
||||
```
|
||||
|
||||
In another command window, get the memory usage from the Heapster service:
|
||||
|
||||
```
|
||||
curl http://localhost:8001/api/v1/namespaces/kube-system/services/heapster/proxy/api/v1/model/namespaces/mem-example/pods/memory-demo/metrics/memory/usage
|
||||
kubectl top pod memory-demo
|
||||
```
|
||||
|
||||
The output shows that the Pod is using about 162,900,000 bytes of memory, which
|
||||
is about 150 MiB. This is greater than the Pod's 100 MiB request, but within the
|
||||
Pod's 200 MiB limit.
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2017-06-20T18:54:00Z",
|
||||
"value": 162856960
|
||||
}
|
||||
```
|
||||
|
||||
NAME CPU(cores) MEMORY(bytes)
|
||||
memory-demo <something> 162856960
|
||||
```
|
||||
|
||||
Delete your Pod:
|
||||
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
---
|
||||
reviewers:
|
||||
- Random-Liu
|
||||
- feiskyer
|
||||
- mrunalp
|
||||
title: Debugging Kubernetes nodes with crictl
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.11" state="stable" >}}
|
||||
|
||||
`crictl` is a command-line interface for CRI-compatible container runtimes.
|
||||
You can use it to inspect and debug container runtimes and applications on a
|
||||
Kubernetes node. `crictl` and its source are hosted in the
|
||||
[cri-tools](https://github.com/kubernetes-incubator/cri-tools) repository.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
`crictl` requires a Linux operating system with a CRI runtime.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Installing crictl
|
||||
|
||||
You can download a compressed archive `crictl` from the cri-tools [release
|
||||
page](https://github.com/kubernetes-incubator/cri-tools/releases), for several
|
||||
different architectures. Download the version that corresponds to your version
|
||||
of Kubernetes. Extract it and move it to a location on your system path, such as
|
||||
`/usr/local/bin/`.
|
||||
|
||||
## General usage
|
||||
|
||||
The `crictl` command has several subcommands and runtime flags. Use
|
||||
`crictl help` or `crictl <subcommand> help` for more details.
|
||||
|
||||
`crictl` connects to `unix:///var/run/dockershim.sock` by default. For other
|
||||
runtimes, you can set the endpoint in multiple different ways:
|
||||
|
||||
- By setting flags `--runtime-endpoint` and `--image-endpoint`
|
||||
- By setting environment variables `CONTAINER_RUNTIME_ENDPOINT` and `IMAGE_SERVICE_ENDPOINT`
|
||||
- By setting the endpoint in the config file `--config=/etc/crictl.yaml`
|
||||
|
||||
You can also specify timeout values when connecting to the server and enable or
|
||||
disable debugging, by specifying `timeout` or `debug` values in the configuration
|
||||
file or using the `--timeout` and `--debug` command-line flags.
|
||||
|
||||
To view or edit the current configuration, view or edit the contents of
|
||||
`/etc/crictl.yaml`.
|
||||
|
||||
```sh
|
||||
$ cat /etc/crictl.yaml
|
||||
runtime-endpoint: unix:///var/run/dockershim.sock
|
||||
image-endpoint: unix:///var/run/dockershim.sock
|
||||
timeout: 10
|
||||
debug: true
|
||||
```
|
||||
|
||||
## Example crictl commands
|
||||
|
||||
The following examples show some `crictl` commands and and example output.
|
||||
|
||||
{{< warning >}}
|
||||
If you use `crictl` to create pod sandboxes or containers on a running
|
||||
Kubernetes cluster, the Kubelet will eventually delete them. `crictl` is not a
|
||||
general purpose workflow tool, but a tool that is useful for debugging.
|
||||
{{< /warning >}}
|
||||
|
||||
### List pods
|
||||
|
||||
List all pods:
|
||||
|
||||
```bash
|
||||
crictl pods
|
||||
```
|
||||
```none
|
||||
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
|
||||
926f1b5a1d33a About a minute ago Ready sh-84d7dcf559-4r2gq default 0
|
||||
4dccb216c4adb About a minute ago Ready nginx-65899c769f-wv2gp default 0
|
||||
a86316e96fa89 17 hours ago Ready kube-proxy-gblk4 kube-system 0
|
||||
919630b8f81f1 17 hours ago Ready nvidia-device-plugin-zgbbv kube-system 0
|
||||
```
|
||||
|
||||
List pods by name:
|
||||
|
||||
```bash
|
||||
crictl pods --name nginx-65899c769f-wv2gp
|
||||
```
|
||||
```none
|
||||
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
|
||||
4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0
|
||||
```
|
||||
|
||||
List pods by label:
|
||||
|
||||
```bash
|
||||
crictl pods --label run=nginx
|
||||
```
|
||||
```none
|
||||
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
|
||||
4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0
|
||||
```
|
||||
|
||||
### List images
|
||||
|
||||
List all images:
|
||||
|
||||
```bash
|
||||
crictl images
|
||||
```
|
||||
```none
|
||||
IMAGE TAG IMAGE ID SIZE
|
||||
busybox latest 8c811b4aec35f 1.15MB
|
||||
k8s-gcrio.azureedge.net/hyperkube-amd64 v1.10.3 e179bbfe5d238 665MB
|
||||
k8s-gcrio.azureedge.net/pause-amd64 3.1 da86e6ba6ca19 742kB
|
||||
nginx latest cd5239a0906a6 109MB
|
||||
```
|
||||
|
||||
List images by repository:
|
||||
|
||||
```bash
|
||||
crictl images nginx
|
||||
```
|
||||
```none
|
||||
IMAGE TAG IMAGE ID SIZE
|
||||
nginx latest cd5239a0906a6 109MB
|
||||
```
|
||||
|
||||
Only list image IDs:
|
||||
|
||||
```bash
|
||||
crictl images -q
|
||||
```
|
||||
```none
|
||||
sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a
|
||||
sha256:e179bbfe5d238de6069f3b03fccbecc3fb4f2019af741bfff1233c4d7b2970c5
|
||||
sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e
|
||||
sha256:cd5239a0906a6ccf0562354852fae04bc5b52d72a2aff9a871ddb6bd57553569
|
||||
```
|
||||
|
||||
### List containers
|
||||
|
||||
List all containers:
|
||||
|
||||
```bash
|
||||
crictl ps -a
|
||||
```
|
||||
```none
|
||||
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
|
||||
1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 7 minutes ago Running sh 1
|
||||
9c5951df22c78 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 8 minutes ago Exited sh 0
|
||||
87d3992f84f74 nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f 8 minutes ago Running nginx 0
|
||||
1941fb4da154f k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab5be80c58e98140dfc69df107f253d7fdd714b30a714260a 18 hours ago Running kube-proxy 0
|
||||
```
|
||||
|
||||
List running containers:
|
||||
|
||||
```bash
|
||||
crictl ps
|
||||
```
|
||||
```none
|
||||
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
|
||||
1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 6 minutes ago Running sh 1
|
||||
87d3992f84f74 nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f 7 minutes ago Running nginx 0
|
||||
1941fb4da154f k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab5be80c58e98140dfc69df107f253d7fdd714b30a714260a 17 hours ago Running kube-proxy 0
|
||||
```
|
||||
|
||||
### Execute a command in a running container
|
||||
|
||||
```bash
|
||||
crictl exec -i -t 1f73f2d81bf98 ls
|
||||
```
|
||||
```none
|
||||
bin dev etc home proc root sys tmp usr var
|
||||
```
|
||||
|
||||
### Get a coontainer's logs
|
||||
|
||||
Get all container logs:
|
||||
|
||||
```bash
|
||||
crictl logs 87d3992f84f74
|
||||
```
|
||||
```none
|
||||
10.240.0.96 - - [06/Jun/2018:02:45:49 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
|
||||
10.240.0.96 - - [06/Jun/2018:02:45:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
|
||||
10.240.0.96 - - [06/Jun/2018:02:45:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
|
||||
```
|
||||
|
||||
Get only the latest `N` lines of logs:
|
||||
|
||||
```bash
|
||||
crictl logs --tail=1 87d3992f84f74
|
||||
```
|
||||
```none
|
||||
10.240.0.96 - - [06/Jun/2018:02:45:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
|
||||
```
|
||||
|
||||
### Run a pod sandbox
|
||||
|
||||
Using `crictl` to run a pod sandbox is useful for debugging container runtimes.
|
||||
On a running Kubernetes cluster, the sandbox will eventually be stopped and
|
||||
deleted by the Kubelet.
|
||||
|
||||
1. Create a JSON file like the following:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"name": "nginx-sandbox",
|
||||
"namespace": "default",
|
||||
"attempt": 1,
|
||||
"uid": "hdishd83djaidwnduwk28bcsb"
|
||||
},
|
||||
"logDirectory": "/tmp",
|
||||
"linux": {
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Use the `crictl runp` command to apply the JSON and run the sandbox.
|
||||
|
||||
```bash
|
||||
crictl runp pod-config.json
|
||||
```
|
||||
|
||||
The ID of the sandbox is returned.
|
||||
|
||||
### Create a container
|
||||
|
||||
Using `crictl` to create a container is useful for debugging container runtimes.
|
||||
On a running Kubernetes cluster, the sandbox will eventually be stopped and
|
||||
deleted by the Kubelet.
|
||||
|
||||
1. Pull a busybox image
|
||||
|
||||
```bash
|
||||
crictl pull busybox
|
||||
Image is up to date for busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
|
||||
```
|
||||
|
||||
2. Create configs for the pod and the container:
|
||||
|
||||
**Pod config**:
|
||||
```yaml
|
||||
{
|
||||
"metadata": {
|
||||
"name": "nginx-sandbox",
|
||||
"namespace": "default",
|
||||
"attempt": 1,
|
||||
"uid": "hdishd83djaidwnduwk28bcsb"
|
||||
},
|
||||
"log_directory": "/tmp",
|
||||
"linux": {
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Container config**:
|
||||
```yaml
|
||||
{
|
||||
"metadata": {
|
||||
"name": "busybox"
|
||||
},
|
||||
"image":{
|
||||
"image": "busybox"
|
||||
},
|
||||
"command": [
|
||||
"top"
|
||||
],
|
||||
"log_path":"busybox/0.log",
|
||||
"linux": {
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. Create the container, passing the ID of the previously-created pod, the
|
||||
container config file, and the pod config file. The ID of the container is
|
||||
returned.
|
||||
|
||||
```bash
|
||||
crictl create f84dd361f8dc51518ed291fbadd6db537b0496536c1d2d6c05ff943ce8c9a54f container-config.json pod-config.json
|
||||
```
|
||||
|
||||
4. List all containers and verify that the newly-created container has its
|
||||
state set to `Created`.
|
||||
|
||||
```bash
|
||||
crictl ps -a
|
||||
```
|
||||
```none
|
||||
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
|
||||
3e025dd50a72d busybox 32 seconds ago Created busybox 0
|
||||
```
|
||||
|
||||
### Start a container
|
||||
|
||||
To start a container, pass its ID to `crictl start`:
|
||||
|
||||
```bash
|
||||
crictl start 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60
|
||||
```
|
||||
```none
|
||||
3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60
|
||||
|
||||
$ crictl ps
|
||||
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
|
||||
3e025dd50a72d busybox About a minute ago Running busybox 0
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{% capture discussion %}}
|
||||
|
||||
See [kubernetes-incubator/cri-tools](https://github.com/kubernetes-incubator/cri-tools)
|
||||
for more information.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -363,6 +363,8 @@ status:
|
||||
systemUUID: ABE5F6B4-D44B-108B-C46A-24CCE16C8B6E
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
Learn about additional debugging tools, including:
|
||||
@@ -372,5 +374,6 @@ Learn about additional debugging tools, including:
|
||||
* [Getting into containers via `exec`](/docs/tasks/debug-application-cluster/get-shell-running-container/)
|
||||
* [Connecting to containers via proxies](/docs/tasks/access-kubernetes-api/http-proxy-access-api/)
|
||||
* [Connecting to containers via port forwarding](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
|
||||
* [Inspect Kubernetes node with crictl](/docs/tasks/debug-application-cluster/crictl/)
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -7,7 +7,14 @@ title: Tools for Monitoring Compute, Storage, and Network Resources
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
Understanding how an application behaves when deployed is crucial to scaling the application and providing a reliable service. In a Kubernetes cluster, application performance can be examined at many different levels: containers, [pods](/docs/user-guide/pods), [services](/docs/user-guide/services), and whole clusters. As part of Kubernetes we want to provide users with detailed resource usage information about their running applications at all these levels. This will give users deep insights into how their applications are performing and where possible application bottlenecks may be found. In comes [Heapster](https://github.com/kubernetes/heapster), a project meant to provide a base monitoring platform on Kubernetes.
|
||||
To scale and application and provide a reliable service, you need to
|
||||
understand how an application behaves when it is deployed. You can examine
|
||||
application performance in a Kubernetes cluster by examining the containers,
|
||||
[pods](/docs/user-guide/pods), [services](/docs/user-guide/services), and
|
||||
the characteristics of the overall cluster. Kubernetes provides detailed
|
||||
information about an application's resource usage at each of these levels.
|
||||
This information allows you to evaluate your application's performance and
|
||||
where bottlenecks can be removed to improve overall performance.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -15,11 +22,31 @@ Understanding how an application behaves when deployed is crucial to scaling the
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
Heapster is a cluster-wide aggregator of monitoring and event data. It currently supports Kubernetes natively and works on all Kubernetes setups. Heapster runs as a pod in the cluster, similar to how any Kubernetes application would run. The Heapster pod discovers all nodes in the cluster and queries usage information from the nodes' [Kubelet](/docs/admin/kubelet/)s, the on-machine Kubernetes agent. The Kubelet itself fetches the data from [cAdvisor](https://github.com/google/cadvisor). Heapster groups the information by pod along with the relevant labels. This data is then pushed to a configurable backend for storage and visualization. Currently supported backends include [InfluxDB](http://influxdb.com/) (with [Grafana](http://grafana.org/) for visualization), [Google Cloud Monitoring](https://cloud.google.com/monitoring/) and many others described in more details [here](https://git.k8s.io/heapster/docs/sink-configuration.md). The overall architecture of the service can be seen below:
|
||||
In Kubernetes, application monitoring does not depend on a single monitoring
|
||||
solution. On new clusters, you can use two separate pipelines to collect
|
||||
monitoring statistics by default:
|
||||
|
||||
|
||||
- The **resource metrics pipeline** provides a limited set of metrics related
|
||||
to cluster components such as the HorizontalPodAutoscaler controller, as well
|
||||
as the `kubectl top` utility. These metrics are collected by
|
||||
[metrics-server](https://github.com/kubernetes-incubator/metrics-server)
|
||||
and are exposed via the `metrics.k8s.io` API. `metrics-server` discovers
|
||||
all nodes on the cluster and queries each node's [Kubelet](/docs/admin/kubelet)
|
||||
for CPU and memory usage. The Kubelet fetches the data from
|
||||
[cAdvisor](https://github.com/google/cadvisor). `metrics-server` is a
|
||||
lightweight short-term in-memory store.
|
||||
|
||||
- A **full monitoring pipeline**, such as Prometheus, gives you access to richer
|
||||
metrics. In addition, Kubernetes can respond to these metrics by automatically
|
||||
scaling or adapting the cluster based on its current state, using mechanisms
|
||||
such as the Horizontal Pod Autoscaler. The monitoring pipeline fetches
|
||||
metrics from the Kubelet, and then exposes them to Kubernetes via an adapter
|
||||
by implementing either the `custom.metrics.k8s.io` or
|
||||
`external.metrics.k8s.io` API. See
|
||||
[Full metrics pipeline](#full-metrics-pipelines) for more information about
|
||||
some popular pipelines that implement these APIs and enable these
|
||||
capabilities.
|
||||
|
||||
Let's look at some of the other components in more detail.
|
||||
|
||||
### cAdvisor
|
||||
|
||||
@@ -33,38 +60,36 @@ On most Kubernetes clusters, cAdvisor exposes a simple UI for on-machine contain
|
||||
|
||||
The Kubelet acts as a bridge between the Kubernetes master and the nodes. It manages the pods and containers running on a machine. Kubelet translates each pod into its constituent containers and fetches individual container usage statistics from cAdvisor. It then exposes the aggregated pod resource usage statistics via a REST API.
|
||||
|
||||
## Storage Backends
|
||||
## Full Metrics Pipelines
|
||||
|
||||
### InfluxDB and Grafana
|
||||
Many full metrics solutions exist for Kubernetes. Prometheus and Google Cloud
|
||||
Monitoring are two of the most popular.
|
||||
|
||||
A Grafana setup with InfluxDB is a very popular combination for monitoring in the open source world. InfluxDB exposes an easy to use API to write and fetch time series data. Heapster is setup to use this storage backend by default on most Kubernetes clusters. A detailed setup guide can be found [here](https://github.com/GoogleCloudPlatform/heapster/blob/master/docs/influxdb.md). InfluxDB and Grafana run in Pods. The pod exposes itself as a Kubernetes service which is how Heapster discovers it.
|
||||
### Prometheus
|
||||
|
||||
The Grafana container serves Grafana's UI which provides an easy to configure dashboard interface. The default dashboard for Kubernetes contains an example dashboard that monitors resource usage of the cluster and the pods inside of it. This dashboard can easily be customized and expanded. Take a look at the storage schema for InfluxDB [here](https://github.com/GoogleCloudPlatform/heapster/blob/master/docs/storage-schema.md#metrics).
|
||||
|
||||
Here is a video showing how to monitor a Kubernetes cluster using heapster, InfluxDB and Grafana:
|
||||
|
||||
[](http://www.youtube.com/watch?v=SZgqjMrxo3g)
|
||||
|
||||
Here is a snapshot of the default Kubernetes Grafana dashboard that shows the CPU and Memory usage of the entire cluster, individual pods and containers:
|
||||
|
||||
|
||||
[Prometheus](https://prometheus.io) natively monitors Prometheus.
|
||||
The [Prometheus Operator](https://coreos.com/operators/prometheus/docs/latest/)
|
||||
simplifies Prometheus setup on Kubernetes, and allows you to serve the
|
||||
custom metrics API using the
|
||||
[Prometheus adapter](https://github.com/directxman12/k8s-prometheus-adapter).
|
||||
Prometheus provides a robust query language and a built-in dashboard for
|
||||
querying and visualizing your data. Prometheus is also a supported
|
||||
data source for [Grafana](https://prometheus.io/docs/visualization/grafana/).
|
||||
|
||||
### Google Cloud Monitoring
|
||||
|
||||
Google Cloud Monitoring is a hosted monitoring service that allows you to visualize and alert on important metrics in your application. Heapster can be setup to automatically push all collected metrics to Google Cloud Monitoring. These metrics are then available in the [Cloud Monitoring Console](https://app.google.stackdriver.com/). This storage backend is the easiest to setup and maintain. The monitoring console allows you to easily create and customize dashboards using the exported data.
|
||||
Google Cloud Monitoring is a hosted monitoring service you can use to
|
||||
visualize and alert on important metrics in your application. can collect
|
||||
metrics from Kubernetes, and you can access them
|
||||
using the [Cloud Monitoring Console](https://app.google.stackdriver.com/).
|
||||
You can create and customize dashboards to visualize the data gathered
|
||||
from your Kubernetes cluster.
|
||||
|
||||
Here is a video showing how to setup and run a Google Cloud Monitoring backed Heapster:
|
||||
This video shows how to configure and run a Google Cloud Monitoring backed Heapster:
|
||||
|
||||
[](http://www.youtube.com/watch?v=xSMNR2fcoLs)
|
||||
|
||||
Here is a snapshot of the Google Cloud Monitoring dashboard showing cluster-wide resource usage.
|
||||
|
||||
|
||||
{{< figure src="/images/docs/gcm.png" alt="Google Cloud Monitoring dashboard example" title="Google Cloud Monitoring dashboard example" caption="This dashboard shows cluster-wide resource usage." >}}
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
Now that you've learned a bit about Heapster, feel free to try it out on your own clusters! The [Heapster repository](https://github.com/kubernetes/heapster) is available on GitHub. It contains detailed instructions to setup Heapster and its storage backends. Heapster runs by default on most Kubernetes clusters, so you may already have it! Feedback is always welcome. Please let us know if you run into any issues via the troubleshooting [channels](/docs/troubleshooting/).
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -152,70 +152,3 @@ spec:
|
||||
|
||||
This will ensure that the pod will be scheduled to a node that has the GPU type
|
||||
you specified.
|
||||
|
||||
## v1.6 and v1.7
|
||||
To enable GPU support in 1.6 and 1.7, a special **alpha** feature gate
|
||||
`Accelerators` has to be set to true across the system:
|
||||
`--feature-gates="Accelerators=true"`. It also requires using the Docker
|
||||
Engine as the container runtime.
|
||||
|
||||
Further, the Kubernetes nodes have to be pre-installed with NVIDIA drivers.
|
||||
Kubelet will not detect NVIDIA GPUs otherwise.
|
||||
|
||||
When you start Kubernetes components after all the above conditions are true,
|
||||
Kubernetes will expose `alpha.kubernetes.io/nvidia-gpu` as a schedulable
|
||||
resource.
|
||||
|
||||
You can consume these GPUs from your containers by requesting
|
||||
`alpha.kubernetes.io/nvidia-gpu` just like you request `cpu` or `memory`.
|
||||
However, there are some limitations in how you specify the resource requirements
|
||||
when using GPUs:
|
||||
- GPUs are only supposed to be specified in the `limits` section, which means:
|
||||
* You can specify GPU `limits` without specifying `requests` because
|
||||
Kubernetes will use the limit as the request value by default.
|
||||
* You can specify GPU in both `limits` and `requests` but these two values
|
||||
must be equal.
|
||||
* You cannot specify GPU `requests` without specifying `limits`.
|
||||
- Containers (and pods) do not share GPUs. There's no overcommitting of GPUs.
|
||||
- Each container can request one or more GPUs. It is not possible to request a
|
||||
fraction of a GPU.
|
||||
|
||||
When using `alpha.kubernetes.io/nvidia-gpu` as the resource, you also have to
|
||||
mount host directories containing NVIDIA libraries (libcuda.so, libnvidia.so
|
||||
etc.) to the container.
|
||||
|
||||
Here's an example:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: cuda-vector-add
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: cuda-vector-add
|
||||
# https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
|
||||
image: "k8s.gcr.io/cuda-vector-add:v0.1"
|
||||
resources:
|
||||
limits:
|
||||
alpha.kubernetes.io/nvidia-gpu: 1 # requesting 1 GPU
|
||||
volumeMounts:
|
||||
- name: "nvidia-libraries"
|
||||
mountPath: "/usr/local/nvidia/lib64"
|
||||
volumes:
|
||||
- name: "nvidia-libraries"
|
||||
hostPath:
|
||||
path: "/usr/lib/nvidia-375"
|
||||
```
|
||||
|
||||
The `Accelerators` feature gate and `alpha.kubernetes.io/nvidia-gpu` resource
|
||||
works on 1.8 and 1.9 as well. It will be deprecated in 1.10 and removed in
|
||||
1.11.
|
||||
|
||||
## Future
|
||||
- Support for hardware accelerators in Kubernetes is still in alpha.
|
||||
- Better APIs will be introduced to provision and consume accelerators in a scalable manner.
|
||||
- Kubernetes will automatically ensure that applications consuming GPUs get the best possible performance.
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -25,10 +25,10 @@ This document walks you through an example of enabling Horizontal Pod Autoscaler
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
This example requires a running Kubernetes cluster and kubectl, version 1.2 or later.
|
||||
[Heapster](https://github.com/kubernetes/heapster) monitoring needs to be deployed in the cluster
|
||||
as Horizontal Pod Autoscaler uses it to collect metrics
|
||||
(if you followed [getting started on GCE guide](/docs/getting-started-guides/gce),
|
||||
heapster monitoring will be turned-on by default).
|
||||
[metrics-server](https://github.com/kubernetes/heapster) monitoring needs to be deployed in the cluster
|
||||
to provide metrics via the resource metrics API, as Horizontal Pod Autoscaler uses this API to collect metrics
|
||||
(if you followed [getting started on GCE guide](/docs/getting-started-guides/gce.md),
|
||||
metrics-server monitoring will be turned-on by default).
|
||||
|
||||
To specify multiple resource metrics for a Horizontal Pod Autoscaler, you must have a Kubernetes cluster
|
||||
and kubectl at version 1.6 or later. Furthermore, in order to make use of custom metrics, your cluster
|
||||
@@ -212,7 +212,7 @@ Notice that the `targetCPUUtilizationPercentage` field has been replaced with an
|
||||
The CPU utilization metric is a *resource metric*, since it is represented as a percentage of a resource
|
||||
specified on pod containers. Notice that you can specify other resource metrics besides CPU. By default,
|
||||
the only other supported resource metric is memory. These resources do not change names from cluster
|
||||
to cluster, and should always be available, as long as Heapster is deployed.
|
||||
to cluster, and should always be available, as long as the `metrics.k8s.io` API is available.
|
||||
|
||||
You can also specify resource metrics in terms of direct values, instead of as percentages of the
|
||||
requested value. To do so, use the `targetAverageValue` field instead of the `targetAverageUtilization`
|
||||
|
||||
@@ -60,19 +60,16 @@ or the custom metrics API (for all other metrics).
|
||||
* For object metrics, a single metric is fetched (which describes the object
|
||||
in question), and compared to the target value, to produce a ratio as above.
|
||||
|
||||
The HorizontalPodAutoscaler controller can fetch metrics in two different ways: direct Heapster
|
||||
access, and REST client access.
|
||||
The HorizontalPodAutoscaler normally fetches metrics a series of aggregated APIs (`metrics.k8s.io`,\
|
||||
`custom.metrics.k8s.io`, and `external.metrics.k8s.io`). It can also fetch metrics directly
|
||||
from Heapster. Fetching metrics from Heapster is deprecated as of Kubernetes 1.11.
|
||||
|
||||
When using direct Heapster access, the HorizontalPodAutoscaler queries Heapster directly
|
||||
through the API server's service proxy subresource. Heapster needs to be deployed on the
|
||||
cluster and running in the kube-system namespace.
|
||||
|
||||
See [Support for custom metrics](#support-for-custom-metrics) for more details on REST client access.
|
||||
|
||||
The autoscaler accesses corresponding replication controller, deployment or replica set by scale sub-resource.
|
||||
Scale is an interface that allows you to dynamically set the number of replicas and examine each of their current states.
|
||||
More details on scale sub-resource can be found [here](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#scale-subresource).
|
||||
See [Support for metrics APIs](#support-for-metrics-apis) for more details.
|
||||
|
||||
The autoscaler accesses corresponding scalable controllers (such as replication controllers, deployments, and replica sets)
|
||||
by using the scale sub-resource. Scale is an interface that allows you to dynamically set the number of replicas and examine
|
||||
each of their current states. More details on scale sub-resource can be found
|
||||
[here](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#scale-subresource).
|
||||
|
||||
## API Object
|
||||
|
||||
@@ -158,30 +155,27 @@ Kubernetes 1.6 adds support for making use of custom metrics in the Horizontal P
|
||||
You can add custom metrics for the Horizontal Pod Autoscaler to use in the `autoscaling/v2beta1` API.
|
||||
Kubernetes then queries the new custom metrics API to fetch the values of the appropriate custom metrics.
|
||||
|
||||
### Requirements
|
||||
See [Support for metrics APIs](#support-for-metrics-APIs) for the requirements.
|
||||
|
||||
To use custom metrics with your Horizontal Pod Autoscaler, you must set the necessary configurations when deploying your cluster:
|
||||
## Support for metrics APIs
|
||||
|
||||
* [Enable the API aggregation layer](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/) if you have not already done so.
|
||||
By default, the HorizontalPodAutoscaler controller retrieves metrics from a series of APIs. In order for it to access these
|
||||
APIs, cluster administrators must ensure that:
|
||||
|
||||
* Register your resource metrics API, your
|
||||
custom metrics API and, optionally, external metrics API with the API aggregation layer. All of these API servers must be running *on* your cluster.
|
||||
* The [API aggregation layer](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/) is enabled.
|
||||
|
||||
* *Resource Metrics API*: You can use Heapster's implementation of the resource metrics API, by running Heapster with its `--api-server` flag set to true.
|
||||
* The corresponding APIs are registered:
|
||||
|
||||
* *Custom Metrics API*: This must be provided by a separate component. To get started with boilerplate code, see the [kubernetes-incubator/custom-metrics-apiserver](https://github.com/kubernetes-incubator/custom-metrics-apiserver) and the [k8s.io/metrics](https://github.com/kubernetes/metrics) repositories.
|
||||
* For resource metrics, this is the `metrics.k8s.io` API, generally provided by [metrics-server](https://github.com/kubernetes-incubator/metrics-server).
|
||||
It can be launched as a cluster addon.
|
||||
|
||||
* *External Metrics API*: Starting from Kubernetes 1.10 you can use this API if you need to autoscale on metrics not related to any Kubernetes object. Similarly to *Custom Metrics API* this must be provided by a separate component.
|
||||
* For custom metrics, this is the `custom.metrics.k8s.io` API. It's provided by "adapter" API servers provided by metrics solution vendors.
|
||||
Check with your metrics pipeline, or the [list of known solutions](https://github.com/kubernetes/metrics/blob/master/IMPLEMENTATIONS.md#custom-metrics-api).
|
||||
If you would like to write your own, check out the [boilerplate](https://github.com/kubernetes-incubator/custom-metrics-apiserver) to get started.
|
||||
|
||||
* Set the appropriate flags for kube-controller-manager:
|
||||
* For external metrics, this is the `external.metrics.k8s.io` API. It may be provided by the custom metrics adapters provided above.
|
||||
|
||||
* `--horizontal-pod-autoscaler-use-rest-clients` should be true.
|
||||
|
||||
* `--kubeconfig <path-to-kubeconfig>` OR `--master <ip-address-of-apiserver>`
|
||||
|
||||
Note that either the `--master` or `--kubeconfig` flag can be used; `--master` will override `--kubeconfig` if both are specified. These flags specify the location of the API aggregation layer, allowing the controller manager to communicate to the API server.
|
||||
|
||||
In Kubernetes 1.7, the standard aggregation layer that Kubernetes provides runs in-process with the kube-apiserver, so the target IP address can be found with `kubectl get pods --selector k8s-app=kube-apiserver --namespace kube-system -o jsonpath='{.items[0].status.podIP}'`.
|
||||
* The `--horizontal-pod-autoscaler-use-rest-clients` is `true` or unset. Setting this to false switches to Heapster-based autoscaling, which is deprecated.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user