Release docs for Kubernetes 1.11 (#9171)
* Seperate priority and preemption (#8144) * Doc about PID pressure condition. (#8211) * Doc about PID pressure condition. Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com> * "so" -> "too" * Update version selector for 1.11 * StorageObjectInUseProtection is GA (#8291) * Feature gate: StorageObjectInUseProtection is GA Update feature gate reference for 1.11 * Trivial commit to re-trigger Netlify * CRIContainerLogRotation is Beta in 1.11 (#8665) * Seperate priority and preemption (#8144) * CRIContainerLogRotation is Beta in 1.11 xref: kubernetes/kubernetes#64046 * Bring StorageObjectInUseProtection feature to GA (#8159) * StorageObjectInUseProtection is GA (#8291) * Feature gate: StorageObjectInUseProtection is GA Update feature gate reference for 1.11 * Trivial commit to re-trigger Netlify * Bring StorageObjectInUseProtection feature to GA StorageObjectInUseProtection is Beta in K8s 1.10. It's brought to GA in K8s 1.11. * Fixed typo and added feature state tags. * Remove KUBE_API_VERSIONS doc (#8292) The support to the KUBER_API_VERSIONS environment variable is completely dropped (no deprecation). This PR removes the related doc in release-1.11. xref: kubernetes/kubernetes#63165 * Remove InitialResources from admission controllers (#8293) The feature (was experimental) is dropped in 1.11. xref: kubernetes/kubernetes#58784 * Remove docs related to in-tree support to GPU (#8294) * Remove docs related to in-tree support to GPU The in-tree support to GPU is completely removed in release 1.11. This PR removes the related docs in release-1.11 branch. xref: kubernetes/kubernetes#61498 * Update content updated by PR to Hugo syntax Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Update the doc about extra volume in kubeadm config (#8453) Signed-off-by: Xianglin Gao <xianglin.gxl@alibaba-inc.com> * Update CRD Subresources for 1.11 (#8519) * coredns: update notes in administer-cluster/coredns.md (#8697) CoreDNS is installed by default in 1.11. Add notes on how to install kube-dns instead. Update notes about CoreDNS->CoreDNS upgrades as in 1.11 the Corefile is retained. Add example on upgrading from kube-dns to CoreDNS. * kubeadm-alpha: CoreDNS related changes (#8727) Update note about CoreDNS feature gate. This change also updates a tab as a kubeadm sub-command will change. It looks for a new generated file: generated/kubeadm_alpha_phase_addon_coredns.md instead of: generated/kubeadm_alpha_phase_addon_kube-dns.md * Update cloud controller manager docs to beta 1.11 (#8756) * Update cloud controller manager docs to beta 1.11 * Use Hugo shortcode for feature state * kubeadm-upgrade: include new command `kubeadm upgrade diff` (#8617) Also: - Include note that this was added in 1.11. - Modify the note about upgrade guidance. * independent: update CoreDNS mentions for kubeadm (#8753) Give CoreDNS instead of kube-dns examples in: - docs/setup/independent/create-cluster-kubeadm.md - docs/setup/independent/troubleshooting-kubeadm.md * update 1.11 --server-print info (#8870) * update 1.11 --server-print info * Copyedit * Mark ExpandPersistentVolumes feature to beta (#8778) * Update version selector for 1.11 * Mark ExpandPersistentVolumes Beta xref: kubernetes/kubernetes#64288 * fix shortcode, add placeholder files to fix deploy failures (#8874) * declare ipvs ga (#8850) * kubeadm: update info about CoreDNS in kubeadm-init.md (#8728) Add info to install kube-dns instead of CoreDNS, as CoreDNS is the default DNS server in 1.11. Add notes that kubeadm config images can be used to list and pull the required images in 1.11. * kubeadm: update implementation-details.md about CoreDNS (#8829) - Replace examples from kube-dns to CoreDNS - Add notes about the CoreDNS feature gate status in 1.11 - Add note that the service name for CoreDNS is also called `kube-dns` * Update block device support for 1.11 (#8895) * Update block device support for 1.11 * Copyedits * Fix typo 'fiber channel' (#8957) Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * kubeadm-upgrade: add the 'node [config]' sub-command (#8960) - Add includes for the generated pages - Include placeholder generated pages * kubeadm-init: update the example for the MasterConfiguration (#8958) - include godocs link for MasterConfiguration - include example MasterConfiguration - add note that `kubeadm config print-default` can be used * kubeadm-config: include new commands (#8862) Add notes and includes for these new commands in 1.11: - kubeadm config print-default - kubeadm config migrate - kubeadm config images list - kubeadm config images pull Include placeholder generated files for the above. * administer-cluster/coredns: include more changes (#8985) It was requested that for this page a couple of methods should be outlined: - manual installation for CoreDNS explained at the Kubernetes section of the GitHub project for CoreDNS - installation and upgrade via kubeadm Make the above changes and also add a section "About CoreDNS". This commit also lowercases a section title. * Update CRD subresources doc for 1.11 (#8918) * Add docs for volume expansion and online resizing (#8896) * Add docs for volume expansion going beta * Copyedit * Address feedback * Update exec plugin docs with TLS credentials (#8826) * Update exec plugin docs with TLS credentials kubernetes/kubernetes#61803 implements TLS client credential support for 1.11. * Copyedit * More copyedits for clarification * Additional copyedit * Change token->credential * NodeRestriction admission prevents kubelet taint removal (#8911) * dns-custom-namerserver: break down the page into mutliple sections (#8900) * dns-custom-namerserver: break down the page into mutliple sections This page is currently about kube-dns and is a bit outdated. Introduce the heading `# Customizing kube-dns`. Introduce a separate section about CoreDNS. * Copyedits, fix headings for customizing DNS Hey Lubomir, I coypedited pretty heavily because this workflow is so much easier for docs and because I'm trying to help improve everything touching kubeadm as much as possible. But there's one outstanding issue wrt headings and intro content: you can't add a heading 1 to a topic to do what you wanted to do. The page title in the front matter is rendered as a heading 1 and everything else has to start at heading 2. (We still need to doc this better in the docs contributing content, I know.) Instead, I think we need to rewrite the top-of-page intro content to explain better the relationship between kube-dns and CoreDNS. I'm happy to write something, but I thought I'd push this commit first so you can see what I'm doing. Hope it's all clear -- ping here or on Slack with any questions ~ Jennifer * Interim fix for talking about CoreDNS * Fix CoreDNS details * PSP readOnly hostPath (#8898) * Add documentation for crictl (#8880) * Add documentation for crictl * Copyedit Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Final copyedit * VolumeSubpathEnvExpansion alpha feature (#8835) * Note that Heapster is deprecated (#8827) * Note that Heapster is deprecated This notes that Heapster is deprecated, and migrates the relevant docs to talk about metrics-server or other solutions by default. * Copyedits and improvements Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Address feedback * fix shortcode to troubleshoot deploy (#9057) * update dynamic kubelet config docs for v1.11 (#8766) * update dynamic kubelet config docs for v1.11 * Substantial copyedit * Address feedback * Reference doc for kubeadm (release-1.11) (#9044) * Reference doc for kubeadm (release-1.11) * fix shortcode to troubleshoot deploy (#9057) * Reference doc for kube-components (release-1.11) (#9045) * Reference doc for kube-components (release-1.11) * Update cloud-controller-manager.md * fix shortcode to troubleshoot deploy (#9057) * Documentation on lowercasing kubeadm init apiserver SANs (#9059) * Documentation on lowercasing kubeadm init apiserver SANs * fix shortcode to troubleshoot deploy (#9057) * Clarification in dynamic Kubelet config doc (#9061) * Promote sysctls to Beta (#8804) * Promote sysctls to Beta * Copyedits Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Review comments * Address feedback * More feedback * kubectl reference docs for 1.11 (#9080) * Update Kubernetes API 1.11 ref docs (#8977) * Update v1alpha1 to v1beta1. * Adjust left nav for 1.11 ref docs. * Trim list of old ref docs. * Update Federation API ref docs for 1.11. (#9064) * Update Federation API ref docs for 1.11. * Add titles. * Update definitions.html * CRD versioning Public Documentation (#8834) * CRD versioning Public Documentation * Copyedit Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Address feedback * More rewrites * Address feedback * Update main CRD page in light of versioning * Reorg CRD docs * Further reorg * Tweak title * CSI documentation update for raw block volume support (#8927) * CSI documetation update for raw block volume support * minor edits for "CSI raw block volume support" Some small grammar and style nits. * minor CSIBlockVolume edits * Update kubectl component ref page for 1.11. (#9094) * Update kubectl component ref page for 1.11. * Add title. Replace stevepe with username. * crd versioning doc: fix nits (#9142) * Update `DynamicKubeletConfig` feature to beta (#9110) xref: kubernetes/kubernetes#64275 * Documentation for dynamic volume limits based on node type (#8871) * add cos for storage limits * Update docs specific for aws and gce * fix some minor things * Update storage-limits.md * Add k8s version to feature-state shortcode * The Doc update for ScheduleDaemonSetPods (#8842) Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com> * Update docs related to PersistentVolumeLabel admission control (#9109) The said admission controller is disabled by default in 1.11 (kubernetes/kubernetes#64326) and scheduled to be removed in future release. * client exec auth: updates for 1.11 (#9154) * Updates HA kubeadm docs (#9066) * Updates HA kubeadm docs Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * kubeadm HA - Add stacked control plane steps * ssh instructions and some typos in the bash scripts Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Fix typos and copypasta errors * Fix rebase issues * Integrate more changes Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * copyedits, layout and formatting fixes * final copyedits * Adds a sanity check for load balancer connection Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * formatting fixes, copyedits * fix typos, formatting * Document the Pod Ready++ feature (#9180) Closes: #9107 Xref: kubernetes/kubernetes#64057 * Mention 'KubeletPluginsWatcher' feature (#9177) * Mention 'KubeletPluginsWatcher' feature This feature is more developers oriented than users oriented, so simply mention it in the feature gate should be fine. In future, when the design doc is migrated from Google doc to the kubernetes/community repo, we can add links to it for users who want to dig deeper. Closes: #9108 Xref: kubernetes/kubernetes#63328, kubernetes/kubernetes#64605 * Copyedit * Amend dynamic volume list docs (#9181) The dynamic volume list feature has been documented but the feature gate related was not there yet. Closes: #9105 * Document for service account projection (#9182) This adds docs for the service account projection feature. Xref: kubernetes/kubernetes#63819, kubernetes/community#1973 Closes: #9102 * Update pod priority and preemption user docs (#9172) * Update pod priority and preemption user docs * Copyedit * Documentation on setting node name with Kubeadm (#8925) * Documentation on setting node name with Kubeadm * copyedit * Add kubeadm upgrade docs for 1.11 (#9089) * Add kubeadm upgrade docs for 1.11 * Initial docs review feedback * Add 1-11 to outline * Fix formatting on tab blocks * Move file to correct location * Add `kubeadm upgrade node config` step * Overzealous ediffing * copyedit, fix lists and headings * clarify --force flag for fixing bad state * Get TOML ready for 1.11 release * Blog post for 1.11 release (#9254) * Blog post for 1.11 release * Update 2018-06-26-kubernetes-1.11-release-announcement.md * Update 2018-06-26-kubernetes-1.11-release-announcement.md * Update 2018-06-26-kubernetes-1.11-release-announcement.md
This commit is contained in:
Misty Linville
committed by
k8s-ci-robot
k8s-ci-robot
parent
e091e29a5d
commit
5f30e7d4c3
@@ -206,9 +206,6 @@ The objects that are stored to disk for a cluster's internal representation of t
|
||||
When the supported API changes, these objects may need to be rewritten in the newer API. Failure to do this will eventually result in resources that are no longer decodable or usable
|
||||
by the Kubernetes API server.
|
||||
|
||||
`KUBE_API_VERSIONS` environment variable for the `kube-apiserver` binary which controls the API versions that are supported in the cluster. The first version in the list is used as the cluster's storage version. Hence, to set a specific version as the storage version, bring it to the front of list of versions in the value of `KUBE_API_VERSIONS`. You need to restart the `kube-apiserver` binary
|
||||
for changes to this variable to take effect.
|
||||
|
||||
### Switching your config files to a new API version
|
||||
|
||||
You can use `kubectl convert` command to convert config files between different API versions.
|
||||
@@ -217,6 +214,4 @@ You can use `kubectl convert` command to convert config files between different
|
||||
kubectl convert -f pod.yaml --output-version v1
|
||||
```
|
||||
|
||||
For more options, please refer to the usage of [kubectl convert](/docs/reference/generated/kubectl/kubectl-commands/#convert) command.
|
||||
|
||||
{{% /capture %}}
|
||||
For more options, please refer to the usage of [kubectl convert](/docs/user-guide/kubectl/{{page.version}}/#convert) command.
|
||||
|
||||
@@ -6,11 +6,8 @@ min-kubernetes-server-version: v1.9
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{< feature-state state="beta" >}}
|
||||
|
||||
{{% capture overview %}}
|
||||
This page describes how to enable CoreDNS instead of kube-dns for service
|
||||
discovery.
|
||||
This page describes the CoreDNS upgrade process and how to install kube-dns instead of CoreDNS.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
@@ -19,41 +16,47 @@ discovery.
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## About CoreDNS
|
||||
|
||||
[CoreDNS](https://coredns.io) is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS.
|
||||
Like Kubernetes, the CoreDNS project is hosted by the [CNCF.](http://www.cncf.io)
|
||||
|
||||
You can use CoreDNS instead of kube-dns in your cluster by replacing kube-dns in an existing
|
||||
deployment, or by using tools like kubeadm that will deploy and upgrade the cluster for you.
|
||||
|
||||
## Installing CoreDNS
|
||||
|
||||
For manual deployment or replacement of kube-dns, see the documentation at the
|
||||
[CoreDNS GitHub project.](https://github.com/coredns/deployment/tree/master/kubernetes)
|
||||
|
||||
## Installing CoreDNS with kubeadm
|
||||
|
||||
In Kubernetes 1.9, [CoreDNS](https://coredns.io) is available as an alpha feature, and
|
||||
in Kubernetes 1.10 it is available as a beta feature. In either case, you may install
|
||||
it during cluster creation by setting the `CoreDNS` feature gate to `true` during `kubeadm init`:
|
||||
|
||||
In Kubernetes 1.11, CoreDNS has graduated to General Availability (GA)
|
||||
and is installed by default. To install kube-dns instead, set the `CoreDNS` feature gate
|
||||
value to `false`:
|
||||
```
|
||||
kubeadm init --feature-gates=CoreDNS=true
|
||||
kubeadm init --feature-gates=CoreDNS=false
|
||||
```
|
||||
|
||||
This installs CoreDNS instead of kube-dns.
|
||||
## Upgrading an existing cluster with kubeadm
|
||||
|
||||
## Using a custom CoreDNS image repository with kubeadm
|
||||
|
||||
To use a custom image repository for the CoreDNS image, e.g. one located in your own Docker registry,
|
||||
you can execute the following command after kubeadm has deployed the CoreDNS manifest:
|
||||
|
||||
```shell
|
||||
kubectl set image -n kube-system deploy/coredns coredns=prefix.example.com/coredns/coredns:1.0.6
|
||||
```
|
||||
|
||||
## Upgrading an Existing Cluster with kubeadm
|
||||
|
||||
In Kubernetes 1.10, you can also move to CoreDNS when you use `kubeadm` to upgrade
|
||||
In Kubernetes version 1.10 and later, you can also move to CoreDNS when you use `kubeadm` to upgrade
|
||||
a cluster that is using `kube-dns`. In this case, `kubeadm` will generate the CoreDNS configuration
|
||||
("Corefile") based upon the `kube-dns` ConfigMap, preserving configurations for federation,
|
||||
stub domains, and upstream name server.
|
||||
|
||||
Note that if you are running CoreDNS in your cluster already, prior to upgrade, your existing Corefile will be
|
||||
**overwritten** by the one created during upgrade. **You should save your existing ConfigMap
|
||||
if you have customized it.** You may re-apply your customizations after the new ConfigMap is
|
||||
up and running.
|
||||
If you are moving from kube-dns to CoreDNS, make sure to set the `CoreDNS` feature gate to `true`
|
||||
during an upgrade. For example, here is what a `v1.11.0` upgrade would look like:
|
||||
```
|
||||
kubeadm upgrade apply v1.11.0 --feature-gates=CoreDNS=true
|
||||
```
|
||||
|
||||
This process will be modified for the GA release of this feature, such that an existing
|
||||
Corefile will not be overwritten.
|
||||
In versions prior to 1.11 the Corefile will be **overwritten** by the one created during upgrade.
|
||||
**You should save your existing ConfigMap if you have customized it.** You may re-apply your
|
||||
customizations after the new ConfigMap is up and running.
|
||||
|
||||
If you are running CoreDNS in Kubernetes version 1.11 and later, during upgrade,
|
||||
your existing Corefile will be retained.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -9,6 +9,11 @@ content_template: templates/concept
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.11" state="beta" >}}
|
||||
In upcoming releases, Cloud Controller Manager will
|
||||
be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers
|
||||
can develop their features independently from the core Kubernetes release cycles.**
|
||||
|
||||
{{< feature-state for_k8s_version="1.8" state="alpha" >}}
|
||||
|
||||
Before going into how to build your own cloud controller manager, some background on how it works under the hood is helpful. The cloud controller manager is code from `kube-controller-manager` utilizing Go interfaces to allow implementations from any cloud to be plugged in. Most of the scaffolding and generic controller implementations will be in core, but it will always exec out to the cloud interfaces it is provided, so long as the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L29-L50) is satisfied.
|
||||
|
||||
@@ -7,56 +7,59 @@ content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page provides hints on configuring DNS Pod and guidance on customizing the
|
||||
DNS resolution process.
|
||||
This page explains how to configure your DNS Pod and customize the
|
||||
DNS resolution process. In Kubernetes version 1.11 and later, CoreDNS is at GA
|
||||
and is installed by default with kubeadm. See [Configuring CoreDNS](#config-coredns)
|
||||
and [Using CoreDNS for Service Discovery](/docs/tasks/administer-cluster/coredns/).
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
||||
* Kubernetes version 1.6 and above.
|
||||
* The cluster must be configured to use the `kube-dns` addon.
|
||||
* Kubernetes version 1.6 or later. To work with CoreDNS, version 1.9 or later.
|
||||
* The appropriate add-on: kube-dns or CoreDNS. To install with kubeadm,
|
||||
see [the kubeadm reference documentation](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-addon).
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Introduction
|
||||
|
||||
Starting from Kubernetes v1.3, DNS is a built-in service launched automatically
|
||||
DNS is a built-in Kubernetes service launched automatically
|
||||
using the addon manager
|
||||
[cluster add-on](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/README.md).
|
||||
|
||||
The running Kubernetes DNS pod holds 3 containers:
|
||||
The running DNS Pod holds 3 containers:
|
||||
|
||||
- "`kubedns`": The `kubedns` process watches the Kubernetes master for changes
|
||||
- "`kubedns`": watches the Kubernetes master for changes
|
||||
in Services and Endpoints, and maintains in-memory lookup structures to serve
|
||||
DNS requests.
|
||||
- "`dnsmasq`": The `dnsmasq` container adds DNS caching to improve performance.
|
||||
- "`sidecar`": The `sidecar` container provides a single health check endpoint
|
||||
while performing dual healthchecks (for `dnsmasq` and `kubedns`).
|
||||
- "`dnsmasq`": adds DNS caching to improve performance.
|
||||
- "`sidecar`": provides a single health check endpoint
|
||||
to perform healthchecks for `dnsmasq` and `kubedns`.
|
||||
|
||||
The DNS pod is exposed as a Kubernetes Service with a static IP. Once assigned
|
||||
the kubelet passes DNS configured using the `--cluster-dns=<dns-service-ip>`
|
||||
flag to each container.
|
||||
The DNS Pod is exposed as a Kubernetes Service with a static IP.
|
||||
The kubelet passes DNS to each container with the `--cluster-dns=<dns-service-ip>`
|
||||
flag.
|
||||
|
||||
DNS names also need domains. The local domain is configurable in the kubelet
|
||||
using the flag `--cluster-domain=<default-local-domain>`.
|
||||
DNS names also need domains. You configure the local domain in the kubelet
|
||||
with the flag `--cluster-domain=<default-local-domain>`.
|
||||
|
||||
The Kubernetes cluster DNS server is based off the
|
||||
The Kubernetes cluster DNS server is based on the
|
||||
[SkyDNS](https://github.com/skynetservices/skydns) library. It supports forward
|
||||
lookups (A records), service lookups (SRV records) and reverse IP address
|
||||
lookups (A records), service lookups (SRV records), and reverse IP address
|
||||
lookups (PTR records).
|
||||
|
||||
## Inheriting DNS from the node
|
||||
|
||||
When running a pod, kubelet will prepend the cluster DNS server and search
|
||||
paths to the node's own DNS settings. If the node is able to resolve DNS names
|
||||
specific to the larger environment, pods should be able to, also.
|
||||
See [Known issues](#known-issues) below for a caveat.
|
||||
When running a Pod, kubelet prepends the cluster DNS server and searches
|
||||
paths to the node's DNS settings. If the node is able to resolve DNS names
|
||||
specific to the larger environment, Pods should also be able to resolve.
|
||||
But see [Known issues](#known-issues).
|
||||
|
||||
If you don't want this, or if you want a different DNS config for pods, you can
|
||||
use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will
|
||||
not inherit DNS. Setting it to a valid file path means that kubelet will use
|
||||
this file instead of `/etc/resolv.conf` for DNS inheritance.
|
||||
use the kubelet's `--resolv-conf` flag. Set this flag to "" to prevent Pods from
|
||||
inheriting DNS. Set it to a valid file path to specify a file other than
|
||||
`/etc/resolv.conf` for DNS inheritance.
|
||||
|
||||
## Configure stub-domain and upstream DNS servers
|
||||
|
||||
@@ -64,7 +67,7 @@ Cluster administrators can specify custom stub domains and upstream nameservers
|
||||
by providing a ConfigMap for kube-dns (`kube-system:kube-dns`).
|
||||
|
||||
For example, the following ConfigMap sets up a DNS configuration with a single stub domain and two
|
||||
upstream nameservers.
|
||||
upstream nameservers:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -79,11 +82,11 @@ data:
|
||||
["8.8.8.8", "8.8.4.4"]
|
||||
```
|
||||
|
||||
As specified, DNS requests with the “.acme.local” suffix
|
||||
DNS requests with the “.acme.local” suffix
|
||||
are forwarded to a DNS listening at 1.2.3.4. Google Public DNS
|
||||
serves the upstream queries.
|
||||
|
||||
The table below describes how queries with certain domain names would map to
|
||||
The table below describes how queries with certain domain names map to
|
||||
their destination DNS servers:
|
||||
|
||||
| Domain name | Server answering the query |
|
||||
@@ -99,9 +102,9 @@ details about the configuration option format.
|
||||
|
||||
{{% capture discussion %}}
|
||||
|
||||
### Impacts on Pods
|
||||
### Effects on Pods
|
||||
|
||||
Custom upstream nameservers and stub domains won't impact Pods that have their
|
||||
Custom upstream nameservers and stub domains do not affect Pods with a
|
||||
`dnsPolicy` set to "`Default`" or "`None`".
|
||||
|
||||
If a Pod's `dnsPolicy` is set to "`ClusterFirst`", its name resolution is
|
||||
@@ -113,23 +116,23 @@ cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream
|
||||
nameserver inherited from the node.
|
||||
|
||||
**With custom configurations**: If stub domains and upstream DNS servers are
|
||||
configured (as in the [previous example](#configuring-stub-domain-and-upstream-dns-servers)),
|
||||
DNS queries will be routed according to the following flow:
|
||||
configured,
|
||||
DNS queries are routed according to the following flow:
|
||||
|
||||
1. The query is first sent to the DNS caching layer in kube-dns.
|
||||
|
||||
1. From the caching layer, the suffix of the request is examined and then
|
||||
forwarded to the appropriate DNS, based on the following cases:
|
||||
|
||||
* *Names with the cluster suffix* (e.g.".cluster.local"):
|
||||
* *Names with the cluster suffix*, for example ".cluster.local":
|
||||
The request is sent to kube-dns.
|
||||
|
||||
* *Names with the stub domain suffix* (e.g. ".acme.local"):
|
||||
The request is sent to the configured custom DNS resolver (e.g. listening at 1.2.3.4).
|
||||
* *Names with the stub domain suffix*, for example ".acme.local":
|
||||
The request is sent to the configured custom DNS resolver, listening for example at 1.2.3.4.
|
||||
|
||||
* *Names without a matching suffix* (e.g."widget.com"):
|
||||
The request is forwarded to the upstream DNS
|
||||
(e.g. Google public DNS servers at 8.8.8.8 and 8.8.4.4).
|
||||
* *Names without a matching suffix*, for example "widget.com":
|
||||
The request is forwarded to the upstream DNS,
|
||||
for example Google public DNS servers at 8.8.8.8 and 8.8.4.4.
|
||||
|
||||
|
||||
|
||||
@@ -139,17 +142,17 @@ Options for the kube-dns `kube-system:kube-dns` ConfigMap:
|
||||
|
||||
| Field | Format | Description |
|
||||
| ----- | ------ | ----------- |
|
||||
| `stubDomains` (optional) | A JSON map using a DNS suffix key (e.g. “acme.local”) and a value consisting of a JSON array of DNS IPs. | The target nameserver may itself be a Kubernetes service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
|
||||
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | Note: If specified, then the values specified replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
|
||||
| `stubDomains` (optional) | A JSON map using a DNS suffix key such as “acme.local”, and a value consisting of a JSON array of DNS IPs. | The target nameserver can itself be a Kubernetes Service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
|
||||
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | If specified, the values replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
|
||||
|
||||
### Examples
|
||||
|
||||
#### Example: Stub domain
|
||||
|
||||
In this example, the user has a Consul DNS service discovery system that they wish to
|
||||
In this example, the user has a Consul DNS service discovery system they want to
|
||||
integrate with kube-dns. The consul domain server is located at 10.150.0.1, and
|
||||
all consul names have the suffix “.consul.local”. To configure Kubernetes, the
|
||||
cluster administrator simply creates a ConfigMap object as shown below.
|
||||
all consul names have the suffix `.consul.local`. To configure Kubernetes, the
|
||||
cluster administrator creates the following ConfigMap:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -162,7 +165,7 @@ data:
|
||||
{"consul.local": ["10.150.0.1"]}
|
||||
```
|
||||
|
||||
Note that the cluster administrator did not wish to override the node’s
|
||||
Note that the cluster administrator does not want to override the node’s
|
||||
upstream nameservers, so they did not specify the optional
|
||||
`upstreamNameservers` field.
|
||||
|
||||
@@ -170,8 +173,8 @@ upstream nameservers, so they did not specify the optional
|
||||
|
||||
In this example the cluster administrator wants to explicitly force all
|
||||
non-cluster DNS lookups to go through their own nameserver at 172.16.0.1.
|
||||
Again, this is easy to accomplish; they just need to create a ConfigMap with the
|
||||
`upstreamNameservers` field specifying the desired nameserver.
|
||||
In this case, they create a ConfigMap with the
|
||||
`upstreamNameservers` field specifying the desired nameserver:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -184,9 +187,97 @@ data:
|
||||
["172.16.0.1"]
|
||||
```
|
||||
|
||||
## What's next
|
||||
- [Debugging DNS Resolution](/docs/tasks/administer-cluster/dns-debugging-resolution/).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
## Configuring CoreDNS {config-coredns}
|
||||
|
||||
You can configure [CoreDNS](https://coredns.io/) as a service discovery.
|
||||
|
||||
CoreDNS is available as an option in Kubernetes starting with version 1.9.
|
||||
It is currently a [GA feature](https://github.com/kubernetes/community/blob/master/keps/sig-network/0010-20180314-coredns-GA-proposal.md) and is on course to be [the default](https://github.com/kubernetes/community/blob/master/keps/sig-network/0012-20180518-coredns-default-proposal.md), replacing kube-dns.
|
||||
|
||||
|
||||
## CoreDNS ConfigMap options
|
||||
|
||||
CoreDNS chains plugins and can be configured by maintaining a Corefile with the ConfigMap. CoreDNS supports all the functionalities and more that is provided by kube-dns.
|
||||
A ConfigMap created for kube-dns to support `StubDomains`and `upstreamnameserver` translates to the `proxy` plugin in CoreDNS.
|
||||
Similarly, the `Federation` plugin translates to the `federation` plugin in CoreDNS.
|
||||
|
||||
### Example
|
||||
|
||||
This example ConfigMap for kube0dns specifies federations, stubdomains and upstreamnameservers:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
data:
|
||||
federations: |
|
||||
{"foo" : "foo.feddomain.com"}
|
||||
stubDomains: |
|
||||
{"abc.com" : ["1.2.3.4"], "my.cluster.local" : ["2.3.4.5"]}
|
||||
upstreamNameservers: |
|
||||
["8.8.8.8", "8.8.4.4"]
|
||||
kind: ConfigMap
|
||||
```
|
||||
|
||||
The equivalent configuration in CoreDNS creates a Corefile:
|
||||
|
||||
* For federations:
|
||||
```yaml
|
||||
federation cluster.local {
|
||||
foo foo.feddomain.com
|
||||
}
|
||||
```
|
||||
|
||||
* For stubDomains:
|
||||
```yaml
|
||||
abc.com:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 1.2.3.4
|
||||
}
|
||||
my.cluster.local:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 2.3.4.5
|
||||
}
|
||||
```
|
||||
|
||||
The complete Corefile with the default plugins:
|
||||
|
||||
```yaml
|
||||
.:53 {
|
||||
errors
|
||||
health
|
||||
kubernetes cluster.local in-addr.arpa ip6.arpa {
|
||||
upstream 8.8.8.8 8.8.4.4
|
||||
pods insecure
|
||||
fallthrough in-addr.arpa ip6.arpa
|
||||
}
|
||||
federation cluster.local {
|
||||
foo foo.feddomain.com
|
||||
}
|
||||
prometheus :9153
|
||||
proxy . 8.8.8.8 8.8.4.4
|
||||
cache 30
|
||||
}
|
||||
abc.com:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 1.2.3.4
|
||||
}
|
||||
my.cluster.local:53 {
|
||||
errors
|
||||
cache 30
|
||||
proxy . 2.3.4.5
|
||||
}
|
||||
```
|
||||
|
||||
In Kubernetes version 1.10 and later, kubeadm supports automatic translation of the CoreDNS ConfigMap from the kube-dns ConfigMap.
|
||||
|
||||
## Migration to CoreDNS
|
||||
|
||||
A number of tools support the installation of CoreDNS instead of kube-dns.
|
||||
To migrate from kube-dns to CoreDNS, [a detailed blog](https://coredns.io/2018/05/21/migration-from-kube-dns-to-coredns/) is available to help users adapt CoreDNS in place of kube-dns.
|
||||
|
||||
## What's next
|
||||
- [Debugging DNS Resolution](/docs/tasks/administer-cluster/dns-debugging-resolution/).
|
||||
|
||||
@@ -83,7 +83,6 @@ The output shows that the Node has a capacity of 4 dongles:
|
||||
|
||||
```
|
||||
"capacity": {
|
||||
"alpha.kubernetes.io/nvidia-gpu": "0",
|
||||
"cpu": "2",
|
||||
"memory": "2049008Ki",
|
||||
"example.com/dongle": "4",
|
||||
@@ -99,7 +98,6 @@ Once again, the output shows the dongle resource:
|
||||
|
||||
```yaml
|
||||
Capacity:
|
||||
alpha.kubernetes.io/nvidia-gpu: 0
|
||||
cpu: 2
|
||||
memory: 2049008Ki
|
||||
example.com/dongle: 4
|
||||
@@ -205,6 +203,3 @@ kubectl describe node <your-node-name> | grep dongle
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ content_template: templates/concept
|
||||
|
||||
In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a master machine
|
||||
there are a number of add-ons which, for various reasons, must run on a regular cluster node (rather than the Kubernetes master).
|
||||
Some of these add-ons are critical to a fully functional cluster, such as Heapster, DNS, and UI.
|
||||
Some of these add-ons are critical to a fully functional cluster, such as metrics-server, DNS, and UI.
|
||||
A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade)
|
||||
and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space
|
||||
vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason).
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
---
|
||||
reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Upgrading kubeadm clusters from v1.10 to v1.11
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
This page explains how to upgrade a Kubernetes cluster created with `kubeadm` from version 1.10.x to version 1.11.x, and from version 1.11.x to 1.11.y, where `y > x`.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- You need to have a `kubeadm` Kubernetes cluster running version 1.10.0 or later. Swap must be disabled. The cluster should use a static control plane and etcd pods.
|
||||
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md) carefully.
|
||||
- Make sure to back up any important components, such as app-level state stored in a database. `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice.
|
||||
|
||||
### Additional information
|
||||
|
||||
- All containers are restarted after upgrade, because the container spec hash value is changed.
|
||||
- You can upgrade only froom one minor version to the next minor version. That is, you cannot skip versions when you upgrade. For example, you can upgrade only from 1.10 to 1.11, not from 1.9 to 1.11.
|
||||
- The default DNS provider in version 1.11 is [CoreDNS](https://coredns.io/) rather than [kube-dns](https://github.com/kubernetes/dns).
|
||||
To keep `kube-dns`, pass `--feature-flags=CoreDNS=false` to `kubeadm upgrade apply`.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Upgrade the control plane
|
||||
|
||||
1. On your master node, run the following (as root:
|
||||
|
||||
export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
|
||||
export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
|
||||
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
|
||||
chmod a+rx /usr/bin/kubeadm
|
||||
|
||||
Note that upgrading the `kubeadm` package on your system prior to upgrading the control plane causes a failed upgrade. Even though `kubeadm` ships in the Kubernetes repositories, it's important to install it manually. The kubeadm team is working on fixing this limitation.
|
||||
|
||||
1. Verify that the download works and has the expected version:
|
||||
|
||||
```shell
|
||||
kubeadm version
|
||||
```
|
||||
|
||||
1. On the master node, run:
|
||||
|
||||
```shell
|
||||
kubeadm upgrade plan
|
||||
```
|
||||
|
||||
You should see output similar to this:
|
||||
|
||||
<!-- TODO: copy-paste actual output once new version is stable -->
|
||||
|
||||
```shell
|
||||
[preflight] Running pre-flight checks.
|
||||
[upgrade] Making sure the cluster is healthy:
|
||||
[upgrade/config] Making sure the configuration is correct:
|
||||
[upgrade/config] Reading configuration from the cluster...
|
||||
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
|
||||
I0618 20:32:32.950358 15307 feature_gate.go:230] feature gates: &{map[]}
|
||||
[upgrade] Fetching available versions to upgrade to
|
||||
[upgrade/versions] Cluster version: v1.10.4
|
||||
[upgrade/versions] kubeadm version: v1.11.0-beta.2.78+e0b33dbc2bde88
|
||||
|
||||
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
|
||||
COMPONENT CURRENT AVAILABLE
|
||||
Kubelet 1 x v1.10.4 v1.11.0
|
||||
|
||||
Upgrade to the latest version in the v1.10 series:
|
||||
|
||||
COMPONENT CURRENT AVAILABLE
|
||||
API Server v1.10.4 v1.11.0
|
||||
Controller Manager v1.10.4 v1.11.0
|
||||
Scheduler v1.10.4 v1.11.0
|
||||
Kube Proxy v1.10.4 v1.11.0
|
||||
CoreDNS 1.1.3
|
||||
Kube DNS 1.14.8
|
||||
Etcd 3.1.12 3.2.18
|
||||
|
||||
You can now apply the upgrade by executing the following command:
|
||||
|
||||
kubeadm upgrade apply v1.11.0
|
||||
|
||||
Note: Before you can perform this upgrade, you have to update kubeadm to v1.11.0.
|
||||
|
||||
_____________________________________________________________________
|
||||
```
|
||||
|
||||
This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to.
|
||||
|
||||
1. Choose a version to upgrade to, and run the appropriate command. For example:
|
||||
|
||||
```shell
|
||||
kubeadm upgrade apply v1.11.0
|
||||
```
|
||||
|
||||
If you currently use `kube-dns` and wish to continue doing so, add `--feature-flags=CoreDNS=false`.
|
||||
|
||||
You should see output similar to this:
|
||||
|
||||
<!-- TODO: output from stable -->
|
||||
|
||||
```shell
|
||||
[preflight] Running pre-flight checks.
|
||||
[upgrade] Making sure the cluster is healthy:
|
||||
[upgrade/config] Making sure the configuration is correct:
|
||||
[upgrade/config] Reading configuration from the cluster...
|
||||
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
|
||||
I0614 20:56:08.320369 30918 feature_gate.go:230] feature gates: &{map[]}
|
||||
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
|
||||
[upgrade/version] You have chosen to change the cluster version to "v1.11.0-beta.2.78+e0b33dbc2bde88"
|
||||
[upgrade/versions] Cluster version: v1.10.4
|
||||
[upgrade/versions] kubeadm version: v1.11.0-beta.2.78+e0b33dbc2bde88
|
||||
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
|
||||
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
|
||||
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.11.0-beta.2.78+e0b33dbc2bde88"...
|
||||
Static pod: kube-apiserver-ip-172-31-85-18 hash: 7a329408b21bc0c44d7b3b78ff8187bf
|
||||
Static pod: kube-controller-manager-ip-172-31-85-18 hash: 24fd3157627c7567b687968967c6a5e8
|
||||
Static pod: kube-scheduler-ip-172-31-85-18 hash: 5179266fb24d4c1834814c4f69486371
|
||||
Static pod: etcd-ip-172-31-85-18 hash: 9dfc197f444be11fcc70ab1467b030b8
|
||||
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/etcd.yaml"
|
||||
[certificates] Using the existing etcd/ca certificate and key.
|
||||
[certificates] Using the existing etcd/server certificate and key.
|
||||
[certificates] Using the existing etcd/peer certificate and key.
|
||||
[certificates] Using the existing etcd/healthcheck-client certificate and key.
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/etcd.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: etcd-ip-172-31-85-18 hash: 9dfc197f444be11fcc70ab1467b030b8
|
||||
< snip >
|
||||
[apiclient] Found 1 Pods for label selector component=etcd
|
||||
[upgrade/staticpods] Component "etcd" upgraded successfully!
|
||||
[upgrade/etcd] Waiting for etcd to become available
|
||||
[util/etcd] Waiting 0s for initial delay
|
||||
[util/etcd] Attempting to see if all cluster endpoints are available 1/10
|
||||
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939"
|
||||
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/kube-apiserver.yaml"
|
||||
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/kube-controller-manager.yaml"
|
||||
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests089436939/kube-scheduler.yaml"
|
||||
[certificates] Using the existing etcd/ca certificate and key.
|
||||
[certificates] Using the existing apiserver-etcd-client certificate and key.
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/kube-apiserver.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: kube-apiserver-ip-172-31-85-18 hash: 7a329408b21bc0c44d7b3b78ff8187bf
|
||||
< snip >
|
||||
[apiclient] Found 1 Pods for label selector component=kube-apiserver
|
||||
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/kube-controller-manager.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: kube-controller-manager-ip-172-31-85-18 hash: 24fd3157627c7567b687968967c6a5e8
|
||||
Static pod: kube-controller-manager-ip-172-31-85-18 hash: 63992ff14733dcb9dcfa6ac0a3b8031a
|
||||
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
|
||||
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
|
||||
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-06-14-20-56-11/kube-scheduler.yaml"
|
||||
[upgrade/staticpods] Waiting for the kubelet to restart the component
|
||||
Static pod: kube-scheduler-ip-172-31-85-18 hash: 5179266fb24d4c1834814c4f69486371
|
||||
Static pod: kube-scheduler-ip-172-31-85-18 hash: 831e4b9425f758e572392976311e56d9
|
||||
[apiclient] Found 1 Pods for label selector component=kube-scheduler
|
||||
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
|
||||
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
|
||||
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
|
||||
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
|
||||
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
|
||||
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ip-172-31-85-18" as an annotation
|
||||
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
|
||||
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
|
||||
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
|
||||
[addons] Applied essential addon: CoreDNS
|
||||
[addons] Applied essential addon: kube-proxy
|
||||
|
||||
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.11.0-beta.2.78+e0b33dbc2bde88". Enjoy!
|
||||
|
||||
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
|
||||
```
|
||||
|
||||
1. Manually upgrade your Software Defined Network (SDN).
|
||||
|
||||
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow.
|
||||
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
|
||||
find your CNI provider and see whther additional upgrade steps are required.
|
||||
|
||||
## Upgrade master and node packages
|
||||
|
||||
1. Prepare each host for maintenance, marking it unschedulable and evicting the workload:
|
||||
|
||||
```shell
|
||||
kubectl drain $HOST --ignore-daemonsets
|
||||
```
|
||||
|
||||
On the master host, you must add `--ignore-daemonsets`:
|
||||
|
||||
```shell
|
||||
kubectl drain ip-172-31-85-18
|
||||
node "ip-172-31-85-18" cordoned
|
||||
error: unable to drain node "ip-172-31-85-18", aborting command...
|
||||
|
||||
There are pending nodes to be drained:
|
||||
ip-172-31-85-18
|
||||
error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-5798d, kube-proxy-thjp9
|
||||
```
|
||||
|
||||
```
|
||||
kubectl drain ip-172-31-85-18 --ignore-daemonsets
|
||||
node "ip-172-31-85-18" already cordoned
|
||||
WARNING: Ignoring DaemonSet-managed pods: calico-node-5798d, kube-proxy-thjp9
|
||||
node "ip-172-31-85-18" drained
|
||||
```
|
||||
|
||||
1. Upgrade the Kubernetes package version on each `$HOST` node by running the Linux package manager for your distribution:
|
||||
|
||||
{{< tabs name="k8s_install" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
apt-get update
|
||||
apt-get upgrade -y kubelet kubeadm
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
yum upgrade -y kubelet kubedam
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
## Upgrade kubelet on each node
|
||||
|
||||
1. On each node except the master node, upgrade the kubelet config:
|
||||
|
||||
```shell
|
||||
sudo kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)
|
||||
```
|
||||
|
||||
1. Restart the kubectl process:
|
||||
|
||||
```shell
|
||||
sudo systemctl restart kubelet
|
||||
```
|
||||
|
||||
1. Verify that the new version of the `kubelet` is running on the host:
|
||||
|
||||
```shell
|
||||
systemctl status kubelet
|
||||
```
|
||||
|
||||
1. Bring the host back online by marking it schedulable:
|
||||
|
||||
```shell
|
||||
kubectl uncordon $HOST
|
||||
```
|
||||
|
||||
1. After the kubelet is upgraded on all hosts, verify that all nodes are available again by running the following command from anywhere -- for example, from outside the cluster:
|
||||
|
||||
```shell
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
The `STATUS` column should show `Ready` for all your hosts, and the version number should be updated.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
## Recovering from a failure state
|
||||
|
||||
If `kubeadm upgrade` fails and does not roll back, for example because of an unexpected shutdown during execution,
|
||||
you can run `kubeadm upgrade` again. This command is idempotent and eventually makes sure that the actual state is the desired state you declare.
|
||||
|
||||
To recover from a bad state, you can also run `kubeadm upgrade --force` without changing the version that your cluster is running.
|
||||
|
||||
## How it works
|
||||
|
||||
`kubeadm upgrade apply` does the following:
|
||||
|
||||
- Checks that your cluster is in an upgradeable state:
|
||||
- The API server is reachable,
|
||||
- All nodes are in the `Ready` state
|
||||
- The control plane is healthy
|
||||
- Enforces the version skew policies.
|
||||
- Makes sure the control plane images are available or available to pull to the machine.
|
||||
- Upgrades the control plane components or rollbacks if any of them fails to come up.
|
||||
- Applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
|
||||
- Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days.
|
||||
@@ -1,145 +0,0 @@
|
||||
---
|
||||
reviewers:
|
||||
- msau42
|
||||
- jsafrane
|
||||
title: Persistent Volume Claim Protection
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state for_k8s_version="v1.9" state="alpha" >}}
|
||||
|
||||
As of Kubernetes 1.9, persistent volume claims (PVCs) that are in active use by a pod can be protected from pre-mature removal.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- A v1.9 or higher Kubernetes must be installed.
|
||||
- As PVC Protection is a Kubernetes v1.9 alpha feature it must be enabled:
|
||||
1. [Admission controller](/docs/admin/admission-controllers/) must be started with the [PVC Protection plugin](/docs/admin/admission-controllers/#persistent-volume-claim-protection-alpha).
|
||||
2. All Kubernetes components must be started with the `PVCProtection` alpha features enabled.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## PVC Protection Verification
|
||||
|
||||
The example below uses a GCE PD `StorageClass`, however, similar steps can be performed for any volume type.
|
||||
|
||||
Create a `StorageClass` for convenient storage provisioning:
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: slow
|
||||
provisioner: kubernetes.io/gce-pd
|
||||
parameters:
|
||||
type: pd-standard
|
||||
```
|
||||
|
||||
There are two scenarios: a PVC deleted by a user is either in active use or not in active use by a pod.
|
||||
|
||||
### Scenario 1: The PVC is not in active use by a pod
|
||||
|
||||
- Create a PVC:
|
||||
|
||||
```yaml
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: slzc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: slow
|
||||
resources:
|
||||
requests:
|
||||
storage: 3.7Gi
|
||||
```
|
||||
|
||||
- Check that the PVC has the finalizer `kubernetes.io/pvc-protection` set:
|
||||
```shell
|
||||
$ kubectl describe pvc slzc
|
||||
Name: slzc
|
||||
Namespace: default
|
||||
StorageClass: slow
|
||||
Status: Bound
|
||||
Volume: pvc-bee8c30a-d6a3-11e7-9af0-42010a800002
|
||||
Labels: <none>
|
||||
Annotations: pv.kubernetes.io/bind-completed=yes
|
||||
pv.kubernetes.io/bound-by-controller=yes
|
||||
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
|
||||
Finalizers: [kubernetes.io/pvc-protection]
|
||||
Capacity: 4Gi
|
||||
Access Modes: RWO
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal ProvisioningSucceeded 2m persistentvolume-controller Successfully provisioned volume pvc-bee8c30a-d6a3-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
|
||||
```
|
||||
|
||||
- Delete the PVC and check that the PVC (not in active use by a pod) was removed successfully.
|
||||
|
||||
### Scenario 2: The PVC is in active use by a pod
|
||||
|
||||
- Again, create the same PVC.
|
||||
- Create a pod that uses the PVC:
|
||||
|
||||
```yaml
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: app1
|
||||
spec:
|
||||
containers:
|
||||
- name: test-pod
|
||||
image: k8s.gcr.io/busybox:1.24
|
||||
command:
|
||||
- "/bin/sh"
|
||||
args:
|
||||
- "-c"
|
||||
- "date > /mnt/app1.txt; sleep 60 && exit 0 || exit 1"
|
||||
volumeMounts:
|
||||
- name: path-pvc
|
||||
mountPath: "/mnt"
|
||||
restartPolicy: "Never"
|
||||
volumes:
|
||||
- name: path-pvc
|
||||
persistentVolumeClaim:
|
||||
claimName: slzc
|
||||
```
|
||||
|
||||
- Wait until the pod status is `Running`, i.e. the PVC becomes in active use.
|
||||
- Delete the PVC that is now in active use by a pod and verify that the PVC is not removed but its status is `Terminating`:
|
||||
|
||||
```shell
|
||||
Name: slzc
|
||||
Namespace: default
|
||||
StorageClass: slow
|
||||
Status: Terminating (since Fri, 01 Dec 2017 14:47:55 +0000)
|
||||
Volume: pvc-803a1f4d-d6a6-11e7-9af0-42010a800002
|
||||
Labels: <none>
|
||||
Annotations: pv.kubernetes.io/bind-completed=yes
|
||||
pv.kubernetes.io/bound-by-controller=yes
|
||||
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
|
||||
Finalizers: [kubernetes.io/pvc-protection]
|
||||
Capacity: 4Gi
|
||||
Access Modes: RWO
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal ProvisioningSucceeded 52s persistentvolume-controller Successfully provisioned volume pvc-803a1f4d-d6a6-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
|
||||
```
|
||||
- Wait until the pod status is `Terminated` (either delete the pod or wait until it finishes). Afterwards, check that the PVC is removed.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture discussion %}}
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -7,31 +7,28 @@ content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state state="alpha" >}}
|
||||
As of Kubernetes 1.8, the new
|
||||
{{< feature-state for_k8s_version="v1.11" state="beta" >}}
|
||||
|
||||
[Dynamic Kubelet Configuration](https://github.com/kubernetes/features/issues/281)
|
||||
feature is available in alpha. This allows you to change the configuration of
|
||||
Kubelets in a live Kubernetes cluster via first-class Kubernetes concepts.
|
||||
Specifically, this feature allows you to configure individual Nodes' Kubelets
|
||||
via ConfigMaps.
|
||||
allows you to change the configuration of each Kubelet in a live Kubernetes
|
||||
cluster by deploying a ConfigMap and configuring each Node to use it.
|
||||
|
||||
{{< warning >}}
|
||||
**Warning:** All Kubelet configuration parameters may be changed dynamically,
|
||||
but not all parameters are safe to change dynamically. This feature is intended
|
||||
for system experts who have a strong understanding of how configuration changes
|
||||
will affect behavior. No documentation currently exists which plainly lists
|
||||
"safe to change" fields, but we plan to add it before this feature graduates
|
||||
from alpha.
|
||||
**Warning:** All Kubelet configuration parameters can be changed dynamically,
|
||||
but this is unsafe for some parameters. Before deciding to change a parameter
|
||||
dynamically, you need a strong understanding of how that change will affect your
|
||||
cluster's behavior. Always carefully test configuration changes on a small set
|
||||
of nodes before rolling them out cluster-wide. Advice on configuring specific
|
||||
fields is available in the inline `KubeletConfiguration`
|
||||
[type documentation](https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go).
|
||||
{{< /warning >}}
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
- A live Kubernetes cluster with both Master and Node at v1.8 or higher must be
|
||||
running, with the `DynamicKubeletConfig` feature gate enabled and the Kubelet's
|
||||
`--dynamic-config-dir` flag set to a writable directory on the Node.
|
||||
This flag must be set to enable Dynamic Kubelet Configuration.
|
||||
- The kubectl command-line tool must be also v1.8 or higher, and must be
|
||||
configured to communicate with the cluster.
|
||||
- Kubernetes v1.11 or higher on both the Master and the Nodes
|
||||
- kubectl v1.11 or higher, configured to communicate with the cluster
|
||||
- The Kubelet's `--dynamic-config-dir` flag must be set to a writable
|
||||
directory on the Node.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
@@ -50,20 +47,19 @@ Kubelet's configuration.
|
||||
Each Kubelet watches a configuration reference on its respective Node object.
|
||||
When this reference changes, the Kubelet downloads the new configuration,
|
||||
updates a local reference to refer to the file, and exits.
|
||||
For the feature to work correctly, you must be running a process manager
|
||||
(like systemd) which will restart the Kubelet when it exits. When the Kubelet is
|
||||
restarted, it will begin using the new configuration.
|
||||
For the feature to work correctly, you must be running an OS-level service
|
||||
manager (such as systemd), which will restart the Kubelet if it exits. When the
|
||||
Kubelet is restarted, it will begin using the new configuration.
|
||||
|
||||
The new configuration completely overrides configuration provided by `--config`,
|
||||
and is overridden by command-line flags. Unspecified values in the new configuration
|
||||
will receive default values appropriate to the configuration version
|
||||
(e.g. `kubelet.config.k8s.io/v1beta1`), unless overridden by flags.
|
||||
|
||||
The status of the Node's Kubelet configuration is reported via the `KubeletConfigOK`
|
||||
condition in the Node status. Once you have updated a Node to use the new
|
||||
ConfigMap, you can observe this condition to confirm that the Node is using the
|
||||
intended configuration. A table describing the possible conditions can be found
|
||||
at the end of this article.
|
||||
The status of the Node's Kubelet configuration is reported via
|
||||
`Node.Spec.Status.Config`. Once you have updated a Node to use the new
|
||||
ConfigMap, you can observe this status to confirm that the Node is using the
|
||||
intended configuration.
|
||||
|
||||
This document describes editing Nodes using `kubectl edit`.
|
||||
There are other ways to modify a Node's spec, including `kubectl patch`, for
|
||||
@@ -72,16 +68,19 @@ example, which facilitate scripted workflows.
|
||||
This document only describes a single Node consuming each ConfigMap. Keep in
|
||||
mind that it is also valid for multiple Nodes to consume the same ConfigMap.
|
||||
|
||||
### Node Authorizer Workarounds
|
||||
{{< warning >}}
|
||||
**Warning:** While it is *possible* to change the configuration by
|
||||
updating the ConfigMap in-place, this causes all Kubelets configured with
|
||||
that ConfigMap to update simultaneously. It is much safer to treat ConfigMaps
|
||||
as immutable by convention, aided by `kubectl`'s `--append-hash` option,
|
||||
and incrementally roll out updates to `Node.Spec.ConfigSource`.
|
||||
{{< /warning >}}
|
||||
|
||||
The Node Authorizer does not yet pay attention to which ConfigMaps are assigned
|
||||
to which Nodes. If you currently use the Node authorizer, your Kubelets will not
|
||||
be automatically granted permission to download their respective ConfigMaps.
|
||||
### Automatic RBAC rules for Node Authorizer
|
||||
|
||||
The temporary workaround used in this document is to manually create the RBAC
|
||||
Roles and RoleBindings for each ConfigMap. The Node Authorizer will be extended
|
||||
before the Dynamic Kubelet Configuration feature graduates from alpha, so doing
|
||||
this in production should never be necessary.
|
||||
Previously, you were required to manually create RBAC rules
|
||||
to allow Nodes to access their assigned ConfigMaps. The Node Authorizer now
|
||||
automatically configures these rules.
|
||||
|
||||
### Generating a file that contains the current configuration
|
||||
|
||||
@@ -92,54 +91,63 @@ and debug issues. The compromise, however, is that you must start with knowledge
|
||||
of the existing configuration to ensure that you only change the fields you
|
||||
intend to change.
|
||||
|
||||
In the future, the Kubelet will be bootstrapped from a file on disk
|
||||
Ideally, the Kubelet would be bootstrapped from a file on disk
|
||||
and you could edit this file (which could also be version-controlled),
|
||||
to create the first Kubelet ConfigMap
|
||||
(see [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)),
|
||||
and you will simply edit a copy of this file (which, as a best practice, should
|
||||
live in version control) while creating the first Kubelet ConfigMap. Today,
|
||||
however, the Kubelet is still bootstrapped with command-line flags. Fortunately,
|
||||
there is a dirty trick you can use to generate a config file containing a Node's
|
||||
current configuration. The trick involves accessing the Kubelet server's `configz`
|
||||
endpoint via the kubectl proxy. This endpoint, in its current implementation, is
|
||||
intended to be used only as a debugging aid, which is part of why this is a
|
||||
dirty trick. The endpoint may be improved in the future, but until then
|
||||
it should not be relied on for production scenarios.
|
||||
This trick also requires the `jq` command to be installed on your machine,
|
||||
for unpacking and editing the JSON response from the endpoint.
|
||||
Currently, the Kubelet is bootstrapped with **a combination of this file and command-line flags**
|
||||
that can override the configuration in the file.
|
||||
As a workaround, you can generate a config file containing a Node's current
|
||||
configuration by accessing the Kubelet server's `configz` endpoint via the
|
||||
kubectl proxy. This endpoint, in its current implementation, is intended to be
|
||||
used only as a debugging aid. Do not rely on the behavior of this endpoint for
|
||||
production scenarios. The examples below use the `jq` command to streamline
|
||||
working with JSON. To follow the tasks as written, you need to have `jq`
|
||||
installed, but you can adapt the tasks if you prefer to extract the
|
||||
`kubeletconfig` subobject manually.
|
||||
|
||||
Do the following to generate the file:
|
||||
#### Generate the configuration file
|
||||
|
||||
1. Pick a Node to reconfigure. We will refer to this Node's name as NODE_NAME.
|
||||
2. Start the kubectl proxy in the background with `kubectl proxy --port=8001 &`
|
||||
3. Run the following command to download and unpack the configuration from the
|
||||
configz endpoint:
|
||||
1. Choose a Node to reconfigure. In this example, the name of this Node is
|
||||
referred to as `NODE_NAME`.
|
||||
2. Start the kubectl proxy in the background using the following command:
|
||||
```bash
|
||||
kubectl proxy --port=8001 &
|
||||
```
|
||||
3. Run the following command to download and unpack the configuration from the
|
||||
`configz` endpoint. The command is long, so be careful when copying and
|
||||
pasting. **If you use zsh**, note that common zsh configurations add backslashes
|
||||
to escape the opening and closing curly braces around the variable name in the URL.
|
||||
For example: `${NODE_NAME}` will be rewritten as `$\{NODE_NAME\}` during the paste.
|
||||
You must remove the backslashes before running the command, or the command will fail.
|
||||
|
||||
```
|
||||
$ export NODE_NAME=the-name-of-the-node-you-are-reconfiguring
|
||||
$ curl -sSL http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
|
||||
```
|
||||
```bash
|
||||
NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
|
||||
```
|
||||
|
||||
Note that we have to manually add the `kind` and `apiVersion` to the downloaded
|
||||
object, as these are not reported by the configz endpoint. This is one of the
|
||||
limitations of the endpoint.
|
||||
{{< note >}}
|
||||
You need to manually add the `kind` and `apiVersion` to the downloaded
|
||||
object, because they are not reported by the `configz` endpoint.
|
||||
{{< /note >}}
|
||||
|
||||
### Edit the configuration file
|
||||
#### Edit the configuration file
|
||||
|
||||
Using your editor of choice, change one of the parameters in the
|
||||
`kubelet_configz_${NODE_NAME}` file from the previous step. A QPS parameter,
|
||||
`eventRecordQPS` for example, is a good candidate.
|
||||
Using a text editor, change one of the parameters in the
|
||||
file generated by the previous procedure. For example, you
|
||||
might edit the QPS parameter `eventRecordQPS`.
|
||||
|
||||
### Push the configuration file to the control plane
|
||||
#### Push the configuration file to the control plane
|
||||
|
||||
Push the edited configuration file to the control plane with the
|
||||
following command:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
|
||||
```bash
|
||||
kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
|
||||
```
|
||||
|
||||
You should see a response similar to:
|
||||
This is an example of a valid response:
|
||||
|
||||
```
|
||||
```none
|
||||
apiVersion: v1
|
||||
data:
|
||||
kubelet: |
|
||||
@@ -154,304 +162,204 @@ metadata:
|
||||
uid: 946d785e-998a-11e7-a8dd-42010a800006
|
||||
```
|
||||
|
||||
Note that the configuration data must appear under the ConfigMap's
|
||||
`kubelet` key.
|
||||
|
||||
We create the ConfigMap in the `kube-system` namespace, which is appropriate
|
||||
because this ConfigMap configures a Kubernetes system component - the Kubelet.
|
||||
The ConfigMap is created in the `kube-system` namespace because this
|
||||
ConfigMap configures a Kubelet, which is Kubernetes system component.
|
||||
|
||||
The `--append-hash` option appends a short checksum of the ConfigMap contents
|
||||
to the name. This is convenient for an edit->push workflow, as it will
|
||||
automatically, yet deterministically, generate new names for new ConfigMaps.
|
||||
to the name. This is convenient for an edit-then-push workflow, because it
|
||||
automatically, yet deterministically, generates new names for new ConfigMaps.
|
||||
The name that includes this generated hash is referred to as `CONFIG_MAP_NAME`
|
||||
in the following examples.
|
||||
|
||||
We use the `-o yaml` output format so that the name, namespace, and uid are all
|
||||
reported following creation. We will need these in the next step. We will refer
|
||||
to the name as CONFIG_MAP_NAME and the uid as CONFIG_MAP_UID.
|
||||
|
||||
### Authorize your Node to read the new ConfigMap
|
||||
|
||||
Now that you've created a new ConfigMap, you need to authorize your node to
|
||||
read it. First, create a Role for your new ConfigMap with the
|
||||
following commands:
|
||||
|
||||
```
|
||||
$ export CONFIG_MAP_NAME=name-from-previous-output
|
||||
$ kubectl -n kube-system create role ${CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${CONFIG_MAP_NAME}
|
||||
```
|
||||
|
||||
Next, create a RoleBinding to associate your Node with the new Role:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system create rolebinding ${CONFIG_MAP_NAME}-reader --role=${CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
|
||||
```
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
|
||||
### Set the Node to use the new configuration
|
||||
#### Set the Node to use the new configuration
|
||||
|
||||
Edit the Node's reference to point to the new ConfigMap with the
|
||||
following command:
|
||||
|
||||
```
|
||||
```bash
|
||||
kubectl edit node ${NODE_NAME}
|
||||
```
|
||||
|
||||
Once in your editor, add the following YAML under `spec`:
|
||||
In your text editor, add the following YAML under `spec`:
|
||||
|
||||
```
|
||||
```yaml
|
||||
configSource:
|
||||
configMapRef:
|
||||
configMap:
|
||||
name: CONFIG_MAP_NAME
|
||||
namespace: kube-system
|
||||
uid: CONFIG_MAP_UID
|
||||
kubeletConfigKey: kubelet
|
||||
```
|
||||
|
||||
Be sure to specify all three of `name`, `namespace`, and `uid`.
|
||||
You must specify all three of `name`, `namespace`, and `kubeletConfigKey`.
|
||||
The `kubeletConfigKey` parameter shows the Kubelet which key of the ConfigMap
|
||||
contains its config.
|
||||
|
||||
### Observe that the Node begins using the new configuration
|
||||
#### Observe that the Node begins using the new configuration
|
||||
|
||||
Retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and look for the
|
||||
`KubeletConfigOK` condition in `status.conditions`. You should see the message
|
||||
`Using current (UID: CONFIG_MAP_UID)` when the Kubelet starts using the new
|
||||
configuration.
|
||||
Retrieve the Node using the `kubectl get node ${NODE_NAME} -o yaml` command and inspect
|
||||
`Node.Status.Config`. The config sources corresponding to the `active`,
|
||||
`assigned`, and `lastKnownGood` configurations are reported in the status.
|
||||
|
||||
For convenience, you can use the following command (using `jq`) to filter down
|
||||
to the `KubeletConfigOK` condition:
|
||||
- The `active` configuration is the version the Kubelet is currently running with.
|
||||
- The `assigned` configuration is the latest version the Kubelet has resolved based on
|
||||
`Node.Spec.ConfigSource`.
|
||||
- The `lastKnownGood` configuration is the version the
|
||||
Kubelet will fall back to if an invalid config is assigned in `Node.Spec.ConfigSource`.
|
||||
|
||||
The`lastKnownGood` configuration might not be present if it is set to its default value,
|
||||
the local config deployed with the node. The status will update `lastKnownGood` to
|
||||
match a valid `assigned` config after the Kubelet becomes comfortable with the config.
|
||||
The details of how the Kubelet determines a config should become the `lastKnownGood` are
|
||||
not guaranteed by the API, but is currently implemented as a 10-minute grace period.
|
||||
|
||||
You can use the following command (using `jq`) to filter down
|
||||
to the config status:
|
||||
|
||||
```bash
|
||||
kubectl get no ${NODE_NAME} -o json | jq '.status.config'
|
||||
```
|
||||
$ kubectl get no ${NODE_NAME} -o json | jq '.status.conditions|map(select(.type=="KubeletConfigOK"))'
|
||||
[
|
||||
{
|
||||
"lastHeartbeatTime": "2017-09-20T18:08:29Z",
|
||||
"lastTransitionTime": "2017-09-20T18:08:17Z",
|
||||
"message": "using current: /api/v1/namespaces/kube-system/configmaps/my-node-config-gkt4c2m4b2",
|
||||
"reason": "passing all checks",
|
||||
"status": "True",
|
||||
"type": "KubeletConfigOK"
|
||||
|
||||
The following is an example response:
|
||||
|
||||
```json
|
||||
{
|
||||
"active": {
|
||||
"configMap": {
|
||||
"kubeletConfigKey": "kubelet",
|
||||
"name": "my-node-config-9mbkccg2cc",
|
||||
"namespace": "kube-system",
|
||||
"resourceVersion": "1326",
|
||||
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
|
||||
}
|
||||
},
|
||||
"assigned": {
|
||||
"configMap": {
|
||||
"kubeletConfigKey": "kubelet",
|
||||
"name": "my-node-config-9mbkccg2cc",
|
||||
"namespace": "kube-system",
|
||||
"resourceVersion": "1326",
|
||||
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
|
||||
}
|
||||
},
|
||||
"lastKnownGood": {
|
||||
"configMap": {
|
||||
"kubeletConfigKey": "kubelet",
|
||||
"name": "my-node-config-9mbkccg2cc",
|
||||
"namespace": "kube-system",
|
||||
"resourceVersion": "1326",
|
||||
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
If something goes wrong, you may see one of several different error conditions,
|
||||
detailed in the table of KubeletConfigOK conditions, below. When this happens, you
|
||||
should check the Kubelet's log for more details.
|
||||
|
||||
### Edit the configuration file again
|
||||
|
||||
To change the configuration again, we simply repeat the above workflow.
|
||||
Try editing the `kubelet` file, changing the previously changed parameter to a
|
||||
new value.
|
||||
|
||||
### Push the newly edited configuration to the control plane
|
||||
|
||||
Push the new configuration to the control plane in a new ConfigMap with the
|
||||
following command:
|
||||
}
|
||||
|
||||
```
|
||||
$ kubectl create configmap my-node-config --namespace=kube-system --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
|
||||
```
|
||||
|
||||
This new ConfigMap will get a new name, as we have changed the contents.
|
||||
We will refer to the new name as NEW_CONFIG_MAP_NAME and the new uid
|
||||
as NEW_CONFIG_MAP_UID.
|
||||
If an error occurs, the Kubelet reports it in the `Node.Status.Config.Error`
|
||||
structure. Possible errors are listed in
|
||||
[Understanding Node.Status.Config.Error messages](#understanding-node-status-config-error-messages).
|
||||
You can search for the identical text in the Kubelet log for additional details
|
||||
and context about the error.
|
||||
|
||||
### Authorize your Node to read the new ConfigMap
|
||||
#### Make more changes
|
||||
|
||||
Now that you've created a new ConfigMap, you need to authorize your node to
|
||||
read it. First, create a Role for your new ConfigMap with the
|
||||
following commands:
|
||||
Follow the workflow above to make more changes and push them again. Each time
|
||||
you push a ConfigMap with new contents, the --append-hash kubectl option creates
|
||||
the ConfigMap with a new name. The safest rollout strategy is to first create a
|
||||
new ConfigMap, and then update the Node to use the new ConfigMap.
|
||||
|
||||
```
|
||||
$ export NEW_CONFIG_MAP_NAME=name-from-previous-output
|
||||
$ kubectl -n kube-system create role ${NEW_CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${NEW_CONFIG_MAP_NAME}
|
||||
```
|
||||
#### Reset the Node to use its local default configuration
|
||||
|
||||
Next, create a RoleBinding to associate your Node with the new Role:
|
||||
To reset the Node to use the configuration it was provisioned with, edit the
|
||||
Node using `kubectl edit node ${NODE_NAME}` and remove the
|
||||
`Node.Spec.ConfigSource` field.
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system create rolebinding ${NEW_CONFIG_MAP_NAME}-reader --role=${NEW_CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
|
||||
```
|
||||
#### Observe that the Node is using its local default configuration
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
|
||||
### Configure the Node to use the new configuration
|
||||
|
||||
Once more, edit the Node's `spec.configSource` with
|
||||
`kubectl edit node ${NODE_NAME}`. Your new `spec.configSource` should look like
|
||||
the following, with `name` and `uid` substituted as necessary:
|
||||
|
||||
```
|
||||
configSource:
|
||||
configMapRef:
|
||||
name: ${NEW_CONFIG_MAP_NAME}
|
||||
namespace: kube-system
|
||||
uid: ${NEW_CONFIG_MAP_UID}
|
||||
```
|
||||
|
||||
### Observe that the Kubelet is using the new configuration
|
||||
|
||||
Once more, retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and
|
||||
look for the `KubeletConfigOK` condition in `status.conditions`. You should see the message
|
||||
`using current: /api/v1/namespaces/kube-system/configmaps/${NEW_CONFIG_MAP_NAME}` when the Kubelet starts using the
|
||||
new configuration.
|
||||
|
||||
### Deauthorize your Node from reading the old ConfigMap
|
||||
|
||||
Once you know your Node is using the new configuration and are confident that
|
||||
the new configuration has not caused any problems, it is a good idea to
|
||||
deauthorize the node from reading the old ConfigMap. Run the following
|
||||
commands to remove the RoleBinding and Role:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete rolebinding ${CONFIG_MAP_NAME}-reader
|
||||
$ kubectl -n kube-system delete role ${CONFIG_MAP_NAME}-reader
|
||||
```
|
||||
|
||||
Note that this does not necessarily prevent the Node from reverting to the old
|
||||
configuration, as it may locally cache the old ConfigMap for an indefinite
|
||||
period of time.
|
||||
|
||||
You may optionally also choose to remove the old ConfigMap:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete configmap ${CONFIG_MAP_NAME}
|
||||
```
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
|
||||
### Reset the Node to use its local default configuration
|
||||
|
||||
Finally, if you wish to reset the Node to use the configuration it was
|
||||
provisioned with, simply edit the Node with `kubectl edit node ${NODE_NAME}` and
|
||||
remove the `spec.configSource` subfield.
|
||||
|
||||
### Observe that the Node is using its local default configuration
|
||||
|
||||
After removing this subfield, you should eventually observe that the KubeletConfigOK
|
||||
condition's message reverts to `using current: local`.
|
||||
|
||||
### Deauthorize your Node from reading the old ConfigMap
|
||||
|
||||
Once you know your Node is using the default configuration again, it is a good
|
||||
idea to deauthorize the node from reading the old ConfigMap. Run the following
|
||||
commands to remove the RoleBinding and Role:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete rolebinding ${NEW_CONFIG_MAP_NAME}-reader
|
||||
$ kubectl -n kube-system delete role ${NEW_CONFIG_MAP_NAME}-reader
|
||||
```
|
||||
|
||||
Note that this does not necessarily prevent the Node from reverting to the old
|
||||
ConfigMap, as it may locally cache the old ConfigMap for an indefinite
|
||||
period of time.
|
||||
|
||||
You may optionally also choose to remove the old ConfigMap:
|
||||
|
||||
```
|
||||
$ kubectl -n kube-system delete configmap ${NEW_CONFIG_MAP_NAME}
|
||||
```
|
||||
|
||||
Once the Node Authorizer is updated to do this automatically, you will
|
||||
be able to skip this step.
|
||||
After removing this subfield, `Node.Status.Config` eventually becomes
|
||||
empty, since all config sources have been reset to `nil`, which indicates that
|
||||
the local default config is `assigned`, `active`, and `lastKnownGood`, and no
|
||||
error is reported.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture discussion %}}
|
||||
## Kubectl Patch Example
|
||||
As mentioned above, there are many ways to change a Node's configSource.
|
||||
Here is an example command that uses `kubectl patch`:
|
||||
|
||||
```
|
||||
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMapRef\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"uid\":\"${CONFIG_MAP_UID}\"}}}}"
|
||||
You can change a Node's configSource using several different mechanisms.
|
||||
This example uses `kubectl patch`:
|
||||
|
||||
```bash
|
||||
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
|
||||
```
|
||||
|
||||
## Understanding KubeletConfigOK Conditions
|
||||
## Understanding how the Kubelet checkpoints config
|
||||
|
||||
The following table describes several of the `KubeletConfigOK` Node conditions you
|
||||
might encounter in a cluster that has Dynamic Kubelet Config enabled. If you
|
||||
observe a condition with `status=False`, you should check the Kubelet log for
|
||||
more error details by searching for the message or reason text.
|
||||
When a new config is assigned to the Node, the Kubelet downloads and unpacks the
|
||||
config payload as a set of files on the local disk. The Kubelet also records metadata
|
||||
that locally tracks the assigned and last-known-good config sources, so that the
|
||||
Kubelet knows which config to use across restarts, even if the API server becomes
|
||||
unavailable. After checkpointing a config and the relevant metadata, the Kubelet
|
||||
exits if it detects that the assigned config has changed. When the Kubelet is
|
||||
restarted by the OS-level service manager (such as `systemd`), it reads the new
|
||||
metadata and uses the new config.
|
||||
|
||||
The recorded metadata is fully resolved, meaning that it contains all necessary
|
||||
information to choose a specific config version - typically a `UID` and `ResourceVersion`.
|
||||
This is in contrast to `Node.Spec.ConfigSource`, where the intended config is declared
|
||||
via the idempotent `namespace/name` that identifies the target ConfigMap; the Kubelet
|
||||
tries to use the latest version of this ConfigMap.
|
||||
|
||||
When you are debugging problems on a node, you can inspect the Kubelet's config
|
||||
metadata and checkpoints. The structure of the Kubelet's checkpointing directory is:
|
||||
|
||||
```none
|
||||
- --dynamic-config-dir (root for managing dynamic config)
|
||||
| - meta
|
||||
| - assigned (encoded kubeletconfig/v1beta1.SerializedNodeConfigSource object, indicating the assigned config)
|
||||
| - last-known-good (encoded kubeletconfig/v1beta1.SerializedNodeConfigSource object, indicating the last-known-good config)
|
||||
| - checkpoints
|
||||
| - uid1 (dir for versions of object identified by uid1)
|
||||
| - resourceVersion1 (dir for unpacked files from resourceVersion1 of object with uid1)
|
||||
| - ...
|
||||
| - ...
|
||||
```
|
||||
|
||||
## Understanding Node.Status.Config.Error messages
|
||||
|
||||
The following table describes error messages that can occur
|
||||
when using Dynamic Kubelet Config. You can search for the identical text
|
||||
in the Kubelet log for additional details and context about the error.
|
||||
|
||||
<table>
|
||||
|
||||
|
||||
<table align="left">
|
||||
<tr>
|
||||
<th>Possible Messages</th>
|
||||
<th>Possible Reasons</th>
|
||||
<th>Status</th>
|
||||
<th>Error Message</th>
|
||||
<th>Possible Causes</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using current: local</p></td>
|
||||
<td><p>when the config source is nil, the Kubelet uses its local config</p></td>
|
||||
<td><p>True</p></td>
|
||||
<td><p>failed to load config, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet likely could not parse the downloaded config payload, or encountered a filesystem error attempting to load the payload from disk.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</p></td>
|
||||
<td><p>passing all checks</p></td>
|
||||
<td><p>True</p></td>
|
||||
<td><p>failed to validate config, see Kubelet log for details</p></td>
|
||||
<td><p>The configuration in the payload, combined with any command-line flag overrides, and the sum of feature gates from flags, the config file, and the remote payload, was determined to be invalid by the Kubelet.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using last-known-good: local</p></td>
|
||||
<td>
|
||||
<ul>
|
||||
<li>failed to load current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to parse current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to validate current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
</ul>
|
||||
</td>
|
||||
<td><p>False</p></td>
|
||||
<td><p>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</p></td>
|
||||
<td><p>Since Node.Spec.ConfigSource is validated by the API server to contain at least one non-nil subfield, this likely means that the Kubelet is older than the API server and does not recognize a newer source type.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>using last-known-good: /api/v1/namespaces/${LAST_KNOWN_GOOD_CONFIG_MAP_NAMESPACE}/configmaps/${LAST_KNOWN_GOOD_CONFIG_MAP_NAME}</p></td>
|
||||
<td>
|
||||
<ul>
|
||||
<li>failed to load current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to parse current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
<li>failed to validate current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
|
||||
</ul>
|
||||
</td>
|
||||
<td><p>False</p></td>
|
||||
<td><p>failed to sync: failed to download config, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet could not download the config. It is possible that Node.Spec.ConfigSource could not be resolved to a concrete API object, or that network errors disrupted the download attempt. The Kubelet will retry the download when in this error state.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<p>
|
||||
The reasons in the next column could potentially appear for any of
|
||||
the above messages.
|
||||
</p>
|
||||
<p>
|
||||
This condition indicates that the Kubelet is having trouble
|
||||
reconciling `spec.configSource`, and thus no change to the in-use
|
||||
configuration has occurred.
|
||||
</p>
|
||||
<p>
|
||||
The "failed to sync" reasons are specific to the failure that
|
||||
occurred, and the next column does not necessarily contain all
|
||||
possible failure reasons.
|
||||
</p>
|
||||
</td>
|
||||
<td>
|
||||
<p>failed to sync, reason:</p>
|
||||
<ul>
|
||||
<li>failed to read Node from informer object cache</li>
|
||||
<li>failed to reset to local config</li>
|
||||
<li>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</li>
|
||||
<li>invalid ObjectReference, all of UID, Name, and Namespace must be specified</li>
|
||||
<li>invalid ConfigSource.ConfigMapRef.UID: ${UID} does not match ${API_PATH}.UID: ${UID_OF_CONFIG_MAP_AT_API_PATH}</li>
|
||||
<li>failed to determine whether object ${API_PATH} with UID ${UID} was already checkpointed</li>
|
||||
<li>failed to download ConfigMap with name ${NAME} from namespace ${NAMESPACE}</li>
|
||||
<li>failed to save config checkpoint for object ${API_PATH} with UID ${UID}</li>
|
||||
<li>failed to set current config checkpoint to local config</li>
|
||||
<li>failed to set current config checkpoint to object ${API_PATH} with UID ${UID}</li>
|
||||
</ul>
|
||||
</td>
|
||||
<td><p>False</p></td>
|
||||
<td><p>failed to sync: internal failure, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet encountered some internal problem and failed to update its config as a result. Examples include filesystem errors and reading objects from the internal informer cache.</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><p>internal failure, see Kubelet log for details</p></td>
|
||||
<td><p>The Kubelet encountered some internal problem while manipulating config, outside of the configuration sync loop.</p></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ content_template: templates/concept
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< feature-state state="alpha" >}}
|
||||
{{< feature-state state="beta" >}}
|
||||
|
||||
Kubernetes v1.6 introduced a new binary called `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops. These cloud-specific control loops were originally in the `kube-controller-manager`. Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud vendors to evolve independently from the core Kubernetes code.
|
||||
|
||||
@@ -37,7 +37,8 @@ Successfully running cloud-controller-manager requires some changes to your clus
|
||||
|
||||
* `kube-apiserver` and `kube-controller-manager` MUST NOT specify the `--cloud-provider` flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.
|
||||
* `kubelet` must run with `--cloud-provider=external`. This is to ensure that the kubelet is aware that it must be initialized by the cloud controller manager before it is scheduled any work.
|
||||
* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller since the cloud controller manager takes over labeling persistent volumes. To prevent the PersistentVolumeLabel admission plugin from running in `kube-apiserver`, include the `PersistentVolumeLabel` as a listed value in the `--disable-admission-plugins` flag.
|
||||
* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller
|
||||
since the cloud controller manager takes over labeling persistent volumes.
|
||||
* For the `cloud-controller-manager` to label persistent volumes, initializers will need to be enabled and an InitializerConifguration needs to be added to the system. Follow [these instructions](/docs/admin/extensible-admission-controllers.md#enable-initializers-alpha-feature) to enable initializers. Use the following YAML to create the InitializerConfiguration:
|
||||
|
||||
{{< code file="persistent-volume-label-initializer-config.yaml" >}}
|
||||
@@ -53,7 +54,7 @@ As of v1.8, cloud controller manager can implement:
|
||||
* node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
|
||||
* service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
|
||||
* route controller - responsible for setting up network routes on your cloud
|
||||
* [PersistentVolumeLabel Admission Controller](/docs/admin/admission-controllers#persistentvolumelabel) - responsible for labeling persistent volumes on your cloud - ensure that the persistent volume label admission plugin is not enabled on your kube-apiserver.
|
||||
* persistent valume labels controller - responsible for setting the zone and region labels on PersistentVolumes created in GCP and AWS clouds.
|
||||
* any other features you would like to implement if you are running an out-of-tree provider.
|
||||
|
||||
|
||||
|
||||
@@ -7,15 +7,16 @@ content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state for_k8s_version="v1.10" state="beta" >}}
|
||||
|
||||
Persistent volume claims (PVCs) that are in active use by a pod and persistent volumes (PVs) that are bound to PVCs can be protected from pre-mature removal.
|
||||
Persistent volume claims (PVCs) that are in active use by a pod and persistent volumes (PVs) that are bound to PVCs can be protected from premature removal.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- The Storage Object in Use Protection feature is enabled in a version of Kubernetes in which it is supported.
|
||||
The Storage Object in Use Protection feature is enabled in one of the below Kubernetes versions:
|
||||
- {% assign for_k8s_version = "1.10" %} {% include feature-state-beta.md %}
|
||||
- {% assign for_k8s_version = "1.11" %} {% include feature-state-stable.md %}
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -312,5 +313,3 @@ No resources found.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
---
|
||||
title: Using Sysctls in a Kubernetes Cluster
|
||||
title: Using sysctls in a Kubernetes Cluster
|
||||
reviewers:
|
||||
- sttts
|
||||
content_template: templates/task
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< feature-state for_k8s_version="v1.11" state="beta" >}}
|
||||
|
||||
This document describes how sysctls are used within a Kubernetes cluster.
|
||||
This document describes how to configure and use kernel parameters within a
|
||||
Kubernetes cluster using the sysctl interface.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -74,7 +76,7 @@ application tuning. _Unsafe_ sysctls are enabled on a node-by-node basis with a
|
||||
flag of the kubelet, e.g.:
|
||||
|
||||
```shell
|
||||
$ kubelet --experimental-allowed-unsafe-sysctls \
|
||||
$ kubelet --allowed-unsafe-sysctls \
|
||||
'kernel.msg*,net.ipv4.route.min_pmtu' ...
|
||||
```
|
||||
|
||||
@@ -89,10 +91,11 @@ Only _namespaced_ sysctls can be enabled this way.
|
||||
## Setting Sysctls for a Pod
|
||||
|
||||
A number of sysctls are _namespaced_ in today's Linux kernels. This means that
|
||||
they can be set independently for each pod on a node. Being namespaced is a
|
||||
requirement for sysctls to be accessible in a pod context within Kubernetes.
|
||||
they can be set independently for each pod on a node. Only namespaced sysctls
|
||||
are configurable via the pod securityContext within Kubernetes.
|
||||
|
||||
The following sysctls are known to be _namespaced_:
|
||||
The following sysctls are known to be namespaced. This list could change
|
||||
in future versions of the Linux kernel.
|
||||
|
||||
- `kernel.shm*`,
|
||||
- `kernel.msg*`,
|
||||
@@ -100,25 +103,37 @@ The following sysctls are known to be _namespaced_:
|
||||
- `fs.mqueue.*`,
|
||||
- `net.*`.
|
||||
|
||||
Sysctls which are not namespaced are called _node-level_ and must be set
|
||||
manually by the cluster admin, either by means of the underlying Linux
|
||||
distribution of the nodes (e.g. via `/etc/sysctls.conf`) or using a DaemonSet
|
||||
with privileged containers.
|
||||
Sysctls with no namespace are called _node-level_ sysctls. If you need to set
|
||||
them, you must manually configure them on each node's operating system, or by
|
||||
using a DaemonSet with privileged containers.
|
||||
|
||||
The sysctl feature is an alpha API. Therefore, sysctls are set using annotations
|
||||
on pods. They apply to all containers in the same pod.
|
||||
Use the pod securityContext to configure namespaced sysctls. The securityContext
|
||||
applies to all containers in the same pod.
|
||||
|
||||
Here is an example, with different annotations for _safe_ and _unsafe_ sysctls:
|
||||
This example uses the pod securityContext to set a safe sysctl
|
||||
`kernel.shm_rmid_forced` and two unsafe sysctls `net.ipv4.route.min_pmtu` and
|
||||
`kernel.msgmax` There is no distinction between _safe_ and _unsafe_ sysctls in
|
||||
the specification.
|
||||
|
||||
{{< warning >}}
|
||||
Only modify sysctl parameters after you understand their effects, to avoid
|
||||
destabilizing your operating system.
|
||||
{{< /warning >}}
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: sysctl-example
|
||||
annotations:
|
||||
security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1
|
||||
security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.route.min_pmtu=1000,kernel.msgmax=1 2 3
|
||||
spec:
|
||||
securityContext:
|
||||
sysctls:
|
||||
- name: kernel.shm_rmid_forced
|
||||
value: "0"
|
||||
- name: net.ipv4.route.min_pmtu
|
||||
value: "552"
|
||||
- name: kernel.msgmax
|
||||
value: "65536"
|
||||
...
|
||||
```
|
||||
{{% /capture %}}
|
||||
@@ -143,27 +158,52 @@ is recommended to use
|
||||
[taints on nodes](/docs/concepts/configuration/taint-and-toleration/)
|
||||
to schedule those pods onto the right nodes.
|
||||
|
||||
## PodSecurityPolicy Annotations
|
||||
## PodSecurityPolicy
|
||||
|
||||
The use of sysctl in pods can be controlled via annotation on the PodSecurityPolicy.
|
||||
You can further control which sysctls can be set in pods by specifying lists of
|
||||
sysctls or sysctl patterns in the `forbiddenSysctls` and/or
|
||||
`allowedUnsafeSysctls` fields of the PodSecurityPolicy. A sysctl pattern ends
|
||||
with a `*` character, such as `kernel.*`. A `*` character on its own matches
|
||||
all sysctls.
|
||||
|
||||
Sysctl annotation represents a whitelist of allowed safe and unsafe sysctls
|
||||
in a pod spec. It's a comma-separated list of plain sysctl names or sysctl patterns
|
||||
(which end in `*`). The string `*` matches all sysctls.
|
||||
By default, all safe sysctls are allowed.
|
||||
|
||||
Here is an example, it authorizes binding user creating pod with corresponding sysctls.
|
||||
Both `forbiddenSysctls` and `allowedUnsafeSysctls` are lists of plain sysctl names
|
||||
or sysctl patterns (which end with `*`). The string `*` matches all sysctls.
|
||||
|
||||
The `forbiddenSysctls` field excludes specific sysctls. You can forbid a
|
||||
combination of safe and unsafe sysctls in the list. To forbid setting any
|
||||
sysctls, use `*` on its own.
|
||||
|
||||
If you specify any unsafe sysctl in the `allowedUnsafeSysctls` field and it is
|
||||
not present in the `forbiddenSysctls` field, that sysctl can be used in Pods
|
||||
using this PodSecurityPolicy. To allow all unsafe sysctls in the
|
||||
PodSecurityPolicy to be set, use `*` on its own.
|
||||
|
||||
Do not configure these two fields such that there is overlap, meaning that a
|
||||
given sysctl is both allowed and forbidden.
|
||||
|
||||
{{< warning >}}
|
||||
**Warning**: If you whitelist unsafe sysctls via the `allowedUnsafeSysctls` field
|
||||
in a PodSecurityPolicy, any pod using such a sysctl will fail to start
|
||||
if the sysctl is not whitelisted via the `--allowed-unsafe-sysctls` kubelet
|
||||
flag as well on that node.
|
||||
{{< /warning >}}
|
||||
|
||||
This example allows unsafe sysctls prefixed with `kernel.msg` to be set and
|
||||
disallows setting of the `kernel.shm_rmid_forced` sysctl.
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: sysctl-psp
|
||||
annotations:
|
||||
security.alpha.kubernetes.io/sysctls: 'net.ipv4.route.*,kernel.msg*'
|
||||
spec:
|
||||
allowedUnsafeSysctls:
|
||||
- kernel.msg*
|
||||
forbiddenSysctls:
|
||||
- kernel.shm_rmid_forced
|
||||
...
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user