whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/main' into dev-1.24

Browse Source
This commit is contained in:
Nate W
2022-04-07 10:41:49 -07:00
parent a8b46072c3 d8dfd81d0c
commit d1bce4cb0d
148 changed files with 3268 additions and 1461 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md
+133
View File
@@ -0,0 +1,133 @@
---
layout: blog
title: "Kubernetes Removals and Deprecations In 1.24"
date: 2022-04-07
slug: upcoming-changes-in-kubernetes-1-24
---
**Author**: Mickey Boxell (Oracle)
As Kubernetes evolves, features and APIs are regularly revisited and removed. New features may offer
an alternative or improved approach to solving existing problems, motivating the team to remove the
old approach.
We want to make sure you are aware of the changes coming in the Kubernetes 1.24 release. The release will
**deprecate** several (beta) APIs in favor of stable versions of the same APIs. The major change coming
in the Kubernetes 1.24 release is the
[removal of Dockershim](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim).
This is discussed below and will be explored in more depth at release time. For an early look at the
changes coming in Kubernetes 1.24, take a look at the in-progress
[CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md).
## A note about Dockershim
It's safe to say that the removal receiving the most attention with the release of Kubernetes 1.24
is Dockershim. Dockershim was deprecated in v1.20. As noted in the [Kubernetes 1.20 changelog](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#deprecation):
"Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet
uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance
issues in the Kubernetes community." With the upcoming release of Kubernetes 1.24, the Dockershim will
finally be removed.
In the article [Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/),
the authors succinctly captured the change's impact and encouraged users to remain calm:
> Docker as an underlying runtime is being deprecated in favor of runtimes that use the
> Container Runtime Interface (CRI) created for Kubernetes. Docker-produced images
> will continue to work in your cluster with all runtimes, as they always have.
Several guides have been created with helpful information about migrating from dockershim
to container runtimes that are directly compatible with Kubernetes. You can find them on the
[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/)
page in the Kubernetes documentation.
For more information about why Kubernetes is moving away from dockershim, check out the aptly
named: [Kubernetes is Moving on From Dockershim](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/)
and the [updated dockershim removal FAQ](/blog/2022/02/17/dockershim-faq/).
Take a look at the [Is Your Cluster Ready for v1.24?](/blog/2022/03/31/ready-for-dockershim-removal/) post to learn about how to ensure your cluster continues to work after upgrading from v1.23 to v1.24.
## The Kubernetes API removal and deprecation process
Kubernetes contains a large number of components that evolve over time. In some cases, this
evolution results in APIs, flags, or entire features, being removed. To prevent users from facing
breaking changes, Kubernetes contributors adopted a feature [deprecation policy](/docs/reference/using-api/deprecation-policy/).
This policy ensures that stable APIs may only be deprecated when a newer stable version of that
same API is available and that APIs have a minimum lifetime as indicated by the following stability levels:
* Generally available (GA) or stable API versions may be marked as deprecated but must not be removed within a major version of Kubernetes.
* Beta or pre-release API versions must be supported for 3 releases after deprecation.
* Alpha or experimental API versions may be removed in any release without prior deprecation notice.
Removals follow the same deprecation policy regardless of whether an API is removed due to a beta feature
graduating to stable or because that API was not proven to be successful. Kubernetes will continue to make
sure migration options are documented whenever APIs are removed.
**Deprecated** APIs are those that have been marked for removal in a future Kubernetes release. **Removed**
APIs are those that are no longer available for use in current, supported Kubernetes versions after having
been deprecated. These removals have been superseded by newer, stable/generally available (GA) APIs.
## API removals, deprecations, and other changes for Kubernetes 1.24
* [Dynamic kubelet configuration](https://github.com/kubernetes/enhancements/issues/281): `DynamicKubeletConfig` is used to enable the dynamic configuration of the kubelet. The `DynamicKubeletConfig` flag was deprecated in Kubernetes 1.22. In v1.24, this feature gate will be removed from the kubelet. See [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). Refer to the ["Dynamic kubelet config is removed" KEP](https://github.com/kubernetes/enhancements/issues/281) for more information.
* [Dynamic log sanitization](https://github.com/kubernetes/kubernetes/pull/107207): The experimental dynamic log sanitization feature is deprecated and will be removed in v1.24. This feature introduced a logging filter that could be applied to all Kubernetes system components logs to prevent various types of sensitive information from leaking via logs. Refer to [KEP-1753: Kubernetes system components logs sanitization](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#deprecation) for more information and an [alternative approach](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#alternatives=).
* In-tree provisioner to CSI driver migration: This applies to a number of in-tree plugins, including [Portworx](https://github.com/kubernetes/enhancements/issues/2589). Refer to the [In-tree Storage Plugin to CSI Migration Design Doc](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/csi-migration.md#background-and-motivations) for more information.
* [Removing Dockershim from kubelet](https://github.com/kubernetes/enhancements/issues/2221): the Container Runtime Interface (CRI) for Docker (i.e. Dockershim) is currently a built-in container runtime in the kubelet code base. It was deprecated in v1.20. As of v1.24, the kubelet will no longer have dockershim. Check out this blog on [what you need to do be ready for v1.24](/blog/2022/03/31/ready-for-dockershim-removal/).
* [Storage capacity tracking for pod scheduling](https://github.com/kubernetes/enhancements/issues/1472): The CSIStorageCapacity API supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances scheduling of pods that use CSI volumes with late binding. In v1.24, the CSIStorageCapacity API will be stable. The API graduating to stable initates the deprecation of the v1beta1 CSIStorageCapacity API. Refer to the [Storage Capacity Constraints for Pod Scheduling KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1472-storage-capacity-tracking) for more information.
* [The `master` label is no longer present on kubeadm control plane nodes](https://github.com/kubernetes/kubernetes/pull/107533). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to [KEP-2067: Rename the kubeadm "master" label and taint](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint).
* [VolumeSnapshot v1beta1 CRD will be removed](https://github.com/kubernetes/enhancements/issues/177). Volume snapshot and restore functionality for Kubernetes and the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, entered beta in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.21 and is now unsupported. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v4.1.0) for more information.
## What to do
### Dockershim removal
As stated earlier, there are several guides about
[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/).
You can start with [Finding what container runtime are on your nodes](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/).
If your nodes are using dockershim, there are other possible Docker Engine dependencies such as
Pods or third-party tools executing Docker commands or private registries in the Docker configuration file. You can follow the
[Check whether Dockershim deprecation affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/) guide to review possible
Docker Engine dependencies. Before upgrading to v1.24, you decide to either remain using Docker Engine and
[Migrate Docker Engine nodes from dockershim to cri-dockerd](/docs/tasks/administer-cluster/migrating-from-dockershim/migrate-dockershim-dockerd/) or migrate to a CRI-compatible runtime. Here's a guide to
[change the container runtime on a node from Docker Engine to containerd](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/).
### `kubectl convert`
The [`kubectl convert`](/docs/tasks/tools/included/kubectl-convert-overview/) plugin for `kubectl`
can be helpful to address migrating off deprecated APIs. The plugin facilitates the conversion of
manifests between different API versions, for example, from a deprecated to a non-deprecated API
version. More general information about the API migration process can be found in the [Deprecated API Migration Guide](/docs/reference/using-api/deprecation-guide/).
Follow the [install `kubectl convert` plugin](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-convert-plugin)
documentation to download and install the `kubectl-convert` binary.
### Looking ahead
The Kubernetes 1.25 and 1.26 releases planned for later this year will stop serving beta versions
of several currently stable Kubernetes APIs. The v1.25 release will also remove PodSecurityPolicy,
which was deprecated with Kubernetes 1.21 and will not graduate to stable. See [PodSecurityPolicy
Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) for more information.
The official [list of API removals planned for Kubernetes 1.25](/docs/reference/using-api/deprecation-guide/#v1-25) is:
* The beta CronJob API (batch/v1beta1)
* The beta EndpointSlice API (discovery.k8s.io/v1beta1)
* The beta Event API (events.k8s.io/v1beta1)
* The beta HorizontalPodAutoscaler API (autoscaling/v2beta1)
* The beta PodDisruptionBudget API (policy/v1beta1)
* The beta PodSecurityPolicy API (policy/v1beta1)
* The beta RuntimeClass API (node.k8s.io/v1beta1)
The official [list of API removals planned for Kubernetes 1.26](/docs/reference/using-api/deprecation-guide/#v1-26) is:
* The beta FlowSchema and PriorityLevelConfiguration APIs (flowcontrol.apiserver.k8s.io/v1beta1)
* The beta HorizontalPodAutoscaler API (autoscaling/v2beta2)
### Want to know more?
Deprecations are announced in the Kubernetes release notes. You can see the announcements of pending deprecations in the release notes for:
* [Kubernetes 1.21](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#deprecation)
* [Kubernetes 1.22](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#deprecation)
* [Kubernetes 1.23](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#deprecation)
* We will formally announce the deprecations that come with [Kubernetes 1.24](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#deprecation) as part of the CHANGELOG for that release.
For information on the process of deprecation and removal, check out the official Kubernetes [deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api) document.
content/en/docs/concepts/architecture/nodes.md
+3
View File
@@ -539,6 +539,9 @@ Using this feature, requires enabling the
config's `ShutdownGracePeriodByPodPriority` to the desired configuration
containing the pod priority class values and their respective shutdown periods.
Metrics `graceful_shutdown_start_time_seconds` and `graceful_shutdown_end_time_seconds`
are emitted under the kubelet subsystem to monitor node shutdowns.
## Swap memory management {#swap-memory}
{{< feature-state state="alpha" for_k8s_version="v1.22" >}}
content/en/docs/concepts/scheduling-eviction/assign-pod-node.md
+2 -2
View File
@@ -124,8 +124,8 @@ For example, consider the following Pod spec:
In this example, the following rules apply:
* The node *must* have a label with the key `kubernetes.io/e2e-az-name` and
the value is either `e2e-az1` or `e2e-az2`.
* The node *must* have a label with the key `kubernetes.io/os` and
the value `linux`.
* The node *preferably* has a label with the key `another-node-label-key` and
the value `another-node-label-value`.
content/en/docs/concepts/services-networking/service.md
+1 -1
View File
@@ -25,7 +25,7 @@ and can load-balance across them.
## Motivation
Kubernetes {{< glossary_tooltip term_id="pod" text="Pods" >}} are created and destroyed
to match the state of your cluster. Pods are nonpermanent resources.
to match the desired state of your cluster. Pods are nonpermanent resources.
If you use a {{< glossary_tooltip term_id="deployment" >}} to run your app,
it can create and destroy Pods dynamically.
content/en/docs/concepts/storage/volumes.md
+14
View File
@@ -151,6 +151,13 @@ Storage Interface (CSI) Driver. In order to use this feature, the
[Azure Disk CSI Driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver)
must be installed on the cluster and the `CSIMigration` feature must be enabled.
#### azureDisk CSI migration complete
{{< feature-state for_k8s_version="v1.21" state="alpha" >}}
To disable the `azureDisk` storage plugin from being loaded by the controller manager
and the kubelet, set the `InTreePluginAzureDiskUnregister` flag to `true`.
### azureFile {#azurefile}
The `azureFile` volume type mounts a Microsoft Azure File volume (SMB 2.1 and 3.0)
@@ -172,6 +179,13 @@ must be installed on the cluster and the `CSIMigration` and `CSIMigrationAzureFi
Azure File CSI driver does not support using same volume with different fsgroups. If
`CSIMigrationAzureFile` is enabled, using same volume with different fsgroups won't be supported at all.
#### azureFile CSI migration complete
{{< feature-state for_k8s_version="v1.21" state="alpha" >}}
To disable the `azureFile` storage plugin from being loaded by the controller manager
and the kubelet, set the `InTreePluginAzureFileUnregister` flag to `true`.
### cephfs
A `cephfs` volume allows an existing CephFS volume to be
content/en/docs/concepts/workloads/controllers/replicaset.md
-2
View File
@@ -223,8 +223,6 @@ In this manner, a ReplicaSet can own a non-homogenous set of Pods
As with all other Kubernetes API objects, a ReplicaSet needs the `apiVersion`, `kind`, and `metadata` fields.
For ReplicaSets, the `kind` is always a ReplicaSet.
In Kubernetes 1.9 the API version `apps/v1` on the ReplicaSet kind is the current version and is enabled by default. The API version `apps/v1beta2` is deprecated.
Refer to the first lines of the `frontend.yaml` example for guidance.
The name of a ReplicaSet object must be a valid
[DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names).
content/en/docs/concepts/workloads/controllers/statefulset.md
+2 -2
View File
@@ -115,7 +115,7 @@ The name of a StatefulSet object must be a valid
### Pod Selector
You must set the `.spec.selector` field of a StatefulSet to match the labels of its `.spec.template.metadata.labels`. In 1.8 and later versions, failing to specify a matching Pod Selector will result in a validation error during StatefulSet creation.
You must set the `.spec.selector` field of a StatefulSet to match the labels of its `.spec.template.metadata.labels`. Failing to specify a matching Pod Selector will result in a validation error during StatefulSet creation.
### Volume Claim Templates
@@ -226,7 +226,7 @@ is completely shutdown, but prior to web-1's termination, web-1 would not be ter
until web-0 is Running and Ready.
### Pod Management Policies
In Kubernetes 1.7 and later, StatefulSet allows you to relax its ordering guarantees while
StatefulSet allows you to relax its ordering guarantees while
preserving its uniqueness and identity guarantees via its `.spec.podManagementPolicy` field.
#### OrderedReady Pod Management
content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md
+4 -2
View File
@@ -300,7 +300,8 @@ apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- pluginConfig:
- schedulerName: default-scheduler
pluginConfig:
- name: PodTopologySpread
args:
defaultConstraints:
@@ -356,7 +357,8 @@ apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- pluginConfig:
- schedulerName: default-scheduler
pluginConfig:
- name: PodTopologySpread
args:
defaultConstraints: []
content/en/docs/contribute/localization.md
+5 -2
View File
@@ -125,11 +125,14 @@ Add a configuration block for the new language to `config.toml`, under the exist
[languages.de]
title = "Kubernetes"
description = "Produktionsreife Container-Verwaltung"
languageName = "Deutsch"
languageName = "Deutsch (German)"
languageNameLatinScript = "German"
contentDir = "content/de"
weight = 3
weight = 8
```
The value for `languageName` will be listed in language selection bar. Assign "language name in native script (language name in latin script)" to `languageName`, for example, `languageName = "한국어 (Korean)"`. `languageNameLatinScript` can be used to access the language name in latin script and use it in the theme. Assign "language name in latin script" to `languageNameLatinScript`, for example, `languageNameLatinScript ="Korean"`.
When assigning a `weight` parameter for your block, find the language block with the highest weight and add 1 to that value.
For more information about Hugo's multilingual support, see "[Multilingual Mode](https://gohugo.io/content-management/multilingual/)".
content/en/docs/reference/glossary/pod-security-policy.md
+1
View File
@@ -17,3 +17,4 @@ tags:
A cluster-level resource that controls security sensitive aspects of the Pod specification. The `PodSecurityPolicy` objects define a set of conditions that a Pod must run with in order to be accepted into the system, as well as defaults for the related fields. Pod Security Policy control is implemented as an optional admission controller.
PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25. We recommend migrating to [Pod Security Admission](/docs/concepts/security/pod-security-admission/), or a 3rd party admission plugin.
content/en/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd.md
+35 -24
View File
@@ -4,20 +4,28 @@ weight: 8
content_type: task
---
This task outlines the steps needed to update your container runtime to containerd from Docker. It is applicable for cluster operators running Kubernetes 1.23 or earlier. Also this covers an example scenario for migrating from dockershim to containerd and alternative container runtimes can be picked from this [page](https://kubernetes.io/docs/setup/production-environment/container-runtimes/).
This task outlines the steps needed to update your container runtime to containerd from Docker. It
is applicable for cluster operators running Kubernetes 1.23 or earlier. Also this covers an
example scenario for migrating from dockershim to containerd and alternative container runtimes
can be picked from this [page](/docs/setup/production-environment/container-runtimes/).
## {{% heading "prerequisites" %}}
{{% thirdparty-content %}}
Install containerd. For more information see, [containerd's installation documentation](https://containerd.io/docs/getting-started/) and for specific prerequisite follow [this](/docs/setup/production-environment/container-runtimes/#containerd).
Install containerd. For more information see
[containerd's installation documentation](https://containerd.io/docs/getting-started/)
and for specific prerequisite follow
[the containerd guide](/docs/setup/production-environment/container-runtimes/#containerd).
## Drain the node
```
# replace <node-to-drain> with the name of your node you are draining
```shell
kubectl drain <node-to-drain> --ignore-daemonsets
```
Replace `<node-to-drain>` with the name of your node you are draining.
## Stop the Docker daemon
```shell
@@ -27,32 +35,33 @@ systemctl disable docker.service --now
## Install Containerd
This [page](/docs/setup/production-environment/container-runtimes/#containerd) contains detailed steps to install containerd.
Follow the [guide](/docs/setup/production-environment/container-runtimes/#containerd)
for detailed steps to install containerd.
{{< tabs name="tab-cri-containerd-installation" >}}
{{% tab name="Linux" %}}
1. Install the `containerd.io` package from the official Docker repositories.
Instructions for setting up the Docker repository for your respective Linux distribution and installing the `containerd.io` package can be found at
[Install Docker Engine](https://docs.docker.com/engine/install/#server).
Instructions for setting up the Docker repository for your respective Linux distribution and
installing the `containerd.io` package can be found at
[Install Docker Engine](https://docs.docker.com/engine/install/#server).
2. Configure containerd:
1. Configure containerd:
```shell
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
```
3. Restart containerd:
1. Restart containerd:
```shell
sudo systemctl restart containerd
```
{{% /tab %}}
{{% tab name="Windows (PowerShell)" %}}
Start a Powershell session, set `$Version` to the desired version (ex: `$Version="1.4.3"`), and then run the following commands:
Start a Powershell session, set `$Version` to the desired version (ex: `$Version="1.4.3"`), and
then run the following commands:
1. Download containerd:
@@ -89,24 +98,25 @@ Start a Powershell session, set `$Version` to the desired version (ex: `$Version
## Configure the kubelet to use containerd as its container runtime
Edit the file `/var/lib/kubelet/kubeadm-flags.env` and add the containerd runtime to the flags. `--container-runtime=remote` and `--container-runtime-endpoint=unix:///run/containerd/containerd.sock"`
Edit the file `/var/lib/kubelet/kubeadm-flags.env` and add the containerd runtime to the flags.
`--container-runtime=remote` and
`--container-runtime-endpoint=unix:///run/containerd/containerd.sock"`.
For users using kubeadm should consider the following:
Users using kubeadm should be aware that the `kubeadm` tool stores the CRI socket for each host as
an annotation in the Node object for that host. To change it you can execute the following command
on a machine that has the kubeadm `/etc/kubernetes/admin.conf` file.
The `kubeadm` tool stores the CRI socket for each host as an annotation in the Node object for that host.
To change it you must do the following:
Execute `kubectl edit no <NODE-NAME>` on a machine that has the kubeadm `/etc/kubernetes/admin.conf` file.
```shell
kubectl edit no <node-name>
```
This will start a text editor where you can edit the Node object.
To choose a text editor you can set the `KUBE_EDITOR` environment variable.
- Change the value of `kubeadm.alpha.kubernetes.io/cri-socket` from `/var/run/dockershim.sock`
to the CRI socket path of your choice (for example `unix:///run/containerd/containerd.sock`).
to the CRI socket path of your choice (for example `unix:///run/containerd/containerd.sock`).
Note that new CRI socket paths must be prefixed with `unix://` ideally.
Note that new CRI socket paths must be prefixed with `unix://` ideally.
- Save the changes in the text editor, which will update the Node object.
@@ -124,7 +134,7 @@ Run `kubectl get nodes -o wide` and containerd appears as the runtime for the no
{{% thirdparty-content %}}
Finally if everything goes well remove docker
Finally if everything goes well, remove Docker.
{{< tabs name="tab-remove-docker-enigine" >}}
{{% tab name="CentOS" %}}
@@ -151,4 +161,5 @@ sudo dnf remove docker-ce docker-ce-cli
sudo apt-get purge docker-ce docker-ce-cli
```
{{% /tab %}}
{{< /tabs >}}
{{< /tabs >}}
content/en/docs/tasks/administer-cluster/nodelocaldns.md
+82 -39
View File
@@ -8,36 +8,49 @@ content_type: task
---
<!-- overview -->
{{< feature-state for_k8s_version="v1.18" state="stable" >}}
This page provides an overview of NodeLocal DNSCache feature in Kubernetes.
## {{% heading "prerequisites" %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
<!-- steps -->
## Introduction
NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet. In today's architecture, Pods in ClusterFirst DNS mode reach out to a kube-dns serviceIP for DNS queries. This is translated to a kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy. With this new architecture, Pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking. The local caching agent will query kube-dns service for cache misses of cluster hostnames(cluster.local suffix by default).
NodeLocal DNSCache improves Cluster DNS performance by running a DNS caching agent
on cluster nodes as a DaemonSet. In today's architecture, Pods in 'ClusterFirst' DNS mode
reach out to a kube-dns `serviceIP` for DNS queries. This is translated to a
kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy.
With this new architecture, Pods will reach out to the DNS caching agent
running on the same node, thereby avoiding iptables DNAT rules and connection tracking.
The local caching agent will query kube-dns service for cache misses of cluster
hostnames ("`cluster.local`" suffix by default).
## Motivation
* With the current DNS architecture, it is possible that Pods with the highest DNS QPS have to reach out to a different node, if there is no local kube-dns/CoreDNS instance.
Having a local cache will help improve the latency in such scenarios.
* With the current DNS architecture, it is possible that Pods with the highest DNS QPS
have to reach out to a different node, if there is no local kube-dns/CoreDNS instance.
Having a local cache will help improve the latency in such scenarios.
* Skipping iptables DNAT and connection tracking will help reduce [conntrack races](https://github.com/kubernetes/kubernetes/issues/56903) and avoid UDP DNS entries filling up conntrack table.
* Skipping iptables DNAT and connection tracking will help reduce
[conntrack races](https://github.com/kubernetes/kubernetes/issues/56903)
and avoid UDP DNS entries filling up conntrack table.
* Connections from local caching agent to kube-dns service can be upgraded to TCP. TCP conntrack entries will be removed on connection close in contrast with UDP entries that have to timeout ([default](https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt) `nf_conntrack_udp_timeout` is 30 seconds)
* Connections from local caching agent to kube-dns service can be upgraded to TCP.
TCP conntrack entries will be removed on connection close in contrast with
UDP entries that have to timeout
([default](https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt)
`nf_conntrack_udp_timeout` is 30 seconds)
* Upgrading DNS queries from UDP to TCP would reduce tail latency attributed to dropped UDP packets and DNS timeouts usually up to 30s (3 retries + 10s timeout). Since the nodelocal cache listens for UDP DNS queries, applications don't need to be changed.
* Upgrading DNS queries from UDP to TCP would reduce tail latency attributed to
dropped UDP packets and DNS timeouts usually up to 30s (3 retries + 10s timeout).
Since the nodelocal cache listens for UDP DNS queries, applications don't need to be changed.
* Metrics & visibility into dns requests at a node level.
* Metrics & visibility into DNS requests at a node level.
* Negative caching can be re-enabled, thereby reducing number of queries to kube-dns service.
@@ -49,48 +62,75 @@ This is the path followed by DNS Queries after NodeLocal DNSCache is enabled:
{{< figure src="/images/docs/nodelocaldns.svg" alt="NodeLocal DNSCache flow" title="Nodelocal DNSCache flow" caption="This image shows how NodeLocal DNSCache handles DNS queries." class="diagram-medium" >}}
## Configuration
{{< note >}} The local listen IP address for NodeLocal DNSCache can be any address that can be guaranteed to not collide with any existing IP in your cluster. It's recommended to use an address with a local scope, per example, from the link-local range 169.254.0.0/16 for IPv4 or from the Unique Local Address range in IPv6 fd00::/8.
{{< note >}}
The local listen IP address for NodeLocal DNSCache can be any address that
can be guaranteed to not collide with any existing IP in your cluster.
It's recommended to use an address with a local scope, per example,
from the 'link-local' range '169.254.0.0/16' for IPv4 or from the
'Unique Local Address' range in IPv6 'fd00::/8'.
{{< /note >}}
This feature can be enabled using the following steps:
* Prepare a manifest similar to the sample [`nodelocaldns.yaml`](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml) and save it as `nodelocaldns.yaml.`
* If using IPv6, the CoreDNS configuration file need to enclose all the IPv6 addresses into square brackets if used in IP:Port format.
If you are using the sample manifest from the previous point, this will require to modify [the configuration line L70](https://github.com/kubernetes/kubernetes/blob/b2ecd1b3a3192fbbe2b9e348e095326f51dc43dd/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L70) like this `health [__PILLAR__LOCAL__DNS__]:8080`
* Prepare a manifest similar to the sample
[`nodelocaldns.yaml`](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml)
and save it as `nodelocaldns.yaml.`
* If using IPv6, the CoreDNS configuration file need to enclose all the IPv6 addresses
into square brackets if used in 'IP:Port' format.
If you are using the sample manifest from the previous point, this will require to modify
[the configuration line L70](https://github.com/kubernetes/kubernetes/blob/b2ecd1b3a3192fbbe2b9e348e095326f51dc43dd/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L70)
like this: "`health [__PILLAR__LOCAL__DNS__]:8080`"
* Substitute the variables in the manifest with the right values:
* kubedns=`kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}`
```shell
kubedns=`kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}`
domain=<cluster-domain>
localdns=<node-local-address>
```
* domain=`<cluster-domain>`
`<cluster-domain>` is "`cluster.local`" by default. `<node-local-address>` is the
local listen IP address chosen for NodeLocal DNSCache.
* localdns=`<node-local-address>`
* If kube-proxy is running in IPTABLES mode:
`<cluster-domain>` is "cluster.local" by default. `<node-local-address>` is the local listen IP address chosen for NodeLocal DNSCache.
``` bash
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
```
* If kube-proxy is running in IPTABLES mode:
``` bash
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
```
`__PILLAR__CLUSTER__DNS__` and `__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
In this mode, node-local-dns pods listen on both the kube-dns service IP as well as `<node-local-address>`, so pods can lookup DNS records using either IP address.
`__PILLAR__CLUSTER__DNS__` and `__PILLAR__UPSTREAM__SERVERS__` will be populated by
the `node-local-dns` pods.
In this mode, the `node-local-dns` pods listen on both the kube-dns service IP
as well as `<node-local-address>`, so pods can lookup DNS records using either IP address.
* If kube-proxy is running in IPVS mode:
``` bash
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
```
In this mode, node-local-dns pods listen only on `<node-local-address>`. The node-local-dns interface cannot bind the kube-dns cluster IP since the interface used for IPVS loadbalancing already uses this address.
`__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
In this mode, the `node-local-dns` pods listen only on `<node-local-address>`.
The `node-local-dns` interface cannot bind the kube-dns cluster IP since the
interface used for IPVS loadbalancing already uses this address.
`__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
* Run `kubectl create -f nodelocaldns.yaml`
* If using kube-proxy in IPVS mode, `--cluster-dns` flag to kubelet needs to be modified to use `<node-local-address>` that NodeLocal DNSCache is listening on.
Otherwise, there is no need to modify the value of the `--cluster-dns` flag, since NodeLocal DNSCache listens on both the kube-dns service IP as well as `<node-local-address>`.
Once enabled, node-local-dns Pods will run in the kube-system namespace on each of the cluster nodes. This Pod runs [CoreDNS](https://github.com/coredns/coredns) in cache mode, so all CoreDNS metrics exposed by the different plugins will be available on a per-node basis.
* If using kube-proxy in IPVS mode, `--cluster-dns` flag to kubelet needs to be modified
to use `<node-local-address>` that NodeLocal DNSCache is listening on.
Otherwise, there is no need to modify the value of the `--cluster-dns` flag,
since NodeLocal DNSCache listens on both the kube-dns service IP as well as
`<node-local-address>`.
You can disable this feature by removing the DaemonSet, using `kubectl delete -f <manifest>` . You should also revert any changes you made to the kubelet configuration.
Once enabled, the `node-local-dns` Pods will run in the `kube-system` namespace
on each of the cluster nodes. This Pod runs [CoreDNS](https://github.com/coredns/coredns)
in cache mode, so all CoreDNS metrics exposed by the different plugins will
be available on a per-node basis.
You can disable this feature by removing the DaemonSet, using `kubectl delete -f <manifest>`.
You should also revert any changes you made to the kubelet configuration.
## StubDomains and Upstream server Configuration
@@ -103,7 +143,9 @@ In those cases, the `kube-dns` ConfigMap can be updated.
## Setting memory limits
node-local-dns pods use memory for storing cache entries and processing queries. Since they do not watch Kubernetes objects, the cluster size or the number of Services/Endpoints do not directly affect memory usage. Memory usage is influenced by the DNS query pattern.
The `node-local-dns` Pods use memory for storing cache entries and processing queries.
Since they do not watch Kubernetes objects, the cluster size or the number of Services/Endpoints
do not directly affect memory usage. Memory usage is influenced by the DNS query pattern.
From [CoreDNS docs](https://github.com/coredns/deployment/blob/master/kubernetes/Scaling_CoreDNS.md),
> The default cache size is 10000 entries, which uses about 30 MB when completely filled.
@@ -114,13 +156,13 @@ The number of concurrent queries is linked to the memory demand, because each ex
goroutine used for handling a query requires an amount of memory. You can set an upper limit
using the `max_concurrent` option in the forward plugin.
If a node-local-dns pod attempts to use more memory than is available (because of total system
If a `node-local-dns` Pod attempts to use more memory than is available (because of total system
resources, or because of a configured
[resource limit](/docs/concepts/configuration/manage-resources-containers/)), the operating system
may shut down that pod's container.
If this happens, the container that is terminated (“OOMKilled”) does not clean up the custom
packet filtering rules that it previously added during startup.
The node-local-dns container should get restarted (since managed as part of a DaemonSet), but this
The `node-local-dns` container should get restarted (since managed as part of a DaemonSet), but this
will lead to a brief DNS downtime each time that the container fails: the packet filtering rules direct
DNS queries to a local Pod that is unhealthy.
@@ -128,3 +170,4 @@ You can determine a suitable memory limit by running node-local-dns pods without
measuring the peak usage. You can also set up and use a
[VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler)
in _recommender mode_, and then check its recommendations.
content/en/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md
+6 -6
View File
@@ -2,7 +2,7 @@
reviewers:
- fgrzadkowski
- piosz
title: Resource metrics pipeline
title: Resource metrics pipeline
content_type: concept
---
@@ -77,7 +77,7 @@ The architecture components, from right to left in the figure, consist of the fo
* [Metrics API](#metrics-api): Kubernetes API supporting access to CPU and memory used for
workload autoscaling. To make this work in your cluster, you need an API extension server that
provides the Metrics API.
{{< note >}}
cAdvisor supports reading metrics from cgroups, which works with typical container runtimes on Linux.
If you use a container runtime that uses another resource isolation mechanism, for example
@@ -85,14 +85,15 @@ The architecture components, from right to left in the figure, consist of the fo
[CRI Container Metrics](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/cri-container-stats.md)
in order for metrics to be available to the kubelet.
{{< /note >}}
<!-- body -->
## Metrics API
{{< feature-state for_k8s_version="1.8" state="beta" >}}
The metrics-server implements the Metrics API. This API allows you to access CPU and memory usage
for the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8s
autoscaler components.
autoscaler components.
Here is an example of the Metrics API request for a `minikube` node piped through `jq` for easier
reading:
@@ -201,7 +202,7 @@ Memory is reported as the working set, measured in bytes, at the instant the met
In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under
memory pressure. However, calculation of the working set varies by host OS, and generally makes
heavy use of heuristics to produce an estimate.
heavy use of heuristics to produce an estimate.
The Kubernetes model for a container's working set expects that the container runtime counts
anonymous memory associated with the container in question. The working set metric typically also
@@ -264,4 +265,3 @@ curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary
The summary API `/stats/summary` endpoint will be replaced by the `/metrics/resource` endpoint
beginning with metrics-server 0.6.x.
{{< /note >}}
content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md
+1 -1
View File
@@ -37,7 +37,7 @@ httpd running some PHP code.
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} If you're running an older
release of Kubernetes, refer to the version of the documentation for that release (see
[available documentation versions](/docs/home/supported-doc-versions/).
[available documentation versions](/docs/home/supported-doc-versions/)).
To follow this walkthrough, you also need to use a cluster that has a
[Metrics Server](https://github.com/kubernetes-sigs/metrics-server#readme) deployed and configured.