Official 1.17 Release Docs (#18011)
* feat: graduate TaintNodesByCondition to GA (#17073) * Promote StartupProbe to beta (enabled by default). (#17164) * Watch bookmarks to GA (#17026) * feat: graduate ScheduleDaemonSetPods to GA (#17350) * Update Docker installation instructions (#17405) * Use exact version numbers for installing Docker in Ubuntu (#17428) * Move CSIMigration and CSIMigrationGCE to Beta in Kubernetes v1.17 (#17478) * Promote NodeLease feature to GA (#17189) * Update docs for csi topology ga (#17408) * Update RunAsUsername to beta (#17460) * doc:Update RunAsUsername to beta * doc: update samples - kubernetes.io/os is no longer beta * Updating based on review feedback * Promote Node-specific volume limits to GA (#17432) * Promote PodShareProcessNamespace to stable (#17192) * Promote PodShareProcessNamespace to stable * Add for_k8s_version to feature-state label Co-Authored-By: Tim Bannister <tim@scalefactory.com> * Readd version-check to shareProcessNamespace task * Update service load balancer finalizer doc for GA (#17438) * Update Topology Manager docs (#17451) * Added information on how device plugins can take advantage of Topology Manager * Updated the Topology Manager documentation to include additionalinformation and update some out of date sections * Fix broken Topology Manager link (#17746) Part of What's Next Device Plugin section * Update CRD defaulting docs for GA (#17450) * Add documentation for VolumeSnapshot Beta (#17233) * Updating EndpointSlice documentation for beta release in 1.17 (#17411) * (docs/dualstack): v1.17 updates (#17457) * Add placehold doc updates for dualstack in 1.17 Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * Add Downward API and /etc/hosts Pod IP validation Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * remove addressed known issue via k/k pr 85246 Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * Remove known issue and add flag as part of k/k 79993 Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * remove follow up placeholders Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * Update verbiage Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * Make IP addressing consistent throughout the task Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * Update to status.podIPs Signed-off-by: Lachlan Evenson <lachlan.evenson@microsoft.com> * Update content/en/docs/tasks/network/validate-dual-stack.md Use set instead of env Co-Authored-By: Khaled Henidak (Kal) <khnidk@outlook.com> * add topology.kubernetes.io/zone, topology.kubernetes.io/region and node.kubernetes.io/instance-type labels to docs (#17498) Signed-off-by: Andrew Sy Kim <kiman@vmware.com> * Service topology alpha documentation (#17459) * Update list of feature flags for in-tree plugins migrated to CSI (#17533) Signed-off-by: Deep Debroy <ddebroy@docker.com> * Update Node concept for TaintNodesByCondition going GA (#17577) * feat: graduate ResourceQuotaScopeSelectors to GA in 1.17 (#17554) * kubeadm: update the upgrade documentation for 1.17 (#17587) * doc: Simplify Windows deployments with RuntimeClass (#16697) * doc: Simplify Windows deployments with RuntimeClass * Updating on review feedback * doc: Adding windows-build label from enhancement 1301 * update doc for kubelet option --reserved-cpus (#17648) * feat: update TaintNodesByCondition in feature gates table (#17377) * Update docs for v1 resource quota configuration (#17547) * AdmissionConfiguration v1 (#17548) * Update WebhookAdmissionConfiguration examples (#17549) * Update AWS EBS Migration Feature state (#16126) * Add resource version section to api-concepts documentation (#16910) * Add Resource Version semantics section to api concepts * Clarify risks of going back in time, add details about compaction and watch cache sizes * Apply suggestions from liggitt Co-Authored-By: Jordan Liggitt <jordan@liggitt.net> * remove pesudocode, apply feedback * Fix typo * Clarify equality rules * Cleanup kubectl generators docs (#17609) * Write ReplicationController without a space * Drop mentioning unsupported cluster versions * Fix capitalization for “API group” * Tweak wording * Avoid using deprecated generator in example * add Antrea description in dev-1.17 (#17919) * Promote VolumeSubpathEnvExpansion to GA * Reference Documentation for the Kubernetes API for 1.17 (#18019) * Update feature-gates.md (#18033) * Reference Documentation for kubectl Commands for 1.17 (#18017) * Update for v1.17 (#18034) * Update config.toml(release-1.17) for 1.17 (#18031)
This commit is contained in:
Damini Satya
committed by
Kubernetes Prow Robot
Kubernetes Prow Robot
parent
cf925bb688
commit
dff2f7b960
@@ -85,7 +85,10 @@ state file `cpu_manager_state` in the kubelet root directory.
|
||||
This policy manages a shared pool of CPUs that initially contains all CPUs in the
|
||||
node. The amount of exclusively allocatable CPUs is equal to the total
|
||||
number of CPUs in the node minus any CPU reservations by the kubelet `--kube-reserved` or
|
||||
`--system-reserved` options. CPUs reserved by these options are taken, in
|
||||
`--system-reserved` options. From 1.17, the CPU reservation list can be specified
|
||||
explictly by kubelet `--reserved-cpus` option. The explicit CPU list specified by
|
||||
`--reserved-cpus` takes precedence over the CPU reservation specified by
|
||||
`--kube-reserved` and `--system-reserved`. CPUs reserved by these options are taken, in
|
||||
integer quantity, from the initial shared pool in ascending order by physical
|
||||
core ID. This shared pool is the set of CPUs on which any containers in
|
||||
`BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional
|
||||
@@ -95,8 +98,8 @@ exclusive CPUs.
|
||||
|
||||
{{< note >}}
|
||||
The kubelet requires a CPU reservation greater than zero be made
|
||||
using either `--kube-reserved` and/or `--system-reserved` when the static
|
||||
policy is enabled. This is because zero CPU reservation would allow the shared
|
||||
using either `--kube-reserved` and/or `--system-reserved` or `--reserved-cpus` when
|
||||
the static policy is enabled. This is because zero CPU reservation would allow the shared
|
||||
pool to become empty.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
+3
-15
@@ -27,7 +27,7 @@ resources instead of a single large Endpoints resource.
|
||||
|
||||
## Enabling Endpoint Slices
|
||||
|
||||
{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
|
||||
{{< feature-state for_k8s_version="v1.17" state="beta" >}}
|
||||
|
||||
{{< note >}}
|
||||
Although Endpoint Slices may eventually replace Endpoints, many Kubernetes
|
||||
@@ -36,19 +36,7 @@ seen as an addition to Endpoints in a cluster, not a replacement for them.
|
||||
{{< /note >}}
|
||||
|
||||
As an alpha feature, Endpoint Slices are not enabled by default in Kubernetes.
|
||||
Enabling Endpoint Slices requires as many as 3 changes to Kubernetes cluster
|
||||
configuration.
|
||||
|
||||
To enable the Discovery API group that includes Endpoint Slices, use the runtime
|
||||
config flag (`--runtime-config=discovery.k8s.io/v1alpha1=true`).
|
||||
|
||||
The logic responsible for watching services, pods, and nodes and creating or
|
||||
updating associated Endpoint Slices lives within the EndpointSlice controller.
|
||||
This is disabled by default but can be enabled with the controllers flag on
|
||||
kube-controller-manager (`--controllers=endpointslice`).
|
||||
|
||||
For Kubernetes components like kube-proxy to actually start using Endpoint
|
||||
Slices, the EndpointSlice feature gate will need to be enabled
|
||||
To enable them, the EndpointSlice feature gate will need to be enabled
|
||||
(`--feature-gates=EndpointSlice=true`).
|
||||
|
||||
## Using Endpoint Slices
|
||||
@@ -57,4 +45,4 @@ With Endpoint Slices fully enabled in your cluster, you should see corresponding
|
||||
EndpointSlice resources for each Endpoints resource. In addition to supporting
|
||||
existing Endpoints functionality, Endpoint Slices should include new bits of
|
||||
information such as topology. They will allow for greater scalability and
|
||||
extensibility of network endpoints in your cluster.
|
||||
extensibility of network endpoints in your cluster.
|
||||
@@ -8,14 +8,14 @@ content_template: templates/task
|
||||
{{% capture overview %}}
|
||||
|
||||
This page explains how to upgrade a Kubernetes cluster created with kubeadm from version
|
||||
1.15.x to version 1.16.x, and from version 1.16.x to 1.16.y (where `y > x`).
|
||||
1.16.x to version 1.17.x, and from version 1.17.x to 1.17.y (where `y > x`).
|
||||
|
||||
To see information about upgrading clusters created using older versions of kubeadm,
|
||||
please refer to following pages instead:
|
||||
|
||||
- [Upgrading kubeadm cluster from 1.15 to 1.16](https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)
|
||||
- [Upgrading kubeadm cluster from 1.14 to 1.15](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/)
|
||||
- [Upgrading kubeadm cluster from 1.13 to 1.14](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-14/)
|
||||
- [Upgrading kubeadm cluster from 1.12 to 1.13](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/) or [Upgrading kubeadm HA clusters from v1.12 to v1.13](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-13/)
|
||||
|
||||
The upgrade workflow at high level is the following:
|
||||
|
||||
@@ -27,7 +27,7 @@ The upgrade workflow at high level is the following:
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
- You need to have a kubeadm Kubernetes cluster running version 1.15.0 or later.
|
||||
- You need to have a kubeadm Kubernetes cluster running version 1.16.0 or later.
|
||||
- [Swap must be disabled](https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux).
|
||||
- The cluster should use a static control plane and etcd pods or external etcd.
|
||||
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md) carefully.
|
||||
@@ -47,19 +47,19 @@ The upgrade workflow at high level is the following:
|
||||
|
||||
## Determine which version to upgrade to
|
||||
|
||||
1. Find the latest stable 1.16 version:
|
||||
1. Find the latest stable 1.17 version:
|
||||
|
||||
{{< tabs name="k8s_install_versions" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
apt update
|
||||
apt-cache policy kubeadm
|
||||
# find the latest 1.16 version in the list
|
||||
# it should look like 1.16.x-00, where x is the latest patch
|
||||
# find the latest 1.17 version in the list
|
||||
# it should look like 1.17.x-00, where x is the latest patch
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
yum list --showduplicates kubeadm --disableexcludes=kubernetes
|
||||
# find the latest 1.16 version in the list
|
||||
# it should look like 1.16.x-0, where x is the latest patch
|
||||
# find the latest 1.17 version in the list
|
||||
# it should look like 1.17.x-0, where x is the latest patch
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
@@ -71,14 +71,14 @@ The upgrade workflow at high level is the following:
|
||||
|
||||
{{< tabs name="k8s_install_kubeadm_first_cp" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
# replace x in 1.16.x-00 with the latest patch version
|
||||
# replace x in 1.17.x-00 with the latest patch version
|
||||
apt-mark unhold kubeadm && \
|
||||
apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
|
||||
apt-get update && apt-get install -y kubeadm=1.17.x-00 && \
|
||||
apt-mark hold kubeadm
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
# replace x in 1.16.x-0 with the latest patch version
|
||||
yum install -y kubeadm-1.16.x-0 --disableexcludes=kubernetes
|
||||
# replace x in 1.17.x-0 with the latest patch version
|
||||
yum install -y kubeadm-1.17.x-0 --disableexcludes=kubernetes
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
@@ -91,7 +91,7 @@ The upgrade workflow at high level is the following:
|
||||
1. Drain the control plane node:
|
||||
|
||||
```shell
|
||||
kubectl drain $MASTER --ignore-daemonsets
|
||||
kubectl drain $CP_NODE --ignore-daemonsets
|
||||
```
|
||||
|
||||
1. On the control plane node, run:
|
||||
@@ -109,26 +109,26 @@ The upgrade workflow at high level is the following:
|
||||
[preflight] Running pre-flight checks.
|
||||
[upgrade] Making sure the cluster is healthy:
|
||||
[upgrade] Fetching available versions to upgrade to
|
||||
[upgrade/versions] Cluster version: v1.15.2
|
||||
[upgrade/versions] kubeadm version: v1.16.0
|
||||
[upgrade/versions] Cluster version: v1.16.0
|
||||
[upgrade/versions] kubeadm version: v1.17.0
|
||||
|
||||
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
|
||||
COMPONENT CURRENT AVAILABLE
|
||||
Kubelet 1 x v1.15.2 v1.16.0
|
||||
Kubelet 1 x v1.16.0 v1.17.0
|
||||
|
||||
Upgrade to the latest version in the v1.16 series:
|
||||
|
||||
COMPONENT CURRENT AVAILABLE
|
||||
API Server v1.15.2 v1.16.0
|
||||
Controller Manager v1.15.2 v1.16.0
|
||||
Scheduler v1.15.2 v1.16.0
|
||||
Kube Proxy v1.15.2 v1.16.0
|
||||
CoreDNS 1.3.1 1.6.2
|
||||
Etcd 3.3.10 3.3.15
|
||||
API Server v1.16.0 v1.17.0
|
||||
Controller Manager v1.16.0 v1.17.0
|
||||
Scheduler v1.16.0 v1.17.0
|
||||
Kube Proxy v1.16.0 v1.17.0
|
||||
CoreDNS 1.6.2 1.6.5
|
||||
Etcd 3.3.15 3.4.3-0
|
||||
|
||||
You can now apply the upgrade by executing the following command:
|
||||
|
||||
kubeadm upgrade apply v1.16.0
|
||||
kubeadm upgrade apply v1.17.0
|
||||
|
||||
_____________________________________________________________________
|
||||
```
|
||||
@@ -144,7 +144,7 @@ The upgrade workflow at high level is the following:
|
||||
1. Choose a version to upgrade to, and run the appropriate command. For example:
|
||||
|
||||
```shell
|
||||
sudo kubeadm upgrade apply v1.16.x
|
||||
sudo kubeadm upgrade apply v1.17.x
|
||||
```
|
||||
|
||||
- Replace `x` with the patch version you picked for this upgrade.
|
||||
@@ -157,9 +157,9 @@ The upgrade workflow at high level is the following:
|
||||
[upgrade/config] Making sure the configuration is correct:
|
||||
[upgrade/config] Reading configuration from the cluster...
|
||||
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
|
||||
[upgrade/version] You have chosen to change the cluster version to "v1.16.0"
|
||||
[upgrade/versions] Cluster version: v1.15.2
|
||||
[upgrade/versions] kubeadm version: v1.16.0
|
||||
[upgrade/version] You have chosen to change the cluster version to "v1.17.0"
|
||||
[upgrade/versions] Cluster version: v1.16.0
|
||||
[upgrade/versions] kubeadm version: v1.17.0
|
||||
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
|
||||
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
|
||||
[upgrade/prepull] Prepulling image for component etcd.
|
||||
@@ -177,7 +177,7 @@ The upgrade workflow at high level is the following:
|
||||
[upgrade/prepull] Prepulled image for component kube-apiserver.
|
||||
[upgrade/prepull] Prepulled image for component kube-scheduler.
|
||||
[upgrade/prepull] Successfully prepulled the images for all the control plane components
|
||||
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.16.0"...
|
||||
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.17.0"...
|
||||
Static pod: kube-apiserver-luboitvbox hash: 8d931c2296a38951e95684cbcbe3b923
|
||||
Static pod: kube-controller-manager-luboitvbox hash: 2480bf6982ad2103c05f6764e20f2787
|
||||
Static pod: kube-scheduler-luboitvbox hash: 9b290132363a92652555896288ca3f88
|
||||
@@ -215,8 +215,8 @@ The upgrade workflow at high level is the following:
|
||||
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
|
||||
[upgrade/staticpods] Renewing certificate embedded in "admin.conf"
|
||||
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
|
||||
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
|
||||
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
|
||||
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
|
||||
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
|
||||
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
|
||||
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
|
||||
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
|
||||
@@ -224,7 +224,7 @@ The upgrade workflow at high level is the following:
|
||||
[addons] Applied essential addon: CoreDNS
|
||||
[addons] Applied essential addon: kube-proxy
|
||||
|
||||
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.16.0". Enjoy!
|
||||
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.17.0". Enjoy!
|
||||
|
||||
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
|
||||
```
|
||||
@@ -240,7 +240,7 @@ The upgrade workflow at high level is the following:
|
||||
1. Uncordon the control plane node
|
||||
|
||||
```shell
|
||||
kubectl uncordon $MASTER
|
||||
kubectl uncordon $CP_NODE
|
||||
```
|
||||
|
||||
### Upgrade additional control plane nodes
|
||||
@@ -265,14 +265,14 @@ Also `sudo kubeadm upgrade plan` is not needed.
|
||||
|
||||
{{< tabs name="k8s_install_kubelet" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
# replace x in 1.16.x-00 with the latest patch version
|
||||
# replace x in 1.17.x-00 with the latest patch version
|
||||
apt-mark unhold kubelet kubectl && \
|
||||
apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
|
||||
apt-get update && apt-get install -y kubelet=1.17.x-00 kubectl=1.17.x-00 && \
|
||||
apt-mark hold kubelet kubectl
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
# replace x in 1.16.x-0 with the latest patch version
|
||||
yum install -y kubelet-1.16.x-0 kubectl-1.16.x-0 --disableexcludes=kubernetes
|
||||
# replace x in 1.17.x-0 with the latest patch version
|
||||
yum install -y kubelet-1.17.x-0 kubectl-1.17.x-0 --disableexcludes=kubernetes
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
@@ -293,14 +293,14 @@ without compromising the minimum required capacity for running your workloads.
|
||||
|
||||
{{< tabs name="k8s_install_kubeadm_worker_nodes" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
# replace x in 1.16.x-00 with the latest patch version
|
||||
# replace x in 1.17.x-00 with the latest patch version
|
||||
apt-mark unhold kubeadm && \
|
||||
apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
|
||||
apt-get update && apt-get install -y kubeadm=1.17.x-00 && \
|
||||
apt-mark hold kubeadm
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
# replace x in 1.16.x-0 with the latest patch version
|
||||
yum install -y kubeadm-1.16.x-0 --disableexcludes=kubernetes
|
||||
# replace x in 1.17.x-0 with the latest patch version
|
||||
yum install -y kubeadm-1.17.x-0 --disableexcludes=kubernetes
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
@@ -334,14 +334,14 @@ without compromising the minimum required capacity for running your workloads.
|
||||
|
||||
{{< tabs name="k8s_kubelet_and_kubectl" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
# replace x in 1.16.x-00 with the latest patch version
|
||||
# replace x in 1.17.x-00 with the latest patch version
|
||||
apt-mark unhold kubelet kubectl && \
|
||||
apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
|
||||
apt-get update && apt-get install -y kubelet=1.17.x-00 kubectl=1.17.x-00 && \
|
||||
apt-mark hold kubelet kubectl
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
# replace x in 1.16.x-0 with the latest patch version
|
||||
yum install -y kubelet-1.16.x-0 kubectl-1.16.x-0 --disableexcludes=kubernetes
|
||||
# replace x in 1.17.x-0 with the latest patch version
|
||||
yum install -y kubelet-1.17.x-0 kubectl-1.17.x-0 --disableexcludes=kubernetes
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
|
||||
@@ -146,6 +146,28 @@ control group (`system.slice` on systemd machines for example).
|
||||
Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't
|
||||
exist. Kubelet will fail if an invalid cgroup is specified.
|
||||
|
||||
### Explictly Reserved CPU List
|
||||
{{< feature-state for_k8s_version="v1.17" state="stable" >}}
|
||||
|
||||
- **Kubelet Flag**: `--reserved-cpus=0-3`
|
||||
|
||||
`reserved-cpus` is meant to define an explict CPU set for OS system daemons and
|
||||
kubernetes system daemons. This option is added in 1.17 release. `reserved-cpus`
|
||||
is for systems that do not intent to define seperate top level cgroups for
|
||||
OS system daemons and kubernetes system daemons with regard to cpuset resource.
|
||||
If the Kubelet **does not** have `--system-reserved-cgroup` and `--kube-reserved-cgroup`,
|
||||
the explicit cpuset provided by `reserved-cpus` will take precedence over the CPUs
|
||||
defined by `--kube-reserved` and `--system-reserved` options.
|
||||
|
||||
This option is specifically designed for Telco/NFV use cases where uncontrolled
|
||||
interrupts/timers may impact the workload performance. you can use this option
|
||||
to define the explict cpuset for the system/kubernetes daemons as well as the
|
||||
interrupts/timers, so the rest CPUs on the system can be used exclusively for
|
||||
workloads, with less impact from uncontrolled interrupts/timers. To move the
|
||||
system daemon, kubernetes daemons and interrupts/timers to the explict cpuset
|
||||
defined by this option, other mechanism outside Kubernetes should be used.
|
||||
For example: in Centos, you can do this using the tuned toolset.
|
||||
|
||||
### Eviction Thresholds
|
||||
|
||||
- **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
|
||||
@@ -259,4 +281,8 @@ for `kube-reserved` and `system-reserved`.
|
||||
As of Kubernetes version 1.8, the `storage` key name was changed to `ephemeral-storage`
|
||||
for the alpha release.
|
||||
|
||||
As of Kubernetes version 1.17, you can optionally specify
|
||||
explicit cpuset by `reserved-cpus` as CPUs reserved for OS system
|
||||
daemons/interrupts/timers and Kubernetes daemons.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Control Topology Management Policies on a node
|
||||
|
||||
reviewers:
|
||||
- ConnorDoyle
|
||||
- klueska
|
||||
@@ -48,9 +49,9 @@ The hint is then stored in the Topology Manager for use by the *Hint Providers*
|
||||
The Topology Manager currently:
|
||||
|
||||
- Works on Nodes with the `static` CPU Manager Policy enabled. See [control CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/)
|
||||
- Works on Pods in the `Guaranteed` {{< glossary_tooltip text="QoS class" term_id="qos-class" >}}
|
||||
- Works on Pods making CPU requests or Device requests via extended resources
|
||||
|
||||
If these conditions are met, Topology Manager will align CPU and device requests.
|
||||
If these conditions are met, Topology Manager will align the requested resources.
|
||||
|
||||
Topology Manager supports four allocation policies. You can set a policy via a Kubelet flag, `--topology-manager-policy`.
|
||||
There are four supported policies:
|
||||
@@ -83,6 +84,9 @@ Using this information, the Topology Manager stores the
|
||||
preferred NUMA Node affinity for that container. If the affinity is not preferred,
|
||||
Topology Manager will reject this pod from the node. This will result in a pod in a `Terminated` state with a pod admission failure.
|
||||
|
||||
Once the pod is in a `Terminated` state, the Kubernetes scheduler will **not** attempt to reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigger a redeploy of the pod.
|
||||
An external control loop could be also implemented to trigger a redeployment of pods that have the `Topology Affinity` error.
|
||||
|
||||
If the pod is admitted, the *Hint Providers* can then use this information when making the
|
||||
resource allocation decision.
|
||||
|
||||
@@ -95,6 +99,8 @@ If it is, Topology Manager will store this and the *Hint Providers* can then use
|
||||
resource allocation decision.
|
||||
If, however, this is not possible then the Topology Manager will reject the pod from the node. This will result in a pod in a `Terminated` state with a pod admission failure.
|
||||
|
||||
Once the pod is in a `Terminated` state, the Kubernetes scheduler will **not** attempt to reschedule the pod. It is recommended a Deployment with Replicas to trigger a redeploy of the pod.
|
||||
An external control loop could be also implemented to trigger a redeployment of pods that have the `Topology Affinity` error.
|
||||
|
||||
### Pod Interactions with Topology Manager Policies
|
||||
|
||||
@@ -146,9 +152,36 @@ spec:
|
||||
|
||||
This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
|
||||
|
||||
Topology Manager would consider this Pod. The Topology Manager consults the CPU Manager `static` policy, which returns the topology of available CPUs.
|
||||
Topology Manager also consults Device Manager to discover the topology of available devices for example.com/device.
|
||||
|
||||
Topology Manager will use this information to store the best Topology for this container. In the case of this Pod, CPU and Device Manager will use this stored information at the resource allocation stage.
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx
|
||||
resources:
|
||||
limits:
|
||||
example.com/deviceA: "1"
|
||||
example.com/deviceB: "1"
|
||||
requests:
|
||||
example.com/deviceA: "1"
|
||||
example.com/deviceB: "1"
|
||||
```
|
||||
This pod runs in the `BestEffort` QoS class because there are no CPU and memory requests.
|
||||
|
||||
The Topology Manager would consider both of the above pods. The Topology Manager would consult the Hint Providers, which are CPU and Device Manager to get topology hints for the pods.
|
||||
In the case of the `Guaranteed` pod the `static` CPU Manager policy would return hints relating to the CPU request and the Device Manager would send back hints for the requested device.
|
||||
|
||||
In the case of the `BestEffort` pod the CPU Manager would send back the default hint as there is no CPU request and the Device Manager would send back the hints for each of the requested devices.
|
||||
|
||||
Using this information the Topology Manager calculates the optimal hint for the pod and stores this information, which will be used by the Hint Providers when they are making their resource assignments.
|
||||
|
||||
### Known Limitations
|
||||
1. As of K8s 1.16 the Topology Manager is currently only guaranteed to work if a *single* container in the pod spec requires aligned resources. This is due to the hint generation being based on current resource allocations, and all containers in a pod generate hints before any resource allocation has been made. This results in unreliable hints for all but the first container in a pod.
|
||||
*Due to this limitation if multiple pods/containers are considered by Kubelet in quick succession they may not respect the Topology Manager policy.
|
||||
|
||||
2. The maximum number of NUMA nodes that Topology Manager will allow is 8, past this there will be a state explosion when trying to enumerate the possible NUMA affinities and generating their hints.
|
||||
|
||||
3. The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail on the node due to the Topology Manager.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
Reference in New Issue
Block a user