Merge master into dev-1.21 to keep in sync, plus latest API reference
This sync merge includes API reference updates.
This commit is contained in:
Tim Bannister
+1
-1
@@ -236,7 +236,7 @@ Cluster DNS (CoreDNS) will not start up before a network is installed.**
|
||||
{{< /caution >}}
|
||||
|
||||
{{< note >}}
|
||||
Currently Calico is the only CNI plugin that the kubeadm project performs e2e tests against.
|
||||
Kubeadm should be CNI agnostic and the validation of CNI providers is out of the scope of our current e2e testing.
|
||||
If you find an issue related to a CNI plugin you should log a ticket in its respective issue
|
||||
tracker instead of the kubeadm or kubernetes issue trackers.
|
||||
{{< /note >}}
|
||||
|
||||
@@ -71,7 +71,7 @@ For more details please see the [Network Plugin Requirements](/docs/concepts/ext
|
||||
|
||||
| Protocol | Direction | Port Range | Purpose | Used By |
|
||||
|----------|-----------|------------|-------------------------|---------------------------|
|
||||
| TCP | Inbound | 6443* | Kubernetes API server | All |
|
||||
| TCP | Inbound | 6443\* | Kubernetes API server | All |
|
||||
| TCP | Inbound | 2379-2380 | etcd server client API | kube-apiserver, etcd |
|
||||
| TCP | Inbound | 10250 | kubelet API | Self, Control plane |
|
||||
| TCP | Inbound | 10251 | kube-scheduler | Self |
|
||||
@@ -303,9 +303,9 @@ for the management of cgroups on Linux machines.
|
||||
|
||||
{{< warning >}}
|
||||
Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.
|
||||
{{< warning >}}
|
||||
|
||||
See [Configuring a cgroup driver](/tasks/administer-cluster/kubeadm/configure-cgroup-driver) for more details.
|
||||
See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/) for more details.
|
||||
{{< /warning >}}
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -313,5 +313,5 @@ If you are running into difficulties with kubeadm, please consult our [troublesh
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
|
||||
* [Using kubeadm to Create a Cluster](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)
|
||||
|
||||
|
||||
@@ -23,10 +23,8 @@ manager instead, but you need to configure it manually.
|
||||
Some kubelet configuration details need to be the same across all kubelets involved in the cluster, while
|
||||
other configuration aspects need to be set on a per-kubelet basis to accommodate the different
|
||||
characteristics of a given machine (such as OS, storage, and networking). You can manage the configuration
|
||||
of your kubelets manually, but kubeadm now provides a `KubeletConfiguration` API type for [managing your
|
||||
kubelet configurations centrally](#configure-kubelets-using-kubeadm).
|
||||
|
||||
|
||||
of your kubelets manually, but kubeadm now provides a `KubeletConfiguration` API type for
|
||||
[managing your kubelet configurations centrally](#configure-kubelets-using-kubeadm).
|
||||
|
||||
<!-- body -->
|
||||
|
||||
@@ -52,8 +50,9 @@ Virtual IPs for services are now allocated from this subnet. You also need to se
|
||||
by the kubelet, using the `--cluster-dns` flag. This setting needs to be the same for every kubelet
|
||||
on every manager and Node in the cluster. The kubelet provides a versioned, structured API object
|
||||
that can configure most parameters in the kubelet and push out this configuration to each running
|
||||
kubelet in the cluster. This object is called **the kubelet's ComponentConfig**.
|
||||
The ComponentConfig allows the user to specify flags such as the cluster DNS IP addresses expressed as
|
||||
kubelet in the cluster. This object is called
|
||||
[`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/).
|
||||
The `KubeletConfiguration` allows the user to specify flags such as the cluster DNS IP addresses expressed as
|
||||
a list of values to a camelCased key, illustrated by the following example:
|
||||
|
||||
```yaml
|
||||
@@ -63,7 +62,7 @@ clusterDNS:
|
||||
- 10.96.0.10
|
||||
```
|
||||
|
||||
For more details on the ComponentConfig have a look at [this section](#configure-kubelets-using-kubeadm).
|
||||
For more details on the `KubeletConfiguration` have a look at [this section](#configure-kubelets-using-kubeadm).
|
||||
|
||||
### Providing instance-specific configuration details
|
||||
|
||||
@@ -99,8 +98,8 @@ API object is passed with a configuration file like so `kubeadm ... --config som
|
||||
By calling `kubeadm config print init-defaults --component-configs KubeletConfiguration` you can
|
||||
see all the default values for this structure.
|
||||
|
||||
Also have a look at the [API reference for the
|
||||
kubelet ComponentConfig](https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/config#KubeletConfiguration)
|
||||
Also have a look at the
|
||||
[reference for the KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/)
|
||||
for more information on the individual fields.
|
||||
|
||||
### Workflow when using `kubeadm init`
|
||||
@@ -160,9 +159,13 @@ has finished performing the TLS Bootstrap.
|
||||
`kubeadm` ships with configuration for how systemd should run the kubelet.
|
||||
Note that the kubeadm CLI command never touches this drop-in file.
|
||||
|
||||
This configuration file installed by the `kubeadm` [DEB](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf) or [RPM package](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/rpm/kubeadm/10-kubeadm.conf) is written to
|
||||
This configuration file installed by the `kubeadm`
|
||||
[DEB](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf) or
|
||||
[RPM package](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/rpm/kubeadm/10-kubeadm.conf) is written to
|
||||
`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` and is used by systemd.
|
||||
It augments the basic [`kubelet.service` for RPM](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/rpm/kubelet/kubelet.service) or [`kubelet.service` for DEB](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service):
|
||||
It augments the basic
|
||||
[`kubelet.service` for RPM](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/rpm/kubelet/kubelet.service) or
|
||||
[`kubelet.service` for DEB](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service):
|
||||
|
||||
```none
|
||||
[Service]
|
||||
|
||||
+11
-6
@@ -99,10 +99,11 @@ This may be caused by a number of problems. The most common are:
|
||||
|
||||
There are two common ways to fix the cgroup driver problem:
|
||||
|
||||
1. Install Docker again following instructions
|
||||
[here](/docs/setup/production-environment/container-runtimes/#docker).
|
||||
1. Install Docker again following instructions
|
||||
[here](/docs/setup/production-environment/container-runtimes/#docker).
|
||||
|
||||
1. Change the kubelet config to match the Docker cgroup driver manually, you can refer to [Configure cgroup driver used by kubelet on control-plane node](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node)
|
||||
1. Change the kubelet config to match the Docker cgroup driver manually, you can refer to
|
||||
[Configure cgroup driver used by kubelet on control-plane node](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node)
|
||||
|
||||
- control plane Docker containers are crashlooping or hanging. You can check this by running `docker ps` and investigating each container by running `docker logs`.
|
||||
|
||||
@@ -110,8 +111,11 @@ This may be caused by a number of problems. The most common are:
|
||||
|
||||
The following could happen if Docker halts and does not remove any Kubernetes-managed containers:
|
||||
|
||||
```bash
|
||||
```shell
|
||||
sudo kubeadm reset
|
||||
```
|
||||
|
||||
```console
|
||||
[preflight] Running pre-flight checks
|
||||
[reset] Stopping the kubelet service
|
||||
[reset] Unmounting mounted directories in "/var/lib/kubelet"
|
||||
@@ -121,14 +125,14 @@ sudo kubeadm reset
|
||||
|
||||
A possible solution is to restart the Docker service and then re-run `kubeadm reset`:
|
||||
|
||||
```bash
|
||||
```shell
|
||||
sudo systemctl restart docker.service
|
||||
sudo kubeadm reset
|
||||
```
|
||||
|
||||
Inspecting the logs for docker may also be useful:
|
||||
|
||||
```sh
|
||||
```shell
|
||||
journalctl -u docker
|
||||
```
|
||||
|
||||
@@ -415,3 +419,4 @@ If `/var/lib/kubelet` is being mounted, performing a `kubeadm reset` will effect
|
||||
To workaround the issue, re-mount the `/var/lib/kubelet` directory after performing the `kubeadm reset` operation.
|
||||
|
||||
This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20.
|
||||
|
||||
|
||||
@@ -68,7 +68,7 @@ Kubespray provides the ability to customize many aspects of the deployment:
|
||||
* {{< glossary_tooltip term_id="cri-o" >}}
|
||||
* Certificate generation methods
|
||||
|
||||
Kubespray customizations can be made to a [variable file](https://docs.ansible.com/ansible/playbooks_variables.html). If you are getting started with Kubespray, consider using the Kubespray defaults to deploy your cluster and explore Kubernetes.
|
||||
Kubespray customizations can be made to a [variable file](https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html). If you are getting started with Kubespray, consider using the Kubespray defaults to deploy your cluster and explore Kubernetes.
|
||||
|
||||
### (4/5) Deploy a Cluster
|
||||
|
||||
|
||||
+28
-19
@@ -1,7 +1,9 @@
|
||||
---
|
||||
reviewers:
|
||||
- michmike
|
||||
- patricklang
|
||||
- jayunit100
|
||||
- jsturtevant
|
||||
- marosset
|
||||
- perithompson
|
||||
title: Intro to Windows support in Kubernetes
|
||||
content_type: concept
|
||||
weight: 65
|
||||
@@ -233,23 +235,33 @@ Overlay (VXLAN) networks on Windows do not support dual-stack networking today.
|
||||
|
||||
### Limitations
|
||||
|
||||
#### Control Plane
|
||||
|
||||
Windows is only supported as a worker node in the Kubernetes architecture and component matrix. This means that a Kubernetes cluster must always include Linux master nodes, zero or more Linux worker nodes, and zero or more Windows worker nodes.
|
||||
|
||||
#### Compute {#compute-limitations}
|
||||
|
||||
##### Resource management and process isolation
|
||||
#### Resource Handling
|
||||
|
||||
Linux cgroups are used as a pod boundary for resource controls in Linux. Containers are created within that boundary for network, process and file system isolation. The cgroups APIs can be used to gather cpu/io/memory stats. In contrast, Windows uses a Job object per container with a system namespace filter to contain all processes in a container and provide logical isolation from the host. There is no way to run a Windows container without the namespace filtering in place. This means that system privileges cannot be asserted in the context of the host, and thus privileged containers are not available on Windows. Containers cannot assume an identity from the host because the Security Account Manager (SAM) is separate.
|
||||
|
||||
##### Operating System Restrictions
|
||||
#### Resource Reservations
|
||||
|
||||
Windows has strict compatibility rules, where the host OS version must match the container base image OS version. Only Windows containers with a container operating system of Windows Server 2019 are supported. Hyper-V isolation of containers, enabling some backward compatibility of Windows container image versions, is planned for a future release.
|
||||
##### Memory Reservations
|
||||
Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance.
|
||||
|
||||
##### Feature Restrictions
|
||||
Keeping memory usage within reasonable bounds is possible using the kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to account for memory usage on the node (outside of containers). This reduces [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
|
||||
|
||||
* TerminationGracePeriod: requires CRI-containerD
|
||||
{{< note >}}
|
||||
As you deploy workloads, use resource limits (must set only limits or limits must equal requests) on containers. This also subtracts from NodeAllocatable and prevents the scheduler from adding more pods once a node is full.
|
||||
{{< /note >}}
|
||||
|
||||
A best practice to avoid over-provisioning is to configure the kubelet with a system reserved memory of at least 2GB to account for Windows, Docker, and Kubernetes processes.
|
||||
|
||||
##### CPU Reservations
|
||||
To account for Windows, Docker and other Kubernetes host processes it is recommended to reserve a percentage of CPU so they are able to respond to events. This value needs to be scaled based on the number of CPU cores available on the Windows node.To determine this percentage a user should identify the maximum pod density for each of their nodes and monitor the CPU usage of the system services choosing a value that meets their workload needs.
|
||||
|
||||
Keeping CPU usage within reasonable bounds is possible using the kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to account for CPU usage on the node (outside of containers). This reduces [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
|
||||
|
||||
#### Feature Restrictions
|
||||
* TerminationGracePeriod: not implemented
|
||||
* Single file mapping: to be implemented with CRI-ContainerD
|
||||
* Termination message: to be implemented with CRI-ContainerD
|
||||
* Privileged Containers: not currently supported in Windows containers
|
||||
@@ -257,15 +269,8 @@ Windows has strict compatibility rules, where the host OS version must match the
|
||||
* The existing node problem detector is Linux-only and requires privileged containers. In general, we don't expect this to be used on Windows because privileged containers are not supported
|
||||
* Not all features of shared namespaces are supported (see API section for more details)
|
||||
|
||||
##### Memory Reservations and Handling
|
||||
|
||||
Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance.
|
||||
|
||||
Keeping memory usage within reasonable bounds is possible with a two-step process. First, use the kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to account for memory usage on the node (outside of containers). This reduces [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)). As you deploy workloads, use resource limits (must set only limits or limits must equal requests) on containers. This also subtracts from NodeAllocatable and prevents the scheduler from adding more pods once a node is full.
|
||||
|
||||
A best practice to avoid over-provisioning is to configure the kubelet with a system reserved memory of at least 2GB to account for Windows, Docker, and Kubernetes processes.
|
||||
|
||||
The behavior of the flags behave differently as described below:
|
||||
#### Difference in behavior of flags when compared to Linux
|
||||
The behavior of the following kubelet flags is different on Windows nodes as described below:
|
||||
|
||||
* `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update Node Allocatable
|
||||
* Eviction by using `--enforce-node-allocable` is not implemented
|
||||
@@ -413,6 +418,10 @@ None of the PodSecurityContext fields work on Windows. They're listed here for r
|
||||
* V1.PodSecurityContext.SupplementalGroups - provides GID, not available on Windows
|
||||
* V1.PodSecurityContext.Sysctls - these are part of the Linux sysctl interface. There's no equivalent on Windows.
|
||||
|
||||
#### Operating System Version Restrictions
|
||||
|
||||
Windows has strict compatibility rules, where the host OS version must match the container base image OS version. Only Windows containers with a container operating system of Windows Server 2019 are supported. Hyper-V isolation of containers, enabling some backward compatibility of Windows container image versions, is planned for a future release.
|
||||
|
||||
## Getting Help and Troubleshooting {#troubleshooting}
|
||||
|
||||
Your main source of help for troubleshooting your Kubernetes cluster should start with this [section](/docs/tasks/debug-application-cluster/troubleshooting/). Some additional, Windows-specific troubleshooting help is included in this section. Logs are an important element of troubleshooting issues in Kubernetes. Make sure to include them any time you seek troubleshooting assistance from other contributors. Follow the instructions in the SIG-Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs).
|
||||
|
||||
+4
-2
@@ -1,7 +1,9 @@
|
||||
---
|
||||
reviewers:
|
||||
- michmike
|
||||
- patricklang
|
||||
- jayunit100
|
||||
- jsturtevant
|
||||
- marosset
|
||||
- perithompson
|
||||
title: Guide for scheduling Windows containers in Kubernetes
|
||||
content_type: concept
|
||||
weight: 75
|
||||
|
||||
Reference in New Issue
Block a user