Merge branch 'main' into patch-1
This commit is contained in:
Adeniyi Stephen Oluwatola
@@ -247,6 +247,8 @@ You can still [manually create](/docs/tasks/configure-pod-container/configure-se
|
||||
a service account token Secret; for example, if you need a token that never expires.
|
||||
However, using the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/)
|
||||
subresource to obtain a token to access the API is recommended instead.
|
||||
You can use the [`kubectl create token`](/docs/reference/generated/kubectl/kubectl-commands#-em-token-em-)
|
||||
command to obtain a token from the `TokenRequest` API.
|
||||
{{< /note >}}
|
||||
|
||||
#### Projection of Secret keys to specific paths
|
||||
@@ -886,15 +888,30 @@ In this case, `0` means you have created an empty Secret.
|
||||
### Service account token Secrets
|
||||
|
||||
A `kubernetes.io/service-account-token` type of Secret is used to store a
|
||||
token that identifies a
|
||||
token credential that identifies a
|
||||
{{< glossary_tooltip text="service account" term_id="service-account" >}}.
|
||||
|
||||
Since 1.22, this type of Secret is no longer used to mount credentials into Pods,
|
||||
and obtaining tokens via the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/)
|
||||
API is recommended instead of using service account token Secret objects.
|
||||
Tokens obtained from the `TokenRequest` API are more secure than ones stored in Secret objects,
|
||||
because they have a bounded lifetime and are not readable by other API clients.
|
||||
You can use the [`kubectl create token`](/docs/reference/generated/kubectl/kubectl-commands#-em-token-em-)
|
||||
command to obtain a token from the `TokenRequest` API.
|
||||
|
||||
You should only create a service account token Secret object
|
||||
if you can't use the `TokenRequest` API to obtain a token,
|
||||
and the security exposure of persisting a non-expiring token credential
|
||||
in a readable API object is acceptable to you.
|
||||
|
||||
When using this Secret type, you need to ensure that the
|
||||
`kubernetes.io/service-account.name` annotation is set to an existing
|
||||
service account name. A Kubernetes
|
||||
{{< glossary_tooltip text="controller" term_id="controller" >}} fills in some
|
||||
other fields such as the `kubernetes.io/service-account.uid` annotation, and the
|
||||
`token` key in the `data` field, which is set to contain an authentication
|
||||
token.
|
||||
service account name. If you are creating both the ServiceAccount and
|
||||
the Secret objects, you should create the ServiceAccount object first.
|
||||
|
||||
After the Secret is created, a Kubernetes {{< glossary_tooltip text="controller" term_id="controller" >}}
|
||||
fills in some other fields such as the `kubernetes.io/service-account.uid` annotation, and the
|
||||
`token` key in the `data` field, which is populated with an authentication token.
|
||||
|
||||
The following example configuration declares a service account token Secret:
|
||||
|
||||
@@ -911,20 +928,14 @@ data:
|
||||
extra: YmFyCg==
|
||||
```
|
||||
|
||||
When creating a `Pod`, Kubernetes automatically finds or creates a service account
|
||||
Secret and then automatically modifies your Pod to use this Secret. The service account
|
||||
token Secret contains credentials for accessing the Kubernetes API.
|
||||
|
||||
The automatic creation and use of API credentials can be disabled or
|
||||
overridden if desired. However, if all you need to do is securely access the
|
||||
API server, this is the recommended workflow.
|
||||
After creating the Secret, wait for Kubernetes to populate the `token` key in the `data` field.
|
||||
|
||||
See the [ServiceAccount](/docs/tasks/configure-pod-container/configure-service-account/)
|
||||
documentation for more information on how service accounts work.
|
||||
You can also check the `automountServiceAccountToken` field and the
|
||||
`serviceAccountName` field of the
|
||||
[`Pod`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core)
|
||||
for information on referencing service account from Pods.
|
||||
for information on referencing service account credentials from within Pods.
|
||||
|
||||
### Docker config Secrets
|
||||
|
||||
@@ -982,7 +993,7 @@ kubectl create secret docker-registry secret-tiger-docker \
|
||||
```
|
||||
|
||||
That command creates a Secret of type `kubernetes.io/dockerconfigjson`.
|
||||
If you dump the `.data.dockercfgjson` field from that new Secret and then
|
||||
If you dump the `.data.dockerconfigjson` field from that new Secret and then
|
||||
decode it from base64:
|
||||
|
||||
```shell
|
||||
@@ -1291,7 +1302,7 @@ on that node.
|
||||
- When deploying applications that interact with the Secret API, you should
|
||||
limit access using
|
||||
[authorization policies](/docs/reference/access-authn-authz/authorization/) such as
|
||||
[RBAC]( /docs/reference/access-authn-authz/rbac/).
|
||||
[RBAC](/docs/reference/access-authn-authz/rbac/).
|
||||
- In the Kubernetes API, `watch` and `list` requests for Secrets within a namespace
|
||||
are extremely powerful capabilities. Avoid granting this access where feasible, since
|
||||
listing Secrets allows the clients to inspect the values of every Secret in that
|
||||
@@ -1310,7 +1321,7 @@ have access to run a Pod that then exposes the Secret.
|
||||
- When deploying applications that interact with the Secret API, you should
|
||||
limit access using
|
||||
[authorization policies](/docs/reference/access-authn-authz/authorization/) such as
|
||||
[RBAC]( /docs/reference/access-authn-authz/rbac/).
|
||||
[RBAC](/docs/reference/access-authn-authz/rbac/).
|
||||
- In the API server, objects (including Secrets) are persisted into
|
||||
{{< glossary_tooltip term_id="etcd" >}}; therefore:
|
||||
- only allow cluster admistrators to access etcd (this includes read-only access);
|
||||
|
||||
@@ -14,6 +14,8 @@ weight: 10
|
||||
Kubernetes {{< skew currentVersion >}} supports [Container Network Interface](https://github.com/containernetworking/cni)
|
||||
(CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with your cluster and that suits your needs. Different plugins are available (both open- and closed- source) in the wider Kubernetes ecosystem.
|
||||
|
||||
A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model).
|
||||
|
||||
You must use a CNI plugin that is compatible with the
|
||||
[v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) or later
|
||||
releases of the CNI specification. The Kubernetes project recommends using a plugin that is
|
||||
@@ -24,26 +26,37 @@ CNI specification (plugins can be compatible with multiple spec versions).
|
||||
|
||||
## Installation
|
||||
|
||||
A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model). The CRI manages its own CNI plugins. There are two Kubelet command line parameters to keep in mind when using plugins:
|
||||
A Container Runtime, in the networking context, is a daemon on a node configured to provide CRI Services for kubelet. In particular, the Container Runtime must be configured to load the CNI plugins required to implement the Kubernetes network model.
|
||||
|
||||
* `cni-bin-dir`: Kubelet probes this directory for plugins on startup
|
||||
* `network-plugin`: The network plugin to use from `cni-bin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is `cni`.
|
||||
{{< note >}}
|
||||
Prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the `cni-bin-dir` and `network-plugin` command-line parameters.
|
||||
These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for kubelet.
|
||||
|
||||
See [Troubleshooting CNI plugin-related errors](/docs/tasks/administer-cluster/migrating-from-dockershim/troubleshooting-cni-plugin-related-errors/)
|
||||
if you are facing issues following the removal of dockershim.
|
||||
{{< /note >}}
|
||||
|
||||
For specific information about how a Container Runtime manages the CNI plugins, see the documentation for that Container Runtime, for example:
|
||||
- [containerd](https://github.com/containerd/containerd/blob/main/script/setup/install-cni)
|
||||
- [CRI-O](https://github.com/cri-o/cri-o/blob/main/contrib/cni/README.md)
|
||||
|
||||
For specific information about how to install and manage a CNI plugin, see the documentation for that plugin or [networking provider](/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model).
|
||||
|
||||
## Network Plugin Requirements
|
||||
|
||||
Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{< param "fullversion" >}}/pkg/kubelet/dockershim/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy.
|
||||
For plugin developers and users who regularly build or deploy Kubernetes, the plugin may also need specific configuration to support kube-proxy.
|
||||
The iptables proxy depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables.
|
||||
For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly.
|
||||
If the plugin does not use a Linux bridge, but uses something like Open vSwitch or some other mechanism instead, it should ensure container traffic is appropriately routed for the proxy.
|
||||
|
||||
By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy.
|
||||
By default, if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy.
|
||||
|
||||
### CNI
|
||||
### Loopback CNI
|
||||
|
||||
The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`).
|
||||
In addition to the CNI plugin installed on the nodes for implementing the Kubernetes network model, Kubernetes also requires the container runtimes to provide a loopback interface `lo`, which is used for each sandbox (pod sandboxes, vm sandboxes, ...).
|
||||
Implementing the loopback interface can be accomplished by re-using the [CNI loopback plugin.](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) or by developing your own code to achieve this (see [this example from CRI-O](https://github.com/cri-o/ocicni/blob/release-1.24/pkg/ocicni/util_linux.go#L91)).
|
||||
|
||||
If there are multiple CNI configuration files in the directory, the kubelet uses the configuration file that comes first by name in lexicographic order.
|
||||
|
||||
In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0
|
||||
|
||||
#### Support hostPort
|
||||
### Support hostPort
|
||||
|
||||
The CNI networking plugin supports `hostPort`. You can use the official [portmap](https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap)
|
||||
plugin offered by the CNI plugin team or use your own plugin with portMapping functionality.
|
||||
@@ -80,7 +93,7 @@ For example:
|
||||
}
|
||||
```
|
||||
|
||||
#### Support traffic shaping
|
||||
### Support traffic shaping
|
||||
|
||||
**Experimental Feature**
|
||||
|
||||
@@ -132,8 +145,4 @@ metadata:
|
||||
...
|
||||
```
|
||||
|
||||
## Usage Summary
|
||||
|
||||
* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`).
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
@@ -111,6 +111,7 @@ Operator.
|
||||
{{% thirdparty-content %}}
|
||||
|
||||
* [Charmed Operator Framework](https://juju.is/)
|
||||
* [Java Operator SDK](https://github.com/java-operator-sdk/java-operator-sdk)
|
||||
* [Kopf](https://github.com/nolar/kopf) (Kubernetes Operator Pythonic Framework)
|
||||
* [kubebuilder](https://book.kubebuilder.io/)
|
||||
* [KubeOps](https://buehler.github.io/dotnet-operator-sdk/) (.NET operator SDK)
|
||||
|
||||
@@ -302,9 +302,8 @@ the Pod onto a node that is in the same zone as one or more Pods with the label
|
||||
`topology.kubernetes.io/zone=R` label if there are other nodes in the
|
||||
same zone currently running Pods with the `Security=S2` Pod label.
|
||||
|
||||
See the
|
||||
[design doc](https://git.k8s.io/community/contributors/design-proposals/scheduling/podaffinity.md)
|
||||
for many more examples of Pod affinity and anti-affinity.
|
||||
To get yourself more familiar with the examples of Pod affinity and anti-affinity,
|
||||
refer to the [design proposal](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/podaffinity.md).
|
||||
|
||||
You can use the `In`, `NotIn`, `Exists` and `DoesNotExist` values in the
|
||||
`operator` field for Pod affinity and anti-affinity.
|
||||
|
||||
@@ -29,10 +29,9 @@ This guide outlines the requirements of each policy.
|
||||
**The _Privileged_ policy is purposely-open, and entirely unrestricted.** This type of policy is
|
||||
typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users.
|
||||
|
||||
The Privileged policy is defined by an absence of restrictions. For allow-by-default enforcement
|
||||
mechanisms (such as gatekeeper), the Privileged policy may be an absence of applied constraints
|
||||
rather than an instantiated profile. In contrast, for a deny-by-default mechanism (such as Pod
|
||||
Security Policy) the Privileged policy should enable all controls (disable all restrictions).
|
||||
The Privileged policy is defined by an absence of restrictions. Allow-by-default
|
||||
mechanisms (such as gatekeeper) may be Privileged by default. In contrast, for a deny-by-default mechanism (such as Pod
|
||||
Security Policy) the Privileged policy should disable all restrictions.
|
||||
|
||||
### Baseline
|
||||
|
||||
@@ -458,6 +457,16 @@ of individual policies are not defined here.
|
||||
- {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}}
|
||||
- {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}}
|
||||
|
||||
### Alternatives
|
||||
|
||||
{{% thirdparty-content %}}
|
||||
|
||||
Other alternatives for enforcing policies are being developed in the Kubernetes ecosystem, such as:
|
||||
- [Kubewarden](https://github.com/kubewarden)
|
||||
- [Kyverno](https://kyverno.io/policies/pod-security/)
|
||||
- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper)
|
||||
|
||||
|
||||
## FAQ
|
||||
|
||||
### Why isn't there a profile between privileged and baseline?
|
||||
@@ -481,14 +490,6 @@ as well as other related parameters outside the Security Context. As of July 202
|
||||
[Pod Security Policies](/docs/concepts/security/pod-security-policy/) are deprecated in favor of the
|
||||
built-in [Pod Security Admission Controller](/docs/concepts/security/pod-security-admission/).
|
||||
|
||||
{{% thirdparty-content %}}
|
||||
|
||||
Other alternatives for enforcing security profiles are being developed in the Kubernetes
|
||||
ecosystem, such as:
|
||||
- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper).
|
||||
- [Kubewarden](https://github.com/kubewarden).
|
||||
- [Kyverno](https://kyverno.io/policies/pod-security/).
|
||||
|
||||
### What profiles should I apply to my Windows Pods?
|
||||
|
||||
Windows in Kubernetes has some limitations and differentiators from standard Linux-based
|
||||
|
||||
@@ -37,13 +37,14 @@ some general rules that can be applied are :
|
||||
can avoid accidental modification of cluster resources.
|
||||
- Avoid adding users to the `system:masters` group. Any user who is a member of this group
|
||||
bypasses all RBAC rights checks and will always have unrestricted superuser access, which cannot be
|
||||
revoked by removing Role Bindings or Cluster Role Bindings. As an aside, if a cluster is
|
||||
revoked by removing RoleBindings or ClusterRoleBindings. As an aside, if a cluster is
|
||||
using an authorization webhook, membership of this group also bypasses that webhook (requests
|
||||
from users who are members of that group are never sent to the webhook)
|
||||
|
||||
### Minimize distribution of privileged tokens
|
||||
|
||||
Ideally, pods shouldn't be assigned service accounts granted powerful permissions (listed [here](#Kubernetes-RBAC---Privilege-Escalation-Risks)).
|
||||
Ideally, pods shouldn't be assigned service accounts that have been granted powerful permissions (for example, any of the rights listed under
|
||||
[privilege escalation risks](#privilege-escalation-risks)).
|
||||
In cases where a workload requires powerful permissions, consider the following practices:
|
||||
|
||||
- Limit the number of nodes running powerful pods. Ensure that any DaemonSets you run
|
||||
@@ -82,7 +83,7 @@ Within Kubernetes RBAC there are a number of privileges which, if granted, can a
|
||||
to escalate their privileges in the cluster or affect systems outside the cluster.
|
||||
|
||||
This section is intended to provide visibility of the areas where cluster operators
|
||||
should take care, to ensure that they do not inadvertantly allow for more access to clusters than intended.
|
||||
should take care, to ensure that they do not inadvertently allow for more access to clusters than intended.
|
||||
|
||||
### Listing secrets
|
||||
|
||||
@@ -172,4 +173,7 @@ specifically relevant in multi-tenant clusters if semi-trusted or untrusted user
|
||||
are allowed limited access to a system.
|
||||
|
||||
One option for mitigation of this issue would be to use [resource quotas](/docs/concepts/policy/resource-quotas/#object-count-quota)
|
||||
to limit the quantity of objects which can be created.
|
||||
to limit the quantity of objects which can be created.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
* To learn more about RBAC, see the [RBAC documentation](/docs/reference/access-authn-authz/rbac/).
|
||||
|
||||
@@ -51,5 +51,5 @@ Windows containers can also run as Active Directory identities by utilizing [Gro
|
||||
Linux-specific pod security context mechanisms (such as SELinux, AppArmor, Seccomp, or custom
|
||||
POSIX capabilities) are not supported on Windows nodes.
|
||||
|
||||
Privileged containers are [not supported](#compatibility-v1-pod-spec-containers-securitycontext) on Windows.
|
||||
Privileged containers are [not supported](/docs/concepts/windows/intro/#compatibility-v1-pod-spec-containers-securitycontext) on Windows.
|
||||
Instead [HostProcess containers](/docs/tasks/configure-pod-container/create-hostprocess-pod) can be used on Windows to perform many of the tasks performed by privileged containers on Linux.
|
||||
|
||||
@@ -7,26 +7,25 @@ description: >
|
||||
|
||||
## The Kubernetes network model
|
||||
|
||||
Every [`Pod`](/docs/concepts/workloads/pods/) gets its own IP address.
|
||||
Every [`Pod`](/docs/concepts/workloads/pods/) in a cluster gets its own unique cluster-wide IP address.
|
||||
This means you do not need to explicitly create links between `Pods` and you
|
||||
almost never need to deal with mapping container ports to host ports.
|
||||
This creates a clean, backwards-compatible model where `Pods` can be treated
|
||||
much like VMs or physical hosts from the perspectives of port allocation,
|
||||
naming, service discovery, [load balancing](/docs/concepts/services-networking/ingress/#load-balancing), application configuration,
|
||||
and migration.
|
||||
naming, service discovery, [load balancing](/docs/concepts/services-networking/ingress/#load-balancing),
|
||||
application configuration, and migration.
|
||||
|
||||
Kubernetes imposes the following fundamental requirements on any networking
|
||||
implementation (barring any intentional network segmentation policies):
|
||||
|
||||
* pods on a [node](/docs/concepts/architecture/nodes/) can communicate with all pods on all nodes without NAT
|
||||
* pods can communicate with all other pods on any other [node](/docs/concepts/architecture/nodes/)
|
||||
without NAT
|
||||
* agents on a node (e.g. system daemons, kubelet) can communicate with all
|
||||
pods on that node
|
||||
|
||||
Note: For those platforms that support `Pods` running in the host network (e.g.
|
||||
Linux):
|
||||
|
||||
* pods in the host network of a node can communicate with all pods on all
|
||||
nodes without NAT
|
||||
Linux), when pods are attached to the host network of a node they can still communicate
|
||||
with all pods on all nodes without NAT.
|
||||
|
||||
This model is not only less complex overall, but it is principally compatible
|
||||
with the desire for Kubernetes to enable low-friction porting of apps from VMs
|
||||
|
||||
@@ -8,8 +8,8 @@ weight: 20
|
||||
---
|
||||
<!-- overview -->
|
||||
|
||||
Kubernetes creates DNS records for services and pods. You can contact
|
||||
services with consistent DNS names instead of IP addresses.
|
||||
Kubernetes creates DNS records for Services and Pods. You can contact
|
||||
Services with consistent DNS names instead of IP addresses.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
@@ -25,20 +25,20 @@ Pod's own namespace and the cluster's default domain.
|
||||
|
||||
### Namespaces of Services
|
||||
|
||||
A DNS query may return different results based on the namespace of the pod making
|
||||
it. DNS queries that don't specify a namespace are limited to the pod's
|
||||
namespace. Access services in other namespaces by specifying it in the DNS query.
|
||||
A DNS query may return different results based on the namespace of the Pod making
|
||||
it. DNS queries that don't specify a namespace are limited to the Pod's
|
||||
namespace. Access Services in other namespaces by specifying it in the DNS query.
|
||||
|
||||
For example, consider a pod in a `test` namespace. A `data` service is in
|
||||
For example, consider a Pod in a `test` namespace. A `data` Service is in
|
||||
the `prod` namespace.
|
||||
|
||||
A query for `data` returns no results, because it uses the pod's `test` namespace.
|
||||
A query for `data` returns no results, because it uses the Pod's `test` namespace.
|
||||
|
||||
A query for `data.prod` returns the intended result, because it specifies the
|
||||
namespace.
|
||||
|
||||
DNS queries may be expanded using the pod's `/etc/resolv.conf`. Kubelet
|
||||
sets this file for each pod. For example, a query for just `data` may be
|
||||
DNS queries may be expanded using the Pod's `/etc/resolv.conf`. Kubelet
|
||||
sets this file for each Pod. For example, a query for just `data` may be
|
||||
expanded to `data.test.svc.cluster.local`. The values of the `search` option
|
||||
are used to expand queries. To learn more about DNS queries, see
|
||||
[the `resolv.conf` manual page.](https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html)
|
||||
@@ -49,7 +49,7 @@ search <namespace>.svc.cluster.local svc.cluster.local cluster.local
|
||||
options ndots:5
|
||||
```
|
||||
|
||||
In summary, a pod in the _test_ namespace can successfully resolve either
|
||||
In summary, a Pod in the _test_ namespace can successfully resolve either
|
||||
`data.prod` or `data.prod.svc.cluster.local`.
|
||||
|
||||
### DNS Records
|
||||
@@ -70,14 +70,14 @@ For more up-to-date specification, see
|
||||
### A/AAAA records
|
||||
|
||||
"Normal" (not headless) Services are assigned a DNS A or AAAA record,
|
||||
depending on the IP family of the service, for a name of the form
|
||||
depending on the IP family of the Service, for a name of the form
|
||||
`my-svc.my-namespace.svc.cluster-domain.example`. This resolves to the cluster IP
|
||||
of the Service.
|
||||
|
||||
"Headless" (without a cluster IP) Services are also assigned a DNS A or AAAA record,
|
||||
depending on the IP family of the service, for a name of the form
|
||||
depending on the IP family of the Service, for a name of the form
|
||||
`my-svc.my-namespace.svc.cluster-domain.example`. Unlike normal
|
||||
Services, this resolves to the set of IPs of the pods selected by the Service.
|
||||
Services, this resolves to the set of IPs of the Pods selected by the Service.
|
||||
Clients are expected to consume the set or else use standard round-robin
|
||||
selection from the set.
|
||||
|
||||
@@ -87,36 +87,36 @@ SRV Records are created for named ports that are part of normal or [Headless
|
||||
Services](/docs/concepts/services-networking/service/#headless-services).
|
||||
For each named port, the SRV record would have the form
|
||||
`_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster-domain.example`.
|
||||
For a regular service, this resolves to the port number and the domain name:
|
||||
For a regular Service, this resolves to the port number and the domain name:
|
||||
`my-svc.my-namespace.svc.cluster-domain.example`.
|
||||
For a headless service, this resolves to multiple answers, one for each pod
|
||||
that is backing the service, and contains the port number and the domain name of the pod
|
||||
For a headless Service, this resolves to multiple answers, one for each Pod
|
||||
that is backing the Service, and contains the port number and the domain name of the Pod
|
||||
of the form `auto-generated-name.my-svc.my-namespace.svc.cluster-domain.example`.
|
||||
|
||||
## Pods
|
||||
|
||||
### A/AAAA records
|
||||
|
||||
In general a pod has the following DNS resolution:
|
||||
In general a Pod has the following DNS resolution:
|
||||
|
||||
`pod-ip-address.my-namespace.pod.cluster-domain.example`.
|
||||
|
||||
For example, if a pod in the `default` namespace has the IP address 172.17.0.3,
|
||||
For example, if a Pod in the `default` namespace has the IP address 172.17.0.3,
|
||||
and the domain name for your cluster is `cluster.local`, then the Pod has a DNS name:
|
||||
|
||||
`172-17-0-3.default.pod.cluster.local`.
|
||||
|
||||
Any pods exposed by a Service have the following DNS resolution available:
|
||||
Any Pods exposed by a Service have the following DNS resolution available:
|
||||
|
||||
`pod-ip-address.service-name.my-namespace.svc.cluster-domain.example`.
|
||||
|
||||
### Pod's hostname and subdomain fields
|
||||
|
||||
Currently when a pod is created, its hostname is the Pod's `metadata.name` value.
|
||||
Currently when a Pod is created, its hostname is the Pod's `metadata.name` value.
|
||||
|
||||
The Pod spec has an optional `hostname` field, which can be used to specify the
|
||||
Pod's hostname. When specified, it takes precedence over the Pod's name to be
|
||||
the hostname of the pod. For example, given a Pod with `hostname` set to
|
||||
the hostname of the Pod. For example, given a Pod with `hostname` set to
|
||||
"`my-host`", the Pod will have its hostname set to "`my-host`".
|
||||
|
||||
The Pod spec also has an optional `subdomain` field which can be used to specify
|
||||
@@ -173,14 +173,14 @@ spec:
|
||||
name: busybox
|
||||
```
|
||||
|
||||
If there exists a headless service in the same namespace as the pod and with
|
||||
If there exists a headless Service in the same namespace as the Pod and with
|
||||
the same name as the subdomain, the cluster's DNS Server also returns an A or AAAA
|
||||
record for the Pod's fully qualified hostname.
|
||||
For example, given a Pod with the hostname set to "`busybox-1`" and the subdomain set to
|
||||
"`default-subdomain`", and a headless Service named "`default-subdomain`" in
|
||||
the same namespace, the pod will see its own FQDN as
|
||||
the same namespace, the Pod will see its own FQDN as
|
||||
"`busybox-1.default-subdomain.my-namespace.svc.cluster-domain.example`". DNS serves an
|
||||
A or AAAA record at that name, pointing to the Pod's IP. Both pods "`busybox1`" and
|
||||
A or AAAA record at that name, pointing to the Pod's IP. Both Pods "`busybox1`" and
|
||||
"`busybox2`" can have their distinct A or AAAA records.
|
||||
|
||||
The Endpoints object can specify the `hostname` for any endpoint addresses,
|
||||
@@ -189,7 +189,7 @@ along with its IP.
|
||||
{{< note >}}
|
||||
Because A or AAAA records are not created for Pod names, `hostname` is required for the Pod's A or AAAA
|
||||
record to be created. A Pod with no `hostname` but with `subdomain` will only create the
|
||||
A or AAAA record for the headless service (`default-subdomain.my-namespace.svc.cluster-domain.example`),
|
||||
A or AAAA record for the headless Service (`default-subdomain.my-namespace.svc.cluster-domain.example`),
|
||||
pointing to the Pod's IP address. Also, Pod needs to become ready in order to have a
|
||||
record unless `publishNotReadyAddresses=True` is set on the Service.
|
||||
{{< /note >}}
|
||||
@@ -205,17 +205,17 @@ When you set `setHostnameAsFQDN: true` in the Pod spec, the kubelet writes the P
|
||||
{{< note >}}
|
||||
In Linux, the hostname field of the kernel (the `nodename` field of `struct utsname`) is limited to 64 characters.
|
||||
|
||||
If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in `Pending` status (`ContainerCreating` as seen by `kubectl`) generating error events, such as Failed to construct FQDN from pod hostname and cluster domain, FQDN `long-FQDN` is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an [admission webhook controller](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) to control FQDN size when users create top level objects, for example, Deployment.
|
||||
If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in `Pending` status (`ContainerCreating` as seen by `kubectl`) generating error events, such as Failed to construct FQDN from Pod hostname and cluster domain, FQDN `long-FQDN` is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an [admission webhook controller](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) to control FQDN size when users create top level objects, for example, Deployment.
|
||||
{{< /note >}}
|
||||
|
||||
### Pod's DNS Policy
|
||||
|
||||
DNS policies can be set on a per-pod basis. Currently Kubernetes supports the
|
||||
following pod-specific DNS policies. These policies are specified in the
|
||||
DNS policies can be set on a per-Pod basis. Currently Kubernetes supports the
|
||||
following Pod-specific DNS policies. These policies are specified in the
|
||||
`dnsPolicy` field of a Pod Spec.
|
||||
|
||||
- "`Default`": The Pod inherits the name resolution configuration from the node
|
||||
that the pods run on.
|
||||
that the Pods run on.
|
||||
See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers)
|
||||
for more details.
|
||||
- "`ClusterFirst`": Any DNS query that does not match the configured cluster
|
||||
@@ -226,6 +226,7 @@ following pod-specific DNS policies. These policies are specified in the
|
||||
for details on how DNS queries are handled in those cases.
|
||||
- "`ClusterFirstWithHostNet`": For Pods running with hostNetwork, you should
|
||||
explicitly set its DNS policy "`ClusterFirstWithHostNet`".
|
||||
- Note: This is not supported on Windows. See [below](#dns-windows) for details
|
||||
- "`None`": It allows a Pod to ignore DNS settings from the Kubernetes
|
||||
environment. All DNS settings are supposed to be provided using the
|
||||
`dnsConfig` field in the Pod Spec.
|
||||
@@ -306,7 +307,7 @@ For IPv6 setup, search path and name server should be setup like this:
|
||||
kubectl exec -it dns-example -- cat /etc/resolv.conf
|
||||
```
|
||||
The output is similar to this:
|
||||
```shell
|
||||
```
|
||||
nameserver fd00:79:30::a
|
||||
search default.svc.cluster-domain.example svc.cluster-domain.example cluster-domain.example
|
||||
options ndots:5
|
||||
@@ -323,8 +324,25 @@ If the feature gate `ExpandedDNSConfig` is enabled for the kube-apiserver and
|
||||
the kubelet, it is allowed for Kubernetes to have at most 32 search domains and
|
||||
a list of search domains of up to 2048 characters.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
## DNS resolution on Windows nodes {#dns-windows}
|
||||
|
||||
- ClusterFirstWithHostNet is not supported for Pods that run on Windows nodes.
|
||||
Windows treats all names with a `.` as a FQDN and skips FQDN resolution.
|
||||
- On Windows, there are multiple DNS resolvers that can be used. As these come with
|
||||
slightly different behaviors, using the
|
||||
[`Resolve-DNSName`](https://docs.microsoft.com/powershell/module/dnsclient/resolve-dnsname)
|
||||
powershell cmdlet for name query resolutions is recommended.
|
||||
- On Linux, you have a DNS suffix list, which is used after resolution of a name as fully
|
||||
qualified has failed.
|
||||
On Windows, you can only have 1 DNS suffix, which is the DNS suffix associated with that
|
||||
Pod's namespace (example: `mydns.svc.cluster.local`). Windows can resolve FQDNs, Services,
|
||||
or network name which can be resolved with this single suffix. For example, a Pod spawned
|
||||
in the `default` namespace, will have the DNS suffix `default.svc.cluster.local`.
|
||||
Inside a Windows Pod, you can resolve both `kubernetes.default.svc.cluster.local`
|
||||
and `kubernetes`, but not the partially qualified names (`kubernetes.default` or
|
||||
`kubernetes.default.svc`).
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
For guidance on administering DNS configurations, check
|
||||
[Configure DNS Service](/docs/tasks/administer-cluster/dns-custom-nameservers/)
|
||||
|
||||
@@ -239,6 +239,21 @@ If you want to enable egress traffic in order to reach off-cluster destinations
|
||||
Ensure your {{< glossary_tooltip text="CNI" term_id="cni" >}} provider supports IPv6.
|
||||
{{< /note >}}
|
||||
|
||||
## Windows support
|
||||
|
||||
Kubernetes on Windows does not support single-stack "IPv6-only" networking. However,
|
||||
dual-stack IPv4/IPv6 networking for pods and nodes with single-family services
|
||||
is supported.
|
||||
|
||||
You can use IPv4/IPv6 dual-stack networking with `l2bridge` networks.
|
||||
|
||||
{{< note >}}
|
||||
Overlay (VXLAN) networks on Windows **do not** support dual-stack networking.
|
||||
{{< /note >}}
|
||||
|
||||
You can read more about the different network modes for Windows within the
|
||||
[Networking on Windows](/docs/concepts/services-networking/windows-networking#network-modes) topic.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
|
||||
|
||||
@@ -30,23 +30,8 @@ For clarity, this guide defines the following terms:
|
||||
Traffic routing is controlled by rules defined on the Ingress resource.
|
||||
|
||||
Here is a simple example where an Ingress sends all its traffic to one Service:
|
||||
{{< mermaid >}}
|
||||
graph LR;
|
||||
client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress];
|
||||
ingress-->|routing rule|service[Service];
|
||||
subgraph cluster
|
||||
ingress;
|
||||
service-->pod1[Pod];
|
||||
service-->pod2[Pod];
|
||||
end
|
||||
classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
|
||||
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
|
||||
classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
|
||||
class ingress,service,pod1,pod2 k8s;
|
||||
class client plain;
|
||||
class cluster cluster;
|
||||
{{</ mermaid >}}
|
||||
|
||||
{{< figure src="/docs/images/ingress.svg" alt="ingress-diagram" class="diagram-large" caption="Figure. Ingress" link="https://mermaid.live/edit#pako:eNqNkstuwyAQRX8F4U0r2VHqPlSRKqt0UamLqlnaWWAYJygYLB59KMm_Fxcix-qmGwbuXA7DwAEzzQETXKutof0Ovb4vaoUQkwKUu6pi3FwXM_QSHGBt0VFFt8DRU2OWSGrKUUMlVQwMmhVLEV1Vcm9-aUksiuXRaO_CEhkv4WjBfAgG1TrGaLa-iaUw6a0DcwGI-WgOsF7zm-pN881fvRx1UDzeiFq7ghb1kgqFWiElyTjnuXVG74FkbdumefEpuNuRu_4rZ1pqQ7L5fL6YQPaPNiFuywcG9_-ihNyUkm6YSONWkjVNM8WUIyaeOJLO3clTB_KhL8NQDmVe-OJjxgZM5FhFiiFTK5zjDkxHBQ9_4zB4a-x20EGNSZhyaKmXrg7f5hSsvufUwTMXThtMWiot5Jh6p9ffimHijIezaSVoeN0uiqcfMJvf7w" >}}
|
||||
|
||||
An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting. An [Ingress controller](/docs/concepts/services-networking/ingress-controllers) is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.
|
||||
|
||||
@@ -398,25 +383,8 @@ A fanout configuration routes traffic from a single IP address to more than one
|
||||
based on the HTTP URI being requested. An Ingress allows you to keep the number of load balancers
|
||||
down to a minimum. For example, a setup like:
|
||||
|
||||
{{< mermaid >}}
|
||||
graph LR;
|
||||
client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress, 178.91.123.132];
|
||||
ingress-->|/foo|service1[Service service1:4200];
|
||||
ingress-->|/bar|service2[Service service2:8080];
|
||||
subgraph cluster
|
||||
ingress;
|
||||
service1-->pod1[Pod];
|
||||
service1-->pod2[Pod];
|
||||
service2-->pod3[Pod];
|
||||
service2-->pod4[Pod];
|
||||
end
|
||||
classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
|
||||
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
|
||||
classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
|
||||
class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s;
|
||||
class client plain;
|
||||
class cluster cluster;
|
||||
{{</ mermaid >}}
|
||||
{{< figure src="/docs/images/ingressFanOut.svg" alt="ingress-fanout-diagram" class="diagram-large" caption="Figure. Ingress Fan Out" link="https://mermaid.live/edit#pako:eNqNUslOwzAQ_RXLvYCUhMQpUFzUUzkgcUBwbHpw4klr4diR7bCo8O8k2FFbFomLPZq3jP00O1xpDpjijWHtFt09zAuFUCUFKHey8vf6NE7QrdoYsDZumGIb4Oi6NAskNeOoZJKpCgxK4oXwrFVgRyi7nCVXWZKRPMlysv5yD6Q4Xryf1Vq_WzDPooJs9egLNDbolKTpT03JzKgh3zWEztJZ0Niu9L-qZGcdmAMfj4cxvWmreba613z9C0B-AMQD-V_AdA-A4j5QZu0SatRKJhSqhZR0wjmPrDP6CeikrutQxy-Cuy2dtq9RpaU2dJKm6fzI5Glmg0VOLio4_5dLjx27hFSC015KJ2VZHtuQvY2fuHcaE43G0MaCREOow_FV5cMxHZ5-oPX75UM5avuXhXuOI9yAaZjg_aLuBl6B3RYaKDDtSw4166QrcKE-emrXcubghgunDaY1kxYizDqnH99UhakzHYykpWD9hjS--fEJoIELqQ" >}}
|
||||
|
||||
|
||||
would require an Ingress such as:
|
||||
|
||||
@@ -460,25 +428,7 @@ you are using, you may need to create a default-http-backend
|
||||
|
||||
Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.
|
||||
|
||||
{{< mermaid >}}
|
||||
graph LR;
|
||||
client([client])-. Ingress-managed <br> load balancer .->ingress[Ingress, 178.91.123.132];
|
||||
ingress-->|Host: foo.bar.com|service1[Service service1:80];
|
||||
ingress-->|Host: bar.foo.com|service2[Service service2:80];
|
||||
subgraph cluster
|
||||
ingress;
|
||||
service1-->pod1[Pod];
|
||||
service1-->pod2[Pod];
|
||||
service2-->pod3[Pod];
|
||||
service2-->pod4[Pod];
|
||||
end
|
||||
classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
|
||||
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
|
||||
classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
|
||||
class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s;
|
||||
class client plain;
|
||||
class cluster cluster;
|
||||
{{</ mermaid >}}
|
||||
{{< figure src="/docs/images/ingressNameBased.svg" alt="ingress-namebase-diagram" class="diagram-large" caption="Figure. Ingress Name Based Virtual hosting" link="https://mermaid.live/edit#pako:eNqNkl9PwyAUxb8KYS-atM1Kp05m9qSJJj4Y97jugcLtRqTQAPVPdN_dVlq3qUt8gZt7zvkBN7xjbgRgiteW1Rt0_zjLNUJcSdD-ZBn21WmcoDu9tuBcXDHN1iDQVWHnSBkmUMEU0xwsSuK5DK5l745QejFNLtMkJVmSZmT1Re9NcTz_uDXOU1QakxTMJtxUHw7ss-SQLhehQEODTsdH4l20Q-zFyc84-Y67pghv5apxHuweMuj9eS2_NiJdPhix-kMgvwQShOyYMNkJoEUYM3PuGkpUKyY1KqVSdCSEiJy35gnoqCzLvo5fpPAbOqlfI26UsXQ0Ho9nB5CnqesRGTnncPYvSqsdUvqp9KRdlI6KojjEkB0mnLgjDRONhqENBYm6oXbLV5V1y6S7-l42_LowlIN2uFm_twqOcAW2YlK0H_i9c-bYb6CCHNO2FFCyRvkc53rbWptaMA83QnpjMS2ZchBh1nizeNMcU28bGEzXkrV_pArN7Sc0rBTu" >}}
|
||||
|
||||
|
||||
The following Ingress tells the backing load balancer to route requests based on
|
||||
|
||||
@@ -54,7 +54,7 @@ POSTing this to the API server for your cluster will have no effect unless your
|
||||
__Mandatory Fields__: As with all other Kubernetes config, a NetworkPolicy
|
||||
needs `apiVersion`, `kind`, and `metadata` fields. For general information
|
||||
about working with config files, see
|
||||
[Configure Containers Using a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/),
|
||||
[Configure a Pod to Use a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/),
|
||||
and [Object Management](/docs/concepts/overview/working-with-objects/object-management).
|
||||
|
||||
__spec__: NetworkPolicy [spec](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status) has all the information needed to define a particular network policy in the given namespace.
|
||||
|
||||
@@ -192,6 +192,7 @@ where it's running, by adding an Endpoints object manually:
|
||||
apiVersion: v1
|
||||
kind: Endpoints
|
||||
metadata:
|
||||
# the name here should match the name of the Service
|
||||
name: my-service
|
||||
subsets:
|
||||
- addresses:
|
||||
@@ -203,6 +204,10 @@ subsets:
|
||||
The name of the Endpoints object must be a valid
|
||||
[DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names).
|
||||
|
||||
When you create an [Endpoints](docs/reference/kubernetes-api/service-resources/endpoints-v1/)
|
||||
object for a Service, you set the name of the new object to be the same as that
|
||||
of the Service.
|
||||
|
||||
{{< note >}}
|
||||
The endpoint IPs _must not_ be: loopback (127.0.0.0/8 for IPv4, ::1/128 for IPv6), or
|
||||
link-local (169.254.0.0/16 and 224.0.0.0/24 for IPv4, fe80::/64 for IPv6).
|
||||
@@ -394,6 +399,10 @@ You can also set the maximum session sticky time by setting
|
||||
`service.spec.sessionAffinityConfig.clientIP.timeoutSeconds` appropriately.
|
||||
(the default value is 10800, which works out to be 3 hours).
|
||||
|
||||
{{< note >}}
|
||||
On Windows, setting the maximum session sticky time for Services is not supported.
|
||||
{{< /note >}}
|
||||
|
||||
## Multi-Port Services
|
||||
|
||||
For some Services, you need to expose more than one port.
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
---
|
||||
reviewers:
|
||||
- aravindhp
|
||||
- jayunit100
|
||||
- jsturtevant
|
||||
- marosset
|
||||
title: Networking on Windows
|
||||
content_type: concept
|
||||
weight: 75
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Kubernetes supports running nodes on either Linux or Windows. You can mix both kinds of node
|
||||
within a single cluster.
|
||||
This page provides an overview to networking specific to the Windows operating system.
|
||||
|
||||
<!-- body -->
|
||||
## Container networking on Windows {#networking}
|
||||
|
||||
Networking for Windows containers is exposed through
|
||||
[CNI plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/).
|
||||
Windows containers function similarly to virtual machines in regards to
|
||||
networking. Each container has a virtual network adapter (vNIC) which is connected
|
||||
to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the
|
||||
Host Compute Service (HCS) work together to create containers and attach container
|
||||
vNICs to networks. HCS is responsible for the management of containers whereas HNS
|
||||
is responsible for the management of networking resources such as:
|
||||
|
||||
* Virtual networks (including creation of vSwitches)
|
||||
* Endpoints / vNICs
|
||||
* Namespaces
|
||||
* Policies including packet encapsulations, load-balancing rules, ACLs, and NAT rules.
|
||||
|
||||
The Windows HNS and vSwitch implement namespacing and can
|
||||
create virtual NICs as needed for a pod or container. However, many configurations such
|
||||
as DNS, routes, and metrics are stored in the Windows registry database rather than as
|
||||
files inside `/etc`, which is how Linux stores those configurations. The Windows registry for the container
|
||||
is separate from that of the host, so concepts like mapping `/etc/resolv.conf` from
|
||||
the host into a container don't have the same effect they would on Linux. These must
|
||||
be configured using Windows APIs run in the context of that container. Therefore
|
||||
CNI implementations need to call the HNS instead of relying on file mappings to pass
|
||||
network details into the pod or container.
|
||||
|
||||
## Network modes
|
||||
|
||||
Windows supports five different networking drivers/modes: L2bridge, L2tunnel,
|
||||
Overlay (Beta), Transparent, and NAT. In a heterogeneous cluster with Windows and Linux
|
||||
worker nodes, you need to select a networking solution that is compatible on both
|
||||
Windows and Linux. The following table lists the out-of-tree plugins are supported on Windows,
|
||||
with recommendations on when to use each CNI:
|
||||
|
||||
| Network Driver | Description | Container Packet Modifications | Network Plugins | Network Plugin Characteristics |
|
||||
| -------------- | ----------- | ------------------------------ | --------------- | ------------------------------ |
|
||||
| L2bridge | Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. | MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS OutboundNAT policy. | [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge), [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), Flannel host-gateway uses win-bridge | win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires user-defined routes (UDR) for inter-node connectivity. |
|
||||
| L2Tunnel | This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. | MAC rewritten, IP visible on the underlay network | [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) | Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that [Azure Virtual Network provides](https://azure.microsoft.com/en-us/services/virtual-network/). For example, securely connect to Azure services or use Azure NSGs. See [azure-cni for some examples](https://docs.microsoft.com/azure/aks/concepts-network#azure-cni-advanced-networking) |
|
||||
| Overlay | Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. | Encapsulated with an outer header. | [win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay), Flannel VXLAN (uses win-overlay) | win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option requires [KB4489899](https://support.microsoft.com/help/4489899) on Windows Server 2019. |
|
||||
| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host. <br/> Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller. <br/> NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. |
|
||||
| NAT (*not used in Kubernetes*) | Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called [WinNAT](https://techcommunity.microsoft.com/t5/virtualization/windows-nat-winnat-capabilities-and-limitations/ba-p/382303) | MAC and IP is rewritten to host MAC/IP. | [nat](https://github.com/Microsoft/windows-container-networking/tree/master/plugins/nat) | Included here for completeness |
|
||||
|
||||
As outlined above, the [Flannel](https://github.com/coreos/flannel)
|
||||
[CNI plugin](https://github.com/flannel-io/cni-plugin)
|
||||
is also [supported](https://github.com/flannel-io/cni-plugin#windows-support-experimental) on Windows via the
|
||||
[VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) (**Beta support** ; delegates to win-overlay)
|
||||
and [host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) (stable support; delegates to win-bridge).
|
||||
|
||||
This plugin supports delegating to one of the reference CNI plugins (win-overlay,
|
||||
win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for
|
||||
automatic node subnet lease assignment and HNS network creation. This plugin reads
|
||||
in its own configuration file (cni.conf), and aggregates it with the environment
|
||||
variables from the FlannelD generated subnet.env file. It then delegates to one of
|
||||
the reference CNI plugins for network plumbing, and sends the correct configuration
|
||||
containing the node-assigned subnet to the IPAM plugin (for example: `host-local`).
|
||||
|
||||
For Node, Pod, and Service objects, the following network flows are supported for
|
||||
TCP/UDP traffic:
|
||||
|
||||
* Pod → Pod (IP)
|
||||
* Pod → Pod (Name)
|
||||
* Pod → Service (Cluster IP)
|
||||
* Pod → Service (PQDN, but only if there are no ".")
|
||||
* Pod → Service (FQDN)
|
||||
* Pod → external (IP)
|
||||
* Pod → external (DNS)
|
||||
* Node → Pod
|
||||
* Pod → Node
|
||||
|
||||
## IP address management (IPAM) {#ipam}
|
||||
|
||||
The following IPAM options are supported on Windows:
|
||||
|
||||
* [host-local](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local)
|
||||
* [azure-vnet-ipam](https://github.com/Azure/azure-container-networking/blob/master/docs/ipam.md) (for azure-cni only)
|
||||
* [Windows Server IPAM](https://docs.microsoft.com/windows-server/networking/technologies/ipam/ipam-top) (fallback option if no IPAM is set)
|
||||
|
||||
## Load balancing and Services
|
||||
|
||||
A Kubernetes {{< glossary_tooltip text="Service" term_id="service" >}} is an abstraction
|
||||
that defines a logical set of Pods and a means to access them over a network.
|
||||
In a cluster that includes Windows nodes, you can use the following types of Service:
|
||||
|
||||
* `NodePort`
|
||||
* `ClusterIP`
|
||||
* `LoadBalancer`
|
||||
* `ExternalName`
|
||||
|
||||
Windows container networking differs in some important ways from Linux networking.
|
||||
The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture)
|
||||
provides additional details and background.
|
||||
|
||||
On Windows, you can use the following settings to configure Services and load
|
||||
balancing behavior:
|
||||
|
||||
{{< table caption="Windows Service Settings" >}}
|
||||
| Feature | Description | Minimum Supported Windows OS build | How to enable |
|
||||
| ------- | ----------- | -------------------------- | ------------- |
|
||||
| Session affinity | Ensures that connections from a particular client are passed to the same Pod each time. | Windows Server 2022 | Set `service.spec.sessionAffinity` to "ClientIP" |
|
||||
| Direct Server Return (DSR) | Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating pod IP. | Windows Server 2019 | Set the following flags in kube-proxy: `--feature-gates="WinDSR=true" --enable-dsr=true` |
|
||||
| Preserve-Destination | Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding. | Windows Server, version 1903 | Set `"preserve-destination": "true"` in service annotations and enable DSR in kube-proxy. |
|
||||
| IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | Windows Server 2019 | See [IPv4/IPv6 dual-stack](#ipv4ipv6-dual-stack) |
|
||||
| Client IP preservation | Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding. | Windows Server 2019 | Set `service.spec.externalTrafficPolicy` to "Local" and enable DSR in kube-proxy |
|
||||
{{< /table >}}
|
||||
|
||||
{{< warning >}}
|
||||
There are known issue with NodePort Services on overlay networking, if the destination node is running Windows Server 2022.
|
||||
To avoid the issue entirely, you can configure the service with `externalTrafficPolicy: Local`.
|
||||
|
||||
There are known issues with Pod to Pod connectivity on l2bridge network on Windows Server 2022 with KB5005619 or higher installed.
|
||||
To workaround the issue and restore Pod to Pod connectivity, you can disable the WinDSR feature in kube-proxy.
|
||||
|
||||
These issues require OS fixes.
|
||||
Please follow https://github.com/microsoft/Windows-Containers/issues/204 for updates.
|
||||
{{< /warning >}}
|
||||
|
||||
## Limitations
|
||||
|
||||
The following networking functionality is _not_ supported on Windows nodes:
|
||||
|
||||
* Host networking mode
|
||||
* Local NodePort access from the node itself (works for other nodes or external clients)
|
||||
* More than 64 backend pods (or unique destination addresses) for a single Service
|
||||
* IPv6 communication between Windows pods connected to overlay networks
|
||||
* Local Traffic Policy in non-DSR mode
|
||||
* Outbound communication using the ICMP protocol via the `win-overlay`, `win-bridge`, or using the Azure-CNI plugin.\
|
||||
Specifically, the Windows data plane ([VFP](https://www.microsoft.com/research/project/azure-virtual-filtering-platform/))
|
||||
doesn't support ICMP packet transpositions, and this means:
|
||||
* ICMP packets directed to destinations within the same network (such as pod to pod communication via ping)
|
||||
work as expected;
|
||||
* TCP/UDP packets work as expected;
|
||||
* ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping)
|
||||
cannot be transposed and thus will not be routed back to their source;
|
||||
* Since TCP/UDP packets can still be transposed, you can substitute `ping <destination>` with
|
||||
`curl <destination>` when debugging connectivity with the outside world.
|
||||
|
||||
Other limitations:
|
||||
|
||||
* Windows reference network plugins win-bridge and win-overlay do not implement
|
||||
[CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md) v0.4.0,
|
||||
due to a missing `CHECK` implementation.
|
||||
* The Flannel VXLAN CNI plugin has the following limitations on Windows:
|
||||
* Node-pod connectivity is only possible for local pods with Flannel v0.12.0 (or higher).
|
||||
* Flannel is restricted to using VNI 4096 and UDP port 4789. See the official
|
||||
[Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)
|
||||
backend docs for more details on these parameters.
|
||||
@@ -540,6 +540,15 @@ In the CLI, the access modes are abbreviated to:
|
||||
* RWX - ReadWriteMany
|
||||
* RWOP - ReadWriteOncePod
|
||||
|
||||
{{< note >}}
|
||||
Kubernetes uses volume access modes to match PersistentVolumeClaims and PersistentVolumes.
|
||||
In some cases, the volume access modes also constrain where the PersistentVolume can be mounted.
|
||||
Volume access modes do **not** enforce write protection once the storage has been mounted.
|
||||
Even if the access modes are specified as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany, they don't set any constraints on the volume.
|
||||
For example, even if a PersistentVolume is created as ReadOnlyMany, it is no guarantee that it will be read-only.
|
||||
If the access modes are specified as ReadWriteOncePod, the volume is constrained and can be mounted on only a single Pod.
|
||||
{{< /note >}}
|
||||
|
||||
> __Important!__ A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.
|
||||
|
||||
|
||||
|
||||
@@ -64,7 +64,9 @@ a different volume.
|
||||
|
||||
Kubernetes supports several types of volumes.
|
||||
|
||||
### awsElasticBlockStore {#awselasticblockstore}
|
||||
### awsElasticBlockStore (deprecated) {#awselasticblockstore}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.17" state="deprecated" >}}
|
||||
|
||||
An `awsElasticBlockStore` volume mounts an Amazon Web Services (AWS)
|
||||
[EBS volume](https://aws.amazon.com/ebs/) into your pod. Unlike
|
||||
@@ -135,7 +137,9 @@ beta features must be enabled.
|
||||
To disable the `awsElasticBlockStore` storage plugin from being loaded by the controller manager
|
||||
and the kubelet, set the `InTreePluginAWSUnregister` flag to `true`.
|
||||
|
||||
### azureDisk {#azuredisk}
|
||||
### azureDisk (deprecated) {#azuredisk}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.19" state="deprecated" >}}
|
||||
|
||||
The `azureDisk` volume type mounts a Microsoft Azure [Data Disk](https://docs.microsoft.com/en-us/azure/aks/csi-storage-drivers) into a pod.
|
||||
|
||||
@@ -158,7 +162,9 @@ must be installed on the cluster and the `CSIMigration` feature must be enabled.
|
||||
To disable the `azureDisk` storage plugin from being loaded by the controller manager
|
||||
and the kubelet, set the `InTreePluginAzureDiskUnregister` flag to `true`.
|
||||
|
||||
### azureFile {#azurefile}
|
||||
### azureFile (deprecated) {#azurefile}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.21" state="deprecated" >}}
|
||||
|
||||
The `azureFile` volume type mounts a Microsoft Azure File volume (SMB 2.1 and 3.0)
|
||||
into a pod.
|
||||
@@ -201,7 +207,9 @@ You must have your own Ceph server running with the share exported before you ca
|
||||
|
||||
See the [CephFS example](https://github.com/kubernetes/examples/tree/master/volumes/cephfs/) for more details.
|
||||
|
||||
### cinder
|
||||
### cinder (deprecated) {#cinder}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.18" state="deprecated" >}}
|
||||
|
||||
{{< note >}}
|
||||
Kubernetes must be configured with the OpenStack cloud provider.
|
||||
@@ -390,7 +398,9 @@ You must have your own Flocker installation running before you can use it.
|
||||
|
||||
See the [Flocker example](https://github.com/kubernetes/examples/tree/master/staging/volumes/flocker) for more details.
|
||||
|
||||
### gcePersistentDisk
|
||||
### gcePersistentDisk (deprecated) {#gcepersistentdisk}
|
||||
|
||||
{{< feature-state for_k8s_version="v1.17" state="deprecated" >}}
|
||||
|
||||
A `gcePersistentDisk` volume mounts a Google Compute Engine (GCE)
|
||||
[persistent disk](https://cloud.google.com/compute/docs/disks) (PD) into your Pod.
|
||||
@@ -1240,6 +1250,20 @@ for more information.
|
||||
For more information on how to develop a CSI driver, refer to the
|
||||
[kubernetes-csi documentation](https://kubernetes-csi.github.io/docs/)
|
||||
|
||||
#### Windows CSI proxy
|
||||
|
||||
{{< feature-state for_k8s_version="v1.22" state="stable" >}}
|
||||
|
||||
CSI node plugins need to perform various privileged
|
||||
operations like scanning of disk devices and mounting of file systems. These operations
|
||||
differ for each host operating system. For Linux worker nodes, containerized CSI node
|
||||
node plugins are typically deployed as privileged containers. For Windows worker nodes,
|
||||
privileged operations for containerized CSI node plugins is supported using
|
||||
[csi-proxy](https://github.com/kubernetes-csi/csi-proxy), a community-managed,
|
||||
stand-alone binary that needs to be pre-installed on each Windows node.
|
||||
|
||||
For more details, refer to the deployment guide of the CSI plugin you wish to deploy.
|
||||
|
||||
#### Migrating to CSI drivers from in-tree plugins
|
||||
|
||||
{{< feature-state for_k8s_version="v1.17" state="beta" >}}
|
||||
@@ -1256,6 +1280,14 @@ provisioning/delete, attach/detach, mount/unmount and resizing of volumes.
|
||||
In-tree plugins that support `CSIMigration` and have a corresponding CSI driver implemented
|
||||
are listed in [Types of Volumes](#volume-types).
|
||||
|
||||
The following in-tree plugins support persistent storage on Windows nodes:
|
||||
|
||||
* [`awsElasticBlockStore`](#awselasticblockstore)
|
||||
* [`azureDisk`](#azuredisk)
|
||||
* [`azureFile`](#azurefile)
|
||||
* [`gcePersistentDisk`](#gcepersistentdisk)
|
||||
* [`vsphereVolume`](#vspherevolume)
|
||||
|
||||
### flexVolume
|
||||
|
||||
{{< feature-state for_k8s_version="v1.23" state="deprecated" >}}
|
||||
@@ -1267,6 +1299,12 @@ volume plugin path on each node and in some cases the control plane nodes as wel
|
||||
Pods interact with FlexVolume drivers through the `flexVolume` in-tree volume plugin.
|
||||
For more details, see the FlexVolume [README](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-storage/flexvolume.md#readme) document.
|
||||
|
||||
The following FlexVolume [plugins](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows),
|
||||
deployed as PowerShell scripts on the host, support Windows nodes:
|
||||
|
||||
* [SMB](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~smb.cmd)
|
||||
* [iSCSI](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~iscsi.cmd)
|
||||
|
||||
{{< note >}}
|
||||
FlexVolume is deprecated. Using an out-of-tree CSI driver is the recommended way to integrate external storage with Kubernetes.
|
||||
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
reviewers:
|
||||
- jingxu97
|
||||
- mauriciopoppe
|
||||
- jayunit100
|
||||
- jsturtevant
|
||||
- marosset
|
||||
- aravindhp
|
||||
title: Windows Storage
|
||||
content_type: concept
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
This page provides an storage overview specific to the Windows operating system.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Persistent storage {#storage}
|
||||
|
||||
Windows has a layered filesystem driver to mount container layers and create a copy
|
||||
filesystem based on NTFS. All file paths in the container are resolved only within
|
||||
the context of that container.
|
||||
|
||||
* With Docker, volume mounts can only target a directory in the container, and not
|
||||
an individual file. This limitation does not apply to containerd.
|
||||
* Volume mounts cannot project files or directories back to the host filesystem.
|
||||
* Read-only filesystems are not supported because write access is always required
|
||||
for the Windows registry and SAM database. However, read-only volumes are supported.
|
||||
* Volume user-masks and permissions are not available. Because the SAM is not shared
|
||||
between the host & container, there's no mapping between them. All permissions are
|
||||
resolved within the context of the container.
|
||||
|
||||
As a result, the following storage functionality is not supported on Windows nodes:
|
||||
|
||||
* Volume subpath mounts: only the entire volume can be mounted in a Windows container
|
||||
* Subpath volume mounting for Secrets
|
||||
* Host mount projection
|
||||
* Read-only root filesystem (mapped volumes still support `readOnly`)
|
||||
* Block device mapping
|
||||
* Memory as the storage medium (for example, `emptyDir.medium` set to `Memory`)
|
||||
* File system features like uid/gid; per-user Linux filesystem permissions
|
||||
* Setting [secret permissions with DefaultMode](/docs/concepts/configuration/secret/#secret-files-permissions) (due to UID/GID dependency)
|
||||
* NFS based storage/volume support
|
||||
* Expanding the mounted volume (resizefs)
|
||||
|
||||
Kubernetes {{< glossary_tooltip text="volumes" term_id="volume" >}} enable complex
|
||||
applications, with data persistence and Pod volume sharing requirements, to be deployed
|
||||
on Kubernetes. Management of persistent volumes associated with a specific storage
|
||||
back-end or protocol includes actions such as provisioning/de-provisioning/resizing
|
||||
of volumes, attaching/detaching a volume to/from a Kubernetes node and
|
||||
mounting/dismounting a volume to/from individual containers in a pod that needs to
|
||||
persist data.
|
||||
|
||||
Volume management components are shipped as Kubernetes volume
|
||||
[plugin](/docs/concepts/storage/volumes/#types-of-volumes).
|
||||
The following broad classes of Kubernetes volume plugins are supported on Windows:
|
||||
|
||||
* [`FlexVolume plugins`](/docs/concepts/storage/volumes/#flexVolume)
|
||||
* Please note that FlexVolumes have been deprecated as of 1.23
|
||||
* [`CSI Plugins`](/docs/concepts/storage/volumes/#csi)
|
||||
|
||||
##### In-tree volume plugins
|
||||
|
||||
The following in-tree plugins support persistent storage on Windows nodes:
|
||||
|
||||
* [`awsElasticBlockStore`](/docs/concepts/storage/volumes/#awselasticblockstore)
|
||||
* [`azureDisk`](/docs/concepts/storage/volumes/#azuredisk)
|
||||
* [`azureFile`](/docs/concepts/storage/volumes/#azurefile)
|
||||
* [`gcePersistentDisk`](/docs/concepts/storage/volumes/#gcepersistentdisk)
|
||||
* [`vsphereVolume`](/docs/concepts/storage/volumes/#vspherevolume)
|
||||
@@ -0,0 +1,4 @@
|
||||
---
|
||||
title: "Windows in Kubernetes"
|
||||
weight: 50
|
||||
---
|
||||
@@ -0,0 +1,384 @@
|
||||
---
|
||||
reviewers:
|
||||
- jayunit100
|
||||
- jsturtevant
|
||||
- marosset
|
||||
- perithompson
|
||||
title: Windows containers in Kubernetes
|
||||
content_type: concept
|
||||
weight: 65
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Windows applications constitute a large portion of the services and applications that
|
||||
run in many organizations. [Windows containers](https://aka.ms/windowscontainers)
|
||||
provide a way to encapsulate processes and package dependencies, making it easier
|
||||
to use DevOps practices and follow cloud native patterns for Windows applications.
|
||||
|
||||
Organizations with investments in Windows-based applications and Linux-based
|
||||
applications don't have to look for separate orchestrators to manage their workloads,
|
||||
leading to increased operational efficiencies across their deployments, regardless
|
||||
of operating system.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Windows nodes in Kubernetes
|
||||
|
||||
To enable the orchestration of Windows containers in Kubernetes, include Windows nodes
|
||||
in your existing Linux cluster. Scheduling Windows containers in
|
||||
{{< glossary_tooltip text="Pods" term_id="pod" >}} on Kubernetes is similar to
|
||||
scheduling Linux-based containers.
|
||||
|
||||
In order to run Windows containers, your Kubernetes cluster must include
|
||||
multiple operating systems.
|
||||
While you can only run the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} on Linux,
|
||||
you can deploy worker nodes running either Windows or Linux.
|
||||
|
||||
Windows {{< glossary_tooltip text="nodes" term_id="node" >}} are
|
||||
[supported](#windows-os-version-support) provided that the operating system is
|
||||
Windows Server 2019.
|
||||
|
||||
This document uses the term *Windows containers* to mean Windows containers with
|
||||
process isolation. Kubernetes does not support running Windows containers with
|
||||
[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container).
|
||||
|
||||
## Compatibility and limitations {#limitations}
|
||||
|
||||
Some node features are only available if you use a specific
|
||||
[container runtime](#container-runtime); others are not available on Windows nodes,
|
||||
including:
|
||||
|
||||
* HugePages: not supported for Windows containers
|
||||
* Privileged containers: not supported for Windows containers
|
||||
* TerminationGracePeriod: requires containerD
|
||||
|
||||
Not all features of shared namespaces are supported. See [API compatibility](#api)
|
||||
for more details.
|
||||
|
||||
See [Windows OS version compatibility](#windows-os-version-support) for details on
|
||||
the Windows versions that Kubernetes is tested against.
|
||||
|
||||
From an API and kubectl perspective, Windows containers behave in much the same
|
||||
way as Linux-based containers. However, there are some notable differences in key
|
||||
functionality which are outlined in this section.
|
||||
|
||||
### Comparison with Linux {#compatibility-linux-similarities}
|
||||
|
||||
Key Kubernetes elements work the same way in Windows as they do in Linux. This
|
||||
section refers to several key workload abstractions and how they map to Windows.
|
||||
|
||||
* [Pods](/docs/concepts/workloads/pods/)
|
||||
|
||||
A Pod is the basic building block of Kubernetes–the smallest and simplest unit in
|
||||
the Kubernetes object model that you create or deploy. You may not deploy Windows and
|
||||
Linux containers in the same Pod. All containers in a Pod are scheduled onto a single
|
||||
Node where each Node represents a specific platform and architecture. The following
|
||||
Pod capabilities, properties and events are supported with Windows containers:
|
||||
|
||||
* Single or multiple containers per Pod with process isolation and volume sharing
|
||||
* Pod `status` fields
|
||||
* Readiness and Liveness probes
|
||||
* postStart & preStop container lifecycle hooks
|
||||
* ConfigMap, Secrets: as environment variables or volumes
|
||||
* `emptyDir` volumes
|
||||
* Named pipe host mounts
|
||||
* Resource limits
|
||||
* OS field:
|
||||
|
||||
The `.spec.os.name` field should be set to `windows` to indicate that the current Pod uses Windows containers.
|
||||
The `IdentifyPodOS` feature gate needs to be enabled for this field to be recognized.
|
||||
|
||||
{{< note >}}
|
||||
Starting from 1.24, the `IdentifyPodOS` feature gate is in Beta stage and defaults to be enabled.
|
||||
{{< /note >}}
|
||||
|
||||
If the `IdentifyPodOS` feature gate is enabled and you set the `.spec.os.name` field to `windows`,
|
||||
you must not set the following fields in the `.spec` of that Pod:
|
||||
|
||||
* `spec.hostPID`
|
||||
* `spec.hostIPC`
|
||||
* `spec.securityContext.seLinuxOptions`
|
||||
* `spec.securityContext.seccompProfile`
|
||||
* `spec.securityContext.fsGroup`
|
||||
* `spec.securityContext.fsGroupChangePolicy`
|
||||
* `spec.securityContext.sysctls`
|
||||
* `spec.shareProcessNamespace`
|
||||
* `spec.securityContext.runAsUser`
|
||||
* `spec.securityContext.runAsGroup`
|
||||
* `spec.securityContext.supplementalGroups`
|
||||
* `spec.containers[*].securityContext.seLinuxOptions`
|
||||
* `spec.containers[*].securityContext.seccompProfile`
|
||||
* `spec.containers[*].securityContext.capabilities`
|
||||
* `spec.containers[*].securityContext.readOnlyRootFilesystem`
|
||||
* `spec.containers[*].securityContext.privileged`
|
||||
* `spec.containers[*].securityContext.allowPrivilegeEscalation`
|
||||
* `spec.containers[*].securityContext.procMount`
|
||||
* `spec.containers[*].securityContext.runAsUser`
|
||||
* `spec.containers[*].securityContext.runAsGroup`
|
||||
|
||||
In the above list, wildcards (`*`) indicate all elements in a list.
|
||||
For example, `spec.containers[*].securityContext` refers to the SecurityContext object
|
||||
for all containers. If any of these fields is specified, the Pod will
|
||||
not be admited by the API server.
|
||||
|
||||
* [Workload resources](/docs/concepts/workloads/controllers/) including:
|
||||
* ReplicaSet
|
||||
* Deployment
|
||||
* StatefulSet
|
||||
* DaemonSet
|
||||
* Job
|
||||
* CronJob
|
||||
* ReplicationController
|
||||
* {{< glossary_tooltip text="Services" term_id="service" >}}
|
||||
See [Load balancing and Services](#load-balancing-and-services) for more details.
|
||||
|
||||
Pods, workload resources, and Services are critical elements to managing Windows
|
||||
workloads on Kubernetes. However, on their own they are not enough to enable
|
||||
the proper lifecycle management of Windows workloads in a dynamic cloud native
|
||||
environment. Kubernetes also supports:
|
||||
|
||||
* `kubectl exec`
|
||||
* Pod and container metrics
|
||||
* {{< glossary_tooltip text="Horizontal pod autoscaling" term_id="horizontal-pod-autoscaler" >}}
|
||||
* {{< glossary_tooltip text="Resource quotas" term_id="resource-quota" >}}
|
||||
* Scheduler preemption
|
||||
|
||||
### Command line options for the kubelet {#kubelet-compatibility}
|
||||
|
||||
Some kubelet command line options behave differently on Windows, as described below:
|
||||
|
||||
* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process
|
||||
(see [CPU resource management](/docs/concepts/configuration/windows-resource-management/#resource-management-cpu))
|
||||
* The `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update
|
||||
[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)
|
||||
* Eviction by using `--enforce-node-allocable` is not implemented
|
||||
* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented
|
||||
* A kubelet running on a Windows node does not have memory
|
||||
restrictions. `--kubelet-reserve` and `--system-reserve` do not set limits on
|
||||
kubelet or processes running on the host. This means kubelet or a process on the host
|
||||
could cause memory resource starvation outside the node-allocatable and scheduler.
|
||||
* The `MemoryPressure` Condition is not implemented
|
||||
* The kubelet does not take OOM eviction actions
|
||||
|
||||
### API compatibility {#api}
|
||||
|
||||
There are subtle differences in the way the Kubernetes APIs work for Windows due to the OS
|
||||
and container runtime. Some workload properties were designed for Linux, and fail to run on Windows.
|
||||
|
||||
At a high level, these OS concepts are different:
|
||||
|
||||
* Identity - Linux uses userID (UID) and groupID (GID) which
|
||||
are represented as integer types. User and group names
|
||||
are not canonical - they are just an alias in `/etc/groups`
|
||||
or `/etc/passwd` back to UID+GID. Windows uses a larger binary
|
||||
[security identifier](https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/security-identifiers) (SID)
|
||||
which is stored in the Windows Security Access Manager (SAM) database. This
|
||||
database is not shared between the host and containers, or between containers.
|
||||
* File permissions - Windows uses an access control list based on (SIDs), whereas
|
||||
POSIX systems such as Linux use a bitmask based on object permissions and UID+GID,
|
||||
plus _optional_ access control lists.
|
||||
* File paths - the convention on Windows is to use `\` instead of `/`. The Go IO
|
||||
libraries typically accept both and just make it work, but when you're setting a
|
||||
path or command line that's interpreted inside a container, `\` may be needed.
|
||||
* Signals - Windows interactive apps handle termination differently, and can
|
||||
implement one or more of these:
|
||||
* A UI thread handles well-defined messages including `WM_CLOSE`.
|
||||
* Console apps handle Ctrl-C or Ctrl-break using a Control Handler.
|
||||
* Services register a Service Control Handler function that can accept
|
||||
`SERVICE_CONTROL_STOP` control codes.
|
||||
|
||||
Container exit codes follow the same convention where 0 is success, and nonzero is failure.
|
||||
The specific error codes may differ across Windows and Linux. However, exit codes
|
||||
passed from the Kubernetes components (kubelet, kube-proxy) are unchanged.
|
||||
|
||||
##### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers}
|
||||
|
||||
The following list documents differences between how Pod container specifications
|
||||
work between Windows and Linux:
|
||||
|
||||
* Huge pages are not implemented in the Windows container
|
||||
runtime, and are not available. They require [asserting a user
|
||||
privilege](https://docs.microsoft.com/en-us/windows/desktop/Memory/large-page-support)
|
||||
that's not configurable for containers.
|
||||
* `requests.cpu` and `requests.memory` - requests are subtracted
|
||||
from node available resources, so they can be used to avoid overprovisioning a
|
||||
node. However, they cannot be used to guarantee resources in an overprovisioned
|
||||
node. They should be applied to all containers as a best practice if the operator
|
||||
wants to avoid overprovisioning entirely.
|
||||
* `securityContext.allowPrivilegeEscalation` -
|
||||
not possible on Windows; none of the capabilities are hooked up
|
||||
* `securityContext.capabilities` -
|
||||
POSIX capabilities are not implemented on Windows
|
||||
* `securityContext.privileged` -
|
||||
Windows doesn't support privileged containers
|
||||
* `securityContext.procMount` -
|
||||
Windows doesn't have a `/proc` filesystem
|
||||
* `securityContext.readOnlyRootFilesystem` -
|
||||
not possible on Windows; write access is required for registry & system
|
||||
processes to run inside the container
|
||||
* `securityContext.runAsGroup` -
|
||||
not possible on Windows as there is no GID support
|
||||
* `securityContext.runAsNonRoot` -
|
||||
this setting will prevent containers from running as `ContainerAdministrator`
|
||||
which is the closest equivalent to a root user on Windows.
|
||||
* `securityContext.runAsUser` -
|
||||
use [`runAsUserName`](/docs/tasks/configure-pod-container/configure-runasusername)
|
||||
instead
|
||||
* `securityContext.seLinuxOptions` -
|
||||
not possible on Windows as SELinux is Linux-specific
|
||||
* `terminationMessagePath` -
|
||||
this has some limitations in that Windows doesn't support mapping single files. The
|
||||
default value is `/dev/termination-log`, which does work because it does not
|
||||
exist on Windows by default.
|
||||
|
||||
##### Field compatibility for Pod specifications {#compatibility-v1-pod}
|
||||
|
||||
The following list documents differences between how Pod specifications work between Windows and Linux:
|
||||
|
||||
* `hostIPC` and `hostpid` - host namespace sharing is not possible on Windows
|
||||
* `hostNetwork` - There is no Windows OS support to share the host network
|
||||
* `dnsPolicy` - setting the Pod `dnsPolicy` to `ClusterFirstWithHostNet` is
|
||||
not supported on Windows because host networking is not provided. Pods always
|
||||
run with a container network.
|
||||
* `podSecurityContext` (see below)
|
||||
* `shareProcessNamespace` - this is a beta feature, and depends on Linux namespaces
|
||||
which are not implemented on Windows. Windows cannot share process namespaces or
|
||||
the container's root filesystem. Only the network can be shared.
|
||||
* `terminationGracePeriodSeconds` - this is not fully implemented in Docker on Windows,
|
||||
see the [GitHub issue](https://github.com/moby/moby/issues/25982).
|
||||
The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT,
|
||||
then Windows waits 5 seconds by default, and finally shuts down
|
||||
all processes using the normal Windows shutdown behavior. The 5
|
||||
second default is actually in the Windows registry
|
||||
[inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183),
|
||||
so it can be overridden when the container is built.
|
||||
* `volumeDevices` - this is a beta feature, and is not implemented on Windows.
|
||||
Windows cannot attach raw block devices to pods.
|
||||
* `volumes`
|
||||
* If you define an `emptyDir` volume, you cannot set its volume source to `memory`.
|
||||
* You cannot enable `mountPropagation` for volume mounts as this is not
|
||||
supported on Windows.
|
||||
|
||||
##### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext}
|
||||
|
||||
None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) fields work on Windows.
|
||||
|
||||
### Node problem detector
|
||||
|
||||
The node problem detector (see
|
||||
[Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/))
|
||||
is not compatible with Windows.
|
||||
|
||||
### Pause container
|
||||
|
||||
In a Kubernetes Pod, an infrastructure or “pause” container is first created
|
||||
to host the container. In Linux, the cgroups and namespaces that make up a pod
|
||||
need a process to maintain their continued existence; the pause process provides
|
||||
this. Containers that belong to the same pod, including infrastructure and worker
|
||||
containers, share a common network endpoint (same IPv4 and / or IPv6 address, same
|
||||
network port spaces). Kubernetes uses pause containers to allow for worker containers
|
||||
crashing or restarting without losing any of the networking configuration.
|
||||
|
||||
Kubernetes maintains a multi-architecture image that includes support for Windows.
|
||||
For Kubernetes v{{< skew currentVersion >}} the recommended pause image is `k8s.gcr.io/pause:3.6`.
|
||||
The [source code](https://github.com/kubernetes/kubernetes/tree/master/build/pause)
|
||||
is available on GitHub.
|
||||
|
||||
Microsoft maintains a different multi-architecture image, with Linux and Windows
|
||||
amd64 support, that you can find as `mcr.microsoft.com/oss/kubernetes/pause:3.6`.
|
||||
This image is built from the same source as the Kubernetes maintained image but
|
||||
all of the Windows binaries are [authenticode signed](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode) by Microsoft.
|
||||
The Kubernetes project recommends using the Microsoft maintained image if you are
|
||||
deploying to a production or production-like environment that requires signed
|
||||
binaries.
|
||||
|
||||
### Container runtimes {#container-runtime}
|
||||
|
||||
You need to install a
|
||||
{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}
|
||||
into each node in the cluster so that Pods can run there.
|
||||
|
||||
The following container runtimes work with Windows:
|
||||
|
||||
{{% thirdparty-content %}}
|
||||
|
||||
#### cri-containerd
|
||||
|
||||
{{< feature-state for_k8s_version="v1.20" state="stable" >}}
|
||||
|
||||
You can use {{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+
|
||||
as the container runtime for Kubernetes nodes that run Windows.
|
||||
|
||||
Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#install-containerd).
|
||||
|
||||
{{< note >}}
|
||||
There is a [known limitation](/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations)
|
||||
when using GMSA with containerd to access Windows network shares, which requires a
|
||||
kernel patch.
|
||||
{{< /note >}}
|
||||
|
||||
#### Mirantis Container Runtime {#mcr}
|
||||
|
||||
[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) is available as a container runtime for all Windows Server 2019 and later versions.
|
||||
|
||||
See [Install MCR on Windows Servers](https://docs.mirantis.com/mcr/20.10/install/mcr-windows.html) for more information.
|
||||
|
||||
## Windows OS version compatibility {#windows-os-version-support}
|
||||
|
||||
On Windows nodes, strict compatibility rules apply where the host OS version must
|
||||
match the container base image OS version. Only Windows containers with a container
|
||||
operating system of Windows Server 2019 are fully supported.
|
||||
|
||||
For Kubernetes v{{< skew currentVersion >}}, operating system compatibility for Windows nodes (and Pods)
|
||||
is as follows:
|
||||
|
||||
Windows Server LTSC release
|
||||
: Windows Server 2019
|
||||
: Windows Server 2022
|
||||
|
||||
Windows Server SAC release
|
||||
: Windows Server version 20H2
|
||||
|
||||
The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) also applies.
|
||||
|
||||
## Getting help and troubleshooting {#troubleshooting}
|
||||
|
||||
Your main source of help for troubleshooting your Kubernetes cluster should start
|
||||
with the [Troubleshooting](/docs/tasks/debug/)
|
||||
page.
|
||||
|
||||
Some additional, Windows-specific troubleshooting help is included
|
||||
in this section. Logs are an important element of troubleshooting
|
||||
issues in Kubernetes. Make sure to include them any time you seek
|
||||
troubleshooting assistance from other contributors. Follow the
|
||||
instructions in the
|
||||
SIG Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs).
|
||||
|
||||
### Reporting issues and feature requests
|
||||
|
||||
If you have what looks like a bug, or you would like to
|
||||
make a feature request, please follow the [SIG Windows contributing guide](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#reporting-issues-and-feature-requests) to create a new issue.
|
||||
You should first search the list of issues in case it was
|
||||
reported previously and comment with your experience on the issue and add additional
|
||||
logs. SIG-Windows Slack is also a great avenue to get some initial support and
|
||||
troubleshooting ideas prior to creating a ticket.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
### Deployment tools
|
||||
|
||||
The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control
|
||||
plane to manage the cluster it, and nodes to run your workloads.
|
||||
[Adding Windows nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/)
|
||||
explains how to deploy Windows nodes to your cluster using kubeadm.
|
||||
|
||||
The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also provides means to automate deployment of Windows nodes.
|
||||
|
||||
### Windows distribution channels
|
||||
|
||||
For a detailed explanation of Windows distribution channels see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19).
|
||||
|
||||
Information on the different Windows Server servicing channels
|
||||
including their support models can be found at
|
||||
[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started/servicing-channels-comparison).
|
||||
@@ -0,0 +1,310 @@
|
||||
---
|
||||
reviewers:
|
||||
- jayunit100
|
||||
- jsturtevant
|
||||
- marosset
|
||||
title: Guide for scheduling Windows containers in Kubernetes
|
||||
content_type: concept
|
||||
weight: 75
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Windows applications constitute a large portion of the services and applications that run in many organizations.
|
||||
This guide walks you through the steps to configure and deploy Windows containers in Kubernetes.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Objectives
|
||||
|
||||
* Configure an example deployment to run Windows containers on the Windows node
|
||||
* Highlight Windows specific funcationality in Kubernetes
|
||||
|
||||
## Before you begin
|
||||
|
||||
* Create a Kubernetes cluster that includes a
|
||||
control plane and a [worker node running Windows Server](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/)
|
||||
* It is important to note that creating and deploying services and workloads on Kubernetes
|
||||
behaves in much the same way for Linux and Windows containers.
|
||||
[Kubectl commands](/docs/reference/kubectl/) to interface with the cluster are identical.
|
||||
The example in the section below is provided to jumpstart your experience with Windows containers.
|
||||
|
||||
## Getting Started: Deploying a Windows container
|
||||
|
||||
The example YAML file below deploys a simple webserver application running inside a Windows container.
|
||||
|
||||
Create a service spec named `win-webserver.yaml` with the contents below:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: win-webserver
|
||||
labels:
|
||||
app: win-webserver
|
||||
spec:
|
||||
ports:
|
||||
# the port that this service should serve on
|
||||
- port: 80
|
||||
targetPort: 80
|
||||
selector:
|
||||
app: win-webserver
|
||||
type: NodePort
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
app: win-webserver
|
||||
name: win-webserver
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: win-webserver
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: win-webserver
|
||||
name: win-webserver
|
||||
spec:
|
||||
containers:
|
||||
- name: windowswebserver
|
||||
image: mcr.microsoft.com/windows/servercore:ltsc2019
|
||||
command:
|
||||
- powershell.exe
|
||||
- -command
|
||||
- "<#code used from https://gist.github.com/19WAS85/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ; ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus) } ; "
|
||||
nodeSelector:
|
||||
kubernetes.io/os: windows
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
Port mapping is also supported, but for simplicity this example exposes
|
||||
port 80 of the container directly to the Service.
|
||||
{{< /note >}}
|
||||
|
||||
1. Check that all nodes are healthy:
|
||||
|
||||
```bash
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
1. Deploy the service and watch for pod updates:
|
||||
|
||||
```bash
|
||||
kubectl apply -f win-webserver.yaml
|
||||
kubectl get pods -o wide -w
|
||||
```
|
||||
|
||||
When the service is deployed correctly both Pods are marked as Ready. To exit the watch command, press Ctrl+C.
|
||||
|
||||
1. Check that the deployment succeeded. To verify:
|
||||
|
||||
* Two pods listed from the Linux control plane node, use `kubectl get pods`
|
||||
* Node-to-pod communication across the network, `curl` port 80 of your pod IPs from the Linux control plane node
|
||||
to check for a web server response
|
||||
* Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node)
|
||||
using docker exec or kubectl exec
|
||||
* Service-to-pod communication, `curl` the virtual service IP (seen under `kubectl get services`)
|
||||
from the Linux control plane node and from individual pods
|
||||
* Service discovery, `curl` the service name with the Kubernetes [default DNS suffix](/docs/concepts/services-networking/dns-pod-service/#services)
|
||||
* Inbound connectivity, `curl` the NodePort from the Linux control plane node or machines outside of the cluster
|
||||
* Outbound connectivity, `curl` external IPs from inside the pod using kubectl exec
|
||||
|
||||
{{< note >}}
|
||||
Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack.
|
||||
Only Windows pods are able to access service IPs.
|
||||
{{< /note >}}
|
||||
|
||||
## Observability
|
||||
|
||||
### Capturing logs from workloads
|
||||
|
||||
Logs are an important element of observability; they enable users to gain insights
|
||||
into the operational aspect of workloads and are a key ingredient to troubleshooting issues.
|
||||
Because Windows containers and workloads inside Windows containers behave differently from Linux containers,
|
||||
users had a hard time collecting logs, limiting operational visibility.
|
||||
Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows)
|
||||
or push entries to the application event log.
|
||||
[LogMonitor](https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor), an open source tool by Microsoft,
|
||||
is the recommended way to monitor configured log sources inside a Windows container.
|
||||
LogMonitor supports monitoring event logs, ETW providers, and custom application logs,
|
||||
piping them to STDOUT for consumption by `kubectl logs <pod>`.
|
||||
|
||||
Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files
|
||||
to all your containers and add the necessary entrypoints for LogMonitor to push your logs to STDOUT.
|
||||
|
||||
## Configuring container user
|
||||
|
||||
### Using configurable Container usernames
|
||||
|
||||
Windows containers can be configured to run their entrypoints and processes
|
||||
with different usernames than the image defaults.
|
||||
Learn more about it [here](/docs/tasks/configure-pod-container/configure-runasusername/).
|
||||
|
||||
### Managing Workload Identity with Group Managed Service Accounts
|
||||
|
||||
Windows container workloads can be configured to use Group Managed Service Accounts (GMSA).
|
||||
Group Managed Service Accounts are a specific type of Active Directory account that provide automatic password management,
|
||||
simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.
|
||||
Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA.
|
||||
Learn more about configuring and using GMSA for Windows containers [here](/docs/tasks/configure-pod-container/configure-gmsa/).
|
||||
|
||||
## Taints and Tolerations
|
||||
|
||||
Users need to use some combination of taints and node selectors in order to
|
||||
schedule Linux and Windows workloads to their respective OS-specific nodes.
|
||||
The recommended approach is outlined below,
|
||||
with one of its main goals being that this approach should not break compatibility for existing Linux workloads.
|
||||
|
||||
If the `IdentifyPodOS` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is
|
||||
enabled, you can (and should) set `.spec.os.name` for a Pod to indicate the operating system
|
||||
that the containers in that Pod are designed for. For Pods that run Linux containers, set
|
||||
`.spec.os.name` to `linux`. For Pods that run Windows containers, set `.spec.os.name`
|
||||
to Windows.
|
||||
|
||||
{{< note >}}
|
||||
Starting from 1.24, the `IdentifyPodOS` feature is in Beta stage and defaults to be enabled.
|
||||
{{< /note >}}
|
||||
|
||||
The scheduler does not use the value of `.spec.os.name` when assigning Pods to nodes. You should
|
||||
use normal Kubernetes mechanisms for
|
||||
[assigning pods to nodes](/docs/concepts/scheduling-eviction/assign-pod-node/)
|
||||
to ensure that the control plane for your cluster places pods onto nodes that are running the
|
||||
appropriate operating system.
|
||||
|
||||
The `.spec.os.name` value has no effect on the scheduling of the Windows pods,
|
||||
so taints and tolerations and node selectors are still required
|
||||
to ensure that the Windows pods land onto appropriate Windows nodes.
|
||||
|
||||
### Ensuring OS-specific workloads land on the appropriate container host
|
||||
|
||||
Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations.
|
||||
All Kubernetes nodes today have the following default labels:
|
||||
|
||||
* kubernetes.io/os = [windows|linux]
|
||||
* kubernetes.io/arch = [amd64|arm64|...]
|
||||
|
||||
If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`,
|
||||
it is possible the Pod can be scheduled on any host, Windows or Linux.
|
||||
This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux.
|
||||
The best practice is to use a nodeSelector.
|
||||
|
||||
However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers,
|
||||
as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators.
|
||||
In those situations, you may be hesitant to make the configuration change to add nodeSelectors.
|
||||
The alternative is to use Taints. Because the kubelet can set Taints during registration,
|
||||
it could easily be modified to automatically add a taint when running on Windows only.
|
||||
|
||||
For example: `--register-with-taints='os=windows:NoSchedule'`
|
||||
|
||||
By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods).
|
||||
In order for a Windows Pod to be scheduled on a Windows node,
|
||||
it would need both the nodeSelector and the appropriate matching toleration to choose Windows.
|
||||
|
||||
```yaml
|
||||
nodeSelector:
|
||||
kubernetes.io/os: windows
|
||||
node.kubernetes.io/windows-build: '10.0.17763'
|
||||
tolerations:
|
||||
- key: "os"
|
||||
operator: "Equal"
|
||||
value: "windows"
|
||||
effect: "NoSchedule"
|
||||
```
|
||||
|
||||
### Handling multiple Windows versions in the same cluster
|
||||
|
||||
The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows
|
||||
Server versions in the same cluster, then you should set additional node labels and nodeSelectors.
|
||||
|
||||
Kubernetes 1.17 automatically adds a new label `node.kubernetes.io/windows-build` to simplify this.
|
||||
If you're running an older version, then it's recommended to add this label manually to Windows nodes.
|
||||
|
||||
This label reflects the Windows major, minor, and build number that need to match for compatibility.
|
||||
Here are values used today for each Windows Server version.
|
||||
|
||||
| Product Name | Build Number(s) |
|
||||
|--------------------------------------|------------------------|
|
||||
| Windows Server 2019 | 10.0.17763 |
|
||||
| Windows Server, Version 20H2 | 10.0.19042 |
|
||||
| Windows Server 2022 | 10.0.20348 |
|
||||
|
||||
### Simplifying with RuntimeClass
|
||||
|
||||
[RuntimeClass] can be used to simplify the process of using taints and tolerations.
|
||||
A cluster administrator can create a `RuntimeClass` object which is used to encapsulate these taints and tolerations.
|
||||
|
||||
1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector`
|
||||
for the Windows OS, architecture, and version.
|
||||
|
||||
```yaml
|
||||
apiVersion: node.k8s.io/v1
|
||||
kind: RuntimeClass
|
||||
metadata:
|
||||
name: windows-2019
|
||||
handler: 'docker'
|
||||
scheduling:
|
||||
nodeSelector:
|
||||
kubernetes.io/os: 'windows'
|
||||
kubernetes.io/arch: 'amd64'
|
||||
node.kubernetes.io/windows-build: '10.0.17763'
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
key: os
|
||||
operator: Equal
|
||||
value: "windows"
|
||||
```
|
||||
|
||||
1. Run `kubectl create -f runtimeClasses.yml` using as a cluster administrator
|
||||
1. Add `runtimeClassName: windows-2019` as appropriate to Pod specs
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: iis-2019
|
||||
labels:
|
||||
app: iis-2019
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
name: iis-2019
|
||||
labels:
|
||||
app: iis-2019
|
||||
spec:
|
||||
runtimeClassName: windows-2019
|
||||
containers:
|
||||
- name: iis
|
||||
image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1
|
||||
memory: 800Mi
|
||||
requests:
|
||||
cpu: .1
|
||||
memory: 300Mi
|
||||
ports:
|
||||
- containerPort: 80
|
||||
selector:
|
||||
matchLabels:
|
||||
app: iis-2019
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: iis
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 80
|
||||
selector:
|
||||
app: iis-2019
|
||||
```
|
||||
|
||||
[RuntimeClass]: https://kubernetes.io/docs/concepts/containers/runtime-class/
|
||||
@@ -119,7 +119,7 @@ kubectl logs $pods
|
||||
|
||||
The output is similar to this:
|
||||
|
||||
```shell
|
||||
```
|
||||
3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901
|
||||
```
|
||||
|
||||
|
||||
@@ -78,7 +78,7 @@ kubectl describe rs/frontend
|
||||
|
||||
And you will see output similar to:
|
||||
|
||||
```shell
|
||||
```
|
||||
Name: frontend
|
||||
Namespace: default
|
||||
Selector: tier=frontend
|
||||
@@ -130,7 +130,7 @@ kubectl get pods frontend-b2zdv -o yaml
|
||||
|
||||
The output will look similar to this, with the frontend ReplicaSet's info set in the metadata's ownerReferences field:
|
||||
|
||||
```shell
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
@@ -181,7 +181,7 @@ kubectl get pods
|
||||
|
||||
The output shows that the new Pods are either already terminated, or in the process of being terminated:
|
||||
|
||||
```shell
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
frontend-b2zdv 1/1 Running 0 10m
|
||||
frontend-vcmts 1/1 Running 0 10m
|
||||
@@ -210,7 +210,7 @@ kubectl get pods
|
||||
```
|
||||
|
||||
Will reveal in its output:
|
||||
```shell
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
frontend-hmmj2 1/1 Running 0 9s
|
||||
pod1 1/1 Running 0 36s
|
||||
|
||||
Reference in New Issue
Block a user