Updates for Windows Server version 1709 with K8s v1.8 (#6180)
* Updated for WSv1709 and K8s v1.8 * Updated picture and CNI config * Fixed formatting on CNI Config * Updated docs to reference Microsoft/SDN GitHub docs * fix typo * Workaround for Jekyllr frontmatter * Added section on features and limitations, with example yaml files. * Update index.md * Added kubeadm section, few other small fixes * Few minor grammar fixes * Update access-cluster.md with a comment that for IPv6 the user should use [::1] for the localhost * Addressed a number of issues brought up against the base PR * Fixed windows-host-setup link * Rewrite PodSecurityPolicy guide * Update index.md Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com> Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org> * Spelling correction and sentence capitalization. - Corrected the spelling error for storing, was put in as 'stoing'. - Capitalized list items. - Added '.' at end of sentences in the list items. * Update index.md * Update index.md * Addressed comments and rebased * Fixed formatting * Fixed formatting * Updated header link * Updated hyperlinks * Updated warning * formatting * formatting * formatting * Revert "Update access-cluster.md with a comment that for IPv6" This reverts commit31e4dbdc25. * Revert "fix typo" This reverts commitc05678752d. * Revert "Workaround for Jekyllr frontmatter" This reverts commitb84ac59624. * Fixed grammatical issues and reverted non-related commits * Revert "Rewrite PodSecurityPolicy guide" This reverts commit5d39cfeae4. * Revert "Spelling correction and sentence capitalization." This reverts commit47eed4346e. * Fixed auto-numbering * Minor formatting updates
This commit is contained in:
Jason Messer
committed by
Zach Corleissen
Zach Corleissen
parent
736fb02fd8
commit
7c7cf17194
@@ -48,24 +48,24 @@ When creating a new API, consider whether to [aggregate your API with the Kubern
|
||||
#### Declarative APIs
|
||||
|
||||
In a Declarative API, typically:
|
||||
- Your API consists of a relatively small number of relatively small objects (resources).
|
||||
- The objects define configuration of applications or infrastructure.
|
||||
- The objects are updated relatively infrequently.
|
||||
- Humans often need to read and write the objects.
|
||||
- The main operations on the objects are CRUD-y (creating, reading, updating and deleting).
|
||||
- Transactions across objects are not required: the API represents a desired state, not an exact state.
|
||||
- your API consists of a relatively small number of relatively small objects (resources).
|
||||
- the objects define configuration of applications or infrastructure
|
||||
- the objects are updated relatively infrequently
|
||||
- humans often need to read and write the objects
|
||||
- the main operations on the objects are CRUD-y (creating, reading, updating and deleting)
|
||||
- transactions across objects are not required: the API represents a desired state, not an exact state.
|
||||
|
||||
Imperative APIs are not declarative.
|
||||
Signs that your API might not be declarative include:
|
||||
- The client says "do this", and then gets a synchornous response back when it is done.
|
||||
- The client says "do this", and then gets an operation ID back, and has to check a separate Operation objects to determine completion of the request.
|
||||
- You talk about Remote Procedure Calls (RPCs).
|
||||
- Directly storing large amounts of data (e.g. > a few kB per object, or >1000s of objects).
|
||||
- High bandwidth access (10s of requests per second sustained) needed.
|
||||
- Store end-user data (such as images, PII, etc) or other large-scale data processed by applications.
|
||||
- The natural operations on the objects are not CRUD-y.
|
||||
- The API is not easily modeled as objects.
|
||||
- You chose to represent pending operations with an operation ID or operation object.
|
||||
- the client says "do this", and then gets a synchornous response back when it is done.
|
||||
- the client says "do this", and then gets an operation ID back, and has to check a separate Operation objects to determine completion of the request.
|
||||
- you talk about Remote Procedure Calls (RPCs)
|
||||
- directly stoing large amounts of data (e.g. > a few kB per object, or >1000s of objects)
|
||||
- high bandwidth access (10s of requests per second sustained) needed
|
||||
- store end-user data (such as images, PII, etc) or other large-scale data processed by applications
|
||||
- the natural operations on the objects are not CRUD-y.
|
||||
- the API is not easily modeled as objects.
|
||||
- you chose to represent pending operations with an operation ID or operation object.
|
||||
|
||||
### Should I use a configMap or a custom resource?
|
||||
|
||||
@@ -102,7 +102,7 @@ Aggregated APIs are subordinate APIServers that sit behind the primary API serve
|
||||
|
||||
Custom Resource Definitions (CRDS) allow users to create new types of resources without adding another APIserver. You do not need to understand API Aggregation to use CRDs.
|
||||
|
||||
Regardless of whether they are installed via CRDs or AA, the new resources are called Custom Resources to distinguish them from built-in Kubernetes resources (like pods).
|
||||
Regardless of whether they are installed via CRDs or AA, the new resources are called Custom Resources to distinguish them from built-in Kubernetes resources (like pods)
|
||||
|
||||
## CustomResourceDefinitions
|
||||
|
||||
@@ -215,9 +215,9 @@ Kubernetes [client libraries](/docs/reference/client-libraries/) can be used to
|
||||
|
||||
When you add a custom resource, you can access it using:
|
||||
- kubectl
|
||||
- The kubernetes dynamic client.
|
||||
- A REST client that you write.
|
||||
- A client generated using Kubernetes client generation tools (generating one is an advanced undertaking, but some projects may provide a client along with the CRD or AA).
|
||||
- the kubernetes dynamic client
|
||||
- a REST client that you write
|
||||
- a client generated using Kubernetes client generation tools (generating one is an advanced undertaking, but some projects may provide a client along with the CRD or AA).
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
@@ -1,492 +1,70 @@
|
||||
---
|
||||
approvers:
|
||||
- pweil-
|
||||
- tallclair
|
||||
title: Pod Security Policies
|
||||
---
|
||||
|
||||
{% include feature-state-beta.md %}
|
||||
Objects of type `PodSecurityPolicy` govern the ability
|
||||
to make requests on a pod that affect the `SecurityContext` that will be
|
||||
applied to a pod and container.
|
||||
|
||||
Pod Security Policies enable fine-grained authorization of pod creation and
|
||||
updates.
|
||||
See [PodSecurityPolicy proposal](https://git.k8s.io/community/contributors/design-proposals/auth/pod-security-policy.md) for more information.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## What is a Pod Security Policy?
|
||||
|
||||
A _Pod Security Policy_ is a cluster-level resource that controls security
|
||||
sensitive aspects of the pod specification. The `PodSecurityPolicy` objects
|
||||
define a set of conditions that a pod must run with in order to be accepted into
|
||||
the system, as well as defaults for the related fields. They allow an
|
||||
A _Pod Security Policy_ is a cluster-level resource that controls the
|
||||
actions that a pod can perform and what it has the ability to access. The
|
||||
`PodSecurityPolicy` objects define a set of conditions that a pod must
|
||||
run with in order to be accepted into the system. They allow an
|
||||
administrator to control the following:
|
||||
|
||||
| Control Aspect | Field Names |
|
||||
| ----------------------------------------------------| ------------------------------------------- |
|
||||
| Running of privileged containers | `privileged` |
|
||||
| Usage of the root namespaces | [`hostPID`, `hostIPC`](#host-namespaces) |
|
||||
| Usage of host networking and ports | [`hostNetwork`, `hostPorts`](#host-namespaces) |
|
||||
| Usage of volume types | [`volumes`](#volumes-and-file-systems) |
|
||||
| Usage of the host filesystem | [`allowedHostPaths`](#volumes-and-file-systems) |
|
||||
| Allocating an FSGroup that owns the pod's volumes | [`fsGroup`](#volumes-and-file-systems) |
|
||||
| Requiring the use of a read only root file system | [`readOnlyRootFilesystem`](#volumes-and-file-systems) |
|
||||
| The user and group IDs of the container | [`runAsUser`, `supplementalGroups`](#users-and-groups) |
|
||||
| Restricting escalation to root privileges | [`allowPrivilegeEscalation`, `defaultAllowPrivilegeEscalation`](#privilege-escalation) |
|
||||
| Linux capabilities | [`defaultAddCapabilities`, `requiredDropCapabilities`, `allowedCapabilities`](#capabilities) |
|
||||
| The SELinux context of the container | [`seLinux`](#selinux) |
|
||||
| The AppArmor profile used by containers | [annotations](#apparmor) |
|
||||
| The seccomp profile used by containers | [annotations](#seccomp) |
|
||||
|
||||
|
||||
## Enabling Pod Security Policies
|
||||
|
||||
Pod security policy control is implemented as an optional (but recommended)
|
||||
[admission
|
||||
controller](/docs/admin/admission-controllers/#podsecuritypolicy). PodSecurityPolicies
|
||||
are enforced by [enabling the admission
|
||||
controller](/docs/admin/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in),
|
||||
but doing so without authorizing any policies **will prevent any pods from being
|
||||
created** in the cluster.
|
||||
|
||||
Since the pod security policy API (`extensions/v1beta1/podsecuritypolicy`) is
|
||||
enabled independently of the admission controller, for existing clusters it is
|
||||
recommended that policies are added and authorized before enabling the admission
|
||||
controller.
|
||||
|
||||
## Authorizing Policies
|
||||
|
||||
When a PodSecurityPolicy resource is created, it does nothing. In order to use
|
||||
it, the requesting user or target pod's [service
|
||||
account](/docs/tasks/configure-pod-container/configure-service-account/) must be
|
||||
authorized to use the policy, by allowing the `use` verb on the policy.
|
||||
|
||||
Most Kubernetes pods are not created directly by users. Instead, they are
|
||||
typically created indirectly as part of a
|
||||
[Deployment](/docs/concepts/workloads/controllers/deployment/),
|
||||
[ReplicaSet](/docs/concepts/workloads/controllers/replicaset/), or other
|
||||
templated controller via the controller manager. Granting the controller access
|
||||
to the policy would grant access for *all* pods created by that the controller,
|
||||
so the preferred method for authorizing policies is to grant access to the
|
||||
pod's service account (see [example](#run-another-pod)).
|
||||
|
||||
### Via RBAC
|
||||
|
||||
[RBAC](/docs/admin/authorization/rbac/) is a standard Kubernetes authorization
|
||||
mode, and can easily be used to authorize use of policies.
|
||||
|
||||
First, a `Role` or `ClusterRole` needs to grant access to `use` the desired
|
||||
policies. The rules to grant access look like this:
|
||||
|
||||
```yaml
|
||||
kind: ClusterRole
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
metadata:
|
||||
name: <role name>
|
||||
rules:
|
||||
- apiGroups: ['extensions']
|
||||
resources: ['podsecuritypolicies']
|
||||
verbs: ['use']
|
||||
resourceNames:
|
||||
- <list of policies to authorize>
|
||||
```
|
||||
|
||||
Then the `(Cluster)Role` is bound to the authorized user(s):
|
||||
|
||||
```yaml
|
||||
kind: ClusterRoleBinding
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
metadata:
|
||||
name: <binding name>
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: <role name>
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
subjects:
|
||||
# Authorize specific service accounts:
|
||||
- kind: ServiceAccount
|
||||
name: <authorized service account name>
|
||||
namespace: <authorized pod namespace>
|
||||
# Authorize specific users (not recommended):
|
||||
- kind: User
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
name: <authorized user name>
|
||||
```
|
||||
|
||||
If a `RoleBinding` (not a `ClusterRoleBinding`) is used, it will only grant
|
||||
usage for pods being run in the same namespace as the binding. This can be
|
||||
paired with system groups to grant access to all pods run in the namespace:
|
||||
```yaml
|
||||
# Authorize all service accounts in a namespace:
|
||||
- kind: Group
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
name: system:serviceaccounts
|
||||
# Or equivalently, all authenticated users in a namespace:
|
||||
- kind: Group
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
name: system:authenticated
|
||||
```
|
||||
|
||||
For more examples of RBAC bindings, see [Role Binding
|
||||
Examples](docs/admin/authorization/rbac/#role-binding-examples). For a complete
|
||||
example of authorizing a PodSecurityPolicy, see
|
||||
[below](#example).
|
||||
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- The [Controller Manager](/docs/admin/kube-controller-manager/) must be run
|
||||
against [the secured API port](/docs/admin/accessing-the-api/), and must not
|
||||
have superuser permissions. Otherwise requests would bypass authentication and
|
||||
authorization modules, all PodSecurityPolicy objects would be allowed, and users
|
||||
would be able to create privileged containers. For more details on configuring
|
||||
Controller Manager authorization, see [Controller
|
||||
Roles](docs/admin/authorization/rbac/#controller-roles).
|
||||
|
||||
## Policy Order
|
||||
|
||||
In addition to restricting pod creation and update, pod security policies can
|
||||
also be used to provide default values for many of the fields that it
|
||||
controls. When multiple policies are available, the pod security policy
|
||||
controller selects policies in the following order:
|
||||
|
||||
1. If any policies successfully validate the pod without altering it, they are
|
||||
used.
|
||||
2. Otherwise, the first valid policy in alphabetical order is used.
|
||||
|
||||
## Example
|
||||
|
||||
_This example assumes you have a running cluster with the PodSecurityPolicy
|
||||
admission controller enabled and you have cluster admin privileges._
|
||||
|
||||
### Set up
|
||||
|
||||
Set up a namespace and a service account to act as for this example. We'll use
|
||||
this service account to mock a non-admin user.
|
||||
|
||||
```shell
|
||||
$ kubectl create namespace psp-example
|
||||
$ kubectl create serviceaccount -n psp-example fake-user
|
||||
$ kubectl create rolebinding -n psp-example fake-editor --clusterrole=edit --serviceaccount=psp-example:fake-user
|
||||
```
|
||||
|
||||
To make it clear which user we're acting as and save some typing, create 2
|
||||
aliases:
|
||||
|
||||
```shell
|
||||
$ alias kubectl-admin='kubectl -n psp-example'
|
||||
$ alias kubectl-user='kubectl --as=system:serviceaccount:psp-example:fake-user -n psp-example'
|
||||
```
|
||||
|
||||
### Create a policy and a pod
|
||||
|
||||
Define the example PodSecurityPolicy object in a file. This is a policy that
|
||||
simply prevents the creation of privileged pods.
|
||||
|
||||
{% include code.html language="yaml" file="example-psp.yaml" ghlink="/docs/concepts/policy/example-psp.yaml" %}
|
||||
|
||||
And create it with kubectl:
|
||||
|
||||
```shell
|
||||
$ kubectl-admin create -f example-psp.yaml
|
||||
```
|
||||
|
||||
Now, as the unprivileged user, try to create a simple pod:
|
||||
|
||||
```shell
|
||||
$ kubectl-user create -f- <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: pause
|
||||
spec:
|
||||
containers:
|
||||
- name: pause
|
||||
image: gcr.io/google-containers/pause
|
||||
EOF
|
||||
Error from server (Forbidden): error when creating "STDIN": pods "pause" is forbidden: unable to validate against any pod security policy: []
|
||||
```
|
||||
|
||||
**What happened?** Although the PodSecurityPolicy was created, neither the
|
||||
pod's service account nor `fake-user` have permission to use the new policy:
|
||||
|
||||
```shell
|
||||
$ kubectl-user auth can-i use podsecuritypolicy/example
|
||||
no
|
||||
```
|
||||
|
||||
Create the rolebinding to grant `fake-user` the `use` verb on the example
|
||||
policy:
|
||||
|
||||
_Note: This is not the recommended way! See the [next section](#run-another-pod)
|
||||
for the preferred approach._
|
||||
|
||||
```shell
|
||||
$ kubectl-admin create role psp:unprivileged \
|
||||
--verb=use \
|
||||
--resource=podsecuritypolicy \
|
||||
--resource-name=example
|
||||
role "psp:unprivileged" created
|
||||
$ kubectl-admin create rolebinding fake-user:psp:unprivileged \
|
||||
--role=psp:unprivileged \
|
||||
--serviceaccount=psp-example:fake-user
|
||||
rolebinding "fake-user:psp:unprivileged" created
|
||||
$ kubectl-user auth can-i use podsecuritypolicy/example
|
||||
yes
|
||||
```
|
||||
|
||||
Now retry creating the pod:
|
||||
|
||||
```shell
|
||||
$ kubectl-user create -f- <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: pause
|
||||
spec:
|
||||
containers:
|
||||
- name: pause
|
||||
image: gcr.io/google-containers/pause
|
||||
EOF
|
||||
pod "pause" created
|
||||
```
|
||||
|
||||
It works as expected! But any attempts to create a privileged pod should still
|
||||
be denied:
|
||||
|
||||
```shell
|
||||
$ kubectl-user create -f- <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: privileged
|
||||
spec:
|
||||
containers:
|
||||
- name: pause
|
||||
image: gcr.io/google-containers/pause
|
||||
securityContext:
|
||||
privileged: true
|
||||
EOF
|
||||
Error from server (Forbidden): error when creating "STDIN": pods "privileged" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
|
||||
```
|
||||
|
||||
Delete the pod before moving on:
|
||||
|
||||
```shell
|
||||
$ kubectl-user delete pause
|
||||
```
|
||||
|
||||
### Run another pod
|
||||
|
||||
Let's try that again, slightly differently:
|
||||
|
||||
```shell
|
||||
$ kubectl-user run pause --image=gcr.io/google-containers/pause
|
||||
deployment "pause" created
|
||||
$ kubectl-user get pods
|
||||
No resources found.
|
||||
$ kubectl-user get events | head -n 2
|
||||
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
|
||||
1m 2m 15 pause-7774d79b5 ReplicaSet Warning FailedCreate replicaset-controller Error creating: pods "pause-7774d79b5-" is forbidden: no providers available to validate pod request
|
||||
```
|
||||
|
||||
**What happened?** We already bound the `psp:unprivileged` role for our `fake-user`,
|
||||
why are we getting the error `Error creating: pods "pause-7774d79b5-" is
|
||||
forbidden: no providers available to validate pod request`? The answer lies in
|
||||
the source - `replicaset-controller`. Fake-user successfully created the
|
||||
deployment (which successfully created a replicaset), but when the replicaset
|
||||
went to create the pod it was not authorized to use the example
|
||||
podsecuritypolicy.
|
||||
|
||||
In order to fix this, bind the `psp:unprivileged` role to the pod's service
|
||||
account instead. In this case (since we didn't specify it) the service account
|
||||
is `default`:
|
||||
|
||||
```shell
|
||||
$ kubectl-admin create rolebinding default:psp:unprivileged \
|
||||
--role=psp:unprivileged \
|
||||
--serviceaccount=psp-example:default
|
||||
rolebinding "default:psp:unprivileged" created
|
||||
```
|
||||
|
||||
Now if you give it a minute to retry, the replicaset-controller should
|
||||
eventually succeed in creating the pod:
|
||||
|
||||
```shell
|
||||
$ kubectl-user get pods --watch
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
pause-7774d79b5-qrgcb 0/1 Pending 0 1s
|
||||
pause-7774d79b5-qrgcb 0/1 Pending 0 1s
|
||||
pause-7774d79b5-qrgcb 0/1 ContainerCreating 0 1s
|
||||
pause-7774d79b5-qrgcb 1/1 Running 0 2s
|
||||
^C
|
||||
```
|
||||
|
||||
### Clean up
|
||||
|
||||
Delete the namespace to clean up most of the example resources:
|
||||
|
||||
```shell
|
||||
$ kubectl-admin delete ns psp-example
|
||||
namespace "psp-example" deleted
|
||||
```
|
||||
|
||||
Note that `PodSecurityPolicy` resources are not namespaced, and must be cleaned
|
||||
up separately:
|
||||
|
||||
```shell
|
||||
$ kubectl-admin delete psp example
|
||||
podsecuritypolicy "example" deleted
|
||||
```
|
||||
|
||||
### Example Policies
|
||||
|
||||
This is the least restricted policy you can create, equivalent to not using the
|
||||
pod security policy admission controller:
|
||||
|
||||
{% include code.html language="yaml" file="privileged-psp.yaml" ghlink="/docs/concepts/policy/privileged-psp.yaml" %}
|
||||
|
||||
This is an example of a restrictive policy that requires users to run as an
|
||||
unprivileged user, blocks possible escalations to root, and requires use of
|
||||
several security mechanisms.
|
||||
|
||||
{% include code.html language="yaml" file="restricted-psp.yaml" ghlink="/docs/concepts/policy/restricted-psp.yaml" %}
|
||||
|
||||
## Policy Reference
|
||||
|
||||
### Host namespaces
|
||||
|
||||
**HostPID** - Controls whether the pod containers can share the host process ID
|
||||
namespace. Note that when paired with ptrace this can be used to escalate
|
||||
privileges outside of the container (ptrace is forbidden by default).
|
||||
|
||||
**HostIPC** - Controls whether the pod containers can share the host IPC
|
||||
namespace.
|
||||
|
||||
**HostNetwork** - Controls whether the pod may use the node network
|
||||
namespace. Doing so gives the pod access to the loopback device, services
|
||||
listening on localhost, and could be used to snoop on network activity of other
|
||||
pods on the same node.
|
||||
|
||||
**HostPorts** - Provides a whitelist of ranges of allowable ports in the host
|
||||
network namespace. Defined as a list of `HostPortRange`, with `min`(inclusive)
|
||||
and `max`(inclusive). Defaults to no allowed host ports.
|
||||
|
||||
**AllowedHostPaths** - See [Volumes and file systems](#volumes-and-file-systems).
|
||||
|
||||
### Volumes and file systems
|
||||
|
||||
**Volumes** - Provides a whitelist of allowed volume types. The allowable values
|
||||
correspond to the volume sources that are defined when creating a volume. For
|
||||
the complete list of volume types, see [Types of
|
||||
Volumes](/docs/concepts/storage/volumes/#types-of-volumes). Additionally, `*`
|
||||
may be used to allow all volume types.
|
||||
|
||||
The **recommended minimum set** of allowed volumes for new PSPs are:
|
||||
|
||||
- configMap
|
||||
- downwardAPI
|
||||
- emptyDir
|
||||
- persistentVolumeClaim
|
||||
- secret
|
||||
- projected
|
||||
|
||||
**FSGroup** - Controls the supplemental group applied to some volumes.
|
||||
|
||||
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
- *RunAsAny* - No default provided. Allows any `fsGroup` ID to be specified.
|
||||
|
||||
**AllowedHostPaths** - This specifies a whitelist of host paths that are allowed
|
||||
to be used by hostPath volumes. An empty list means there is no restriction on
|
||||
host paths used. This is defined as a list of objects with a single `pathPrefix`
|
||||
field, which allows hostPath volumes to mount a path that begins with an
|
||||
allowed prefix. For example:
|
||||
|
||||
```yaml
|
||||
allowedHostPaths:
|
||||
# This allows "/foo", "/foo/", "/foo/bar" etc., but
|
||||
# disallows "/fool", "/etc/foo" etc.
|
||||
# "/foo/../" is never valid.
|
||||
- pathPrefix: "/foo"
|
||||
```
|
||||
|
||||
_Note: There are many ways a container with unrestricted access to the host
|
||||
filesystem can escalate privileges, including reading data from other
|
||||
containers, and abusing the credentials of system services, such as Kubelet._
|
||||
|
||||
**ReadOnlyRootFilesystem** - Requires that containers must run with a read-only
|
||||
root filesystem (i.e. no writeable layer).
|
||||
|
||||
### Users and groups
|
||||
|
||||
**RunAsUser** - Controls the what user ID containers run as.
|
||||
|
||||
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
| Control Aspect | Field Name |
|
||||
| ---------------------------------------------------------------------- | ------------------------------------------- |
|
||||
| Running of privileged containers | `privileged` |
|
||||
| Default set of capabilities that will be added to a container | `defaultAddCapabilities` |
|
||||
| Capabilities that will be dropped from a container | `requiredDropCapabilities` |
|
||||
| Capabilities a container can request to be added | `allowedCapabilities` |
|
||||
| Controlling the usage of volume types | [`volumes`](#controlling-volumes) |
|
||||
| The use of host networking | [`hostNetwork`](#host-network) |
|
||||
| The use of host ports | `hostPorts` |
|
||||
| The use of host's PID namespace | `hostPID` |
|
||||
| The use of host's IPC namespace | `hostIPC` |
|
||||
| The SELinux context of the container | [`seLinux`](#selinux) |
|
||||
| The user ID | [`runAsUser`](#runasuser) |
|
||||
| Configuring allowable supplemental groups | [`supplementalGroups`](#supplementalgroups) |
|
||||
| Allocating an FSGroup that owns the pod's volumes | [`fsGroup`](#fsgroup) |
|
||||
| Requiring the use of a read only root file system | `readOnlyRootFilesystem` |
|
||||
| Running of a container that allow privilege escalation from its parent | [`allowPrivilegeEscalation`](#allowprivilegeescalation) |
|
||||
| Control whether a process can gain more privileges than its parent process | [`defaultAllowPrivilegeEscalation`](#defaultallowprivilegeescalation) |
|
||||
| Whitelist of allowed host paths | [`allowedHostPaths`](#allowedhostpaths) |
|
||||
|
||||
_Pod Security Policies_ are comprised of settings and strategies that
|
||||
control the security features a pod has access to. These settings fall
|
||||
into three categories:
|
||||
|
||||
- *Controlled by a Boolean*: Fields of this type default to the most
|
||||
restrictive value.
|
||||
- *Controlled by an allowable set*: Fields of this type are checked
|
||||
against the set to ensure their values are allowed.
|
||||
- *Controlled by a strategy*: Items that have a strategy to provide
|
||||
a mechanism to generate the value and a mechanism to ensure that a
|
||||
specified value falls into the set of allowable values.
|
||||
|
||||
|
||||
## Strategies
|
||||
|
||||
### RunAsUser
|
||||
|
||||
- *MustRunAs* - Requires a `range` to be configured. Uses the first value
|
||||
of the range as the default. Validates against the configured range.
|
||||
- *MustRunAsNonRoot* - Requires that the pod be submitted with a non-zero
|
||||
`runAsUser` or have the `USER` directive defined (using a numeric UID) in the
|
||||
image. No default provided. Setting `allowPrivilegeEscalation=false` is strongly
|
||||
recommended with this strategy.
|
||||
`runAsUser` or have the `USER` directive defined in the image. No default
|
||||
provided.
|
||||
- *RunAsAny* - No default provided. Allows any `runAsUser` to be specified.
|
||||
|
||||
**SupplementalGroups** - Controls which group IDs containers add.
|
||||
|
||||
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
- *RunAsAny* - No default provided. Allows any `supplementalGroups` to be
|
||||
specified.
|
||||
|
||||
### Privilege Escalation
|
||||
|
||||
These options control the `allowPrivilegeEscalation` container option. This bool
|
||||
directly controls whether the
|
||||
[`no_new_privs`](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt)
|
||||
flag gets set on the container process. This flag will prevent `setuid` binaries
|
||||
from changing the effective user ID, and prevent files from enabling extra
|
||||
capabilities (e.g. it will prevent the use of the `ping` tool). This behavior is
|
||||
required to effectively enforce `MustRunAsNonRoot`.
|
||||
|
||||
It defaults to `nil`. The default behavior of `nil` allows privilege escalation
|
||||
so as to not break setuid binaries. Setting it to `false` ensures that no child
|
||||
process of a container can gain more privileges than its parent.
|
||||
|
||||
**AllowPrivilegeEscalation** - Gates whether or not a user is allowed to set the
|
||||
security context of a container to `allowPrivilegeEscalation=true`. This
|
||||
defaults to allowed. When set to false, the container's
|
||||
`allowPrivilegeEscalation` is defaulted to false.
|
||||
|
||||
**DefaultAllowPrivilegeEscalation** - Sets the default for the
|
||||
`allowPrivilegeEscalation` option. The default behavior without this is to allow
|
||||
privilege escalation so as to not break setuid binaries. If that behavior is not
|
||||
desired, this field can be used to default to disallow, while still permitting
|
||||
pods to request `allowPrivilegeEscalation` explicitly.
|
||||
|
||||
### Capabilities
|
||||
|
||||
Linux capabilities provide a finer grained breakdown of the privileges
|
||||
traditionally associated with the superuser. Some of these capabilities can be
|
||||
used to escalate privileges or for container breakout, and may be restricted by
|
||||
the PodSecurityPolicy. For more details on Linux capabilities, see
|
||||
[capabilities(7)](http://man7.org/linux/man-pages/man7/capabilities.7.html).
|
||||
|
||||
The following fields take a list of capabilities, specified as the capability
|
||||
name in ALL_CAPS without the `CAP_` prefix.
|
||||
|
||||
**AllowedCapabilities** - Provides a whitelist of capabilities that may be added
|
||||
to a container. The default set of capabilities are implicitly allowed. The
|
||||
empty set means that no additional capabilities may be added beyond the default
|
||||
set. `*` can be used to allow all capabilities.
|
||||
|
||||
**RequiredDropCapabilities** - The capabilities which must be dropped from
|
||||
containers. These capabilities are removed from the default set, and must not be
|
||||
added. Capabilities listed in `RequiredDropCapabilities` must not be included in
|
||||
`AllowedCapabilities` or `DefaultAddCapabilities`.
|
||||
|
||||
**DefaultAddCapabilities** - The capabilities which are added to containers by
|
||||
default, in addition to the runtime defaults. See the [Docker
|
||||
documentation](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)
|
||||
for the default list of capabilities when using the Docker runtime.
|
||||
|
||||
### SELinux
|
||||
|
||||
- *MustRunAs* - Requires `seLinuxOptions` to be configured if not using
|
||||
@@ -495,29 +73,193 @@ pre-allocated values. Uses `seLinuxOptions` as the default. Validates against
|
||||
- *RunAsAny* - No default provided. Allows any `seLinuxOptions` to be
|
||||
specified.
|
||||
|
||||
### AppArmor
|
||||
### SupplementalGroups
|
||||
|
||||
Controlled via annotations on the PodSecurityPolicy. Refer to the [AppArmor
|
||||
documentation](/docs/tutorials/clusters/apparmor/#podsecuritypolicy-annotations).
|
||||
- *MustRunAs* - Requires at least one range to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against all ranges.
|
||||
- *RunAsAny* - No default provided. Allows any `supplementalGroups` to be
|
||||
specified.
|
||||
|
||||
### Seccomp
|
||||
### FSGroup
|
||||
|
||||
The use of seccomp profiles in pods can be controlled via annotations on the
|
||||
PodSecurityPolicy. Seccomp is an alpha feature in Kubernetes.
|
||||
- *MustRunAs* - Requires at least one range to be specified. Uses the
|
||||
minimum value of the first range as the default. Validates against the
|
||||
first ID in the first range.
|
||||
- *RunAsAny* - No default provided. Allows any `fsGroup` ID to be specified.
|
||||
|
||||
**seccomp.security.alpha.kubernetes.io/defaultProfileName** - Annotation that
|
||||
specifies the default seccomp profile to apply to containers. Possible values
|
||||
are:
|
||||
### Controlling Volumes
|
||||
|
||||
- `unconfined` - Seccomp is not applied to the container processes (this is the
|
||||
default in Kubernetes), if no alternative is provided.
|
||||
- `docker/default` - The Docker default seccomp profile is used.
|
||||
- `localhost/<path>` - Specify a profile as a file on the node located at
|
||||
`<seccomp_root>/<path>`, where `<seccomp_root>` is defined via the
|
||||
`--seccomp-profile-root` flag on the Kubelet.
|
||||
The usage of specific volume types can be controlled by setting the
|
||||
volumes field of the PSP. The allowable values of this field correspond
|
||||
to the volume sources that are defined when creating a volume:
|
||||
|
||||
**seccomp.security.alpha.kubernetes.io/allowedProfileNames** - Annotation that
|
||||
specifies which values are allowed for the pod seccomp annotations. Specified as
|
||||
a comma-delimited list of allowed values. Possible values are those listed
|
||||
above, plus `*` to allow all profiles. Absence of this annotation means that the
|
||||
default cannot be changed.
|
||||
1. azureFile
|
||||
1. azureDisk
|
||||
1. flocker
|
||||
1. flexVolume
|
||||
1. hostPath
|
||||
1. emptyDir
|
||||
1. gcePersistentDisk
|
||||
1. awsElasticBlockStore
|
||||
1. gitRepo
|
||||
1. secret
|
||||
1. nfs
|
||||
1. iscsi
|
||||
1. glusterfs
|
||||
1. persistentVolumeClaim
|
||||
1. rbd
|
||||
1. cinder
|
||||
1. cephFS
|
||||
1. downwardAPI
|
||||
1. fc
|
||||
1. configMap
|
||||
1. vsphereVolume
|
||||
1. quobyte
|
||||
1. photonPersistentDisk
|
||||
1. projected
|
||||
1. portworxVolume
|
||||
1. scaleIO
|
||||
1. storageos
|
||||
1. \* (allow all volumes)
|
||||
|
||||
The recommended minimum set of allowed volumes for new PSPs are
|
||||
configMap, downwardAPI, emptyDir, persistentVolumeClaim, secret, and projected.
|
||||
|
||||
### Host Network
|
||||
- *HostPorts*, default `empty`. List of `HostPortRange`, defined by `min`(inclusive) and `max`(inclusive), which define the allowed host ports.
|
||||
|
||||
### AllowPrivilegeEscalation
|
||||
|
||||
Gates whether or not a user is allowed to set the security context of a container
|
||||
to `allowPrivilegeEscalation=true`. This field defaults to `false`.
|
||||
|
||||
### DefaultAllowPrivilegeEscalation
|
||||
|
||||
Sets the default for the security context `AllowPrivilegeEscalation` of a container.
|
||||
This bool directly controls whether the `no_new_privs` flag gets set on the
|
||||
container process. It defaults to `nil`. The default behavior of `nil`
|
||||
allows privilege escalation so as to not break setuid binaries. Setting it to `false`
|
||||
ensures that no child process of a container can gain more privileges than
|
||||
its parent.
|
||||
|
||||
### AllowedHostPaths
|
||||
|
||||
This specifies a whitelist of host paths that are allowed to be used by Pods.
|
||||
An empty list means there is no restriction on host paths used.
|
||||
Each item in the list must specify a string value named `pathPrefix` that
|
||||
defines a host path to match. The value cannot be "`*`" though.
|
||||
An example is shown below:
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: custom-paths
|
||||
spec:
|
||||
allowedHostPaths:
|
||||
# This allows "/foo", "/foo/", "/foo/bar" etc., but
|
||||
# disallows "/fool", "/etc/foo" etc.
|
||||
- pathPrefix: "/foo"
|
||||
```
|
||||
|
||||
## Admission
|
||||
|
||||
[_Admission control_ with `PodSecurityPolicy`](/docs/admin/admission-controllers/#podsecuritypolicy)
|
||||
allows for control over the creation and modification of resources based on the
|
||||
capabilities allowed in the cluster.
|
||||
|
||||
Admission uses the following approach to create the final security context for
|
||||
the pod:
|
||||
|
||||
1. Retrieve all PSPs available for use.
|
||||
1. Generate field values for security context settings that were not specified
|
||||
on the request.
|
||||
1. Validate the final settings against the available policies.
|
||||
|
||||
If a matching policy is found, then the pod is accepted. If the
|
||||
request cannot be matched to a PSP, the pod is rejected.
|
||||
|
||||
A pod must validate every field against the PSP.
|
||||
|
||||
## Creating a Pod Security Policy
|
||||
|
||||
Here is an example Pod Security Policy. It has permissive settings for
|
||||
all fields
|
||||
|
||||
{% include code.html language="yaml" file="psp.yaml" ghlink="/docs/concepts/policy/psp.yaml" %}
|
||||
|
||||
Create the policy by downloading the example file and then running this command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f ./psp.yaml
|
||||
podsecuritypolicy "permissive" created
|
||||
```
|
||||
|
||||
## Getting a list of Pod Security Policies
|
||||
|
||||
To get a list of existing policies, use `kubectl get`:
|
||||
|
||||
```shell
|
||||
$ kubectl get psp
|
||||
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
|
||||
permissive false [] RunAsAny RunAsAny RunAsAny RunAsAny false [*]
|
||||
privileged true [] RunAsAny RunAsAny RunAsAny RunAsAny false [*]
|
||||
restricted false [] RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false [emptyDir secret downwardAPI configMap persistentVolumeClaim projected]
|
||||
```
|
||||
|
||||
## Editing a Pod Security Policy
|
||||
|
||||
To modify policy interactively, use `kubectl edit`:
|
||||
|
||||
```shell
|
||||
$ kubectl edit psp permissive
|
||||
```
|
||||
|
||||
This command will open a default text editor where you will be able to modify policy.
|
||||
|
||||
## Deleting a Pod Security Policy
|
||||
|
||||
Once you don't need a policy anymore, simply delete it with `kubectl`:
|
||||
|
||||
```shell
|
||||
$ kubectl delete psp permissive
|
||||
podsecuritypolicy "permissive" deleted
|
||||
```
|
||||
|
||||
## Enabling Pod Security Policies
|
||||
|
||||
In order to use Pod Security Policies in your cluster you must ensure the
|
||||
following
|
||||
|
||||
1. You have enabled the API type `extensions/v1beta1/podsecuritypolicy` (only for versions prior 1.6)
|
||||
1. [You have enabled the admission control plug-in `PodSecurityPolicy`](/docs/admin/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in)
|
||||
1. You have defined your policies
|
||||
|
||||
## Working With RBAC
|
||||
|
||||
In Kubernetes 1.5 and newer, you can use PodSecurityPolicy to control access to
|
||||
privileged containers based on user role and groups. Access to different
|
||||
PodSecurityPolicy objects can be controlled via authorization.
|
||||
|
||||
Note that [Controller Manager](/docs/admin/kube-controller-manager/) must be run
|
||||
against [the secured API port](/docs/admin/accessing-the-api/), and must not
|
||||
have superuser permissions. Otherwise requests would bypass authentication and
|
||||
authorization modules, all PodSecurityPolicy objects would be allowed,
|
||||
and user will be able to create privileged containers.
|
||||
|
||||
PodSecurityPolicy authorization uses the union of all policies available to the
|
||||
user creating the pod and
|
||||
[the service account specified on the pod](/docs/tasks/configure-pod-container/configure-service-account/).
|
||||
|
||||
Access to given PSP policies for a user will be effective only when creating
|
||||
Pods directly.
|
||||
|
||||
For pods created on behalf of a user, in most cases by Controller Manager,
|
||||
access should be given to the service account specified on the pod spec
|
||||
template. Examples of resources that create pods on behalf of a user are
|
||||
Deployments, ReplicaSets, etc.
|
||||
|
||||
For more details, see the
|
||||
[PodSecurityPolicy RBAC example](https://git.k8s.io/examples/staging/podsecuritypolicy/rbac/README.md)
|
||||
of applying PodSecurityPolicy to control access to privileged containers based
|
||||
on role and groups when deploying Pods directly.
|
||||
|
||||
@@ -1,27 +0,0 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: privileged
|
||||
annotations:
|
||||
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
|
||||
spec:
|
||||
privileged: true
|
||||
allowPrivilegeEscalation: true
|
||||
allowedCapabilities:
|
||||
- '*'
|
||||
volumes:
|
||||
- '*'
|
||||
hostNetwork: true
|
||||
hostPorts:
|
||||
- min: 0
|
||||
max: 65535
|
||||
hostIPC: true
|
||||
hostPID: true
|
||||
runAsUser:
|
||||
rule: 'RunAsAny'
|
||||
seLinux:
|
||||
rule: 'RunAsAny'
|
||||
supplementalGroups:
|
||||
rule: 'RunAsAny'
|
||||
fsGroup:
|
||||
rule: 'RunAsAny'
|
||||
@@ -1,10 +1,8 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: example
|
||||
name: permissive
|
||||
spec:
|
||||
privileged: false # Don't allow privileged pods!
|
||||
# The rest fills in some required fields.
|
||||
seLinux:
|
||||
rule: RunAsAny
|
||||
supplementalGroups:
|
||||
@@ -13,5 +11,10 @@ spec:
|
||||
rule: RunAsAny
|
||||
fsGroup:
|
||||
rule: RunAsAny
|
||||
hostPorts:
|
||||
- min: 8000
|
||||
max: 8080
|
||||
volumes:
|
||||
- '*'
|
||||
allowedCapabilities:
|
||||
- '*'
|
||||
@@ -1,48 +0,0 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: restricted
|
||||
annotations:
|
||||
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
|
||||
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
|
||||
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
|
||||
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
|
||||
spec:
|
||||
privileged: false
|
||||
# Required to prevent escalations to root.
|
||||
allowPrivilegeEscalation: false
|
||||
# This is redundant with non-root + disallow privilege escalation,
|
||||
# but we can provide it for defense in depth.
|
||||
requiredDropCapabilities:
|
||||
- ALL
|
||||
# Allow core volume types.
|
||||
volumes:
|
||||
- 'configMap'
|
||||
- 'emptyDir'
|
||||
- 'projected'
|
||||
- 'secret'
|
||||
- 'downwardAPI'
|
||||
# Assume that persistentVolumes set up by the cluster admin are safe to use.
|
||||
- 'persistentVolumeClaim'
|
||||
hostNetwork: false
|
||||
hostIPC: false
|
||||
hostPID: false
|
||||
runAsUser:
|
||||
# Require the container to run without root privileges.
|
||||
rule: 'MustRunAsNonRoot'
|
||||
seLinux:
|
||||
# This policy assumes the nodes are using AppArmor rather than SELinux.
|
||||
rule: 'RunAsAny'
|
||||
supplementalGroups:
|
||||
rule: 'MustRunAs'
|
||||
ranges:
|
||||
# Forbid adding the root group.
|
||||
- min: 1
|
||||
max: 65535
|
||||
fsGroup:
|
||||
rule: 'MustRunAs'
|
||||
ranges:
|
||||
# Forbid adding the root group.
|
||||
- min: 1
|
||||
max: 65535
|
||||
readOnlyRootFilesystem: false
|
||||
@@ -705,7 +705,7 @@ Complete this template for the scheduler pod:
|
||||
"containers": [
|
||||
{
|
||||
"name": "kube-scheduler",
|
||||
"image": "$HYPERKUBE_IMAGE",
|
||||
"image": "$HYBERKUBE_IMAGE",
|
||||
"command": [
|
||||
"/hyperkube",
|
||||
"scheduler",
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 87 KiB |
@@ -0,0 +1,31 @@
|
||||
kind: ConfigMap
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: example-config
|
||||
data:
|
||||
example.property.1: hello
|
||||
example.property.2: world
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: configmap-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: configmap-redis
|
||||
image: redis:3.0-nanoserver
|
||||
env:
|
||||
- name: EXAMPLE_PROPERTY_1
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: example-config
|
||||
key: example.property.1
|
||||
- name: EXAMPLE_PROPERTY_2
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: example-config
|
||||
key: example.property.2
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
@@ -0,0 +1,20 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: empty-dir-pod
|
||||
spec:
|
||||
containers:
|
||||
- image: redis:3.0-nanoserver
|
||||
name: empty-dir-redis
|
||||
volumeMounts:
|
||||
- mountPath: /cache
|
||||
name: cache-volume
|
||||
- mountPath: C:/scratch
|
||||
name: scratch-volume
|
||||
volumes:
|
||||
- name: cache-volume
|
||||
emptyDir: {}
|
||||
- name: scratch-volume
|
||||
emptyDir: {}
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
@@ -0,0 +1,18 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: hostpath-volume-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: hostpath-redis
|
||||
image: redis:3.0-nanoserver
|
||||
volumeMounts:
|
||||
- name: blah
|
||||
mountPath: "C:\\etc\\foo"
|
||||
readOnly: true
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
volumes:
|
||||
- name: blah
|
||||
hostPath:
|
||||
path: "C:\\etc\\foo"
|
||||
@@ -1,144 +1,263 @@
|
||||
---
|
||||
title: Windows Server Containers
|
||||
title: Using Windows Server Containers in Kubernetes
|
||||
---
|
||||
**Note:** These instructions were recently updated based on Windows Server platform enhancements and the Kubernetes v1.9 release
|
||||
{: .note}
|
||||
|
||||
Kubernetes version 1.5 introduces support for Windows Server Containers. In version 1.5, the Kubernetes control plane (API Server, Scheduler, Controller Manager, etc) continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server.
|
||||
Kubernetes version 1.5 introduced Alpha support for Windows Server Containers based on the Windows Server 2016 operating system. With the release of Windows Server version 1709 and using Kubernetes v1.9 users are able to deploy a Kubernetes cluster either on-premises or in a private/public cloud using a number of different network topologies and CNI plugins. Some key feature improvements for Windows Server Containers on Kubernetes include:
|
||||
- Improved support for pods! Shared network namespace (compartment) with multiple Windows Server containers (shared kernel)
|
||||
- Reduced network complexity by using a single network endpoint per pod
|
||||
- Kernel-Based load-balancing using the Virtual Filtering Platform (VFP) Hyper-v Switch Extension (analogous to Linux iptables)
|
||||
- Container Runtime Interface (CRI) pod and node level statistics
|
||||
- Support for kubeadm commands to add Windows Server nodes to a Kubernetes environment
|
||||
|
||||
**Note:** Windows Server Containers on Kubernetes is an Alpha feature in Kubernetes 1.5.
|
||||
The Kubernetes control plane (API Server, Scheduler, Controller Manager, etc) continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server 2016 or later
|
||||
|
||||
**Note:** Windows Server Containers on Kubernetes is a Beta feature in Kubernetes v1.9
|
||||
{: .note}
|
||||
|
||||
## Build
|
||||
We recommend using the release binaries that can be found at [https://github.com/kubernetes/kubernetes/releases](https://github.com/kubernetes/kubernetes/releases). Look for the Node Binaries section by visiting the binary downloads link.
|
||||
|
||||
If you wish to build the code yourself, please follow the next instructions:
|
||||
|
||||
1. Install the pre-requisites on a Linux host:
|
||||
|
||||
```
|
||||
sudo apt-get install curl git build-essential docker.io conntrack
|
||||
```
|
||||
2. Run the following commands to build kubelet and kube-proxy:
|
||||
|
||||
```bash
|
||||
K8SREPO="github.com/kubernetes/kubernetes"
|
||||
go get -d $K8SREPO
|
||||
# Note: the above command may spit out a message about
|
||||
# "no Go files in...", but it can be safely ignored!
|
||||
|
||||
cd $GOPATH/src/k8s.io/kubernetes
|
||||
# Build the kubelet
|
||||
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet
|
||||
|
||||
# Build the kube-proxy
|
||||
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy
|
||||
|
||||
# You will find the output binaries under the folder _output/local/bin/windows/
|
||||
```
|
||||
|
||||
More detailed build instructions will be maintained and kept up to date [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/compiling-kubernetes-binaries.md).
|
||||
|
||||
## Prerequisites
|
||||
In Kubernetes version 1.5, Windows Server Containers for Kubernetes is supported using the following:
|
||||
In Kubernetes version 1.9 or later, Windows Server Containers for Kubernetes are supported using the following:
|
||||
|
||||
1. Kubernetes control plane running on existing Linux infrastructure (version 1.5 or later).
|
||||
1. Kubernetes control plane running on existing Linux infrastructure (version 1.9 or later).
|
||||
2. Kubenet network plugin setup on the Linux nodes.
|
||||
3. Windows Server 2016 (RTM version 10.0.14393 or later).
|
||||
4. Docker Version 1.12.2-cs2-ws-beta or later for Windows Server nodes (Linux nodes and Kubernetes control plane can run any Kubernetes supported Docker Version).
|
||||
3. Windows Server 2016 RTM or later. Windows Server version 1709 or later is preferred; it unlocks key capabilities like shared network namespace.
|
||||
4. Docker Version 17.06.1-ee-2 or later for Windows Server nodes (Linux nodes and Kubernetes control plane can run any Kubernetes supported Docker Version).
|
||||
|
||||
## Networking
|
||||
Network is achieved using L3 routing. Because third-party networking plugins (e.g. flannel, calico, etc) don't natively work on Windows Server, existing technology that is built into the Windows and Linux operating systems is relied on. In this L3 networking approach, a /16 subnet is chosen for the cluster nodes, and a /24 subnet is assigned to each worker node. All pods on a given worker node will be connected to the /24 subnet. This allows pods on the same node to communicate with each other. In order to enable networking between pods running on different nodes, routing features that are built into Windows Server 2016 and Linux are used.
|
||||
There are several supported network configurations with Kubernetes v1.9 on Windows, including both Layer-3 routed and overlay topologies using third-party network plugins.
|
||||
|
||||
1. [Upstream L3 Routing](#upstream-l3-routing-topology) - IP routes configured in upstream ToR
|
||||
2. [Host-Gateway](#host-gateway-topology) - IP routes configured on each host
|
||||
3. [Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay](#using-ovn-with-ovs) - overlay networks (supports STT and Geneve tunneling types)
|
||||
4. [Future - In Review] Overlay - VXLAN or IP-in-IP encapsulation using Flannel
|
||||
5. [Future] Layer-3 Routing with BGP (Calico)
|
||||
|
||||
The selection of which network configuration and topology to deploy depends on the physical network topology and a user's ability to configure routes, performance concerns with encapsulation, and requirement to integrate with third-party network plugins.
|
||||
|
||||
### Future CNI Plugins
|
||||
An additional two CNI plugins [win-l2bridge (host-gateway) and win-overlay (vxlan)] are in [PR review](https://github.com/containernetworking/plugins/pull/85). These two CNI plugins, when ready, can either be used directly or with Flannel.
|
||||
|
||||
### Linux
|
||||
The above networking approach is already supported on Linux using a bridge interface, which essentially creates a private network local to the node. Similar to the Windows side, routes to all other pod CIDRs must be created in order to send packets via the "public" NIC.
|
||||
The above networking approaches are already supported on Linux using a bridge interface, which essentially creates a private network local to the node. Similar to the Windows side, routes to all other pod CIDRs must be created in order to send packets via the "public" NIC.
|
||||
|
||||
### Windows
|
||||
Windows supports the CNI network model and uses plugins to interface with the Windows Host Networking Service (HNS) to configure host networking and policy. At the time of this writing, the only publicly available CNI plugin from Microsoft is built from a private repo and available here [wincni.exe](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/cni/wincni.exe). It uses an l2bridge network created through the Windows Host Networking Service (HNS) by an administrator using HNS PowerShell commands on each node as documented in the [Windows Host Setup](#windows-host-setup) section below. Source code for the future CNI plugins will be made available publicly.
|
||||
|
||||
#### Upstream L3 Routing Topology
|
||||
In this topology, networking is achieved using L3 routing with static IP routes configured in an upstream Top of Rack (ToR) switch/router. Each cluster node is connected to the management network with a host IP. Additionally, each node uses a local 'l2bridge' network with a pod CIDR assigned. All pods on a given worker node will be connected to the pod CIDR subnet ('l2bridge' network). In order to enable network communication between pods running on different nodes, the upstream router has static routes configured with pod CIDR prefix => Host IP.
|
||||
|
||||
Each Window Server node should have the following configuration:
|
||||
|
||||
1. Two NICs (virtual networking adapters) are required on each Windows Server node - The two Windows container networking modes of interest (transparent and L2 bridge) use an external Hyper-V virtual switch. This means that one of the NICs is entirely allocated to the bridge, creating the need for the second NIC.
|
||||
2. Transparent container network created - This is a manual configuration step and is shown in **_Route Setup_** section below.
|
||||
3. RRAS (Routing) Windows feature enabled - Allows routing between NICs on the box, and also "captures" packets that have the destination IP of a POD running on the node. To enable, open "Server Manager". Click on "Roles", "Add Roles". Click "Next". Select "Network Policy and Access Services". Click on "Routing and Remote Access Service" and the underlying checkboxes.
|
||||
4. Routes defined pointing to the other pod CIDRs via the "public" NIC - These routes are added to the built-in routing table as shown in **_Route Setup_** section below.
|
||||
The following diagram illustrates the Windows Server networking setup for Kubernetes using Upstream L3 Routing Setup:
|
||||
|
||||
|
||||
The following diagram illustrates the Windows Server networking setup for Kubernetes Setup:
|
||||
|
||||
#### Host-Gateway Topology
|
||||
This topology is similar to the Upstream L3 Routing topology with the only difference being that static IP routes are configured directly on each cluster node and not in the upstream ToR. Each node uses a local 'l2bridge' network with a pod CIDR assigned as before and has routing table entries for all other pod CIDR subnets assigned to the remote cluster nodes.
|
||||
|
||||
#### Using OVN with OVS
|
||||
The following diagram gives a general overview of the architecture and interaction between components:
|
||||
|
||||
|
||||
|
||||
(The above image is from [https://github.com/openvswitch/ovn-kubernetes#overlay-mode-architecture-diagram](https://github.com/openvswitch/ovn-kubernetes#overlay-mode-architecture-diagram))
|
||||
|
||||
Due to its architecture, OVN has a central component which stores your networking intent in a database. Other components i.e. kube-apiserver, kube-controller-manager, kube-scheduler etc. can be deployed on that central node as well.
|
||||
|
||||
## Setting up Windows Server Containers on Kubernetes
|
||||
To run Windows Server Containers on Kubernetes, you'll need to set up both your host machines and the Kubernetes node components for Windows and setup Routes for Pod communication on different nodes.
|
||||
To run Windows Server Containers on Kubernetes, you'll need to set up both your host machines and the Kubernetes node components for Windows. Depending on your network topology, routes may need to be set up for pod communication on different nodes.
|
||||
|
||||
### Host Setup
|
||||
**Windows Host Setup**
|
||||
|
||||
1. Windows Server container host running Windows Server 2016 and Docker v1.12. Follow the setup instructions outlined by this blog post: https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/quick_start_windows_server.
|
||||
2. DNS support for Windows recently got merged to docker master and is currently not supported in a stable docker release. To use DNS build docker from master or download the binary from [Docker master](https://master.dockerproject.org/).
|
||||
3. Pull the `apprenda/pause` image from `https://hub.docker.com/r/apprenda/pause`.
|
||||
4. RRAS (Routing) Windows feature enabled.
|
||||
5. Install a VMSwitch of type `Internal`, by running `New-VMSwitch -Name KubeProxySwitch -SwitchType Internal` command in *PowerShell* window. This will create a new Network Interface with name `vEthernet (KubeProxySwitch)`. This interface will be used by kube-proxy to add Service IPs.
|
||||
#### For 1. Upstream L3 Routing Topology and 2. Host-Gateway Topology
|
||||
|
||||
**Linux Host Setup**
|
||||
##### Linux Host Setup
|
||||
|
||||
1. Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using.
|
||||
2. CNI network plugin installed.
|
||||
2. Configure Linux Master node using steps [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/creating-a-linux-master.md)
|
||||
3. [Optional] CNI network plugin installed.
|
||||
|
||||
### Component Setup
|
||||
##### Windows Host Setup
|
||||
|
||||
Requirements
|
||||
|
||||
* Git
|
||||
* Go 1.7.1+
|
||||
* make (if using Linux or MacOS)
|
||||
* Important notes and other dependencies are listed [here](https://git.k8s.io/community/contributors/devel/development.md#building-kubernetes-on-a-local-osshell-environment)
|
||||
1. Windows Server container host running the required Windows Server and Docker versions. Follow the setup instructions outlined by this help topic: https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server.
|
||||
2. [Build](#Build) or download kubelet.exe, kube-proxy.exe, and kubectl.exe using instructions
|
||||
3. Copy Node spec file (kube config) from Linux master node with X.509 keys
|
||||
4. Create the HNS Network, ensure the correct CNI network config, and start kubelet.exe using this script [start-kubelet.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubelet.ps1)
|
||||
5. Start kube-proxy using this script [start-kubeproxy.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubeproxy.ps1)
|
||||
6. [Only required for #2 Host-Gateway mode] Add static routes on Windows host using this script [AddRoutes.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/AddRoutes.ps1)
|
||||
|
||||
**kubelet**
|
||||
More detailed instructions can be found [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows.md).
|
||||
|
||||
To build the *kubelet*, run:
|
||||
**Windows CNI Config Example**
|
||||
Today, Windows CNI plugin is based on wincni.exe code with the following example, configuration file.
|
||||
|
||||
1. `cd $GOPATH/src/k8s.io/kubernetes`
|
||||
2. Build *kubelet*
|
||||
1. Linux/MacOS: `KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet`
|
||||
2. Windows: `go build cmd/kubelet/kubelet.go`
|
||||
Note: this file assumes that a user previous created 'l2bridge' host networks on each Windows node using `<Verb>-HNSNetwork` cmdlets as shown in the `start-kubelet.ps1` and `start-kubeproxy.ps1` scripts linked above
|
||||
|
||||
**kube-proxy**
|
||||
|
||||
To build *kube-proxy*, run:
|
||||
|
||||
1. `cd $GOPATH/src/k8s.io/kubernetes`
|
||||
2. Build *kube-proxy*
|
||||
1. Linux/MacOS: `KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy`
|
||||
2. Windows: `go build cmd/kube-proxy/proxy.go`
|
||||
|
||||
### Route Setup
|
||||
The below example setup assumes one Linux and two Windows Server 2016 nodes and a cluster CIDR 192.168.0.0/16
|
||||
|
||||
| Hostname | Routable IP address | Pod CIDR |
|
||||
| --- | --- | --- |
|
||||
| Lin01 | `<IP of Lin01 host>` | 192.168.0.0/24 |
|
||||
| Win01 | `<IP of Win01 host>` | 192.168.1.0/24 |
|
||||
| Win02 | `<IP of Win02 host>` | 192.168.2.0/24 |
|
||||
|
||||
**Lin01**
|
||||
|
||||
```
|
||||
ip route add 192.168.1.0/24 via <IP of Win01 host>
|
||||
ip route add 192.168.2.0/24 via <IP of Win02 host>
|
||||
```json
|
||||
{
|
||||
"cniVersion": "0.2.0",
|
||||
"name": "l2bridge",
|
||||
"type": "wincni.exe",
|
||||
"master": "Ethernet",
|
||||
"ipam": {
|
||||
"environment": "azure",
|
||||
"subnet": "10.10.187.64/26",
|
||||
"routes": [{
|
||||
"GW": "10.10.187.66"
|
||||
}]
|
||||
},
|
||||
"dns": {
|
||||
"Nameservers": [
|
||||
"11.0.0.10"
|
||||
]
|
||||
},
|
||||
"AdditionalArgs": [{
|
||||
"Name": "EndpointPolicy",
|
||||
"Value": {
|
||||
"Type": "OutBoundNAT",
|
||||
"ExceptionList": [
|
||||
"11.0.0.0/8",
|
||||
"10.10.0.0/16",
|
||||
"10.127.132.128/25"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"Name": "EndpointPolicy",
|
||||
"Value": {
|
||||
"Type": "ROUTE",
|
||||
"DestinationPrefix": "11.0.0.0/8",
|
||||
"NeedEncap": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"Name": "EndpointPolicy",
|
||||
"Value": {
|
||||
"Type": "ROUTE",
|
||||
"DestinationPrefix": "10.127.132.213/32",
|
||||
"NeedEncap": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Win01**
|
||||
#### For 3. Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay
|
||||
|
||||
##### Linux Host Setup
|
||||
|
||||
Setting up the central node and the components needed is out of scope of this document. You can read [these instructions](https://github.com/openvswitch/ovn-kubernetes#k8s-master-node-initialization) for that.
|
||||
|
||||
Adding a Linux minion is also out of scope and you can read it here: [Linux minion](https://github.com/openvswitch/ovn-kubernetes#k8s-minion-node-initializations).
|
||||
|
||||
|
||||
##### Windows Host Setup
|
||||
|
||||
Adding a Windows minion requires you to install OVS and OVN binaries. Windows Server container host running the required Windows Server and Docker versions. Follow the setup instructions outlined by [this help topic](https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server). This type of deployment is supported starting with Windows Server 2016 RTM.
|
||||
|
||||
Compiling OVS and generating the installer will not be treated in this document. For a step by step instruction please visit [this link](http://docs.openvswitch.org/en/latest/intro/install/windows/#open-vswitch-on-windows).
|
||||
For a prebuilt certified installer please visit [this link](https://cloudbase.it/openvswitch/#download) and download the latest version of it.
|
||||
|
||||
The following guide uses the prebuilt certified installer.
|
||||
|
||||
Installing OVS can be done either via the GUI dialogs or unattended. Adding a Windows host to your setup requires you to have `OVN Host` together with the default installation features. Below is the dialog image on what needs to be installed:
|
||||
|
||||
|
||||
|
||||
For an unattended installation please use the following command:
|
||||
```
|
||||
docker network create -d transparent --gateway 192.168.1.1 --subnet 192.168.1.0/24 <network name>
|
||||
# A bridge is created with Adapter name "vEthernet (HNSTransparent)". Set its IP address to transparent network gateway
|
||||
netsh interface ipv4 set address "vEthernet (HNSTransparent)" addr=192.168.1.1
|
||||
route add 192.168.0.0 mask 255.255.255.0 192.168.0.1 if <Interface Id of the Routable Ethernet Adapter> -p
|
||||
route add 192.168.2.0 mask 255.255.255.0 192.168.2.1 if <Interface Id of the Routable Ethernet Adapter> -p
|
||||
cmd /c 'msiexec /i openvswitch.msi ADDLOCAL="OpenvSwitchCLI,OpenvSwitchDriver,OVNHost" /qn'
|
||||
```
|
||||
|
||||
**Win02**
|
||||
The installer propagates new environment variables. Please open a new command shell or logoff/logon to ensure the environment variables are refreshed.
|
||||
|
||||
For overlay, OVS on Windows requires a transparent docker network to function properly. Please use the following to create a transparent docker network which will be used by OVS. From powershell:
|
||||
```
|
||||
docker network create -d transparent --gateway 192.168.2.1 --subnet 192.168.2.0/24 <network name>
|
||||
# A bridge is created with Adapter name "vEthernet (HNSTransparent)". Set its IP address to transparent network gateway
|
||||
netsh interface ipv4 set address "vEthernet (HNSTransparent)" addr=192.168.2.1
|
||||
route add 192.168.0.0 mask 255.255.255.0 192.168.0.1 if <Interface Id of the Routable Ethernet Adapter> -p
|
||||
route add 192.168.1.0 mask 255.255.255.0 192.168.1.1 if <Interface Id of the Routable Ethernet Adapter> -p
|
||||
docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET `
|
||||
-o com.docker.network.windowsshim.interface="$INTERFACE_ALIAS" external
|
||||
```
|
||||
Where $SUBNET is the minion subnet which will be used to spawn pods on (the one which will be used by kubernetes), $GATEWAY_IP is the first IP of the $SUBNET and $INTERFACE_ALIAS is the interface used for creating the overlay tunnels (must have connectivity with the rests of the OVN hosts).
|
||||
Example:
|
||||
```
|
||||
docker network create -d transparent --gateway 10.0.1.1 --subnet 10.0.1.0/24 `
|
||||
-o com.docker.network.windowsshim.interface="Ethernet0" external
|
||||
```
|
||||
After creating the docker network please run the next commands from powershell. (creates an OVS bridge, adds the interface under the bridge and enables the OVS forwarding switch extension)
|
||||
```
|
||||
$a = Get-NetAdapter | where Name -Match HNSTransparent
|
||||
Rename-NetAdapter $a[0].Name -NewName HNSTransparent
|
||||
Stop-Service ovs-vswitchd -force; Disable-VMSwitchExtension "Cloudbase Open vSwitch Extension";
|
||||
ovs-vsctl --no-wait del-br br-ex
|
||||
ovs-vsctl --no-wait --may-exist add-br br-ex
|
||||
ovs-vsctl --no-wait add-port br-ex HNSTransparent -- set interface HNSTransparent type=internal
|
||||
ovs-vsctl --no-wait add-port br-ex $INTERFACE_ALIAS
|
||||
Enable-VMSwitchExtension "Cloudbase Open vSwitch Extension"; sleep 2; Restart-Service ovs-vswitchd
|
||||
```
|
||||
Besides of the above, setting up a Windows host is the same as the Linux host. Follow the steps from [here](https://github.com/openvswitch/ovn-kubernetes#k8s-minion-node-initializations).
|
||||
|
||||
**Windows CNI Setup**
|
||||
|
||||
Today, Windows OVN&OVS CNI plugin is based on ovn_cni.exe which can be downloaded from [here](https://cloudbase.it/downloads/ovn_cni.exe). A sample of CNI config file is the following:
|
||||
```
|
||||
{
|
||||
"name": "net",
|
||||
"type": "ovn_cni.exe",
|
||||
"bridge": "br-int",
|
||||
"isGateway": "true",
|
||||
"ipMasq": "false",
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"subnet": "$SUBNET"
|
||||
}
|
||||
}
|
||||
```
|
||||
Where $SUBNET is the subnet that was used in the previous ```docker network create``` command.
|
||||
|
||||
For a complete guide on Google Cloud Platform (GCP), namely Google Compute Engine (GCE) visit [this](https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster#heterogeneous-kubernetes-cluster-on-top-of-ovn).
|
||||
|
||||
For a complete guide on Amazon Web Services (AWS) visit [this](https://github.com/justeat/kubernetes-windows-aws-ovs#kubernetes-on-windows-in-aws-using-ovn).
|
||||
|
||||
## Starting the Cluster
|
||||
To start your cluster, you'll need to start both the Linux-based Kubernetes control plane, and the Windows Server-based Kubernetes node components.
|
||||
To start your cluster, you'll need to start both the Linux-based Kubernetes control plane, and the Windows Server-based Kubernetes node components (kubelet and kube-proxy). For the OVS & OVN only the kubelet is required.
|
||||
|
||||
## Starting the Linux-based Control Plane
|
||||
Use your preferred method to start Kubernetes cluster on Linux. Please note that Cluster CIDR might need to be updated.
|
||||
## Starting the Windows Node Components
|
||||
To start kubelet on your Windows node:
|
||||
Run the following in a PowerShell window. Be aware that if the node reboots or the process exits, you will have to rerun the commands below to restart the kubelet.
|
||||
|
||||
1. Set environment variable *CONTAINER_NETWORK* value to the docker container network to use
|
||||
`$env:CONTAINER_NETWORK = "<docker network>"`
|
||||
|
||||
2. Run *kubelet* executable using the below command
|
||||
`kubelet.exe --hostname-override=<ip address/hostname of the windows node> --pod-infra-container-image="apprenda/pause" --resolv-conf="" --api_servers=<api server location>`
|
||||
|
||||
To start kube-proxy on your Windows node:
|
||||
|
||||
Run the following in a PowerShell window with administrative privileges. Be aware that if the node reboots or the process exits, you will have to rerun the commands below to restart the kube-proxy.
|
||||
|
||||
1. Set environment variable *INTERFACE_TO_ADD_SERVICE_IP* value to `vEthernet (KubeProxySwitch)` which we created in **_Windows Host Setup_** above
|
||||
`$env:INTERFACE_TO_ADD_SERVICE_IP = "vEthernet (KubeProxySwitch)"`
|
||||
|
||||
2. Run *kube-proxy* executable using the below command
|
||||
`.\proxy.exe --v=3 --proxy-mode=userspace --hostname-override=<ip address/hostname of the windows node> --master=<api server location> --bind-address=<ip address of the windows node>`
|
||||
|
||||
## Scheduling Pods on Windows
|
||||
Because your cluster has both Linux and Windows nodes, you must explicitly set the nodeSelector constraint to be able to schedule Pods to Windows nodes. You must set nodeSelector with the label beta.kubernetes.io/os to the value windows; see the following example:
|
||||
Because your cluster has both Linux and Windows nodes, you must explicitly set the nodeSelector constraint to be able to schedule pods to Windows nodes. You must set nodeSelector with the label beta.kubernetes.io/os to the value windows; see the following example:
|
||||
|
||||
```
|
||||
```yaml
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "Pod",
|
||||
@@ -166,9 +285,161 @@ Because your cluster has both Linux and Windows nodes, you must explicitly set t
|
||||
}
|
||||
}
|
||||
```
|
||||
## Support for kubeadm join
|
||||
|
||||
## Known Limitations:
|
||||
1. There is no network namespace in Windows and as a result currently only one container per pod is supported.
|
||||
2. Secrets currently do not work because of a bug in Windows Server Containers described [here](https://github.com/docker/docker/issues/28401).
|
||||
3. ConfigMaps have not been implemented yet.
|
||||
4. `kube-proxy` implementation uses `netsh portproxy` and as it only supports TCP, DNS currently works only if the client retries DNS query using TCP.
|
||||
If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
|
||||
and your networking is setup correctly using one of the methods listed above (networking is setup outside of kubeadm), you can use kubeadm to add a Windows node to your cluster. At a high level, you first have to initialize the master with kubeadm (Linux), then set up the CNI based networking (outside of kubeadm), and finally start joining Windows or Linux worker nodes to the cluster. For additional documentation and reference material, visit the kubeadm link above.
|
||||
|
||||
The kubeadm binary can be found at [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases), inside the node binaries archive. Adding a Windows node is not any different than adding a Linux node:
|
||||
|
||||
`kubeadm.exe join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>`
|
||||
|
||||
See [joining-your-nodes](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#44-joining-your-nodes) for more details.
|
||||
|
||||
## Supported Features
|
||||
|
||||
### Secrets and ConfigMaps
|
||||
Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be used as environment variables. See limitations section below for additional details.
|
||||
|
||||
**Examples:**
|
||||
|
||||
Windows pod with secrets mapped to environment variables
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: mysecret
|
||||
type: Opaque
|
||||
data:
|
||||
username: YWRtaW4=
|
||||
password: MWYyZDFlMmU2N2Rm
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: mypod-secret
|
||||
spec:
|
||||
containers:
|
||||
- name: mypod-secret
|
||||
image: redis:3.0-nanoserver
|
||||
env:
|
||||
- name: USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: username
|
||||
- name: PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: password
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
```
|
||||
|
||||
Windows pod with configMap values mapped to environment variables
|
||||
|
||||
```yaml
|
||||
kind: ConfigMap
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: example-config
|
||||
data:
|
||||
example.property.1: hello
|
||||
example.property.2: world
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: configmap-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: configmap-redis
|
||||
image: redis:3.0-nanoserver
|
||||
env:
|
||||
- name: EXAMPLE_PROPERTY_1
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: example-config
|
||||
key: example.property.1
|
||||
- name: EXAMPLE_PROPERTY_2
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: example-config
|
||||
key: example.property.2
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
```
|
||||
|
||||
### Volumes
|
||||
Some supported Volume Mounts are local, emptyDir, hostPath. One thing to remember is that paths must either be escaped, or use forward slashes, for example `mountPath: "C:\\etc\\foo"` or `mountPath: "C:/etc/foo"`.
|
||||
|
||||
Persistent Volume Claims are supported for supported volume types.
|
||||
|
||||
**Examples:**
|
||||
|
||||
Windows pod with a hostPath volume
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: hostpath-volume-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: hostpath-redis
|
||||
image: redis:3.0-nanoserver
|
||||
volumeMounts:
|
||||
- name: blah
|
||||
mountPath: "C:\\etc\\foo"
|
||||
readOnly: true
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
volumes:
|
||||
- name: blah
|
||||
hostPath:
|
||||
path: "C:\\etc\\foo"
|
||||
```
|
||||
|
||||
Windows pod with multiple emptyDir volumes
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: empty-dir-pod
|
||||
spec:
|
||||
containers:
|
||||
- image: redis:3.0-nanoserver
|
||||
name: empty-dir-redis
|
||||
volumeMounts:
|
||||
- mountPath: /cache
|
||||
name: cache-volume
|
||||
- mountPath: C:/scratch
|
||||
name: scratch-volume
|
||||
volumes:
|
||||
- name: cache-volume
|
||||
emptyDir: {}
|
||||
- name: scratch-volume
|
||||
emptyDir: {}
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
```
|
||||
|
||||
### Metrics
|
||||
|
||||
Windows Stats use a hybrid model: pod and container level stats come from CRI (via dockershim), while node level stats come from the "winstats" package that exports cadvisor like datastructures using windows specific perf counters from the node.
|
||||
|
||||
## Known Limitations for Windows Server Containers with v1.9
|
||||
Some of these limitations will be addressed by the community in future releases of Kubernetes
|
||||
- Shared network namespace (compartment) with multiple Windows Server containers (shared kernel) per pod is only supported on Windows Server 1709 or later
|
||||
- Using Secrets and ConfigMaps as volume mounts is not supported
|
||||
- The StatefulSet functionality for stateful applications is not supported
|
||||
- Horizontal Pod Autoscaling for Windows Server Container pods has not been verified to work end-to-end
|
||||
- Hyper-V Containers are not supported
|
||||
|
||||
|
||||
> As of this writing, the Kube-proxy binary requires a pending Kubernetes [pull request](https://github.com/kubernetes/kubernetes/pull/56529) to work properly. You may need to [build](#build) the binaries manually to work around this.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 104 KiB |
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"cniVersion": "0.2.0",
|
||||
"name": "l2bridge",
|
||||
"type": "wincni.exe",
|
||||
"master": "Ethernet",
|
||||
"ipam": {
|
||||
"environment": "azure",
|
||||
"subnet": "10.10.187.64/26",
|
||||
"routes": [
|
||||
{
|
||||
"GW": "10.10.187.66"
|
||||
}
|
||||
]
|
||||
},
|
||||
"dns": {
|
||||
"Nameservers": [
|
||||
"11.0.0.10"
|
||||
]
|
||||
},
|
||||
"AdditionalArgs": [
|
||||
{
|
||||
"Name": "EndpointPolicy",
|
||||
"Value": {
|
||||
"Type": "OutBoundNAT",
|
||||
"ExceptionList": [
|
||||
"11.0.0.0/8",
|
||||
"10.10.0.0/16",
|
||||
"10.127.132.128/25"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"Name": "EndpointPolicy",
|
||||
"Value": {
|
||||
"Type": "ROUTE",
|
||||
"DestinationPrefix": "11.0.0.0/8",
|
||||
"NeedEncap": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"Name": "EndpointPolicy",
|
||||
"Value": {
|
||||
"Type": "ROUTE",
|
||||
"DestinationPrefix": "10.127.132.213/32",
|
||||
"NeedEncap": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: mysecret
|
||||
type: Opaque
|
||||
data:
|
||||
username: YWRtaW4=
|
||||
password: MWYyZDFlMmU2N2Rm
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: mypod-secret
|
||||
spec:
|
||||
containers:
|
||||
- name: mypod-secret
|
||||
image: redis:3.0-nanoserver
|
||||
env:
|
||||
- name: USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: username
|
||||
- name: PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: password
|
||||
nodeSelector:
|
||||
beta.kubernetes.io/os: windows
|
||||
@@ -55,8 +55,7 @@ $ kubectl proxy --port=8080 &
|
||||
|
||||
See [kubectl proxy](/docs/user-guide/kubectl/{{page.version}}/#proxy) for more details.
|
||||
|
||||
Then you can explore the API with curl, wget, or a browser, replacing localhost
|
||||
with [::1] for IPv6, like so:
|
||||
Then you can explore the API with curl, wget, or a browser, like so:
|
||||
|
||||
```shell
|
||||
$ curl http://localhost:8080/api/
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: frontend
|
||||
spec:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
@@ -44,4 +45,4 @@ spec:
|
||||
accessModes: [ "ReadWriteOnce" ]
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
storage: 1Gi
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ReplicationController
|
||||
metadata:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ReplicationController
|
||||
metadata:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
|
||||
Reference in New Issue
Block a user