whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into release-1.8

Browse Source
This commit is contained in:
zacharysarah
2017-12-06 14:45:26 -06:00
parent 538dd33ff2 93646b023a
commit 258fc90f5a
50 changed files with 1010 additions and 440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concepts/api-extension/custom-resources.md
+19 -19
View File
@@ -48,24 +48,24 @@ When creating a new API, consider whether to [aggregate your API with the Kubern
#### Declarative APIs
In a Declarative API, typically:
- your API consists of a relatively small number of relatively small objects (resources).
- the objects define configuration of applications or infrastructure
- the objects are updated relatively infrequently
- humans often need to read and write the objects
- the main operations on the objects are CRUD-y (creating, reading, updating and deleting)
- transactions across objects are not required: the API represents a desired state, not an exact state.
- Your API consists of a relatively small number of relatively small objects (resources).
- The objects define configuration of applications or infrastructure.
- The objects are updated relatively infrequently.
- Humans often need to read and write the objects.
- The main operations on the objects are CRUD-y (creating, reading, updating and deleting).
- Transactions across objects are not required: the API represents a desired state, not an exact state.
Imperative APIs are not declarative.
Signs that your API might not be declarative include:
- the client says "do this", and then gets a synchornous response back when it is done.
- the client says "do this", and then gets an operation ID back, and has to check a separate Operation objects to determine completion of the request.
- you talk about Remote Procedure Calls (RPCs)
- directly stoing large amounts of data (e.g. > a few kB per object, or >1000s of objects)
- high bandwidth access (10s of requests per second sustained) needed
- store end-user data (such as images, PII, etc) or other large-scale data processed by applications
- the natural operations on the objects are not CRUD-y.
- the API is not easily modeled as objects.
- you chose to represent pending operations with an operation ID or operation object.
- The client says "do this", and then gets a synchornous response back when it is done.
- The client says "do this", and then gets an operation ID back, and has to check a separate Operation objects to determine completion of the request.
- You talk about Remote Procedure Calls (RPCs).
- Directly storing large amounts of data (e.g. > a few kB per object, or >1000s of objects).
- High bandwidth access (10s of requests per second sustained) needed.
- Store end-user data (such as images, PII, etc) or other large-scale data processed by applications.
- The natural operations on the objects are not CRUD-y.
- The API is not easily modeled as objects.
- You chose to represent pending operations with an operation ID or operation object.
### Should I use a configMap or a custom resource?
@@ -102,7 +102,7 @@ Aggregated APIs are subordinate APIServers that sit behind the primary API serve
Custom Resource Definitions (CRDS) allow users to create new types of resources without adding another APIserver. You do not need to understand API Aggregation to use CRDs.
Regardless of whether they are installed via CRDs or AA, the new resources are called Custom Resources to distinguish them from built-in Kubernetes resources (like pods)
Regardless of whether they are installed via CRDs or AA, the new resources are called Custom Resources to distinguish them from built-in Kubernetes resources (like pods).
## CustomResourceDefinitions
@@ -215,9 +215,9 @@ Kubernetes [client libraries](/docs/reference/client-libraries/) can be used to
When you add a custom resource, you can access it using:
- kubectl
- the kubernetes dynamic client
- a REST client that you write
- a client generated using Kubernetes client generation tools (generating one is an advanced undertaking, but some projects may provide a client along with the CRD or AA).
- The kubernetes dynamic client.
- A REST client that you write.
- A client generated using Kubernetes client generation tools (generating one is an advanced undertaking, but some projects may provide a client along with the CRD or AA).
{% endcapture %}
docs/concepts/policy/psp.yaml → docs/concepts/policy/example-psp.yaml
+3 -6
View File
@@ -1,8 +1,10 @@
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: permissive
name: example
spec:
privileged: false # Don't allow privileged pods!
# The rest fills in some required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
@@ -11,10 +13,5 @@ spec:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'
allowedCapabilities:
- '*'
docs/concepts/policy/pod-security-policy.md
+487 -229
View File
@@ -1,70 +1,492 @@
---
approvers:
- pweil-
- tallclair
title: Pod Security Policies
---
Objects of type `PodSecurityPolicy` govern the ability
to make requests on a pod that affect the `SecurityContext` that will be
applied to a pod and container.
{% include feature-state-beta.md %}
See [PodSecurityPolicy proposal](https://git.k8s.io/community/contributors/design-proposals/auth/pod-security-policy.md) for more information.
Pod Security Policies enable fine-grained authorization of pod creation and
updates.
* TOC
{:toc}
## What is a Pod Security Policy?
A _Pod Security Policy_ is a cluster-level resource that controls the
actions that a pod can perform and what it has the ability to access. The
`PodSecurityPolicy` objects define a set of conditions that a pod must
run with in order to be accepted into the system. They allow an
A _Pod Security Policy_ is a cluster-level resource that controls security
sensitive aspects of the pod specification. The `PodSecurityPolicy` objects
define a set of conditions that a pod must run with in order to be accepted into
the system, as well as defaults for the related fields. They allow an
administrator to control the following:
| Control Aspect | Field Name |
| ---------------------------------------------------------------------- | ------------------------------------------- |
| Running of privileged containers | `privileged` |
| Default set of capabilities that will be added to a container | `defaultAddCapabilities` |
| Capabilities that will be dropped from a container | `requiredDropCapabilities` |
| Capabilities a container can request to be added | `allowedCapabilities` |
| Controlling the usage of volume types | [`volumes`](#controlling-volumes) |
| The use of host networking | [`hostNetwork`](#host-network) |
| The use of host ports | `hostPorts` |
| The use of host's PID namespace | `hostPID` |
| The use of host's IPC namespace | `hostIPC` |
| The SELinux context of the container | [`seLinux`](#selinux) |
| The user ID | [`runAsUser`](#runasuser) |
| Configuring allowable supplemental groups | [`supplementalGroups`](#supplementalgroups) |
| Allocating an FSGroup that owns the pod's volumes | [`fsGroup`](#fsgroup) |
| Requiring the use of a read only root file system | `readOnlyRootFilesystem` |
| Running of a container that allow privilege escalation from its parent | [`allowPrivilegeEscalation`](#allowprivilegeescalation) |
| Control whether a process can gain more privileges than its parent process | [`defaultAllowPrivilegeEscalation`](#defaultallowprivilegeescalation) |
| Whitelist of allowed host paths | [`allowedHostPaths`](#allowedhostpaths) |
_Pod Security Policies_ are comprised of settings and strategies that
control the security features a pod has access to. These settings fall
into three categories:
- *Controlled by a Boolean*: Fields of this type default to the most
restrictive value.
- *Controlled by an allowable set*: Fields of this type are checked
against the set to ensure their values are allowed.
- *Controlled by a strategy*: Items that have a strategy to provide
a mechanism to generate the value and a mechanism to ensure that a
specified value falls into the set of allowable values.
| Control Aspect | Field Names |
| ----------------------------------------------------| ------------------------------------------- |
| Running of privileged containers | `privileged` |
| Usage of the root namespaces | [`hostPID`, `hostIPC`](#host-namespaces) |
| Usage of host networking and ports | [`hostNetwork`, `hostPorts`](#host-namespaces) |
| Usage of volume types | [`volumes`](#volumes-and-file-systems) |
| Usage of the host filesystem | [`allowedHostPaths`](#volumes-and-file-systems) |
| Allocating an FSGroup that owns the pod's volumes | [`fsGroup`](#volumes-and-file-systems) |
| Requiring the use of a read only root file system | [`readOnlyRootFilesystem`](#volumes-and-file-systems) |
| The user and group IDs of the container | [`runAsUser`, `supplementalGroups`](#users-and-groups) |
| Restricting escalation to root privileges | [`allowPrivilegeEscalation`, `defaultAllowPrivilegeEscalation`](#privilege-escalation) |
| Linux capabilities | [`defaultAddCapabilities`, `requiredDropCapabilities`, `allowedCapabilities`](#capabilities) |
| The SELinux context of the container | [`seLinux`](#selinux) |
| The AppArmor profile used by containers | [annotations](#apparmor) |
| The seccomp profile used by containers | [annotations](#seccomp) |
## Strategies
## Enabling Pod Security Policies
### RunAsUser
Pod security policy control is implemented as an optional (but recommended)
[admission
controller](/docs/admin/admission-controllers/#podsecuritypolicy). PodSecurityPolicies
are enforced by [enabling the admission
controller](/docs/admin/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in),
but doing so without authorizing any policies **will prevent any pods from being
created** in the cluster.
- *MustRunAs* - Requires a `range` to be configured. Uses the first value
of the range as the default. Validates against the configured range.
Since the pod security policy API (`extensions/v1beta1/podsecuritypolicy`) is
enabled independently of the admission controller, for existing clusters it is
recommended that policies are added and authorized before enabling the admission
controller.
## Authorizing Policies
When a PodSecurityPolicy resource is created, it does nothing. In order to use
it, the requesting user or target pod's [service
account](/docs/tasks/configure-pod-container/configure-service-account/) must be
authorized to use the policy, by allowing the `use` verb on the policy.
Most Kubernetes pods are not created directly by users. Instead, they are
typically created indirectly as part of a
[Deployment](/docs/concepts/workloads/controllers/deployment/),
[ReplicaSet](/docs/concepts/workloads/controllers/replicaset/), or other
templated controller via the controller manager. Granting the controller access
to the policy would grant access for *all* pods created by that the controller,
so the preferred method for authorizing policies is to grant access to the
pod's service account (see [example](#run-another-pod)).
### Via RBAC
[RBAC](/docs/admin/authorization/rbac/) is a standard Kubernetes authorization
mode, and can easily be used to authorize use of policies.
First, a `Role` or `ClusterRole` needs to grant access to `use` the desired
policies. The rules to grant access look like this:
```yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: <role name>
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- <list of policies to authorize>
```
Then the `(Cluster)Role` is bound to the authorized user(s):
```yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: <binding name>
roleRef:
kind: ClusterRole
name: <role name>
apiGroup: rbac.authorization.k8s.io
subjects:
# Authorize specific service accounts:
- kind: ServiceAccount
name: <authorized service account name>
namespace: <authorized pod namespace>
# Authorize specific users (not recommended):
- kind: User
apiGroup: rbac.authorization.k8s.io
name: <authorized user name>
```
If a `RoleBinding` (not a `ClusterRoleBinding`) is used, it will only grant
usage for pods being run in the same namespace as the binding. This can be
paired with system groups to grant access to all pods run in the namespace:
```yaml
# Authorize all service accounts in a namespace:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccounts
# Or equivalently, all authenticated users in a namespace:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:authenticated
```
For more examples of RBAC bindings, see [Role Binding
Examples](docs/admin/authorization/rbac/#role-binding-examples). For a complete
example of authorizing a PodSecurityPolicy, see
[below](#example).
### Troubleshooting
- The [Controller Manager](/docs/admin/kube-controller-manager/) must be run
against [the secured API port](/docs/admin/accessing-the-api/), and must not
have superuser permissions. Otherwise requests would bypass authentication and
authorization modules, all PodSecurityPolicy objects would be allowed, and users
would be able to create privileged containers. For more details on configuring
Controller Manager authorization, see [Controller
Roles](docs/admin/authorization/rbac/#controller-roles).
## Policy Order
In addition to restricting pod creation and update, pod security policies can
also be used to provide default values for many of the fields that it
controls. When multiple policies are available, the pod security policy
controller selects policies in the following order:
1. If any policies successfully validate the pod without altering it, they are
used.
2. Otherwise, the first valid policy in alphabetical order is used.
## Example
_This example assumes you have a running cluster with the PodSecurityPolicy
admission controller enabled and you have cluster admin privileges._
### Set up
Set up a namespace and a service account to act as for this example. We'll use
this service account to mock a non-admin user.
```shell
$ kubectl create namespace psp-example
$ kubectl create serviceaccount -n psp-example fake-user
$ kubectl create rolebinding -n psp-example fake-editor --clusterrole=edit --serviceaccount=psp-example:fake-user
```
To make it clear which user we're acting as and save some typing, create 2
aliases:
```shell
$ alias kubectl-admin='kubectl -n psp-example'
$ alias kubectl-user='kubectl --as=system:serviceaccount:psp-example:fake-user -n psp-example'
```
### Create a policy and a pod
Define the example PodSecurityPolicy object in a file. This is a policy that
simply prevents the creation of privileged pods.
{% include code.html language="yaml" file="example-psp.yaml" ghlink="/docs/concepts/policy/example-psp.yaml" %}
And create it with kubectl:
```shell
$ kubectl-admin create -f example-psp.yaml
```
Now, as the unprivileged user, try to create a simple pod:
```shell
$ kubectl-user create -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: pause
spec:
containers:
- name: pause
image: gcr.io/google-containers/pause
EOF
Error from server (Forbidden): error when creating "STDIN": pods "pause" is forbidden: unable to validate against any pod security policy: []
```
**What happened?** Although the PodSecurityPolicy was created, neither the
pod's service account nor `fake-user` have permission to use the new policy:
```shell
$ kubectl-user auth can-i use podsecuritypolicy/example
no
```
Create the rolebinding to grant `fake-user` the `use` verb on the example
policy:
_Note: This is not the recommended way! See the [next section](#run-another-pod)
for the preferred approach._
```shell
$ kubectl-admin create role psp:unprivileged \
--verb=use \
--resource=podsecuritypolicy \
--resource-name=example
role "psp:unprivileged" created
$ kubectl-admin create rolebinding fake-user:psp:unprivileged \
--role=psp:unprivileged \
--serviceaccount=psp-example:fake-user
rolebinding "fake-user:psp:unprivileged" created
$ kubectl-user auth can-i use podsecuritypolicy/example
yes
```
Now retry creating the pod:
```shell
$ kubectl-user create -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: pause
spec:
containers:
- name: pause
image: gcr.io/google-containers/pause
EOF
pod "pause" created
```
It works as expected! But any attempts to create a privileged pod should still
be denied:
```shell
$ kubectl-user create -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: privileged
spec:
containers:
- name: pause
image: gcr.io/google-containers/pause
securityContext:
privileged: true
EOF
Error from server (Forbidden): error when creating "STDIN": pods "privileged" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
```
Delete the pod before moving on:
```shell
$ kubectl-user delete pause
```
### Run another pod
Let's try that again, slightly differently:
```shell
$ kubectl-user run pause --image=gcr.io/google-containers/pause
deployment "pause" created
$ kubectl-user get pods
No resources found.
$ kubectl-user get events | head -n 2
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
1m 2m 15 pause-7774d79b5 ReplicaSet Warning FailedCreate replicaset-controller Error creating: pods "pause-7774d79b5-" is forbidden: no providers available to validate pod request
```
**What happened?** We already bound the `psp:unprivileged` role for our `fake-user`,
why are we getting the error `Error creating: pods "pause-7774d79b5-" is
forbidden: no providers available to validate pod request`? The answer lies in
the source - `replicaset-controller`. Fake-user successfully created the
deployment (which successfully created a replicaset), but when the replicaset
went to create the pod it was not authorized to use the example
podsecuritypolicy.
In order to fix this, bind the `psp:unprivileged` role to the pod's service
account instead. In this case (since we didn't specify it) the service account
is `default`:
```shell
$ kubectl-admin create rolebinding default:psp:unprivileged \
--role=psp:unprivileged \
--serviceaccount=psp-example:default
rolebinding "default:psp:unprivileged" created
```
Now if you give it a minute to retry, the replicaset-controller should
eventually succeed in creating the pod:
```shell
$ kubectl-user get pods --watch
NAME READY STATUS RESTARTS AGE
pause-7774d79b5-qrgcb 0/1 Pending 0 1s
pause-7774d79b5-qrgcb 0/1 Pending 0 1s
pause-7774d79b5-qrgcb 0/1 ContainerCreating 0 1s
pause-7774d79b5-qrgcb 1/1 Running 0 2s
^C
```
### Clean up
Delete the namespace to clean up most of the example resources:
```shell
$ kubectl-admin delete ns psp-example
namespace "psp-example" deleted
```
Note that `PodSecurityPolicy` resources are not namespaced, and must be cleaned
up separately:
```shell
$ kubectl-admin delete psp example
podsecuritypolicy "example" deleted
```
### Example Policies
This is the least restricted policy you can create, equivalent to not using the
pod security policy admission controller:
{% include code.html language="yaml" file="privileged-psp.yaml" ghlink="/docs/concepts/policy/privileged-psp.yaml" %}
This is an example of a restrictive policy that requires users to run as an
unprivileged user, blocks possible escalations to root, and requires use of
several security mechanisms.
{% include code.html language="yaml" file="restricted-psp.yaml" ghlink="/docs/concepts/policy/restricted-psp.yaml" %}
## Policy Reference
### Host namespaces
**HostPID** - Controls whether the pod containers can share the host process ID
namespace. Note that when paired with ptrace this can be used to escalate
privileges outside of the container (ptrace is forbidden by default).
**HostIPC** - Controls whether the pod containers can share the host IPC
namespace.
**HostNetwork** - Controls whether the pod may use the node network
namespace. Doing so gives the pod access to the loopback device, services
listening on localhost, and could be used to snoop on network activity of other
pods on the same node.
**HostPorts** - Provides a whitelist of ranges of allowable ports in the host
network namespace. Defined as a list of `HostPortRange`, with `min`(inclusive)
and `max`(inclusive). Defaults to no allowed host ports.
**AllowedHostPaths** - See [Volumes and file systems](#volumes-and-file-systems).
### Volumes and file systems
**Volumes** - Provides a whitelist of allowed volume types. The allowable values
correspond to the volume sources that are defined when creating a volume. For
the complete list of volume types, see [Types of
Volumes](/docs/concepts/storage/volumes/#types-of-volumes). Additionally, `*`
may be used to allow all volume types.
The **recommended minimum set** of allowed volumes for new PSPs are:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- secret
- projected
**FSGroup** - Controls the supplemental group applied to some volumes.
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
minimum value of the first range as the default. Validates against all ranges.
- *RunAsAny* - No default provided. Allows any `fsGroup` ID to be specified.
**AllowedHostPaths** - This specifies a whitelist of host paths that are allowed
to be used by hostPath volumes. An empty list means there is no restriction on
host paths used. This is defined as a list of objects with a single `pathPrefix`
field, which allows hostPath volumes to mount a path that begins with an
allowed prefix. For example:
```yaml
allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
```
_Note: There are many ways a container with unrestricted access to the host
filesystem can escalate privileges, including reading data from other
containers, and abusing the credentials of system services, such as Kubelet._
**ReadOnlyRootFilesystem** - Requires that containers must run with a read-only
root filesystem (i.e. no writeable layer).
### Users and groups
**RunAsUser** - Controls the what user ID containers run as.
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
minimum value of the first range as the default. Validates against all ranges.
- *MustRunAsNonRoot* - Requires that the pod be submitted with a non-zero
`runAsUser` or have the `USER` directive defined in the image. No default
provided.
`runAsUser` or have the `USER` directive defined (using a numeric UID) in the
image. No default provided. Setting `allowPrivilegeEscalation=false` is strongly
recommended with this strategy.
- *RunAsAny* - No default provided. Allows any `runAsUser` to be specified.
**SupplementalGroups** - Controls which group IDs containers add.
- *MustRunAs* - Requires at least one `range` to be specified. Uses the
minimum value of the first range as the default. Validates against all ranges.
- *RunAsAny* - No default provided. Allows any `supplementalGroups` to be
specified.
### Privilege Escalation
These options control the `allowPrivilegeEscalation` container option. This bool
directly controls whether the
[`no_new_privs`](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt)
flag gets set on the container process. This flag will prevent `setuid` binaries
from changing the effective user ID, and prevent files from enabling extra
capabilities (e.g. it will prevent the use of the `ping` tool). This behavior is
required to effectively enforce `MustRunAsNonRoot`.
It defaults to `nil`. The default behavior of `nil` allows privilege escalation
so as to not break setuid binaries. Setting it to `false` ensures that no child
process of a container can gain more privileges than its parent.
**AllowPrivilegeEscalation** - Gates whether or not a user is allowed to set the
security context of a container to `allowPrivilegeEscalation=true`. This
defaults to allowed. When set to false, the container's
`allowPrivilegeEscalation` is defaulted to false.
**DefaultAllowPrivilegeEscalation** - Sets the default for the
`allowPrivilegeEscalation` option. The default behavior without this is to allow
privilege escalation so as to not break setuid binaries. If that behavior is not
desired, this field can be used to default to disallow, while still permitting
pods to request `allowPrivilegeEscalation` explicitly.
### Capabilities
Linux capabilities provide a finer grained breakdown of the privileges
traditionally associated with the superuser. Some of these capabilities can be
used to escalate privileges or for container breakout, and may be restricted by
the PodSecurityPolicy. For more details on Linux capabilities, see
[capabilities(7)](http://man7.org/linux/man-pages/man7/capabilities.7.html).
The following fields take a list of capabilities, specified as the capability
name in ALL_CAPS without the `CAP_` prefix.
**AllowedCapabilities** - Provides a whitelist of capabilities that may be added
to a container. The default set of capabilities are implicitly allowed. The
empty set means that no additional capabilities may be added beyond the default
set. `*` can be used to allow all capabilities.
**RequiredDropCapabilities** - The capabilities which must be dropped from
containers. These capabilities are removed from the default set, and must not be
added. Capabilities listed in `RequiredDropCapabilities` must not be included in
`AllowedCapabilities` or `DefaultAddCapabilities`.
**DefaultAddCapabilities** - The capabilities which are added to containers by
default, in addition to the runtime defaults. See the [Docker
documentation](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)
for the default list of capabilities when using the Docker runtime.
### SELinux
- *MustRunAs* - Requires `seLinuxOptions` to be configured if not using
@@ -73,193 +495,29 @@ pre-allocated values. Uses `seLinuxOptions` as the default. Validates against
- *RunAsAny* - No default provided. Allows any `seLinuxOptions` to be
specified.
### SupplementalGroups
### AppArmor
- *MustRunAs* - Requires at least one range to be specified. Uses the
minimum value of the first range as the default. Validates against all ranges.
- *RunAsAny* - No default provided. Allows any `supplementalGroups` to be
specified.
Controlled via annotations on the PodSecurityPolicy. Refer to the [AppArmor
documentation](/docs/tutorials/clusters/apparmor/#podsecuritypolicy-annotations).
### FSGroup
### Seccomp
- *MustRunAs* - Requires at least one range to be specified. Uses the
minimum value of the first range as the default. Validates against the
first ID in the first range.
- *RunAsAny* - No default provided. Allows any `fsGroup` ID to be specified.
The use of seccomp profiles in pods can be controlled via annotations on the
PodSecurityPolicy. Seccomp is an alpha feature in Kubernetes.
### Controlling Volumes
**seccomp.security.alpha.kubernetes.io/defaultProfileName** - Annotation that
specifies the default seccomp profile to apply to containers. Possible values
are:
The usage of specific volume types can be controlled by setting the
volumes field of the PSP. The allowable values of this field correspond
to the volume sources that are defined when creating a volume:
- `unconfined` - Seccomp is not applied to the container processes (this is the
default in Kubernetes), if no alternative is provided.
- `docker/default` - The Docker default seccomp profile is used.
- `localhost/<path>` - Specify a profile as a file on the node located at
`<seccomp_root>/<path>`, where `<seccomp_root>` is defined via the
`--seccomp-profile-root` flag on the Kubelet.
1. azureFile
1. azureDisk
1. flocker
1. flexVolume
1. hostPath
1. emptyDir
1. gcePersistentDisk
1. awsElasticBlockStore
1. gitRepo
1. secret
1. nfs
1. iscsi
1. glusterfs
1. persistentVolumeClaim
1. rbd
1. cinder
1. cephFS
1. downwardAPI
1. fc
1. configMap
1. vsphereVolume
1. quobyte
1. photonPersistentDisk
1. projected
1. portworxVolume
1. scaleIO
1. storageos
1. \* (allow all volumes)
The recommended minimum set of allowed volumes for new PSPs are
configMap, downwardAPI, emptyDir, persistentVolumeClaim, secret, and projected.
### Host Network
- *HostPorts*, default `empty`. List of `HostPortRange`, defined by `min`(inclusive) and `max`(inclusive), which define the allowed host ports.
### AllowPrivilegeEscalation
Gates whether or not a user is allowed to set the security context of a container
to `allowPrivilegeEscalation=true`. This field defaults to `false`.
### DefaultAllowPrivilegeEscalation
Sets the default for the security context `AllowPrivilegeEscalation` of a container.
This bool directly controls whether the `no_new_privs` flag gets set on the
container process. It defaults to `nil`. The default behavior of `nil`
allows privilege escalation so as to not break setuid binaries. Setting it to `false`
ensures that no child process of a container can gain more privileges than
its parent.
### AllowedHostPaths
This specifies a whitelist of host paths that are allowed to be used by Pods.
An empty list means there is no restriction on host paths used.
Each item in the list must specify a string value named `pathPrefix` that
defines a host path to match. The value cannot be "`*`" though.
An example is shown below:
```yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: custom-paths
spec:
allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
- pathPrefix: "/foo"
```
## Admission
[_Admission control_ with `PodSecurityPolicy`](/docs/admin/admission-controllers/#podsecuritypolicy)
allows for control over the creation and modification of resources based on the
capabilities allowed in the cluster.
Admission uses the following approach to create the final security context for
the pod:
1. Retrieve all PSPs available for use.
1. Generate field values for security context settings that were not specified
on the request.
1. Validate the final settings against the available policies.
If a matching policy is found, then the pod is accepted. If the
request cannot be matched to a PSP, the pod is rejected.
A pod must validate every field against the PSP.
## Creating a Pod Security Policy
Here is an example Pod Security Policy. It has permissive settings for
all fields
{% include code.html language="yaml" file="psp.yaml" ghlink="/docs/concepts/policy/psp.yaml" %}
Create the policy by downloading the example file and then running this command:
```shell
$ kubectl create -f ./psp.yaml
podsecuritypolicy "permissive" created
```
## Getting a list of Pod Security Policies
To get a list of existing policies, use `kubectl get`:
```shell
$ kubectl get psp
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
permissive false [] RunAsAny RunAsAny RunAsAny RunAsAny false [*]
privileged true [] RunAsAny RunAsAny RunAsAny RunAsAny false [*]
restricted false [] RunAsAny MustRunAsNonRoot RunAsAny RunAsAny false [emptyDir secret downwardAPI configMap persistentVolumeClaim projected]
```
## Editing a Pod Security Policy
To modify policy interactively, use `kubectl edit`:
```shell
$ kubectl edit psp permissive
```
This command will open a default text editor where you will be able to modify policy.
## Deleting a Pod Security Policy
Once you don't need a policy anymore, simply delete it with `kubectl`:
```shell
$ kubectl delete psp permissive
podsecuritypolicy "permissive" deleted
```
## Enabling Pod Security Policies
In order to use Pod Security Policies in your cluster you must ensure the
following
1. You have enabled the API type `extensions/v1beta1/podsecuritypolicy` (only for versions prior 1.6)
1. [You have enabled the admission control plug-in `PodSecurityPolicy`](/docs/admin/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in)
1. You have defined your policies
## Working With RBAC
In Kubernetes 1.5 and newer, you can use PodSecurityPolicy to control access to
privileged containers based on user role and groups. Access to different
PodSecurityPolicy objects can be controlled via authorization.
Note that [Controller Manager](/docs/admin/kube-controller-manager/) must be run
against [the secured API port](/docs/admin/accessing-the-api/), and must not
have superuser permissions. Otherwise requests would bypass authentication and
authorization modules, all PodSecurityPolicy objects would be allowed,
and user will be able to create privileged containers.
PodSecurityPolicy authorization uses the union of all policies available to the
user creating the pod and
[the service account specified on the pod](/docs/tasks/configure-pod-container/configure-service-account/).
Access to given PSP policies for a user will be effective only when creating
Pods directly.
For pods created on behalf of a user, in most cases by Controller Manager,
access should be given to the service account specified on the pod spec
template. Examples of resources that create pods on behalf of a user are
Deployments, ReplicaSets, etc.
For more details, see the
[PodSecurityPolicy RBAC example](https://git.k8s.io/examples/staging/podsecuritypolicy/rbac/README.md)
of applying PodSecurityPolicy to control access to privileged containers based
on role and groups when deploying Pods directly.
**seccomp.security.alpha.kubernetes.io/allowedProfileNames** - Annotation that
specifies which values are allowed for the pod seccomp annotations. Specified as
a comma-delimited list of allowed values. Possible values are those listed
above, plus `*` to allow all profiles. Absence of this annotation means that the
default cannot be changed.
docs/concepts/policy/privileged-psp.yaml
+27
View File
@@ -0,0 +1,27 @@
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
volumes:
- '*'
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
docs/concepts/policy/restricted-psp.yaml
+48
View File
@@ -0,0 +1,48 @@
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
# This is redundant with non-root + disallow privilege escalation,
# but we can provide it for defense in depth.
requiredDropCapabilities:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that persistentVolumes set up by the cluster admin are safe to use.
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
docs/getting-started-guides/scratch.md
+1 -1
View File
@@ -705,7 +705,7 @@ Complete this template for the scheduler pod:
"containers": [
{
"name": "kube-scheduler",
"image": "$HYBERKUBE_IMAGE",
"image": "$HYPERKUBE_IMAGE",
"command": [
"/hyperkube",
"scheduler",
docs/tasks/access-application-cluster/access-cluster.md
+2 -1
View File
@@ -55,7 +55,8 @@ $ kubectl proxy --port=8080 &
See [kubectl proxy](/docs/user-guide/kubectl/{{page.version}}/#proxy) for more details.
Then you can explore the API with curl, wget, or a browser, like so:
Then you can explore the API with curl, wget, or a browser, replacing localhost
with [::1] for IPv6, like so:
```shell
$ curl http://localhost:8080/api/
docs/tasks/access-application-cluster/frontend.yaml
+1 -1
View File
@@ -1,5 +1,5 @@
kind: Service
apiVersion: v1
kind: Service
metadata:
name: frontend
spec:
docs/tasks/administer-cluster/out-of-resource.md
+116 -125
View File
@@ -9,27 +9,25 @@ title: Configure Out Of Resource Handling
* TOC
{:toc}
This page explains how to configure out of resource handling with `kubelet`.
The `kubelet` needs to preserve node stability when available compute resources
are low.
This is especially important when dealing with incompressible resources such as
memory or disk.
If either resource is exhausted, the node would become unstable.
are low. This is especially important when dealing with incompressible
compute resources, such as memory or disk space. If such resources are exhausted,
nodes become unstable.
## Eviction Policy
The `kubelet` can pro-actively monitor for and prevent against total starvation
of a compute resource. In those cases, the `kubelet` can pro-actively fail one
or more pods in order to reclaim the starved resource. When the `kubelet` fails
a pod, it terminates all containers in the pod, and the `PodPhase` is
transitioned to `Failed`.
The `kubelet` can proactively monitor for and prevent total starvation of a
compute resource. In those cases, the `kubelet` can reclaim the starved
resource by proactively failing one or more Pods. When the `kubelet` fails
a Pod, it terminates all of its containers and transitions its `PodPhase` to `Failed`.
### Eviction Signals
The `kubelet` can support the ability to trigger eviction decisions on the
signals described in the table below. The value of each signal is described in
the description column based on the `kubelet` summary API.
The `kubelet` supports eviction decisions based on the signals described in the following
table. The value of each signal is described in the Description column, which is based on
the `kubelet` summary API.
| Eviction Signal | Description |
|----------------------------|-----------------------------------------------------------------------|
@@ -47,7 +45,7 @@ The value for `memory.available` is derived from the cgroupfs instead of tools
like `free -m`. This is important because `free -m` does not work in a
container, and if users use the [node
allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) feature, out of resource decisions
are made local to the end user pod part of the cgroup hierarchy as well as the
are made local to the end user Pod part of the cgroup hierarchy as well as the
root node. This
[script](/docs/tasks/administer-cluster/out-of-resource/memory-available.sh)
reproduces the same set of steps that the `kubelet` performs to calculate
@@ -67,64 +65,64 @@ of configurations are not currently supported by the kubelet. For example, it is
*not OK* to store volumes and logs in a dedicated `filesystem`.
In future releases, the `kubelet` will deprecate the existing [garbage
collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/) support in favor of eviction in
response to disk pressure.
collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/)
support in favor of eviction in response to disk pressure.
### Eviction Thresholds
The `kubelet` supports the ability to specify eviction thresholds that trigger the `kubelet` to reclaim resources.
Each threshold is of the following form:
Each threshold has the following form:
`<eviction-signal><operator><quantity>`
`[eviction-signal][operator][quantity]`
* valid `eviction-signal` tokens as defined above.
* valid `operator` tokens are `<`
* valid `quantity` tokens must match the quantity representation used by Kubernetes
* an eviction threshold can be expressed as a percentage if ends with `%` token.
where:
For example, if a node has `10Gi` of memory, and the desire is to induce eviction
if available memory falls below `1Gi`, an eviction threshold can be specified as either
of the following (but not both).
* `eviction-signal` is a eviction signal token as defined in the previous table.
* `operator` is the desired relational operator, such as `<` (less than).
* `quantity` is the eviction threshhold quantity, such as `1Gi`. These tokens must
match the quantity representation used by Kubernetes. An eviction threshold can also
be expressed as a percentage using the `%` token.
* `memory.available<10%`
* `memory.available<1Gi`
For example, if a node has `10Gi` of total memory and you want trigger eviction if
the available memory falls below `1Gi`, you can define the eviction threshold as
either `memory.available<10%` or `memory.available<1Gi`. You cannot use both.
#### Soft Eviction Thresholds
A soft eviction threshold pairs an eviction threshold with a required
administrator specified grace period. No action is taken by the `kubelet`
administrator-specified grace period. No action is taken by the `kubelet`
to reclaim resources associated with the eviction signal until that grace
period has been exceeded. If no grace period is provided, the `kubelet` will
error on startup.
period has been exceeded. If no grace period is provided, the `kubelet`
returns an error on startup.
In addition, if a soft eviction threshold has been met, an operator can
specify a maximum allowed pod termination grace period to use when evicting
pods from the node. If specified, the `kubelet` will use the lesser value among
specify a maximum allowed Pod termination grace period to use when evicting
pods from the node. If specified, the `kubelet` uses the lesser value among
the `pod.Spec.TerminationGracePeriodSeconds` and the max allowed grace period.
If not specified, the `kubelet` will kill pods immediately with no graceful
If not specified, the `kubelet` kills Pods immediately with no graceful
termination.
To configure soft eviction thresholds, the following flags are supported:
* `eviction-soft` describes a set of eviction thresholds (e.g. `memory.available<1.5Gi`) that if met over a
corresponding grace period would trigger a pod eviction.
corresponding grace period would trigger a Pod eviction.
* `eviction-soft-grace-period` describes a set of eviction grace periods (e.g. `memory.available=1m30s`) that
correspond to how long a soft eviction threshold must hold before triggering a pod eviction.
correspond to how long a soft eviction threshold must hold before triggering a Pod eviction.
* `eviction-max-pod-grace-period` describes the maximum allowed grace period (in seconds) to use when terminating
pods in response to a soft eviction threshold being met.
#### Hard Eviction Thresholds
A hard eviction threshold has no grace period, and if observed, the `kubelet`
will take immediate action to reclaim the associated starved resource. If a
hard eviction threshold is met, the `kubelet` will kill the pod immediately
will take immediate action to reclaim the associated starved resource. If a
hard eviction threshold is met, the `kubelet` kills the Pod immediately
with no graceful termination.
To configure hard eviction thresholds, the following flag is supported:
* `eviction-hard` describes a set of eviction thresholds (e.g. `memory.available<1Gi`) that if met
would trigger a pod eviction.
would trigger a Pod eviction.
The `kubelet` has the following default hard eviction threshold:
@@ -138,10 +136,10 @@ The `kubelet` evaluates eviction thresholds per its configured housekeeping inte
### Node Conditions
The `kubelet` will map one or more eviction signals to a corresponding node condition.
The `kubelet` maps one or more eviction signals to a corresponding node condition.
If a hard eviction threshold has been met, or a soft eviction threshold has been met
independent of its associated grace period, the `kubelet` will report a condition that
independent of its associated grace period, the `kubelet` reports a condition that
reflects the node is under pressure.
The following node conditions are defined that correspond to the specified eviction signal.
@@ -151,7 +149,7 @@ The following node conditions are defined that correspond to the specified evict
| `MemoryPressure` | `memory.available` | Available memory on the node has satisfied an eviction threshold |
| `DiskPressure` | `nodefs.available`, `nodefs.inodesFree`, `imagefs.available`, or `imagefs.inodesFree` | Available disk space and inodes on either the node's root filesytem or image filesystem has satisfied an eviction threshold |
The `kubelet` will continue to report node status updates at the frequency specified by
The `kubelet` continues to report node status updates at the frequency specified by
`--node-status-update-frequency` which defaults to `10s`.
### Oscillation of node conditions
@@ -174,85 +172,76 @@ condition back to `false`.
### Reclaiming node level resources
If an eviction threshold has been met and the grace period has passed,
the `kubelet` will initiate the process of reclaiming the pressured resource
the `kubelet` initiates the process of reclaiming the pressured resource
until it has observed the signal has gone below its defined threshold.
The `kubelet` attempts to reclaim node level resources prior to evicting end-user pods. If
The `kubelet` attempts to reclaim node level resources prior to evicting end-user Pods. If
disk pressure is observed, the `kubelet` reclaims node level resources differently if the
machine has a dedicated `imagefs` configured for the container runtime.
#### With Imagefs
#### With `imagefs`
If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order:
If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting the dead Pods and their containers.
1. Delete dead pods/containers
If `imagefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting all unused images.
If `imagefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order:
#### Without `imagefs`
If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space in the following order:
1. Delete dead Pods and their containers
1. Delete all unused images
#### Without Imagefs
### Evicting end-user Pods
If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order:
If the `kubelet` is unable to reclaim sufficient resource on the node, `kubelet` begins evicting Pods.
1. Delete dead pods/containers
1. Delete all unused images
The `kubelet` ranks Pods for eviction first by their quality of service, and then by the consumption
of the starved compute resource relative to the Pods' scheduling requests.
### Evicting end-user pods
As a result, `kubectl` ranks and evicts Pods in the following order:
If the `kubelet` is unable to reclaim sufficient resource on the node,
it will begin evicting pods.
The `kubelet` ranks pods for eviction as follows:
* by their quality of service.
* by the consumption of the starved compute resource relative to the pods scheduling request.
As a result, pod eviction occurs in the following order:
* `BestEffort` pods that consume the most of the starved resource are failed
first.
* `Burstable` pods that consume the greatest amount of the starved resource
relative to their request for that resource are killed first. If no pod
* `BestEffort` Pods consume the most of the starved resource are failed first.
Local disk is a `BestEffort` resource.
* `Burstable` Pods consume the greatest amount of the starved resource
relative to their request for that resource are killed first. If no Pod
has exceeded its request, the strategy targets the largest consumer of the
starved resource.
* `Guaranteed` pods that consume the greatest amount of the starved resource
relative to their request are killed first. If no pod has exceeded its request,
the strategy targets the largest consumer of the starved resource.
A `Guaranteed` pod is guaranteed to never be evicted because of another pod's
resource consumption. If a system daemon (i.e. `kubelet`, `docker`, `journald`, etc.)
is consuming more resources than were reserved via `system-reserved` or `kube-reserved` allocations,
and the node only has `Guaranteed` pod(s) remaining, then the node must choose to evict a
`Guaranteed` pod in order to preserve node stability, and to limit the impact
of the unexpected consumption to other `Guaranteed` pod(s).
Local disk is a `BestEffort` resource. If necessary, `kubelet` will evict pods one at a time to reclaim
disk when `DiskPressure` is encountered. The `kubelet` will rank pods by quality of service. If the `kubelet`
is responding to `inode` starvation, it will reclaim `inodes` by evicting pods with the lowest quality of service
first. If the `kubelet` is responding to lack of available disk, it will rank pods within a quality of service
* `Guaranteed` Pods are guaranteed only when requests and limits are specified
for all the containers and they are equal A `Guaranteed` Pod is guaranteed to
never be evicted because of another Pod's resource consumption. If a system
daemon (such as `kubelet`, `docker`, and `journald`) is consuming more resources
than were reserved via `system-reserved` or `kube-reserved` allocations, and the
node only has `Guaranteed` Pods remaining, then the node must choose to evict a
`Guaranteed` Pod in order to preserve node stability and to limit the impact
of the unexpected consumption to other `Guaranteed` Pods.
If necessary, `kubelet` evicts Pods one at a time to reclaim disk when `DiskPressure`
is encountered. If the `kubelet` is responding to `inode` starvation, it reclaims
`inodes` by evicting Pods with the lowest quality of service first. If the `kubelet`
is responding to lack of available disk, it ranks Pods within a quality of service
that consumes the largest amount of disk and kill those first.
#### With Imagefs
#### With `imagefs`
If `nodefs` is triggering evictions, `kubelet` will sort pods based on the usage on `nodefs`
If `nodefs` is triggering evictions, `kubelet` sorts Pods based on the usage on `nodefs`
- local volumes + logs of all its containers.
If `imagefs` is triggering evictions, `kubelet` will sort pods based on the writable layer usage of all its containers.
If `imagefs` is triggering evictions, `kubelet` sorts Pods based on the writable layer usage of all its containers.
#### Without Imagefs
#### Without `imagefs`
If `nodefs` is triggering evictions, `kubelet` will sort pods based on their total disk usage
If `nodefs` is triggering evictions, `kubelet` sorts Pods based on their total disk usage
- local volumes + logs & writable layer of all its containers.
### Minimum eviction reclaim
In certain scenarios, eviction of pods could result in reclamation of small amount of resources. This can result in
In certain scenarios, eviction of Pods could result in reclamation of small amount of resources. This can result in
`kubelet` hitting eviction thresholds in repeated successions. In addition to that, eviction of resources like `disk`,
is time consuming.
To mitigate these issues, `kubelet` can have a per-resource `minimum-reclaim`. Whenever `kubelet` observes
resource pressure, `kubelet` will attempt to reclaim at least `minimum-reclaim` amount of resource below
resource pressure, `kubelet` attempts to reclaim at least `minimum-reclaim` amount of resource below
the configured eviction threshold.
For example, with the following configuration:
@@ -262,31 +251,31 @@ For example, with the following configuration:
--eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"`
```
If an eviction threshold is triggered for `memory.available`, the `kubelet` will work to ensure
that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` will work
to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it will
work to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure
If an eviction threshold is triggered for `memory.available`, the `kubelet` works to ensure
that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` works
to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it
works to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure
on their associated resources.
The default `eviction-minimum-reclaim` is `0` for all resources.
### Scheduler
The node will report a condition when a compute resource is under pressure. The
The node reports a condition when a compute resource is under pressure. The
scheduler views that condition as a signal to dissuade placing additional
pods on the node.
| Node Condition | Scheduler Behavior |
| ---------------- | ------------------------------------------------ |
| `MemoryPressure` | No new `BestEffort` pods are scheduled to the node. |
| `DiskPressure` | No new pods are scheduled to the node. |
| `MemoryPressure` | No new `BestEffort` Pods are scheduled to the node. |
| `DiskPressure` | No new Pods are scheduled to the node. |
## Node OOM Behavior
If the node experiences a system OOM (out of memory) event prior to the `kubelet` is able to reclaim memory,
the node depends on the [oom_killer](https://lwn.net/Articles/391222/) to respond.
The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the pod.
The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the Pod.
| Quality of Service | oom_score_adj |
|----------------------------|-----------------------------------------------------------------------|
@@ -294,7 +283,7 @@ The `kubelet` sets a `oom_score_adj` value for each container based on the quali
| `BestEffort` | 1000 |
| `Burstable` | min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999) |
If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` will calculate
If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` calculates
an `oom_score` based on the percentage of memory it's using on the node, and then add the `oom_score_adj` to get an
effective `oom_score` for the container, and then kills the container with the highest score.
@@ -302,17 +291,19 @@ The intended behavior should be that containers with the lowest quality of servi
are consuming the largest amount of memory relative to the scheduling request should be killed first in order
to reclaim memory.
Unlike pod eviction, if a pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`.
Unlike Pod eviction, if a Pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`.
## Best Practices
The following sections describe best practices for out of resource handling.
### Schedulable resources and eviction policies
Let's imagine the following scenario:
Consider the following scenario:
* Node memory capacity: `10Gi`
* Operator wants to reserve 10% of memory capacity for system daemons (kernel, `kubelet`, etc.)
* Operator wants to evict pods at 95% memory utilization to reduce thrashing and incidence of system OOM.
* Operator wants to evict Pods at 95% memory utilization to reduce thrashing and incidence of system OOM.
To facilitate this scenario, the `kubelet` would be launched as follows:
@@ -324,31 +315,30 @@ To facilitate this scenario, the `kubelet` would be launched as follows:
Implicit in this configuration is the understanding that "System reserved" should include the amount of memory
covered by the eviction threshold.
To reach that capacity, either some pod is using more than its request, or the system is using more than `500Mi`.
To reach that capacity, either some Pod is using more than its request, or the system is using more than `500Mi`.
This configuration will ensure that the scheduler does not place pods on a node that immediately induce memory pressure
and trigger eviction assuming those pods use less than their configured request.
This configuration ensures that the scheduler does not place Pods on a node that immediately induce memory pressure
and trigger eviction assuming those Pods use less than their configured request.
### DaemonSet
It is never desired for a `kubelet` to evict a pod that was derived from
a `DaemonSet` since the pod will immediately be recreated and rescheduled
back to the same node.
It is never desired for a `kubelet` to evict a `DaemonSet` Pod, since the Pod is
immediately recreated and rescheduled back to the same node.
At the moment, the `kubelet` has no ability to distinguish a pod created
from `DaemonSet` versus any other object. If/when that information is
available, the `kubelet` could pro-actively filter those pods from the
candidate set of pods provided to the eviction strategy.
At the moment, the `kubelet` has no ability to distinguish a Pod created
from `DaemonSet` versus any other object. If/when that information is
available, the `kubelet` could pro-actively filter those Pods from the
candidate set of Pods provided to the eviction strategy.
In general, it is strongly recommended that `DaemonSet` not
create `BestEffort` pods to avoid being identified as a candidate pod
for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` pods.
create `BestEffort` Pods to avoid being identified as a candidate Pod
for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` Pods.
## Deprecation of existing feature flags to reclaim disk
`kubelet` has been freeing up disk space on demand to keep the node stable.
As disk based eviction matures, the following `kubelet` flags will be marked for deprecation
As disk based eviction matures, the following `kubelet` flags are marked for deprecation
in favor of the simpler configuration supported around eviction.
| Existing Flag | New Flag |
@@ -363,26 +353,27 @@ in favor of the simpler configuration supported around eviction.
## Known issues
The following sections describe known issues related to out of resource handling.
### kubelet may not observe memory pressure right away
The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage
The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage
increases within that window rapidly, the `kubelet` may not observe `MemoryPressure` fast enough, and the `OOMKiller`
will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this
will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this
latency, and instead have the kernel tell us when a threshold has been crossed immediately.
If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for
this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature
this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature
to prevent system OOMs, and promote eviction of workloads so cluster state can rebalance.
### kubelet may evict more pods than needed
### kubelet may evict more Pods than needed
The pod eviction may evict more pods than needed due to stats collection timing gap. This can be mitigated by adding
The Pod eviction may evict more Pods than needed due to stats collection timing gap. This can be mitigated by adding
the ability to get root container stats on an on-demand basis [(https://github.com/google/cadvisor/issues/1247)](https://github.com/google/cadvisor/issues/1247) in the future.
### How kubelet ranks pods for eviction in response to inode exhaustion
### How kubelet ranks Pods for eviction in response to inode exhaustion
At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes
inode exhaustion, it will evict pods by ranking them by quality of service. The following issue has been opened in cadvisor
to track per container inode consumption [(https://github.com/google/cadvisor/issues/1422)](https://github.com/google/cadvisor/issues/1422) which would allow us to rank pods
by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict
that pod over others.
At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes
inode exhaustion, it evicts Pods by ranking them by quality of service. The following issue has been opened in cadvisor
to track per container inode consumption [(https://github.com/google/cadvisor/issues/1422)](https://github.com/google/cadvisor/issues/1422) which would allow us to rank Pods
by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict that Pod over others.
docs/tasks/job/parallel-processing-expansion.md
+5 -5
View File
@@ -42,7 +42,7 @@ Next, expand the template into multiple files, one for each item to be processed
$ mkdir ./jobs
$ for i in apple banana cherry
do
cat job.yaml.txt | sed "s/\$ITEM/$i/" > ./jobs/job-$i.yaml
cat job.yaml | sed "s/\$ITEM/$i/" > ./jobs/job-$i.yaml
done
```
@@ -72,10 +72,10 @@ Now, check on the jobs:
```shell
$ kubectl get jobs -l jobgroup=jobexample
JOB CONTAINER(S) IMAGE(S) SELECTOR SUCCESSFUL
process-item-apple c busybox app in (jobexample),item in (apple) 1
process-item-banana c busybox app in (jobexample),item in (banana) 1
process-item-cherry c busybox app in (jobexample),item in (cherry) 1
NAME DESIRED SUCCESSFUL AGE
process-item-apple 1 1 31s
process-item-banana 1 1 31s
process-item-cherry 1 1 31s
```
Here we use the `-l` option to select all jobs that are part of this
docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md
+5 -18
View File
@@ -311,29 +311,16 @@ HorizontalPodAutoscaler.
## Appendix: Other possible scenarios
### Creating the autoscaler from a .yaml file
### Creating the autoscaler declaratively
Instead of using `kubectl autoscale` command we can use the [hpa-php-apache.yaml](/docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml) file, which looks like this:
Instead of using `kubectl autoscale` command to create a HorizontalPodAutoscaler imperatively we
can use the following file to create it declaratively:
```yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
```
{% include code.html language="yaml" file="hpa-php-apache.yaml" ghlink="/docs/tasks/run-application/hpa-php-apache.yaml" %}
We will create the autoscaler by executing the following command:
```shell
$ kubectl create -f docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml
$ kubectl create -f https://k8s.io/docs/tasks/run-application/hpa-php-apache.yaml
horizontalpodautoscaler "php-apache" created
```
docs/tasks/run-application/hpa-php-apache.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
docs/tutorials/stateful-application/web.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: Service
metadata:
docs/tutorials/stateful-application/webp.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: Service
metadata:
docs/tutorials/stateful-application/zookeeper.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: Service
metadata:
docs/user-guide/environment-guide/backend-rc.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: ReplicationController
metadata:
docs/user-guide/environment-guide/backend-srv.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: Service
metadata:
docs/user-guide/environment-guide/show-rc.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: ReplicationController
metadata:
docs/user-guide/environment-guide/show-srv.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: Service
metadata:
docs/user-guide/multi-pod.yaml
-1
View File
@@ -1,4 +1,3 @@
---
apiVersion: v1
kind: Pod
metadata: