whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into spell_fixes

Browse Source
This commit is contained in:
Madhuri Kumari
2017-12-27 09:49:06 +05:30
committed by GitHub
parent bc1accdc25 d946e32cb7
commit bf283d4d41
4083 changed files with 128909 additions and 543989 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tasks/access-application-cluster/access-cluster.md
+6 -7
View File
@@ -55,7 +55,8 @@ $ kubectl proxy --port=8080 &
See [kubectl proxy](/docs/user-guide/kubectl/{{page.version}}/#proxy) for more details.
Then you can explore the API with curl, wget, or a browser, like so:
Then you can explore the API with curl, wget, or a browser, replacing localhost
with [::1] for IPv6, like so:
```shell
$ curl http://localhost:8080/api/
@@ -154,7 +155,7 @@ the `kubernetes` DNS name, which resolves to a Service IP which in turn
will be routed to an apiserver.
The recommended way to authenticate to the apiserver is with a
[service account](/docs/tasks/configure-pod-container/configure-service-account/) credential. By kube-system, a pod
[service account](/docs/tasks/configure-pod-container/configure-service-account/) credential. By kube-system, a pod
is associated with a service account, and a credential (token) for that
service account is placed into the filesystem tree of each container in that pod,
at `/var/run/secrets/kubernetes.io/serviceaccount/token`.
@@ -168,17 +169,15 @@ at `/var/run/secrets/kubernetes.io/serviceaccount/namespace` in each container.
From within a pod the recommended ways to connect to API are:
- run a kubectl proxy as one of the containers in the pod, or as a background
process within a container. This proxies the
- run `kubectl proxy` in a sidecar container in the pod, or as a background
process within the container. This proxies the
Kubernetes API to the localhost interface of the pod, so that other processes
in any container of the pod can access it. See this [example of using kubectl proxy
in a pod](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/kubectl-container/).
in any container of the pod can access it.
- use the Go client library, and create a client using the `rest.InClusterConfig()` and `kubernetes.NewForConfig()` functions.
They handle locating and authenticating to the apiserver. [example](https://git.k8s.io/client-go/examples/in-cluster-client-configuration/main.go)
In each case, the credentials of the pod are used to communicate securely with the apiserver.
## Accessing services running on the cluster
The previous section was about connecting the Kubernetes API server. This section is about
docs/tasks/access-application-cluster/frontend.yaml
+1 -1
View File
@@ -1,5 +1,5 @@
kind: Service
apiVersion: v1
kind: Service
metadata:
name: frontend
spec:
docs/tasks/access-application-cluster/redis-master.yaml
+1 -1
View File
@@ -9,7 +9,7 @@ metadata:
spec:
containers:
- name: master
image: gcr.io/google_containers/redis:v1
image: k8s.gcr.io/redis:v1
env:
- name: MASTER
value: "true"
docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions.md
+8 -4
View File
@@ -74,6 +74,10 @@ This endpoint URL can then be used to create and manage custom objects.
The `kind` of these objects will be `CronTab` from the spec of the
CustomResourceDefinition object you created above.
Please note that it might take a few seconds for the endpoint to be created.
You can watch the `Established` condition of your CustomResourceDefinition
to be true or watch the discovery information of the API server for your
resource to show up.
## Create custom objects
@@ -207,12 +211,12 @@ Additionally, the following restrictions are applied to the schema:
- The field `uniqueItems` cannot be set to true.
- The field `additionalProperties` cannot be set to false.
This feature is __alpha__ in v1.8 and may change in backward incompatible ways.
Enable this feature using the `CustomResourceValidation` feature gate on
This feature is __beta__ in v1.9.
You can disable this feature using the `CustomResourceValidation` feature gate on
the [kube-apiserver](/docs/admin/kube-apiserver):
```
--feature-gates=CustomResourceValidation=true
```
--feature-gates=CustomResourceValidation=false
```
The schema is defined in the CustomResourceDefinition. In the following example, the
docs/tasks/access-kubernetes-api/setup-extension-api-server.md
+3 -3
View File
@@ -23,7 +23,7 @@ Setting up an extension API server to work the aggregation layer allows the Kube
## Setup an extension api-server to work with the aggregation layer
The following steps describe how to set up an extension-apiserver *at a high level*. For a concrete example of how they can be implemented, you can look at the [sample-apiserver](https://github.com/kubernetes/sample-apiserver/blob/master/README.md) in the Kubernetes repo.
The following steps describe how to set up an extension-apiserver *at a high level*. These steps apply regardless if you're using YAML configs or using APIs. An attempt is made to specifically identify any differences between the two. For a concrete example of how they can be implemented using YAML configs, you can look at the [sample-apiserver](https://github.com/kubernetes/sample-apiserver/blob/master/README.md) in the Kubernetes repo.
Alternatively, you can use an existing 3rd party solution, such as [apiserver-builder](https://github.com/Kubernetes-incubator/apiserver-builder/blob/master/README.md), which should generate a skeleton and automate all of the following steps for you.
@@ -38,7 +38,7 @@ Alternatively, you can use an existing 3rd party solution, such as [apiserver-bu
1. Create a Kubernetes service account in your namespace.
1. Create a Kubernetes cluster role for the operations you want to allow on your resources.
1. Create a Kubernetes cluster role binding from the default service account in your namespace to the cluster role you just created.
1. Create a Kubernetes apiservice. The CA cert above should be base 64 encoded, stripped of new lines and used as the spec.caBundle in the apiservice. This should not be namespaced.
1. Create a Kubernetes apiservice. The CA cert above should be base64 encoded, stripped of new lines and used as the spec.caBundle in the apiservice. This should not be namespaced.
1. Use kubectl to get your resource. It should return "No resources found." Which means that everything worked but you currently have no objects of that resource type created yet.
{% endcapture %}
@@ -46,7 +46,7 @@ Alternatively, you can use an existing 3rd party solution, such as [apiserver-bu
{% capture whatsnext %}
* If you haven't already, [configure the aggregation layer](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/) and enable the apiserver flags.
* For a high level overview, see [Extending the Kubernetes API with the aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/).
* For a high level overview, see [Extending the Kubernetes API with the aggregation layer](/docs/concepts/api-extension/apiserver-aggregation).
* Learn how to [Extend the Kubernetes API Using Custom Resource Definitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
{% endcapture %}
docs/tasks/administer-cluster/access-cluster-api.md
+13 -13
View File
@@ -21,7 +21,7 @@ When accessing the Kubernetes API for the first time, use the
Kubernetes command-line tool, `kubectl`.
To access a cluster, you need to know the location of the cluster and have credentials
to access it. Typically, this is automatically set-up when you work through
to access it. Typically, this is automatically set-up when you work through
a [Getting started guide](/docs/setup/),
or someone else setup the cluster and provided you with credentials and a location.
@@ -32,21 +32,21 @@ $ kubectl config view
```
Many of the [examples](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/) provide an introduction to using
kubectl. Complete documentation is found in the [kubectl manual](/docs/user-guide/kubectl/index).
kubectl. Complete documentation is found in the [kubectl manual](/docs/reference/kubectl/overview/).
### Directly accessing the REST API
kubectl handles locating and authenticating to the API server. If you want to directly access the REST API with an http client like
`curl` or `wget`, or a browser, there are multiple ways you can locate and authenticate against the API server:
1. Run kubectl in proxy mode (recommended). This method is recommended, since it uses the stored apiserver location and verifies the identity of the API server using a self-signed cert. No man-in-the-middle (MITM) attack is possible using this method.
1. Alternatively, you can provide the location and credentials directly to the http client. This works with for client code that is confused by proxies. To protect against man in the middle attacks, you'll need to import a root cert into your browser.
1. Run kubectl in proxy mode (recommended). This method is recommended, since it uses the stored apiserver location and verifies the identity of the API server using a self-signed cert. No man-in-the-middle (MITM) attack is possible using this method.
1. Alternatively, you can provide the location and credentials directly to the http client. This works with client code that is confused by proxies. To protect against man in the middle attacks, you'll need to import a root cert into your browser.
Using the Go or Python client libraries provides accessing kubectl in proxy mode.
#### Using kubectl proxy
The following command runs kubectl in a mode where it acts as a reverse proxy. It handles
The following command runs kubectl in a mode where it acts as a reverse proxy. It handles
locating the API server and authenticating.
Run it like this:
@@ -97,17 +97,17 @@ $ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
}
```
The above example uses the `--insecure` flag. This leaves it subject to MITM
attacks. When kubectl accesses the cluster it uses a stored root certificate
and client certificates to access the server. (These are installed in the
`~/.kube` directory). Since cluster certificates are typically self-signed, it
The above example uses the `--insecure` flag. This leaves it subject to MITM
attacks. When kubectl accesses the cluster it uses a stored root certificate
and client certificates to access the server. (These are installed in the
`~/.kube` directory). Since cluster certificates are typically self-signed, it
may take special configuration to get your http client to use root
certificate.
On some clusters, the API server does not require authentication; it may serve
on localhost, or be protected by a firewall. There is not a standard
for this. [Configuring Access to the API](/docs/admin/accessing-the-api)
describes how a cluster admin can configure this. Such approaches may conflict
on localhost, or be protected by a firewall. There is not a standard
for this. [Configuring Access to the API](/docs/admin/accessing-the-api)
describes how a cluster admin can configure this. Such approaches may conflict
with future high-availability support.
### Programmatic access to the API
@@ -136,7 +136,7 @@ import (
// creates the clientset
clientset, _:= kubernetes.NewForConfig(config)
// access the API to list pods
pods, _:= clientset.Core().Pods("").List(v1.ListOptions{})
pods, _:= clientset.CoreV1().Pods("").List(v1.ListOptions{})
fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
...
```
docs/tasks/administer-cluster/access-cluster-services.md
+5 -5
View File
@@ -32,7 +32,7 @@ You have several options for connecting to nodes, pods and services from outside
or it may expose it to the internet. Think about whether the service being exposed is secure.
Does it do its own authentication?
- Place pods behind services. To access one specific pod from a set of replicas, such as for debugging,
place a unique label on the pod it and create a new service which selects this label.
place a unique label on the pod and create a new service which selects this label.
- In most cases, it should not be necessary for application developer to directly access
nodes via their nodeIPs.
- Access services, nodes, or pods using the Proxy Verb.
@@ -45,9 +45,9 @@ You have several options for connecting to nodes, pods and services from outside
- Access from a node or pod in the cluster.
- Run a pod, and then connect to a shell in it using [kubectl exec](/docs/user-guide/kubectl/{{page.version}}/#exec).
Connect to other nodes, pods, and services from that shell.
- Some clusters may allow you to ssh to a node in the cluster. From there you may be able to
access cluster services. This is a non-standard method, and will work on some clusters but
not others. Browsers and other tools may or may not be installed. Cluster DNS may not work.
- Some clusters may allow you to ssh to a node in the cluster. From there you may be able to
access cluster services. This is a non-standard method, and will work on some clusters but
not others. Browsers and other tools may or may not be installed. Cluster DNS may not work.
### Discovering builtin services
@@ -102,7 +102,7 @@ If you haven't specified a name for your port, you don't have to specify *port_n
You may be able to put an apiserver proxy URL into the address bar of a browser. However:
- Web browsers cannot usually pass tokens, so you may need to use basic (password) auth. Apiserver can be configured to accept basic auth,
- Web browsers cannot usually pass tokens, so you may need to use basic (password) auth. Apiserver can be configured to accept basic auth,
but your cluster may not be configured to accept basic auth.
- Some web apps may not work, particularly those with client side javascript that construct URLs in a
way that is unaware of the proxy path prefix.
docs/tasks/administer-cluster/busybox.yaml
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml
+2 -2
View File
@@ -42,9 +42,9 @@ spec:
serviceAccountName: cloud-controller-manager
containers:
- name: cloud-controller-manager
# for in-tree providers we use gcr.io/google_containers/cloud-controller-manager
# for in-tree providers we use k8s.gcr.io/cloud-controller-manager
# this can be replaced with any other image for out-of-tree providers
image: gcr.io/google_containers/cloud-controller-manager:v1.8.0
image: k8s.gcr.io/cloud-controller-manager:v1.8.0
command:
- /usr/local/bin/cloud-controller-manager
- --cloud-provider=<YOUR_CLOUD_PROVIDER> # Add your own cloud provider here!
docs/tasks/administer-cluster/configure-upgrade-etcd.md
+9 -9
View File
@@ -184,7 +184,7 @@ Before starting the restore operation, a snapshot file must be present. It can e
If the access URLs of the restored cluster is changed from the previous cluster, the Kubernetes API server must be reconfigured accordingly. In this case, restart Kubernetes API server with the flag `--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag `--etcd-servers=$OLD_ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and `$OLD_ETCD_CLUSTER` with the respective IP addresses. If a load balancer is used in front of an etcd cluster, you might need to update the load balancer instead.
If the majority of etcd members have permanently failed, the etcd cluster is considered failed. In this scenario, Kubernetes cannot make any changes to its current state. Although the scheduled pods might continue to run, no new pods can be scheduled. In such cases, recover the etcd cluster and potentially reconfigure Kubernetes API server to fix the issue.
If the majority of etcd members have permanently failed, the etcd cluster is considered failed. In this scenario, Kubernetes cannot make any changes to its current state. Although the scheduled pods might continue to run, no new pods can be scheduled. In such cases, recover the etcd cluster and potentially reconfigure Kubernetes API server to fix the issue.
## Upgrading and rolling back etcd clusters
@@ -212,7 +212,7 @@ Note that we need to migrate both the etcd versions that we are using (from 2.2.
to at least 3.0.x) as well as the version of the etcd API that Kubernetes talks to. The etcd 3.0.x
binaries support both the v2 and v3 API.
This document describes how to do this migration. If you want to skip the
This document describes how to do this migration. If you want to skip the
background and cut right to the procedure, see [Upgrade
Procedure](#upgrade-procedure).
@@ -227,7 +227,7 @@ There are requirements on how an etcd cluster upgrade can be performed. The prim
Upgrade only one minor release at a time. For example, we cannot upgrade directly from 2.1.x to 2.3.x.
Within patch releases it is possible to upgrade and downgrade between arbitrary versions. Starting a cluster for
any intermediate minor release, waiting until the cluster is healthy, and then
shutting down the cluster down will perform the migration. For example, to upgrade from version 2.1.x to 2.3.y,
shutting down the cluster will perform the migration. For example, to upgrade from version 2.1.x to 2.3.y,
it is enough to start etcd in 2.2.z version, wait until it is healthy, stop it, and then start the
2.3.y version.
@@ -239,7 +239,7 @@ The etcd team has provided a [custom rollback tool](https://git.k8s.io/kubernete
but the rollback tool has these limitations:
* This custom rollback tool is not part of the etcd repo and does not receive the same
testing as the rest of etcd. We are testing it in a couple of end-to-end tests.
testing as the rest of etcd. We are testing it in a couple of end-to-end tests.
There is only community support here.
* The rollback can be done only from the 3.0.x version (that is using the v3 API) to the
@@ -263,13 +263,13 @@ rollback might require restarting all Kubernetes components on all nodes.
**Note**: At the time of writing, both Kubelet and KubeProxy are using “resource
version” only for watching (i.e. are not using resource versions for anything
else). And both are using reflector and/or informer frameworks for watching
(i.e. they don’t send watch requests themselves). Both those frameworks if they
(i.e. they don’t send watch requests themselves). Both those frameworks if they
can’t renew watch, they will start from “current version” by doing “list + watch
from the resource version returned by list”. That means that if the apiserver
will be down for the period of rollback, all of node components should basically
restart their watches and start from “now” when apiserver is back. And it will
be back with new resource version. That would mean that restarting node
components is not needed. But the assumptions here may not hold forever.
components is not needed. But the assumptions here may not hold forever.
{: .note}
### Design
@@ -284,7 +284,7 @@ focus on them at all. We focus only on the upgrade/rollback here.
### New etcd Docker image
We decided to completely change the content of the etcd image and the way it works.
So far, the Docker image for etcd in version X has contained only the etcd and
So far, the Docker image for etcd in version X has contained only the etcd and
etcdctl binaries.
Going forward, the Docker image for etcd in version X will contain multiple
@@ -337,7 +337,7 @@ script works as follows:
1. Verify that the detected version is 3.0.x with the v3 API, and the
desired version is 2.2.1 with the v2 API. We don’t support any other rollback.
1. If so, we run the custom tool provided by etcd team to do the offline
rollback. This tool reads the v3 formatted data and writes it back to disk
rollback. This tool reads the v3 formatted data and writes it back to disk
in v2 format.
1. Finally update the contents of the version file.
@@ -350,7 +350,7 @@ Simply modify the command line in the etcd manifest to:
Starting in Kubernetes version 1.6, this has been done in the manifests for new
Google Compute Engine clusters. You should also specify these environment
variables. In particular,you must keep `STORAGE_MEDIA_TYPE` set to
variables. In particular, you must keep `STORAGE_MEDIA_TYPE` set to
`application/json` if you wish to preserve the option to roll back.
```
docs/tasks/administer-cluster/coredns.md
+42
View File
@@ -0,0 +1,42 @@
---
approvers:
- johnbelamaric
title: Using CoreDNS for Service Discovery
min-kubernetes-server-version: v1.9
---
{% include feature-state-alpha.md %}
{% capture overview %}
This page describes how to enable CoreDNS instead of kube-dns for service
discovery.
{% endcapture %}
{% capture prerequisites %}
{% include task-tutorial-prereqs.md %}
{% endcapture %}
{% capture steps %}
## Installing CoreDNS with kubeadm
In Kubernetes 1.9, [CoreDNS](https://coredns.io) is available as an alpha feature and
may be installed by setting the `CoreDNS` feature gate to `true` during `kubeadm init`:
```
kubeadm init --feature-gates=CoreDNS=true
```
This installs CoreDNS instead of kube-dns.
{% endcapture %}
{% capture whatsnext %}
You can configure [CoreDNS](https://coredns.io) to support many more use cases than
kube-dns by modifying the `Corefile`. For more information, see the
[CoreDNS site](https://coredns.io/2017/05/08/custom-dns-entries-for-kubernetes/).
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-cluster/cpu-management-policies.md
+8 -2
View File
@@ -1,14 +1,20 @@
---
title: Control CPU Management Policies on the Node
approvers:
- sjenning
- ConnorDoyle
- balajismaniam
---
{% include feature-state-beta.md %}
* TOC
{:toc}
Kubernetes keeps many aspects of how pods execute on nodes abstracted
from the user. This is by design.  However, some workloads require
stronger guarantees in terms of latency and/or performance in order to operate
acceptably.  The kubelet provides methods to enable more complex workload
acceptably. The kubelet provides methods to enable more complex workload
placement policies while keeping the abstraction free from explicit placement
directives.
@@ -188,5 +194,5 @@ spec:
This pod runs in the `Guaranteed` QoS class because only `limits` are specified
and `requests` are set equal to `limits` when not explicitly specified. And the
container's resource limit for the CPU resource is an integer greater than or
equal to one.The `nginx` container is granted 2 exclusive CPUs.
equal to one. The `nginx` container is granted 2 exclusive CPUs.
docs/tasks/administer-cluster/developing-cloud-controller-manager.md
+1 -1
View File
@@ -17,7 +17,7 @@ can develop their features independently from the core Kubernetes release cycles
Before going into how to build your own cloud controller manager, some background on how it works under the hood is helpful. The cloud controller manager is code from `kube-controller-manager` utilizing Go interfaces to allow implementations from any cloud to be plugged in. Most of the scaffolding and generic controller implementations will be in core, but it will always exec out to the cloud interfaces it is provided, so long as the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L29-L50) is satisfied.
To dive a little deeper into implementation details, all cloud controller managers will import packages from Kubernetes core, the only difference being each project will register their own cloud providers by calling [cloudprovider.RegisterCloudProvier](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52) where a global variable of available cloud providers is updated.
To dive a little deeper into implementation details, all cloud controller managers will import packages from Kubernetes core, the only difference being each project will register their own cloud providers by calling [cloudprovider.RegisterCloudProvider](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52) where a global variable of available cloud providers is updated.
## Developing
docs/tasks/administer-cluster/dns-custom-nameservers.md
+245 -27
View File
@@ -2,12 +2,12 @@
approvers:
- bowei
- zihongz
title: Configure private DNS zones and upstream nameservers in Kubernetes
title: Configure DNS Service
---
{% capture overview %}
This page shows how to add custom private DNS zones (stub domains) and upstream
nameservers.
This page provides hints on configuring DNS Pod and guidance on customizing the
DNS resolution process and diagnosing DNS problems.
{% endcapture %}
{% capture prerequisites %}
@@ -18,6 +18,45 @@ nameservers.
{% capture steps %}
## Introduction
Starting from Kubernetes v1.3, DNS is a built-in service launched automatically
using the addon manager
[cluster add-on](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/README.md).
The running Kubernetes DNS pod holds 3 containers:
- "`kubedns`": The `kubedns` process watches the Kubernetes master for changes
in Services and Endpoints, and maintains in-memory lookup structures to serve
DNS requests.
- "`dnsmasq`": The `dnsmasq` container adds DNS caching to improve performance.
- "`healthz`": The `healthz` container provides a single health check endpoint
while performing dual healthchecks (for `dnsmasq` and `kubedns`).
The DNS pod is exposed as a Kubernetes Service with a static IP. Once assigned
the kubelet passes DNS configured using the `--cluster-dns=<dns-service-ip>`
flag to each container.
DNS names also need domains. The local domain is configurable in the kubelet
using the flag `--cluster-domain=<default-local-domain>`.
The Kubernetes cluster DNS server is based off the
[SkyDNS](https://github.com/skynetservices/skydns) library. It supports forward
lookups (A records), service lookups (SRV records) and reverse IP address
lookups (PTR records).
## Inheriting DNS from the node
When running a pod, kubelet will prepend the cluster DNS server and search
paths to the node's own DNS settings. If the node is able to resolve DNS names
specific to the larger environment, pods should be able to, also.
See [Known issues](#known-issues) below for a caveat.
If you don't want this, or if you want a different DNS config for pods, you can
use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will
not inherit DNS. Setting it to a valid file path means that kubelet will use
this file instead of `/etc/resolv.conf` for DNS inheritance.
## Configure stub-domain and upstream DNS servers
Cluster administrators can specify custom stub domains and upstream nameservers
@@ -43,7 +82,8 @@ As specified, DNS requests with the “.acme.local” suffix
are forwarded to a DNS listening at 1.2.3.4. Google Public DNS
serves the upstream queries.
The table below describes how queries with certain domain names would map to their destination DNS servers:
The table below describes how queries with certain domain names would map to
their destination DNS servers:
| Domain name | Server answering the query |
| ----------- | -------------------------- |
@@ -58,36 +98,37 @@ details about the configuration option format.
{% capture discussion %}
## Understanding name resolution in Kubernetes
### Impacts on Pods
DNS policies can be set on a per-pod basis. Currently Kubernetes supports two pod-specific DNS policies: “Default” and “ClusterFirst”. These policies are specified with the `dnsPolicy` flag.
Custom upstream nameservers and stub domains won't impact Pods that have their
`dnsPolicy` set to "`Default`" or "`None`".
*NOTE: "Default" is not the default DNS policy. If `dnsPolicy` is not
explicitly specified, then “ClusterFirst” is used.*
If a Pod's `dnsPolicy` is set to "`ClusterFirst`", its name resolution is
handled differently, depending on whether stub-domain and upstream DNS servers
are configured.
### "Default" DNS Policy
**Without custom configurations**: Any query that does not match the configured
cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream
nameserver inherited from the node.
If `dnsPolicy` is set to “Default”, then the name resolution configuration is
inherited from the node that the pods run on. Custom upstream nameservers and stub domains cannot be used in conjunction with this policy.
### "ClusterFirst" DNS Policy
If the `dnsPolicy` is set to "ClusterFirst", name resolution is handled differently, *depending on whether stub-domain and upstream DNS servers are configured*.
**Without custom configurations**: Any query that does not match the configured cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream nameserver inherited from the node.
**With custom configurations**: If stub domains and upstream DNS servers are configured (as in the [previous example](#configuring-stub-domain-and-upstream-dns-servers)), DNS queries will be
routed according to the following flow:
**With custom configurations**: If stub domains and upstream DNS servers are
configured (as in the [previous example](#configuring-stub-domain-and-upstream-dns-servers)),
DNS queries will be routed according to the following flow:
1. The query is first sent to the DNS caching layer in kube-dns.
1. From the caching layer, the suffix of the request is examined and then forwarded to the appropriate DNS, based on the following cases:
1. From the caching layer, the suffix of the request is examined and then
forwarded to the appropriate DNS, based on the following cases:
* *Names with the cluster suffix* (e.g.".cluster.local"): The request is sent to kube-dns.
* *Names with the cluster suffix* (e.g.".cluster.local"):
The request is sent to kube-dns.
* *Names with the stub domain suffix* (e.g. ".acme.local"): The request is sent to the configured custom DNS resolver (e.g. listening at 1.2.3.4).
* *Names with the stub domain suffix* (e.g. ".acme.local"):
The request is sent to the configured custom DNS resolver (e.g. listening at 1.2.3.4).
* *Names without a matching suffix* (e.g."widget.com"): The request is forwarded to the upstream DNS (e.g. Google public DNS servers at 8.8.8.8 and 8.8.4.4).
* *Names without a matching suffix* (e.g."widget.com"):
The request is forwarded to the upstream DNS
(e.g. Google public DNS servers at 8.8.8.8 and 8.8.4.4).
![DNS lookup flow](/docs/tasks/administer-cluster/dns-custom-nameservers/dns.png)
@@ -100,9 +141,9 @@ Options for the kube-dns `kube-system:kube-dns` ConfigMap:
| `stubDomains` (optional) | A JSON map using a DNS suffix key (e.g. “acme.local”) and a value consisting of a JSON array of DNS IPs. | The target nameserver may itself be a Kubernetes service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | Note: If specified, then the values specified replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
## Additional examples
### Examples
### Example: Stub domain
#### Example: Stub domain
In this example, the user has a Consul DNS service discovery system that they wish to
integrate with kube-dns. The consul domain server is located at 10.150.0.1, and
@@ -124,7 +165,7 @@ Note that the cluster administrator did not wish to override the node’s
upstream nameservers, so they did not specify the optional
`upstreamNameservers` field.
### Example: Upstream nameserver
#### Example: Upstream nameserver
In this example the cluster administrator wants to explicitly force all
non-cluster DNS lookups to go through their own nameserver at 172.16.0.1.
@@ -142,6 +183,183 @@ data:
[“172.16.0.1”]
```
## Debugging DNS resolution
### Create a simple Pod to use as a test environment
Create a file named busybox.yaml with the following contents:
{% include code.html language="yaml" file="busybox.yaml" ghlink="/docs/tasks/administer-cluster/busybox.yaml" %}
Then create a pod using this file and verify its status:
```shell
$ kubectl create -f busybox.yaml
pod "busybox" created
$ kubectl get pods busybox
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 <some-time>
```
Once that pod is running, you can exec `nslookup` in that environment.
If you see something like the following, DNS is working correctly.
```shell
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10
Name: kubernetes.default
Address 1: 10.0.0.1
```
If the `nslookup` command fails, check the following:
### Check the local DNS configuration first
Take a look inside the resolv.conf file.
(See [Inheriting DNS from the node](#inheriting-dns-from-the-node) and
[Known issues](#known-issues) below for more information)
```shell
$ kubectl exec busybox cat /etc/resolv.conf
```
Verify that the search path and name server are set up like the following
(note that search path may vary for different cloud providers):
```
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
nameserver 10.0.0.10
options ndots:5
```
Errors such as the following indicate a problem with the kube-dns add-on or
associated Services:
```
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10
nslookup: can't resolve 'kubernetes.default'
```
or
```
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'kubernetes.default'
```
### Check if the DNS pod is running
Use the `kubectl get pods` command to verify that the DNS pod is running.
```shell
$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
...
kube-dns-v19-ezo1y 3/3 Running 0 1h
...
```
If you see that no pod is running or that the pod has failed/completed, the DNS
add-on may not be deployed by default in your current environment and you will
have to deploy it manually.
### Check for Errors in the DNS pod
Use `kubectl logs` command to see logs for the DNS daemons.
```shell
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
```
See if there is any suspicious log. Letter '`W`', '`E`', '`F`' at the beginning
represent Warning, Error and Failure. Please search for entries that have these
as the logging level and use
[kubernetes issues](https://github.com/kubernetes/kubernetes/issues)
to report unexpected errors.
### Is DNS service up?
Verify that the DNS service is up by using the `kubectl get service` command.
```shell
$ kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
kube-dns 10.0.0.10 <none> 53/UDP,53/TCP 1h
...
```
If you have created the service or in the case it should be created by default
but it does not appear, see
[debugging services](/docs/tasks/debug-application-cluster/debug-service/) for
more information.
### Are DNS endpoints exposed?
You can verify that DNS endpoints are exposed by using the `kubectl get endpoints`
command.
```shell
$ kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 10.180.3.17:53,10.180.3.17:53 1h
```
If you do not see the endpoints, see endpoints section in the
[debugging services](/docs/tasks/debug-application-cluster/debug-service/) documentation.
For additional Kubernetes DNS examples, see the
[cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns)
in the Kubernetes GitHub repository.
## Known issues
Kubernetes installs do not configure the nodes' resolv.conf files to use the
cluster DNS by default, because that process is inherently distro-specific.
This should probably be implemented eventually.
Linux's libc is impossibly stuck ([see this bug from
2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just
3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to
consume 1 `nameserver` record and 3 `search` records. This means that if a
local installation already uses 3 `nameserver`s or uses more than 3 `search`es,
some of those settings will be lost. As a partial workaround, the node can run
`dnsmasq` which will provide more `nameserver` entries, but not more `search`
entries. You can also use kubelet's `--resolv-conf` flag.
If you are using Alpine version 3.3 or earlier as your base image, DNS may not
work properly owing to a known issue with Alpine.
Check [here](https://github.com/kubernetes/kubernetes/issues/30215)
for more information.
## Kubernetes Federation (Multiple Zone support)
Release 1.3 introduced Cluster Federation support for multi-site Kubernetes
installations. This required some minor (backward-compatible) changes to the
way the Kubernetes cluster DNS server processes DNS queries, to facilitate
the lookup of federated services (which span multiple Kubernetes clusters).
See the [Cluster Federation Administrators' Guide](/docs/concepts/cluster-administration/federation/)
for more details on Cluster Federation and multi-site support.
## References
- [DNS for Services and Pods](/docs/concepts/services-networking/dns-pod-service/)
- [Docs for the DNS cluster addon](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/dns/README.md)
## What's next
- [Autoscaling the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/).
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-cluster/dns-horizontal-autoscaler.yaml
+1 -1
View File
@@ -13,7 +13,7 @@ spec:
spec:
containers:
- name: autoscaler
image: gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.0.0
image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.0.0
resources:
requests:
cpu: "20m"
docs/tasks/administer-cluster/opaque-integer-resource-node.md → docs/tasks/administer-cluster/extended-resource-node.md
+26 -27
View File
@@ -1,15 +1,15 @@
---
title: Advertise Opaque Integer Resources for a Node
title: Advertise Extended Resources for a Node
---
{% capture overview %}
This page shows how to specify opaque integer resources for a Node.
Opaque integer resources allow cluster administrators to advertise node-level
This page shows how to specify extended resources for a Node.
Extended resources allow cluster administrators to advertise node-level
resources that would otherwise be unknown to Kubernetes.
{% include feature-state-deprecated.md %}
{% include feature-state-stable.md %}
{% endcapture %}
@@ -31,9 +31,9 @@ kubectl get nodes
Choose one of your Nodes to use for this exercise.
## Advertise a new opaque integer resource on one of your Nodes
## Advertise a new extended resource on one of your Nodes
To advertise a new opaque integer resource on a Node, send an HTTP PATCH request to
To advertise a new extended resource on a Node, send an HTTP PATCH request to
the Kubernetes API server. For example, suppose one of your Nodes has four dongles
attached. Here's an example of a PATCH request that advertises four dongle resources
for your Node.
@@ -47,7 +47,7 @@ Host: k8s-master:8080
[
{
"op": "add",
"path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-dongle",
"path": "/status/capacity/example.com~1dongle",
"value": "4"
}
]
@@ -69,7 +69,7 @@ Replace `<your-node-name>` with the name of your Node:
```shell
curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-dongle", "value": "4"}]' \
--data '[{"op": "add", "path": "/status/capacity/example.com~1dongle", "value": "4"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status
```
@@ -85,7 +85,7 @@ The output shows that the Node has a capacity of 4 dongles:
"alpha.kubernetes.io/nvidia-gpu": "0",
"cpu": "2",
"memory": "2049008Ki",
"pod.alpha.kubernetes.io/opaque-int-resource-dongle": "4",
"example.com/dongle": "4",
```
Describe your Node:
@@ -98,53 +98,52 @@ Once again, the output shows the dongle resource:
```yaml
Capacity:
alpha.kubernetes.io/nvidia-gpu: 0
cpu: 2
memory: 2049008Ki
pod.alpha.kubernetes.io/opaque-int-resource-dongle: 4
alpha.kubernetes.io/nvidia-gpu: 0
cpu: 2
memory: 2049008Ki
example.com/dongle: 4
```
Now, application developers can create Pods that request a certain
number of dongles. See
[Assign Opaque Integer Resources to a Container](/docs/tasks/configure-pod-container/opaque-integer-resource/).
[Assign Extended Resources to a Container](/docs/tasks/configure-pod-container/extended-resource/).
## Discussion
Opaque integer resources are similar to memory and CPU resources. For example,
Extended resources are similar to memory and CPU resources. For example,
just as a Node has a certain amount of memory and CPU to be shared by all components
running on the Node, it can have a certain number of dongles to be shared
by all components running on the Node. And just as application developers
can create Pods that request a certain amount of memory and CPU, they can
create Pods that request a certain number of dongles.
Opaque integer resources are called opaque because Kubernetes does not
Extended resources are opaque to Kubernetes; Kubernetes does not
know anything about what they are. Kubernetes knows only that a Node
has a certain number of them. They are called integer resources because
they must be advertised in integer amounts. For example, a Node can advertise
four dongles, but not 4.5 dongles.
has a certain number of them. Extended resources must be advertised in integer
amounts. For example, a Node can advertise four dongles, but not 4.5 dongles.
### Storage example
Suppose a Node has 800 GiB of a special kind of disk storage. You could
create a name for the special storage, say opaque-int-resource-special-storage.
create a name for the special storage, say example.com/special-storage.
Then you could advertise it in chunks of a certain size, say 100 GiB. In that case,
your Node would advertise that it has eight resources of type
opaque-int-resource-special-storage.
example.com/special-storage.
```yaml
Capacity:
...
pod.alpha.kubernetes.io/opaque-int-resource-special-storage: 8
example.com/special-storage: 8
```
If you want to allow arbitrary requests for special storage, you
could advertise special storage in chunks of size 1 byte. In that case, you would advertise
800Gi resources of type opaque-int-resource-special-storage.
800Gi resources of type example.com/special-storage.
```yaml
Capacity:
...
pod.alpha.kubernetes.io/opaque-int-resource-special-storage: 800Gi
example.com/special-storage: 800Gi
```
Then a Container could request any number of bytes of special storage, up to 800Gi.
@@ -162,7 +161,7 @@ Host: k8s-master:8080
[
{
"op": "remove",
"path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-dongle",
"path": "/status/capacity/example.com~1dongle",
}
]
```
@@ -179,7 +178,7 @@ Replace `<your-node-name>` with the name of your Node:
```shell
curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "remove", "path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-dongle"}]' \
--data '[{"op": "remove", "path": "/status/capacity/example.com~1dongle"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status
```
@@ -196,7 +195,7 @@ kubectl describe node <your-node-name> | grep dongle
### For application developers
* [Assign Opaque Integer Resources to a Container](/docs/tasks/configure-pod-container/opaque-integer-resource/)
* [Assign Extended Resources to a Container](/docs/tasks/configure-pod-container/extended-resource/)
### For cluster administrators
docs/tasks/administer-cluster/kubeadm-upgrade-1-9.md
+244
View File
@@ -0,0 +1,244 @@
---
approvers:
- pipejakob
- luxas
- roberthbailey
- jbeda
title: Upgrading/downgrading kubeadm clusters between v1.8 to v1.9
---
{% capture overview %}
This guide is for upgrading `kubeadm` clusters from version 1.8.x to 1.9.x, as well as 1.8.x to 1.8.y and 1.9.x to 1.9.y where `y > x`.
See also [upgrading kubeadm clusters from 1.7 to 1.8](/docs/tasks/administer-cluster/kubeadm-upgrade-1-8/) if you're on a 1.7 cluster currently.
{% endcapture %}
{% capture prerequisites %}
Before proceeding:
- You need to have a functional `kubeadm` Kubernetes cluster running version 1.8.0 or higher in order to use the process described here. Swap also needs to be disabled.
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md) carefully.
- `kubeadm upgrade` now allows you to upgrade etcd. `kubeadm upgrade` will also upgrade of etcd to 3.1.10 as part of upgrading from v1.8 to v1.9 by default. This is due to the fact that etcd 3.1.10 is the officially validated etcd version for Kubernetes v1.9. The upgrade is handled automatically by kubeadm for you.
- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up what's important to you. For example, any app-level state, such as a database an app might depend on (like MySQL or MongoDB) must be backed up beforehand.
Also, note that only one minor version upgrade is supported. For example, you can only upgrade from 1.8 to 1.9, not from 1.7 to 1.9.
{% endcapture %}
{% capture steps %}
## Upgrading your control plane
Execute these commands on your master node:
1. Install the most recent version of `kubeadm` using `curl` like so:
```shell
$ export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
$ export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
$ curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
$ chmod a+rx /usr/bin/kubeadm
```
**Caution:** Upgrading the `kubeadm` package on your system prior to upgrading the control plane causes a failed upgrade.
Even though `kubeadm` ships in the Kubernetes repositories, it's important to install `kubeadm` manually. The kubeadm
team is working on fixing this limitation.
{: .caution}
Verify that this download of kubeadm works and has the expected version:
```shell
$ kubeadm version
```
2. On the master node, run the following:
```shell
$ kubeadm upgrade plan
[preflight] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Fetching available versions to upgrade to:
[upgrade/versions] Cluster version: v1.8.1
[upgrade/versions] kubeadm version: v1.9.0
[upgrade/versions] Latest stable version: v1.9.0
[upgrade/versions] Latest version in the v1.8 series: v1.8.6
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.8.1 v1.8.6
Upgrade to the latest version in the v1.8 series:
COMPONENT CURRENT AVAILABLE
API Server v1.8.1 v1.8.6
Controller Manager v1.8.1 v1.8.6
Scheduler v1.8.1 v1.8.6
Kube Proxy v1.8.1 v1.8.6
Kube DNS 1.14.4 1.14.5
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.8.6
_____________________________________________________________________
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.8.1 v1.9.0
Upgrade to the latest experimental version:
COMPONENT CURRENT AVAILABLE
API Server v1.8.1 v1.9.0
Controller Manager v1.8.1 v1.9.0
Scheduler v1.8.1 v1.9.0
Kube Proxy v1.8.1 v1.9.0
Kube DNS 1.14.5 1.14.7
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.9.0
Note: Before you do can perform this upgrade, you have to update kubeadm to v1.9.0
_____________________________________________________________________
```
The `kubeadm upgrade plan` checks that your cluster is upgradeable and fetches the versions available to upgrade to in an user-friendly way.
3. Pick a version to upgrade to and run. For example:
```shell
$ kubeadm upgrade apply v1.9.0
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/version] You have chosen to upgrade to version "v1.9.0"
[upgrade/versions] Cluster version: v1.8.1
[upgrade/versions] kubeadm version: v1.9.0
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.9.0"...
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/etcd.yaml"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/staticpods] Writing upgraded Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/kube-scheduler.yaml"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.9.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.
```
`kubeadm upgrade apply` does the following:
- Checks that your cluster is in an upgradeable state:
- The API server is reachable,
- All nodes are in the `Ready` state
- The control plane is healthy
- Enforces the version skew policies.
- Makes sure the control plane images are available or available to pull to the machine.
- Upgrades the control plane components or rollbacks if any of them fails to come up.
- Applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
4. Manually upgrade your Software Defined Network (SDN).
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow.
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
find your CNI provider and see if there are additional upgrade steps
necessary.
## Upgrading your master and node packages
For each host (referred to as `$HOST` below) in your cluster, upgrade `kubelet` by executing the following commands:
1. Prepare the host for maintenance, marking it unschedulable and evicting the workload:
```shell
$ kubectl drain $HOST --ignore-daemonsets
```
When running this command against the master host, this error is expected and can be safely ignored (since there are static pods running on the master):
```shell
node "master" already cordoned
error: pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): etcd-kubeadm, kube-apiserver-kubeadm, kube-controller-manager-kubeadm, kube-scheduler-kubeadm
```
2. Upgrade the Kubernetes package versions on the `$HOST` node by using a Linux distribution-specific package manager:
If the host is running a Debian-based distro such as Ubuntu, run:
```shell
$ apt-get update
$ apt-get upgrade
```
If the host is running CentOS or the like, run:
```shell
$ yum update
```
Now the new version of the `kubelet` should be running on the host. Verify this using the following command on `$HOST`:
```shell
$ systemctl status kubelet
```
3. Bring the host back online by marking it schedulable:
```shell
$ kubectl uncordon $HOST
```
4. After upgrading `kubelet` on each host in your cluster, verify that all nodes are available again by executing the following (from anywhere, for example, from outside the cluster):
```shell
$ kubectl get nodes
```
If the `STATUS` column of the above command shows `Ready` for all of your hosts, you are done.
## Recovering from a failure state
If `kubeadm upgrade` somehow fails and fails to roll back, for example due to an unexpected shutdown during execution,
you can run `kubeadm upgrade` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring.
You can use `kubeadm upgrade` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-cluster/namespace-prod.json
+10
View File
@@ -0,0 +1,10 @@
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "production",
"labels": {
"name": "production"
}
}
}
docs/tasks/administer-cluster/namespaces-walkthrough.md
+1 -1
View File
@@ -66,7 +66,7 @@ $ kubectl create -f docs/admin/namespaces/namespace-dev.json
And then let's create the production namespace using kubectl.
```shell
$ kubectl create -f docs/admin/namespaces/namespace-prod.json
$ kubectl create -f docs/tasks/administer-cluster/namespace-prod.json
```
To be sure things are right, let's list all of the namespaces in our cluster.
docs/tasks/administer-cluster/namespaces.md
+1 -1
View File
@@ -152,7 +152,7 @@ $ kubectl create -f docs/admin/namespaces/namespace-dev.json
And then let's create the production namespace using kubectl.
```shell
$ kubectl create -f docs/admin/namespaces/namespace-prod.json
$ kubectl create -f docs/tasks/administer-cluster/namespace-prod.json
```
To be sure things are right, list all of the namespaces in our cluster.
docs/tasks/administer-cluster/out-of-resource.md
+119 -128
View File
@@ -9,27 +9,25 @@ title: Configure Out Of Resource Handling
* TOC
{:toc}
This page explains how to configure out of resource handling with `kubelet`.
The `kubelet` needs to preserve node stability when available compute resources
are low.
This is especially important when dealing with incompressible resources such as
memory or disk.
If either resource is exhausted, the node would become unstable.
are low. This is especially important when dealing with incompressible
compute resources, such as memory or disk space. If such resources are exhausted,
nodes become unstable.
## Eviction Policy
The `kubelet` can pro-actively monitor for and prevent against total starvation
of a compute resource. In those cases, the `kubelet` can pro-actively fail one
or more pods in order to reclaim the starved resource. When the `kubelet` fails
a pod, it terminates all containers in the pod, and the `PodPhase` is
transitioned to `Failed`.
The `kubelet` can proactively monitor for and prevent total starvation of a
compute resource. In those cases, the `kubelet` can reclaim the starved
resource by proactively failing one or more Pods. When the `kubelet` fails
a Pod, it terminates all of its containers and transitions its `PodPhase` to `Failed`.
### Eviction Signals
The `kubelet` can support the ability to trigger eviction decisions on the
signals described in the table below. The value of each signal is described in
the description column based on the `kubelet` summary API.
The `kubelet` supports eviction decisions based on the signals described in the following
table. The value of each signal is described in the Description column, which is based on
the `kubelet` summary API.
| Eviction Signal | Description |
|----------------------------|-----------------------------------------------------------------------|
@@ -44,11 +42,11 @@ The percentage based value is calculated relative to the total capacity
associated with each signal.
The value for `memory.available` is derived from the cgroupfs instead of tools
like `free -m`. This is important because `free -m` does not work in a
like `free -m`. This is important because `free -m` does not work in a
container, and if users use the [node
allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) feature, out of resource decisions
are made local to the end user pod part of the cgroup hierarchy as well as the
root node. This
are made local to the end user Pod part of the cgroup hierarchy as well as the
root node. This
[script](/docs/tasks/administer-cluster/out-of-resource/memory-available.sh)
reproduces the same set of steps that the `kubelet` performs to calculate
`memory.available`. The `kubelet` excludes inactive_file (i.e. # of bytes of
@@ -62,69 +60,69 @@ memory is reclaimable under pressure.
container writable layers.
`imagefs` is optional. `kubelet` auto-discovers these filesystems using
cAdvisor. `kubelet` does not care about any other filesystems. Any other types
cAdvisor. `kubelet` does not care about any other filesystems. Any other types
of configurations are not currently supported by the kubelet. For example, it is
*not OK* to store volumes and logs in a dedicated `filesystem`.
In future releases, the `kubelet` will deprecate the existing [garbage
collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/) support in favor of eviction in
response to disk pressure.
collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/)
support in favor of eviction in response to disk pressure.
### Eviction Thresholds
The `kubelet` supports the ability to specify eviction thresholds that trigger the `kubelet` to reclaim resources.
Each threshold is of the following form:
Each threshold has the following form:
`<eviction-signal><operator><quantity>`
`[eviction-signal][operator][quantity]`
* valid `eviction-signal` tokens as defined above.
* valid `operator` tokens are `<`
* valid `quantity` tokens must match the quantity representation used by Kubernetes
* an eviction threshold can be expressed as a percentage if ends with `%` token.
where:
For example, if a node has `10Gi` of memory, and the desire is to induce eviction
if available memory falls below `1Gi`, an eviction threshold can be specified as either
of the following (but not both).
* `eviction-signal` is an eviction signal token as defined in the previous table.
* `operator` is the desired relational operator, such as `<` (less than).
* `quantity` is the eviction threshhold quantity, such as `1Gi`. These tokens must
match the quantity representation used by Kubernetes. An eviction threshold can also
be expressed as a percentage using the `%` token.
* `memory.available<10%`
* `memory.available<1Gi`
For example, if a node has `10Gi` of total memory and you want trigger eviction if
the available memory falls below `1Gi`, you can define the eviction threshold as
either `memory.available<10%` or `memory.available<1Gi`. You cannot use both.
#### Soft Eviction Thresholds
A soft eviction threshold pairs an eviction threshold with a required
administrator specified grace period. No action is taken by the `kubelet`
administrator-specified grace period. No action is taken by the `kubelet`
to reclaim resources associated with the eviction signal until that grace
period has been exceeded. If no grace period is provided, the `kubelet` will
error on startup.
period has been exceeded. If no grace period is provided, the `kubelet`
returns an error on startup.
In addition, if a soft eviction threshold has been met, an operator can
specify a maximum allowed pod termination grace period to use when evicting
pods from the node. If specified, the `kubelet` will use the lesser value among
specify a maximum allowed Pod termination grace period to use when evicting
pods from the node. If specified, the `kubelet` uses the lesser value among
the `pod.Spec.TerminationGracePeriodSeconds` and the max allowed grace period.
If not specified, the `kubelet` will kill pods immediately with no graceful
If not specified, the `kubelet` kills Pods immediately with no graceful
termination.
To configure soft eviction thresholds, the following flags are supported:
* `eviction-soft` describes a set of eviction thresholds (e.g. `memory.available<1.5Gi`) that if met over a
corresponding grace period would trigger a pod eviction.
corresponding grace period would trigger a Pod eviction.
* `eviction-soft-grace-period` describes a set of eviction grace periods (e.g. `memory.available=1m30s`) that
correspond to how long a soft eviction threshold must hold before triggering a pod eviction.
correspond to how long a soft eviction threshold must hold before triggering a Pod eviction.
* `eviction-max-pod-grace-period` describes the maximum allowed grace period (in seconds) to use when terminating
pods in response to a soft eviction threshold being met.
#### Hard Eviction Thresholds
A hard eviction threshold has no grace period, and if observed, the `kubelet`
will take immediate action to reclaim the associated starved resource. If a
hard eviction threshold is met, the `kubelet` will kill the pod immediately
will take immediate action to reclaim the associated starved resource. If a
hard eviction threshold is met, the `kubelet` kills the Pod immediately
with no graceful termination.
To configure hard eviction thresholds, the following flag is supported:
* `eviction-hard` describes a set of eviction thresholds (e.g. `memory.available<1Gi`) that if met
would trigger a pod eviction.
would trigger a Pod eviction.
The `kubelet` has the following default hard eviction threshold:
@@ -138,10 +136,10 @@ The `kubelet` evaluates eviction thresholds per its configured housekeeping inte
### Node Conditions
The `kubelet` will map one or more eviction signals to a corresponding node condition.
The `kubelet` maps one or more eviction signals to a corresponding node condition.
If a hard eviction threshold has been met, or a soft eviction threshold has been met
independent of its associated grace period, the `kubelet` will report a condition that
independent of its associated grace period, the `kubelet` reports a condition that
reflects the node is under pressure.
The following node conditions are defined that correspond to the specified eviction signal.
@@ -151,7 +149,7 @@ The following node conditions are defined that correspond to the specified evict
| `MemoryPressure` | `memory.available` | Available memory on the node has satisfied an eviction threshold |
| `DiskPressure` | `nodefs.available`, `nodefs.inodesFree`, `imagefs.available`, or `imagefs.inodesFree` | Available disk space and inodes on either the node's root filesystem or image filesystem has satisfied an eviction threshold |
The `kubelet` will continue to report node status updates at the frequency specified by
The `kubelet` continues to report node status updates at the frequency specified by
`--node-status-update-frequency` which defaults to `10s`.
### Oscillation of node conditions
@@ -174,85 +172,76 @@ condition back to `false`.
### Reclaiming node level resources
If an eviction threshold has been met and the grace period has passed,
the `kubelet` will initiate the process of reclaiming the pressured resource
the `kubelet` initiates the process of reclaiming the pressured resource
until it has observed the signal has gone below its defined threshold.
The `kubelet` attempts to reclaim node level resources prior to evicting end-user pods. If
The `kubelet` attempts to reclaim node level resources prior to evicting end-user Pods. If
disk pressure is observed, the `kubelet` reclaims node level resources differently if the
machine has a dedicated `imagefs` configured for the container runtime.
#### With Imagefs
#### With `imagefs`
If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order:
If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting the dead Pods and their containers.
1. Delete dead pods/containers
If `imagefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting all unused images.
If `imagefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order:
#### Without `imagefs`
If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space in the following order:
1. Delete dead Pods and their containers
1. Delete all unused images
#### Without Imagefs
### Evicting end-user Pods
If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order:
If the `kubelet` is unable to reclaim sufficient resource on the node, `kubelet` begins evicting Pods.
1. Delete dead pods/containers
1. Delete all unused images
The `kubelet` ranks Pods for eviction first by their quality of service, and then by the consumption
of the starved compute resource relative to the Pods' scheduling requests.
### Evicting end-user pods
As a result, `kubectl` ranks and evicts Pods in the following order:
If the `kubelet` is unable to reclaim sufficient resource on the node,
it will begin evicting pods.
The `kubelet` ranks pods for eviction as follows:
* by their quality of service.
* by the consumption of the starved compute resource relative to the pods scheduling request.
As a result, pod eviction occurs in the following order:
* `BestEffort` pods that consume the most of the starved resource are failed
first.
* `Burstable` pods that consume the greatest amount of the starved resource
relative to their request for that resource are killed first. If no pod
* `BestEffort` Pods consume the most of the starved resource are failed first.
Local disk is a `BestEffort` resource.
* `Burstable` Pods consume the greatest amount of the starved resource
relative to their request for that resource are killed first. If no Pod
has exceeded its request, the strategy targets the largest consumer of the
starved resource.
* `Guaranteed` pods that consume the greatest amount of the starved resource
relative to their request are killed first. If no pod has exceeded its request,
the strategy targets the largest consumer of the starved resource.
A `Guaranteed` pod is guaranteed to never be evicted because of another pod's
resource consumption. If a system daemon (i.e. `kubelet`, `docker`, `journald`, etc.)
is consuming more resources than were reserved via `system-reserved` or `kube-reserved` allocations,
and the node only has `Guaranteed` pod(s) remaining, then the node must choose to evict a
`Guaranteed` pod in order to preserve node stability, and to limit the impact
of the unexpected consumption to other `Guaranteed` pod(s).
Local disk is a `BestEffort` resource. If necessary, `kubelet` will evict pods one at a time to reclaim
disk when `DiskPressure` is encountered. The `kubelet` will rank pods by quality of service. If the `kubelet`
is responding to `inode` starvation, it will reclaim `inodes` by evicting pods with the lowest quality of service
first. If the `kubelet` is responding to lack of available disk, it will rank pods within a quality of service
* `Guaranteed` Pods are guaranteed only when requests and limits are specified
for all the containers and they are equal. A `Guaranteed` Pod is guaranteed to
never be evicted because of another Pod's resource consumption. If a system
daemon (such as `kubelet`, `docker`, and `journald`) is consuming more resources
than were reserved via `system-reserved` or `kube-reserved` allocations, and the
node only has `Guaranteed` Pods remaining, then the node must choose to evict a
`Guaranteed` Pod in order to preserve node stability and to limit the impact
of the unexpected consumption to other `Guaranteed` Pods.
If necessary, `kubelet` evicts Pods one at a time to reclaim disk when `DiskPressure`
is encountered. If the `kubelet` is responding to `inode` starvation, it reclaims
`inodes` by evicting Pods with the lowest quality of service first. If the `kubelet`
is responding to lack of available disk, it ranks Pods within a quality of service
that consumes the largest amount of disk and kill those first.
#### With Imagefs
#### With `imagefs`
If `nodefs` is triggering evictions, `kubelet` will sort pods based on the usage on `nodefs`
If `nodefs` is triggering evictions, `kubelet` sorts Pods based on the usage on `nodefs`
- local volumes + logs of all its containers.
If `imagefs` is triggering evictions, `kubelet` will sort pods based on the writable layer usage of all its containers.
If `imagefs` is triggering evictions, `kubelet` sorts Pods based on the writable layer usage of all its containers.
#### Without Imagefs
#### Without `imagefs`
If `nodefs` is triggering evictions, `kubelet` will sort pods based on their total disk usage
If `nodefs` is triggering evictions, `kubelet` sorts Pods based on their total disk usage
- local volumes + logs & writable layer of all its containers.
### Minimum eviction reclaim
In certain scenarios, eviction of pods could result in reclamation of small amount of resources. This can result in
In certain scenarios, eviction of Pods could result in reclamation of small amount of resources. This can result in
`kubelet` hitting eviction thresholds in repeated successions. In addition to that, eviction of resources like `disk`,
is time consuming.
To mitigate these issues, `kubelet` can have a per-resource `minimum-reclaim`. Whenever `kubelet` observes
resource pressure, `kubelet` will attempt to reclaim at least `minimum-reclaim` amount of resource below
resource pressure, `kubelet` attempts to reclaim at least `minimum-reclaim` amount of resource below
the configured eviction threshold.
For example, with the following configuration:
@@ -262,31 +251,31 @@ For example, with the following configuration:
--eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"`
```
If an eviction threshold is triggered for `memory.available`, the `kubelet` will work to ensure
that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` will work
to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it will
work to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure
If an eviction threshold is triggered for `memory.available`, the `kubelet` works to ensure
that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` works
to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it
works to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure
on their associated resources.
The default `eviction-minimum-reclaim` is `0` for all resources.
### Scheduler
The node will report a condition when a compute resource is under pressure. The
The node reports a condition when a compute resource is under pressure. The
scheduler views that condition as a signal to dissuade placing additional
pods on the node.
| Node Condition | Scheduler Behavior |
| ---------------- | ------------------------------------------------ |
| `MemoryPressure` | No new `BestEffort` pods are scheduled to the node. |
| `DiskPressure` | No new pods are scheduled to the node. |
| `MemoryPressure` | No new `BestEffort` Pods are scheduled to the node. |
| `DiskPressure` | No new Pods are scheduled to the node. |
## Node OOM Behavior
If the node experiences a system OOM (out of memory) event prior to the `kubelet` is able to reclaim memory,
the node depends on the [oom_killer](https://lwn.net/Articles/391222/) to respond.
The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the pod.
The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the Pod.
| Quality of Service | oom_score_adj |
|----------------------------|-----------------------------------------------------------------------|
@@ -294,7 +283,7 @@ The `kubelet` sets a `oom_score_adj` value for each container based on the quali
| `BestEffort` | 1000 |
| `Burstable` | min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999) |
If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` will calculate
If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` calculates
an `oom_score` based on the percentage of memory it's using on the node, and then add the `oom_score_adj` to get an
effective `oom_score` for the container, and then kills the container with the highest score.
@@ -302,17 +291,19 @@ The intended behavior should be that containers with the lowest quality of servi
are consuming the largest amount of memory relative to the scheduling request should be killed first in order
to reclaim memory.
Unlike pod eviction, if a pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`.
Unlike Pod eviction, if a Pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`.
## Best Practices
The following sections describe best practices for out of resource handling.
### Schedulable resources and eviction policies
Let's imagine the following scenario:
Consider the following scenario:
* Node memory capacity: `10Gi`
* Operator wants to reserve 10% of memory capacity for system daemons (kernel, `kubelet`, etc.)
* Operator wants to evict pods at 95% memory utilization to reduce thrashing and incidence of system OOM.
* Operator wants to evict Pods at 95% memory utilization to reduce thrashing and incidence of system OOM.
To facilitate this scenario, the `kubelet` would be launched as follows:
@@ -324,31 +315,30 @@ To facilitate this scenario, the `kubelet` would be launched as follows:
Implicit in this configuration is the understanding that "System reserved" should include the amount of memory
covered by the eviction threshold.
To reach that capacity, either some pod is using more than its request, or the system is using more than `500Mi`.
To reach that capacity, either some Pod is using more than its request, or the system is using more than `500Mi`.
This configuration will ensure that the scheduler does not place pods on a node that immediately induce memory pressure
and trigger eviction assuming those pods use less than their configured request.
This configuration ensures that the scheduler does not place Pods on a node that immediately induce memory pressure
and trigger eviction assuming those Pods use less than their configured request.
### DaemonSet
It is never desired for a `kubelet` to evict a pod that was derived from
a `DaemonSet` since the pod will immediately be recreated and rescheduled
back to the same node.
It is never desired for `kubelet` to evict a `DaemonSet` Pod, since the Pod is
immediately recreated and rescheduled back to the same node.
At the moment, the `kubelet` has no ability to distinguish a pod created
from `DaemonSet` versus any other object. If/when that information is
available, the `kubelet` could pro-actively filter those pods from the
candidate set of pods provided to the eviction strategy.
At the moment, the `kubelet` has no ability to distinguish a Pod created
from `DaemonSet` versus any other object. If/when that information is
available, the `kubelet` could pro-actively filter those Pods from the
candidate set of Pods provided to the eviction strategy.
In general, it is strongly recommended that `DaemonSet` not
create `BestEffort` pods to avoid being identified as a candidate pod
for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` pods.
create `BestEffort` Pods to avoid being identified as a candidate Pod
for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` Pods.
## Deprecation of existing feature flags to reclaim disk
`kubelet` has been freeing up disk space on demand to keep the node stable.
As disk based eviction matures, the following `kubelet` flags will be marked for deprecation
As disk based eviction matures, the following `kubelet` flags are marked for deprecation
in favor of the simpler configuration supported around eviction.
| Existing Flag | New Flag |
@@ -363,26 +353,27 @@ in favor of the simpler configuration supported around eviction.
## Known issues
The following sections describe known issues related to out of resource handling.
### kubelet may not observe memory pressure right away
The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage
The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage
increases within that window rapidly, the `kubelet` may not observe `MemoryPressure` fast enough, and the `OOMKiller`
will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this
will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this
latency, and instead have the kernel tell us when a threshold has been crossed immediately.
If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for
this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature
this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature
to prevent system OOMs, and promote eviction of workloads so cluster state can rebalance.
### kubelet may evict more pods than needed
### kubelet may evict more Pods than needed
The pod eviction may evict more pods than needed due to stats collection timing gap. This can be mitigated by adding
The Pod eviction may evict more Pods than needed due to stats collection timing gap. This can be mitigated by adding
the ability to get root container stats on an on-demand basis [(https://github.com/google/cadvisor/issues/1247)](https://github.com/google/cadvisor/issues/1247) in the future.
### How kubelet ranks pods for eviction in response to inode exhaustion
### How kubelet ranks Pods for eviction in response to inode exhaustion
At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes
inode exhaustion, it will evict pods by ranking them by quality of service. The following issue has been opened in cadvisor
to track per container inode consumption [(https://github.com/google/cadvisor/issues/1422)](https://github.com/google/cadvisor/issues/1422) which would allow us to rank pods
by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict
that pod over others.
At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes
inode exhaustion, it evicts Pods by ranking them by quality of service. The following issue has been opened in cadvisor
to track per container inode consumption [(https://github.com/google/cadvisor/issues/1422)](https://github.com/google/cadvisor/issues/1422) which would allow us to rank Pods
by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict that Pod over others.
docs/tasks/administer-cluster/pod1.yaml
+1 -1
View File
@@ -7,4 +7,4 @@ metadata:
spec:
containers:
- name: pod-with-no-annotation-container
image: gcr.io/google_containers/pause:2.0
image: k8s.gcr.io/pause:2.0
docs/tasks/administer-cluster/pod2.yaml
+1 -1
View File
@@ -8,4 +8,4 @@ spec:
schedulerName: default-scheduler
containers:
- name: pod-with-default-annotation-container
image: gcr.io/google_containers/pause:2.0
image: k8s.gcr.io/pause:2.0
docs/tasks/administer-cluster/pod3.yaml
+1 -1
View File
@@ -8,4 +8,4 @@ spec:
schedulerName: my-scheduler
containers:
- name: pod-with-second-annotation-container
image: gcr.io/google_containers/pause:2.0
image: k8s.gcr.io/pause:2.0
docs/tasks/administer-cluster/pvc-protection.md
+144
View File
@@ -0,0 +1,144 @@
---
approvers:
- msau42
- jsafrane
title: Persistent Volume Claim Protection
---
{% capture overview %}
{% assign for_k8s_version="v1.9" %}{% include feature-state-alpha.md %}
As of Kubernetes 1.9, persistent volume claims (PVCs) that are in active use by a pod can be protected from pre-mature removal.
{% endcapture %}
{% capture prerequisites %}
- A v1.9 or higher Kubernetes must be installed.
- As PVC Protection is a Kubernetes v1.9 alpha feature it must be enabled:
1. [Admission controller](/docs/admin/admission-controllers/) must be started with the [PVC Protection plugin](/docs/admin/admission-controllers/#persistent-volume-claim-protection-alpha).
2. All Kubernetes components must be started with the `PVCProtection` alpha features enabled.
{% endcapture %}
{% capture steps %}
## PVC Protection Verification
The example below uses a GCE PD `StorageClass`, however, similar steps can be performed for any volume type.
Create a `StorageClass` for convenient storage provisioning:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
```
There are two scenarios: a PVC deleted by a user is either in active use or not in active use by a pod.
### Scenario 1: The PVC is not in active use by a pod
- Create a PVC:
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: slzc
spec:
accessModes:
- ReadWriteOnce
storageClassName: slow
resources:
requests:
storage: 3.7Gi
```
- Check that the PVC has the finalizer `kubernetes.io/pvc-protection` set:
```shell
$ kubectl describe pvc slzc
Name: slzc
Namespace: default
StorageClass: slow
Status: Bound
Volume: pvc-bee8c30a-d6a3-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 2m persistentvolume-controller Successfully provisioned volume pvc-bee8c30a-d6a3-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Delete the PVC and check that the PVC (not in active use by a pod) was removed successfully.
### Scenario 2: The PVC is in active use by a pod
- Again, create the same PVC.
- Create a pod that uses the PVC:
```yaml
kind: Pod
apiVersion: v1
metadata:
name: app1
spec:
containers:
- name: test-pod
image: k8s.gcr.io/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "date > /mnt/app1.txt; sleep 60 && exit 0 || exit 1"
volumeMounts:
- name: path-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: path-pvc
persistentVolumeClaim:
claimName: slzc
```
- Wait until the pod status is `Running`, i.e. the PVC becomes in active use.
- Delete the PVC that is now in active use by a pod and verify that the PVC is not removed but its status is `Terminating`:
```shell
Name: slzc
Namespace: default
StorageClass: slow
Status: Terminating (since Fri, 01 Dec 2017 14:47:55 +0000)
Volume: pvc-803a1f4d-d6a6-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 52s persistentvolume-controller Successfully provisioned volume pvc-803a1f4d-d6a6-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Wait until the pod status is `Terminated` (either delete the pod or wait until it finishes). Afterwards, check that the PVC is removed.
{% endcapture %}
{% capture discussion %}
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-cluster/quota-api-object.md
+2 -2
View File
@@ -67,7 +67,7 @@ status:
services.nodeports: "0"
```
## Create a PersistentVolumeClaim:
## Create a PersistentVolumeClaim
Here is the configuration file for a PersistentVolumeClaim object:
@@ -92,7 +92,7 @@ NAME STATUS
pvc-quota-demo Pending
```
## Attempt to create a second PersistentVolumeClaim:
## Attempt to create a second PersistentVolumeClaim
Here is the configuration file for a second PersistentVolumeClaim:
docs/tasks/administer-cluster/reconfigure-kubelet.md
+1 -1
View File
@@ -27,7 +27,7 @@ from alpha.
running, with the `DynamicKubeletConfig` feature gate enabled and the Kubelet's
`--dynamic-config-dir` flag set to a writeable directory on the Node.
This flag must be set to enable Dynamic Kubelet Configuration.
- The kubectl command-line tool must be also be v1.8 or higher, and must be
- The kubectl command-line tool must be also v1.8 or higher, and must be
configured to communicate with the cluster.
{% endcapture %}
docs/tasks/administer-cluster/reserve-compute-resources.md
+5 -5
View File
@@ -52,8 +52,8 @@ Resources can be reserved for two categories of system daemons in the `kubelet`.
### Enabling QoS and Pod level cgroups
To properly enforce node allocatable constraints on the node, you must
enable the new cgroup hierarchy via the `--cgroups-per-qos` flag. This flag is
enabled by default. When enabled, the `kubelet` will parent all end-user pods
enable the new cgroup hierarchy via the `--cgroups-per-qos` flag. This flag is
enabled by default. When enabled, the `kubelet` will parent all end-user pods
under a cgroup hierarchy managed by the `kubelet`.
### Configuring a cgroup driver
@@ -71,7 +71,7 @@ transient slices for resources that are supported by that init system.
Depending on the configuration of the associated container runtime,
operators may have to choose a particular cgroup driver to ensure
proper system behavior. For example, if operators use the `systemd`
proper system behavior. For example, if operators use the `systemd`
cgroup driver provided by the `docker` runtime, the `kubelet` must
be configured to use the `systemd` cgroup driver.
@@ -199,7 +199,7 @@ Under this scenario, `Allocatable` will be `14.5 CPUs`, `28.5Gi` of memory and
`98Gi` of local storage.
Scheduler ensures that the total memory `requests` across all pods on this node does
not exceed `28.5Gi` and storage doesn't exceed `88Gi`.
Kubelet evicts pods whenever the overall memory usage exceeds across pods exceed `28.5Gi`,
Kubelet evicts pods whenever the overall memory usage across pods exceeds `28.5Gi`,
or if overall disk usage exceeds `88Gi` If all processes on the node consume as
much CPU as they can, pods together cannot consume more than `14.5 CPUs`.
@@ -224,7 +224,7 @@ kubelet flag. Note that unless `--kube-reserved`, or `--system-reserved` or
not affect existing deployments.
As of Kubernetes version 1.6, `kubelet` launches pods in their own cgroup
sandbox in a dedicated part of the cgroup hierarchy it manages. Operators are
sandbox in a dedicated part of the cgroup hierarchy it manages. Operators are
required to drain their nodes prior to upgrade of the `kubelet` from prior
versions in order to ensure pods and their associated containers are launched in
the proper part of the cgroup hierarchy.
docs/tasks/administer-cluster/safely-drain-node.md
+8 -8
View File
@@ -98,8 +98,8 @@ You should first be familiar with using [Kubernetes language clients](/docs/task
The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
@@ -123,7 +123,7 @@ The API can respond in one of three ways:
- If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
- If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
@@ -131,21 +131,21 @@ The API can respond in one of three ways:
- If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases.
For a given eviction request, there are two cases:
- There is no budget that matches this pod. In this case, the server always
- There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
- There is at least one budget. In this case, any of the three above responses may
- There is at least one budget. In this case, any of the three above responses may
apply.
In some cases, an application may reach a broken state where it will never return anything
other than 429 or 500. This can happen, for example, if the replacement pod created by the
other than 429 or 500. This can happen, for example, if the replacement pod created by the
application's controller does not become ready, or if the last pod evicted has a very long
termination grace period.
In this case, there are two potential solutions:
- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation.
- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation.
- After a suitably long wait, `DELETE` the pod instead of using the eviction API.
Kubernetes does not specify what the behavior should be in this case; it is up to the
docs/tasks/administer-cluster/securing-a-cluster.md
+13 -1
View File
@@ -1,6 +1,9 @@
---
approvers:
- smarterclayton
- liggitt
- ericchiang
- destijl
title: Securing a Cluster
---
@@ -68,6 +71,15 @@ to prevent accidental escalation. You can make roles specific to your use case i
Consult the [authorization reference section](/docs/admin/authorization/) for more information.
## Controlling access to the Kubelet
Kubelets expose HTTPS endpoints which give access to data of varying sensitivity, and allow performing operations with varying levels of power on the node and within containers.
By default, Kubelets allow full access to those endpoints.
To secure access to those endpoints, enable Kubelet authentication and authorization.
Consult the [Kubelet authentication/authorization reference](/docs/admin/kubelet-authentication-authorization) for more information.
## Controlling the capabilities of a workload or user at runtime
@@ -151,7 +163,7 @@ access to a subset of the keyspace is strongly recommended.
### Enable audit logging
The [audit logger](/docs/tasks/debug-application-cluster/audit/) is an alpha feature that records actions taken by the
The [audit logger](/docs/tasks/debug-application-cluster/audit/) is a beta feature that records actions taken by the
API for later analysis in the event of a compromise. It is recommended to enable audit logging
and archive the audit file on a secure server.
docs/tasks/configure-pod-container/attach-handler-lifecycle-event.md
+3 -3
View File
@@ -85,9 +85,9 @@ unless the Pod's grace period expires. For more details, see
### Reference
* [Lifecycle](/docs/resources-reference/{{page.version}}/#lifecycle-v1-core)
* [Container](/docs/resources-reference/{{page.version}}/#container-v1-core)
* See `terminationGracePeriodSeconds` in [PodSpec](/docs/resources-reference/{{page.version}}/#podspec-v1-core)
* [Lifecycle](/docs/api-reference/{{page.version}}/#lifecycle-v1-core)
* [Container](/docs/api-reference/{{page.version}}/#container-v1-core)
* See `terminationGracePeriodSeconds` in [PodSpec](/docs/api-reference/{{page.version}}/#podspec-v1-core)
{% endcapture %}
docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md
+6 -6
View File
@@ -34,7 +34,7 @@ broken states, and cannot recover except by being restarted. Kubernetes provides
liveness probes to detect and remedy such situations.
In this exercise, you create a Pod that runs a Container based on the
`gcr.io/google_containers/busybox` image. Here is the configuration file for the Pod:
`k8s.gcr.io/busybox` image. Here is the configuration file for the Pod:
{% include code.html language="yaml" file="exec-liveness.yaml" ghlink="/docs/tasks/configure-pod-container/exec-liveness.yaml" %}
@@ -75,8 +75,8 @@ The output indicates that no liveness probes have failed yet:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
24s 24s 1 {default-scheduler } Normal Scheduled Successfully assigned liveness-exec to worker0
23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Pulling pulling image "gcr.io/google_containers/busybox"
23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Pulled Successfully pulled image "gcr.io/google_containers/busybox"
23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Pulling pulling image "k8s.gcr.io/busybox"
23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Pulled Successfully pulled image "k8s.gcr.io/busybox"
23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Created Created container with docker id 86849c15382e; Security:[seccomp=unconfined]
23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Started Started container with docker id 86849c15382e
```
@@ -94,8 +94,8 @@ probes have failed, and the containers have been killed and recreated.
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
37s 37s 1 {default-scheduler } Normal Scheduled Successfully assigned liveness-exec to worker0
36s 36s 1 {kubelet worker0} spec.containers{liveness} Normal Pulling pulling image "gcr.io/google_containers/busybox"
36s 36s 1 {kubelet worker0} spec.containers{liveness} Normal Pulled Successfully pulled image "gcr.io/google_containers/busybox"
36s 36s 1 {kubelet worker0} spec.containers{liveness} Normal Pulling pulling image "k8s.gcr.io/busybox"
36s 36s 1 {kubelet worker0} spec.containers{liveness} Normal Pulled Successfully pulled image "k8s.gcr.io/busybox"
36s 36s 1 {kubelet worker0} spec.containers{liveness} Normal Created Created container with docker id 86849c15382e; Security:[seccomp=unconfined]
36s 36s 1 {kubelet worker0} spec.containers{liveness} Normal Started Started container with docker id 86849c15382e
2s 2s 1 {kubelet worker0} spec.containers{liveness} Warning Unhealthy Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
@@ -117,7 +117,7 @@ liveness-exec 1/1 Running 1 1m
## Define a liveness HTTP request
Another kind of liveness probe uses an HTTP GET request. Here is the configuration
file for a Pod that runs a container based on the `gcr.io/google_containers/liveness`
file for a Pod that runs a container based on the `k8s.gcr.io/liveness`
image.
{% include code.html language="yaml" file="http-liveness.yaml" ghlink="/docs/tasks/configure-pod-container/http-liveness.yaml" %}
docs/tasks/configure-pod-container/configure-persistent-volume-storage.md
+4 -4
View File
@@ -200,10 +200,10 @@ PersistentVolume are not present on the Pod resource itself.
### Reference
* [PersistentVolume](/docs/resources-reference/{{page.version}}/#persistentvolume-v1-core)
* [PersistentVolumeSpec](/docs/resources-reference/{{page.version}}/#persistentvolumespec-v1-core)
* [PersistentVolumeClaim](/docs/resources-reference/{{page.version}}/#persistentvolumeclaim-v1-core)
* [PersistentVolumeClaimSpec](/docs/resources-reference/{{page.version}}/#persistentvolumeclaimspec-v1-core)
* [PersistentVolume](/docs/api-reference/{{page.version}}/#persistentvolume-v1-core)
* [PersistentVolumeSpec](/docs/api-reference/{{page.version}}/#persistentvolumespec-v1-core)
* [PersistentVolumeClaim](/docs/api-reference/{{page.version}}/#persistentvolumeclaim-v1-core)
* [PersistentVolumeClaimSpec](/docs/api-reference/{{page.version}}/#persistentvolumeclaimspec-v1-core)
{% endcapture %}
docs/tasks/configure-pod-container/configure-pod-configmap.md
+6 -6
View File
@@ -38,7 +38,7 @@ This page provides a series of usage examples demonstrating how to configure Pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
# Define the environment variable
@@ -88,7 +88,7 @@ This page provides a series of usage examples demonstrating how to configure Pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: SPECIAL_LEVEL_KEY
@@ -134,7 +134,7 @@ This page provides a series of usage examples demonstrating how to configure Pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- configMapRef:
@@ -161,7 +161,7 @@ metadata:
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "echo $(SPECIAL_LEVEL_KEY) $(SPECIAL_TYPE_KEY)" ]
env:
- name: SPECIAL_LEVEL_KEY
@@ -214,7 +214,7 @@ metadata:
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "ls /etc/config/" ]
volumeMounts:
- name: config-volume
@@ -248,7 +248,7 @@ metadata:
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "/bin/sh","-c","cat /etc/config/keys" ]
volumeMounts:
- name: config-volume
docs/tasks/configure-pod-container/exec-liveness.yaml
+1 -1
View File
@@ -7,7 +7,7 @@ metadata:
spec:
containers:
- name: liveness
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
docs/tasks/configure-pod-container/extended-resource-pod-2.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: extended-resource-demo-2
spec:
containers:
- name: extended-resource-demo-2-ctr
image: nginx
resources:
requests:
example.com/dongle: 2
limits:
example.com/dongle: 2
docs/tasks/configure-pod-container/extended-resource-pod.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: extended-resource-demo
spec:
containers:
- name: extended-resource-demo-ctr
image: nginx
resources:
requests:
example.com/dongle: 3
limits:
example.com/dongle: 3
docs/tasks/configure-pod-container/opaque-integer-resource.md → docs/tasks/configure-pod-container/extended-resource.md
+27 -22
View File
@@ -1,12 +1,12 @@
---
title: Assign Opaque Integer Resources to a Container
title: Assign Extended Resources to a Container
---
{% capture overview %}
This page shows how to assign opaque integer resources to a Container.
This page shows how to assign extended resources to a Container.
{% include feature-state-deprecated.md %}
{% include feature-state-stable.md %}
{% endcapture %}
@@ -16,7 +16,7 @@ This page shows how to assign opaque integer resources to a Container.
{% include task-tutorial-prereqs.md %}
Before you do this exercise, do the exercise in
[Advertise Opaque Integer Resources for a Node](/docs/tasks/administer-cluster/opaque-integer-resource-node/).
[Advertise Extended Resources for a Node](/docs/tasks/administer-cluster/extended-resource-node/).
That will configure one of your Nodes to advertise a dongle resource.
{% endcapture %}
@@ -24,40 +24,45 @@ That will configure one of your Nodes to advertise a dongle resource.
{% capture steps %}
## Assign an opaque integer resource to a Pod
## Assign an extended resource to a Pod
To request an opaque integer resource, include the `resources:requests` field in your
Container manifest. Opaque integer resources have the prefix `pod.alpha.kubernetes.io/opaque-int-resource-`.
To request an extended resource, include the `resources:requests` field in your
Container manifest. Extended resources are fully qualified with any domain outside of
`*.kubernetes.io/`. Valid extended resource names have the form `example.com/foo` where
`example.com` is replaced with your organization's domain and `foo` is a
descriptive resource name.
Here is the configuration file for a Pod that has one Container:
{% include code.html language="yaml" file="oir-pod.yaml" ghlink="/docs/tasks/configure-pod-container/oir-pod.yaml" %}
{% include code.html language="yaml" file="extended-resource-pod.yaml" ghlink="/docs/tasks/configure-pod-container/extended-resource-pod.yaml" %}
In the configuration file, you can see that the Container requests 3 dongles.
Create a Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/oir-pod.yaml
kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/extended-resource-pod.yaml
```
Verify that the Pod is running:
```shell
kubectl get pod oir-demo
kubectl get pod extended-resource-demo
```
Describe the Pod:
```shell
kubectl describe pod oir-demo
kubectl describe pod extended-resource-demo
```
The output shows dongle requests:
```yaml
Limits:
example.com/dongle: 3
Requests:
pod.alpha.kubernetes.io/opaque-int-resource-dongle: 3
example.com/dongle: 3
```
## Attempt to create a second Pod
@@ -65,7 +70,7 @@ Requests:
Here is the configuration file for a Pod that has one Container. The Container requests
two dongles.
{% include code.html language="yaml" file="oir-pod-2.yaml" ghlink="/docs/tasks/configure-pod-container/oir-pod-2.yaml" %}
{% include code.html language="yaml" file="extended-resource-pod-2.yaml" ghlink="/docs/tasks/configure-pod-container/extended-resource-pod-2.yaml" %}
Kubernetes will not be able to satisfy the request for two dongles, because the first Pod
used three of the four available dongles.
@@ -73,13 +78,13 @@ used three of the four available dongles.
Attempt to create a Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/oir-pod-2.yaml
kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/extended-resource-pod-2.yaml
```
Describe the Pod
```shell
kubectl describe pod oir-demo-2
kubectl describe pod extended-resource-demo-2
```
The output shows that the Pod cannot be scheduled, because there is no Node that has
@@ -93,22 +98,22 @@ Conditions:
...
Events:
...
... Warning FailedScheduling pod (oir-demo-2) failed to fit in any node
fit failure summary on nodes : Insufficient pod.alpha.kubernetes.io/opaque-int-resource-dongle (1)
... Warning FailedScheduling pod (extended-resource-demo-2) failed to fit in any node
fit failure summary on nodes : Insufficient example.com/dongle (1)
```
View the Pod status:
```shell
kubectl get pod oir-demo-2
kubectl get pod extended-resource-demo-2
```
The output shows that the Pod was created, but not scheduled to run on a Node.
It has a status of Pending:
```yaml
NAME READY STATUS RESTARTS AGE
oir-demo-2 0/1 Pending 0 6m
NAME READY STATUS RESTARTS AGE
extended-resource-demo-2 0/1 Pending 0 6m
```
## Clean up
@@ -116,7 +121,7 @@ oir-demo-2 0/1 Pending 0 6m
Delete the Pod that you created for this exercise:
```shell
kubectl delete pod oir-demo-2
kubectl delete pod extended-resource-demo-2
```
{% endcapture %}
@@ -130,7 +135,7 @@ kubectl delete pod oir-demo-2
### For cluster administrators
* [Advertise Opaque Integer Resources for a Node](/docs/tasks/administer-cluster/opaque-integer-resource-node/)
* [Advertise Extended Resources for a Node](/docs/tasks/administer-cluster/extended-resource-node/)
{% endcapture %}
docs/tasks/configure-pod-container/http-liveness.yaml
+1 -1
View File
@@ -7,7 +7,7 @@ metadata:
spec:
containers:
- name: liveness
image: gcr.io/google_containers/liveness
image: k8s.gcr.io/liveness
args:
- /server
livenessProbe:
docs/tasks/configure-pod-container/oir-pod-2.yaml
-11
View File
@@ -1,11 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: oir-demo-2
spec:
containers:
- name: oir-demo-2-ctr
image: nginx
resources:
requests:
pod.alpha.kubernetes.io/opaque-int-resource-dongle: 2
docs/tasks/configure-pod-container/oir-pod.yaml
-11
View File
@@ -1,11 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: oir-demo
spec:
containers:
- name: oir-demo-ctr
image: nginx
resources:
requests:
pod.alpha.kubernetes.io/opaque-int-resource-dongle: 3
docs/tasks/configure-pod-container/tcp-liveness-readiness.yaml
+1 -1
View File
@@ -7,7 +7,7 @@ metadata:
spec:
containers:
- name: goproxy
image: gcr.io/google_containers/goproxy:0.1
image: k8s.gcr.io/goproxy:0.1
ports:
- containerPort: 8080
readinessProbe:
docs/tasks/debug-application-cluster/audit-policy.yaml
+68
View File
@@ -0,0 +1,68 @@
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
docs/tasks/debug-application-cluster/audit.md
+241 -426
View File
@@ -9,7 +9,9 @@ title: Auditing
* TOC
{:toc}
Kubernetes Audit provides a security-relevant chronological set of records documenting
{% include feature-state-beta.md %}
Kubernetes auditing provides a security-relevant chronological set of records documenting
the sequence of activities that have affected system by individual users, administrators
or other components of the system. It allows cluster administrator to
answer the following questions:
@@ -22,10 +24,241 @@ answer the following questions:
- from where was it initiated?
- to where was it going?
[Kube-apiserver][kube-apiserver] performs auditing. Each request on each stage
of its execution generates an event, which is then pre-processed according to
a certain policy and written to a backend. You can find more details about the
pipeline in the [design proposal][auditing-proposal].
## Audit Policy
Audit policy defines rules about what events should be recorded and what data
they should include. When an event is processed, it's compared against the list
of rules in order. The first matching rule sets the [audit level][auditing-level]
of the event. The audit policy object structure is defined in the
[`audit.k8s.io` API group][auditing-api].
You can pass a file with the policy to [kube-apiserver][kube-apiserver]
using the `--audit-policy-file` flag. If the flag is omitted, no events are logged.
__Note:__ `kind` and `apiVersion` fields along with `rules` __must__ be provided
in the audit policy file. A policy with no (0) rules, or a policy that doesn't
provide valid `apiVersion` and `kind` values is treated as illegal.
Some example audit policy files:
{% include code.html language="yaml" file="audit-policy.yaml" ghlink="/docs/tasks/debug-application-cluster/audit-policy.yaml" %}
You can use a minimal audit policy file to log all requests at the `Metadata` level:
```yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
```
The [audit profile used by GCE][gce-audit-profile] should be used as reference by
admins constructing their own audit profiles.
## Audit backends
Audit backends implement exporting audit events to an external storage.
[Kube-apiserver][kube-apiserver] out of the box provides two backends:
- Log backend, which writes events to a disk
- Webhook backend, which sends events to an external API
In both cases, audit events structure is defined by the API in the
`audit.k8s.io` API group. The current version of the API is
[`v1beta1`][auditing-api].
### Log backend
Log backend writes audit events to a file in JSON format. You can configure
log audit backend using the following [kube-apiserver][kube-apiserver] flags:
- `--audit-log-path` specifies the log file path that log backend uses to write
audit events. Not specifying this flag disables log backend. `-` means standard out
- `--audit-log-maxage` defined the maximum number of days to retain old audit log files
- `--audit-log-maxbackup` defines the maximum number of audit log files to retain
- `--audit-log-maxsize` defines the maximum size in megabytes of the audit log file before it gets rotated
### Webhook backend
Webhook backend sends audit events to a remote API, which is assumed to be the
same API as [kube-apiserver][kube-apiserver] exposes. You can configure webhook
audit backend using the following kube-apiserver flags:
- `--audit-webhook-config-file` specifies the path to a file with a webhook
configuration. Webhook configuration is effectively a [kubeconfig][kubeconfig].
- `--audit-webhook-mode` define the buffering strategy, one of the following:
- `batch` - buffer events and asynchronously send the set of events to the external service
This is the default
- `blocking` - block API server responses on sending each event to the external service
The webhook config file uses the kubeconfig format to specify the remote address of
the service and credentials used to connect to it.
## Log Collector Examples
### Use fluentd to collect and distribute audit events from log file
[Fluentd][fluentd] is an open source data collector for unified logging layer.
In this example, we will use fluentd to split audit events by different namespaces.
1. install [fluentd, fluent-plugin-forest and fluent-plugin-rewrite-tag-filter][fluentd_install_doc] in the kube-apiserver node
1. create a config file for fluentd
```shell
$ cat <<EOF > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
@type tail
# audit log path of kube-apiserver
path /var/log/audit
pos_file /var/log/audit.pos
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%N%z
tag audit
</source>
<filter audit>
#https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
type record_transformer
enable_ruby
<record>
namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ? "none":record["objectRef"]["namespace"])}
</record>
</filter>
<match audit>
# route audit according to namespace element in context
@type rewrite_tag_filter
rewriterule1 namespace ^(.+) ${tag}.$1
</match>
<filter audit.**>
@type record_transformer
remove_keys namespace
</filter>
<match audit.**>
@type forest
subtype file
remove_prefix audit
<template>
time_slice_format %Y%m%d%H
compress gz
path /var/log/audit-${tag}.*.log
format json
include_time_key true
</template>
</match>
```
1. start fluentd
```shell
$ fluentd -c /etc/fluentd/config -vv
```
1. start kube-apiserver with the following options:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
```
1. check audits for different namespaces in /var/log/audit-*.log
### Use logstash to collect and distribute audit events from webhook backend
[Logstash][logstash] is an open source, server-side data processing tool. In this example,
we will use logstash to collect audit events from webhook backend, and save events of
different users into different files.
1. install [logstash][logstash_install_doc]
1. create config file for logstash
```shell
$ cat <<EOF > /etc/logstash/config
input{
http{
#TODO, figure out a way to use kubeconfig file to authenticate to logstash
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
port=>8888
}
}
filter{
split{
# Webhook audit backend sends several events together with EventList
# split each event here.
field=>[items]
# We only need event subelement, remove others.
remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
}
mutate{
rename => {items=>event}
}
}
output{
file{
# Audit events from different users will be saved into different files.
path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
}
}
```
1. start logstash
```shell
$ bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/
```
1. create a [kubeconfig file](/docs/tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/) for kube-apiserver webhook audit backend
```shell
$ cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
clusters:
- cluster:
server: http://<ip_of_logstash>:8888
name: logstash
contexts:
- context:
cluster: logstash
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
EOF
```
1. start kube-apiserver with the following options:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
```
1. check audits in logstash node's directories /var/log/kube-audit-*/audit
Note that in addition to file output plugin, logstash has a variety of outputs that
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.
## Legacy Audit
Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests
processed by the server. Each audit log entry contains two lines:
__Note:__ Legacy Audit is deprecated and is disabled by default since Kubernetes 1.8.
To fallback to this legacy audit, disable the advanced auditing feature
using the `AdvancedAuditing` feature gate in [kube-apiserver][kube-apiserver]:
```
--feature-gates=AdvancedAuditing=false
```
In legacy format, each audit log entry contains two lines:
1. The request line containing a unique ID to match the response and request metadata, such as the source IP, requesting user, impersonation information, resource being requested, etc.
2. The response line containing a unique ID matching the request line and the response code.
@@ -37,14 +270,6 @@ Example output for `admin` user listing pods in the `default` namespace:
2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"
```
Note that Kubernetes 1.8 has switched to use the advanced structured audit log by default.
To fallback to this legacy audit, disable the advanced auditing feature
using the `AdvancedAuditing` feature gate on the [kube-apiserver][kube-apiserver]:
```
--feature-gates=AdvancedAuditing=false
```
### Configuration
[Kube-apiserver][kube-apiserver] provides the following options which are responsible
@@ -64,422 +289,12 @@ Kubernetes may delete old log files when creating a new log file; you can config
how many files are retained and how old they can be by specifying the `audit-log-maxbackup`
and `audit-log-maxage` options.
## Advanced audit
Kubernetes 1.7 expands auditing with experimental functionality such as event
filtering and a webhook for integration with external systems. Kubernetes 1.8
upgrades the advanced audit feature to beta, and some backward incompatible changes
have been committed.
`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded,
and backends, which persist records. Backend implementations include logs files and
webhooks.
The structure of audit events changes when enabling the `AdvancedAuditing` feature
flag. This includes some cleanups, such as the `method` reflecting the verb evaluated
by the [authorization layer](/docs/admin/authorization/) instead of the [HTTP verb](/docs/admin/authorization/#determine-the-request-verb).
Also, instead of always generating two events per request, events are recorded with an associated "stage".
The known stages are:
- `RequestReceived` - The stage for events generated as soon as the audit handler receives the request.
- `ResponseStarted` - Once the response headers are sent, but before the response body is sent. This stage is only generated for long-running requests (e.g. watch).
- `ResponseComplete` - Once the response body has been completed.
- `Panic` - Events generated when a panic occurred.
### Audit Policy
Audit policy is a document defining rules about what events should be recorded.
The policy is passed to the [kube-apiserver][kube-apiserver] using the
`--audit-policy-file` flag.
```
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
```
If `AdvancedAuditing` is enabled and this flag is omitted, no events are logged.
The policy file holds rules that determine the level of an event. Known audit levels are:
- `None` - don't log events that match this rule.
- `Metadata` - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
- `Request` - log event metadata and request body but not response body.
- `RequestResponse` - log event metadata, request and response bodies.
When an event is processed, it's compared against the list of rules in order.
The first matching rule sets the audit level of the event. The audit policy is
defined by the [`audit.k8s.io` API group][audit-api].
Some new fields are supported in beta version, like `resourceNames` and `omitStages`.
In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be provided in
the audit policy file. A policy file with 0 rules, or a policy file that doesn't provide
a valid `apiVersion` and `kind` value will be treated as illgal.
Some example audit policy files:
```yaml
apiVersion: audit.k8s.io/v1beta1 #this is required in Kubernetes 1.8
kind: Policy
rules:
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
```
The next audit policy file shows new features introduced in Kubernetes 1.8:
```yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# Log pod changes at Request level
- level: Request
resources:
- group: ""
# Resource "pods" no longer matches requests to any subresource of pods,
# This behavior is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# A catch-all rule to log all other requests at the Metadata level.
# For this rule we use "omitStages" to omit events at "ReqeustReceived" stage.
# Events in this stage will not be sent to backend.
- level: Metadata
omitStages:
- "RequestReceived"
```
You can use a minimal audit policy file to log all requests at the `Metadata` level:
```yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
```
The [audit profile used by GCE][gce-audit-profile] should be used as reference by
admins constructing their own audit profiles.
### Audit backends
Audit backends implement strategies for emitting events. The [kube-apiserver][kube-apiserver]
provides a logging and webhook backend.
Each request to the API server can generate multiple events, one when the request is received,
another when the response is sent, and additional events for long running requests (such as
watches). The ID of events will be the same if they were generated from the same request.
The event format is defined by the `audit.k8s.io` API group. The `v1alpha1` format of this
API can be found [here][audit-api] with more details about the exact fields captured.
#### Log backend
The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing`
feature flag. All generated events defined by `--audit-policy-file` are recorded in structured
json format:
```
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200}}
```
In alpha version, objectRef.apiVersion holds both the api group and version.
In beta version these were break out into objectRef.apiGroup and objectRef.apiVersion.
Starting from Kubernetes 1.8, structured json format is used for log backend by default.
Use the following option to switch log to legacy format:
```
--audit-log-format=legacy
```
With legacy format, events are formatted as follows:
```
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="RequestReceived" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="<deferred>"
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="ResponseComplete" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="200"
```
Logged events omit the request and response bodies. The `Request` and
`RequestResponse` levels are equivalent to `Metadata` for legacy format. This legacy format
of advanced audit is different from the [Legacy Audit](# Legacy Audit) discussed above, such
as changes to the method values and the introduction of a "stage" for each event.
#### Webhook backend
The audit webhook backend can be used to have [kube-apiserver][kube-apiserver]
send audit events to a remote service. The webhook requires the `AdvancedAuditing`
feature flag and is configured using the following command line flags:
```
--audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
--audit-webhook-mode=batch
```
`audit-webhook-mode` controls buffering strategies used by the webhook. Known modes are:
- `batch` - buffer events and asynchronously send the set of events to the external service.
- `blocking` - block API server responses on sending each event to the external service.
The webhook config file uses the kubeconfig format to specify the remote address of
the service and credentials used to connect to it.
```
# clusters refers to the remote service.
clusters:
- name: name-of-remote-audit-service
cluster:
certificate-authority: /path/to/ca.pem # CA for verifying the remote service.
server: https://audit.example.com/audit # URL of remote service to query. Must use 'https'.
# users refers to the API server's webhook configuration.
users:
- name: name-of-api-server
user:
client-certificate: /path/to/cert.pem # cert for the webhook plugin to use
client-key: /path/to/key.pem # key matching the cert
# kubeconfig files require a context. Provide one for the API server.
current-context: webhook
contexts:
- context:
cluster: name-of-remote-audit-service
user: name-of-api-sever
name: webhook
```
Events are POSTed as a JSON serialized `EventList`. An example payload:
```json
{
"apiVersion": "audit.k8s.io/v1beta1",
"items": [
{
"auditID": "24f30caf-d7d4-45d5-b7bd-e7af300d7886",
"level": "Metadata",
"metadata": {
"creationTimestamp": null
},
"objectRef": {
"apiGroup": "rbac.authorization.k8s.io",
"apiVersion": "v1",
"name": "jane",
"namespace": "default",
"resource": "roles"
},
"requestURI": "/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/jane",
"responseStatus": {
"code": 200,
"metadata": {}
},
"sourceIPs": [
"172.16.116.128"
],
"stage": "ResponseComplete",
"timestamp": "2017-09-05T10:20:24Z",
"user": {
"groups": [
"system:masters",
"system:authenticated"
],
"username": "kubecfg"
},
"verb": "get"
}
],
"kind": "EventList",
"metadata": {}
}
```
### Audit-Id
Audit-Id is a unique ID for each http request to kube-apiserver. The ID of events will be the
same if they were generated from the same request. Starting from Kubernetes 1.8, if an audit
event is generated for the request, kube-apiserver will respond with an Audit-Id in the HTTP header.
Note that for some special requests like `kubectl exec`, `kubectl attach`, kube-apiserver works
like a proxy, no Audit-Id will be returned even if audit events are recorded.
### Log Collector Examples
#### Use fluentd to collect and distribute audit events from log file
[Fluentd][fluentd] is an open source data collector for unified logging layer.
In this example, we will use fluentd to split audit events by different namespaces.
Note that this example requires json format output support in Kubernetes 1.8.
1. install [fluentd, fluent-plugin-forest and fluent-plugin-rewrite-tag-filter][fluentd_install_doc] in the kube-apiserver node
1. create a config file for fluentd
$ cat <<EOF > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
@type tail
# audit log path of kube-apiserver
path /var/log/audit
pos_file /var/log/audit.pos
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%N%z
tag audit
</source>
<filter audit>
#https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
type record_transformer
enable_ruby
<record>
namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ? "none":record["objectRef"]["namespace"])}
</record>
</filter>
<match audit>
# route audit according to namespace element in context
@type rewrite_tag_filter
rewriterule1 namespace ^(.+) ${tag}.$1
</match>
<filter audit.**>
@type record_transformer
remove_keys namespace
</filter>
<match audit.**>
@type forest
subtype file
remove_prefix audit
<template>
time_slice_format %Y%m%d%H
compress gz
path /var/log/audit-${tag}.*.log
format json
include_time_key true
</template>
</match>
1. start fluentd
$ fluentd -c /etc/fluentd/config -vv
1. start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
1. check audits for different namespaces in /var/log/audit-*.log
#### Use logstash to collect and distribute audit events from webhook backend
[Logstash][logstash] is an open source, server-side data processing tool. In this example,
we will use logstash to collect audit events from webhook backend, and save events of
different users into different files.
1. install [logstash][logstash_install_doc]
1. create config file for logstash
$ cat <<EOF > /etc/logstash/config
input{
http{
#TODO, figure out a way to use kubeconfig file to authenticate to logstash
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
port=>8888
}
}
filter{
split{
# Webhook audit backend sends several events together with EventList
# split each event here.
field=>[items]
# We only need event subelement, remove others.
remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
}
mutate{
rename => {items=>event}
}
}
output{
file{
# Audit events from different users will be saved into different files.
path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
}
}
1. start logstash
$ bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/
1. create a [kubeconfig file](/docs/tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/) for kube-apiserver webhook audit backend
$ cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
clusters:
- cluster:
server: http://<ip_of_logstash>:8888
name: logstash
contexts:
- context:
cluster: logstash
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
EOF
1. start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
1. check audits in logstash node's directories /var/log/kube-audit-*/audit
Note that in addition to file output plugin, logstash has a variety of outputs that
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go
[kube-apiserver]: /docs/admin/kube-apiserver
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/cluster/gce/gci/configure-helper.sh#L532
[auditing-proposal]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md
[auditing-level]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md#levels
[auditing-api]: https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh#L532
[kubeconfig]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd
[logstash]: https://www.elastic.co/products/logstash
docs/tasks/debug-application-cluster/debug-application-introspection.md
+3 -24
View File
@@ -16,33 +16,12 @@ your pods. But there are a number of ways to get even more information about you
For this example we'll use a Deployment to create two pods, similar to the earlier example.
```yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
```
{% include code.html language="yaml" file="nginx-dep.yaml" ghlink="/docs/tasks/debug-application-cluster/nginx-dep.yaml" %}
Copy this to a file *./my-nginx-dep.yaml*
Create deployment by running following command:
```shell
$ kubectl create -f ./my-nginx-dep.yaml
$ kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/nginx-dep.yaml
deployment "nginx-deployment" created
```
docs/tasks/debug-application-cluster/debug-service.md
+2 -2
View File
@@ -66,7 +66,7 @@ probably debugging your own `Service` you can substitute your own details, or yo
can follow along and get a second data point.
```shell
$ kubectl run hostnames --image=gcr.io/google_containers/serve_hostname \
$ kubectl run hostnames --image=k8s.gcr.io/serve_hostname \
--labels=app=hostnames \
--port=9376 \
--replicas=3
@@ -93,7 +93,7 @@ spec:
spec:
containers:
- name: hostnames
image: gcr.io/google_containers/serve_hostname
image: k8s.gcr.io/serve_hostname
ports:
- containerPort: 9376
protocol: TCP
docs/tasks/debug-application-cluster/event-exporter-deploy.yaml
+1 -2
View File
@@ -10,7 +10,6 @@ apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: event-exporter-rb
namespace: default
labels:
app: event-exporter
roleRef:
@@ -42,4 +41,4 @@ spec:
image: gcr.io/google-containers/event-exporter:v0.1.0
command:
- '/event-exporter'
terminationGracePeriodSeconds: 30
terminationGracePeriodSeconds: 30
docs/tasks/debug-application-cluster/monitor-node-health.md
+2 -2
View File
@@ -77,7 +77,7 @@ spec:
hostNetwork: true
containers:
- name: node-problem-detector
image: gcr.io/google_containers/node-problem-detector:v0.1
image: k8s.gcr.io/node-problem-detector:v0.1
securityContext:
privileged: true
resources:
@@ -149,7 +149,7 @@ spec:
hostNetwork: true
containers:
- name: node-problem-detector
image: gcr.io/google_containers/node-problem-detector:v0.1
image: k8s.gcr.io/node-problem-detector:v0.1
securityContext:
privileged: true
resources:
docs/tasks/debug-application-cluster/nginx-dep.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
docs/tasks/federation/set-up-cluster-federation-kubefed.md
+23 -32
View File
@@ -371,44 +371,35 @@ For more information see
## Adding a cluster to a federation
Once you've deployed a federation control plane, you'll need to make
that control plane aware of the clusters it should manage. You can add
a cluster to your federation by using the [`kubefed join`](/docs/admin/kubefed_join/)
command. A new context will have been added to your kubeconfig named `fellowship`
(after the name of your federation). To join clusters into the federation, you will
need to change the context.
After you've deployed a federation control plane, you'll need to make that control plane aware of the clusters it should manage.
```
kubectl config use-context fellowship
```
To join clusters into the federation:
To use `kubefed join`, you'll need to provide the name of the cluster
you want to add to the federation, and the `--host-cluster-context`
for the federation control plane's host cluster.
1. Change the context:
> Note: The name that you provide to the `join` command is used as the
joining cluster's identity in federation. This name should adhere to
the rules described in the
[identifiers doc](/docs/concepts/overview/working-with-objects/names/). If the context
corresponding to your joining cluster conforms to these rules then you
can use the same name in the join command. Otherwise, you will have to
choose a different name for your cluster's identity. For more
information, please see the
[naming rules and customization](#naming-rules-and-customization)
section below.
kubectl config use-context fellowship
The following example command adds the cluster `gondor` to the
federation running on host cluster `rivendell`:
1. If you are using a managed cluster service, allow the service to access the cluster. To do this, create a `clusterrolebinding` for the account associated with your cluster service:
```
kubefed join gondor --host-cluster-context=rivendell
```
kubectl create clusterrolebinding <your_user>-cluster-admin-binding --clusterrole=cluster-admin --user=<your_user>@example.org --context=<joining_cluster_context
1. Join the cluster to the federation, using `kubefed join`, and make sure you provide the following:
* The name of the cluster that you are joining to the federation
* `--host-cluster-context`, the kubeconfig context for the host cluster
For example, this command adds the cluster `gondor` to the federation running on host cluster `rivendell`:
```
kubefed join gondor --host-cluster-context=rivendell
```
A new context has now been added to your kubeconfig named `fellowship` (after the name of your federation).
> Note: The name that you provide to the `join` command is used as the joining cluster's identity in federation. If this name adheres to the rules described in the [identifiers doc](/docs/concepts/overview/working-with-objects/names/). If the context
corresponding to your joining cluster conforms to these rules then you can use the same name in the join command. Otherwise, you will have to choose a different name for your cluster's identity.
> Note: Kubernetes requires that you manually join clusters to a
federation because the federation control plane manages only those
clusters that it is responsible for managing. Adding a cluster tells
the federation control plane that it is responsible for managing that
cluster.
### Naming rules and customization
docs/tasks/inject-data-application/dapi-envars-container.yaml
+1 -1
View File
@@ -5,7 +5,7 @@ metadata:
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox:1.24
image: k8s.gcr.io/busybox:1.24
command: [ "sh", "-c"]
args:
- while true; do
docs/tasks/inject-data-application/dapi-envars-pod.yaml
+1 -1
View File
@@ -5,7 +5,7 @@ metadata:
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: [ "sh", "-c"]
args:
- while true; do
docs/tasks/inject-data-application/dapi-volume-resources.yaml
+1 -1
View File
@@ -5,7 +5,7 @@ metadata:
spec:
containers:
- name: client-container
image: gcr.io/google_containers/busybox:1.24
image: k8s.gcr.io/busybox:1.24
command: ["sh", "-c"]
args:
- while true; do
docs/tasks/inject-data-application/dapi-volume.yaml
+1 -1
View File
@@ -12,7 +12,7 @@ metadata:
spec:
containers:
- name: client-container
image: gcr.io/google_containers/busybox
image: k8s.gcr.io/busybox
command: ["sh", "-c"]
args:
- while true; do
docs/tasks/inject-data-application/distribute-credentials-secure.md
+3 -3
View File
@@ -37,7 +37,7 @@ username and password:
1. Create the Secret
kubectl create -f secret.yaml
kubectl create -f https://k8s.io/docs/tasks/inject-data-application/secret.yaml
**Note:** If you want to skip the Base64 encoding step, you can create a Secret
by using the `kubectl create secret` command:
@@ -81,7 +81,7 @@ Here is a configuration file you can use to create a Pod:
1. Create the Pod:
kubectl create -f secret-pod.yaml
kubectl create -f https://k8s.io/docs/tasks/inject-data-application/secret-pod.yaml
1. Verify that your Pod is running:
@@ -128,7 +128,7 @@ Here is a configuration file you can use to create a Pod:
1. Create the Pod:
kubectl create -f secret-envars-pod.yaml
kubectl create -f https://k8s.io/docs/tasks/inject-data-application/secret-envars-pod.yaml
1. Verify that your Pod is running:
docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.md
+6 -6
View File
@@ -40,7 +40,7 @@ In the configuration file, you can see that the Pod has a `downwardAPI` Volume,
and the Container mounts the Volume at `/etc`.
Look at the `items` array under `downwardAPI`. Each element of the array is a
[DownwardAPIVolumeFile](/docs/resources-reference/{{page.version}}/#downwardapivolumefile-v1-core).
[DownwardAPIVolumeFile](/docs/api-reference/{{page.version}}/#downwardapivolumefile-v1-core).
The first element specifies that the value of the Pod's
`metadata.labels` field should be stored in a file named `labels`.
The second element specifies that the value of the Pod's `annotations`
@@ -234,11 +234,11 @@ inject the Pod's name into the well-known environment variable.
{% capture whatsnext %}
* [PodSpec](/docs/resources-reference/{{page.version}}/#podspec-v1-core)
* [Volume](/docs/resources-reference/{{page.version}}/#volume-v1-core)
* [DownwardAPIVolumeSource](/docs/resources-reference/{{page.version}}/#downwardapivolumesource-v1-core)
* [DownwardAPIVolumeFile](/docs/resources-reference/{{page.version}}/#downwardapivolumefile-v1-core)
* [ResourceFieldSelector](/docs/resources-reference/{{page.version}}/#resourcefieldselector-v1-core)
* [PodSpec](/docs/api-reference/{{page.version}}/#podspec-v1-core)
* [Volume](/docs/api-reference/{{page.version}}/#volume-v1-core)
* [DownwardAPIVolumeSource](/docs/api-reference/{{page.version}}/#downwardapivolumesource-v1-core)
* [DownwardAPIVolumeFile](/docs/api-reference/{{page.version}}/#downwardapivolumefile-v1-core)
* [ResourceFieldSelector](/docs/api-reference/{{page.version}}/#resourcefieldselector-v1-core)
{% endcapture %}
docs/tasks/inject-data-application/environment-variable-expose-pod-information.md
+9 -9
View File
@@ -25,7 +25,7 @@ Pod fields and Container fields.
There are two ways to expose Pod and Container fields to a running Container:
* Environment variables
* [DownwardAPIVolumeFiles](/docs/resources-reference/{{page.version}}/#downwardapivolumefile-v1-core)
* [DownwardAPIVolumeFiles](/docs/api-reference/{{page.version}}/#downwardapivolumefile-v1-core)
Together, these two ways of exposing Pod and Container fields are called the
*Downward API*.
@@ -40,7 +40,7 @@ configuration file for the Pod:
In the configuration file, you can see five environment variables. The `env`
field is an array of
[EnvVars](/docs/resources-reference/{{page.version}}/#envvar-v1-core).
[EnvVars](/docs/api-reference/{{page.version}}/#envvar-v1-core).
The first element in the array specifies that the `MY_NODE_NAME` environment
variable gets its value from the Pod's `spec.nodeName` field. Similarly, the
other environment variables get their names from Pod fields.
@@ -118,7 +118,7 @@ container:
In the configuration file, you can see four environment variables. The `env`
field is an array of
[EnvVars](/docs/resources-reference/{{page.version}}/#envvar-v1-core).
[EnvVars](/docs/api-reference/{{page.version}}/#envvar-v1-core).
The first element in the array specifies that the `MY_CPU_REQUEST` environment
variable gets its value from the `requests.cpu` field of a Container named
`test-container`. Similarly, the other environment variables get their values
@@ -156,12 +156,12 @@ The output shows the values of selected environment variables:
{% capture whatsnext %}
* [Defining Environment Variables for a Container](/docs/tasks/inject-data-application/define-environment-variable-container/)
* [PodSpec](/docs/resources-reference/{{page.version}}/#podspec-v1-core)
* [Container](/docs/resources-reference/{{page.version}}/#container-v1-core)
* [EnvVar](/docs/resources-reference/{{page.version}}/#envvar-v1-core)
* [EnvVarSource](/docs/resources-reference/{{page.version}}/#envvarsource-v1-core)
* [ObjectFieldSelector](/docs/resources-reference/{{page.version}}/#objectfieldselector-v1-core)
* [ResourceFieldSelector](/docs/resources-reference/{{page.version}}/#resourcefieldselector-v1-core)
* [PodSpec](/docs/api-reference/{{page.version}}/#podspec-v1-core)
* [Container](/docs/api-reference/{{page.version}}/#container-v1-core)
* [EnvVar](/docs/api-reference/{{page.version}}/#envvar-v1-core)
* [EnvVarSource](/docs/api-reference/{{page.version}}/#envvarsource-v1-core)
* [ObjectFieldSelector](/docs/api-reference/{{page.version}}/#objectfieldselector-v1-core)
* [ResourceFieldSelector](/docs/api-reference/{{page.version}}/#resourcefieldselector-v1-core)
{% endcapture %}
docs/tasks/inject-data-application/podpreset-allow-db-merged.yaml
+2 -2
View File
@@ -10,7 +10,7 @@ metadata:
spec:
containers:
- name: website
image: ecorp/website
image: nginx
volumeMounts:
- mountPath: /cache
name: cache-volume
@@ -28,7 +28,7 @@ spec:
value: $(REPLACE_ME)
envFrom:
- configMapRef:
name: etcd-env-config
name: etcd-env-config
volumes:
- name: cache-volume
emptyDir: {}
docs/tasks/inject-data-application/podpreset-allow-db.yaml
+1 -2
View File
@@ -2,14 +2,13 @@ apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
role: frontend
env:
- name: DB_PORT
value: 6379
value: "6379"
- name: duplicate_key
value: FROM_ENV
- name: expansion
docs/tasks/inject-data-application/podpreset-conflict-pod.yaml
+2 -2
View File
@@ -8,12 +8,12 @@ metadata:
spec:
containers:
- name: website
image: ecorp/website
image: nginx
volumeMounts:
- mountPath: /cache
name: cache-volume
ports:
- containerPort: 80
volumes:
- name: cache-volume
emptyDir: {}
- containerPort: 80
docs/tasks/inject-data-application/podpreset-conflict-preset.yaml
-1
View File
@@ -2,7 +2,6 @@ apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
docs/tasks/inject-data-application/podpreset-merged.yaml
+1 -1
View File
@@ -10,7 +10,7 @@ metadata:
spec:
containers:
- name: website
image: ecorp/website
image: nginx
volumeMounts:
- mountPath: /cache
name: cache-volume
docs/tasks/inject-data-application/podpreset-multi-merged.yaml
+1 -1
View File
@@ -11,7 +11,7 @@ metadata:
spec:
containers:
- name: website
image: ecorp/website
image: nginx
volumeMounts:
- mountPath: /cache
name: cache-volume
docs/tasks/inject-data-application/podpreset-pod.yaml
+1 -1
View File
@@ -8,7 +8,7 @@ metadata:
spec:
containers:
- name: website
image: ecorp/website
image: nginx
ports:
- containerPort: 80
docs/tasks/inject-data-application/podpreset-preset.yaml
-1
View File
@@ -2,7 +2,6 @@ apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
name: allow-database
namespace: myns
spec:
selector:
matchLabels:
docs/tasks/inject-data-application/podpreset-proxy.yaml
-1
View File
@@ -2,7 +2,6 @@ apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
name: proxy
namespace: myns
spec:
selector:
matchLabels:
docs/tasks/inject-data-application/podpreset-replicaset-merged.yaml
+1
View File
@@ -1,6 +1,7 @@
apiVersion: v1
kind: Pod
metadata:
name: frontend
labels:
app: guestbook
tier: frontend
docs/tasks/inject-data-application/podpreset.md
+36 -4
View File
@@ -20,18 +20,50 @@ You can get an overview of PodPresets at
This is a simple example to show how a Pod spec is modified by the Pod
Preset.
**User submitted pod spec:**
{% include code.html language="yaml" file="podpreset-preset.yaml" ghlink="/docs/tasks/inject-data-application/podpreset-preset.yaml" %}
Create the PodPreset:
```shell
kubectl create -f https://k8s.io/docs/tasks/inject-data-application/podpreset-preset.yaml
```
Examine the created PodPreset:
```shell
$ kubectl get podpreset
NAME AGE
allow-database 1m
```
The new PodPreset will act upon any pod that has label `role: frontend`.
{% include code.html language="yaml" file="podpreset-pod.yaml" ghlink="/docs/tasks/inject-data-application/podpreset-pod.yaml" %}
**Example Pod Preset:**
Create a pod:
{% include code.html language="yaml" file="podpreset-preset.yaml" ghlink="/docs/tasks/inject-data-application/podpreset-preset.yaml" %}
```shell
$ kubectl create -f https://k8s.io/docs/tasks/inject-data-application/podpreset-pod.yaml
```
List the running Pods:
```shell
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
website 1/1 Running 0 4m
```
**Pod spec after admission controller:**
{% include code.html language="yaml" file="podpreset-merged.yaml" ghlink="/docs/tasks/inject-data-application/podpreset-merged.yaml" %}
To see above output, run the following command:
```shell
$ kubectl get pod website -o yaml
```
### Pod Spec with `ConfigMap` Example
This is an example to show how a Pod spec is modified by the Pod Preset
@@ -113,7 +145,7 @@ when there is a conflict.
**If we run `kubectl describe...` we can see the event:**
```
```shell
$ kubectl describe ...
....
Events:
docs/tasks/job/parallel-processing-expansion.md
+5 -5
View File
@@ -42,7 +42,7 @@ Next, expand the template into multiple files, one for each item to be processed
$ mkdir ./jobs
$ for i in apple banana cherry
do
cat job.yaml.txt | sed "s/\$ITEM/$i/" > ./jobs/job-$i.yaml
cat job.yaml | sed "s/\$ITEM/$i/" > ./jobs/job-$i.yaml
done
```
@@ -72,10 +72,10 @@ Now, check on the jobs:
```shell
$ kubectl get jobs -l jobgroup=jobexample
JOB CONTAINER(S) IMAGE(S) SELECTOR SUCCESSFUL
process-item-apple c busybox app in (jobexample),item in (apple) 1
process-item-banana c busybox app in (jobexample),item in (banana) 1
process-item-cherry c busybox app in (jobexample),item in (cherry) 1
NAME DESIRED SUCCESSFUL AGE
process-item-apple 1 1 31s
process-item-banana 1 1 31s
process-item-cherry 1 1 31s
```
Here we use the `-l` option to select all jobs that are part of this
docs/tasks/manage-gpus/scheduling-gpus.md
+3 -3
View File
@@ -36,13 +36,13 @@ spec:
containers:
-
name: gpu-container-1
image: gcr.io/google_containers/pause:2.0
image: k8s.gcr.io/pause:2.0
resources:
limits:
alpha.kubernetes.io/nvidia-gpu: 2 # requesting 2 GPUs
-
name: gpu-container-2
image: gcr.io/google_containers/pause:2.0
image: k8s.gcr.io/pause:2.0
resources:
limits:
alpha.kubernetes.io/nvidia-gpu: 3 # requesting 3 GPUs
@@ -126,7 +126,7 @@ metadata:
spec:
containers:
- name: gpu-container-1
image: gcr.io/google_containers/pause:2.0
image: k8s.gcr.io/pause:2.0
resources:
limits:
alpha.kubernetes.io/nvidia-gpu: 1
docs/tasks/manage-hugepages/scheduling-hugepages.md
+1 -1
View File
@@ -19,7 +19,7 @@ can consume huge pages and the current limitations.
its huge page capacity. A node may only pre-allocate huge pages for a single
size.
1. A special **alpha** feature gate `HugePages` has to be set to true across the
system: `--feature-gates="HugePages=true"`.
system: `--feature-gates=HugePages=true`.
The nodes will automatically discover and report all huge page resources as a
schedulable resource.
docs/tasks/run-application/configure-pdb.md
+5
View File
@@ -88,6 +88,11 @@ of the evicted pod. `minAvailable` can be either an absolute number or a percent
of the number of pods from that set that can be unavailable after the eviction.
It can be either an absolute number or a percentage.
**Note:** For versions 1.8 and earlier: When creating a `PodDisruptionBudget`
object using the `kubectl` command line tool, the `minAvailable` field has a
default value of 1 if neither `minAvailable` nor `maxAvailable` is specified.
{: .note}
You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`.
`maxUnavailable` can only be used to control the eviction of pods
that have an associated controller managing them. In the examples below, "desired replicas"
docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md
+10 -23
View File
@@ -4,14 +4,14 @@ approvers:
- jszczepkowski
- justinsb
- directxman12
title: Horizontal Pod Autoscaling Walkthrough
title: Horizontal Pod Autoscaler Walkthrough
---
Horizontal Pod Autoscaling automatically scales the number of pods
Horizontal Pod Autoscaler automatically scales the number of pods
in a replication controller, deployment or replica set based on observed CPU utilization
(or, with beta support, on some other, application-provided metrics).
This document walks you through an example of enabling Horizontal Pod Autoscaling for the php-apache server. For more information on how Horizontal Pod Autoscaling behaves, see the [Horizontal Pod Autoscaling user guide](/docs/tasks/run-application/horizontal-pod-autoscale/).
This document walks you through an example of enabling Horizontal Pod Autoscaler for the php-apache server. For more information on how Horizontal Pod Autoscaler behaves, see the [Horizontal Pod Autoscaler user guide](/docs/tasks/run-application/horizontal-pod-autoscale/).
## Prerequisites
@@ -24,7 +24,7 @@ heapster monitoring will be turned-on by default).
To specify multiple resource metrics for a Horizontal Pod Autoscaler, you must have a Kubernetes cluster
and kubectl at version 1.6 or later. Furthermore, in order to make use of custom metrics, your cluster
must be able to communicate with the API server providing the custom metrics API.
See the [Horizontal Pod Autoscaling user guide](/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics) for more details.
See the [Horizontal Pod Autoscaler user guide](/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics) for more details.
## Step One: Run & expose php-apache server
@@ -35,7 +35,7 @@ It defines an [index.php](/docs/user-guide/horizontal-pod-autoscaling/image/inde
First, we will start a deployment running the image and expose it as a service:
```shell
$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
$ kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80
service "php-apache" created
deployment "php-apache" created
```
@@ -311,29 +311,16 @@ HorizontalPodAutoscaler.
## Appendix: Other possible scenarios
### Creating the autoscaler from a .yaml file
### Creating the autoscaler declaratively
Instead of using `kubectl autoscale` command we can use the [hpa-php-apache.yaml](/docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml) file, which looks like this:
Instead of using `kubectl autoscale` command to create a HorizontalPodAutoscaler imperatively we
can use the following file to create it declaratively:
```yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
```
{% include code.html language="yaml" file="hpa-php-apache.yaml" ghlink="/docs/tasks/run-application/hpa-php-apache.yaml" %}
We will create the autoscaler by executing the following command:
```shell
$ kubectl create -f docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml
$ kubectl create -f https://k8s.io/docs/tasks/run-application/hpa-php-apache.yaml
horizontalpodautoscaler "php-apache" created
```
docs/tasks/run-application/horizontal-pod-autoscale.md
+5 -5
View File
@@ -3,18 +3,18 @@ approvers:
- fgrzadkowski
- jszczepkowski
- directxman12
title: Horizontal Pod Autoscaling
title: Horizontal Pod Autoscaler
---
This document describes the current state of Horizontal Pod Autoscaling in Kubernetes.
This document describes the current state of the Horizontal Pod Autoscaler in Kubernetes.
## What is Horizontal Pod Autoscaling?
## What is the Horizontal Pod Autoscaler?
With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of pods
The Horizontal Pod Autoscaler automatically scales the number of pods
in a replication controller, deployment or replica set based on observed CPU utilization (or, with
[custom metrics](https://git.k8s.io/community/contributors/design-proposals/instrumentation/custom-metrics-api.md)
support, on some other application-provided metrics). Note that Horizontal
Pod Autoscaling does not apply to objects that can't be scaled, for example, DaemonSet.
Pod Autoscaling does not apply to objects that can't be scaled, for example, DaemonSets.
The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller.
The resource determines the behavior of the controller.
docs/tasks/run-application/hpa-php-apache.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
docs/tasks/run-application/mysql-statefulset.yaml
+1 -1
View File
@@ -1,4 +1,4 @@
apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
docs/tasks/service-catalog/install-service-catalog-using-helm.md
+2 -2
View File
@@ -5,7 +5,7 @@ approvers:
---
{% capture overview %}
{% glossary_definition term_id="service-catalog" length="long" %}
{% glossary_definition term_id="service-catalog" length="all" prepend="Service Catalog is" %}
Use [Helm](https://helm.sh/) to install Service Catalog on your Kubernetes cluster. Up to date information on this process can be found at the [kubernetes-incubator/service-catalog](https://github.com/kubernetes-incubator/service-catalog/blob/master/docs/install.md) repo.
@@ -97,4 +97,4 @@ helm install svc-cat/catalog \
{% endcapture %}
{% include templates/task.md %}
{% include templates/task.md %}
docs/tasks/service-catalog/install-service-catalog-using-sc.md
+5 -5
View File
@@ -5,7 +5,7 @@ approvers:
---
{% capture overview %}
{% glossary_definition term_id="service-catalog" length="long" %}
{% glossary_definition term_id="service-catalog" length="all" prepend="Service Catalog is" %}
Use the [Service Catalog Installer](https://github.com/GoogleCloudPlatform/k8s-service-catalog#installation) tool to easily install or uninstall Service Catalog on your Kubernetes cluster. This CLI tool is installed as `sc` in your local environment.
@@ -15,7 +15,7 @@ Use the [Service Catalog Installer](https://github.com/GoogleCloudPlatform/k8s-s
{% capture prerequisites %}
* Understand the key concepts of [Service Catalog](/docs/concepts/service-catalog/).
* Install [Go 1.6+](https://golang.org/dl/) and set the `GOPATH`.
* Install the [cfssl](https://github.com/cloudflare/cfssl) tool needed for generating SSL artifacts.
* Install the [cfssl](https://github.com/cloudflare/cfssl) tool needed for generating SSL artifacts.
* Service Catalog requires Kubernetes version 1.7+.
* [Install and setup kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) so that it is configured to connect to a Kubernetes v1.7+ cluster.
* The kubectl user must be bound to the *cluster-admin* role for it to install Service Catalog. To ensure that this is true, run the following command:
@@ -44,11 +44,11 @@ First, verify that all dependencies have been installed. Run:
sc check
```
If the check is successful, it should return:
If the check is successful, it should return:
```
Dependency check passed. You are good to go.
```
```
Next, run the install command and specify the `storageclass` that you want to use for the backup:
@@ -74,4 +74,4 @@ sc uninstall
{% endcapture %}
{% include templates/task.md %}
{% include templates/task.md %}