whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Convert site to Hugo (#8316)

Browse Source 
This commit converts content and layout to use Hugo.
This commit is contained in:
Bjørn Erik Pedersen
2018-05-05 18:00:51 +02:00
committed by k8s-ci-robot
parent 7745f0e0c5
commit 7f3b633aa0
2327 changed files with 10928 additions and 171503 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/tasks/administer-cluster/_index.md
Executable
+5
View File
@@ -0,0 +1,5 @@
---
title: "Manage Memory, CPU, and API Resources"
weight: 20
---
content/en/docs/tasks/administer-cluster/access-cluster-api.md
+215
View File
@@ -0,0 +1,215 @@
---
title: Access Clusters Using the Kubernetes API
content_template: templates/task
---
{{% capture overview %}}
This page shows how to access clusters using the Kubernetes API.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Accessing the cluster API
### Accessing for the first time with kubectl
When accessing the Kubernetes API for the first time, use the
Kubernetes command-line tool, `kubectl`.
To access a cluster, you need to know the location of the cluster and have credentials
to access it. Typically, this is automatically set-up when you work through
a [Getting started guide](/docs/setup/),
or someone else setup the cluster and provided you with credentials and a location.
Check the location and credentials that kubectl knows about with this command:
```shell
$ kubectl config view
```
Many of the [examples](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/) provide an introduction to using
kubectl. Complete documentation is found in the [kubectl manual](/docs/reference/kubectl/overview/).
### Directly accessing the REST API
kubectl handles locating and authenticating to the API server. If you want to directly access the REST API with an http client like
`curl` or `wget`, or a browser, there are multiple ways you can locate and authenticate against the API server:
1. Run kubectl in proxy mode (recommended). This method is recommended, since it uses the stored apiserver location and verifies the identity of the API server using a self-signed cert. No man-in-the-middle (MITM) attack is possible using this method.
1. Alternatively, you can provide the location and credentials directly to the http client. This works with client code that is confused by proxies. To protect against man in the middle attacks, you'll need to import a root cert into your browser.
Using the Go or Python client libraries provides accessing kubectl in proxy mode.
#### Using kubectl proxy
The following command runs kubectl in a mode where it acts as a reverse proxy. It handles
locating the API server and authenticating.
Run it like this:
```shell
$ kubectl proxy --port=8080 &
```
See [kubectl proxy](/docs/reference/generated/kubectl/kubectl-commands/#proxy) for more details.
Then you can explore the API with curl, wget, or a browser, like so:
```shell
$ curl http://localhost:8080/api/
{
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.0.1.149:443"
}
]
}
```
#### Without kubectl proxy
It is possible to avoid using kubectl proxy by passing an authentication token
directly to the API server, like this:
``` shell
$ APISERVER=$(kubectl config view | grep server | cut -f 2- -d ":" | tr -d " ")
$ TOKEN=$(kubectl describe secret $(kubectl get secrets | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d '\t')
$ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.0.1.149:443"
}
]
}
```
The above example uses the `--insecure` flag. This leaves it subject to MITM
attacks. When kubectl accesses the cluster it uses a stored root certificate
and client certificates to access the server. (These are installed in the
`~/.kube` directory). Since cluster certificates are typically self-signed, it
may take special configuration to get your http client to use root
certificate.
On some clusters, the API server does not require authentication; it may serve
on localhost, or be protected by a firewall. There is not a standard
for this. [Configuring Access to the API](/docs/admin/accessing-the-api)
describes how a cluster admin can configure this. Such approaches may conflict
with future high-availability support.
### Programmatic access to the API
Kubernetes officially supports client libraries for [Go](#go-client) and
[Python](#python-client).
#### Go client
* To get the library, run the following command: `go get k8s.io/client-go/<version number>/kubernetes` See [https://github.com/kubernetes/client-go](https://github.com/kubernetes/client-go) to see which versions are supported.
* Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g., `import "k8s.io/client-go/1.4/pkg/api/v1"` is correct.
The Go client can use the same [kubeconfig file](/docs/concepts/cluster-administration/authenticate-across-clusters-kubeconfig/)
as the kubectl CLI does to locate and authenticate to the API server. See this [example](https://git.k8s.io/client-go/examples/out-of-cluster-client-configuration/main.go):
```golang
import (
"fmt"
"k8s.io/client-go/1.4/kubernetes"
"k8s.io/client-go/1.4/pkg/api/v1"
"k8s.io/client-go/1.4/tools/clientcmd"
)
...
// uses the current context in kubeconfig
config, _ := clientcmd.BuildConfigFromFlags("", "path to kubeconfig")
// creates the clientset
clientset, _:= kubernetes.NewForConfig(config)
// access the API to list pods
pods, _:= clientset.CoreV1().Pods("").List(v1.ListOptions{})
fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
...
```
If the application is deployed as a Pod in the cluster, please refer to the [next section](#accessing-the-api-from-a-pod).
#### Python client
To use [Python client](https://github.com/kubernetes-client/python), run the following command: `pip install kubernetes` See [Python Client Library page](https://github.com/kubernetes-client/python) for more installation options.
The Python client can use the same [kubeconfig file](/docs/concepts/cluster-administration/authenticate-across-clusters-kubeconfig/)
as the kubectl CLI does to locate and authenticate to the API server. See this [example](https://github.com/kubernetes-client/python/tree/master/examples/example1.py):
```python
from kubernetes import client, config
config.load_kube_config()
v1=client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
```
#### Other languages
There are [client libraries](/docs/reference/client-libraries/) for accessing the API from other languages. See documentation for other libraries for how they authenticate.
### Accessing the API from a Pod
When accessing the API from a Pod, locating and authenticating
to the API server are somewhat different.
The easiest way to use the Kubernetes API from a Pod is to use
one of the official [client libraries](/docs/reference/client-libraries/). These
libraries can automatically discover the API server and authenticate.
While running in a Pod, the Kubernetes apiserver is accessible via a Service named
`kubernetes` in the `default` namespace. Therefore, Pods can use the
`kubernetes.default.svc` hostname to query the API server. Official client libraries
do this automatically.
From within a Pod, the recommended way to authenticate to the API server is with a
[service account](/docs/user-guide/service-accounts) credential. By default, a Pod
is associated with a service account, and a credential (token) for that
service account is placed into the filesystem tree of each container in that Pod,
at `/var/run/secrets/kubernetes.io/serviceaccount/token`.
If available, a certificate bundle is placed into the filesystem tree of each
container at `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`, and should be
used to verify the serving certificate of the API server.
Finally, the default namespace to be used for namespaced API operations is placed in a file
at `/var/run/secrets/kubernetes.io/serviceaccount/namespace` in each container.
From within a Pod, the recommended ways to connect to the Kubernetes API are:
- Use one of the official [client libraries](/docs/reference/client-libraries/)
as they handle API host discovery and authentication automatically.
For Go client, the `rest.InClusterConfig()` function assists with this.
See [an example here](https://git.k8s.io/client-go/examples/in-cluster-client-configuration/main.go).
- If you would like to query the API without an official client library, you can run `kubectl proxy`
as the [command](/docs/tasks/inject-data-application/define-command-argument-container/)
of a new sidecar container in the Pod. This way, `kubectl proxy` will authenticate
to the API and expose it on the `localhost` interface of the Pod, so that other containers
in the Pod can use it directly.
In each case, the service account credentials of the Pod are used to communicate
securely with the API server.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/access-cluster-services.md
+113
View File
@@ -0,0 +1,113 @@
---
title: Access Services Running on Clusters
content_template: templates/task
---
{{% capture overview %}}
This page shows how to connect to services running on the Kubernetes cluster.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Accessing services running on the cluster
In Kubernetes, [nodes](/docs/admin/node), [pods](/docs/user-guide/pods) and [services](/docs/user-guide/services) all have
their own IPs. In many cases, the node IPs, pod IPs, and some service IPs on a cluster will not be
routable, so they will not be reachable from a machine outside the cluster,
such as your desktop machine.
### Ways to connect
You have several options for connecting to nodes, pods and services from outside the cluster:
- Access services through public IPs.
- Use a service with type `NodePort` or `LoadBalancer` to make the service reachable outside
the cluster. See the [services](/docs/user-guide/services) and
[kubectl expose](/docs/reference/generated/kubectl/kubectl-commands/#expose) documentation.
- Depending on your cluster environment, this may just expose the service to your corporate network,
or it may expose it to the internet. Think about whether the service being exposed is secure.
Does it do its own authentication?
- Place pods behind services. To access one specific pod from a set of replicas, such as for debugging,
place a unique label on the pod and create a new service which selects this label.
- In most cases, it should not be necessary for application developer to directly access
nodes via their nodeIPs.
- Access services, nodes, or pods using the Proxy Verb.
- Does apiserver authentication and authorization prior to accessing the remote service.
Use this if the services are not secure enough to expose to the internet, or to gain
access to ports on the node IP, or for debugging.
- Proxies may cause problems for some web applications.
- Only works for HTTP/HTTPS.
- Described [here](#manually-constructing-apiserver-proxy-urls).
- Access from a node or pod in the cluster.
- Run a pod, and then connect to a shell in it using [kubectl exec](/docs/reference/generated/kubectl/kubectl-commands/#exec).
Connect to other nodes, pods, and services from that shell.
- Some clusters may allow you to ssh to a node in the cluster. From there you may be able to
access cluster services. This is a non-standard method, and will work on some clusters but
not others. Browsers and other tools may or may not be installed. Cluster DNS may not work.
### Discovering builtin services
Typically, there are several services which are started on a cluster by kube-system. Get a list of these
with the `kubectl cluster-info` command:
```shell
$ kubectl cluster-info
Kubernetes master is running at https://104.197.5.247
elasticsearch-logging is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy
kibana-logging is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/kibana-logging/proxy
kube-dns is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/kube-dns/proxy
grafana is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
heapster is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/monitoring-heapster/proxy
```
This shows the proxy-verb URL for accessing each service.
For example, this cluster has cluster-level logging enabled (using Elasticsearch), which can be reached
at `https://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/` if suitable credentials are passed, or through a kubectl proxy at, for example:
`http://localhost:8080/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/`.
(See [Access Clusters Using the Kubernetes API](/docs/tasks/administer-cluster/access-cluster-api/#accessing-the-cluster-api) for how to pass credentials or use kubectl proxy.)
#### Manually constructing apiserver proxy URLs
As mentioned above, you use the `kubectl cluster-info` command to retrieve the service's proxy URL. To create proxy URLs that include service endpoints, suffixes, and parameters, you simply append to the service's proxy URL:
`http://`*`kubernetes_master_address`*`/api/v1/namespaces/`*`namespace_name`*`/services/`*`service_name[:port_name]`*`/proxy`
If you haven't specified a name for your port, you don't have to specify *port_name* in the URL
##### Examples
* To access the Elasticsearch service endpoint `_search?q=user:kimchy`, you would use: `http://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_search?q=user:kimchy`
* To access the Elasticsearch cluster health information `_cluster/health?pretty=true`, you would use: `https://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_cluster/health?pretty=true`
```json
{
"cluster_name" : "kubernetes_logging",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5
}
```
#### Using web browsers to access services running on the cluster
You may be able to put an apiserver proxy URL into the address bar of a browser. However:
- Web browsers cannot usually pass tokens, so you may need to use basic (password) auth. Apiserver can be configured to accept basic auth,
but your cluster may not be configured to accept basic auth.
- Some web apps may not work, particularly those with client side javascript that construct URLs in a
way that is unaware of the proxy path prefix.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/busybox.yaml
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
content/en/docs/tasks/administer-cluster/calico-network-policy.md
+52
View File
@@ -0,0 +1,52 @@
---
reviewers:
- caseydavenport
title: Use Calico for NetworkPolicy
content_template: templates/task
---
{{% capture overview %}}
This page shows a couple of quick ways to create a Calico cluster on Kubernetes.
{{% /capture %}}
{{% capture prerequisites %}}
Decide whether you want to deploy a [cloud](#creating-a-calico-cluster-with-google-kubernetes-engine-gke) or [local](#creating-a-local-calico-cluster-with-kubeadm) cluster.
{{% /capture %}}
{{% capture steps %}}
## Creating a Calico cluster with Google Kubernetes Engine (GKE)
**Prerequisite**: [gcloud](https://cloud.google.com/sdk/docs/quickstarts).
1. To launch a GKE cluster with Calico, just include the `--enable-network-policy` flag.
**Syntax**
```shell
gcloud container clusters create [CLUSTER_NAME] --enable-network-policy
```
**Example**
```shell
gcloud container clusters create my-calico-cluster --enable-network-policy
```
1. To verify the deployment, use the following command.
```shell
kubectl get pods --namespace=kube-system
```
The Calico pods begin with `calico`. Check to make sure each one has a status of `Running`.
## Creating a local Calico cluster with kubeadm
To get a local single-host Calico cluster in fifteen minutes using kubeadm, refer to the
[Calico Quickstart](https://docs.projectcalico.org/latest/getting-started/kubernetes/).
{{% /capture %}}
{{% capture whatsnext %}}
Once your cluster is running, you can follow the [Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/) to try out Kubernetes NetworkPolicy.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/change-default-storage-class.md
+91
View File
@@ -0,0 +1,91 @@
---
title: Change the default StorageClass
content_template: templates/task
---
{{% capture overview %}}
This page shows how to change the default Storage Class that is used to
provision volumes for PersistentVolumeClaims that have no special requirements.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Why change the default storage class?
Depending on the installation method, your Kubernetes cluster may be deployed with
an existing StorageClass that is marked as default. This default StorageClass
is then used to dynamically provision storage for PersistentVolumeClaims
that do not require any specific storage class. See
[PersistentVolumeClaim documentation](/docs/concepts/storage/persistent-volumes/#class-1)
for details.
The pre-installed default StorageClass may not fit well with your expected workload;
for example, it might provision storage that is too expensive. If this is the case,
you can either change the default StorageClass or disable it completely to avoid
dynamic provisioning of storage.
Simply deleting the default StorageClass may not work, as it may be re-created
automatically by the addon manager running in your cluster. Please consult the docs for your installation
for details about addon manager and how to disable individual addons.
## Changing the default StorageClass
1. List the StorageClasses in your cluster:
kubectl get storageclass
The output is similar to this:
NAME TYPE
standard (default) kubernetes.io/gce-pd
gold kubernetes.io/gce-pd
The default StorageClass is marked by `(default)`.
1. Mark the default StorageClass as non-default:
The default StorageClass has an annotation
`storageclass.kubernetes.io/is-default-class` set to `true`. Any other value
or absence of the annotation is interpreted as `false`.
To mark a StorageClass as non-default, you need to change its value to `false`:
kubectl patch storageclass <your-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
where `<your-class-name>` is the name of your chosen StorageClass.
1. Mark a StorageClass as default:
Similarly to the previous step, you need to add/set the annotation
`storageclass.kubernetes.io/is-default-class=true`.
kubectl patch storageclass <your-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Please note that at most one StorageClass can be marked as default. If two
or more of them are marked as default, Kubernetes ignores the annotation,
i.e. it behaves as if there is no default StorageClass.
1. Verify that your chosen StorageClass is default:
kubectl get storageclass
The output is similar to this:
NAME TYPE
standard kubernetes.io/gce-pd
gold (default) kubernetes.io/gce-pd
{{% /capture %}}
{{% capture whatsnext %}}
* Learn more about [StorageClasses](/docs/concepts/storage/persistent-volumes/).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/change-pv-reclaim-policy.md
+81
View File
@@ -0,0 +1,81 @@
---
title: Change the Reclaim Policy of a PersistentVolume
content_template: templates/task
---
{{% capture overview %}}
This page shows how to change the reclaim policy of a Kubernetes
PersistentVolume.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Why change reclaim policy of a PersistentVolume
`PersistentVolumes` can have various reclaim policies, including "Retain",
"Recycle", and "Delete". For dynamically provisioned `PersistentVolumes`,
the default reclaim policy is "Delete". This means that a dynamically provisioned
volume is automatically deleted when a user deletes the corresponding
`PersistentVolumeClaim`. This automatic behavior might be inappropriate if the volume
contains precious data. In that case, it is more appropriate to use the "Retain"
policy. With the "Retain" policy, if a user deletes a `PersistentVolumeClaim`,
the corresponding `PersistentVolume` is not be deleted. Instead, it is moved to the
`Released` phase, where all of its data can be manually recovered.
## Changing the reclaim policy of a PersistentVolume
1. List the PersistentVolumes in your cluster:
kubectl get pv
The output is similar to this:
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94 4Gi RWO Delete Bound default/claim1 10s
pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94 4Gi RWO Delete Bound default/claim2 6s
pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94 4Gi RWO Delete Bound default/claim3 3s
This list also includes the name of the claims that are bound to each volume
for easier identification of dynamically provisioned volumes.
1. Choose one of your PersistentVolumes and change its reclaim policy:
kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
where `<your-pv-name>` is the name of your chosen PersistentVolume.
1. Verify that your chosen PersistentVolume has the right policy:
kubectl get pv
The output is similar to this:
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94 4Gi RWO Delete Bound default/claim1 40s
pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94 4Gi RWO Delete Bound default/claim2 36s
pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94 4Gi RWO Retain Bound default/claim3 33s
In the preceding output, you can see that the volume bound to claim
`default/claim3` has reclaim policy `Retain`. It will not be automatically
deleted when a user deletes claim `default/claim3`.
{{% /capture %}}
{{% capture whatsnext %}}
* Learn more about [PersistentVolumes](/docs/concepts/storage/persistent-volumes/).
* Learn more about [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims).
### Reference
* [PersistentVolume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolume-v1-core)
* [PersistentVolumeClaim](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumeclaim-v1-core)
* See the `persistentVolumeReclaimPolicy` field of [PersistentVolumeSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumeclaim-v1-core).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/cilium-network-policy.md
+93
View File
@@ -0,0 +1,93 @@
---
reviewers:
- danwent
title: Use Cilium for NetworkPolicy
content_template: templates/task
---
{{% capture overview %}}
This page shows how to use Cilium for NetworkPolicy.
For background on Cilium, read the [Introduction to Cilium](https://cilium.readthedocs.io/en/latest/intro).
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Deploying Cilium on Minikube for Basic Testing
To get familiar with Cilium easily you can follow the
[Cilium Kubernetes Getting Started Guide](https://docs.cilium.io/en/latest/gettingstarted/minikube/)
to perform a basic DaemonSet installation of Cilium in minikube.
Installation in a minikube setup uses a simple ''all-in-one'' YAML
file that includes DaemonSet configurations for Cilium, to connect
to the minikube's etcd instance as well as appropriate RBAC settings:
```shell
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/master/examples/kubernetes/cilium.yaml
configmap "cilium-config" created
secret "cilium-etcd-secrets" created
serviceaccount "cilium" created
clusterrolebinding "cilium" created
daemonset "cilium" created
clusterrole "cilium" created
```
The remainder of the Getting Started Guide explains how to enforce both L3/L4
(i.e., IP address + port) security policies, as well as L7 (e.g., HTTP) security
policies using an example application.
## Deploying Cilium for Production Use
For detailed instructions around deploying Cilium for production, see:
[Cilium Kubernetes Installation Guide](https://cilium.readthedocs.io/en/latest/kubernetes/install/)
This documentation includes detailed requirements, instructions and example
production DaemonSet files.
{{% /capture %}}
{{% capture discussion %}}
## Understanding Cilium components
Deploying a cluster with Cilium adds Pods to the `kube-system` namespace. To see
this list of Pods run:
```shell
kubectl get pods --namespace=kube-system
```
You'll see a list of Pods similar to this:
```console
NAME DESIRED CURRENT READY NODE-SELECTOR AGE
cilium 1 1 1 <none> 2m
...
```
There are two main components to be aware of:
- One `cilium` Pod runs on each node in your cluster and enforces network policy
on the traffic to/from Pods on that node using Linux BPF.
- For production deployments, Cilium should leverage the key-value store cluster
(e.g., etcd) used by Kubernetes, which typically runs on the Kubernetes master nodes.
The [Cilium Kubernetes Installation Guide](https://cilium.readthedocs.io/en/latest/kubernetes/install/)
includes an example DaemonSet which can be customized to point to this key-value
store cluster. The simple ''all-in-one'' DaemonSet for minikube requires no such
configuration because it automatically connects to the minikube's etcd instance.
{{% /capture %}}
{{% capture whatsnext %}}
Once your cluster is running, you can follow the
[Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/)
to try out Kubernetes NetworkPolicy with Cilium.
Have fun, and if you have questions, contact us using the
[Cilium Slack Channel](https://cilium.herokuapp.com/).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml
+69
View File
@@ -0,0 +1,69 @@
# This is an example of how to setup cloud-controller-manger as a Daemonset in your cluster.
# It assumes that your masters can run pods and has the role node-role.kubernetes.io/master
# Note that this Daemonset will not work straight out of the box for your cloud, this is
# meant to be a guideline.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloud-controller-manager
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: system:cloud-controller-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: cloud-controller-manager
name: cloud-controller-manager
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: cloud-controller-manager
template:
metadata:
labels:
k8s-app: cloud-controller-manager
spec:
serviceAccountName: cloud-controller-manager
containers:
- name: cloud-controller-manager
# for in-tree providers we use k8s.gcr.io/cloud-controller-manager
# this can be replaced with any other image for out-of-tree providers
image: k8s.gcr.io/cloud-controller-manager:v1.8.0
command:
- /usr/local/bin/cloud-controller-manager
- --cloud-provider=<YOUR_CLOUD_PROVIDER> # Add your own cloud provider here!
- --leader-elect=true
- --use-service-account-credentials
# these flags will vary for every cloud provider
- --allocate-node-cidrs=true
- --configure-cloud-routes=true
- --cluster-cidr=172.17.0.0/16
tolerations:
# this is required so CCM can bootstrap itself
- key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
effect: NoSchedule
# this is to have the daemonset runnable on master nodes
# the taint may vary depending on your cluster setup
- key: node-role.kubernetes.io/master
effect: NoSchedule
# this is to restrict CCM to only run on master nodes
# the node selector may vary depending on your cluster setup
nodeSelector:
node-role.kubernetes.io/master: ""
content/en/docs/tasks/administer-cluster/cluster-management.md
+213
View File
@@ -0,0 +1,213 @@
---
reviewers:
- lavalamp
- thockin
title: Cluster Management
---
{{< toc >}}
This document describes several topics related to the lifecycle of a cluster: creating a new cluster,
upgrading your cluster's
master and worker nodes, performing node maintenance (e.g. kernel upgrades), and upgrading the Kubernetes API version of a
running cluster.
## Creating and configuring a Cluster
To install Kubernetes on a set of machines, consult one of the existing [Getting Started guides](/docs/setup/) depending on your environment.
## Upgrading a cluster
The current state of cluster upgrades is provider dependent, and some releases may require special care when upgrading. It is recommended that administrators consult both the [release notes](https://git.k8s.io/kubernetes/CHANGELOG.md), as well as the version specific upgrade notes prior to upgrading their clusters.
* [Upgrading to 1.6](/docs/admin/upgrade-1-6)
### Upgrading an Azure Kubernetes Service (AKS) cluster
Azure Kubernetes Service enables easy self-service upgrades of the control plane and nodes in your cluster. The process is
currently user-initiated and is described in the [Azure AKS documentation](https://docs.microsoft.com/en-us/azure/aks/upgrade-cluster).
### Upgrading Google Compute Engine clusters
Google Compute Engine Open Source (GCE-OSS) support master upgrades by deleting and
recreating the master, while maintaining the same Persistent Disk (PD) to ensure that data is retained across the
upgrade.
Node upgrades for GCE use a [Managed Instance Group](https://cloud.google.com/compute/docs/instance-groups/), each node
is sequentially destroyed and then recreated with new software. Any Pods that are running on that node need to be
controlled by a Replication Controller, or manually re-created after the roll out.
Upgrades on open source Google Compute Engine (GCE) clusters are controlled by the `cluster/gce/upgrade.sh` script.
Get its usage by running `cluster/gce/upgrade.sh -h`.
For example, to upgrade just your master to a specific version (v1.0.2):
```shell
cluster/gce/upgrade.sh -M v1.0.2
```
Alternatively, to upgrade your entire cluster to the latest stable release:
```shell
cluster/gce/upgrade.sh release/stable
```
### Upgrading Google Kubernetes Engine clusters
Google Kubernetes Engine automatically updates master components (e.g. `kube-apiserver`, `kube-scheduler`) to the latest version. It also handles upgrading the operating system and other components that the master runs on.
The node upgrade process is user-initiated and is described in the [Google Kubernetes Engine documentation](https://cloud.google.com/kubernetes-engine/docs/clusters/upgrade).
### Upgrading clusters on other platforms
Different providers, and tools, will manage upgrades differently. It is recommended that you consult their main documentation regarding upgrades.
* [kops](https://github.com/kubernetes/kops)
* [kubespray](https://github.com/kubernetes-incubator/kubespray)
* [CoreOS Tectonic](https://coreos.com/tectonic/docs/latest/admin/upgrade.html)
* ...
## Resizing a cluster
If your cluster runs short on resources you can easily add more machines to it if your cluster is running in [Node self-registration mode](/docs/admin/node/#self-registration-of-nodes).
If you're using GCE or Google Kubernetes Engine it's done by resizing Instance Group managing your Nodes. It can be accomplished by modifying number of instances on `Compute > Compute Engine > Instance groups > your group > Edit group` [Google Cloud Console page](https://console.developers.google.com) or using gcloud CLI:
```shell
gcloud compute instance-groups managed resize kubernetes-minion-group --size=42 --zone=$ZONE
```
Instance Group will take care of putting appropriate image on new machines and start them, while Kubelet will register its Node with API server to make it available for scheduling. If you scale the instance group down, system will randomly choose Nodes to kill.
In other environments you may need to configure the machine yourself and tell the Kubelet on which machine API server is running.
### Resizing an Azure Kubernetes Service (AKS) cluster
Azure Kubernetes Service enables user-initiated resizing of the cluster from either the CLI or the Azure Portal and is described in the [Azure AKS documentation](https://docs.microsoft.com/en-us/azure/aks/scale-cluster).
### Cluster autoscaling
If you are using GCE or Google Kubernetes Engine, you can configure your cluster so that it is automatically rescaled based on
pod needs.
As described in [Compute Resource](/docs/concepts/configuration/manage-compute-resources-container/), users can reserve how much CPU and memory is allocated to pods.
This information is used by the Kubernetes scheduler to find a place to run the pod. If there is
no node that has enough free capacity (or doesn't match other pod requirements) then the pod has
to wait until some pods are terminated or a new node is added.
Cluster autoscaler looks for the pods that cannot be scheduled and checks if adding a new node, similar
to the other in the cluster, would help. If yes, then it resizes the cluster to accommodate the waiting pods.
Cluster autoscaler also scales down the cluster if it notices that one or more nodes are not needed anymore for
an extended period of time (10min but it may change in the future).
Cluster autoscaler is configured per instance group (GCE) or node pool (Google Kubernetes Engine).
If you are using GCE then you can either enable it while creating a cluster with kube-up.sh script.
To configure cluster autoscaler you have to set three environment variables:
* `KUBE_ENABLE_CLUSTER_AUTOSCALER` - it enables cluster autoscaler if set to true.
* `KUBE_AUTOSCALER_MIN_NODES` - minimum number of nodes in the cluster.
* `KUBE_AUTOSCALER_MAX_NODES` - maximum number of nodes in the cluster.
Example:
```shell
KUBE_ENABLE_CLUSTER_AUTOSCALER=true KUBE_AUTOSCALER_MIN_NODES=3 KUBE_AUTOSCALER_MAX_NODES=10 NUM_NODES=5 ./cluster/kube-up.sh
```
On Google Kubernetes Engine you configure cluster autoscaler either on cluster creation or update or when creating a particular node pool
(which you want to be autoscaled) by passing flags `--enable-autoscaling` `--min-nodes` and `--max-nodes`
to the corresponding `gcloud` commands.
Examples:
```shell
gcloud container clusters create mytestcluster --zone=us-central1-b --enable-autoscaling --min-nodes=3 --max-nodes=10 --num-nodes=5
```
```shell
gcloud container clusters update mytestcluster --enable-autoscaling --min-nodes=1 --max-nodes=15
```
**Cluster autoscaler expects that nodes have not been manually modified (e.g. by adding labels via kubectl) as those properties would not be propagated to the new nodes within the same instance group.**
For more details about how the cluster autoscaler decides whether, when and how
to scale a cluster, please refer to the [FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md)
documentation from the autoscaler project.
## Maintenance on a Node
If you need to reboot a node (such as for a kernel upgrade, libc upgrade, hardware repair, etc.), and the downtime is
brief, then when the Kubelet restarts, it will attempt to restart the pods scheduled to it. If the reboot takes longer
(the default time is 5 minutes, controlled by `--pod-eviction-timeout` on the controller-manager),
then the node controller will terminate the pods that are bound to the unavailable node. If there is a corresponding
replica set (or replication controller), then a new copy of the pod will be started on a different node. So, in the case where all
pods are replicated, upgrades can be done without special coordination, assuming that not all nodes will go down at the same time.
If you want more control over the upgrading process, you may use the following workflow:
Use `kubectl drain` to gracefully terminate all pods on the node while marking the node as unschedulable:
```shell
kubectl drain $NODENAME
```
This keeps new pods from landing on the node while you are trying to get them off.
For pods with a replica set, the pod will be replaced by a new pod which will be scheduled to a new node. Additionally, if the pod is part of a service, then clients will automatically be redirected to the new pod.
For pods with no replica set, you need to bring up a new copy of the pod, and assuming it is not part of a service, redirect clients to it.
Perform maintenance work on the node.
Make the node schedulable again:
```shell
kubectl uncordon $NODENAME
```
If you deleted the node's VM instance and created a new one, then a new schedulable node resource will
be created automatically (if you're using a cloud provider that supports
node discovery; currently this is only Google Compute Engine, not including CoreOS on Google Compute Engine using kube-register). See [Node](/docs/admin/node) for more details.
## Advanced Topics
### Upgrading to a different API version
When a new API version is released, you may need to upgrade a cluster to support the new API version (e.g. switching from 'v1' to 'v2' when 'v2' is launched).
This is an infrequent event, but it requires careful management. There is a sequence of steps to upgrade to a new API version.
1. Turn on the new API version.
1. Upgrade the cluster's storage to use the new version.
1. Upgrade all config files. Identify users of the old API version endpoints.
1. Update existing objects in the storage to new version by running `cluster/update-storage-objects.sh`.
1. Turn off the old API version.
### Turn on or off an API version for your cluster
Specific API versions can be turned on or off by passing `--runtime-config=api/<version>` flag while bringing up the API server. For example: to turn off v1 API, pass `--runtime-config=api/v1=false`.
runtime-config also supports 2 special keys: api/all and api/legacy to control all and legacy APIs respectively.
For example, for turning off all API versions except v1, pass `--runtime-config=api/all=false,api/v1=true`.
For the purposes of these flags, _legacy_ APIs are those APIs which have been explicitly deprecated (e.g. `v1beta3`).
### Switching your cluster's storage API version
The objects that are stored to disk for a cluster's internal representation of the Kubernetes resources active in the cluster are written using a particular version of the API.
When the supported API changes, these objects may need to be rewritten in the newer API. Failure to do this will eventually result in resources that are no longer decodable or usable
by the Kubernetes API server.
`KUBE_API_VERSIONS` environment variable for the `kube-apiserver` binary which controls the API versions that are supported in the cluster. The first version in the list is used as the cluster's storage version. Hence, to set a specific version as the storage version, bring it to the front of list of versions in the value of `KUBE_API_VERSIONS`. You need to restart the `kube-apiserver` binary
for changes to this variable to take effect.
### Switching your config files to a new API version
You can use `kubectl convert` command to convert config files between different API versions.
```shell
kubectl convert -f pod.yaml --output-version v1
```
For more options, please refer to the usage of [kubectl convert](/docs/reference/generated/kubectl/kubectl-commands/#convert) command.
content/en/docs/tasks/administer-cluster/configure-multiple-schedulers.md
+194
View File
@@ -0,0 +1,194 @@
---
reviewers:
- davidopp
- madhusudancs
title: Configure Multiple Schedulers
---
Kubernetes ships with a default scheduler that is described [here](/docs/admin/kube-scheduler/).
If the default scheduler does not suit your needs you can implement your own scheduler.
Not just that, you can even run multiple schedulers simultaneously alongside the default
scheduler and instruct Kubernetes what scheduler to use for each of your pods. Let's
learn how to run multiple schedulers in Kubernetes with an example.
A detailed description of how to implement a scheduler is outside the scope of this
document. Please refer to the kube-scheduler implementation in
[pkg/scheduler](https://github.com/kubernetes/kubernetes/tree/{{< param "githubbranch" >}}/pkg/scheduler)
in the Kubernetes source directory for a canonical example.
### 1. Package the scheduler
Package your scheduler binary into a container image. For the purposes of this example,
let's just use the default scheduler (kube-scheduler) as our second scheduler as well.
Clone the [Kubernetes source code from Github](https://github.com/kubernetes/kubernetes)
and build the source.
```shell
git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
make
```
Create a container image containing the kube-scheduler binary. Here is the `Dockerfile`
to build the image:
```docker
FROM busybox
ADD ./_output/dockerized/bin/linux/amd64/kube-scheduler /usr/local/bin/kube-scheduler
```
Save the file as `Dockerfile`, build the image and push it to a registry. This example
pushes the image to
[Google Container Registry (GCR)](https://cloud.google.com/container-registry/).
For more details, please read the GCR
[documentation](https://cloud.google.com/container-registry/docs/).
```shell
docker build -t my-kube-scheduler:1.0 .
gcloud docker -- push gcr.io/my-gcp-project/my-kube-scheduler:1.0
```
### 2. Define a Kubernetes Deployment for the scheduler
Now that we have our scheduler in a container image, we can just create a pod
config for it and run it in our Kubernetes cluster. But instead of creating a pod
directly in the cluster, let's use a [Deployment](/docs/concepts/workloads/controllers/deployment/)
for this example. A [Deployment](/docs/concepts/workloads/controllers/deployment/) manages a
[Replica Set](/docs/concepts/workloads/controllers/replicaset/) which in turn manages the pods,
thereby making the scheduler resilient to failures. Here is the deployment
config. Save it as `my-scheduler.yaml`:
{{< code file="my-scheduler.yaml" >}}
An important thing to note here is that the name of the scheduler specified as an
argument to the scheduler command in the container spec should be unique. This is the name that is matched against the value of the optional `spec.schedulerName` on pods, to determine whether this scheduler is responsible for scheduling a particular pod.
Please see the
[kube-scheduler documentation](/docs/admin/kube-scheduler/) for
detailed description of other command line arguments.
### 3. Run the second scheduler in the cluster
In order to run your scheduler in a Kubernetes cluster, just create the deployment
specified in the config above in a Kubernetes cluster:
```shell
kubectl create -f my-scheduler.yaml
```
Verify that the scheduler pod is running:
```shell
$ kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
....
my-scheduler-lnf4s-4744f 1/1 Running 0 2m
...
```
You should see a "Running" my-scheduler pod, in addition to the default kube-scheduler
pod in this list.
To run multiple-scheduler with leader election enabled, you must do the following:
First, update the following fields in your YAML file:
* `--leader-elect=true`
* `--lock-object-namespace=lock-object-namespace`
* `--lock-object-name=lock-object-name`
If RBAC is enabled on your cluster, you must update the `system:kube-scheduler` cluster role. Add you scheduler name to the resourceNames of the rule applied for endpoints resources, as in the following example:
```
$ kubectl edit clusterrole system:kube-scheduler
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-scheduler
rules:
- apiGroups:
- ""
resourceNames:
- kube-scheduler
- my-scheduler
resources:
- endpoints
verbs:
- delete
- get
- patch
- update
```
### 4. Specify schedulers for pods
Now that our second scheduler is running, let's create some pods, and direct them to be scheduled by either the default scheduler or the one we just deployed. In order to schedule a given pod using a specific scheduler, we specify the name of the
scheduler in that pod spec. Let's look at three examples.
- Pod spec without any scheduler name
{{< code file="pod1.yaml" >}}
When no scheduler name is supplied, the pod is automatically scheduled using the
default-scheduler.
Save this file as `pod1.yaml` and submit it to the Kubernetes cluster.
```shell
kubectl create -f pod1.yaml
```
- Pod spec with `default-scheduler`
{{< code file="pod2.yaml" >}}
A scheduler is specified by supplying the scheduler name as a value to `spec.schedulerName`. In this case, we supply the name of the
default scheduler which is `default-scheduler`.
Save this file as `pod2.yaml` and submit it to the Kubernetes cluster.
```shell
kubectl create -f pod2.yaml
```
- Pod spec with `my-scheduler`
{{< code file="pod3.yaml" >}}
In this case, we specify that this pod should be scheduled using the scheduler that we
deployed - `my-scheduler`. Note that the value of `spec.schedulerName` should match the name supplied to the scheduler
command as an argument in the deployment config for the scheduler.
Save this file as `pod3.yaml` and submit it to the Kubernetes cluster.
```shell
kubectl create -f pod3.yaml
```
Verify that all three pods are running.
```shell
kubectl get pods
```
### Verifying that the pods were scheduled using the desired schedulers
In order to make it easier to work through these examples, we did not verify that the
pods were actually scheduled using the desired schedulers. We can verify that by
changing the order of pod and deployment config submissions above. If we submit all the
pod configs to a Kubernetes cluster before submitting the scheduler deployment config,
we see that the pod `annotation-second-scheduler` remains in "Pending" state forever
while the other two pods get scheduled. Once we submit the scheduler deployment config
and our new scheduler starts running, the `annotation-second-scheduler` pod gets
scheduled as well.
Alternatively, one could just look at the "Scheduled" entries in the event logs to
verify that the pods were scheduled by the desired schedulers.
```shell
kubectl get events
```
content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md
+401
View File
@@ -0,0 +1,401 @@
---
reviewers:
- mml
- wojtek-t
title: Operating etcd clusters for Kubernetes
---
{{< glossary_definition term_id="etcd" length="all" prepend="etcd is a ">}}
<!-- TODO(mml): Write this doc.
For the mechanics behind how kubernetes builds, distributes and deploys etcd,
see _some doc_.
-->
## Prerequisites
* Run etcd as a cluster of odd members.
* etcd is a leader-based distributed system. Ensure that the leader periodically send heartbeats on time to all followers to keep the cluster stable.
* Ensure that no resource starvation occurs.
Performance and stability of the cluster is sensitive to network and disk IO. Any resource starvation can lead to heartbeat timeout, causing instability of the cluster. An unstable etcd indicates that no leader is elected. Under such circumstances, a cluster cannot make any changes to its current state, which implies no new pods can be scheduled.
* Keeping stable etcd clusters is critical to the stability of Kubernetes clusters. Therefore, run etcd clusters on dedicated machines or isolated environments for [guaranteed resource requirements](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations).
## Resource requirements
Operating etcd with limited resources is suitable only for testing purposes. For deploying in production, advanced hardware configuration is required. Before deploying etcd in production, see [resource requirement reference documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#example-hardware-configurations).
## Starting Kubernetes API server
This section covers starting a Kubernetes API server with an etcd cluster in the deployment.
### Single-node etcd cluster
Use a single-node etcd cluster only for testing purpose.
1. Run the following:
./etcd --listen-client-urls=http://$PRIVATE_IP:2379 --advertise-client-urls=http://$PRIVATE_IP:2379
2. Start Kubernetes API server with the flag `--etcd-servers=$PRIVATE_IP:2379`.
Replace `PRIVATE_IP` with your etcd client IP.
### Multi-node etcd cluster
For durability and high availability, run etcd as a multi-node cluster in production and back it up periodically. A five-member cluster is recommended in production. For more information, see [FAQ Documentation](https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-is-failure-tolerance).
Configure an etcd cluster either by static member information or by dynamic discovery. For more information on clustering, see [etcd Clustering Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md).
For an example, consider a five-member etcd cluster running with the following client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`, `http://$IP4:2379`, and `http://$IP5:2379`. To start a Kubernetes API server:
1. Run the following:
./etcd --listen-client-urls=http://$IP1:2379, http://$IP2:2379, http://$IP3:2379, http://$IP4:2379, http://$IP5:2379 --advertise-client-urls=http://$IP1:2379, http://$IP2:2379, http://$IP3:2379, http://$IP4:2379, http://$IP5:2379
2. Start Kubernetes API servers with the flag `--etcd-servers=$IP1:2379, $IP2:2379, $IP3:2379, $IP4:2379, $IP5:2379`.
Replace `IP` with your client IP addresses.
### Multi-node etcd cluster with load balancer
To run a load balancing etcd cluster:
1. Set up an etcd cluster.
2. Configure a load balancer in front of the etcd cluster.
For example, let the address of the load balancer be `$LB`.
3. Start Kubernetes API Servers with the flag `--etcd-servers=$LB:2379`.
## Securing etcd clusters
Access to etcd is equivalent to root permission in the cluster so ideally only the API server should have access to it. Considering the sensitivity of the data, it is recommended to grant permission to only those nodes that require access to etcd clusters.
To secure etcd, either set up firewall rules or use the security features provided by etcd. etcd security features depend on x509 Public Key Infrastructure (PKI). To begin, establish secure communication channels by generating a key and certificate pair. For example, use key pairs `peer.key` and `peer.cert` for securing communication between etcd members, and `client.key` and `client.cert` for securing communication between etcd and its clients. See the [example scripts](https://github.com/coreos/etcd/tree/master/hack/tls-setup) provided by the etcd project to generate key pairs and CA files for client authentication.
### Securing communication
To configure etcd with secure peer communication, specify flags `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use https as URL schema.
Similarly, to configure etcd with secure client communication, specify flags `--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use https as URL schema.
### Limiting access of etcd clusters
After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API server. Use TLS authentication to do so.
For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` along with TLS, it verifies the certificates from clients by using system CAs or the CA passed in by `--trusted-ca-file` flag. Specifying flags `--client-cert-auth=true` and `--trusted-ca-file=etcd.ca` will restrict the access to clients with the certificate `k8sclient.cert`.
Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API server the access, configure it with the flags `--etcd-certfile=k8sclient.cert` and `--etcd-keyfile=k8sclient.key`.
{{< note >}}
**Note**: etcd authentication is not currently supported by Kubernetes. For more information, see the related issue [Support Basic Auth for Etcd v2](https://github.com/kubernetes/kubernetes/issues/23398).
{{< /note >}}
## Replacing a failed etcd member
etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member.
Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, member1=http://10.0.0.1, member2=http://10.0.0.2, and member3=http://10.0.0.3. When member1 fails, replace it with member4=http://10.0.0.4.
1. Get the member ID of the failed member1:
`etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list`
The following message is displayed:
8211f1d0f64f3269, started, member1, http://10.0.0.1:12380, http://10.0.0.1:2379
91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379
fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379
2. Remove the failed member:
`etcdctl member remove 8211f1d0f64f3269`
The following message is displayed:
Removed member 8211f1d0f64f3269 from cluster
3. Add the new member:
`./etcdctl member add member4 --peer-urls=http://10.0.0.4:2380`
The following message is displayed:
Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4
4. Start the newly added member on a machine with the IP `10.0.0.4`:
export ETCD_NAME="member4"
export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380"
export ETCD_INITIAL_CLUSTER_STATE=existing
etcd [flags]
5. Do either of the following:
1. Update its `--etcd-servers` flag to make Kubernetes aware of the configuration changes, then restart the Kubernetes API server.
2. Update the load balancer configuration if a load balancer is used in the deployment.
For more information on cluster reconfiguration, see [etcd Reconfiguration Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member).
## Backing up an etcd cluster
All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all master nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files.
Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshot.
### Built-in snapshot
etcd supports built-in snapshot, so backing up an etcd cluster is easy. A snapshot may either be taken from a live member with the `etcdctl snapshot save` command or by copying the `member/snap/db` file from an etcd [data directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir) that is not currently used by an etcd process. `datadir` is located at `$DATA_DIR/member/snap/db`. Taking the snapshot will normally not affect the performance of the member.
Below is an example for taking a snapshot of the keyspace served by `$ENDPOINT` to the file `snapshotdb`:
```sh
ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshotdb
# exit 0
# verify the snapshot
ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshotdb
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| fe01cf57 | 10 | 7 | 2.1 MB |
+----------+----------+------------+------------+
```
### Volume snapshot
If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.
## Scaling up etcd clusters
Scaling up etcd clusters increases availability by trading off performance. Scaling does not increase cluster performance nor capability. A general rule is not to scale up or down etcd clusters. Do not configure any auto scaling groups for etcd clusters. It is highly recommended to always run a static five-member etcd cluster for production Kubernetes clusters at any officially supported scale.
A reasonable scaling is to upgrade a three-member cluster to a five-member one, when more reliability is desired. See [etcd Reconfiguration Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member) for information on how to add members into an existing cluster.
## Restoring an etcd cluster
etcd supports restoring from snapshots that are taken from an etcd process of the [major.minor](http://semver.org/) version. Restoring a version from a different patch version of etcd also is supported. A restore operation is employed to recover the data of a failed cluster.
Before starting the restore operation, a snapshot file must be present. It can either be a snapshot file from a previous backup operation, or from a remaining [data directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir). `datadir` is located at `$DATA_DIR/member/snap/db`. For more information and examples on restoring a cluster from a snapshot file, see [etcd disaster recovery documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster).
If the access URLs of the restored cluster is changed from the previous cluster, the Kubernetes API server must be reconfigured accordingly. In this case, restart Kubernetes API server with the flag `--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag `--etcd-servers=$OLD_ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and `$OLD_ETCD_CLUSTER` with the respective IP addresses. If a load balancer is used in front of an etcd cluster, you might need to update the load balancer instead.
If the majority of etcd members have permanently failed, the etcd cluster is considered failed. In this scenario, Kubernetes cannot make any changes to its current state. Although the scheduled pods might continue to run, no new pods can be scheduled. In such cases, recover the etcd cluster and potentially reconfigure Kubernetes API server to fix the issue.
## Upgrading and rolling back etcd clusters
### Important assumptions
The upgrade procedure described in this document assumes that either:
1. The etcd cluster has only a single node.
2. The etcd cluster has multiple nodes.
In this case, the upgrade procedure requires shutting down the
etcd cluster. During the time the etcd cluster is shut down, the Kubernetes API Server will be read only.
{{< warning >}}
**Warning**: Deviations from the assumptions are untested by continuous
integration, and deviations might create undesirable consequences. Additional information about operating an etcd cluster is available [from the etcd maintainers](https://github.com/coreos/etcd/tree/master/Documentation).
{{< /warning >}}
### Background
As of Kubernetes version 1.5.1, we are still using etcd from the 2.2.1 release with
the v2 API. Also, we have no pre-existing process for updating etcd, as we have
never updated etcd by either minor or major version.
Note that we need to migrate both the etcd versions that we are using (from 2.2.1
to at least 3.0.x) as well as the version of the etcd API that Kubernetes talks to. The etcd 3.0.x
binaries support both the v2 and v3 API.
This document describes how to do this migration. If you want to skip the
background and cut right to the procedure, see [Upgrade
Procedure](#upgrade-procedure).
### etcd upgrade requirements
There are requirements on how an etcd cluster upgrade can be performed. The primary considerations are:
- Upgrade between one minor release at a time
- Rollback supported through additional tooling
#### One minor release at a time
Upgrade only one minor release at a time. For example, we cannot upgrade directly from 2.1.x to 2.3.x.
Within patch releases it is possible to upgrade and downgrade between arbitrary versions. Starting a cluster for
any intermediate minor release, waiting until the cluster is healthy, and then
shutting down the cluster will perform the migration. For example, to upgrade from version 2.1.x to 2.3.y,
it is enough to start etcd in 2.2.z version, wait until it is healthy, stop it, and then start the
2.3.y version.
#### Rollback via additional tooling
Versions 3.0+ of etcd do not support general rollback. That is,
after migrating from M.N to M.N+1, there is no way to go back to M.N.
The etcd team has provided a [custom rollback tool](https://git.k8s.io/kubernetes/cluster/images/etcd/rollback)
but the rollback tool has these limitations:
* This custom rollback tool is not part of the etcd repo and does not receive the same
testing as the rest of etcd. We are testing it in a couple of end-to-end tests.
There is only community support here.
* The rollback can be done only from the 3.0.x version (that is using the v3 API) to the
2.2.1 version (that is using the v2 API).
* The tool only works if the data is stored in `application/json` format.
* Rollback doesn’t preserve resource versions of objects stored in etcd.
{{< warning >}}
**Warning**: If the data is not kept in `application/json` format (see [Upgrade
Procedure](#upgrade-procedure)), you will lose the option to roll back to etcd
2.2.
{{< /warning >}}
The last bullet means that any component or user that has some logic
depending on resource versions may require restart after etcd rollback. This
includes that all clients using the watch API, which depends on
resource versions. Since both the kubelet and kube-proxy use the watch API, a
rollback might require restarting all Kubernetes components on all nodes.
{{< note >}}
**Note**: At the time of writing, both Kubelet and KubeProxy are using “resource
version” only for watching (i.e. are not using resource versions for anything
else). And both are using reflector and/or informer frameworks for watching
(i.e. they don’t send watch requests themselves). Both those frameworks if they
can’t renew watch, they will start from “current version” by doing “list + watch
from the resource version returned by list”. That means that if the apiserver
will be down for the period of rollback, all of node components should basically
restart their watches and start from “now” when apiserver is back. And it will
be back with new resource version. That would mean that restarting node
components is not needed. But the assumptions here may not hold forever.
{{< /note >}}
### Design
This section describes how we are going to do the migration, given the
[etcd upgrade requirements](#etcd-upgrade-requirements).
Note that because the code changes in Kubernetes code needed
to support the etcd v3 API are local and straightforward, we do not
focus on them at all. We focus only on the upgrade/rollback here.
### New etcd Docker image
We decided to completely change the content of the etcd image and the way it works.
So far, the Docker image for etcd in version X has contained only the etcd and
etcdctl binaries.
Going forward, the Docker image for etcd in version X will contain multiple
versions of etcd. For example, the 3.0.17 image will contain the 2.2.1, 2.3.7, and
3.0.17 binaries of etcd and etcdctl. This will allow running etcd in multiple
different versions using the same Docker image.
Additionally, the image will contain a custom script, written by the Kubernetes team,
for doing migration between versions. The image will also contain the rollback tool
provided by the etcd team.
### Migration script
The migration script that will be part of the etcd Docker image is a bash
script that works as follows:
1. Detect which version of etcd we were previously running.
For that purpose, we have added a dedicated file, `version.txt`, that
holds that information and is stored in the etcd-data-specific directory,
next to the etcd data. If the file doesn’t exist, we default it to version 2.2.1.
1. If we are in version 2.2.1 and are supposed to upgrade, backup
data.
1. Based on the detected previous etcd version and the desired one
(communicated via environment variable), do the upgrade steps as
needed. This means that for every minor etcd release greater than the detected one and
less than or equal to the desired one:
1. Start etcd in that version.
1. Wait until it is healthy. Healthy means that you can write some data to it.
1. Stop this etcd. Note that this etcd will not listen on the default
etcd port. It is hard coded to listen on ports that the API server is not
configured to connect to, which means that API server won’t be able to connect
to it. Assuming no other client goes out of its way to try to
connect and write to this obscure port, no new data will be written during
this period.
1. If the desired API version is v3 and the detected version is v2, do the offline
migration from the v2 to v3 data format. For that we use two tools:
* ./etcdctl migrate: This is the official tool for migration provided by the etcd team.
* A custom script that is attaching TTLs to events in the etcd. Note that etcdctl
migrate doesn’t support TTLs.
1. After every successful step, update contents of the version file.
This will protect us from the situation where something crashes in the
meantime ,and the version file gets completely unsynchronized with the
real data. Note that it is safe if the script crashes after the step is
done and before the file is updated. This will only result in redoing one
step in the next try.
All the previous steps are for the case where the detected version is less than or
equal to the desired version. In the opposite case, that is for a rollback, the
script works as follows:
1. Verify that the detected version is 3.0.x with the v3 API, and the
desired version is 2.2.1 with the v2 API. We don’t support any other rollback.
1. If so, we run the custom tool provided by etcd team to do the offline
rollback. This tool reads the v3 formatted data and writes it back to disk
in v2 format.
1. Finally update the contents of the version file.
### Upgrade procedure
Simply modify the command line in the etcd manifest to:
1. Run the migration script. If the previously run version is already in the
desired version, this will be no-op.
1. Start etcd in the desired version.
Starting in Kubernetes version 1.6, this has been done in the manifests for new
Google Compute Engine clusters. You should also specify these environment
variables. In particular, you must keep `STORAGE_MEDIA_TYPE` set to
`application/json` if you wish to preserve the option to roll back.
```
TARGET_STORAGE=etcd3
ETCD_IMAGE=3.0.17
TARGET_VERSION=3.0.17
STORAGE_MEDIA_TYPE=application/json
```
To roll back, use these:
```
TARGET_STORAGE=etcd2
ETCD_IMAGE=3.0.17
TARGET_VERSION=2.2.1
STORAGE_MEDIA_TYPE=application/json
```
## Notes for etcd Version 2.2.1
### Default configuration
The default setup scripts use kubelet's file-based static pods feature to run etcd in a
[pod](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/gce/manifests/etcd.manifest). This manifest should only
be run on master VMs. The default location that kubelet scans for manifests is
`/etc/kubernetes/manifests/`.
### Kubernetes's usage of etcd
By default, Kubernetes objects are stored under the `/registry` key in etcd.
This path can be prefixed by using the [kube-apiserver](/docs/admin/kube-apiserver) flag
`--etcd-prefix="/foo"`.
`etcd` is the only place that Kubernetes keeps state.
### Troubleshooting
To test whether `etcd` is running correctly, you can try writing a value to a
test key. On your master VM (or somewhere with firewalls configured such that
you can talk to your cluster's etcd), try:
```shell
curl -X PUT "http://${host}:${port}/v2/keys/_test"
```
content/en/docs/tasks/administer-cluster/coredns.md
+59
View File
@@ -0,0 +1,59 @@
---
reviewers:
- johnbelamaric
title: Using CoreDNS for Service Discovery
min-kubernetes-server-version: v1.9
content_template: templates/task
---
{{< feature-state state="beta" >}}
{{% capture overview %}}
This page describes how to enable CoreDNS instead of kube-dns for service
discovery.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Installing CoreDNS with kubeadm
In Kubernetes 1.9, [CoreDNS](https://coredns.io) is available as an alpha feature, and
in Kubernetes 1.10 it is available as a beta feature. In either case, you may install
it during cluster creation by setting the `CoreDNS` feature gate to `true` during `kubeadm init`:
```
kubeadm init --feature-gates=CoreDNS=true
```
This installs CoreDNS instead of kube-dns.
## Upgrading an Existing Cluster with kubeadm
In Kubernetes 1.10, you can also move to CoreDNS when you use `kubeadm` to upgrade
a cluster that is using `kube-dns`. In this case, `kubeadm` will generate the CoreDNS configuration
("Corefile") based upon the `kube-dns` ConfigMap, preserving configurations for federation,
stub domains, and upstream name server.
Note that if you are running CoreDNS in your cluster already, prior to upgrade, your existing Corefile will be
**overwritten** by the one created during upgrade. **You should save your existing ConfigMap
if you have customized it.** You may re-apply your customizations after the new ConfigMap is
up and running.
This process will be modified for the GA release of this feature, such that an existing
Corefile will not be overwritten.
{{% /capture %}}
{{% capture whatsnext %}}
You can configure [CoreDNS](https://coredns.io) to support many more use cases than
kube-dns by modifying the `Corefile`. For more information, see the
[CoreDNS site](https://coredns.io/2017/05/08/custom-dns-entries-for-kubernetes/).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/cpu-constraint-namespace.md
+277
View File
@@ -0,0 +1,277 @@
---
title: Configure Minimum and Maximum CPU Constraints for a Namespace
content_template: templates/task
---
{{% capture overview %}}
This page shows how to set minimum and maximum values for the CPU resources used by Containers
and Pods in a namespace. You specify minimum and maximum CPU values in a
[LimitRange](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#limitrange-v1-core)
object. If a Pod does not meet the constraints imposed by the LimitRange, it cannot be created
in the namespace.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
Each node in your cluster must have at least 1 CPU.
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace constraints-cpu-example
```
## Create a LimitRange and a Pod
Here's the configuration file for a LimitRange:
{{< code file="cpu-constraints.yaml" >}}
Create the LimitRange:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-constraints.yaml --namespace=constraints-cpu-example
```
View detailed information about the LimitRange:
```shell
kubectl get limitrange cpu-min-max-demo-lr --output=yaml --namespace=constraints-cpu-example
```
The output shows the minimum and maximum CPU constraints as expected. But
notice that even though you didn't specify default values in the configuration
file for the LimitRange, they were created automatically.
```yaml
limits:
- default:
cpu: 800m
defaultRequest:
cpu: 800m
max:
cpu: 800m
min:
cpu: 200m
type: Container
```
Now whenever a Container is created in the constraints-cpu-example namespace, Kubernetes
performs these steps:
* If the Container does not specify its own CPU request and limit, assign the default
CPU request and limit to the Container.
* Verify that the Container specifies a CPU request that is greater than or equal to 200 millicpu.
* Verify that the Container specifies a CPU limit that is less than or equal to 800 millicpu.
{{< note >}}
**Note:** When creating a `LimitRange` object, you can specify limits on huge-pages
or GPUs as well. However, when both `default` and `defaultRequest` are specified
on these resources, the two values must be the same.
{{< /note >}}
Here's the configuration file for a Pod that has one Container. The Container manifest
specifies a CPU request of 500 millicpu and a CPU limit of 800 millicpu. These satisfy the
minimum and maximum CPU constraints imposed by the LimitRange.
{{< code file="cpu-constraints-pod.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-constraints-pod.yaml --namespace=constraints-cpu-example
```
Verify that the Pod's Container is running:
```shell
kubectl get pod constraints-cpu-demo --namespace=constraints-cpu-example
```
View detailed information about the Pod:
```shell
kubectl get pod constraints-cpu-demo --output=yaml --namespace=constraints-cpu-example
```
The output shows that the Container has a CPU request of 500 millicpu and CPU limit
of 800 millicpu. These satisfy the constraints imposed by the LimitRange.
```yaml
resources:
limits:
cpu: 800m
requests:
cpu: 500m
```
## Delete the Pod
```shell
kubectl delete pod constraints-cpu-demo --namespace=constraints-cpu-example
```
## Attempt to create a Pod that exceeds the maximum CPU constraint
Here's the configuration file for a Pod that has one Container. The Container specifies a
CPU request of 500 millicpu and a cpu limit of 1.5 cpu.
{{< code file="cpu-constraints-pod-2.yaml" >}}
Attempt to create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-constraints-pod-2.yaml --namespace=constraints-cpu-example
```
The output shows that the Pod does not get created, because the Container specifies a CPU limit that is
too large:
```
Error from server (Forbidden): error when creating "docs/tasks/administer-cluster/cpu-constraints-pod-2.yaml":
pods "constraints-cpu-demo-2" is forbidden: maximum cpu usage per Container is 800m, but limit is 1500m.
```
## Attempt to create a Pod that does not meet the minimum CPU request
Here's the configuration file for a Pod that has one Container. The Container specifies a
CPU request of 100 millicpu and a CPU limit of 800 millicpu.
{{< code file="cpu-constraints-pod-3.yaml" >}}
Attempt to create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-constraints-pod-3.yaml --namespace=constraints-cpu-example
```
The output shows that the Pod does not get created, because the Container specifies a CPU
request that is too small:
```
Error from server (Forbidden): error when creating "docs/tasks/administer-cluster/cpu-constraints-pod-3.yaml":
pods "constraints-cpu-demo-4" is forbidden: minimum cpu usage per Container is 200m, but request is 100m.
```
## Create a Pod that does not specify any CPU request or limit
Here's the configuration file for a Pod that has one Container. The Container does not
specify a CPU request, and it does not specify a CPU limit.
{{< code file="cpu-constraints-pod-4.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-constraints-pod-4.yaml --namespace=constraints-cpu-example
```
View detailed information about the Pod:
```
kubectl get pod constraints-cpu-demo-4 --namespace=constraints-cpu-example --output=yaml
```
The output shows that the Pod's Container has a CPU request of 800 millicpu and a CPU limit of 800 millicpu.
How did the Container get those values?
```yaml
resources:
limits:
cpu: 800m
requests:
cpu: 800m
```
Because your Container did not specify its own CPU request and limit, it was given the
[default CPU request and limit](/docs/tasks/administer-cluster/cpu-default-namespace/)
from the LimitRange.
At this point, your Container might be running or it might not be running. Recall that a prerequisite
for this task is that your Nodes have at least 1 CPU. If each of your Nodes has only
1 CPU, then there might not be enough allocatable CPU on any Node to accommodate a request
of 800 millicpu. If you happen to be using Nodes with 2 CPU, then you probably have
enough CPU to accommodate the 800 millicpu request.
Delete your Pod:
```
kubectl delete pod constraints-cpu-demo-4 --namespace=constraints-cpu-example
```
## Enforcement of minimum and maximum CPU constraints
The maximum and minimum CPU constraints imposed on a namespace by a LimitRange are enforced only
when a Pod is created or updated. If you change the LimitRange, it does not affect
Pods that were created previously.
## Motivation for minimum and maximum CPU constraints
As a cluster administrator, you might want to impose restrictions on the CPU resources that Pods can use.
For example:
* Each Node in a cluster has 2 CPU. You do not want to accept any Pod that requests
more than 2 CPU, because no Node in the cluster can support the request.
* A cluster is shared by your production and development departments.
You want to allow production workloads to consume up to 3 CPU, but you want development workloads to be limited
to 1 CPU. You create separate namespaces for production and development, and you apply CPU constraints to
each namespace.
## Clean up
Delete your namespace:
```shell
kubectl delete namespace constraints-cpu-example
```
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default Memory Requests and Limits for a Namespace](/docs/tasks/administer-cluster/memory-default-namespace/)
* [Configure Default CPU Requests and Limits for a Namespace](/docs/tasks/administer-cluster/cpu-default-namespace/)
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Memory and CPU Quotas for a Namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/)
* [Configure a Pod Quota for a Namespace](/docs/tasks/administer-cluster/quota-pod-namespace/)
* [Configure Quotas for API Objects](/docs/tasks/administer-cluster/quota-api-object/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/cpu-constraints-pod-2.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-cpu-demo-2
spec:
containers:
- name: constraints-cpu-demo-2-ctr
image: nginx
resources:
limits:
cpu: "1.5"
requests:
cpu: "500m"
content/en/docs/tasks/administer-cluster/cpu-constraints-pod-3.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-cpu-demo-4
spec:
containers:
- name: constraints-cpu-demo-4-ctr
image: nginx
resources:
limits:
cpu: "800m"
requests:
cpu: "100m"
content/en/docs/tasks/administer-cluster/cpu-constraints-pod-4.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-cpu-demo-4
spec:
containers:
- name: constraints-cpu-demo-4-ctr
image: vish/stress
content/en/docs/tasks/administer-cluster/cpu-constraints-pod.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-cpu-demo
spec:
containers:
- name: constraints-cpu-demo-ctr
image: nginx
resources:
limits:
cpu: "800m"
requests:
cpu: "500m"
content/en/docs/tasks/administer-cluster/cpu-constraints.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-min-max-demo-lr
spec:
limits:
- max:
cpu: "800m"
min:
cpu: "200m"
type: Container
content/en/docs/tasks/administer-cluster/cpu-default-namespace.md
+185
View File
@@ -0,0 +1,185 @@
---
title: Configure Default CPU Requests and Limits for a Namespace
content_template: templates/task
---
{{% capture overview %}}
This page shows how to configure default CPU requests and limits for a namespace.
A Kubernetes cluster can be divided into namespaces. If a Container is created in a namespace
that has a default CPU limit, and the Container does not specify its own CPU limit, then
the Container is assigned the default CPU limit. Kubernetes assigns a default CPU request
under certain conditions that are explained later in this topic.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace default-cpu-example
```
## Create a LimitRange and a Pod
Here's the configuration file for a LimitRange object. The configuration specifies
a default CPU request and a default CPU limit.
{{< code file="cpu-defaults.yaml" >}}
Create the LimitRange in the default-cpu-example namespace:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-defaults.yaml --namespace=default-cpu-example
```
Now if a Container is created in the default-cpu-example namespace, and the
Container does not specify its own values for CPU request and CPU limit,
the Container is given a default CPU request of 0.5 and a default
CPU limit of 1.
Here's the configuration file for a Pod that has one Container. The Container
does not specify a CPU request and limit.
{{< code file="cpu-defaults-pod.yaml" >}}
Create the Pod.
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-defaults-pod.yaml --namespace=default-cpu-example
```
View the Pod's specification:
```shell
kubectl get pod default-cpu-demo --output=yaml --namespace=default-cpu-example
```
The output shows that the Pod's Container has a CPU request of 500 millicpus and
a CPU limit of 1 cpu. These are the default values specified by the LimitRange.
```shel
containers:
- image: nginx
imagePullPolicy: Always
name: default-cpu-demo-ctr
resources:
limits:
cpu: "1"
requests:
cpu: 500m
```
## What if you specify a Container's limit, but not its request?
Here's the configuration file for a Pod that has one Container. The Container
specifies a CPU limit, but not a request:
{{< code file="cpu-defaults-pod-2.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-defaults-pod-2.yaml --namespace=default-cpu-example
```
View the Pod specification:
```
kubectl get pod default-cpu-demo-2 --output=yaml --namespace=default-cpu-example
```
The output shows that the Container's CPU request is set to match its CPU limit.
Notice that the Container was not assigned the default CPU request value of 0.5 cpu.
```
resources:
limits:
cpu: "1"
requests:
cpu: "1"
```
## What if you specify a Container's request, but not its limit?
Here's the configuration file for a Pod that has one Container. The Container
specifies a CPU request, but not a limit:
{{< code file="cpu-defaults-pod-3.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/cpu-defaults-pod-3.yaml --namespace=default-cpu-example
```
View the Pod specification:
```
kubectl get pod default-cpu-demo-3 --output=yaml --namespace=default-cpu-example
```
The output shows that the Container's CPU request is set to the value specified in the
Container's configuration file. The Container's CPU limit is set to 1 cpu, which is the
default CPU limit for the namespace.
```
resources:
limits:
cpu: "1"
requests:
cpu: 750m
```
## Motivation for default CPU limits and requests
If your namespace has a
[resource quota](),
it is helpful to have a default value in place for CPU limit.
Here are two of the restrictions that a resource quota imposes on a namespace:
* Every Container that runs in the namespace must have its own CPU limit.
* The total amount of CPU used by all Containers in the namespace must not exceed a specified limit.
If a Container does not specify its own CPU limit, it is given the default limit, and then
it can be allowed to run in a namespace that is restricted by a quota.
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default Memory Requests and Limits for a Namespace](/docs/tasks/administer-cluster/memory-default-namespace/)
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
* [Configure Memory and CPU Quotas for a Namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/)
* [Configure a Pod Quota for a Namespace](/docs/tasks/administer-cluster/quota-pod-namespace/)
* [Configure Quotas for API Objects](/docs/tasks/administer-cluster/quota-api-object/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/cpu-defaults-pod-2.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: default-cpu-demo-2
spec:
containers:
- name: default-cpu-demo-2-ctr
image: nginx
resources:
limits:
cpu: "1"
content/en/docs/tasks/administer-cluster/cpu-defaults-pod-3.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: default-cpu-demo-3
spec:
containers:
- name: default-cpu-demo-3-ctr
image: nginx
resources:
requests:
cpu: "0.75"
content/en/docs/tasks/administer-cluster/cpu-defaults-pod.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Pod
metadata:
name: default-cpu-demo
spec:
containers:
- name: default-cpu-demo-ctr
image: nginx
content/en/docs/tasks/administer-cluster/cpu-defaults.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-limit-range
spec:
limits:
- default:
cpu: 1
defaultRequest:
cpu: 0.5
type: Container
content/en/docs/tasks/administer-cluster/cpu-management-policies.md
+199
View File
@@ -0,0 +1,199 @@
---
title: Control CPU Management Policies on the Node
reviewers:
- sjenning
- ConnorDoyle
- balajismaniam
---
{{< feature-state state="beta" >}}
{{< toc >}}
Kubernetes keeps many aspects of how pods execute on nodes abstracted
from the user. This is by design.  However, some workloads require
stronger guarantees in terms of latency and/or performance in order to operate
acceptably. The kubelet provides methods to enable more complex workload
placement policies while keeping the abstraction free from explicit placement
directives.
## CPU Management Policies
By default, the kubelet uses [CFS quota](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler)
to enforce pod CPU limits.  When the node runs many CPU-bound pods,
the workload can move to different CPU cores depending on
whether the pod is throttled and which CPU cores are available at
scheduling time.  Many workloads are not sensitive to this migration and thus
work fine without any intervention.
However, in workloads where CPU cache affinity and scheduling latency
significantly affect workload performance, the kubelet allows alternative CPU
management policies to determine some placement preferences on the node.
### Configuration
The CPU Manager is an alpha feature in Kubernetes v1.8. It was enabled by
default as a beta feature since v1.10.
The CPU Manager policy is set with the `--cpu-manager-policy` kubelet
option. There are two supported policies:
* `none`: the default, which represents the existing scheduling behavior.
* `static`: allows pods with certain resource characteristics to be
granted increased CPU affinity and exclusivity on the node.
The CPU manager periodically writes resource updates through the CRI in
order to reconcile in-memory CPU assignments with cgroupfs. The reconcile
frequency is set through a new Kubelet configuration value
`--cpu-manager-reconcile-period`. If not specified, it defaults to the same
duration as `--node-status-update-frequency`.
### None policy
The `none` policy explicitly enables the existing default CPU
affinity scheme, providing no affinity beyond what the OS scheduler does
automatically.  Limits on CPU usage for
[Guaranteed pods](/docs/tasks/configure-pod-container/quality-service-pod/)
are enforced using CFS quota.
### Static policy
The `static` policy allows containers in `Guaranteed` pods with integer CPU
`requests` access to exclusive CPUs on the node. This exclusivity is enforced
using the [cpuset cgroup controller](https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt).
{{< note >}}
**Note:** System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods.
{{< /note >}}
{{< note >}}
**Note:** The alpha version of this policy does not guarantee static
exclusive allocations across Kubelet restarts.
{{< /note >}}
This policy manages a shared pool of CPUs that initially contains all CPUs in the
node. The amount of exclusively allocatable CPUs is equal to the total
number of CPUs in the node minus any CPU reservations by the kubelet `--kube-reserved` or
`--system-reserved` options. CPUs reserved by these options are taken, in
integer quantity, from the initial shared pool in ascending order by physical
core ID.  This shared pool is the set of CPUs on which any containers in
`BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional
CPU `requests` also run on CPUs in the shared pool. Only containers that are
both part of a `Guaranteed` pod and have integer CPU `requests` are assigned
exclusive CPUs.
{{< note >}}
**Note:** The kubelet requires a CPU reservation greater than zero be made
using either `--kube-reserved` and/or `--system-reserved` when the static
policy is enabled. This is because zero CPU reservation would allow the shared
pool to become empty.
{{< /note >}}
As `Guaranteed` pods whose containers fit the requirements for being statically
assigned are scheduled to the node, CPUs are removed from the shared pool and
placed in the cpuset for the container. CFS quota is not used to bound
the CPU usage of these containers as their usage is bound by the scheduling domain
itself. In others words, the number of CPUs in the container cpuset is equal to the integer
CPU `limit` specified in the pod spec. This static assignment increases CPU
affinity and decreases context switches due to throttling for the CPU-bound
workload.
Consider the containers in the following pod specs:
```yaml
spec:
containers:
- name: nginx
image: nginx
```
This pod runs in the `BestEffort` QoS class because no resource `requests` or
`limits` are specified. It runs in the shared pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
```
This pod runs in the `Burstable` QoS class because resource `requests` do not
equal `limits` and the `cpu` quantity is not specified. It runs in the shared
pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
requests:
memory: "100Mi"
cpu: "1"
```
This pod runs in the `Burstable` QoS class because resource `requests` do not
equal `limits`. It runs in the shared pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
requests:
memory: "200Mi"
cpu: "2"
```
This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
And the container's resource limit for the CPU resource is an integer greater than
or equal to one. The `nginx` container is granted 2 exclusive CPUs.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "1.5"
requests:
memory: "200Mi"
cpu: "1.5"
```
This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
But the container's resource limit for the CPU resource is a fraction. It runs in
the shared pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
```
This pod runs in the `Guaranteed` QoS class because only `limits` are specified
and `requests` are set equal to `limits` when not explicitly specified. And the
container's resource limit for the CPU resource is an integer greater than or
equal to one. The `nginx` container is granted 2 exclusive CPUs.
content/en/docs/tasks/administer-cluster/declare-network-policy.md
+128
View File
@@ -0,0 +1,128 @@
---
reviewers:
- caseydavenport
- danwinship
title: Declare Network Policy
content_template: templates/task
---
{{% capture overview %}}
This document helps you get started using the Kubernetes [NetworkPolicy API](/docs/concepts/services-networking/network-policies/) to declare network policies that govern how pods communicate with each other.
{{% /capture %}}
{{% capture prerequisites %}}
You'll need to have a Kubernetes cluster in place, with network policy support. There are a number of network providers that support NetworkPolicy, including:
* [Calico](/docs/tasks/configure-pod-container/calico-network-policy/)
* [Cilium](/docs/tasks/administer-cluster/cilium-network-policy/)
* [Kube-router](/docs/tasks/administer-cluster/kube-router-network-policy/)
* [Romana](/docs/tasks/configure-pod-container/romana-network-policy/)
* [Weave Net](/docs/tasks/administer-cluster/weave-network-policy/)
**Note**: The above list is sorted alphabetically by product name, not by recommendation or preference. This example is valid for a Kubernetes cluster using any of these providers.
{{% /capture %}}
{{% capture steps %}}
## Create an `nginx` deployment and expose it via a service
To see how Kubernetes network policy works, start off by creating an `nginx` deployment and exposing it via a service.
```console
$ kubectl run nginx --image=nginx --replicas=2
deployment "nginx" created
$ kubectl expose deployment nginx --port=80
service "nginx" exposed
```
This runs two `nginx` pods in the default namespace, and exposes them through a service called `nginx`.
```console
$ kubectl get svc,pod
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.100.0.1 <none> 443/TCP 46m
svc/nginx 10.100.0.16 <none> 80/TCP 33s
NAME READY STATUS RESTARTS AGE
po/nginx-701339712-e0qfq 1/1 Running 0 35s
po/nginx-701339712-o00ef 1/1 Running 0 35s
```
## Test the service by accessing it from another pod
You should be able to access the new `nginx` service from other pods. To test, access the service from another pod in the default namespace. Make sure you haven't enabled isolation on the namespace.
Start a busybox container, and use `wget` on the `nginx` service:
```console
$ kubectl run busybox --rm -ti --image=busybox /bin/sh
Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending, pod ready: false
Hit enter for command prompt
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.100.0.16:80)
/ #
```
## Limit access to the `nginx` service
Let's say you want to limit access to the `nginx` service so that only pods with the label `access: true` can query it. To do that, create a `NetworkPolicy` that allows connections only from those pods:
```yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-nginx
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- podSelector:
matchLabels:
access: "true"
```
## Assign the policy to the service
Use kubectl to create a NetworkPolicy from the above nginx-policy.yaml file:
```console
$ kubectl create -f nginx-policy.yaml
networkpolicy "access-nginx" created
```
## Test access to the service when access label is not defined
If we attempt to access the nginx Service from a pod without the correct labels, the request will now time out:
```console
$ kubectl run busybox --rm -ti --image=busybox /bin/sh
Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending, pod ready: false
Hit enter for command prompt
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.100.0.16:80)
wget: download timed out
/ #
```
## Define access label and test again
Create a pod with the correct labels, and you'll see that the request is allowed:
```console
$ kubectl run busybox --rm -ti --labels="access=true" --image=busybox /bin/sh
Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending, pod ready: false
Hit enter for command prompt
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.100.0.16:80)
/ #
```
{{% /capture %}}
content/en/docs/tasks/administer-cluster/developing-cloud-controller-manager.md
+35
View File
@@ -0,0 +1,35 @@
---
reviewers:
- luxas
- thockin
- wlan0
title: Developing Cloud Controller Manager
---
**Cloud Controller Manager is an alpha feature in 1.8. In upcoming releases it will
be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers
can develop their features independently from the core Kubernetes release cycles.**
{{< toc >}}
## Background
Before going into how to build your own cloud controller manager, some background on how it works under the hood is helpful. The cloud controller manager is code from `kube-controller-manager` utilizing Go interfaces to allow implementations from any cloud to be plugged in. Most of the scaffolding and generic controller implementations will be in core, but it will always exec out to the cloud interfaces it is provided, so long as the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L29-L50) is satisfied.
To dive a little deeper into implementation details, all cloud controller managers will import packages from Kubernetes core, the only difference being each project will register their own cloud providers by calling [cloudprovider.RegisterCloudProvider](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52) where a global variable of available cloud providers is updated.
## Developing
### Out of Tree
To build an out-of-tree cloud-controller-manager for your cloud, follow these steps:
1. Create a go package with an implementation that satisfies [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
2. Use [main.go in cloud-controller-manager](https://github.com/kubernetes/kubernetes/blob/master/cmd/cloud-controller-manager/controller-manager.go) from Kubernetes core as a template for your main.go. As mentioned above, the only difference should be the cloud package that will be imported.
3. Import your cloud package in `main.go`, ensure your package has an `init` block to run [cloudprovider.RegisterCloudProvider](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52).
Using existing out-of-tree cloud providers as an example may be helpful. You can find the list [here](/docs/tasks/administer-cluster/running-cloud-controller.md#examples).
### In Tree
For in-tree cloud providers, you can run the in-tree cloud controller manager as a [Daemonset](/docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml) in your cluster. See the [running cloud controller manager docs](/docs/tasks/administer-cluster/running-cloud-controller.md) for more details.
content/en/docs/tasks/administer-cluster/dns-custom-nameservers.md
+192
View File
@@ -0,0 +1,192 @@
---
reviewers:
- bowei
- zihongz
title: Customizing DNS Service
content_template: templates/task
---
{{% capture overview %}}
This page provides hints on configuring DNS Pod and guidance on customizing the
DNS resolution process.
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Kubernetes version 1.6 and above.
* The cluster must be configured to use the `kube-dns` addon.
{{% /capture %}}
{{% capture steps %}}
## Introduction
Starting from Kubernetes v1.3, DNS is a built-in service launched automatically
using the addon manager
[cluster add-on](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/README.md).
The running Kubernetes DNS pod holds 3 containers:
- "`kubedns`": The `kubedns` process watches the Kubernetes master for changes
in Services and Endpoints, and maintains in-memory lookup structures to serve
DNS requests.
- "`dnsmasq`": The `dnsmasq` container adds DNS caching to improve performance.
- "`sidecar`": The `sidecar` container provides a single health check endpoint
while performing dual healthchecks (for `dnsmasq` and `kubedns`).
The DNS pod is exposed as a Kubernetes Service with a static IP. Once assigned
the kubelet passes DNS configured using the `--cluster-dns=<dns-service-ip>`
flag to each container.
DNS names also need domains. The local domain is configurable in the kubelet
using the flag `--cluster-domain=<default-local-domain>`.
The Kubernetes cluster DNS server is based off the
[SkyDNS](https://github.com/skynetservices/skydns) library. It supports forward
lookups (A records), service lookups (SRV records) and reverse IP address
lookups (PTR records).
## Inheriting DNS from the node
When running a pod, kubelet will prepend the cluster DNS server and search
paths to the node's own DNS settings. If the node is able to resolve DNS names
specific to the larger environment, pods should be able to, also.
See [Known issues](#known-issues) below for a caveat.
If you don't want this, or if you want a different DNS config for pods, you can
use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will
not inherit DNS. Setting it to a valid file path means that kubelet will use
this file instead of `/etc/resolv.conf` for DNS inheritance.
## Configure stub-domain and upstream DNS servers
Cluster administrators can specify custom stub domains and upstream nameservers
by providing a ConfigMap for kube-dns (`kube-system:kube-dns`).
For example, the following ConfigMap sets up a DNS configuration with a single stub domain and two
upstream nameservers.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-dns
namespace: kube-system
data:
stubDomains: |
{"acme.local": ["1.2.3.4"]}
upstreamNameservers: |
["8.8.8.8", "8.8.4.4"]
```
As specified, DNS requests with the “.acme.local” suffix
are forwarded to a DNS listening at 1.2.3.4. Google Public DNS
serves the upstream queries.
The table below describes how queries with certain domain names would map to
their destination DNS servers:
| Domain name | Server answering the query |
| ----------- | -------------------------- |
| kubernetes.default.svc.cluster.local| kube-dns |
| foo.acme.local| custom DNS (1.2.3.4) |
| widget.com | upstream DNS (one of 8.8.8.8, 8.8.4.4) |
See [ConfigMap options](#configmap-options) for
details about the configuration option format.
{{% /capture %}}
{{% capture discussion %}}
### Impacts on Pods
Custom upstream nameservers and stub domains won't impact Pods that have their
`dnsPolicy` set to "`Default`" or "`None`".
If a Pod's `dnsPolicy` is set to "`ClusterFirst`", its name resolution is
handled differently, depending on whether stub-domain and upstream DNS servers
are configured.
**Without custom configurations**: Any query that does not match the configured
cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream
nameserver inherited from the node.
**With custom configurations**: If stub domains and upstream DNS servers are
configured (as in the [previous example](#configuring-stub-domain-and-upstream-dns-servers)),
DNS queries will be routed according to the following flow:
1. The query is first sent to the DNS caching layer in kube-dns.
1. From the caching layer, the suffix of the request is examined and then
forwarded to the appropriate DNS, based on the following cases:
* *Names with the cluster suffix* (e.g.".cluster.local"):
The request is sent to kube-dns.
* *Names with the stub domain suffix* (e.g. ".acme.local"):
The request is sent to the configured custom DNS resolver (e.g. listening at 1.2.3.4).
* *Names without a matching suffix* (e.g."widget.com"):
The request is forwarded to the upstream DNS
(e.g. Google public DNS servers at 8.8.8.8 and 8.8.4.4).
![DNS lookup flow](/docs/tasks/administer-cluster/dns-custom-nameservers/dns.png)
## ConfigMap options
Options for the kube-dns `kube-system:kube-dns` ConfigMap:
| Field | Format | Description |
| ----- | ------ | ----------- |
| `stubDomains` (optional) | A JSON map using a DNS suffix key (e.g. “acme.local”) and a value consisting of a JSON array of DNS IPs. | The target nameserver may itself be a Kubernetes service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | Note: If specified, then the values specified replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
### Examples
#### Example: Stub domain
In this example, the user has a Consul DNS service discovery system that they wish to
integrate with kube-dns. The consul domain server is located at 10.150.0.1, and
all consul names have the suffix “.consul.local”. To configure Kubernetes, the
cluster administrator simply creates a ConfigMap object as shown below.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-dns
namespace: kube-system
data:
stubDomains: |
{"consul.local": ["10.150.0.1"]}
```
Note that the cluster administrator did not wish to override the node’s
upstream nameservers, so they did not specify the optional
`upstreamNameservers` field.
#### Example: Upstream nameserver
In this example the cluster administrator wants to explicitly force all
non-cluster DNS lookups to go through their own nameserver at 172.16.0.1.
Again, this is easy to accomplish; they just need to create a ConfigMap with the
`upstreamNameservers` field specifying the desired nameserver.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-dns
namespace: kube-system
data:
upstreamNameservers: |
["172.16.0.1"]
```
## What's next
- [Debugging DNS Resolution](/docs/tasks/administer-cluster/dns-debugging-resolution/).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/dns-custom-nameservers/dns.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md
+197
View File
@@ -0,0 +1,197 @@
---
approvers:
- bowei
- zihongz
title: Debugging DNS Resolution
content_template: templates/task
---
{{% capture overview %}}
This page provides hints on diagnosing DNS problems.
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Kubernetes version 1.6 and above.
* The cluster must be configured to use the `kube-dns` addon.
{{% /capture %}}
{{% capture steps %}}
### Create a simple Pod to use as a test environment
Create a file named busybox.yaml with the following contents:
{{< code file="busybox.yaml" >}}
Then create a pod using this file and verify its status:
```shell
$ kubectl create -f busybox.yaml
pod "busybox" created
$ kubectl get pods busybox
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 <some-time>
```
Once that pod is running, you can exec `nslookup` in that environment.
If you see something like the following, DNS is working correctly.
```shell
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10
Name: kubernetes.default
Address 1: 10.0.0.1
```
If the `nslookup` command fails, check the following:
### Check the local DNS configuration first
Take a look inside the resolv.conf file.
(See [Inheriting DNS from the node](#inheriting-dns-from-the-node) and
[Known issues](#known-issues) below for more information)
```shell
$ kubectl exec busybox cat /etc/resolv.conf
```
Verify that the search path and name server are set up like the following
(note that search path may vary for different cloud providers):
```
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
nameserver 10.0.0.10
options ndots:5
```
Errors such as the following indicate a problem with the kube-dns add-on or
associated Services:
```
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10
nslookup: can't resolve 'kubernetes.default'
```
or
```
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'kubernetes.default'
```
### Check if the DNS pod is running
Use the `kubectl get pods` command to verify that the DNS pod is running.
```shell
$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
...
kube-dns-v19-ezo1y 3/3 Running 0 1h
...
```
If you see that no pod is running or that the pod has failed/completed, the DNS
add-on may not be deployed by default in your current environment and you will
have to deploy it manually.
### Check for Errors in the DNS pod
Use `kubectl logs` command to see logs for the DNS daemons.
```shell
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
```
See if there is any suspicious log. Letter '`W`', '`E`', '`F`' at the beginning
represent Warning, Error and Failure. Please search for entries that have these
as the logging level and use
[kubernetes issues](https://github.com/kubernetes/kubernetes/issues)
to report unexpected errors.
### Is DNS service up?
Verify that the DNS service is up by using the `kubectl get service` command.
```shell
$ kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
kube-dns 10.0.0.10 <none> 53/UDP,53/TCP 1h
...
```
If you have created the service or in the case it should be created by default
but it does not appear, see
[debugging services](/docs/tasks/debug-application-cluster/debug-service/) for
more information.
### Are DNS endpoints exposed?
You can verify that DNS endpoints are exposed by using the `kubectl get endpoints`
command.
```shell
$ kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 10.180.3.17:53,10.180.3.17:53 1h
```
If you do not see the endpoints, see endpoints section in the
[debugging services](/docs/tasks/debug-application-cluster/debug-service/) documentation.
For additional Kubernetes DNS examples, see the
[cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns)
in the Kubernetes GitHub repository.
## Known issues
Kubernetes installs do not configure the nodes' resolv.conf files to use the
cluster DNS by default, because that process is inherently distro-specific.
This should probably be implemented eventually.
Linux's libc is impossibly stuck ([see this bug from
2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just
3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to
consume 1 `nameserver` record and 3 `search` records. This means that if a
local installation already uses 3 `nameserver`s or uses more than 3 `search`es,
some of those settings will be lost. As a partial workaround, the node can run
`dnsmasq` which will provide more `nameserver` entries, but not more `search`
entries. You can also use kubelet's `--resolv-conf` flag.
If you are using Alpine version 3.3 or earlier as your base image, DNS may not
work properly owing to a known issue with Alpine.
Check [here](https://github.com/kubernetes/kubernetes/issues/30215)
for more information.
## Kubernetes Federation (Multiple Zone support)
Release 1.3 introduced Cluster Federation support for multi-site Kubernetes
installations. This required some minor (backward-compatible) changes to the
way the Kubernetes cluster DNS server processes DNS queries, to facilitate
the lookup of federated services (which span multiple Kubernetes clusters).
See the [Cluster Federation Administrators' Guide](/docs/concepts/cluster-administration/federation/)
for more details on Cluster Federation and multi-site support.
## References
- [DNS for Services and Pods](/docs/concepts/services-networking/dns-pod-service/)
- [Docs for the DNS cluster addon](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/README.md)
## What's next
- [Autoscaling the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/dns-horizontal-autoscaler.yaml
+33
View File
@@ -0,0 +1,33 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-dns-autoscaler
namespace: kube-system
labels:
k8s-app: kube-dns-autoscaler
spec:
selector:
matchLabels:
k8s-app: kube-dns-autoscaler
template:
metadata:
labels:
k8s-app: kube-dns-autoscaler
spec:
containers:
- name: autoscaler
image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.1
resources:
requests:
cpu: "20m"
memory: "10Mi"
command:
- /cluster-proportional-autoscaler
- --namespace=kube-system
- --configmap=kube-dns-autoscaler
- --target=<SCALE_TARGET>
# When cluster is using large nodes(with more cores), "coresPerReplica" should dominate.
# If using small nodes, "nodesPerReplica" should dominate.
- --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"min":1}}
- --logtostderr=true
- --v=2
content/en/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md
+240
View File
@@ -0,0 +1,240 @@
---
title: Autoscale the DNS Service in a Cluster
content_template: templates/task
---
{{% capture overview %}}
This page shows how to enable and configure autoscaling of the DNS service in a
Kubernetes cluster.
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Make sure the [DNS feature](/docs/concepts/services-networking/dns-pod-service/) itself is enabled.
* Kubernetes version 1.4.0 or later is recommended.
{{% /capture %}}
{{% capture steps %}}
## Determining whether DNS horizontal autoscaling is already enabled
List the Deployments in your cluster in the kube-system namespace:
kubectl get deployment --namespace=kube-system
The output is similar to this:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
...
kube-dns-autoscaler 1 1 1 1 ...
...
If you see "kube-dns-autoscaler" in the output, DNS horizontal autoscaling is
already enabled, and you can skip to
[Tuning autoscaling parameters](#tuning-autoscaling-parameters).
## Getting the name of your DNS Deployment or ReplicationController
List the Deployments in your cluster in the kube-system namespace:
kubectl get deployment --namespace=kube-system
The output is similar to this:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
...
kube-dns 1 1 1 1 ...
...
In Kubernetes versions earlier than 1.5 DNS is implemented using a
ReplicationController instead of a Deployment. So if you don't see kube-dns,
or a similar name, in the preceding output, list the ReplicationControllers in
your cluster in the kube-system namespace:
kubectl get rc --namespace=kube-system
The output is similar to this:
NAME DESIRED CURRENT READY AGE
...
kube-dns-v20 1 1 1 ...
...
## Determining your scale target
If you have a DNS Deployment, your scale target is:
Deployment/<your-deployment-name>
where <dns-deployment-name> is the name of your DNS Deployment. For example, if
your DNS Deployment name is kube-dns, your scale target is Deployment/kube-dns.
If you have a DNS ReplicationController, your scale target is:
ReplicationController/<your-rc-name>
where <your-rc-name> is the name of your DNS ReplicationController. For example,
if your DNS ReplicationController name is kube-dns-v20, your scale target is
ReplicationController/kube-dns-v20.
## Enabling DNS horizontal autoscaling
In this section, you create a Deployment. The Pods in the Deployment run a
container based on the `cluster-proportional-autoscaler-amd64` image.
Create a file named `dns-horizontal-autoscaler.yaml` with this content:
{{< code file="dns-horizontal-autoscaler.yaml" >}}
In the file, replace `<SCALE_TARGET>` with your scale target.
Go to the directory that contains your configuration file, and enter this
command to create the Deployment:
kubectl create -f dns-horizontal-autoscaler.yaml
The output of a successful command is:
deployment "kube-dns-autoscaler" created
DNS horizontal autoscaling is now enabled.
## Tuning autoscaling parameters
Verify that the kube-dns-autoscaler ConfigMap exists:
kubectl get configmap --namespace=kube-system
The output is similar to this:
NAME DATA AGE
...
kube-dns-autoscaler 1 ...
...
Modify the data in the ConfigMap:
kubectl edit configmap kube-dns-autoscaler --namespace=kube-system
Look for this line:
linear: '{"coresPerReplica":256,"min":1,"nodesPerReplica":16}'
Modify the fields according to your needs. The "min" field indicates the
minimal number of DNS backends. The actual number of backends number is
calculated using this equation:
replicas = max( ceil( cores * 1/coresPerReplica ) , ceil( nodes * 1/nodesPerReplica ) )
Note that the values of both `coresPerReplica` and `nodesPerReplica` are
integers.
The idea is that when a cluster is using nodes that have many cores,
`coresPerReplica` dominates. When a cluster is using nodes that have fewer
cores, `nodesPerReplica` dominates.
There are other supported scaling patterns. For details, see
[cluster-proportional-autoscaler](https://github.com/kubernetes-incubator/cluster-proportional-autoscaler).
## Disable DNS horizontal autoscaling
There are a few options for turning DNS horizontal autoscaling. Which option to
use depends on different conditions.
### Option 1: Scale down the kube-dns-autoscaler deployment to 0 replicas
This option works for all situations. Enter this command:
kubectl scale deployment --replicas=0 kube-dns-autoscaler --namespace=kube-system
The output is:
deployment "kube-dns-autoscaler" scaled
Verify that the replica count is zero:
kubectl get deployment --namespace=kube-system
The output displays 0 in the DESIRED and CURRENT columns:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
...
kube-dns-autoscaler 0 0 0 0 ...
...
### Option 2: Delete the kube-dns-autoscaler deployment
This option works if kube-dns-autoscaler is under your own control, which means
no one will re-create it:
kubectl delete deployment kube-dns-autoscaler --namespace=kube-system
The output is:
deployment "kube-dns-autoscaler" deleted
### Option 3: Delete the kube-dns-autoscaler manifest file from the master node
This option works if kube-dns-autoscaler is under control of the
[Addon Manager](https://git.k8s.io/kubernetes/cluster/addons/README.md)'s
control, and you have write access to the master node.
Sign in to the master node and delete the corresponding manifest file.
The common path for this kube-dns-autoscaler is:
/etc/kubernetes/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml
After the manifest file is deleted, the Addon Manager will delete the
kube-dns-autoscaler Deployment.
{{% /capture %}}
{{% capture discussion %}}
## Understanding how DNS horizontal autoscaling works
* The cluster-proportional-autoscaler application is deployed separately from
the DNS service.
* An autoscaler Pod runs a client that polls the Kubernetes API server for the
number of nodes and cores in the cluster.
* A desired replica count is calculated and applied to the DNS backends based on
the current schedulable nodes and cores and the given scaling parameters.
* The scaling parameters and data points are provided via a ConfigMap to the
autoscaler, and it refreshes its parameters table every poll interval to be up
to date with the latest desired scaling parameters.
* Changes to the scaling parameters are allowed without rebuilding or restarting
the autoscaler Pod.
* The autoscaler provides a controller interface to support two control
patterns: *linear* and *ladder*.
## Future enhancements
Control patterns, in addition to linear and ladder, that consider custom metrics
are under consideration as a future development.
Scaling of DNS backends based on DNS-specific metrics is under consideration as
a future development. The current implementation, which uses the number of nodes
and cores in cluster, is limited.
Support for custom metrics, similar to that provided by
[Horizontal Pod Autoscaling](/docs/tasks/run-application/horizontal-pod-autoscale/),
is under consideration as a future development.
{{% /capture %}}
{{% capture whatsnext %}}
Learn more about the
[implementation of cluster-proportional-autoscaler](https://github.com/kubernetes-incubator/cluster-proportional-autoscaler).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/encrypt-data.md
+201
View File
@@ -0,0 +1,201 @@
---
reviewers:
- smarterclayton
title: Encrypting Secret Data at Rest
content_template: templates/task
---
{{% capture overview %}}
This page shows how to enable and configure encryption of secret data at rest.
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Kubernetes version 1.7.0 or later is required
* etcd v3 or later is required
* Encryption at rest is alpha in 1.7.0 which means it may change without notice. Users may be required to decrypt their data prior to upgrading to 1.8.0.
{{% /capture %}}
{{% capture steps %}}
## Configuration and determining whether encryption at rest is already enabled
The `kube-apiserver` process accepts an argument `--experimental-encryption-provider-config`
that controls how API data is encrypted in etcd. An example configuration
is provided below.
## Understanding the encryption at rest configuration.
```yaml
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- identity: {}
- aesgcm:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- aescbc:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- secretbox:
keys:
- name: key1
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
```
Each `resources` array item is a separate config and contains a complete configuration. The
`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`)
that should be encrypted. The `providers` array is an ordered list of the possible encryption
providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided,
but not both in the same item).
The first provider in the list is used to encrypt resources going into storage. When reading
resources from storage each provider that matches the stored data attempts to decrypt the data in
order. If no provider can read the stored data due to a mismatch in format or secret key, an error
is returned which prevents clients from accessing that resource.
**IMPORTANT:** If any resource is not readable via the encryption config (because keys were changed),
the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to
read that resource will fail until it is deleted or a valid decryption key is provided.
### Providers:
Name | Encryption | Strength | Speed | Key Length | Other Considerations
-----|------------|----------|-------|------------|---------------------
`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
`aescbc` | AES-CBC with PKCS#7 padding | Strongest | Fast | 32-byte | The recommended choice for encryption at rest but may be slightly slower than `secretbox`.
`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review.
`aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented.
`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/)
Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
is the first provider, the first key is used for encryption.
## Encrypting your data
Create a new encryption config file:
```yaml
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>
- identity: {}
```
To create a new secret perform the following steps:
1. Generate a 32 byte random key and base64 encode it. If you're on Linux or Mac OS X, run the following command:
```
head -c 32 /dev/urandom | base64
```
2. Place that value in the secret field.
3. Set the `--experimental-encryption-provider-config` flag on the `kube-apiserver` to point to the location of the config file.
4. Restart your API server.
**IMPORTANT:** Your config file contains keys that can decrypt content in etcd, so you must properly restrict permissions on your masters so only the user who runs the kube-apiserver can read it.
## Verifying that data is encrypted
Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or
updated secret should be encrypted when stored. To check, you can use the `etcdctl` command line
program to retrieve the contents of your secret.
1. Create a new secret called `secret1` in the `default` namespace:
```
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
```
2. Using the etcdctl commandline, read that secret out of etcd:
```
   ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C
```
where `[...]` must be the additional arguments for connecting to the etcd server.
3. Verify the stored secret is prefixed with `k8s:enc:aescbc:v1:` which indicates the `aescbc` provider has encrypted the resulting data.
4. Verify the secret is correctly decrypted when retrieved via the API:
```
kubectl describe secret secret1 -n default
```
should match `mykey: mydata`
## Ensure all secrets are encrypted
Since secrets are encrypted on write, performing an update on a secret will encrypt that content.
```
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
The command above reads all secrets and then updates them to apply server side encryption.
If an error occurs due to a conflicting write, retry the command.
For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
## Rotating a decryption key
Changing the secret without incurring downtime requires a multi step operation, especially in
the presence of a highly available deployment where multiple `kube-apiserver` processes are running.
1. Generate a new key and add it as the second key entry for the current provider on all servers
2. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key
3. Make the new key the first entry in the `keys` array so that it is used for encryption in the config
4. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key
5. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all existing secrets with the new key
6. Remove the old decryption key from the config after you back up etcd with the new key in use and update all secrets
With a single `kube-apiserver`, step 2 may be skipped.
## Decrypting all data
To disable encryption at rest place the `identity` provider as the first entry in the config:
```yaml
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- identity: {}
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>
```
and restart all `kube-apiserver` processes. Then run the command `kubectl get secrets --all-namespaces -o json | kubectl replace -f -`
to force all secrets to be decrypted.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/extended-resource-node.md
+210
View File
@@ -0,0 +1,210 @@
---
title: Advertise Extended Resources for a Node
content_template: templates/task
---
{{% capture overview %}}
This page shows how to specify extended resources for a Node.
Extended resources allow cluster administrators to advertise node-level
resources that would otherwise be unknown to Kubernetes.
{{< feature-state state="stable" >}}
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Get the names of your Nodes
```shell
kubectl get nodes
```
Choose one of your Nodes to use for this exercise.
## Advertise a new extended resource on one of your Nodes
To advertise a new extended resource on a Node, send an HTTP PATCH request to
the Kubernetes API server. For example, suppose one of your Nodes has four dongles
attached. Here's an example of a PATCH request that advertises four dongle resources
for your Node.
```shell
PATCH /api/v1/nodes/<your-node-name>/status HTTP/1.1
Accept: application/json
Content-Type: application/json-patch+json
Host: k8s-master:8080
[
{
"op": "add",
"path": "/status/capacity/example.com~1dongle",
"value": "4"
}
]
```
Note that Kubernetes does not need to know what a dongle is or what a dongle is for.
The preceding PATCH request just tells Kubernetes that your Node has four things that
you call dongles.
Start a proxy, so that you can easily send requests to the Kubernetes API server:
```
kubectl proxy
```
In another command window, send the HTTP PATCH request.
Replace `<your-node-name>` with the name of your Node:
```shell
curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/example.com~1dongle", "value": "4"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status
```
**Note**: In the preceding request, `~1` is the encoding for the character / in
the patch path. The operation path value in JSON-Patch is interpreted as a
JSON-Pointer. For more details, see
[IETF RFC 6901](https://tools.ietf.org/html/rfc6901), section 3.
The output shows that the Node has a capacity of 4 dongles:
```
"capacity": {
"alpha.kubernetes.io/nvidia-gpu": "0",
"cpu": "2",
"memory": "2049008Ki",
"example.com/dongle": "4",
```
Describe your Node:
```
kubectl describe node <your-node-name>
```
Once again, the output shows the dongle resource:
```yaml
Capacity:
alpha.kubernetes.io/nvidia-gpu: 0
cpu: 2
memory: 2049008Ki
example.com/dongle: 4
```
Now, application developers can create Pods that request a certain
number of dongles. See
[Assign Extended Resources to a Container](/docs/tasks/configure-pod-container/extended-resource/).
## Discussion
Extended resources are similar to memory and CPU resources. For example,
just as a Node has a certain amount of memory and CPU to be shared by all components
running on the Node, it can have a certain number of dongles to be shared
by all components running on the Node. And just as application developers
can create Pods that request a certain amount of memory and CPU, they can
create Pods that request a certain number of dongles.
Extended resources are opaque to Kubernetes; Kubernetes does not
know anything about what they are. Kubernetes knows only that a Node
has a certain number of them. Extended resources must be advertised in integer
amounts. For example, a Node can advertise four dongles, but not 4.5 dongles.
### Storage example
Suppose a Node has 800 GiB of a special kind of disk storage. You could
create a name for the special storage, say example.com/special-storage.
Then you could advertise it in chunks of a certain size, say 100 GiB. In that case,
your Node would advertise that it has eight resources of type
example.com/special-storage.
```yaml
Capacity:
...
example.com/special-storage: 8
```
If you want to allow arbitrary requests for special storage, you
could advertise special storage in chunks of size 1 byte. In that case, you would advertise
800Gi resources of type example.com/special-storage.
```yaml
Capacity:
...
example.com/special-storage: 800Gi
```
Then a Container could request any number of bytes of special storage, up to 800Gi.
## Clean up
Here is a PATCH request that removes the dongle advertisement from a Node.
```shell
PATCH /api/v1/nodes/<your-node-name>/status HTTP/1.1
Accept: application/json
Content-Type: application/json-patch+json
Host: k8s-master:8080
[
{
"op": "remove",
"path": "/status/capacity/example.com~1dongle",
}
]
```
Start a proxy, so that you can easily send requests to the Kubernetes API server:
```
kubectl proxy
```
In another command window, send the HTTP PATCH request.
Replace `<your-node-name>` with the name of your Node:
```shell
curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "remove", "path": "/status/capacity/example.com~1dongle"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status
```
Verify that the dongle advertisement has been removed:
```
kubectl describe node <your-node-name> | grep dongle
```
{{% /capture %}}
{{% capture whatsnext %}}
### For application developers
* [Assign Extended Resources to a Container](/docs/tasks/configure-pod-container/extended-resource/)
### For cluster administrators
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md
+62
View File
@@ -0,0 +1,62 @@
---
reviewers:
- davidopp
- filipg
- piosz
title: Guaranteed Scheduling For Critical Add-On Pods
---
{{< toc >}}
## Overview
In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a master machine
there are a number of add-ons which, for various reasons, must run on a regular cluster node (rather than the Kubernetes master).
Some of these add-ons are critical to a fully functional cluster, such as Heapster, DNS, and UI.
A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade)
and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space
vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason).
## Rescheduler: guaranteed scheduling of critical add-ons
**Rescheduler is deprecated as of Kubernetes 1.10 and will be removed in version 1.11 in
accordance with the [deprecation policy](/docs/reference/deprecation-policy) for beta features.**
**To avoid eviction of critical pods, you must
[enable priorities in scheduler](/docs/concepts/configuration/pod-priority-preemption/)
before upgrading to Kubernetes 1.10 or higher.**
Rescheduler ensures that critical add-ons are always scheduled
(assuming the cluster has enough resources to run the critical add-on pods in the absence of regular pods).
If the scheduler determines that no node has enough free resources to run the critical add-on pod
given the pods that are already running in the cluster
(indicated by critical add-on pod's pod condition PodScheduled set to false, the reason set to Unschedulable)
the rescheduler tries to free up space for the add-on by evicting some pods; then the scheduler will schedule the add-on pod.
To avoid situation when another pod is scheduled into the space prepared for the critical add-on,
the chosen node gets a temporary taint "CriticalAddonsOnly" before the eviction(s)
(see [more details](https://git.k8s.io/community/contributors/design-proposals/scheduling/taint-toleration-dedicated.md)).
Each critical add-on has to tolerate it,
while the other pods shouldn't tolerate the taint. The taint is removed once the add-on is successfully scheduled.
*Warning:* currently there is no guarantee which node is chosen and which pods are being killed
in order to schedule critical pods, so if rescheduler is enabled your pods might be occasionally
killed for this purpose. Please ensure that rescheduler is not enabled along with priorities & preemptions in default-scheduler as rescheduler is oblivious to priorities and it may evict high priority pods, instead of low priority ones.
## Config
Rescheduler doesn't have any user facing configuration (component config) or API.
### Marking pod as critical when using Rescheduler.
To be considered critical, the pod has to run in the `kube-system` namespace (configurable via flag) and
* have the `scheduler.alpha.kubernetes.io/critical-pod` annotation set to empty string, and
* have the PodSpec's `tolerations` field set to `[{"key":"CriticalAddonsOnly", "operator":"Exists"}]`.
The first one marks a pod a critical. The second one is required by Rescheduler algorithm.
### Marking pod as critical when priorites are enabled.
To be considered critical, the pod has to run in the `kube-system` namespace (configurable via flag) and
* Have the priorityClass set as "system-cluster-critical" or "system-node-critical", the latter being the highest for entire cluster and `scheduler.alpha.kubernetes.io/critical-pod` annotation set to empty string(This will be deprecated too).
content/en/docs/tasks/administer-cluster/highly-available-master.md
+156
View File
@@ -0,0 +1,156 @@
---
reviewers:
- jszczepkowski
title: Set up High-Availability Kubernetes Masters
---
{{< toc >}}
Kubernetes version 1.5 adds alpha support for replicating Kubernetes masters in `kube-up` or `kube-down` scripts for Google Compute Engine.
This document describes how to use kube-up/down scripts to manage highly available (HA) masters and how HA masters are implemented for use with GCE.
## Starting an HA-compatible cluster
To create a new HA-compatible cluster, you must set the following flags in your `kube-up` script:
* `MULTIZONE=true` - to prevent removal of master replicas kubelets from zones different than server's default zone.
Required if you want to run master replicas in different zones, which is recommended.
* `ENABLE_ETCD_QUORUM_READS=true` - to ensure that reads from all API servers will return most up-to-date data.
If true, reads will be directed to leader etcd replica.
Setting this value to true is optional: reads will be more reliable but will also be slower.
Optionally, you can specify a GCE zone where the first master replica is to be created.
Set the following flag:
* `KUBE_GCE_ZONE=zone` - zone where the first master replica will run.
The following sample command sets up a HA-compatible cluster in the GCE zone europe-west1-b:
```shell
$ MULTIZONE=true KUBE_GCE_ZONE=europe-west1-b ENABLE_ETCD_QUORUM_READS=true ./cluster/kube-up.sh
```
Note that the commands above create a cluster with one master;
however, you can add new master replicas to the cluster with subsequent commands.
## Adding a new master replica
After you have created an HA-compatible cluster, you can add master replicas to it.
You add master replicas by using a `kube-up` script with the following flags:
* `KUBE_REPLICATE_EXISTING_MASTER=true` - to create a replica of an existing
master.
* `KUBE_GCE_ZONE=zone` - zone where the master replica will run.
Must be in the same region as other replicas' zones.
You don't need to set the `MULTIZONE` or `ENABLE_ETCD_QUORUM_READS` flags,
as those are inherited from when you started your HA-compatible cluster.
The following sample command replicates the master on an existing HA-compatible cluster:
```shell
$ KUBE_GCE_ZONE=europe-west1-c KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
```
## Removing a master replica
You can remove a master replica from an HA cluster by using a `kube-down` script with the following flags:
* `KUBE_DELETE_NODES=false` - to restrain deletion of kubelets.
* `KUBE_GCE_ZONE=zone` - the zone from where master replica will be removed.
* `KUBE_REPLICA_NAME=replica_name` - (optional) the name of master replica to remove.
If empty: any replica from the given zone will be removed.
The following sample command removes a master replica from an existing HA cluster:
```shell
$ KUBE_DELETE_NODES=false KUBE_GCE_ZONE=europe-west1-c ./cluster/kube-down.sh
```
## Handling master replica failures
If one of the master replicas in your HA cluster fails,
the best practice is to remove the replica from your cluster and add a new replica in the same zone.
The following sample commands demonstrate this process:
1. Remove the broken replica:
```shell
$ KUBE_DELETE_NODES=false KUBE_GCE_ZONE=replica_zone KUBE_REPLICA_NAME=replica_name ./cluster/kube-down.sh
```
<ol start="2"><li>Add a new replica in place of the old one:</li></ol>
```shell
$ KUBE_GCE_ZONE=replica-zone KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
```
## Best practices for replicating masters for HA clusters
* Try to place master replicas in different zones. During a zone failure, all masters placed inside the zone will fail.
To survive zone failure, also place nodes in multiple zones
(see [multiple-zones](/docs/admin/multiple-zones/) for details).
* Do not use a cluster with two master replicas. Consensus on a two-replica cluster requires both replicas running when changing persistent state.
As a result, both replicas are needed and a failure of any replica turns cluster into majority failure state.
A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster.
* When you add a master replica, cluster state (etcd) is copied to a new instance.
If the cluster is large, it may take a long time to duplicate its state.
This operation may be sped up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration)
(we are considering adding support for etcd data dir migration in future).
## Implementation notes
![ha-master-gce](/images/docs/ha-master-gce.png)
### Overview
Each of master replicas will run the following components in the following mode:
* etcd instance: all instances will be clustered together using consensus;
* API server: each server will talk to local etcd - all API servers in the cluster will be available;
* controllers, scheduler, and cluster auto-scaler: will use lease mechanism - only one instance of each of them will be active in the cluster;
* add-on manager: each manager will work independently trying to keep add-ons in sync.
In addition, there will be a load balancer in front of API servers that will route external and internal traffic to them.
### Load balancing
When starting the second master replica, a load balancer containing the two replicas will be created
and the IP address of the first replica will be promoted to IP address of load balancer.
Similarly, after removal of the penultimate master replica, the load balancer will be removed and its IP address will be assigned to the last remaining replica.
Please note that creation and removal of load balancer are complex operations and it may take some time (~20 minutes) for them to propagate.
### Master service & kubelets
Instead of trying to keep an up-to-date list of Kubernetes apiserver in the Kubernetes service,
the system directs all traffic to the external IP:
* in one master cluster the IP points to the single master,
* in multi-master cluster the IP points to the load balancer in-front of the masters.
Similarly, the external IP will be used by kubelets to communicate with master.
### Master certificates
Kubernetes generates Master TLS certificates for the external public IP and local IP for each replica.
There are no certificates for the ephemeral public IP for replicas;
to access a replica via its ephemeral public IP, you must skip TLS verification.
### Clustering etcd
To allow etcd clustering, ports needed to communicate between etcd instances will be opened (for inside cluster communication).
To make such deployment secure, communication between etcd instances is authorized using SSL.
## Additional reading
[Automated HA master deployment - design doc](https://git.k8s.io/community/contributors/design-proposals/cluster-lifecycle/ha_master.md)
content/en/docs/tasks/administer-cluster/ip-masq-agent.md
+113
View File
@@ -0,0 +1,113 @@
---
title: IP Masquerade Agent User Guide
content_template: templates/task
---
{{% capture overview %}}
This page shows how to configure and enable the ip-masq-agent.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture discussion %}}
## IP Masquerade Agent User Guide
The ip-masq-agent configures iptables rules to hide a pod's IP address behind the cluster node's IP address. This is typically done when sending traffic to destinations outside the cluster's pod [CIDR](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) range.
### **Key Terms**
* **NAT (Network Address Translation)**
Is a method of remapping one IP address to another by modifying either the source and/or destination address information in the IP header. Typically performed by a device doing IP routing.
* **Masquerading**
A form of NAT that is typically used to perform a many to one address translation, where multiple source IP addresses are masked behind a single address, which is typically the device doing the IP routing. In Kubernetes this is the Node's IP address.
* **CIDR (Classless Inter-Domain Routing)**
Based on the variable-length subnet masking, allows specifying arbitrary-length prefixes. CIDR introduced a new method of representation for IP addresses, now commonly known as **CIDR notation**, in which an address or routing prefix is written with a suffix indicating the number of bits of the prefix, such as 192.168.2.0/24.
* **Link Local**
A link-local address is a network address that is valid only for communications within the network segment or the broadcast domain that the host is connected to. Link-local addresses for IPv4 are defined in the address block 169.254.0.0/16 in CIDR notation.
The ip-masq-agent configures iptables rules to handle masquerading node/pod IP addresses when sending traffic to destinations outside the cluster node's IP and the Cluster IP range. This essentially hides pod IP addresses behind the cluster node's IP address. In some environments, traffic to "external" addresses must come from a known machine address. For example, in Google Cloud, any traffic to the internet must come from a VM's IP. When containers are used, as in Google Kubernetes Engine, the Pod IP will be rejected for egress. To avoid this, we must hide the Pod IP behind the VM's own IP address - generally known as "masquerade". By default, the agent is configured to treat the three private IP ranges specified by [RFC 1918](https://tools.ietf.org/html/rfc1918) as non-masquerade [CIDR](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing). These ranges are 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. The agent will also treat link-local (169.254.0.0/16) as a non-masquerade CIDR by default. The agent is configured to reload its configuration from the location */etc/config/ip-masq-agent* every 60 seconds, which is also configurable.
![masq/non-masq example](/images/docs/ip-masq.png)
The agent configuration file must be written in YAML or JSON syntax, and may contain three optional keys:
* **nonMasqueradeCIDRs:** A list of strings in [CIDR](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) notation that specify the non-masquerade ranges.
* **masqLinkLocal:** A Boolean (true / false) which indicates whether to masquerade traffic to the link local prefix 169.254.0.0/16. False by default.
* **resyncInterval:** An interval at which the agent attempts to reload config from disk. e.g. '30s' where 's' is seconds, 'ms' is milliseconds etc...
Traffic to 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) ranges will NOT be masqueraded. Any other traffic (assumed to be internet) will be masqueraded. An example of a local destination from a pod could be its Node's IP address as well as another node's address or one of the IP addresses in Cluster's IP range. Any other traffic will be masqueraded by default. The below entries show the default set of rules that are applied by the ip-masq-agent:
```
iptables -t nat -L IP-MASQ-AGENT
RETURN all -- anywhere 169.254.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 10.0.0.0/8 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 172.16.0.0/12 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 192.168.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
MASQUERADE all -- anywhere anywhere /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
```
By default, in GCE/Google Kubernetes Engine starting with Kubernetes version 1.7.0, if network policy is enabled or you are using a cluster CIDR not in the 10.0.0.0/8 range, the ip-masq-agent will run in your cluster. If you are running in another environment, you can add the ip-masq-agent [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to your cluster:
{{% /capture %}}
{{% capture steps %}}
## Create an ip-masq-agent
To create an ip-masq-agent, run the following kubectl command:
`
kubectl create -f https://raw.githubusercontent.com/kubernetes-incubator/ip-masq-agent/master/ip-masq-agent.yaml
`
You must also apply the appropriate node label to any nodes in your cluster that you want the agent to run on.
`
kubectl label nodes my-node beta.kubernetes.io/masq-agent-ds-ready=true
`
More information can be found in the ip-masq-agent documentation [here](https://github.com/kubernetes-incubator/ip-masq-agent)
In most cases, the default set of rules should be sufficient; however, if this is not the case for your cluster, you can create and apply a [ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/) to customize the IP ranges that are affected. For example, to allow only 10.0.0.0/8 to be considered by the ip-masq-agent, you can create the following [ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/) in a file called "config".
**Note:** It is important that the file is called config since, by default, that will be used as the key for lookup by the ip-masq-agent:
```
nonMasqueradeCIDRs:
- 10.0.0.0/8
resyncInterval: 60s
```
Run the following command to add the config map to your cluster:
```
kubectl create configmap ip-masq-agent --from-file=config --namespace=kube-system
```
This will update a file located at */etc/config/ip-masq-agent* which is periodically checked every *resyscInterval* and applied to the cluster node.
After the resync interval has expired, you should see the iptables rules reflect your changes:
```
iptables -t nat -L IP-MASQ-AGENT
Chain IP-MASQ-AGENT (1 references)
target prot opt source destination
RETURN all -- anywhere 169.254.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 10.0.0.0/8 /* ip-masq-agent: cluster-local
MASQUERADE all -- anywhere anywhere /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
```
By default, the link local range (169.254.0.0/16) is also handled by the ip-masq agent, which sets up the appropriate iptables rules. To have the ip-masq-agent ignore link local, you can set *masqLinkLocal* to true in the config map.
```
nonMasqueradeCIDRs:
- 10.0.0.0/8
resyncInterval: 60s
masqLinkLocal: true
```
{{% /capture %}}
content/en/docs/tasks/administer-cluster/kms-provider.md
+182
View File
@@ -0,0 +1,182 @@
---
approvers:
- smarterclayton
title: Using a KMS provider for data encryption
content_template: templates/task
---
{{% capture overview %}}
This page shows how to configure a Key Management Service (KMS) provider and plugin to enable secret data encryption.
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Kubernetes version 1.10.0 or later is required
* etcd v3 or later is required
{{< feature-state for_k8s_version="v1.10" state="alpha" >}}
{{% /capture %}}
{{% capture steps %}}
The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd. The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption. The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS. The KMS provider uses gRPC to communicate with a specific KMS
plugin. The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s) as the Kubernetes master(s), is responsible for all communication with the remote KMS.
## Configuring the KMS provider
To configure a KMS provider on the API server, include a provider of type ```kms``` in the providers array in the encryption configuration file and set the following properties:
* `name`: Display name of the KMS plugin.
* `endpoint`: Listen address of the gRPC server (KMS plugin). The endpoint is a UNIX domain socket.
* `cachesize`: Number of data encryption keys (DEKs) to be cached in the clear. When cached, DEKs can be used without another call to the KMS; whereas DEKs that are not cached require a call to the KMS to unwrap..
See [Understanding the encryption at rest configuration.](/docs/tasks/administer-cluster/encrypt-data)
## Implementing a KMS plugin
To implement a KMS plugin, you can develop a new plugin gRPC server or enable a KMS plugin already provided by your cloud provider. You then integrate the plugin with the remote KMS and deploy it on the Kubernetes master.
### Enabling the KMS supported by your cloud provider
Refer to your cloud provider for instructions on enabling the cloud provider-specific KMS plugin.
### Developing a KMS plugin gRPC server
You can develop a KMS plugin gRPC server using a stub file available for Go. For other languages, you use a proto file to create a stub file that you can use to develop the gRPC server code.
* Using Go: Use the functions and data structures in the stub file: [service.pb.go](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/service.pb.go) to develop the gRPC server code
* Using languages other than Go: Use the protoc compiler with the proto file: [service.proto](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/service.proto) to generate a stub file for the specific language
Then use the functions and data structures in the stub file to develop the server code.
**Notes:**
* kms plugin version: `v1beta1`
In response to procedure call Version, a compatible KMS plugin should return v1beta1 as VersionResponse.version
* message version: `v1beta1`
All messages from KMS provider have the version field set to current version v1beta1
* protocol: UNIX domain socket (`unix`)
The gRPC server should listen at UNIX domain socket
### Integrating a KMS plugin with the remote KMS
The KMS plugin can communicate with the remote KMS using any protocol supported by the KMS.
All configuration data, including authentication credentials the KMS plugin uses to communicate with the remote KMS,
are stored and managed by the KMS plugin independently. The KMS plugin can encode the ciphertext with additional metadata that may be required before sending it to the KMS for decryption.
### Deploying the KMS plugin
Ensure that the KMS plugin runs on the same host(s) as the Kubernetes master(s).
## Encrypting your data with the KMS provider
To encrypt the data:
1. Create a new encryption configuration file using the appropriate properties for the `kms` provider:
```yaml
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- kms:
name: myKmsPlugin
endpoint: unix:///tmp/socketfile.sock
cachesize: 100
- identity: {}
```
2. Set the `--experimental-encryption-provider-config` flag on the kube-apiserver to point to the location of the configuration file.
3. Restart your API server.
## Verifying that the data is encrypted
Data is encrypted when written to etcd. After restarting your kube-apiserver, any newly created or updated secret should be encrypted when stored. To verify, you can use the etcdctl command line program to retrieve the contents of your secret.
1. Create a new secret called secret1 in the default namespace:
```
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
```
2. Using the etcdctl command line, read that secret out of etcd:
```
ETCDCTL_API=3 etcdctl get /kubernetes.io/secrets/default/secret1 [...] | hexdump -C
```
where `[...]` must be the additional arguments for connecting to the etcd server.
3. Verify the stored secret is prefixed with `k8s:enc:kms:v1:`, which indicates that the `kms` provider has encrypted the resulting data.
4. Verify that the secret is correctly decrypted when retrieved via the API:
```
kubectl describe secret secret1 -n default
```
should match `mykey: mydata`
## Ensuring all secrets are encrypted
Because secrets are encrypted on write, performing an update on a secret encrypts that content.
The following command reads all secrets and then updates them to apply server side encryption. If an error occurs due to a conflicting write, retry the command. For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
```
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
## Switching from a local encryption provider to the KMS provider
To switch from a local encryption provider to the `kms` provider and re-encrypt all of the secrets:
1. Add the `kms` provider as the first entry in the configuration file as shown in the following example.
```yaml
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- kms:
name : myKmsPlugin
endpoint: unix:///tmp/socketfile.sock
cachesize: 100
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>
```
2. Restart all kube-apiserver processes.
3. Run the following command to force all secrets to be re-encrypted using the `kms` provider.
```
kubectl get secrets --all-namespaces -o json| kubectl replace -f -
```
## Disabling encryption at rest
To disable encryption at rest:
1. Place the `identity` provider as the first entry in the configuration file:
```yaml
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- identity: {}
- kms:
name : myKmsPlugin
endpoint: unix:///tmp/socketfile.sock
cachesize: 100
```
2. Restart all kube-apiserver processes.
3. Run the following command to force all secrets to be decrypted.
```
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
{{% /capture %}}
content/en/docs/tasks/administer-cluster/kube-router-network-policy.md
+25
View File
@@ -0,0 +1,25 @@
---
reviewers:
- murali-reddy
title: Use Kube-router for NetworkPolicy
content_template: templates/task
---
{{% capture overview %}}
This page shows how to use [Kube-router](https://github.com/cloudnativelabs/kube-router) for NetworkPolicy.
{{% /capture %}}
{{% capture prerequisites %}}
You need to have a Kubernetes cluster running. If you do not already have a cluster, you can create one by using any of the cluster installers like Kops, Bootkube, Kubeadm etc.
{{% /capture %}}
{{% capture steps %}}
## Installing Kube-router addon
The Kube-router Addon comes with a Network Policy Controller that watches Kubernetes API server for any NetworkPolicy and pods updated and configures iptables rules and ipsets to allow or block traffic as directed by the policies. Please follow the [trying Kube-router with cluster installers](https://github.com/cloudnativelabs/kube-router/tree/master/Documentation#try-kube-router-with-cluster-installers) guide to install Kube-router addon.
{{% /capture %}}
{{% capture whatsnext %}}
Once you have installed the Kube-router addon, you can follow the [Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/) to try out Kubernetes NetworkPolicy.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/kubelet-config-file.md
+82
View File
@@ -0,0 +1,82 @@
---
reviewers:
- mtaufen
- dawnchen
title: Set Kubelet parameters via a config file
content_template: templates/task
---
{{% capture overview %}}
{{< feature-state state="beta" >}}
A subset of the Kubelet's configuration parameters may be
set via an on-disk config file, as a substitute for command-line flags.
This functionality is considered beta in v1.10.
Providing parameters via a config file is the recommended approach because
it simplifies node deployment and configuration management.
{{% /capture %}}
{{% capture prerequisites %}}
- A v1.10 or higher Kubelet binary must be installed for beta functionality.
{{% /capture %}}
{{% capture steps %}}
## Create the config file
The subset of the Kubelet's configuration that can be configured via a file
is defined by the `KubeletConfiguration` struct
[here (v1beta1)](https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go).
The configuration file must be a JSON or YAML representation of the parameters
in this struct. Make sure the Kubelet has read permissions on the file.
Here is an example of what this file might look like:
```
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
evictionHard:
memory.available: "200Mi"
```
In the example, the Kubelet is configured to evict Pods when available memory drops below 200Mi.
All other Kubelet configuration values are left at their built-in defaults, unless overridden
by flags. Command line flags which target the same value as a config file will override that value.
For a trick to generate a configuration file from a live node, see
[Reconfigure a Node's Kubelet in a Live Cluster](/docs/tasks/administer-cluster/reconfigure-kubelet).
## Start a Kubelet process configured via the config file
Start the Kubelet with the `--config` flag set to the path of the Kubelet's config file.
The Kubelet will then load its config from this file.
Note that command line flags which target the same value as a config file will override that value.
This helps ensure backwards compatibility with the command-line API.
Note that relative file paths in the Kubelet config file are resolved relative to the
location of the Kubelet config file, whereas relative paths in command line flags are resolved
relative to the Kubelet's current working directory.
Note that some default values differ between command-line flags and the Kubelet config file.
If `--config` is provided and the values are not specified via the command line, the
defaults for the `KubeletConfiguration` version apply.
In the above example, this version is `kubelet.config.k8s.io/v1beta1`.
{{% /capture %}}
{{% capture discussion %}}
## Relationship to Dynamic Kubelet Config
If you are using the [Dynamic Kubelet Configuration](/docs/tasks/administer-cluster/reconfigure-kubelet)
feature, the combination of configuration provided via `--config` and any flags which override these values
is considered the default "last known good" configuration by the automatic rollback mechanism.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/limit-storage-consumption.md
+92
View File
@@ -0,0 +1,92 @@
---
title: Limit Storage Consumption
content_template: templates/task
---
{{% capture overview %}}
This example demonstrates an easy way to limit the amount of storage consumed in a namespace.
The following resources are used in the demonstration: [ResourceQuota](/docs/concepts/policy/resource-quotas/),
[LimitRange](/docs/tasks/administer-cluster/memory-default-namespace/),
and [PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/).
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Scenario: Limiting Storage Consumption
The cluster-admin is operating a cluster on behalf of a user population and the admin wants to control
how much storage a single namespace can consume in order to control cost.
The admin would like to limit:
1. The number of persistent volume claims in a namespace
2. The amount of storage each claim can request
3. The amount of cumulative storage the namespace can have
## LimitRange to limit requests for storage
Adding a `LimitRange` to a namespace enforces storage request sizes to a minimum and maximum. Storage is requested
via `PersistentVolumeClaim`. The admission controller that enforces limit ranges will reject any PVC that is above or below
the values set by the admin.
In this example, a PVC requesting 10Gi of storage would be rejected because it exceeds the 2Gi max.
```
apiVersion: v1
kind: LimitRange
metadata:
name: storagelimits
spec:
limits:
- type: PersistentVolumeClaim
max:
storage: 2Gi
min:
storage: 1Gi
```
Minimum storage requests are used when the underlying storage provider requires certain minimums. For example,
AWS EBS volumes have a 1Gi minimum requirement.
## StorageQuota to limit PVC count and cumulative storage capacity
Admins can limit the number of PVCs in a namespace as well as the cumulative capacity of those PVCs. New PVCs that exceed
either maximum value will be rejected.
In this example, a 6th PVC in the namespace would be rejected because it exceeds the maximum count of 5. Alternatively,
a 5Gi maximum quota when combined with the 2Gi max limit above, cannot have 3 PVCs where each has 2Gi. That would be 6Gi requested
for a namespace capped at 5Gi.
```
apiVersion: v1
kind: ResourceQuota
metadata:
name: storagequota
spec:
hard:
persistentvolumeclaims: "5"
requests.storage: "5Gi"
```
{{% /capture %}}
{{% capture discussion %}}
## Summary
A limit range can put a ceiling on how much storage is requested while a resource quota can effectively cap the storage
consumed by a namespace through claim counts and cumulative storage capacity. The allows a cluster-admin to plan their
cluster's storage budget without risk of any one project going over their allotment.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/memory-available.sh
+31
View File
@@ -0,0 +1,31 @@
#!/bin/bash
#!/usr/bin/env bash
# This script reproduces what the kubelet does
# to calculate memory.available relative to root cgroup.
# current memory usage
memory_capacity_in_kb=$(cat /proc/meminfo | grep MemTotal | awk '{print $2}')
memory_capacity_in_bytes=$((memory_capacity_in_kb * 1024))
memory_usage_in_bytes=$(cat /sys/fs/cgroup/memory/memory.usage_in_bytes)
memory_total_inactive_file=$(cat /sys/fs/cgroup/memory/memory.stat | grep total_inactive_file | awk '{print $2}')
memory_working_set=${memory_usage_in_bytes}
if [ "$memory_working_set" -lt "$memory_total_inactive_file" ];
then
memory_working_set=0
else
memory_working_set=$((memory_usage_in_bytes - memory_total_inactive_file))
fi
memory_available_in_bytes=$((memory_capacity_in_bytes - memory_working_set))
memory_available_in_kb=$((memory_available_in_bytes / 1024))
memory_available_in_mb=$((memory_available_in_kb / 1024))
echo "memory.capacity_in_bytes $memory_capacity_in_bytes"
echo "memory.usage_in_bytes $memory_usage_in_bytes"
echo "memory.total_inactive_file $memory_total_inactive_file"
echo "memory.working_set $memory_working_set"
echo "memory.available_in_bytes $memory_available_in_bytes"
echo "memory.available_in_kb $memory_available_in_kb"
echo "memory.available_in_mb $memory_available_in_mb"
content/en/docs/tasks/administer-cluster/memory-constraint-namespace.md
+272
View File
@@ -0,0 +1,272 @@
---
title: Configure Minimum and Maximum Memory Constraints for a Namespace
content_template: templates/task
---
{{% capture overview %}}
This page shows how to set minimum and maximum values for memory used by Containers
running in a namespace. You specify minimum and maximum memory values in a
[LimitRange](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#limitrange-v1-core)
object. If a Pod does not meet the constraints imposed by the LimitRange,
it cannot be created in the namespace.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
Each node in your cluster must have at least 1 GiB of memory.
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace constraints-mem-example
```
## Create a LimitRange and a Pod
Here's the configuration file for a LimitRange:
{{< code file="memory-constraints.yaml" >}}
Create the LimitRange:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-constraints.yaml --namespace=constraints-mem-example
```
View detailed information about the LimitRange:
```shell
kubectl get limitrange mem-min-max-demo-lr --namespace=constraints-mem-example --output=yaml
```
The output shows the minimum and maximum memory constraints as expected. But
notice that even though you didn't specify default values in the configuration
file for the LimitRange, they were created automatically.
```
limits:
- default:
memory: 1Gi
defaultRequest:
memory: 1Gi
max:
memory: 1Gi
min:
memory: 500Mi
type: Container
```
Now whenever a Container is created in the constraints-mem-example namespace, Kubernetes
performs these steps:
* If the Container does not specify its own memory request and limit, assign the default
memory request and limit to the Container.
* Verify that the Container has a memory request that is greater than or equal to 500 MiB.
* Verify that the Container has a memory limit that is less than or equal to 1 GiB.
Here's the configuration file for a Pod that has one Container. The Container manifest
specifies a memory request of 600 MiB and a memory limit of 800 MiB. These satisfy the
minimum and maximum memory constraints imposed by the LimitRange.
{{< code file="memory-constraints-pod.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-constraints-pod.yaml --namespace=constraints-mem-example
```
Verify that the Pod's Container is running:
```shell
kubectl get pod constraints-mem-demo --namespace=constraints-mem-example
```
View detailed information about the Pod:
```shell
kubectl get pod constraints-mem-demo --output=yaml --namespace=constraints-mem-example
```
The output shows that the Container has a memory request of 600 MiB and a memory limit
of 800 MiB. These satisfy the constraints imposed by the LimitRange.
```yaml
resources:
limits:
memory: 800Mi
requests:
memory: 600Mi
```
Delete your Pod:
```shell
kubectl delete pod constraints-mem-demo --namespace=constraints-mem-example
```
## Attempt to create a Pod that exceeds the maximum memory constraint
Here's the configuration file for a Pod that has one Container. The Container specifies a
memory request of 800 MiB and a memory limit of 1.5 GiB.
{{< code file="memory-constraints-pod-2.yaml" >}}
Attempt to create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-constraints-pod-2.yaml --namespace=constraints-mem-example
```
The output shows that the Pod does not get created, because the Container specifies a memory limit that is
too large:
```
Error from server (Forbidden): error when creating "docs/tasks/administer-cluster/memory-constraints-pod-2.yaml":
pods "constraints-mem-demo-2" is forbidden: maximum memory usage per Container is 1Gi, but limit is 1536Mi.
```
## Attempt to create a Pod that does not meet the minimum memory request
Here's the configuration file for a Pod that has one Container. The Container specifies a
memory request of 200 MiB and a memory limit of 800 MiB.
{{< code file="memory-constraints-pod-3.yaml" >}}
Attempt to create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-constraints-pod-3.yaml --namespace=constraints-mem-example
```
The output shows that the Pod does not get created, because the Container specifies a memory
request that is too small:
```
Error from server (Forbidden): error when creating "docs/tasks/administer-cluster/memory-constraints-pod-3.yaml":
pods "constraints-mem-demo-3" is forbidden: minimum memory usage per Container is 500Mi, but request is 100Mi.
```
## Create a Pod that does not specify any memory request or limit
Here's the configuration file for a Pod that has one Container. The Container does not
specify a memory request, and it does not specify a memory limit.
{{< code file="memory-constraints-pod-4.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-constraints-pod-4.yaml --namespace=constraints-mem-example
```
View detailed information about the Pod:
```
kubectl get pod constraints-mem-demo-4 --namespace=constraints-mem-example --output=yaml
```
The output shows that the Pod's Container has a memory request of 1 GiB and a memory limit of 1 GiB.
How did the Container get those values?
```
resources:
limits:
memory: 1Gi
requests:
memory: 1Gi
```
Because your Container did not specify its own memory request and limit, it was given the
[default memory request and limit](/docs/tasks/administer-cluster/memory-default-namespace/)
from the LimitRange.
At this point, your Container might be running or it might not be running. Recall that a prerequisite
for this task is that your Nodes have at least 1 GiB of memory. If each of your Nodes has only
1 GiB of memory, then there is not enough allocatable memory on any Node to accommodate a memory
request of 1 GiB. If you happen to be using Nodes with 2 GiB of memory, then you probably have
enough space to accommodate the 1 GiB request.
Delete your Pod:
```
kubectl delete pod constraints-mem-demo-4 --namespace=constraints-mem-example
```
## Enforcement of minimum and maximum memory constraints
The maximum and minimum memory constraints imposed on a namespace by a LimitRange are enforced only
when a Pod is created or updated. If you change the LimitRange, it does not affect
Pods that were created previously.
## Motivation for minimum and maximum memory constraints
As a cluster administrator, you might want to impose restrictions on the amount of memory that Pods can use.
For example:
* Each Node in a cluster has 2 GB of memory. You do not want to accept any Pod that requests
more than 2 GB of memory, because no Node in the cluster can support the request.
* A cluster is shared by your production and development departments.
You want to allow production workloads to consume up to 8 GB of memory, but
you want development workloads to be limited to 512 MB. You create separate namespaces
for production and development, and you apply memory constraints to each namespace.
## Clean up
Delete your namespace:
```shell
kubectl delete namespace constraints-mem-example
```
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default Memory Requests and Limits for a Namespace](/docs/tasks/administer-cluster/memory-default-namespace/)
* [Configure Default CPU Requests and Limits for a Namespace](/docs/tasks/administer-cluster/cpu-default-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
* [Configure Memory and CPU Quotas for a Namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/)
* [Configure a Pod Quota for a Namespace](/docs/tasks/administer-cluster/quota-pod-namespace/)
* [Configure Quotas for API Objects](/docs/tasks/administer-cluster/quota-api-object/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/memory-constraints-pod-2.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-mem-demo-2
spec:
containers:
- name: constraints-mem-demo-2-ctr
image: nginx
resources:
limits:
memory: "1.5Gi"
requests:
memory: "800Mi"
content/en/docs/tasks/administer-cluster/memory-constraints-pod-3.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-mem-demo-3
spec:
containers:
- name: constraints-mem-demo-3-ctr
image: nginx
resources:
limits:
memory: "800Mi"
requests:
memory: "100Mi"
content/en/docs/tasks/administer-cluster/memory-constraints-pod-4.yaml
+9
View File
@@ -0,0 +1,9 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-mem-demo-4
spec:
containers:
- name: constraints-mem-demo-4-ctr
image: nginx
content/en/docs/tasks/administer-cluster/memory-constraints-pod.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Pod
metadata:
name: constraints-mem-demo
spec:
containers:
- name: constraints-mem-demo-ctr
image: nginx
resources:
limits:
memory: "800Mi"
requests:
memory: "600Mi"
content/en/docs/tasks/administer-cluster/memory-constraints.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: LimitRange
metadata:
name: mem-min-max-demo-lr
spec:
limits:
- max:
memory: 1Gi
min:
memory: 500Mi
type: Container
content/en/docs/tasks/administer-cluster/memory-default-namespace.md
+192
View File
@@ -0,0 +1,192 @@
---
title: Configure Default Memory Requests and Limits for a Namespace
content_template: templates/task
---
{{% capture overview %}}
This page shows how to configure default memory requests and limits for a namespace.
If a Container is created in a namespace that has a default memory limit, and the Container
does not specify its own memory limit, then the Container is assigned the default memory limit.
Kubernetes assigns a default memory request under certain conditions that are explained later in this topic.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
Each node in your cluster must have at least 2 GiB of memory.
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace default-mem-example
```
## Create a LimitRange and a Pod
Here's the configuration file for a LimitRange object. The configuration specifies
a default memory request and a default memory limit.
{{< code file="memory-defaults.yaml" >}}
Create the LimitRange in the default-mem-example namespace:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-defaults.yaml --namespace=default-mem-example
```
Now if a Container is created in the default-mem-example namespace, and the
Container does not specify its own values for memory request and memory limit,
the Container is given a default memory request of 256 MiB and a default
memory limit of 512 MiB.
Here's the configuration file for a Pod that has one Container. The Container
does not specify a memory request and limit.
{{< code file="memory-defaults-pod.yaml" >}}
Create the Pod.
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-defaults-pod.yaml --namespace=default-mem-example
```
View detailed information about the Pod:
```shell
kubectl get pod default-mem-demo --output=yaml --namespace=default-mem-example
```
The output shows that the Pod's Container has a memory request of 256 MiB and
a memory limit of 512 MiB. These are the default values specified by the LimitRange.
```shel
containers:
- image: nginx
imagePullPolicy: Always
name: default-mem-demo-ctr
resources:
limits:
memory: 512Mi
requests:
memory: 256Mi
```
Delete your Pod:
```shell
kubectl delete pod default-mem-demo --namespace=default-mem-example
```
## What if you specify a Container's limit, but not its request?
Here's the configuration file for a Pod that has one Container. The Container
specifies a memory limit, but not a request:
{{< code file="memory-defaults-pod-2.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-defaults-pod-2.yaml --namespace=default-mem-example
```
View detailed information about the Pod:
```shell
kubectl get pod default-mem-demo-2 --output=yaml --namespace=default-mem-example
```
The output shows that the Container's memory request is set to match its memory limit.
Notice that the Container was not assigned the default memory request value of 256Mi.
```
resources:
limits:
memory: 1Gi
requests:
memory: 1Gi
```
## What if you specify a Container's request, but not its limit?
Here's the configuration file for a Pod that has one Container. The Container
specifies a memory request, but not a limit:
{{< code file="memory-defaults-pod-3.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/memory-defaults-pod-3.yaml --namespace=default-mem-example
```
View the Pod's specification:
```shell
kubectl get pod default-mem-demo-3 --output=yaml --namespace=default-mem-example
```
The output shows that the Container's memory request is set to the value specified in the
Container's configuration file. The Container's memory limit is set to 512Mi, which is the
default memory limit for the namespace.
```
resources:
limits:
memory: 512Mi
requests:
memory: 128Mi
```
## Motivation for default memory limits and requests
If your namespace has a resource quota,
it is helpful to have a default value in place for memory limit.
Here are two of the restrictions that a resource quota imposes on a namespace:
* Every Container that runs in the namespace must have its own memory limit.
* The total amount of memory used by all Containers in the namespace must not exceed a specified limit.
If a Container does not specify its own memory limit, it is given the default limit, and then
it can be allowed to run in a namespace that is restricted by a quota.
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default CPU Requests and Limits for a Namespace](/docs/tasks/administer-cluster/cpu-default-namespace/)
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
* [Configure Memory and CPU Quotas for a Namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/)
* [Configure a Pod Quota for a Namespace](/docs/tasks/administer-cluster/quota-pod-namespace/)
* [Configure Quotas for API Objects](/docs/tasks/administer-cluster/quota-api-object/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/memory-defaults-pod-2.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: default-mem-demo-2
spec:
containers:
- name: defalt-mem-demo-2-ctr
image: nginx
resources:
limits:
memory: "1Gi"
content/en/docs/tasks/administer-cluster/memory-defaults-pod-3.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: default-mem-demo-3
spec:
containers:
- name: default-mem-demo-3-ctr
image: nginx
resources:
requests:
memory: "128Mi"
content/en/docs/tasks/administer-cluster/memory-defaults-pod.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Pod
metadata:
name: default-mem-demo
spec:
containers:
- name: default-mem-demo-ctr
image: nginx
content/en/docs/tasks/administer-cluster/memory-defaults.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: LimitRange
metadata:
name: mem-limit-range
spec:
limits:
- default:
memory: 512Mi
defaultRequest:
memory: 256Mi
type: Container
content/en/docs/tasks/administer-cluster/my-scheduler.yaml
+47
View File
@@ -0,0 +1,47 @@
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: scheduler
tier: control-plane
name: my-scheduler
namespace: kube-system
spec:
selector:
matchLabels:
component: scheduler
tier: control-plane
replicas: 1
template:
metadata:
labels:
component: scheduler
tier: control-plane
version: second
spec:
containers:
- command:
- /usr/local/bin/kube-scheduler
- --address=0.0.0.0
- --leader-elect=false
- --scheduler-name=my-scheduler
image: gcr.io/my-gcp-project/my-kube-scheduler:1.0
livenessProbe:
httpGet:
path: /healthz
port: 10251
initialDelaySeconds: 15
name: kube-second-scheduler
readinessProbe:
httpGet:
path: /healthz
port: 10251
resources:
requests:
cpu: '0.1'
securityContext:
privileged: false
volumeMounts: []
hostNetwork: false
hostPID: false
volumes: []
content/en/docs/tasks/administer-cluster/namespace-dev.json
+10
View File
@@ -0,0 +1,10 @@
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "development",
"labels": {
"name": "development"
}
}
}
content/en/docs/tasks/administer-cluster/namespace-prod.json
+10
View File
@@ -0,0 +1,10 @@
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "production",
"labels": {
"name": "production"
}
}
}
content/en/docs/tasks/administer-cluster/namespaces-walkthrough.md
+253
View File
@@ -0,0 +1,253 @@
---
reviewers:
- derekwaynecarr
- janetkuo
title: Namespaces Walkthrough
---
Kubernetes _namespaces_ help different projects, teams, or customers to share a Kubernetes cluster.
It does this by providing the following:
1. A scope for [Names](/docs/concepts/overview/working-with-objects/names/).
2. A mechanism to attach authorization and policy to a subsection of the cluster.
Use of multiple namespaces is optional.
This example demonstrates how to use Kubernetes namespaces to subdivide your cluster.
### Step Zero: Prerequisites
This example assumes the following:
1. You have an [existing Kubernetes cluster](/docs/setup/).
2. You have a basic understanding of Kubernetes _[Pods](/docs/concepts/workloads/pods/pod/)_, _[Services](/docs/concepts/services-networking/service/)_, and _[Deployments](/docs/concepts/workloads/controllers/deployment/)_.
### Step One: Understand the default namespace
By default, a Kubernetes cluster will instantiate a default namespace when provisioning the cluster to hold the default set of Pods,
Services, and Deployments used by the cluster.
Assuming you have a fresh cluster, you can inspect the available namespaces by doing the following:
```shell
$ kubectl get namespaces
NAME STATUS AGE
default Active 13m
```
### Step Two: Create new namespaces
For this exercise, we will create two additional Kubernetes namespaces to hold our content.
Let's imagine a scenario where an organization is using a shared Kubernetes cluster for development and production use cases.
The development team would like to maintain a space in the cluster where they can get a view on the list of Pods, Services, and Deployments
they use to build and run their application. In this space, Kubernetes resources come and go, and the restrictions on who can or cannot modify resources
are relaxed to enable agile development.
The operations team would like to maintain a space in the cluster where they can enforce strict procedures on who can or cannot manipulate the set of
Pods, Services, and Deployments that run the production site.
One pattern this organization could follow is to partition the Kubernetes cluster into two namespaces: development and production.
Let's create two new namespaces to hold our work.
Use the file [`namespace-dev.json`](/docs/tasks/administer-cluster/namespace-dev.json) which describes a development namespace:
{{< code language="json" file="namespace-dev.json" >}}
Create the development namespace using kubectl.
```shell
$ kubectl create -f https://k8s.io/docs/tasks/administer-cluster/namespace-dev.json
```
Save the following contents into file [`namespace-prod.json`](/docs/tasks/administer-cluster/namespace-prod.json) which describes a production namespace:
{{< code language="json" file="namespace-prod.json" >}}
And then let's create the production namespace using kubectl.
```shell
$ kubectl create -f https://k8s.io/docs/tasks/administer-cluster/namespace-prod.json
```
To be sure things are right, let's list all of the namespaces in our cluster.
```shell
$ kubectl get namespaces --show-labels
NAME STATUS AGE LABELS
default Active 32m <none>
development Active 29s name=development
production Active 23s name=production
```
### Step Three: Create pods in each namespace
A Kubernetes namespace provides the scope for Pods, Services, and Deployments in the cluster.
Users interacting with one namespace do not see the content in another namespace.
To demonstrate this, let's spin up a simple Deployment and Pods in the development namespace.
We first check what is the current context:
```shell
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://130.211.122.180
name: lithe-cocoa-92103_kubernetes
contexts:
- context:
cluster: lithe-cocoa-92103_kubernetes
user: lithe-cocoa-92103_kubernetes
name: lithe-cocoa-92103_kubernetes
current-context: lithe-cocoa-92103_kubernetes
kind: Config
preferences: {}
users:
- name: lithe-cocoa-92103_kubernetes
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: 65rZW78y8HbwXXtSXuUw9DbP4FLjHi4b
- name: lithe-cocoa-92103_kubernetes-basic-auth
user:
password: h5M0FtUUIflBSdI7
username: admin
$ kubectl config current-context
lithe-cocoa-92103_kubernetes
```
The next step is to define a context for the kubectl client to work in each namespace. The value of "cluster" and "user" fields are copied from the current context.
```shell
$ kubectl config set-context dev --namespace=development \
--cluster=lithe-cocoa-92103_kubernetes \
--user=lithe-cocoa-92103_kubernetes
$ kubectl config set-context prod --namespace=production \
--cluster=lithe-cocoa-92103_kubernetes \
--user=lithe-cocoa-92103_kubernetes
```
By default, the above commands adds two contexts that are saved into file
`.kube/config`. You can now view the contexts and alternate against the two
new request contexts depending on which namespace you wish to work against.
To view the new contexts:
```shell
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://130.211.122.180
name: lithe-cocoa-92103_kubernetes
contexts:
- context:
cluster: lithe-cocoa-92103_kubernetes
user: lithe-cocoa-92103_kubernetes
name: lithe-cocoa-92103_kubernetes
- context:
cluster: lithe-cocoa-92103_kubernetes
namespace: development
user: lithe-cocoa-92103_kubernetes
name: dev
- context:
cluster: lithe-cocoa-92103_kubernetes
namespace: production
user: lithe-cocoa-92103_kubernetes
name: prod
current-context: lithe-cocoa-92103_kubernetes
kind: Config
preferences: {}
users:
- name: lithe-cocoa-92103_kubernetes
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: 65rZW78y8HbwXXtSXuUw9DbP4FLjHi4b
- name: lithe-cocoa-92103_kubernetes-basic-auth
user:
password: h5M0FtUUIflBSdI7
username: admin
```
Let's switch to operate in the development namespace.
```shell
$ kubectl config use-context dev
```
You can verify your current context by doing the following:
```shell
$ kubectl config current-context
dev
```
At this point, all requests we make to the Kubernetes cluster from the command line are scoped to the development namespace.
Let's create some contents.
```shell
$ kubectl run snowflake --image=kubernetes/serve_hostname --replicas=2
```
We have just created a deployment whose replica size is 2 that is running the pod called snowflake with a basic container that just serves the hostname.
Note that `kubectl run` creates deployments only on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead.
If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/reference/generated/kubectl/kubectl-commands/#run) for more details.
```shell
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
snowflake 2 2 2 2 2m
$ kubectl get pods -l run=snowflake
NAME READY STATUS RESTARTS AGE
snowflake-3968820950-9dgr8 1/1 Running 0 2m
snowflake-3968820950-vgc4n 1/1 Running 0 2m
```
And this is great, developers are able to do what they want, and they do not have to worry about affecting content in the production namespace.
Let's switch to the production namespace and show how resources in one namespace are hidden from the other.
```shell
$ kubectl config use-context prod
```
The production namespace should be empty, and the following commands should return nothing.
```shell
$ kubectl get deployment
$ kubectl get pods
```
Production likes to run cattle, so let's create some cattle pods.
```shell
$ kubectl run cattle --image=kubernetes/serve_hostname --replicas=5
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
cattle 5 5 5 5 10s
kubectl get pods -l run=cattle
NAME READY STATUS RESTARTS AGE
cattle-2263376956-41xy6 1/1 Running 0 34s
cattle-2263376956-kw466 1/1 Running 0 34s
cattle-2263376956-n4v97 1/1 Running 0 34s
cattle-2263376956-p5p3i 1/1 Running 0 34s
cattle-2263376956-sxpth 1/1 Running 0 34s
```
At this point, it should be clear that the resources users create in one namespace are hidden from the other namespace.
As the policy support in Kubernetes evolves, we will extend this scenario to show how you can provide different
authorization rules for each namespace.
content/en/docs/tasks/administer-cluster/namespaces.md
+353
View File
@@ -0,0 +1,353 @@
---
reviewers:
- derekwaynecarr
- janetkuo
title: Share a Cluster with Namespaces
content_template: templates/task
---
{{% capture overview %}}
This page shows how to view, work in, and delete namespaces. The page also shows how to use Kubernetes namespaces to subdivide your cluster.
{{% /capture %}}
{{% capture prerequisites %}}
* Have an [existing Kubernetes cluster](/docs/setup/).
* Have a basic understanding of Kubernetes _[Pods](/docs/concepts/workloads/pods/pod/)_, _[Services](/docs/concepts/services-networking/service/)_, and _[Deployments](/docs/concepts/workloads/controllers/deployment/)_.
{{% /capture %}}
{{% capture steps %}}
## Viewing namespaces
1. List the current namespaces in a cluster using:
```shell
$ kubectl get namespaces
NAME STATUS AGE
default Active 11d
kube-system Active 11d
```
Kubernetes starts with two initial namespaces:
* `default` The default namespace for objects with no other namespace
* `kube-system` The namespace for objects created by the Kubernetes system
You can also get the summary of a specific namespace using:
```shell
$ kubectl get namespaces <name>
```
Or you can get detailed information with:
```shell
$ kubectl describe namespaces <name>
Name: default
Labels: <none>
Annotations: <none>
Status: Active
No resource quota.
Resource Limits
Type Resource Min Max Default
---- -------- --- --- ---
Container cpu - - 100m
```
Note that these details show both resource quota (if present) as well as resource limit ranges.
Resource quota tracks aggregate usage of resources in the *Namespace* and allows cluster operators
to define *Hard* resource usage limits that a *Namespace* may consume.
A limit range defines min/max constraints on the amount of resources a single entity can consume in
a *Namespace*.
See [Admission control: Limit Range](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md)
A namespace can be in one of two phases:
* `Active` the namespace is in use
* `Terminating` the namespace is being deleted, and can not be used for new objects
See the [design doc](https://git.k8s.io/community/contributors/design-proposals/architecture/namespaces.md#phases) for more details.
## Creating a new namespace
1. Create a new YAML file called `my-namespace.yaml` with the contents:
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: <insert-namespace-name-here>
```
Then run:
```shell
$ kubectl create -f ./my-namespace.yaml
```
Note that the name of your namespace must be a DNS compatible label.
There's an optional field `finalizers`, which allows observables to purge resources whenever the namespace is deleted. Keep in mind that if you specify a nonexistent finalizer, the namespace will be created but will get stuck in the `Terminating` state if the user tries to delete it.
More information on `finalizers` can be found in the namespace [design doc](https://git.k8s.io/community/contributors/design-proposals/architecture/namespaces.md#finalizers).
## Deleting a namespace
1. Delete a namespace with
```shell
$ kubectl delete namespaces <insert-some-namespace-name>
```
**WARNING, this deletes _everything_ under the namespace!**
This delete is asynchronous, so for a time you will see the namespace in the `Terminating` state.
## Subdividing your cluster using Kubernetes namespaces
1. Understand the default namespace
By default, a Kubernetes cluster will instantiate a default namespace when provisioning the cluster to hold the default set of Pods,
Services, and Deployments used by the cluster.
Assuming you have a fresh cluster, you can introspect the available namespace's by doing the following:
```shell
$ kubectl get namespaces
NAME STATUS AGE
default Active 13m
```
2. Create new namespaces
For this exercise, we will create two additional Kubernetes namespaces to hold our content.
In a scenario where an organization is using a shared Kubernetes cluster for development and production use cases:
The development team would like to maintain a space in the cluster where they can get a view on the list of Pods, Services, and Deployments
they use to build and run their application. In this space, Kubernetes resources come and go, and the restrictions on who can or cannot modify resources
are relaxed to enable agile development.
The operations team would like to maintain a space in the cluster where they can enforce strict procedures on who can or cannot manipulate the set of
Pods, Services, and Deployments that run the production site.
One pattern this organization could follow is to partition the Kubernetes cluster into two namespaces: development and production.
Let's create two new namespaces to hold our work.
Use the file [`namespace-dev.json`](/docs/tasks/administer-cluster/namespace-dev.json) which describes a development namespace:
{{< code language="json" file="namespace-dev.json" >}}
Create the development namespace using kubectl.
```shell
$ kubectl create -f docs/tasks/administer-cluster/namespace-dev.json
```
And then let's create the production namespace using kubectl.
```shell
$ kubectl create -f docs/tasks/administer-cluster/namespace-prod.json
```
To be sure things are right, list all of the namespaces in our cluster.
```shell
$ kubectl get namespaces --show-labels
NAME STATUS AGE LABELS
default Active 32m <none>
development Active 29s name=development
production Active 23s name=production
```
3. Create pods in each namespace
A Kubernetes namespace provides the scope for Pods, Services, and Deployments in the cluster.
Users interacting with one namespace do not see the content in another namespace.
To demonstrate this, let's spin up a simple Deployment and Pods in the development namespace.
We first check what is the current context:
```shell
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://130.211.122.180
name: lithe-cocoa-92103_kubernetes
contexts:
- context:
cluster: lithe-cocoa-92103_kubernetes
user: lithe-cocoa-92103_kubernetes
name: lithe-cocoa-92103_kubernetes
current-context: lithe-cocoa-92103_kubernetes
kind: Config
preferences: {}
users:
- name: lithe-cocoa-92103_kubernetes
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: 65rZW78y8HbwXXtSXuUw9DbP4FLjHi4b
- name: lithe-cocoa-92103_kubernetes-basic-auth
user:
password: h5M0FtUUIflBSdI7
username: admin
$ kubectl config current-context
lithe-cocoa-92103_kubernetes
```
The next step is to define a context for the kubectl client to work in each namespace. The values of "cluster" and "user" fields are copied from the current context.
```shell
$ kubectl config set-context dev --namespace=development --cluster=lithe-cocoa-92103_kubernetes --user=lithe-cocoa-92103_kubernetes
$ kubectl config set-context prod --namespace=production --cluster=lithe-cocoa-92103_kubernetes --user=lithe-cocoa-92103_kubernetes
```
The above commands provided two request contexts you can alternate against depending on what namespace you
wish to work against.
Let's switch to operate in the development namespace.
```shell
$ kubectl config use-context dev
```
You can verify your current context by doing the following:
```shell
$ kubectl config current-context
dev
```
At this point, all requests we make to the Kubernetes cluster from the command line are scoped to the development namespace.
Let's create some contents.
```shell
$ kubectl run snowflake --image=kubernetes/serve_hostname --replicas=2
```
We have just created a deployment whose replica size is 2 that is running the pod called snowflake with a basic container that just serves the hostname.
Note that `kubectl run` creates deployments only on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead.
If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/reference/generated/kubectl/kubectl-commands/#run) for more details.
```shell
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
snowflake 2 2 2 2 2m
$ kubectl get pods -l run=snowflake
NAME READY STATUS RESTARTS AGE
snowflake-3968820950-9dgr8 1/1 Running 0 2m
snowflake-3968820950-vgc4n 1/1 Running 0 2m
```
And this is great, developers are able to do what they want, and they do not have to worry about affecting content in the production namespace.
Let's switch to the production namespace and show how resources in one namespace are hidden from the other.
```shell
$ kubectl config use-context prod
```
The production namespace should be empty, and the following commands should return nothing.
```shell
$ kubectl get deployment
$ kubectl get pods
```
Production likes to run cattle, so let's create some cattle pods.
```shell
$ kubectl run cattle --image=kubernetes/serve_hostname --replicas=5
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
cattle 5 5 5 5 10s
kubectl get pods -l run=cattle
NAME READY STATUS RESTARTS AGE
cattle-2263376956-41xy6 1/1 Running 0 34s
cattle-2263376956-kw466 1/1 Running 0 34s
cattle-2263376956-n4v97 1/1 Running 0 34s
cattle-2263376956-p5p3i 1/1 Running 0 34s
cattle-2263376956-sxpth 1/1 Running 0 34s
```
At this point, it should be clear that the resources users create in one namespace are hidden from the other namespace.
As the policy support in Kubernetes evolves, we will extend this scenario to show how you can provide different
authorization rules for each namespace.
{{% /capture %}}
{{% capture discussion %}}
## Understanding the motivation for using namespaces
A single cluster should be able to satisfy the needs of multiple users or groups of users (henceforth a 'user community').
Kubernetes _namespaces_ help different projects, teams, or customers to share a Kubernetes cluster.
It does this by providing the following:
1. A scope for [Names](/docs/concepts/overview/working-with-objects/names/).
2. A mechanism to attach authorization and policy to a subsection of the cluster.
Use of multiple namespaces is optional.
Each user community wants to be able to work in isolation from other communities.
Each user community has its own:
1. resources (pods, services, replication controllers, etc.)
2. policies (who can or cannot perform actions in their community)
3. constraints (this community is allowed this much quota, etc.)
A cluster operator may create a Namespace for each unique user community.
The Namespace provides a unique scope for:
1. named resources (to avoid basic naming collisions)
2. delegated management authority to trusted users
3. ability to limit community resource consumption
Use cases include:
1. As a cluster operator, I want to support multiple user communities on a single cluster.
2. As a cluster operator, I want to delegate authority to partitions of the cluster to trusted users
in those communities.
3. As a cluster operator, I want to limit the amount of resources each community can consume in order
to limit the impact to other communities using the cluster.
4. As a cluster user, I want to interact with resources that are pertinent to my user community in
isolation of what other user communities are doing on the cluster.
## Understanding namespaces and DNS
When you create a [Service](/docs/concepts/services-networking/service/), it creates a corresponding [DNS entry](/docs/concepts/services-networking/dns-pod-service/).
This entry is of the form `<service-name>.<namespace-name>.svc.cluster.local`, which means
that if a container just uses `<service-name>` it will resolve to the service which
is local to a namespace. This is useful for using the same configuration across
multiple namespaces such as Development, Staging and Production. If you want to reach
across namespaces, you need to use the fully qualified domain name (FQDN).
{{% /capture %}}
{{% capture whatsnext %}}
* Learn more about [setting the namespace preference](/docs/concepts/overview/working-with-objects/namespaces/#setting-the-namespace-preference).
* Learn more about [setting the namespace for a request](/docs/concepts/overview/working-with-objects/namespaces/#setting-the-namespace-for-a-request)
* See [namespaces design](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/architecture/namespaces.md).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/out-of-resource.md
+371
View File
@@ -0,0 +1,371 @@
---
reviewers:
- derekwaynecarr
- vishh
- timstclair
title: Configure Out Of Resource Handling
---
{{< toc >}}
This page explains how to configure out of resource handling with `kubelet`.
The `kubelet` needs to preserve node stability when available compute resources
are low. This is especially important when dealing with incompressible
compute resources, such as memory or disk space. If such resources are exhausted,
nodes become unstable.
## Eviction Policy
The `kubelet` can proactively monitor for and prevent total starvation of a
compute resource. In those cases, the `kubelet` can reclaim the starved
resource by proactively failing one or more Pods. When the `kubelet` fails
a Pod, it terminates all of its containers and transitions its `PodPhase` to `Failed`.
### Eviction Signals
The `kubelet` supports eviction decisions based on the signals described in the following
table. The value of each signal is described in the Description column, which is based on
the `kubelet` summary API.
| Eviction Signal | Description |
|----------------------------|-----------------------------------------------------------------------|
| `memory.available` | `memory.available` := `node.status.capacity[memory]` - `node.stats.memory.workingSet` |
| `nodefs.available` | `nodefs.available` := `node.stats.fs.available` |
| `nodefs.inodesFree` | `nodefs.inodesFree` := `node.stats.fs.inodesFree` |
| `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available` |
| `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree` |
Each of the above signals supports either a literal or percentage based value.
The percentage based value is calculated relative to the total capacity
associated with each signal.
The value for `memory.available` is derived from the cgroupfs instead of tools
like `free -m`. This is important because `free -m` does not work in a
container, and if users use the [node
allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) feature, out of resource decisions
are made local to the end user Pod part of the cgroup hierarchy as well as the
root node. This
[script](/docs/tasks/administer-cluster/out-of-resource/memory-available.sh)
reproduces the same set of steps that the `kubelet` performs to calculate
`memory.available`. The `kubelet` excludes inactive_file (i.e. # of bytes of
file-backed memory on inactive LRU list) from its calculation as it assumes that
memory is reclaimable under pressure.
`kubelet` supports only two filesystem partitions.
1. The `nodefs` filesystem that kubelet uses for volumes, daemon logs, etc.
1. The `imagefs` filesystem that container runtimes uses for storing images and
container writable layers.
`imagefs` is optional. `kubelet` auto-discovers these filesystems using
cAdvisor. `kubelet` does not care about any other filesystems. Any other types
of configurations are not currently supported by the kubelet. For example, it is
*not OK* to store volumes and logs in a dedicated `filesystem`.
In future releases, the `kubelet` will deprecate the existing [garbage
collection](/docs/concepts/cluster-administration/kubelet-garbage-collection/)
support in favor of eviction in response to disk pressure.
### Eviction Thresholds
The `kubelet` supports the ability to specify eviction thresholds that trigger the `kubelet` to reclaim resources.
Each threshold has the following form:
`[eviction-signal][operator][quantity]`
where:
* `eviction-signal` is an eviction signal token as defined in the previous table.
* `operator` is the desired relational operator, such as `<` (less than).
* `quantity` is the eviction threshold quantity, such as `1Gi`. These tokens must
match the quantity representation used by Kubernetes. An eviction threshold can also
be expressed as a percentage using the `%` token.
For example, if a node has `10Gi` of total memory and you want trigger eviction if
the available memory falls below `1Gi`, you can define the eviction threshold as
either `memory.available<10%` or `memory.available<1Gi`. You cannot use both.
#### Soft Eviction Thresholds
A soft eviction threshold pairs an eviction threshold with a required
administrator-specified grace period. No action is taken by the `kubelet`
to reclaim resources associated with the eviction signal until that grace
period has been exceeded. If no grace period is provided, the `kubelet`
returns an error on startup.
In addition, if a soft eviction threshold has been met, an operator can
specify a maximum allowed Pod termination grace period to use when evicting
pods from the node. If specified, the `kubelet` uses the lesser value among
the `pod.Spec.TerminationGracePeriodSeconds` and the max allowed grace period.
If not specified, the `kubelet` kills Pods immediately with no graceful
termination.
To configure soft eviction thresholds, the following flags are supported:
* `eviction-soft` describes a set of eviction thresholds (e.g. `memory.available<1.5Gi`) that if met over a
corresponding grace period would trigger a Pod eviction.
* `eviction-soft-grace-period` describes a set of eviction grace periods (e.g. `memory.available=1m30s`) that
correspond to how long a soft eviction threshold must hold before triggering a Pod eviction.
* `eviction-max-pod-grace-period` describes the maximum allowed grace period (in seconds) to use when terminating
pods in response to a soft eviction threshold being met.
#### Hard Eviction Thresholds
A hard eviction threshold has no grace period, and if observed, the `kubelet`
will take immediate action to reclaim the associated starved resource. If a
hard eviction threshold is met, the `kubelet` kills the Pod immediately
with no graceful termination.
To configure hard eviction thresholds, the following flag is supported:
* `eviction-hard` describes a set of eviction thresholds (e.g. `memory.available<1Gi`) that if met
would trigger a Pod eviction.
The `kubelet` has the following default hard eviction threshold:
* `memory.available<100Mi`
* `nodefs.available<10%`
* `nodefs.inodesFree<5%`
* `imagefs.available<15%`
### Eviction Monitoring Interval
The `kubelet` evaluates eviction thresholds per its configured housekeeping interval.
* `housekeeping-interval` is the interval between container housekeepings.
### Node Conditions
The `kubelet` maps one or more eviction signals to a corresponding node condition.
If a hard eviction threshold has been met, or a soft eviction threshold has been met
independent of its associated grace period, the `kubelet` reports a condition that
reflects the node is under pressure.
The following node conditions are defined that correspond to the specified eviction signal.
| Node Condition | Eviction Signal | Description |
|-------------------------|-------------------------------|--------------------------------------------|
| `MemoryPressure` | `memory.available` | Available memory on the node has satisfied an eviction threshold |
| `DiskPressure` | `nodefs.available`, `nodefs.inodesFree`, `imagefs.available`, or `imagefs.inodesFree` | Available disk space and inodes on either the node's root filesystem or image filesystem has satisfied an eviction threshold |
The `kubelet` continues to report node status updates at the frequency specified by
`--node-status-update-frequency` which defaults to `10s`.
### Oscillation of node conditions
If a node is oscillating above and below a soft eviction threshold, but not exceeding
its associated grace period, it would cause the corresponding node condition to
constantly oscillate between true and false, and could cause poor scheduling decisions
as a consequence.
To protect against this oscillation, the following flag is defined to control how
long the `kubelet` must wait before transitioning out of a pressure condition.
* `eviction-pressure-transition-period` is the duration for which the `kubelet` has
to wait before transitioning out of an eviction pressure condition.
The `kubelet` would ensure that it has not observed an eviction threshold being met
for the specified pressure condition for the period specified before toggling the
condition back to `false`.
### Reclaiming node level resources
If an eviction threshold has been met and the grace period has passed,
the `kubelet` initiates the process of reclaiming the pressured resource
until it has observed the signal has gone below its defined threshold.
The `kubelet` attempts to reclaim node level resources prior to evicting end-user Pods. If
disk pressure is observed, the `kubelet` reclaims node level resources differently if the
machine has a dedicated `imagefs` configured for the container runtime.
#### With `imagefs`
If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting the dead Pods and their containers.
If `imagefs` filesystem has met eviction thresholds, `kubelet` frees up disk space by deleting all unused images.
#### Without `imagefs`
If `nodefs` filesystem has met eviction thresholds, `kubelet` frees up disk space in the following order:
1. Delete dead Pods and their containers
1. Delete all unused images
### Evicting end-user Pods
If the `kubelet` is unable to reclaim sufficient resource on the node, `kubelet` begins evicting Pods.
The `kubelet` ranks Pods for eviction first by whether or not their usage of the starved resource exceeds requests,
then by [Priority](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/), and then by the consumption of the starved compute resource relative to the Pods' scheduling requests.
As a result, `kubelet` ranks and evicts Pods in the following order:
* `BestEffort` or `Burstable` Pods whose usage of a starved resource exceeds its request.
Such pods are ranked by Priority, and then usage above request.
* `Guaranteed` pods and `Burstable` pods whose usage is beneath requests are evicted last.
`Guaranteed` Pods are guaranteed only when requests and limits are specified for all
the containers and they are equal. Such pods are guaranteed to never be evicted because
of another Pod's resource consumption. If a system daemon (such as `kubelet`, `docker`,
and `journald`) is consuming more resources than were reserved via `system-reserved` or
`kube-reserved` allocations, and the node only has `Guaranteed` or `Burstable` Pods using
less than requests remaining, then the node must choose to evict such a Pod in order to
preserve node stability and to limit the impact of the unexpected consumption to other Pods.
In this case, it will choose to evict pods of Lowest Priority first.
If necessary, `kubelet` evicts Pods one at a time to reclaim disk when `DiskPressure`
is encountered. If the `kubelet` is responding to `inode` starvation, it reclaims
`inodes` by evicting Pods with the lowest quality of service first. If the `kubelet`
is responding to lack of available disk, it ranks Pods within a quality of service
that consumes the largest amount of disk and kill those first.
#### With `imagefs`
If `nodefs` is triggering evictions, `kubelet` sorts Pods based on the usage on `nodefs`
- local volumes + logs of all its containers.
If `imagefs` is triggering evictions, `kubelet` sorts Pods based on the writable layer usage of all its containers.
#### Without `imagefs`
If `nodefs` is triggering evictions, `kubelet` sorts Pods based on their total disk usage
- local volumes + logs & writable layer of all its containers.
### Minimum eviction reclaim
In certain scenarios, eviction of Pods could result in reclamation of small amount of resources. This can result in
`kubelet` hitting eviction thresholds in repeated successions. In addition to that, eviction of resources like `disk`,
is time consuming.
To mitigate these issues, `kubelet` can have a per-resource `minimum-reclaim`. Whenever `kubelet` observes
resource pressure, `kubelet` attempts to reclaim at least `minimum-reclaim` amount of resource below
the configured eviction threshold.
For example, with the following configuration:
```
--eviction-hard=memory.available<500Mi,nodefs.available<1Gi,imagefs.available<100Gi
--eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"`
```
If an eviction threshold is triggered for `memory.available`, the `kubelet` works to ensure
that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` works
to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it
works to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure
on their associated resources.
The default `eviction-minimum-reclaim` is `0` for all resources.
### Scheduler
The node reports a condition when a compute resource is under pressure. The
scheduler views that condition as a signal to dissuade placing additional
pods on the node.
| Node Condition | Scheduler Behavior |
| ---------------- | ------------------------------------------------ |
| `MemoryPressure` | No new `BestEffort` Pods are scheduled to the node. |
| `DiskPressure` | No new Pods are scheduled to the node. |
## Node OOM Behavior
If the node experiences a system OOM (out of memory) event prior to the `kubelet` is able to reclaim memory,
the node depends on the [oom_killer](https://lwn.net/Articles/391222/) to respond.
The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the Pod.
| Quality of Service | oom_score_adj |
|----------------------------|-----------------------------------------------------------------------|
| `Guaranteed` | -998 |
| `BestEffort` | 1000 |
| `Burstable` | min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999) |
If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` calculates
an `oom_score` based on the percentage of memory it's using on the node, and then add the `oom_score_adj` to get an
effective `oom_score` for the container, and then kills the container with the highest score.
The intended behavior should be that containers with the lowest quality of service that
are consuming the largest amount of memory relative to the scheduling request should be killed first in order
to reclaim memory.
Unlike Pod eviction, if a Pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`.
## Best Practices
The following sections describe best practices for out of resource handling.
### Schedulable resources and eviction policies
Consider the following scenario:
* Node memory capacity: `10Gi`
* Operator wants to reserve 10% of memory capacity for system daemons (kernel, `kubelet`, etc.)
* Operator wants to evict Pods at 95% memory utilization to reduce thrashing and incidence of system OOM.
To facilitate this scenario, the `kubelet` would be launched as follows:
```
--eviction-hard=memory.available<500Mi
--system-reserved=memory=1.5Gi
```
Implicit in this configuration is the understanding that "System reserved" should include the amount of memory
covered by the eviction threshold.
To reach that capacity, either some Pod is using more than its request, or the system is using more than `1.5Gi - 500Mi = 1Gi`.
This configuration ensures that the scheduler does not place Pods on a node that immediately induce memory pressure
and trigger eviction assuming those Pods use less than their configured request.
### DaemonSet
It is never desired for `kubelet` to evict a `DaemonSet` Pod, since the Pod is
immediately recreated and rescheduled back to the same node.
At the moment, the `kubelet` has no ability to distinguish a Pod created
from `DaemonSet` versus any other object. If/when that information is
available, the `kubelet` could pro-actively filter those Pods from the
candidate set of Pods provided to the eviction strategy.
In general, it is strongly recommended that `DaemonSet` not
create `BestEffort` Pods to avoid being identified as a candidate Pod
for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` Pods.
## Deprecation of existing feature flags to reclaim disk
`kubelet` has been freeing up disk space on demand to keep the node stable.
As disk based eviction matures, the following `kubelet` flags are marked for deprecation
in favor of the simpler configuration supported around eviction.
| Existing Flag | New Flag |
| ------------- | -------- |
| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` |
| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` |
| `--maximum-dead-containers` | deprecated |
| `--maximum-dead-containers-per-container` | deprecated |
| `--minimum-container-ttl-duration` | deprecated |
| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` |
| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` |
## Known issues
The following sections describe known issues related to out of resource handling.
### kubelet may not observe memory pressure right away
The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage
increases within that window rapidly, the `kubelet` may not observe `MemoryPressure` fast enough, and the `OOMKiller`
will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this
latency, and instead have the kernel tell us when a threshold has been crossed immediately.
If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for
this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature
to prevent system OOMs, and promote eviction of workloads so cluster state can rebalance.
### kubelet may evict more Pods than needed
The Pod eviction may evict more Pods than needed due to stats collection timing gap. This can be mitigated by adding
the ability to get root container stats on an on-demand basis [(https://github.com/google/cadvisor/issues/1247)](https://github.com/google/cadvisor/issues/1247) in the future.
content/en/docs/tasks/administer-cluster/persistent-volume-label-initializer-config.yaml
+13
View File
@@ -0,0 +1,13 @@
kind: InitializerConfiguration
apiVersion: admissionregistration.k8s.io/v1alpha1
metadata:
name: pvlabel.kubernetes.io
initializers:
- name: pvlabel.kubernetes.io
rules:
- apiGroups:
- ""
apiVersions:
- "*"
resources:
- persistentvolumes
content/en/docs/tasks/administer-cluster/pod1.yaml
+10
View File
@@ -0,0 +1,10 @@
apiVersion: v1
kind: Pod
metadata:
name: no-annotation
labels:
name: multischeduler-example
spec:
containers:
- name: pod-with-no-annotation-container
image: k8s.gcr.io/pause:2.0
content/en/docs/tasks/administer-cluster/pod2.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: annotation-default-scheduler
labels:
name: multischeduler-example
spec:
schedulerName: default-scheduler
containers:
- name: pod-with-default-annotation-container
image: k8s.gcr.io/pause:2.0
content/en/docs/tasks/administer-cluster/pod3.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: annotation-second-scheduler
labels:
name: multischeduler-example
spec:
schedulerName: my-scheduler
containers:
- name: pod-with-second-annotation-container
image: k8s.gcr.io/pause:2.0
content/en/docs/tasks/administer-cluster/pvc-protection.md
+145
View File
@@ -0,0 +1,145 @@
---
reviewers:
- msau42
- jsafrane
title: Persistent Volume Claim Protection
content_template: templates/task
---
{{% capture overview %}}
{{< feature-state for_k8s_version="v1.9" state="alpha" >}}
As of Kubernetes 1.9, persistent volume claims (PVCs) that are in active use by a pod can be protected from pre-mature removal.
{{% /capture %}}
{{% capture prerequisites %}}
- A v1.9 or higher Kubernetes must be installed.
- As PVC Protection is a Kubernetes v1.9 alpha feature it must be enabled:
1. [Admission controller](/docs/admin/admission-controllers/) must be started with the [PVC Protection plugin](/docs/admin/admission-controllers/#persistent-volume-claim-protection-alpha).
2. All Kubernetes components must be started with the `PVCProtection` alpha features enabled.
{{% /capture %}}
{{% capture steps %}}
## PVC Protection Verification
The example below uses a GCE PD `StorageClass`, however, similar steps can be performed for any volume type.
Create a `StorageClass` for convenient storage provisioning:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
```
There are two scenarios: a PVC deleted by a user is either in active use or not in active use by a pod.
### Scenario 1: The PVC is not in active use by a pod
- Create a PVC:
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: slzc
spec:
accessModes:
- ReadWriteOnce
storageClassName: slow
resources:
requests:
storage: 3.7Gi
```
- Check that the PVC has the finalizer `kubernetes.io/pvc-protection` set:
```shell
$ kubectl describe pvc slzc
Name: slzc
Namespace: default
StorageClass: slow
Status: Bound
Volume: pvc-bee8c30a-d6a3-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 2m persistentvolume-controller Successfully provisioned volume pvc-bee8c30a-d6a3-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Delete the PVC and check that the PVC (not in active use by a pod) was removed successfully.
### Scenario 2: The PVC is in active use by a pod
- Again, create the same PVC.
- Create a pod that uses the PVC:
```yaml
kind: Pod
apiVersion: v1
metadata:
name: app1
spec:
containers:
- name: test-pod
image: k8s.gcr.io/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "date > /mnt/app1.txt; sleep 60 && exit 0 || exit 1"
volumeMounts:
- name: path-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: path-pvc
persistentVolumeClaim:
claimName: slzc
```
- Wait until the pod status is `Running`, i.e. the PVC becomes in active use.
- Delete the PVC that is now in active use by a pod and verify that the PVC is not removed but its status is `Terminating`:
```shell
Name: slzc
Namespace: default
StorageClass: slow
Status: Terminating (since Fri, 01 Dec 2017 14:47:55 +0000)
Volume: pvc-803a1f4d-d6a6-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 52s persistentvolume-controller Successfully provisioned volume pvc-803a1f4d-d6a6-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Wait until the pod status is `Terminated` (either delete the pod or wait until it finishes). Afterwards, check that the PVC is removed.
{{% /capture %}}
{{% capture discussion %}}
{{% /capture %}}
content/en/docs/tasks/administer-cluster/quota-api-object.md
+175
View File
@@ -0,0 +1,175 @@
---
title: Configure Quotas for API Objects
content_template: templates/task
---
{{% capture overview %}}
This page shows how to configure quotas for API objects, including
PersistentVolumeClaims and Services. A quota restricts the number of
objects, of a particular type, that can be created in a namespace.
You specify quotas in a
[ResourceQuota](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcequota-v1-core)
object.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace quota-object-example
```
## Create a ResourceQuota
Here is the configuration file for a ResourceQuota object:
{{< code file="quota-objects.yaml" >}}
Create the ResourceQuota:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-objects.yaml --namespace=quota-object-example
```
View detailed information about the ResourceQuota:
```shell
kubectl get resourcequota object-quota-demo --namespace=quota-object-example --output=yaml
```
The output shows that in the quota-object-example namespace, there can be at most
one PersistentVolumeClaim, at most two Services of type LoadBalancer, and no Services
of type NodePort.
```yaml
status:
hard:
persistentvolumeclaims: "1"
services.loadbalancers: "2"
services.nodeports: "0"
used:
persistentvolumeclaims: "0"
services.loadbalancers: "0"
services.nodeports: "0"
```
## Create a PersistentVolumeClaim
Here is the configuration file for a PersistentVolumeClaim object:
{{< code file="quota-objects-pvc.yaml" >}}
Create the PersistentVolumeClaim:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-objects-pvc.yaml --namespace=quota-object-example
```
Verify that the PersistentVolumeClaim was created:
```shell
kubectl get persistentvolumeclaims --namespace=quota-object-example
```
The output shows that the PersistentVolumeClaim exists and has status Pending:
```shell
NAME STATUS
pvc-quota-demo Pending
```
## Attempt to create a second PersistentVolumeClaim
Here is the configuration file for a second PersistentVolumeClaim:
{{< code file="quota-objects-pvc-2.yaml" >}}
Attempt to create the second PersistentVolumeClaim:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-objects-pvc-2.yaml --namespace=quota-object-example
```
The output shows that the second PersistentVolumeClaim was not created,
because it would have exceeded the quota for the namespace.
```
persistentvolumeclaims "pvc-quota-demo-2" is forbidden:
exceeded quota: object-quota-demo, requested: persistentvolumeclaims=1,
used: persistentvolumeclaims=1, limited: persistentvolumeclaims=1
```
## Notes
These are the strings used to identify API resources that can be constrained
by quotas:
<table>
<tr><th>String</th><th>API Object</th></tr>
<tr><td>"pods"</td><td>Pod</td></tr>
<tr><td>"services</td><td>Service</td></tr>
<tr><td>"replicationcontrollers"</td><td>ReplicationController</td></tr>
<tr><td>"resourcequotas"</td><td>ResourceQuota</td></tr>
<tr><td>"secrets"</td><td>Secret</td></tr>
<tr><td>"configmaps"</td><td>ConfigMap</td></tr>
<tr><td>"persistentvolumeclaims"</td><td>PersistentVolumeClaim</td></tr>
<tr><td>"services.nodeports"</td><td>Service of type NodePort</td></tr>
<tr><td>"services.loadbalancers"</td><td>Service of type LoadBalancer</td></tr>
</table>
## Clean up
Delete your namespace:
```shell
kubectl delete namespace quota-object-example
```
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default Memory Requests and Limits for a Namespace](/docs/tasks/administer-cluster/memory-default-namespace/)
* [Configure Default CPU Requests and Limits for a Namespace](/docs/tasks/administer-cluster/cpu-default-namespace/)
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
* [Configure Memory and CPU Quotas for a Namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/)
* [Configure a Pod Quota for a Namespace](/docs/tasks/administer-cluster/quota-pod-namespace/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/quota-mem-cpu-pod-2.yaml
+16
View File
@@ -0,0 +1,16 @@
apiVersion: v1
kind: Pod
metadata:
name: quota-mem-cpu-demo-2
spec:
containers:
- name: quota-mem-cpu-demo-2-ctr
image: redis
resources:
limits:
memory: "1Gi"
cpu: "800m"
requests:
memory: "700Mi"
cpu: "400m"
content/en/docs/tasks/administer-cluster/quota-mem-cpu-pod.yaml
+16
View File
@@ -0,0 +1,16 @@
apiVersion: v1
kind: Pod
metadata:
name: quota-mem-cpu-demo
spec:
containers:
- name: quota-mem-cpu-demo-ctr
image: nginx
resources:
limits:
memory: "800Mi"
cpu: "800m"
requests:
memory: "600Mi"
cpu: "400m"
content/en/docs/tasks/administer-cluster/quota-mem-cpu.yaml
+10
View File
@@ -0,0 +1,10 @@
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
content/en/docs/tasks/administer-cluster/quota-memory-cpu-namespace.md
+179
View File
@@ -0,0 +1,179 @@
---
title: Configure Memory and CPU Quotas for a Namespace
content_template: templates/task
---
{{% capture overview %}}
This page shows how to set quotas for the total amount memory and CPU that
can be used by all Containers running in a namespace. You specify quotas in a
[ResourceQuota](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcequota-v1-core)
object.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
Each node in your cluster must have at least 1 GiB of memory.
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace quota-mem-cpu-example
```
## Create a ResourceQuota
Here is the configuration file for a ResourceQuota object:
{{< code file="quota-mem-cpu.yaml" >}}
Create the ResourceQuota:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-mem-cpu.yaml --namespace=quota-mem-cpu-example
```
View detailed information about the ResourceQuota:
```shell
kubectl get resourcequota mem-cpu-demo --namespace=quota-mem-cpu-example --output=yaml
```
The ResourceQuota places these requirements on the quota-mem-cpu-example namespace:
* Every Container must have a memory request, memory limit, cpu request, and cpu limit.
* The memory request total for all Containers must not exceed 1 GiB.
* The memory limit total for all Containers must not exceed 2 GiB.
* The CPU request total for all Containers must not exceed 1 cpu.
* The CPU limit total for all Containers must not exceed 2 cpu.
## Create a Pod
Here is the configuration file for a Pod:
{{< code file="quota-mem-cpu-pod.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-mem-cpu-pod.yaml --namespace=quota-mem-cpu-example
```
Verify that the Pod's Container is running:
```
kubectl get pod quota-mem-cpu-demo --namespace=quota-mem-cpu-example
```
Once again, view detailed information about the ResourceQuota:
```
kubectl get resourcequota mem-cpu-demo --namespace=quota-mem-cpu-example --output=yaml
```
The output shows the quota along with how much of the quota has been used.
You can see that the memory and CPU requests and limits for your Pod do not
exceed the quota.
```
status:
hard:
limits.cpu: "2"
limits.memory: 2Gi
requests.cpu: "1"
requests.memory: 1Gi
used:
limits.cpu: 800m
limits.memory: 800Mi
requests.cpu: 400m
requests.memory: 600Mi
```
## Attempt to create a second Pod
Here is the configuration file for a second Pod:
{{< code file="quota-mem-cpu-pod-2.yaml" >}}
In the configuration file, you can see that the Pod has a memory request of 700 MiB.
Notice that the sum of the used memory request and this new memory
request exceeds the memory request quota. 600 MiB + 700 MiB > 1 GiB.
Attempt to create the Pod:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-mem-cpu-pod-2.yaml --namespace=quota-mem-cpu-example
```
The second Pod does not get created. The output shows that creating the second Pod
would cause the memory request total to exceed the memory request quota.
```
Error from server (Forbidden): error when creating "docs/tasks/administer-cluster/quota-mem-cpu-pod-2.yaml":
pods "quota-mem-cpu-demo-2" is forbidden: exceeded quota: mem-cpu-demo,
requested: requests.memory=700Mi,used: requests.memory=600Mi, limited: requests.memory=1Gi
```
## Discussion
As you have seen in this exercise, you can use a ResourceQuota to restrict
the memory request total for all Containers running in a namespace.
You can also restrict the totals for memory limit, cpu request, and cpu limit.
If you want to restrict individual Containers, instead of totals for all Containers, use a
[LimitRange](/docs/tasks/administer-cluster/memory-constraint-namespace/).
## Clean up
Delete your namespace:
```shell
kubectl delete namespace quota-mem-cpu-example
```
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default Memory Requests and Limits for a Namespace](/docs/tasks/administer-cluster/memory-default-namespace/)
* [Configure Default CPU Requests and Limits for a Namespace](/docs/tasks/administer-cluster/cpu-default-namespace/)
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
* [Configure a Pod Quota for a Namespace](/docs/tasks/administer-cluster/quota-pod-namespace/)
* [Configure Quotas for API Objects](/docs/tasks/administer-cluster/quota-api-object/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/quota-objects-pvc-2.yaml
+11
View File
@@ -0,0 +1,11 @@
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-quota-demo-2
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
content/en/docs/tasks/administer-cluster/quota-objects-pvc.yaml
+11
View File
@@ -0,0 +1,11 @@
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-quota-demo
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
content/en/docs/tasks/administer-cluster/quota-objects.yaml
+9
View File
@@ -0,0 +1,9 @@
apiVersion: v1
kind: ResourceQuota
metadata:
name: object-quota-demo
spec:
hard:
persistentvolumeclaims: "1"
services.loadbalancers: "2"
services.nodeports: "0"
content/en/docs/tasks/administer-cluster/quota-pod-deployment.yaml
+17
View File
@@ -0,0 +1,17 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: pod-quota-demo
spec:
selector:
matchLabels:
purpose: quota-demo
replicas: 3
template:
metadata:
labels:
purpose: quota-demo
spec:
containers:
- name: pod-quota-demo
image: nginx
content/en/docs/tasks/administer-cluster/quota-pod-namespace.md
+140
View File
@@ -0,0 +1,140 @@
---
title: Configure a Pod Quota for a Namespace
content_template: templates/task
---
{{% capture overview %}}
This page shows how to set a quota for the total number of Pods that can run
in a namespace. You specify quotas in a
[ResourceQuota](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcequota-v1-core)
object.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Create a namespace
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
```shell
kubectl create namespace quota-pod-example
```
## Create a ResourceQuota
Here is the configuration file for a ResourceQuota object:
{{< code file="quota-pod.yaml" >}}
Create the ResourceQuota:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-pod.yaml --namespace=quota-pod-example
```
View detailed information about the ResourceQuota:
```shell
kubectl get resourcequota pod-demo --namespace=quota-pod-example --output=yaml
```
The output shows that the namespace has a quota of two Pods, and that currently there are
no Pods; that is, none of the quota is used.
```yaml
spec:
hard:
pods: "2"
status:
hard:
pods: "2"
used:
pods: "0"
```
Here is the configuration file for a Deployment:
{{< code file="quota-pod-deployment.yaml" >}}
In the configuration file, `replicas: 3` tells Kubernetes to attempt to create three Pods, all running the same application.
Create the Deployment:
```shell
kubectl create -f https://k8s.io/docs/tasks/administer-cluster/quota-pod-deployment.yaml --namespace=quota-pod-example
```
View detailed information about the Deployment:
```shell
kubectl get deployment pod-quota-demo --namespace=quota-pod-example --output=yaml
```
The output shows that even though the Deployment specifies three replicas, only two
Pods were created because of the quota.
```yaml
spec:
...
replicas: 3
...
status:
availableReplicas: 2
...
lastUpdateTime: 2017-07-07T20:57:05Z
message: 'unable to create pods: pods "pod-quota-demo-1650323038-" is forbidden:
exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited: pods=2'
```
## Clean up
Delete your namespace:
```shell
kubectl delete namespace quota-pod-example
```
{{% /capture %}}
{{% capture whatsnext %}}
### For cluster administrators
* [Configure Default Memory Requests and Limits for a Namespace](/docs/tasks/administer-cluster/memory-default-namespace/)
* [Configure Default CPU Requests and Limits for a Namespace](/docs/tasks/administer-cluster/cpu-default-namespace/)
* [Configure Minimum and Maximum Memory Constraints for a Namespace](/docs/tasks/administer-cluster/memory-constraint-namespace/)
* [Configure Minimum and Maximum CPU Constraints for a Namespace](/docs/tasks/administer-cluster/cpu-constraint-namespace/)
* [Configure Memory and CPU Quotas for a Namespace](/docs/tasks/administer-cluster/quota-memory-cpu-namespace/)
* [Configure Quotas for API Objects](/docs/tasks/administer-cluster/quota-api-object/)
### For app developers
* [Assign Memory Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-memory-resource/)
* [Assign CPU Resources to Containers and Pods](/docs/tasks/configure-pod-container/assign-cpu-resource/)
* [Configure Quality of Service for Pods](/docs/tasks/configure-pod-container/quality-service-pod/)
{{% /capture %}}
content/en/docs/tasks/administer-cluster/quota-pod.yaml
+7
View File
@@ -0,0 +1,7 @@
apiVersion: v1
kind: ResourceQuota
metadata:
name: pod-demo
spec:
hard:
pods: "2"
content/en/docs/tasks/administer-cluster/quota-pvc-2.yaml
+11
View File
@@ -0,0 +1,11 @@
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-quota-demo-2
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
content/en/docs/tasks/administer-cluster/reconfigure-kubelet.md
+455
View File
@@ -0,0 +1,455 @@
---
reviewers:
- mtaufen
- dawnchen
title: Reconfigure a Node's Kubelet in a Live Cluster
content_template: templates/task
---
{{% capture overview %}}
{{< feature-state state="alpha" >}}
As of Kubernetes 1.8, the new
[Dynamic Kubelet Configuration](https://github.com/kubernetes/features/issues/281)
feature is available in alpha. This allows you to change the configuration of
Kubelets in a live Kubernetes cluster via first-class Kubernetes concepts.
Specifically, this feature allows you to configure individual Nodes' Kubelets
via ConfigMaps.
**Warning:** All Kubelet configuration parameters may be changed dynamically,
but not all parameters are safe to change dynamically. This feature is intended
for system experts who have a strong understanding of how configuration changes
will affect behavior. No documentation currently exists which plainly lists
"safe to change" fields, but we plan to add it before this feature graduates
from alpha.
{{% /capture %}}
{{% capture prerequisites %}}
- A live Kubernetes cluster with both Master and Node at v1.8 or higher must be
running, with the `DynamicKubeletConfig` feature gate enabled and the Kubelet's
`--dynamic-config-dir` flag set to a writeable directory on the Node.
This flag must be set to enable Dynamic Kubelet Configuration.
- The kubectl command-line tool must be also v1.8 or higher, and must be
configured to communicate with the cluster.
{{% /capture %}}
{{% capture steps %}}
## Reconfiguring the Kubelet on a Live Node in your Cluster
### Basic Workflow Overview
The basic workflow for configuring a Kubelet in a live cluster is as follows:
1. Write a YAML or JSON configuration file containing the
Kubelet's configuration.
2. Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
3. Update the Kubelet's corresponding Node object to use this ConfigMap.
Each Kubelet watches a configuration reference on its respective Node object.
When this reference changes, the Kubelet downloads the new configuration,
updates a local reference to refer to the file, and exits.
For the feature to work correctly, you must be running a process manager
(like systemd) which will restart the Kubelet when it exits. When the Kubelet is
restarted, it will begin using the new configuration.
The new configuration completely overrides configuration provided by `--config`,
and is overridden by command-line flags. Unspecified values in the new configuration
will receive default values appropriate to the configuration version
(e.g. `kubelet.config.k8s.io/v1beta1`), unless overridden by flags.
The status of the Node's Kubelet configuration is reported via the `KubeletConfigOK`
condition in the Node status. Once you have updated a Node to use the new
ConfigMap, you can observe this condition to confirm that the Node is using the
intended configuration. A table describing the possible conditions can be found
at the end of this article.
This document describes editing Nodes using `kubectl edit`.
There are other ways to modify a Node's spec, including `kubectl patch`, for
example, which facilitate scripted workflows.
This document only describes a single Node consuming each ConfigMap. Keep in
mind that it is also valid for multiple Nodes to consume the same ConfigMap.
### Node Authorizer Workarounds
The Node Authorizer does not yet pay attention to which ConfigMaps are assigned
to which Nodes. If you currently use the Node authorizer, your Kubelets will not
be automatically granted permission to download their respective ConfigMaps.
The temporary workaround used in this document is to manually create the RBAC
Roles and RoleBindings for each ConfigMap. The Node Authorizer will be extended
before the Dynamic Kubelet Configuration feature graduates from alpha, so doing
this in production should never be necessary.
### Generating a file that contains the current configuration
The Dynamic Kubelet Configuration feature allows you to provide an override for
the entire configuration object, rather than a per-field overlay. This is a
simpler model that makes it easier to trace the source of configuration values
and debug issues. The compromise, however, is that you must start with knowledge
of the existing configuration to ensure that you only change the fields you
intend to change.
In the future, the Kubelet will be bootstrapped from a file on disk
(see [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)),
and you will simply edit a copy of this file (which, as a best practice, should
live in version control) while creating the first Kubelet ConfigMap. Today,
however, the Kubelet is still bootstrapped with command-line flags. Fortunately,
there is a dirty trick you can use to generate a config file containing a Node's
current configuration. The trick involves accessing the Kubelet server's `configz`
endpoint via the kubectl proxy. This endpoint, in its current implementation, is
intended to be used only as a debugging aid, which is part of why this is a
dirty trick. The endpoint may be improved in the future, but until then
it should not be relied on for production scenarios.
This trick also requires the `jq` command to be installed on your machine,
for unpacking and editing the JSON response from the endpoint.
Do the following to generate the file:
1. Pick a Node to reconfigure. We will refer to this Node's name as NODE_NAME.
2. Start the kubectl proxy in the background with `kubectl proxy --port=8001 &`
3. Run the following command to download and unpack the configuration from the
configz endpoint:
```
$ export NODE_NAME=the-name-of-the-node-you-are-reconfiguring
$ curl -sSL http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
```
Note that we have to manually add the `kind` and `apiVersion` to the downloaded
object, as these are not reported by the configz endpoint. This is one of the
limitations of the endpoint.
### Edit the configuration file
Using your editor of choice, change one of the parameters in the
`kubelet_configz_${NODE_NAME}` file from the previous step. A QPS parameter,
`eventRecordQPS` for example, is a good candidate.
### Push the configuration file to the control plane
Push the edited configuration file to the control plane with the
following command:
```
$ kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
```
You should see a response similar to:
```
apiVersion: v1
data:
kubelet: |
{...}
kind: ConfigMap
metadata:
creationTimestamp: 2017-09-14T20:23:33Z
name: my-node-config-gkt4c2m4b2
namespace: kube-system
resourceVersion: "119980"
selfLink: /api/v1/namespaces/kube-system/configmaps/my-node-config-gkt4c2m4b2
uid: 946d785e-998a-11e7-a8dd-42010a800006
```
Note that the configuration data must appear under the ConfigMap's
`kubelet` key.
We create the ConfigMap in the `kube-system` namespace, which is appropriate
because this ConfigMap configures a Kubernetes system component - the Kubelet.
The `--append-hash` option appends a short checksum of the ConfigMap contents
to the name. This is convenient for an edit->push workflow, as it will
automatically, yet deterministically, generate new names for new ConfigMaps.
We use the `-o yaml` output format so that the name, namespace, and uid are all
reported following creation. We will need these in the next step. We will refer
to the name as CONFIG_MAP_NAME and the uid as CONFIG_MAP_UID.
### Authorize your Node to read the new ConfigMap
Now that you've created a new ConfigMap, you need to authorize your node to
read it. First, create a Role for your new ConfigMap with the
following commands:
```
$ export CONFIG_MAP_NAME=name-from-previous-output
$ kubectl -n kube-system create role ${CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${CONFIG_MAP_NAME}
```
Next, create a RoleBinding to associate your Node with the new Role:
```
$ kubectl -n kube-system create rolebinding ${CONFIG_MAP_NAME}-reader --role=${CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
### Set the Node to use the new configuration
Edit the Node's reference to point to the new ConfigMap with the
following command:
```
kubectl edit node ${NODE_NAME}
```
Once in your editor, add the following YAML under `spec`:
```
configSource:
configMapRef:
name: CONFIG_MAP_NAME
namespace: kube-system
uid: CONFIG_MAP_UID
```
Be sure to specify all three of `name`, `namespace`, and `uid`.
### Observe that the Node begins using the new configuration
Retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and look for the
`KubeletConfigOK` condition in `status.conditions`. You should see the message
`Using current (UID: CONFIG_MAP_UID)` when the Kubelet starts using the new
configuration.
For convenience, you can use the following command (using `jq`) to filter down
to the `KubeletConfigOK` condition:
```
$ kubectl get no ${NODE_NAME} -o json | jq '.status.conditions|map(select(.type=="KubeletConfigOK"))'
[
{
"lastHeartbeatTime": "2017-09-20T18:08:29Z",
"lastTransitionTime": "2017-09-20T18:08:17Z",
"message": "using current: /api/v1/namespaces/kube-system/configmaps/my-node-config-gkt4c2m4b2",
"reason": "passing all checks",
"status": "True",
"type": "KubeletConfigOK"
}
]
```
If something goes wrong, you may see one of several different error conditions,
detailed in the table of KubeletConfigOK conditions, below. When this happens, you
should check the Kubelet's log for more details.
### Edit the configuration file again
To change the configuration again, we simply repeat the above workflow.
Try editing the `kubelet` file, changing the previously changed parameter to a
new value.
### Push the newly edited configuration to the control plane
Push the new configuration to the control plane in a new ConfigMap with the
following command:
```
$ kubectl create configmap my-node-config --namespace=kube-system --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
```
This new ConfigMap will get a new name, as we have changed the contents.
We will refer to the new name as NEW_CONFIG_MAP_NAME and the new uid
as NEW_CONFIG_MAP_UID.
### Authorize your Node to read the new ConfigMap
Now that you've created a new ConfigMap, you need to authorize your node to
read it. First, create a Role for your new ConfigMap with the
following commands:
```
$ export NEW_CONFIG_MAP_NAME=name-from-previous-output
$ kubectl -n kube-system create role ${NEW_CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${NEW_CONFIG_MAP_NAME}
```
Next, create a RoleBinding to associate your Node with the new Role:
```
$ kubectl -n kube-system create rolebinding ${NEW_CONFIG_MAP_NAME}-reader --role=${NEW_CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
### Configure the Node to use the new configuration
Once more, edit the Node's `spec.configSource` with
`kubectl edit node ${NODE_NAME}`. Your new `spec.configSource` should look like
the following, with `name` and `uid` substituted as necessary:
```
configSource:
configMapRef:
name: ${NEW_CONFIG_MAP_NAME}
namespace: kube-system
uid: ${NEW_CONFIG_MAP_UID}
```
### Observe that the Kubelet is using the new configuration
Once more, retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and
look for the `KubeletConfigOK` condition in `status.conditions`. You should see the message
`using current: /api/v1/namespaces/kube-system/configmaps/${NEW_CONFIG_MAP_NAME}` when the Kubelet starts using the
new configuration.
### Deauthorize your Node fom reading the old ConfigMap
Once you know your Node is using the new configuration and are confident that
the new configuration has not caused any problems, it is a good idea to
deauthorize the node from reading the old ConfigMap. Run the following
commands to remove the RoleBinding and Role:
```
$ kubectl -n kube-system delete rolebinding ${CONFIG_MAP_NAME}-reader
$ kubectl -n kube-system delete role ${CONFIG_MAP_NAME}-reader
```
Note that this does not necessarily prevent the Node from reverting to the old
configuration, as it may locally cache the old ConfigMap for an indefinite
period of time.
You may optionally also choose to remove the old ConfigMap:
```
$ kubectl -n kube-system delete configmap ${CONFIG_MAP_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
### Reset the Node to use its local default configuration
Finally, if you wish to reset the Node to use the configuration it was
provisioned with, simply edit the Node with `kubectl edit node ${NODE_NAME}` and
remove the `spec.configSource` subfield.
### Observe that the Node is using its local default configuration
After removing this subfield, you should eventually observe that the KubeletConfigOK
condition's message reverts to `using current: local`.
### Deauthorize your Node fom reading the old ConfigMap
Once you know your Node is using the default configuration again, it is a good
idea to deauthorize the node from reading the old ConfigMap. Run the following
commands to remove the RoleBinding and Role:
```
$ kubectl -n kube-system delete rolebinding ${NEW_CONFIG_MAP_NAME}-reader
$ kubectl -n kube-system delete role ${NEW_CONFIG_MAP_NAME}-reader
```
Note that this does not necessarily prevent the Node from reverting to the old
ConfigMap, as it may locally cache the old ConfigMap for an indefinite
period of time.
You may optionally also choose to remove the old ConfigMap:
```
$ kubectl -n kube-system delete configmap ${NEW_CONFIG_MAP_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
{{% /capture %}}
{{% capture discussion %}}
## Kubectl Patch Example
As mentioned above, there are many ways to change a Node's configSource.
Here is an example command that uses `kubectl patch`:
```
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMapRef\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"uid\":\"${CONFIG_MAP_UID}\"}}}}"
```
## Understanding KubeletConfigOK Conditions
The following table describes several of the `KubeletConfigOK` Node conditions you
might encounter in a cluster that has Dynamic Kubelet Config enabled. If you
observe a condition with `status=False`, you should check the Kubelet log for
more error details by searching for the message or reason text.
<table>
<table align="left">
<tr>
<th>Possible Messages</th>
<th>Possible Reasons</th>
<th>Status</th>
</tr>
<tr>
<td><p>using current: local</p></td>
<td><p>when the config source is nil, the Kubelet uses its local config</p></td>
<td><p>True</p></td>
</tr>
<tr>
<td><p>using current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</p></td>
<td><p>passing all checks</p></td>
<td><p>True</p></td>
</tr>
<tr>
<td><p>using last-known-good: local</p></td>
<td>
<ul>
<li>failed to load current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
<li>failed to parse current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
<li>failed to validate current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
<tr>
<td><p>using last-known-good: /api/v1/namespaces/${LAST_KNOWN_GOOD_CONFIG_MAP_NAMESPACE}/configmaps/${LAST_KNOWN_GOOD_CONFIG_MAP_NAME}</p></td>
<td>
<ul>
<li>failed to load current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
<li>failed to parse current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
<li>failed to validate current: /api/v1/namespaces/${CURRENT_CONFIG_MAP_NAMESPACE}/configmaps/${CURRENT_CONFIG_MAP_NAME}</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
<tr>
<td>
<p>
The reasons in the next column could potentially appear for any of
the above messages.
</p>
<p>
This condition indicates that the Kubelet is having trouble
reconciling `spec.configSource`, and thus no change to the in-use
configuration has occurred.
</p>
<p>
The "failed to sync" reasons are specific to the failure that
occurred, and the next column does not necessarily contain all
possible failure reasons.
</p>
</td>
<td>
<p>failed to sync, reason:</p>
<ul>
<li>failed to read Node from informer object cache</li>
<li>failed to reset to local config</li>
<li>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</li>
<li>invalid ObjectReference, all of UID, Name, and Namespace must be specified</li>
<li>invalid ConfigSource.ConfigMapRef.UID: ${UID} does not match ${API_PATH}.UID: ${UID_OF_CONFIG_MAP_AT_API_PATH}</li>
<li>failed to determine whether object ${API_PATH} with UID ${UID} was already checkpointed</li>
<li>failed to download ConfigMap with name ${NAME} from namespace ${NAMESPACE}</li>
<li>failed to save config checkpoint for object ${API_PATH} with UID ${UID}</li>
<li>failed to set current config checkpoint to local config</li>
<li>failed to set current config checkpoint to object ${API_PATH} with UID ${UID}</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
</table>
{{% /capture %}}
content/en/docs/tasks/administer-cluster/reserve-compute-resources.md
+232
View File
@@ -0,0 +1,232 @@
---
reviewers:
- vishh
- derekwaynecarr
- dashpole
title: Reserve Compute Resources for System Daemons
---
{{< toc >}}
Kubernetes nodes can be scheduled to `Capacity`. Pods can consume all the
available capacity on a node by default. This is an issue because nodes
typically run quite a few system daemons that power the OS and Kubernetes
itself. Unless resources are set aside for these system daemons, pods and system
daemons compete for resources and lead to resource starvation issues on the
node.
The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve
compute resources for system daemons. Kubernetes recommends cluster
administrators to configure `Node Allocatable` based on their workload density
on each node.
## Node Allocatable
```text
Node Capacity
---------------------------
| kube-reserved |
|-------------------------|
| system-reserved |
|-------------------------|
| eviction-threshold |
|-------------------------|
| |
| allocatable |
| (available for pods) |
| |
| |
---------------------------
```
`Allocatable` on a Kubernetes node is defined as the amount of compute resources
that are available for pods. The scheduler does not over-subscribe
`Allocatable`. `CPU`, `memory` and `ephemeral-storage` are supported as of now.
Node Allocatable is exposed as part of `v1.Node` object in the API and as part
of `kubectl describe node` in the CLI.
Resources can be reserved for two categories of system daemons in the `kubelet`.
### Enabling QoS and Pod level cgroups
To properly enforce node allocatable constraints on the node, you must
enable the new cgroup hierarchy via the `--cgroups-per-qos` flag. This flag is
enabled by default. When enabled, the `kubelet` will parent all end-user pods
under a cgroup hierarchy managed by the `kubelet`.
### Configuring a cgroup driver
The `kubelet` supports manipulation of the cgroup hierarchy on
the host using a cgroup driver. The driver is configured via the
`--cgroup-driver` flag.
The supported values are the following:
* `cgroupfs` is the default driver that performs direct manipulation of the
cgroup filesystem on the host in order to manage cgroup sandboxes.
* `systemd` is an alternative driver that manages cgroup sandboxes using
transient slices for resources that are supported by that init system.
Depending on the configuration of the associated container runtime,
operators may have to choose a particular cgroup driver to ensure
proper system behavior. For example, if operators use the `systemd`
cgroup driver provided by the `docker` runtime, the `kubelet` must
be configured to use the `systemd` cgroup driver.
### Kube Reserved
- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi]`
- **Kubelet Flag**: `--kube-reserved-cgroup=`
`kube-reserved` is meant to capture resource reservation for kubernetes system
daemons like the `kubelet`, `container runtime`, `node problem detector`, etc.
It is not meant to reserve resources for system daemons that are run as pods.
`kube-reserved` is typically a function of `pod density` on the nodes. [This
performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and
`memory` usage profiles of `kubelet` and `docker engine` at multiple levels of
pod density. [This blog
post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html)
explains how the dashboard can be interpreted to come up with a suitable
`kube-reserved` reservation.
To optionally enforce `kube-reserved` on system daemons, specify the parent
control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet
flag.
It is recommended that the kubernetes system daemons are placed under a top
level control group (`runtime.slice` on systemd machines for example). Each
system daemon should ideally run within its own child control group. Refer to
[this
doc](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md#recommended-cgroups-setup)
for more details on recommended control group hierarchy.
Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't
exist. Kubelet will fail if an invalid cgroup is specified.
### System Reserved
- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi][,][ephemeral-storage=1Gi]`
- **Kubelet Flag**: `--system-reserved-cgroup=`
`system-reserved` is meant to capture resource reservation for OS system daemons
like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the
`kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time.
Reserving resources for user login sessions is also recommended (`user.slice` in
systemd world).
To optionally enforce `system-reserved` on system daemons, specify the parent
control group for OS system daemons as the value for `--system-reserved-cgroup`
kubelet flag.
It is recommended that the OS system daemons are placed under a top level
control group (`system.slice` on systemd machines for example).
Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't
exist. Kubelet will fail if an invalid cgroup is specified.
### Eviction Thresholds
- **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
Memory pressure at the node level leads to System OOMs which affects the entire
node and all pods running on it. Nodes can go offline temporarily until memory
has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
provides [`Out of Resource`](./out-of-resource.md) management. Evictions are
supported for `memory` and `ephemeral-storage` only. By reserving some memory via
`--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory
availability on the node drops below the reserved value. Hypothetically, if
system daemons did not exist on a node, pods cannot use more than `capacity -
eviction-hard`. For this reason, resources reserved for evictions are not
available for pods.
### Enforcing Node Allocatable
- **Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]`
The scheduler treats `Allocatable` as the available `capacity` for pods.
`kubelet` enforce `Allocatable` across pods by default. Enforcement is performed
by evicting pods whenever the overall usage across all pods exceeds
`Allocatable`. More details on eviction policy can be found
[here](./out-of-resource.md#eviction-policy). This enforcement is controlled by
specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.
Optionally, `kubelet` can be made to enforce `kube-reserved` and
`system-reserved` by specifying `kube-reserved` & `system-reserved` values in
the same flag. Note that to enforce `kube-reserved` or `system-reserved`,
`--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified
respectively.
## General Guidelines
System daemons are expected to be treated similar to `Guaranteed` pods. System
daemons can burst within their bounding control groups and this behavior needs
to be managed as part of kubernetes deployments. For example, `kubelet` should
have its own control group and share `Kube-reserved` resources with the
container runtime. However, Kubelet cannot burst and use up all available Node
resources if `kube-reserved` is enforced.
Be extra careful while enforcing `system-reserved` reservation since it can lead
to critical system services being CPU starved or OOM killed on the node. The
recommendation is to enforce `system-reserved` only if a user has profiled their
nodes exhaustively to come up with precise estimates and is confident in their
ability to recover if any process in that group is oom_killed.
* To begin with enforce `Allocatable` on `pods`.
* Once adequate monitoring and alerting is in place to track kube system
daemons, attempt to enforce `kube-reserved` based on usage heuristics.
* If absolutely necessary, enforce `system-reserved` over time.
The resource requirements of kube system daemons may grow over time as more and
more features are added. Over time, kubernetes project will attempt to bring
down utilization of node system daemons, but that is not a priority as of now.
So expect a drop in `Allocatable` capacity in future releases.
## Example Scenario
Here is an example to illustrate Node Allocatable computation:
* Node has `32Gi` of `memory`, `16 CPUs` and `100Gi` of `Storage`
* `--kube-reserved` is set to `cpu=1,memory=2Gi,ephemeral-storage=1Gi`
* `--system-reserved` is set to `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`
* `--eviction-hard` is set to `memory.available<500Mi,nodefs.available<10%`
Under this scenario, `Allocatable` will be `14.5 CPUs`, `28.5Gi` of memory and
`98Gi` of local storage.
Scheduler ensures that the total memory `requests` across all pods on this node does
not exceed `28.5Gi` and storage doesn't exceed `88Gi`.
Kubelet evicts pods whenever the overall memory usage across pods exceeds `28.5Gi`,
or if overall disk usage exceeds `88Gi` If all processes on the node consume as
much CPU as they can, pods together cannot consume more than `14.5 CPUs`.
If `kube-reserved` and/or `system-reserved` is not enforced and system daemons
exceed their reservation, `kubelet` evicts pods whenever the overall node memory
usage is higher than `31.5Gi` or `storage` is greater than `90Gi`
## Feature Availability
As of Kubernetes version 1.2, it has been possible to **optionally** specify
`kube-reserved` and `system-reserved` reservations. The scheduler switched to
using `Allocatable` instead of `Capacity` when available in the same release.
As of Kubernetes version 1.6, `eviction-thresholds` are being considered by
computing `Allocatable`. To revert to the old behavior set
`--experimental-allocatable-ignore-eviction` kubelet flag to `true`.
As of Kubernetes version 1.6, `kubelet` enforces `Allocatable` on pods using
control groups. To revert to the old behavior unset `--enforce-node-allocatable`
kubelet flag. Note that unless `--kube-reserved`, or `--system-reserved` or
`--eviction-hard` flags have non-default values, `Allocatable` enforcement does
not affect existing deployments.
As of Kubernetes version 1.6, `kubelet` launches pods in their own cgroup
sandbox in a dedicated part of the cgroup hierarchy it manages. Operators are
required to drain their nodes prior to upgrade of the `kubelet` from prior
versions in order to ensure pods and their associated containers are launched in
the proper part of the cgroup hierarchy.
As of Kubernetes version 1.7, `kubelet` supports specifying `storage` as a resource
for `kube-reserved` and `system-reserved`.
content/en/docs/tasks/administer-cluster/romana-network-policy.md
+42
View File
@@ -0,0 +1,42 @@
---
reviewers:
- chrismarino
title: Romana for NetworkPolicy
content_template: templates/task
---
{{% capture overview %}}
This page shows how to use Romana for NetworkPolicy.
{{% /capture %}}
{{% capture prerequisites %}}
Complete steps 1, 2, and 3 of the [kubeadm getting started guide](/docs/getting-started-guides/kubeadm/).
{{% /capture %}}
{{% capture steps %}}
## Installing Romana with kubeadm
Follow the [containerized installation guide](https://github.com/romana/romana/tree/master/containerize) for kubeadmin.
## Applying network policies
To apply network policies use one of the following:
* [Romana network policies](https://github.com/romana/romana/wiki/Romana-policies).
* [Example of Romana network policy](https://github.com/romana/core/blob/master/doc/policy.md).
* The NetworkPolicy API.
{{% /capture %}}
{{% capture whatsnext %}}
Once your have installed Romana, you can follow the [Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/) to try out Kubernetes NetworkPolicy.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/running-cloud-controller.md
+93
View File
@@ -0,0 +1,93 @@
---
reviewers:
- luxas
- thockin
- wlan0
title: Kubernetes Cloud Controller Manager
---
{{< feature-state state="alpha" >}}
{{< toc >}}
## Cloud Controller Manager
Kubernetes v1.6 introduced a new binary called `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops. These cloud-specific control loops were originally in the `kube-controller-manager`. Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud vendors to evolve independently from the core Kubernetes code.
The `cloud-controller-manager` can be linked to any cloud provider that satisfies [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go). For backwards compatibility, the [cloud-controller-manager](https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager) provided in the core Kubernetes project uses the same cloud libraries as `kube-controller-manager`. Cloud providers already supported in Kubernetes core are expected to use the in-tree cloud-controller-manager to transition out of Kubernetes core. In future Kubernetes releases, all cloud controller managers will be developed outside of the core Kubernetes project managed by sig leads or cloud vendors.
## Administration
### Requirements
Every cloud has their own set of requirements for running their own cloud provider integration, it should not be too different from the requirements when running `kube-controller-manager`. As a general rule of thumb you'll need:
* cloud authentication/authorization: your cloud may require a token or IAM rules to allow access to their APIs
* kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the kubernetes apiserver
* high availability: like kube-controller-manager, you may want a high available setup for cloud controller manager using leader election (on by default).
### Running cloud-controller-manager
Successfully running cloud-controller-manager requires some changes to your cluster configuration.
* `kube-apiserver` and `kube-controller-manager` MUST NOT specify the `--cloud-provider` flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.
* `kubelet` must run with `--cloud-provider=external`. This is to ensure that the kubelet is aware that it must be initialized by the cloud controller manager before it is scheduled any work.
* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller since the cloud controller manager takes over labeling persistent volumes. To prevent the PersistentVolumeLabel admission plugin from running in `kube-apiserver`, include the `PersistentVolumeLabel` as a listed value in the `--disable-admission-plugins` flag.
* For the `cloud-controller-manager` to label persistent volumes, initializers will need to be enabled and an InitializerConifguration needs to be added to the system. Follow [these instructions](/docs/admin/extensible-admission-controllers.md#enable-initializers-alpha-feature) to enable initializers. Use the following YAML to create the InitializerConfiguration:
{{< code file="persistent-volume-label-initializer-config.yaml" >}}
Keep in mind that setting up your cluster to use cloud controller manager will change your cluster behaviour in a few ways:
* kubelets specifying `--cloud-provider=external` will add a taint `node.cloudprovider.kubernetes.io/uninitialized` with an effect `NoSchedule` during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that cloud controller manager is not available, new nodes in the cluster will be left unschedulable. The taint is important since the scheduler may require cloud specific information about nodes such as their region or type (high cpu, gpu, high memory, spot instance, etc).
* cloud information about nodes in the cluster will no longer be retrieved using local metadata, but instead all API calls to retrieve node information will go through cloud controller manager. This may mean you can restrict access to your cloud API on the kubelets for better security. For larger clusters you may want to consider if cloud controller manager will hit rate limits since it is now responsible for almost all API calls to your cloud from within the cluster.
As of v1.8, cloud controller manager can implement:
* node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
* service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
* route controller - responsible for setting up network routes on your cloud
* [PersistentVolumeLabel Admission Controller](/docs/admin/admission-controllers#persistentvolumelabel) - responsible for labeling persistent volumes on your cloud - ensure that the persistent volume label admission plugin is not enabled on your kube-apiserver.
* any other features you would like to implement if you are running an out-of-tree provider.
## Examples
If you are using a cloud that is currently supported in Kubernetes core and would like to adopt cloud controller manager, see the [cloud controller manager in kubernetes core](https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager).
For cloud controller managers not in Kubernetes core, you can find the respective projects in repos maintained by cloud vendors or sig leads.
* [DigitalOcean](https://github.com/digitalocean/digitalocean-cloud-controller-manager)
* [keepalived](https://github.com/munnerz/keepalived-cloud-provider)
* [Oracle Cloud Infrastructure](https://github.com/oracle/oci-cloud-controller-manager)
* [Rancher](https://github.com/rancher/rancher-cloud-controller-manager)
For providers already in Kubernetes core, you can run the in-tree cloud controller manager as a Daemonset in your cluster, use the following as a guideline:
{{< code file="cloud-controller-manager-daemonset-example.yaml" >}}
## Limitations
Running cloud controller manager comes with a few possible limitations. Although these limitations are being addressed in upcoming releases, it's important that you are aware of these limitations for production workloads.
### Support for Volumes
Cloud controller manager does not implement any of the volume controllers found in `kube-controller-manager` as the volume integrations also require coordination with kubelets. As we evolve CSI (container storage interface) and add stronger support for flex volume plugins, necessary support will be added to cloud controller manager so that clouds can fully integrate with volumes. Learn more about out-of-tree CSI volume plugins [here](https://github.com/kubernetes/features/issues/178).
### Scalability
In the previous architecture for cloud providers, we relied on kubelets using a local metadata service to retrieve node information about itself. With this new architecture, we now fully rely on the cloud controller managers to retrieve information for all nodes. For very larger clusters, you should consider possible bottle necks such as resource requirements and API rate limiting.
### Chicken and Egg
The goal of the cloud controller manager project is to decouple development of cloud features from the core Kubernetes project. Unfortunately, many aspects of the Kubernetes project has assumptions that cloud provider features are tightly integrated into the project. As a result, adopting this new architecture can create several situations where a request is being made for information from a cloud provider, but the cloud controller manager may not be able to return that information without the original request being complete.
A good example of this is the TLS bootstrapping feature in the Kubelet. Currently, TLS bootstrapping assumes that the Kubelet has the ability to ask the cloud provider (or a local metadata service) for all its address types (private, public, etc) but cloud controller manager cannot set a node's address types without being initialized in the first place which requires that the kubelet has TLS certificates to communicate with the apiserver.
As this initiative evolves, changes will be made to address these issues in upcoming releases.
## Developing your own Cloud Controller Manager
To build and develop your own cloud controller manager, read the [Developing Cloud Controller Manager](/docs/tasks/administer-cluster/developing-cloud-controller-manager.md) doc.
content/en/docs/tasks/administer-cluster/safely-drain-node.md
+162
View File
@@ -0,0 +1,162 @@
---
reviewers:
- davidopp
- mml
- foxish
- kow3ns
title: Safely Drain a Node while Respecting Application SLOs
content_template: templates/task
---
{{% capture overview %}}
This page shows how to safely drain a machine, respecting the application-level
disruption SLOs you have specified using PodDisruptionBudget.
{{% /capture %}}
{{% capture prerequisites %}}
This task assumes that you have met the following prerequisites:
* You are using Kubernetes release >= 1.5.
* Either:
1. You do not require your applications to be highly available during the
node drain, or
1. You have read about the [PodDisruptionBudget concept](/docs/concepts/workloads/pods/disruptions/)
and [Configured PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb/) for
applications that need them.
{{% /capture %}}
{{% capture steps %}}
## Use `kubectl drain` to remove a node from service
You can use `kubectl drain` to safely evict all of your pods from a
node before you perform maintenance on the node (e.g. kernel upgrade,
hardware maintenance, etc.). Safe evictions allow the pod's containers
to [gracefully terminate](/docs/concepts/workloads/pods/pod/#termination-of-pods)
and will respect the `PodDisruptionBudgets` you have specified.
**Note:** By default `kubectl drain` will ignore certain system pods on the node
that cannot be killed; see
the [kubectl drain](/docs/reference/generated/kubectl/kubectl-commands/#drain)
documentation for more details.
When `kubectl drain` returns successfully, that indicates that all of
the pods (except the ones excluded as described in the previous paragraph)
have been safely evicted (respecting the desired graceful
termination period, and without violating any application-level
disruption SLOs). It is then safe to bring down the node by powering
down its physical machine or, if running on a cloud platform, deleting its
virtual machine.
First, identify the name of the node you wish to drain. You can list all of the nodes in your cluster with
```shell
kubectl get nodes
```
Next, tell Kubernetes to drain the node:
```shell
kubectl drain <node name>
```
Once it returns (without giving an error), you can power down the node
(or equivalently, if on a cloud platform, delete the virtual machine backing the node).
If you leave the node in the cluster during the maintenance operation, you need to run
```shell
kubectl uncordon <node name>
```
afterwards to tell Kubernetes that it can resume scheduling new pods onto the node.
## Draining multiple nodes in parallel
The `kubectl drain` command should only be issued to a single node at a
time. However, you can run multiple `kubectl drain` commands for
different node in parallel, in different terminals or in the
background. Multiple drain commands running concurrently will still
respect the `PodDisruptionBudget` you specify.
For example, if you have a StatefulSet with three replicas and have
set a `PodDisruptionBudget` for that set specifying `minAvailable:
2`. `kubectl drain` will only evict a pod from the StatefulSet if all
three pods are ready, and if you issue multiple drain commands in
parallel, Kubernetes will respect the PodDisruptionBudget and ensure
that only one pod is unavailable at any given time. Any drains that
would cause the number of ready replicas to fall below the specified
budget are blocked.
## The Eviction API
If you prefer not to use [kubectl drain](/docs/reference/generated/kubectl/kubectl-commands/#drain) (such as
to avoid calling to an external command, or to get finer control over the pod
eviction process), you can also programmatically cause evictions using the eviction API.
You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api/#programmatic-access-to-the-api).
The eviction subresource of a
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
*create* an eviction), you POST an attempted operation. Here's an example:
```json
{
"apiVersion": "policy/v1beta1",
"kind": "Eviction",
"metadata": {
"name": "quux",
"namespace": "default"
}
}
```
You can attempt an eviction using `curl`:
```bash
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
```
The API can respond in one of three ways:
- If the eviction is granted, then the pod is deleted just as if you had sent
a `DELETE` request to the pod's URL and you get back `200 OK`.
- If the current state of affairs wouldn't allow an eviction by the rules set
forth in the budget, you get back `429 Too Many Requests`. This is
typically used for generic rate limiting of *any* requests, but here we mean
that this request isn't allowed *right now* but it may be allowed later.
Currently, callers do not get any `Retry-After` advice, but they may in
future versions.
- If there is some kind of misconfiguration, like multiple budgets pointing at
the same pod, you will get `500 Internal Server Error`.
For a given eviction request, there are two cases:
- There is no budget that matches this pod. In this case, the server always
returns `200 OK`.
- There is at least one budget. In this case, any of the three above responses may
apply.
In some cases, an application may reach a broken state where it will never return anything
other than 429 or 500. This can happen, for example, if the replacement pod created by the
application's controller does not become ready, or if the last pod evicted has a very long
termination grace period.
In this case, there are two potential solutions:
- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation.
- After a suitably long wait, `DELETE` the pod instead of using the eviction API.
Kubernetes does not specify what the behavior should be in this case; it is up to the
application owners and cluster owners to establish an agreement on behavior in these cases.
{{% /capture %}}
{{% capture whatsnext %}}
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application/configure-pdb/).
{{% /capture %}}
content/en/docs/tasks/administer-cluster/securing-a-cluster.md
+227
View File
@@ -0,0 +1,227 @@
---
reviewers:
- smarterclayton
- liggitt
- ericchiang
- destijl
title: Securing a Cluster
content_template: templates/task
---
{{% capture overview %}}
This document covers topics related to protecting a cluster from accidental or malicious access
and provides recommendations on overall security.
{{% /capture %}}
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Controlling access to the Kubernetes API
As Kubernetes is entirely API driven, controlling and limiting who can access the cluster and what actions
they are allowed to perform is the first line of defense.
### Use Transport Level Security (TLS) for all API traffic
Kubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the
majority of installation methods will allow the necessary certificates to be created and distributed to
the cluster components. Note that some components and installation methods may enable local ports over
HTTP and administrators should familiarize themselves with the settings of each component to identify
potentially unsecured traffic.
### API Authentication
Choose an authentication mechanism for the API servers to use that matches the common access patterns
when you install a cluster. For instance, small single user clusters may wish to use a simple certificate
or static Bearer token approach. Larger clusters may wish to integrate an existing OIDC or LDAP server that
allow users to be subdivided into groups.
All API clients must be authenticated, even those that are part of the infrastructure like nodes,
proxies, the scheduler, and volume plugins. These clients are typically [service accounts](/docs/admin/service-accounts-admin/) or use x509 client certificates, and they are created automatically at cluster startup or are setup as part of the cluster installation.
Consult the [authentication reference document](/docs/admin/authentication/) for more information.
### API Authorization
Once authenticated, every API call is also expected to pass an authorization check. Kubernetes ships
an integrated [Role-Based Access Control (RBAC)](/docs/admin/authorization/rbac/) component that matches an incoming user or group to a
set of permissions bundled into roles. These permissions combine verbs (get, create, delete) with
resources (pods, services, nodes) and can be namespace or cluster scoped. A set of out of the box
roles are provided that offer reasonable default separation of responsibility depending on what
actions a client might want to perform. It is recommended that you use the [Node](/docs/admin/authorization/node/) and [RBAC](/docs/admin/authorization/rbac/) authorizers together, in combination with the
[NodeRestriction](/docs/admin/admission-controllers/#noderestriction) admission plugin.
As with authentication, simple and broad roles may be appropriate for smaller clusters, but as
more users interact with the cluster, it may become necessary to separate teams into separate
namespaces with more limited roles.
With authorization, it is important to understand how updates on one object may cause actions in
other places. For instance, a user may not be able to create pods directly, but allowing them to
create a deployment, which creates pods on their behalf, will let them create those pods
indirectly. Likewise, deleting a node from the API will result in the pods scheduled to that node
being terminated and recreated on other nodes. The out of the box roles represent a balance
between flexibility and the common use cases, but more limited roles should be carefully reviewed
to prevent accidental escalation. You can make roles specific to your use case if the out-of-box ones don't meet your needs.
Consult the [authorization reference section](/docs/admin/authorization/) for more information.
## Controlling access to the Kubelet
Kubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default Kubelets allow unauthenticated access to this API.
Production clusters should enable Kubelet authentication and authorization.
Consult the [Kubelet authentication/authorization reference](/docs/admin/kubelet-authentication-authorization) for more information.
## Controlling the capabilities of a workload or user at runtime
Authorization in Kubernetes is intentionally high level, focused on coarse actions on resources.
More powerful controls exist as **policies** to limit by use case how those objects act on the
cluster, themselves, and other resources.
### Limiting resource usage on a cluster
[Resource quota](/docs/concepts/policy/resource-quotas/) limits the number or capacity of
resources granted to a namespace. This is most often used to limit the amount of CPU, memory,
or persistent disk a namespace can allocate, but can also control how many pods, services, or
volumes exist in each namespace.
[Limit ranges](/docs/tasks/administer-cluster/memory-default-namespace/) restrict the maximum or minimum size of some of the
resources above, to prevent users from requesting unreasonably high or low values for commonly
reserved resources like memory, or to provide default limits when none are specified.
### Controlling what privileges containers run with
A pod definition contains a [security context](/docs/tasks/configure-pod-container/security-context/)
that allows it to request access to running as a specific Linux user on a node (like root),
access to run privileged or access the host network, and other controls that would otherwise
allow it to run unfettered on a hosting node. [Pod security policies](/docs/concepts/policy/pod-security-policy/)
can limit which users or service accounts can provide dangerous security context settings. For example, pod security policies can limit volume mounts, especially `hostPath`, which are aspects of a pod that should be controlled.
Generally, most application workloads need limited access to host resources so they can
successfully run as a root process (uid 0) without access to host information. However,
considering the privileges associated with the root user, you should write application
containers to run as a non-root user. Similarly, administrators who wish to prevent
client applications from escaping their containers should use a restrictive pod security
policy.
### Restricting network access
The [network policies](/docs/tasks/administer-cluster/declare-network-policy/) for a namespace
allows application authors to restrict which pods in other namespaces may access pods and ports
within their namespaces. Many of the supported [Kubernetes networking providers](/docs/concepts/cluster-administration/networking/)
now respect network policy.
Quota and limit ranges can also be used to control whether users may request node ports or
load balanced services, which on many clusters can control whether those users applications
are visible outside of the cluster.
Additional protections may be available that control network rules on a per plugin or per
environment basis, such as per-node firewalls, physically separating cluster nodes to
prevent cross talk, or advanced networking policy.
### Restricting cloud metadata API access
Cloud platforms (AWS, Azure, GCE, etc.) often expose metadata services locally to instances.
By default these APIs are accessible by pods running on an instance and can contain cloud
credentials for that node, or provisioning data such as kubelet credentials. These credentials
can be used to escalate within the cluster or to other cloud services under the same account.
When running Kubernetes on a cloud platform limit permissions given to instance credentials, use
[network policies](/docs/tasks/administer-cluster/declare-network-policy/) to restrict pod access
to the metadata API, and avoid using provisioning data to deliver secrets.
### Controlling which nodes pods may access
By default, there are no restrictions on which nodes may run a pod. Kubernetes offers a
[rich set of policies for controlling placement of pods onto nodes](/docs/concepts/configuration/assign-pod-node/)
and the [taint based pod placement and eviction](/docs/concepts/configuration/taint-and-toleration/)
that are available to end users. For many clusters use of these policies to separate workloads
can be a convention that authors adopt or enforce via tooling.
As an administrator, a beta admission plugin `PodNodeSelector` can be used to force pods
within a namespace to default or require a specific node selector, and if end users cannot
alter namespaces, this can strongly limit the placement of all of the pods in a specific workload.
## Protecting cluster components from compromise
This section describes some common patterns for protecting clusters from compromise.
### Restrict access to etcd
Write access to the etcd backend for the API is equivalent to gaining root on the entire cluster,
and read access can be used to escalate fairly quickly. Administrators should always use strong
credentials from the API servers to their etcd server, such as mutual auth via TLS client certificates,
and it is often recommended to isolate the etcd servers behind a firewall that only the API servers
may access.
**CAUTION:** Allowing other components within the cluster to access the master etcd instance with
read or write access to the full keyspace is equivalent to granting cluster-admin access. Using
separate etcd instances for non-master components or using etcd ACLs to restrict read and write
access to a subset of the keyspace is strongly recommended.
### Enable audit logging
The [audit logger](/docs/tasks/debug-application-cluster/audit/) is a beta feature that records actions taken by the
API for later analysis in the event of a compromise. It is recommended to enable audit logging
and archive the audit file on a secure server.
### Restrict access to alpha or beta features
Alpha and beta Kubernetes features are in active development and may have limitations or bugs
that result in security vulnerabilities. Always assess the value an alpha or beta feature may
provide against the possible risk to your security posture. When in doubt, disable features you
do not use.
### Rotate infrastructure credentials frequently
The shorter the lifetime of a secret or credential the harder it is for an attacker to make
use of that credential. Set short lifetimes on certificates and automate their rotation. Use
an authentication provider that can control how long issued tokens are available and use short
lifetimes where possible. If you use service account tokens in external integrations, plan to
rotate those tokens frequently. For example, once the bootstrap phase is complete, a bootstrap token used for setting up nodes should be revoked or its authorization removed.
### Review third party integrations before enabling them
Many third party integrations to Kubernetes may alter the security profile of your cluster. When
enabling an integration, always review the permissions that an extension requests before granting
it access. For example, many security integrations may request access to view all secrets on
your cluster which is effectively making that component a cluster admin. When in doubt,
restrict the integration to functioning in a single namespace if possible.
Components that create pods may also be unexpectedly powerful if they can do so inside namespaces
like the `kube-system` namespace, because those pods can gain access to service account secrets
or run with elevated permissions if those service accounts are granted access to permissive
[pod security policies](/docs/concepts/policy/pod-security-policy/).
### Encrypt secrets at rest
In general, the etcd database will contain any information accessible via the Kubernetes API
and may grant an attacker significant visibility into the state of your cluster. Always encrypt
your backups using a well reviewed backup and encryption solution, and consider using full disk
encryption where possible.
Kubernetes 1.7 contains [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/), an alpha feature that will encrypt `Secret` resources in etcd, preventing
parties that gain access to your etcd backups from viewing the content of those secrets. While
this feature is currently experimental, it may offer an additional level of defense when backups
are not encrypted or an attacker gains read access to etcd.
### Receiving alerts for security updates and reporting vulnerabilities
Join the [kubernetes-announce](https://groups.google.com/forum/#!forum/kubernetes-announce)
group for emails about security announcements. See the [security reporting](/security/)
page for more on how to report vulnerabilities.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/static-pod.md
+127
View File
@@ -0,0 +1,127 @@
---
reviewers:
- jsafrane
title: Static Pods
---
**If you are running clustered Kubernetes and are using static pods to run a pod on every node, you should probably be using a [DaemonSet](/docs/concepts/workloads/controllers/daemonset/)!**
*Static pods* are managed directly by kubelet daemon on a specific node, without the API server observing it. It does not have an associated replication controller, and kubelet daemon itself watches it and restarts it when it crashes. There is no health check. Static pods are always bound to one kubelet daemon and always run on the same node with it.
Kubelet automatically creates so-called *mirror pod* on the Kubernetes API server for each static pod, so the pods are visible there, but they cannot be controlled from the API server.
## Static pod creation
Static pod can be created in two ways: either by using configuration file(s) or by HTTP.
### Configuration files
The configuration files are just standard pod definitions in json or yaml format in a specific directory. Use `kubelet --pod-manifest-path=<the directory>` to start kubelet daemon, which periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there.
Note that kubelet will ignore files starting with dots when scanning the specified directory.
For example, this is how to start a simple web server as a static pod:
1. Choose a node where we want to run the static pod. In this example, it's `my-node1`.
```
[joe@host ~] $ ssh my-node1
```
2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`:
```shell
[root@my-node1 ~] $ mkdir /etc/kubelet.d/
[root@my-node1 ~] $ cat <<EOF >/etc/kubelet.d/static-web.yaml
apiVersion: v1
kind: Pod
metadata:
name: static-web
labels:
role: myrole
spec:
containers:
- name: web
image: nginx
ports:
- name: web
containerPort: 80
protocol: TCP
EOF
```
3. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument.
On Fedora edit `/etc/kubernetes/kubelet` to include this line:
```
KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/"
```
Instructions for other distributions or Kubernetes installations may vary.
4. Restart kubelet. On Fedora, this is:
```
[root@my-node1 ~] $ systemctl restart kubelet
```
### Pods created via HTTP
Kubelet periodically downloads a file specified by `--manifest-url=<URL>` argument and interprets it as a json/yaml file with a pod definition. It works the same as `--pod-manifest-path=<directory>`, i.e. it's reloaded every now and then and changes are applied to running static pods (see below).
## Behavior of static pods
When kubelet starts, it automatically starts all pods defined in directory specified in `--pod-manifest-path=` or `--manifest-url=` arguments, i.e. our static-web. (It may take some time to pull nginx image, be patient…):
```shell
[joe@my-node1 ~] $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c
```
If we look at our Kubernetes API server (running on host `my-master`), we see that a new mirror-pod was created there too:
```shell
[joe@host ~] $ ssh my-master
[joe@my-master ~] $ kubectl get pods
NAME READY STATUS RESTARTS AGE
static-web-my-node1 1/1 Running 0 2m
```
Labels from the static pod are propagated into the mirror-pod and can be used as usual for filtering.
Notice we cannot delete the pod with the API server (e.g. via [`kubectl`](/docs/user-guide/kubectl/) command), kubelet simply won't remove it.
```shell
[joe@my-master ~] $ kubectl delete pod static-web-my-node1
pod "static-web-my-node1" deleted
[joe@my-master ~] $ kubectl get pods
NAME READY STATUS RESTARTS AGE
static-web-my-node1 1/1 Running 0 12s
```
Back to our `my-node1` host, we can try to stop the container manually and see, that kubelet automatically restarts it in a while:
```shell
[joe@host ~] $ ssh my-node1
[joe@my-node1 ~] $ docker stop f6d05272b57e
[joe@my-node1 ~] $ sleep 20
[joe@my-node1 ~] $ docker ps
CONTAINER ID IMAGE COMMAND CREATED ...
5b920cbaf8b1 nginx:latest "nginx -g 'daemon of 2 seconds ago ...
```
## Dynamic addition and removal of static pods
Running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes pods as files appear/disappear in this directory.
```shell
[joe@my-node1 ~] $ mv /etc/kubelet.d/static-web.yaml /tmp
[joe@my-node1 ~] $ sleep 20
[joe@my-node1 ~] $ docker ps
// no nginx container is running
[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubelet.d/
[joe@my-node1 ~] $ sleep 20
[joe@my-node1 ~] $ docker ps
CONTAINER ID IMAGE COMMAND CREATED ...
e7a62e3427f1 nginx:latest "nginx -g 'daemon of 27 seconds ago
```
content/en/docs/tasks/administer-cluster/storage-object-in-use-protection.md
+316
View File
@@ -0,0 +1,316 @@
---
approvers:
- msau42
- jsafrane
title: Storage Object in Use Protection
content_template: templates/task
---
{{% capture overview %}}
{{< feature-state for_k8s_version="v1.10" state="beta" >}}
Persistent volume claims (PVCs) that are in active use by a pod and persistent volumes (PVs) that are bound to PVCs can be protected from pre-mature removal.
{{% /capture %}}
{{% capture prerequisites %}}
- The Storage Object in Use Protection feature is enabled in a version of Kubernetes in which it is supported.
{{% /capture %}}
{{% capture steps %}}
## Storage Object in Use Protection feature used for PVC Protection
The example below uses a GCE PD `StorageClass`, however, similar steps can be performed for any volume type.
Create a `StorageClass` for convenient storage provisioning:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
```
Verification scenarios follow below.
### Scenario 1: The PVC is not in active use by a pod
- Create a PVC:
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: slzc
spec:
accessModes:
- ReadWriteOnce
storageClassName: slow
resources:
requests:
storage: 3.7Gi
```
- Check that the PVC has the finalizer `kubernetes.io/pvc-protection` set:
```shell
kubectl describe pvc slzc
Name: slzc
Namespace: default
StorageClass: slow
Status: Bound
Volume: pvc-bee8c30a-d6a3-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 2m persistentvolume-controller Successfully provisioned volume pvc-bee8c30a-d6a3-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Delete the PVC and check that the PVC (not in active use by a pod) was removed successfully.
### Scenario 2: The PVC is in active use by a pod
- Again, create the same PVC.
- Create a pod that uses the PVC:
```yaml
kind: Pod
apiVersion: v1
metadata:
name: app1
spec:
containers:
- name: test-pod
image: k8s.gcr.io/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "date > /mnt/app1.txt; sleep 60 && exit 0 || exit 1"
volumeMounts:
- name: path-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: path-pvc
persistentVolumeClaim:
claimName: slzc
```
- Wait until the pod status is `Running`, i.e. the PVC becomes in active use.
- Delete the PVC that is now in active use by a pod and verify that the PVC is not removed but its status is `Terminating`:
```shell
Name: slzc
Namespace: default
StorageClass: slow
Status: Terminating (since Fri, 01 Dec 2017 14:47:55 +0000)
Volume: pvc-803a1f4d-d6a6-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 52s persistentvolume-controller Successfully provisioned volume pvc-803a1f4d-d6a6-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Wait until the pod status is `Terminated` (either delete the pod or wait until it finishes). Afterwards, check that the PVC is removed.
### Scenario 3: A pod starts using a PVC that is in Terminating state
- Again, create the same PVC.
- Create a first pod that uses the PVC:
```yaml
kind: Pod
apiVersion: v1
metadata:
name: app1
spec:
containers:
- name: test-pod
image: k8s.gcr.io/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "date > /mnt/app1.txt; sleep 600 && exit 0 || exit 1"
volumeMounts:
- name: path-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: path-pvc
persistentVolumeClaim:
claimName: slzc
```
- Wait until the pod status is `Running`, i.e. the PVC becomes in active use.
- Delete the PVC that is now in active use by a pod and verify that the PVC is not removed but its status is `Terminating`:
```shell
Name: slzc
Namespace: default
StorageClass: slow
Status: Terminating (since Fri, 01 Dec 2017 14:47:55 +0000)
Volume: pvc-803a1f4d-d6a6-11e7-9af0-42010a800002
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 4Gi
Access Modes: RWO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 52s persistentvolume-controller Successfully provisioned volume pvc-803a1f4d-d6a6-11e7-9af0-42010a800002 using kubernetes.io/gce-pd
```
- Create a second pod that uses the same PVC:
```
kind: Pod
apiVersion: v1
metadata:
name: app2
spec:
containers:
- name: test-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "date > /mnt/app1.txt; sleep 600 && exit 0 || exit 1"
volumeMounts:
- name: path-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: path-pvc
persistentVolumeClaim:
claimName: slzc
```
- Verify that the scheduling of the second pod fails with the below warning:
```
Warning FailedScheduling 18s (x4 over 21s) default-scheduler persistentvolumeclaim "slzc" is being deleted
```
- Wait until the pod status of both pods is `Terminated` or `Completed` (either delete the pods or wait until they finish). Afterwards, check that the PVC is removed.
## Storage Object in Use Protection feature used for PV Protection
The example below uses a `HostPath` PV.
Verification scenarios follow below.
### Scenario 1: The PV is not bound to a PVC
- Create a PV:
```yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: task-pv-volume
labels:
type: local
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: standard
hostPath:
path: "/tmp/data"
```
- Check that the PV has the finalizer `kubernetes.io/pv-protection` set:
```shell
Name: task-pv-volume
Labels: type=local
Annotations: pv.kubernetes.io/bound-by-controller=yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Terminating (lasts 1m)
Claim: default/task-pv-claim
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 1Gi
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /tmp/data
HostPathType:
Events: <none>
```
- Delete the PV and check that the PV (not bound to a PVC) is removed successfully.
### Scenario 2: The PV is bound to a PVC
- Again, create the same PV.
- Create a PVC
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: task-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
- Wait until the PV and PVC are bound to each other.
- Delete the PV and verify that the PV is not removed but its status is `Terminating`:
```shell
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
task-pv-volume 1Gi RWO Delete Terminating default/task-pv-claim standard 59s
```
- Delete the PVC and verify that the PV is removed too.
```shell
kubectl delete pvc task-pv-claim
persistentvolumeclaim "task-pv-claim" deleted
$ kubectl get pvc
No resources found.
$ kubectl get pv
No resources found.
```
{{% /capture %}}
{{% capture discussion %}}
{{% /capture %}}
content/en/docs/tasks/administer-cluster/sysctl-cluster.md
+169
View File
@@ -0,0 +1,169 @@
---
title: Using Sysctls in a Kubernetes Cluster
reviewers:
- sttts
content_template: templates/task
---
{{% capture overview %}}
This document describes how sysctls are used within a Kubernetes cluster.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
{{% capture steps %}}
## Listing all Sysctl Parameters
In Linux, the sysctl interface allows an administrator to modify kernel
parameters at runtime. Parameters are available via the `/proc/sys/` virtual
process file system. The parameters cover various subsystems such as:
- kernel (common prefix: `kernel.`)
- networking (common prefix: `net.`)
- virtual memory (common prefix: `vm.`)
- MDADM (common prefix: `dev.`)
- More subsystems are described in [Kernel docs](https://www.kernel.org/doc/Documentation/sysctl/README).
To get a list of all parameters, you can run
```shell
$ sudo sysctl -a
```
## Enabling Unsafe Sysctls
Sysctls are grouped into _safe_ and _unsafe_ sysctls. In addition to proper
namespacing a _safe_ sysctl must be properly _isolated_ between pods on the same
node. This means that setting a _safe_ sysctl for one pod
- must not have any influence on any other pod on the node
- must not allow to harm the node's health
- must not allow to gain CPU or memory resources outside of the resource limits
of a pod.
By far, most of the _namespaced_ sysctls are not necessarily considered _safe_.
The following sysctls are supported in the _safe_ set:
- `kernel.shm_rmid_forced`,
- `net.ipv4.ip_local_port_range`,
- `net.ipv4.tcp_syncookies`.
{{< note >}}
**Note**: The example `net.ipv4.tcp_syncookies` is not namespaced on Linux kernel version 4.4 or lower.
{{< /note >}}
This list will be extended in future Kubernetes versions when the kubelet
supports better isolation mechanisms.
All _safe_ sysctls are enabled by default.
All _unsafe_ sysctls are disabled by default and must be allowed manually by the
cluster admin on a per-node basis. Pods with disabled unsafe sysctls will be
scheduled, but will fail to launch.
With the warning above in mind, the cluster admin can allow certain _unsafe_
sysctls for very special situations like e.g. high-performance or real-time
application tuning. _Unsafe_ sysctls are enabled on a node-by-node basis with a
flag of the kubelet, e.g.:
```shell
$ kubelet --experimental-allowed-unsafe-sysctls \
'kernel.msg*,net.ipv4.route.min_pmtu' ...
```
For minikube, this can be done via the `extra-config` flag:
```shell
$ minikube start --extra-config="kubelet.AllowedUnsafeSysctls=kernel.msg*,net.ipv4.route.min_pmtu"...
```
Only _namespaced_ sysctls can be enabled this way.
## Setting Sysctls for a Pod
A number of sysctls are _namespaced_ in today's Linux kernels. This means that
they can be set independently for each pod on a node. Being namespaced is a
requirement for sysctls to be accessible in a pod context within Kubernetes.
The following sysctls are known to be _namespaced_:
- `kernel.shm*`,
- `kernel.msg*`,
- `kernel.sem`,
- `fs.mqueue.*`,
- `net.*`.
Sysctls which are not namespaced are called _node-level_ and must be set
manually by the cluster admin, either by means of the underlying Linux
distribution of the nodes (e.g. via `/etc/sysctls.conf`) or using a DaemonSet
with privileged containers.
The sysctl feature is an alpha API. Therefore, sysctls are set using annotations
on pods. They apply to all containers in the same pod.
Here is an example, with different annotations for _safe_ and _unsafe_ sysctls:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: sysctl-example
annotations:
security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1
security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.route.min_pmtu=1000,kernel.msgmax=1 2 3
spec:
...
```
{{% /capture %}}
{{% capture discussion %}}
{{< warning >}}
**Warning**: Due to their nature of being _unsafe_, the use of _unsafe_ sysctls
is at-your-own-risk and can lead to severe problems like wrong behavior of
containers, resource shortage or complete breakage of a node.
{{< /warning >}}
It is good practice to consider nodes with special sysctl settings as
_tainted_ within a cluster, and only schedule pods onto them which need those
sysctl settings. It is suggested to use the Kubernetes [_taints and toleration_
feature](/docs/reference/generated/kubectl/kubectl-commands/#taint) to implement this.
A pod with the _unsafe_ sysctls will fail to launch on any node which has not
enabled those two _unsafe_ sysctls explicitly. As with _node-level_ sysctls it
is recommended to use
[_taints and toleration_ feature](/docs/reference/generated/kubectl/kubectl-commands/#taint) or
[taints on nodes](/docs/concepts/configuration/taint-and-toleration/)
to schedule those pods onto the right nodes.
## PodSecurityPolicy Annotations
The use of sysctl in pods can be controlled via annotation on the PodSecurityPolicy.
Sysctl annotation represents a whitelist of allowed safe and unsafe sysctls
in a pod spec. It's a comma-separated list of plain sysctl names or sysctl patterns
(which end in `*`). The string `*` matches all sysctls.
Here is an example, it authorizes binding user creating pod with corresponding sysctls.
```yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: sysctl-psp
annotations:
security.alpha.kubernetes.io/sysctls: 'net.ipv4.route.*,kernel.msg*'
spec:
...
```
{{% /capture %}}
content/en/docs/tasks/administer-cluster/upgrade-downgrade/_index.md
Executable
+5
View File
@@ -0,0 +1,5 @@
---
title: "Upgrading or downgrading Kubernetes"
weight: 10
---
content/en/docs/tasks/administer-cluster/upgrade-downgrade/kubeadm-upgrade-1-7.md
+96
View File
@@ -0,0 +1,96 @@
---
reviewers:
- pipejakob
title: Upgrading kubeadm clusters from 1.6 to 1.7
content_template: templates/task
---
{{% capture overview %}}
This guide is for upgrading kubeadm clusters from version 1.6.x to 1.7.x.
Upgrades are not supported for clusters lower than 1.6, which is when kubeadm
became Beta.
**WARNING**: These instructions will **overwrite** all of the resources managed
by kubeadm (static pod manifest files, service accounts and RBAC rules in the
`kube-system` namespace, etc.), so any customizations you may have made to these
resources after cluster setup will need to be reapplied after the upgrade. The
upgrade will not disturb other static pod manifest files or objects outside the
`kube-system` namespace.
{{% /capture %}}
{{% capture prerequisites %}}
You need to have a Kubernetes cluster running version 1.6.x.
{{% /capture %}}
{{% capture steps %}}
## On the master
1. Upgrade system packages.
Upgrade your OS packages for kubectl, kubeadm, kubelet, and kubernetes-cni.
a. On Debian, this can be accomplished with:
sudo apt-get update
sudo apt-get upgrade
b. On CentOS/Fedora, you would instead run:
sudo yum update
2. Restart kubelet.
systemctl restart kubelet
3. Delete the `kube-proxy` DaemonSet.
Although most components are automatically upgraded by the next step,
`kube-proxy` currently needs to be manually deleted so it can be recreated at
the correct version:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete daemonset kube-proxy -n kube-system
4. Perform kubeadm upgrade.
**WARNING**: All parameters you passed to the first `kubeadm init` when you bootstrapped your
cluster **MUST** be specified here in the upgrade-`kubeadm init`-command. This is a limitation
we plan to address in v1.8.
sudo kubeadm init --skip-preflight-checks --kubernetes-version <DESIRED_VERSION>
For instance, if you want to upgrade to `1.7.0`, you would run:
sudo kubeadm init --skip-preflight-checks --kubernetes-version v1.7.0
5. Upgrade CNI provider.
Your CNI provider might have its own upgrade instructions to follow now.
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
find your CNI provider and see if there are additional upgrade steps
necessary.
## On each node
1. Upgrade system packages.
Upgrade your OS packages for kubectl, kubeadm, kubelet, and kubernetes-cni.
a. On Debian, this can be accomplished with:
sudo apt-get update
sudo apt-get upgrade
b. On CentOS/Fedora, you would instead run:
sudo yum update
2. Restart kubelet.
systemctl restart kubelet
{{% /capture %}}
content/en/docs/tasks/administer-cluster/upgrade-downgrade/kubeadm-upgrade-1-8.md
+290
View File
@@ -0,0 +1,290 @@
---
reviewers:
- pipejakob
- luxas
- roberthbailey
- jbeda
title: Upgrading kubeadm clusters from 1.7 to 1.8
content_template: templates/task
---
{{% capture overview %}}
This guide is for upgrading `kubeadm` clusters from version 1.7.x to 1.8.x, as well as 1.7.x to 1.7.y and 1.8.x to 1.8.y where `y > x`.
See also [upgrading kubeadm clusters from 1.6 to 1.7](/docs/tasks/administer-cluster/kubeadm-upgrade-1-7/) if you're on a 1.6 cluster currently.
{{% /capture %}}
{{% capture prerequisites %}}
Before proceeding:
- You need to have a functional `kubeadm` Kubernetes cluster running version 1.7.0 or higher in order to use the process described here.
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#v180-beta1) carefully.
- As `kubeadm upgrade` does not upgrade etcd make sure to back it up. You can, for example, use `etcdctl backup` to take care of this.
- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up what's important to you. For example, any app-level state, such as a database an app might depend on (like MySQL or MongoDB) must be backed up beforehand.
Also, note that only one minor version upgrade is supported. That is, you can only upgrade from, say 1.7 to 1.8, not from 1.7 to 1.9.
{{% /capture %}}
{{% capture steps %}}
## Upgrading your control plane
You have to carry out the following steps by executing these commands on your master node:
1. Install the most recent version of `kubeadm` using `curl` like so:
{{< caution >}}
```shell
export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
chmod a+rx /usr/bin/kubeadm
```
**Caution:** Upgrading the `kubeadm` package on your system prior to
upgrading the control plane causes a failed upgrade. Even though
`kubeadm` is shipped in the Kubernetes repositories, it's important
to install `kubeadm` manually. The kubeadm team is working on fixing
this limitation.
{{< /caution >}}
Verify that this download of kubeadm works, and has the expected version:
```shell
kubeadm version
```
2. If this the first time you use `kubeadm upgrade`, in order to preserve the configuration for future upgrades, do:
Note that for below you will need to recall what CLI args you passed to `kubeadm init` the first time.
If you used flags, do:
```shell
kubeadm config upload from-flags [flags]
```
Where `flags` can be empty.
If you used a config file, do:
```shell
kubeadm config upload from-file --config [config]
```
Where the `config` is mandatory.
3. On the master node, run the following:
```shell
kubeadm upgrade plan
```
You should see output similar to this:
```shell
[preflight] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Fetching available versions to upgrade to:
[upgrade/versions] Cluster version: v1.7.1
[upgrade/versions] kubeadm version: v1.8.0
[upgrade/versions] Latest stable version: v1.8.0
[upgrade/versions] Latest version in the v1.7 series: v1.7.6
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.7.1 v1.7.6
Upgrade to the latest version in the v1.7 series:
COMPONENT CURRENT AVAILABLE
API Server v1.7.1 v1.7.6
Controller Manager v1.7.1 v1.7.6
Scheduler v1.7.1 v1.7.6
Kube Proxy v1.7.1 v1.7.6
Kube DNS 1.14.4 1.14.4
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.7.6
_____________________________________________________________________
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.7.1 v1.8.0
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.7.1 v1.8.0
Controller Manager v1.7.1 v1.8.0
Scheduler v1.7.1 v1.8.0
Kube Proxy v1.7.1 v1.8.0
Kube DNS 1.14.4 1.14.4
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.8.0
Note: Before you do can perform this upgrade, you have to update kubeadm to v1.8.0
_____________________________________________________________________
```
The `kubeadm upgrade plan` checks that your cluster is in an upgradeable state and fetches the versions available to upgrade to in an user-friendly way.
4. Pick a version to upgrade to and run, for example, `kubeadm upgrade apply` as follows:
```shell
kubeadm upgrade apply v1.8.0
```
You should see output similar to this:
```shell
[preflight] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade/version] You have chosen to upgrade to version "v1.8.0"
[upgrade/versions] Cluster version: v1.7.1
[upgrade/versions] kubeadm version: v1.8.0
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/prepull] Prepulling image for component kube-scheduler.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.8.0"...
[upgrade/staticpods] Writing upgraded Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-scheduler.yaml"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.8.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.
```
`kubeadm upgrade apply` does the following:
- It checks that your cluster is in an upgradeable state, that is:
- The API Server is reachable,
- All nodes are in the `Ready` state, and
- The control plane is healthy
- It enforces the version skew policies.
- It makes sure the control plane images are available or available to pull to the machine.
- It upgrades the control plane components or rollbacks if any of them fails to come up.
- It applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
5. Manually upgrade your Software Defined Network (SDN).
Your Container Network Interface (CNI) provider might have its own upgrade instructions to follow now.
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
find your CNI provider and see if there are additional upgrade steps
necessary.
6. Add RBAC permissions for automated certificate rotation. In the future, kubeadm will perform this step automatically:
```shell
kubectl create clusterrolebinding kubeadm:node-autoapprove-certificate-rotation --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
```
## Upgrading your master and node packages
For each host (referred to as `$HOST` below) in your cluster, upgrade `kubelet` by executing the following commands:
1. Prepare the host for maintenance, marking it unschedulable and evicting the workload:
```shell
kubectl drain $HOST --ignore-daemonsets
```
When running this command against the master host, this error is expected and can be safely ignored (since there are static pods running on the master):
```shell
node "master" already cordoned
error: pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): etcd-kubeadm, kube-apiserver-kubeadm, kube-controller-manager-kubeadm, kube-scheduler-kubeadm
```
2. Upgrade the Kubernetes package versions on the `$HOST` node by using a Linux distribution-specific package manager:
If the host is running a Debian-based distro such as Ubuntu, run:
```shell
apt-get update
apt-get upgrade
```
If the host is running CentOS or the like, run:
```shell
yum update
```
Now the new version of the `kubelet` should be running on the host. Verify this using the following command on `$HOST`:
```shell
systemctl status kubelet
```
3. Bring the host back online by marking it schedulable:
```shell
kubectl uncordon $HOST
```
4. After upgrading `kubelet` on each host in your cluster, verify that all nodes are available again by executing the following (from anywhere, for example, from outside the cluster):
```shell
kubectl get nodes
```
If the `STATUS` column of the above command shows `Ready` for all of your hosts, you are done.
## Recovering from a bad state
If `kubeadm upgrade` somehow fails and fails to roll back, due to an unexpected shutdown during execution for instance,
you may run `kubeadm upgrade` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring.
You can use `kubeadm upgrade` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/upgrade-downgrade/kubeadm-upgrade-1-9.md
+264
View File
@@ -0,0 +1,264 @@
---
reviewers:
- pipejakob
- luxas
- roberthbailey
- jbeda
title: Upgrading/downgrading kubeadm clusters between v1.8 to v1.9
content_template: templates/task
---
{{% capture overview %}}
This guide is for upgrading `kubeadm` clusters from version 1.8.x to 1.9.x, as well as 1.8.x to 1.8.y and 1.9.x to 1.9.y where `y > x`.
See also [upgrading kubeadm clusters from 1.7 to 1.8](/docs/tasks/administer-cluster/kubeadm-upgrade-1-8/) if you're on a 1.7 cluster currently.
{{% /capture %}}
{{% capture prerequisites %}}
Before proceeding:
- You need to have a functional `kubeadm` Kubernetes cluster running version 1.8.0 or higher in order to use the process described here. Swap also needs to be disabled.
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md) carefully.
- `kubeadm upgrade` now allows you to upgrade etcd. `kubeadm upgrade` will also upgrade of etcd to 3.1.10 as part of upgrading from v1.8 to v1.9 by default. This is due to the fact that etcd 3.1.10 is the officially validated etcd version for Kubernetes v1.9. The upgrade is handled automatically by kubeadm for you.
- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up what's important to you. For example, any app-level state, such as a database an app might depend on (like MySQL or MongoDB) must be backed up beforehand.
{{< caution >}}
**Caution:** All the containers will get restarted after the upgrade, due to container spec hash value gets changed.
{{< /caution >}}
Also, note that only one minor version upgrade is supported. For example, you can only upgrade from 1.8 to 1.9, not from 1.7 to 1.9.
{{% /capture %}}
{{% capture steps %}}
## Upgrading your control plane
Execute these commands on your master node:
1. Install the most recent version of `kubeadm` using `curl` like so:
```shell
export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
chmod a+rx /usr/bin/kubeadm
```
{{< caution >}}
**Caution:** Upgrading the `kubeadm` package on your system prior to upgrading the control plane causes a failed upgrade.
Even though `kubeadm` ships in the Kubernetes repositories, it's important to install `kubeadm` manually. The kubeadm
team is working on fixing this limitation.
{{< /caution >}}
Verify that this download of kubeadm works and has the expected version:
```shell
kubeadm version
```
2. On the master node, run the following:
```shell
kubeadm upgrade plan
```
You should see output similar to this:
```shell
[preflight] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Fetching available versions to upgrade to:
[upgrade/versions] Cluster version: v1.8.1
[upgrade/versions] kubeadm version: v1.9.0
[upgrade/versions] Latest stable version: v1.9.0
[upgrade/versions] Latest version in the v1.8 series: v1.8.6
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.8.1 v1.8.6
Upgrade to the latest version in the v1.8 series:
COMPONENT CURRENT AVAILABLE
API Server v1.8.1 v1.8.6
Controller Manager v1.8.1 v1.8.6
Scheduler v1.8.1 v1.8.6
Kube Proxy v1.8.1 v1.8.6
Kube DNS 1.14.4 1.14.5
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.8.6
_____________________________________________________________________
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.8.1 v1.9.0
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.8.1 v1.9.0
Controller Manager v1.8.1 v1.9.0
Scheduler v1.8.1 v1.9.0
Kube Proxy v1.8.1 v1.9.0
Kube DNS 1.14.5 1.14.7
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.9.0
Note: Before you do can perform this upgrade, you have to update kubeadm to v1.9.0
_____________________________________________________________________
```
The `kubeadm upgrade plan` checks that your cluster is upgradeable and fetches the versions available to upgrade to in an user-friendly way.
To check CoreDNS version, include the `--feature-gates=CoreDNS=true` flag to verify the CoreDNS version which will be installed in place of kube-dns.
3. Pick a version to upgrade to and run. For example:
```shell
kubeadm upgrade apply v1.9.0
```
You should see output similar to this:
```shell
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/version] You have chosen to upgrade to version "v1.9.0"
[upgrade/versions] Cluster version: v1.8.1
[upgrade/versions] kubeadm version: v1.9.0
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.9.0"...
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/etcd.yaml"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/staticpods] Writing upgraded Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests802453804/kube-scheduler.yaml"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests502223003/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.9.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.
```
To upgrade the cluster with CoreDNS as the default internal DNS, invoke `kubeadm upgrade apply` with the `--feature-gates=CoreDNS=true` flag.
`kubeadm upgrade apply` does the following:
- Checks that your cluster is in an upgradeable state:
- The API server is reachable,
- All nodes are in the `Ready` state
- The control plane is healthy
- Enforces the version skew policies.
- Makes sure the control plane images are available or available to pull to the machine.
- Upgrades the control plane components or rollbacks if any of them fails to come up.
- Applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
- Creates new certificate and key files of apiserver and backs up old files if they're about to expire in 180 days.
4. Manually upgrade your Software Defined Network (SDN).
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow.
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
find your CNI provider and see if there are additional upgrade steps
necessary.
## Upgrading your master and node packages
For each host (referred to as `$HOST` below) in your cluster, upgrade `kubelet` by executing the following commands:
1. Prepare the host for maintenance, marking it unschedulable and evicting the workload:
```shell
kubectl drain $HOST --ignore-daemonsets
```
When running this command against the master host, this error is expected and can be safely ignored (since there are static pods running on the master):
```shell
node "master" already cordoned
error: pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): etcd-kubeadm, kube-apiserver-kubeadm, kube-controller-manager-kubeadm, kube-scheduler-kubeadm
```
2. Upgrade the Kubernetes package versions on the `$HOST` node by using a Linux distribution-specific package manager:
If the host is running a Debian-based distro such as Ubuntu, run:
```shell
apt-get update
apt-get upgrade
```
If the host is running CentOS or the like, run:
```shell
yum update
```
Now the new version of the `kubelet` should be running on the host. Verify this using the following command on `$HOST`:
```shell
systemctl status kubelet
```
3. Bring the host back online by marking it schedulable:
```shell
kubectl uncordon $HOST
```
4. After upgrading `kubelet` on each host in your cluster, verify that all nodes are available again by executing the following (from anywhere, for example, from outside the cluster):
```shell
kubectl get nodes
```
If the `STATUS` column of the above command shows `Ready` for all of your hosts, you are done.
## Recovering from a failure state
If `kubeadm upgrade` somehow fails and fails to roll back, for example due to an unexpected shutdown during execution,
you can run `kubeadm upgrade` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring.
You can use `kubeadm upgrade` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/upgrade-downgrade/kubeadm-upgrade-ha.md
+136
View File
@@ -0,0 +1,136 @@
---
reviewers:
- jamiehannaford
- luxas
- timothysc
- jbeda
title: Upgrading kubeadm HA clusters from 1.9.x to 1.9.y
content_template: templates/task
---
{{% capture overview %}}
This guide is for upgrading `kubeadm` HA clusters from version 1.9.x to 1.9.y where `y > x`. The term "`kubeadm` HA clusters" refers to clusters of more than one master node created with `kubeadm`. To set up an HA cluster for Kubernetes version 1.9.x `kubeadm` requires additional manual steps. See [Creating HA clusters with kubeadm](/docs/setup/independent/high-availability/) for instructions on how to do this. The upgrade procedure described here targets clusters created following those very instructions. See [Upgrading/downgrading kubeadm clusters between v1.8 to v1.9](/docs/tasks/administer-cluster/kubeadm-upgrade-1-9/) for more instructions on how to create an HA cluster with `kubeadm`.
{{% /capture %}}
{{% capture prerequisites %}}
Before proceeding:
- You need to have a functional `kubeadm` HA cluster running version 1.9.0 or higher in order to use the process described here.
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md) carefully.
- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up anything important to you. For example, any application-level state, such as a database and application might depend on (like MySQL or MongoDB) should be backed up beforehand.
- Read [Upgrading/downgrading kubeadm clusters between v1.8 to v1.9](/docs/tasks/administer-cluster/kubeadm-upgrade-1-9/) to learn about the relevant prerequisites.
{{% /capture %}}
{{% capture steps %}}
## Preparation
Some preparation is needed prior to starting the upgrade. First download the version of `kubeadm` that matches the version of Kubernetes that you are upgrading to:
```shell
# Use the latest stable release or manually specify a
# released Kubernetes version
export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt)
export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /tmp/kubeadm
chmod a+rx /tmp/kubeadm
```
Copy this file to `/tmp` on your primary master if necessary. Run this command for checking prerequisites and determining the versions you will receive:
```shell
/tmp/kubeadm upgrade plan
```
If the prerequisites are met you'll get a summary of the software versions kubeadm will upgrade to, like this:
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.9.0 v1.9.2
Controller Manager v1.9.0 v1.9.2
Scheduler v1.9.0 v1.9.2
Kube Proxy v1.9.0 v1.9.2
Kube DNS 1.14.5 1.14.7
Etcd 3.2.7 3.1.11
{{< caution >}}
**Caution:** Currently the only supported configuration for kubeadm HA clusters requires the use of an externally managed etcd cluster. Upgrading etcd is not supported as a part of the upgrade. If necessary you will have to upgrade the etcd cluster according to [etcd's upgrade instructions](/docs/tasks/administer-cluster/configure-upgrade-etcd/), which is beyond the scope of these instructions.
{{< /caution >}}
## Upgrading your control plane
The following procedure must be applied on a single master node and repeated for each subsequent master node sequentially.
Before initiating the upgrade with `kubeadm` `configmap/kubeadm-config` needs to be modified for the current master host. Replace any hard reference to a master host name with the current master hosts' name:
```shell
kubectl get configmap -n kube-system kubeadm-config -o yaml >/tmp/kubeadm-config-cm.yaml
sed -i 's/^\([ \t]*nodeName:\).*/\1 <CURRENT-MASTER-NAME>/' /tmp/kubeadm-config-cm.yaml
kubectl apply -f /tmp/kubeadm-config-cm.yaml --force
```
Now the upgrade process can start. Use the target version determined in the preparation step and run the following command (press “y” when prompted):
```shell
/tmp/kubeadm upgrade apply v<YOUR-CHOSEN-VERSION-HERE>
```
If the operation was successful you’ll get a message like this:
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.9.2". Enjoy!
To upgrade the cluster with CoreDNS as the default internal DNS, invoke `kubeadm upgrade apply` with the `--feature-gates=CoreDNS=true` flag.
Next, manually upgrade your CNI provider
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. Check the [addons](/docs/concepts/cluster-administration/addons/) page to find your CNI provider and see if there are additional upgrade steps necessary.
{{< note >}}
**Note:** The `kubeadm upgrade apply` step has been known to fail when run initially on the secondary masters (timed out waiting for the restarted static pods to come up). It should succeed if retried after a minute or two.
{{< /note >}}
## Upgrade base software packages
At this point all the static pod manifests in your cluster, for example API Server, Controller Manager, Scheduler, Kube Proxy have been upgraded, however the base software, for example `kubelet`, `kubectl`, `kubeadm` installed on your nodes’ OS are still of the old version. For upgrading the base software packages we will upgrade them and restart services on all nodes one by one:
```shell
# use your distro's package manager, e.g. 'yum' on RH-based systems
# for the versions stick to kubeadm's output (see above)
yum install -y kubelet-<NEW-K8S-VERSION> kubectl-<NEW-K8S-VERSION> kubeadm-<NEW-K8S-VERSION> kubernetes-cni-<NEW-CNI-VERSION>
systemctl restart kubelet
```
In this example an _rpm_-based system is assumed and `yum` is used for installing the upgraded software. On _deb_-based systems it will be `apt-get update` and then `apt-get install <PACKAGE>=<NEW-K8S-VERSION>` for all packages.
Now the new version of the `kubelet` should be running on the host. Verify this using the following command on the respective host:
```shell
systemctl status kubelet
```
Verify that the upgraded node is available again by executing the following from wherever you run `kubectl` commands:
```shell
kubectl get nodes
```
If the `STATUS` column of the above command shows `Ready` for the upgraded host, you can continue (you may have to repeat this for a couple of time before the node gets `Ready`).
## If something goes wrong
If the upgrade fails the situation afterwards depends on the phase in which things went wrong:
1. If `/tmp/kubeadm upgrade apply` failed to upgrade the cluster it will try to perform a rollback. Hence if that happened on the first master, chances are pretty good that the cluster is still intact.
You can run `/tmp/kubeadm upgrade apply` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring. You can use `/tmp/kubeadm upgrade apply` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
2. If `/tmp/kubeadm upgrade apply` on one of the secondary masters failed you still have a working, upgraded cluster, but with the secondary masters in a somewhat undefined condition. You will have to find out what went wrong and join the secondaries manually. As mentioned above, sometimes upgrading one of the secondary masters fails waiting for the restarted static pods first, but succeeds when the operation is simply repeated after a little pause of one or two minutes.
{{% /capture %}}
content/en/docs/tasks/administer-cluster/upgrade-downgrade/upgrade-1-6.md
+25
View File
@@ -0,0 +1,25 @@
---
reviewers:
- mml
title: Cluster Management Guide for Version 1.6
---
{{< toc >}}
This document outlines the potentially disruptive changes that exist in the 1.6 release cycle. Operators, administrators, and developers should
take note of the changes below in order to maintain continuity across their upgrade process.
## Cluster defaults set to etcd 3
In the 1.6 release cycle, the default backend storage layer has been upgraded to fully leverage [etcd 3 capabilities](https://coreos.com/blog/etcd3-a-new-etcd.html) by default.
For new clusters, there is nothing an operator will need to do, it should "just work". However, if you are upgrading from a 1.5 cluster, care should be taken to ensure
continuity.
It is possible to maintain v2 compatibility mode while running etcd 3 for an interim period of time. To do this, you will simply need to update an argument passed to your apiserver during
startup:
```
$ kube-apiserver --storage-backend='etcd2' $(EXISTING_ARGS)
```
However, for long-term maintenance of the cluster, we recommend that the operator plan an outage window in order to perform a [v2->v3 data upgrade](https://coreos.com/etcd/docs/latest/upgrades/upgrade_3_0.html).
content/en/docs/tasks/administer-cluster/weave-network-policy.md
+58
View File
@@ -0,0 +1,58 @@
---
reviewers:
- bboreham
title: Weave Net for NetworkPolicy
content_template: templates/task
---
{{% capture overview %}}
This page shows how to use Weave Net for NetworkPolicy.
{{% /capture %}}
{{% capture prerequisites %}}
You need to have a Kubernetes cluster. Follow the [kubeadm getting started guide](/docs/getting-started-guides/kubeadm/) to bootstrap one.
{{% /capture %}}
{{% capture steps %}}
## Install the Weave Net addon
Follow the [Integrating Kubernetes via the Addon](https://www.weave.works/docs/net/latest/kube-addon/) guide.
The Weave Net addon for Kubernetes comes with a [Network Policy Controller](https://www.weave.works/docs/net/latest/kube-addon/#npc) that automatically monitors Kubernetes for any NetworkPolicy annotations on all namespaces and configures `iptables` rules to allow or block traffic as directed by the policies.
## Test the installation
Verify that the weave works.
Enter the following command:
```shell
kubectl get po -n kube-system -o wide
```
The output is similar to this:
```
NAME READY STATUS RESTARTS AGE IP NODE
weave-net-1t1qg 2/2 Running 0 9d 192.168.2.10 worknode3
weave-net-231d7 2/2 Running 1 7d 10.2.0.17 worknodegpu
weave-net-7nmwt 2/2 Running 3 9d 192.168.2.131 masternode
weave-net-pmw8w 2/2 Running 0 9d 192.168.2.216 worknode2
```
Each Node has a weave Pod, and all Pods are `Running` and `2/2 READY`. (`2/2` means that each Pod has `weave` and `weave-npc`.)
{{% /capture %}}
{{% capture whatsnext %}}
Once you have installed the Weave Net addon, you can follow the [Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/) to try out Kubernetes NetworkPolicy. If you have any question, contact us at [#weave-community on Slack or Weave User Group](https://github.com/weaveworks/weave#getting-help).
{{% /capture %}}