Surfaces previously hidden topics that had URL conflicts, moves pages to index.md as often as possible
This commit is contained in:
johndmulhausen
@@ -171,7 +171,7 @@ Here you can see from the `Allocated resources` section that that a pod which as
|
||||
|
||||
Looking at the `Pods` section, you can see which pods are taking up space on the node.
|
||||
|
||||
The [resource quota](/docs/admin/resource-quota) feature can be configured
|
||||
The [resource quota](/docs/admin/resourcequota/) feature can be configured
|
||||
to limit the total amount of resources that can be consumed. If used in conjunction
|
||||
with namespaces, it can prevent one team from hogging all the resources.
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ case you can try several things:
|
||||
kubectl get nodes -o yaml | grep '\sname\|cpu\|memory'
|
||||
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, cap: .status.capacity}'
|
||||
|
||||
The [resource quota](/docs/admin/resource-quota/)
|
||||
The [resource quota](/docs/admin/resourcequota/)
|
||||
feature can be configured to limit the total amount of
|
||||
resources that can be consumed. If used in conjunction with namespaces, it can
|
||||
prevent one team from hogging all the resources.
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
---
|
||||
---
|
||||
|
||||
This document describes the current state of Horizontal Pod Autoscaler in Kubernetes.
|
||||
|
||||
## What is Horizontal Pod Autoscaler?
|
||||
|
||||
Horizontal pod autoscaling allows to automatically scale the number of pods
|
||||
in a replication controller, deployment or replica set based on observed CPU utilization.
|
||||
|
||||
The autoscaler is implemented as a Kubernetes API resource and a controller.
|
||||
The resource describes behavior of the controller.
|
||||
The controller periodically adjusts the number of replicas in a replication controller or deployment
|
||||
to match the observed average CPU utilization to the target specified by user.
|
||||
|
||||
|
||||
## How does Horizontal Pod Autoscaler work?
|
||||
|
||||
|
||||
|
||||
The autoscaler is implemented as a control loop.
|
||||
It periodically queries CPU utilization for the pods it targets.
|
||||
(The period of the autoscaler is controlled by `--horizontal-pod-autoscaler-sync-period` flag of controller manager.
|
||||
The default value is 30 seconds).
|
||||
Then, it compares the arithmetic mean of the pods' CPU utilization with the target and adjust the number of replicas if needed.
|
||||
|
||||
CPU utilization is the recent CPU usage of a pod divided by the sum of CPU requested by the pod's containers.
|
||||
Please note that if some of the pod's containers do not have CPU request set,
|
||||
CPU utilization for the pod will not be defined and the autoscaler will not take any action.
|
||||
Further details of the autoscaling algorithm are given [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#autoscaling-algorithm).
|
||||
|
||||
Autoscaler uses heapster to collect CPU utilization.
|
||||
Therefore, it is required to deploy heapster monitoring in your cluster for autoscaling to work.
|
||||
|
||||
Autoscaler accesses corresponding replication controller, deployment or replica set by scale sub-resource.
|
||||
Scale is an interface which allows to dynamically set the number of replicas and to learn the current state of them.
|
||||
More details on scale sub-resource can be found [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#scale-subresource).
|
||||
|
||||
|
||||
## API Object
|
||||
|
||||
Horizontal pod autoscaler is a top-level resource in the Kubernetes REST API.
|
||||
In Kubernetes 1.2 HPA was graduated from beta to stable (more details about [api versioning](/docs/api/#api-versioning)) with compatibility between versions.
|
||||
The stable version is available in `autoscaling/v1` api group whereas the beta vesion is available in `extensions/v1beta1` api group as before.
|
||||
The transition plan is to depracate beta version of HPA in Kubernetes 1.3 and get it rid off completely in Kubernetes 1.4.
|
||||
|
||||
**Warning!** Please have in mind that all Kubernetes components still use HPA in version `extensions/v1beta1` in Kubernetes 1.2.
|
||||
|
||||
More details about the API object can be found at
|
||||
[HorizontalPodAutoscaler Object](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object).
|
||||
|
||||
## Support for horizontal pod autoscaler in kubectl
|
||||
|
||||
Horizontal pod autoscaler, like every API resource, is supported in a standard way by `kubectl`.
|
||||
We can create a new autoscaler using `kubectl create` command.
|
||||
We can list autoscalers by `kubectl get hpa` and get detailed description by `kubectl describe hpa`.
|
||||
Finally, we can delete an autoscaler using `kubectl delete hpa`.
|
||||
|
||||
In addition, there is a special `kubectl autoscale` command that allows for easy creation of horizontal pod autoscaler.
|
||||
For instance, executing `kubectl autoscale rc foo --min=2 --max=5 --cpu-percent=80`
|
||||
will create an autoscaler for replication controller *foo*, with target CPU utilization set to `80%`
|
||||
and the number of replicas between 2 and 5.
|
||||
The detailed documentation of `kubectl autoscale` can be found [here](/docs/user-guide/kubectl/kubectl_autoscale).
|
||||
|
||||
|
||||
## Autoscaling during rolling update
|
||||
|
||||
Currently in Kubernetes, it is possible to perform a rolling update by managing replication controllers directly,
|
||||
or by using the deployment object, which manages the underlying replication controllers for you.
|
||||
Horizontal pod autoscaler only supports the latter approach: the horizontal pod autoscaler is bound to the deployment object,
|
||||
it sets the size for the deployment object, and the deployment is responsible for setting sizes of underlying replication controllers.
|
||||
|
||||
Horizontal pod autoscaler does not work with rolling update using direct manipulation of replication controllers,
|
||||
i.e. you cannot bind a horizontal pod autoscaler to a replication controller and do rolling update (e.g. using `kubectl rolling-update`).
|
||||
The reason this doesn't work is that when rolling update creates a new replication controller,
|
||||
the horizontal pod autoscaler will not be bound to the new replication controller.
|
||||
|
||||
|
||||
## Further reading
|
||||
|
||||
* Design documentation: [Horizontal Pod Autoscaling](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md).
|
||||
* Manual of autoscale command in kubectl: [kubectl autoscale](/docs/user-guide/kubectl/kubectl_autoscale).
|
||||
* Usage example of [Horizontal Pod Autoscaler](/docs/user-guide/horizontal-pod-autoscaling/).
|
||||
@@ -1,147 +1,83 @@
|
||||
---
|
||||
---
|
||||
|
||||
This document describes the current state of Horizontal Pod Autoscaler in Kubernetes.
|
||||
|
||||
## What is Horizontal Pod Autoscaler?
|
||||
|
||||
Horizontal pod autoscaling allows to automatically scale the number of pods
|
||||
in a replication controller, deployment or replica set based on observed CPU utilization.
|
||||
In the future also other metrics will be supported.
|
||||
|
||||
In this document we explain how this feature works by walking you through an example of enabling horizontal pod autoscaling for the php-apache server.
|
||||
The autoscaler is implemented as a Kubernetes API resource and a controller.
|
||||
The resource describes behavior of the controller.
|
||||
The controller periodically adjusts the number of replicas in a replication controller or deployment
|
||||
to match the observed average CPU utilization to the target specified by user.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This example requires a running Kubernetes cluster and kubectl in the version at least 1.2.
|
||||
[Heapster](https://github.com/kubernetes/heapster) monitoring needs to be deployed in the cluster
|
||||
as horizontal pod autoscaler uses it to collect metrics
|
||||
(if you followed [getting started on GCE guide](/docs/getting-started-guides/gce),
|
||||
heapster monitoring will be turned-on by default).
|
||||
## How does Horizontal Pod Autoscaler work?
|
||||
|
||||
## Step One: Run & expose php-apache server
|
||||
|
||||
|
||||
To demonstrate horizontal pod autoscaler we will use a custom docker image based on php-apache server.
|
||||
The image can be found [here](/docs/user-guide/horizontal-pod-autoscaling/image).
|
||||
It defines [index.php](/docs/user-guide/horizontal-pod-autoscaling/image/index.php) page which performs some CPU intensive computations.
|
||||
The autoscaler is implemented as a control loop.
|
||||
It periodically queries CPU utilization for the pods it targets.
|
||||
(The period of the autoscaler is controlled by `--horizontal-pod-autoscaler-sync-period` flag of controller manager.
|
||||
The default value is 30 seconds).
|
||||
Then, it compares the arithmetic mean of the pods' CPU utilization with the target and adjust the number of replicas if needed.
|
||||
|
||||
First, we will start a deployment running the image and expose it as a service:
|
||||
CPU utilization is the recent CPU usage of a pod divided by the sum of CPU requested by the pod's containers.
|
||||
Please note that if some of the pod's containers do not have CPU request set,
|
||||
CPU utilization for the pod will not be defined and the autoscaler will not take any action.
|
||||
Further details of the autoscaling algorithm are given [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#autoscaling-algorithm).
|
||||
|
||||
```shell
|
||||
$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
|
||||
service "php-apache" created
|
||||
deployment "php-apache" created
|
||||
```
|
||||
Autoscaler uses heapster to collect CPU utilization.
|
||||
Therefore, it is required to deploy heapster monitoring in your cluster for autoscaling to work.
|
||||
|
||||
## Step Two: Create horizontal pod autoscaler
|
||||
Autoscaler accesses corresponding replication controller, deployment or replica set by scale sub-resource.
|
||||
Scale is an interface which allows to dynamically set the number of replicas and to learn the current state of them.
|
||||
More details on scale sub-resource can be found [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#scale-subresource).
|
||||
|
||||
Now that the server is running, we will create the autoscaler using
|
||||
[kubectl autoscale](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/user-guide/kubectl/kubectl_autoscale.md).
|
||||
The following command will create a horizontal pod autoscaler that maintains between 1 and 10 replicas of the Pods
|
||||
controlled by the php-apache deployment we created in the first step of these instructions.
|
||||
Roughly speaking, the horizontal autoscaler will increase and decrease the number of replicas
|
||||
(via the deployment) to maintain an average CPU utilization across all Pods of 50%
|
||||
(since each pod requests 200 milli-cores by [kubectl run](#kubectl-run), this means average CPU usage of 100 milli-cores).
|
||||
See [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#autoscaling-algorithm) for more details on the algorithm.
|
||||
|
||||
```shell
|
||||
$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
|
||||
deployment "php-apache" autoscaled
|
||||
```
|
||||
## API Object
|
||||
|
||||
We may check the current status of autoscaler by running:
|
||||
Horizontal pod autoscaler is a top-level resource in the Kubernetes REST API.
|
||||
In Kubernetes 1.2 HPA was graduated from beta to stable (more details about [api versioning](/docs/api/#api-versioning)) with compatibility between versions.
|
||||
The stable version is available in `autoscaling/v1` api group whereas the beta vesion is available in `extensions/v1beta1` api group as before.
|
||||
The transition plan is to depracate beta version of HPA in Kubernetes 1.3 and get it rid off completely in Kubernetes 1.4.
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa
|
||||
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
|
||||
php-apache Deployment/php-apache/scale 50% 0% 1 10 18s
|
||||
**Warning!** Please have in mind that all Kubernetes components still use HPA in version `extensions/v1beta1` in Kubernetes 1.2.
|
||||
|
||||
```
|
||||
More details about the API object can be found at
|
||||
[HorizontalPodAutoscaler Object](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#horizontalpodautoscaler-object).
|
||||
|
||||
Please note that the current CPU consumption is 0% as we are not sending any requests to the server
|
||||
(the ``CURRENT`` column shows the average across all the pods controlled by the corresponding deployment).
|
||||
## Support for horizontal pod autoscaler in kubectl
|
||||
|
||||
## Step Three: Increase load
|
||||
Horizontal pod autoscaler, like every API resource, is supported in a standard way by `kubectl`.
|
||||
We can create a new autoscaler using `kubectl create` command.
|
||||
We can list autoscalers by `kubectl get hpa` and get detailed description by `kubectl describe hpa`.
|
||||
Finally, we can delete an autoscaler using `kubectl delete hpa`.
|
||||
|
||||
Now, we will see how the autoscaler reacts on the increased load on the server.
|
||||
We will start a container with `busybox` image and an infinite loop of queries to our server inside (please run it in a different terminal):
|
||||
In addition, there is a special `kubectl autoscale` command that allows for easy creation of horizontal pod autoscaler.
|
||||
For instance, executing `kubectl autoscale rc foo --min=2 --max=5 --cpu-percent=80`
|
||||
will create an autoscaler for replication controller *foo*, with target CPU utilization set to `80%`
|
||||
and the number of replicas between 2 and 5.
|
||||
The detailed documentation of `kubectl autoscale` can be found [here](/docs/user-guide/kubectl/kubectl_autoscale).
|
||||
|
||||
```shell
|
||||
$ kubectl run -i --tty load-generator --image=busybox /bin/sh
|
||||
|
||||
Hit enter for command prompt
|
||||
## Autoscaling during rolling update
|
||||
|
||||
$ while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done
|
||||
```
|
||||
Currently in Kubernetes, it is possible to perform a rolling update by managing replication controllers directly,
|
||||
or by using the deployment object, which manages the underlying replication controllers for you.
|
||||
Horizontal pod autoscaler only supports the latter approach: the horizontal pod autoscaler is bound to the deployment object,
|
||||
it sets the size for the deployment object, and the deployment is responsible for setting sizes of underlying replication controllers.
|
||||
|
||||
We may examine, how CPU load was increased by executing (it usually takes 1 minute):
|
||||
Horizontal pod autoscaler does not work with rolling update using direct manipulation of replication controllers,
|
||||
i.e. you cannot bind a horizontal pod autoscaler to a replication controller and do rolling update (e.g. using `kubectl rolling-update`).
|
||||
The reason this doesn't work is that when rolling update creates a new replication controller,
|
||||
the horizontal pod autoscaler will not be bound to the new replication controller.
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa
|
||||
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
|
||||
php-apache Deployment/php-apache/scale 50% 305% 1 10 3m
|
||||
|
||||
```
|
||||
## Further reading
|
||||
|
||||
In the case presented here, it bumped CPU consumption to 305% of the request.
|
||||
As a result, the deployment was resized to 7 replicas:
|
||||
|
||||
```shell
|
||||
$ kubectl get deployment php-apache
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
php-apache 7 7 7 7 19m
|
||||
```
|
||||
|
||||
**Warning!** Sometimes it may take few steps to stabilize the number of replicas.
|
||||
Since the amount of load is not controlled in any way it may happen that the final number of replicas will
|
||||
differ from this example.
|
||||
|
||||
## Step Four: Stop load
|
||||
|
||||
We will finish our example by stopping the user load.
|
||||
|
||||
In the terminal where we created container with `busybox` image we will terminate
|
||||
infinite ``while`` loop by sending `SIGINT` signal,
|
||||
which can be done using `<Ctrl> + C` combination.
|
||||
|
||||
Then we will verify the result state:
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa
|
||||
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
|
||||
php-apache Deployment/php-apache/scale 50% 0% 1 10 11m
|
||||
|
||||
$ kubectl get deployment php-apache
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
php-apache 1 1 1 1 27m
|
||||
```
|
||||
|
||||
As we see, in the presented case CPU utilization dropped to 0, and the number of replicas dropped to 1.
|
||||
|
||||
**Warning!** Sometimes dropping number of replicas may take few steps.
|
||||
|
||||
## Appendix: Other possible scenarios
|
||||
|
||||
### Creating the autoscaler from a .yaml file
|
||||
|
||||
Instead of using `kubectl autoscale` command we can use the [hpa-php-apache.yaml](/docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml) file, which looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: php-apache
|
||||
namespace: default
|
||||
spec:
|
||||
scaleRef:
|
||||
kind: Deployment
|
||||
name: php-apache
|
||||
subresource: scale
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
cpuUtilization:
|
||||
targetPercentage: 50
|
||||
```
|
||||
|
||||
We will create the autoscaler by executing the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml
|
||||
horizontalpodautoscaler "php-apache" created
|
||||
```
|
||||
* Design documentation: [Horizontal Pod Autoscaling](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md).
|
||||
* Manual of autoscale command in kubectl: [kubectl autoscale](/docs/user-guide/kubectl/kubectl_autoscale).
|
||||
* Usage example of [Horizontal Pod Autoscaler](/docs/user-guide/horizontal-pod-autoscaling/).
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
---
|
||||
|
||||
Horizontal pod autoscaling allows to automatically scale the number of pods
|
||||
in a replication controller, deployment or replica set based on observed CPU utilization.
|
||||
In the future also other metrics will be supported.
|
||||
|
||||
In this document we explain how this feature works by walking you through an example of enabling horizontal pod autoscaling for the php-apache server.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This example requires a running Kubernetes cluster and kubectl in the version at least 1.2.
|
||||
[Heapster](https://github.com/kubernetes/heapster) monitoring needs to be deployed in the cluster
|
||||
as horizontal pod autoscaler uses it to collect metrics
|
||||
(if you followed [getting started on GCE guide](/docs/getting-started-guides/gce),
|
||||
heapster monitoring will be turned-on by default).
|
||||
|
||||
## Step One: Run & expose php-apache server
|
||||
|
||||
To demonstrate horizontal pod autoscaler we will use a custom docker image based on php-apache server.
|
||||
The image can be found [here](/docs/user-guide/horizontal-pod-autoscaling/image).
|
||||
It defines [index.php](/docs/user-guide/horizontal-pod-autoscaling/image/index.php) page which performs some CPU intensive computations.
|
||||
|
||||
First, we will start a deployment running the image and expose it as a service:
|
||||
|
||||
```shell
|
||||
$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
|
||||
service "php-apache" created
|
||||
deployment "php-apache" created
|
||||
```
|
||||
|
||||
## Step Two: Create horizontal pod autoscaler
|
||||
|
||||
Now that the server is running, we will create the autoscaler using
|
||||
[kubectl autoscale](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/user-guide/kubectl/kubectl_autoscale.md).
|
||||
The following command will create a horizontal pod autoscaler that maintains between 1 and 10 replicas of the Pods
|
||||
controlled by the php-apache deployment we created in the first step of these instructions.
|
||||
Roughly speaking, the horizontal autoscaler will increase and decrease the number of replicas
|
||||
(via the deployment) to maintain an average CPU utilization across all Pods of 50%
|
||||
(since each pod requests 200 milli-cores by [kubectl run](#kubectl-run), this means average CPU usage of 100 milli-cores).
|
||||
See [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/horizontal-pod-autoscaler.md#autoscaling-algorithm) for more details on the algorithm.
|
||||
|
||||
```shell
|
||||
$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
|
||||
deployment "php-apache" autoscaled
|
||||
```
|
||||
|
||||
We may check the current status of autoscaler by running:
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa
|
||||
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
|
||||
php-apache Deployment/php-apache/scale 50% 0% 1 10 18s
|
||||
|
||||
```
|
||||
|
||||
Please note that the current CPU consumption is 0% as we are not sending any requests to the server
|
||||
(the ``CURRENT`` column shows the average across all the pods controlled by the corresponding deployment).
|
||||
|
||||
## Step Three: Increase load
|
||||
|
||||
Now, we will see how the autoscaler reacts on the increased load on the server.
|
||||
We will start a container with `busybox` image and an infinite loop of queries to our server inside (please run it in a different terminal):
|
||||
|
||||
```shell
|
||||
$ kubectl run -i --tty load-generator --image=busybox /bin/sh
|
||||
|
||||
Hit enter for command prompt
|
||||
|
||||
$ while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done
|
||||
```
|
||||
|
||||
We may examine, how CPU load was increased by executing (it usually takes 1 minute):
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa
|
||||
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
|
||||
php-apache Deployment/php-apache/scale 50% 305% 1 10 3m
|
||||
|
||||
```
|
||||
|
||||
In the case presented here, it bumped CPU consumption to 305% of the request.
|
||||
As a result, the deployment was resized to 7 replicas:
|
||||
|
||||
```shell
|
||||
$ kubectl get deployment php-apache
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
php-apache 7 7 7 7 19m
|
||||
```
|
||||
|
||||
**Warning!** Sometimes it may take few steps to stabilize the number of replicas.
|
||||
Since the amount of load is not controlled in any way it may happen that the final number of replicas will
|
||||
differ from this example.
|
||||
|
||||
## Step Four: Stop load
|
||||
|
||||
We will finish our example by stopping the user load.
|
||||
|
||||
In the terminal where we created container with `busybox` image we will terminate
|
||||
infinite ``while`` loop by sending `SIGINT` signal,
|
||||
which can be done using `<Ctrl> + C` combination.
|
||||
|
||||
Then we will verify the result state:
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa
|
||||
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
|
||||
php-apache Deployment/php-apache/scale 50% 0% 1 10 11m
|
||||
|
||||
$ kubectl get deployment php-apache
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
php-apache 1 1 1 1 27m
|
||||
```
|
||||
|
||||
As we see, in the presented case CPU utilization dropped to 0, and the number of replicas dropped to 1.
|
||||
|
||||
**Warning!** Sometimes dropping number of replicas may take few steps.
|
||||
|
||||
## Appendix: Other possible scenarios
|
||||
|
||||
### Creating the autoscaler from a .yaml file
|
||||
|
||||
Instead of using `kubectl autoscale` command we can use the [hpa-php-apache.yaml](/docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml) file, which looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: php-apache
|
||||
namespace: default
|
||||
spec:
|
||||
scaleRef:
|
||||
kind: Deployment
|
||||
name: php-apache
|
||||
subresource: scale
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
cpuUtilization:
|
||||
targetPercentage: 50
|
||||
```
|
||||
|
||||
We will create the autoscaler by executing the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/horizontal-pod-autoscaling/hpa-php-apache.yaml
|
||||
horizontalpodautoscaler "php-apache" created
|
||||
```
|
||||
@@ -13,7 +13,7 @@ need the features they provide.
|
||||
|
||||
Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces.
|
||||
|
||||
Namespaces are a way to divide cluster resources between multiple uses (via [resource quota](/docs/admin/resource-quota)).
|
||||
Namespaces are a way to divide cluster resources between multiple uses (via [resource quota](/docs/admin/resourcequota/)).
|
||||
|
||||
In future versions of Kubernetes, objects in the same namespace will have the same
|
||||
access control policies by default.
|
||||
|
||||
@@ -1,171 +0,0 @@
|
||||
---
|
||||
---
|
||||
|
||||
This document describes the current state of `PersistentVolumes` in Kubernetes. Familiarity with [volumes](/docs/user-guide/volumes/) is suggested.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Introduction
|
||||
|
||||
Managing storage is a distinct problem from managing compute. The `PersistentVolume` subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this we introduce two new API resources: `PersistentVolume` and `PersistentVolumeClaim`.
|
||||
|
||||
A `PersistentVolume` (PV) is a piece of networked storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
|
||||
|
||||
A `PersistentVolumeClaim` (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g, can be mounted once read/write or many times read-only).
|
||||
|
||||
Please see the [detailed walkthrough with working examples](/docs/user-guide/persistent-volumes/).
|
||||
|
||||
|
||||
## Lifecycle of a volume and claim
|
||||
|
||||
PVs are resources in the cluster. PVCs are requests for those resources and also act as claim checks to the resource. The interaction between PVs and PVCs follows this lifecycle:
|
||||
|
||||
### Provisioning
|
||||
|
||||
A cluster administrator will create a number of PVs. They carry the details of the real storage which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
|
||||
|
||||
### Binding
|
||||
|
||||
A user creates a `PersistentVolumeClaim` with a specific amount of storage requested and with certain access modes. A control loop in the master watches for new PVCs, finds a matching PV (if possible), and binds them together. The user will always get at least what they asked for, but the volume may be in excess of what was requested.
|
||||
|
||||
Claims will remain unbound indefinitely if a matching volume does not exist. Claims will be bound as matching volumes become available. For example, a cluster provisioned with many 50Gi PVs would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.
|
||||
|
||||
### Using
|
||||
|
||||
Pods use claims as volumes. The cluster inspects the claim to find the bound volume and mounts that volume for a pod. For volumes which support multiple access modes, the user specifies which mode desired when using their claim as a volume in a pod.
|
||||
|
||||
Once a user has a claim and that claim is bound, the bound PV belongs to the user for as long as they need it. Users schedule Pods and access their claimed PVs by including a persistentVolumeClaim in their Pod's volumes block. [See below for syntax details](#claims-as-volumes).
|
||||
|
||||
### Releasing
|
||||
|
||||
When a user is done with their volume, they can delete the PVC objects from the API which allows reclamation of the resource. The volume is considered "released" when the claim is deleted, but it is not yet available for another claim. The previous claimant's data remains on the volume which must be handled according to policy.
|
||||
|
||||
### Reclaiming
|
||||
|
||||
The reclaim policy for a `PersistentVolume` tells the cluster what to do with the volume after it has been released. Currently, volumes can either be Retained or Recycled. Retention allows for manual reclamation of the resource. For those volume plugins that support it, recycling performs a basic scrub (`rm -rf /thevolume/*`) on the volume and makes it available again for a new claim.
|
||||
|
||||
## Types of Persistent Volumes
|
||||
|
||||
`PersistentVolume` types are implemented as plugins. Kubernetes currently supports the following plugins:
|
||||
|
||||
* GCEPersistentDisk
|
||||
* AWSElasticBlockStore
|
||||
* NFS
|
||||
* iSCSI
|
||||
* RBD (Ceph Block Device)
|
||||
* Glusterfs
|
||||
* HostPath (single node testing only -- local storage is not supported in any way and WILL NOT WORK in a multi-node cluster)
|
||||
|
||||
|
||||
## Persistent Volumes
|
||||
|
||||
Each PV contains a spec and status, which is the specification and status of the volume.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: pv0003
|
||||
spec:
|
||||
capacity:
|
||||
storage: 5Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Recycle
|
||||
nfs:
|
||||
path: /tmp
|
||||
server: 172.17.0.2
|
||||
```
|
||||
|
||||
### Capacity
|
||||
|
||||
Generally, a PV will have a specific storage capacity. This is set using the PV's `capacity` attribute. See the Kubernetes [Resource Model](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/resources.md) to understand the units expected by `capacity`.
|
||||
|
||||
Currently, storage size is the only resource that can be set or requested. Future attributes may include IOPS, throughput, etc.
|
||||
|
||||
### Access Modes
|
||||
|
||||
A `PersistentVolume` can be mounted on a host in any way supported by the resource provider. Providers will have different capabilities and each PV's access modes are set to the specific modes supported by that particular volume. For example, NFS can support multiple read/write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV's capabilities.
|
||||
|
||||
The access modes are:
|
||||
|
||||
* ReadWriteOnce -- the volume can be mounted as read-write by a single node
|
||||
* ReadOnlyMany -- the volume can be mounted read-only by many nodes
|
||||
* ReadWriteMany -- the volume can be mounted as read-write by many nodes
|
||||
|
||||
In the CLI, the access modes are abbreviated to:
|
||||
|
||||
* RWO - ReadWriteOnce
|
||||
* ROX - ReadOnlyMany
|
||||
* RWX - ReadWriteMany
|
||||
|
||||
> __Important!__ A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.
|
||||
|
||||
|
||||
### Recycling Policy
|
||||
|
||||
Current recycling policies are:
|
||||
|
||||
* Retain -- manual reclamation
|
||||
* Recycle -- basic scrub ("rm -rf /thevolume/*")
|
||||
|
||||
Currently, NFS and HostPath support recycling.
|
||||
|
||||
### Phase
|
||||
|
||||
A volume will be in one of the following phases:
|
||||
|
||||
* Available -- a free resource that is not yet bound to a claim
|
||||
* Bound -- the volume is bound to a claim
|
||||
* Released -- the claim has been deleted, but the resource is not yet reclaimed by the cluster
|
||||
* Failed -- the volume has failed its automatic reclamation
|
||||
|
||||
The CLI will show the name of the PVC bound to the PV.
|
||||
|
||||
## PersistentVolumeClaims
|
||||
|
||||
Each PVC contains a spec and status, which is the specification and status of the claim.
|
||||
|
||||
```yaml
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: myclaim
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 8Gi
|
||||
```
|
||||
|
||||
### Access Modes
|
||||
|
||||
Claims use the same conventions as volumes when requesting storage with specific access modes.
|
||||
|
||||
### Resources
|
||||
|
||||
Claims, like pods, can request specific quantities of a resource. In this case, the request is for storage. The same [resource model](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/resources.md) applies to both volumes and claims.
|
||||
|
||||
## Claims As Volumes
|
||||
|
||||
Pods access storage by using the claim as a volume. Claims must exist in the same namespace as the pod using the claim. The cluster finds the claim in the pod's namespace and uses it to get the `PersistentVolume` backing the claim. The volume is then mounted to the host and into the pod.
|
||||
|
||||
```yaml
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: mypod
|
||||
spec:
|
||||
containers:
|
||||
- name: myfrontend
|
||||
image: dockerfile/nginx
|
||||
volumeMounts:
|
||||
- mountPath: "/var/www/html"
|
||||
name: mypd
|
||||
volumes:
|
||||
- name: mypd
|
||||
persistentVolumeClaim:
|
||||
claimName: myclaim
|
||||
```
|
||||
@@ -1,96 +1,171 @@
|
||||
---
|
||||
---
|
||||
|
||||
The purpose of this guide is to help you become familiar with [Kubernetes Persistent Volumes](/docs/user-guide/persistent-volumes/). By the end of the guide, we'll have
|
||||
nginx serving content from your persistent volume.
|
||||
This document describes the current state of `PersistentVolumes` in Kubernetes. Familiarity with [volumes](/docs/user-guide/volumes/) is suggested.
|
||||
|
||||
You can view all the files for this example in [the docs repo
|
||||
here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/persistent-volumes).
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
This guide assumes knowledge of Kubernetes fundamentals and that you have a cluster up and running.
|
||||
## Introduction
|
||||
|
||||
See [Persistent Storage design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/persistent-storage.md) for more information.
|
||||
Managing storage is a distinct problem from managing compute. The `PersistentVolume` subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this we introduce two new API resources: `PersistentVolume` and `PersistentVolumeClaim`.
|
||||
|
||||
## Provisioning
|
||||
A `PersistentVolume` (PV) is a piece of networked storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
|
||||
|
||||
A Persistent Volume (PV) in Kubernetes represents a real piece of underlying storage capacity in the infrastructure. Cluster administrators
|
||||
must first create storage (create their Google Compute Engine (GCE) disks, export their NFS shares, etc.) in order for Kubernetes to mount it.
|
||||
A `PersistentVolumeClaim` (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g, can be mounted once read/write or many times read-only).
|
||||
|
||||
PVs are intended for "network volumes" like GCE Persistent Disks, NFS shares, and AWS ElasticBlockStore volumes. `HostPath` was included
|
||||
for ease of development and testing. You'll create a local `HostPath` for this example.
|
||||
Please see the [detailed walkthrough with working examples](/docs/user-guide/persistent-volumes/).
|
||||
|
||||
> IMPORTANT! For `HostPath` to work, you will need to run a single node cluster. Kubernetes does not
|
||||
support local storage on the host at this time. There is no guarantee your pod ends up on the correct node where the `HostPath` resides.
|
||||
|
||||
```shell
|
||||
# This will be nginx's webroot
|
||||
$ mkdir /tmp/data01
|
||||
$ echo 'I love Kubernetes storage!' > /tmp/data01/index.html
|
||||
## Lifecycle of a volume and claim
|
||||
|
||||
PVs are resources in the cluster. PVCs are requests for those resources and also act as claim checks to the resource. The interaction between PVs and PVCs follows this lifecycle:
|
||||
|
||||
### Provisioning
|
||||
|
||||
A cluster administrator will create a number of PVs. They carry the details of the real storage which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
|
||||
|
||||
### Binding
|
||||
|
||||
A user creates a `PersistentVolumeClaim` with a specific amount of storage requested and with certain access modes. A control loop in the master watches for new PVCs, finds a matching PV (if possible), and binds them together. The user will always get at least what they asked for, but the volume may be in excess of what was requested.
|
||||
|
||||
Claims will remain unbound indefinitely if a matching volume does not exist. Claims will be bound as matching volumes become available. For example, a cluster provisioned with many 50Gi PVs would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.
|
||||
|
||||
### Using
|
||||
|
||||
Pods use claims as volumes. The cluster inspects the claim to find the bound volume and mounts that volume for a pod. For volumes which support multiple access modes, the user specifies which mode desired when using their claim as a volume in a pod.
|
||||
|
||||
Once a user has a claim and that claim is bound, the bound PV belongs to the user for as long as they need it. Users schedule Pods and access their claimed PVs by including a persistentVolumeClaim in their Pod's volumes block. [See below for syntax details](#claims-as-volumes).
|
||||
|
||||
### Releasing
|
||||
|
||||
When a user is done with their volume, they can delete the PVC objects from the API which allows reclamation of the resource. The volume is considered "released" when the claim is deleted, but it is not yet available for another claim. The previous claimant's data remains on the volume which must be handled according to policy.
|
||||
|
||||
### Reclaiming
|
||||
|
||||
The reclaim policy for a `PersistentVolume` tells the cluster what to do with the volume after it has been released. Currently, volumes can either be Retained or Recycled. Retention allows for manual reclamation of the resource. For those volume plugins that support it, recycling performs a basic scrub (`rm -rf /thevolume/*`) on the volume and makes it available again for a new claim.
|
||||
|
||||
## Types of Persistent Volumes
|
||||
|
||||
`PersistentVolume` types are implemented as plugins. Kubernetes currently supports the following plugins:
|
||||
|
||||
* GCEPersistentDisk
|
||||
* AWSElasticBlockStore
|
||||
* NFS
|
||||
* iSCSI
|
||||
* RBD (Ceph Block Device)
|
||||
* Glusterfs
|
||||
* HostPath (single node testing only -- local storage is not supported in any way and WILL NOT WORK in a multi-node cluster)
|
||||
|
||||
|
||||
## Persistent Volumes
|
||||
|
||||
Each PV contains a spec and status, which is the specification and status of the volume.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: pv0003
|
||||
spec:
|
||||
capacity:
|
||||
storage: 5Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Recycle
|
||||
nfs:
|
||||
path: /tmp
|
||||
server: 172.17.0.2
|
||||
```
|
||||
|
||||
PVs are created by posting them to the API server.
|
||||
### Capacity
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/volumes/local-01.yaml
|
||||
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON
|
||||
pv0001 type=local 10737418240 RWO Available
|
||||
Generally, a PV will have a specific storage capacity. This is set using the PV's `capacity` attribute. See the Kubernetes [Resource Model](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/resources.md) to understand the units expected by `capacity`.
|
||||
|
||||
Currently, storage size is the only resource that can be set or requested. Future attributes may include IOPS, throughput, etc.
|
||||
|
||||
### Access Modes
|
||||
|
||||
A `PersistentVolume` can be mounted on a host in any way supported by the resource provider. Providers will have different capabilities and each PV's access modes are set to the specific modes supported by that particular volume. For example, NFS can support multiple read/write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV's capabilities.
|
||||
|
||||
The access modes are:
|
||||
|
||||
* ReadWriteOnce -- the volume can be mounted as read-write by a single node
|
||||
* ReadOnlyMany -- the volume can be mounted read-only by many nodes
|
||||
* ReadWriteMany -- the volume can be mounted as read-write by many nodes
|
||||
|
||||
In the CLI, the access modes are abbreviated to:
|
||||
|
||||
* RWO - ReadWriteOnce
|
||||
* ROX - ReadOnlyMany
|
||||
* RWX - ReadWriteMany
|
||||
|
||||
> __Important!__ A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.
|
||||
|
||||
|
||||
### Recycling Policy
|
||||
|
||||
Current recycling policies are:
|
||||
|
||||
* Retain -- manual reclamation
|
||||
* Recycle -- basic scrub ("rm -rf /thevolume/*")
|
||||
|
||||
Currently, NFS and HostPath support recycling.
|
||||
|
||||
### Phase
|
||||
|
||||
A volume will be in one of the following phases:
|
||||
|
||||
* Available -- a free resource that is not yet bound to a claim
|
||||
* Bound -- the volume is bound to a claim
|
||||
* Released -- the claim has been deleted, but the resource is not yet reclaimed by the cluster
|
||||
* Failed -- the volume has failed its automatic reclamation
|
||||
|
||||
The CLI will show the name of the PVC bound to the PV.
|
||||
|
||||
## PersistentVolumeClaims
|
||||
|
||||
Each PVC contains a spec and status, which is the specification and status of the claim.
|
||||
|
||||
```yaml
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: myclaim
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 8Gi
|
||||
```
|
||||
|
||||
## Requesting storage
|
||||
### Access Modes
|
||||
|
||||
Users of Kubernetes request persistent storage for their pods. They don't know how the underlying cluster is provisioned.
|
||||
They just know they can rely on their claim to storage and can manage its lifecycle independently from the many pods that may use it.
|
||||
Claims use the same conventions as volumes when requesting storage with specific access modes.
|
||||
|
||||
Claims must be created in the same namespace as the pods that use them.
|
||||
### Resources
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/claims/claim-01.yaml
|
||||
Claims, like pods, can request specific quantities of a resource. In this case, the request is for storage. The same [resource model](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/resources.md) applies to both volumes and claims.
|
||||
|
||||
$ kubectl get pvc
|
||||
NAME LABELS STATUS VOLUME
|
||||
myclaim-1 map[]
|
||||
|
||||
|
||||
# A background process will attempt to match this claim to a volume.
|
||||
# The eventual state of your claim will look something like this:
|
||||
## Claims As Volumes
|
||||
|
||||
$ kubectl get pvc
|
||||
NAME LABELS STATUS VOLUME
|
||||
myclaim-1 map[] Bound pv0001
|
||||
Pods access storage by using the claim as a volume. Claims must exist in the same namespace as the pod using the claim. The cluster finds the claim in the pod's namespace and uses it to get the `PersistentVolume` backing the claim. The volume is then mounted to the host and into the pod.
|
||||
|
||||
$ kubectl get pv
|
||||
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON
|
||||
pv0001 type=local 10737418240 RWO Bound default/myclaim-1
|
||||
```
|
||||
|
||||
## Using your claim as a volume
|
||||
|
||||
Claims are used as volumes in pods. Kubernetes uses the claim to look up its bound PV. The PV is then exposed to the pod.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/simpletest/pod.yaml
|
||||
|
||||
$ kubectl get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
mypod 1/1 Running 0 1h
|
||||
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/simpletest/service.json
|
||||
$ kubectl get services
|
||||
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
|
||||
frontendservice 10.0.0.241 <none> 3000/TCP name=frontendhttp 1d
|
||||
kubernetes 10.0.0.2 <none> 443/TCP <none> 2d
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
You should be able to query your service endpoint and see what content nginx is serving. A "forbidden" error might mean you
|
||||
need to disable SELinux (setenforce 0).
|
||||
|
||||
```shell
|
||||
$ curl 10.0.0.241:3000
|
||||
I love Kubernetes storage!
|
||||
```
|
||||
|
||||
Hopefully this simple guide is enough to get you started with PersistentVolumes. If you have any questions, join the team on [Slack](/docs/troubleshooting/#slack) and ask!
|
||||
|
||||
Enjoy!
|
||||
```yaml
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: mypod
|
||||
spec:
|
||||
containers:
|
||||
- name: myfrontend
|
||||
image: dockerfile/nginx
|
||||
volumeMounts:
|
||||
- mountPath: "/var/www/html"
|
||||
name: mypd
|
||||
volumes:
|
||||
- name: mypd
|
||||
persistentVolumeClaim:
|
||||
claimName: myclaim
|
||||
```
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
---
|
||||
|
||||
The purpose of this guide is to help you become familiar with [Kubernetes Persistent Volumes](/docs/user-guide/persistent-volumes/). By the end of the guide, we'll have
|
||||
nginx serving content from your persistent volume.
|
||||
|
||||
You can view all the files for this example in [the docs repo
|
||||
here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/persistent-volumes).
|
||||
|
||||
This guide assumes knowledge of Kubernetes fundamentals and that you have a cluster up and running.
|
||||
|
||||
See [Persistent Storage design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/persistent-storage.md) for more information.
|
||||
|
||||
## Provisioning
|
||||
|
||||
A Persistent Volume (PV) in Kubernetes represents a real piece of underlying storage capacity in the infrastructure. Cluster administrators
|
||||
must first create storage (create their Google Compute Engine (GCE) disks, export their NFS shares, etc.) in order for Kubernetes to mount it.
|
||||
|
||||
PVs are intended for "network volumes" like GCE Persistent Disks, NFS shares, and AWS ElasticBlockStore volumes. `HostPath` was included
|
||||
for ease of development and testing. You'll create a local `HostPath` for this example.
|
||||
|
||||
> IMPORTANT! For `HostPath` to work, you will need to run a single node cluster. Kubernetes does not
|
||||
support local storage on the host at this time. There is no guarantee your pod ends up on the correct node where the `HostPath` resides.
|
||||
|
||||
```shell
|
||||
# This will be nginx's webroot
|
||||
$ mkdir /tmp/data01
|
||||
$ echo 'I love Kubernetes storage!' > /tmp/data01/index.html
|
||||
```
|
||||
|
||||
PVs are created by posting them to the API server.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/volumes/local-01.yaml
|
||||
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON
|
||||
pv0001 type=local 10737418240 RWO Available
|
||||
```
|
||||
|
||||
## Requesting storage
|
||||
|
||||
Users of Kubernetes request persistent storage for their pods. They don't know how the underlying cluster is provisioned.
|
||||
They just know they can rely on their claim to storage and can manage its lifecycle independently from the many pods that may use it.
|
||||
|
||||
Claims must be created in the same namespace as the pods that use them.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/claims/claim-01.yaml
|
||||
|
||||
$ kubectl get pvc
|
||||
NAME LABELS STATUS VOLUME
|
||||
myclaim-1 map[]
|
||||
|
||||
|
||||
# A background process will attempt to match this claim to a volume.
|
||||
# The eventual state of your claim will look something like this:
|
||||
|
||||
$ kubectl get pvc
|
||||
NAME LABELS STATUS VOLUME
|
||||
myclaim-1 map[] Bound pv0001
|
||||
|
||||
$ kubectl get pv
|
||||
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON
|
||||
pv0001 type=local 10737418240 RWO Bound default/myclaim-1
|
||||
```
|
||||
|
||||
## Using your claim as a volume
|
||||
|
||||
Claims are used as volumes in pods. Kubernetes uses the claim to look up its bound PV. The PV is then exposed to the pod.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/simpletest/pod.yaml
|
||||
|
||||
$ kubectl get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
mypod 1/1 Running 0 1h
|
||||
|
||||
$ kubectl create -f docs/user-guide/persistent-volumes/simpletest/service.json
|
||||
$ kubectl get services
|
||||
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
|
||||
frontendservice 10.0.0.241 <none> 3000/TCP name=frontendhttp 1d
|
||||
kubernetes 10.0.0.2 <none> 443/TCP <none> 2d
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
You should be able to query your service endpoint and see what content nginx is serving. A "forbidden" error might mean you
|
||||
need to disable SELinux (setenforce 0).
|
||||
|
||||
```shell
|
||||
$ curl 10.0.0.241:3000
|
||||
I love Kubernetes storage!
|
||||
```
|
||||
|
||||
Hopefully this simple guide is enough to get you started with PersistentVolumes. If you have any questions, join the team on [Slack](/docs/troubleshooting/#slack) and ask!
|
||||
|
||||
Enjoy!
|
||||
@@ -1,672 +0,0 @@
|
||||
---
|
||||
---
|
||||
|
||||
Objects of type `secret` are intended to hold sensitive information, such as
|
||||
passwords, OAuth tokens, and ssh keys. Putting this information in a `secret`
|
||||
is safer and more flexible than putting it verbatim in a `pod` definition or in
|
||||
a docker image. See [Secrets design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/secrets.md) for more information.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Overview of Secrets
|
||||
|
||||
A Secret is an object that contains a small amount of sensitive data such as
|
||||
a password, a token, or a key. Such information might otherwise be put in a
|
||||
Pod specification or in an image; putting it in a Secret object allows for
|
||||
more control over how it is used, and reduces the risk of accidental exposure.
|
||||
|
||||
Users can create secrets, and the system also creates some secrets.
|
||||
|
||||
To use a secret, a pod needs to reference the secret.
|
||||
A secret can be used with a pod in two ways: as files in a [volume](/docs/user-guide/volumes) mounted on one or more of
|
||||
its containers, or used by kubelet when pulling images for the pod.
|
||||
|
||||
### Built-in Secrets
|
||||
|
||||
#### Service Accounts Automatically Create and Attach Secrets with API Credentials
|
||||
|
||||
Kubernetes automatically creates secrets which contain credentials for
|
||||
accessing the API and it automatically modifies your pods to use this type of
|
||||
secret.
|
||||
|
||||
The automatic creation and use of API credentials can be disabled or overridden
|
||||
if desired. However, if all you need to do is securely access the apiserver,
|
||||
this is the recommended workflow.
|
||||
|
||||
See the [Service Account](/docs/user-guide/service-accounts) documentation for more
|
||||
information on how Service Accounts work.
|
||||
|
||||
### Creating your own Secrets
|
||||
|
||||
#### Creating a Secret Using kubectl create secret
|
||||
|
||||
Say that some pods need to access a database. The
|
||||
username and password that the pods should use is in the files
|
||||
`./username.txt` and `./password.txt` on your local machine.
|
||||
|
||||
```shell
|
||||
# Create files needed for rest of example.
|
||||
$ echo "admin" > ./username.txt
|
||||
$ echo "1f2d1e2e67df" > ./password.txt
|
||||
```
|
||||
|
||||
The `kubectl create secret` command
|
||||
packages these files into a Secret and creates
|
||||
the object on the Apiserver.
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt
|
||||
secret "db-user-pass" created
|
||||
```
|
||||
|
||||
You can check that the secret was created like this:
|
||||
|
||||
```shell
|
||||
$ kubectl get secrets
|
||||
NAME TYPE DATA AGE
|
||||
db-user-pass Opaque 2 51s
|
||||
$ kubectl describe secrets/db-user-pass
|
||||
Name: db-user-pass
|
||||
Namespace: default
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
|
||||
Type: Opaque
|
||||
|
||||
Data
|
||||
====
|
||||
password.txt: 13 bytes
|
||||
username.txt: 6 bytes
|
||||
```
|
||||
|
||||
Note that neither `get` nor `describe` shows the contents of the file by default.
|
||||
This is to protect the secret from being exposed accidentally to someone looking
|
||||
or from being stored in a terminal log.
|
||||
|
||||
See [decoding a secret](#decoding-a-secret) for how to see the contents.
|
||||
|
||||
#### Creating a Secret Manually
|
||||
|
||||
You can also create a secret object in a file first,
|
||||
in json or yaml format, and then create that object.
|
||||
|
||||
Each item must be base64 encoded:
|
||||
|
||||
```shell
|
||||
$ echo "admin" | base64
|
||||
YWRtaW4K
|
||||
$ echo "1f2d1e2e67df" | base64
|
||||
MWYyZDFlMmU2N2RmCg==
|
||||
```
|
||||
|
||||
Now write a secret object that looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: mysecret
|
||||
type: Opaque
|
||||
data:
|
||||
password: MWYyZDFlMmU2N2RmCg==
|
||||
username: YWRtaW4K
|
||||
```
|
||||
|
||||
The data field is a map. Its keys must match
|
||||
[`DNS_SUBDOMAIN`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/design/identifiers.md), except that leading dots are also
|
||||
allowed. The values are arbitrary data, encoded using base64.
|
||||
|
||||
Create the secret using [`kubectl create`](/docs/user-guide/kubectl/kubectl_create/):
|
||||
|
||||
```shell
|
||||
$ kubectl create -f ./secret.yaml
|
||||
secret "mysecret" created
|
||||
```
|
||||
|
||||
**Encoding Note:** The serialized JSON and YAML values of secret data are encoded as
|
||||
base64 strings. Newlines are not valid within these strings and must be
|
||||
omitted (i.e. do not use `-b` option of `base64` which breaks long lines.)
|
||||
|
||||
#### Decoding a Secret
|
||||
|
||||
Get back the secret created in the previous section:
|
||||
|
||||
```shell
|
||||
$ kubectl get secret mysecret -o yaml
|
||||
apiVersion: v1
|
||||
data:
|
||||
password: MWYyZDFlMmU2N2RmCg==
|
||||
username: YWRtaW4K
|
||||
kind: Secret
|
||||
metadata:
|
||||
creationTimestamp: 2016-01-22T18:41:56Z
|
||||
name: mysecret
|
||||
namespace: default
|
||||
resourceVersion: "164619"
|
||||
selfLink: /api/v1/namespaces/default/secrets/mysecret
|
||||
uid: cfee02d6-c137-11e5-8d73-42010af00002
|
||||
type: Opaque
|
||||
```
|
||||
|
||||
Decode the password field:
|
||||
|
||||
```shell
|
||||
$ echo "MWYyZDFlMmU2N2RmCg==" | base64 -D
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
### Using Secrets
|
||||
|
||||
Secrets can be mounted as data volumes or be exposed as environment variables to
|
||||
be used by a container in a pod. They can also be used by other parts of the
|
||||
system, without being directly exposed to the pod. For example, they can hold
|
||||
credentials that other parts of the system should use to interact with external
|
||||
systems on your behalf.
|
||||
|
||||
#### Using Secrets as Files from a Pod
|
||||
|
||||
To consume a Secret in a volume in a Pod:
|
||||
|
||||
1. Create a secret or use an existing one. Multiple pods can reference the same secret.
|
||||
1. Modify your Pod definition to add a volume under `spec.volumes[]`. Name the volume anything, and have a `spec.volumes[].secret.secretName` field equal to the name of the secret object.
|
||||
1. Add a `spec.containers[].volumeMounts[]` to each container that needs the secret. Specify `spec.containers[].volumeMounts[].readOnly = true` and `spec.containers[].volumeMounts[].mountPath` to an unused directory name where you would like the secrets to appear.
|
||||
1. Modify your image and/or command line so that the the program looks for files in that directory. Each key in the secret `data` map becomes the filename under `mountPath`.
|
||||
|
||||
This is an example of a pod that mounts a secret in a volume:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "Pod",
|
||||
"metadata": {
|
||||
"name": "mypod",
|
||||
"namespace": "myns"
|
||||
},
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "mypod",
|
||||
"image": "redis",
|
||||
"volumeMounts": [{
|
||||
"name": "foo",
|
||||
"mountPath": "/etc/foo",
|
||||
"readOnly": true
|
||||
}]
|
||||
}],
|
||||
"volumes": [{
|
||||
"name": "foo",
|
||||
"secret": {
|
||||
"secretName": "mysecret"
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each secret you want to use needs to be referred to in `spec.volumes`.
|
||||
|
||||
If there are multiple containers in the pod, then each container needs its
|
||||
own `volumeMounts` block, but only one `spec.volumes` is needed per secret.
|
||||
|
||||
You can package many files into one secret, or use many secrets, whichever is convenient.
|
||||
|
||||
See another example of creating a secret and a pod that consumes that secret in a volume [here](/docs/user-guide/secrets/).
|
||||
|
||||
##### Consuming Secret Values from Volumes
|
||||
|
||||
Inside the container that mounts a secret volume, the secret keys appear as
|
||||
files and the secret values are base-64 decoded and stored inside these files.
|
||||
This is the result of commands
|
||||
executed inside the container from the example above:
|
||||
|
||||
```shell
|
||||
$ ls /etc/foo/
|
||||
username
|
||||
password
|
||||
$ cat /etc/foo/username
|
||||
admin
|
||||
$ cat /etc/foo/password
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
The program in a container is responsible for reading the secret(s) from the
|
||||
files.
|
||||
|
||||
#### Using Secrets as Environment Variables
|
||||
|
||||
To use a secret in an environment variable in a pod:
|
||||
|
||||
1. Create a secret or use an existing one. Multiple pods can reference the same secret.
|
||||
1. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in `env[x].valueFrom.secretKeyRef`.
|
||||
1. Modify your image and/or command line so that the the program looks for values in the specified environment variabless
|
||||
|
||||
This is an example of a pod that mounts a secret in a volume:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: secret-env-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: mycontainer
|
||||
image: redis
|
||||
env:
|
||||
- name: SECRET_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: username
|
||||
- name: SECRET_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: password
|
||||
restartPolicy: Never
|
||||
```
|
||||
|
||||
##### Consuming Secret Values from Environment Variables
|
||||
|
||||
Inside a container that consumes a secret in an environment variables, the secret keys appear as
|
||||
normal environment variables containing the base-64 decoded values of the secret data.
|
||||
This is the result of commands executed inside the container from the example above:
|
||||
|
||||
```shell
|
||||
$ echo $SECRET_USERNAME
|
||||
admin
|
||||
$ cat /etc/foo/password
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
#### Using imagePullSecrets
|
||||
|
||||
An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry
|
||||
password to the Kubelet so it can pull a private image on behalf of your Pod.
|
||||
|
||||
##### Manually specifying an imagePullSecret
|
||||
|
||||
Use of imagePullSecrets is described in the [images documentation](/docs/user-guide/images/#specifying-imagepullsecrets-on-a-pod)
|
||||
|
||||
### Arranging for imagePullSecrets to be Automatically Attached
|
||||
|
||||
You can manually create an imagePullSecret, and reference it from
|
||||
a serviceAccount. Any pods created with that serviceAccount
|
||||
or that default to use that serviceAccount, will get their imagePullSecret
|
||||
field set to that of the service account.
|
||||
See [here](/docs/user-guide/service-accounts/#adding-imagepullsecrets-to-a-service-account)
|
||||
for a detailed explanation of that process.
|
||||
|
||||
#### Automatic Mounting of Manually Created Secrets
|
||||
|
||||
We plan to extend the service account behavior so that manually created
|
||||
secrets (e.g. one containing a token for accessing a github account)
|
||||
can be automatically attached to pods based on their service account.
|
||||
*This is not implemented yet. See [issue 9902](http://issue.k8s.io/9902).*
|
||||
|
||||
## Details
|
||||
|
||||
### Restrictions
|
||||
|
||||
Secret volume sources are validated to ensure that the specified object
|
||||
reference actually points to an object of type `Secret`. Therefore, a secret
|
||||
needs to be created before any pods that depend on it.
|
||||
|
||||
Secret API objects reside in a namespace. They can only be referenced by pods
|
||||
in that same namespace.
|
||||
|
||||
Individual secrets are limited to 1MB in size. This is to discourage creation
|
||||
of very large secrets which would exhaust apiserver and kubelet memory.
|
||||
However, creation of many smaller secrets could also exhaust memory. More
|
||||
comprehensive limits on memory usage due to secrets is a planned feature.
|
||||
|
||||
Kubelet only supports use of secrets for Pods it gets from the API server.
|
||||
This includes any pods created using kubectl, or indirectly via a replication
|
||||
controller. It does not include pods created via the kubelets
|
||||
`--manifest-url` flag, its `--config` flag, or its REST API (these are
|
||||
not common ways to create pods.)
|
||||
|
||||
### Secret and Pod Lifetime interaction
|
||||
|
||||
When a pod is created via the API, there is no check whether a referenced
|
||||
secret exists. Once a pod is scheduled, the kubelet will try to fetch the
|
||||
secret value. If the secret cannot be fetched because it does not exist or
|
||||
because of a temporary lack of connection to the API server, kubelet will
|
||||
periodically retry. It will report an event about the pod explaining the
|
||||
reason it is not started yet. Once the a secret is fetched, the kubelet will
|
||||
create and mount a volume containing it. None of the pod's containers will
|
||||
start until all the pod's volumes are mounted.
|
||||
|
||||
Once the kubelet has started a pod's containers, its secret volumes will not
|
||||
change, even if the secret resource is modified. To change the secret used,
|
||||
the original pod must be deleted, and a new pod (perhaps with an identical
|
||||
`PodSpec`) must be created. Therefore, updating a secret follows the same
|
||||
workflow as deploying a new container image. The `kubectl rolling-update`
|
||||
command can be used ([man page](/docs/user-guide/kubectl/kubectl_rolling-update)).
|
||||
|
||||
The [`resourceVersion`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#concurrency-control-and-consistency)
|
||||
of the secret is not specified when it is referenced.
|
||||
Therefore, if a secret is updated at about the same time as pods are starting,
|
||||
then it is not defined which version of the secret will be used for the pod. It
|
||||
is not possible currently to check what resource version of a secret object was
|
||||
used when a pod was created. It is planned that pods will report this
|
||||
information, so that a replication controller restarts ones using an old
|
||||
`resourceVersion`. In the interim, if this is a concern, it is recommended to not
|
||||
update the data of existing secrets, but to create new ones with distinct names.
|
||||
|
||||
## Use cases
|
||||
|
||||
### Use-Case: Pod with ssh keys
|
||||
|
||||
Create a secret containing some ssh keys:
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic my-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
**Security Note:** think carefully before sending your own ssh keys: other users of the cluster may have access to the secret. Use a service account which you want to have accessible to all the users with whom you share the kubernetes cluster, and can revoke if they are compromised.
|
||||
|
||||
|
||||
Now we can create a pod which references the secret with the ssh key and
|
||||
consumes it in a volume:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "secret-test-pod",
|
||||
"labels": {
|
||||
"name": "secret-test"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "ssh-key-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "ssh-test-container",
|
||||
"image": "mySshImage",
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When the container's command runs, the pieces of the key will be available in:
|
||||
|
||||
```shell
|
||||
/etc/secret-volume/id-rsa.pub
|
||||
/etc/secret-volume/id-rsa
|
||||
```
|
||||
|
||||
The container is then free to use the secret data to establish an ssh connection.
|
||||
|
||||
### Use-Case: Pods with prod / test credentials
|
||||
|
||||
This example illustrates a pod which consumes a secret containing prod
|
||||
credentials and another pod which consumes a secret with test environment
|
||||
credentials.
|
||||
|
||||
Make the secrets:
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic prod-db-password --from-literal=user=produser --from-literal=password=Y4nys7f11
|
||||
secret "prod-db-password" created
|
||||
$ kubectl create secret generic test-db-password --from-literal=user=testuser --from-literal=password=iluvtests
|
||||
secret "test-db-password" created
|
||||
```
|
||||
|
||||
Now make the pods:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "List",
|
||||
"items":
|
||||
[{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "prod-db-client-pod",
|
||||
"labels": {
|
||||
"name": "prod-db-client"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "prod-db-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "db-client-container",
|
||||
"image": "myClientImage",
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "test-db-client-pod",
|
||||
"labels": {
|
||||
"name": "test-db-client"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "test-db-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "db-client-container",
|
||||
"image": "myClientImage",
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
Both containers will have the following files present on their filesystems:
|
||||
|
||||
```shell
|
||||
/etc/secret-volume/username
|
||||
/etc/secret-volume/password
|
||||
```
|
||||
|
||||
Note how the specs for the two pods differ only in one field; this facilitates
|
||||
creating pods with different capabilities from a common pod config template.
|
||||
|
||||
You could further simplify the base pod specification by using two Service Accounts:
|
||||
one called, say, `prod-user` with the `prod-db-secret`, and one called, say,
|
||||
`test-user` with the `test-db-secret`. Then, the pod spec can be shortened to, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "prod-db-client-pod",
|
||||
"labels": {
|
||||
"name": "prod-db-client"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"serviceAccount": "prod-db-client",
|
||||
"containers": [
|
||||
{
|
||||
"name": "db-client-container",
|
||||
"image": "myClientImage"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Use-case: Dotfiles in secret volume
|
||||
|
||||
In order to make piece of data 'hidden' (ie, in a file whose name begins with a dot character), simply
|
||||
make that key begin with a dot. For example, when the following secret secret is mounted into a volume:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Secret",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "dotfile-secret"
|
||||
},
|
||||
"data": {
|
||||
".secret-file": "dmFsdWUtMg0KDQo=",
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "secret-dotfiles-pod",
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "dotfile-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "dotfile-test-container",
|
||||
"image": "gcr.io/google_containers/busybox",
|
||||
"command": "ls -l /etc/secret-volume"
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
The `secret-volume` will contain a single file, called `.secret-file`, and
|
||||
the `dotfile-test-container` will have this file present at the path
|
||||
`/etc/secret-volume/.secret-file`.
|
||||
|
||||
**NOTE**
|
||||
|
||||
Files beginning with dot characters are hidden from the output of `ls -l`;
|
||||
you must use `ls -la` to see them when listing directory contents.
|
||||
|
||||
|
||||
### Use-case: Secret visible to one container in a pod
|
||||
|
||||
<a name="use-case-two-containers"></a>
|
||||
|
||||
Consider a program that needs to handle HTTP requests, do some complex business
|
||||
logic, and then sign some messages with an HMAC. Because it has complex
|
||||
application logic, there might be an unnoticed remote file reading exploit in
|
||||
the server, which could expose the private key to an attacker.
|
||||
|
||||
This could be divided into two processes in two containers: a frontend container
|
||||
which handles user interaction and business logic, but which cannot see the
|
||||
private key; and a signer container that can see the private key, and responds
|
||||
to simple signing requests from the frontend (e.g. over localhost networking).
|
||||
|
||||
With this partitioned approach, an attacker now has to trick the application
|
||||
server into doing something rather arbitrary, which may be harder than getting
|
||||
it to read a file.
|
||||
|
||||
<!-- TODO: explain how to do this while still using automation. -->
|
||||
|
||||
## Security Properties
|
||||
|
||||
### Protections
|
||||
|
||||
Because `secret` objects can be created independently of the `pods` that use
|
||||
them, there is less risk of the secret being exposed during the workflow of
|
||||
creating, viewing, and editing pods. The system can also take additional
|
||||
precautions with `secret` objects, such as avoiding writing them to disk where
|
||||
possible.
|
||||
|
||||
A secret is only sent to a node if a pod on that node requires it. It is not
|
||||
written to disk. It is stored in a tmpfs. It is deleted once the pod that
|
||||
depends on it is deleted.
|
||||
|
||||
On most Kubernetes-project-maintained distributions, communication between user
|
||||
to the apiserver, and from apiserver to the kubelets, is protected by SSL/TLS.
|
||||
Secrets are protected when transmitted over these channels.
|
||||
|
||||
Secret data on nodes is stored in tmpfs volumes and thus does not come to rest
|
||||
on the node.
|
||||
|
||||
There may be secrets for several pods on the same node. However, only the
|
||||
secrets that a pod requests are potentially visible within its containers.
|
||||
Therefore, one Pod does not have access to the secrets of another pod.
|
||||
|
||||
There may be several containers in a pod. However, each container in a pod has
|
||||
to request the secret volume in its `volumeMounts` for it to be visible within
|
||||
the container. This can be used to construct useful [security partitions at the
|
||||
Pod level](#use-case-two-containers).
|
||||
|
||||
### Risks
|
||||
|
||||
- In the API server secret data is stored as plaintext in etcd; therefore:
|
||||
- Administrators should limit access to etcd to admin users
|
||||
- Secret data in the API server is at rest on the disk that etcd uses; admins may want to wipe/shred disks
|
||||
used by etcd when no longer in use
|
||||
- Applications still need to protect the value of secret after reading it from the volume,
|
||||
such as not accidentally logging it or transmitting it to an untrusted party.
|
||||
- A user who can create a pod that uses a secret can also see the value of that secret. Even
|
||||
if apiserver policy does not allow that user to read the secret object, the user could
|
||||
run a pod which exposes the secret.
|
||||
- If multiple replicas of etcd are run, then the secrets will be shared between them.
|
||||
By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.
|
||||
- It is not possible currently to control which users of a Kubernetes cluster can
|
||||
access a secret. Support for this is planned.
|
||||
- Currently, anyone with root on any node can read any secret from the apiserver,
|
||||
by impersonating the kubelet. It is a planned feature to only send secrets to
|
||||
nodes that actually require them, to restrict the impact of a root exploit on a
|
||||
single node.
|
||||
@@ -1,59 +1,672 @@
|
||||
---
|
||||
---
|
||||
|
||||
Following this example, you will create a secret and a [pod](/docs/user-guide/pods/) that consumes that secret in a [volume](/docs/user-guide/volumes/). See [Secrets design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/secrets.md) for more information.
|
||||
Objects of type `secret` are intended to hold sensitive information, such as
|
||||
passwords, OAuth tokens, and ssh keys. Putting this information in a `secret`
|
||||
is safer and more flexible than putting it verbatim in a `pod` definition or in
|
||||
a docker image. See [Secrets design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/secrets.md) for more information.
|
||||
|
||||
## Step Zero: Prerequisites
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
This example assumes you have a Kubernetes cluster installed and running, and that you have
|
||||
installed the `kubectl` command line tool somewhere in your path. Please see the [getting
|
||||
started](/docs/getting-started-guides/) for installation instructions for your platform.
|
||||
## Overview of Secrets
|
||||
|
||||
## Step One: Create the secret
|
||||
A Secret is an object that contains a small amount of sensitive data such as
|
||||
a password, a token, or a key. Such information might otherwise be put in a
|
||||
Pod specification or in an image; putting it in a Secret object allows for
|
||||
more control over how it is used, and reduces the risk of accidental exposure.
|
||||
|
||||
A secret contains a set of named byte arrays.
|
||||
Users can create secrets, and the system also creates some secrets.
|
||||
|
||||
Use the [`secret.yaml`](/docs/user-guide/secrets/secret.yaml) file to create a secret:
|
||||
To use a secret, a pod needs to reference the secret.
|
||||
A secret can be used with a pod in two ways: as files in a [volume](/docs/user-guide/volumes) mounted on one or more of
|
||||
its containers, or used by kubelet when pulling images for the pod.
|
||||
|
||||
### Built-in Secrets
|
||||
|
||||
#### Service Accounts Automatically Create and Attach Secrets with API Credentials
|
||||
|
||||
Kubernetes automatically creates secrets which contain credentials for
|
||||
accessing the API and it automatically modifies your pods to use this type of
|
||||
secret.
|
||||
|
||||
The automatic creation and use of API credentials can be disabled or overridden
|
||||
if desired. However, if all you need to do is securely access the apiserver,
|
||||
this is the recommended workflow.
|
||||
|
||||
See the [Service Account](/docs/user-guide/service-accounts) documentation for more
|
||||
information on how Service Accounts work.
|
||||
|
||||
### Creating your own Secrets
|
||||
|
||||
#### Creating a Secret Using kubectl create secret
|
||||
|
||||
Say that some pods need to access a database. The
|
||||
username and password that the pods should use is in the files
|
||||
`./username.txt` and `./password.txt` on your local machine.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/secrets/secret.yaml
|
||||
# Create files needed for rest of example.
|
||||
$ echo "admin" > ./username.txt
|
||||
$ echo "1f2d1e2e67df" > ./password.txt
|
||||
```
|
||||
|
||||
You can use `kubectl` to see information about the secret:
|
||||
The `kubectl create secret` command
|
||||
packages these files into a Secret and creates
|
||||
the object on the Apiserver.
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt
|
||||
secret "db-user-pass" created
|
||||
```
|
||||
|
||||
You can check that the secret was created like this:
|
||||
|
||||
```shell
|
||||
$ kubectl get secrets
|
||||
NAME TYPE DATA
|
||||
test-secret Opaque 2
|
||||
NAME TYPE DATA AGE
|
||||
db-user-pass Opaque 2 51s
|
||||
$ kubectl describe secrets/db-user-pass
|
||||
Name: db-user-pass
|
||||
Namespace: default
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
|
||||
$ kubectl describe secret test-secret
|
||||
Name: test-secret
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
|
||||
Type: Opaque
|
||||
Type: Opaque
|
||||
|
||||
Data
|
||||
====
|
||||
data-1: 9 bytes
|
||||
data-2: 11 bytes
|
||||
password.txt: 13 bytes
|
||||
username.txt: 6 bytes
|
||||
```
|
||||
|
||||
## Step Two: Create a pod that consumes a secret
|
||||
Note that neither `get` nor `describe` shows the contents of the file by default.
|
||||
This is to protect the secret from being exposed accidentally to someone looking
|
||||
or from being stored in a terminal log.
|
||||
|
||||
Pods consume secrets in volumes. Now that you have created a secret, you can create a pod that
|
||||
consumes it.
|
||||
See [decoding a secret](#decoding-a-secret) for how to see the contents.
|
||||
|
||||
Use the [`secret-pod.yaml`](/docs/user-guide/secrets/secret-pod.yaml) file to create a Pod that consumes the secret.
|
||||
#### Creating a Secret Manually
|
||||
|
||||
You can also create a secret object in a file first,
|
||||
in json or yaml format, and then create that object.
|
||||
|
||||
Each item must be base64 encoded:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/secrets/secret-pod.yaml
|
||||
$ echo "admin" | base64
|
||||
YWRtaW4K
|
||||
$ echo "1f2d1e2e67df" | base64
|
||||
MWYyZDFlMmU2N2RmCg==
|
||||
```
|
||||
|
||||
This pod runs a binary that displays the content of one of the pieces of secret data in the secret
|
||||
volume:
|
||||
Now write a secret object that looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: mysecret
|
||||
type: Opaque
|
||||
data:
|
||||
password: MWYyZDFlMmU2N2RmCg==
|
||||
username: YWRtaW4K
|
||||
```
|
||||
|
||||
The data field is a map. Its keys must match
|
||||
[`DNS_SUBDOMAIN`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/design/identifiers.md), except that leading dots are also
|
||||
allowed. The values are arbitrary data, encoded using base64.
|
||||
|
||||
Create the secret using [`kubectl create`](/docs/user-guide/kubectl/kubectl_create/):
|
||||
|
||||
```shell
|
||||
$ kubectl logs secret-test-pod
|
||||
2015-04-29T21:17:24.712206409Z content of file "/etc/secret-volume/data-1": value-1
|
||||
```
|
||||
$ kubectl create -f ./secret.yaml
|
||||
secret "mysecret" created
|
||||
```
|
||||
|
||||
**Encoding Note:** The serialized JSON and YAML values of secret data are encoded as
|
||||
base64 strings. Newlines are not valid within these strings and must be
|
||||
omitted (i.e. do not use `-b` option of `base64` which breaks long lines.)
|
||||
|
||||
#### Decoding a Secret
|
||||
|
||||
Get back the secret created in the previous section:
|
||||
|
||||
```shell
|
||||
$ kubectl get secret mysecret -o yaml
|
||||
apiVersion: v1
|
||||
data:
|
||||
password: MWYyZDFlMmU2N2RmCg==
|
||||
username: YWRtaW4K
|
||||
kind: Secret
|
||||
metadata:
|
||||
creationTimestamp: 2016-01-22T18:41:56Z
|
||||
name: mysecret
|
||||
namespace: default
|
||||
resourceVersion: "164619"
|
||||
selfLink: /api/v1/namespaces/default/secrets/mysecret
|
||||
uid: cfee02d6-c137-11e5-8d73-42010af00002
|
||||
type: Opaque
|
||||
```
|
||||
|
||||
Decode the password field:
|
||||
|
||||
```shell
|
||||
$ echo "MWYyZDFlMmU2N2RmCg==" | base64 -D
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
### Using Secrets
|
||||
|
||||
Secrets can be mounted as data volumes or be exposed as environment variables to
|
||||
be used by a container in a pod. They can also be used by other parts of the
|
||||
system, without being directly exposed to the pod. For example, they can hold
|
||||
credentials that other parts of the system should use to interact with external
|
||||
systems on your behalf.
|
||||
|
||||
#### Using Secrets as Files from a Pod
|
||||
|
||||
To consume a Secret in a volume in a Pod:
|
||||
|
||||
1. Create a secret or use an existing one. Multiple pods can reference the same secret.
|
||||
1. Modify your Pod definition to add a volume under `spec.volumes[]`. Name the volume anything, and have a `spec.volumes[].secret.secretName` field equal to the name of the secret object.
|
||||
1. Add a `spec.containers[].volumeMounts[]` to each container that needs the secret. Specify `spec.containers[].volumeMounts[].readOnly = true` and `spec.containers[].volumeMounts[].mountPath` to an unused directory name where you would like the secrets to appear.
|
||||
1. Modify your image and/or command line so that the the program looks for files in that directory. Each key in the secret `data` map becomes the filename under `mountPath`.
|
||||
|
||||
This is an example of a pod that mounts a secret in a volume:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "Pod",
|
||||
"metadata": {
|
||||
"name": "mypod",
|
||||
"namespace": "myns"
|
||||
},
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "mypod",
|
||||
"image": "redis",
|
||||
"volumeMounts": [{
|
||||
"name": "foo",
|
||||
"mountPath": "/etc/foo",
|
||||
"readOnly": true
|
||||
}]
|
||||
}],
|
||||
"volumes": [{
|
||||
"name": "foo",
|
||||
"secret": {
|
||||
"secretName": "mysecret"
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each secret you want to use needs to be referred to in `spec.volumes`.
|
||||
|
||||
If there are multiple containers in the pod, then each container needs its
|
||||
own `volumeMounts` block, but only one `spec.volumes` is needed per secret.
|
||||
|
||||
You can package many files into one secret, or use many secrets, whichever is convenient.
|
||||
|
||||
See another example of creating a secret and a pod that consumes that secret in a volume [here](/docs/user-guide/secrets/).
|
||||
|
||||
##### Consuming Secret Values from Volumes
|
||||
|
||||
Inside the container that mounts a secret volume, the secret keys appear as
|
||||
files and the secret values are base-64 decoded and stored inside these files.
|
||||
This is the result of commands
|
||||
executed inside the container from the example above:
|
||||
|
||||
```shell
|
||||
$ ls /etc/foo/
|
||||
username
|
||||
password
|
||||
$ cat /etc/foo/username
|
||||
admin
|
||||
$ cat /etc/foo/password
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
The program in a container is responsible for reading the secret(s) from the
|
||||
files.
|
||||
|
||||
#### Using Secrets as Environment Variables
|
||||
|
||||
To use a secret in an environment variable in a pod:
|
||||
|
||||
1. Create a secret or use an existing one. Multiple pods can reference the same secret.
|
||||
1. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in `env[x].valueFrom.secretKeyRef`.
|
||||
1. Modify your image and/or command line so that the the program looks for values in the specified environment variabless
|
||||
|
||||
This is an example of a pod that mounts a secret in a volume:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: secret-env-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: mycontainer
|
||||
image: redis
|
||||
env:
|
||||
- name: SECRET_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: username
|
||||
- name: SECRET_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: mysecret
|
||||
key: password
|
||||
restartPolicy: Never
|
||||
```
|
||||
|
||||
##### Consuming Secret Values from Environment Variables
|
||||
|
||||
Inside a container that consumes a secret in an environment variables, the secret keys appear as
|
||||
normal environment variables containing the base-64 decoded values of the secret data.
|
||||
This is the result of commands executed inside the container from the example above:
|
||||
|
||||
```shell
|
||||
$ echo $SECRET_USERNAME
|
||||
admin
|
||||
$ cat /etc/foo/password
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
#### Using imagePullSecrets
|
||||
|
||||
An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry
|
||||
password to the Kubelet so it can pull a private image on behalf of your Pod.
|
||||
|
||||
##### Manually specifying an imagePullSecret
|
||||
|
||||
Use of imagePullSecrets is described in the [images documentation](/docs/user-guide/images/#specifying-imagepullsecrets-on-a-pod)
|
||||
|
||||
### Arranging for imagePullSecrets to be Automatically Attached
|
||||
|
||||
You can manually create an imagePullSecret, and reference it from
|
||||
a serviceAccount. Any pods created with that serviceAccount
|
||||
or that default to use that serviceAccount, will get their imagePullSecret
|
||||
field set to that of the service account.
|
||||
See [here](/docs/user-guide/service-accounts/#adding-imagepullsecrets-to-a-service-account)
|
||||
for a detailed explanation of that process.
|
||||
|
||||
#### Automatic Mounting of Manually Created Secrets
|
||||
|
||||
We plan to extend the service account behavior so that manually created
|
||||
secrets (e.g. one containing a token for accessing a github account)
|
||||
can be automatically attached to pods based on their service account.
|
||||
*This is not implemented yet. See [issue 9902](http://issue.k8s.io/9902).*
|
||||
|
||||
## Details
|
||||
|
||||
### Restrictions
|
||||
|
||||
Secret volume sources are validated to ensure that the specified object
|
||||
reference actually points to an object of type `Secret`. Therefore, a secret
|
||||
needs to be created before any pods that depend on it.
|
||||
|
||||
Secret API objects reside in a namespace. They can only be referenced by pods
|
||||
in that same namespace.
|
||||
|
||||
Individual secrets are limited to 1MB in size. This is to discourage creation
|
||||
of very large secrets which would exhaust apiserver and kubelet memory.
|
||||
However, creation of many smaller secrets could also exhaust memory. More
|
||||
comprehensive limits on memory usage due to secrets is a planned feature.
|
||||
|
||||
Kubelet only supports use of secrets for Pods it gets from the API server.
|
||||
This includes any pods created using kubectl, or indirectly via a replication
|
||||
controller. It does not include pods created via the kubelets
|
||||
`--manifest-url` flag, its `--config` flag, or its REST API (these are
|
||||
not common ways to create pods.)
|
||||
|
||||
### Secret and Pod Lifetime interaction
|
||||
|
||||
When a pod is created via the API, there is no check whether a referenced
|
||||
secret exists. Once a pod is scheduled, the kubelet will try to fetch the
|
||||
secret value. If the secret cannot be fetched because it does not exist or
|
||||
because of a temporary lack of connection to the API server, kubelet will
|
||||
periodically retry. It will report an event about the pod explaining the
|
||||
reason it is not started yet. Once the a secret is fetched, the kubelet will
|
||||
create and mount a volume containing it. None of the pod's containers will
|
||||
start until all the pod's volumes are mounted.
|
||||
|
||||
Once the kubelet has started a pod's containers, its secret volumes will not
|
||||
change, even if the secret resource is modified. To change the secret used,
|
||||
the original pod must be deleted, and a new pod (perhaps with an identical
|
||||
`PodSpec`) must be created. Therefore, updating a secret follows the same
|
||||
workflow as deploying a new container image. The `kubectl rolling-update`
|
||||
command can be used ([man page](/docs/user-guide/kubectl/kubectl_rolling-update)).
|
||||
|
||||
The [`resourceVersion`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#concurrency-control-and-consistency)
|
||||
of the secret is not specified when it is referenced.
|
||||
Therefore, if a secret is updated at about the same time as pods are starting,
|
||||
then it is not defined which version of the secret will be used for the pod. It
|
||||
is not possible currently to check what resource version of a secret object was
|
||||
used when a pod was created. It is planned that pods will report this
|
||||
information, so that a replication controller restarts ones using an old
|
||||
`resourceVersion`. In the interim, if this is a concern, it is recommended to not
|
||||
update the data of existing secrets, but to create new ones with distinct names.
|
||||
|
||||
## Use cases
|
||||
|
||||
### Use-Case: Pod with ssh keys
|
||||
|
||||
Create a secret containing some ssh keys:
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic my-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
**Security Note:** think carefully before sending your own ssh keys: other users of the cluster may have access to the secret. Use a service account which you want to have accessible to all the users with whom you share the kubernetes cluster, and can revoke if they are compromised.
|
||||
|
||||
|
||||
Now we can create a pod which references the secret with the ssh key and
|
||||
consumes it in a volume:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "secret-test-pod",
|
||||
"labels": {
|
||||
"name": "secret-test"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "ssh-key-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "ssh-test-container",
|
||||
"image": "mySshImage",
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When the container's command runs, the pieces of the key will be available in:
|
||||
|
||||
```shell
|
||||
/etc/secret-volume/id-rsa.pub
|
||||
/etc/secret-volume/id-rsa
|
||||
```
|
||||
|
||||
The container is then free to use the secret data to establish an ssh connection.
|
||||
|
||||
### Use-Case: Pods with prod / test credentials
|
||||
|
||||
This example illustrates a pod which consumes a secret containing prod
|
||||
credentials and another pod which consumes a secret with test environment
|
||||
credentials.
|
||||
|
||||
Make the secrets:
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic prod-db-password --from-literal=user=produser --from-literal=password=Y4nys7f11
|
||||
secret "prod-db-password" created
|
||||
$ kubectl create secret generic test-db-password --from-literal=user=testuser --from-literal=password=iluvtests
|
||||
secret "test-db-password" created
|
||||
```
|
||||
|
||||
Now make the pods:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "List",
|
||||
"items":
|
||||
[{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "prod-db-client-pod",
|
||||
"labels": {
|
||||
"name": "prod-db-client"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "prod-db-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "db-client-container",
|
||||
"image": "myClientImage",
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "test-db-client-pod",
|
||||
"labels": {
|
||||
"name": "test-db-client"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "test-db-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "db-client-container",
|
||||
"image": "myClientImage",
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
Both containers will have the following files present on their filesystems:
|
||||
|
||||
```shell
|
||||
/etc/secret-volume/username
|
||||
/etc/secret-volume/password
|
||||
```
|
||||
|
||||
Note how the specs for the two pods differ only in one field; this facilitates
|
||||
creating pods with different capabilities from a common pod config template.
|
||||
|
||||
You could further simplify the base pod specification by using two Service Accounts:
|
||||
one called, say, `prod-user` with the `prod-db-secret`, and one called, say,
|
||||
`test-user` with the `test-db-secret`. Then, the pod spec can be shortened to, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "prod-db-client-pod",
|
||||
"labels": {
|
||||
"name": "prod-db-client"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"serviceAccount": "prod-db-client",
|
||||
"containers": [
|
||||
{
|
||||
"name": "db-client-container",
|
||||
"image": "myClientImage"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Use-case: Dotfiles in secret volume
|
||||
|
||||
In order to make piece of data 'hidden' (ie, in a file whose name begins with a dot character), simply
|
||||
make that key begin with a dot. For example, when the following secret secret is mounted into a volume:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Secret",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "dotfile-secret"
|
||||
},
|
||||
"data": {
|
||||
".secret-file": "dmFsdWUtMg0KDQo=",
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "secret-dotfiles-pod",
|
||||
},
|
||||
"spec": {
|
||||
"volumes": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"secret": {
|
||||
"secretName": "dotfile-secret"
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "dotfile-test-container",
|
||||
"image": "gcr.io/google_containers/busybox",
|
||||
"command": "ls -l /etc/secret-volume"
|
||||
"volumeMounts": [
|
||||
{
|
||||
"name": "secret-volume",
|
||||
"readOnly": true,
|
||||
"mountPath": "/etc/secret-volume"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
The `secret-volume` will contain a single file, called `.secret-file`, and
|
||||
the `dotfile-test-container` will have this file present at the path
|
||||
`/etc/secret-volume/.secret-file`.
|
||||
|
||||
**NOTE**
|
||||
|
||||
Files beginning with dot characters are hidden from the output of `ls -l`;
|
||||
you must use `ls -la` to see them when listing directory contents.
|
||||
|
||||
|
||||
### Use-case: Secret visible to one container in a pod
|
||||
|
||||
<a name="use-case-two-containers"></a>
|
||||
|
||||
Consider a program that needs to handle HTTP requests, do some complex business
|
||||
logic, and then sign some messages with an HMAC. Because it has complex
|
||||
application logic, there might be an unnoticed remote file reading exploit in
|
||||
the server, which could expose the private key to an attacker.
|
||||
|
||||
This could be divided into two processes in two containers: a frontend container
|
||||
which handles user interaction and business logic, but which cannot see the
|
||||
private key; and a signer container that can see the private key, and responds
|
||||
to simple signing requests from the frontend (e.g. over localhost networking).
|
||||
|
||||
With this partitioned approach, an attacker now has to trick the application
|
||||
server into doing something rather arbitrary, which may be harder than getting
|
||||
it to read a file.
|
||||
|
||||
<!-- TODO: explain how to do this while still using automation. -->
|
||||
|
||||
## Security Properties
|
||||
|
||||
### Protections
|
||||
|
||||
Because `secret` objects can be created independently of the `pods` that use
|
||||
them, there is less risk of the secret being exposed during the workflow of
|
||||
creating, viewing, and editing pods. The system can also take additional
|
||||
precautions with `secret` objects, such as avoiding writing them to disk where
|
||||
possible.
|
||||
|
||||
A secret is only sent to a node if a pod on that node requires it. It is not
|
||||
written to disk. It is stored in a tmpfs. It is deleted once the pod that
|
||||
depends on it is deleted.
|
||||
|
||||
On most Kubernetes-project-maintained distributions, communication between user
|
||||
to the apiserver, and from apiserver to the kubelets, is protected by SSL/TLS.
|
||||
Secrets are protected when transmitted over these channels.
|
||||
|
||||
Secret data on nodes is stored in tmpfs volumes and thus does not come to rest
|
||||
on the node.
|
||||
|
||||
There may be secrets for several pods on the same node. However, only the
|
||||
secrets that a pod requests are potentially visible within its containers.
|
||||
Therefore, one Pod does not have access to the secrets of another pod.
|
||||
|
||||
There may be several containers in a pod. However, each container in a pod has
|
||||
to request the secret volume in its `volumeMounts` for it to be visible within
|
||||
the container. This can be used to construct useful [security partitions at the
|
||||
Pod level](#use-case-two-containers).
|
||||
|
||||
### Risks
|
||||
|
||||
- In the API server secret data is stored as plaintext in etcd; therefore:
|
||||
- Administrators should limit access to etcd to admin users
|
||||
- Secret data in the API server is at rest on the disk that etcd uses; admins may want to wipe/shred disks
|
||||
used by etcd when no longer in use
|
||||
- Applications still need to protect the value of secret after reading it from the volume,
|
||||
such as not accidentally logging it or transmitting it to an untrusted party.
|
||||
- A user who can create a pod that uses a secret can also see the value of that secret. Even
|
||||
if apiserver policy does not allow that user to read the secret object, the user could
|
||||
run a pod which exposes the secret.
|
||||
- If multiple replicas of etcd are run, then the secrets will be shared between them.
|
||||
By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.
|
||||
- It is not possible currently to control which users of a Kubernetes cluster can
|
||||
access a secret. Support for this is planned.
|
||||
- Currently, anyone with root on any node can read any secret from the apiserver,
|
||||
by impersonating the kubelet. It is a planned feature to only send secrets to
|
||||
nodes that actually require them, to restrict the impact of a root exploit on a
|
||||
single node.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
---
|
||||
|
||||
Following this example, you will create a secret and a [pod](/docs/user-guide/pods/) that consumes that secret in a [volume](/docs/user-guide/volumes/). See [Secrets design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/secrets.md) for more information.
|
||||
|
||||
## Step Zero: Prerequisites
|
||||
|
||||
This example assumes you have a Kubernetes cluster installed and running, and that you have
|
||||
installed the `kubectl` command line tool somewhere in your path. Please see the [getting
|
||||
started](/docs/getting-started-guides/) for installation instructions for your platform.
|
||||
|
||||
## Step One: Create the secret
|
||||
|
||||
A secret contains a set of named byte arrays.
|
||||
|
||||
Use the [`secret.yaml`](/docs/user-guide/secrets/secret.yaml) file to create a secret:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/secrets/secret.yaml
|
||||
```
|
||||
|
||||
You can use `kubectl` to see information about the secret:
|
||||
|
||||
```shell
|
||||
$ kubectl get secrets
|
||||
NAME TYPE DATA
|
||||
test-secret Opaque 2
|
||||
|
||||
$ kubectl describe secret test-secret
|
||||
Name: test-secret
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
|
||||
Type: Opaque
|
||||
|
||||
Data
|
||||
====
|
||||
data-1: 9 bytes
|
||||
data-2: 11 bytes
|
||||
```
|
||||
|
||||
## Step Two: Create a pod that consumes a secret
|
||||
|
||||
Pods consume secrets in volumes. Now that you have created a secret, you can create a pod that
|
||||
consumes it.
|
||||
|
||||
Use the [`secret-pod.yaml`](/docs/user-guide/secrets/secret-pod.yaml) file to create a Pod that consumes the secret.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f docs/user-guide/secrets/secret-pod.yaml
|
||||
```
|
||||
|
||||
This pod runs a binary that displays the content of one of the pieces of secret data in the secret
|
||||
volume:
|
||||
|
||||
```shell
|
||||
$ kubectl logs secret-test-pod
|
||||
2015-04-29T21:17:24.712206409Z content of file "/etc/secret-volume/data-1": value-1
|
||||
```
|
||||
Reference in New Issue
Block a user