Merge branch 'master' of https://github.com/kubernetes/kubernetes.github.io into release-1.7
* 'master' of https://github.com/kubernetes/kubernetes.github.io: whatisk8s - Chinese version (#3243) split rows and nest to stack columns (#4129) Updates to docs/admin/kubelet-tls-bootstrapping (#3617) Replace "Pod Injection Policy" with "Pod Preset" (#3667) Change example PersistentVolumeClaim add PodNodeSelector to admission-controllers (#3691) Get pods using a label selector instead of hardcoded indexes. Use JsonPath instead of go-template. correct_init_container_state (#3972) kubernetes.github.io/docs/tasks/administer-cluster: operating etcd clusters Remove --no-check-certificate flag in bare_metal_offline.md
This commit is contained in:
Andrew Chen
@@ -257,6 +257,36 @@ storage classes and how to mark a storage class as default.
|
||||
This plug-in sets the default forgiveness toleration for pods, which have no forgiveness tolerations, to tolerate
|
||||
the taints `notready:NoExecute` and `unreachable:NoExecute` for 5 minutes.
|
||||
|
||||
### PodNodeSelector
|
||||
|
||||
This plug-in defaults and limits what node selectors may be used within a namespace by reading a namespace annotation and a global configuration.
|
||||
|
||||
#### Configuration File Format
|
||||
PodNodeSelector uses the admission config file `--admission-control-config-file` to set configuration options for the behavior of the backend.
|
||||
|
||||
Note that the configuration file format will move to a versioned file in a future release.
|
||||
|
||||
This file may be json or yaml and has the following format:
|
||||
|
||||
```yaml
|
||||
podNodeSelectorPluginConfig:
|
||||
clusterDefaultNodeSelector: <node-selectors-labels>
|
||||
namespace1: <node-selectors-labels>
|
||||
namespace2: <node-selectors-labels>
|
||||
```
|
||||
|
||||
#### Configuration Annotation Format
|
||||
PodNodeSelector uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign node selectors to namespaces.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
annotations:
|
||||
scheduler.alpha.kubernetes.io/node-selector: <node-selectors-labels>
|
||||
name: namespace3
|
||||
```
|
||||
|
||||
### PodSecurityPolicy
|
||||
|
||||
This plug-in acts on creation and modification of the pod and determines if it should be admitted
|
||||
|
||||
@@ -10,87 +10,87 @@ title: TLS bootstrapping
|
||||
## Overview
|
||||
|
||||
This document describes how to set up TLS client certificate bootstrapping for kubelets.
|
||||
Kubernetes 1.4 introduces an experimental API for requesting certificates from a cluster-level
|
||||
Certificate Authority (CA). The first supported use of this API is the provisioning of TLS client
|
||||
certificates for kubelets. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439)
|
||||
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43).
|
||||
Kubernetes 1.4 introduced an API for requesting certificates from a cluster-level Certificate Authority (CA). The original intent of this API is to enable provisioning of TLS client certificates for kubelets. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439)
|
||||
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43).
|
||||
|
||||
## apiserver configuration
|
||||
## kube-apiserver configuration
|
||||
|
||||
You must provide a token file which specifies at least one "bootstrap token" assigned to a kubelet bootstrap-specific group.
|
||||
This group will later be used in the controller-manager configuration to scope approvals in the default approval
|
||||
controller. As this feature matures, you should ensure tokens are bound to an RBAC policy which limits requests
|
||||
using the bootstrap token to only be able to make requests related to certificate provisioning. When RBAC policy
|
||||
is in place, scoping the tokens to a group will allow great flexibility (e.g. you could disable a particular
|
||||
bootstrap group's access when you are done provisioning the nodes).
|
||||
controller. As this feature matures, you should ensure tokens are bound to a Role-Based Access Control (RBAC) policy which limits requests
|
||||
(using the bootstrap token) strictly to client requests related to certificate provisioning. With RBAC in place, scoping the tokens to a group allows for great flexibility (e.g. you could disable a particular bootstrap group's access when you are done provisioning the nodes).
|
||||
|
||||
### Token auth file
|
||||
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
|
||||
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example:
|
||||
### Token authentication file
|
||||
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
|
||||
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example:
|
||||
|
||||
`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
|
||||
`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
|
||||
|
||||
will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`
|
||||
will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`
|
||||
|
||||
The token file will look like the following example, where the first three values can be anything and the quoted group
|
||||
name should be as depicted:
|
||||
The token file should look like the following example, where the first three values can be anything and the quoted group
|
||||
name should be as depicted:
|
||||
|
||||
```
|
||||
02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
|
||||
```
|
||||
```
|
||||
|
||||
Add the `--token-auth-file=FILENAME` flag to the apiserver command to enable the token file.
|
||||
See docs at http://kubernetes.io/docs/admin/authentication/#static-token-file for further details.
|
||||
Add the `--token-auth-file=FILENAME` flag to the kube-apiserver command (in your systemd unit file perhaps) to enable the token file.
|
||||
See docs [here](http://kubernetes.io/docs/admin/authentication/#static-token-file) for further details.
|
||||
|
||||
### Client certificate CA bundle
|
||||
|
||||
Add the `--client-ca-file=FILENAME` flag to the apiserver command to enable client certificate authentication,
|
||||
referencing a certificate authority bundle containing the signing certificate.
|
||||
Add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable client certificate authentication,
|
||||
referencing a certificate authority bundle containing the signing certificate (e.g. `--client-ca-file=/var/lib/kubernetes/ca.pem`).
|
||||
|
||||
## controller-manager configuration
|
||||
The API for requesting certificates adds a certificate-issuing control loop to the KCM. This takes the form of a
|
||||
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk.
|
||||
Currently, all certificates issued have one year validity and a default set of key usages.
|
||||
## kube-controller-manager configuration
|
||||
The API for requesting certificates adds a certificate-issuing control loop to the Kubernetes Controller Manager. This takes the form of a
|
||||
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk. Currently, all certificates issued have one year validity and a default set of key usages.
|
||||
|
||||
### Signing assets
|
||||
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
|
||||
This CA should be trusted by the apiserver for authentication with the `--client-ca-file=SOMEFILE` flag. The management
|
||||
### Signing assets
|
||||
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
|
||||
This CA should be trusted by kube-apiserver for authentication with the `--client-ca-file=FILENAME` flag. The management
|
||||
of the CA is beyond the scope of this document but it is recommended that you generate a dedicated CA for Kubernetes.
|
||||
Both certificate and key are assumed to be PEM-encoded.
|
||||
Both certificate and key are assumed to be PEM-encoded.
|
||||
|
||||
The kube-controller-manager flags are:
|
||||
|
||||
The new controller-manager flags are:
|
||||
```
|
||||
--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"
|
||||
```
|
||||
```
|
||||
|
||||
### Auto-approval
|
||||
To ease deployment and testing, the alpha version of the certificate request API includes a flag to approve all certificate
|
||||
### Automatic approval
|
||||
To ease deployment and testing, there is an experimental flag in the certificate bootstrapping API to approve all certificate
|
||||
requests made by users in a certain group. The intended use of this is to whitelist only the group corresponding to the bootstrap
|
||||
token in the token file above. Use of this flag circumvents makes the "approval" process described below and is not recommended
|
||||
for production use.
|
||||
token in the token file above. Use of this flag circumvents the approval process described below and is not recommended
|
||||
for production use.
|
||||
|
||||
The flag is:
|
||||
|
||||
The flag is:
|
||||
```
|
||||
--insecure-experimental-approve-all-kubelet-csrs-for-group="system:kubelet-bootstrap"
|
||||
```
|
||||
```
|
||||
|
||||
## kubelet configuration
|
||||
To use request a client cert from the certificate request API, the kubelet needs a path to a kubeconfig file that contains the
|
||||
bootstrap auth token. If the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a
|
||||
client certificate from the API server. On success, a kubeconfig file referencing the generated key and obtained certificate
|
||||
is written to the path specified by `--kubeconfig`. The certificate and key file will be stored in the directory pointed
|
||||
by `--cert-dir`. The new flag is:
|
||||
## kubelet configuration
|
||||
To request a client certificate from kube-apiserver, the kubelet first needs a path to a kubeconfig file that contains the
|
||||
bootstrap authentication token. You can use `kubectl config set-cluster`, `set-credentials`, and `set-context` to build this kubeconfig. Provide the name `kubelet-bootstrap` to `kubectl config set-credentials` and include `--token=<token-value>` as follows:
|
||||
|
||||
```
|
||||
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig
|
||||
```
|
||||
|
||||
When starting the kubelet, if the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On approval of the certificate request and receipt back by the kubelet, a kubeconfig file referencing the generated key and obtained certificate is written to the path specified by `--kubeconfig`. The certificate and key file will be placed in the directory specified by `--cert-dir`.
|
||||
|
||||
The flag to enable this bootstrapping when starting the kubelet is:
|
||||
|
||||
```
|
||||
--experimental-bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig"
|
||||
```
|
||||
```
|
||||
|
||||
## kubectl approval
|
||||
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
|
||||
## kubectl approval
|
||||
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
|
||||
"Approved" status by an appropriately-privileged user. This is intended to eventually be an automated process handled by an external
|
||||
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl.
|
||||
An administrator can list CSRs with `kubectl get csr`, describe one in detail with `kubectl describe <name>`. There are
|
||||
[currently no direct approve/deny commands](https://github.com/kubernetes/kubernetes/issues/30163) so an approver will need to update
|
||||
the Status field directly. A rough example of how to do this in bash which should only be used until the porcelain merges is available
|
||||
at [https://github.com/gtank/csrctl](https://github.com/gtank/csrctl).
|
||||
|
||||
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl.
|
||||
An administrator can list CSRs with `kubectl get csr` and describe one in detail with `kubectl describe csr <name>`. Before the 1.6 release there were
|
||||
[no direct approve/deny commands](https://github.com/kubernetes/kubernetes/issues/30163) so an approver had to update
|
||||
the Status field directly ([rough how-to](https://github.com/gtank/csrctl)). Later versions of Kubernetes offer `kubectl certificate approve <name>` and `kubectl certificate deny <name>` commands.
|
||||
|
||||
@@ -199,7 +199,9 @@ Init Containers:
|
||||
[...]
|
||||
init-mydb:
|
||||
[...]
|
||||
State: Running
|
||||
State: Waiting
|
||||
Reason: PodInitializing
|
||||
Ready: False
|
||||
[...]
|
||||
Containers:
|
||||
myapp-container:
|
||||
@@ -209,13 +211,13 @@ Containers:
|
||||
Ready: False
|
||||
[...]
|
||||
Events:
|
||||
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
|
||||
--------- -------- ----- ---- ------------- -------- ------ -------
|
||||
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
|
||||
--------- -------- ----- ---- ------------- -------- ------ -------
|
||||
16s 16s 1 {default-scheduler } Normal Scheduled Successfully assigned myapp-pod to 172.17.4.201
|
||||
16s 16s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulling pulling image "busybox"
|
||||
16s 16s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulling pulling image "busybox"
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulled Successfully pulled image "busybox"
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container with docker id 5ced34a04634; Security:[seccomp=unconfined]
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container with docker id 5ced34a04634
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container with docker id 5ced34a04634; Security:[seccomp=unconfined]
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container with docker id 5ced34a04634
|
||||
$ kubectl logs myapp-pod -c init-myservice # Inspect the first init container
|
||||
$ kubectl logs myapp-pod -c init-mydb # Inspect the second init container
|
||||
```
|
||||
|
||||
@@ -5,7 +5,7 @@ assignees:
|
||||
title: Offline
|
||||
---
|
||||
|
||||
Deploy a CoreOS running Kubernetes environment. This particular guide is made to help those in an OFFLINE system, wither for testing a POC before the real deal, or you are restricted to be totally offline for your applications.
|
||||
Deploy a CoreOS running Kubernetes environment. This particular guide is made to help those in an OFFLINE system, whether for testing a POC before the real deal, or you are restricted to be totally offline for your applications.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
@@ -214,15 +214,15 @@ rm /etc/httpd/conf.d/welcome.conf
|
||||
cd /var/www/html/
|
||||
wget -O kube-register https://github.com/kelseyhightower/kube-register/releases/download/v0.0.2/kube-register-0.0.2-linux-amd64
|
||||
wget -O setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubernetes --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-apiserver --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-controller-manager --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-scheduler --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubectl --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubecfg --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubelet --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-proxy --no-check-certificate
|
||||
wget -O flanneld https://storage.googleapis.com/k8s/flanneld --no-check-certificate
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubernetes
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-apiserver
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-controller-manager
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-scheduler
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubectl
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubecfg
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kubelet
|
||||
wget https://storage.googleapis.com/kubernetes-release/release/v0.15.0/bin/linux/amd64/kube-proxy
|
||||
wget -O flanneld https://storage.googleapis.com/k8s/flanneld
|
||||
```
|
||||
|
||||
This sets up our binaries we need to run Kubernetes. This would need to be enhanced to download from the Internet for updates in the future.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
assignees:
|
||||
- mml
|
||||
- wojtek-t
|
||||
title: Configure, Upgrade, and Roll Back etcd
|
||||
title: Operating etcd clusters for Kubernetes
|
||||
redirect_from:
|
||||
- "/docs/concepts/storage/etcd-store-api-object/"
|
||||
- "/docs/concepts/storage/etcd-store-api-object.html"
|
||||
@@ -16,9 +16,7 @@ redirect_from:
|
||||
- "/docs/concepts/cluster-administration/etcd-upgrade.html"
|
||||
---
|
||||
|
||||
[etcd](https://coreos.com/etcd/docs/latest/) is a highly-available key value
|
||||
store which Kubernetes uses for persistent storage of all of its REST API
|
||||
objects.
|
||||
etcd is a strong, consistent, and highly-available key value store which Kubernetes uses for persistent storage of all of its API objects. This documentation provides specific instruction on operating, upgrading, and rolling back etcd clusters for Kubernetes. For in-depth information on etcd, see [etcd documentation](https://github.com/coreos/etcd/blob/master/Documentation/docs.md).
|
||||
|
||||
<!-- TODO(mml): Write this doc.
|
||||
|
||||
@@ -27,36 +25,196 @@ see _some doc_.
|
||||
|
||||
-->
|
||||
|
||||
## Important assumptions
|
||||
## Prerequisites
|
||||
|
||||
* Run etcd as a cluster of odd members.
|
||||
|
||||
* etcd is a leader-based distributed system. Ensure that the leader periodically send heartbeats on time to all followers to keep the cluster stable.
|
||||
|
||||
* Ensure that no resource starvation occurs.
|
||||
|
||||
Performance and stability of the cluster is sensitive to network and disk IO. Any resource starvation can lead to heartbeat timeout, causing instability of the cluster. An unstable etcd indicates that no leader is elected. Under such circumstances, a cluster cannot make any changes to its current state, which implies no new pods can be scheduled.
|
||||
|
||||
* Keeping stable etcd clusters is critical to the stability of Kubernetes clusters. Therefore, run etcd clusters on dedicated machines or isolated environments for [guaranteed resource requirements](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations).
|
||||
|
||||
## Resource requirements
|
||||
|
||||
Operating etcd with limited resources is suitable only for testing purposes. For deploying in production, advanced hardware configuration is required. Before deploying etcd in production, see [resource requirement reference documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#example-hardware-configurations).
|
||||
|
||||
## Starting Kubernetes API server
|
||||
|
||||
This section covers starting a Kubernetes API server with an etcd cluster in the deployment.
|
||||
|
||||
### Single-node etcd cluster
|
||||
|
||||
Use a single-node etcd cluster only for testing purpose.
|
||||
|
||||
1. Run the following:
|
||||
|
||||
``` bash
|
||||
./etcd --client-listen-urls=http://$PRIVATE_IP:2379 --client-advertise-urls=http://$PRIVATE_IP:2379
|
||||
```
|
||||
|
||||
2. Start Kubernetes API server with the flag `--etcd-servers=$PRIVATE_IP:2379`.
|
||||
|
||||
Replace `PRIVATE_IP` with your etcd client IP.
|
||||
|
||||
### Multi-node etcd cluster
|
||||
|
||||
For durability and high availability, run etcd as a multi-node cluster in production and back it up periodically. A five-member cluster is recommended in production. For more information, see [FAQ Documentation](https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-is-failure-tolerance).
|
||||
|
||||
Configure an etcd cluster either by static member information or by dynamic discovery. For more information on clustering, see [etcd Clustering Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md).
|
||||
|
||||
For an example, consider a five-member etcd cluster running with the following client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`, `http://$IP4:2379`, and `http://$IP5:2379`. To start a Kubernetes API server:
|
||||
|
||||
1. Run the following:
|
||||
|
||||
./etcd --client-listen-urls=http://$IP1:2379, http://$IP2:2379, http://$IP3:2379, http://$IP4:2379, http://$IP5:2379 --client-advertise-urls=http://$IP1:2379, http://$IP2:2379, http://$IP3:2379, http://$IP4:2379, http://$IP5:2379
|
||||
|
||||
|
||||
2. Start Kubernetes API servers with the flag `--etcd-servers=$IP1:2379, $IP2:2379, $IP3:2379, $IP4:2379, $IP5:2379`.
|
||||
|
||||
Replace `IP` with your client IP addresses.
|
||||
|
||||
### Multi-node etcd cluster with load balancer
|
||||
|
||||
To run a load balancing etcd cluster:
|
||||
|
||||
1. Set up an etcd cluster.
|
||||
2. Configure a load balancer in front of the etcd cluster.
|
||||
For example, let the address of the load balancer be `$LB`.
|
||||
3. Start Kubernetes API Servers with the flag `--etcd-servers=$LB:2379`.
|
||||
|
||||
## Securing etcd clusters
|
||||
|
||||
Access to etcd is equivalent to root permission in the cluster so ideally only the API server should have access to it. Considering the sensitivity of the data, it is recommended to grant permission to only those nodes that require access to etcd clusters.
|
||||
|
||||
To secure etcd, either set up firewall rules or use the security features provided by etcd. etcd security features depend on x509 Public Key Infrastructure (PKI). To begin, establish secure communication channels by generating a key and certificate pair. For example, use key pairs `peer.key` and `peer.cert` for securing communication between etcd members, and `client.key` and `client.cert` for securing communication between etcd and its clients. See the [example scripts](https://github.com/coreos/etcd/tree/master/hack/tls-setup) provided by the etcd project to generate key pairs and CA files for client authentication.
|
||||
|
||||
### Securing communication
|
||||
|
||||
To configure etcd with secure peer communication, specify flags `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use https as URL schema.
|
||||
|
||||
Similarly, to configure etcd with secure client communication, specify flags `--key-file=peer.key` and `--cert-file=peer.cert`, and use https as URL schema.
|
||||
|
||||
### Limiting access of etcd clusters
|
||||
|
||||
After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API server. Use TLS authentication to do so.
|
||||
|
||||
For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` along with TLS, it verifies the certificates from clients by using system CAs or the CA passed in by `--trusted-ca-file` flag. Specifying flags `--client-cert-auth=true` and `--trust-ca-file=etcd.ca` will restrict the access to clients with the certificate `k8sclient.cert`.
|
||||
|
||||
Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API server the access, configure it with the flags `--etcd-certfile=k8sclient.cert` and `--etcd-keyfile=k8sclient.key`.
|
||||
|
||||
**Note**: etcd authentication is not currently supported by Kubernetes. For more information, see see the related issue [Support Basic Auth for Etcd v2](https://github.com/kubernetes/kubernetes/issues/23398).
|
||||
|
||||
## Replacing a failed etcd member
|
||||
|
||||
etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member.
|
||||
|
||||
Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, member1=http://10.0.0.1, member2=http://10.0.0.2, and member3=http://10.0.0.3. When member1 fails, replace it with member4=http://10.0.0.4.
|
||||
|
||||
1. Get the member ID of the failed member1:
|
||||
|
||||
`etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list`
|
||||
|
||||
The following message is displayed:
|
||||
|
||||
8211f1d0f64f3269, started, member1, http://10.0.0.1:12380, http://10.0.0.1:2379
|
||||
91bc3c398fb3c146, started, member2, http://10.0.0.1:2380, http://10.0.0.2:2379
|
||||
fd422379fda50e48, started, member3, http://10.0.0.1:2380, http://10.0.0.3:2379
|
||||
|
||||
2. Remove the failed member:
|
||||
|
||||
`etcdctl member remove 8211f1d0f64f3269`
|
||||
|
||||
The following message is displayed:
|
||||
|
||||
Removed member 8211f1d0f64f3269 from cluster
|
||||
|
||||
3. Add the new member:
|
||||
|
||||
`./etcdctl member add member4 --peer-urls=http://10.0.0.4:2380`
|
||||
|
||||
The following message is displayed:
|
||||
|
||||
Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4
|
||||
|
||||
4. Start the newly added member on a machine with the IP `10.0.0.4`:
|
||||
|
||||
```bash
|
||||
export ETCD_NAME="member4"
|
||||
export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380"
|
||||
export ETCD_INITIAL_CLUSTER_STATE=existing
|
||||
etcd [flags]
|
||||
```
|
||||
5. Do either of the following:
|
||||
|
||||
1. Update its `--etcd-servers` flag to make Kubernetes aware of the configuration changes, then restart the Kubernetes API server.
|
||||
2. Update the load balancer configuration if a load balancer is used in the deployment.
|
||||
|
||||
For more information on cluster reconfiguration, see [etcd Reconfiguration Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member).
|
||||
|
||||
## Backing up an etcd cluster
|
||||
|
||||
All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all master nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files.
|
||||
|
||||
Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshot.
|
||||
|
||||
### Built-in snapshot
|
||||
|
||||
etcd supports built-in snapshot, so backing up an etcd cluster is easy. A snapshot may either be taken from a live member with the `etcdctl snapshot save` command or by copying the `member/snap/db` file from an etcd [data directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir) that is not currently used by an etcd process. `datadir` is located at `$DATA_DIR/member/snap/db`. Taking the snapshot will normally not affect the performance of the member.
|
||||
|
||||
Below is an example for taking a snapshot of the keyspace served by `$ENDPOINT` to the file `snapshotdb`:
|
||||
|
||||
```sh
|
||||
ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshotdb
|
||||
# exit 0
|
||||
|
||||
# verify the snapshot
|
||||
ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshotdb
|
||||
+----------+----------+------------+------------+
|
||||
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
|
||||
+----------+----------+------------+------------+
|
||||
| fe01cf57 | 10 | 7 | 2.1 MB |
|
||||
+----------+----------+------------+------------+
|
||||
```
|
||||
|
||||
### Volume snapshot
|
||||
|
||||
If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.
|
||||
|
||||
## Scaling up etcd clusters
|
||||
|
||||
Scaling up etcd clusters increases availability by trading off performance. Scaling does not increase cluster performance nor capability. A general rule is not to scale up or down etcd clusters. Do not configure any auto scaling groups for etcd clusters. It is highly recommended to always run a static five-member etcd cluster for production Kubernetes clusters at any officially supported scale.
|
||||
|
||||
A reasonable scaling is to upgrade a three-member cluster to a five-member one, when more reliability is desired. See [etcd Reconfiguration Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member) for information on how to add members into an existing cluster.
|
||||
|
||||
## Restoring an etcd cluster
|
||||
|
||||
etcd supports restoring from snapshots that are taken from an etcd process of the [major.minor](http://semver.org/) version. Restoring a version from a different patch version of etcd also is supported. A restore operation is employed to recover the data of a failed cluster.
|
||||
|
||||
Before starting the restore operation, a snapshot file must be present. It can either be a snapshot file from a previous backup operation, or from a remaining [data directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir). `datadir` is located at `$DATA_DIR/member/snap/db`. For more information and examples on restoring a cluster from a snapshot file, see [etcd disaster recovery documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster).
|
||||
|
||||
If the access URLs of the restored cluster is changed from the previous cluster, the Kubernetes API server must be reconfigured accordingly. In this case, restart Kubernetes API server with the flag `--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag `--etcd-servers=$OLD__ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and `$OLD__ETCD_CLUSTER` with the respective IP addresses. If a load balancer is used in front of an etcd cluster, you might need to update the load balancer instead.
|
||||
|
||||
If the majority of etcd members have permanently failed, the etcd cluster is considered failed. In this scenario, Kubernetes cannot make any changes to its current state. Although the scheduled pods might continue to run, no new pods can be scheduled. In such cases, recover the etcd cluster and potentially reconfigure Kubernetes API server to fix the issue.
|
||||
|
||||
## Upgrading and rolling back etcd clusters
|
||||
|
||||
### Important assumptions
|
||||
|
||||
The upgrade procedure described in this document assumes that either:
|
||||
|
||||
1. The etcd cluster has only a single node, or
|
||||
1. The etcd cluster has multiple nodes. In this case, the upgrade procedure requires shutting down the
|
||||
etcd cluster. During the time the etcd cluster is shutdown, the Kubernetes API Server
|
||||
will be read only.
|
||||
1. The etcd cluster has only a single node
|
||||
2. The etcd cluster has multiple nodes.
|
||||
|
||||
In this case, the upgrade procedure requires shutting down the
|
||||
etcd cluster. During the time the etcd cluster is shut down, the Kubernetes API Server will be read only.
|
||||
|
||||
**Warning**: Deviations from the assumptions are untested by continuous
|
||||
integration, and deviations might create undesirable consequences. Additional
|
||||
information about operating an etcd cluster is available [from the etcd
|
||||
maintainers](https://github.com/coreos/etcd/tree/master/Documentation).
|
||||
integration, and deviations might create undesirable consequences. Additional information about operating an etcd cluster is available [from the etcd maintainers](https://github.com/coreos/etcd/tree/master/Documentation).
|
||||
|
||||
## etcd for Kubernetes: high-level goals
|
||||
|
||||
Access control: Give only kube-apiserver read/write access to etcd. You do not
|
||||
want the API server's etcd exposed to every node in your cluster, or worse, to the
|
||||
internet at large. Access to etcd is equivalent to root access in your
|
||||
cluster.
|
||||
|
||||
Data reliability: For reasonable safety, either etcd needs to be run as a
|
||||
[cluster](/docs/admin/high-availability/#clustering-etcd), or etcd's data directory
|
||||
should be located on durable storage, for example, a Google Compute Engine persistent disk.
|
||||
In either case, if high availability is
|
||||
required--as it might be in a production cluster--the data directory ought to be
|
||||
[backed up periodically](https://coreos.com/etcd/docs/latest/op-guide/recovery.html)
|
||||
to reduce downtime in case of corruption.
|
||||
|
||||
## Background
|
||||
### Background
|
||||
|
||||
As of Kubernetes version 1.5.1, we are still using etcd from the 2.2.1 release with
|
||||
the v2 API. Also, we have no pre-existing process for updating etcd, as we have
|
||||
@@ -79,9 +237,9 @@ There are requirements on how an etcd cluster upgrade can be performed. The prim
|
||||
#### One minor release at a time
|
||||
|
||||
Upgrade only one minor release at a time. For example, we cannot upgrade directly from 2.1.x to 2.3.x.
|
||||
Within patch releases it is possible to upgrade and downgrade between arbitrary versions. Starting a cluster for
|
||||
Within patch releases it is possible to upgrade and downgrade between arbitrary versions. Starting a cluster for
|
||||
any intermediate minor release, waiting until the cluster is healthy, and then
|
||||
shutting down the cluster down will perform the migration. For example, to upgrade from version 2.1.x to 2.3.y,
|
||||
shutting down the cluster down will perform the migration. For example, to upgrade from version 2.1.x to 2.3.y,
|
||||
it is enough to start etcd in 2.2.z version, wait until it is healthy, stop it, and then start the
|
||||
2.3.y version.
|
||||
|
||||
@@ -193,7 +351,7 @@ script works as follows:
|
||||
in v2 format.
|
||||
1. Finally update the contents of the version file.
|
||||
|
||||
## Upgrade procedure
|
||||
### Upgrade procedure
|
||||
Simply modify the command line in the etcd manifest to:
|
||||
|
||||
1. Run the migration script. If the previously run version is already in the
|
||||
|
||||
@@ -9,3 +9,4 @@ spec:
|
||||
resources:
|
||||
requests:
|
||||
storage: 3Gi
|
||||
storageClassName: ""
|
||||
|
||||
@@ -74,7 +74,7 @@ of the form: `podpreset.admission.kubernetes.io/exclude: "true"`.
|
||||
### Simple Pod Spec Example
|
||||
|
||||
This is a simple example to show how a Pod spec is modified by the Pod
|
||||
Injection Policy.
|
||||
Preset.
|
||||
|
||||
**User submitted pod spec:**
|
||||
|
||||
@@ -148,8 +148,8 @@ spec:
|
||||
|
||||
### Pod Spec with `ConfigMap` Example
|
||||
|
||||
This is an example to show how a Pod spec is modified by the Pod Injection
|
||||
Policy that defines a `ConfigMap` for Environment Variables.
|
||||
This is an example to show how a Pod spec is modified by the Pod Preset
|
||||
that defines a `ConfigMap` for Environment Variables.
|
||||
|
||||
**User submitted pod spec:**
|
||||
|
||||
@@ -266,7 +266,7 @@ spec:
|
||||
### ReplicaSet with Pod Spec Example
|
||||
|
||||
The following example shows that only the pod spec is modified by the Pod
|
||||
Injection Policy.
|
||||
Preset.
|
||||
|
||||
**User submitted ReplicaSet:**
|
||||
|
||||
@@ -460,8 +460,8 @@ spec:
|
||||
|
||||
### Conflict Example
|
||||
|
||||
This is a example to show how a Pod spec is not modified by the Pod Injection
|
||||
Policy when there is a conflict.
|
||||
This is an example to show how a Pod spec is not modified by the Pod Preset
|
||||
when there is a conflict.
|
||||
|
||||
**User submitted pod spec:**
|
||||
|
||||
|
||||
@@ -41,51 +41,59 @@ title: Overview
|
||||
<div class="content__modules">
|
||||
<h2>Kubernetes Basics Modules</h2>
|
||||
<div class="row">
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/cluster-intro/"><img src="./public/images/module_01.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="cluster-intro/"><h5>1. Create a Kubernetes cluster</h5></a>
|
||||
<div class="col-md-12">
|
||||
<div class="row">
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/cluster-intro/"><img src="./public/images/module_01.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="cluster-intro/"><h5>1. Create a Kubernetes cluster</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/deploy-intro/"><img src="./public/images/module_02.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="deploy-intro/"><h5>2. Deploy an app</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/explore-intro/"><img src="./public/images/module_03.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="explore-intro/"><h5>3. Explore your app</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/deploy-intro/"><img src="./public/images/module_02.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="deploy-intro/"><h5>2. Deploy an app</h5></a>
|
||||
<div class="col-md-12">
|
||||
<div class="row">
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/expose-intro/"><img src="./public/images/module_04.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="expose-intro/"><h5>4. Expose your app publicly</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/explore-intro/"><img src="./public/images/module_03.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="explore-intro/"><h5>3. Explore your app</h5></a>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/scale-intro/"><img src="./public/images/module_05.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="scale-intro/"><h5>5. Scale up your app</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/expose-intro/"><img src="./public/images/module_04.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="expose-intro/"><h5>4. Expose your app publicly</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/scale-intro/"><img src="./public/images/module_05.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="scale-intro/"><h5>5. Scale up your app</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/update-intro/"><img src="./public/images/module_06.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="update-intro/"><h5>6. Update your app</h5></a>
|
||||
<div class="col-md-4">
|
||||
<div class="thumbnail">
|
||||
<a href="/docs/tutorials/kubernetes-basics/update-intro/"><img src="./public/images/module_06.svg?v=1469803628347" alt=""></a>
|
||||
<div class="caption">
|
||||
<a href="update-intro/"><h5>6. Update your app</h5></a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -491,10 +491,10 @@ web-0 1/1 Running 0 3s
|
||||
Get the Pods to view their container images.
|
||||
|
||||
```shell{% raw %}
|
||||
for p in 0 1 2; do kubectl get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
gcr.io/google_containers/nginx-slim:0.8
|
||||
kubectl get pod -l app=nginx -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
|
||||
web-0 gcr.io/google_containers/nginx-slim:0.7
|
||||
web-1 gcr.io/google_containers/nginx-slim:0.8
|
||||
web-2 gcr.io/google_containers/nginx-slim:0.8
|
||||
{% endraw %}```
|
||||
|
||||
`web-0` has had its image updated. Complete the update by deleting the remaining
|
||||
@@ -532,10 +532,10 @@ web-2 1/1 Running 0 36s
|
||||
Get the Pods to view their container images.
|
||||
|
||||
```shell{% raw %}
|
||||
for p in 0 1 2; do kubectl get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
gcr.io/google_containers/nginx-slim:0.7
|
||||
kubectl get pod -l app=nginx -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
|
||||
web-0 gcr.io/google_containers/nginx-slim:0.7
|
||||
web-1 gcr.io/google_containers/nginx-slim:0.7
|
||||
web-2 gcr.io/google_containers/nginx-slim:0.7
|
||||
{% endraw %}```
|
||||
|
||||
All the Pods in the StatefulSet are now running a new container image.
|
||||
|
||||
Reference in New Issue
Block a user