From 9702579d38702877379c6f278b2694fdb213d310 Mon Sep 17 00:00:00 2001 From: Chuck Ha Date: Fri, 11 May 2018 11:05:55 -0400 Subject: [PATCH] Adds a task for ha etcd with kubeadm (#8301) Signed-off-by: Chuck Ha --- .../setup-ha-etcd-with-kubeadm.md | 203 ++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 content/en/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md diff --git a/content/en/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md b/content/en/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md new file mode 100644 index 0000000000..9bc4d95059 --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md @@ -0,0 +1,203 @@ +--- +reviewers: +- chuckha +title: Set up a High-Availablity Etcd Cluster With Kubeadm +content_template: templates/task +--- + +{{% capture overview %}} + +Kubeadm defaults to running a single member etcd cluster in a static pod managed +by the kubelet on the control plane node. This is not a highly-available setup +as the the etcd cluster contains only one member and cannot sustain any members +becoming unavailable. This task walks through the process of creating a highly +available etcd cluster of three members that can be used as an external etcd +when using kubeadm to set up a kubernetes cluster. + +{{% /capture %}} + +{{% capture prerequisites %}} + +* Three hosts that can talk to each other over ports 2379 and 2380. This + document assumes these default ports. However, they are configurable through + the kubeadm config file. +* Each host must [have docker, kubelet, and kubeadm installed][toolbox]. +* Some infrastructure to copy files between hosts (e.g., ssh). + +[toolbox]: /docs/setup/independent/install-kubeadm/ + +{{% /capture %}} + +{{% capture steps %}} + +The general approach is to generate all certs on one node and only distribute +the *necessary* files to the other nodes. + +## Create configuration files for kubeadm + +Using the template provided below, create one kubeadm configuration file for +each host that will have an etcd member running on it. Update the value of +`CURRENT_HOST` and `NAME` before running the `cat` command. + +``` +export HOST0=10.0.0.1 # Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts +export HOST1=10.0.0.2 +export HOST2=10.0.0.3 + +# Create temp directories to store files that will end up on other hosts. +mkdir -p /tmp/${HOST0}/certs /tmp/${HOST1}/certs /tmp/${HOST2}/certs + +export CURRENT_HOST="${HOST0}" # Update on each ranging through HOST0, HOST1 and HOST2 +export NAME=infra0 # Update to use infra0 for HOST0, infra1 for HOST1 and infra2 for HOST2 + +cat << EOF > /tmp/${CURRENT_HOST}/kubeadmcfg.yaml +apiVersion: "kubeadm.k8s.io/v1alpha1" +kind: MasterConfiguration +etcd: + serverCertSANs: + - "${CURRENT_HOST}" + peerCertSANs: + - "${CURRENT_HOST}" + extraArgs: + initial-cluster: infra0=https://${HOST0}:2380,infra1=https://${HOST1}:2380,infra2=https://${HOST2}:2380 + initial-cluster-state: new + name: ${NAME} + listen-peer-urls: https://${CURRENT_HOST}:2380 + listen-client-urls: https://${CURRENT_HOST}:2379 + advertise-client-urls: https://${CURRENT_HOST}:2379 + initial-advertise-peer-urls: https://${CURRENT_HOST}:2380 +EOF +``` + +## Generate certificates needed for the etcd cluster + +### Certificate Authority + +If you already have a CA then the only action that is copying the CA's `crt` and +`key` file to `/etc/kubernetes/pki/etcd/ca.crt` and +`/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied, please +skip to the Certificate Swizzling section below. + +If you do not already have a CA then run this command on `$HOST0` (where you +generated the configuration files for kubeadm). + +``` +kubeadm alpha phase certs etcd-ca +``` + +This creates two files + +1. `/etc/kubernetes/pki/etcd/ca.crt` +2. `/etc/kubernetes/pki/etcd/ca.key` + +### Create certificates for each member + +In this step we create all the certs for each host in our cluster. + +``` +kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml +kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml +kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml +# Move the generated certs out of the generated directory +find /etc/kubernetes/pki/etcd -not -name ca.crt -not -name ca.key -type f -exec mv {} /tmp/${HOST2}/certs \; +cp /etc/kubernetes/pki/etcd/ca.crt /tmp/${HOST2}/certs + +kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml +kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml +kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml +# Move the generated certs out of the generated directory +find /etc/kubernetes/pki/etcd -not -name ca.crt -not -name ca.key -type f -exec mv {} /tmp/${HOST1}/certs \; +cp /etc/kubernetes/pki/etcd/ca.crt /tmp/${HOST1}/certs + +kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml +kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml +kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml +# No need to move the certs because they are for HOST0 +``` + +### Copy certs and configs to other hosts + +Copy the certs and configs in each tmp directory to the respective hosts and put +the certs owned by root:root in `/etc/kubernetes/pki/etcd/`. + +The steps to get these files on `$HOST1` might look like this if you can ssh +between hosts: + +``` +root@HOST0 $ scp -i /home/ubuntu/.ssh/id_rsa -r /tmp/${HOST1}/* ubuntu@${HOST1}:/home/ubuntu +root@HOST0 $ ssh -i /home/ubuntu/.ssh/id_rsa ubuntu@${HOST1} +ubuntu@HOST1 $ sudo -s +root@HOST1 $ chown -R root:root certs +root@HOST1 $ mv certs/* /etc/kubernetes/pki/etcd/ +# Repeat for HOST2 +``` + +### List of all generated certs + +This is a list of all the files you have generated and where on which host they +should live. + +#### Host 0 + +1. `/etc/kubernetes/pki/etcd/ca.crt` +1. `/etc/kubernetes/pki/etcd/ca.key` +1. `/etc/kubernetes/pki/etcd/server.crt` +1. `/etc/kubernetes/pki/etcd/server.key` +1. `/etc/kubernetes/pki/etcd/peer.crt` +1. `/etc/kubernetes/pki/etcd/peer.key` +1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt` +1. `/etc/kubernetes/pki/etcd/healthcheck-client.key` +1. `/tmp/${HOST0}/kubeadmcfg.yaml` + +#### Host 1 + +1. `/etc/kubernetes/pki/etcd/ca.crt` +1. `/etc/kubernetes/pki/etcd/server.crt` +1. `/etc/kubernetes/pki/etcd/server.key` +1. `/etc/kubernetes/pki/etcd/peer.crt` +1. `/etc/kubernetes/pki/etcd/peer.key` +1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt` +1. `/etc/kubernetes/pki/etcd/healthcheck-client.key` +1. `/home/ubuntu/kubeadmcfg.yaml` + +#### Host 2 + +1. `/etc/kubernetes/pki/etcd/ca.crt` +1. `/etc/kubernetes/pki/etcd/server.crt` +1. `/etc/kubernetes/pki/etcd/server.key` +1. `/etc/kubernetes/pki/etcd/peer.crt` +1. `/etc/kubernetes/pki/etcd/peer.key` +1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt` +1. `/etc/kubernetes/pki/etcd/healthcheck-client.key` +1. `/home/ubuntu/kubeadmcfg.yaml` + +## Manifests + +Now that the certs and configs are in place we can create the manifest. On each +host run the `kubeadm` command to generate a static manifest for etcd. + +``` +root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml +root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml +root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml +``` + +## Optional: Check the cluster health + +``` +docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.14 etcdctl --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --ca-file /etc/kubernetes/pki/etcd/ca.crt --endpoints https://${HOST0}:2379 cluster-health +... +cluster is healthy +``` + +{{% /capture %}} + +{{% capture whatsnext %}} + +Once your have a working 3 member etcd cluster, you can continue [setting up an +HA control plane using +kubeadm](/docs/tasks/administer-cluster/highly-available-master/). + +{{% /capture %}} + +