From 96d116e29fffc799370e56e7a68ec6685465fa7e Mon Sep 17 00:00:00 2001 From: Andrew Smith Date: Tue, 22 Mar 2016 11:45:53 +0000 Subject: [PATCH 1/5] Add leader elect flag --- docs/admin/high-availability/kube-controller-manager.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/high-availability/kube-controller-manager.yaml b/docs/admin/high-availability/kube-controller-manager.yaml index 736181fc7c..0ecbebb276 100644 --- a/docs/admin/high-availability/kube-controller-manager.yaml +++ b/docs/admin/high-availability/kube-controller-manager.yaml @@ -9,7 +9,7 @@ spec: - -c - /usr/local/bin/kube-controller-manager --master=127.0.0.1:8080 --cluster-name=e2e-test-bburns --cluster-cidr=10.245.0.0/16 --allocate-node-cidrs=true --cloud-provider=gce --service-account-private-key-file=/srv/kubernetes/server.key - --v=2 1>>/var/log/kube-controller-manager.log 2>&1 + --v=2 --leader-elect=true 1>>/var/log/kube-controller-manager.log 2>&1 image: gcr.io/google_containers/kube-controller-manager:fda24638d51a48baa13c35337fcd4793 livenessProbe: httpGet: From 06b593b6ee232bceb50f4936d5c0e43cf6a56993 Mon Sep 17 00:00:00 2001 From: Andrew Smith Date: Tue, 22 Mar 2016 11:46:10 +0000 Subject: [PATCH 2/5] Update kube-scheduler.yaml --- docs/admin/high-availability/kube-scheduler.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/high-availability/kube-scheduler.yaml b/docs/admin/high-availability/kube-scheduler.yaml index 1b23ee1bb8..9e997cbc47 100644 --- a/docs/admin/high-availability/kube-scheduler.yaml +++ b/docs/admin/high-availability/kube-scheduler.yaml @@ -10,7 +10,7 @@ spec: command: - /bin/sh - -c - - /usr/local/bin/kube-scheduler --master=127.0.0.1:8080 --v=2 1>>/var/log/kube-scheduler.log + - /usr/local/bin/kube-scheduler --master=127.0.0.1:8080 --v=2 --leader-elect=true 1>>/var/log/kube-scheduler.log 2>&1 livenessProbe: httpGet: From 810247a1fa64002f83b8f945750e1e2e4134fa25 Mon Sep 17 00:00:00 2001 From: Andrew Smith Date: Tue, 22 Mar 2016 11:49:41 +0000 Subject: [PATCH 3/5] Replace podmaster method with --leader-elect --- docs/admin/high-availability.md | 36 ++++++++------------------------- 1 file changed, 8 insertions(+), 28 deletions(-) diff --git a/docs/admin/high-availability.md b/docs/admin/high-availability.md index beaf7b1db9..1446f1fed1 100644 --- a/docs/admin/high-availability.md +++ b/docs/admin/high-availability.md @@ -1,13 +1,8 @@ ---- ---- - -## Introduction - -PLEASE NOTE: Note that the podmaster implementation is obsoleted by https://github.com/kubernetes/kubernetes/pull/16830, -which provides a primitive for leader election in the experimental kubernetes API. - -Nevertheless, the concepts and implementation in this document are still valid, as is the podmaster implementation itself. - +--- +--- + +## Introduction + This document describes how to build a high-availability (HA) Kubernetes cluster. This is a fairly advanced topic. Users who merely want to experiment with Kubernetes are encouraged to use configurations that are simpler to set up such as the simple [Docker based single node cluster instructions](/docs/getting-started-guides/docker), @@ -188,11 +183,7 @@ them to talk to the external load balancer's IP address. So far we have set up state storage, and we have set up the API server, but we haven't run anything that actually modifies cluster state, such as the controller manager and scheduler. To achieve this reliably, we only want to have one actor modifying state at a time, but we want replicated instances of these actors, in case a machine dies. To achieve this, we are going to use a lease-lock in etcd to perform -master election. On each of the three apiserver nodes, we run a small utility application named `podmaster`. It's job is to implement a master -election protocol using etcd "compare and swap". If the apiserver node wins the election, it starts the master component it is managing (e.g. the scheduler), if it -loses the election, it ensures that any master components running on the node (e.g. the scheduler) are stopped. - -In the future, we expect to more tightly integrate this lease-locking into the scheduler and controller-manager binaries directly, as described in the [high availability design proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/high-availability.md) +master election. We will use the `--leader-elect` flag for each scheduler and controller-manager, using etcd locks this will ensure that only 1 instance of the scheduler and controller-manager are running at once. ### Installing configuration files @@ -204,18 +195,7 @@ touch /var/log/kube-controller-manager.log ``` Next, set up the descriptions of the scheduler and controller manager pods on each node. -by copying [kube-scheduler.yaml](/docs/admin/high-availability/kube-scheduler.yaml) and [kube-controller-manager.yaml](/docs/admin/high-availability/kube-controller-manager.yaml) into the `/srv/kubernetes/` directory. - -### Running the podmaster - -Now that the configuration files are in place, copy the [podmaster.yaml](/docs/admin/high-availability/podmaster.yaml) config file into `/etc/kubernetes/manifests/` - -As before, the kubelet on the node monitors this directory, and will start an instance of the podmaster using the pod specification provided in `podmaster.yaml`. - -Now you will have one instance of the scheduler process running on a single master node, and likewise one -controller-manager process running on a single (possibly different) master node. If either of these processes fail, -the kubelet will restart them. If any of these nodes fail, the process will move to a different instance of a master -node. +by copying [kube-scheduler.yaml](/docs/admin/high-availability/kube-scheduler.yaml) and [kube-controller-manager.yaml](/docs/admin/high-availability/kube-controller-manager.yaml) into the `/etc/kubernetes/manifests/` directory. ## Conclusion @@ -225,4 +205,4 @@ If you have an existing cluster, this is as simple as reconfiguring your kubelet restarting the kubelets on each node. If you are turning up a fresh cluster, you will need to install the kubelet and kube-proxy on each worker node, and -set the `--apiserver` flag to your replicated endpoint. +set the `--apiserver` flag to your replicated endpoint. From 1484749ca30e4a8800e1fcbe64cb3e24c673aea9 Mon Sep 17 00:00:00 2001 From: Andrew Smith Date: Tue, 22 Mar 2016 18:03:15 +0000 Subject: [PATCH 4/5] Correction: We use API locks, not etcd locks --- docs/admin/high-availability.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/admin/high-availability.md b/docs/admin/high-availability.md index 1446f1fed1..27657db757 100644 --- a/docs/admin/high-availability.md +++ b/docs/admin/high-availability.md @@ -7,7 +7,7 @@ This document describes how to build a high-availability (HA) Kubernetes cluster Users who merely want to experiment with Kubernetes are encouraged to use configurations that are simpler to set up such as the simple [Docker based single node cluster instructions](/docs/getting-started-guides/docker), or try [Google Container Engine](https://cloud.google.com/container-engine/) for hosted Kubernetes. - +le Also, at this time high availability support for Kubernetes is not continuously tested in our end-to-end (e2e) testing. We will be working to add this continuous testing, but for now the single-node master installations are more heavily tested. @@ -182,8 +182,8 @@ them to talk to the external load balancer's IP address. So far we have set up state storage, and we have set up the API server, but we haven't run anything that actually modifies cluster state, such as the controller manager and scheduler. To achieve this reliably, we only want to have one actor modifying state at a time, but we want replicated -instances of these actors, in case a machine dies. To achieve this, we are going to use a lease-lock in etcd to perform -master election. We will use the `--leader-elect` flag for each scheduler and controller-manager, using etcd locks this will ensure that only 1 instance of the scheduler and controller-manager are running at once. +instances of these actors, in case a machine dies. To achieve this, we are going to use a lease-lock in the API to perform +master election. We will use the `--leader-elect` flag for each scheduler and controller-manager, using a lease in the API will ensure that only 1 instance of the scheduler and controller-manager are running at once. ### Installing configuration files From 4c79e1c381ede9a0714a341ddfec37d501b4230f Mon Sep 17 00:00:00 2001 From: Andrew Smith Date: Tue, 22 Mar 2016 18:03:52 +0000 Subject: [PATCH 5/5] Update high-availability.md --- docs/admin/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/high-availability.md b/docs/admin/high-availability.md index 27657db757..16a596f2fe 100644 --- a/docs/admin/high-availability.md +++ b/docs/admin/high-availability.md @@ -7,7 +7,7 @@ This document describes how to build a high-availability (HA) Kubernetes cluster Users who merely want to experiment with Kubernetes are encouraged to use configurations that are simpler to set up such as the simple [Docker based single node cluster instructions](/docs/getting-started-guides/docker), or try [Google Container Engine](https://cloud.google.com/container-engine/) for hosted Kubernetes. -le + Also, at this time high availability support for Kubernetes is not continuously tested in our end-to-end (e2e) testing. We will be working to add this continuous testing, but for now the single-node master installations are more heavily tested.