From b786d89c9057b8f2d1bf4904515953eb8142ecb0 Mon Sep 17 00:00:00 2001 From: Martin Dietze Date: Wed, 14 Feb 2018 17:08:35 +0100 Subject: [PATCH 1/4] HA guide for kubeadm: - tabified section on load balancing - added tab for keepalived configuration --- docs/setup/independent/high-availability.md | 84 ++++++++++++++++++++- 1 file changed, 83 insertions(+), 1 deletion(-) diff --git a/docs/setup/independent/high-availability.md b/docs/setup/independent/high-availability.md index c4c6b72d8c..6e1b6b1a80 100644 --- a/docs/setup/independent/high-availability.md +++ b/docs/setup/independent/high-availability.md @@ -385,7 +385,14 @@ Please select one of the tabs to see installation instructions for the respectiv ## Set up master Load Balancer -The next step is to create a Load Balancer that sits in front of your master nodes. How you do this depends on your environment; you could, for example, leverage a cloud provider Load Balancer, or set up your own using nginx, keepalived, or HAproxy. Some examples of cloud provider solutions are: +The next step is to create a Load Balancer that sits in front of your master nodes. How you do this depends on your environment; you could, for example, leverage a cloud provider Load Balancer, or set up your own using nginx, keepalived, or HAproxy. + +{% capture choose %} +Please select one of the tabs to see installation instructions for information on load balancing in the respective environment. +{% endcapture %} + +{% capture cloud %} +Some examples of cloud provider solutions are: * [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/) * [GCE Load Balancing](https://cloud.google.com/compute/docs/load-balancing/) @@ -394,6 +401,81 @@ The next step is to create a Load Balancer that sits in front of your master nod You will need to ensure that the load balancer routes to **just `master0` on port 6443**. This is because kubeadm will perform health checks using the load balancer IP. Since `master0` is set up individually first, the other masters will not have running apiservers, which will result in kubeadm hanging indefinitely. If possible, use a smart load balancing algorithm like "least connections", and use health checks so unhealthy nodes can be removed from circulation. Most providers will provide these features. +{% endcapture %} + +{% capture onsite %} +In an on-site environment there may not be a physical load balancer available. Instead, keepalived can be used to setup a virtual IP pointing to a healthy master node. The configuration shown here provides an _active/passive_ setup rather than _real_ load balancing, but it can be extended for this purpose quite easily by setting up HAProxy, nginx or similar on the master nodes (not covered here). + +1. Install keepalived, e.g. using your distribution's package manager. The configuration shown here works with version 1.3.5 and supposedly many others. Make sure to have it enabled (chkconfig, systemd, ...) so that it starts automatically when the respective node comes up. + +2. Create the following configuration file _/etc/keepalived/keepalived.conf_ on all master nodes: + + ```shell + ! Configuration File for keepalived + global_defs { + router_id LVS_DEVEL + } + + vrrp_script check_apiserver { + script "/etc/keepalived/check_apiserver.sh" + interval 3 + weight -2 + fall 10 + rise 2 + } + + vrrp_instance VI_1 { + state + interface + virtual_router_id 51 + priority + authentication { + auth_type PASS + auth_pass 4be37dc3b4c90194d1600c483e10ad1d + } + virtual_ipaddress { + + } + track_script { + check_apiserver + } + } + ``` + + In the section `vrrp_instance VI_1`, change few lines depending on your setup: + + * `state` is either `MASTER` (on the first master nodes) or `BACKUP` (the other master nodes). + * `interface` is the name of an existing public interface to bind the virtual IP to (usually the primary interface). + * `priority` should be higher for the first master node, e.g. 101, and lower for the others, e.g. 100. + * `auth_pass` use any random string here. + * `virtual_ipaddresses` should contain the virtual IP for the master nodes. + +3. Install the following health check script to _/etc/keepalived/check_apiserver.sh_ on all master nodes: + + ```shell + #!/bin/sh + + errorExit() { + echo "*** $*" 1>&2 + exit 1 + } + + curl --silent --max-time 2 --insecure https://localhost:6443/ -o /dev/null || errorExit "Error GET https://localhost:6443/" + if ip addr | grep -q ; then + curl --silent --max-time 2 --insecure https://:6443/ -o /dev/null || errorExit "Error GET https://:6443/" + fi + ``` + + Replace the `` by your chosen virtual IP. + +4. Restart keepalived. While no Kubernetes services are up yet it will log health check fails on all master nodes. This will stop as soon as the first master node has been bootstrapped. + +{% endcapture %} + +{% assign tab_names = "Choose one...,Cloud,On-Site" | split: ',' | compact %} +{% assign tab_contents = site.emptyArray | push: choose | push: cloud | push: onsite %} + +{% include tabs.md %} ## Acquire etcd certs From 7c32550bb72ebd8114f456beccb2e883709ab37b Mon Sep 17 00:00:00 2001 From: Martin Dietze Date: Wed, 14 Feb 2018 18:51:18 +0100 Subject: [PATCH 2/4] HA guide for kubeadm: fixed tab navigation for load balancer setup. --- docs/setup/independent/high-availability.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/setup/independent/high-availability.md b/docs/setup/independent/high-availability.md index 6e1b6b1a80..e9ce05c1d1 100644 --- a/docs/setup/independent/high-availability.md +++ b/docs/setup/independent/high-availability.md @@ -378,6 +378,7 @@ Please select one of the tabs to see installation instructions for the respectiv {% endcapture %} +{% assign tab_set_name = "etcd_mode" %} {% assign tab_names = "Choose one...,systemd,Static Pods" | split: ',' | compact %} {% assign tab_contents = site.emptyArray | push: choose | push: systemd | push: static_pods %} @@ -472,6 +473,7 @@ In an on-site environment there may not be a physical load balancer available. I {% endcapture %} +{% assign tab_set_name = "lb_mode" %} {% assign tab_names = "Choose one...,Cloud,On-Site" | split: ',' | compact %} {% assign tab_contents = site.emptyArray | push: choose | push: cloud | push: onsite %} From 747d8037220e444ab226efdac186c81b9d49a9d9 Mon Sep 17 00:00:00 2001 From: Martin Dietze Date: Thu, 22 Feb 2018 11:53:12 +0100 Subject: [PATCH 3/4] HA guide for kubeadm: text change to highlight the fact that keepalived is not the prescribed solution to setting up a virtual IP --- docs/setup/independent/high-availability.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/setup/independent/high-availability.md b/docs/setup/independent/high-availability.md index e9ce05c1d1..c78673783f 100644 --- a/docs/setup/independent/high-availability.md +++ b/docs/setup/independent/high-availability.md @@ -386,7 +386,7 @@ Please select one of the tabs to see installation instructions for the respectiv ## Set up master Load Balancer -The next step is to create a Load Balancer that sits in front of your master nodes. How you do this depends on your environment; you could, for example, leverage a cloud provider Load Balancer, or set up your own using nginx, keepalived, or HAproxy. +The next step is to create a Load Balancer that sits in front of your master nodes. How you do this depends on your environment; you could, for example, leverage a cloud provider Load Balancer, or set up your own using NGINX, keepalived, or HAproxy. {% capture choose %} Please select one of the tabs to see installation instructions for information on load balancing in the respective environment. @@ -405,7 +405,9 @@ If possible, use a smart load balancing algorithm like "least connections", and {% endcapture %} {% capture onsite %} -In an on-site environment there may not be a physical load balancer available. Instead, keepalived can be used to setup a virtual IP pointing to a healthy master node. The configuration shown here provides an _active/passive_ setup rather than _real_ load balancing, but it can be extended for this purpose quite easily by setting up HAProxy, nginx or similar on the master nodes (not covered here). +In an on-site environment there may not be a physical load balancer available. Instead, a virtual IP pointing to a healthy master node can be used. There are a number of solutions for this including keepalived, Pacemaker and probably many others, some with and some without load balancing. + +As an example we outline a simple setup based on keepalived. Depending on environment and requirements people may prefer different solutions. The configuration shown here provides an _active/passive_ failover without load balancing. If required, load balancing can by added quite easily by setting up HAProxy, NGINX or similar on the master nodes (not covered in this guide). 1. Install keepalived, e.g. using your distribution's package manager. The configuration shown here works with version 1.3.5 and supposedly many others. Make sure to have it enabled (chkconfig, systemd, ...) so that it starts automatically when the respective node comes up. From 773def6cdc13c2d5f25d7bfe89d11fbd17622ff8 Mon Sep 17 00:00:00 2001 From: Martin Dietze Date: Thu, 22 Feb 2018 15:46:29 +0100 Subject: [PATCH 4/4] High Availability guide for kubeadm: text change as proposed by @mattkelly. --- docs/setup/independent/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/setup/independent/high-availability.md b/docs/setup/independent/high-availability.md index c78673783f..b75826675c 100644 --- a/docs/setup/independent/high-availability.md +++ b/docs/setup/independent/high-availability.md @@ -409,7 +409,7 @@ In an on-site environment there may not be a physical load balancer available. I As an example we outline a simple setup based on keepalived. Depending on environment and requirements people may prefer different solutions. The configuration shown here provides an _active/passive_ failover without load balancing. If required, load balancing can by added quite easily by setting up HAProxy, NGINX or similar on the master nodes (not covered in this guide). -1. Install keepalived, e.g. using your distribution's package manager. The configuration shown here works with version 1.3.5 and supposedly many others. Make sure to have it enabled (chkconfig, systemd, ...) so that it starts automatically when the respective node comes up. +1. Install keepalived, e.g. using your distribution's package manager. The configuration shown here works with version `1.3.5` but is expected to work with may other versions. Make sure to have it enabled (chkconfig, systemd, ...) so that it starts automatically when the respective node comes up. 2. Create the following configuration file _/etc/keepalived/keepalived.conf_ on all master nodes: