|
|
|
@@ -11,15 +11,15 @@ title: Running ZooKeeper, A CP Distributed System
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
{% capture overview %}
|
|
|
|
|
This tutorial demonstrates [Apache Zookeeper](https://zookeeper.apache.org) on
|
|
|
|
|
Kubernetes using [StatefulSets](/docs/concepts/abstractions/controllers/statefulsets/),
|
|
|
|
|
[PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget),
|
|
|
|
|
This tutorial demonstrates [Apache Zookeeper](https://zookeeper.apache.org) on
|
|
|
|
|
Kubernetes using [StatefulSets](/docs/concepts/workloads/controllers/statefulset/),
|
|
|
|
|
[PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget),
|
|
|
|
|
and [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature).
|
|
|
|
|
{% endcapture %}
|
|
|
|
|
|
|
|
|
|
{% capture prerequisites %}
|
|
|
|
|
|
|
|
|
|
Before starting this tutorial, you should be familiar with the following
|
|
|
|
|
Before starting this tutorial, you should be familiar with the following
|
|
|
|
|
Kubernetes concepts.
|
|
|
|
|
|
|
|
|
|
* [Pods](/docs/user-guide/pods/single-container/)
|
|
|
|
@@ -27,22 +27,22 @@ Kubernetes concepts.
|
|
|
|
|
* [Headless Services](/docs/concepts/services-networking/service/#headless-services)
|
|
|
|
|
* [PersistentVolumes](/docs/concepts/storage/volumes/)
|
|
|
|
|
* [PersistentVolume Provisioning](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/persistent-volume-provisioning/)
|
|
|
|
|
* [StatefulSets](/docs/concepts/abstractions/controllers/statefulsets/)
|
|
|
|
|
* [StatefulSets](/docs/concepts/workloads/controllers/statefulset/)
|
|
|
|
|
* [PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget)
|
|
|
|
|
* [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature)
|
|
|
|
|
* [kubectl CLI](/docs/user-guide/kubectl)
|
|
|
|
|
* [kubectl CLI](/docs/user-guide/kubectl/)
|
|
|
|
|
|
|
|
|
|
You will require a cluster with at least four nodes, and each node will require
|
|
|
|
|
at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and
|
|
|
|
|
drain the cluster's nodes. **This means that all Pods on the cluster's nodes
|
|
|
|
|
will be terminated and evicted, and the nodes will, temporarily, become
|
|
|
|
|
unschedulable.** You should use a dedicated cluster for this tutorial, or you
|
|
|
|
|
should ensure that the disruption you cause will not interfere with other
|
|
|
|
|
at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and
|
|
|
|
|
drain the cluster's nodes. **This means that all Pods on the cluster's nodes
|
|
|
|
|
will be terminated and evicted, and the nodes will, temporarily, become
|
|
|
|
|
unschedulable.** You should use a dedicated cluster for this tutorial, or you
|
|
|
|
|
should ensure that the disruption you cause will not interfere with other
|
|
|
|
|
tenants.
|
|
|
|
|
|
|
|
|
|
This tutorial assumes that your cluster is configured to dynamically provision
|
|
|
|
|
This tutorial assumes that your cluster is configured to dynamically provision
|
|
|
|
|
PersistentVolumes. If your cluster is not configured to do so, you
|
|
|
|
|
will have to manually provision three 20 GiB volumes prior to starting this
|
|
|
|
|
will have to manually provision three 20 GiB volumes prior to starting this
|
|
|
|
|
tutorial.
|
|
|
|
|
{% endcapture %}
|
|
|
|
|
|
|
|
|
@@ -59,51 +59,59 @@ After this tutorial, you will know the following.
|
|
|
|
|
|
|
|
|
|
### ZooKeeper Basics
|
|
|
|
|
|
|
|
|
|
[Apache ZooKeeper](https://zookeeper.apache.org/doc/current/) is a
|
|
|
|
|
[Apache ZooKeeper](https://zookeeper.apache.org/doc/current/) is a
|
|
|
|
|
distributed, open-source coordination service for distributed applications.
|
|
|
|
|
ZooKeeper allows you to read, write, and observe updates to data. Data are
|
|
|
|
|
organized in a file system like hierarchy and replicated to all ZooKeeper
|
|
|
|
|
servers in the ensemble (a set of ZooKeeper servers). All operations on data
|
|
|
|
|
are atomic and sequentially consistent. ZooKeeper ensures this by using the
|
|
|
|
|
[Zab](https://pdfs.semanticscholar.org/b02c/6b00bd5dbdbd951fddb00b906c82fa80f0b3.pdf)
|
|
|
|
|
ZooKeeper allows you to read, write, and observe updates to data. Data are
|
|
|
|
|
organized in a file system like hierarchy and replicated to all ZooKeeper
|
|
|
|
|
servers in the ensemble (a set of ZooKeeper servers). All operations on data
|
|
|
|
|
are atomic and sequentially consistent. ZooKeeper ensures this by using the
|
|
|
|
|
[Zab](https://pdfs.semanticscholar.org/b02c/6b00bd5dbdbd951fddb00b906c82fa80f0b3.pdf)
|
|
|
|
|
consensus protocol to replicate a state machine across all servers in the ensemble.
|
|
|
|
|
|
|
|
|
|
The ensemble uses the Zab protocol to elect a leader, and
|
|
|
|
|
data can not be written until a leader is elected. Once a leader is
|
|
|
|
|
elected, the ensemble uses Zab to ensure that all writes are replicated to a
|
|
|
|
|
data can not be written until a leader is elected. Once a leader is
|
|
|
|
|
elected, the ensemble uses Zab to ensure that all writes are replicated to a
|
|
|
|
|
quorum before they are acknowledged and made visible to clients. Without respect
|
|
|
|
|
to weighted quorums, a quorum is a majority component of the ensemble containing
|
|
|
|
|
the current leader. For instance, if the ensemble has three servers, a component
|
|
|
|
|
that contains the leader and one other server constitutes a quorum. If the
|
|
|
|
|
to weighted quorums, a quorum is a majority component of the ensemble containing
|
|
|
|
|
the current leader. For instance, if the ensemble has three servers, a component
|
|
|
|
|
that contains the leader and one other server constitutes a quorum. If the
|
|
|
|
|
ensemble can not achieve a quorum, data can not be written.
|
|
|
|
|
|
|
|
|
|
ZooKeeper servers keep their entire state machine in memory, but every mutation
|
|
|
|
|
is written to a durable WAL (Write Ahead Log) on storage media. When a server
|
|
|
|
|
crashes, it can recover its previous state by replaying the WAL. In order to
|
|
|
|
|
prevent the WAL from growing without bound, ZooKeeper servers will periodically
|
|
|
|
|
snapshot their in memory state to storage media. These snapshots can be loaded
|
|
|
|
|
directly into memory, and all WAL entries that preceded the snapshot may be
|
|
|
|
|
ZooKeeper servers keep their entire state machine in memory, but every mutation
|
|
|
|
|
is written to a durable WAL (Write Ahead Log) on storage media. When a server
|
|
|
|
|
crashes, it can recover its previous state by replaying the WAL. In order to
|
|
|
|
|
prevent the WAL from growing without bound, ZooKeeper servers will periodically
|
|
|
|
|
snapshot their in memory state to storage media. These snapshots can be loaded
|
|
|
|
|
directly into memory, and all WAL entries that preceded the snapshot may be
|
|
|
|
|
safely discarded.
|
|
|
|
|
|
|
|
|
|
## Creating a ZooKeeper Ensemble
|
|
|
|
|
|
|
|
|
|
The manifest below contains a
|
|
|
|
|
[Headless Service](/docs/concepts/services-networking/service/#headless-services),
|
|
|
|
|
The manifest below contains a
|
|
|
|
|
[Headless Service](/docs/concepts/services-networking/service/#headless-services),
|
|
|
|
|
<<<<<<< HEAD
|
|
|
|
|
a [Service](/docs/concepts/services-networking/service),
|
|
|
|
|
a [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions//#specifying-a-poddisruptionbudget),
|
|
|
|
|
and a [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets/).
|
|
|
|
|
=======
|
|
|
|
|
a [Service](/docs/concepts/services-networking/service/),
|
|
|
|
|
>>>>>>> master
|
|
|
|
|
a [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions//#specifying-a-poddisruptionbudget),
|
|
|
|
|
and a [StatefulSet](/docs/concepts/workloads/controllers/statefulset/).
|
|
|
|
|
|
|
|
|
|
{% include code.html language="yaml" file="zookeeper.yaml" ghlink="/docs/tutorials/stateful-application/zookeeper.yaml" %}
|
|
|
|
|
|
|
|
|
|
Open a command terminal, and use
|
|
|
|
|
[`kubectl apply`](/docs/user-guide/kubectl/{{page.version}}/#apply) to create the
|
|
|
|
|
Open a command terminal, and use
|
|
|
|
|
[`kubectl apply`](/docs/user-guide/kubectl/{{page.version}}/#apply) to create the
|
|
|
|
|
manifest.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
<<<<<<< HEAD
|
|
|
|
|
kubectl apply -f https://k8s.io/docs/tutorials/stateful-application/zookeeper.yaml
|
|
|
|
|
=======
|
|
|
|
|
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/docs/tutorials/stateful-application/zookeeper.yaml
|
|
|
|
|
>>>>>>> master
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This creates the `zk-hs` Headless Service, the `zk-cs` Service,
|
|
|
|
|
This creates the `zk-hs` Headless Service, the `zk-cs` Service,
|
|
|
|
|
the `zk-pdb` PodDisruptionBudget, and the `zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -141,29 +149,29 @@ zk-2 0/1 Running 0 19s
|
|
|
|
|
zk-2 1/1 Running 0 40s
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The StatefulSet controller creates three Pods, and each Pod has a container with
|
|
|
|
|
a [ZooKeeper 3.4.9](http://www-us.apache.org/dist/zookeeper/zookeeper-3.4.9/) server.
|
|
|
|
|
The StatefulSet controller creates three Pods, and each Pod has a container with
|
|
|
|
|
a [ZooKeeper](http://www-us.apache.org/dist/zookeeper/stable/) server.
|
|
|
|
|
|
|
|
|
|
### Facilitating Leader Election
|
|
|
|
|
|
|
|
|
|
As there is no terminating algorithm for electing a leader in an anonymous
|
|
|
|
|
network, Zab requires explicit membership configuration in order to perform
|
|
|
|
|
leader election. Each server in the ensemble needs to have a unique
|
|
|
|
|
As there is no terminating algorithm for electing a leader in an anonymous
|
|
|
|
|
network, Zab requires explicit membership configuration in order to perform
|
|
|
|
|
leader election. Each server in the ensemble needs to have a unique
|
|
|
|
|
identifier, all servers need to know the global set of identifiers, and each
|
|
|
|
|
identifier needs to be associated with a network address.
|
|
|
|
|
|
|
|
|
|
Use [`kubectl exec`](/docs/user-guide/kubectl/{{page.version}}/#exec) to get the hostnames
|
|
|
|
|
Use [`kubectl exec`](/docs/user-guide/kubectl/{{page.version}}/#exec) to get the hostnames
|
|
|
|
|
of the Pods in the `zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
for i in 0 1 2; do kubectl exec zk-$i -- hostname; done
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The StatefulSet controller provides each Pod with a unique hostname based on its
|
|
|
|
|
ordinal index. The hostnames take the form `<statefulset name>-<ordinal index>`.
|
|
|
|
|
As the `replicas` field of the `zk` StatefulSet is set to `3`, the Set's
|
|
|
|
|
controller creates three Pods with their hostnames set to `zk-0`, `zk-1`, and
|
|
|
|
|
`zk-2`.
|
|
|
|
|
The StatefulSet controller provides each Pod with a unique hostname based on its
|
|
|
|
|
ordinal index. The hostnames take the form `<statefulset name>-<ordinal index>`.
|
|
|
|
|
As the `replicas` field of the `zk` StatefulSet is set to `3`, the Set's
|
|
|
|
|
controller creates three Pods with their hostnames set to `zk-0`, `zk-1`, and
|
|
|
|
|
`zk-2`.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
zk-0
|
|
|
|
@@ -171,9 +179,9 @@ zk-1
|
|
|
|
|
zk-2
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The servers in a ZooKeeper ensemble use natural numbers as unique identifiers, and
|
|
|
|
|
each server's identifier is stored in a file called `myid` in the server's
|
|
|
|
|
data directory.
|
|
|
|
|
The servers in a ZooKeeper ensemble use natural numbers as unique identifiers, and
|
|
|
|
|
each server's identifier is stored in a file called `myid` in the server's
|
|
|
|
|
data directory.
|
|
|
|
|
|
|
|
|
|
Examine the contents of the `myid` file for each server.
|
|
|
|
|
|
|
|
|
@@ -181,7 +189,7 @@ Examine the contents of the `myid` file for each server.
|
|
|
|
|
for i in 0 1 2; do echo "myid zk-$i";kubectl exec zk-$i -- cat /var/lib/zookeeper/data/myid; done
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
As the identifiers are natural numbers and the ordinal indices are non-negative
|
|
|
|
|
As the identifiers are natural numbers and the ordinal indices are non-negative
|
|
|
|
|
integers, you can generate an identifier by adding one to the ordinal.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -199,7 +207,7 @@ Get the FQDN (Fully Qualified Domain Name) of each Pod in the `zk` StatefulSet.
|
|
|
|
|
for i in 0 1 2; do kubectl exec zk-$i -- hostname -f; done
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The `zk-hs` Service creates a domain for all of the Pods,
|
|
|
|
|
The `zk-hs` Service creates a domain for all of the Pods,
|
|
|
|
|
`zk-headless.default.svc.cluster.local`.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -208,11 +216,11 @@ zk-1.zk-hs.default.svc.cluster.local
|
|
|
|
|
zk-2.zk-hs.default.svc.cluster.local
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The A records in [Kubernetes DNS](/docs/concepts/services-networking/dns-pod-service/) resolve the FQDNs to the Pods' IP addresses.
|
|
|
|
|
If the Pods are rescheduled, the A records will be updated with the Pods' new IP
|
|
|
|
|
The A records in [Kubernetes DNS](/docs/concepts/services-networking/dns-pod-service/) resolve the FQDNs to the Pods' IP addresses.
|
|
|
|
|
If the Pods are rescheduled, the A records will be updated with the Pods' new IP
|
|
|
|
|
addresses, but the A record's names will not change.
|
|
|
|
|
|
|
|
|
|
ZooKeeper stores its application configuration in a file named `zoo.cfg`. Use
|
|
|
|
|
ZooKeeper stores its application configuration in a file named `zoo.cfg`. Use
|
|
|
|
|
`kubectl exec` to view the contents of the `zoo.cfg` file in the `zk-0` Pod.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
@@ -221,8 +229,8 @@ kubectl exec zk-0 -- cat /opt/zookeeper/conf/zoo.cfg
|
|
|
|
|
|
|
|
|
|
For the `server.1`, `server.2`, and `server.3` properties at the bottom of
|
|
|
|
|
the file, the `1`, `2`, and `3` correspond to the identifiers in the
|
|
|
|
|
ZooKeeper servers' `myid` files. They are set to the FQDNs for the Pods in
|
|
|
|
|
the `zk` StatefulSet.
|
|
|
|
|
ZooKeeper servers' `myid` files. They are set to the FQDNs for the Pods in
|
|
|
|
|
the `zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
clientPort=2181
|
|
|
|
@@ -235,7 +243,7 @@ maxClientCnxns=60
|
|
|
|
|
minSessionTimeout= 4000
|
|
|
|
|
maxSessionTimeout= 40000
|
|
|
|
|
autopurge.snapRetainCount=3
|
|
|
|
|
autopurge.purgeInteval=0
|
|
|
|
|
autopurge.purgeInterval=0
|
|
|
|
|
server.1=zk-0.zk-headless.default.svc.cluster.local:2888:3888
|
|
|
|
|
server.2=zk-1.zk-headless.default.svc.cluster.local:2888:3888
|
|
|
|
|
server.3=zk-2.zk-headless.default.svc.cluster.local:2888:3888
|
|
|
|
@@ -243,10 +251,10 @@ server.3=zk-2.zk-headless.default.svc.cluster.local:2888:3888
|
|
|
|
|
|
|
|
|
|
### Achieving Consensus
|
|
|
|
|
|
|
|
|
|
Consensus protocols require that the identifiers of each participant be
|
|
|
|
|
unique. No two participants in the Zab protocol should claim the same unique
|
|
|
|
|
identifier. This is necessary to allow the processes in the system to agree on
|
|
|
|
|
which processes have committed which data. If two Pods were launched with the
|
|
|
|
|
Consensus protocols require that the identifiers of each participant be
|
|
|
|
|
unique. No two participants in the Zab protocol should claim the same unique
|
|
|
|
|
identifier. This is necessary to allow the processes in the system to agree on
|
|
|
|
|
which processes have committed which data. If two Pods were launched with the
|
|
|
|
|
same ordinal, two ZooKeeper servers would both identify themselves as the same
|
|
|
|
|
server.
|
|
|
|
|
|
|
|
|
@@ -272,7 +280,7 @@ zk-2 1/1 Running 0 40s
|
|
|
|
|
|
|
|
|
|
The A records for each Pod are only entered when the Pod becomes Ready. Therefore,
|
|
|
|
|
the FQDNs of the ZooKeeper servers will only resolve to a single endpoint, and that
|
|
|
|
|
endpoint will be the unique ZooKeeper server claiming the identity configured
|
|
|
|
|
endpoint will be the unique ZooKeeper server claiming the identity configured
|
|
|
|
|
in its `myid` file.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -281,7 +289,7 @@ zk-1.zk-hs.default.svc.cluster.local
|
|
|
|
|
zk-2.zk-hs.default.svc.cluster.local
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This ensures that the `servers` properties in the ZooKeepers' `zoo.cfg` files
|
|
|
|
|
This ensures that the `servers` properties in the ZooKeepers' `zoo.cfg` files
|
|
|
|
|
represents a correctly configured ensemble.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -290,16 +298,16 @@ server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888
|
|
|
|
|
server.3=zk-2.zk-hsdefault.svc.cluster.local:2888:3888
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
When the servers use the Zab protocol to attempt to commit a value, they will
|
|
|
|
|
either achieve consensus and commit the value (if leader election has succeeded
|
|
|
|
|
and at least two of the Pods are Running and Ready), or they will fail to do so
|
|
|
|
|
(if either of the aforementioned conditions are not met). No state will arise
|
|
|
|
|
When the servers use the Zab protocol to attempt to commit a value, they will
|
|
|
|
|
either achieve consensus and commit the value (if leader election has succeeded
|
|
|
|
|
and at least two of the Pods are Running and Ready), or they will fail to do so
|
|
|
|
|
(if either of the aforementioned conditions are not met). No state will arise
|
|
|
|
|
where one server acknowledges a write on behalf of another.
|
|
|
|
|
|
|
|
|
|
### Sanity Testing the Ensemble
|
|
|
|
|
|
|
|
|
|
The most basic sanity test is to write some data to one ZooKeeper server and
|
|
|
|
|
to read the data from another.
|
|
|
|
|
The most basic sanity test is to write some data to one ZooKeeper server and
|
|
|
|
|
to read the data from another.
|
|
|
|
|
|
|
|
|
|
Use the `zkCli.sh` script to write `world` to the path `/hello` on the `zk-0` Pod.
|
|
|
|
|
|
|
|
|
@@ -322,7 +330,7 @@ Get the data from the `zk-1` Pod.
|
|
|
|
|
kubectl exec zk-1 zkCli.sh get /hello
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The data that you created on `zk-0` is available on all of the servers in the
|
|
|
|
|
The data that you created on `zk-0` is available on all of the servers in the
|
|
|
|
|
ensemble.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -346,12 +354,12 @@ numChildren = 0
|
|
|
|
|
### Providing Durable Storage
|
|
|
|
|
|
|
|
|
|
As mentioned in the [ZooKeeper Basics](#zookeeper-basics) section,
|
|
|
|
|
ZooKeeper commits all entries to a durable WAL, and periodically writes snapshots
|
|
|
|
|
in memory state, to storage media. Using WALs to provide durability is a common
|
|
|
|
|
ZooKeeper commits all entries to a durable WAL, and periodically writes snapshots
|
|
|
|
|
in memory state, to storage media. Using WALs to provide durability is a common
|
|
|
|
|
technique for applications that use consensus protocols to achieve a replicated
|
|
|
|
|
state machine and for storage applications in general.
|
|
|
|
|
|
|
|
|
|
Use [`kubectl delete`](/docs/user-guide/kubectl/{{page.version}}/#delete) to delete the
|
|
|
|
|
Use [`kubectl delete`](/docs/user-guide/kubectl/{{page.version}}/#delete) to delete the
|
|
|
|
|
`zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -384,10 +392,10 @@ zk-0 0/1 Terminating 0 11m
|
|
|
|
|
Reapply the manifest in `zookeeper.yaml`.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
kubectl apply -f https://k8s.io/docs/tutorials/stateful-application/zookeeper.yaml
|
|
|
|
|
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/docs/tutorials/stateful-application/zookeeper.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The `zk` StatefulSet will be created, but, as they already exist, the other API
|
|
|
|
|
The `zk` StatefulSet will be created, but, as they already exist, the other API
|
|
|
|
|
Objects in the manifest will not be modified.
|
|
|
|
|
|
|
|
|
|
Watch the StatefulSet controller recreate the StatefulSet's Pods.
|
|
|
|
@@ -417,14 +425,14 @@ zk-2 0/1 Running 0 19s
|
|
|
|
|
zk-2 1/1 Running 0 40s
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Get the value you entered during the [sanity test](#sanity-testing-the-ensemble),
|
|
|
|
|
Get the value you entered during the [sanity test](#sanity-testing-the-ensemble),
|
|
|
|
|
from the `zk-2` Pod.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
kubectl exec zk-2 zkCli.sh get /hello
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Even though all of the Pods in the `zk` StatefulSet have been terminated and
|
|
|
|
|
Even though all of the Pods in the `zk` StatefulSet have been terminated and
|
|
|
|
|
recreated, the ensemble still serves the original value.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -445,8 +453,8 @@ dataLength = 5
|
|
|
|
|
numChildren = 0
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The `volumeClaimTemplates` field, of the `zk` StatefulSet's `spec`, specifies a
|
|
|
|
|
PersistentVolume that will be provisioned for each Pod.
|
|
|
|
|
The `volumeClaimTemplates` field, of the `zk` StatefulSet's `spec`, specifies a
|
|
|
|
|
PersistentVolume that will be provisioned for each Pod.
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
volumeClaimTemplates:
|
|
|
|
@@ -462,8 +470,8 @@ volumeClaimTemplates:
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The StatefulSet controller generates a PersistentVolumeClaim for each Pod in
|
|
|
|
|
the StatefulSet.
|
|
|
|
|
The StatefulSet controller generates a PersistentVolumeClaim for each Pod in
|
|
|
|
|
the StatefulSet.
|
|
|
|
|
|
|
|
|
|
Get the StatefulSet's PersistentVolumeClaims.
|
|
|
|
|
|
|
|
|
@@ -471,7 +479,7 @@ Get the StatefulSet's PersistentVolumeClaims.
|
|
|
|
|
kubectl get pvc -l app=zk
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
When the StatefulSet recreated its Pods, the Pods' PersistentVolumes were
|
|
|
|
|
When the StatefulSet recreated its Pods, the Pods' PersistentVolumes were
|
|
|
|
|
remounted.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -490,23 +498,29 @@ volumeMounts:
|
|
|
|
|
mountPath: /var/lib/zookeeper
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
When a Pod in the `zk` StatefulSet is (re)scheduled, it will always have the
|
|
|
|
|
same PersistentVolume mounted to the ZooKeeper server's data directory.
|
|
|
|
|
Even when the Pods are rescheduled, all of the writes made to the ZooKeeper
|
|
|
|
|
When a Pod in the `zk` StatefulSet is (re)scheduled, it will always have the
|
|
|
|
|
same PersistentVolume mounted to the ZooKeeper server's data directory.
|
|
|
|
|
Even when the Pods are rescheduled, all of the writes made to the ZooKeeper
|
|
|
|
|
servers' WALs, and all of their snapshots, remain durable.
|
|
|
|
|
|
|
|
|
|
## Ensuring Consistent Configuration
|
|
|
|
|
|
|
|
|
|
As noted in the [Facilitating Leader Election](#facilitating-leader-election) and
|
|
|
|
|
[Achieving Consensus](#achieving-consensus) sections, the servers in a
|
|
|
|
|
ZooKeeper ensemble require consistent configuration in order to elect a leader
|
|
|
|
|
[Achieving Consensus](#achieving-consensus) sections, the servers in a
|
|
|
|
|
ZooKeeper ensemble require consistent configuration in order to elect a leader
|
|
|
|
|
and form a quorum. They also require consistent configuration of the Zab protocol
|
|
|
|
|
in order for the protocol to work correctly over a network. In our example we
|
|
|
|
|
achive consistent configuration by embedding the configuration directly into
|
|
|
|
|
in order for the protocol to work correctly over a network. In our example we
|
|
|
|
|
achive consistent configuration by embedding the configuration directly into
|
|
|
|
|
the manifest.
|
|
|
|
|
<<<<<<< HEAD
|
|
|
|
|
|
|
|
|
|
Get the `zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
=======
|
|
|
|
|
|
|
|
|
|
Get the `zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
>>>>>>> master
|
|
|
|
|
```shell{% raw %}
|
|
|
|
|
kubectl get sts zk -o yaml
|
|
|
|
|
...
|
|
|
|
@@ -534,22 +548,26 @@ Get the `zk` StatefulSet.
|
|
|
|
|
...
|
|
|
|
|
```{% endraw %}
|
|
|
|
|
|
|
|
|
|
Notice that the command used to start the ZooKeeper servers passed the configuration
|
|
|
|
|
as command line parameter. Enviornment variables are another, equally good, way to
|
|
|
|
|
Notice that the command used to start the ZooKeeper servers passed the configuration
|
|
|
|
|
<<<<<<< HEAD
|
|
|
|
|
as command line parameter. Enviornment variables are another, equally good, way to
|
|
|
|
|
=======
|
|
|
|
|
as command line parameter. Environment variables are another, equally good, way to
|
|
|
|
|
>>>>>>> master
|
|
|
|
|
pass configuration to ensemble.
|
|
|
|
|
|
|
|
|
|
### Configuring Logging
|
|
|
|
|
|
|
|
|
|
One of the files generated by the `zkGenConfig.sh` script controls ZooKeeper's logging.
|
|
|
|
|
ZooKeeper uses [Log4j](http://logging.apache.org/log4j/2.x/), and, by default,
|
|
|
|
|
it uses a time and size based rolling file appender for its logging configuration.
|
|
|
|
|
One of the files generated by the `zkGenConfig.sh` script controls ZooKeeper's logging.
|
|
|
|
|
ZooKeeper uses [Log4j](http://logging.apache.org/log4j/2.x/), and, by default,
|
|
|
|
|
it uses a time and size based rolling file appender for its logging configuration.
|
|
|
|
|
Get the logging configuration from one of Pods in the `zk` StatefulSet.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
kubectl exec zk-0 cat /usr/etc/zookeeper/log4j.properties
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The logging configuration below will cause the ZooKeeper process to write all
|
|
|
|
|
The logging configuration below will cause the ZooKeeper process to write all
|
|
|
|
|
of its logs to the standard output file stream.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -562,20 +580,20 @@ log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
|
|
|
|
|
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This is the simplest possible way to safely log inside the container. As the
|
|
|
|
|
application's logs are being written to standard out, Kubernetes will handle
|
|
|
|
|
log rotation for you. Kubernetes also implements a sane retention policy that
|
|
|
|
|
ensures application logs written to standard out and standard error do not
|
|
|
|
|
This is the simplest possible way to safely log inside the container. As the
|
|
|
|
|
application's logs are being written to standard out, Kubernetes will handle
|
|
|
|
|
log rotation for you. Kubernetes also implements a sane retention policy that
|
|
|
|
|
ensures application logs written to standard out and standard error do not
|
|
|
|
|
exhaust local storage media.
|
|
|
|
|
|
|
|
|
|
Use [`kubectl logs`](/docs/user-guide/kubectl/{{page.version}}/#logs) to retrieve the last
|
|
|
|
|
Use [`kubectl logs`](/docs/user-guide/kubectl/{{page.version}}/#logs) to retrieve the last
|
|
|
|
|
few log lines from one of the Pods.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
kubectl logs zk-0 --tail 20
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Application logs that are written to standard out or standard error are viewable
|
|
|
|
|
Application logs that are written to standard out or standard error are viewable
|
|
|
|
|
using `kubectl logs` and from the Kubernetes Dashboard.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -601,19 +619,19 @@ using `kubectl logs` and from the Kubernetes Dashboard.
|
|
|
|
|
2016-12-06 19:34:46,230 [myid:1] - INFO [Thread-1142:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52768 (no session established for client)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Kubernetes also supports more powerful, but more complex, logging integrations
|
|
|
|
|
with [Logging Using Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/)
|
|
|
|
|
Kubernetes also supports more powerful, but more complex, logging integrations
|
|
|
|
|
with [Logging Using Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/)
|
|
|
|
|
and [Logging Using Elasticsearch and Kibana](/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/).
|
|
|
|
|
For cluster level log shipping and aggregation, you should consider deploying a
|
|
|
|
|
[sidecar](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html)
|
|
|
|
|
[sidecar](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html)
|
|
|
|
|
container to rotate and ship your logs.
|
|
|
|
|
|
|
|
|
|
### Configuring a Non-Privileged User
|
|
|
|
|
|
|
|
|
|
The best practices with respect to allowing an application to run as a privileged
|
|
|
|
|
user inside of a container are a matter of debate. If your organization requires
|
|
|
|
|
that applications be run as a non-privileged user you can use a
|
|
|
|
|
[SecurityContext](/docs/tasks/configure-pod-container/security-context/) to control the user that
|
|
|
|
|
The best practices with respect to allowing an application to run as a privileged
|
|
|
|
|
user inside of a container are a matter of debate. If your organization requires
|
|
|
|
|
that applications be run as a non-privileged user you can use a
|
|
|
|
|
[SecurityContext](/docs/tasks/configure-pod-container/security-context/) to control the user that
|
|
|
|
|
the entry point runs as.
|
|
|
|
|
|
|
|
|
|
The `zk` StatefulSet's Pod `template` contains a SecurityContext.
|
|
|
|
@@ -624,7 +642,7 @@ securityContext:
|
|
|
|
|
fsGroup: 1000
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In the Pods' containers, UID 1000 corresponds to the zookeeper user and GID 1000
|
|
|
|
|
In the Pods' containers, UID 1000 corresponds to the zookeeper user and GID 1000
|
|
|
|
|
corresponds to the zookeeper group.
|
|
|
|
|
|
|
|
|
|
Get the ZooKeeper process information from the `zk-0` Pod.
|
|
|
|
@@ -633,7 +651,7 @@ Get the ZooKeeper process information from the `zk-0` Pod.
|
|
|
|
|
kubectl exec zk-0 -- ps -elf
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
As the `runAsUser` field of the `securityContext` object is set to 1000,
|
|
|
|
|
As the `runAsUser` field of the `securityContext` object is set to 1000,
|
|
|
|
|
instead of running as root, the ZooKeeper process runs as the zookeeper user.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -642,8 +660,8 @@ F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
|
|
|
|
|
0 S zookeep+ 27 1 0 80 0 - 1155556 - 20:46 ? 00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
By default, when the Pod's PersistentVolume is mounted to the ZooKeeper server's
|
|
|
|
|
data directory, it is only accessible by the root user. This configuration
|
|
|
|
|
By default, when the Pod's PersistentVolume is mounted to the ZooKeeper server's
|
|
|
|
|
data directory, it is only accessible by the root user. This configuration
|
|
|
|
|
prevents the ZooKeeper process from writing to its WAL and storing its snapshots.
|
|
|
|
|
|
|
|
|
|
Get the file permissions of the ZooKeeper data directory on the `zk-0` Pod.
|
|
|
|
@@ -652,8 +670,8 @@ Get the file permissions of the ZooKeeper data directory on the `zk-0` Pod.
|
|
|
|
|
kubectl exec -ti zk-0 -- ls -ld /var/lib/zookeeper/data
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
As the `fsGroup` field of the `securityContext` object is set to 1000,
|
|
|
|
|
the ownership of the Pods' PersistentVolumes is set to the zookeeper group,
|
|
|
|
|
As the `fsGroup` field of the `securityContext` object is set to 1000,
|
|
|
|
|
the ownership of the Pods' PersistentVolumes is set to the zookeeper group,
|
|
|
|
|
and the ZooKeeper process is able to successfully read and write its data.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -662,12 +680,12 @@ drwxr-sr-x 3 zookeeper zookeeper 4096 Dec 5 20:45 /var/lib/zookeeper/data
|
|
|
|
|
|
|
|
|
|
## Managing the ZooKeeper Process
|
|
|
|
|
|
|
|
|
|
The [ZooKeeper documentation](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_supervision)
|
|
|
|
|
indicates that "You will want to have a supervisory process that
|
|
|
|
|
manages each of your ZooKeeper server processes (JVM)." Utilizing a watchdog
|
|
|
|
|
(supervisory process) to restart failed processes in a distributed system is a
|
|
|
|
|
common pattern. When deploying an application in Kubernetes, rather than using
|
|
|
|
|
an external utility as a supervisory process, you should use Kubernetes as the
|
|
|
|
|
The [ZooKeeper documentation](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_supervision)
|
|
|
|
|
indicates that "You will want to have a supervisory process that
|
|
|
|
|
manages each of your ZooKeeper server processes (JVM)." Utilizing a watchdog
|
|
|
|
|
(supervisory process) to restart failed processes in a distributed system is a
|
|
|
|
|
common pattern. When deploying an application in Kubernetes, rather than using
|
|
|
|
|
an external utility as a supervisory process, you should use Kubernetes as the
|
|
|
|
|
watchdog for your application.
|
|
|
|
|
|
|
|
|
|
### Updating the Ensemble
|
|
|
|
@@ -698,7 +716,7 @@ Waiting for 1 pods to be ready...
|
|
|
|
|
statefulset rolling update complete 3 pods at revision zk-5db4499664...
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The Pods are terminated, one at a time, in reverse ordinal order, and they
|
|
|
|
|
The Pods are terminated, one at a time, in reverse ordinal order, and they
|
|
|
|
|
are recreated with the new configuration. This ensures that quorum is maintained
|
|
|
|
|
during a rolling update.
|
|
|
|
|
|
|
|
|
@@ -718,13 +736,13 @@ kubectl rollout undo sts/zk
|
|
|
|
|
statefulset "zk" rolled back
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Handling Process Failure
|
|
|
|
|
### Handling Process Failure
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[Restart Policies](/docs/user-guide/pod-states/#restartpolicy) control how
|
|
|
|
|
[Restart Policies](/docs/user-guide/pod-states/#restartpolicy) control how
|
|
|
|
|
Kubernetes handles process failures for the entry point of the container in a Pod.
|
|
|
|
|
For Pods in a StatefulSet, the only appropriate RestartPolicy is Always, and this
|
|
|
|
|
is the default value. For stateful applications you should **never** override
|
|
|
|
|
is the default value. For stateful applications you should **never** override
|
|
|
|
|
the default policy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -734,7 +752,7 @@ Examine the process tree for the ZooKeeper server running in the `zk-0` Pod.
|
|
|
|
|
kubectl exec zk-0 -- ps -ef
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The command used as the container's entry point has PID 1, and
|
|
|
|
|
The command used as the container's entry point has PID 1, and
|
|
|
|
|
the ZooKeeper process, a child of the entry point, has PID 23.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -759,8 +777,8 @@ In another terminal, kill the ZooKeeper process in Pod `zk-0`.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The death of the ZooKeeper process caused its parent process to terminate. As
|
|
|
|
|
the RestartPolicy of the container is Always, the parent process was relaunched.
|
|
|
|
|
The death of the ZooKeeper process caused its parent process to terminate. As
|
|
|
|
|
the RestartPolicy of the container is Always, the parent process was relaunched.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -775,19 +793,19 @@ zk-0 1/1 Running 1 29m
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If your application uses a script (such as zkServer.sh) to launch the process
|
|
|
|
|
If your application uses a script (such as zkServer.sh) to launch the process
|
|
|
|
|
that implements the application's business logic, the script must terminate with the
|
|
|
|
|
child process. This ensures that Kubernetes will restart the application's
|
|
|
|
|
container when the process implementing the application's business logic fails.
|
|
|
|
|
container when the process implementing the application's business logic fails.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Testing for Liveness
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Configuring your application to restart failed processes is not sufficient to
|
|
|
|
|
keep a distributed system healthy. There are many scenarios where
|
|
|
|
|
a system's processes can be both alive and unresponsive, or otherwise
|
|
|
|
|
unhealthy. You should use liveness probes in order to notify Kubernetes
|
|
|
|
|
Configuring your application to restart failed processes is not sufficient to
|
|
|
|
|
keep a distributed system healthy. There are many scenarios where
|
|
|
|
|
a system's processes can be both alive and unresponsive, or otherwise
|
|
|
|
|
unhealthy. You should use liveness probes in order to notify Kubernetes
|
|
|
|
|
that your application's processes are unhealthy and should be restarted.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -804,7 +822,7 @@ The Pod `template` for the `zk` StatefulSet specifies a liveness probe.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The probe calls a simple bash script that uses the ZooKeeper `ruok` four letter
|
|
|
|
|
The probe calls a simple bash script that uses the ZooKeeper `ruok` four letter
|
|
|
|
|
word to test the server's health.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -835,7 +853,7 @@ kubectl exec zk-0 -- rm /opt/zookeeper/bin/zkOk.sh
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When the liveness probe for the ZooKeeper process fails, Kubernetes will
|
|
|
|
|
When the liveness probe for the ZooKeeper process fails, Kubernetes will
|
|
|
|
|
automatically restart the process for you, ensuring that unhealthy processes in
|
|
|
|
|
the ensemble are restarted.
|
|
|
|
|
|
|
|
|
@@ -856,10 +874,10 @@ zk-0 1/1 Running 1 1h
|
|
|
|
|
### Testing for Readiness
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Readiness is not the same as liveness. If a process is alive, it is scheduled
|
|
|
|
|
and healthy. If a process is ready, it is able to process input. Liveness is
|
|
|
|
|
Readiness is not the same as liveness. If a process is alive, it is scheduled
|
|
|
|
|
and healthy. If a process is ready, it is able to process input. Liveness is
|
|
|
|
|
a necessary, but not sufficient, condition for readiness. There are many cases,
|
|
|
|
|
particularly during initialization and termination, when a process can be
|
|
|
|
|
particularly during initialization and termination, when a process can be
|
|
|
|
|
alive but not ready.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -867,8 +885,8 @@ If you specify a readiness probe, Kubernetes will ensure that your application's
|
|
|
|
|
processes will not receive network traffic until their readiness checks pass.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For a ZooKeeper server, liveness implies readiness. Therefore, the readiness
|
|
|
|
|
probe from the `zookeeper.yaml` manifest is identical to the liveness probe.
|
|
|
|
|
For a ZooKeeper server, liveness implies readiness. Therefore, the readiness
|
|
|
|
|
probe from the `zookeeper.yaml` manifest is identical to the liveness probe.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
@@ -881,28 +899,28 @@ probe from the `zookeeper.yaml` manifest is identical to the liveness probe.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Even though the liveness and readiness probes are identical, it is important
|
|
|
|
|
to specify both. This ensures that only healthy servers in the ZooKeeper
|
|
|
|
|
Even though the liveness and readiness probes are identical, it is important
|
|
|
|
|
to specify both. This ensures that only healthy servers in the ZooKeeper
|
|
|
|
|
ensemble receive network traffic.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Tolerating Node Failure
|
|
|
|
|
|
|
|
|
|
ZooKeeper needs a quorum of servers in order to successfully commit mutations
|
|
|
|
|
to data. For a three server ensemble, two servers must be healthy in order for
|
|
|
|
|
writes to succeed. In quorum based systems, members are deployed across failure
|
|
|
|
|
domains to ensure availability. In order to avoid an outage, due to the loss of an
|
|
|
|
|
individual machine, best practices preclude co-locating multiple instances of the
|
|
|
|
|
ZooKeeper needs a quorum of servers in order to successfully commit mutations
|
|
|
|
|
to data. For a three server ensemble, two servers must be healthy in order for
|
|
|
|
|
writes to succeed. In quorum based systems, members are deployed across failure
|
|
|
|
|
domains to ensure availability. In order to avoid an outage, due to the loss of an
|
|
|
|
|
individual machine, best practices preclude co-locating multiple instances of the
|
|
|
|
|
application on the same machine.
|
|
|
|
|
|
|
|
|
|
By default, Kubernetes may co-locate Pods in a StatefulSet on the same node.
|
|
|
|
|
By default, Kubernetes may co-locate Pods in a StatefulSet on the same node.
|
|
|
|
|
For the three server ensemble you created, if two servers reside on the same
|
|
|
|
|
node, and that node fails, the clients of your ZooKeeper service will experience
|
|
|
|
|
an outage until at least one of the Pods can be rescheduled.
|
|
|
|
|
an outage until at least one of the Pods can be rescheduled.
|
|
|
|
|
|
|
|
|
|
You should always provision additional capacity to allow the processes of critical
|
|
|
|
|
systems to be rescheduled in the event of node failures. If you do so, then the
|
|
|
|
|
outage will only last until the Kubernetes scheduler reschedules one of the ZooKeeper
|
|
|
|
|
systems to be rescheduled in the event of node failures. If you do so, then the
|
|
|
|
|
outage will only last until the Kubernetes scheduler reschedules one of the ZooKeeper
|
|
|
|
|
servers. However, if you want your service to tolerate node failures with no downtime,
|
|
|
|
|
you should set `podAntiAffinity`.
|
|
|
|
|
|
|
|
|
@@ -930,16 +948,16 @@ This is because the Pods in the `zk` StatefulSet have a PodAntiAffinity specifie
|
|
|
|
|
matchExpressions:
|
|
|
|
|
- key: "app"
|
|
|
|
|
operator: In
|
|
|
|
|
values:
|
|
|
|
|
values:
|
|
|
|
|
- zk-headless
|
|
|
|
|
topologyKey: "kubernetes.io/hostname"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The `requiredDuringSchedulingIgnoredDuringExecution` field tells the
|
|
|
|
|
The `requiredDuringSchedulingIgnoredDuringExecution` field tells the
|
|
|
|
|
Kubernetes Scheduler that it should never co-locate two Pods from the `zk-headless`
|
|
|
|
|
Service in the domain defined by the `topologyKey`. The `topologyKey`
|
|
|
|
|
`kubernetes.io/hostname` indicates that the domain is an individual node. Using
|
|
|
|
|
different rules, labels, and selectors, you can extend this technique to spread
|
|
|
|
|
`kubernetes.io/hostname` indicates that the domain is an individual node. Using
|
|
|
|
|
different rules, labels, and selectors, you can extend this technique to spread
|
|
|
|
|
your ensemble across physical, network, and power failure domains.
|
|
|
|
|
|
|
|
|
|
## Surviving Maintenance
|
|
|
|
@@ -947,8 +965,8 @@ your ensemble across physical, network, and power failure domains.
|
|
|
|
|
**In this section you will cordon and drain nodes. If you are using this tutorial
|
|
|
|
|
on a shared cluster, be sure that this will not adversely affect other tenants.**
|
|
|
|
|
|
|
|
|
|
The previous section showed you how to spread your Pods across nodes to survive
|
|
|
|
|
unplanned node failures, but you also need to plan for temporary node failures
|
|
|
|
|
The previous section showed you how to spread your Pods across nodes to survive
|
|
|
|
|
unplanned node failures, but you also need to plan for temporary node failures
|
|
|
|
|
that occur due to planned maintenance.
|
|
|
|
|
|
|
|
|
|
Get the nodes in your cluster.
|
|
|
|
@@ -957,7 +975,7 @@ Get the nodes in your cluster.
|
|
|
|
|
kubectl get nodes
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use [`kubectl cordon`](/docs/user-guide/kubectl/{{page.version}}/#cordon) to
|
|
|
|
|
Use [`kubectl cordon`](/docs/user-guide/kubectl/{{page.version}}/#cordon) to
|
|
|
|
|
cordon all but four of the nodes in your cluster.
|
|
|
|
|
|
|
|
|
|
```shell{% raw %}
|
|
|
|
@@ -970,8 +988,8 @@ Get the `zk-pdb` PodDisruptionBudget.
|
|
|
|
|
kubectl get pdb zk-pdb
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The `max-unavailable` field indicates to Kubernetes that at most one Pod from
|
|
|
|
|
`zk` StatefulSet can be unavailable at any time.
|
|
|
|
|
The `max-unavailable` field indicates to Kubernetes that at most one Pod from
|
|
|
|
|
`zk` StatefulSet can be unavailable at any time.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
NAME MIN-AVAILABLE MAX-UNAVAILABLE ALLOWED-DISRUPTIONS AGE
|
|
|
|
@@ -994,7 +1012,7 @@ kubernetes-minion-group-i4c4
|
|
|
|
|
{% endraw %}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use [`kubectl drain`](/docs/user-guide/kubectl/{{page.version}}/#drain) to cordon and
|
|
|
|
|
Use [`kubectl drain`](/docs/user-guide/kubectl/{{page.version}}/#drain) to cordon and
|
|
|
|
|
drain the node on which the `zk-0` Pod is scheduled.
|
|
|
|
|
|
|
|
|
|
```shell {% raw %}
|
|
|
|
@@ -1005,7 +1023,7 @@ pod "zk-0" deleted
|
|
|
|
|
node "kubernetes-minion-group-pb41" drained
|
|
|
|
|
{% endraw %}```
|
|
|
|
|
|
|
|
|
|
As there are four nodes in your cluster, `kubectl drain`, succeeds and the
|
|
|
|
|
As there are four nodes in your cluster, `kubectl drain`, succeeds and the
|
|
|
|
|
`zk-0` is rescheduled to another node.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
@@ -1025,7 +1043,7 @@ zk-0 0/1 Running 0 51s
|
|
|
|
|
zk-0 1/1 Running 0 1m
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Keep watching the StatefulSet's Pods in the first terminal and drain the node on which
|
|
|
|
|
Keep watching the StatefulSet's Pods in the first terminal and drain the node on which
|
|
|
|
|
`zk-1` is scheduled.
|
|
|
|
|
|
|
|
|
|
```shell{% raw %}
|
|
|
|
@@ -1035,8 +1053,8 @@ pod "zk-1" deleted
|
|
|
|
|
node "kubernetes-minion-group-ixsl" drained
|
|
|
|
|
{% endraw %}```
|
|
|
|
|
|
|
|
|
|
The `zk-1` Pod can not be scheduled. As the `zk` StatefulSet contains a
|
|
|
|
|
PodAntiAffinity rule preventing co-location of the Pods, and as only
|
|
|
|
|
The `zk-1` Pod can not be scheduled. As the `zk` StatefulSet contains a
|
|
|
|
|
PodAntiAffinity rule preventing co-location of the Pods, and as only
|
|
|
|
|
two nodes are schedulable, the Pod will remain in a Pending state.
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
@@ -1063,7 +1081,7 @@ zk-1 0/1 Pending 0 0s
|
|
|
|
|
zk-1 0/1 Pending 0 0s
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Continue to watch the Pods of the stateful set, and drain the node on which
|
|
|
|
|
Continue to watch the Pods of the stateful set, and drain the node on which
|
|
|
|
|
`zk-2` is scheduled.
|
|
|
|
|
|
|
|
|
|
```shell{% raw %}
|
|
|
|
@@ -1075,9 +1093,9 @@ There are pending pods when an error occurred: Cannot evict pod as it would viol
|
|
|
|
|
pod/zk-2
|
|
|
|
|
{% endraw %}```
|
|
|
|
|
|
|
|
|
|
Use `CRTL-C` to terminate to kubectl.
|
|
|
|
|
Use `CRTL-C` to terminate to kubectl.
|
|
|
|
|
|
|
|
|
|
You can not drain the third node because evicting `zk-2` would violate `zk-budget`. However,
|
|
|
|
|
You can not drain the third node because evicting `zk-2` would violate `zk-budget`. However,
|
|
|
|
|
the node will remain cordoned.
|
|
|
|
|
|
|
|
|
|
Use `zkCli.sh` to retrieve the value you entered during the sanity test from `zk-0`.
|
|
|
|
@@ -1162,9 +1180,9 @@ node "kubernetes-minion-group-ixsl" uncordoned
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
You can use `kubectl drain` in conjunction with PodDisruptionBudgets to ensure that your service
|
|
|
|
|
remains available during maintenance. If drain is used to cordon nodes and evict pods prior to
|
|
|
|
|
taking the node offline for maintenance, services that express a disruption budget will have that
|
|
|
|
|
budget respected. You should always allocate additional capacity for critical services so that
|
|
|
|
|
remains available during maintenance. If drain is used to cordon nodes and evict pods prior to
|
|
|
|
|
taking the node offline for maintenance, services that express a disruption budget will have that
|
|
|
|
|
budget respected. You should always allocate additional capacity for critical services so that
|
|
|
|
|
their Pods can be immediately rescheduled.
|
|
|
|
|
|
|
|
|
|
{% endcapture %}
|
|
|
|
@@ -1172,8 +1190,8 @@ their Pods can be immediately rescheduled.
|
|
|
|
|
{% capture cleanup %}
|
|
|
|
|
* Use `kubectl uncordon` to uncordon all the nodes in your cluster.
|
|
|
|
|
* You will need to delete the persistent storage media for the PersistentVolumes
|
|
|
|
|
used in this tutorial. Follow the necessary steps, based on your environment,
|
|
|
|
|
storage configuration, and provisioning method, to ensure that all storage is
|
|
|
|
|
used in this tutorial. Follow the necessary steps, based on your environment,
|
|
|
|
|
storage configuration, and provisioning method, to ensure that all storage is
|
|
|
|
|
reclaimed.
|
|
|
|
|
{% endcapture %}
|
|
|
|
|
{% include templates/tutorial.md %}
|
|
|
|
|
zacharysarah