whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update zookeeper.md (#6565)

Browse Source 
* Update zookeeper.md

As part of a docs sprint in Melbourne I made a very initial pass at trying to tidy this up a little, I would like to have done much much more, but it's a start…

* Updates from feedback

* Format markdown

* FInal tidy
This commit is contained in:
Chris Chinchilla
2018-04-26 23:46:27 +02:00
committed by k8s-ci-robot
parent 89376986cb
commit 691edcfe8d
1 changed files with 253 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tutorials/stateful-application/zookeeper.md
+184 -270
View File
@@ -7,11 +7,11 @@ reviewers:
- janetkuo - janetkuo
- kow3ns - kow3ns
- smarterclayton - smarterclayton
title: Running ZooKeeper, A CP Distributed System title: Running ZooKeeper, A Distributed System Coordinator
--- ---
{% capture overview %} {% capture overview %}
This tutorial demonstrates [Apache Zookeeper](https://zookeeper.apache.org) on This tutorial demonstrates running [Apache Zookeeper](https://zookeeper.apache.org) on
Kubernetes using [StatefulSets](/docs/concepts/workloads/controllers/statefulset/), Kubernetes using [StatefulSets](/docs/concepts/workloads/controllers/statefulset/),
[PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget), [PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget),
and [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature). and [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature).
@@ -22,38 +22,32 @@ and [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-an
Before starting this tutorial, you should be familiar with the following Before starting this tutorial, you should be familiar with the following
Kubernetes concepts. Kubernetes concepts.
* [Pods](/docs/user-guide/pods/single-container/) - [Pods](/docs/user-guide/pods/single-container/)
* [Cluster DNS](/docs/concepts/services-networking/dns-pod-service/) - [Cluster DNS](/docs/concepts/services-networking/dns-pod-service/)
* [Headless Services](/docs/concepts/services-networking/service/#headless-services) - [Headless Services](/docs/concepts/services-networking/service/#headless-services)
* [PersistentVolumes](/docs/concepts/storage/volumes/) - [PersistentVolumes](/docs/concepts/storage/volumes/)
* [PersistentVolume Provisioning](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/persistent-volume-provisioning/) - [PersistentVolume Provisioning](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/staging/persistent-volume-provisioning/)
* [StatefulSets](/docs/concepts/workloads/controllers/statefulset/) - [StatefulSets](/docs/concepts/workloads/controllers/statefulset/)
* [PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget) - [PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#specifying-a-poddisruptionbudget)
* [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature) - [PodAntiAffinity](/docs/user-guide/node-selection/#inter-pod-affinity-and-anti-affinity-beta-feature)
* [kubectl CLI](/docs/user-guide/kubectl/) - [kubectl CLI](/docs/user-guide/kubectl/)
You will require a cluster with at least four nodes, and each node will require You will require a cluster with at least four nodes, and each node requires at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and drain the cluster's nodes. **This means that the cluster will terminate and evict all Pods on its nodes, and the nodes will temporarily become unschedulable.** You should use a dedicated cluster for this tutorial, or you should ensure that the disruption you cause will not interfere with other tenants.
at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and
drain the cluster's nodes. **This means that all Pods on the cluster's nodes
will be terminated and evicted, and the nodes will, temporarily, become
unschedulable.** You should use a dedicated cluster for this tutorial, or you
should ensure that the disruption you cause will not interfere with other
tenants.
This tutorial assumes that your cluster is configured to dynamically provision This tutorial assumes that you have configured your cluster to dynamically provision
PersistentVolumes. If your cluster is not configured to do so, you PersistentVolumes. If your cluster is not configured to do so, you
will have to manually provision three 20 GiB volumes prior to starting this will have to manually provision three 20 GiB volumes before starting this
tutorial. tutorial.
{% endcapture %} {% endcapture %}
{% capture objectives %} {% capture objectives %}
After this tutorial, you will know the following. After this tutorial, you will know the following.
* How to deploy a ZooKeeper ensemble using StatefulSet. - How to deploy a ZooKeeper ensemble using StatefulSet.
* How to consistently configure the ensemble using ConfigMaps. - How to consistently configure the ensemble using ConfigMaps.
* How to spread the deployment of ZooKeeper servers in the ensemble. - How to spread the deployment of ZooKeeper servers in the ensemble.
* How to use PodDisruptionBudgets to ensure service availability during planned maintenance. - How to use PodDisruptionBudgets to ensure service availability during planned maintenance.
{% endcapture %} {% endcapture %}
{% capture lessoncontent %} {% capture lessoncontent %}
@@ -68,22 +62,9 @@ are atomic and sequentially consistent. ZooKeeper ensures this by using the
[Zab](https://pdfs.semanticscholar.org/b02c/6b00bd5dbdbd951fddb00b906c82fa80f0b3.pdf) [Zab](https://pdfs.semanticscholar.org/b02c/6b00bd5dbdbd951fddb00b906c82fa80f0b3.pdf)
consensus protocol to replicate a state machine across all servers in the ensemble. consensus protocol to replicate a state machine across all servers in the ensemble.
The ensemble uses the Zab protocol to elect a leader, and The ensemble uses the Zab protocol to elect a leader, and the ensemble cannot write data until that election is complete. Once complete, the ensemble uses Zab to ensure that it replicates all writes to a quorum before it acknowledges and makes them visible to clients. Without respect to weighted quorums, a quorum is a majority component of the ensemble containing the current leader. For instance, if the ensemble has three servers, a component that contains the leader and one other server constitutes a quorum. If the ensemble can not achieve a quorum, the ensemble cannot write data.
data can not be written until a leader is elected. Once a leader is
elected, the ensemble uses Zab to ensure that all writes are replicated to a
quorum before they are acknowledged and made visible to clients. Without respect
to weighted quorums, a quorum is a majority component of the ensemble containing
the current leader. For instance, if the ensemble has three servers, a component
that contains the leader and one other server constitutes a quorum. If the
ensemble can not achieve a quorum, data can not be written.
ZooKeeper servers keep their entire state machine in memory, but every mutation ZooKeeper servers keep their entire state machine in memory, and write every mutation to a durable WAL (Write Ahead Log) on storage media. When a server crashes, it can recover its previous state by replaying the WAL. To prevent the WAL from growing without bound, ZooKeeper servers will periodically snapshot their in memory state to storage media. These snapshots can be loaded directly into memory, and all WAL entries that preceded the snapshot may be discarded.
is written to a durable WAL (Write Ahead Log) on storage media. When a server
crashes, it can recover its previous state by replaying the WAL. In order to
prevent the WAL from growing without bound, ZooKeeper servers will periodically
snapshot their in memory state to storage media. These snapshots can be loaded
directly into memory, and all WAL entries that preceded the snapshot may be
safely discarded.
## Creating a ZooKeeper Ensemble ## Creating a ZooKeeper Ensemble
@@ -95,12 +76,12 @@ and a [StatefulSet](/docs/concepts/workloads/controllers/statefulset/).
{% include code.html language="yaml" file="zookeeper.yaml" ghlink="/docs/tutorials/stateful-application/zookeeper.yaml" %} {% include code.html language="yaml" file="zookeeper.yaml" ghlink="/docs/tutorials/stateful-application/zookeeper.yaml" %}
Open a command terminal, and use Open a terminal, and use the
[`kubectl apply`](/docs/user-guide/kubectl/{{page.version}}/#apply) to create the [`kubectl apply`](/docs/user-guide/kubectl/{{page.version}}/#apply) command to create the
manifest. manifest.
```shell ```shell
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/docs/tutorials/stateful-application/zookeeper.yaml kubectl apply -f https://k8s.io/docs/tutorials/stateful-application/zookeeper.yaml
``` ```
This creates the `zk-hs` Headless Service, the `zk-cs` Service, This creates the `zk-hs` Headless Service, the `zk-cs` Service,
@@ -113,7 +94,7 @@ poddisruptionbudget "zk-pdb" created
statefulset "zk" created statefulset "zk" created
``` ```
Use [`kubectl get`](/docs/user-guide/kubectl/{{page.version}}/#get) to watch the Use [`kubectl get`](/docs/user-guide/kubectl/{{page.version}}/#get) to watch the
StatefulSet controller create the StatefulSet's Pods. StatefulSet controller create the StatefulSet's Pods.
```shell ```shell
@@ -146,11 +127,7 @@ a [ZooKeeper](http://www-us.apache.org/dist/zookeeper/stable/) server.
### Facilitating Leader Election ### Facilitating Leader Election
As there is no terminating algorithm for electing a leader in an anonymous Because there is no terminating algorithm for electing a leader in an anonymous network, Zab requires explicit membership configuration to perform leader election. Each server in the ensemble needs to have a unique identifier, all servers need to know the global set of identifiers, and each identifier needs to be associated with a network address.
network, Zab requires explicit membership configuration in order to perform
leader election. Each server in the ensemble needs to have a unique
identifier, all servers need to know the global set of identifiers, and each
identifier needs to be associated with a network address.
Use [`kubectl exec`](/docs/user-guide/kubectl/{{page.version}}/#exec) to get the hostnames Use [`kubectl exec`](/docs/user-guide/kubectl/{{page.version}}/#exec) to get the hostnames
of the Pods in the `zk` StatefulSet. of the Pods in the `zk` StatefulSet.
@@ -159,10 +136,7 @@ of the Pods in the `zk` StatefulSet.
for i in 0 1 2; do kubectl exec zk-$i -- hostname; done for i in 0 1 2; do kubectl exec zk-$i -- hostname; done
``` ```
The StatefulSet controller provides each Pod with a unique hostname based on its The StatefulSet controller provides each Pod with a unique hostname based on its ordinal index. The hostnames take the form of `<statefulset name>-<ordinal index>`. Because the `replicas` field of the `zk` StatefulSet is set to `3`, the Set's controller creates three Pods with their hostnames set to `zk-0`, `zk-1`, and
ordinal index. The hostnames take the form `<statefulset name>-<ordinal index>`.
As the `replicas` field of the `zk` StatefulSet is set to `3`, the Set's
controller creates three Pods with their hostnames set to `zk-0`, `zk-1`, and
`zk-2`. `zk-2`.
```shell ```shell
@@ -171,18 +145,15 @@ zk-1
zk-2 zk-2
``` ```
The servers in a ZooKeeper ensemble use natural numbers as unique identifiers, and The servers in a ZooKeeper ensemble use natural numbers as unique identifiers, and store each server's identifier in a file called `myid` in the server's data directory.
each server's identifier is stored in a file called `myid` in the server's
data directory.
Examine the contents of the `myid` file for each server. To examine the contents of the `myid` file for each server use the following command.
```shell ```shell
for i in 0 1 2; do echo "myid zk-$i";kubectl exec zk-$i -- cat /var/lib/zookeeper/data/myid; done for i in 0 1 2; do echo "myid zk-$i";kubectl exec zk-$i -- cat /var/lib/zookeeper/data/myid; done
``` ```
As the identifiers are natural numbers and the ordinal indices are non-negative Because the identifiers are natural numbers and the ordinal indices are non-negative integers, you can generate an identifier by adding 1 to the ordinal.
integers, you can generate an identifier by adding one to the ordinal.
```shell ```shell
myid zk-0 myid zk-0
@@ -193,7 +164,7 @@ myid zk-2
3 3
``` ```
Get the FQDN (Fully Qualified Domain Name) of each Pod in the `zk` StatefulSet. To get the Fully Qualified Domain Name (FQDN) of each Pod in the `zk` StatefulSet use the following command.
```shell ```shell
for i in 0 1 2; do kubectl exec zk-$i -- hostname -f; done for i in 0 1 2; do kubectl exec zk-$i -- hostname -f; done
@@ -208,18 +179,15 @@ zk-1.zk-hs.default.svc.cluster.local
zk-2.zk-hs.default.svc.cluster.local zk-2.zk-hs.default.svc.cluster.local
``` ```
The A records in [Kubernetes DNS](/docs/concepts/services-networking/dns-pod-service/) resolve the FQDNs to the Pods' IP addresses. The A records in [Kubernetes DNS](/docs/concepts/services-networking/dns-pod-service/) resolve the FQDNs to the Pods' IP addresses. If Kubernetes reschedules the Pods, it will update the A records with the Pods' new IP addresses, but the A records names will not change.
If the Pods are rescheduled, the A records will be updated with the Pods' new IP
addresses, but the A record's names will not change.
ZooKeeper stores its application configuration in a file named `zoo.cfg`. Use ZooKeeper stores its application configuration in a file named `zoo.cfg`. Use `kubectl exec` to view the contents of the `zoo.cfg` file in the `zk-0` Pod.
`kubectl exec` to view the contents of the `zoo.cfg` file in the `zk-0` Pod.
``` ```shell
kubectl exec zk-0 -- cat /opt/zookeeper/conf/zoo.cfg kubectl exec zk-0 -- cat /opt/zookeeper/conf/zoo.cfg
``` ```
For the `server.1`, `server.2`, and `server.3` properties at the bottom of In the `server.1`, `server.2`, and `server.3` properties at the bottom of
the file, the `1`, `2`, and `3` correspond to the identifiers in the the file, the `1`, `2`, and `3` correspond to the identifiers in the
ZooKeeper servers' `myid` files. They are set to the FQDNs for the Pods in ZooKeeper servers' `myid` files. They are set to the FQDNs for the Pods in
the `zk` StatefulSet. the `zk` StatefulSet.
@@ -243,15 +211,11 @@ server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888
### Achieving Consensus ### Achieving Consensus
Consensus protocols require that the identifiers of each participant be Consensus protocols require that the identifiers of each participant be unique. No two participants in the Zab protocol should claim the same unique identifier. This is necessary to allow the processes in the system to agree on which processes have committed which data. If two Pods are launched with the same ordinal, two ZooKeeper servers would both identify themselves as the same server.
unique. No two participants in the Zab protocol should claim the same unique
identifier. This is necessary to allow the processes in the system to agree on
which processes have committed which data. If two Pods were launched with the
same ordinal, two ZooKeeper servers would both identify themselves as the same
server.
```shell ```shell
kubectl get pods -w -l app=zk kubectl get pods -w -l app=zk
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s
zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s
@@ -270,8 +234,8 @@ zk-2 0/1 Running 0 19s
zk-2 1/1 Running 0 40s zk-2 1/1 Running 0 40s
``` ```
The A records for each Pod are only entered when the Pod becomes Ready. Therefore, The A records for each Pod are entered when the Pod becomes Ready. Therefore,
the FQDNs of the ZooKeeper servers will only resolve to a single endpoint, and that the FQDNs of the ZooKeeper servers will resolve to a single endpoint, and that
endpoint will be the unique ZooKeeper server claiming the identity configured endpoint will be the unique ZooKeeper server claiming the identity configured
in its `myid` file. in its `myid` file.
@@ -290,39 +254,31 @@ server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888
server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888 server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888
``` ```
When the servers use the Zab protocol to attempt to commit a value, they will When the servers use the Zab protocol to attempt to commit a value, they will either achieve consensus and commit the value (if leader election has succeeded and at least two of the Pods are Running and Ready), or they will fail to do so (if either of the conditions are not met). No state will arise where one server acknowledges a write on behalf of another.
either achieve consensus and commit the value (if leader election has succeeded
and at least two of the Pods are Running and Ready), or they will fail to do so
(if either of the aforementioned conditions are not met). No state will arise
where one server acknowledges a write on behalf of another.
### Sanity Testing the Ensemble ### Sanity Testing the Ensemble
The most basic sanity test is to write some data to one ZooKeeper server and The most basic sanity test is to write data to one ZooKeeper server and
to read the data from another. to read the data from another.
Use the `zkCli.sh` script to write `world` to the path `/hello` on the `zk-0` Pod. The command below executes the `zkCli.sh` script to write `world` to the path `/hello` on the `zk-0` Pod in the ensemble.
```shell ```shell
kubectl exec zk-0 zkCli.sh create /hello world kubectl exec zk-0 zkCli.sh create /hello world
```
This will write `world` to the `/hello` path in the ensemble.
```shell
WATCHER:: WATCHER::
WatchedEvent state:SyncConnected type:None path:null WatchedEvent state:SyncConnected type:None path:null
Created /hello Created /hello
``` ```
Get the data from the `zk-1` Pod. To get the data from the `zk-1` Pod use the following command.
```shell ```shell
kubectl exec zk-1 zkCli.sh get /hello kubectl exec zk-1 zkCli.sh get /hello
``` ```
The data that you created on `zk-0` is available on all of the servers in the The data that you created on `zk-0` is available on all the servers in the
ensemble. ensemble.
```shell ```shell
@@ -349,9 +305,9 @@ As mentioned in the [ZooKeeper Basics](#zookeeper-basics) section,
ZooKeeper commits all entries to a durable WAL, and periodically writes snapshots ZooKeeper commits all entries to a durable WAL, and periodically writes snapshots
in memory state, to storage media. Using WALs to provide durability is a common in memory state, to storage media. Using WALs to provide durability is a common
technique for applications that use consensus protocols to achieve a replicated technique for applications that use consensus protocols to achieve a replicated
state machine and for storage applications in general. state machine.
Use [`kubectl delete`](/docs/user-guide/kubectl/{{page.version}}/#delete) to delete the Use the [`kubectl delete`](/docs/user-guide/kubectl/{{page.version}}/#delete) command to delete the
`zk` StatefulSet. `zk` StatefulSet.
```shell ```shell
@@ -381,14 +337,14 @@ zk-0 0/1 Terminating 0 11m
zk-0 0/1 Terminating 0 11m zk-0 0/1 Terminating 0 11m
zk-0 0/1 Terminating 0 11m zk-0 0/1 Terminating 0 11m
``` ```
Reapply the manifest in `zookeeper.yaml`. Reapply the manifest in `zookeeper.yaml`.
```shell ```shell
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/docs/tutorials/stateful-application/zookeeper.yaml kubectl apply -f https://k8s.io/docs/tutorials/stateful-application/zookeeper.yaml
``` ```
The `zk` StatefulSet will be created, but, as they already exist, the other API This creates the `zk` StatefulSet object, but the other API objects in the manifest are not modified because they already exist.
Objects in the manifest will not be modified.
Watch the StatefulSet controller recreate the StatefulSet's Pods. Watch the StatefulSet controller recreate the StatefulSet's Pods.
@@ -417,15 +373,14 @@ zk-2 0/1 Running 0 19s
zk-2 1/1 Running 0 40s zk-2 1/1 Running 0 40s
``` ```
Get the value you entered during the [sanity test](#sanity-testing-the-ensemble), Use the command below to get the value you entered during the [sanity test](#sanity-testing-the-ensemble),
from the `zk-2` Pod. from the `zk-2` Pod.
```shell ```shell
kubectl exec zk-2 zkCli.sh get /hello kubectl exec zk-2 zkCli.sh get /hello
``` ```
Even though all of the Pods in the `zk` StatefulSet have been terminated and Even though you terminated and recreated all of the Pods in the `zk` StatefulSet, the ensemble still serves the original value.
recreated, the ensemble still serves the original value.
```shell ```shell
WATCHER:: WATCHER::
@@ -445,8 +400,7 @@ dataLength = 5
numChildren = 0 numChildren = 0
``` ```
The `volumeClaimTemplates` field, of the `zk` StatefulSet's `spec`, specifies a The `volumeClaimTemplates` field of the `zk` StatefulSet's `spec` specifies a PersistentVolume provisioned for each Pod.
PersistentVolume that will be provisioned for each Pod.
```yaml ```yaml
volumeClaimTemplates: volumeClaimTemplates:
@@ -461,18 +415,16 @@ volumeClaimTemplates:
storage: 20Gi storage: 20Gi
``` ```
The `StatefulSet` controller generates a `PersistentVolumeClaim` for each Pod in
the `StatefulSet`.
The StatefulSet controller generates a PersistentVolumeClaim for each Pod in Use the following command to get the `StatefulSet`'s `PersistentVolumeClaims`.
the StatefulSet.
Get the StatefulSet's PersistentVolumeClaims.
```shell ```shell
kubectl get pvc -l app=zk kubectl get pvc -l app=zk
``` ```
When the StatefulSet recreated its Pods, the Pods' PersistentVolumes were When the `StatefulSet` recreated its Pods, it remounts the Pods' PersistentVolumes.
remounted.
```shell ```shell
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
@@ -481,8 +433,7 @@ datadir-zk-1 Bound pvc-bedd27d2-bcb1-11e6-994f-42010a800002 20Gi R
datadir-zk-2 Bound pvc-bee0817e-bcb1-11e6-994f-42010a800002 20Gi RWO 1h datadir-zk-2 Bound pvc-bee0817e-bcb1-11e6-994f-42010a800002 20Gi RWO 1h
``` ```
The `volumeMounts` section of the StatefulSet's container `template` causes the The `volumeMounts` section of the `StatefulSet`'s container `template` mounts the PersistentVolumes in the ZooKeeper servers' data directories.
PersistentVolumes to be mounted to the ZooKeeper servers' data directories.
```shell ```shell
volumeMounts: volumeMounts:
@@ -490,16 +441,16 @@ volumeMounts:
mountPath: /var/lib/zookeeper mountPath: /var/lib/zookeeper
``` ```
When a Pod in the `zk` StatefulSet is (re)scheduled, it will always have the When a Pod in the `zk` `StatefulSet` is (re)scheduled, it will always have the
same PersistentVolume mounted to the ZooKeeper server's data directory. same `PersistentVolume` mounted to the ZooKeeper server's data directory.
Even when the Pods are rescheduled, all of the writes made to the ZooKeeper Even when the Pods are rescheduled, all the writes made to the ZooKeeper
servers' WALs, and all of their snapshots, remain durable. servers' WALs, and all their snapshots, remain durable.
## Ensuring Consistent Configuration ## Ensuring Consistent Configuration
As noted in the [Facilitating Leader Election](#facilitating-leader-election) and As noted in the [Facilitating Leader Election](#facilitating-leader-election) and
[Achieving Consensus](#achieving-consensus) sections, the servers in a [Achieving Consensus](#achieving-consensus) sections, the servers in a
ZooKeeper ensemble require consistent configuration in order to elect a leader ZooKeeper ensemble require consistent configuration to elect a leader
and form a quorum. They also require consistent configuration of the Zab protocol and form a quorum. They also require consistent configuration of the Zab protocol
in order for the protocol to work correctly over a network. In our example we in order for the protocol to work correctly over a network. In our example we
achieve consistent configuration by embedding the configuration directly into achieve consistent configuration by embedding the configuration directly into
@@ -507,43 +458,42 @@ the manifest.
Get the `zk` StatefulSet. Get the `zk` StatefulSet.
```shell{% raw %} ```shell
kubectl get sts zk -o yaml kubectl get sts zk -o yaml
... …
command: command:
- sh - sh
- -c - -c
- "start-zookeeper \ - "start-zookeeper \
--servers=3 \ --servers=3 \
--data_dir=/var/lib/zookeeper/data \ --data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \ --data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \ --conf_dir=/opt/zookeeper/conf \
--client_port=2181 \ --client_port=2181 \
--election_port=3888 \ --election_port=3888 \
--server_port=2888 \ --server_port=2888 \
--tick_time=2000 \ --tick_time=2000 \
--init_limit=10 \ --init_limit=10 \
--sync_limit=5 \ --sync_limit=5 \
--heap=512M \ --heap=512M \
--max_client_cnxns=60 \ --max_client_cnxns=60 \
--snap_retain_count=3 \ --snap_retain_count=3 \
--purge_interval=12 \ --purge_interval=12 \
--max_session_timeout=40000 \ --max_session_timeout=40000 \
--min_session_timeout=4000 \ --min_session_timeout=4000 \
--log_level=INFO" --log_level=INFO"
... …
```{% endraw %} ```
Notice that the command used to start the ZooKeeper servers passed the configuration The command used to start the ZooKeeper servers passed the configuration as command line parameter. You can also use environment variables to pass configuration to the ensemble.
as command line parameter. Environment variables are another, equally good, way to
pass configuration to ensemble.
### Configuring Logging ### Configuring Logging
One of the files generated by the `zkGenConfig.sh` script controls ZooKeeper's logging. One of the files generated by the `zkGenConfig.sh` script controls ZooKeeper's logging.
ZooKeeper uses [Log4j](http://logging.apache.org/log4j/2.x/), and, by default, ZooKeeper uses [Log4j](http://logging.apache.org/log4j/2.x/), and, by default,
it uses a time and size based rolling file appender for its logging configuration. it uses a time and size based rolling file appender for its logging configuration.
Get the logging configuration from one of Pods in the `zk` StatefulSet.
Use the command below to get the logging configuration from one of Pods in the `zk` `StatefulSet`.
```shell ```shell
kubectl exec zk-0 cat /usr/etc/zookeeper/log4j.properties kubectl exec zk-0 cat /usr/etc/zookeeper/log4j.properties
@@ -562,21 +512,15 @@ log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
``` ```
This is the simplest possible way to safely log inside the container. As the This is the simplest possible way to safely log inside the container. Because the applications write logs to standard out, Kubernetes will handle log rotation for you. Kubernetes also implements a sane retention policy that ensures application logs written to standard out and standard error do not exhaust local storage media.
application's logs are being written to standard out, Kubernetes will handle
log rotation for you. Kubernetes also implements a sane retention policy that
ensures application logs written to standard out and standard error do not
exhaust local storage media.
Use [`kubectl logs`](/docs/user-guide/kubectl/{{page.version}}/#logs) to retrieve the last Use [`kubectl logs`](/docs/user-guide/kubectl/{{page.version}}/#logs) to retrieve the last 20 log lines from one of the Pods.
few log lines from one of the Pods.
```shell ```shell
kubectl logs zk-0 --tail 20 kubectl logs zk-0 --tail 20
``` ```
Application logs that are written to standard out or standard error are viewable You can view application logs written to standard out or standard error using `kubectl logs` and from the Kubernetes Dashboard.
using `kubectl logs` and from the Kubernetes Dashboard.
```shell ```shell
2016-12-06 19:34:16,236 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52740 2016-12-06 19:34:16,236 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52740
@@ -601,22 +545,21 @@ using `kubectl logs` and from the Kubernetes Dashboard.
2016-12-06 19:34:46,230 [myid:1] - INFO [Thread-1142:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52768 (no session established for client) 2016-12-06 19:34:46,230 [myid:1] - INFO [Thread-1142:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52768 (no session established for client)
``` ```
Kubernetes also supports more powerful, but more complex, logging integrations Kubernetes supports more powerful, but more complex, logging integrations
with [Logging Using Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/) with [Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/)
and [Logging Using Elasticsearch and Kibana](/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/). and [Elasticsearch and Kibana](/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/).
For cluster level log shipping and aggregation, you should consider deploying a For cluster level log shipping and aggregation, consider deploying a [sidecar](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html)
[sidecar](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html)
container to rotate and ship your logs. container to rotate and ship your logs.
### Configuring a Non-Privileged User ### Configuring a Non-Privileged User
The best practices with respect to allowing an application to run as a privileged The best practices to allow an application to run as a privileged
user inside of a container are a matter of debate. If your organization requires user inside of a container are a matter of debate. If your organization requires
that applications be run as a non-privileged user you can use a that applications run as a non-privileged user you can use a
[SecurityContext](/docs/tasks/configure-pod-container/security-context/) to control the user that [SecurityContext](/docs/tasks/configure-pod-container/security-context/) to control the user that
the entry point runs as. the entry point runs as.
The `zk` StatefulSet's Pod `template` contains a SecurityContext. The `zk` `StatefulSet`'s Pod `template` contains a `SecurityContext`.
```yaml ```yaml
securityContext: securityContext:
@@ -642,19 +585,15 @@ F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S zookeep+ 27 1 0 80 0 - 1155556 - 20:46 ? 00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg 0 S zookeep+ 27 1 0 80 0 - 1155556 - 20:46 ? 00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
``` ```
By default, when the Pod's PersistentVolume is mounted to the ZooKeeper server's By default, when the Pod's PersistentVolumes is mounted to the ZooKeeper server's data directory, it is only accessible by the root user. This configuration prevents the ZooKeeper process from writing to its WAL and storing its snapshots.
data directory, it is only accessible by the root user. This configuration
prevents the ZooKeeper process from writing to its WAL and storing its snapshots.
Get the file permissions of the ZooKeeper data directory on the `zk-0` Pod. Use the command below to get the file permissions of the ZooKeeper data directory on the `zk-0` Pod.
```shell ```shell
kubectl exec -ti zk-0 -- ls -ld /var/lib/zookeeper/data kubectl exec -ti zk-0 -- ls -ld /var/lib/zookeeper/data
``` ```
As the `fsGroup` field of the `securityContext` object is set to 1000, Because the `fsGroup` field of the `securityContext` object is set to 1000, the ownership of the Pods' PersistentVolumes is set to the zookeeper group, and the ZooKeeper process is able to read and write its data.
the ownership of the Pods' PersistentVolumes is set to the zookeeper group,
and the ZooKeeper process is able to successfully read and write its data.
```shell ```shell
drwxr-sr-x 3 zookeeper zookeeper 4096 Dec 5 20:45 /var/lib/zookeeper/data drwxr-sr-x 3 zookeeper zookeeper 4096 Dec 5 20:45 /var/lib/zookeeper/data
@@ -663,7 +602,7 @@ drwxr-sr-x 3 zookeeper zookeeper 4096 Dec 5 20:45 /var/lib/zookeeper/data
## Managing the ZooKeeper Process ## Managing the ZooKeeper Process
The [ZooKeeper documentation](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_supervision) The [ZooKeeper documentation](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_supervision)
indicates that "You will want to have a supervisory process that mentions that "You will want to have a supervisory process that
manages each of your ZooKeeper server processes (JVM)." Utilizing a watchdog manages each of your ZooKeeper server processes (JVM)." Utilizing a watchdog
(supervisory process) to restart failed processes in a distributed system is a (supervisory process) to restart failed processes in a distributed system is a
common pattern. When deploying an application in Kubernetes, rather than using common pattern. When deploying an application in Kubernetes, rather than using
@@ -672,20 +611,21 @@ watchdog for your application.
### Updating the Ensemble ### Updating the Ensemble
The `zk` StatefulSet is configured to use the RollingUpdate update strategy. The `zk` `StatefulSet` is configured to use the `RollingUpdate` update strategy.
You can use `kubectl patch` to update the number of `cpus` allocated to the servers. You can use `kubectl patch` to update the number of `cpus` allocated to the servers.
```shell{% raw %} ```shell
kubectl patch sts zk --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value":"0.3"}]' kubectl patch sts zk --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value":"0.3"}]'
statefulset "zk" patched statefulset "zk" patched
```{% endraw %} ```
Use `kubectl rollout status` to watch the status of the update. Use `kubectl rollout status` to watch the status of the update.
```shell ```shell
kubectl rollout status sts/zk kubectl rollout status sts/zk
waiting for statefulset rolling update to complete 0 pods at revision zk-5db4499664... waiting for statefulset rolling update to complete 0 pods at revision zk-5db4499664...
Waiting for 1 pods to be ready... Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready... Waiting for 1 pods to be ready...
@@ -698,37 +638,36 @@ Waiting for 1 pods to be ready...
statefulset rolling update complete 3 pods at revision zk-5db4499664... statefulset rolling update complete 3 pods at revision zk-5db4499664...
``` ```
The Pods are terminated, one at a time, in reverse ordinal order, and they This terminates the Pods, one at a time, in reverse ordinal order, and recreates them with the new configuration. This ensures that quorum is maintained during a rolling update.
are recreated with the new configuration. This ensures that quorum is maintained
during a rolling update.
Use `kubectl rollout history` to view a history or previous configurations. Use the `kubectl rollout history` command to view a history or previous configurations.
```shell ```shell
kubectl rollout history sts/zk kubectl rollout history sts/zk
statefulsets "zk" statefulsets "zk"
REVISION REVISION
1 1
2 2
``` ```
Use `kubectl rollout undo` to roll back the modification.
Use the `kubectl rollout undo` command to roll back the modification.
```shell ```shell
kubectl rollout undo sts/zk kubectl rollout undo sts/zk
statefulset "zk" rolled back statefulset "zk" rolled back
``` ```
### Handling Process Failure ### Handling Process Failure
[Restart Policies](/docs/user-guide/pod-states/#restartpolicy) control how [Restart Policies](/docs/user-guide/pod-states/#restartpolicy) control how
Kubernetes handles process failures for the entry point of the container in a Pod. Kubernetes handles process failures for the entry point of the container in a Pod.
For Pods in a StatefulSet, the only appropriate RestartPolicy is Always, and this For Pods in a `StatefulSet`, the only appropriate `RestartPolicy` is Always, and this
is the default value. For stateful applications you should **never** override is the default value. For stateful applications you should **never** override
the default policy. the default policy.
Use the following command to examine the process tree for the ZooKeeper server running in the `zk-0` Pod.
Examine the process tree for the ZooKeeper server running in the `zk-0` Pod.
```shell ```shell
kubectl exec zk-0 -- ps -ef kubectl exec zk-0 -- ps -ef
@@ -737,31 +676,25 @@ kubectl exec zk-0 -- ps -ef
The command used as the container's entry point has PID 1, and The command used as the container's entry point has PID 1, and
the ZooKeeper process, a child of the entry point, has PID 23. the ZooKeeper process, a child of the entry point, has PID 23.
```shell
```
UID PID PPID C STIME TTY TIME CMD UID PID PPID C STIME TTY TIME CMD
zookeep+ 1 0 0 15:03 ? 00:00:00 sh -c zkGenConfig.sh && zkServer.sh start-foreground zookeep+ 1 0 0 15:03 ? 00:00:00 sh -c zkGenConfig.sh && zkServer.sh start-foreground
zookeep+ 27 1 0 15:03 ? 00:00:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg zookeep+ 27 1 0 15:03 ? 00:00:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
``` ```
In another terminal watch the Pods in the `zk` `StatefulSet` with the following command.
In one terminal watch the Pods in the `zk` StatefulSet.
```shell ```shell
kubectl get pod -w -l app=zk kubectl get pod -w -l app=zk
``` ```
In another terminal, terminate the ZooKeeper process in Pod `zk-0` with the following command.
In another terminal, kill the ZooKeeper process in Pod `zk-0`.
```shell ```shell
kubectl exec zk-0 -- pkill java kubectl exec zk-0 -- pkill java
``` ```
The termination of the ZooKeeper process caused its parent process to terminate. Because the `RestartPolicy` of the container is Always, it restarted the parent process.
The death of the ZooKeeper process caused its parent process to terminate. As
the RestartPolicy of the container is Always, the parent process was relaunched.
```shell ```shell
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
@@ -774,25 +707,21 @@ zk-0 0/1 Running 1 29m
zk-0 1/1 Running 1 29m zk-0 1/1 Running 1 29m
``` ```
If your application uses a script (such as `zkServer.sh`) to launch the process
If your application uses a script (such as zkServer.sh) to launch the process
that implements the application's business logic, the script must terminate with the that implements the application's business logic, the script must terminate with the
child process. This ensures that Kubernetes will restart the application's child process. This ensures that Kubernetes will restart the application's
container when the process implementing the application's business logic fails. container when the process implementing the application's business logic fails.
### Testing for Liveness ### Testing for Liveness
Configuring your application to restart failed processes is not enough to
Configuring your application to restart failed processes is not sufficient to keep a distributed system healthy. There are scenarios where
keep a distributed system healthy. There are many scenarios where
a system's processes can be both alive and unresponsive, or otherwise a system's processes can be both alive and unresponsive, or otherwise
unhealthy. You should use liveness probes in order to notify Kubernetes unhealthy. You should use liveness probes to notify Kubernetes
that your application's processes are unhealthy and should be restarted. that your application's processes are unhealthy and it should restart them.
The Pod `template` for the `zk` StatefulSet specifies a liveness probe.
The Pod `template` for the `zk` `StatefulSet` specifies a liveness probe.
``
```yaml ```yaml
livenessProbe: livenessProbe:
@@ -805,11 +734,9 @@ The Pod `template` for the `zk` StatefulSet specifies a liveness probe.
timeoutSeconds: 5 timeoutSeconds: 5
``` ```
The probe calls a bash script that uses the ZooKeeper `ruok` four letter
The probe calls a simple bash script that uses the ZooKeeper `ruok` four letter
word to test the server's health. word to test the server's health.
```bash ```bash
OK=$(echo ruok | nc 127.0.0.1 $1) OK=$(echo ruok | nc 127.0.0.1 $1)
if [ "$OK" == "imok" ]; then if [ "$OK" == "imok" ]; then
@@ -819,30 +746,25 @@ else
fi fi
``` ```
In one terminal window, use the following command to watch the Pods in the `zk` StatefulSet.
In one terminal window, watch the Pods in the `zk` StatefulSet.
```shell ```shell
kubectl get pod -w -l app=zk kubectl get pod -w -l app=zk
``` ```
In another window, using the following command to delete the `zkOk.sh` script from the file system of Pod `zk-0`.
In another window, delete the `zkOk.sh` script from the file system of Pod `zk-0`.
```shell ```shell
kubectl exec zk-0 -- rm /usr/bin/zookeeper-ready kubectl exec zk-0 -- rm /usr/bin/zookeeper-ready
``` ```
When the liveness probe for the ZooKeeper process fails, Kubernetes will When the liveness probe for the ZooKeeper process fails, Kubernetes will
automatically restart the process for you, ensuring that unhealthy processes in automatically restart the process for you, ensuring that unhealthy processes in
the ensemble are restarted. the ensemble are restarted.
```shell ```shell
kubectl get pod -w -l app=zk kubectl get pod -w -l app=zk
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
zk-0 1/1 Running 0 1h zk-0 1/1 Running 0 1h
zk-1 1/1 Running 0 1h zk-1 1/1 Running 0 1h
@@ -853,25 +775,20 @@ zk-0 0/1 Running 1 1h
zk-0 1/1 Running 1 1h zk-0 1/1 Running 1 1h
``` ```
### Testing for Readiness ### Testing for Readiness
Readiness is not the same as liveness. If a process is alive, it is scheduled Readiness is not the same as liveness. If a process is alive, it is scheduled
and healthy. If a process is ready, it is able to process input. Liveness is and healthy. If a process is ready, it is able to process input. Liveness is
a necessary, but not sufficient, condition for readiness. There are many cases, a necessary, but not sufficient, condition for readiness. There are cases,
particularly during initialization and termination, when a process can be particularly during initialization and termination, when a process can be
alive but not ready. alive but not ready.
If you specify a readiness probe, Kubernetes will ensure that your application's If you specify a readiness probe, Kubernetes will ensure that your application's
processes will not receive network traffic until their readiness checks pass. processes will not receive network traffic until their readiness checks pass.
For a ZooKeeper server, liveness implies readiness. Therefore, the readiness For a ZooKeeper server, liveness implies readiness. Therefore, the readiness
probe from the `zookeeper.yaml` manifest is identical to the liveness probe. probe from the `zookeeper.yaml` manifest is identical to the liveness probe.
```yaml ```yaml
readinessProbe: readinessProbe:
exec: exec:
@@ -883,25 +800,20 @@ probe from the `zookeeper.yaml` manifest is identical to the liveness probe.
timeoutSeconds: 5 timeoutSeconds: 5
``` ```
Even though the liveness and readiness probes are identical, it is important Even though the liveness and readiness probes are identical, it is important
to specify both. This ensures that only healthy servers in the ZooKeeper to specify both. This ensures that only healthy servers in the ZooKeeper
ensemble receive network traffic. ensemble receive network traffic.
## Tolerating Node Failure ## Tolerating Node Failure
ZooKeeper needs a quorum of servers in order to successfully commit mutations ZooKeeper needs a quorum of servers to successfully commit mutations
to data. For a three server ensemble, two servers must be healthy in order for to data. For a three server ensemble, two servers must be healthy for
writes to succeed. In quorum based systems, members are deployed across failure writes to succeed. In quorum based systems, members are deployed across failure
domains to ensure availability. In order to avoid an outage, due to the loss of an domains to ensure availability. To avoid an outage, due to the loss of an
individual machine, best practices preclude co-locating multiple instances of the individual machine, best practices preclude co-locating multiple instances of the
application on the same machine. application on the same machine.
By default, Kubernetes may co-locate Pods in a StatefulSet on the same node. By default, Kubernetes may co-locate Pods in a `StatefulSet` on the same node. For the three server ensemble you created, if two servers are on the same node, and that node fails, the clients of your ZooKeeper service will experience an outage until at least one of the Pods can be rescheduled.
For the three server ensemble you created, if two servers reside on the same
node, and that node fails, the clients of your ZooKeeper service will experience
an outage until at least one of the Pods can be rescheduled.
You should always provision additional capacity to allow the processes of critical You should always provision additional capacity to allow the processes of critical
systems to be rescheduled in the event of node failures. If you do so, then the systems to be rescheduled in the event of node failures. If you do so, then the
@@ -909,13 +821,13 @@ outage will only last until the Kubernetes scheduler reschedules one of the ZooK
servers. However, if you want your service to tolerate node failures with no downtime, servers. However, if you want your service to tolerate node failures with no downtime,
you should set `podAntiAffinity`. you should set `podAntiAffinity`.
Get the nodes for Pods in the `zk` Stateful Set. Use the command below to get the nodes for Pods in the `zk` `StatefulSet`.
```shell{% raw %} ```shell
for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done
``` {% endraw %} ```
All of the Pods in the `zk` StatefulSet are deployed on different nodes. All of the Pods in the `zk` `StatefulSet` are deployed on different nodes.
```shell ```shell
kubernetes-minion-group-cxpk kubernetes-minion-group-cxpk
@@ -923,7 +835,7 @@ kubernetes-minion-group-a5aq
kubernetes-minion-group-2g2d kubernetes-minion-group-2g2d
``` ```
This is because the Pods in the `zk` StatefulSet have a PodAntiAffinity specified. This is because the Pods in the `zk` `StatefulSet` have a `PodAntiAffinity` specified.
```yaml ```yaml
affinity: affinity:
@@ -954,7 +866,7 @@ The previous section showed you how to spread your Pods across nodes to survive
unplanned node failures, but you also need to plan for temporary node failures unplanned node failures, but you also need to plan for temporary node failures
that occur due to planned maintenance. that occur due to planned maintenance.
Get the nodes in your cluster. Use this command to get the nodes in your cluster.
```shell ```shell
kubectl get nodes kubectl get nodes
@@ -963,55 +875,56 @@ kubectl get nodes
Use [`kubectl cordon`](/docs/user-guide/kubectl/{{page.version}}/#cordon) to Use [`kubectl cordon`](/docs/user-guide/kubectl/{{page.version}}/#cordon) to
cordon all but four of the nodes in your cluster. cordon all but four of the nodes in your cluster.
```shell{% raw %} ```shell
kubectl cordon < node name > kubectl cordon <node-name>
```{% endraw %} ```
Get the `zk-pdb` PodDisruptionBudget. Use this command to get the `zk-pdb` `PodDisruptionBudget`.
```shell ```shell
kubectl get pdb zk-pdb kubectl get pdb zk-pdb
``` ```
The `max-unavailable` field indicates to Kubernetes that at most one Pod from The `max-unavailable` field indicates to Kubernetes that at most one Pod from
`zk` StatefulSet can be unavailable at any time. `zk` `StatefulSet` can be unavailable at any time.
```shell ```shell
NAME MIN-AVAILABLE MAX-UNAVAILABLE ALLOWED-DISRUPTIONS AGE NAME MIN-AVAILABLE MAX-UNAVAILABLE ALLOWED-DISRUPTIONS AGE
zk-pdb N/A 1 1 zk-pdb N/A 1 1
``` ```
In one terminal, watch the Pods in the `zk` StatefulSet. In one terminal, use this command to watch the Pods in the `zk` `StatefulSet`.
```shell ```shell
kubectl get pods -w -l app=zk kubectl get pods -w -l app=zk
``` ```
In another terminal, get the nodes that the Pods are currently scheduled on. In another terminal, use this command to get the nodes that the Pods are currently scheduled on.
```shell{% raw %} ```shell
for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done
kubernetes-minion-group-pb41 kubernetes-minion-group-pb41
kubernetes-minion-group-ixsl kubernetes-minion-group-ixsl
kubernetes-minion-group-i4c4 kubernetes-minion-group-i4c4
{% endraw %}
``` ```
Use [`kubectl drain`](/docs/user-guide/kubectl/{{page.version}}/#drain) to cordon and Use [`kubectl drain`](/docs/user-guide/kubectl/{{page.version}}/#drain) to cordon and
drain the node on which the `zk-0` Pod is scheduled. drain the node on which the `zk-0` Pod is scheduled.
```shell {% raw %} ```shell
kubectl drain $(kubectl get pod zk-0 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data kubectl drain $(kubectl get pod zk-0 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data
node "kubernetes-minion-group-pb41" cordoned node "kubernetes-minion-group-pb41" cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-pb41, kube-proxy-kubernetes-minion-group-pb41; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-o5elz WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-pb41, kube-proxy-kubernetes-minion-group-pb41; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-o5elz
pod "zk-0" deleted pod "zk-0" deleted
node "kubernetes-minion-group-pb41" drained node "kubernetes-minion-group-pb41" drained
{% endraw %}``` ```
As there are four nodes in your cluster, `kubectl drain`, succeeds and the As there are four nodes in your cluster, `kubectl drain`, succeeds and the
`zk-0` is rescheduled to another node. `zk-0` is rescheduled to another node.
``` ```shell
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
zk-0 1/1 Running 2 1h zk-0 1/1 Running 2 1h
zk-1 1/1 Running 0 1h zk-1 1/1 Running 0 1h
@@ -1028,22 +941,22 @@ zk-0 0/1 Running 0 51s
zk-0 1/1 Running 0 1m zk-0 1/1 Running 0 1m
``` ```
Keep watching the StatefulSet's Pods in the first terminal and drain the node on which Keep watching the `StatefulSet`'s Pods in the first terminal and drain the node on which
`zk-1` is scheduled. `zk-1` is scheduled.
```shell{% raw %} ```shell
kubectl drain $(kubectl get pod zk-1 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data "kubernetes-minion-group-ixsl" cordoned kubectl drain $(kubectl get pod zk-1 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data "kubernetes-minion-group-ixsl" cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-ixsl, kube-proxy-kubernetes-minion-group-ixsl; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-voc74 WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-ixsl, kube-proxy-kubernetes-minion-group-ixsl; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-voc74
pod "zk-1" deleted pod "zk-1" deleted
node "kubernetes-minion-group-ixsl" drained node "kubernetes-minion-group-ixsl" drained
{% endraw %}``` ```
The `zk-1` Pod can not be scheduled. As the `zk` StatefulSet contains a The `zk-1` Pod cannot be scheduled because the `zk` `StatefulSet` contains a `PodAntiAffinity` rule preventing co-location of the Pods, and as only two nodes are schedulable, the Pod will remain in a Pending state.
PodAntiAffinity rule preventing co-location of the Pods, and as only
two nodes are schedulable, the Pod will remain in a Pending state.
```shell ```shell
kubectl get pods -w -l app=zk kubectl get pods -w -l app=zk
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
zk-0 1/1 Running 2 1h zk-0 1/1 Running 2 1h
zk-1 1/1 Running 0 1h zk-1 1/1 Running 0 1h
@@ -1069,19 +982,19 @@ zk-1 0/1 Pending 0 0s
Continue to watch the Pods of the stateful set, and drain the node on which Continue to watch the Pods of the stateful set, and drain the node on which
`zk-2` is scheduled. `zk-2` is scheduled.
```shell{% raw %} ```shell
kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data
node "kubernetes-minion-group-i4c4" cordoned node "kubernetes-minion-group-i4c4" cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-i4c4, kube-proxy-kubernetes-minion-group-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-i4c4, kube-proxy-kubernetes-minion-group-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog
WARNING: Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog; Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-i4c4, kube-proxy-kubernetes-minion-group-i4c4 WARNING: Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog; Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-i4c4, kube-proxy-kubernetes-minion-group-i4c4
There are pending pods when an error occurred: Cannot evict pod as it would violate the pod's disruption budget. There are pending pods when an error occurred: Cannot evict pod as it would violate the pod's disruption budget.
pod/zk-2 pod/zk-2
{% endraw %}``` ```
Use `CTRL-C` to terminate to kubectl. Use `CTRL-C` to terminate to kubectl.
You can not drain the third node because evicting `zk-2` would violate `zk-budget`. However, You cannot drain the third node because evicting `zk-2` would violate `zk-budget`. However, the node will remain cordoned.
the node will remain cordoned.
Use `zkCli.sh` to retrieve the value you entered during the sanity test from `zk-0`. Use `zkCli.sh` to retrieve the value you entered during the sanity test from `zk-0`.
@@ -1089,9 +1002,9 @@ Use `zkCli.sh` to retrieve the value you entered during the sanity test from `zk
kubectl exec zk-0 zkCli.sh get /hello kubectl exec zk-0 zkCli.sh get /hello
``` ```
The service is still available because its PodDisruptionBudget is respected. The service is still available because its `PodDisruptionBudget` is respected.
``` ```shell
WatchedEvent state:SyncConnected type:None path:null WatchedEvent state:SyncConnected type:None path:null
world world
cZxid = 0x200000002 cZxid = 0x200000002
@@ -1111,6 +1024,7 @@ Use [`kubectl uncordon`](/docs/user-guide/kubectl/{{page.version}}/#uncordon) to
```shell ```shell
kubectl uncordon kubernetes-minion-group-pb41 kubectl uncordon kubernetes-minion-group-pb41
node "kubernetes-minion-group-pb41" uncordoned node "kubernetes-minion-group-pb41" uncordoned
``` ```
@@ -1118,6 +1032,7 @@ node "kubernetes-minion-group-pb41" uncordoned
```shell ```shell
kubectl get pods -w -l app=zk kubectl get pods -w -l app=zk
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
zk-0 1/1 Running 2 1h zk-0 1/1 Running 2 1h
zk-1 1/1 Running 0 1h zk-1 1/1 Running 0 1h
@@ -1146,14 +1061,15 @@ zk-1 1/1 Running 0 13m
Attempt to drain the node on which `zk-2` is scheduled. Attempt to drain the node on which `zk-2` is scheduled.
```shell{% raw %} ```shell
kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data
node "kubernetes-minion-group-i4c4" already cordoned node "kubernetes-minion-group-i4c4" already cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-i4c4, kube-proxy-kubernetes-minion-group-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-minion-group-i4c4, kube-proxy-kubernetes-minion-group-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog
pod "heapster-v1.2.0-2604621511-wht1r" deleted pod "heapster-v1.2.0-2604621511-wht1r" deleted
pod "zk-2" deleted pod "zk-2" deleted
node "kubernetes-minion-group-i4c4" drained node "kubernetes-minion-group-i4c4" drained
{% endraw %}``` ```
This time `kubectl drain` succeeds. This time `kubectl drain` succeeds.
@@ -1161,22 +1077,20 @@ Uncordon the second node to allow `zk-2` to be rescheduled.
```shell ```shell
kubectl uncordon kubernetes-minion-group-ixsl kubectl uncordon kubernetes-minion-group-ixsl
node "kubernetes-minion-group-ixsl" uncordoned node "kubernetes-minion-group-ixsl" uncordoned
``` ```
You can use `kubectl drain` in conjunction with PodDisruptionBudgets to ensure that your service You can use `kubectl drain` in conjunction with `PodDisruptionBudgets` to ensure that your services remain available during maintenance. If drain is used to cordon nodes and evict pods prior to taking the node offline for maintenance, services that express a disruption budget will have that budget respected. You should always allocate additional capacity for critical services so that their Pods can be immediately rescheduled.
remains available during maintenance. If drain is used to cordon nodes and evict pods prior to
taking the node offline for maintenance, services that express a disruption budget will have that
budget respected. You should always allocate additional capacity for critical services so that
their Pods can be immediately rescheduled.
{% endcapture %} {% endcapture %}
{% capture cleanup %} {% capture cleanup %}
* Use `kubectl uncordon` to uncordon all the nodes in your cluster.
* You will need to delete the persistent storage media for the PersistentVolumes - Use `kubectl uncordon` to uncordon all the nodes in your cluster.
used in this tutorial. Follow the necessary steps, based on your environment, - You will need to delete the persistent storage media for the PersistentVolumes
storage configuration, and provisioning method, to ensure that all storage is used in this tutorial. Follow the necessary steps, based on your environment,
reclaimed. storage configuration, and provisioning method, to ensure that all storage is
{% endcapture %} reclaimed.
{% include templates/tutorial.md %} {% endcapture %}
{% include templates/tutorial.md %}