From 99f7d6d2f144776e21477305902f2edfb73cd6a2 Mon Sep 17 00:00:00 2001 From: gmarek Date: Wed, 31 Aug 2016 14:52:13 +0200 Subject: [PATCH] update Node admin doc --- docs/admin/node.md | 57 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 17 deletions(-) diff --git a/docs/admin/node.md b/docs/admin/node.md index 1f1c89b4cb..c64169b9bc 100644 --- a/docs/admin/node.md +++ b/docs/admin/node.md @@ -120,25 +120,48 @@ Currently, there are three components that interact with the Kubernetes node int ### Node Controller Node controller is a component in Kubernetes master which manages Node -objects. It performs two major functions: cluster-wide node synchronization -and single node life-cycle management. +objects. -Node controller has a sync loop that deletes Nodes from Kubernetes -based on all matching VM instances listed from the cloud provider. The sync period -can be controlled via flag `--node-sync-period`. If a new VM instance -gets created, Node Controller creates a representation for it. If an existing -instance gets deleted, Node Controller deletes the representation. Note however, -that Node Controller is unable to provision the node for you, i.e. it won't install -any binary; therefore, to -join a node to a Kubernetes cluster, you as an admin need to make sure proper services are -running in the node. In the future, we plan to automatically provision some node -services. +Node controller has mutliple roles in Node's life. First is assigning a CIDR block to +the Node when it is registered (if CIDR assignment is turned on). Second is keeping the +node controller's list of nodes up to date with the cloud provider's list of available +machines. When running in cloud environment whenever a node is unhealthy node controller +asks cloud provider if the VM for that node is still available. If not, the node +controller deletes the node from its list of nodes. -In general, node controller is responsible for updating the NodeReady condition of node -status to ConditionUnknown when a node becomes unreachable (e.g. due to the node being down), -and then later evicting all the pods from the node (using graceful termination) if the node -continues to be unreachable. (The current timeouts for those are 40s and 5m, respectively.) -It also allocates CIDR blocks to the new nodes. +Third responsibiliy is monitoring Node's health. Node controller is responsible for updating +the NodeReady condition of NodeStatus to ConditionUnknown when a node becomes unreachable +(i.e. node controller stops receiving heartbeats e.g. due to the node being down), and then +later evicting all the pods from the node (using graceful termination) if the node continues +to be unreachable (the current timeouts are 40s to start reporting ConditionUnknown and 5m +after that to start evicting pods). Node controller checks the state of each node every +`--node-monitor-period` seconds. + +In 1.4 release we updated the logic of node controller to better handle cases when a +big number of Nodes have problems with reaching the master machine (e.g. because +master machine has networking problem). Starting with 1.4 node controller will look at the +state of all Nodes in the cluster when making a decision about pod eviction. + +In most cases, node controller limits the eviction rate to `--node-eviction-rate` (default 0.1) +per second, meaning it won't evict pods from more than 1 node per 10 seconds. + +The node eviction behavior changes when a node in a given availability zone becomes unhealthy, +node controller checks what percentage of nodes in the zone are unhealthy (NodeReady condition +is ConditionUnknown or ConditionFalse) at the same time. If the fraction of unhealthy nodes is +at least `--unhealthy-zone-threshold` (default 0.55) then the eviction rate is reduced: if +the cluster is small (i.e. has less than or equal to `--large-cluster-size-threshold` +nodes - default 50) then evictions are stopped, otherwise the eviction rate is reduced to +`--secondary-node-eviction-rate` (default 0.01) per second. The reason these policies are +implemented per availability zone is because one availability zone might become partitioned +from the master while the others remain connected. If your cluster does not span multiple cloud +provider availability zones, then there is only one availability zone, namely the whole cluster. + +A key reason for spreading your nodes across availability zones is so that workload can be +shifted to healthy zones when one entire zone goes down. To enable this behavior, if all +nodes in a zone are unhealthy then node controller evicts at the normal rate `--node-eviction-rate`. +The corner case for that is when all zones are completely unhealthy (i.e. there's no healthy node in +the cluster). In such case node controller assumes that there's some problem with master machine +connectivity and stops all evictions until any connectivity is restored. ### Self-Registration of Nodes