Remove or move topics under docs/admin. (#9140)
This commit is contained in:
Steve Perry
committed by
k8s-ci-robot
k8s-ci-robot
parent
3fd4c6b3d1
commit
7a780bef1e
@@ -0,0 +1,126 @@
|
||||
---
|
||||
reviewers:
|
||||
- davidopp
|
||||
- lavalamp
|
||||
title: Building Large Clusters
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
At {{< param "version" >}}, Kubernetes supports clusters with up to 5000 nodes. More specifically, we support configurations that meet *all* of the following criteria:
|
||||
|
||||
* No more than 5000 nodes
|
||||
* No more than 150000 total pods
|
||||
* No more than 300000 total containers
|
||||
* No more than 100 pods per node
|
||||
|
||||
<br>
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Setup
|
||||
|
||||
A cluster is a set of nodes (physical or virtual machines) running Kubernetes agents, managed by a "master" (the cluster-level control plane).
|
||||
|
||||
Normally the number of nodes in a cluster is controlled by the value `NUM_NODES` in the platform-specific `config-default.sh` file (for example, see [GCE's `config-default.sh`](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/gce/config-default.sh)).
|
||||
|
||||
Simply changing that value to something very large, however, may cause the setup script to fail for many cloud providers. A GCE deployment, for example, will run in to quota issues and fail to bring the cluster up.
|
||||
|
||||
When setting up a large Kubernetes cluster, the following issues must be considered.
|
||||
|
||||
### Quota Issues
|
||||
|
||||
To avoid running into cloud provider quota issues, when creating a cluster with many nodes, consider:
|
||||
|
||||
* Increase the quota for things like CPU, IPs, etc.
|
||||
* In [GCE, for example,](https://cloud.google.com/compute/docs/resource-quotas) you'll want to increase the quota for:
|
||||
* CPUs
|
||||
* VM instances
|
||||
* Total persistent disk reserved
|
||||
* In-use IP addresses
|
||||
* Firewall Rules
|
||||
* Forwarding rules
|
||||
* Routes
|
||||
* Target pools
|
||||
* Gating the setup script so that it brings up new node VMs in smaller batches with waits in between, because some cloud providers rate limit the creation of VMs.
|
||||
|
||||
### Etcd storage
|
||||
|
||||
To improve performance of large clusters, we store events in a separate dedicated etcd instance.
|
||||
|
||||
When creating a cluster, existing salt scripts:
|
||||
|
||||
* start and configure additional etcd instance
|
||||
* configure api-server to use it for storing events
|
||||
|
||||
### Size of master and master components
|
||||
|
||||
On GCE/Google Kubernetes Engine, and AWS, `kube-up` automatically configures the proper VM size for your master depending on the number of nodes
|
||||
in your cluster. On other providers, you will need to configure it manually. For reference, the sizes we use on GCE are
|
||||
|
||||
* 1-5 nodes: n1-standard-1
|
||||
* 6-10 nodes: n1-standard-2
|
||||
* 11-100 nodes: n1-standard-4
|
||||
* 101-250 nodes: n1-standard-8
|
||||
* 251-500 nodes: n1-standard-16
|
||||
* more than 500 nodes: n1-standard-32
|
||||
|
||||
And the sizes we use on AWS are
|
||||
|
||||
* 1-5 nodes: m3.medium
|
||||
* 6-10 nodes: m3.large
|
||||
* 11-100 nodes: m3.xlarge
|
||||
* 101-250 nodes: m3.2xlarge
|
||||
* 251-500 nodes: c4.4xlarge
|
||||
* more than 500 nodes: c4.8xlarge
|
||||
|
||||
Note that these master node sizes are currently only set at cluster startup time, and are not adjusted if you later scale your cluster up or down (e.g. manually removing or adding nodes, or using a cluster autoscaler).
|
||||
|
||||
### Addon Resources
|
||||
|
||||
To prevent memory leaks or other resource issues in [cluster addons](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons) from consuming all the resources available on a node, Kubernetes sets resource limits on addon containers to limit the CPU and Memory resources they can consume (See PR [#10653](http://pr.k8s.io/10653/files) and [#10778](http://pr.k8s.io/10778/files)).
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
containers:
|
||||
- name: fluentd-cloud-logging
|
||||
image: k8s.gcr.io/fluentd-gcp:1.16
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 200Mi
|
||||
```
|
||||
|
||||
Except for Heapster, these limits are static and are based on data we collected from addons running on 4-node clusters (see [#10335](http://issue.k8s.io/10335#issuecomment-117861225)). The addons consume a lot more resources when running on large deployment clusters (see [#5880](http://issue.k8s.io/5880#issuecomment-113984085)). So, if a large cluster is deployed without adjusting these values, the addons may continuously get killed because they keep hitting the limits.
|
||||
|
||||
To avoid running into cluster addon resource issues, when creating a cluster with many nodes, consider the following:
|
||||
|
||||
* Scale memory and CPU limits for each of the following addons, if used, as you scale up the size of cluster (there is one replica of each handling the entire cluster so memory and CPU usage tends to grow proportionally with size/load on cluster):
|
||||
* [InfluxDB and Grafana](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml)
|
||||
* [kubedns, dnsmasq, and sidecar](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/kube-dns.yaml.in)
|
||||
* [Kibana](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/kibana-deployment.yaml)
|
||||
* Scale number of replicas for the following addons, if used, along with the size of cluster (there are multiple replicas of each so increasing replicas should help handle increased load, but, since load per replica also increases slightly, also consider increasing CPU/memory limits):
|
||||
* [elasticsearch](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/es-statefulset.yaml)
|
||||
* Increase memory and CPU limits slightly for each of the following addons, if used, along with the size of cluster (there is one replica per node but CPU/memory usage increases slightly along with cluster load/size as well):
|
||||
* [FluentD with ElasticSearch Plugin](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml)
|
||||
* [FluentD with GCP Plugin](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml)
|
||||
|
||||
Heapster's resource limits are set dynamically based on the initial size of your cluster (see [#16185](http://issue.k8s.io/16185)
|
||||
and [#22940](http://issue.k8s.io/22940)). If you find that Heapster is running
|
||||
out of resources, you should adjust the formulas that compute heapster memory request (see those PRs for details).
|
||||
|
||||
For directions on how to detect if addon containers are hitting resource limits, see the [Troubleshooting section of Compute Resources](/docs/concepts/configuration/manage-compute-resources-container/#troubleshooting).
|
||||
|
||||
In the [future](http://issue.k8s.io/13048), we anticipate to set all cluster addon resource limits based on cluster size, and to dynamically adjust them if you grow or shrink your cluster.
|
||||
We welcome PRs that implement those features.
|
||||
|
||||
### Allowing minor node failure at startup
|
||||
|
||||
For various reasons (see [#18969](https://github.com/kubernetes/kubernetes/issues/18969) for more details) running
|
||||
`kube-up.sh` with a very large `NUM_NODES` may fail due to a very small number of nodes not coming up properly.
|
||||
Currently you have two choices: restart the cluster (`kube-down.sh` and then `kube-up.sh` again), or before
|
||||
running `kube-up.sh` set the environment variable `ALLOWED_NOTREADY_NODES` to whatever value you feel comfortable
|
||||
with. This will allow `kube-up.sh` to succeed with fewer than `NUM_NODES` coming up. Depending on the
|
||||
reason for the failure, those additional nodes may join later or the cluster may remain at a size of
|
||||
`NUM_NODES - ALLOWED_NOTREADY_NODES`.
|
||||
@@ -0,0 +1,332 @@
|
||||
---
|
||||
reviewers:
|
||||
- jlowdermilk
|
||||
- justinsb
|
||||
- quinton-hoole
|
||||
title: Running in Multiple Zones
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
Kubernetes 1.2 adds support for running a single cluster in multiple failure zones
|
||||
(GCE calls them simply "zones", AWS calls them "availability zones", here we'll refer to them as "zones").
|
||||
This is a lightweight version of a broader Cluster Federation feature (previously referred to by the affectionate
|
||||
nickname ["Ubernetes"](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/multicluster/federation.md)).
|
||||
Full Cluster Federation allows combining separate
|
||||
Kubernetes clusters running in different regions or cloud providers
|
||||
(or on-premises data centers). However, many
|
||||
users simply want to run a more available Kubernetes cluster in multiple zones
|
||||
of their single cloud provider, and this is what the multizone support in 1.2 allows
|
||||
(this previously went by the nickname "Ubernetes Lite").
|
||||
|
||||
Multizone support is deliberately limited: a single Kubernetes cluster can run
|
||||
in multiple zones, but only within the same region (and cloud provider). Only
|
||||
GCE and AWS are currently supported automatically (though it is easy to
|
||||
add similar support for other clouds or even bare metal, by simply arranging
|
||||
for the appropriate labels to be added to nodes and volumes).
|
||||
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Functionality
|
||||
|
||||
When nodes are started, the kubelet automatically adds labels to them with
|
||||
zone information.
|
||||
|
||||
Kubernetes will automatically spread the pods in a replication controller
|
||||
or service across nodes in a single-zone cluster (to reduce the impact of
|
||||
failures.) With multiple-zone clusters, this spreading behavior is
|
||||
extended across zones (to reduce the impact of zone failures.) (This is
|
||||
achieved via `SelectorSpreadPriority`). This is a best-effort
|
||||
placement, and so if the zones in your cluster are heterogeneous
|
||||
(e.g. different numbers of nodes, different types of nodes, or
|
||||
different pod resource requirements), this might prevent perfectly
|
||||
even spreading of your pods across zones. If desired, you can use
|
||||
homogeneous zones (same number and types of nodes) to reduce the
|
||||
probability of unequal spreading.
|
||||
|
||||
When persistent volumes are created, the `PersistentVolumeLabel`
|
||||
admission controller automatically adds zone labels to them. The scheduler (via the
|
||||
`VolumeZonePredicate` predicate) will then ensure that pods that claim a
|
||||
given volume are only placed into the same zone as that volume, as volumes
|
||||
cannot be attached across zones.
|
||||
|
||||
## Limitations
|
||||
|
||||
There are some important limitations of the multizone support:
|
||||
|
||||
* We assume that the different zones are located close to each other in the
|
||||
network, so we don't perform any zone-aware routing. In particular, traffic
|
||||
that goes via services might cross zones (even if pods in some pods backing that service
|
||||
exist in the same zone as the client), and this may incur additional latency and cost.
|
||||
|
||||
* Volume zone-affinity will only work with a `PersistentVolume`, and will not
|
||||
work if you directly specify an EBS volume in the pod spec (for example).
|
||||
|
||||
* Clusters cannot span clouds or regions (this functionality will require full
|
||||
federation support).
|
||||
|
||||
* Although your nodes are in multiple zones, kube-up currently builds
|
||||
a single master node by default. While services are highly
|
||||
available and can tolerate the loss of a zone, the control plane is
|
||||
located in a single zone. Users that want a highly available control
|
||||
plane should follow the [high availability](/docs/admin/high-availability) instructions.
|
||||
|
||||
* StatefulSet volume zone spreading when using dynamic provisioning is currently not compatible with
|
||||
pod affinity or anti-affinity policies.
|
||||
|
||||
* If the name of the StatefulSet contains dashes ("-"), volume zone spreading
|
||||
may not provide a uniform distribution of storage across zones.
|
||||
|
||||
* When specifying multiple PVCs in a Deployment or Pod spec, the StorageClass
|
||||
needs to be configured for a specific, single zone, or the PVs need to be
|
||||
statically provisioned in a specific zone. Another workaround is to use a
|
||||
StatefulSet, which will ensure that all the volumes for a replica are
|
||||
provisioned in the same zone.
|
||||
|
||||
|
||||
## Walkthrough
|
||||
|
||||
We're now going to walk through setting up and using a multi-zone
|
||||
cluster on both GCE & AWS. To do so, you bring up a full cluster
|
||||
(specifying `MULTIZONE=true`), and then you add nodes in additional zones
|
||||
by running `kube-up` again (specifying `KUBE_USE_EXISTING_MASTER=true`).
|
||||
|
||||
### Bringing up your cluster
|
||||
|
||||
Create the cluster as normal, but pass MULTIZONE to tell the cluster to manage multiple zones; creating nodes in us-central1-a.
|
||||
|
||||
GCE:
|
||||
|
||||
```shell
|
||||
curl -sS https://get.k8s.io | MULTIZONE=true KUBERNETES_PROVIDER=gce KUBE_GCE_ZONE=us-central1-a NUM_NODES=3 bash
|
||||
```
|
||||
|
||||
AWS:
|
||||
|
||||
```shell
|
||||
curl -sS https://get.k8s.io | MULTIZONE=true KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2a NUM_NODES=3 bash
|
||||
```
|
||||
|
||||
This step brings up a cluster as normal, still running in a single zone
|
||||
(but `MULTIZONE=true` has enabled multi-zone capabilities).
|
||||
|
||||
### Nodes are labeled
|
||||
|
||||
View the nodes; you can see that they are labeled with zone information.
|
||||
They are all in `us-central1-a` (GCE) or `us-west-2a` (AWS) so far. The
|
||||
labels are `failure-domain.beta.kubernetes.io/region` for the region,
|
||||
and `failure-domain.beta.kubernetes.io/zone` for the zone:
|
||||
|
||||
```shell
|
||||
> kubectl get nodes --show-labels
|
||||
|
||||
|
||||
NAME STATUS AGE VERSION LABELS
|
||||
kubernetes-master Ready,SchedulingDisabled 6m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-master
|
||||
kubernetes-minion-87j9 Ready 6m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-87j9
|
||||
kubernetes-minion-9vlv Ready 6m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv
|
||||
kubernetes-minion-a12q Ready 6m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-a12q
|
||||
```
|
||||
|
||||
### Add more nodes in a second zone
|
||||
|
||||
Let's add another set of nodes to the existing cluster, reusing the
|
||||
existing master, running in a different zone (us-central1-b or us-west-2b).
|
||||
We run kube-up again, but by specifying `KUBE_USE_EXISTING_MASTER=true`
|
||||
kube-up will not create a new master, but will reuse one that was previously
|
||||
created instead.
|
||||
|
||||
GCE:
|
||||
|
||||
```shell
|
||||
KUBE_USE_EXISTING_MASTER=true MULTIZONE=true KUBERNETES_PROVIDER=gce KUBE_GCE_ZONE=us-central1-b NUM_NODES=3 kubernetes/cluster/kube-up.sh
|
||||
```
|
||||
|
||||
On AWS we also need to specify the network CIDR for the additional
|
||||
subnet, along with the master internal IP address:
|
||||
|
||||
```shell
|
||||
KUBE_USE_EXISTING_MASTER=true MULTIZONE=true KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2b NUM_NODES=3 KUBE_SUBNET_CIDR=172.20.1.0/24 MASTER_INTERNAL_IP=172.20.0.9 kubernetes/cluster/kube-up.sh
|
||||
```
|
||||
|
||||
|
||||
View the nodes again; 3 more nodes should have launched and be tagged
|
||||
in us-central1-b:
|
||||
|
||||
```shell
|
||||
> kubectl get nodes --show-labels
|
||||
|
||||
NAME STATUS AGE VERSION LABELS
|
||||
kubernetes-master Ready,SchedulingDisabled 16m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-master
|
||||
kubernetes-minion-281d Ready 2m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-281d
|
||||
kubernetes-minion-87j9 Ready 16m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-87j9
|
||||
kubernetes-minion-9vlv Ready 16m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv
|
||||
kubernetes-minion-a12q Ready 17m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-a12q
|
||||
kubernetes-minion-pp2f Ready 2m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-pp2f
|
||||
kubernetes-minion-wf8i Ready 2m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-wf8i
|
||||
```
|
||||
|
||||
### Volume affinity
|
||||
|
||||
Create a volume using the dynamic volume creation (only PersistentVolumes are supported for zone affinity):
|
||||
|
||||
```json
|
||||
kubectl create -f - <<EOF
|
||||
{
|
||||
"kind": "PersistentVolumeClaim",
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "claim1",
|
||||
"annotations": {
|
||||
"volume.alpha.kubernetes.io/storage-class": "foo"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"accessModes": [
|
||||
"ReadWriteOnce"
|
||||
],
|
||||
"resources": {
|
||||
"requests": {
|
||||
"storage": "5Gi"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
**NOTE:** For version 1.3+ Kubernetes will distribute dynamic PV claims across
|
||||
the configured zones. For version 1.2, dynamic persistent volumes were
|
||||
always created in the zone of the cluster master
|
||||
(here us-central1-a / us-west-2a); that issue
|
||||
([#23330](https://github.com/kubernetes/kubernetes/issues/23330))
|
||||
was addressed in 1.3+.
|
||||
|
||||
Now lets validate that Kubernetes automatically labeled the zone & region the PV was created in.
|
||||
|
||||
```shell
|
||||
> kubectl get pv --show-labels
|
||||
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE LABELS
|
||||
pv-gce-mj4gm 5Gi RWO Bound default/claim1 46s failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a
|
||||
```
|
||||
|
||||
So now we will create a pod that uses the persistent volume claim.
|
||||
Because GCE PDs / AWS EBS volumes cannot be attached across zones,
|
||||
this means that this pod can only be created in the same zone as the volume:
|
||||
|
||||
```yaml
|
||||
kubectl create -f - <<EOF
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: mypod
|
||||
spec:
|
||||
containers:
|
||||
- name: myfrontend
|
||||
image: nginx
|
||||
volumeMounts:
|
||||
- mountPath: "/var/www/html"
|
||||
name: mypd
|
||||
volumes:
|
||||
- name: mypd
|
||||
persistentVolumeClaim:
|
||||
claimName: claim1
|
||||
EOF
|
||||
```
|
||||
|
||||
Note that the pod was automatically created in the same zone as the volume, as
|
||||
cross-zone attachments are not generally permitted by cloud providers:
|
||||
|
||||
```shell
|
||||
> kubectl describe pod mypod | grep Node
|
||||
Node: kubernetes-minion-9vlv/10.240.0.5
|
||||
> kubectl get node kubernetes-minion-9vlv --show-labels
|
||||
NAME STATUS AGE VERSION LABELS
|
||||
kubernetes-minion-9vlv Ready 22m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv
|
||||
```
|
||||
|
||||
### Pods are spread across zones
|
||||
|
||||
Pods in a replication controller or service are automatically spread
|
||||
across zones. First, let's launch more nodes in a third zone:
|
||||
|
||||
GCE:
|
||||
|
||||
```shell
|
||||
KUBE_USE_EXISTING_MASTER=true MULTIZONE=true KUBERNETES_PROVIDER=gce KUBE_GCE_ZONE=us-central1-f NUM_NODES=3 kubernetes/cluster/kube-up.sh
|
||||
```
|
||||
|
||||
AWS:
|
||||
|
||||
```shell
|
||||
KUBE_USE_EXISTING_MASTER=true MULTIZONE=true KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2c NUM_NODES=3 KUBE_SUBNET_CIDR=172.20.2.0/24 MASTER_INTERNAL_IP=172.20.0.9 kubernetes/cluster/kube-up.sh
|
||||
```
|
||||
|
||||
Verify that you now have nodes in 3 zones:
|
||||
|
||||
```shell
|
||||
kubectl get nodes --show-labels
|
||||
```
|
||||
|
||||
Create the guestbook-go example, which includes an RC of size 3, running a simple web app:
|
||||
|
||||
```shell
|
||||
find kubernetes/examples/guestbook-go/ -name '*.json' | xargs -I {} kubectl create -f {}
|
||||
```
|
||||
|
||||
The pods should be spread across all 3 zones:
|
||||
|
||||
```shell
|
||||
> kubectl describe pod -l app=guestbook | grep Node
|
||||
Node: kubernetes-minion-9vlv/10.240.0.5
|
||||
Node: kubernetes-minion-281d/10.240.0.8
|
||||
Node: kubernetes-minion-olsh/10.240.0.11
|
||||
|
||||
> kubectl get node kubernetes-minion-9vlv kubernetes-minion-281d kubernetes-minion-olsh --show-labels
|
||||
NAME STATUS AGE VERSION LABELS
|
||||
kubernetes-minion-9vlv Ready 34m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv
|
||||
kubernetes-minion-281d Ready 20m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-281d
|
||||
kubernetes-minion-olsh Ready 3m v1.6.0+fff5156 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-f,kubernetes.io/hostname=kubernetes-minion-olsh
|
||||
```
|
||||
|
||||
|
||||
Load-balancers span all zones in a cluster; the guestbook-go example
|
||||
includes an example load-balanced service:
|
||||
|
||||
```shell
|
||||
> kubectl describe service guestbook | grep LoadBalancer.Ingress
|
||||
LoadBalancer Ingress: 130.211.126.21
|
||||
|
||||
> ip=130.211.126.21
|
||||
|
||||
> curl -s http://${ip}:3000/env | grep HOSTNAME
|
||||
"HOSTNAME": "guestbook-44sep",
|
||||
|
||||
> (for i in `seq 20`; do curl -s http://${ip}:3000/env | grep HOSTNAME; done) | sort | uniq
|
||||
"HOSTNAME": "guestbook-44sep",
|
||||
"HOSTNAME": "guestbook-hum5n",
|
||||
"HOSTNAME": "guestbook-ppm40",
|
||||
```
|
||||
|
||||
The load balancer correctly targets all the pods, even though they are in multiple zones.
|
||||
|
||||
### Shutting down the cluster
|
||||
|
||||
When you're done, clean up:
|
||||
|
||||
GCE:
|
||||
|
||||
```shell
|
||||
KUBERNETES_PROVIDER=gce KUBE_USE_EXISTING_MASTER=true KUBE_GCE_ZONE=us-central1-f kubernetes/cluster/kube-down.sh
|
||||
KUBERNETES_PROVIDER=gce KUBE_USE_EXISTING_MASTER=true KUBE_GCE_ZONE=us-central1-b kubernetes/cluster/kube-down.sh
|
||||
KUBERNETES_PROVIDER=gce KUBE_GCE_ZONE=us-central1-a kubernetes/cluster/kube-down.sh
|
||||
```
|
||||
|
||||
AWS:
|
||||
|
||||
```shell
|
||||
KUBERNETES_PROVIDER=aws KUBE_USE_EXISTING_MASTER=true KUBE_AWS_ZONE=us-west-2c kubernetes/cluster/kube-down.sh
|
||||
KUBERNETES_PROVIDER=aws KUBE_USE_EXISTING_MASTER=true KUBE_AWS_ZONE=us-west-2b kubernetes/cluster/kube-down.sh
|
||||
KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2a kubernetes/cluster/kube-down.sh
|
||||
```
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
reviewers:
|
||||
- Random-Liu
|
||||
title: Validate Node Setup
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Node Conformance Test
|
||||
|
||||
*Node conformance test* is a containerized test framework that provides a system
|
||||
verification and functionality test for a node. The test validates whether the
|
||||
node meets the minimum requirements for Kubernetes; a node that passes the test
|
||||
is qualified to join a Kubernetes cluster.
|
||||
|
||||
## Limitations
|
||||
|
||||
In Kubernetes version 1.5, node conformance test has the following limitations:
|
||||
|
||||
* Node conformance test only supports Docker as the container runtime.
|
||||
|
||||
## Node Prerequisite
|
||||
|
||||
To run node conformance test, a node must satisfy the same prerequisites as a
|
||||
standard Kubernetes node. At a minimum, the node should have the following
|
||||
daemons installed:
|
||||
|
||||
* Container Runtime (Docker)
|
||||
* Kubelet
|
||||
|
||||
## Running Node Conformance Test
|
||||
|
||||
To run the node conformance test, perform the following steps:
|
||||
|
||||
1. Point your Kubelet to localhost `--api-servers="http://localhost:8080"`,
|
||||
because the test framework starts a local master to test Kubelet. There are some
|
||||
other Kubelet flags you may care:
|
||||
* `--pod-cidr`: If you are using `kubenet`, you should specify an arbitrary CIDR
|
||||
to Kubelet, for example `--pod-cidr=10.180.0.0/24`.
|
||||
* `--cloud-provider`: If you are using `--cloud-provider=gce`, you should
|
||||
remove the flag to run the test.
|
||||
|
||||
2. Run the node conformance test with command:
|
||||
|
||||
```shell
|
||||
# $CONFIG_DIR is the pod manifest path of your Kubelet.
|
||||
# $LOG_DIR is the test output path.
|
||||
sudo docker run -it --rm --privileged --net=host \
|
||||
-v /:/rootfs -v $CONFIG_DIR:$CONFIG_DIR -v $LOG_DIR:/var/result \
|
||||
k8s.gcr.io/node-test:0.2
|
||||
```
|
||||
|
||||
## Running Node Conformance Test for Other Architectures
|
||||
|
||||
Kubernetes also provides node conformance test docker images for other
|
||||
architectures:
|
||||
|
||||
Arch | Image |
|
||||
--------|:-----------------:|
|
||||
amd64 | node-test-amd64 |
|
||||
arm | node-test-arm |
|
||||
arm64 | node-test-arm64 |
|
||||
|
||||
## Running Selected Test
|
||||
|
||||
To run specific tests, overwrite the environment variable `FOCUS` with the
|
||||
regular expression of tests you want to run.
|
||||
|
||||
```shell
|
||||
sudo docker run -it --rm --privileged --net=host \
|
||||
-v /:/rootfs:ro -v $CONFIG_DIR:$CONFIG_DIR -v $LOG_DIR:/var/result \
|
||||
-e FOCUS=MirrorPod \ # Only run MirrorPod test
|
||||
k8s.gcr.io/node-test:0.2
|
||||
```
|
||||
|
||||
To skip specific tests, overwrite the environment variable `SKIP` with the
|
||||
regular expression of tests you want to skip.
|
||||
|
||||
```shell
|
||||
sudo docker run -it --rm --privileged --net=host \
|
||||
-v /:/rootfs:ro -v $CONFIG_DIR:$CONFIG_DIR -v $LOG_DIR:/var/result \
|
||||
-e SKIP=MirrorPod \ # Run all conformance tests but skip MirrorPod test
|
||||
k8s.gcr.io/node-test:0.2
|
||||
```
|
||||
|
||||
Node conformance test is a containerized version of [node e2e test](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/devel/e2e-node-tests.md).
|
||||
By default, it runs all conformance tests.
|
||||
|
||||
Theoretically, you can run any node e2e test if you configure the container and
|
||||
mount required volumes properly. But **it is strongly recommended to only run conformance
|
||||
test**, because it requires much more complex configuration to run non-conformance test.
|
||||
|
||||
## Caveats
|
||||
|
||||
* The test leaves some docker images on the node, including the node conformance
|
||||
test image and images of containers used in the functionality
|
||||
test.
|
||||
* The test leaves dead containers on the node. These containers are created
|
||||
during the functionality test.
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
reviewers:
|
||||
- davidopp
|
||||
title: Configuring Kubernetes with Salt
|
||||
---
|
||||
|
||||
The Kubernetes cluster can be configured using Salt.
|
||||
|
||||
The Salt scripts are shared across multiple hosting providers and depending on where you host your Kubernetes cluster, you may be using different operating systems and different networking configurations. As a result, it's important to understand some background information before making Salt changes in order to minimize introducing failures for other hosting providers.
|
||||
|
||||
## Salt cluster setup
|
||||
|
||||
The **salt-master** service runs on the kubernetes-master [(except on the default GCE and OpenStack-Heat setup)](#standalone-salt-configuration-on-gce-and-others).
|
||||
|
||||
The **salt-minion** service runs on the kubernetes-master and each kubernetes-node in the cluster.
|
||||
|
||||
Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file [(except on GCE and OpenStack-Heat)](#standalone-salt-configuration-on-gce-and-others).
|
||||
|
||||
```shell
|
||||
[root@kubernetes-master] $ cat /etc/salt/minion.d/master.conf
|
||||
master: kubernetes-master
|
||||
```
|
||||
|
||||
The salt-master is contacted by each salt-minion and depending upon the machine information presented, the salt-master will provision the machine as either a kubernetes-master or kubernetes-node with all the required capabilities needed to run Kubernetes.
|
||||
|
||||
If you are running the Vagrant based environment, the **salt-api** service is running on the kubernetes-master. It is configured to enable the vagrant user to introspect the salt cluster in order to find out about machines in the Vagrant environment via a REST API.
|
||||
|
||||
## Standalone Salt Configuration on GCE and others
|
||||
|
||||
On GCE and OpenStack, using the Openstack-Heat provider, the master and nodes are all configured as [standalone minions](http://docs.saltstack.com/en/latest/topics/tutorials/standalone_minion.html). The configuration for each VM is derived from the VM's [instance metadata](https://cloud.google.com/compute/docs/metadata) and then stored in Salt grains (`/etc/salt/minion.d/grains.conf`) and pillars (`/srv/salt-overlay/pillar/cluster-params.sls`) that local Salt uses to enforce state.
|
||||
|
||||
All remaining sections that refer to master/minion setups should be ignored for GCE and OpenStack. One fallout of this setup is that the Salt mine doesn't exist - there is no sharing of configuration amongst nodes.
|
||||
|
||||
## Salt security
|
||||
|
||||
*(Not applicable on default GCE and OpenStack-Heat setup.)*
|
||||
|
||||
Security is not enabled on the salt-master, and the salt-master is configured to auto-accept incoming requests from minions. It is not recommended to use this security configuration in production environments without deeper study. (In some environments this isn't as bad as it might sound if the salt master port isn't externally accessible and you trust everyone on your network.)
|
||||
|
||||
```shell
|
||||
[root@kubernetes-master] $ cat /etc/salt/master.d/auto-accept.conf
|
||||
open_mode: True
|
||||
auto_accept: True
|
||||
```
|
||||
|
||||
## Salt minion configuration
|
||||
|
||||
Each minion in the salt cluster has an associated configuration that instructs the salt-master how to provision the required resources on the machine.
|
||||
|
||||
An example file is presented below using the Vagrant based environment.
|
||||
|
||||
```shell
|
||||
[root@kubernetes-master] $ cat /etc/salt/minion.d/grains.conf
|
||||
grains:
|
||||
etcd_servers: $MASTER_IP
|
||||
cloud: vagrant
|
||||
roles:
|
||||
- kubernetes-master
|
||||
```
|
||||
|
||||
Each hosting environment has a slightly different grains.conf file that is used to build conditional logic where required in the Salt files.
|
||||
|
||||
The following enumerates the set of defined key/value pairs that are supported today. If you add new ones, please make sure to update this list.
|
||||
|
||||
Key | Value
|
||||
-----------------------------------|----------------------------------------------------------------
|
||||
`api_servers` | (Optional) The IP address / host name where a kubelet can get read-only access to kube-apiserver
|
||||
`cbr-cidr` | (Optional) The minion IP address range used for the docker container bridge.
|
||||
`cloud` | (Optional) Which IaaS platform is used to host Kubernetes, *gce*, *azure*, *aws*, *vagrant*
|
||||
`etcd_servers` | (Optional) Comma-delimited list of IP addresses the kube-apiserver and kubelet use to reach etcd. Uses the IP of the first machine in the kubernetes_master role, or 127.0.0.1 on GCE.
|
||||
`hostnamef` | (Optional) The full host name of the machine, i.e. uname -n
|
||||
`node_ip` | (Optional) The IP address to use to address this node
|
||||
`hostname_override` | (Optional) Mapped to the kubelet hostname-override
|
||||
`network_mode` | (Optional) Networking model to use among nodes: *openvswitch*
|
||||
`networkInterfaceName` | (Optional) Networking interface to use to bind addresses, default value *eth0*
|
||||
`publicAddressOverride` | (Optional) The IP address the kube-apiserver should use to bind against for external read-only access
|
||||
`roles` | (Required) 1. `kubernetes-master` means this machine is the master in the Kubernetes cluster. 2. `kubernetes-pool` means this machine is a kubernetes-node. Depending on the role, the Salt scripts will provision different resources on the machine.
|
||||
|
||||
These keys may be leveraged by the Salt sls files to branch behavior.
|
||||
|
||||
In addition, a cluster may be running a Debian based operating system or Red Hat based operating system (Centos, Fedora, RHEL, etc.). As a result, it's important to sometimes distinguish behavior based on operating system using if branches like the following.
|
||||
|
||||
```liquid
|
||||
|
||||
{% if grains['os_family'] == 'RedHat' %}
|
||||
// something specific to a RedHat environment (Centos, Fedora, RHEL) where you may use yum, systemd, etc.
|
||||
{% else %}
|
||||
// something specific to Debian environment (apt-get, initd)
|
||||
{% endif %}
|
||||
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
When configuring default arguments for processes, it's best to avoid the use of EnvironmentFiles (Systemd in Red Hat environments) or init.d files (Debian distributions) to hold default values that should be common across operating system environments. This helps keep our Salt template files easy to understand for editors who may not be familiar with the particulars of each distribution.
|
||||
|
||||
## Future enhancements (Networking)
|
||||
|
||||
Per pod IP configuration is provider-specific, so when making networking changes, it's important to sandbox these as all providers may not use the same mechanisms (iptables, openvswitch, etc.)
|
||||
|
||||
We should define a grains.conf key that captures more specifically what network configuration environment is being used to avoid future confusion across providers.
|
||||
Reference in New Issue
Block a user