From 29b3f2b99a720bf209c4795f3a3d6623da43e10a Mon Sep 17 00:00:00 2001 From: Stephen Gordon Date: Sat, 25 Feb 2017 16:12:07 -0500 Subject: [PATCH 01/13] Provide correct location for KUBE_ETCD_SERVERS Provide correct location for KUBE_ETCD_SERVERS configuration key. It was previously listed as being in /etc/kubernetes/config but is actually in /etc/kubernetes/apiserver. Related: https://github.com/kubernetes/kubernetes.github.io/issues/1600 --- docs/getting-started-guides/centos/centos_manual_config.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/centos/centos_manual_config.md b/docs/getting-started-guides/centos/centos_manual_config.md index 031a5bb3f5..b64af60475 100644 --- a/docs/getting-started-guides/centos/centos_manual_config.md +++ b/docs/getting-started-guides/centos/centos_manual_config.md @@ -61,9 +61,6 @@ echo "192.168.121.9 centos-master * Edit /etc/kubernetes/config which will be the same on all hosts to contain: ```shell -# Comma separated list of nodes in the etcd cluster -KUBE_ETCD_SERVERS="--etcd-servers=http://centos-master:2379" - # logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" @@ -111,6 +108,9 @@ KUBE_API_PORT="--port=8080" # Port kubelets listen on KUBELET_PORT="--kubelet-port=10250" +# Comma separated list of nodes in the etcd cluster +KUBE_ETCD_SERVERS="--etcd-servers=http://centos-master:2379" + # Address range to use for services KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" From 650d632519d1e03cf113825448fd4f2f09a9c0ad Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Mon, 27 Feb 2017 14:16:14 -0800 Subject: [PATCH 02/13] Move Compute Resources topic to Concepts. (#2410) --- _data/concepts.yml | 1 + .../manage-compute-resources-container.md | 430 ++++++++++++++++++ docs/user-guide/compute-resources.md | 366 +-------------- 3 files changed, 433 insertions(+), 364 deletions(-) create mode 100644 docs/concepts/configuration/manage-compute-resources-container.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 56a556a801..bd187c6cee 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -35,6 +35,7 @@ toc: - title: Configuration section: - docs/concepts/configuration/container-command-args.md + - docs/concepts/configuration/manage-compute-resources-container.md - title: Policies section: diff --git a/docs/concepts/configuration/manage-compute-resources-container.md b/docs/concepts/configuration/manage-compute-resources-container.md new file mode 100644 index 0000000000..2754260d65 --- /dev/null +++ b/docs/concepts/configuration/manage-compute-resources-container.md @@ -0,0 +1,430 @@ +--- +title: Managing Compute Resources for Containers +--- + +{% capture overview %} + +When you specify a [Pod](/docs/user-guide/pods), you can optionally specify how +much CPU and memory (RAM) each Container needs. When Containers have resource +requests specified, the scheduler can make better decisions about which nodes to +place Pods on. And when Containers have their limits specified, contention for +resources on a node can be handled in a specified manner. For more details about +the difference between requests and limits, see +[Resource QoS](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md). + +{% endcapture %} + + +{% capture body %} + +## Resource types + +*CPU* and *memory* are each a *resource type*. A resource type has a base unit. +CPU is specified in units of cores, and memory is specified in units of bytes. + +CPU and memory are collectively referred to as *compute resources*, or just +*resources*. Compute +resources are measurable quantities that can be requested, allocated, and +consumed. They are distinct from +[API resources](/docs/api/). API resources, such as Pods and +[Services](/docs/user-guide/services) are objects that can be read and modified +through the Kubernetes API server. + +## Resource requests and limits of Pod and Container + +Each Container of a Pod can specify one or more of the following: + +* `spec.containers[].resources.limits.cpu` +* `spec.containers[].resources.limits.memory` +* `spec.containers[].resources.requests.cpu` +* `spec.containers[].resources.requests.memory` + +Although requests and limits can only be specified on individual Containers, it +is convenient to talk about Pod resource requests and limits. A +*Pod resource request/limit* for a particular resource type is the sum of the +resource requests/limits of that type for each Container in the Pod. + +## Meaning of CPU + +Limits and requests for CPU resources are measured in *cpu* units. +One cpu, in Kubernetes, is equivalent to: + +- 1 AWS vCPU +- 1 GCP Core +- 1 Azure vCore +- 1 *Hyperthread* on a bare-metal Intel processor with Hyperthreading + +Fractional requests are allowed. A Container with +`spec.containers[].resources.requests.cpu` of `0.5` is guaranteed half as much +CPU as one that asks for 1 CPU. The expression `0.1` is equivalent to the +expression `100m`, which can be read as "one hundred millicpu". Some people say +"one hundred millicores", and this is understood to mean the same thing. A +request with a decimal point, like `0.1`, is converted to `100m` by the API, and +precision finer than `1m` is not allowed. For this reason, the form `100m` might +be preferred. + +CPU is always requested as an absolute quantity, never as a relative quantity; +0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine. + +## Meaning of memory + +Limits and requests for `memory` are measured in bytes. You can express memory as +a plain integer or as a fixed-point integer using one of these SI suffixes: +E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, +Mi, Ki. For example, the following represent roughly the same value: + +```shell +128974848, 129e6, 129M, 123Mi +``` + +Here's an example. +The following Pod has two Containers. Each Container has a request of 0.25 cpu +and 64MiB (226 bytes) of memory Each Container has a limit of 0.5 +cpu and 128MiB of memory. You can say the Pod has a request of 0.5 cpu and 128 +MiB of memory, and a limit of 1 core and 256MiB of memory. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: frontend +spec: + containers: + - name: db + image: mysql + resources: + requests: + memory: "64Mi" + cpu: "250m" + limits: + memory: "128Mi" + cpu: "500m" + - name: wp + image: wordpress + resources: + requests: + memory: "64Mi" + cpu: "250m" + limits: + memory: "128Mi" + cpu: "500m" +``` + +## How Pods with resource requests are scheduled + +When you create a Pod, the Kubernetes scheduler selects a node for the Pod to +run on. Each node has a maximum capacity for each of the resource types: the +amount of CPU and memory it can provide for Pods. The scheduler ensures that, +for each resource type, the sum of the resource requests of the scheduled +Containers is less than the capacity of the node. Note that although actual memory +or CPU resource usage on nodes is very low, the scheduler still refuses to place +a Pod on a node if the capacity check fails. This protects against a resource +shortage on a node when resource usage later increases, for example, during a +daily peak in request rate. + +## How Pods with resource limits are run + +When the kubelet starts a Container of a Pod, it passes the CPU and memory limits +to the container runtime. + +When using Docker: + +- The `spec.containers[].resources.requests.cpu` is converted to its core value, + which is potentially fractional, and multiplied by 1024. This number is used + as the value of the + [`--cpu-shares`](https://docs.docker.com/engine/reference/run/#/cpu-share-constraint) + flag in the `docker run` command. + +- The `spec.containers[].resources.limits.cpu` is converted to its millicore value, + multiplied by 100000, and then divided by 1000. This number is used as the value + of the [`--cpu-quota`](https://docs.docker.com/engine/reference/run/#/cpu-quota-constraint) + flag in the `docker run` command. he [`--cpu-period`] flag is set to 100000, + which represents the default 100ms period for measuring quota usage. The + kubelet enforces cpu limits if it is started with the + [`--cpu-cfs-quota`] flag set to true. As of Kubernetes version 1.2, this flag + defaults to true. + +- The `spec.containers[].resources.limits.memory` is converted to an integer, and + used as the value of the + [`--memory`](https://docs.docker.com/engine/reference/run/#/user-memory-constraints) + flag in the `docker run` command. + +If a Container exceeds its memory limit, it might be terminated. If it is +restartable, the kubelet will restart it, as with any other type of runtime +failure. + +If a Container exceeds its memory request, it is likely that its Pod will +be evicted whenever the node runs out of memory. + +A Container might or might not be allowed to exceed its CPU limit for extended +periods of time. However, it will not be killed for excessive CPU usage. + +To determine whether a Container cannot be scheduled or is being killed due to +resource limits, see the +[Troubleshooting](#troubleshooting) section. + +## Monitoring compute resource usage + +The resource usage of a Pod is reported as part of the Pod status. + +If [optional monitoring](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/README.md) +is configured for your cluster, then Pod resource usage can be retrieved from +the monitoring system. + +## Troubleshooting + +### My Pods are pending with event message failedScheduling + +If the scheduler cannot find any node where a Pod can fit, the Pod remains +unscheduled until a place can be found. An event is produced each time the +scheduler fails to find a place for the Pod, like this: + +```shell +$ kubectl describe pod frontend | grep -A 3 Events +Events: + FirstSeen LastSeen Count From Subobject PathReason Message + 36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others +``` + +In the preceding example, the Pod named "frontend" fails to be scheduled due to +insufficient CPU resource on the node. Similar error messages can also suggest +failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod +is pending with a message of this type, there are several things to try: + +- Add more nodes to the cluster. +- Terminate unneeded Pods to make room for pending Pods. +- Check that the Pod is not larger than all the nodes. For example, if all the + nodes have a capacity of `cpu: 1`, then a Pod with a limit of `cpu: 1.1` will + never be scheduled. + +You can check node capacities and amounts allocated with the +`kubectl describe nodes` command. For example: + +```shell +$ kubectl.sh describe nodes e2e-test-minion-group-4lw4 +Name: e2e-test-minion-group-4lw4 +[ ... lines removed for clarity ...] +Capacity: + alpha.kubernetes.io/nvidia-gpu: 0 + cpu: 2 + memory: 7679792Ki + pods: 110 +Allocatable: + alpha.kubernetes.io/nvidia-gpu: 0 + cpu: 1800m + memory: 7474992Ki + pods: 110 +[ ... lines removed for clarity ...] +Non-terminated Pods: (5 in total) + Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits + --------- ---- ------------ ---------- --------------- ------------- + kube-system fluentd-gcp-v1.38-28bv1 100m (5%) 0 (0%) 200Mi (2%) 200Mi (2%) + kube-system kube-dns-3297075139-61lj3 260m (13%) 0 (0%) 100Mi (1%) 170Mi (2%) + kube-system kube-proxy-e2e-test-... 100m (5%) 0 (0%) 0 (0%) 0 (0%) + kube-system monitoring-influxdb-grafana-v4-z1m12 200m (10%) 200m (10%) 600Mi (8%) 600Mi (8%) + kube-system node-problem-detector-v0.1-fj7m3 20m (1%) 200m (10%) 20Mi (0%) 100Mi (1%) +Allocated resources: + (Total limits may be over 100 percent, i.e., overcommitted.) + CPU Requests CPU Limits Memory Requests Memory Limits + ------------ ---------- --------------- ------------- + 680m (34%) 400m (20%) 920Mi (12%) 1070Mi (14%) +``` + +In the preceding output, you can see that if a Pod requests more than 1120m +CPUs or 6.23Gi of memory, it will not fit on the node. + +By looking at the `Pods` section, you can see which Pods are taking up space on +the node. + +The amount of resources available to Pods is less than the node capacity, because +system daemons use a portion of the available resources. The `allocatable` field +[NodeStatus](/docs/resources-reference/v1.5/#nodestatus-v1) +gives the amount of resources that are available to Pods. For more information, see +[Node Allocatable Resources](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md). + +The [resource quota](/docs/admin/resourcequota/) feature can be configured +to limit the total amount of resources that can be consumed. If used in conjunction +with namespaces, it can prevent one team from hogging all the resources. + +### My Container is terminated + +Your Container might get terminated because it is resource-starved. To check +whether a Container is being killed because it is hitting a resource limit, call +`kubectl describe pod` on the Pod of interest: + +```shell +[12:54:41] $ ./cluster/kubectl.sh describe pod simmemleak-hra99 +Name: simmemleak-hra99 +Namespace: default +Image(s): saadali/simmemleak +Node: kubernetes-node-tf0f/10.240.216.66 +Labels: name=simmemleak +Status: Running +Reason: +Message: +IP: 10.244.2.75 +Replication Controllers: simmemleak (1/1 replicas created) +Containers: + simmemleak: + Image: saadali/simmemleak + Limits: + cpu: 100m + memory: 50Mi + State: Running + Started: Tue, 07 Jul 2015 12:54:41 -0700 + Last Termination State: Terminated + Exit Code: 1 + Started: Fri, 07 Jul 2015 12:54:30 -0700 + Finished: Fri, 07 Jul 2015 12:54:33 -0700 + Ready: False + Restart Count: 5 +Conditions: + Type Status + Ready False +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {scheduler } scheduled Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD created Created with docker id 6a41280f516d + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD started Started with docker id 6a41280f516d + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} spec.containers{simmemleak} created Created with docker id 87348f12526a +``` + +In the preceding example, the `Restart Count: 5` indicates that the `simmemleak` +Container in the Pod was terminated and restarted five times. + +You can call `get pod` with the `-o go-template=...` option to fetch the status +of previously terminated Containers: + +```shell{% raw %} +[13:59:01] $ ./cluster/kubectl.sh get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-60xbc +Container Name: simmemleak +LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]{% endraw %} +``` + +You can see that the Container was terminated because of `reason:OOM Killed`, +where `OOM` stands for Out Of Memory. + +## Opaque integer resources (Alpha feature) + +Kubernetes version 1.5 introduces Opaque integer resources. Opaque +integer resources allow cluster operators to advertise new node-level +resources that would be otherwise unknown to the system. + +Users can consume these resources in Pod specs just like CPU and memory. +The scheduler takes care of the resource accounting so that no more than the +available amount is simultaneously allocated to Pods. + +**Note:** Opaque integer resources are Alpha in Kubernetes version 1.5. +Only resource accounting is implemented; node-level isolation is still +under active development. + +Opaque integer resources are resources that begin with the prefix +`pod.alpha.kubernetes.io/opaque-int-resource-`. The API server +restricts quantities of these resources to whole numbers. Examples of +_valid_ quantities are `3`, `3000m` and `3Ki`. Examples of _invalid_ +quantities are `0.5` and `1500m`. + +There are two steps required to use opaque integer resources. First, the +cluster operator must advertise a per-node opaque resource on one or more +nodes. Second, users must request the opaque resource in Pods. + +To advertise a new opaque integer resource, the cluster operator should +submit a `PATCH` HTTP request to the API server to specify the available +quantity in the `status.capacity` for a node in the cluster. After this +operation, the node's `status.capacity` will include a new resource. The +`status.allocatable` field is updated automatically with the new resource +asynchronously by the kubelet. Note that because the scheduler uses the +node `status.allocatable` value when evaluating Pod fitness, there may +be a short delay between patching the node capacity with a new resource and the +first pod that requests the resource to be scheduled on that node. + +**Example:** + +Here is an HTTP request that advertises five "foo" resources on node `k8s-node-1`. + +```http +PATCH /api/v1/nodes/k8s-node-1/status HTTP/1.1 +Accept: application/json +Content-Type: application/json-patch+json +Host: k8s-master:8080 + +[ + { + "op": "add", + "path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-foo", + "value": "5" + } +] +``` + +**Note**: In the preceding request, `~1` is the encoding for the character `/` +in the patch path. The operation path value in JSON-Patch is interpreted as a +JSON-Pointer. For more details, see +[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3). + +To consume an opaque resource in a Pod, include the name of the opaque +resource as a key in the `spec.containers[].resources.requests` map. + +The Pod is scheduled only if all of the resource requests are +satisfied, including cpu, memory and any opaque resources. The Pod will +remain in the `PENDING` state as long as the resource request cannot be met by +any node. + +**Example:** + +The Pod below requests 2 cpus and 1 "foo" (an opaque resource.) + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: my-pod +spec: + containers: + - name: my-container + image: myimage + resources: + requests: + cpu: 2 + pod.alpha.kubernetes.io/opaque-int-resource-foo: 1 +``` + +## Planned Improvements + +Kubernetes version 1.5 only allows resource quantities to be specified on a +Container. It is planned to improve accounting for resources that are shared by +all Containers in a Pod, such as +[emptyDir volumes](/docs/user-guide/volumes/#emptydir). + +Kubernetes version 1.5 only supports Container requests and limits for CPU and +memory. It is planned to add new resource types, including a node disk space +resource, and a framework for adding custom +[resource types](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/resources.md). + +Kubernetes supports overcommitment of resources by supporting multiple levels of +[Quality of Service](http://issue.k8s.io/168). + +In Kubernetes version 1.5, one unit of CPU means different things on different +cloud providers, and on different machine types within the same cloud providers. +For example, on AWS, the capacity of a node is reported in +[ECUs](http://aws.amazon.com/ec2/faqs/), while in GCE it is reported in logical +cores. We plan to revise the definition of the cpu resource to allow for more +consistency across providers and platforms. + +{% endcapture %} + + +{% capture whatsnext %} + +* Get hands-on experience +[assigning CPU and RAM resources to a container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/). + +* [Container](/docs/api-reference/v1/definitions/#_v1_container) + +* [ResourceRequirements](/docs/resources-reference/v1.5/#resourcerequirements-v1) + +{% endcapture %} + +{% include templates/concept.md %} + diff --git a/docs/user-guide/compute-resources.md b/docs/user-guide/compute-resources.md index d2856f50aa..51fcaafa9d 100644 --- a/docs/user-guide/compute-resources.md +++ b/docs/user-guide/compute-resources.md @@ -5,368 +5,6 @@ assignees: title: Managing Compute Resources --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -When specifying a [pod](/docs/user-guide/pods), you can optionally specify how much CPU and memory (RAM) each -container needs. When containers have their resource requests specified, the scheduler is -able to make better decisions about which nodes to place pods on; and when containers have their -limits specified, contention for resources on a node can be handled in a specified manner. For -more details about the difference between requests and limits, please refer to -[Resource QoS](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/resource-qos.md). - -*CPU* and *memory* are each a *resource type*. A resource type has a base unit. CPU is specified -in units of cores. Memory is specified in units of bytes. - -CPU and RAM are collectively referred to as *compute resources*, or just *resources*. Compute -resources are measureable quantities which can be requested, allocated, and consumed. They are -distinct from [API resources](/docs/user-guide/working-with-resources). API resources, such as pods and -[services](/docs/user-guide/services) are objects that can be written to and retrieved from the Kubernetes API -server. - -## Resource Requests and Limits of Pod and Container - -Each container of a pod can optionally specify one or more of the following: - -* `spec.containers[].resources.limits.cpu` -* `spec.containers[].resources.limits.memory` -* `spec.containers[].resources.requests.cpu` -* `spec.containers[].resources.requests.memory`. - -Specifying resource requests and/or limits is optional. In some clusters, unset limits or requests -may be replaced with default values when a pod is created or updated. The default value depends on -how the cluster is configured. If the requests values are not specified, they are set to be equal -to the limits values by default. Please note that limits must always be greater than or equal to -requests. - -Although requests/limits can only be specified on individual containers, it is convenient to talk -about pod resource requests/limits. A *pod resource request/limit* for a particular resource -type is the sum of the resource requests/limits of that type for each container in the pod, with -unset values treated as zero (or equal to default values in some cluster configurations). - -### Meaning of CPU -Limits and requests for `cpu` are measured in cpus. -One cpu, in Kubernetes, is equivalent to: - -- 1 AWS vCPU -- 1 GCP Core -- 1 Azure vCore -- 1 *Hyperthread* on a bare-metal Intel processor with Hyperthreading - -Fractional requests are allowed. A container with `spec.containers[].resources.requests.cpu` of `0.5` will -be guaranteed half as much CPU as one that asks for `1`. The expression `0.1` is equivalent to the expression -`100m`, which can be read as "one hundred millicpu" (some may say "one hundred millicores", and this is understood -to mean the same thing when talking about Kubernetes). A request with a decimal point, like `0.1` is converted to -`100m` by the API, and precision finer than `1m` is not allowed. For this reason, the form `100m` may be preferred. - -CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same amount of cpu on a single -core, dual core, or 48 core machine. - -# Meaning of Memory - -Limits and requests for `memory` are measured in bytes. -Memory can be expressed a plain integer or as fixed-point integers with one of these SI suffixes (E, P, T, G, M, K) -or their power-of-two equivalents (Ei, Pi, Ti, Gi, Mi, Ki). For example, the following represent roughly the same value: -`128974848`, `129e6`, `129M` , `123Mi`. - -### Example -The following pod has two containers. Each has a request of 0.25 core of cpu and 64MiB -(226 bytes) of memory and a limit of 0.5 core of cpu and 128MiB of memory. The pod can -be said to have a request of 0.5 core and 128 MiB of memory and a limit of 1 core and 256MiB of -memory. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: frontend -spec: - containers: - - name: db - image: mysql - resources: - requests: - memory: "64Mi" - cpu: "250m" - limits: - memory: "128Mi" - cpu: "500m" - - name: wp - image: wordpress - resources: - requests: - memory: "64Mi" - cpu: "250m" - limits: - memory: "128Mi" - cpu: "500m" -``` - -## How Pods with Resource Requests are Scheduled - -When a pod is created, the Kubernetes scheduler selects a node for the pod to -run on. Each node has a maximum capacity for each of the resource types: the -amount of CPU and memory it can provide for pods. The scheduler ensures that, -for each resource type (CPU and memory), the sum of the resource requests of the -containers scheduled to the node is less than the capacity of the node. Note -that although actual memory or CPU resource usage on nodes is very low, the -scheduler will still refuse to place pods onto nodes if the capacity check -fails. This protects against a resource shortage on a node when resource usage -later increases, such as due to a daily peak in request rate. - -## How Pods with Resource Limits are Run - -When kubelet starts a container of a pod, it passes the CPU and memory limits to the container -runner (Docker or rkt). - -When using Docker: - -- The `spec.containers[].resources.requests.cpu` is converted to its core value (potentially fractional), - and multiplied by 1024, and used as the value of the [`--cpu-shares`](https://docs.docker.com/engine/reference/run/#/cpu-share-constraint) - flag to the `docker run` command. -- The `spec.containers[].resources.limits.cpu` is converted to its millicore value, - multiplied by 100000, and then divided by 1000, and used as the value of the [`--cpu-quota`]( - https://docs.docker.com/engine/reference/run/#/cpu-quota-constraint) flag to the `docker run` - command. The [`--cpu-period`] flag is set to 100000 which represents the default 100ms period - for measuring quota usage. The kubelet enforces cpu limits if it was started with the - [`--cpu-cfs-quota`] flag set to true. As of version 1.2, this flag will now default to true. -- The `spec.containers[].resources.limits.memory` is converted to an integer, and used as the value - of the [`--memory`](https://docs.docker.com/engine/reference/run/#/user-memory-constraints) flag - to the `docker run` command. - -**TODO: document behavior for rkt** - -If a container exceeds its memory limit, it may be terminated. If it is restartable, it will be -restarted by kubelet, as will any other type of runtime failure. - -A container may or may not be allowed to exceed its CPU limit for extended periods of time. -However, it will not be killed for excessive CPU usage. - -To determine if a container cannot be scheduled or is being killed due to resource limits, see the -"Troubleshooting" section below. - -## Monitoring Compute Resource Usage - -The resource usage of a pod is reported as part of the Pod status. - -If [optional monitoring](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/README.md) is configured for your cluster, -then pod resource usage can be retrieved from the monitoring system. - -## Troubleshooting - -### My pods are pending with event message failedScheduling - -If the scheduler cannot find any node where a pod can fit, then the pod will remain unscheduled -until a place can be found. An event will be produced each time the scheduler fails to find a -place for the pod, like this: - -```shell -$ kubectl describe pod frontend | grep -A 3 Events -Events: - FirstSeen LastSeen Count From Subobject PathReason Message - 36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others -``` - -In the case shown above, the pod "frontend" fails to be scheduled due to insufficient -CPU resource on the node. Similar error messages can also suggest failure due to insufficient -memory (PodExceedsFreeMemory). In general, if a pod or pods are pending with this message and -alike, then there are several things to try: - -- Add more nodes to the cluster. -- Terminate unneeded pods to make room for pending pods. -- Check that the pod is not larger than all the nodes. For example, if all the nodes -have a capacity of `cpu: 1`, then a pod with a limit of `cpu: 1.1` will never be scheduled. - -You can check node capacities and amounts allocated with the `kubectl describe nodes` command. -For example: - -```shell -$ kubectl describe nodes gke-cluster-4-386701dd-node-ww4p -Name: gke-cluster-4-386701dd-node-ww4p -[ ... lines removed for clarity ...] -Capacity: - cpu: 1 - memory: 464Mi - pods: 40 -Allocated resources (total requests): - cpu: 910m - memory: 2370Mi - pods: 4 -[ ... lines removed for clarity ...] -Pods: (4 in total) - Namespace Name CPU(milliCPU) Memory(bytes) - frontend webserver-ffj8j 500 (50% of total) 2097152000 (50% of total) - kube-system fluentd-cloud-logging-gke-cluster-4-386701dd-node-ww4p 100 (10% of total) 209715200 (5% of total) - kube-system kube-dns-v8-qopgw 310 (31% of total) 178257920 (4% of total) -TotalResourceLimits: - CPU(milliCPU): 910 (91% of total) - Memory(bytes): 2485125120 (59% of total) -[ ... lines removed for clarity ...] -``` - -Here you can see from the `Allocated resources` section that that a pod which ask for more than -90 millicpus or more than 1341MiB of memory will not be able to fit on this node. - -Looking at the `Pods` section, you can see which pods are taking up space on the node. - -The [resource quota](/docs/admin/resourcequota/) feature can be configured -to limit the total amount of resources that can be consumed. If used in conjunction -with namespaces, it can prevent one team from hogging all the resources. - -### My container is terminated - -Your container may be terminated because it's resource-starved. To check if a container is being killed because it is hitting a resource limit, call `kubectl describe pod` -on the pod you are interested in: - -```shell -[12:54:41] $ ./cluster/kubectl.sh describe pod simmemleak-hra99 -Name: simmemleak-hra99 -Namespace: default -Image(s): saadali/simmemleak -Node: kubernetes-node-tf0f/10.240.216.66 -Labels: name=simmemleak -Status: Running -Reason: -Message: -IP: 10.244.2.75 -Replication Controllers: simmemleak (1/1 replicas created) -Containers: - simmemleak: - Image: saadali/simmemleak - Limits: - cpu: 100m - memory: 50Mi - State: Running - Started: Tue, 07 Jul 2015 12:54:41 -0700 - Last Termination State: Terminated - Exit Code: 1 - Started: Fri, 07 Jul 2015 12:54:30 -0700 - Finished: Fri, 07 Jul 2015 12:54:33 -0700 - Ready: False - Restart Count: 5 -Conditions: - Type Status - Ready False -Events: - FirstSeen LastSeen Count From SubobjectPath Reason Message - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {scheduler } scheduled Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD created Created with docker id 6a41280f516d - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD started Started with docker id 6a41280f516d - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} spec.containers{simmemleak} created Created with docker id 87348f12526a -``` - -The `Restart Count: 5` indicates that the `simmemleak` container in this pod was terminated and restarted 5 times. - -You can call `get pod` with the `-o go-template=...` option to fetch the status of previously terminated containers: - -```shell{% raw %} -[13:59:01] $ ./cluster/kubectl.sh get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-60xbc -Container Name: simmemleak -LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]{% endraw %} -``` - -We can see that this container was terminated because `reason:OOM Killed`, where *OOM* stands for Out Of Memory. - -## Opaque Integer Resources (Alpha Feature) - -Kubernetes version 1.5 introduces Opaque integer resources. Opaque -integer resources allow cluster operators to advertise new node-level -resources that would be otherwise unknown to the system. - -Users can consume these resources in pod specs just like CPU and memory. -The scheduler takes care of the resource accounting so that no more than the -available amount is simultaneously allocated to pods. - -**Note:** Opaque integer resources are Alpha in Kubernetes version 1.5. -Only resource accounting is implemented; node-level isolation is still -under active development. - -Opaque integer resources are resources that begin with the prefix -`pod.alpha.kubernetes.io/opaque-int-resource-`. The API server -restricts quantities of these resources to whole numbers. Examples of -_valid_ quantities are `3`, `3000m` and `3Ki`. Examples of _invalid_ -quantities are `0.5` and `1500m`. - -There are two steps required to use opaque integer resources. First, the -cluster operator must advertise a per-node opaque resource on one or more -nodes. Second, users must request the opaque resource in pods. - -To advertise a new opaque integer resource, the cluster operator should -submit a `PATCH` HTTP request to the API server to specify the available -quantity in the `status.capacity` for a node in the cluster. After this -operation, the node's `status.capacity` will include a new resource. The -`status.allocatable` field is updated automatically with the new resource -asychronously by the Kubelet. Note that since the scheduler uses the -node `status.allocatable` value when evaluating pod fitness, there may -be a short delay between patching the node capacity with a new resource and the -first pod that requests the resource to be scheduled on that node. - -**Example:** - -The HTTP request below advertises 5 "foo" resources on node `k8s-node-1`. - -_NOTE: `~1` is the encoding for the character `/` in the patch path. -The operation path value in JSON-Patch is interpreted as a JSON-Pointer. -For more details, please refer to -[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3)._ - -```http -PATCH /api/v1/nodes/k8s-node-1/status HTTP/1.1 -Accept: application/json -Content-Type: application/json-patch+json -Host: k8s-master:8080 - -[ - { - "op": "add", - "path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-foo", - "value": "5" - } -] -``` - -To consume opaque resources in pods, include the name of the opaque -resource as a key in the `spec.containers[].resources.requests` map. - -The pod will be scheduled only if all of the resource requests are -satisfied (including cpu, memory and any opaque resources.) The pod will -remain in the `PENDING` state while the resource request cannot be met by any -node. - -**Example:** - -The pod below requests 2 cpus and 1 "foo" (an opaque resource.) - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: my-pod -spec: - containers: - - name: my-container - image: myimage - resources: - requests: - cpu: 2 - pod.alpha.kubernetes.io/opaque-int-resource-foo: 1 -``` - -## Planned Improvements - -The current system only allows resource quantities to be specified on a container. -It is planned to improve accounting for resources which are shared by all containers in a pod, -such as [EmptyDir volumes](/docs/user-guide/volumes/#emptydir). - -The current system only supports container requests and limits for CPU and Memory. -It is planned to add new resource types, including a node disk space -resource, and a framework for adding custom [resource types](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/resources.md). - -Kubernetes supports overcommitment of resources by supporting multiple levels of [Quality of Service](http://issue.k8s.io/168). - -Currently, one unit of CPU means different things on different cloud providers, and on different -machine types within the same cloud providers. For example, on AWS, the capacity of a node -is reported in [ECUs](http://aws.amazon.com/ec2/faqs/), while in GCE it is reported in logical -cores. We plan to revise the definition of the cpu resource to allow for more consistency -across providers and platforms. +[Managing Compute Resources for Containers](/docs/concepts/configuration/manage-compute-resources-container/) From 11f3e0fec14504ff2f86b02953c5275f2febfc7c Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Mon, 27 Feb 2017 15:24:11 +0800 Subject: [PATCH 03/13] Delete the parameter "--google-json-key string" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit “# kube-scheduler -help”can not find --google-json-key option # kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-23T00:04:39Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-22T23:56:57Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} --- docs/admin/kube-scheduler.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/admin/kube-scheduler.md b/docs/admin/kube-scheduler.md index 15e47d2f46..91ffc303aa 100644 --- a/docs/admin/kube-scheduler.md +++ b/docs/admin/kube-scheduler.md @@ -36,7 +36,6 @@ DynamicKubeletConfig=true|false (ALPHA - default=false) DynamicVolumeProvisioning=true|false (ALPHA - default=true) ExperimentalHostUserNamespaceDefaulting=true|false (ALPHA - default=false) StreamingProxyRedirects=true|false (ALPHA - default=false) - --google-json-key string The Google Cloud Platform Service Account JSON Key to use for authentication. --hard-pod-affinity-symmetric-weight int RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule corresponding to every RequiredDuringScheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit PreferredDuringScheduling affinity rule. (default 1) --kube-api-burst int32 Burst to use while talking with Kubernetes apiserver (default 100) --kube-api-content-type string Content type of requests sent to apiserver. (default "application/vnd.kubernetes.protobuf") From d30c4a7c7885a6a2e8e4ca5bcf732cc804112382 Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Sun, 26 Feb 2017 11:39:46 +0800 Subject: [PATCH 04/13] Add diagnose tips when you face problem. --- docs/getting-started-guides/kubeadm.md | 32 ++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index b5a9845b44..229ef993f0 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -195,6 +195,38 @@ Once a pod network has been installed, you can confirm that it is working by che And once the `kube-dns` pod is up and running, you can continue by joining your nodes. +If you see following status +``` +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system canal-node-f0lqp 2/3 RunContainerError 2 48s +``` +Or + +``` +kube-system canal-node-77d0h 2/3 CrashLoopBackOff 3 3m +kube-system kube-dns-2924299975-7q1vq 0/4 ContainerCreating 0 15m +``` +The three status ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` very common. + +You may have trouble in configure. to diagnose what happened. you can using ```kubectl describe -n kube-system po {YOUR_POD_NAME}``` to check what's in logs. do not using kubectl logs. you will got +``` +# kubectl logs -n kube-system canal-node-f0lqp +Error from server (BadRequest): the server rejected our request for an unknown reason (get pods canal-node-f0lqp) +``` +The kubectl describe will gave you more details about the logs + +``` +# kubectl describe -n kube-system po kube-dns-2924299975-1l2t7 + 2m 2m 1 {kubelet nac} spec.containers{flannel} Warning Failed Failed to start container with docker id 927e7ccdc32b with error: Error response from daemon: {"message":"chown /etc/resolv.conf: operation not permitted"} + +``` +Or +``` + 6m 1m 191 {kubelet nac} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1l2t7_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1l2t7_kube-system(dee8ef21-fbcb-11e6-ba19-38d547e0006a)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod" +``` + +Then you can do some search with google, and you will find solutions. + ### (4/4) Joining your nodes The nodes are where your workloads (containers and pods, etc) run. From 06104e35a3d89d3b00139d165a0a6a907be7e5f5 Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Tue, 28 Feb 2017 08:01:23 +0800 Subject: [PATCH 05/13] Update grammar for kubeadm.md, thanks @chenopis --- docs/getting-started-guides/kubeadm.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index 229ef993f0..aba4e1d52d 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -195,37 +195,47 @@ Once a pod network has been installed, you can confirm that it is working by che And once the `kube-dns` pod is up and running, you can continue by joining your nodes. -If you see following status +If you see the following status ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system canal-node-f0lqp 2/3 RunContainerError 2 48s ``` + Or ``` kube-system canal-node-77d0h 2/3 CrashLoopBackOff 3 3m kube-system kube-dns-2924299975-7q1vq 0/4 ContainerCreating 0 15m ``` -The three status ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` very common. +The three statuses ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` are very common. + +You may have trouble in the configuration. To help diagnose what happened, you can use the following command to check what is in the logs: + +```bash +kubectl describe -n kube-system po {YOUR_POD_NAME} +``` + +Do not using kubectl logs. you will got the following error: -You may have trouble in configure. to diagnose what happened. you can using ```kubectl describe -n kube-system po {YOUR_POD_NAME}``` to check what's in logs. do not using kubectl logs. you will got ``` # kubectl logs -n kube-system canal-node-f0lqp Error from server (BadRequest): the server rejected our request for an unknown reason (get pods canal-node-f0lqp) ``` -The kubectl describe will gave you more details about the logs + +The ```kubectl describe``` gives you more details about the logs ``` # kubectl describe -n kube-system po kube-dns-2924299975-1l2t7 2m 2m 1 {kubelet nac} spec.containers{flannel} Warning Failed Failed to start container with docker id 927e7ccdc32b with error: Error response from daemon: {"message":"chown /etc/resolv.conf: operation not permitted"} ``` + Or ``` 6m 1m 191 {kubelet nac} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1l2t7_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1l2t7_kube-system(dee8ef21-fbcb-11e6-ba19-38d547e0006a)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod" ``` -Then you can do some search with google, and you will find solutions. +You can then do some Google searches on the error messages, which may help you to find some solutions. ### (4/4) Joining your nodes From 1cbd24e888ae2f952bf8cb2db5d374f934a6f15f Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Tue, 28 Feb 2017 09:13:50 +0800 Subject: [PATCH 06/13] Merge two pieces to one words --- docs/getting-started-guides/kubeadm.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index aba4e1d52d..24c9498e44 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -195,34 +195,32 @@ Once a pod network has been installed, you can confirm that it is working by che And once the `kube-dns` pod is up and running, you can continue by joining your nodes. -If you see the following status + +You may have trouble in the configuration if you see the following statuses + ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system canal-node-f0lqp 2/3 RunContainerError 2 48s -``` - -Or - -``` kube-system canal-node-77d0h 2/3 CrashLoopBackOff 3 3m kube-system kube-dns-2924299975-7q1vq 0/4 ContainerCreating 0 15m ``` + The three statuses ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` are very common. -You may have trouble in the configuration. To help diagnose what happened, you can use the following command to check what is in the logs: +To help diagnose what happened, you can use the following command to check what is in the logs: ```bash kubectl describe -n kube-system po {YOUR_POD_NAME} ``` -Do not using kubectl logs. you will got the following error: +Do not using kubectl logs. You will got the following error: ``` # kubectl logs -n kube-system canal-node-f0lqp Error from server (BadRequest): the server rejected our request for an unknown reason (get pods canal-node-f0lqp) ``` -The ```kubectl describe``` gives you more details about the logs +The ```kubectl describe``` comand gives you more details about the logs ``` # kubectl describe -n kube-system po kube-dns-2924299975-1l2t7 From eb57603fcbf52f8114aa000172e482fd46e5262b Mon Sep 17 00:00:00 2001 From: xilabao Date: Sun, 26 Feb 2017 21:09:41 -0600 Subject: [PATCH 07/13] add http proxy infomation in kubeadm --- docs/admin/kubeadm.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index a43beec497..24f39217f0 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -258,6 +258,8 @@ These environment variables are a short-term solution, eventually they will be i | `KUBE_ETCD_IMAGE` | `gcr.io/google_containers/etcd-:2.2.5` | The etcd container image to use. | | `KUBE_REPO_PREFIX` | `gcr.io/google_containers` | The image prefix for all images that are used. | +If you want to use kubeadm with an http proxy, you may need to configure it to support http_proxy, https_proxy, or no_proxy. + ## Releases and release notes If you already have kubeadm installed and want to upgrade, run `apt-get update && apt-get upgrade` or `yum update` to get the latest version of kubeadm. From 3c20c4da09024eaa4874f2ed68b57fdd24e7016c Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Mon, 27 Feb 2017 17:45:20 -0800 Subject: [PATCH 08/13] Update landing pages for Tasks and Tutorials. (#2634) --- _data/tasks.yml | 2 +- docs/tasks/index.md | 6 ++++++ docs/tasks/kubectl/list-all-running-container-images.md | 2 +- docs/tutorials/index.md | 6 +++--- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/_data/tasks.yml b/_data/tasks.yml index 5dae817f8e..ab237c3474 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -3,7 +3,7 @@ abstract: "Step-by-step instructions for performing operations with Kubernetes." toc: - docs/tasks/index.md -- title: Using the Kubectl Command-Line +- title: Using the kubectl Command-Line section: - docs/tasks/kubectl/list-all-running-container-images.md - docs/tasks/kubectl/get-shell-running-container.md diff --git a/docs/tasks/index.md b/docs/tasks/index.md index d490fe5532..6bd8db2e6c 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -6,12 +6,18 @@ This section of the Kubernetes documentation contains pages that show how to do individual tasks. A task page shows how to do a single thing, typically by giving a short sequence of steps. +#### Using the kubectl Command Line + +* [Listing Alll Container Images Running in a Cluster](/docs/tasks/kubectl/list-all-running-container-images/) +* [Getting a Shell to a Running Container](/docs/tasks/kubectl/get-shell-running-container/) + #### Configuring Pods and Containers * [Defining Environment Variables for a Container](/docs/tasks/configure-pod-container/define-environment-variable-container/) * [Defining a Command and Arguments for a Container](/docs/tasks/configure-pod-container/define-command-argument-container/) * [Assigning CPU and RAM Resources to a Container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/) * [Configuring a Pod to Use a Volume for Storage](/docs/tasks/configure-pod-container/configure-volume-storage/) +* [Configuring a Pod to Use a PersistentVolume for Storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/) * [Exposing Pod Information to Containers Through Environment Variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) * [Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) * [Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) diff --git a/docs/tasks/kubectl/list-all-running-container-images.md b/docs/tasks/kubectl/list-all-running-container-images.md index 4fb64ee442..070de69004 100644 --- a/docs/tasks/kubectl/list-all-running-container-images.md +++ b/docs/tasks/kubectl/list-all-running-container-images.md @@ -1,5 +1,5 @@ --- -title: Listing all Container images running in the cluster +title: Listing All Container Images Running in a Cluster --- {% capture overview %} diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 76e42570b1..25b8cca32e 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -7,14 +7,14 @@ A tutorial shows how to accomplish a goal that is larger than a single [task](/docs/tasks/). Typically a tutorial has several sections, each of which has a sequence of steps. -#### Kubernetes Basics - * [Kubernetes Basics](/docs/tutorials/kubernetes-basics/) is an in-depth interactive tutorial that helps you understand the Kubernetes system and try out some basic Kubernetes features. -#### Stateless Applications +* [Online Training Course](https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615) * [Hello Minikube](/docs/tutorials/stateless-application/hello-minikube/) +#### Stateless Applications + * [Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) * [Using a Service to Access an Application in a Cluster](/docs/tutorials/stateless-application/expose-external-ip-address-service/) From 6f83f5070a1dd42c58259cda5308b44c39e943eb Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 28 Feb 2017 09:08:11 -0800 Subject: [PATCH 09/13] Move Guide Topic: Multi-container pods. (#2642) --- docs/user-guide/pods/multi-container.md | 170 +----------------------- 1 file changed, 2 insertions(+), 168 deletions(-) diff --git a/docs/user-guide/pods/multi-container.md b/docs/user-guide/pods/multi-container.md index 55e0e56f84..465379e51c 100644 --- a/docs/user-guide/pods/multi-container.md +++ b/docs/user-guide/pods/multi-container.md @@ -4,172 +4,6 @@ assignees: title: Creating Multi-Container Pods --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -A pod is a group of containers that are scheduled -onto the same host. Pods serve as units of scheduling, deployment, and -horizontal scaling/replication. Pods share fate, and share some resources, such -as storage volumes and IP addresses. - -## Creating a pod - -Multi-container pods must be created with the `create` command. Properties -are passed to the command as a YAML- or JSON-formatted configuration file. - -The `create` command can be used to create a pod directly, or it can create -a pod or pods through a `Deployment`. It is highly recommended that -you use a -[Deployment](/docs/user-guide/deployments/) -to create your pods. It watches for failed pods and will start up -new pods as required to maintain the specified number. - -If you don't want a Deployment to monitor your pod (e.g. your pod -is writing non-persistent data which won't survive a restart, or your pod is -intended to be very short-lived), you can create a pod directly with the -`create` command. - -### Using `create` - -Note: We recommend using a -[Deployment](/docs/user-guide/deployments/) -to create pods. You should use the instructions below only if you don't want -to create a Deployment. - -If your pod will contain more than one container, or if you don't want to -create a Deployment to manage your pod, use the -`kubectl create` command and pass a pod specification as a JSON- or -YAML-formatted configuration file. - -```shell -$ kubectl create -f FILE -``` - -Where: - -* `-f FILE` or `--filename FILE` is the name of a - [pod configuration file](#pod-configuration-file) in either JSON or YAML - format. - -A successful create request returns the pod name. Use the -[`kubectl get`](#viewing_a_pod) command to view status after creation. - -### Pod configuration file - -A pod configuration file specifies required information about the pod. -It can be formatted as YAML or as JSON, and supports the following fields: - -{% capture tabspec %}configfiles -JSON,json,pod-config.json,/docs/user-guide/pods/pod-config.json -YAML,yaml,pod-config.yaml,/docs/user-guide/pods/pod-config.yaml{% endcapture %} -{% include tabs.html %} - -Required fields are: - -* `kind`: Always `Pod`. -* `apiVersion`: Currently `v1`. -* `metadata`: An object containing: - * `name`: Required if `generateName` is not specified. The name of this pod. - It must be an - [RFC1035](https://www.ietf.org/rfc/rfc1035.txt) compatible value and be - unique within the namespace. - * `labels`: Optional. Labels are arbitrary key:value pairs that can be used - by - [Deployment](/docs/user-guide/deployments/) - and [services](/docs/user-guide/services/) for grouping and targeting - pods. - * `generateName`: Required if `name` is not set. A prefix to use to generate - a unique name. Has the same validation rules as `name`. - * `namespace`: Required. The namespace of the pod. - * `annotations`: Optional. A map of string keys and values that can be used - by external tooling to store and retrieve arbitrary metadata about - objects. -* `spec`: The pod specification. See [The `spec` schema](#the_spec_schema) for - details. - - -### The `spec` schema - -A full description of the `spec` schema is contained in the -[Kubernetes API reference](/docs/api-reference/v1/definitions/#_v1_podspec). - -The following fields are required or commonly used in the `spec` schema: - -{% capture tabspec %}specfiles -JSON,json,pod-spec-common.json,/docs/user-guide/pods/pod-spec-common.json -YAML,yaml,pod-spec-common.yaml,/docs/user-guide/pods/pod-spec-common.yaml{% endcapture %} -{% include tabs.html %} - -#### `containers[]` - -A list of containers belonging to the pod. Containers cannot be added or removed once the pod is created, and there must be at least one container in a pod. - -The `containers` object **must contain**: - -* `name`: Name of the container. It must be a DNS_LABEL and be unique within the pod. Cannot be updated. -* `image`: Docker image name. - -The `containers` object **commonly contains** the following optional properties: - -* `command[]`: The entrypoint array. Commands are not executed within a shell. The docker image's entrypoint is used if this is not provided. Cannot be updated. -* `args[]`: A command array containing arguments to the entrypoint. The docker image's `cmd` is used if this is not provided. Cannot be updated. -* `env[]`: A list of environment variables in key:value format to set in the container. Cannot be updated. - * `name`: The name of the environment variable; must be a `C_IDENTIFIER`. - * `value`: The value of the environment variable. Defaults to empty string. -* `imagePullPolicy`: The image pull policy. Accepted values are: - * `Always` - * `Never` - * `IfNotPresent`Defaults to `Always` if `:latest` tag is specified, or `IfNotPresent` otherwise. Cannot be updated. -* `ports[]`: A list of ports to expose from the container. Cannot be updated. - * `containerPort`: The port number to expose on the pod's IP address. - * `name`: The name for the port that can be referred to by services. Must be a `DNS_LABEL` and be unique without the pod. - * `protocol`: Protocol for the port. Must be UDP or TCP. Default is TCP. -* `resources`: The Compute resources required by this container. Contains: - * `cpu`: CPUs to reserve for each container. Default is whole CPUs; scale suffixes (e.g. `100m` for one hundred milli-CPUs) are supported. If the host does not have enough available resources, your pod will not be scheduled. - * `memory`: Memory to reserve for each container. Default is bytes; [binary scale suffixes](http://en.wikipedia.org/wiki/Binary_prefix) (e.g. `100Mi` for one hundred mebibytes) are supported. If the host does not have enough available resources, your pod will not be scheduled.Cannot be updated. - -#### `restartPolicy` - -Restart policy for all containers within the pod. Options are: - -* `Always` -* `OnFailure` -* `Never` - -#### `volumes[]` - -A list of volumes that can be mounted by containers belonging to the pod. You must specify a `name` and a source for each volume. The container must also include a `volumeMount` with matching `name`. Source is one of: - -* `emptyDir`: A temporary directory that shares a pod's lifetime. Contains: - * `medium`: The type of storage used to back the volume. Must be an empty string (default) or `Memory`. -* `hostPath`: A pre-existing host file or directory. This is generally used for privileged system daemons or other agents tied to the host. Contains: - * `path`: The path of the directory on the host. -* `secret`: Secret to populate volume. Secrets are used to hold sensitive information, such as passwords, OAuth tokens, and SSH keys. Learn more from [the docs on secrets](/docs/user-guide/secrets/). Contains: - * `secretName`: The name of a secret in the pod's namespace. - -The `name` must be a DNS_LABEL and unique within the pod. - - -### Sample file - -For example, the following configuration file creates two containers: a -`redis` key-value store image, and a `django` frontend image. - -{% capture tabspec %}samplefiles -JSON,json,pod-sample.json,/docs/user-guide/pods/pod-sample.json -YAML,yaml,pod-sample.yaml,/docs/user-guide/pods/pod-sample.yaml{% endcapture %} -{% include tabs.html %} - -## Viewing a pod - -{% include_relative _viewing-a-pod.md %} - -## Deleting a pod - -If you created your pod directly with `kubectl create`, use `kubectl delete`: - -```shell -$ kubectl delete pod NAME -``` - -A successful delete request returns the name of the deleted pod. +[Communicating Between Containers Running in the Same Pod](/docs/tasks/configure-pod-container/communicate-containers-same-pod/) From 8b97c4265f09ca4a520aaf475062192ddf467f3d Mon Sep 17 00:00:00 2001 From: EJ Date: Wed, 15 Feb 2017 19:05:45 -0800 Subject: [PATCH 10/13] fix link to go to pod-lifecycle page --- .../configure-liveness-readiness-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md index ff05756350..a5c3d8ee56 100644 --- a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md +++ b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md @@ -255,7 +255,7 @@ In addition to command probes and HTTP probes, Kubernetes supports {% capture whatsnext %} * Learn more about -[Container Probes](/docs/user-guide/pod-states/#container-probes). +[Container Probes](/docs/concepts/workloads/pods/pod-lifecycle/#container-probes). * Learn more about [Health Checking section](/docs/user-guide/walkthrough/k8s201/#health-checking). From 764225a7a018a7853d66690ab887516e133cde58 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Tue, 28 Feb 2017 14:58:36 +0800 Subject: [PATCH 11/13] fix the command output fix the command output --- docs/getting-started-guides/gce.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting-started-guides/gce.md b/docs/getting-started-guides/gce.md index 245de295f9..109dcb8d95 100644 --- a/docs/getting-started-guides/gce.md +++ b/docs/getting-started-guides/gce.md @@ -134,10 +134,10 @@ $ kubectl get --all-namespaces services should show a set of [services](/docs/user-guide/services) that look something like this: ```shell -NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE -default kubernetes 10.0.0.1 443/TCP 1d -kube-system kube-dns 10.0.0.2 53/TCP,53/UDP k8s-app=kube-dns 1d -kube-system kube-ui 10.0.0.3 80/TCP k8s-app=kube-ui 1d +NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) AGE +default kubernetes 10.0.0.1 443/TCP 1d +kube-system kube-dns 10.0.0.2 53/TCP,53/UDP 1d +kube-system kube-ui 10.0.0.3 80/TCP 1d ... ``` From 2ef4477df3d8a2b84baf592decbfee4c28ef6fd0 Mon Sep 17 00:00:00 2001 From: mlambert890b Date: Sat, 4 Feb 2017 00:06:16 -0800 Subject: [PATCH 12/13] mirantis_logo.png ,/images/community_logos/mirantis_logo.png updated per Mirantis request --- images/community_logos/mirantis_logo.png | Bin 11543 -> 8607 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/images/community_logos/mirantis_logo.png b/images/community_logos/mirantis_logo.png index d2c395e323e80f86b3ba57d55731ab0707023f77..4f407a37bb6a3b411f8209c789232391adb43e83 100644 GIT binary patch literal 8607 zcmb`NWl$Sm*tdhb6e#Z6;$B<}4Nh?@Qb>W~P^<(94#k7J1t{(koZ=d!KufVuq)^-; z1s?wMetPHm`rI?S=flqIIkWf9?)AIo#OP|jA;PD{2LJ#>YQWcePvg?l5r~WZbQZe1 zl6@L5y_D1paGwf{YZnawFay+HD;oIa9_IOjzc0WaDqR%U9F7_)x2{(0bImJhZmF?9>)e(_P2E-0|JWyvCf$iK^RE4V}-j!cM8u5YvD= z)Q-~`|2>5LT()4HNzIxQ#rtyu71V9A14j7g+Y9^eI3xUtH<15-35{>K(+TDp+-a0e zY)Z`e62>?(TflurVWAr5tK!GL=XQ^EJC}ZEc-eZmx)jFlYuO;OJ1_^x6oz$SiZ=WZ9Noy$8m>e_DI|9l}-@oG6?<1x+ml9Uki=0)SFf3 zT^V%a{g(92LucL-6OWKN<#$UBwI8dt6|w~AF)>gO|gl1;s&5yCdvELAHFRzK( z__APo?blTFlTG#tW-kWxaJkTPaan!)lwhv_{80<)CK0wrM}P2g#9IjS7xqLp^PG^) z5&pI{5Aic_Ksr=(z-8W=%^?R;ASHUuyWYA}gKXuZ$=*RQc`>IO&-S!_I} zHKX+jg`HA%;4(H+@y(I#{HyQ1xi%tExzBbjo~!i3i9KSYy@15vAMTiJ z6)|)13$u4`B7ac=ehB#<{rHg32@qt$6vCNvhDjAC87X%l6p%E36RRb6qdjVMk)i%4 zSpwu;cI_@S4M0S6S65lXQA?3gxBptPU~A9k7Q5;8A=4?{w^nf0!Om z9;%mK+6-+};?hVYvyfS7jiY1_aMFT3?DQ|riv~YrhPgg)pv&Q$!J5PIgh+eGJr7;x z#R=~-2HXXmr(5jQW7a>Qj#xaV-G2&ZC(i)e3%({)-Lf_xy__*esTiFoL#`nOYJ6^fXV7dDEIehIBgQk5A-k{LjPJDE9(f4x+_HLVe z$^-RhFGx z=7Dl{)8Ap+_Mls9e`x+8k9)J@5SLA@jy(idQXv?(nLJk0KH7taABI2h+jJJj4`?qO|%C0RRoV-k#I&nW;Zm{ve>FJsn`v|m4-lEx@H8PU9bQjZlnr?hq4P27+_|cc4ovzv-8F~f`jI$zop^2 z$L;?vF5-jDGiYKHwdZ_lbqg;vV7GFxuZ2|~Sub@ye5*&o*jOLcL)sEx(crKF?dKTv zqEzj6as4t@0VJ)mtNH4Fj18>>A9@zk(~>(9R;At%4gs7 zg4Kd}z*Mmv{u0Ja((g)>?cG(iI|szzhtjPT(rNo~e&=zQh?>#pen!CD zjf_&rMz6+;>iEdsxDxB%_1vWYut(j9TP$(!YSQcw4fe;P=-Vw8wxoWB8`C|`G8B2B)_>Yyv7&tTt$%U5aJ8tI zUvL5kEL?!QO{T9EptBJAJ#So{FG6)GF|guj*D`Z%L+Id*wkjI~%CDC;uBccsz^z#f zoBlTjw`cVEc{F(=ZLHj^vgC4I zZO=-!_pM*A;l^?6Kv#J_stEYk?U5=$G^7KMg~+ZD9{Gk{zLze13Z)2bKV zZqN9a@#z2(?4f2O%ug$#J{-GJW#(Yz zX1oEj7L2NUHJ}45s$^B{o>)rxAnk!J-0CT%-hr`t^<>6%$` z)3+_QCbR4Kk!eSD@!sIoiww&P#sMo+ zSbl|?5znU?*}AL4$nJ`9>L~+l-Yfv1qAM5y$4-VvfZBFxS2ikT8#m+VH#Wgx5t4_> z1Hch6fFrL_*G5z~FKYWlezPFYsCnZ6{r5lm*B-;-4b}OL?sZl1;?7g4qsiK(y$AG4 zTy3i74*}1OhM+FTl1t29dqVw{M(+M9uBe~|lU8X%p1GLnnbxtbS8Gj2aH~OiY5$KW zsm0F!1PLr;99Tq*NX+(&mVx)r=(r7^y~uOv>jbpdxO6x#Oy89mZ23)#jlNzt_o1-~ zk{oI#dA#E3zOTd^7BEVQFA*%RmV^iKK``a~f^0=b?e|+%U z0tFzFXW5I%2Htdi+VYfZr=Lpt%Td&zrEkk=Vo zh%&fj#ts2(ZJgPhH2OytzH^oJAO{-Nt0^|ww{H!6@`XBbw|dbbc5c-%-yfOZ!Jpp4 zOuq9_CeQ5^71l$78>t;bgf^Aj(4~|3kUY<2M$jiRf3DF!ARcilDaM798OkBn*3xW2 zLRg1Cqhuw+@VjSdQ9#0|KoTzipw?J#$iwR7BvLfuRZG}#yJh{{nh{%4?42nk1fyGG zU}5AZ$L0CU2o8H}`duLn%6N+MeIm6-%nNoEc;xAC+FZ%GaE41-;NDfRlD9N^3 z@wxan>yS1 z-5J-&Qu>^J8C_N!`>-u(3~jzPxKk|49apNeS2WhKGBzf+vNVwho|NIbUgJG0W}B;k zDnPL|(Ak-z?3R1mpxxb^dXEu=nM*wRUPMVXUK7*qHa+NJr8Li8e5Z2?ma$b{9XK5t z(I(0lUX6PCE(+}70#V{~q$k;Sbu?cBJD%kr9bhNJYO&fWvtj(VYjL=#mg0SeB9@j- z9pxilfj2Ib?pF@E#_^vp?~S!=Z~iKmBD`uXe+c^`Rro!+qpdlm{1i!MTYM2Kq?Ysc z-IS@^#`4U2Dy8s(f+dK_fHs3qRw{Bd(v8Q9)IFWClr85CT5q5h6bP$VKQj8m+L0Wwr`Fi0mkz ze{M*FyJkO%u=`eBf~D5Udw`X-OsJh!vJAJBcbvsh2!w8dfYJlCmKhh#IyX}nm)?jE z!{2yGlikCiysCKFyG`O^4#M^`n8PJyE~&HYrbzNJU1n8fiC{|oSQ_Fz-}vfn}E3U--$a{#qa+E739U?S;_Xv{aH_zOOm zFS86`A?MB+JWU$0b+uk`$8LxVG^YXbIt4|iX`;d=_Bn3;TBd{S7d_GTNr z==LG>Ik}`R=q5Ujd_+r{Z2?Q3_p>Rct;N>;9>ex>#3uvclp&g8^9mo5#nmm)mbYtM zrh-jdyzV;(T=Y0;o=vy-tNQbM=GbTQifAn6IuPBbegWi$h74Y)9Y_0xeXAn%!S##% z&L3R|I3f^PDo?lG0&d0O9p`1{6g;9Sqbwm5KLsedU>G2=mw?6n*)p@^ZR~evb@W#Fq<^t7($G1^k3qmxW|wyJHvH2FMKYbYobP zLk?Me-Y#PaeylDoW+k3n46Er}ld+lqGqEZN{ff^kF&x_Ta*|y_*;=Tm0>BxP$;`|I zQHHSM_LDWUE;Ap7orn{zg*LzYCJu4?wsFbZc3JQrN=vRByGc-kca`ABHKsUc*E_-p zF8k`LRu57y2IdK|Pk)vv|B6hN8KN4IN9VW1^vh(dmAj3?USn?ecjJr;l zv@5KSl~no#H3X}cuG3@aDzdlC*1oj&0)GyS_WdtuBFn*XCEMjAja50TZpSFZm-Gh* zv3mN}80q4|>u<(`R7O~o^cWJ?1gw2VdPB7b@fdIia78csBlA=#<}{W>q8;-CJUkH7 zvSsEYY2t!-Y{X#(OS4t90;Er;621R^A^AiAYWO3KBchaRB9lwXs_9$+ znVP@9TQch*RPlwufBVY&Gq<`1jz>(2NLg5!sJ8d;v+svJLQB~m31KrxM7rQcbI%s{1#+~i|G;oQpFjTcm#t0{*3UV> zd8)LqqZRq<;-o<$SAKE+tBfvzcm4ncB*yBcJL_4lP8mzPZpZtr0K(g>u74}%*vp)tT~?v#Q%BRvLo{XzTsg(f zxKDP3G_ZAhFcC{|JvW08$BQv^fB!`8a6~Cv=Nm$Snll{4RQrkq5JXZ229PFQ<-IRb z_({O7f%?vNc{{|Hjb=5Pu~>7(u0wzoDqj^7UZ>gWGHUf%axAf|ME*HD>g7CTz~Uu{ zYA7S6)*6YCD+J4X;VBnVrI?^Ysvu)(+I;eTxlD{sZ&qH?x>h0M^qyrlvZ?AHh~p_* zMH<*$^=!?>3XqD!h7A^2Dwjp^tN> zKw-M!Acj7b3YxxHsd8L~bttR<9}MzOXb?has*!tCbg1J?H(bg?K{+LEmghw zQhr99zVVyI9=P--<0El^CEK4ESYB+Ocu4OaClGASe0Yr+ssoXc?9rU515^VS(~Feq zUybx~zWre)Ve@t_l-@hf$#6+^psmdcwEQy1*30^HZVE@ljaDe0IgRK}0(bCOZZjxR zJi0M_X)cJSTHQz!gLZk*+1$&z@G|8X4k>Qk30oT754rKA<;TZTXF9BM0To`*LYdSk zzG{iZBQ;*qYd+1oc^I9prcQlMC&k6d_WNS2Jdtf`;S{u39O$Nf^G<1-a8dTx7#-jSZ`0(2`)#iHG&FGoT`0vU^ONIV%kWu8fJ4U+}Q;NCKiWGpqTORi4%eaE=uyd3%k@K`{$-9u5FF8Mu?oa!sl!0d|M7cxhuB zPqNK5aOROCZeLQaXNYZy!laQ zmGe?`TG?=@Kb&ae91UkMu@ev4U0depGPimzn0(SiRxe*M%uuD}amZxWQ^kT0c*36A zum$~7L>}+El**#RGa|G?PP50-(!K z5j<`dSmXqE-V7beS8&G~8?;Hi7}*B~w&gpsd-}{)=qu(^Se5b1npzp$YV1HXZ&+Dp zIlyr`oLywKCNhpT9%qX_Bh+3+uy>~*)`#6E!GKl=j{opuM~<<#9RtAr*v(C9(Pp>r z*o3X&W;TrlXqi1Zgg($)sSfB@q4&!J6w-PxIDNvI8B`y?CWJ?U6L}1mHK$84&o(Ej z;}?jUyEjnIn@%;=u7rOR;$QLn3vd{e+8=I^MZo`7B2vb+H_lEMD^9TT*Tb9f=0pP5@QkLKo*W}gD;ym6t z4v7Ziv0?MLfm}VBO7BM2v>y+j=#XIlTewn{tKu-vy4gG~aw(?tUmJ|*C}O|t>t!Q;tokzGxgl+orNm%w;cM=DHMY|&H z6v6Otf{$#M1FZ+OnGEhX>MC#ddOr{lv3rq6Y&3+7kB-rLagK!#(b{UYIRpdTOFPN% z2{Vi^)l;m_jw7B@X$sU*=LH4h*!L4Jm_9jP!OPg=Ys2!@Sxpy1=Jr<eXhcBDn?pe-+^%p5!YGV!9PC}RMRUIJV8im}+m9hO}>9^cqOC#6dBs8!OHh5L@?n8tC7X@_J=uQJC@X{Hk z+gXAv&33q1&IbBr7wP~jk8AitWT5M`Df5e;jI0&)0&-})`2i;?YY)-YmWO7HvB>$X zMX^-iovIc)_=B)ZTy8i#9X54O6l0ED<3ygkVwYs}|DY!P8C4>#cO9q}oG2i&)ds!a zmPFTmN^QPk$?o-9NAx^kGb5cVla$Hp{^~k>UccZ(L;$nxz&Gu58-0y5N-(2@%mt){ zs&3`&(%*Fup^FBcj+gVxq3xq%t!Bpc-!J z#O`y`_ao>g3+tZYvS;{tDy^_{v-OYi&u!9v^0XDJlNH(tKEx^=rBSv;z3W6)%PiU2 zM68A(DUV%m%`7Z(WTAkXDG}Daa#a3@ZscqH=I;}2%T_S&*#IA5lHkTYV- z;k@Q=Lk~i^IBe4bKpJxp0BiRaMK<{s@|C z*B!&3)SBKL`6&nnuWXcPuWu9oPwAN9v!+(Yce`sxZleLR%^UfXQ%MIucBmxMcUP}< ze`X2~_&oufrpSYLx9r)?X}w=Z}f#j@*_3fdDW(p7(*7qpp1vt zi9-S`sP=H>7!WYugyw~Y)U|T(f43^ACewkYG(EXsWrbt7W&)>0hK@I%Nd5gzB2BXm z(g8HpDu(mKzHVsky-jg;A>NlRfP%wr5dA@`@STG19C7lDgyf9QsN_{p*brrZSK!$0;k6)A1Di?&%-;7_ZChXOwZEv;Kq~ z1$8mseZ8_X$9f{|g1}kcs{tL08rY_5%3&J6FLc)FUUcJ!eWI)tH}lnaK}ranbTAlo zR~jx2U~@xv*XI>=Q!k% zI`inZ0tvp5t1o{fqa`KTziMa$^9?QO4@9^=OI1gI@8u?P2YsUGOaDO$bSvX|(^ya$ zABmG2%+9gl77oNU~PCbz{7uvopsanH8->g`@2g9Vu@@q3umTK}*q8tCXpuDhO6w-pNPmWGO z`07%cm;HpgnwGEr7pZAzwTtVoHLFg$oL2G*0u2e>L)yIq%&>{9=>yST`Vc zoY8l0biB7)U7=%eJciqDyewxrI`^YkOw&Q{=DQ>;AG*GXeogf}(+I*oF3@Eh;M0W^*xemWOb|zkO0cnS353)?8zP+13Ro?-ZKYyKEt2nDZo7p0~1+4w%kOuL<1A zexgp}LYVh`ObiKDls|G4798~&B5uHCV`ix5YV5Pvo=}h3s=)tH4#OQJIbJvQ@*}b7 zNEZT-QX{wqfGdoYk~L2;Uyr|+J?xh^bPH~fG^CT2`QHV3Br$OkrC)i4>U`R90Z>!X KeqF0%9r{1m>yx$s literal 11543 zcmb`tRZ|>X6E!+WaCZsr?(PnQ1$P;2kl+phhT!fH+})i)2Zvz6A!u;t!9s$byysM% zAMn+8v3l+9i(S=gUvyXZj`^gago#Fm1^@ssRg~qm|MBWS1fd}Pvpi3~CH@hro3fE7 z0DzAFKfnQU@<;#xG))INxlf<$UAV5!qe@8i+TTl=cyaot^h>l`u+1tU|LPfGxTQ7ub}yBjt>GWP3$ zQ#url-VK0*(rQEF`AP=|_!BHH&PEGE?gzm6Euy0UVCuOr?vGLL*k`iYrf`sOIG9(8 zuqq0K03aQZC|d!LR)mA(e`C}E6d?jEr>w2^0iQSkmP{c>ivURepX@+5fJr(HAzV=k zfEwR6S{`682B@3Uk5vE|asvqLR7XSrYa9S>6$3kEKvO3GHj9Ja0zkn4aBD?JumBJP z0hSYVbbf&FYyg4ct%3LrTLaE9>p!KkTg0LCA_}1-v{1_qpz%(E)QYy^TJcu@;UK$yydhpvZ*Yi3=8izdpqpU2$DMjfF?&txWE*SDl=<@->OVF zhF-9}M*!fq$8~Ut83h4iANFg`_x)PxRjG&p0I^d^bq4^<<>@(fzqd+_qW}Q%MWL)s zGUWGzBb}Y{N6s`+uv&60NmKB{wCp3( z+^{%@^t!Ml+_0ZOC=s#?{ zcyf}6igJH=w2?`~3l<;$sI|xbnItJU&J8;uLFOCz%*{DAEZZ#1j+$mY)l6H5Q9DM~ zq+f^76Z2p_*X)ak7pee56-4xvr3O*TC>yF7YOE<&QGcZ8$Nh;ago+TZHN-@lR;Aj; z*p2&lgv5%MC(2Nnoo)p$37-omFj{Uzh#oISj+Y6V46o8$rn`W*pt#_oO?S#(oy)IG zMwd4E)5fURKNStkhhz8IL1gkj^+U}n->5@ZrEsV_<6H?<1V#W>P&U2Y+s^KZx5*x9an zhO8Z|UkFpaqh$ay7&FLOxpb>*Dr&ZC(Dm;0rgW34&2@qr6!rLY_N!5sGHdl}@;{yH zkn4b}#j2uez*VlgQf0wLuQ5nQ!)n8ta%EaY%@+H;W7O!@q^*AJG)|bck=~&KBaF>6 z>tB3Wd%nF=^u$TjR2PfL@d5b^TD9-e?WB)X0FV-3y$3;xMta7A+O+9 zgYwv#I@M%mWg6Bh*J{-&o%abgTvp~+^eOZhJ%?h+j02fWnUuguvmf%0b=GyZD$y#5 z)Fnh88aQj5%E^m+b^QZRX!+T7$UV|&?9YN z%#w(Y3b?z9Q*e!M+-S*}4NJ?FgEBx_zrs_X^Wvq;iV2F?@eX6A`}F(HW6ooxj7W?$ zjH>Eq>Q_11>d*?73NbY?wegwhnUa~6Y=K+}o>kuC+~eHU+?`HKV?Sf|j;f9>Ta}gr z;|imzj$W4tV-cfrLp@`&R>k`KdWMyr`kqSMN~lJfMs2|gc)01NW3r>VleFn@4QZpH z!?_~}1hygucaAm;U#0b>1$74{yhengGEJ@B>^0BRnz5}4wgS*+UfQ5(r z+}7No%c_mWE5R#=tEkPwY5qc%0wX@pAHx%JyIP0yQs{*l+)g`n!J2ZKq2TsoIxcIj z6odNJrp|KTnn%zzvE*+_kHUS)eQ}{}-fbg`X(3qWMCX*|uxo)u#Ex7D-Mh_u!23L) z62brnMu-WO2{nO#{uA(pW*|{Ybi?*{%lf5Ed=XTJP$ubX{nx|6dK&wQi{yHm$K!9a zDYFh(5ubujR(>FZjzDq4!5B;tQ4t1_Qi*QShGg?RSpvTEn{u8z@=x-a^4qy!9!-%( z4xj+5WDTdI_)np3{%)x;PCKLNWy5?!mCvdlML1a-1YJJ&@4Q^zMeXJ9q3;J0f8&}k zbMr5r9lDRGB&z1bV;9YoR*TjUPXA8^hqa0*p5YeXmmv|&e~4lTOv^PdI3K+xXUa;c zajVR#9*}yG(ktLkKgw6meh3X`s!r}z!&KYK6)hmj@u7C+l@RUY2}?~$c9*T8QBUgt zpMj6;e5UGbBU;`@UE(^qC^L{Jom!LYJ{WPJb;wqn=o&MxvLd&DZ4K-OW@5gJWY!k2 z&0J2me&0wI%DXOnzr-8D`<2>J9;s@;L1B2*SLZ0wBJS?LfwoWhfh|%WsZP8rwu9Aq zS+n-iAgdv`k1;^qDb#%M``5vCIHz-TJ+#A!p!3buOEXPFE~A}x&{JeSxg=psYy-gt zaTa078uO;#A$gSR_krxq^tS5S5va|a-Gbfe(Kbhc307uJ)=?Ig3+DRIHq0IUw#$d# z3L9KmGt3)3VI6YN(gE?VUfasa*%>h@-7?*K-6vhatA(w0(CQzT`783k-rxKe%Nr)W zkFS!G{Tq64dg{w|%eK!d&#jMA4KX#Re=?t)&FVFr-7m)x77$*M>99Ihp-amY3lh58 zNh@sUN?7Am{cri#z8si>n0g6$V|#RN^8N>>kElD3l$fGFec}V_ejzVX`A~oYMZ|CCkGxCFiIsa~2 zZj^Kc9$U}uIAXhAcjI{dT2Oe8I~MB|n>D@d^15x=IK(&TGRSEZwQxT3Iun0OThtuy z$?M5!fSNi%uLAno1MMD9$qR&Lx9hpVRLUO4@uZ^ACj_@rXn#T z&kKYlMMUE*MGho^xgEOxGvqn);#z^9m4!>|F`);=SSsp8fjYEJ0iljyBBLd zj*Wl)CbhM)wmJY1zyJV1!T^B3@BjD|0PyAp0M0A{0MTy%0I_ST#i$|xKr5>vFQe=C z^CI83H(xK~ao}0xFtaTeQ5%vRjwfqhtc+VColc9P?gx@)8H%Q}Khl#&Mj}y^DaYLH z-4jkkV)+h3iEaZde3d;}P|>88DBBGF8T@W|KDqwr_9XFmv}yA$S$OQZ0iE-oy}Q^k zcY6J8>_qOwK}kv3uC(hK8WqKq_5b1jKm0H7f9fcO&=^mpUGjqVr2C0C5X2DnAI?cE z$t~%*;YrXi;E@m#Ad{Y0=w1l*(qc_;hUj$*jj9lT*q2>1YNK=P6!o)eTO=lHk=v#&KTH%#S}w_#nTb z`XP6r`>DkU;z)=iU@Tfk^Hs-daTO~_-MNiFxGcW~W%T$UR3QdrUAxg#U8 zG5AFB*nxpldT9HQxoU6G;^@E?;u8Q7GtCh^6})PO8y$ZGIratMqH0{@tVB@scpagO zsOw`Fm7jEySN`@Lp5~vr}TMOV&SYN{LG^A zt-%O<0dn+)kL7YM3ndiglS0`Nfwao zxLp!qA_0wri-ONOTkUUwvFzIJf_6zMaLyl;g{>wHy+atQmp64dwBpR2B(p-ojEgPk zfS?jc&x8LsZIZX$TOmN`ynQLhRL;BPDKGWqQ$hkOZpaE86%FtNA%IGZ0=I;K@jS5K zwB9h~yrdkW$^H^(kh(sJ5q9SnYQhKIJM#!4|JF~sS-cE$dV*PK`5uMAV}~?#`$tvl zZK-b!d?C8T+RYLbBOa39{k*GcQ+^8e{+g|W;DPz!Hxkw---`)W#`<=c$=Q_z#&iR* zZS6WN!HmVur;@~Fn$ck)P1TM6Kkwai(b&7pGgv`sGi*x1yob1@!6zJR8|44>X{QRl zqiw?9Shq^VET#kZTQQG{z=6kB->_$RhR02O@Y^xu_*}uS8AgrZjv;jg^zik~`6c4A zyQ0x>LuQ?48k%H*+MUb}5je9d_Zq`W2W(WG{4m`Ur#i0-l1erEp|$JH!|>B<*VG1X zV>B=kX!w0W4Ve>868(kTO)vX^3?r|CXG=Gsm~|Ady{lFC)I`S>%^;3vb31?UuWsb3 zjziAWCxlKS4*`G$0_UP%1y^`1I8>^d{4OPyh=S~$q%V?t--^El%e!cdIxXD|+Mt7% zvi)w7WUJ5s3O8=KcB&&<-SKr)d$0yJx?{p==z44wew`Xes$j#0TAdjOo5cu>`44!^ zBlQAafY)1&XE+H+WZauu`FoKK^p>|{1}1FK-;1H%x>Qn87)D$mXpJEk^EwgRFSHMf z8`ZS|1WjDDg~xc#m~Qwj)V*aR?RiC4TVRUS&x|cqy@MH7snR{|5)0n!ZT#e(3YE5l zBhlIZt3`f+yyuEnjb|Gyq9EN^>?PT)ip+LSQ0O4iQ3Z!<8K3D+cG>#3*p|gk8;?;4 z`ZDIG9iLP!1DRBB(VjxHpQ8!GxL1P7*wM>t&@Rz#r^4na4ww};ajEc~*vQcke@u4+ zIWnviZmZ+?^Xsuq4#ii?0b5eBYu~VaOqQ6**-yAtWa2Rg7q9*1^}5P6NR~F1e*7s% z&lD=Dz&4=LV_8mn-JcbCB?|7aWUW)u$C(t0<8)0n?*NN$9nSXadhAEz})@qG|5zLhjJ>i?*m1lzo3Uq1KU*ey!t_mQL`UyGz zm7|@xu*nf*BPCh3RHa{yIg`B9J)YwUvuTkk%dGlV+ZmR6vG6gq9 z^u~0>rFqdBZdG7>eOFuzeK;3$2WUJSPwsPG_0n9i419yvk~+`SZMb%ZB-Pm}nxrxa z%Ej@>7oUBO$|S;#*)b1NcSs8YSw}~RNT#~ z&)iXqL&`41jBPb9*kazTo%p$vsP`o?;q1)}{Ut$E9j{};SP#iCFVnAe5T-|=LS&*^ znE|)7G2dIB?1Y%Q;$fbFV8x-GPC~@kL8l$iWBkn};a3T84-}0^VF$)u*=he|Y{wNz z2=k6+@2d!}eGFMP!|+6=LgaLNp94@3bvHpD6y!Mas0KIP-K}kApqS0gmn#KT$9cM= z-o90PzI3VD%Z(*+b^f+ocoH<1+Rj zA*guz_(q8SuXllQOcSqnJ7qQ+mv=H6qqXiA6uzZl)&nyZF1Y1UKA3>xhgeVxE998v zlu9Ypa}~ngP4N+RO^obE{@v#VytX7sHcFA;zC(CAQtIVhxb|-BrC9C+#TXFi8>`n} zMg7uHDX&VS!P5HcGG#1 z{k<_m>_J&~AX0zd;LkY&ULCQVr>U~JN9|;V^za7Ix=ns6#Ma8UETz}5-grbLrl{FtGSSWMPV`& z*kGL!vE1zcwi0Wjwo~f()7XZ`+ta$sxXVEq@0_1rj8^*~=1hThfBH>z?2 z;dcw6o!J#P!h&mBF_ysK&~wE3&IA$Rn4&eHk*wh?J?=DYz?u{pTM?Bq@b6 z=}nsoccqWuOZ&m&(P6r!ZdM__sr$DdeW&X+E$xtnp8YoqWuJ={S~?Rg!jJ_!=+z%T zx9fIw_m%ISq>Y>hLRFl{uI^#Kq_|u)wi;D0jK@TyQq6%?T_8=w^d9w8-!wyPUZ=E@f2F&6gGjHl z?@2ZlAuM12-zn!HM$=$yhOjW%fAYNQVZT^*|L9M})D!W_?l#s82^Ytnn>OIDB*69r zxmmh?z&e*~gF2QltnP=$0=m2o6PDn@-C+Y~tDX}qJESO?=LGGBAfi9l$!ZbfYaW&m zF52o(;2(UrzrDSgvJkvXgzIUB3Q{1iC2-gb+Wa*V%JBPiM|EUoio|O<*Y%QC3K%WIBVS1<+s1zL+;yMQ5e&2_%6QZSamlly2 zR!ia;7@Ic=b(@SThQ?+@;XZS$VXAmYYVAMKrEE^Zl@n1`vl;?Si)9s`_3v?%XhpvA zd2#P_5B?5BP5gNVWVUNE~Nz{?ofPHj>%;Mo>jOrv^NFh z)!zCamxgidoI4TeGkKqX9S%l{?c;eNodL}gbXg5zZQ(YX=WiDq+ZejpFC2E;nRK!> z|JGtk%F#?RJu-_4Rx(s=-h3<6}27u1HfWG`{GkUX_@<1#%^l6108IO%&DI`5g|4%wKV*5tOZ{(dx-b^z+W zpv0W|efn?g73+xZBrfXmCj@toyvV9<@DH7NvuJvSfu&>XvN4xmHf6oP)TCkJ+Z85-KO8;D+b$=JoZiAfeXNGnf-a~MD!jJB7y)t-Y0*rBF z_ipPBex_~-=qC&ng0upmJ^k&*q4OIKQhh%hLe1zHxk8QW52tRJE5}RQj$W3}@X2(o z+A>m>y0XShE49RH>2|=zR1twLr$BQtUT)QGpb4jaG%r(zoj=m7b94)>^^4ZrKO#AtLcb(wCwuy`e(XVyPAP#T0Ij@2K?f~n_&94 zM!RZsb6%CfG&arYMdx2d_OOv?8i&9{h$@+NJjShNBa4anT@kS_7`}!3uNggEzqa^- zO&3Um_3mEiC%!cjd8;a-9aG)GpAXX7W-DWw7B4)k%n1sq?7@*$?d8CGO{MRazt2ZY zjlQ4v4=W4r$y7$Pl9gN#@X*)U=uqLuH7VUM^88SAV$;aI{$Lf2_@1O3pj-Yefq?z3 zVKUiYAmdl=Wekm6yodqvPj6eS^x@>ugUP46FD>ZPY;i9;_PbQchoSBQy>BhE|Z@b!~~iQ6$F z*p{+XVg|#uRa2@v()xi#T>8xmywR$$(Fnq}%+B@sQ&8`0%mZ^t=4|6pVJEW(sl0|H zvtw6-NWTgV-|Qk*YrXTU#6*6|cI-~Y&>j$dDvFdcCx80owq^EUGmgBJ{JmBz!{AA% z@a=||Va3r#{RS@9)e&b5WU%O&rI_6E&4MnbpX!s}B9zf)>x7*U_pE+rlFY{FmuNsU zXqZTbv((p8ee8DBFuYK3TrkBRV9Bpt;l`5zu5X0DiW{1K$jl1a%-HUrD{m)1(E3GE8I}0$nnz}yao3icvj5=f*?@xg({%q<6^D+C` z+XeP<)CQFi`E-#3NlGHZWZdjU7o%%4(ZaN+1xfdYjfmuW%{|C|4#}aFI;5PAs@#Iy zfkqBEtz;~*rk{>P9_9Ahn#_iD&N#P?tbT8+XAFi=hY0N_rFa(`;F7){(q_7UT!`^A zNI-t1?vYcmu;y^F)g+wBP4NgNaW!`S+jT666w`Y_D0g~>JrMRH>g)U z*xjKeg@-J8zA|Kh6a&8&<7ujYDh8Dp7n8?!*>gtHNnpM!Sl3{r=HHOhH#V;op`Xv& zC5IprCA$pN;>Nr>mvwndrvB;Xjq)nhlV8VoNDROTTHK+cSGslVE64#!r>b^8Mehge zc;@BPMkF=1#z3V9{&%bdy$=UjbkCl%WE!{&o>;Y&Nqwe`V-9Y& zD#IEK{U@;|hJHhpQ5B10XyOZrsKt8N^$fhi@LMmd4t}blie-Ec6x07|kQ~0pbTTgg zADm1B!2;!Z&5pTIgUKt#e^e75Fs4@t z?y~|@RBBre1Hn-d8cL9L(1-!1=rd)!O%;`%9ecAe>VtTTp;#`NO`w5t15R2p{ZgqX zd!3%gR@lUiDsvv3B+$A>2j}mXK*4J=y(kTBO@8R}XZ zu}Vgr{`<8wWJ;ICF7wUlvuUby>-(MpoPRw@c{D;&rrK@5G|uo?vwSMOF&COIHL=9# z6NQWd`g88lc)r9n8`H(w;Zq+Po$8*(`w3V&P&_WhL9~@)bZRk(Uw9b%`h^gh8J*Oc z2D)W+PS>OPskW7f)GZ9G`sDtiLvzDSX(DOGw1V|H-22TxIwkYIDmc;{ojD5&$-)#0z$R4fH z@{FQ+7n`p}i4&)k8_XwNwwAV|8Va6Gzmw2vT~{HB8FTzpmKWKKEXWo7Ol|TsUrys; zXl*ONc>ZrHwDK%&g8I;(3cV6!yq*G^@~OoDotMY9aoK8kqzyxyK#bNU)w^S z%vme=+_%(Z-fy8jd>+1irx%d4BfC((t##|GEi{V$8kf@@aeXNOC8|Ufl3bG+NR+)@ z9TR5N@0{%>PiH@0?lY~)#e+23 zrJ=}p4n*Y~Jhs|HLk>46*0hmrGcJ2X+b8p(ARpctU(r;r2dkZJb5>7RXBUOpo(Pi` zD5rFhDWbC6j0}m3W|^Nz#`;npgA2wS!(yG=?`)v4A6q_1p1R@2y%}YEl$O4UU1L}v zUu9bZY~P+KGiLev=kDOll?``Fi*A9;Jy)EVN;q64)9nhuhOg6yZF#0x8SKqgh_bHs zL%1d7!S+&R50)Bln?DnaLo}JODd@H-i@;URYXc$3d|V*Cy3k0GA^LF)=H(~fk2##w z6G6%~HcbXOjJ5nUrUlmVj5P&Wu327IycR~13~zcx>NBXLeZwJ9BxxoSjB3I>!w<^q zcxhvs>}}i9Z7cvZxWqO)Taq|n;9B;#o35wWKx47A60$rlEv}yVb1!x_IKEWzI72>z z(fq9ZJ@gWl@cM`&kVw0PQjVayId&MAhKaEXf3(?$isvC(p&;|0w`#02(+vbvHtd+v z(W!`Ko;IVX!{jFU=@P0P z$Dug5j)hbQJdmLzq(fymB9QcBedQbL=@iHs}F zh4KID{L<dNhT_N2iOKni1D3eIceC$+yHxh0w4 zF~FP;Xa5)_I)Zl#E)k+usdFz|bFFry&ea_^3{J5XPhE0K{&12M`=3ChO#}uln_`84 zjgu0QZjaM{{=N%yDe^7AnssJ36Z%*5KB_qMUKjQjzmGD!Mw!GS$36kGK=?E4$wDB} z>haIAW0No&pY$-?S=z%eb008|*U`jhkgK%pGz%ck&|57gSnOGY-ZH?rypOjJrdCx%qMM_O&!clV6KN9MAyd2R~+V_I+HjJ+?r&rh$& z%2AbH5T6USYlXFw{Ty&zzaTAqtK8%z4?EyFGO41By1!d9oAX6qz%C`wSe^3;=d;6C zZh6pA_FQBrFg*m-&1&;dxCIhYuY#A*Leo{-(p4IqCpzB9RBm)9P*O+8jzmjZkKO1a z>_f_4_MKo1$}zxl;q(i0ZMtMrnB#;%`sdE_uw-YIkb+}x4jZ-)#^0N2uE1Q#DYWJ= zl4FUfN!J&+mMC}jBtr2|NkV5SrZV}dwG78_m)g7DSBl6WSw|6N!ax0=x{i-nd}^mM zHW2+R@@4)FtnM@>|N3KBITBI}72d`j?BFq*9fWw``UIe@-g;UTx(8EHX45|M8Vv>( z#^1Yp#k8ZTj9g8NaQ!bUzuET7bg&@C2~=#KyJn-NQhT#F)BEFpzGs=?YlJM1Rfzi^ z&&jQ*@_z-!4KJTPDk^o8vDlJU-mC`Sv-kF@Fr~ZCYQ^>b_}1_HXLGgMms{(44zWyZ z_48TT509d*2uMN997|=pf*#NmrW)GtL>j7o!UkCd_6xPaDFA=i9B|9Dv?xmCL32P1{2;&%Y^YsZm{$Z zBK{Mrr$#Ohju;Iq?ar`uv&b)Z#;q^10Mp8t%?(vaghxwgp@Cy$5V+)+;(oOyf{<*A zo^R;*F(Pn1&_fS0hy^vC>~Pzja>qv!6M9w$b0-zO2kzMI-unmfqYap}{Tff?W2}X`GZ4{P#oCx*< zK;x>%X)!%yU!d?)0+Y=eY~Ce)8-Jf=`ttsQPangOiXM#P`L|Ks#an11nG6yp+A$np z*d-*9v75hb2s^T#vrX}n-!3iWeiOcBcv2P0iBe>cyF#0Pe8k+`@d_5I&$@bGw{c$v zRBFGQG#u@r<+R_n{NDoRf( zEa9t-y4XV`T|V?pwQ>m4lP-Fkeu;%b7$emSv{2=$mhOJVQsvf5;9BAH-JWKJAFOeDBc9n@XrOr|U8KZUyFz zS%sLMQ-J8aFndV-{3s^{yzonXa0)^S?mAzc+-3$m`rq*oG&9HDv*+Y}*s@xUJrYAi`SM4J?A$Xb1g z#9yR{sw1sHVjk3|6|lP^>KJeF$3sElhYE@gDRg4>4?p(@qYNKkKBNZv=~#j4=l^lb)-ziC)ZH$`SS8Lcx%dEP%iny{q~;w zaWK-S#trSPgvUv{p)MBB0@Dvaqn6QkJqfD>Gnln`!oS#pP)0Ew9*52HxzGroB zO~<)-d+Upg-+t_LP9jSNp_Vwhu(+{d(w-8y%htKiJT6JMZftKbRX~MrpV_W=>tu%kxiOswhXOAp5?W45cgJ(eMHcwM+~T||!VLEC;CJ(I zLX_x%vTv6)ASZ}!<&Op3yokR1?uJhHp??lpXYU~&mM}Z_OsmooarLNse0->|nH_Qe z*{Lanbd@aq21;21)*Nv(J+aTQp3wa>aOGs;W#yqz%2}4NX(0~%-c#U0U+E?(abB`L z50~|0eVc-sLOEwi)O#6-`OW(b=u;&IQtb*ZwBPJZENBUdR!ZKEsDFC&oUl;%*x}{0 z_!yM3_X>DFFg-^EawT`33bZZqDX^{5N>(U2H^^Fk{eJ*G4j@bb From 4bb9a3a6af602f6a26ab88104935ba862868f7c1 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Tue, 28 Feb 2017 14:34:01 +0800 Subject: [PATCH 13/13] kubeadm reference -- / set up/manage mean set up or manage ? it's better use or? --- docs/admin/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 24f39217f0..2145b75310 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -31,7 +31,7 @@ server, as well as an additional kubeconfig file for administration. controller manager and scheduler, and placing them in `/etc/kubernetes/manifests`. The kubelet watches this directory for static resources to create on startup. These are the core components of Kubernetes, and -once they are up and running we can use `kubectl` to set up/manage any +once they are up and running we can use `kubectl` to set up or manage any additional components. 1. kubeadm installs any add-on components, such as DNS or discovery, via the API