From e8727e886eac9952e6153fdd8eba5a977b478e51 Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 5 Jan 2017 09:48:47 -0800 Subject: [PATCH 001/407] Add Bundler to staging a doc request page Fixing the following PR: https://github.com/kubernetes/kubernetes.github.io/pull/1796 --- docs/contribute/stage-documentation-changes.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/contribute/stage-documentation-changes.md b/docs/contribute/stage-documentation-changes.md index 50e22a1b8b..86e19ca42b 100644 --- a/docs/contribute/stage-documentation-changes.md +++ b/docs/contribute/stage-documentation-changes.md @@ -87,6 +87,12 @@ web server: jekyll serve +NOTE: If you do not want jekyll to interfere with your other globally installed gems, you can use bundler: + + gem install bundler + bundle install + bundler exec jekyll serve + 1. View your staged content at [http://localhost:4000](http://localhost:4000){: target="_blank"}. From 582d2079eabe55e308221c5ed1b3b25484a86e5b Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 5 Jan 2017 09:55:12 -0800 Subject: [PATCH 002/407] Update stage-documentation-changes.md --- docs/contribute/stage-documentation-changes.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/contribute/stage-documentation-changes.md b/docs/contribute/stage-documentation-changes.md index 86e19ca42b..cb8858509d 100644 --- a/docs/contribute/stage-documentation-changes.md +++ b/docs/contribute/stage-documentation-changes.md @@ -87,14 +87,16 @@ web server: jekyll serve -NOTE: If you do not want jekyll to interfere with your other globally installed gems, you can use bundler: +1. View your staged content at +[http://localhost:4000](http://localhost:4000){: target="_blank"}. + +NOTE: If you do not want Jekyll to interfere with your other globally installed gems, you can use `bundler`: gem install bundler bundle install bundler exec jekyll serve -1. View your staged content at -[http://localhost:4000](http://localhost:4000){: target="_blank"}. + Regardless of whether you use `bundler` or not, your copy of the site will then be viewable at: [http://localhost:4000](http://localhost:4000) {% endcapture %} From 592d61c55ca2a27b534e0accc96746f27eb0b54e Mon Sep 17 00:00:00 2001 From: Eric Chiang Date: Thu, 2 Feb 2017 13:46:28 -0800 Subject: [PATCH 003/407] docs/admin/authorization: discourage the use rbac-super-user flag Even though this flag works in 1.5, its use should be discouraged. It will be removed in 1.6. --- docs/admin/authorization.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/admin/authorization.md b/docs/admin/authorization.md index d398bba2c8..ce75c189f0 100644 --- a/docs/admin/authorization.md +++ b/docs/admin/authorization.md @@ -221,20 +221,20 @@ don't already have even when the RBAC authorizer it disabled__. If "user-1" does not have the ability to read secrets in "namespace-a", they cannot create a binding that would grant that permission to themselves or any other user. -For bootstrapping the first roles, it becomes necessary for someone to get -around these limitations. For the alpha release of RBAC, an API Server flag was -added to allow one user to step around all RBAC authorization and privilege -escalation checks. NOTE: _This is subject to change with future releases._ +When bootstrapping, superuser credentials should include the `system:masters` +group, for example by creating a client cert with `/O=system:masters`. This +gives those credentials full access to the API and allows an admin to then set +up bindings for other users. + +In Kubernetes versions 1.4 and 1.5, there was a similar flag that gave a user +full access: ``` --authorization-rbac-super-user=admin ``` -Once set the specified super user, in this case "admin", can be used to create -the roles and role bindings to initialize the system. - -This flag is optional and once the initial bootstrapping is performed can be -unset. +__This flag will be removed in 1.6__. Admins should prefer the `system:masters` +group when setting up clusters. ### Roles, RolesBindings, ClusterRoles, and ClusterRoleBindings From d268a3eaf4e82d61b94ae7d19271d26905b6da42 Mon Sep 17 00:00:00 2001 From: Aneesh Kumar Date: Thu, 2 Feb 2017 01:05:15 +0530 Subject: [PATCH 004/407] Fix for redundant footer section in home page. This commit removes redundant footer section in the Home page. --- index.html | 1 - 1 file changed, 1 deletion(-) diff --git a/index.html b/index.html index 5e860832d7..30cb9264b5 100644 --- a/index.html +++ b/index.html @@ -172,7 +172,6 @@ cid: home -{% include footer.html %}
From e64c482469c7d28bfa94c4b2d5540959956a7eac Mon Sep 17 00:00:00 2001 From: David Calavera Date: Tue, 17 Jan 2017 08:41:49 -0800 Subject: [PATCH 005/407] Add redirect generator to make Netlify's CDN redirect to the right places. Netlify will redirect with a 301 status to the right places without the need of a custom javascript file. The current redirects.js is not removed so production can keep working as it is for now. Signed-off-by: David Calavera --- .gitignore | 1 + Makefile | 5 ++++- netlify.toml | 4 ++-- redirects.rb | 22 ++++++++++++++++++++++ 4 files changed, 29 insertions(+), 3 deletions(-) create mode 100644 redirects.rb diff --git a/.gitignore b/.gitignore index 460a88c521..efdcc2a53e 100644 --- a/.gitignore +++ b/.gitignore @@ -24,3 +24,4 @@ Session.vim tags kubernetes.github.io.iml +_redirects diff --git a/Makefile b/Makefile index 3657cb577a..36a4842840 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: all build build-preview help serve +.PHONY: all build build-preview generate-redirects help serve help: ## Show this help. @awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {sub("\\\\n",sprintf("\n%22c"," "), $$2);printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) @@ -11,5 +11,8 @@ build: ## Build site with production settings and put deliverables in _site. build-preview: ## Build site with drafts and future posts enabled. jekyll build --drafts --future +generate-redirects: ## Generate a redirects file and copy it into the _site directory. + mkdir -p _site && REDIRECTS_PATH=_site/_redirects ruby redirects.rb + serve: ## Boot the development server. jekyll serve diff --git a/netlify.toml b/netlify.toml index bac7e0b5ab..37e285cb2d 100644 --- a/netlify.toml +++ b/netlify.toml @@ -1,6 +1,6 @@ [build] - command = "make build" + command = "make build generate-redirects" publish = "_site" [context.deploy-preview] - command = "make build-preview" + command = "make build-preview generate-redirects" diff --git a/redirects.rb b/redirects.rb new file mode 100644 index 0000000000..3aeabbd326 --- /dev/null +++ b/redirects.rb @@ -0,0 +1,22 @@ +REPO_TMPL = "https://github.com/kubernetes/kubernetes/tree/%s/%s/:splat" + +fixed_redirects = """# 301 redirects (301 is the default status when no other one is provided for each line) +/third_party/swagger-ui /kubernetes/third_party/swagger-ui/ +/resource-quota /docs/admin/resourcequota/ +/horizontal-pod-autoscaler /docs/user-guide/horizontal-pod-autoscaling/ +/docs/user-guide/overview /docs/whatisk8s/ +/docs/roadmap https://github.com/kubernetes/kubernetes/milestones/ +/api-ref https://github.com/kubernetes/kubernetes/milestones/ +""" + +branch_redirects = ["examples" , "cluster", "docs/devel", "docs/design"] + +branch_redirects.each do |name| + dest = REPO_TMPL % [ENV.fetch("HEAD", "master"), name] + rule = "\n/#{name}/* #{dest}" + + fixed_redirects << rule +end + +output = ENV["DEBUG"] ? STDOUT : File.open(ENV.fetch("REDIRECTS_PATH", "_redirects"), "w+") +output.puts fixed_redirects From 631d1d1135b8f505b6decabd8b5c684a505c49a0 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Fri, 3 Feb 2017 10:30:33 -0800 Subject: [PATCH 006/407] Change healthz port Reimplemented changes for PR #1826 and Issue #10003 --- docs/admin/kubelet.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/kubelet.md b/docs/admin/kubelet.md index 258db1e3aa..4a57e00d60 100644 --- a/docs/admin/kubelet.md +++ b/docs/admin/kubelet.md @@ -96,7 +96,7 @@ StreamingProxyRedirects=true|false (ALPHA - default=false) --google-json-key string The Google Cloud Platform Service Account JSON Key to use for authentication. --hairpin-mode string How should the kubelet setup hairpin NAT. This allows endpoints of a Service to loadbalance back to themselves if they should try to access their own Service. Valid values are "promiscuous-bridge", "hairpin-veth" and "none". (default "promiscuous-bridge") --healthz-bind-address ip The IP address for the healthz server to serve on, defaulting to 127.0.0.1 (set to 0.0.0.0 for all interfaces) (default 127.0.0.1) - --healthz-port int32 The port of the localhost healthz endpoint (default 10248) + --healthz-port int32 (Deprecated) The port of the localhost healthz endpoint (default 10248) --host-ipc-sources stringSlice Comma-separated list of sources from which the Kubelet allows pods to use the host ipc namespace. [default="*"] (default [*]) --host-network-sources stringSlice Comma-separated list of sources from which the Kubelet allows pods to use of host network. [default="*"] (default [*]) --host-pid-sources stringSlice Comma-separated list of sources from which the Kubelet allows pods to use the host pid namespace. [default="*"] (default [*]) @@ -137,7 +137,7 @@ StreamingProxyRedirects=true|false (ALPHA - default=false) --pods-per-core int32 Number of Pods per core that can run on this Kubelet. The total number of Pods on this Kubelet cannot exceed max-pods, so max-pods will be used if this calculation results in a larger number of Pods allowed on the Kubelet. A value of 0 disables this limit. --port int32 The port for the Kubelet to serve on. (default 10250) --protect-kernel-defaults Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. - --read-only-port int32 The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255) + --read-only-port int32 The read-only port for the Kubelet to serve on with no authentication/authorization, and for localhost healthz endpoint (set to 0 to disable) (default 10255) --really-crash-for-testing If true, when panics occur crash. Intended for testing. --register-node Register the node with the apiserver (defaults to true if --api-servers is set) (default true) --register-schedulable Register the node as schedulable. Won't have any effect if register-node is false. [default=true] (default true) From 69cc2e60a225f84a55be76d2a920febde18f9131 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Fri, 3 Feb 2017 11:55:13 -0800 Subject: [PATCH 007/407] Add object fields to style guide. --- docs/contribute/style-guide.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/docs/contribute/style-guide.md b/docs/contribute/style-guide.md index 6bb1aa5837..d44cab2ea4 100644 --- a/docs/contribute/style-guide.md +++ b/docs/contribute/style-guide.md @@ -17,7 +17,7 @@ docs, follow the instructions on ## Documentation formatting standards -### Use Camel Case for API objects +### Use camel case for API objects When you refer to an API object, use the same uppercase and lowercase letters that are used in the actual object name. Typically, the names of API @@ -75,7 +75,7 @@ represents. Open the /_data/concepts.yaml file.Open the /_data/concepts.yaml file. -## Code snippet formatting +## Inline code formatting ### Use code style for inline code and commands @@ -84,10 +84,31 @@ document, use the backtick (`). - +
DoDon't
Set the value of the replicas field in the configuration file.Set the value of the "replicas" field in the configuration file.
The kubectl run command creates a Deployment.The "kubectl run" command creates a Deployment.
For declarative management, use kubectl apply.For declarative management, use "kubectl apply".
+### Use code style for object field names + + + + + +
DoDon't
Set the value of the replicas field in the configuration file.Set the value of the "replicas" field in the configuration file.
The value of the exec field is an ExecAction object.The value of the "exec" field is an ExecAction object.
+ +### Use normal style for string and integer field values + +For field values of type string or integer, use normal style without quotation marks. + + + + + + +
DoDon't
Set the value of imagePullPolicy to Always.Set the value of imagePullPolicy to "Always".
Set the value of image to nginx:1.8.Set the value of image to nginx:1.8.
Set the value of the replicas field to 2.Set the value of the replicas field to 2.
+ +## Code snippet formatting + ### Don't include the command prompt From 70a2934d02d79a8ad03c270ed58438d4267ec624 Mon Sep 17 00:00:00 2001 From: Cole Mickens Date: Fri, 3 Feb 2017 13:16:20 -0800 Subject: [PATCH 008/407] userguide: PV docs clarify Azure storage account restriction --- docs/user-guide/persistent-volumes/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index d6245e8198..ae34668c9e 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -532,7 +532,7 @@ parameters: * `skuName`: Azure storage account Sku tier. Default is empty. * `location`: Azure storage account location. Default is empty. -* `storageAccount`: Azure storage account name. If storage account is not provided, all storage accounts associated with the resource group are searched to find one that matches `skuName` and `location`. If storage account is provided, `skuName` and `location` are ignored. +* `storageAccount`: Azure storage account name. If storage account is not provided, all storage accounts associated with the resource group are searched to find one that matches `skuName` and `location`. If storage account is provided, it must reside in the same resource group as the cluster, and `skuName` and `location` are ignored. ## Writing Portable Configuration From 87eae080f7c51c9103046ea9d597e5be3dfd93a4 Mon Sep 17 00:00:00 2001 From: Wei Ma Date: Sat, 4 Feb 2017 09:20:39 +0800 Subject: [PATCH 009/407] Add code comment for "replicas" unnecessary code comment been removed. --- docs/tutorials/stateless-application/deployment-scale.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/stateless-application/deployment-scale.yaml b/docs/tutorials/stateless-application/deployment-scale.yaml index 2968b88360..7259c97603 100644 --- a/docs/tutorials/stateless-application/deployment-scale.yaml +++ b/docs/tutorials/stateless-application/deployment-scale.yaml @@ -3,7 +3,7 @@ kind: Deployment metadata: name: nginx-deployment spec: - replicas: 4 + replicas: 4 # Update the replicas from 2 to 4 template: metadata: labels: @@ -11,6 +11,6 @@ spec: spec: containers: - name: nginx - image: nginx:1.8 # Update the version of nginx from 1.7.9 to 1.8 + image: nginx:1.8 ports: - containerPort: 80 From bfe683c3fd5ccef3e2559508b2c7c1032fa1ee5b Mon Sep 17 00:00:00 2001 From: David Calavera Date: Sat, 4 Feb 2017 20:25:31 +0100 Subject: [PATCH 010/407] Add documentation to the redirects.rb script. Signed-off-by: David Calavera --- redirects.rb | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/redirects.rb b/redirects.rb index 3aeabbd326..4cb238a811 100644 --- a/redirects.rb +++ b/redirects.rb @@ -1,3 +1,53 @@ +# This script generates a redirects file that Netlify's CDN +# can use to forward visitor to the right content. +# +# You can read more details about this file in Netlify documentation: +# +# https://www.netlify.com/docs/redirects/ +# +# USAGE: +# +# Run this script with Ruby to generate the _redirects file in this repository. +# It works with any Ruby version higher than 1.8. +# +# ruby redirects.rb +# +# If you want to test a change without modifying the content of the current file, +# you can print the output of the script setting the DEBUG environment variable +# when you run the script: +# +# DEBUG=1 ruby redirects.rb +# +# You can test if the content generated is correct in this playground: +# +# https://play.netlify.com/redirects +# +# HOW TO ADD NEW RULES: +# +# This script is divided in two sections. +# +# The first section handles static redirects, +# those that you know the old path and the new path and never change. +# +# If you want to add one of these redirects, add the rule to the `fixed_redirects` variable, +# in a new line before the closing """. The format for basic 301 redirects is the following one, +# check Netlify's documentation linked above for other rules: +# +# /OLD_PATH /NEW_PATH_OR_URL +# +# The second section handles redirects that change depending on the branch that's deployed +# in the site. For instance, when you want to redirect a path to content in +# the kubernetes main repository but the content is specific to a branch that +# matches the deployed branch. +# +# If you want to add one of there redirects, add the old path to the `branch_redirects` list. For instance, +# when you deploy the branch release-1.5 on Netlify, this script will generate the following redirects: +# +# /examples/* https://github.com/kubernetes/kubernetes/tree/release-1.5/examples/:splat +# /cluster/* https://github.com/kubernetes/kubernetes/tree/release-1.5/cluster/:splat +# /docs/devel/* https://github.com/kubernetes/kubernetes/tree/release-1.5/docs/devel/:splat +# /docs/design/* https://github.com/kubernetes/kubernetes/tree/release-1.5/docs/design/:splat +# REPO_TMPL = "https://github.com/kubernetes/kubernetes/tree/%s/%s/:splat" fixed_redirects = """# 301 redirects (301 is the default status when no other one is provided for each line) From ced286161c2151c780d27f8debd9fd1b31b26b7f Mon Sep 17 00:00:00 2001 From: Shiyang Wang Date: Sat, 4 Feb 2017 13:40:26 +0800 Subject: [PATCH 011/407] fix typo --- .../tools/kubectl/object-management-using-declarative-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/tools/kubectl/object-management-using-declarative-config.md b/docs/concepts/tools/kubectl/object-management-using-declarative-config.md index 5033bbac8c..6c8dfbd250 100644 --- a/docs/concepts/tools/kubectl/object-management-using-declarative-config.md +++ b/docs/concepts/tools/kubectl/object-management-using-declarative-config.md @@ -382,7 +382,7 @@ is used to identify fields that have been removed from the configuration file and need to be cleared from the live configuration. Here are the steps used to caluculate which fields should be deleted or set: -1. Calculate the fields to delete. Thes are the fields present in `last-applied-configuration` and missing from the configuration file. +1. Calculate the fields to delete. These are the fields present in `last-applied-configuration` and missing from the configuration file. 2. Calculate the fields to add or set. These are the fields present in the configuration file whose values don't match the live configuration. Here's an example. Suppose this is the configuration file for a Deployment object: From 7ce1b540c61f794c6a3ec82fb7c3bd89a6e501ce Mon Sep 17 00:00:00 2001 From: Pan Luo Date: Fri, 3 Feb 2017 16:13:58 -0800 Subject: [PATCH 012/407] Add generic to create secret command The generic is missing from the command and will cause error: `Error: unknown flag: --type` when running the command. --- docs/user-guide/persistent-volumes/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index ae34668c9e..3ecc8ea99f 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -423,7 +423,7 @@ parameters: * `restuserkey` : Gluster REST service/Heketi user's password which will be used for authentication to the REST server. This parameter is deprecated in favor of `secretNamespace` + `secretName`. * `secretNamespace` + `secretName` : Identification of Secret instance that containes user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must have type "kubernetes.io/glusterfs", e.g. created in this way: ``` - $ kubectl create secret heketi-secret --type="kubernetes.io/glusterfs" --from-literal=key='opensesame' --namespace=default + $ kubectl create secret generic heketi-secret --type="kubernetes.io/glusterfs" --from-literal=key='opensesame' --namespace=default ``` #### OpenStack Cinder @@ -482,7 +482,7 @@ parameters: * `userId`: Ceph client ID that is used to map the RBD image. Default is the same as `adminId`. * `userSecretName`: The name of Ceph Secret for `userId` to map RBD image. It must exist in the same namespace as PVCs. This parameter is required. The provided secret must have type "kubernetes.io/rbd", e.g. created in this way: ``` - $ kubectl create secret ceph-secret --type="kubernetes.io/rbd" --from-literal=key='QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ==' --namespace=kube-system + $ kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-literal=key='QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ==' --namespace=kube-system ``` #### Quobyte @@ -509,7 +509,7 @@ parameters: * `adminSecretNamespace`: The namespace for `adminSecretName`. Default is "default". * `adminSecretName`: secret that holds information about the Quobyte user and the password to authenticate agains the API server. The provided secret must have type "kubernetes.io/quobyte", e.g. created in this way: ``` - $ kubectl create secret quobyte-admin-secret --type="kubernetes.io/quobyte" --from-literal=key='opensesame' --namespace=kube-system + $ kubectl create secret generic quobyte-admin-secret --type="kubernetes.io/quobyte" --from-literal=key='opensesame' --namespace=kube-system ``` * `user`: maps all access to this user. Default is "root". * `group`: maps all access to this group. Default is "nfsnobody". From b6149571ce8b377bb5a8e5e9252ddaf4bee24bdb Mon Sep 17 00:00:00 2001 From: Philipp T Date: Sat, 4 Feb 2017 19:38:07 +0100 Subject: [PATCH 013/407] Update kubeadm.md grammar ocd --- docs/getting-started-guides/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index a005ed0c56..a4e5679ffb 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -357,6 +357,6 @@ Please note: `kubeadm` is a work in progress and these limitations will be addre 1. If you are using VirtualBox (directly or via Vagrant), you will need to ensure that `hostname -i` returns a routable IP address (i.e. one on the second network interface, not the first one). By default, it doesn't do this and kubelet ends-up using first non-loopback network interface, which is usually NATed. - Workaround: Modify `/etc/hosts`, take a look at this [`Vagrantfile`][ubuntu-vagrantfile] for how you this can be achieved. + Workaround: Modify `/etc/hosts`, take a look at this [`Vagrantfile`][ubuntu-vagrantfile] for how this can be achieved. [ubuntu-vagrantfile]: https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11), From 7815d386ef289e13eb23651acf0dc0ea0c821863 Mon Sep 17 00:00:00 2001 From: Vasily Upornikov Date: Sun, 5 Feb 2017 13:38:13 +0300 Subject: [PATCH 014/407] Update container-command-args.md --- docs/concepts/configuration/container-command-args.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/configuration/container-command-args.md b/docs/concepts/configuration/container-command-args.md index 54fe3afcb9..25d555f702 100644 --- a/docs/concepts/configuration/container-command-args.md +++ b/docs/concepts/configuration/container-command-args.md @@ -13,7 +13,7 @@ fields to override the default Entrypoint and Cmd of the the Container's image. ## Container entry points and arguments -The configuration file for a Container has an `image` field that specifies the +The configuration file for a Container has an `image` field that specifies the Docker image to be run in the Container. A Docker image has metadata that includes a default Entrypoint and a default Cmd. From ae0844f4773862dc2d5a2d6ea62a341796d2f53b Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 6 Feb 2017 11:08:46 +0800 Subject: [PATCH 015/407] Update kubefed.md --- docs/admin/federation/kubefed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/federation/kubefed.md b/docs/admin/federation/kubefed.md index 9685271b85..695edf7bee 100644 --- a/docs/admin/federation/kubefed.md +++ b/docs/admin/federation/kubefed.md @@ -36,7 +36,7 @@ in your `$PATH` and set the executable permission on those binaries. Note: The URL in the curl command below downloads the binaries for Linux amd64. If you are on a different platform, please use the URL for the binaries appropriate for your platform. You can find the list -of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-3) +of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-3). ```shell From 4f390bc64f5958ed050478eb225caba4d70b0dd2 Mon Sep 17 00:00:00 2001 From: caiyixiang Date: Mon, 6 Feb 2017 11:35:24 +0800 Subject: [PATCH 016/407] Update minikube.md --- docs/getting-started-guides/minikube.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting-started-guides/minikube.md b/docs/getting-started-guides/minikube.md index 4f4a432185..e0e79364cf 100644 --- a/docs/getting-started-guides/minikube.md +++ b/docs/getting-started-guides/minikube.md @@ -40,7 +40,7 @@ See the installation instructions for the [latest release](https://github.com/ku ## Quickstart Here's a brief demo of minikube usage. -If you want to change the VM driver add the appropriate `--vm-driver=xxx` flag to `minikube start`. Minikube Supports +If you want to change the VM driver add the appropriate `--vm-driver=xxx` flag to `minikube start`. Minikube supports the following drivers: * virtualbox @@ -106,7 +106,7 @@ plugins, if required. When using a single VM of Kubernetes, it's really handy to reuse the minikube's built-in Docker daemon; as this means you don't have to build a docker registry on your host machine and push the image into it - you can just build inside the same docker daemon as minikube which speeds up local experiments. Just make sure you tag your Docker image with something other than 'latest' and use that tag while you pull the image. Otherwise, if you do not specify version of your image, it will be assumed as `:latest`, with pull image policy of `Always` correspondingly, which may eventually result in `ErrImagePull` as you may not have any versions of your Docker image out there in the default docker registry (usually DockerHub) yet. -To be able to work with the docker daemon on your mac/linux host use the [docker-env command](./docs/minikube_docker-env.md) in your shell: +To be able to work with the docker daemon on your mac/linux host use the [docker-env command](https://github.com/kubernetes/minikube/blob/master/docs/minikube_docker-env.md) in your shell: ``` eval $(minikube docker-env) From 5424759ab141a6cfd1be9fad1ff613cf5e2cdcef Mon Sep 17 00:00:00 2001 From: yupengzte Date: Sat, 4 Feb 2017 10:46:04 +0800 Subject: [PATCH 017/407] Fix the url error Signed-off-by: yupengzte --- docs/admin/limitrange/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/limitrange/index.md b/docs/admin/limitrange/index.md index 3a31b9e732..82bed781f1 100644 --- a/docs/admin/limitrange/index.md +++ b/docs/admin/limitrange/index.md @@ -30,7 +30,7 @@ This example demonstrates how limits can be applied to a Kubernetes [namespace]( min/max resource limits per pod. In addition, this example demonstrates how you can apply default resource limits to pods in the absence of an end-user specified value. -See [LimitRange design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/) +See [LimitRange design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/) ## Step 0: Prerequisites From 5253297f52e9122374d56a8b8e3957955da40c99 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Mon, 6 Feb 2017 15:06:55 +0800 Subject: [PATCH 018/407] Update namspace and service part of ui.md 1. Namespace can not contain capital letter. 2. For service, internal endpoints for cluster connections and external endpoints for external users. Here, it lost "external". --- docs/user-guide/ui.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index da2b2e611c..b2acc5e8ed 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -97,7 +97,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. - Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-). + Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-), can not contain capital letters. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. @@ -147,7 +147,7 @@ Detail views for workloads show status and specification information and surface ![Deployment detail view](/images/docs/ui-dashboard-deployment-detail.png) #### Services and discovery -Services and discovery view shows Kubernetes resources that allow for exposing services to external world and discovering them within a cluster. For that reason, Service and Ingress views show Pods targeted by them, internal endpoints for cluster connections and endpoints for external users. +Services and discovery view shows Kubernetes resources that allow for exposing services to external world and discovering them within a cluster. For that reason, Service and Ingress views show Pods targeted by them, internal endpoints for cluster connections and external endpoints for external users. ![Service list partial view](/images/docs/ui-dashboard-service-list.png) From cff68f53af6c5276838e3a8a4922ded579d5e0dc Mon Sep 17 00:00:00 2001 From: sallydeng Date: Tue, 7 Feb 2017 08:35:21 +0800 Subject: [PATCH 019/407] Update ui.md base on comments Former: The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) , can not contain capital letters. Now: The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. --- docs/user-guide/ui.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index b2acc5e8ed..5d6567a62c 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -97,7 +97,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. - Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-), can not contain capital letters. + Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. From 494f011dd25c36d9ed25586012c9b20311500453 Mon Sep 17 00:00:00 2001 From: Ben Spoon Date: Tue, 7 Feb 2017 13:38:47 -0800 Subject: [PATCH 020/407] Fix overflow scroll on code blocks Fixes https://github.com/kubernetes/kubernetes.github.io/issues/633 --- _sass/_base.sass | 1 + 1 file changed, 1 insertion(+) diff --git a/_sass/_base.sass b/_sass/_base.sass index 3ad4f81dc5..72557bb983 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -856,6 +856,7 @@ dd display: block margin: 20px 0 padding: 15px + position: relative overflow-x: auto h1 code, h2 code, h3 code, h4 code, h5 code, h6 code From 007caa5ef19d1fc5abaa497117d135f1187929dd Mon Sep 17 00:00:00 2001 From: tailnode Date: Tue, 7 Feb 2017 19:40:15 +0800 Subject: [PATCH 021/407] fix link --- _includes/v1.5/v1-definitions.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_includes/v1.5/v1-definitions.html b/_includes/v1.5/v1-definitions.html index faaa0e847d..16858392bc 100755 --- a/_includes/v1.5/v1-definitions.html +++ b/_includes/v1.5/v1-definitions.html @@ -8146,7 +8146,7 @@ The resulting set of endpoints can be viewed as:
 - + @@ -8263,4 +8263,4 @@ Last updated 2016-11-17 06:26:10 UTC - \ No newline at end of file + From 24d587a9c593e073a2c36c32d8dc286f81124ccb Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Tue, 7 Feb 2017 10:59:04 +0800 Subject: [PATCH 022/407] fix unreachable links and typos 1. fix unreachable federated resources links 2. federation proposal doc has been relocated, so i fix this link 3. make federation resources supporting cascading deletion clickable 4. make link to `Kubernetes Scaling and Performance Goals` clickable 5. other typo fixes --- docs/user-guide/federation/index.md | 36 ++++++++++++++--------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/user-guide/federation/index.md b/docs/user-guide/federation/index.md index 7ba0ca9c60..10cea0ad6e 100644 --- a/docs/user-guide/federation/index.md +++ b/docs/user-guide/federation/index.md @@ -43,7 +43,7 @@ why you might want multiple clusters are: [Multi cluster guide](/docs/admin/multi-cluster) has more details on this. * Scalability: There are scalability limits to a single kubernetes cluster (this should not be the case for most users. For more details: - https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md). + [Kubernetes Scaling and Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md)). * Hybrid cloud: You can have multiple clusters on different cloud providers or on-premises data centers. @@ -53,7 +53,7 @@ why you might want multiple clusters are: While there are a lot of attractive use cases for federation, there are also some caveats. -* Increased network bandwidth and cost: The dederation control plane watches all +* Increased network bandwidth and cost: The federation control plane watches all clusters to ensure that the current state is as expected. This can lead to significant network cost if the clusters are running in different regions on a cloud provider or on different cloud providers. @@ -93,15 +93,15 @@ Once we have the control plane setup, we can start creating federation API resources. The following guides explain some of the resources in detail: -* [ConfigMap](/docs/user-guide/federation/configmap/) -* [DaemonSets](/docs/user-guide/federation/daemonsets/) -* [Deployment](/docs/user-guide/federation/deployment/) -* [Events](/docs/user-guide/federation/events/) -* [Ingress](/docs/user-guide/federation/federated-ingress/) -* [Namespaces](/docs/user-guide/federation/namespaces/) -* [ReplicaSets](/docs/user-guide/federation/replicasets/) -* [Secrets](/docs/user-guide/federation/secrets/) -* [Services](/docs/user-guide/federation/federated-services/) +* [ConfigMap](https://kubernetes.io/docs/user-guide/federation/configmap/) +* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) +* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) +* [Events](https://kubernetes.io/docs/user-guide/federation/events/) +* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) +* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) +* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) +* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) +* [Services](https://kubernetes.io/docs/user-guide/federation/federated-services/) [API reference docs](/docs/federation/api-reference/) lists all the resources supported by federation apiserver. @@ -119,12 +119,12 @@ federation control plane. The following Federated resources are affected by cascading deletion: -* Ingress -* Namespace -* ReplicaSet -* Secret -* Deployment -* DaemonSet +* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) +* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) +* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) +* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) +* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) +* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) Note: By default, deleting a resource from federation control plane does not delete the corresponding resources from underlying clusters. @@ -133,5 +133,5 @@ delete the corresponding resources from underlying clusters. ## For more information * [Federation - proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/federation.md) + proposal](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/federation.md) * [Kubecon2016 talk on federation](https://www.youtube.com/watch?v=pq9lbkmxpS8) From 72d37e9a33252b69f7a59e149114e7068419135b Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 8 Feb 2017 09:48:41 +0800 Subject: [PATCH 023/407] Update assign-cpu-ram-container.md --- docs/tasks/configure-pod-container/assign-cpu-ram-container.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/assign-cpu-ram-container.md b/docs/tasks/configure-pod-container/assign-cpu-ram-container.md index 24cca3eca5..fc62af79d9 100644 --- a/docs/tasks/configure-pod-container/assign-cpu-ram-container.md +++ b/docs/tasks/configure-pod-container/assign-cpu-ram-container.md @@ -4,7 +4,7 @@ title: Assigning CPU and RAM Resources to a Container {% capture overview %} -This page shows how assign CPU and RAM resources to containers running +This page shows how to assign CPU and RAM resources to containers running in a Kubernetes Pod. {% endcapture %} From 02d938a3d572fafbf889b0e531947595542a8429 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Wed, 8 Feb 2017 16:38:00 +0800 Subject: [PATCH 024/407] Update ui.md: namespace name can't be number only When the namespace name only contains number, such as "10", the pod can't be created in this kind of namespace, the pod will be put in default namespace. --- docs/user-guide/ui.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index 5d6567a62c..3456a7e98a 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -98,6 +98,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. + When the namespace name only contains number, such as "10", the pod can't be created in this kind of namespace, the pod will be put in default namespace. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. From 5dda4058f1d15ad939b168d85359d76cdc8bcabf Mon Sep 17 00:00:00 2001 From: Andrew Martin Date: Wed, 8 Feb 2017 14:55:50 +0000 Subject: [PATCH 025/407] Fix typo `pages` -> `page` --- docs/getting-started-guides/ubuntu/backups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/backups.md b/docs/getting-started-guides/ubuntu/backups.md index 9c0c741d9b..b839cafee4 100644 --- a/docs/getting-started-guides/ubuntu/backups.md +++ b/docs/getting-started-guides/ubuntu/backups.md @@ -3,7 +3,7 @@ title: Backups --- {% capture overview %} -This pages shows you how to backup and restore data from the different deployed services in a given cluster. +This page shows you how to backup and restore data from the different deployed services in a given cluster. {% endcapture %} {% capture prerequisites %} From 5cc692a6fd563df082fb0cefa4b2df41724e6edb Mon Sep 17 00:00:00 2001 From: Ghe Rivero Date: Wed, 8 Feb 2017 11:19:11 +0100 Subject: [PATCH 026/407] Fix examples display --- docs/admin/authorization.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/admin/authorization.md b/docs/admin/authorization.md index ce75c189f0..e5c091b772 100644 --- a/docs/admin/authorization.md +++ b/docs/admin/authorization.md @@ -445,6 +445,7 @@ subjects: ``` For all authenticated users: + ```yaml subjects: - kind: Group @@ -452,6 +453,7 @@ subjects: ``` For all unauthenticated users: + ```yaml subjects: - kind: Group @@ -459,6 +461,7 @@ subjects: ``` For all users: + ```yaml subjects: - kind: Group From 3bac04bf7a00fe48403bdb71e0c9ce5dd5d9ff38 Mon Sep 17 00:00:00 2001 From: Eamon Taaffe Date: Wed, 8 Feb 2017 15:43:16 +1100 Subject: [PATCH 027/407] Missing export statement in example --- docs/user-guide/jobs/work-queue-1/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-1/index.md b/docs/user-guide/jobs/work-queue-1/index.md index b2b68cff23..4ce1ffb667 100644 --- a/docs/user-guide/jobs/work-queue-1/index.md +++ b/docs/user-guide/jobs/work-queue-1/index.md @@ -101,7 +101,7 @@ Next we will verify we can create a queue, and publish and consume messages. # In the next line, rabbitmq-service is the hostname where the rabbitmq-service # can be reached. 5672 is the standard port for rabbitmq. -root@temp-loe07:/# BROKER_URL=amqp://guest:guest@rabbitmq-service:5672 +root@temp-loe07:/# export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672 # If you could not resolve "rabbitmq-service" in the previous step, # then use this command instead: # root@temp-loe07:/# BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672 From 839854fe7bd1802a847883eb973b90562944a394 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 10:39:32 -0800 Subject: [PATCH 028/407] Remove User Guide topic: Launching, Exposing, --- docs/user-guide/quick-start.md | 67 +--------------------------------- 1 file changed, 2 insertions(+), 65 deletions(-) diff --git a/docs/user-guide/quick-start.md b/docs/user-guide/quick-start.md index 6cef04810d..67dfb1dc77 100644 --- a/docs/user-guide/quick-start.md +++ b/docs/user-guide/quick-start.md @@ -5,69 +5,6 @@ assignees: title: Launching, Exposing, and Killing Applications --- -This guide will help you get oriented to Kubernetes and running your first containers on the cluster. If you are already familiar with the docker-cli, you can also checkout the docker-cli to kubectl migration guide [here](/docs/user-guide/docker-cli-to-kubectl). +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Launching a simple application, and exposing it to the Internet - -Once your application is packaged into a container and pushed to an image registry, you're ready to deploy it to Kubernetes. -Through integration with some cloud providers (for example Google Compute Engine, AWS EC2, and Azure ACS), Kubernetes also enables you to request it to provision a public IP address for your application. - -For example, [nginx](http://wiki.nginx.org/Main) is a popular HTTP server, with a [pre-built container on Docker hub](https://registry.hub.docker.com/_/nginx/). The [`kubectl run`](/docs/user-guide/kubectl/kubectl_run) commands below will create two nginx replicas, listening on port 80, and a public IP address for your application. - -```shell -$ kubectl run my-nginx --image=nginx --replicas=2 --port=80 -deployment "my-nginx" created -``` - -To expose your service to the public Internet, run: - -```shell -$ kubectl expose deployment my-nginx --target-port=80 --type=LoadBalancer -service "my-nginx" exposed -``` -Note: The type, LoadBalancer, is highly dependent upon the underlying platform that Kubernetes is running on. If your cloud provider doesn't have a load balancer implementation (e.g. OpenStack) for Kubernetes, you can simply use the allocated [NodePort](http://kubernetes.io/docs/user-guide/services/#type-nodeport) as a rudimentary form of load balancing across your endpoints. - -You can see that they are running by: - -```shell -$ kubectl get po -NAME READY STATUS RESTARTS AGE -my-nginx-3800858182-h9v8d 1/1 Running 0 1m -my-nginx-3800858182-wqafx 1/1 Running 0 1m -``` - -Kubernetes will ensure that your application keeps running, by automatically restarting containers that fail, spreading containers across nodes, and recreating containers on new nodes when nodes fail. - -To find the public IP address assigned to your application, execute: - -```shell -$ kubectl get service my-nginx -NAME CLUSTER_IP EXTERNAL_IP PORT(S) AGE -my-nginx 10.179.240.1 25.1.2.3 80/TCP 8s -``` - -You may need to wait for a minute or two for the external IP address to be provisioned. - -In order to access your nginx landing page, you also have to make sure that traffic from external IPs is allowed. Do this by opening a [firewall to allow traffic on port 80](/docs/user-guide/services-firewalls). - -If you're running on AWS, Kubernetes creates an ELB for you. ELBs use host -names, not IPs, so you will have to do `kubectl describe service/my-nginx` and look -for the `LoadBalancer Ingress` host name. Traffic from external IPs is allowed -automatically. - -## Killing the application - -To kill the application and delete its containers and public IP address, do: - -```shell -$ kubectl delete deployment,service my-nginx -deployment "my-nginx" deleted -service "my-nginx" deleted -``` - -## What's next? - -* [Learn about how to configure common container parameters, such as commands and environment variables.](/docs/user-guide/configuring-containers) +[Using a Service to Access an Application in a Cluster](https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address-service/) From bb6684e16f1b4c4d15c0949a58dde3bc81248af1 Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 8 Feb 2017 17:56:59 +0800 Subject: [PATCH 029/407] Update debugging-services.md --- docs/user-guide/debugging-services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/debugging-services.md b/docs/user-guide/debugging-services.md index de9cec5c3f..6a1a692e8a 100644 --- a/docs/user-guide/debugging-services.md +++ b/docs/user-guide/debugging-services.md @@ -352,7 +352,7 @@ they are running fine and not crashing. The `-l app=hostnames` argument is a label selector - just like our `Service` has. Inside the Kubernetes system is a control loop which evaluates the -selector of every `Service` and save the results into an `Endpoints` object. +selector of every `Service` and saves the results into an `Endpoints` object. ```shell $ kubectl get endpoints hostnames From 09af0b070f185df7305d4563ebd42c892edc2a2d Mon Sep 17 00:00:00 2001 From: James Duncan Date: Thu, 9 Feb 2017 08:27:42 +1300 Subject: [PATCH 030/407] Corrected syntax of Networking Section Removed an unnecessary 'the' from a sentence. --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index 07f0d18934..f8b9cf2f72 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -41,7 +41,7 @@ Pods provide two kinds of shared resources for their constituent containers: *ne #### Networking -Each Pod is assigned a unique IP address. Every the container in a Pod shares the network namespace, including the IP address and network ports. Containers *inside a Pod* can communicate with one another using `localhost`. When containers in a Pod communicate with entities *outside the Pod*, they must coordinate how they use the shared network resources (such as ports). +Each Pod is assigned a unique IP address. Every container in a Pod shares the network namespace, including the IP address and network ports. Containers *inside a Pod* can communicate with one another using `localhost`. When containers in a Pod communicate with entities *outside the Pod*, they must coordinate how they use the shared network resources (such as ports). #### Storage From c125531513f030a8d563330bf1799b2a3142fff0 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 13:56:18 -0800 Subject: [PATCH 031/407] Remove Guide topic: Deploying Applications. --- docs/user-guide/deploying-applications.md | 94 +---------------------- 1 file changed, 2 insertions(+), 92 deletions(-) diff --git a/docs/user-guide/deploying-applications.md b/docs/user-guide/deploying-applications.md index 9bbcd34282..a1c04e1003 100644 --- a/docs/user-guide/deploying-applications.md +++ b/docs/user-guide/deploying-applications.md @@ -6,96 +6,6 @@ assignees: title: Deploying Applications --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Launching a set of replicas using a configuration file - -Kubernetes creates and manages sets of replicated containers (actually, replicated [Pods](/docs/user-guide/pods)) using [*Deployments*](/docs/user-guide/deployments). - -A Deployment simply ensures that a specified number of pod "replicas" are running at any one time. If there are too many, it will kill some. If there are too few, it will start more. It's analogous to Google Compute Engine's [Instance Group Manager](https://cloud.google.com/compute/docs/instance-groups/manager/) or AWS's [Auto-scaling Group](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AutoScalingGroup.html) (with no scaling policies). - -The Deployment created to run nginx by `kubectl run` in the [Quick start](/docs/user-guide/quick-start) could be specified using YAML as follows: - -{% include code.html language="yaml" file="run-my-nginx.yaml" ghlink="/docs/user-guide/run-my-nginx.yaml" %} - -Some differences compared to specifying just a pod are that the `kind` is `Deployment`, the number of `replicas` desired is specified, and the pod specification is under the `template` field. The names of the pods don't need to be specified explicitly because they are generated from the name of the Deployment. -View the [Deployment API -object](/docs/api-reference/extensions/v1beta1/definitions/#_v1beta1_deployment) -to view the list of supported fields. - -This Deployment can be created using `create`, just as with pods: - -```shell -$ kubectl create -f ./run-my-nginx.yaml -deployment "my-nginx" created -``` - -Unlike in the case where you directly create pods, a Deployment replaces pods that are deleted or terminated for any reason, such as in the case of node failure. For this reason, we recommend that you use a Deployment for a continuously running application even if your application requires only a single pod, in which case you can omit `replicas` and it will default to a single replica. - -## Viewing Deployment status - -You can view the Deployment you created using `get`: - -```shell -$ kubectl get deployment -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE -my-nginx 2 2 2 2 6s -``` - -This tells you that your Deployment will ensure that you have two nginx replicas (desired replicas = 2). - -You can see those replicas using `get`, just as with pods you created directly: - -```shell -$ kubectl get pods -NAME READY STATUS RESTARTS AGE -my-nginx-3800858182-9hk43 1/1 Running 0 8m -my-nginx-3800858182-e529s 1/1 Running 0 8m -``` - -## Deleting Deployments - -When you want to kill your application, delete your Deployment, as in the [Quick start](/docs/user-guide/quick-start): - -```shell -$ kubectl delete deployment/my-nginx -deployment "my-nginx" deleted -``` - -By default, this will also cause the pods managed by the Deployment to be deleted. If there were a large number of pods, this may take a while to complete. If you want to leave the pods running instead, specify `--cascade=false`. - -If you try to delete the pods before deleting the Deployments, it will just replace them, as it is supposed to do. - -## Labels - -Kubernetes uses user-defined key-value attributes called [*labels*](/docs/user-guide/labels) to categorize and identify sets of resources, such as pods and Deployments. The example above specified a single label in the pod template, with key `run` and value `my-nginx`. All pods created carry that label, which can be viewed using `-L`: - -```shell -$ kubectl get pods -L run -NAME READY STATUS RESTARTS AGE RUN -my-nginx-3800858182-1v53o 1/1 Running 0 46s my-nginx -my-nginx-3800858182-2ds1q 1/1 Running 0 46s my-nginx -``` - -The labels from the pod template are copied to the Deployment's labels by default, as well -- all resources in Kubernetes support labels: - -```shell -$ kubectl get deployment/my-nginx -L run -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE RUN -my-nginx 2 2 2 2 2m my-nginx -``` - -More importantly, the pod template's labels are used to create a [`selector`](/docs/user-guide/labels/#label-selectors) that will match pods carrying those labels. You can see this field by requesting it using the [Go template output format of `kubectl get`](/docs/user-guide/kubectl/kubectl_get): - -```shell{% raw %} -$ kubectl get deployment/my-nginx -o template --template="{{.spec.selector}}" -map[matchLabels:map[run:my-nginx]]{% endraw %} -``` - -You could also specify the `selector` explicitly, such as if you wanted to specify labels in the pod template that you didn't want to select on, but you should ensure that the selector will match the labels of the pods created from the pod template, and that it won't match pods created by other Deployments. The most straightforward way to ensure the latter is to create a unique label value for the Deployment, and to specify it in both the pod template's labels and in the selector's -matchLabels. - -## What's next? - -[Learn about exposing applications to users and clients, and connecting tiers of your application together.](/docs/user-guide/connecting-applications) +[Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) From 97a5a10ae4e27b42bd0d0bf9047ead10309fd7e9 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 14:10:23 -0800 Subject: [PATCH 032/407] Remove Guide topic: Secrets Walkthrough. --- docs/user-guide/secrets/walkthrough.md | 58 +------------------------- 1 file changed, 2 insertions(+), 56 deletions(-) diff --git a/docs/user-guide/secrets/walkthrough.md b/docs/user-guide/secrets/walkthrough.md index 11ca7672df..bb069c675e 100644 --- a/docs/user-guide/secrets/walkthrough.md +++ b/docs/user-guide/secrets/walkthrough.md @@ -1,61 +1,7 @@ --- -assignees: title: Secrets Walkthrough --- -Following this example, you will create a secret and a [pod](/docs/user-guide/pods/) that consumes that secret in a [volume](/docs/user-guide/volumes/). See [Secrets design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/secrets.md) for more information. +{% include user-guide-content-moved.md %} -## Step Zero: Prerequisites - -This example assumes you have a Kubernetes cluster installed and running, and that you have -installed the `kubectl` command line tool somewhere in your path. Please see the [getting -started](/docs/getting-started-guides/) for installation instructions for your platform. - -## Step One: Create the secret - -A secret contains a set of named byte arrays. - -Use the [`secret.yaml`](/docs/user-guide/secrets/secret.yaml) file to create a secret: - -```shell -$ kubectl create -f docs/user-guide/secrets/secret.yaml -``` - -You can use `kubectl` to see information about the secret: - -```shell -$ kubectl get secrets -NAME TYPE DATA -test-secret Opaque 2 - -$ kubectl describe secret test-secret -Name: test-secret -Labels: -Annotations: - -Type: Opaque - -Data -==== -data-1: 9 bytes -data-2: 11 bytes -``` - -## Step Two: Create a pod that consumes a secret - -Pods consume secrets in volumes. Now that you have created a secret, you can create a pod that -consumes it. - -Use the [`secret-pod.yaml`](/docs/user-guide/secrets/secret-pod.yaml) file to create a Pod that consumes the secret. - -```shell -$ kubectl create -f docs/user-guide/secrets/secret-pod.yaml -``` - -This pod runs a binary that displays the content of one of the pieces of secret data in the secret -volume: - -```shell -$ kubectl logs secret-test-pod -2015-04-29T21:17:24.712206409Z content of file "/etc/secret-volume/data-1": value-1 -``` \ No newline at end of file +[Distributing Credentials Securely](docs/tasks/configure-pod-container/distribute-credentials-secure/) From ada6673c7de0239500688e1bf106b1b504c8cb39 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 8 Feb 2017 14:20:30 -0800 Subject: [PATCH 033/407] Update walkthrough.md --- docs/user-guide/secrets/walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/secrets/walkthrough.md b/docs/user-guide/secrets/walkthrough.md index bb069c675e..a58364859f 100644 --- a/docs/user-guide/secrets/walkthrough.md +++ b/docs/user-guide/secrets/walkthrough.md @@ -4,4 +4,4 @@ title: Secrets Walkthrough {% include user-guide-content-moved.md %} -[Distributing Credentials Securely](docs/tasks/configure-pod-container/distribute-credentials-secure/) +[Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) From 2d98e5c1a584afacdb9291d061f97add23452953 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 16:58:11 -0800 Subject: [PATCH 034/407] Remove Guide topic: kubectl to Manage Resources. --- docs/user-guide/working-with-resources.md | 52 +---------------------- 1 file changed, 2 insertions(+), 50 deletions(-) diff --git a/docs/user-guide/working-with-resources.md b/docs/user-guide/working-with-resources.md index 07a9bd55b7..7b55db4e25 100644 --- a/docs/user-guide/working-with-resources.md +++ b/docs/user-guide/working-with-resources.md @@ -5,54 +5,6 @@ assignees: title: Using kubectl to Manage Resources --- -*This document is aimed at users who have worked through some of the examples, -and who want to learn more about using kubectl to manage resources such -as pods and services. Users who want to access the REST API directly, -and developers who want to extend the Kubernetes API should -refer to the [api conventions](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md) and -the [api document](/docs/api/).* +{% include user-guide-content-moved.md %} -## Resources are Automatically Modified - -When you create a resource such as pod, and then retrieve the created -resource, a number of the fields of the resource are added. -You can see this at work in the following example: - -```shell -$ cat > /tmp/original.yaml < /tmp/current.yaml -pods/original -$ wc -l /tmp/original.yaml /tmp/current.yaml - 51 /tmp/current.yaml - 9 /tmp/original.yaml - 60 total -``` - -The resource we posted had only 9 lines, but the one we got back had 51 lines. -If you `diff -u /tmp/original.yaml /tmp/current.yaml`, you can see the fields added to the pod. -The system adds fields in several ways: - - - Some fields are added synchronously with creation of the resource and some are set asynchronously. - - For example: `metadata.uid` is set synchronously. (Read more about [metadata](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#metadata)). - - For example, `status.hostIP` is set only after the pod has been scheduled. This often happens fast, but you may notice pods which do not have this set yet. This is called Late Initialization. (Read more about [status](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status) and [late initialization](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#late-initialization)). - - Some fields are set to default values. Some defaults vary by cluster and some are fixed for the API at a certain version. (Read more about [defaulting](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#defaulting)). - - For example, `spec.containers[0].imagePullPolicy` always defaults to `IfNotPresent` in api v1. - - For example, `spec.containers[0].resources.limits.cpu` may be defaulted to `100m` on some clusters, to some other value on others, and not defaulted at all on others. - -The API will generally not modify fields that you have set; it just sets ones which were unspecified. - -## Finding Documentation on Resource Fields - -You can browse auto-generated API documentation [here](/docs/api/). +[Kubernetes Object Management](/docs/concepts/tools/kubectl/object-management-overview/) From b7e410bb574aa58a0675973ee808e0da9aa2fa9e Mon Sep 17 00:00:00 2001 From: sallydeng Date: Thu, 9 Feb 2017 15:49:59 +0800 Subject: [PATCH 035/407] Update ui.md I reword it base on comment. --- docs/user-guide/ui.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index 3456a7e98a..cb330f1f9b 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -98,7 +98,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. - When the namespace name only contains number, such as "10", the pod can't be created in this kind of namespace, the pod will be put in default namespace. + Namespace names should not consist of only numbers. If the name is set as a number, such as 10, the pod will be put in the default namespace. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. From 446606a43a18e963374d5309c9c4b23f7de3fe2c Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Thu, 9 Feb 2017 10:18:59 -0800 Subject: [PATCH 036/407] Migrate Init Containers to Concepts (#2438) * Initial commit for Init Containers migration to Concepts section * Add 1.5 beta include file * Change include to renamed user-guide-content-moved.md file * Fix Concepts/KO/Pods ToC * reformat examples to bullet points * fix formatting * Add back Detailed Behavior, Support and compatibility * Adjust formatting issues * revised based on feedback from Steve * complete sentence rewrite --- _data/concepts.yml | 8 +- _includes/1-5-beta.md | 1 + docs/concepts/abstractions/init-containers.md | 184 ++++++++++++++++++ docs/user-guide/pods/init-container.md | 165 +--------------- 4 files changed, 192 insertions(+), 166 deletions(-) create mode 100644 _includes/1-5-beta.md create mode 100644 docs/concepts/abstractions/init-containers.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 4f41c2c818..4227231cc1 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -9,12 +9,14 @@ toc: - docs/concepts/tools/kubectl/object-management-using-imperative-commands.md - docs/concepts/tools/kubectl/object-management-using-imperative-config.md - docs/concepts/tools/kubectl/object-management-using-declarative-config.md - + - title: Kubernetes Objects section: - docs/concepts/abstractions/overview.md - - docs/concepts/abstractions/pod.md - + - title: Pods + section: + - docs/concepts/abstractions/pod.md + - docs/concepts/abstractions/init-containers.md - title: Controllers section: - docs/concepts/abstractions/controllers/statefulsets.md diff --git a/_includes/1-5-beta.md b/_includes/1-5-beta.md new file mode 100644 index 0000000000..abccaa303c --- /dev/null +++ b/_includes/1-5-beta.md @@ -0,0 +1 @@ +***NOTE: This feature is beta in Kubernetes 1.5.*** \ No newline at end of file diff --git a/docs/concepts/abstractions/init-containers.md b/docs/concepts/abstractions/init-containers.md new file mode 100644 index 0000000000..1bea00b3a2 --- /dev/null +++ b/docs/concepts/abstractions/init-containers.md @@ -0,0 +1,184 @@ +--- +assignees: +- erictune +title: Init Containers +--- + +{% capture overview %} +This page provides an overview of Init Containers, which are specialized +Containers that run before app Containers and can contain utilities or setup +scripts not present in an app image. +{% endcapture %} + +{:toc} + +{% include 1-5-beta.md %} + +**Once the feature exits beta, Init Containers will be specified in the PodSpec +alongside the app `containers` array.** + +{% capture body %} +## Understanding Init Containers + +A [Pod](/docs/concepts/abstractions/pod/) can have multiple Containers running +apps within it, but it can also have one or more Init Containers, which are run +before the app Containers are started. + +Init Containers are exactly like regular Containers, except: + +* They always run to completion. +* Each one must complete successfully before the next one is started. + +If an Init Container fails for a Pod, Kubernetes restarts the Pod repeatedly until the Init +Container succeeds. However, if the Pod has a `restartPolicy` of Never, it is not restarted. + +To specify a Container as an Init Container, add the `annotations` key +`pod.beta.kubernetes.io/init-containers`. Its value should be a +JSON array of objects of type +[Container](http://kubernetes.io/docs/api-reference/v1/definitions/#_v1_container). + +The status of an Init Container is returned as another annotation, +`pod.beta.kubernetes.io/init-container-statuses`, which is an array of +container statuses similar to the `status.containerStatuses` field. + +### Differences from regular Containers + +Init Containers support all the fields and features of app Containers, +including resource limits, volumes, and security settings. However, the +resource requests and limits for an Init Container are handled slightly +differently, which are documented in [Resources](#resources) below. Also, Init Containers do not +support readiness probes because they must run to completion before the Pod can +be ready. + +If multiple Init Containers are specified for a Pod, those Containers are run +one at a time in sequential order. Each must succeed before the next can run. +When all of the Init Containers have run to completion, Kubernetes initializes +the Pod and runs the application Containers as usual. + +## What can Init Containers be used for? + +Because Init Containers have separate images from app Containers, they +have some advantages for start-up related code: + +* They can contain and run utilities that are not desirable to include in the + app Container image for security reasons. +* They can contain utilities or custom code for setup that is not present in an app + image. For example, there is no need to make an image `FROM` another image just to use a tool like + `sed`, `awk`, `python`, or `dig` during setup. +* The application image builder and deployer roles can work independently without + the need to jointly build a single app image. +* They use Linux namespaces so they have a different filesystem view from app Containers. + Consequently, they can be given access to Secrets that app Containers are not able to + access. +* They run to completion before any app Containers start, whereas app + Containers run in parallel, so Init Containers provide an easy way to block or + delay the startup of app Containers until some set of preconditions are met. + +### Examples +Here are some ideas for how to use Init Containers: + +* Wait for a service to be created with a shell command like: + + for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; exit 1 + +* Register this Pod with a remote server from the downward API with a command like: + + curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$()&ip=$()' + +* Wait for some time before starting the app Container with a command like `sleep 60`. +* Clone a git repository into a volume. +* Place values into a configuration file and run a template tool to dynamically + generate a configuration file for the the main app Container. For example, + place the POD_IP value in a configuration and generate the main app + configuration file using Jinja. + +More detailed usage examples can be found in the [StatefulSets documentation](/docs/concepts/abstractions/controllers/statefulsets/) +and the [Production Pods guide](/docs/user-guide/production-pods.md#handling-initialization). + +## Detailed behavior + +During the startup of a Pod, the Init Containers are started in order, after the +network and volumes are initialized. Each Container must exit successfully before +the next is started. If a Container fails to start due to the runtime or +exits with failure, it is retried according to the Pod `restartPolicy`. However, +if the Pod `restartPolicy` is set to Always, the Init Containers use +`RestartPolicy` OnFailure. + +A Pod cannot be `Ready` until all Init Containers have succeeded. The ports on an +Init Container are not aggregated under a service. A Pod that is initializing +is in the `Pending` state but should have a condition `Initializing` set to true. + +If the Pod is [restarted](#pod-restart-reasons), all Init Containers must +execute again. + +Changes to the Init Container spec are limited to the container image field. +Altering an Init Container image field is equivalent to restarting the Pod. + +Because Init Containers can be restarted, retried, or re-executed, Init Container +code should be idempotent. In particular, code that writes to files on `EmptyDirs` +should be prepared for the possibility that an output file already exists. + +Init Containers have all of the fields of an app Container. However, Kubernetes +prohibits `readinessProbe` from being used because Init Containers cannot +define readiness distinct from completion. This is enforced during validation. + +Use `activeDeadlineSeconds` on the Pod and `livenessProbe` on the Container to +prevent Init Containers from failing forever. The active deadline includes Init +Containers. + +The name of each app and Init Container in a Pod must be unique; a +validation error is thrown for any Container sharing a name with another. + +### Resources + +Given the ordering and execution for Init Containers, the following rules +for resource usage apply: + +* The highest of any particular resource request or limit defined on all Init + Containers is the *effective init request/limit* +* The Pod's *effective request/limit* for a resource is the higher of: + * the sum of all app Containers request/limit for a resource + * the effective init request/limit for a resource +* Scheduling is done based on effective requests/limits, which means + Init Containers can reserve resources for initialization that are not used + during the life of the Pod. +* QoS tier of the Pod's *effective QoS tier* is the QoS tier for Init Containers + and app containers alike. + +Quota and limits are applied based on the effective Pod request and +limit. + +Pod level cgroups are based on the effective Pod request and limit, the +same as the scheduler. + + +### Pod restart reasons + +A Pod can restart, causing re-execution of Init Containers, for the following +reasons: + +* A user updates the PodSpec causing the Init Container image to change. + App Container image changes only restart the app Container. +* The Pod infrastructure container is restarted. This is uncommon and would + have to be done by someone with root access to nodes. +* All containers in a Pod are terminated while `restartPolicy` is set to Always, + forcing a restart, and the Init Container completion record has been lost due + to garbage collection. + +## Support and compatibility + +A cluster with Kubelet and Apiserver version 1.4.0 or greater supports Init +Containers with the beta annotations. Support varies for other combinations of +Kubelet and Apiserver versions; see the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md) for details. + +{% endcapture %} + + +{% capture whatsnext %} + +* [Creating a Pod that has an Init Container](/docs/tasks/configure-pod-container/configure-pod-initialization/#creating-a-pod-that-has-an-init-container) + +{% endcapture %} + + +{% include templates/concept.md %} \ No newline at end of file diff --git a/docs/user-guide/pods/init-container.md b/docs/user-guide/pods/init-container.md index ce7679d40e..90a862ea02 100644 --- a/docs/user-guide/pods/init-container.md +++ b/docs/user-guide/pods/init-container.md @@ -1,168 +1,7 @@ --- -assignees: -- erictune title: Init Containers --- -* TOC -{:toc} - -In addition to having one or more main containers (or **app containers**), a -pod can also have one or more **init containers** which run before the app -containers. Init containers allow you to reduce and reorganize setup scripts -and "glue code". - -## Overview - -An init container is exactly like a regular container, except that it always -runs to completion and each init container must complete successfully before -the next one is started. If the init container fails, Kubernetes will restart -the pod until the init container succeeds. If a pod is marked as `RestartNever`, -the pod will fail if the init container fails. - -You specify a container as an init container by adding an annotation. -The annotation key is `pod.beta.kubernetes.io/init-containers`. The annotation -value is a JSON array of [objects of type `v1.Container` -](http://kubernetes.io/docs/api-reference/v1/definitions/#_v1_container) - -Once the feature exits beta, the init containers will be specified on the Pod -Spec alongside the app `containers` array. -The status of the init containers is returned as another annotation - -`pod.beta.kubernetes.io/init-container-statuses` -- as an array of the -container statuses (similar to the `status.containerStatuses` field). - -Init containers support all of the same features as normal containers, -including resource limits, volumes, and security settings. The resource -requests and limits for an init container are [handled slightly differently]( -#resources). Init containers do not support readiness probes since they will -run to completion before the pod can be ready. -An init container has all of the fields of an app container. - -If you specify multiple init containers for a pod, those containers run one at -a time in sequential order. Each must succeed before the next can run. Once all -init containers have run to completion, Kubernetes initializes the pod and runs -the application containers as usual. - -## What are Init Containers Good For? - -Because init containers have separate images from application containers, they -have some advantages for start-up related code. These include: - -* they can contain utilities that are not desirable to include in the app container - image for security reasons, -* they can contain utilities or custom code for setup that is not present in an app - image. (No need to make an image `FROM` another image just to use a tool like - `sed`, `awk`, `python`, `dig`, etc during setup). -* the application image builder and the deployer roles can work independently without - the need to jointly build a single app image. - -Because init containers have different filesystem view (Linux namespaces) from -app containers, they can be given access to Secrets that the app containers are -not able to access. - -Since init containers run to completion before any app containers start, and -since app containers run in parallel, they provide an easier way to block or -delay the startup of application containers until some precondition is met. - -Because init containers run in sequence and there can be multiple init containers, -they can be composed easily. - -Here are some ideas for how to use init containers: -- Wait for a service to be created with a shell command like: - `for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; exit 1` -- Register this pod with a remote server with a command like: - `curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$(POD_NAME)&ip=$(POD_IP)'` - using `POD_NAME` and `POD_IP` from the downward API. -- Wait for some time before starting the app container with a command like `sleep 60`. -- Clone a git repository into a volume -- Place values like a POD_IP into a configuration file, and run a template tool (e.g. jinja) - to generate a configuration file to be consumed by the main app contianer. - -Complete usage examples can be found in the [StatefulSets -documentation](/docs/concepts/abstractions/controllers/statefulsets/) and the [Production Pods -guide](/docs/user-guide/production-pods.md#handling-initialization). - - -## Detailed Behavior - -Each pod may have 0..N init containers defined along with the existing -1..M app containers. - -On startup of the pod, after the network and volumes are initialized, the init -containers are started in order. Each container must exit successfully before -the next is invoked. If a container fails to start (due to the runtime) or -exits with failure, it is retried according to the pod RestartPolicy, except -when the pod restart policy is RestartPolicyAlways, in which case just the init -containers use RestartPolicyOnFailure. - -A pod cannot be ready until all init containers have succeeded. The ports on an -init container are not aggregated under a service. A pod that is being -initialized is in the `Pending` phase but should has a condition `Initializing` -set to `true`. - -If the pod is [restarted](#pod-restart-reasons) all init containers must -execute again. - -Changes to the init container spec are limited to the container image field. -Altering an init container image field is equivalent to restarting the pod. - -Because init containers can be restarted, retried, or reexecuted, init container -code should be idempotent. In particular, code that writes to files on EmptyDirs -should be prepared for the possibility that an output file already exists. - -An init container has all of the fields of an app container. The following -fields are prohibited from being used on init containers by validation: - -* `readinessProbe` - init containers must exit for pod startup to continue, - are not included in rotation, and so cannot define readiness distinct from - completion. - -Init container authors may use `activeDeadlineSeconds` on the pod and -`livenessProbe` on the container to prevent init containers from failing -forever. The active deadline includes init containers. - -The name of each app and init container in a pod must be unique - it is a -validation error for any container to share a name. - -### Resources - -Given the ordering and execution for init containers, the following rules -for resource usage apply: - -* The highest of any particular resource request or limit defined on all init - containers is the **effective init request/limit** -* The pod's **effective request/limit** for a resource is the higher of: - * sum of all app containers request/limit for a resource - * effective init request/limit for a resource -* Scheduling is done based on effective requests/limits, which means - init containers can reserve resources for initialization that are not used - during the life of the pod. -* QoS tier of the pod's **effective QoS tier** is the QoS tier for init containers - and app containers alike. - -Quota and limits are applied based on the effective pod request and -limit. - -Pod level cGroups are based on the effective pod request and limit, the -same as the scheduler. - - -## Pod Restart Reasons - -A Pod may "restart", causing reexecution of init containers, for the following -reasons: - -* An init container image is changed by a user updating the Pod Spec. - * App container image changes only restart the app container. -* The pod infrastructure container is restarted. - * This is uncommon and would have to be done by someone with root access to nodes. -* All containers in a pod are terminated, requiring a restart (RestartPolicyAlways) AND the record of init container completion has been lost due to garbage collection. - -## Support and compatibility - -A cluster with Kubelet and Apiserver version 1.4.0 or greater supports init -containers with the beta annotations. Support varies for other combinations of -Kubelet and Apiserver version; see the [release notes -](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md) for details. - +{% include user-guide-content-moved.md %} +* [Init Containers](/docs/concepts/abstractions/init-containers/) \ No newline at end of file From 51662f0c848cd668e952342287157bc2eb5f6500 Mon Sep 17 00:00:00 2001 From: scjane Date: Thu, 9 Feb 2017 16:18:48 +0800 Subject: [PATCH 037/407] Update accessing-the-cluster.md --- docs/user-guide/accessing-the-cluster.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/user-guide/accessing-the-cluster.md b/docs/user-guide/accessing-the-cluster.md index 8fb9e6b0b4..1f5a65aa61 100644 --- a/docs/user-guide/accessing-the-cluster.md +++ b/docs/user-guide/accessing-the-cluster.md @@ -1,10 +1,10 @@ ---- -assignees: -- lavalamp -- mikedanese -title: Accessing Clusters ---- - +--- +assignees: +- lavalamp +- mikedanese +title: Accessing Clusters +--- + * TOC {:toc} @@ -125,6 +125,7 @@ with future high-availability support. The Kubernetes project-supported Go client library is at [https://github.com/kubernetes/client-go](https://github.com/kubernetes/client-go). To use it, + * To get the library, run the following command: `go get k8s.io/client-go//kubernetes` See [https://github.com/kubernetes/client-go](https://github.com/kubernetes/client-go) to see which versions are supported. * Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g., `import "k8s.io/client-go/1.4/pkg/api/v1"` is correct. From 6620b309e597b2d1cc0292c382b4a355ac3f89ee Mon Sep 17 00:00:00 2001 From: Himanshu Raj Date: Thu, 9 Feb 2017 00:09:44 -0800 Subject: [PATCH 038/407] Add Contiv Information to addons.md --- docs/admin/addons.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/admin/addons.md b/docs/admin/addons.md index aeee68cc30..fc1c4ec83c 100644 --- a/docs/admin/addons.md +++ b/docs/admin/addons.md @@ -14,6 +14,7 @@ Add-ons in each section are sorted alphabetically - the ordering does not imply * [Calico](http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/) is a secure L3 networking and network policy provider. * [Canal](https://github.com/tigera/canal/tree/master/k8s-install/kubeadm) unites Flannel and Calico, providing networking and network policy. +* [Contiv](http://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](http://github.com/contiv). The [installer](http://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options. * [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml) is an overlay network provider that can be used with Kubernetes. * [Romana](http://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/user-guide/networkpolicies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize). * [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database. From fdd72405947a85f428f9eeb5700b10fe0e608bee Mon Sep 17 00:00:00 2001 From: xilabao Date: Wed, 8 Feb 2017 20:43:18 -0600 Subject: [PATCH 039/407] add authorizationMode to kubeadm config ref to https://github.com/kubernetes/kubernetes/pull/39846 --- docs/admin/kubeadm.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 1014ee1ab6..edf9c87a7d 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -191,6 +191,7 @@ available as configuration file options. externalDNSNames: - - + authorizationMode: cloudProvider: discovery: bindPort: From 123b54b101181be2e730efadc3c156ae7f10ed01 Mon Sep 17 00:00:00 2001 From: Klaus Ma Date: Sat, 28 Jan 2017 21:59:13 +0800 Subject: [PATCH 040/407] Updated multiple scheduler beta docs. --- docs/admin/multiple-schedulers.md | 22 ++++++++++------------ docs/admin/multiple-schedulers/pod2.yaml | 3 +-- docs/admin/multiple-schedulers/pod3.yaml | 3 +-- 3 files changed, 12 insertions(+), 16 deletions(-) diff --git a/docs/admin/multiple-schedulers.md b/docs/admin/multiple-schedulers.md index bf7386e492..3e3b4d270b 100644 --- a/docs/admin/multiple-schedulers.md +++ b/docs/admin/multiple-schedulers.md @@ -61,7 +61,7 @@ config. Save it as `my-scheduler.yaml`: {% include code.html language="yaml" file="multiple-schedulers/my-scheduler.yaml" ghlink="/docs/admin/multiple-schedulers/my-scheduler.yaml" %} An important thing to note here is that the name of the scheduler specified as an -argument to the scheduler command in the container spec should be unique. This is the name that is matched against the value of the optional `scheduler.alpha.kubernetes.io/name` annotation on pods, to determine whether this scheduler is responsible for scheduling a particular pod. +argument to the scheduler command in the container spec should be unique. This is the name that is matched against the value of the optional `spec.schedulername` on pods, to determine whether this scheduler is responsible for scheduling a particular pod. Please see the [kube-scheduler documentation](/docs/admin/kube-scheduler/) for @@ -92,14 +92,14 @@ pod in this list. ### 4. Specify schedulers for pods Now that our second scheduler is running, let's create some pods, and direct them to be scheduled by either the default scheduler or the one we just deployed. In order to schedule a given pod using a specific scheduler, we specify the name of the -scheduler as an annotation in that pod spec. Let's look at three examples. +scheduler in that pod spec. Let's look at three examples. -1. Pod spec without any scheduler annotation +1. Pod spec without any scheduler name {% include code.html language="yaml" file="multiple-schedulers/pod1.yaml" ghlink="/docs/admin/multiple-schedulers/pod1.yaml" %} - When no scheduler annotation is supplied, the pod is automatically scheduled using the + When no scheduler name is supplied, the pod is automatically scheduled using the default-scheduler. Save this file as `pod1.yaml` and submit it to the Kubernetes cluster. @@ -108,12 +108,11 @@ scheduler as an annotation in that pod spec. Let's look at three examples. kubectl create -f pod1.yaml ``` -2. Pod spec with `default-scheduler` annotation +2. Pod spec with `default-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod2.yaml" ghlink="/docs/admin/multiple-schedulers/pod2.yaml" %} - A scheduler is specified by supplying the scheduler name as a value to the annotation - with key `scheduler.alpha.kubernetes.io/name`. In this case, we supply the name of the + A scheduler is specified by supplying the scheduler name as a value to `spec.schedulername`. In this case, we supply the name of the default scheduler which is `default-scheduler`. Save this file as `pod2.yaml` and submit it to the Kubernetes cluster. @@ -122,13 +121,12 @@ scheduler as an annotation in that pod spec. Let's look at three examples. kubectl create -f pod2.yaml ``` -3. Pod spec with `my-scheduler` annotation +3. Pod spec with `my-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod3.yaml" ghlink="/docs/admin/multiple-schedulers/pod3.yaml" %} In this case, we specify that this pod should be scheduled using the scheduler that we - deployed - `my-scheduler`. Note that the value of the annotation with key - `scheduler.alpha.kubernetes.io/name` should match the name supplied to the scheduler + deployed - `my-scheduler`. Note that the value of `spec.schedulername` should match the name supplied to the scheduler command as an argument in the deployment config for the scheduler. Save this file as `pod3.yaml` and submit it to the Kubernetes cluster. @@ -149,9 +147,9 @@ In order to make it easier to work through these examples, we did not verify tha pods were actually scheduled using the desired schedulers. We can verify that by changing the order of pod and deployment config submissions above. If we submit all the pod configs to a Kubernetes cluster before submitting the scheduler deployment config, -we see that the pod `annotation-second-scheduler` remains in "Pending" state forever +we see that the pod `second-scheduler` remains in "Pending" state forever while the other two pods get scheduled. Once we submit the scheduler deployment config -and our new scheduler starts running, the `annotation-second-scheduler` pod gets +and our new scheduler starts running, the `second-scheduler` pod gets scheduled as well. Alternatively, one could just look at the "Scheduled" entries in the event logs to diff --git a/docs/admin/multiple-schedulers/pod2.yaml b/docs/admin/multiple-schedulers/pod2.yaml index 2f9fdf9875..26b0376967 100644 --- a/docs/admin/multiple-schedulers/pod2.yaml +++ b/docs/admin/multiple-schedulers/pod2.yaml @@ -2,11 +2,10 @@ apiVersion: v1 kind: Pod metadata: name: annotation-default-scheduler - annotations: - scheduler.alpha.kubernetes.io/name: default-scheduler labels: name: multischeduler-example spec: + schedulername: default-scheduler containers: - name: pod-with-default-annotation-container image: gcr.io/google_containers/pause:2.0 \ No newline at end of file diff --git a/docs/admin/multiple-schedulers/pod3.yaml b/docs/admin/multiple-schedulers/pod3.yaml index 52276df5f6..e64070abfa 100644 --- a/docs/admin/multiple-schedulers/pod3.yaml +++ b/docs/admin/multiple-schedulers/pod3.yaml @@ -2,11 +2,10 @@ apiVersion: v1 kind: Pod metadata: name: annotation-second-scheduler - annotations: - scheduler.alpha.kubernetes.io/name: my-scheduler labels: name: multischeduler-example spec: + schedulername: my-scheduler containers: - name: pod-with-second-annotation-container image: gcr.io/google_containers/pause:2.0 \ No newline at end of file From 2f1ae18bfc5acb0e2747c3107c315ade6dfc289a Mon Sep 17 00:00:00 2001 From: Robert Roland Date: Fri, 3 Feb 2017 11:23:45 -0800 Subject: [PATCH 041/407] Adds documentation for new vSphere vm-uuid parameter. Related to kubernetes/kubernetes#40892 --- docs/getting-started-guides/vsphere.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/getting-started-guides/vsphere.md b/docs/getting-started-guides/vsphere.md index ba07d14890..61ea733bf3 100644 --- a/docs/getting-started-guides/vsphere.md +++ b/docs/getting-started-guides/vsphere.md @@ -69,6 +69,7 @@ Sample Config: datacenter = datastore = working-dir = + vm-uuid = [Disk] scsicontrollertype = pvscsi ``` From 4499f488565c23c217610397d512572c550cc9fc Mon Sep 17 00:00:00 2001 From: cwitte4191 Date: Fri, 3 Feb 2017 13:52:34 -0600 Subject: [PATCH 042/407] Update service-accounts.md Added an automated code snippet to populate the registrykey on a serviceaccount using jq. --- docs/user-guide/service-accounts.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/service-accounts.md b/docs/user-guide/service-accounts.md index d1129a1983..ad1c0a68bf 100644 --- a/docs/user-guide/service-accounts.md +++ b/docs/user-guide/service-accounts.md @@ -147,8 +147,18 @@ NAME TYPE DATA myregistrykey kubernetes.io/.dockerconfigjson 1 ``` -Next, read/modify/write the service account for the namespace to use this secret as an imagePullSecret +Next, read/modify/write the service account for the namespace to use this secret as an imagePullSecret. +Automated version using json and the jq utility: +```shell +kubectl get serviceaccounts default -o json | + jq 'del(.metadata.resourceVersion)'| + jq 'setpath(["imagePullSecrets"];[{"name":"myregistrykey"}])' | + kubectl replace serviceaccount default -f - + +``` + +Interactive version requiring manual edit: ```shell $ kubectl get serviceaccounts default -o yaml > ./sa.yaml $ cat sa.yaml From 13c366c09e7b52ae397e0dae061b566699886ba7 Mon Sep 17 00:00:00 2001 From: Phillip Wittrock Date: Wed, 8 Feb 2017 17:03:17 -0800 Subject: [PATCH 043/407] Kubectl task document for printing out containers in a cluster. --- _data/tasks.yml | 4 + .../list-all-running-container-images.md | 122 ++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 docs/tasks/kubectl/list-all-running-container-images.md diff --git a/_data/tasks.yml b/_data/tasks.yml index de066c9ed2..f301de1edc 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -3,6 +3,10 @@ abstract: "Step-by-step instructions for performing operations with Kubernetes." toc: - docs/tasks/index.md +- title: Using the Kubectl Command-Line + section: + - docs/tasks/kubectl/list-all-running-container-images.md + - title: Configuring Pods and Containers section: - docs/tasks/configure-pod-container/define-environment-variable-container.md diff --git a/docs/tasks/kubectl/list-all-running-container-images.md b/docs/tasks/kubectl/list-all-running-container-images.md new file mode 100644 index 0000000000..4fb64ee442 --- /dev/null +++ b/docs/tasks/kubectl/list-all-running-container-images.md @@ -0,0 +1,122 @@ +--- +title: Listing all Container images running in the cluster +--- + +{% capture overview %} + +This page shows how to use kubectl to list all of the Container images +for Pods running in a cluster. + +{% endcapture %} + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + +{% capture steps %} + +In this exercise you will use kubectl to fetch all of the Pods +running in a cluster, and format the output to pull out the list +of Containers for each. + +## List all Containers in all namespaces + +- Fetch all Pods in all namespaces using `kubectl get pods --all-namespaces` +- Format the output to include only the list of Container image names + using `-o jsonpath={..image}`. This will recursively parse out the + `image` field from the returned json. + - See the [jsonpath reference](/docs/user-guide/jsonpath/) + for further information on how to use jsonpath. +- Format the output using standard tools: `tr`, `sort`, `uniq` + - Use `tr` to replace spaces with newlines + - Use `sort` to sort the results + - Use `uniq` to aggregate image counts + +```sh +kubectl get pods --all-namespaces -o jsonpath="{..image}" |\ +tr -s '[[:space:]]' '\n' |\ +sort |\ +uniq -c +``` + +The above command will recursively return all fields named `image` +for all items returned. + +As an alternative, it is possible to use the absolute path to the image +field within the Pod. This ensures the correct field is retrieved +in the even the field name is repeated, +e.g. many fields are called `name` within a given item: + +```sh +kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" +``` + +The jsonpath is interpreted as follows: + +- `.items[*]`: for each returned value +- `.spec`: get the spec +- `.containers[*]`: for each container +- `.image`: get the image + +**Note:** When fetching a single Pod by name, e.g. `kubect get pod nginx`, +the `.items[*]` portion of the path should be omitted because a single +Pod is returned instead of a list of items. + +## List Containers by Pod + +The formatting can be controlled further by using the `range` operation to +iterate over elements individually. + +```sh +kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |\ +sort +``` + +## List Containers filtering by Pod label + +To target only Pods matching a specific label, use the -l flag. The +following matches only Pods with labels matching `app=nginx`. + +```sh +kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx +``` + +## List Containers filtering by Pod namespace + +To target only pods in a specific namespace, use the namespace flag. The +following matches only Pods in the `kube-system` namespace. + +```sh +kubectl get pods --namespace kube-system -o jsonpath="{..image}" +``` + +## List Containers using a go-template instead of jsonpath + +As an alternative to jsonpath, Kubectl supports using [go-templates](https://golang.org/pkg/text/template/) +for formatting the output: + +{% raw %} +```sh +kubectl get pods --all-namespaces -o go-template --template="{{range .items}}{{range .spec.containers}}{{.image}} {{end}}{{end}}" +``` +{% endraw %} + + +{% endcapture %} + +{% capture discussion %} + +{% endcapture %} + +{% capture whatsnext %} + +### Reference + +* [Jsonpath](/docs/user-guide/jsonpath/) reference guide +* [Go template](https://golang.org/pkg/text/template/) reference guide + +{% endcapture %} + +{% include templates/task.md %} From a65374ff0c18f19cdfc5f30b2fdb411d666beea2 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Thu, 9 Feb 2017 14:08:54 -0800 Subject: [PATCH 044/407] New Task: Exposing Pod Info ... Env Vars --- _data/tasks.yml | 1 + .../dapi-envars-container.yaml | 45 +++++ .../dapi-envars-pod.yaml | 38 ++++ ...ronment-variable-expose-pod-information.md | 163 ++++++++++++++++++ 4 files changed, 247 insertions(+) create mode 100644 docs/tasks/configure-pod-container/dapi-envars-container.yaml create mode 100644 docs/tasks/configure-pod-container/dapi-envars-pod.yaml create mode 100644 docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md diff --git a/_data/tasks.yml b/_data/tasks.yml index f301de1edc..cb61b1a05a 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -14,6 +14,7 @@ toc: - docs/tasks/configure-pod-container/assign-cpu-ram-container.md - docs/tasks/configure-pod-container/configure-volume-storage.md - docs/tasks/configure-pod-container/configure-persistent-volume-storage.md + - docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md - docs/tasks/configure-pod-container/distribute-credentials-secure.md - docs/tasks/configure-pod-container/pull-image-private-registry.md - docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md diff --git a/docs/tasks/configure-pod-container/dapi-envars-container.yaml b/docs/tasks/configure-pod-container/dapi-envars-container.yaml new file mode 100644 index 0000000000..8b3b3a39d3 --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-envars-container.yaml @@ -0,0 +1,45 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-resourcefieldref +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox:1.24 + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_CPU_REQUEST MY_CPU_LIMIT; + printenv MY_MEM_REQUEST MY_MEM_LIMIT; + sleep 10; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + env: + - name: MY_CPU_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.cpu + - name: MY_CPU_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.cpu + - name: MY_MEM_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.memory + - name: MY_MEM_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.memory + restartPolicy: Never diff --git a/docs/tasks/configure-pod-container/dapi-envars-pod.yaml b/docs/tasks/configure-pod-container/dapi-envars-pod.yaml new file mode 100644 index 0000000000..00762373b3 --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-envars-pod.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-fieldref +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_NODE_NAME MY_POD_NAME MY_POD_NAMESPACE; + printenv MY_POD_IP MY_POD_SERVICE_ACCOUNT; + sleep 10; + done; + env: + - name: MY_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: MY_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: MY_POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: MY_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: MY_POD_SERVICE_ACCOUNT + valueFrom: + fieldRef: + fieldPath: spec.serviceAccountName + restartPolicy: Never diff --git a/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md new file mode 100644 index 0000000000..1390727000 --- /dev/null +++ b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md @@ -0,0 +1,163 @@ +--- +title: Exposing Pod Information to Containers Through Environment Variables +--- + +{% capture overview %} + +This page shows how a Pod can use environment variables to expose information +about itself to Containers running in the Pod. Environment variables can expose +Pod fields and Container fields. + +There are two ways to expose Pod and Container fields to a running Container: +environment variables and +[DownwardAPIVolumeFiles](/docs/resources-reference/v1.5/#downwardapivolumefile-v1). +Together, these two ways of exposing Pod and Container fields are called the +*Downward API*. + +{% endcapture %} + + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + + +{% capture steps %} + +## Using Pod fields as values for environment variables + +In this exercise, you create a Pod that has one Container. Here is the +configuration file for the Pod: + +{% include code.html language="yaml" file="dapi-envars-pod.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-envars-pod.yaml" %} + +In the configuration file, you can see five environment variables. The `env` +field is an array of +[EnvVars](/docs/resources-reference/v1.5/#envvar-v1). +The first element in the array specifies that the `MY_NODE_NAME` environment +variable gets its value from the Pod's `spec.nodeName` field. Similarly, the +other environment variables get their names from Pod fields. + +**Note**: The fields in this example are Pod fields. They are not fields of the +Container in the Pod. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-envars-pod.yaml +``` + +Verify that the Container in the Pod is running: + +``` +kubectl get pods +``` + +View the Container's logs: + +``` +kubectl logs dapi-envars-fieldref +``` + +The output shows the values of selected environment variables: + +``` +minikube +dapi-envars-fieldref +default +172.17.0.4 +default +``` + +To see why these values are in the log, look at the `command` and `args` fields +in the configuration file. When the Container starts, it writes the values of +five environment variables to stdout. It repeats this every ten seconds. + +Next, get a shell into the Container that is running in your Pod: + +``` +kubectl exec -it dapi-envars-fieldref -- sh +``` + +In your shell, view the environment variables: + +``` +/# printenv +``` + +The output shows that certain environment variables have been assigned the +values of Pod fields: + +``` +MY_POD_SERVICE_ACCOUNT=default +... +MY_POD_NAMESPACE=default +MY_POD_IP=172.17.0.4 +... +MY_NODE_NAME=minikube +... +MY_POD_NAME=dapi-envars-fieldref +``` + +## Using Container fields as values for environment variables + +In the preceding exercise, you used Pod fields as the values for environment +variables. In this next exercise, you use Container fields as the values for +environment variables. Here is the configuration file for a Pod that has one +container: + +{% include code.html language="yaml" file="dapi-envars-container.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-envars-container.yaml" %} + +In the configuration file, you can see four environment variables. The `env` +field is an array of +[EnvVars](/docs/resources-reference/v1.5/#envvar-v1). +The first element in the array specifies that the `MY_CPU_REQUEST` environment +variable gets its value from the `requests.cpu` field of a Container named +`test-container`. Similarly, the other environment variables get their values +from Container fields. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-envars-container.yaml +``` + +Verify that the Container in the Pod is running: + +``` +kubectl get pods +``` + +View the Container's logs: + +``` +kubectl logs dapi-envars-resourcefieldref +``` + +The output shows the values of selected environment variables: + +``` +1 +1 +33554432 +67108864 +``` + +{% endcapture %} + +{% capture whatsnext %} + +* [Defining Environment Variables for a Container](/docs/tasks/configure-pod-container/define-environment-variable-container/) +* [PodSpec](/docs/resources-reference/v1.5/#podspec-v1) +* [Container](/docs/resources-reference/v1.5/#container-v1) +* [EnvVar](/docs/resources-reference/v1.5/#envvar-v1) +* [EnvVarSource](/docs/resources-reference/v1.5/#envvarsource-v1) +* [ObjectFieldSelector](/docs/resources-reference/v1.5/#objectfieldselector-v1) +* [ResourceFieldSelector](/docs/resources-reference/v1.5/#resourcefieldselector-v1) + +{% endcapture %} + + +{% include templates/task.md %} From baf1b7ef0dfc6de4df68edf5fe8adfd826eda587 Mon Sep 17 00:00:00 2001 From: Jeff Mendoza Date: Fri, 10 Feb 2017 13:15:15 -0800 Subject: [PATCH 045/407] Fix Travis test builds' vendoring workaround. --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index e527e16dbc..09a28b78cb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,6 +13,7 @@ install: - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apiserver - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/client-go +- rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/sample-apiserver - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ From 926d049a8481c2b0be3e12f4f13ff480cd9a102c Mon Sep 17 00:00:00 2001 From: Jasmine Hegman Date: Wed, 8 Feb 2017 17:22:46 -0700 Subject: [PATCH 046/407] Fixed a tiny typo Kubernets -> Kubernetes --- docs/user-guide/logging/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/logging/overview.md b/docs/user-guide/logging/overview.md index a9f00471cc..42b9087172 100644 --- a/docs/user-guide/logging/overview.md +++ b/docs/user-guide/logging/overview.md @@ -68,7 +68,7 @@ When you run [`kubectl logs`](/docs/user-guide/kubectl/kubectl_logs), as in the There are two types of system components: those that run in a container and those that do not run in a container. For example: -* The Kubernets scheduler and kube-proxy run in a container. +* The Kubernetes scheduler and kube-proxy run in a container. * The kubelet and container runtime, for example Docker, do not run in containers. On machines with systemd, the kubelet and container runtime write to journald. If From 32e912fc87842e891c44fd33320d573ba9ac0505 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Wimsingues?= Date: Wed, 8 Feb 2017 11:28:26 +0100 Subject: [PATCH 047/407] redundnacy => redundancy --- docs/concepts/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 3f131c6358..93e4533caf 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -44,7 +44,7 @@ For example, when you use the Kubernetes API to create a Deployment object, you The Kubernetes master is responsible for maintaining the desired state for your cluster. When you interact with Kubernetes, such as by using the `kubectl` command-line interface, you're communicating with your cluster's Kubernetes master. -> The "master" refers to a collection of processes managing the cluster state. Typically these processes are all run on a single node in the cluster, and this node is also referred to as the master. The master can also be replicated for availability and redundnacy. +> The "master" refers to a collection of processes managing the cluster state. Typically these processes are all run on a single node in the cluster, and this node is also referred to as the master. The master can also be replicated for availability and redundancy. ### Kubernetes Nodes From 9f3ab47c8b4f647ba7b23b9e5ea61e1f730f6a83 Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 13 Feb 2017 11:08:33 +0800 Subject: [PATCH 048/407] Update cron-jobs.md --- docs/user-guide/cron-jobs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/cron-jobs.md b/docs/user-guide/cron-jobs.md index 55b85adf46..34402d5568 100644 --- a/docs/user-guide/cron-jobs.md +++ b/docs/user-guide/cron-jobs.md @@ -19,10 +19,10 @@ A _Cron Job_ manages time based [Jobs](/docs/user-guide/jobs/), namely: One CronJob object is like one line of a _crontab_ (cron table) file. It runs a job periodically on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) format. -**Note:**: The question mark (`?`) in the schedule has the same meaning as an asterisk `*`, +**Note:** The question mark (`?`) in the schedule has the same meaning as an asterisk `*`, that is, it stands for any of available value for a given field. -**Note:**: ScheduledJob resource was introduced in Kubernetes version 1.4, but starting +**Note:** ScheduledJob resource was introduced in Kubernetes version 1.4, but starting from version 1.5 its current name is CronJob. A typical use case is: From 87401ac54be948fe741ef20fddf6ff346ca0534b Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 13 Feb 2017 10:40:57 +0800 Subject: [PATCH 049/407] Update index.md --- docs/user-guide/jobs/expansions/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/expansions/index.md b/docs/user-guide/jobs/expansions/index.md index 767ac65215..f401574519 100644 --- a/docs/user-guide/jobs/expansions/index.md +++ b/docs/user-guide/jobs/expansions/index.md @@ -111,7 +111,7 @@ In the first example, each instance of the template had one parameter, and that used as a label. However label keys are limited in [what characters they can contain](/docs/user-guide/labels/#syntax-and-character-set). -This slightly more complex example uses a the jinja2 template language to generate our objects. +This slightly more complex example uses the jinja2 template language to generate our objects. We will use a one-line python script to convert the template to a file. First, copy and paste the following template of a Job object, into a file called `job.yaml.jinja2`: @@ -179,6 +179,7 @@ cat job.yaml.jinja2 | render_template | kubectl create -f - ## Alternatives If you have a large number of job objects, you may find that: + - even using labels, managing so many Job objects is cumbersome. - You exceed resource quota when creating all the Jobs at once, and do not want to wait to create them incrementally. From 93729a843aaf49e780afb6d9ee94e8048b44b151 Mon Sep 17 00:00:00 2001 From: PassionForPie Date: Sun, 12 Feb 2017 23:57:39 +0000 Subject: [PATCH 050/407] Minor grammatical fix See https://en.wiktionary.org/wiki/greengrocer's_apostrophe --- docs/tutorials/kubernetes-basics/expose-intro.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kubernetes-basics/expose-intro.html b/docs/tutorials/kubernetes-basics/expose-intro.html index 9ee7a4117a..ad4394b0e4 100644 --- a/docs/tutorials/kubernetes-basics/expose-intro.html +++ b/docs/tutorials/kubernetes-basics/expose-intro.html @@ -28,7 +28,7 @@ title: Using a Service to Expose Your App

Kubernetes Services

-

While Pods do have their own unique IP across the cluster, those IP's are not exposed outside Kubernetes. Taking into account that over time Pods may be terminated, deleted or replaced by other Pods, we need a way to let other Pods and applications automatically discover each other. Kubernetes addresses this by grouping Pods in Services. A Kubernetes Service is an abstraction layer which defines a logical set of Pods and enables external traffic exposure, load balancing and service discovery for those Pods.

+

While Pods do have their own unique IP across the cluster, those IPs are not exposed outside Kubernetes. Taking into account that over time Pods may be terminated, deleted or replaced by other Pods, we need a way to let other Pods and applications automatically discover each other. Kubernetes addresses this by grouping Pods in Services. A Kubernetes Service is an abstraction layer which defines a logical set of Pods and enables external traffic exposure, load balancing and service discovery for those Pods.

This abstraction will allow us to expose Pods to traffic originating from outside the cluster. Services have their own unique cluster-private IP address and expose a port to receive traffic. If you choose to expose the service outside the cluster, the options are:

    From 6e1ec38daa434b811c24083ea32bb9436f9ef996 Mon Sep 17 00:00:00 2001 From: scjane Date: Fri, 10 Feb 2017 18:32:47 +0800 Subject: [PATCH 051/407] Update jobs.md --- docs/user-guide/jobs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index cc2f9d38c3..d48b856ae3 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -63,7 +63,7 @@ Events: To view completed pods of a job, use `kubectl get pods --show-all`. The `--show-all` will show completed pods too. -To list all the pods that belong to job in a machine readable form, you can use a command like this: +To list all the pods that belong to a job in a machine readable form, you can use a command like this: ```shell $ pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}) @@ -100,7 +100,7 @@ the same schema as a [pod](/docs/user-guide/pods), except it is nested and does In addition to required fields for a Pod, a pod template in a job must specify appropriate labels (see [pod selector](#pod-selector)) and an appropriate restart policy. -Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` are allowed. +Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` is allowed. ### Pod Selector From e42b16b6d2c568bab4dab1d4094c2d4982b79f1f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Wimsingues?= Date: Wed, 8 Feb 2017 11:05:16 +0100 Subject: [PATCH 052/407] Update overview.md comamnd => command --- docs/concepts/abstractions/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/abstractions/overview.md index dc2e8cf7ab..1a84d74534 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/abstractions/overview.md @@ -17,7 +17,7 @@ This page explains how Kubernetes objects are represented in the Kubernetes API, A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system will constantly work to ensure that that object exists. By creating an object, you're effectively telling the Kubernetes system what you want your cluster's workload to look like; this is your cluster's **desired state**. -To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the [Kubernetes API](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md). When you use the `kubectl` comamnd-line interface, for example, the CLI makes the necessary Kubernetes API calls for you; you can also use the Kubernetes API directly in your own programs. Kubernetes currently provides a `golang` [client library](https://github.com/kubernetes/client-go) for this purpose, and other language libraries (such as [Python](https://github.com/kubernetes-incubator/client-python)) are being developed. +To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the [Kubernetes API](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md). When you use the `kubectl` command-line interface, for example, the CLI makes the necessary Kubernetes API calls for you; you can also use the Kubernetes API directly in your own programs. Kubernetes currently provides a `golang` [client library](https://github.com/kubernetes/client-go) for this purpose, and other language libraries (such as [Python](https://github.com/kubernetes-incubator/client-python)) are being developed. ### Object Spec and Status From 010b294cb2c6934967684f407d1f27faa045136c Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Fri, 10 Feb 2017 11:02:18 +0800 Subject: [PATCH 053/407] Update kubefed.md --- docs/admin/federation/kubefed.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/admin/federation/kubefed.md b/docs/admin/federation/kubefed.md index 695edf7bee..4af8db60b5 100644 --- a/docs/admin/federation/kubefed.md +++ b/docs/admin/federation/kubefed.md @@ -36,11 +36,11 @@ in your `$PATH` and set the executable permission on those binaries. Note: The URL in the curl command below downloads the binaries for Linux amd64. If you are on a different platform, please use the URL for the binaries appropriate for your platform. You can find the list -of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-3). +of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1). ```shell -curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.0/kubernetes-client-linux-amd64.tar.gz +curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.2/kubernetes-client-linux-amd64.tar.gz tar -xzvf kubernetes-client-linux-amd64.tar.gz sudo cp kubernetes/client/bin/kubefed /usr/local/bin sudo chmod +x /usr/local/bin/kubefed @@ -78,7 +78,7 @@ control plane. ## Deploying a federation control plane. -"To deploy a federation control plane on your host cluster, run +To deploy a federation control plane on your host cluster, run `kubefed init` command. When you use `kubefed init`, you must provide the following: @@ -94,7 +94,7 @@ domain suffix `example.com`: kubefed init fellowship --host-cluster-context=rivendell --dns-zone-name="example.com" ``` -The domain suffix you specify in `--dns-zone-name` must be an existing +The domain suffix specified in `--dns-zone-name` must be an existing domain that you control, and that is programmable by your DNS provider. `kubefed init` sets up the federation control plane in the host From 49169de6aa61c8f6938c23300a951e15d84d9f89 Mon Sep 17 00:00:00 2001 From: Ahmet Alp Balkan Date: Mon, 13 Feb 2017 09:21:14 -0800 Subject: [PATCH 054/407] Add SIG meetings calendar to /community/ Signed-off-by: Ahmet Alp Balkan --- community/index.html | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/community/index.html b/community/index.html index 201c70a776..5843b4829e 100644 --- a/community/index.html +++ b/community/index.html @@ -27,6 +27,13 @@ cid: community lists of SIGs, from AWS and Openstack to Big Data and Scalability, there's a place for you to contribute and instructions for forming a new SIG if your special interest isn't covered (yet).

    + +

    As a member of the Kubernetes community, you are welcome to join any of the SIG meetings + you are interested in. No registration required.

    +
    + +

Events

From 885864b9bcede01610137dd6e5bd0e6958ffcf68 Mon Sep 17 00:00:00 2001 From: Jason Lock Date: Fri, 30 Dec 2016 09:31:06 -0800 Subject: [PATCH 055/407] Increase font-size and line-height CSS Properties Increase the font-size and line-height CSS properties for the paragraph and list items within the documentation body content. Adjusting these properties will make the information a little easier to read. --- _sass/_base.sass | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/_sass/_base.sass b/_sass/_base.sass index 72557bb983..3149232bec 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -816,9 +816,9 @@ dd font-weight: 500 p - font-size: 14px + font-size: 16px font-weight: 300 - line-height: 1.25em + line-height: 1.75em p + p margin-top: 10px @@ -894,6 +894,7 @@ dd li margin-bottom: 0.75em + font-size: 16px table width: 100% From a9e63ea61d16565712f96cc7a5f6b6b61d5d9dae Mon Sep 17 00:00:00 2001 From: Jason Lock Date: Fri, 30 Dec 2016 09:58:26 -0800 Subject: [PATCH 056/407] Add line-height to List Item Add line-height to the body content list items. The same line-height as the paragraph. --- _sass/_base.sass | 1 + 1 file changed, 1 insertion(+) diff --git a/_sass/_base.sass b/_sass/_base.sass index 3149232bec..4b9e051af4 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -895,6 +895,7 @@ dd li margin-bottom: 0.75em font-size: 16px + line-height: 1.75em table width: 100% From 6ceec5095e30160b9e37856e5b799073b0783a7c Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Mon, 13 Feb 2017 10:05:51 -0800 Subject: [PATCH 057/407] reimplement PR #2217 socks shop port changed --- docs/getting-started-guides/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index a4e5679ffb..a2d4e1b834 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -269,7 +269,7 @@ It takes several minutes to download and start all the containers, watch the out Then go to the IP address of your cluster's master node in your browser, and specify the given port. So for example, `http://:`. -In the example above, this was `31869`, but it is a different port for you. +In the example above, this was `30001`, but it is a different port for you. If there is a firewall, make sure it exposes this port to the internet before you try to access it. From 4a1f2c2b55be57fe2a90557c99e8d771ce2d8ed1 Mon Sep 17 00:00:00 2001 From: samiam Date: Mon, 13 Feb 2017 11:18:16 -0500 Subject: [PATCH 058/407] Update kubectl_completion.md Quote shell commands to improve format --- docs/user-guide/kubectl/kubectl_completion.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/kubectl/kubectl_completion.md b/docs/user-guide/kubectl/kubectl_completion.md index 125b7791cd..f870bbd482 100644 --- a/docs/user-guide/kubectl/kubectl_completion.md +++ b/docs/user-guide/kubectl/kubectl_completion.md @@ -11,17 +11,19 @@ Output shell completion code for the given shell (bash or zsh). This command prints shell code which must be evaluation to provide interactive completion of kubectl commands. - $ source <(kubectl completion bash) + `$ source <(kubectl completion bash)` will load the kubectl completion code for bash. Note that this depends on the bash-completion framework. It must be sourced before sourcing the kubectl completion, e.g. on the Mac: +``` $ brew install bash-completion $ source $(brew --prefix)/etc/bash_completion $ source <(kubectl completion bash) +``` If you use zsh [1], the following will load kubectl zsh completion: - $ source <(kubectl completion zsh) + `$ source <(kubectl completion zsh)` [1] zsh completions are only supported in versions of zsh >= 5.2 From 27a450f45053b7024e155639cc2cf0b2250281b0 Mon Sep 17 00:00:00 2001 From: Andreas Kohn Date: Mon, 13 Feb 2017 15:42:53 +0100 Subject: [PATCH 059/407] Improve grammar --- docs/user-guide/jsonpath.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/jsonpath.md b/docs/user-guide/jsonpath.md index 74ce7bbc77..54c2780da8 100644 --- a/docs/user-guide/jsonpath.md +++ b/docs/user-guide/jsonpath.md @@ -6,8 +6,8 @@ JSONPath template is composed of JSONPath expressions enclosed by {}. And we add three functions in addition to the original JSONPath syntax: 1. The `$` operator is optional since the expression always start from the root object by default. -2. We can use `""` to quote text inside JSONPath expression. -3. We can use `range` operator to iterate list. +2. We can use `""` to quote text inside JSONPath expressions. +3. We can use `range` operator to iterate lists. The result object is printed as its String() function. From f59b154f6f644a42eb83e950df53ae4569da0e73 Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 13 Feb 2017 17:10:28 +0800 Subject: [PATCH 060/407] Update index.md --- docs/user-guide/jobs/work-queue-2/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-2/index.md b/docs/user-guide/jobs/work-queue-2/index.md index 5e2e8a03ec..4fd806d392 100644 --- a/docs/user-guide/jobs/work-queue-2/index.md +++ b/docs/user-guide/jobs/work-queue-2/index.md @@ -47,7 +47,7 @@ If you're not working from the source tree, you could also download [`redis-pod. ## Filling the Queue with tasks -Now lets fill the queue with some "tasks". In our example, our tasks are just strings to be +Now let's fill the queue with some "tasks". In our example, our tasks are just strings to be printed. Start a temporary interactive pod for running the Redis CLI From 62b3570acd0b24b509a44dcf6dfa4e3cbb1fcc2b Mon Sep 17 00:00:00 2001 From: caiyixiang Date: Mon, 13 Feb 2017 16:54:22 +0800 Subject: [PATCH 061/407] Update source-ip.md --- docs/tutorials/services/source-ip.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index 56daa8b345..e1eac87c4e 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -20,8 +20,8 @@ of Services, and how you can toggle this behavior according to your needs. This document makes use of the following terms: * [NAT](https://en.wikipedia.org/wiki/Network_address_translation): network address translation -* [Source NAT](/docs/user-guide/services/#ips-and-vips): replacing the source IP on a packet, usually with a node's IP -* [Destination NAT](/docs/user-guide/services/#ips-and-vips): replacing the destination IP on a packet, usually with a pod IP +* [Source NAT](https://en.wikipedia.org/wiki/Network_address_translation#SNAT): replacing the source IP on a packet, usually with a node's IP +* [Destination NAT](https://en.wikipedia.org/wiki/Network_address_translation#DNAT): replacing the destination IP on a packet, usually with a pod IP * [VIP](/docs/user-guide/services/#ips-and-vips): a virtual IP, such as the one assigned to every Kubernetes Service * [Kube-proxy](/docs/user-guide/services/#virtual-ips-and-service-proxies): a network daemon that orchestrates Service VIP management on every node From 57260ac2d4a6690adb84333ada11d2baed8203ef Mon Sep 17 00:00:00 2001 From: foxyriver Date: Mon, 13 Feb 2017 15:24:26 +0800 Subject: [PATCH 062/407] fix parallelism result According to job.yaml, the parallelism is 2 --- docs/user-guide/jobs/work-queue-1/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-1/index.md b/docs/user-guide/jobs/work-queue-1/index.md index 4ce1ffb667..f926f4211f 100644 --- a/docs/user-guide/jobs/work-queue-1/index.md +++ b/docs/user-guide/jobs/work-queue-1/index.md @@ -227,7 +227,7 @@ Name: job-wq-1 Namespace: default Image(s): gcr.io/causal-jigsaw-637/job-wq-1 Selector: app in (job-wq-1) -Parallelism: 4 +Parallelism: 2 Completions: 8 Labels: app=job-wq-1 Pods Statuses: 0 Running / 8 Succeeded / 0 Failed From 1b33b95669c7a404219edc1bff77a5dba8ce893a Mon Sep 17 00:00:00 2001 From: Christoph Blecker Date: Thu, 9 Feb 2017 17:41:08 -0800 Subject: [PATCH 063/407] Fix link to StatefulSet docs from Pods page --- docs/user-guide/pods/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user-guide/pods/index.md b/docs/user-guide/pods/index.md index 321900c948..47e2dcaf3b 100644 --- a/docs/user-guide/pods/index.md +++ b/docs/user-guide/pods/index.md @@ -1,5 +1,5 @@ --- -assignees: +assignees: title: Pods --- @@ -60,7 +60,7 @@ related thing (e.g. volume) is also destroyed and created anew. ![pod diagram](/images/docs/pod.svg){: style="max-width: 50%" } -*A multi-container pod that contains a file puller and a +*A multi-container pod that contains a file puller and a web server that uses a persistent volume for shared storage between the containers.* ## Motivation for pods @@ -150,7 +150,7 @@ Pod is exposed as a primitive in order to facilitate: * clean composition of Kubelet-level functionality with cluster-level functionality — Kubelet is effectively the "pod controller" * high-availability applications, which will expect pods to be replaced in advance of their termination and certainly in advance of deletion, such as in the case of planned evictions, image prefetching, or live pod migration [#3949](http://issue.k8s.io/3949) -There is new first-class support for stateful pods with the [StatefulSet](/docs/concepts/controllers/statefulsets/) controller (currently in beta). The feature was alpha in 1.4 and was called [PetSet](/docs/user-guide/petset/). For prior versions of Kubernetes, best practice for having stateful pods is to create a replication controller with `replicas` equal to `1` and a corresponding service, see [this MySQL deployment example](/docs/tutorials/stateful-application/run-stateful-application/). +There is new first-class support for stateful pods with the [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets/) controller (currently in beta). The feature was alpha in 1.4 and was called [PetSet](/docs/user-guide/petset/). For prior versions of Kubernetes, best practice for having stateful pods is to create a replication controller with `replicas` equal to `1` and a corresponding service, see [this MySQL deployment example](/docs/tutorials/stateful-application/run-stateful-application/). ## Termination of Pods @@ -168,7 +168,7 @@ An example flow: 6. When the grace period expires, any processes still running in the Pod are killed with SIGKILL. 7. The Kubelet will finish deleting the Pod on the API server by setting grace period 0 (immediate deletion). The Pod disappears from the API and is no longer visible from the client. -By default, all deletes are graceful within 30 seconds. The `kubectl delete` command supports the `--grace-period=` option which allows a user to override the default and specify their own value. The value `0` [force deletes](/docs/user-guide/pods/#force-termination-of-pods) the pod. In kubectl version >= 1.5, you must specify an additional flag `--force` along with `--grace-period=0` in order to perform force deletions. +By default, all deletes are graceful within 30 seconds. The `kubectl delete` command supports the `--grace-period=` option which allows a user to override the default and specify their own value. The value `0` [force deletes](/docs/user-guide/pods/#force-termination-of-pods) the pod. In kubectl version >= 1.5, you must specify an additional flag `--force` along with `--grace-period=0` in order to perform force deletions. ### Force deletion of pods From 17050a7fd2c749dbdf7057153e072086cfd8ff31 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Fri, 10 Feb 2017 11:47:39 +0800 Subject: [PATCH 064/407] replace 30 seconds with 35 seconds replace 30 seconds with 35 seconds --- .../configure-liveness-readiness-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md index 8c877ac16c..d168ceb211 100644 --- a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md +++ b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md @@ -84,7 +84,7 @@ FirstSeen LastSeen Count From SubobjectPath Type 23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Started Started container with docker id 86849c15382e ``` -After 30 seconds, view the Pod events again: +After 35 seconds, view the Pod events again: ```shell kubectl describe pod liveness-exec From ae45c11b3bc7527d2cdee42d75bce24aaf96d805 Mon Sep 17 00:00:00 2001 From: Ahmet Alp Balkan Date: Fri, 10 Feb 2017 14:48:21 -0800 Subject: [PATCH 065/407] Move "User Guide" up in the sidebar So that it is above "Accessing the Cluster" article which describes more advanced topics and does not look coherent when it is read linearly. Signed-off-by: Ahmet Alp Balkan --- _data/guides.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data/guides.yml b/_data/guides.yml index 511ece0885..98ef2db801 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -5,6 +5,8 @@ toc: - docs/whatisk8s.md +- docs/user-guide/index.md + - title: Accessing the Cluster section: - docs/user-guide/prereqs.md @@ -12,8 +14,6 @@ toc: - docs/user-guide/sharing-clusters.md - docs/user-guide/kubeconfig-file.md -- docs/user-guide/index.md - - docs/user-guide/ui.md - title: Workload Deployment and Management From 515f62be0895b9e1bbfea319c31cccf2e235b8ad Mon Sep 17 00:00:00 2001 From: Jesse Johnston Date: Thu, 9 Feb 2017 09:14:26 -0800 Subject: [PATCH 066/407] Update overview.md Update link to Kubernetes API Conventions to point to current document location. --- docs/concepts/abstractions/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/abstractions/overview.md index 1a84d74534..712ad0f99f 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/abstractions/overview.md @@ -25,7 +25,7 @@ Every Kubernetes object includes two nested object fields that govern the object For example, a Kubernetes Deployment is an object that can represent an application running on your cluster. When you create the Deployment, you might set the Deployment spec to specify that you want three replicas of the application to be running. The Kubernetes system reads the Deployment spec and starts three instances of your desired application--updating the status to match your spec. If any of those instances should fail (a status change), the Kubernetes system responds to the difference between spec and status by making a correction--in this case, starting a replacement instance. -For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#spec-and-status). +For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md). ### Describing a Kubernetes Object From c72ac0ab6be2f63d81391494418aee60d1487ef2 Mon Sep 17 00:00:00 2001 From: Erik Bean Date: Mon, 13 Feb 2017 13:25:40 -0800 Subject: [PATCH 067/407] Fix unmatched closing paren End of line 60 had a closing parenthesis, with no opening one to match. Probably left over from a previous version of the document --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index f8b9cf2f72..5c364c5376 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -57,7 +57,7 @@ Pods do not, by themselves, self-heal. If a Pod is scheduled to a Node that fail ### Pods and Controllers -A Controller can create and manage multiple Pods for you, handling replication and rollout and providing self-healing capabilities at cluster scope. For example, if a Node fails, the Controller might automatically replace the Pod by scheduling an identical replacement on a different Node). +A Controller can create and manage multiple Pods for you, handling replication and rollout and providing self-healing capabilities at cluster scope. For example, if a Node fails, the Controller might automatically replace the Pod by scheduling an identical replacement on a different Node. Some examples of Controllers that contain one or more pods include: From 4c85470449bd0813f926269e3cf05c793ef50a85 Mon Sep 17 00:00:00 2001 From: Albert Vaca Date: Mon, 13 Feb 2017 20:58:10 +0100 Subject: [PATCH 068/407] Updated dead links --- docs/user-guide/ingress.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ingress.md b/docs/user-guide/ingress.md index 809a025c57..009fd13f45 100644 --- a/docs/user-guide/ingress.md +++ b/docs/user-guide/ingress.md @@ -220,7 +220,7 @@ Note that there is a gap between TLS features supported by various Ingress contr An Ingress controller is bootstrapped with some loadbalancing policy settings that it applies to all Ingress, such as the loadbalancing algorithm, backend weight scheme etc. More advanced loadbalancing concepts (e.g.: persistent sessions, dynamic weights) are not yet exposed through the Ingress. You can still get these features through the [service loadbalancer](https://github.com/kubernetes/contrib/tree/master/service-loadbalancer). With time, we plan to distill loadbalancing patterns that are applicable cross platform into the Ingress resource. -It's also worth noting that even though health checks are not exposed directly through the Ingress, there exist parallel concepts in Kubernetes such as [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) which allow you to achieve the same end result. Please review the controller specific docs to see how they handle health checks ([nginx](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/README.md), [GCE](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/README.md#health-checks)). +It's also worth noting that even though health checks are not exposed directly through the Ingress, there exist parallel concepts in Kubernetes such as [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) which allow you to achieve the same end result. Please review the controller specific docs to see how they handle health checks ([nginx](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md), [GCE](https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md#health-checks)). ## Updating an Ingress From f4c8ba82c5ef2e3b6e09dea5bf35f686cf7f0d5b Mon Sep 17 00:00:00 2001 From: James Gregory Date: Tue, 14 Feb 2017 11:01:07 +1100 Subject: [PATCH 069/407] Update links to ingress repository Ingress moved out of contrib to kubernetes/ingress. --- docs/user-guide/ingress.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user-guide/ingress.md b/docs/user-guide/ingress.md index 009fd13f45..4628d5050d 100644 --- a/docs/user-guide/ingress.md +++ b/docs/user-guide/ingress.md @@ -81,11 +81,11 @@ __Global Parameters__: For the sake of simplicity the example Ingress has no glo ## Ingress controllers -In order for the Ingress resource to work, the cluster must have an Ingress controller running. This is unlike other types of controllers, which typically run as part of the `kube-controller-manager` binary, and which are typically started automatically as part of cluster creation. You need to choose the ingress controller implementation that is the best fit for your cluster, or implement one. Examples and instructions can be found [here](https://github.com/kubernetes/contrib/tree/master/ingress/controllers). +In order for the Ingress resource to work, the cluster must have an Ingress controller running. This is unlike other types of controllers, which typically run as part of the `kube-controller-manager` binary, and which are typically started automatically as part of cluster creation. You need to choose the ingress controller implementation that is the best fit for your cluster, or implement one. Examples and instructions can be found [here](https://github.com/kubernetes/ingress/tree/master/controllers). ## Before you begin -The following document describes a set of cross platform features exposed through the Ingress resource. Ideally, all Ingress controllers should fulfill this specification, but we're not there yet. The docs for the GCE and nginx controllers are [here](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/README.md) and [here](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/README.md) respectively. **Make sure you review controller specific docs so you understand the caveats of each one**. +The following document describes a set of cross platform features exposed through the Ingress resource. Ideally, all Ingress controllers should fulfill this specification, but we're not there yet. The docs for the GCE and nginx controllers are [here](https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md) and [here](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md) respectively. **Make sure you review controller specific docs so you understand the caveats of each one**. ## Types of Ingress @@ -214,7 +214,7 @@ spec: servicePort: 80 ``` -Note that there is a gap between TLS features supported by various Ingress controllers. Please refer to documentation on [nginx](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#https), [GCE](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/gce#tls), or any other platform specific Ingress controller to understand how TLS works in your environment. +Note that there is a gap between TLS features supported by various Ingress controllers. Please refer to documentation on [nginx](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md#https), [GCE](https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md#tls), or any other platform specific Ingress controller to understand how TLS works in your environment. ### Loadbalancing @@ -282,7 +282,7 @@ Techniques for spreading traffic across failure domains differs between cloud pr * Combining L4 and L7 Ingress * More Ingress controllers -Please track the [L7 and Ingress proposal](https://github.com/kubernetes/kubernetes/pull/12827) for more details on the evolution of the resource, and the [Ingress sub-repository](https://github.com/kubernetes/contrib/tree/master/ingress) for more details on the evolution of various Ingress controllers. +Please track the [L7 and Ingress proposal](https://github.com/kubernetes/kubernetes/pull/12827) for more details on the evolution of the resource, and the [Ingress repository](https://github.com/kubernetes/ingress/tree/master) for more details on the evolution of various Ingress controllers. ## Alternatives From 7267a33db7c39564de884f083da1b5def6d36b78 Mon Sep 17 00:00:00 2001 From: scjane Date: Tue, 14 Feb 2017 14:15:27 +0800 Subject: [PATCH 070/407] Update static-pods.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Some of the serial numbers of the ordered list in the "Static pod creation" section are incorrect because items in the list are segmented by code blocks. I think this can be solved in at least the following ways: 1.Remove the ordered list style and just use plain text each beginning with "1", "2"，...,or "step 1", "step 2"，... (My modofication adopts this one) 2.Replace the ordered list with an unordered list in which each item beginning with "step 1", "step 2"，... 3.Use HTML tags, such as
    ,
  1. to give the lists involved Are there any other better ways for this problem? I would be appreciated to know. --- docs/admin/static-pods.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 8c9e482d0d..7ac6e7c890 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -20,13 +20,13 @@ The configuration files are just standard pod definition in json or yaml format For example, this is how to start a simple web server as a static pod: -1. Choose a node where we want to run the static pod. In this example, it's `my-node1`. +1.Choose a node where we want to run the static pod. In this example, it's `my-node1`. ```shell [joe@host ~] $ ssh my-node1 ``` -2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: +2.Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: ```shell [root@my-node1 ~] $ mkdir /etc/kubernetes.d/ @@ -48,7 +48,7 @@ spec: EOF ``` -2. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. On Fedora edit `/etc/kubernetes/kubelet` to include this line: +3.Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. On Fedora edit `/etc/kubernetes/kubelet` to include this line: ```conf KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" @@ -56,7 +56,7 @@ KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manife Instructions for other distributions or Kubernetes installations may vary. -3. Restart kubelet. On Fedora, this is: +4.Restart kubelet. On Fedora, this is: ```shell [root@my-node1 ~] $ systemctl restart kubelet From 3512bc673ba3e453b8284a6338c3ddbcb9349cb4 Mon Sep 17 00:00:00 2001 From: Zhou Tao Date: Sat, 11 Feb 2017 12:03:38 +0800 Subject: [PATCH 071/407] doc(kubeadm.md) - change base64 decode option to '--decode' `base64 -D` will lead to error on Ubuntu: "base64: invalid option -- 'D'" --- docs/getting-started-guides/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index a2d4e1b834..5fc815ea9c 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -352,7 +352,7 @@ Please note: `kubeadm` is a work in progress and these limitations will be addre 1. There is no built-in way of fetching the token easily once the cluster is up and running, but here is a `kubectl` command you can copy and paste that will print out the token for you: ```console - # kubectl -n kube-system get secret clusterinfo -o yaml | grep token-map | awk '{print $2}' | base64 -D | sed "s|{||g;s|}||g;s|:|.|g;s/\"//g;" | xargs echo + # kubectl -n kube-system get secret clusterinfo -o yaml | grep token-map | awk '{print $2}' | base64 --decode | sed "s|{||g;s|}||g;s|:|.|g;s/\"//g;" | xargs echo ``` 1. If you are using VirtualBox (directly or via Vagrant), you will need to ensure that `hostname -i` returns a routable IP address (i.e. one on the second network interface, not the first one). From 4fa487f7223991a3596822e08b7e33d48c2db241 Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Mon, 13 Feb 2017 16:39:37 +0800 Subject: [PATCH 072/407] Update index.md --- docs/user-guide/persistent-volumes/index.md | 34 ++++++++++----------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index 3ecc8ea99f..8b1178e45d 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -172,23 +172,23 @@ In the CLI, the access modes are abbreviated to: | Volume Plugin | ReadWriteOnce| ReadOnlyMany| ReadWriteMany| | :--- | :---: | :---: | :---: | -| AWSElasticBlockStore | x | - | - | -| AzureFile | x | x | x | -| AzureDisk | x | - | - | -| CephFS | x | x | x | -| Cinder | x | - | - | -| FC | x | x | - | -| FlexVolume | x | x | - | -| Flocker | x | - | - | -| GCEPersistentDisk | x | x | - | -| Glusterfs | x | x | x | -| HostPath | x | - | - | -| iSCSI | x | x | - | -| PhotonPersistentDisk | x | - | - | -| Quobyte | x | x | x | -| NFS | x | x | x | -| RBD | x | x | - | -| VsphereVolume | x | - | - | +| AWSElasticBlockStore | ✓ | - | - | +| AzureFile | ✓ | ✓ | ✓ | +| AzureDisk | ✓ | - | - | +| CephFS | ✓ | ✓ | ✓ | +| Cinder | ✓ | - | - | +| FC | ✓ | ✓ | - | +| FlexVolume | ✓ | ✓ | - | +| Flocker | ✓ | - | - | +| GCEPersistentDisk | ✓ | ✓ | - | +| Glusterfs | ✓ | ✓ | ✓ | +| HostPath | ✓ | - | - | +| iSCSI | ✓ | ✓ | - | +| PhotonPersistentDisk | ✓ | - | - | +| Quobyte | ✓ | ✓ | ✓ | +| NFS | ✓ | ✓ | ✓ | +| RBD | ✓ | ✓ | - | +| VsphereVolume | ✓ | - | - | ### Class From ff088c22568518159778174ecc25c3df5592937e Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 14 Feb 2017 14:25:31 +0800 Subject: [PATCH 073/407] fix typo --- docs/user-guide/kubeconfig-file.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubeconfig-file.md b/docs/user-guide/kubeconfig-file.md index b4d7425127..93877f8a84 100644 --- a/docs/user-guide/kubeconfig-file.md +++ b/docs/user-guide/kubeconfig-file.md @@ -305,7 +305,7 @@ $ kubectl config use-context federal-context ### Final notes for tying it all together -So, tying this all together, a quick start to creating your own kubeconfig file: +So, tying this all together, a quick start to create your own kubeconfig file: - Take a good look and understand how your api-server is being launched: You need to know YOUR security requirements and policies before you can design a kubeconfig file for convenient authentication. From 8a865cf1d310d05bed04b910e3560c77f6a04df8 Mon Sep 17 00:00:00 2001 From: yupengzte Date: Tue, 14 Feb 2017 10:36:00 +0800 Subject: [PATCH 074/407] fix typo Signed-off-by: yupengzte --- docs/deprecation-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecation-policy.md b/docs/deprecation-policy.md index c51326bbc5..cdc0bb0102 100644 --- a/docs/deprecation-policy.md +++ b/docs/deprecation-policy.md @@ -233,7 +233,7 @@ after their announced deprecation for no less than:** * **Beta: 3 months or 1 release (whichever is longer)** * **Alpha: 0 releases** -**Rule #6: Deprecated CLI elements must emit warnings (optionally disableable) +**Rule #6: Deprecated CLI elements must emit warnings (optionally disable) when used.** ## Deprecating a feature or behavior From e56a7296ca805029ce4536bc1a2b6e6f5836dc2e Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Tue, 14 Feb 2017 15:18:45 +0000 Subject: [PATCH 075/407] Fix the standard storageClass for GCE As you can see here: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go#L117-L121 The default is not ssd --- docs/user-guide/persistent-volumes/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index 8b1178e45d..4508a1a25f 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -396,7 +396,7 @@ parameters: zone: us-central1-a ``` -* `type`: `pd-standard` or `pd-ssd`. Default: `pd-ssd` +* `type`: `pd-standard` or `pd-ssd`. Default: `pd-standard` * `zone`: GCE zone. If not specified, a random zone in the same region as controller-manager will be chosen. #### Glusterfs From 7b42b5f9dd0a57fc94664f3b54627b3732216cca Mon Sep 17 00:00:00 2001 From: scjane Date: Tue, 14 Feb 2017 15:03:43 +0800 Subject: [PATCH 076/407] Update multiple-schedulers.md Similar to #2518. But I think an unordered list can be used here :) --- docs/admin/multiple-schedulers.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/admin/multiple-schedulers.md b/docs/admin/multiple-schedulers.md index 3e3b4d270b..c20e129b19 100644 --- a/docs/admin/multiple-schedulers.md +++ b/docs/admin/multiple-schedulers.md @@ -95,7 +95,7 @@ Now that our second scheduler is running, let's create some pods, and direct the scheduler in that pod spec. Let's look at three examples. -1. Pod spec without any scheduler name +- Pod spec without any scheduler name {% include code.html language="yaml" file="multiple-schedulers/pod1.yaml" ghlink="/docs/admin/multiple-schedulers/pod1.yaml" %} @@ -108,7 +108,7 @@ scheduler in that pod spec. Let's look at three examples. kubectl create -f pod1.yaml ``` -2. Pod spec with `default-scheduler` +- Pod spec with `default-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod2.yaml" ghlink="/docs/admin/multiple-schedulers/pod2.yaml" %} @@ -121,7 +121,7 @@ scheduler in that pod spec. Let's look at three examples. kubectl create -f pod2.yaml ``` -3. Pod spec with `my-scheduler` +- Pod spec with `my-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod3.yaml" ghlink="/docs/admin/multiple-schedulers/pod3.yaml" %} From 7bafe3557f466cd2213adf799084099e5f9daad7 Mon Sep 17 00:00:00 2001 From: Fabrizio Milo Date: Tue, 14 Feb 2017 09:29:08 -0800 Subject: [PATCH 077/407] Update pod.md --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index 5c364c5376..59a7beaee5 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -27,7 +27,7 @@ The [Kubernetes Blog](http://blog.kubernetes.io) has some additional information * [The Distributed System Toolkit: Patterns for Composite Containers](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html) * [Container Design Patterns](http://blog.kubernetes.io/2016/06/container-design-patterns.html) -Each Pod is meant to run a single instance of a given application. If you want to scale your application horizontally (e.g., run muliple instances), you should use multiple Pods, one for each instance. In Kubernetes, this is generally referred to as _replication_. Replicated Pods are usually created and managed as a group by an abstraction called a Controller. See [Pods and Controllers](#pods-and-controllers) for more information. +Each Pod is meant to run a single instance of a given application. If you want to scale your application horizontally (e.g., run multiple instances), you should use multiple Pods, one for each instance. In Kubernetes, this is generally referred to as _replication_. Replicated Pods are usually created and managed as a group by an abstraction called a Controller. See [Pods and Controllers](#pods-and-controllers) for more information. ### How Pods Manage Multiple Containers From 38033ae6bf377e14947c4844674e174abc4aa5cb Mon Sep 17 00:00:00 2001 From: Fabrizio Milo Date: Tue, 14 Feb 2017 09:20:59 -0800 Subject: [PATCH 078/407] Update overview.md --- docs/concepts/abstractions/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/abstractions/overview.md index 712ad0f99f..4e832eaffb 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/abstractions/overview.md @@ -9,7 +9,7 @@ This page explains how Kubernetes objects are represented in the Kubernetes API, {% capture body %} ## Understanding Kubernetes Objects -*Kubernetes Objects* are persistent entities in the Kubernetes system. Kubenetes uses these entities to represent the state of your cluster. Specifically, they can describe: +*Kubernetes Objects* are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe: * What containerized applications are running (and on which nodes) * The resources available to those applications From 2fb161e033f9a4f84b95c1bd04cb87806ebcf57a Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Feb 2017 11:07:33 -0800 Subject: [PATCH 079/407] fix Kubenetes typo Kubenetes -> Kubernetes --- .../configure-liveness-readiness-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md index d168ceb211..ff05756350 100644 --- a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md +++ b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md @@ -247,7 +247,7 @@ where you would set it. Suppose the Container listens on 127.0.0.1 and the Pod's If your pod relies on virtual hosts, which is probably the more common case, you should not use `host`, but rather set the `Host` header in `httpHeaders`. -In addition to command probes and HTTP probes, Kubenetes supports +In addition to command probes and HTTP probes, Kubernetes supports [TCP probes](/docs/api-reference/v1/definitions/#_v1_tcpsocketaction). {% endcapture %} From 710a17c10d56180677587028c3a1481ceb0eafa6 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Tue, 14 Feb 2017 16:37:49 +0800 Subject: [PATCH 080/407] column READY is missed column READY is missed --- docs/user-guide/deployments.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index 0b4ae0d85a..52d24391c4 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -62,8 +62,8 @@ This indicates that the Deployment has created all three replicas, and all repli ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-2035384211 3 3 18s +NAME DESIRED CURRENT READY AGE +nginx-deployment-2035384211 3 3 0 18s ``` You may notice that the name of the Replica Set is always `-`. @@ -180,9 +180,9 @@ We can run `kubectl get rs` to see that the Deployment updated the Pods by creat ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 3 3 6s -nginx-deployment-2035384211 0 0 36s +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 3 3 0 6s +nginx-deployment-2035384211 0 0 0 36s ``` Running `get pods` should now show only the new Pods: @@ -287,10 +287,10 @@ You will also see that both the number of old replicas (nginx-deployment-1564180 ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 2 2 25s -nginx-deployment-2035384211 0 0 36s -nginx-deployment-3066724191 2 2 6s +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 2 2 0 25s +nginx-deployment-2035384211 0 0 0 36s +nginx-deployment-3066724191 2 2 2 6s ``` Looking at the Pods created, you will see that the 2 Pods created by new Replica Set are stuck in an image pull loop. @@ -514,10 +514,10 @@ The Deployment was still in progress when we paused it, so the actions of scalin ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 2 2 1h -nginx-deployment-2035384211 2 2 1h -nginx-deployment-3066724191 0 0 1h +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 2 2 2 1h +nginx-deployment-2035384211 2 2 0 1h +nginx-deployment-3066724191 0 0 0 1h ``` In a separate terminal, watch for rollout status changes and you'll see the rollout won't continue: @@ -546,10 +546,10 @@ deployment nginx-deployment successfully rolled out ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 3 3 1h -nginx-deployment-2035384211 0 0 1h -nginx-deployment-3066724191 0 0 1h +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 3 3 3 1h +nginx-deployment-2035384211 0 0 0 1h +nginx-deployment-3066724191 0 0 0 1h ``` Note: You cannot rollback a paused Deployment until you resume it. From 36ebe59068f338e63c86dbc7d3ceedfb55a936a0 Mon Sep 17 00:00:00 2001 From: Arthur Miranda Date: Tue, 14 Feb 2017 15:41:28 -0300 Subject: [PATCH 081/407] Fix broken/outdated links in the ingress.md file The Ingress controllers have moved to the kubernetes/ingress repository, then some links are broken or outdated. --- docs/user-guide/ingress.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/ingress.md b/docs/user-guide/ingress.md index 4628d5050d..0a4f13da2f 100644 --- a/docs/user-guide/ingress.md +++ b/docs/user-guide/ingress.md @@ -44,9 +44,9 @@ It can be configured to give services externally-reachable urls, load balance tr Before you start using the Ingress resource, there are a few things you should understand. The Ingress is a beta resource, not available in any Kubernetes release prior to 1.1. You need an Ingress controller to satisfy an Ingress, simply creating the resource will have no effect. -GCE/GKE deploys an ingress controller on the master. You can deploy any number of custom ingress controllers in a pod. You must annotate each ingress with the appropriate class, as indicated [here](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#running-multiple-ingress-controllers) and [here](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#disabling-glbc). +GCE/GKE deploys an ingress controller on the master. You can deploy any number of custom ingress controllers in a pod. You must annotate each ingress with the appropriate class, as indicated [here](https://github.com/kubernetes/ingress/tree/master/controllers/nginx#running-multiple-ingress-controllers) and [here](https://github.com/kubernetes/ingress/blob/master/controllers/gce/BETA_LIMITATIONS.md#disabling-glbc). -Make sure you review the [beta limitations](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/gce/BETA_LIMITATIONS.md) of this controller. In environments other than GCE/GKE, you need to [deploy a controller](https://github.com/kubernetes/contrib/tree/master/ingress/controllers) as a pod. +Make sure you review the [beta limitations](https://github.com/kubernetes/ingress/blob/master/controllers/gce/BETA_LIMITATIONS.md) of this controller. In environments other than GCE/GKE, you need to [deploy a controller](https://github.com/kubernetes/ingress/tree/master/controllers) as a pod. ## The Ingress Resource @@ -71,7 +71,7 @@ spec: __Lines 1-4__: As with all other Kubernetes config, an Ingress needs `apiVersion`, `kind`, and `metadata` fields. For general information about working with config files, see [here](/docs/user-guide/deploying-applications), [here](/docs/user-guide/configuring-containers), and [here](/docs/user-guide/working-with-resources). -__Lines 5-7__: Ingress [spec](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status) has all the information needed to configure a loadbalancer or proxy server. Most importantly, it contains a list of rules matched against all incoming requests. Currently the Ingress resource only supports http rules. +__Lines 5-7__: Ingress [spec](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#spec-and-status) has all the information needed to configure a loadbalancer or proxy server. Most importantly, it contains a list of rules matched against all incoming requests. Currently the Ingress resource only supports http rules. __Lines 8-9__: Each http rule contains the following information: A host (e.g.: foo.bar.com, defaults to * in this example), a list of paths (e.g.: /testpath) each of which has an associated backend (test:80). Both the host and path must match the content of an incoming request before the loadbalancer directs traffic to the backend. From 7c97c392ed7edcd75afd8ccf6a60c671847250ef Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Tue, 14 Feb 2017 10:36:08 +0800 Subject: [PATCH 082/407] add "--show-all" to kubectl get pods add "--show-all" to "kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}" ,because the pod was ended --- docs/user-guide/jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index d48b856ae3..d64ae9d730 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -66,7 +66,7 @@ To view completed pods of a job, use `kubectl get pods --show-all`. The `--show To list all the pods that belong to a job in a machine readable form, you can use a command like this: ```shell -$ pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}) +$ pods=$(kubectl get pods --show-all --selector=job-name=pi --output=jsonpath={.items..metadata.name}) echo $pods pi-aiw0a ``` From cfaf34835cec826a3452858135477c5b6e82c8ea Mon Sep 17 00:00:00 2001 From: Robert Kubis Date: Sat, 11 Feb 2017 16:05:47 +0000 Subject: [PATCH 083/407] Update cheatsheet for multi-container handling Added how to retrieve logs for containers in multi-container pods --- docs/user-guide/kubectl-cheatsheet.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user-guide/kubectl-cheatsheet.md b/docs/user-guide/kubectl-cheatsheet.md index c147304a7d..7fc0142f04 100644 --- a/docs/user-guide/kubectl-cheatsheet.md +++ b/docs/user-guide/kubectl-cheatsheet.md @@ -197,7 +197,9 @@ $ kubectl -n my-ns delete po,svc --all # Delete all pods and servic ```console $ kubectl logs my-pod # dump pod logs (stdout) +$ kubectl logs my-pod -c my-container # dump pod container logs (stdout, multi-container case) $ kubectl logs -f my-pod # stream pod logs (stdout) +$ kubectl logs -f my-pod -c my-container # stream pod container logs (stdout, multi-container case) $ kubectl run -i --tty busybox --image=busybox -- sh # Run pod as interactive shell $ kubectl attach my-pod -i # Attach to Running Container $ kubectl port-forward my-pod 5000:6000 # Forward port 6000 of Pod to your to 5000 on your local machine From 04d6793544c18280b934ae9c94d2357bfe9594e2 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Feb 2017 13:49:14 -0800 Subject: [PATCH 084/407] re: issue #1671; update to direct link Update Docker Volumes link to https://docs.docker.com/engine/tutorials/dockervolumes/ --- docs/user-guide/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 9c69683525..0d4e77a97d 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -54,7 +54,7 @@ Before running examples in the user guides, please ensure you have completed the : A service defines a set of pods and a means by which to access them, such as single stable IP address and corresponding DNS name. [**Volume**](/docs/user-guide/volumes/) -: A volume is a directory, possibly with some data in it, which is accessible to a Container as part of its filesystem. Kubernetes volumes build upon [Docker Volumes](https://docs.docker.com/userguide/dockervolumes/), adding provisioning of the volume directory and/or device. +: A volume is a directory, possibly with some data in it, which is accessible to a Container as part of its filesystem. Kubernetes volumes build upon [Docker Volumes](https://docs.docker.com/engine/tutorials/dockervolumes/), adding provisioning of the volume directory and/or device. [**Secret**](/docs/user-guide/secrets/) : A secret stores sensitive data, such as authentication tokens, which can be made available to containers upon request. From 566f61f4b939a319eee3400cf2977afe5693605b Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Feb 2017 14:34:13 -0800 Subject: [PATCH 085/407] re: issue #2034; fix docker image link (#2532) * re: issue #2034; fix docker image link change link in line 26 to: https://gcr.io/google_containers/hpa-example * point link to local Dockerfile * specify it as the Dockerfile, not image --- docs/user-guide/horizontal-pod-autoscaling/walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md b/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md index f0a5cbc64a..616061e930 100644 --- a/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md +++ b/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md @@ -23,7 +23,7 @@ heapster monitoring will be turned-on by default). ## Step One: Run & expose php-apache server To demonstrate Horizontal Pod Autoscaler we will use a custom docker image based on the php-apache image. -The image can be found [here](/docs/user-guide/horizontal-pod-autoscaling/image). +The Dockerfile can be found [here](/docs/user-guide/horizontal-pod-autoscaling/image/Dockerfile). It defines an [index.php](/docs/user-guide/horizontal-pod-autoscaling/image/index.php) page which performs some CPU intensive computations. First, we will start a deployment running the image and expose it as a service: From 255364959892e9818d0f9b84f0df7a15aefd1118 Mon Sep 17 00:00:00 2001 From: foxish Date: Wed, 15 Feb 2017 13:27:00 -0800 Subject: [PATCH 086/407] Fix travis.yml --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 09a28b78cb..d05639d195 100644 --- a/.travis.yml +++ b/.travis.yml @@ -14,6 +14,7 @@ install: - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apiserver - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/client-go - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/sample-apiserver +- rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ From 5df783425dc2f01b7be1208becd3eb3e04f125d9 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Tue, 14 Feb 2017 20:03:22 +0800 Subject: [PATCH 087/407] The key-value "optional:true" doesn't work The key-value "optional:true" doesn't work in 1.5.2 yet. So should we add a note here? "The key-value "optional:true" will be work after kubernetes 1.5.2." --- docs/user-guide/configmap/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 3227e416c0..e2de3a7aec 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -590,3 +590,5 @@ Kubelet only supports use of ConfigMap for pods it gets from the API server. Th created using kubectl, or indirectly via a replication controller. It does not include pods created via the Kubelet's `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) + +The key-value "optional:true" will be work after kubernetes 1.5.2. From 2ab877198bcc16c24f637f1a6e6905cf469db331 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Wed, 15 Feb 2017 13:15:20 +0800 Subject: [PATCH 088/407] Change sentence base on reviewer's comment "NOTE: The key-value optional:true is supported for kubernetes 1.5.3 and above." --- docs/user-guide/configmap/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index e2de3a7aec..5e074e5787 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -591,4 +591,4 @@ created using kubectl, or indirectly via a replication controller. It does not via the Kubelet's `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) -The key-value "optional:true" will be work after kubernetes 1.5.2. +NOTE: The key-value optional:true is supported for kubernetes 1.5.3 and above. From 0366fcc114d2d21d2b68c863f23afec1048ab1fd Mon Sep 17 00:00:00 2001 From: Michael Mrowetz Date: Wed, 15 Feb 2017 11:58:43 +0900 Subject: [PATCH 089/407] #2534 mark openstack-heat as standalone-salt-conf --- docs/admin/salt.md | 12 ++++++------ docs/getting-started-guides/openstack-heat.md | 9 +++++++-- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/admin/salt.md b/docs/admin/salt.md index 05d0ae9250..b570dc8473 100644 --- a/docs/admin/salt.md +++ b/docs/admin/salt.md @@ -10,11 +10,11 @@ The Salt scripts are shared across multiple hosting providers and depending on w ## Salt cluster setup -The **salt-master** service runs on the kubernetes-master [(except on the default GCE setup)](#standalone-salt-configuration-on-gce). +The **salt-master** service runs on the kubernetes-master [(except on the default GCE and OpenStack-Heat setup)](#standalone-salt-configuration-on-gce-and-others). The **salt-minion** service runs on the kubernetes-master and each kubernetes-node in the cluster. -Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file [(except on GCE)](#standalone-salt-configuration-on-gce). +Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file [(except on GCE and OpenStack-Heat)](#standalone-salt-configuration-on-gce-and-others). ```shell [root@kubernetes-master] $ cat /etc/salt/minion.d/master.conf @@ -25,15 +25,15 @@ The salt-master is contacted by each salt-minion and depending upon the machine If you are running the Vagrant based environment, the **salt-api** service is running on the kubernetes-master. It is configured to enable the vagrant user to introspect the salt cluster in order to find out about machines in the Vagrant environment via a REST API. -## Standalone Salt Configuration on GCE +## Standalone Salt Configuration on GCE and others -On GCE, the master and nodes are all configured as [standalone minions](http://docs.saltstack.com/en/latest/topics/tutorials/standalone_minion.html). The configuration for each VM is derived from the VM's [instance metadata](https://cloud.google.com/compute/docs/metadata) and then stored in Salt grains (`/etc/salt/minion.d/grains.conf`) and pillars (`/srv/salt-overlay/pillar/cluster-params.sls`) that local Salt uses to enforce state. +On GCE and OpenStack, using the Openstack-Heat provider, the master and nodes are all configured as [standalone minions](http://docs.saltstack.com/en/latest/topics/tutorials/standalone_minion.html). The configuration for each VM is derived from the VM's [instance metadata](https://cloud.google.com/compute/docs/metadata) and then stored in Salt grains (`/etc/salt/minion.d/grains.conf`) and pillars (`/srv/salt-overlay/pillar/cluster-params.sls`) that local Salt uses to enforce state. -All remaining sections that refer to master/minion setups should be ignored for GCE. One fallout of the GCE setup is that the Salt mine doesn't exist - there is no sharing of configuration amongst nodes. +All remaining sections that refer to master/minion setups should be ignored for GCE and OpenStack. One fallout of this setup is that the Salt mine doesn't exist - there is no sharing of configuration amongst nodes. ## Salt security -*(Not applicable on default GCE setup.)* +*(Not applicable on default GCE and OpenStack-Heat setup.)* Security is not enabled on the salt-master, and the salt-master is configured to auto-accept incoming requests from minions. It is not recommended to use this security configuration in production environments without deeper study. (In some environments this isn't as bad as it might sound if the salt master port isn't externally accessible and you trust everyone on your network.) diff --git a/docs/getting-started-guides/openstack-heat.md b/docs/getting-started-guides/openstack-heat.md index e4fb7c277d..8a2fd52d11 100644 --- a/docs/getting-started-guides/openstack-heat.md +++ b/docs/getting-started-guides/openstack-heat.md @@ -23,7 +23,7 @@ This guide assumes you have access to a working OpenStack cluster with the follo - Heat - DNS resolution of instance names -By default this provider provisions 4 m1.medium instances. If you do not have resources available, please see the [Set additional configuration values](#set-additional-configuration-values) section for information on reducing the footprint of your cluster. +By default this provider provisions 4 `m1.medium` instances. If you do not have resources available, please see the [Set additional configuration values](#set-additional-configuration-values) section for information on reducing the footprint of your cluster. ## Pre-Requisites If you already have the required versions of the OpenStack CLI tools installed and configured, you can move on to the [Starting a cluster](#starting-a-cluster) section. @@ -92,7 +92,7 @@ Please see the contents of these files for documentation regarding each variable ## Starting a cluster -Once Kubernetes version 1.3 is released, and you've installed the OpenStack CLI tools and have set your OpenStack environment variables, issue this command: +Once you've installed the OpenStack CLI tools and have set your OpenStack environment variables, issue this command: ```sh export KUBERNETES_PROVIDER=openstack-heat; curl -sS https://get.k8s.io | bash @@ -194,6 +194,11 @@ nova list --name=$STACK_NAME See the [OpenStack CLI Reference](http://docs.openstack.org/cli-reference/) for more details. +### Salt + +The OpenStack-Heat provider uses a [standalone Salt configuration](/docs/admin/salt/#standalone-salt-configuration-on-gce-and-others). +It only uses Salt for bootstraping the machines and creates no salt-master and does not auto-start the salt-minion service on the nodes. + ## SSHing to your nodes Your public key was added during the cluster turn-up, so you can easily ssh to them for troubleshooting purposes. From 76e1bfc55d40b3c70d25a68eb16379951b16fb48 Mon Sep 17 00:00:00 2001 From: Arthur Miranda Date: Wed, 15 Feb 2017 15:07:08 -0300 Subject: [PATCH 090/407] Update the links of Deployment User Guide The file 'blob/master/docs/devel/api-conventions.md' has moved to 'community/blob/master/contributors/devel/api-conventions.md' Updates the links associated with these file. --- docs/user-guide/deployments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index 52d24391c4..fb8f2cc0e0 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -615,7 +615,7 @@ the Deployment's `status.conditions`: * Status=False * Reason=ProgressDeadlineExceeded -See the [Kubernetes API conventions](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties) for more information on status conditions. +See the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#typical-status-properties) for more information on status conditions. Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with `Reason=ProgressDeadlineExceeded`. @@ -725,7 +725,7 @@ As with all other Kubernetes configs, a Deployment needs `apiVersion`, `kind`, a `metadata` fields. For general information about working with config files, see [deploying applications](/docs/user-guide/deploying-applications), [configuring containers](/docs/user-guide/configuring-containers), and [using kubectl to manage resources](/docs/user-guide/working-with-resources) documents. -A Deployment also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). +A Deployment also needs a [`.spec` section](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#spec-and-status). ### Pod Template From 9b3f420bd9153fa930d004eb018fbf5a04052c54 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Wed, 15 Feb 2017 16:15:20 +0800 Subject: [PATCH 091/407] kubectl_apply.md-change it for label key When I test this command, if the configmap has a label, it just delete the configmaps which has the same label key and that are not in the file. kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap So I advise change the description like this: "Apply the configuration in manifest.yaml and delete all the other configmaps with the same label key that are not in the file." --- docs/user-guide/kubectl/kubectl_apply.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl/kubectl_apply.md b/docs/user-guide/kubectl/kubectl_apply.md index c729263d6c..81c1a12137 100644 --- a/docs/user-guide/kubectl/kubectl_apply.md +++ b/docs/user-guide/kubectl/kubectl_apply.md @@ -32,7 +32,7 @@ kubectl apply -f FILENAME # Apply the configuration in manifest.yaml that matches label app=nginx and delete all the other resources that are not in the file and match label app=nginx. kubectl apply --prune -f manifest.yaml -l app=nginx - # Apply the configuration in manifest.yaml and delete all the other configmaps that are not in the file. + # Apply the configuration in manifest.yaml and delete all the other configmaps with the same label key that are not in the file. kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap ``` From 4702554cb68556457811ac9c0ce79c08facd8d0f Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Feb 2017 14:40:39 -0800 Subject: [PATCH 092/407] rollback PR #2522 Rollingback PR #2522 until we can verify which version optional:true will be available in. --- docs/user-guide/configmap/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 5e074e5787..40b3fa181b 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -591,4 +591,3 @@ created using kubectl, or indirectly via a replication controller. It does not via the Kubelet's `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) -NOTE: The key-value optional:true is supported for kubernetes 1.5.3 and above. From c222e9da33df792408351234bd1a9fc92eb36134 Mon Sep 17 00:00:00 2001 From: Michail Kargakis Date: Wed, 25 Jan 2017 10:44:31 +0100 Subject: [PATCH 093/407] Update deployment completeness documentation --- docs/user-guide/deployments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index fb8f2cc0e0..f2e854690e 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -578,6 +578,7 @@ Kubernetes marks a Deployment as _complete_ when it has the following characteri equals or exceeds the number required by the Deployment strategy. * All of the replicas associated with the Deployment have been updated to the latest version you've specified, meaning any updates you've requested have been completed. +* No old pods for the Deployment are running. You can check if a Deployment has completed by using `kubectl rollout status`. If the rollout completed successfully, `kubectl rollout status` returns a zero exit code. From a3a2233e6c9dc7c4e56b26762f75ec4456e394e7 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Feb 2017 14:56:39 -0800 Subject: [PATCH 094/407] Move Pod Lifecycle to Concepts. (#2420) * Move Pod Lifecycle to Concepts. * Put moved note on old page. * Update pod-lifecycle.md --- _data/concepts.yml | 6 + docs/concepts/workloads/pods/pod-lifecycle.md | 282 ++++++++++++++++++ docs/user-guide/pod-states.md | 166 +---------- 3 files changed, 290 insertions(+), 164 deletions(-) create mode 100644 docs/concepts/workloads/pods/pod-lifecycle.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 4227231cc1..432f4c86e1 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -25,6 +25,12 @@ toc: section: - docs/concepts/object-metadata/annotations.md +- title: Workloads + section: + - title: Pods + section: + - docs/concepts/workloads/pods/pod-lifecycle.md + - title: Configuration section: - docs/concepts/configuration/container-command-args.md diff --git a/docs/concepts/workloads/pods/pod-lifecycle.md b/docs/concepts/workloads/pods/pod-lifecycle.md new file mode 100644 index 0000000000..540ba506fc --- /dev/null +++ b/docs/concepts/workloads/pods/pod-lifecycle.md @@ -0,0 +1,282 @@ +--- +title: Pod Lifecycle +--- + +{% capture overview %} + +{% comment %}Updated: 4/14/2015{% endcomment %} +{% comment %}Edited and moved to Concepts section: 2/2/17{% endcomment %} + +This page describes the lifecycle of a Pod. + +{% endcapture %} + + +{% capture body %} + +## Pod phase + +A Pod's `status` field is a +[PodStatus](/docs/resources-reference/v1.5/#podstatus-v1) +object, which has a `phase` field. + +The phase of a Pod is a simple, high-level summary of where the Pod is in its +lifecycle. The phase is not intended to be a comprehensive rollup of observations +of Container or Pod state, nor is it intended to be a comprehensive state machine. + +The number and meanings of Pod phase values are tightly guarded. +Other than what is documented here, nothing should be assumed about Pods that +have a given `phase` value. + +Here are the possible values for `phase`: + +* Pending: The Pod has been accepted by the Kubernetes system, but one or more of + the Container images has not been created. This includes time before being + scheduled as well as time spent downloading images over the network, + which could take a while. + +* Running: The Pod has been bound to a node, and all of the Containers have been + created. At least one Container is still running, or is in the process of + starting or restarting. + +* Succeeded: All Containers in the Pod have terminated in success, and will not + be restarted. + +* Failed: All Containers in the Pod have terminated, and at least one Container + has terminated in failure. That is, the Container either exited with non-zero + status or was terminated by the system. + +* Unknown: For some reason the state of the Pod could not be obtained, typically + due to an error in communicating with the host of the Pod. + +## Pod conditions + +A Pod has a PodStatus, which has an array of +[PodConditions](docs/resources-reference/v1.5/#podcondition). Each element +of the PodCondition array has a `type` field and a `status` field. The `type` +field is a string, with possible values PodScheduled, Ready, Initialized, and +Unschedulable. The `status` field is a string, with possible values True, False, +and Unknown. + +## Container probes + +A [Probe](/docs/resources-reference/v1.5/#probe-v1) is a diagnostic +performed periodically by the [kubelet](/docs/admin/kubelet/) +on a Container. To perform a diagnostic, +the kublet calls a +[Handler](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#Handler) implemented by +the Container. There are three types of handlers: + +* [ExecAction](/docs/resources-reference/v1.5/#execaction-v1): + Executes a specified command inside the Container. The diagnostic + is considered successful if the command exits with a status code of 0. + +* [TCPSocketAction](/docs/resources-reference/v1.5/#tcpsocketaction-v1): + Performs a TCP check against the Container's IP address on + a specified port. The diagnostic is considered successful if the port is open. + +* [HTTPGetAction](/docs/resources-reference/v1.5/#httpgetaction-v1): + Performs an HTTP Get request against the Container's IP + address on a specified port and path. The diagnostic is considered successful + if the response has a status code greater than or equal to 200 and less than 400. + +Each probe has one of three results: + +* Success: The Container passed the diagnostic. +* Failure: The Container failed the diagnostic. +* Unknown: The diagnostic failed, so no action should be taken. + +The kubelet can optionally perform and react to two kinds of probes on running +Containers: + +* `livenessProbe`: Indicates whether the Container is running. If + the liveness probe fails, the kubelet kills the Container, and the Container + is subjected to its [restart policy](#restart-policy). If a Container does not + provide a liveness probe, the default state is `Success`. + +* `readinessProbe`: Indicates whether the Container is ready to service requests. + If the readiness probe fails, the endpoints controller removes the Pod's IP + address from the endpoints of all Services that match the Pod. The default + state of readiness before the initial delay is `Failure`. If a Container does + not provide a readiness probe, the default state is `Success`. + +### When should you use liveness or readiness probes? + +If the process in your Container is able to crash on its own whenever it +encounters an issue or becomes unhealthy, you do not necessarily need a liveness +probe; the kubelet will automatically perform the correct action in accordance +with the Pod's `restartPolicy`. + +If you'd like your Container to be killed and restarted if a probe fails, then +specify a liveness probe, and specify a `restartPolicy` of Always or OnFailure. + +If you'd like to start sending traffic to a Pod only when a probe succeeds, +specify a readiness probe. In this case, the readiness probe might be the same +as the liveness probe, but the existence of the readiness probe in the spec means +that the Pod will start without receiving any traffic and only start receiving +traffic after the probe starts succeeding. + +If you want your Container to be able to take itself down for maintenance, you +can specify a readiness probe that checks an endpoint specific to readiness that +is different from the liveness probe. + +Note that if you just want to be able to drain requests when the Pod is deleted, +you do not necessarily need a readiness probe; on deletion, the Pod automatically +puts itself into an unready state regardless of whether the readiness probe exists. +The Pod remains in the unready state while it waits for the Containers in the Pod +to stop. + +## Pod and Container status + +For detailed information about Pod Container status, see +[PodStatus](/docs/resources-reference/v1.5/#podstatus-v1) +and +[ContainerStatus](/docs/resources-reference/v1.5/#containerstatus-v1). +Note that the information reported as Pod status depends on the current +[ContainerState](/docs/resources-reference/v1.5/#containerstate-v1). + +## Restart policy + +A PodSpec has a `restartPolicy` field with possible values Always, OnFailure, +and Never. The default value is Always. +`restartPolicy` applies to all Containers in the Pod. `restartPolicy` only +refers to restarts of the Containers by the kubelet on the same node. Failed +Containers that are restarted by the kubelet are restarted with an exponential +back-off delay (10s, 20s, 40s ...) capped at five minutes, and is reset after ten +minutes of successful execution. As discussed in the +[Pods document](/docs/user-guide/pods/#durability-of-pods-or-lack-thereof), +once bound to a node, a Pod will never be rebound to another node. + + + +## Pod lifetime + +In general, Pods do not disappear until someone destroys them. This might be a +human or a controller. The only exception to +this rule is that Pods with a `phase` of Succeeded or Failed for more than some +duration (determined by the master) will expire and be automatically destroyed. + +Three types of controllers are available: + +- Use a [Job](/docs/user-guide/jobs/) for Pods that are expected to terminate, + for example, batch computations. Jobs are appropriate only for Pods with + `restartPolicy` equal to OnFailure or Never. + +- Use a [ReplicationController](/docs/user-guide/replication-controller/), + [ReplicaSet](/docs/user-guide/replicasets/), or + [Deployment](/docs/user-guide/deployments/) + for Pods that are not expected to terminate, for example, web servers. + ReplicationControllers are appropriate only for Pods with a `restartPolicy` of + Always. + +- Use a [DaemonSet](/docs/admin/daemons/) for Pods that need to run one per + machine, because they provide a machine-specific system service. + +All three types of controllers contain a PodTemplate. It +is recommended to create the appropriate controller and let +it create Pods, rather than directly create Pods yourself. That is because Pods +alone are not resilient to machine failures, but controllers are. + +If a node dies or is disconnected from the rest of the cluster, Kubernetes +applies a policy for setting the `phase` of all Pods on the lost node to Failed. + +## Examples + +### Advanced liveness probe example + +Liveness probes are executed by the kubelet, so all requests are made in the +kubelet network namespace. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + labels: + test: liveness + name: liveness-http +spec: + containers: + - args: + - /server + image: gcr.io/google_containers/liveness + livenessProbe: + httpGet: + # when "host" is not defined, "PodIP" will be used + # host: my-host + # when "scheme" is not defined, "HTTP" scheme will be used. Only "HTTP" and "HTTPS" are allowed + # scheme: HTTPS + path: /healthz + port: 8080 + httpHeaders: + - name: X-Custom-Header + value: Awesome + initialDelaySeconds: 15 + timeoutSeconds: 1 + name: liveness +``` + +### Example states + + * Pod is running and has one Container. Container exits with success. + * Log completion event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Pod `phase` becomes Succeeded. + * Never: Pod `phase` becomes Succeeded. + + * Pod is running and has one Container. Container exits with failure. + * Log failure event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Pod `phase` becomes Failed. + + * Pod is running and has two Containers. Container 1 exits with failure. + * Log failure event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Do not restart Container; Pod `phase` stays Running. + * If Container 1 is not running, and Container 2 exits: + * Log failure event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Pod `phase` becomes Failed. + + * Pod is running and has one Container. Container runs out of memory. + * Container terminates in failure. + * Log OOM event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Log failure event; Pod `phase` becomes Failed. + + * Pod is running, and a disk dies. + * Kill all Containers. + * Log appropriate event. + * Pod `phase` becomes Failed. + * If running under a controller, Pod is recreated elsewhere. + + * Pod is running, and its node is segmented out. + * Node controller waits for timeout. + * Node controller sets Pod `phase` to Failed. + * If running under a controller, Pod is recreated elsewhere. + +{% endcapture %} + + +{% capture whatsnext %} + +* Get hands-on experience + [attaching handlers to Container lifecycle events](/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/). + +* Get hands-on experience + [configuring liveness and readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/). + +* [Container Lifecycle Hooks](/docs/user-guide/container-environment/) + +{% endcapture %} + +{% include templates/concept.md %} + diff --git a/docs/user-guide/pod-states.md b/docs/user-guide/pod-states.md index 462f8b65bd..a3cea728cc 100644 --- a/docs/user-guide/pod-states.md +++ b/docs/user-guide/pod-states.md @@ -4,168 +4,6 @@ assignees: title: The Lifecycle of a Pod --- -Updated: 4/14/2015 - -This document covers the lifecycle of a pod. It is not an exhaustive document, but an introduction to the topic. - -## Pod Phase - -As consistent with the overall [API convention](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties), phase is a simple, high-level summary of the phase of the lifecycle of a pod. It is not intended to be a comprehensive rollup of observations of container-level or even pod-level conditions or other state, nor is it intended to be a comprehensive state machine. - -The number and meanings of `PodPhase` values are tightly guarded. Other than what is documented here, nothing should be assumed about pods with a given `PodPhase`. - -* Pending: The pod has been accepted by the system, but one or more of the container images has not been created. This includes time before being scheduled as well as time spent downloading images over the network, which could take a while. -* Running: The pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. -* Succeeded: All containers in the pod have terminated in success, and will not be restarted. -* Failed: All containers in the pod have terminated, at least one container has terminated in failure (exited with non-zero exit status or was terminated by the system). -* Unknown: For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod. - -## Pod Conditions - -A pod containing containers that specify readiness probes will also report the Ready condition. Condition status values may be `True`, `False`, or `Unknown`. - -## Container Probes - -A [Probe](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#Probe) is a diagnostic performed periodically by the kubelet on a container. Specifically the diagnostic is one of three [Handlers](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#Handler): - -* `ExecAction`: executes a specified command inside the container expecting on success that the command exits with status code 0. -* `TCPSocketAction`: performs a tcp check against the container's IP address on a specified port expecting on success that the port is open. -* `HTTPGetAction`: performs an HTTP Get against the container's IP address on a specified port and path expecting on success that the response has a status code greater than or equal to 200 and less than 400. - -Each probe will have one of three results: - -* `Success`: indicates that the container passed the diagnostic. -* `Failure`: indicates that the container failed the diagnostic. -* `Unknown`: indicates that the diagnostic failed so no action should be taken. - -The kubelet can optionally perform and react to two kinds of probes on running containers: - -* `LivenessProbe`: indicates whether the container is *live*, i.e. running. If the LivenessProbe fails, the kubelet will kill the container and the container will be subjected to its [RestartPolicy](#restartpolicy). The default state of Liveness before the initial delay is `Success`. The state of Liveness for a container when no probe is provided is assumed to be `Success`. -* `ReadinessProbe`: indicates whether the container is *ready* to service requests. If the ReadinessProbe fails, the endpoints controller will remove the pod's IP address from the endpoints of all services that match the pod. The default state of Readiness before the initial delay is `Failure`. The state of Readiness for a container when no probe is provided is assumed to be `Success`. - -### When should I use liveness or readiness probes? - -If the process in your container is able to crash on its own whenever it encounters an issue or becomes unhealthy, you do not necessarily need a liveness probe - the kubelet will automatically perform the correct action in accordance with the RestartPolicy when the process crashes. - -If you'd like your container to be killed and restarted if a probe fails, then specify a LivenessProbe and a RestartPolicy of `Always` or `OnFailure`. - -If you'd like to start sending traffic to a pod only when a probe succeeds, specify a ReadinessProbe. In this case, the ReadinessProbe may be the same as the LivenessProbe, but the existence of the ReadinessProbe in the spec means that the pod will start without receiving any traffic and only start receiving traffic once the probe starts succeeding. - -If a container wants the ability to take itself down for maintenance, you can specify a ReadinessProbe that checks an endpoint specific to readiness which is different than the LivenessProbe. - -Note that if you just want to be able to drain requests when the pod is deleted, you do not necessarily need a ReadinessProbe - on deletion, the pod automatically puts itself into an unready state regardless of whether the ReadinessProbe exists or not while it waits for the containers in the pod to stop. - -## Container Statuses - -More detailed information about the current (and previous) container statuses can be found in [ContainerStatuses](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#PodStatus). The information reported depends on the current [ContainerState](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#ContainerState), which may be Waiting, Running, or Terminated. - -## RestartPolicy - -The possible values for RestartPolicy are `Always`, `OnFailure`, or `Never`. If RestartPolicy is not set, the default value is `Always`. RestartPolicy applies to all containers in the pod. RestartPolicy only refers to restarts of the containers by the Kubelet on the same node. Failed containers that are restarted by Kubelet, are restarted with an exponential back-off delay, the delay is in multiples of sync-frequency 0, 1x, 2x, 4x, 8x ... capped at 5 minutes and is reset after 10 minutes of successful execution. As discussed in the [pods document](/docs/user-guide/pods/#durability-of-pods-or-lack-thereof), once bound to a node, a pod will never be rebound to another node. This means that some kind of controller is necessary in order for a pod to survive node failure, even if just a single pod at a time is desired. - -Three types of controllers are currently available: - -- Use a [`Job`](/docs/user-guide/jobs/) for pods which are expected to terminate (e.g. batch computations). -- Use a [`ReplicationController`](/docs/user-guide/replication-controller/) or [`Deployment`](/docs/user-guide/deployments/) - for pods which are not expected to terminate (e.g. web servers). -- Use a [`DaemonSet`](/docs/admin/daemons/): Use for pods which need to run 1 per machine because they provide a - machine-specific system service. -If you are unsure whether to use ReplicationController or Daemon, then see [Daemon Set versus -Replication Controller](/docs/admin/daemons/#daemon-set-versus-replication-controller). - -`ReplicationController` is *only* appropriate for pods with `RestartPolicy = Always`. -`Job` is *only* appropriate for pods with `RestartPolicy` equal to `OnFailure` or `Never`. - -All 3 types of controllers contain a PodTemplate, which has all the same fields as a Pod. -It is recommended to create the appropriate controller and let it create pods, rather than to -directly create pods yourself. That is because pods alone are not resilient to machine failures, -but Controllers are. - -## Pod lifetime - -In general, pods which are created do not disappear until someone destroys them. This might be a human or a `ReplicationController`, or another controller. The only exception to this rule is that pods with a `PodPhase` of `Succeeded` or `Failed` for more than some duration (determined by the master) will expire and be automatically reaped. - -If a node dies or is disconnected from the rest of the cluster, some entity within the system (call it the NodeController for now) is responsible for applying policy (e.g. a timeout) and marking any pods on the lost node as `Failed`. - -## Examples - -### Advanced livenessProbe example - -Liveness probes are executed by `kubelet`, so all requests will be made within kubelet network namespace. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - labels: - test: liveness - name: liveness-http -spec: - containers: - - args: - - /server - image: gcr.io/google_containers/liveness - livenessProbe: - httpGet: - # when "host" is not defined, "PodIP" will be used - # host: my-host - # when "scheme" is not defined, "HTTP" scheme will be used. Only "HTTP" and "HTTPS" are allowed - # scheme: HTTPS - path: /healthz - port: 8080 - httpHeaders: - - name: X-Custom-Header - value: Awesome - initialDelaySeconds: 15 - timeoutSeconds: 1 - name: liveness -``` - -### Example states - - * Pod is `Running`, 1 container, container exits success - * Log completion event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: pod becomes `Succeeded` - * Never: pod becomes `Succeeded` - - * Pod is `Running`, 1 container, container exits failure - * Log failure event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: pod becomes `Failed` - - * Pod is `Running`, 2 containers, container 1 exits failure - * Log failure event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: pod stays `Running` - * When container 2 exits... - * Log failure event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: pod becomes `Failed` - - * Pod is `Running`, container becomes OOM - * Container terminates in failure - * Log OOM event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: log failure event, pod becomes `Failed` - - * Pod is `Running`, a disk dies - * All containers are killed - * Log appropriate event - * Pod becomes `Failed` - * If running under a controller, pod will be recreated elsewhere - - * Pod is `Running`, its node is segmented out - * NodeController waits for timeout - * NodeController marks pod `Failed` - * If running under a controller, pod will be recreated elsewhere +{% include user-guide-content-moved.md %} +[Pod Lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/) From f3487280605588e31fa6b69480b5a8f1013aee72 Mon Sep 17 00:00:00 2001 From: foxish Date: Wed, 15 Feb 2017 16:06:49 -0800 Subject: [PATCH 095/407] Fix travis and add comments --- .travis.yml | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index d05639d195..b0a565803c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,9 +7,17 @@ install: - export PATH=$GOPATH/bin:$PATH - mkdir -p $HOME/gopath/src/k8s.io - mv $TRAVIS_BUILD_DIR $HOME/gopath/src/k8s.io/kubernetes.github.io + +# (1) Fetch dependencies for us to run the tests in test/examples_test.go - go get -t -v k8s.io/kubernetes.github.io/test -- git clone --depth=50 --branch=master https://github.com/kubernetes/md-check $HOME/gopath/src/k8s.io/md-check -- go get -t -v k8s.io/md-check + +# The dependencies are complicated for test/examples_test.go +# k8s.io/kubernetes/pkg is a dependency, which in turn depends on apimachinery +# but we also have apimachinery directly as one of our dependencies, which causes a conflict. +# Additionally, we get symlinks when we clone the directory. The below steps do the following: + +# (a) Replace the symlink with the actual dependencies from kubernetes/staging/src/ +# (b) copy all the vendored files to $GOPATH/src - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apiserver - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/client-go @@ -18,6 +26,12 @@ install: - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ + +# (2) Fetch md-check along with all its dependencies. +- git clone --depth=50 --branch=master https://github.com/kubernetes/md-check $HOME/gopath/src/k8s.io/md-check +- go get -t -v k8s.io/md-check + +# (3) Fetch mungedocs - go get -v k8s.io/kubernetes/cmd/mungedocs script: From 2e3758381d7a2dbec084939898366d6879f02c96 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Mon, 13 Feb 2017 15:03:37 -0800 Subject: [PATCH 096/407] Move Guide toic to Tasks: kubectl exec. --- _data/tasks.yml | 1 + .../kubectl/get-shell-running-container.md | 148 ++++++++++++++++++ docs/tasks/kubectl/shell-demo.yaml | 14 ++ docs/user-guide/getting-into-containers.md | 68 +------- 4 files changed, 165 insertions(+), 66 deletions(-) create mode 100644 docs/tasks/kubectl/get-shell-running-container.md create mode 100644 docs/tasks/kubectl/shell-demo.yaml diff --git a/_data/tasks.yml b/_data/tasks.yml index cb61b1a05a..6d7cdb921e 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -6,6 +6,7 @@ toc: - title: Using the Kubectl Command-Line section: - docs/tasks/kubectl/list-all-running-container-images.md + - docs/tasks/kubectl/get-shell-running-container.md - title: Configuring Pods and Containers section: diff --git a/docs/tasks/kubectl/get-shell-running-container.md b/docs/tasks/kubectl/get-shell-running-container.md new file mode 100644 index 0000000000..a005f3e0fe --- /dev/null +++ b/docs/tasks/kubectl/get-shell-running-container.md @@ -0,0 +1,148 @@ +--- +assignees: +- caesarxuchao +- mikedanese +title: Getting a Shell to a Running Container +--- + +{% capture overview %} + +This page shows how to use `kubectl exec` to get a shell to a +running Container. + +{% endcapture %} + + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + + +{% capture steps %} + +## Getting a shell to a Container + +In this exercise, you create a Pod that has one Container. The Container +runs the nginx image. Here is the configuration file for the Pod: + +{% include code.html language="yaml" file="shell-demo.yaml" ghlink="/docs/tasks/kubectl/shell-demo.yaml" %} + +Create the Pod: + +```shell +kubectl create -f https://k8s.io/docs/tasks/kubectl/shell-demo.yaml +``` + +Verify that the Container is running: + +```shell +kubectl get pod shell-demo +``` + +Get a shell to the running Container: + +```shell +kubectl exec -it shell-demo -- /bin/bash +``` + +In your shell, list the running processes: + +```shell +root@shell-demo:/# ps aux +``` + +In your shell, list the nginx processes: + +```shell +root@shell-demo:/# ps aux | grep nginx +``` + +In your shell, experiment with other commands. Here are +some examples: + +```shell +root@shell-demo:/# ls / +root@shell-demo:/# cat /proc/mounts +root@shell-demo:/# cat /proc/1/maps +root@shell-demo:/# apt-get update +root@shell-demo:/# apt-get install tcpdump +root@shell-demo:/# tcpdump +root@shell-demo:/# apt-get install lsof +root@shell-demo:/# lsof +``` + +## Writing the root page for nginx + +Look again at the configuration file for your Pod. The Pod +has an `emptyDir` volume, and the Container mounts the volume +at `/usr/share/nginx/html`. + +In your shell, create an `index.html` file in the `/usr/share/nginx/html` +directory: + +```shell +root@shell-demo:/# echo Hello shell demo > /usr/share/nginx/html/index.html +``` + +In your shell, send a GET request to the nginx server: + +```shell +root@shell-demo:/# apt-get update +root@shell-demo:/# apt-get install curl +root@shell-demo:/# curl localhost +``` + +The output shows the text that you wrote to the `index.html` file: + +```shell +Hello shell demo +``` + +When you are finished with your shell, enter `exit`. + +## Running individual commands in a Container + +In an ordinary command window, not your shell, list the environment +variables in the running Container: + +```shell +kubectl exec shell-demo env +``` + +Experiment running other commands. Here are some examples: + +```shell +kubectl exec shell-demo ps aux +kubectl exec shell-demo ls / +kubectl exec shell-demo cat /proc/1/mounts +``` + +{% endcapture %} + +{% capture discussion %} + +## Opening a shell when a Pod has more than one Container + +If a Pod has more than one Container, use `--container` or `-c` to +specify a Container in the `kubectl exec` command. For example, +suppose you have a Pod named my-pod, and the Pod has two containers +named main-app and helper-app. The following command would open a +shell to the main-app Container. + +```shell +kubectl exec -it my-pod --container main-app -- /bin/bash +``` + +{% endcapture %} + + +{% capture whatsnext %} + +* [kubectl exec](/docs/user-guide/kubectl/v1.5/#exec) + +{% endcapture %} + + +{% include templates/task.md %} diff --git a/docs/tasks/kubectl/shell-demo.yaml b/docs/tasks/kubectl/shell-demo.yaml new file mode 100644 index 0000000000..2a7d274a64 --- /dev/null +++ b/docs/tasks/kubectl/shell-demo.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Pod +metadata: + name: shell-demo +spec: + volumes: + - name: shared-data + emptyDir: {} + containers: + - name: nginx + image: nginx + volumeMounts: + - name: shared-data + mountPath: /usr/share/nginx/html diff --git a/docs/user-guide/getting-into-containers.md b/docs/user-guide/getting-into-containers.md index bf6a5a8a2f..ff89f111f6 100644 --- a/docs/user-guide/getting-into-containers.md +++ b/docs/user-guide/getting-into-containers.md @@ -5,70 +5,6 @@ assignees: title: Running Commands in a Container with kubectl exec --- -Developers can use `kubectl exec` to run commands in a container. This guide demonstrates two use cases. +{% include user-guide-content-moved.md %} -## Using kubectl exec to check the environment variables of a container - -Kubernetes exposes [services](/docs/user-guide/services/#environment-variables) through environment variables. It is convenient to check these environment variables using `kubectl exec`. - -We first create a pod and a service, - -```shell -$ kubectl create -f examples/guestbook/redis-master-controller.yaml -$ kubectl create -f examples/guestbook/redis-master-service.yaml -``` -wait until the pod is Running and Ready, - -```shell -$ kubectl get pod -NAME READY REASON RESTARTS AGE -redis-master-ft9ex 1/1 Running 0 12s -``` - -then we can check the environment variables of the pod, - -```shell -$ kubectl exec redis-master-ft9ex env -... -REDIS_MASTER_SERVICE_PORT=6379 -REDIS_MASTER_SERVICE_HOST=10.0.0.219 -... -``` - -We can use these environment variables in applications to find the service. - - -## Using kubectl exec to check the mounted volumes - -It is convenient to use `kubectl exec` to check if the volumes are mounted as expected. -We first create a Pod with a volume mounted at /data/redis, - -```shell -kubectl create -f docs/user-guide/walkthrough/pod-redis.yaml -``` - -wait until the pod is Running and Ready, - -```shell -$ kubectl get pods -NAME READY REASON RESTARTS AGE -storage 1/1 Running 0 1m -``` - -we then use `kubectl exec` to verify that the volume is mounted at /data/redis, - -```shell -$ kubectl exec storage ls /data -redis -``` - -## Using kubectl exec to open a bash terminal in a pod - -After all, open a terminal in a pod is the most direct way to introspect the pod. Assuming the pod/storage is still running, run - -```shell -$ kubectl exec -ti storage -- bash -root@storage:/data# -``` - -This gets you a terminal. \ No newline at end of file +[Getting a Shell to a Running Container](/docs/tasks/kubectl/get-shell-running-container/) From ed63f9a8ce3f40c088cde8b172d4119e6a5c5134 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Thu, 16 Feb 2017 07:44:49 -0500 Subject: [PATCH 097/407] Let's put kubectl in ~/bin. --- docs/getting-started-guides/ubuntu/installation.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/installation.md b/docs/getting-started-guides/ubuntu/installation.md index cc93995144..9e3bdd8b2b 100644 --- a/docs/getting-started-guides/ubuntu/installation.md +++ b/docs/getting-started-guides/ubuntu/installation.md @@ -159,15 +159,17 @@ juju scp kubernetes-master/0:config ~/.kube/config Fetch a binary for the architecture you have deployed. If your client is a different architecture you will need to get the appropriate `kubectl` binary -through other means. +through other means. In this example we copy kubectl to `~/bin` for convenience, +by default this should be in your $PATH. ``` -juju scp kubernetes-master/0:kubectl ./kubectl +mkdir -p ~/bin +juju scp kubernetes-master/0:kubectl ~/bin/kubectl ``` Query the cluster: - ./kubectl cluster-info + kubectl cluster-info Output: From 0d9e568d1bd2be06655c857e963abfea9d3236e0 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Thu, 16 Feb 2017 16:20:08 +0800 Subject: [PATCH 098/407] remove its name from file content --- docs/admin/service-accounts-admin.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/admin/service-accounts-admin.md b/docs/admin/service-accounts-admin.md index 4a31fbeced..0c7b495a10 100644 --- a/docs/admin/service-accounts-admin.md +++ b/docs/admin/service-accounts-admin.md @@ -71,8 +71,9 @@ account. To create additional API tokens for a service account, create a secret of type `ServiceAccountToken` with an annotation referencing the service account, and the controller will update it with a generated token: -```json secret.json: + +```json { "kind": "Secret", "apiVersion": "v1", @@ -100,4 +101,4 @@ kubectl delete secret mysecretname ### Service Account Controller Service Account Controller manages ServiceAccount inside namespaces, and ensures -a ServiceAccount named "default" exists in every active namespace. \ No newline at end of file +a ServiceAccount named "default" exists in every active namespace. From ac44956a7595561f0768b5cf6130d9e183f78496 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Thu, 16 Feb 2017 17:17:54 +0800 Subject: [PATCH 099/407] replace kubernetes.d with kubelet.d replace kubernetes.d with kubelet.d --- docs/admin/static-pods.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 8c9e482d0d..cacf6e2010 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -26,11 +26,11 @@ For example, this is how to start a simple web server as a static pod: [joe@host ~] $ ssh my-node1 ``` -2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: +2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`: ```shell -[root@my-node1 ~] $ mkdir /etc/kubernetes.d/ -[root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml +[root@my-node1 ~] $ mkdir /etc/kubelet.d/ +[root@my-node1 ~] $ cat </etc/kubelet.d/static-web.yaml apiVersion: v1 kind: Pod metadata: @@ -114,11 +114,11 @@ CONTAINER ID IMAGE COMMAND CREATED ... Running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes pods as files appear/disappear in this directory. ```shell -[joe@my-node1 ~] $ mv /etc/kubernetes.d/static-web.yaml /tmp +[joe@my-node1 ~] $ mv /etc/kubelet.d/static-web.yaml /tmp [joe@my-node1 ~] $ sleep 20 [joe@my-node1 ~] $ docker ps // no nginx container is running -[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubernetes.d/ +[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubelet.d/ [joe@my-node1 ~] $ sleep 20 [joe@my-node1 ~] $ docker ps CONTAINER ID IMAGE COMMAND CREATED ... From 37a51088f7ba4230c27bbb7c4a2fb1f6a044ae09 Mon Sep 17 00:00:00 2001 From: liangxiaoping Date: Thu, 16 Feb 2017 18:37:34 +0800 Subject: [PATCH 100/407] fix etcd disaster-recovery hyperlink --- docs/admin/etcd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/etcd.md b/docs/admin/etcd.md index 41a3e35ddb..06b0eda07c 100644 --- a/docs/admin/etcd.md +++ b/docs/admin/etcd.md @@ -20,7 +20,7 @@ Data Reliability: for reasonable safety, either etcd needs to be run as a etcd) or etcd's data directory should be located on durable storage (e.g., GCE's persistent disk). In either case, if high availability is required--as it might be in a production cluster--the data directory ought to be [backed up -periodically](https://coreos.com/etcd/docs/2.2.1/admin_guide.html#disaster-recovery), +periodically](https://coreos.com/etcd/docs/latest/op-guide/recovery.html), to reduce downtime in case of corruption. ## Default configuration From 2638f4f0879584c8712dca004374119d13780a16 Mon Sep 17 00:00:00 2001 From: "architect.bian" Date: Thu, 16 Feb 2017 18:43:21 +0800 Subject: [PATCH 101/407] modify typora modify namespace-kube-system to namespace=kube-system --- docs/tasks/administer-cluster/dns-horizontal-autoscaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md b/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md index 069409058a..3d80cc77b2 100644 --- a/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md +++ b/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md @@ -156,7 +156,7 @@ The output is: Verify that the replica count is zero: - kubectl get deployment --namespace-kube-system + kubectl get deployment --namespace=kube-system The output displays 0 in the DESIRED and CURRENT columns: From 0da9c85fd44a643aa39fbbbfb7c01f0977866d03 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 16 Feb 2017 12:59:54 -0800 Subject: [PATCH 102/407] Move Guide topic to Tasks: Downward API (#2439) --- _data/tasks.yml | 1 + .../dapi-volume-resources.yaml | 54 ++++ .../configure-pod-container/dapi-volume.yaml | 39 +++ ...nward-api-volume-expose-pod-information.md | 242 ++++++++++++++++++ ...ronment-variable-expose-pod-information.md | 12 + docs/tasks/index.md | 3 + 6 files changed, 351 insertions(+) create mode 100644 docs/tasks/configure-pod-container/dapi-volume-resources.yaml create mode 100644 docs/tasks/configure-pod-container/dapi-volume.yaml create mode 100644 docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 6d7cdb921e..5dae817f8e 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -16,6 +16,7 @@ toc: - docs/tasks/configure-pod-container/configure-volume-storage.md - docs/tasks/configure-pod-container/configure-persistent-volume-storage.md - docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md + - docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md - docs/tasks/configure-pod-container/distribute-credentials-secure.md - docs/tasks/configure-pod-container/pull-image-private-registry.md - docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md diff --git a/docs/tasks/configure-pod-container/dapi-volume-resources.yaml b/docs/tasks/configure-pod-container/dapi-volume-resources.yaml new file mode 100644 index 0000000000..65770f283f --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-volume-resources.yaml @@ -0,0 +1,54 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kubernetes-downwardapi-volume-example-2 +spec: + containers: + - name: client-container + image: gcr.io/google_containers/busybox:1.24 + command: ["sh", "-c"] + args: + - while true; do + echo -en '\n'; + if [[ -e /etc/cpu_limit ]]; then + echo -en '\n'; cat /etc/cpu_limit; fi; + if [[ -e /etc/cpu_request ]]; then + echo -en '\n'; cat /etc/cpu_request; fi; + if [[ -e /etc/mem_limit ]]; then + echo -en '\n'; cat /etc/mem_limit; fi; + if [[ -e /etc/mem_request ]]; then + echo -en '\n'; cat /etc/mem_request; fi; + sleep 5; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + volumeMounts: + - name: podinfo + mountPath: /etc + readOnly: false + volumes: + - name: podinfo + downwardAPI: + items: + - path: "cpu_limit" + resourceFieldRef: + containerName: client-container + resource: limits.cpu + - path: "cpu_request" + resourceFieldRef: + containerName: client-container + resource: requests.cpu + - path: "mem_limit" + resourceFieldRef: + containerName: client-container + resource: limits.memory + - path: "mem_request" + resourceFieldRef: + containerName: client-container + resource: requests.memory + diff --git a/docs/tasks/configure-pod-container/dapi-volume.yaml b/docs/tasks/configure-pod-container/dapi-volume.yaml new file mode 100644 index 0000000000..7126cefae5 --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-volume.yaml @@ -0,0 +1,39 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kubernetes-downwardapi-volume-example + labels: + zone: us-est-coast + cluster: test-cluster1 + rack: rack-22 + annotations: + build: two + builder: john-doe +spec: + containers: + - name: client-container + image: gcr.io/google_containers/busybox + command: ["sh", "-c"] + args: + - while true; do + if [[ -e /etc/labels ]]; then + echo -en '\n\n'; cat /etc/labels; fi; + if [[ -e /etc/annotations ]]; then + echo -en '\n\n'; cat /etc/annotations; fi; + sleep 5; + done; + volumeMounts: + - name: podinfo + mountPath: /etc + readOnly: false + volumes: + - name: podinfo + downwardAPI: + items: + - path: "labels" + fieldRef: + fieldPath: metadata.labels + - path: "annotations" + fieldRef: + fieldPath: metadata.annotations + diff --git a/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md b/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md new file mode 100644 index 0000000000..a55d22f387 --- /dev/null +++ b/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md @@ -0,0 +1,242 @@ +--- +title: Exposing Pod Information to Containers Using a DownwardApiVolumeFile +--- + +{% capture overview %} + +This page shows how a Pod can use a DownwardAPIVolumeFile to expose information +about itself to Containers running in the Pod. A DownwardAPIVolumeFile can expose +Pod fields and Container fields. + +{% endcapture %} + + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + +{% capture steps %} + +## The Downward API + +There are two ways to expose Pod and Container fields to a running Container: + +* [Environment variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) +* DownwardAPIVolumeFiles + +Together, these two ways of exposing Pod and Container fields are called the +*Downward API*. + +## Storing Pod fields + +In this exercise, you create a Pod that has one Container. +Here is the configuration file for the Pod: + +{% include code.html language="yaml" file="dapi-volume.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-volume.yaml" %} + +In the configuration file, you can see that the Pod has a `downwardAPI` Volume, +and the Container mounts the Volume at `/etc`. + +Look at the `items` array under `downwardAPI`. Each element of the array is a +[DownwardAPIVolumeFile](/docs/resources-reference/v1.5/#downwardapivolumefile-v1). +The first element specifies that the value of the Pod's +`metadata.labels` field should be stored in a file named `labels`. +The second element specifies that the value of the Pod's `annotations` +field should be stored in a file named `annotations`. + +**Note**: The fields in this example are Pod fields. They are not +fields of the Container in the Pod. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-volume.yaml +``` + +Verify that Container in the Pod is running: + +```shell +kubectl get pods +``` + +View the Container's logs: + +```shell +kubectl logs kubernetes-downwardapi-volume-example +``` + +The output shows the contents of the `labels` file and the `annotations` file: + +```shell +cluster="test-cluster1" +rack="rack-22" +zone="us-est-coast" + +build="two" +builder="john-doe" +``` + +Get a shell into the Container that is running in your Pod: + +``` +kubectl exec -it kubernetes-downwardapi-volume-example -- sh +``` + +In your shell, view the `labels` file: + +```shell +/# cat /etc/labels +``` + +The output shows that all of the Pod's labels have been written +to the `labels` file: + +```shell +cluster="test-cluster1" +rack="rack-22" +zone="us-est-coast" +``` + +Similarly, view the `annotations` file: + +```shell +/# cat /etc/annotations +``` + +View the files in the `/etc` directory: + +```shell +/# ls -laR /etc +``` + +In the output, you can see that the `labels` and `annotations` files +are in a temporary subdirectory: in this example, +`..2982_06_02_21_47_53.299460680`. In the `/etc` directory, `..data` is +a symbolic link to the temporary subdirectory. Also in the `/etc` directory, +`labels` and `annotations` are symbolic links. + +``` +drwxr-xr-x ... Feb 6 21:47 ..2982_06_02_21_47_53.299460680 +lrwxrwxrwx ... Feb 6 21:47 ..data -> ..2982_06_02_21_47_53.299460680 +lrwxrwxrwx ... Feb 6 21:47 annotations -> ..data/annotations +lrwxrwxrwx ... Feb 6 21:47 labels -> ..data/labels + +/etc/..2982_06_02_21_47_53.299460680: +total 8 +-rw-r--r-- ... Feb 6 21:47 annotations +-rw-r--r-- ... Feb 6 21:47 labels +``` + +Using symbolic links enables dynamic atomic refresh of the metadata; updates are +written to a new temporary directory, and the `..data` symlink is updated +atomically using +[rename(2)](http://man7.org/linux/man-pages/man2/rename.2.html). + +Exit the shell: + +```shell +/# exit +``` + +## Storing Container fields + +The preceding exercise, you stored Pod fields in a DownwardAPIVolumeFile. +In this next exercise, you store Container fields. Here is the configuration +file for a Pod that has one Container: + +{% include code.html language="yaml" file="dapi-volume-resources.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-volume-resources.yaml" %} + +In the configuration file, you can see that the Pod has a `downwardAPI` Volume, +and the Container mounts the Volume at `/etc`. + +Look at the `items` array under `downwardAPI`. Each element of the array is a +DownwardAPIVolumeFile. + +The first element specifies that in the Container named `client-container`, +the value of the `limits.cpu` field +`metadata.labels` field should be stored in a file named `cpu_limit`. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-volume-resources.yaml +``` + +Get a shell into the Container that is running in your Pod: + +``` +kubectl exec -it kubernetes-downwardapi-volume-example-2 -- sh +``` + +In your shell, view the `cpu_limit` file: + +```shell +/# cat /etc/cpu_limit +``` +You can use similar commands to view the `cpu_request`, `mem_limit` and +`mem_request` files. + +{% endcapture %} + +{% capture discussion %} + +## Capabilities of the Downward API + +The following information is available to Containers through environment +variables and DownwardAPIVolumeFiles: + +* The node’s name +* The Pod’s name +* The Pod’s namespace +* The Pod’s IP address +* The Pod’s service account name +* A Container’s CPU limit +* A container’s CPU request +* A Container’s memory limit +* A Container’s memory request + +In addition, the following information is available through +DownwardAPIVolumeFiles. + +* The Pod's labels +* The Pod's annotations + +**Note**: If CPU and memory limits are not specified for a Container, the +Downward API defaults to the node allocatable value for CPU and memory. + +## Projecting keys to specific paths and file permissions + +You can project keys to specific paths and specific permissions on a per-file +basis. For more information, see +[Secrets](/docs/user-guide/secrets/). + +## Motivation for the Downward API + +It is sometimes useful for a Container to have information about itself, without +being overly coupled to Kubernetes. The Downward API allows containers to consume +information about themselves or the cluster without using the Kubernetes client +or API server. + +An example is an existing application that assumes a particular well-known +environment variable holds a unique identifier. One possibility is to wrap the +application, but that is tedious and error prone, and it violates the goal of low +coupling. A better option would be to use the Pod's name as an identifier, and +inject the Pod's name into the well-known environment variable. + +{% endcapture %} + + +{% capture whatsnext %} + +* [PodSpec](/docs/resources-reference/v1.5/#podspec-v1) +* [Volume](/docs/resources-reference/v1.5/#volume-v1) +* [DownwardAPIVolumeSource](/docs/resources-reference/v1.5/#downwardapivolumesource-v1) +* [DownwardAPIVolumeFile](/docs/resources-reference/v1.5/#downwardapivolumefile-v1) +* [ResourceFieldSelector](/docs/resources-reference/v1.5/#resourcefieldselector-v1) + +{% endcapture %} + +{% include templates/task.md %} + diff --git a/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md index 1390727000..10ac238c5e 100644 --- a/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md +++ b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md @@ -26,6 +26,17 @@ Together, these two ways of exposing Pod and Container fields are called the {% capture steps %} +## The Downward API + +There are two ways to expose Pod and Container fields to a running Container: + +* Environment variables +* [DownwardAPIVolumeFiles](/docs/resources-reference/v1.5/#downwardapivolumefile-v1) + +Together, these two ways of exposing Pod and Container fields are called the +*Downward API*. + + ## Using Pod fields as values for environment variables In this exercise, you create a Pod that has one Container. Here is the @@ -161,3 +172,4 @@ The output shows the values of selected environment variables: {% include templates/task.md %} + diff --git a/docs/tasks/index.md b/docs/tasks/index.md index 39f81c6751..d490fe5532 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -12,6 +12,8 @@ single thing, typically by giving a short sequence of steps. * [Defining a Command and Arguments for a Container](/docs/tasks/configure-pod-container/define-command-argument-container/) * [Assigning CPU and RAM Resources to a Container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/) * [Configuring a Pod to Use a Volume for Storage](/docs/tasks/configure-pod-container/configure-volume-storage/) +* [Exposing Pod Information to Containers Through Environment Variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) +* [Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) * [Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) * [Pulling an Image from a Private Registry](/docs/tasks/configure-pod-container/pull-image-private-registry/) * [Configuring Liveness and Readiness Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) @@ -55,3 +57,4 @@ single thing, typically by giving a short sequence of steps. If you would like to write a task page, see [Creating a Documentation Pull Request](/docs/contribute/create-pull-request/). + From 05c1dfe76c4b182d706954eba6ef8846d3018bc7 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 16 Feb 2017 13:20:48 -0800 Subject: [PATCH 103/407] Add note about moved content. (#2563) --- docs/user-guide/downward-api/index.md | 135 +------------------ docs/user-guide/downward-api/volume/index.md | 116 +--------------- 2 files changed, 4 insertions(+), 247 deletions(-) diff --git a/docs/user-guide/downward-api/index.md b/docs/user-guide/downward-api/index.md index 33b2b06c49..beb1cddd9b 100644 --- a/docs/user-guide/downward-api/index.md +++ b/docs/user-guide/downward-api/index.md @@ -5,137 +5,6 @@ assignees: title: Using the Downward API to Convey Pod Properties --- -It is sometimes useful for a container to have information about itself, but we -want to be careful not to over-couple containers to Kubernetes. The downward -API allows containers to consume information about themselves or the system and -expose that information how they want it, without necessarily coupling to the -Kubernetes client or REST API. +{% include user-guide-content-moved.md %} -An example of this is a "legacy" app that is already written assuming -that a particular environment variable will hold a unique identifier. While it -is often possible to "wrap" such applications, this is tedious and error prone, -and violates the goal of low coupling. Instead, the user should be able to use -the Pod's name, for example, and inject it into this well-known variable. - - -## Capabilities - -The following information is available to a `Pod` through the downward API: - -* The node's name -*   The pod's name -* The pod's namespace -* The pod's IP -* The pod's service account name -* A container's cpu limit -* A container's cpu request -* A container's memory limit -* A container's memory request - -More information will be exposed through this same API over time. - - -## Exposing pod information into a container - -Containers consume information from the downward API using environment -variables or using a volume plugin. - - -## Environment variables - -Most environment variables in the Kubernetes API use the `value` field to carry -simple values. However, the alternate `valueFrom` field allows you to specify -a `fieldRef` to select fields from the pod's definition, and a `resourceFieldRef` -to select fields from one of its container's definition. - -The `fieldRef` field is a structure that has an `apiVersion` field and a `fieldPath` -field. The `fieldPath` field is an expression designating a field of the pod. The -`apiVersion` field is the version of the API schema that the `fieldPath` is -written in terms of. If the `apiVersion` field is not specified it is -defaulted to the API version of the enclosing object. - -The `fieldRef` is evaluated and the resulting value is used as the value for -the environment variable. This allows users to publish their pod's name in any -environment variable they want. - -The `resourceFieldRef` is a structure that has a `containerName` field, a `resource` -field, and a `divisor` field. The `containerName` is the name of a container, -whose resource (cpu or memory) information is to be exposed. The `containerName` is -optional for environment variables and defaults to the current container. The -`resource` field is an expression designating a resource in a container, and the `divisor` -field specifies an output format of the resource being exposed. If the `divisor` -is not specified, it defaults to "1" for cpu and memory. The table shows possible -values for cpu and memory resources for `resource` and `divisor` settings: - - -| Setting | Cpu | Memory | -| ------------- |-------------| -----| -| resource | limits.cpu, requests.cpu| limits.memory, requests.memory| -| divisor | 1(cores), 1m(millicores) | 1(bytes), 1k(kilobytes), 1M(megabytes), 1G(gigabytes), 1T(terabytes), 1P(petabytes), 1E(exabytes), 1Ki(kibibyte), 1Mi(mebibyte), 1Gi(gibibyte), 1Ti(tebibyte), 1Pi(pebibyte), 1Ei(exbibyte)| - - -### Example - -This is an example of a pod that consumes its name and namespace via the -downward API: - -{% include code.html language="yaml" file="dapi-pod.yaml" ghlink="/docs/user-guide/downward-api/dapi-pod.yaml" %} - -This is an example of a pod that consumes its container's resources via the downward API: - -{% include code.html language="yaml" file="dapi-container-resources.yaml" ghlink="/docs/user-guide/downward-api/dapi-container-resources.yaml" %} - -## Downward API volume - -Using a similar syntax it's possible to expose pod information to containers using plain text files. -Downward API are dumped to a mounted volume. This is achieved using a `downwardAPI` -volume type and the different items represent the files to be created. `fieldPath` references the field to be exposed. -For exposing a container's resources limits and requests, `containerName` must be specified with `resourceFieldRef`. - -Downward API volume permits to store more complex data like [`metadata.labels`](/docs/user-guide/labels) and [`metadata.annotations`](/docs/user-guide/annotations). Currently key/value pair set fields are saved using `key="value"` format: - -```conf -key1="value1" -key2="value2" -``` - -In future, it will be possible to specify an output format option. - -Downward API volumes can expose: - -* The node's name -* The pod's name -* The pod's namespace -* The pod's labels -* The pod's annotations -* The pod's service account name -* A container's cpu limit -* A container's cpu request -* A container's memory limit -* A container's memory request - -The downward API volume refreshes its data in step with the kubelet refresh loop. When labels will be modifiable on the fly without respawning the pod containers will be able to detect changes through mechanisms such as [inotify](https://en.wikipedia.org/wiki/Inotify). - -In future, it will be possible to specify a specific annotation or label. - -#### Projecting keys to specific paths and file permissions - -You can project keys to specific paths and specific permissions on a per-file -basis. The [Secrets](/docs/user-guide/secrets/) user guide explains the syntax. - -### Example - -This is an example of a pod that consumes its labels and annotations via the downward API volume, labels and annotations are dumped in `/etc/labels` and in `/etc/annotations`, respectively: - -{% include code.html language="yaml" file="volume/dapi-volume.yaml" ghlink="/docs/user-guide/downward-api/volume/dapi-volume.yaml" %} - -This is an example of a pod that consumes its container's resources via the downward API volume. - -{% include code.html language="yaml" file="volume/dapi-volume-resources.yaml" ghlink="/docs/user-guide/downward-api/volume/dapi-volume-resources.yaml" %} - -For a more thorough example, see -[environment variables](/docs/user-guide/environment-guide/). - -## Default values for container resource limits - -If cpu and memory limits are not specified for a container, the downward API will default to the node allocatable value for cpu and memory. +[Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) diff --git a/docs/user-guide/downward-api/volume/index.md b/docs/user-guide/downward-api/volume/index.md index 882b2da1e9..6f359cc797 100644 --- a/docs/user-guide/downward-api/volume/index.md +++ b/docs/user-guide/downward-api/volume/index.md @@ -2,118 +2,6 @@ title: Downward API Volumes --- -Following this example, you will create a pod with a downward API volume. -A downward API volume is a k8s volume plugin with the ability to save some pod information in a plain text file. The pod information can be for example some [metadata](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#metadata) or a container's [resources](/docs/user-guide/compute-resources). +{% include user-guide-content-moved.md %} -Supported metadata fields: - -1. `metadata.annotations` -2. `metadata.namespace` -3. `metadata.name` -4. `metadata.labels` - -Supported container's resources: - -1. `limits.cpu` -2. `limits.memory` -3. `requests.cpu` -4. `requests.memory` - -### Step Zero: Prerequisites - -This example assumes you have a Kubernetes cluster installed and running, and the `kubectl` command line tool somewhere in your path. Please see the [gettingstarted](/docs/getting-started-guides/) for installation instructions for your platform. - -### Step One: Create the pod - -Use the [dapi-volume.yaml](/docs/user-guide/downward-api/volume/dapi-volume.yaml) file to create a Pod with a downward API volume which stores pod labels and pod annotations to `/etc/labels` and `/etc/annotations` respectively. - -```shell -$ kubectl create -f docs/user-guide/downward-api/volume/dapi-volume.yaml -``` - -### Step Two: Examine pod/container output - -The pod displays (every 5 seconds) the content of the dump files which can be executed via the usual `kubectl log` command - -```shell -$ kubectl logs kubernetes-downwardapi-volume-example -cluster="test-cluster1" -rack="rack-22" -zone="us-est-coast" -build="two" -builder="john-doe" -kubernetes.io/config.seen="2015-08-24T13:47:23.432459138Z" -kubernetes.io/config.source="api" -``` - -### Internals - -In pod's `/etc` directory one may find the file created by the plugin (system files elided): - -```shell -$ kubectl exec kubernetes-downwardapi-volume-example -i -t -- sh -/ # ls -laR /etc -/etc: -total 4 -drwxrwxrwt 3 0 0 120 Jun 1 19:55 . -drwxr-xr-x 17 0 0 4096 Jun 1 19:55 .. -drwxr-xr-x 2 0 0 80 Jun 1 19:55 ..6986_01_06_15_55_10.473583074 -lrwxrwxrwx 1 0 0 31 Jun 1 19:55 ..data -> ..6986_01_06_15_55_10.473583074 -lrwxrwxrwx 1 0 0 18 Jun 1 19:55 annotations -> ..data/annotations -lrwxrwxrwx 1 0 0 13 Jun 1 19:55 labels -> ..data/labels - -/etc/..6986_01_06_15_55_10.473583074: -total 8 -drwxr-xr-x 2 0 0 80 Jun 1 19:55 . -drwxrwxrwt 3 0 0 120 Jun 1 19:55 .. --rw-r--r-- 1 0 0 129 Jun 1 19:55 annotations --rw-r--r-- 1 0 0 59 Jun 1 19:55 labels -/ # -``` - -The file `labels` is stored in a temporary directory (`..6986_01_06_15_55_10.473583074` in the example above) which is symlinked to by `..data`. Symlinks for annotations and labels in `/etc` point to files containing the actual metadata through the `..data` indirection.  This structure allows for dynamic atomic refresh of the metadata: updates are written to a new temporary directory, and the `..data` symlink is updated atomically using `rename(2)`. - -## Example of downward API volume with container resources - -Use the `docs/user-guide/downward-api/volume/dapi-volume-resources.yaml` file to create a Pod with a downward API volume which stores its container's limits and requests in /etc. - -```shell -$ kubectl create -f docs/user-guide/downward-api/volume/dapi-volume-resources.yaml -``` - -### Examine pod/container output - -In pod's `/etc` directory one may find the files created by the plugin: - -```shell -$ kubectl exec kubernetes-downwardapi-volume-example -i -t -- sh -/ # ls -alR /etc -/etc: -total 4 -drwxrwxrwt 3 0 0 160 Jun 1 19:47 . -drwxr-xr-x 17 0 0 4096 Jun 1 19:48 .. -drwxr-xr-x 2 0 0 120 Jun 1 19:47 ..6986_01_06_15_47_23.076909525 -lrwxrwxrwx 1 0 0 31 Jun 1 19:47 ..data -> ..6986_01_06_15_47_23.076909525 -lrwxrwxrwx 1 0 0 16 Jun 1 19:47 cpu_limit -> ..data/cpu_limit -lrwxrwxrwx 1 0 0 18 Jun 1 19:47 cpu_request -> ..data/cpu_request -lrwxrwxrwx 1 0 0 16 Jun 1 19:47 mem_limit -> ..data/mem_limit -lrwxrwxrwx 1 0 0 18 Jun 1 19:47 mem_request -> ..data/mem_request - -/etc/..6986_01_06_15_47_23.076909525: -total 16 -drwxr-xr-x 2 0 0 120 Jun 1 19:47 . -drwxrwxrwt 3 0 0 160 Jun 1 19:47 .. --rw-r--r-- 1 0 0 1 Jun 1 19:47 cpu_limit --rw-r--r-- 1 0 0 1 Jun 1 19:47 cpu_request --rw-r--r-- 1 0 0 8 Jun 1 19:47 mem_limit --rw-r--r-- 1 0 0 8 Jun 1 19:47 mem_request - -/ # cat /etc/cpu_limit -1 -/ # cat /etc/mem_limit -67108864 -/ # cat /etc/cpu_request -1 -/ # cat /etc/mem_request -33554432 -``` +[Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) From d0bdf37c1f84245e8e8e997cfecce84dba2a5f82 Mon Sep 17 00:00:00 2001 From: Guillaume Breton Date: Thu, 16 Feb 2017 18:07:10 +0000 Subject: [PATCH 104/407] Fix typos in running zookeeper article --- docs/tutorials/stateful-application/zookeeper.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/stateful-application/zookeeper.md b/docs/tutorials/stateful-application/zookeeper.md index ba2bb219d6..e31d8e280b 100644 --- a/docs/tutorials/stateful-application/zookeeper.md +++ b/docs/tutorials/stateful-application/zookeeper.md @@ -580,7 +580,7 @@ env: key: purge.interval ``` -The entry point of the container invokes a bash script, `zkConfig.sh`, prior to +The entry point of the container invokes a bash script, `zkGenConfig.sh`, prior to launching the ZooKeeper server process. This bash script generates the ZooKeeper configuration files from the supplied environment variables. @@ -653,7 +653,7 @@ ZK_LOG_DIR=/var/log/zookeeper ### Configuring Logging -One of the files generated by the `zkConfigGen.sh` script controls ZooKeeper's logging. +One of the files generated by the `zkGenConfig.sh` script controls ZooKeeper's logging. ZooKeeper uses [Log4j](http://logging.apache.org/log4j/2.x/), and, by default, it uses a time and size based rolling file appender for its logging configuration. Get the logging configuration from one of Pods in the `zk` StatefulSet. From 041100bd2a82fc83e496d75beed881b859779e79 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Fri, 17 Feb 2017 16:54:23 +0800 Subject: [PATCH 105/407] fix typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1、column PORTS is missed 2、fix command `kubectl get pods`output --- docs/admin/static-pods.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index cacf6e2010..610da03da7 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -72,8 +72,8 @@ When kubelet starts, it automatically starts all pods defined in directory speci ```shell [joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES -f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c ``` If we look at our Kubernetes API server (running on host `my-master`), we see that a new mirror-pod was created there too: @@ -81,9 +81,9 @@ If we look at our Kubernetes API server (running on host `my-master`), we see th ```shell [joe@host ~] $ ssh my-master [joe@my-master ~] $ kubectl get pods -POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE -static-web-my-node1 172.17.0.3 my-node1/192.168.100.71 role=myrole Running 11 minutes - web nginx Running 11 minutes +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 2m + ``` Labels from the static pod are propagated into the mirror-pod and can be used as usual for filtering. @@ -94,8 +94,9 @@ Notice we cannot delete the pod with the API server (e.g. via [`kubectl`](/docs/ [joe@my-master ~] $ kubectl delete pod static-web-my-node1 pods/static-web-my-node1 [joe@my-master ~] $ kubectl get pods -POD IP CONTAINER(S) IMAGE(S) HOST ... -static-web-my-node1 172.17.0.3 my-node1/192.168.100.71 ... +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 12s + ``` Back to our `my-node1` host, we can try to stop the container manually and see, that kubelet automatically restarts it in a while: From 77b4ffa69649769c4a15eb06fc736c6740c71d97 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Wed, 15 Feb 2017 16:33:38 +0800 Subject: [PATCH 106/407] modify one word examples to example --- docs/user-guide/walkthrough/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/walkthrough/index.md b/docs/user-guide/walkthrough/index.md index d7872b2260..e0400ec7ed 100644 --- a/docs/user-guide/walkthrough/index.md +++ b/docs/user-guide/walkthrough/index.md @@ -9,7 +9,7 @@ title: Kubernetes 101 For Kubernetes 101, we will cover kubectl, pods, volumes, and multiple containers -In order for the kubectl usage examples to work, make sure you have an examples directory locally, either from [a release](https://github.com/kubernetes/kubernetes/releases) or [the source](https://github.com/kubernetes/kubernetes). +In order for the kubectl usage examples to work, make sure you have an example directory locally, either from [a release](https://github.com/kubernetes/kubernetes/releases) or [the source](https://github.com/kubernetes/kubernetes). * TOC {:toc} From 265a2fc996d3ba79015bd570863a2e6f4ba99722 Mon Sep 17 00:00:00 2001 From: Michael Fraenkel Date: Fri, 17 Feb 2017 08:34:06 -0500 Subject: [PATCH 107/407] Revert "mend" This reverts commit 324451c3378b4ba01d27e3db6117f86d0517a598. --- docs/user-guide/configmap/index.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 40b3fa181b..36d3e89394 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -317,7 +317,10 @@ spec: restartPolicy: Never ``` -When this pod is run, the output will be empty. +When this pod is run, its output will include the lines: + +```shell +``` ### Use-Case: Set command-line arguments with ConfigMap From 4635787521f89c4396c6a82d08d8a92ac89f7a77 Mon Sep 17 00:00:00 2001 From: Michael Fraenkel Date: Fri, 17 Feb 2017 08:34:15 -0500 Subject: [PATCH 108/407] Revert "Document new optional support for ConfigMap and Secret" This reverts commit 7abcc6c8540a7b0a4f6712e77e0f5cb134271bed. --- docs/user-guide/configmap/index.md | 70 ++-------------------------- docs/user-guide/secrets/index.md | 74 +++--------------------------- 2 files changed, 10 insertions(+), 134 deletions(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 36d3e89394..1355e599f2 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -291,37 +291,6 @@ SPECIAL_LEVEL_KEY=very SPECIAL_TYPE_KEY=charm ``` -#### Optional ConfigMap in environment variables - -There might be situations where environment variables are not -always required. These environment variables can be marked as optional in a -pod like so: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: dapi-test-pod -spec: - containers: - - name: test-container - image: gcr.io/google_containers/busybox - command: [ "/bin/sh", "-c", "env" ] - env: - - name: SPECIAL_LEVEL_KEY - valueFrom: - configMapKeyRef: - name: a-config - key: akey - optional: true - restartPolicy: Never -``` - -When this pod is run, its output will include the lines: - -```shell -``` - ### Use-Case: Set command-line arguments with ConfigMap ConfigMaps can also be used to set the value of the command or arguments in a container. This is @@ -453,38 +422,6 @@ very You can project keys to specific paths and specific permissions on a per-file basis. The [Secrets](/docs/user-guide/secrets/) user guide explains the syntax. -#### Optional ConfigMap via volume plugin - -Volumes and files provided by a ConfigMap can be also be marked as optional. -The ConfigMap or the key specified does not have to exist. The mount path for -such items will always be created. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: dapi-test-pod -spec: - containers: - - name: test-container - image: gcr.io/google_containers/busybox - command: [ "/bin/sh", "-c", "ls /etc/config" ] - volumeMounts: - - name: config-volume - mountPath: /etc/config - volumes: - - name: config-volume - configMap: - name: no-config - optional: true - restartPolicy: Never -``` - -When this pod is run, the output will be: - -```shell -``` - ## Real World Example: Configuring Redis Let's take a look at a real-world example: configuring redis using ConfigMap. Say we want to inject @@ -580,10 +517,9 @@ $ kubectl exec -it redis redis-cli ## Restrictions -ConfigMaps must be created before they are consumed in pods unless they are -marked as optional. Controllers may be written to tolerate missing -configuration data; consult individual components configured via ConfigMap on -a case-by-case basis. +ConfigMaps must be created before they are consumed in pods. Controllers may be written to tolerate +missing configuration data; consult individual components configured via ConfigMap on a case-by-case +basis. ConfigMaps reside in a namespace. They can only be referenced by pods in the same namespace. diff --git a/docs/user-guide/secrets/index.md b/docs/user-guide/secrets/index.md index 9723e00aee..29e9044fde 100644 --- a/docs/user-guide/secrets/index.md +++ b/docs/user-guide/secrets/index.md @@ -375,41 +375,6 @@ However, it is using its local ttl-based cache for getting the current value of As a result, the total delay from the moment when the secret is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of secrets cache in kubelet. -#### Optional Secrets as Files from a Pod - -Volumes and files provided by a Secret can be also be marked as optional. -The Secret or the key within a Secret does not have to exist. The mount path for -such items will always be created. - -```json -{ - "apiVersion": "v1", - "kind": "Pod", - "metadata": { - "name": "mypod", - "namespace": "myns" - }, - "spec": { - "containers": [{ - "name": "mypod", - "image": "redis", - "volumeMounts": [{ - "name": "foo", - "mountPath": "/etc/foo" - }] - }], - "volumes": [{ - "name": "foo", - "secret": { - "secretName": "mysecret", - "defaultMode": 256, - "optional": true - } - }] - } -} -``` - #### Using Secrets as Environment Variables To use a secret in an environment variable in a pod: @@ -456,30 +421,6 @@ $ echo $SECRET_PASSWORD 1f2d1e2e67df ``` -#### Optional Secrets from Environment Variables - -You may not want to require all your secrets to exist. They can be marked as -optional as shown in the pod: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: optional-secret-env-pod -spec: - containers: - - name: mycontainer - image: redis - env: - - name: OPTIONAL_SECRET - valueFrom: - secretKeyRef: - name: mysecret - key: username - optional: true - restartPolicy: Never -``` - #### Using imagePullSecrets An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry @@ -511,8 +452,7 @@ can be automatically attached to pods based on their service account. Secret volume sources are validated to ensure that the specified object reference actually points to an object of type `Secret`. Therefore, a secret -needs to be created before any pods that depend on it, unless it is marked as -optional. +needs to be created before any pods that depend on it. Secret API objects reside in a namespace. They can only be referenced by pods in that same namespace. @@ -532,12 +472,12 @@ not common ways to create pods.) When a pod is created via the API, there is no check whether a referenced secret exists. Once a pod is scheduled, the kubelet will try to fetch the -secret value. If a required secret cannot be fetched because it does not -exist or because of a temporary lack of connection to the API server, the -kubelet will periodically retry. It will report an event about the pod -explaining the reason it is not started yet. Once the secret is fetched, the -kubelet will create and mount a volume containing it. None of the pod's -containers will start until all the pod's volumes are mounted. +secret value. If the secret cannot be fetched because it does not exist or +because of a temporary lack of connection to the API server, kubelet will +periodically retry. It will report an event about the pod explaining the +reason it is not started yet. Once the secret is fetched, the kubelet will +create and mount a volume containing it. None of the pod's containers will +start until all the pod's volumes are mounted. ## Use cases From b01dd765a9c1ea8169c264dc70ed3fee0f9dc73f Mon Sep 17 00:00:00 2001 From: Harry Zhang Date: Sat, 18 Feb 2017 03:06:01 +0800 Subject: [PATCH 109/407] Add kubelet eviction in GC doc (#2484) --- docs/admin/garbage-collection.md | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index a960172426..9717a41dd8 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -42,15 +42,34 @@ to free. Default is 80%. We also allow users to customize garbage collection policy through the following kubelet flags: 1. `minimum-container-ttl-duration`, minimum age for a finished container before it is -garbage collected. Default is 1 minute. +garbage collected. Default is 0 minute, which means any finished container will be garbaged collected. 2. `maximum-dead-containers-per-container`, maximum number of old instances to retain -per container. Default is 2. +per container. Default is 1. 3. `maximum-dead-containers`, maximum number of old instances of containers to retain globally. -Default is 100. +Default is -1, which means there is no global limit. Containers can potentially be garbage collected before their usefulness has expired. These containers can contain logs and other data that can be useful for troubleshooting. A sufficiently large value for -`maximum-dead-containers-per-container` is highly recommended to allow at least 2 dead containers to be +`maximum-dead-containers-per-container` is highly recommended to allow at least 1 dead container to be retained per expected container. A higher value for `maximum-dead-containers` is also recommended for a similar reason. See [this issue](https://github.com/kubernetes/kubernetes/issues/13287) for more details. + + +### Deprecation + +Some kubelet Garbage Collection features in this doc will be replaced by kubelet eviction in the future. + +Including: + +| Existing Flag | New Flag | Rationale | +| ------------- | -------- | --------- | +| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` | existing eviction signals can trigger image garbage collection | +| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | eviction reclaims achieve the same behavior | +| `--maximum-dead-containers` | | deprecated once old logs are stored outside of container's context | +| `--maximum-dead-containers-per-container` | | deprecated once old logs are stored outside of container's context | +| `--minimum-container-ttl-duration` | | deprecated once old logs are stored outside of container's context | +| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources | +| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources | + +See [kubelet eviction design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-eviction.md) for more details. From 180091299beb46a23234e9745a41e857f1e1af54 Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Mon, 20 Feb 2017 10:24:13 +0800 Subject: [PATCH 110/407] dedent yaml code blocks otherwise extra redundant ````yaml` characters will be displayed because github code fencing doesn't need any indentation --- docs/admin/kubeadm.md | 80 +++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index edf9c87a7d..a43beec497 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -180,49 +180,49 @@ available as configuration file options. ### Sample Master Configuration - ```yaml - apiVersion: kubeadm.k8s.io/v1alpha1 - kind: MasterConfiguration - api: - advertiseAddresses: - - - - - bindPort: - externalDNSNames: - - - - - authorizationMode: - cloudProvider: - discovery: - bindPort: - etcd: - endpoints: - - - - - caFile: - certFile: - keyFile: - kubernetesVersion: - networking: - dnsDomain: - serviceSubnet: - podSubnet: - secrets: - givenToken: - ``` +```yaml +apiVersion: kubeadm.k8s.io/v1alpha1 +kind: MasterConfiguration +api: + advertiseAddresses: + - + - + bindPort: + externalDNSNames: + - + - +authorizationMode: +cloudProvider: +discovery: + bindPort: +etcd: + endpoints: + - + - + caFile: + certFile: + keyFile: +kubernetesVersion: +networking: + dnsDomain: + serviceSubnet: + podSubnet: +secrets: + givenToken: +``` ### Sample Node Configuration - ```yaml - apiVersion: kubeadm.k8s.io/v1alpha1 - kind: NodeConfiguration - apiPort: - discoveryPort: - masterAddresses: - - - secrets: - givenToken: - ``` +```yaml +apiVersion: kubeadm.k8s.io/v1alpha1 +kind: NodeConfiguration +apiPort: +discoveryPort: +masterAddresses: +- +secrets: + givenToken: +``` ## Automating kubeadm From abf9560874889fe2b953cf2bda01b833e1280238 Mon Sep 17 00:00:00 2001 From: xuxinkun Date: Fri, 17 Feb 2017 18:19:33 +0800 Subject: [PATCH 111/407] Add more options for self-registration. --- docs/admin/node.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/admin/node.md b/docs/admin/node.md index 0923169256..f91f155919 100644 --- a/docs/admin/node.md +++ b/docs/admin/node.md @@ -176,6 +176,9 @@ For self-registration, the kubelet is started with the following options: - `--kubeconfig=` - Path to credentials to authenticate itself to the apiserver. - `--cloud-provider=` - How to talk to a cloud provider to read metadata about itself. - `--register-node` - Automatically register with the API server. + - `--node-ip` IP address of the node. + - `--node-labels` - Labels to add when registering the node in the cluster. + - `--node-status-update-frequency` - Specifies how often kubelet posts node status to master. Currently, any kubelet is authorized to create/modify any node resource, but in practice it only creates/modifies its own. (In the future, we plan to only allow a kubelet to modify its own node resource.) From 8dd5d7b8adb0494d24a16307853ccefac97122d6 Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Thu, 16 Feb 2017 16:44:45 +0800 Subject: [PATCH 112/407] Parallel Jobs with a *work queue*: do not specify `.spec.completions`, default to `.spec.Parallelism`, not default to 1 --- docs/user-guide/jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index d64ae9d730..438b63dff7 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -120,7 +120,7 @@ There are three main types of jobs: - the job is complete when there is one successful pod for each value in the range 1 to `.spec.completions`. - **not implemented yet:** each pod passed a different index in the range 1 to `.spec.completions`. 1. Parallel Jobs with a *work queue*: - - do not specify `.spec.completions` + - do not specify `.spec.completions`, default to `.spec.Parallelism` - the pods must coordinate with themselves or an external service to determine what each should work on - each pod is independently capable of determining whether or not all its peers are done, thus the entire Job is done. - when _any_ pod terminates with success, no new pods are created. From ab7d2c1b3c703066865192cdf0a4ca06a9119b74 Mon Sep 17 00:00:00 2001 From: Olegs Jeremejevs Date: Tue, 21 Feb 2017 17:48:51 +0200 Subject: [PATCH 113/407] Fix typo in concepts --- docs/concepts/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 93e4533caf..8e738e809e 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -17,7 +17,7 @@ Once you've set your desired state, the *Kubernetes Control Plane* works to make ## Kubernetes Objects -Kubernetes contains a number of abstractions that represent your the state of your system: deployed containerized applications and workloads, their associated network and disk resources, and other information about what your cluster is doing. These abstractions are represented by objects in the Kubernetes API; see the [Kubernetes Objects overview](/docs/concepts/abstractions/overview/) for more details. +Kubernetes contains a number of abstractions that represent the state of your system: deployed containerized applications and workloads, their associated network and disk resources, and other information about what your cluster is doing. These abstractions are represented by objects in the Kubernetes API; see the [Kubernetes Objects overview](/docs/concepts/abstractions/overview/) for more details. The basic Kubernetes objects include: From 03b53f2aa90cd2f7a574f304a39a9e823b0998a4 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 21 Feb 2017 15:23:24 +0800 Subject: [PATCH 114/407] fix typo --- .../configure-pod-container/pull-image-private-registry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/pull-image-private-registry.md b/docs/tasks/configure-pod-container/pull-image-private-registry.md index e9b640873a..f54d881178 100644 --- a/docs/tasks/configure-pod-container/pull-image-private-registry.md +++ b/docs/tasks/configure-pod-container/pull-image-private-registry.md @@ -80,7 +80,7 @@ Copy the base64 representation of the secret data into a file named `secret64`. **Important**: Make sure there are no line breaks in your `secret64` file. -To understand what is in the `dockercfg` field, convert the secret data to a +To understand what is in the `.dockercfg` field, convert the secret data to a readable format: base64 -d secret64 From 27039ac3b6859fe1786b12cf00cf8b3746ee3907 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 21 Feb 2017 11:52:18 +0800 Subject: [PATCH 115/407] add an example --- docs/concepts/configuration/container-command-args.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/concepts/configuration/container-command-args.md b/docs/concepts/configuration/container-command-args.md index 25d555f702..3f39baff57 100644 --- a/docs/concepts/configuration/container-command-args.md +++ b/docs/concepts/configuration/container-command-args.md @@ -66,6 +66,7 @@ Here are some examples: | `[/ep-1]` | `[foo bar]` | <not set> | <not set> | `[ep-1 foo bar]` | | `[/ep-1]` | `[foo bar]` | `[/ep-2]` | <not set> | `[ep-2]` | | `[/ep-1]` | `[foo bar]` | <not set> | `[zoo boo]` | `[ep-1 zoo boo]` | +| `[/ep-1]` | `[foo bar]` | `[/ep-2]` | `[zoo boo]` | `[ep-2 zoo boo]` | {% endcapture %} From d4c2da211e8b3aa9cd477f5b03813cc6b55a0a94 Mon Sep 17 00:00:00 2001 From: Ben Mathews Date: Fri, 17 Feb 2017 14:39:48 -0700 Subject: [PATCH 116/407] Remove reference to port-forward to services port forwarding to services isn't supported. --- docs/user-guide/kubectl-cheatsheet.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/user-guide/kubectl-cheatsheet.md b/docs/user-guide/kubectl-cheatsheet.md index 7fc0142f04..6b67cba4a0 100644 --- a/docs/user-guide/kubectl-cheatsheet.md +++ b/docs/user-guide/kubectl-cheatsheet.md @@ -203,7 +203,6 @@ $ kubectl logs -f my-pod -c my-container # stream pod container log $ kubectl run -i --tty busybox --image=busybox -- sh # Run pod as interactive shell $ kubectl attach my-pod -i # Attach to Running Container $ kubectl port-forward my-pod 5000:6000 # Forward port 6000 of Pod to your to 5000 on your local machine -$ kubectl port-forward my-svc 6000 # Forward port to service $ kubectl exec my-pod -- ls / # Run command in existing pod (1 container case) $ kubectl exec my-pod -c my-container -- ls / # Run command in existing pod (multi-container case) $ kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers From 58f20b10fa575145f8c10b5e8bbf1656719db8ce Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Mon, 20 Feb 2017 22:37:54 +0800 Subject: [PATCH 117/407] Fixed typos in persistent volume index.md --- docs/user-guide/persistent-volumes/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index 4508a1a25f..e2d2661d13 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -421,7 +421,7 @@ parameters: * `restauthenabled` : Gluster REST service authentication boolean that enables authentication to the REST server. If this value is 'true', `restuser` and `restuserkey` or `secretNamespace` + `secretName` have to be filled. This option is deprecated, authentication is enabled when any of `restuser`, `restuserkey`, `secretName` or `secretNamespace` is specified. * `restuser` : Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool. * `restuserkey` : Gluster REST service/Heketi user's password which will be used for authentication to the REST server. This parameter is deprecated in favor of `secretNamespace` + `secretName`. -* `secretNamespace` + `secretName` : Identification of Secret instance that containes user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must have type "kubernetes.io/glusterfs", e.g. created in this way: +* `secretNamespace` + `secretName` : Identification of Secret instance that contains user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must have type "kubernetes.io/glusterfs", e.g. created in this way: ``` $ kubectl create secret generic heketi-secret --type="kubernetes.io/glusterfs" --from-literal=key='opensesame' --namespace=default ``` @@ -507,7 +507,7 @@ parameters: * `quobyteAPIServer`: API Server of Quobyte in the format `http(s)://api-server:7860` * `registry`: Quobyte registry to use to mount the volume. You can specify the registry as ``:`` pair or if you want to specify multiple registries you just have to put a comma between them e.q. ``:,:,:``. The host can be an IP address or if you have a working DNS you can also provide the DNS names. * `adminSecretNamespace`: The namespace for `adminSecretName`. Default is "default". -* `adminSecretName`: secret that holds information about the Quobyte user and the password to authenticate agains the API server. The provided secret must have type "kubernetes.io/quobyte", e.g. created in this way: +* `adminSecretName`: secret that holds information about the Quobyte user and the password to authenticate against the API server. The provided secret must have type "kubernetes.io/quobyte", e.g. created in this way: ``` $ kubectl create secret generic quobyte-admin-secret --type="kubernetes.io/quobyte" --from-literal=key='opensesame' --namespace=kube-system ``` From 74d8327f6dfbef515b1e2246e7c48fdf6d0538cb Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Mon, 20 Feb 2017 21:29:07 +0800 Subject: [PATCH 118/407] Highlight some keyworks in garbage-collection.md. --- docs/admin/garbage-collection.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 9717a41dd8..8164fbb7d9 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -24,9 +24,9 @@ threshold has been met. ### Container Collection The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers any single -pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting 'MinAge' to zero and setting 'MaxPerPodContainer' and 'MaxContainers' respectively to less than zero. +pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting `MinAge` to zero and setting `MaxPerPodContainer` and `MaxContainers` respectively to less than zero. -Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. 'MaxPerPodContainer' and 'MaxContainer' may potentially conflict with each other in situations where retaining the maximum number of containers per pod ('MaxPerPodContainer') would go outside the allowable range of global dead containers ('MaxContainers'). 'MaxPerPodContainer' would be adjusted in this situation: A worst case scenario would be to downgrade 'MaxPerPodContainer' to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. +Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. `MaxPerPodContainer` and `MaxContainer` may potentially conflict with each other in situations where retaining the maximum number of containers per pod (`MaxPerPodContainer`) would go outside the allowable range of global dead containers (`MaxContainers`). `MaxPerPodContainer` would be adjusted in this situation: A worst case scenario would be to downgrade `MaxPerPodContainer` to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. Containers that are not managed by kubelet are not subject to container garbage collection. From 960bb03b30b98ee47718950840e45f126f836285 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Sat, 18 Feb 2017 15:14:37 +0800 Subject: [PATCH 119/407] replace argumment with argument replace argumment with argument --- docs/admin/network-plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/network-plugins.md b/docs/admin/network-plugins.md index d5e05ac7b1..f06d22fb87 100644 --- a/docs/admin/network-plugins.md +++ b/docs/admin/network-plugins.md @@ -49,7 +49,7 @@ The plugin requires a few things: * The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `network-plugin-dir` to supply additional search path. The first found match will take effect. * Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin -* Kubelet should also be run with the `--non-masquerade-cidr=` argumment to ensure traffic to IPs outside this range will use IP masquerade. +* Kubelet should also be run with the `--non-masquerade-cidr=` argument to ensure traffic to IPs outside this range will use IP masquerade. * The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=` controller-manager command-line options. ### Customizing the MTU (with kubenet) From 72c93f9b52e612ec75163e4896ddb4dc9f68de17 Mon Sep 17 00:00:00 2001 From: "Ricardo M. Augusto" Date: Tue, 21 Feb 2017 13:16:27 -0300 Subject: [PATCH 120/407] Updated result of command Actually according to the command in example: kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub The keys will be available with names ssh-publickey and ssh-privatekey. --- docs/user-guide/secrets/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/secrets/index.md b/docs/user-guide/secrets/index.md index 29e9044fde..8b7c7ae2fb 100644 --- a/docs/user-guide/secrets/index.md +++ b/docs/user-guide/secrets/index.md @@ -534,8 +534,8 @@ consumes it in a volume: When the container's command runs, the pieces of the key will be available in: ```shell -/etc/secret-volume/id-rsa.pub -/etc/secret-volume/id-rsa +/etc/secret-volume/ssh-publickey +/etc/secret-volume/ssh-privatekey ``` The container is then free to use the secret data to establish an ssh connection. From 60b50187389ab0d9a31075e5e4805e69ae4b6302 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 21 Feb 2017 16:08:27 +0800 Subject: [PATCH 121/407] replace petset with statefulsets --- docs/user-guide/kubectl-cheatsheet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl-cheatsheet.md b/docs/user-guide/kubectl-cheatsheet.md index 6b67cba4a0..56f6d71de4 100644 --- a/docs/user-guide/kubectl-cheatsheet.md +++ b/docs/user-guide/kubectl-cheatsheet.md @@ -243,7 +243,7 @@ Resource type | Abbreviated alias `namespaces` |`ns` `networkpolicies` | `nodes` |`no` -`petset` | +`statefulsets` | `persistentvolumeclaims` |`pvc` `persistentvolumes` |`pv` `pods` |`po` From e82a2e2a58575302ee757d3df81db52316031f0f Mon Sep 17 00:00:00 2001 From: Cao Shufeng Date: Mon, 20 Feb 2017 23:45:50 -0500 Subject: [PATCH 122/407] Remove details about where authenticator is implemented It not necessary to mention where the functions are implemented in document. Further more, the directories recorded here is out of date. Now they are in: staging/src/k8s.io/apiserver/plugin/pkg/authenticator/... --- docs/admin/authentication.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/admin/authentication.md b/docs/admin/authentication.md index 7c59c955ac..7a7d8f1bc3 100644 --- a/docs/admin/authentication.md +++ b/docs/admin/authentication.md @@ -85,9 +85,9 @@ See [APPENDIX](#appendix) for how to generate a client cert. The API server reads bearer tokens from a file when given the `--token-auth-file=SOMEFILE` option on the command line. Currently, tokens last indefinitely, and the token list cannot be changed without restarting API server. -The token file format is implemented in `plugin/pkg/auth/authenticator/token/tokenfile/...` -and is a csv file with a minimum of 3 columns: token, user name, user uid, followed by -optional group names. Note, if you have more than one group the column must be double quoted e.g. +The token file is a csv file with a minimum of 3 columns: token, user name, user uid, +followed by optional group names. Note, if you have more than one group the column must be +double quoted e.g. ```conf token,user,uid,"group1,group2,group3" @@ -115,9 +115,9 @@ and the password cannot be changed without restarting API server. Note that basi authentication is currently supported for convenience while we finish making the more secure modes described above easier to use. -The basic auth file format is implemented in `plugin/pkg/auth/authenticator/password/passwordfile/...` -and is a csv file with a minimum of 3 columns: password, user name, user id, followed by -optional group names. Note, if you have more than one group the column must be double quoted e.g. +The basic auth file is a csv file with a minimum of 3 columns: password, +user name, user id, followed by optional group names. Note, if you have more than +one group the column must be double quoted e.g. ```conf password,user,uid,"group1,group2,group3" From 91356cfbd23f70faf65094dd203f524e258cac11 Mon Sep 17 00:00:00 2001 From: Alexander Kanevskiy Date: Tue, 21 Feb 2017 20:41:52 +0200 Subject: [PATCH 123/407] Updated description for kubeadm init --kubernetes-version --- docs/admin/kubeadm.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index a43beec497..32bd70812f 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -143,10 +143,11 @@ By default, `kubeadm init` automatically generates the token used to initialise each new node. If you would like to manually specify this token, you can use the `--token` flag. The token must be of the format `<6 character string>.<16 character string>`. -- `--use-kubernetes-version` (default 'v1.5.1') the kubernetes version to initialise +- `--kubernetes-version` (default 'latest') the kubernetes version to initialise `kubeadm` was originally built for Kubernetes version **v1.4.0**, older versions are not -supported. With this flag you can try any future version, e.g. **v1.6.0-beta.1** +supported. The current version of `kubeadm` requires at least **v1.6.0-alpha.3** due to RBAC being enabled by default. +With this flag you can try any future version, e.g. **v1.6.0-beta.1** whenever it comes out (check [releases page](https://github.com/kubernetes/kubernetes/releases) for a full list of available versions). From bd7607a066b2472eb70021f9fbf1d6a1e522fd77 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 21 Feb 2017 12:35:15 -0800 Subject: [PATCH 124/407] Revert "Updated description for kubeadm init --kubernetes-version" --- docs/admin/kubeadm.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 32bd70812f..a43beec497 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -143,11 +143,10 @@ By default, `kubeadm init` automatically generates the token used to initialise each new node. If you would like to manually specify this token, you can use the `--token` flag. The token must be of the format `<6 character string>.<16 character string>`. -- `--kubernetes-version` (default 'latest') the kubernetes version to initialise +- `--use-kubernetes-version` (default 'v1.5.1') the kubernetes version to initialise `kubeadm` was originally built for Kubernetes version **v1.4.0**, older versions are not -supported. The current version of `kubeadm` requires at least **v1.6.0-alpha.3** due to RBAC being enabled by default. -With this flag you can try any future version, e.g. **v1.6.0-beta.1** +supported. With this flag you can try any future version, e.g. **v1.6.0-beta.1** whenever it comes out (check [releases page](https://github.com/kubernetes/kubernetes/releases) for a full list of available versions). From f1ecafcdb1b251e7416f176d763ee38c95a6b36b Mon Sep 17 00:00:00 2001 From: divyenpatel Date: Fri, 17 Feb 2017 14:58:30 -0800 Subject: [PATCH 125/407] updated vsphere getting started guide addressed review comments addressed chenopis's review comments --- docs/getting-started-guides/vsphere.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/vsphere.md b/docs/getting-started-guides/vsphere.md index 61ea733bf3..a22d38609f 100644 --- a/docs/getting-started-guides/vsphere.md +++ b/docs/getting-started-guides/vsphere.md @@ -79,8 +79,7 @@ Sample Config: #### Known issues -* [Volumes are not removed from a VM configuration if the VM is down](https://github.com/kubernetes/kubernetes/issues/33061). The workaround is to manually remove the disk from VM settings before powering it up. -* [FS groups are not supported in 1.4.7](https://github.com/kubernetes/kubernetes/issues/34039) - This issue is fixed in 1.4.8 +* [Unable to execute command on pod container using kubectl exec](https://github.com/kubernetes/kubernetes-anywhere/issues/337) ### Kube-up (Deprecated) @@ -216,7 +215,7 @@ going on (find yourself authorized with your SSH key, or use the password IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level -------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ---------------------------- -Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu))([@luomiao](https://github.com/luomiao)) +Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu)), ([@luomiao](https://github.com/luomiao)), ([@divyenpatel](https://github.com/divyenpatel)) For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart. From a7b66a78a38619f8962bbf3c425141a5919188e3 Mon Sep 17 00:00:00 2001 From: "Elijah C. Voigt" Date: Fri, 17 Feb 2017 17:12:13 -0800 Subject: [PATCH 126/407] concepts/abstractions: init containers example. --- docs/concepts/abstractions/init-containers.md | 94 ++++++++++++++++++- 1 file changed, 93 insertions(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/init-containers.md b/docs/concepts/abstractions/init-containers.md index 1bea00b3a2..89738eef56 100644 --- a/docs/concepts/abstractions/init-containers.md +++ b/docs/concepts/abstractions/init-containers.md @@ -95,6 +95,98 @@ Here are some ideas for how to use Init Containers: More detailed usage examples can be found in the [StatefulSets documentation](/docs/concepts/abstractions/controllers/statefulsets/) and the [Production Pods guide](/docs/user-guide/production-pods.md#handling-initialization). +### Init Containers in use + +The following yaml file outlines a simple Pod which has two Init Containers. +The first waits for `myservice` and the second waits for `mydb`. Once both +containers complete the Pod will begin. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: myapp-pod + labels: + app: myapp + annotations: + pod.beta.kubernetes.io/init-containers: '[ + { + "name": "init-myservice", + "image": "busybox", + "command": ["sh", "-c", "until nslookup myservice; do echo waiting for myservice; sleep 2; done;"] + }, + { + "name": "init-mydb", + "image": "busybox", + "command": ["sh", "-c", "until nslookup mydb; do echo waiting for mydb; sleep 2; done;"] + } + ]' +spec: + containers: + - name: myapp-container + image: busybox + command: ['sh', '-c', 'echo The app is running! && sleep 3600'] +``` + +This Pod can be started and debugged with the following commands: + +``` +$ kubectl create -f myapp.yaml +pod "myapp-pod" created +$ kubectl get -f myapp.yaml +NAME READY STATUS RESTARTS AGE +myapp-pod 0/1 Init:0/2 0 6m +$ kubectl describe -f myapp.yaml +i11:32 $ kubectl describe -f examples/init-container.yaml +Name: myapp-pod +Namespace: default +[...] +Labels: app=myapp +Status: Pending +[...] +Init Containers: + init-myservice: +[...] + State: Running +[...] + init-mydb: +[...] + State: Running +[...] +Containers: + myapp-container: +[...] + State: Waiting + Reason: PodInitializing + Ready: False +[...] +Events: + FirstSeen LastSeen Count From SubObjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 16s 16s 1 {default-scheduler } Normal Scheduled Successfully assigned myapp-pod to 172.17.4.201 + 16s 16s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulling pulling image "busybox" + 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulled Successfully pulled image "busybox" + 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container with docker id 5ced34a04634; Security:[seccomp=unconfined] + 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container with docker id 5ced34a04634 +$ kubectl logs myapp-pod -c init-myservice # Inspect the first init container +$ kubectl logs myapp-pod -c init-mydd # Inspect the second init container +``` + +Once we start the `mydb` and `myservice` Services we can see the Init Containers +complete and the `myapp-pod` is created: + +``` +$ kubectl create -f services.yaml +service "myservice" created +service "mydb" created +$ kubectl get -f myapp.yaml +NAME READY STATUS RESTARTS AGE +myapp-pod 1/1 Running 0 9m +``` + +This example is very simple but should provide some inspiration for you to +create your own Init Containers. + ## Detailed behavior During the startup of a Pod, the Init Containers are started in order, after the @@ -181,4 +273,4 @@ Kubelet and Apiserver versions; see the [release notes](https://github.com/kuber {% endcapture %} -{% include templates/concept.md %} \ No newline at end of file +{% include templates/concept.md %} From 13b7c5eb5cfe9c8408db641158a06cf0f5cc06e1 Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Tue, 21 Feb 2017 09:15:27 +0800 Subject: [PATCH 127/407] Update source-ip.md --- docs/tutorials/services/source-ip.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index e1eac87c4e..6366690e88 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -132,7 +132,7 @@ client_address=10.240.0.5 client_address=10.240.0.3 ``` -Note that these are not your IPs, they're cluster internal IPs. This is what happens: +Note that these are not right client IPs, they're cluster internal IPs. This is what happens: * Client sends packet to `node2:nodePort` * `node2` replaces the source IP address (SNAT) in the packet with its own IP address From 93c5edce64b3a4ac79f9e9da6113bfa7c0a2486a Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Wed, 22 Feb 2017 08:40:09 +0800 Subject: [PATCH 128/407] Update source-ip.md --- docs/tutorials/services/source-ip.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index 6366690e88..4f5f7d884a 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -132,7 +132,7 @@ client_address=10.240.0.5 client_address=10.240.0.3 ``` -Note that these are not right client IPs, they're cluster internal IPs. This is what happens: +Note that these are not the correct client IPs, they're cluster internal IPs. This is what happens: * Client sends packet to `node2:nodePort` * `node2` replaces the source IP address (SNAT) in the packet with its own IP address From 5095cf60df3381caa356c02a3fa12821163274a6 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 21 Feb 2017 19:12:47 -0800 Subject: [PATCH 129/407] Move Guide topic: Garbage Collection. (#2488) --- _data/concepts.yml | 1 + .../controllers/garbage-collection.md | 110 ++++++++++++++++++ .../abstractions/controllers/my-repset.yaml | 17 +++ docs/user-guide/garbage-collection.md | 33 +----- 4 files changed, 130 insertions(+), 31 deletions(-) create mode 100644 docs/concepts/abstractions/controllers/garbage-collection.md create mode 100644 docs/concepts/abstractions/controllers/my-repset.yaml diff --git a/_data/concepts.yml b/_data/concepts.yml index 432f4c86e1..56a556a801 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -20,6 +20,7 @@ toc: - title: Controllers section: - docs/concepts/abstractions/controllers/statefulsets.md + - docs/concepts/abstractions/controllers/garbage-collection.md - title: Object Metadata section: diff --git a/docs/concepts/abstractions/controllers/garbage-collection.md b/docs/concepts/abstractions/controllers/garbage-collection.md new file mode 100644 index 0000000000..889049c1ba --- /dev/null +++ b/docs/concepts/abstractions/controllers/garbage-collection.md @@ -0,0 +1,110 @@ +--- +title: Garbage Collection +--- + +{% capture overview %} + +The role of the Kubernetes garbage collector is to delete certain objects +that once had an owner, but no longer have an owner. + +**Note**: Garbage collection is a beta feature and is enabled by default in +Kubernetes version 1.4 and later. + +{% endcapture %} + + +{% capture body %} + +## Owners and dependents + +Some Kubernetes objects are owners of other objects. For example, a ReplicaSet +is the owner of a set of Pods. The owned objects are called *dependents* of the +owner object. Every dependent object has a `metadata.ownerReferences` field that +points to the owning object. + +Sometimes, Kubernetes sets the value of `ownerReference` automatically. For +example, when you create a ReplicaSet, Kubernetes automatically sets the +`ownerReference` field of each Pod in the ReplicaSet. You can also specify +relationships between owners and dependents by manually setting the +`ownerReference` field. + +Here's a configuration file for a ReplicaSet that has three Pods: + +{% include code.html language="yaml" file="my-repset.yaml" ghlink="/docs/concepts/abstractions/controllers/my-repset.yaml" %} + +If you create the ReplicaSet and then view the Pod metadata, you can see +OwnerReferences field: + +```shell +kubectl create -f http://k8s.io/docs/concepts/abstractions/controllers/my-repset.yaml +kubectl get pods --output=yaml +``` + +The output shows that the Pod owner is a ReplicaSet named my-repset: + +```shell +apiVersion: v1 +kind: Pod +metadata: + ... + ownerReferences: + - apiVersion: extensions/v1beta1 + controller: true + kind: ReplicaSet + name: my-repset + uid: d9607e19-f88f-11e6-a518-42010a800195 + ... +``` + +## Controlling whether the garbage collector deletes dependents + +When you delete object, you can specify whether the object's dependents +are deleted automatically. Deleting dependents automatically is called +*cascading deletion*. If you delete an object without deleting its +dependents automatically, the dependents are said to be *orphaned*. + +To delete dependent objects automatically, set the `orphanDependents` query +parameter to false in your request to delete the owner object. + +To orphan the dependents of an owner object, set the `orphanDependents` query +parameter to true in your request to delete the owner object. + +The default value for `orphanDependents` is true. So unless you specify +otherwise, dependent objects are orphaned. + +Here's an example that deletes dependents automatically: + +```shell +kubectl proxy --port=8080 +curl -X DELETE localhost:8080/apis/extensions/v1beta1/namespaces/default/replicasets/my-repset?orphanDependents=false +``` + +To delete dependents automatically using kubectl, set `--cascade` to true. +To orphan dependents, set `--cascade` to false. The default value for +`--cascade` is true. + +Here's an example that orphans the dependents of a ReplicaSet: + +```shell +kubectl delete replicaset my-repset --cascade=false +``` + +## Ongoing development + +In Kubernetes version 1.5, synchronous garbage collection is under active +development. See the tracking +[issue](https://github.com/kubernetes/kubernetes/issues/29891) for more details. + +{% endcapture %} + + +{% capture whatsnext %} + +[Design Doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/garbage-collection.md) + +[Known issues](https://github.com/kubernetes/kubernetes/issues/26120) + +{% endcapture %} + + +{% include templates/concept.md %} diff --git a/docs/concepts/abstractions/controllers/my-repset.yaml b/docs/concepts/abstractions/controllers/my-repset.yaml new file mode 100644 index 0000000000..54befd8f9d --- /dev/null +++ b/docs/concepts/abstractions/controllers/my-repset.yaml @@ -0,0 +1,17 @@ +apiVersion: extensions/v1beta1 +kind: ReplicaSet +metadata: + name: my-repset +spec: + replicas: 3 + selector: + matchLabels: + pod-is-for: garbage-collection-example + template: + metadata: + labels: + pod-is-for: garbage-collection-example + spec: + containers: + - name: nginx + image: nginx diff --git a/docs/user-guide/garbage-collection.md b/docs/user-guide/garbage-collection.md index af90b4dd1a..16f9380866 100644 --- a/docs/user-guide/garbage-collection.md +++ b/docs/user-guide/garbage-collection.md @@ -4,35 +4,6 @@ assignees: title: Garbage Collection (Beta) --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Garbage Collection - -Note: the Garbage Collection is a beta feature and is enabled by default in Kubernetes version 1.4. - -### What does Garbage Collector do - -When you delete, for example, a ReplicaSet, it is often desirable for the server to automatically garbage collect all the Pods that the ReplicaSet creates. The Garbage Collector (GC) implements this. In general, when you delete an owner object, GC deletes that owner's dependent objects. - -### How to establish an owner-dependent relationship between objects - -Kubernetes 1.3 added a metadata.ownerReferences field to every Kubernetes API object. If an API object is a dependent of another object, ownerReference should point to the owning API object. - -When you create a ReplicationController or a ReplicaSet in Kubernetes 1.4, the Kubernetes control plane automatically sets the ownerReference field in each created pod to point to the owning ReplicationController or ReplicaSet. - -You can set up owner-dependent relationships among other objects by manually setting the ownerReference field on dependent objects. - -### Controlling whether Garbage Collector deletes dependents - -When deleting an object, you can request the GC to ***asynchronously*** delete its dependents by ***explicitly*** specifying `deleteOptions.orphanDependents=false` in the deletion request that you send to the API server. A 200 OK response from the API server indicates the owner is deleted. - -In Kubernetes version 1.5, synchronous garbage collection is under active development. See the tracking [issue](https://github.com/kubernetes/kubernetes/issues/29891) for more details. - -If you specify `deleteOptions.orphanDependents=true`, or leave it blank, then the GC will first reset the `ownerReferences` in the dependents, then delete the owner. Note that the deletion of the owner object is asynchronous, that is, a 200 OK response will be sent by the API server before the owner object gets deleted. - -### Other references - -[Design Doc](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/garbage-collection.md) - -[Known issues](https://github.com/kubernetes/kubernetes/issues/26120) +[Garbage Collection](/docs/concepts/abstractions/controllers/garbage-collection/) From c5ca45729707f04617cb5fa211d80f2bd042207d Mon Sep 17 00:00:00 2001 From: Jared Date: Wed, 22 Feb 2017 10:54:28 -0800 Subject: [PATCH 130/407] Update review-issues.md --- docs/contribute/review-issues.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/contribute/review-issues.md b/docs/contribute/review-issues.md index f5430d0b11..1353620e81 100644 --- a/docs/contribute/review-issues.md +++ b/docs/contribute/review-issues.md @@ -12,6 +12,8 @@ This page explains how documentation issues are reviewed and prioritized for the ## Categorizing issues Issues should be sorted into different buckets of work using the following labels and definitions. If an issue doesn't have enough information to identify a problem that can be researched, reviewed, or worked on (i.e. the issue doesn't fit into any of the categories below) you should close the issue with a comment explaining why it is being closed. +### Needs Clarification +* Issues that need more information from the original submitter to make them actionable. Issues with this label that aren't followed up within a week may be closed. ### Actionable * Issues that can be worked on with current information (or may need a comment to explain what needs to be done to make it more clear) @@ -26,8 +28,9 @@ Issues should be sorted into different buckets of work using the following label * Issues that are suggestions for better processes or site improvements that require community agreement to be implemented * Topics can be brought to SIG meetings as agenda items -#### Needs UX Review -* Issues that are suggestions for improving the user interface of the site or fixing a broken UX. +### Needs UX Review +* Issues that are suggestions for improving the user interface of the site. +* Fixing broken site elements. ## Prioritizing Issues From 8904a7f8626bb2ff00169003114628668d241da2 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Wed, 22 Feb 2017 10:11:42 +0800 Subject: [PATCH 131/407] Fixed flag for eviction-soft. --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 8164fbb7d9..ad55b773a5 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -64,7 +64,7 @@ Including: | Existing Flag | New Flag | Rationale | | ------------- | -------- | --------- | -| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` | existing eviction signals can trigger image garbage collection | +| `--image-gc-high-threshold` | `--eviction-hard` or `--eviction-soft` | existing eviction signals can trigger image garbage collection | | `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | eviction reclaims achieve the same behavior | | `--maximum-dead-containers` | | deprecated once old logs are stored outside of container's context | | `--maximum-dead-containers-per-container` | | deprecated once old logs are stored outside of container's context | From 7b69821e5eac5ba69d4dce36780dd3565fed5cd2 Mon Sep 17 00:00:00 2001 From: Xiuyu Li Date: Tue, 21 Feb 2017 16:18:52 +0800 Subject: [PATCH 132/407] show kubectl_rollingupdate svg --- docs/user-guide/kubectl/kubectl_rolling-update.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl/kubectl_rolling-update.md b/docs/user-guide/kubectl/kubectl_rolling-update.md index 382e630c78..f20483e125 100644 --- a/docs/user-guide/kubectl/kubectl_rolling-update.md +++ b/docs/user-guide/kubectl/kubectl_rolling-update.md @@ -13,7 +13,7 @@ Perform a rolling update of the given ReplicationController. Replaces the specified replication controller with a new replication controller by updating one pod at a time to use the new PodTemplate. The new-controller.json must specify the same namespace as the existing replication controller and overwrite at least one (common) label in its replicaSelector. -! http://kubernetes.io/images/docs/kubectl_rollingupdate.svg +![kubectl_rollingupdate](http://kubernetes.io/images/docs/kubectl_rollingupdate.svg) ``` kubectl rolling-update OLD_CONTROLLER_NAME ([NEW_CONTROLLER_NAME] --image=NEW_CONTAINER_IMAGE | -f NEW_CONTROLLER_SPEC) From 9dc31d63022c7d4c47411bb958d5e442823e30aa Mon Sep 17 00:00:00 2001 From: Andres Villarroel Date: Tue, 7 Feb 2017 09:30:13 -0800 Subject: [PATCH 133/407] Update hello-minikube.md - Proposing a different way to run curl without proxy, without touching current environment - There's a formatting problem and the whole thing is being displayed in a single line ![screenshot](http://imgur.com/download/nfk02hd) --- docs/tutorials/stateless-application/hello-minikube.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/tutorials/stateless-application/hello-minikube.md b/docs/tutorials/stateless-application/hello-minikube.md index 6f2a47b897..70fed7db16 100644 --- a/docs/tutorials/stateless-application/hello-minikube.md +++ b/docs/tutorials/stateless-application/hello-minikube.md @@ -74,10 +74,9 @@ chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl ``` Determine whether you can access sites like [https://cloud.google.com/container-registry/](https://cloud.google.com/container-registry/) directly without a proxy, by opening a new terminal and using + ```shell -export http_proxy="" -export https_proxy="" -curl https://cloud.google.com/container-registry/ +curl --proxy "" https://cloud.google.com/container-registry/ ``` If NO proxy is required, start the Minikube cluster: From aa0424825195d83a2a0c1d2eed295b0a85de6249 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Fri, 3 Feb 2017 11:55:13 -0800 Subject: [PATCH 134/407] Add object fields to style guide. --- docs/contribute/style-guide.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/docs/contribute/style-guide.md b/docs/contribute/style-guide.md index 6bb1aa5837..d44cab2ea4 100644 --- a/docs/contribute/style-guide.md +++ b/docs/contribute/style-guide.md @@ -17,7 +17,7 @@ docs, follow the instructions on ## Documentation formatting standards -### Use Camel Case for API objects +### Use camel case for API objects When you refer to an API object, use the same uppercase and lowercase letters that are used in the actual object name. Typically, the names of API @@ -75,7 +75,7 @@ represents.

unschedulable

Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info: http://releases.k8s.io/HEAD/docs/admin/node.md#manual-node-administration"

Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info: http://releases.k8s.io/HEAD/docs/admin/node.md#manual-node-administration

false

boolean

false
Open the /_data/concepts.yaml file.Open the /_data/concepts.yaml file.
-## Code snippet formatting +## Inline code formatting ### Use code style for inline code and commands @@ -84,10 +84,31 @@ document, use the backtick (`). - +
DoDon't
Set the value of the replicas field in the configuration file.Set the value of the "replicas" field in the configuration file.
The kubectl run command creates a Deployment.The "kubectl run" command creates a Deployment.
For declarative management, use kubectl apply.For declarative management, use "kubectl apply".
+### Use code style for object field names + + + + + +
DoDon't
Set the value of the replicas field in the configuration file.Set the value of the "replicas" field in the configuration file.
The value of the exec field is an ExecAction object.The value of the "exec" field is an ExecAction object.
+ +### Use normal style for string and integer field values + +For field values of type string or integer, use normal style without quotation marks. + + + + + + +
DoDon't
Set the value of imagePullPolicy to Always.Set the value of imagePullPolicy to "Always".
Set the value of image to nginx:1.8.Set the value of image to nginx:1.8.
Set the value of the replicas field to 2.Set the value of the replicas field to 2.
+ +## Code snippet formatting + ### Don't include the command prompt From d2266ed0eb804aa3e9f319324e3c6ffc8274aa00 Mon Sep 17 00:00:00 2001 From: David Calavera Date: Sat, 4 Feb 2017 20:25:31 +0100 Subject: [PATCH 135/407] Add documentation to the redirects.rb script. Signed-off-by: David Calavera --- redirects.rb | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/redirects.rb b/redirects.rb index 3aeabbd326..4cb238a811 100644 --- a/redirects.rb +++ b/redirects.rb @@ -1,3 +1,53 @@ +# This script generates a redirects file that Netlify's CDN +# can use to forward visitor to the right content. +# +# You can read more details about this file in Netlify documentation: +# +# https://www.netlify.com/docs/redirects/ +# +# USAGE: +# +# Run this script with Ruby to generate the _redirects file in this repository. +# It works with any Ruby version higher than 1.8. +# +# ruby redirects.rb +# +# If you want to test a change without modifying the content of the current file, +# you can print the output of the script setting the DEBUG environment variable +# when you run the script: +# +# DEBUG=1 ruby redirects.rb +# +# You can test if the content generated is correct in this playground: +# +# https://play.netlify.com/redirects +# +# HOW TO ADD NEW RULES: +# +# This script is divided in two sections. +# +# The first section handles static redirects, +# those that you know the old path and the new path and never change. +# +# If you want to add one of these redirects, add the rule to the `fixed_redirects` variable, +# in a new line before the closing """. The format for basic 301 redirects is the following one, +# check Netlify's documentation linked above for other rules: +# +# /OLD_PATH /NEW_PATH_OR_URL +# +# The second section handles redirects that change depending on the branch that's deployed +# in the site. For instance, when you want to redirect a path to content in +# the kubernetes main repository but the content is specific to a branch that +# matches the deployed branch. +# +# If you want to add one of there redirects, add the old path to the `branch_redirects` list. For instance, +# when you deploy the branch release-1.5 on Netlify, this script will generate the following redirects: +# +# /examples/* https://github.com/kubernetes/kubernetes/tree/release-1.5/examples/:splat +# /cluster/* https://github.com/kubernetes/kubernetes/tree/release-1.5/cluster/:splat +# /docs/devel/* https://github.com/kubernetes/kubernetes/tree/release-1.5/docs/devel/:splat +# /docs/design/* https://github.com/kubernetes/kubernetes/tree/release-1.5/docs/design/:splat +# REPO_TMPL = "https://github.com/kubernetes/kubernetes/tree/%s/%s/:splat" fixed_redirects = """# 301 redirects (301 is the default status when no other one is provided for each line) From 5cf0e08b5bde9be8bf2860c21df3331eb9db1fa6 Mon Sep 17 00:00:00 2001 From: Shiyang Wang Date: Sat, 4 Feb 2017 13:40:26 +0800 Subject: [PATCH 136/407] fix typo --- .../tools/kubectl/object-management-using-declarative-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/tools/kubectl/object-management-using-declarative-config.md b/docs/concepts/tools/kubectl/object-management-using-declarative-config.md index 5033bbac8c..6c8dfbd250 100644 --- a/docs/concepts/tools/kubectl/object-management-using-declarative-config.md +++ b/docs/concepts/tools/kubectl/object-management-using-declarative-config.md @@ -382,7 +382,7 @@ is used to identify fields that have been removed from the configuration file and need to be cleared from the live configuration. Here are the steps used to caluculate which fields should be deleted or set: -1. Calculate the fields to delete. Thes are the fields present in `last-applied-configuration` and missing from the configuration file. +1. Calculate the fields to delete. These are the fields present in `last-applied-configuration` and missing from the configuration file. 2. Calculate the fields to add or set. These are the fields present in the configuration file whose values don't match the live configuration. Here's an example. Suppose this is the configuration file for a Deployment object: From a21e061dcbfd4edc3f6dfbb6df19c3452709f698 Mon Sep 17 00:00:00 2001 From: Pan Luo Date: Fri, 3 Feb 2017 16:13:58 -0800 Subject: [PATCH 137/407] Add generic to create secret command The generic is missing from the command and will cause error: `Error: unknown flag: --type` when running the command. --- docs/user-guide/persistent-volumes/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index ae34668c9e..3ecc8ea99f 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -423,7 +423,7 @@ parameters: * `restuserkey` : Gluster REST service/Heketi user's password which will be used for authentication to the REST server. This parameter is deprecated in favor of `secretNamespace` + `secretName`. * `secretNamespace` + `secretName` : Identification of Secret instance that containes user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must have type "kubernetes.io/glusterfs", e.g. created in this way: ``` - $ kubectl create secret heketi-secret --type="kubernetes.io/glusterfs" --from-literal=key='opensesame' --namespace=default + $ kubectl create secret generic heketi-secret --type="kubernetes.io/glusterfs" --from-literal=key='opensesame' --namespace=default ``` #### OpenStack Cinder @@ -482,7 +482,7 @@ parameters: * `userId`: Ceph client ID that is used to map the RBD image. Default is the same as `adminId`. * `userSecretName`: The name of Ceph Secret for `userId` to map RBD image. It must exist in the same namespace as PVCs. This parameter is required. The provided secret must have type "kubernetes.io/rbd", e.g. created in this way: ``` - $ kubectl create secret ceph-secret --type="kubernetes.io/rbd" --from-literal=key='QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ==' --namespace=kube-system + $ kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-literal=key='QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ==' --namespace=kube-system ``` #### Quobyte @@ -509,7 +509,7 @@ parameters: * `adminSecretNamespace`: The namespace for `adminSecretName`. Default is "default". * `adminSecretName`: secret that holds information about the Quobyte user and the password to authenticate agains the API server. The provided secret must have type "kubernetes.io/quobyte", e.g. created in this way: ``` - $ kubectl create secret quobyte-admin-secret --type="kubernetes.io/quobyte" --from-literal=key='opensesame' --namespace=kube-system + $ kubectl create secret generic quobyte-admin-secret --type="kubernetes.io/quobyte" --from-literal=key='opensesame' --namespace=kube-system ``` * `user`: maps all access to this user. Default is "root". * `group`: maps all access to this group. Default is "nfsnobody". From 24fc013242d033b86e54b28b85a1ca7652784aab Mon Sep 17 00:00:00 2001 From: Philipp T Date: Sat, 4 Feb 2017 19:38:07 +0100 Subject: [PATCH 138/407] Update kubeadm.md grammar ocd --- docs/getting-started-guides/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index a005ed0c56..a4e5679ffb 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -357,6 +357,6 @@ Please note: `kubeadm` is a work in progress and these limitations will be addre 1. If you are using VirtualBox (directly or via Vagrant), you will need to ensure that `hostname -i` returns a routable IP address (i.e. one on the second network interface, not the first one). By default, it doesn't do this and kubelet ends-up using first non-loopback network interface, which is usually NATed. - Workaround: Modify `/etc/hosts`, take a look at this [`Vagrantfile`][ubuntu-vagrantfile] for how you this can be achieved. + Workaround: Modify `/etc/hosts`, take a look at this [`Vagrantfile`][ubuntu-vagrantfile] for how this can be achieved. [ubuntu-vagrantfile]: https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11), From 29e432d41987ed8deb51d627b10142124afa51a6 Mon Sep 17 00:00:00 2001 From: Vasily Upornikov Date: Sun, 5 Feb 2017 13:38:13 +0300 Subject: [PATCH 139/407] Update container-command-args.md --- docs/concepts/configuration/container-command-args.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/configuration/container-command-args.md b/docs/concepts/configuration/container-command-args.md index 54fe3afcb9..25d555f702 100644 --- a/docs/concepts/configuration/container-command-args.md +++ b/docs/concepts/configuration/container-command-args.md @@ -13,7 +13,7 @@ fields to override the default Entrypoint and Cmd of the the Container's image. ## Container entry points and arguments -The configuration file for a Container has an `image` field that specifies the +The configuration file for a Container has an `image` field that specifies the Docker image to be run in the Container. A Docker image has metadata that includes a default Entrypoint and a default Cmd. From 456a3ddcd6283a00155f71036bd6969fbd253c88 Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 6 Feb 2017 11:08:46 +0800 Subject: [PATCH 140/407] Update kubefed.md --- docs/admin/federation/kubefed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/federation/kubefed.md b/docs/admin/federation/kubefed.md index 9685271b85..695edf7bee 100644 --- a/docs/admin/federation/kubefed.md +++ b/docs/admin/federation/kubefed.md @@ -36,7 +36,7 @@ in your `$PATH` and set the executable permission on those binaries. Note: The URL in the curl command below downloads the binaries for Linux amd64. If you are on a different platform, please use the URL for the binaries appropriate for your platform. You can find the list -of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-3) +of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-3). ```shell From af059b3b3c95758daf11d3275df5abdb527d8db6 Mon Sep 17 00:00:00 2001 From: caiyixiang Date: Mon, 6 Feb 2017 11:35:24 +0800 Subject: [PATCH 141/407] Update minikube.md --- docs/getting-started-guides/minikube.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting-started-guides/minikube.md b/docs/getting-started-guides/minikube.md index 4f4a432185..e0e79364cf 100644 --- a/docs/getting-started-guides/minikube.md +++ b/docs/getting-started-guides/minikube.md @@ -40,7 +40,7 @@ See the installation instructions for the [latest release](https://github.com/ku ## Quickstart Here's a brief demo of minikube usage. -If you want to change the VM driver add the appropriate `--vm-driver=xxx` flag to `minikube start`. Minikube Supports +If you want to change the VM driver add the appropriate `--vm-driver=xxx` flag to `minikube start`. Minikube supports the following drivers: * virtualbox @@ -106,7 +106,7 @@ plugins, if required. When using a single VM of Kubernetes, it's really handy to reuse the minikube's built-in Docker daemon; as this means you don't have to build a docker registry on your host machine and push the image into it - you can just build inside the same docker daemon as minikube which speeds up local experiments. Just make sure you tag your Docker image with something other than 'latest' and use that tag while you pull the image. Otherwise, if you do not specify version of your image, it will be assumed as `:latest`, with pull image policy of `Always` correspondingly, which may eventually result in `ErrImagePull` as you may not have any versions of your Docker image out there in the default docker registry (usually DockerHub) yet. -To be able to work with the docker daemon on your mac/linux host use the [docker-env command](./docs/minikube_docker-env.md) in your shell: +To be able to work with the docker daemon on your mac/linux host use the [docker-env command](https://github.com/kubernetes/minikube/blob/master/docs/minikube_docker-env.md) in your shell: ``` eval $(minikube docker-env) From a045b3987d0e4291913e8bb6fea77e18b2b2964b Mon Sep 17 00:00:00 2001 From: yupengzte Date: Sat, 4 Feb 2017 10:46:04 +0800 Subject: [PATCH 142/407] Fix the url error Signed-off-by: yupengzte --- docs/admin/limitrange/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/limitrange/index.md b/docs/admin/limitrange/index.md index 3a31b9e732..82bed781f1 100644 --- a/docs/admin/limitrange/index.md +++ b/docs/admin/limitrange/index.md @@ -30,7 +30,7 @@ This example demonstrates how limits can be applied to a Kubernetes [namespace]( min/max resource limits per pod. In addition, this example demonstrates how you can apply default resource limits to pods in the absence of an end-user specified value. -See [LimitRange design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/) +See [LimitRange design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/) ## Step 0: Prerequisites From f2184920afab181399b7a7f8ce648959ee94aa8d Mon Sep 17 00:00:00 2001 From: sallydeng Date: Mon, 6 Feb 2017 15:06:55 +0800 Subject: [PATCH 143/407] Update namspace and service part of ui.md 1. Namespace can not contain capital letter. 2. For service, internal endpoints for cluster connections and external endpoints for external users. Here, it lost "external". --- docs/user-guide/ui.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index da2b2e611c..b2acc5e8ed 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -97,7 +97,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. - Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-). + Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-), can not contain capital letters. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. @@ -147,7 +147,7 @@ Detail views for workloads show status and specification information and surface ![Deployment detail view](/images/docs/ui-dashboard-deployment-detail.png) #### Services and discovery -Services and discovery view shows Kubernetes resources that allow for exposing services to external world and discovering them within a cluster. For that reason, Service and Ingress views show Pods targeted by them, internal endpoints for cluster connections and endpoints for external users. +Services and discovery view shows Kubernetes resources that allow for exposing services to external world and discovering them within a cluster. For that reason, Service and Ingress views show Pods targeted by them, internal endpoints for cluster connections and external endpoints for external users. ![Service list partial view](/images/docs/ui-dashboard-service-list.png) From 83d5b27667f6dc34cd6169092ca81344862d8106 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Tue, 7 Feb 2017 08:35:21 +0800 Subject: [PATCH 144/407] Update ui.md base on comments Former: The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) , can not contain capital letters. Now: The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. --- docs/user-guide/ui.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index b2acc5e8ed..5d6567a62c 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -97,7 +97,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. - Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-), can not contain capital letters. + Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. From 34f61c10e89fa8ae92b3e283212a027482a0819e Mon Sep 17 00:00:00 2001 From: Ben Spoon Date: Tue, 7 Feb 2017 13:38:47 -0800 Subject: [PATCH 145/407] Fix overflow scroll on code blocks Fixes https://github.com/kubernetes/kubernetes.github.io/issues/633 --- _sass/_base.sass | 1 + 1 file changed, 1 insertion(+) diff --git a/_sass/_base.sass b/_sass/_base.sass index 3ad4f81dc5..72557bb983 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -856,6 +856,7 @@ dd display: block margin: 20px 0 padding: 15px + position: relative overflow-x: auto h1 code, h2 code, h3 code, h4 code, h5 code, h6 code From c999e614f25ccf38fc2a0c88b4629c787753de87 Mon Sep 17 00:00:00 2001 From: tailnode Date: Tue, 7 Feb 2017 19:40:15 +0800 Subject: [PATCH 146/407] fix link --- _includes/v1.5/v1-definitions.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_includes/v1.5/v1-definitions.html b/_includes/v1.5/v1-definitions.html index faaa0e847d..16858392bc 100755 --- a/_includes/v1.5/v1-definitions.html +++ b/_includes/v1.5/v1-definitions.html @@ -8146,7 +8146,7 @@ The resulting set of endpoints can be viewed as:
 - + @@ -8263,4 +8263,4 @@ Last updated 2016-11-17 06:26:10 UTC - \ No newline at end of file + From ee10fefec806ad0273847f7f16e5fa4d89cc626a Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Tue, 7 Feb 2017 10:59:04 +0800 Subject: [PATCH 147/407] fix unreachable links and typos 1. fix unreachable federated resources links 2. federation proposal doc has been relocated, so i fix this link 3. make federation resources supporting cascading deletion clickable 4. make link to `Kubernetes Scaling and Performance Goals` clickable 5. other typo fixes --- docs/user-guide/federation/index.md | 36 ++++++++++++++--------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/user-guide/federation/index.md b/docs/user-guide/federation/index.md index 7ba0ca9c60..10cea0ad6e 100644 --- a/docs/user-guide/federation/index.md +++ b/docs/user-guide/federation/index.md @@ -43,7 +43,7 @@ why you might want multiple clusters are: [Multi cluster guide](/docs/admin/multi-cluster) has more details on this. * Scalability: There are scalability limits to a single kubernetes cluster (this should not be the case for most users. For more details: - https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md). + [Kubernetes Scaling and Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md)). * Hybrid cloud: You can have multiple clusters on different cloud providers or on-premises data centers. @@ -53,7 +53,7 @@ why you might want multiple clusters are: While there are a lot of attractive use cases for federation, there are also some caveats. -* Increased network bandwidth and cost: The dederation control plane watches all +* Increased network bandwidth and cost: The federation control plane watches all clusters to ensure that the current state is as expected. This can lead to significant network cost if the clusters are running in different regions on a cloud provider or on different cloud providers. @@ -93,15 +93,15 @@ Once we have the control plane setup, we can start creating federation API resources. The following guides explain some of the resources in detail: -* [ConfigMap](/docs/user-guide/federation/configmap/) -* [DaemonSets](/docs/user-guide/federation/daemonsets/) -* [Deployment](/docs/user-guide/federation/deployment/) -* [Events](/docs/user-guide/federation/events/) -* [Ingress](/docs/user-guide/federation/federated-ingress/) -* [Namespaces](/docs/user-guide/federation/namespaces/) -* [ReplicaSets](/docs/user-guide/federation/replicasets/) -* [Secrets](/docs/user-guide/federation/secrets/) -* [Services](/docs/user-guide/federation/federated-services/) +* [ConfigMap](https://kubernetes.io/docs/user-guide/federation/configmap/) +* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) +* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) +* [Events](https://kubernetes.io/docs/user-guide/federation/events/) +* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) +* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) +* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) +* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) +* [Services](https://kubernetes.io/docs/user-guide/federation/federated-services/) [API reference docs](/docs/federation/api-reference/) lists all the resources supported by federation apiserver. @@ -119,12 +119,12 @@ federation control plane. The following Federated resources are affected by cascading deletion: -* Ingress -* Namespace -* ReplicaSet -* Secret -* Deployment -* DaemonSet +* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) +* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) +* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) +* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) +* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) +* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) Note: By default, deleting a resource from federation control plane does not delete the corresponding resources from underlying clusters. @@ -133,5 +133,5 @@ delete the corresponding resources from underlying clusters. ## For more information * [Federation - proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/federation.md) + proposal](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/federation.md) * [Kubecon2016 talk on federation](https://www.youtube.com/watch?v=pq9lbkmxpS8) From 8c1b7a6f5c0777bf3f7d4e133ec52a3a29153b3b Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 8 Feb 2017 09:48:41 +0800 Subject: [PATCH 148/407] Update assign-cpu-ram-container.md --- docs/tasks/configure-pod-container/assign-cpu-ram-container.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/assign-cpu-ram-container.md b/docs/tasks/configure-pod-container/assign-cpu-ram-container.md index 24cca3eca5..fc62af79d9 100644 --- a/docs/tasks/configure-pod-container/assign-cpu-ram-container.md +++ b/docs/tasks/configure-pod-container/assign-cpu-ram-container.md @@ -4,7 +4,7 @@ title: Assigning CPU and RAM Resources to a Container {% capture overview %} -This page shows how assign CPU and RAM resources to containers running +This page shows how to assign CPU and RAM resources to containers running in a Kubernetes Pod. {% endcapture %} From aa1c435df38f4bdad8d6be9e747a47cee5e1b21e Mon Sep 17 00:00:00 2001 From: sallydeng Date: Wed, 8 Feb 2017 16:38:00 +0800 Subject: [PATCH 149/407] Update ui.md: namespace name can't be number only When the namespace name only contains number, such as "10", the pod can't be created in this kind of namespace, the pod will be put in default namespace. --- docs/user-guide/ui.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index 5d6567a62c..3456a7e98a 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -98,6 +98,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. + When the namespace name only contains number, such as "10", the pod can't be created in this kind of namespace, the pod will be put in default namespace. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. From 30b425b99128afd0d25846d7d9295b9678fd09c7 Mon Sep 17 00:00:00 2001 From: Andrew Martin Date: Wed, 8 Feb 2017 14:55:50 +0000 Subject: [PATCH 150/407] Fix typo `pages` -> `page` --- docs/getting-started-guides/ubuntu/backups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/backups.md b/docs/getting-started-guides/ubuntu/backups.md index 9c0c741d9b..b839cafee4 100644 --- a/docs/getting-started-guides/ubuntu/backups.md +++ b/docs/getting-started-guides/ubuntu/backups.md @@ -3,7 +3,7 @@ title: Backups --- {% capture overview %} -This pages shows you how to backup and restore data from the different deployed services in a given cluster. +This page shows you how to backup and restore data from the different deployed services in a given cluster. {% endcapture %} {% capture prerequisites %} From 0b192ad9ec3219e7fe8ba83c85a460629a278bbf Mon Sep 17 00:00:00 2001 From: Ghe Rivero Date: Wed, 8 Feb 2017 11:19:11 +0100 Subject: [PATCH 151/407] Fix examples display --- docs/admin/authorization.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/admin/authorization.md b/docs/admin/authorization.md index ce75c189f0..e5c091b772 100644 --- a/docs/admin/authorization.md +++ b/docs/admin/authorization.md @@ -445,6 +445,7 @@ subjects: ``` For all authenticated users: + ```yaml subjects: - kind: Group @@ -452,6 +453,7 @@ subjects: ``` For all unauthenticated users: + ```yaml subjects: - kind: Group @@ -459,6 +461,7 @@ subjects: ``` For all users: + ```yaml subjects: - kind: Group From e93aaf7028a80ddbceb5542502fcf6c159a7bf9a Mon Sep 17 00:00:00 2001 From: Eamon Taaffe Date: Wed, 8 Feb 2017 15:43:16 +1100 Subject: [PATCH 152/407] Missing export statement in example --- docs/user-guide/jobs/work-queue-1/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-1/index.md b/docs/user-guide/jobs/work-queue-1/index.md index b2b68cff23..4ce1ffb667 100644 --- a/docs/user-guide/jobs/work-queue-1/index.md +++ b/docs/user-guide/jobs/work-queue-1/index.md @@ -101,7 +101,7 @@ Next we will verify we can create a queue, and publish and consume messages. # In the next line, rabbitmq-service is the hostname where the rabbitmq-service # can be reached. 5672 is the standard port for rabbitmq. -root@temp-loe07:/# BROKER_URL=amqp://guest:guest@rabbitmq-service:5672 +root@temp-loe07:/# export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672 # If you could not resolve "rabbitmq-service" in the previous step, # then use this command instead: # root@temp-loe07:/# BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672 From c4e19401707e265d36a10827311cd5efb9092e7c Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 10:39:32 -0800 Subject: [PATCH 153/407] Remove User Guide topic: Launching, Exposing, --- docs/user-guide/quick-start.md | 67 +--------------------------------- 1 file changed, 2 insertions(+), 65 deletions(-) diff --git a/docs/user-guide/quick-start.md b/docs/user-guide/quick-start.md index 6cef04810d..67dfb1dc77 100644 --- a/docs/user-guide/quick-start.md +++ b/docs/user-guide/quick-start.md @@ -5,69 +5,6 @@ assignees: title: Launching, Exposing, and Killing Applications --- -This guide will help you get oriented to Kubernetes and running your first containers on the cluster. If you are already familiar with the docker-cli, you can also checkout the docker-cli to kubectl migration guide [here](/docs/user-guide/docker-cli-to-kubectl). +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Launching a simple application, and exposing it to the Internet - -Once your application is packaged into a container and pushed to an image registry, you're ready to deploy it to Kubernetes. -Through integration with some cloud providers (for example Google Compute Engine, AWS EC2, and Azure ACS), Kubernetes also enables you to request it to provision a public IP address for your application. - -For example, [nginx](http://wiki.nginx.org/Main) is a popular HTTP server, with a [pre-built container on Docker hub](https://registry.hub.docker.com/_/nginx/). The [`kubectl run`](/docs/user-guide/kubectl/kubectl_run) commands below will create two nginx replicas, listening on port 80, and a public IP address for your application. - -```shell -$ kubectl run my-nginx --image=nginx --replicas=2 --port=80 -deployment "my-nginx" created -``` - -To expose your service to the public Internet, run: - -```shell -$ kubectl expose deployment my-nginx --target-port=80 --type=LoadBalancer -service "my-nginx" exposed -``` -Note: The type, LoadBalancer, is highly dependent upon the underlying platform that Kubernetes is running on. If your cloud provider doesn't have a load balancer implementation (e.g. OpenStack) for Kubernetes, you can simply use the allocated [NodePort](http://kubernetes.io/docs/user-guide/services/#type-nodeport) as a rudimentary form of load balancing across your endpoints. - -You can see that they are running by: - -```shell -$ kubectl get po -NAME READY STATUS RESTARTS AGE -my-nginx-3800858182-h9v8d 1/1 Running 0 1m -my-nginx-3800858182-wqafx 1/1 Running 0 1m -``` - -Kubernetes will ensure that your application keeps running, by automatically restarting containers that fail, spreading containers across nodes, and recreating containers on new nodes when nodes fail. - -To find the public IP address assigned to your application, execute: - -```shell -$ kubectl get service my-nginx -NAME CLUSTER_IP EXTERNAL_IP PORT(S) AGE -my-nginx 10.179.240.1 25.1.2.3 80/TCP 8s -``` - -You may need to wait for a minute or two for the external IP address to be provisioned. - -In order to access your nginx landing page, you also have to make sure that traffic from external IPs is allowed. Do this by opening a [firewall to allow traffic on port 80](/docs/user-guide/services-firewalls). - -If you're running on AWS, Kubernetes creates an ELB for you. ELBs use host -names, not IPs, so you will have to do `kubectl describe service/my-nginx` and look -for the `LoadBalancer Ingress` host name. Traffic from external IPs is allowed -automatically. - -## Killing the application - -To kill the application and delete its containers and public IP address, do: - -```shell -$ kubectl delete deployment,service my-nginx -deployment "my-nginx" deleted -service "my-nginx" deleted -``` - -## What's next? - -* [Learn about how to configure common container parameters, such as commands and environment variables.](/docs/user-guide/configuring-containers) +[Using a Service to Access an Application in a Cluster](https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address-service/) From 7bd4d8ba44079136835ece48e4f8d07ef46928ed Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 8 Feb 2017 17:56:59 +0800 Subject: [PATCH 154/407] Update debugging-services.md --- docs/user-guide/debugging-services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/debugging-services.md b/docs/user-guide/debugging-services.md index de9cec5c3f..6a1a692e8a 100644 --- a/docs/user-guide/debugging-services.md +++ b/docs/user-guide/debugging-services.md @@ -352,7 +352,7 @@ they are running fine and not crashing. The `-l app=hostnames` argument is a label selector - just like our `Service` has. Inside the Kubernetes system is a control loop which evaluates the -selector of every `Service` and save the results into an `Endpoints` object. +selector of every `Service` and saves the results into an `Endpoints` object. ```shell $ kubectl get endpoints hostnames From d584b78bed11263c0f16f61d8fb8e73735955ed8 Mon Sep 17 00:00:00 2001 From: James Duncan Date: Thu, 9 Feb 2017 08:27:42 +1300 Subject: [PATCH 155/407] Corrected syntax of Networking Section Removed an unnecessary 'the' from a sentence. --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index 07f0d18934..f8b9cf2f72 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -41,7 +41,7 @@ Pods provide two kinds of shared resources for their constituent containers: *ne #### Networking -Each Pod is assigned a unique IP address. Every the container in a Pod shares the network namespace, including the IP address and network ports. Containers *inside a Pod* can communicate with one another using `localhost`. When containers in a Pod communicate with entities *outside the Pod*, they must coordinate how they use the shared network resources (such as ports). +Each Pod is assigned a unique IP address. Every container in a Pod shares the network namespace, including the IP address and network ports. Containers *inside a Pod* can communicate with one another using `localhost`. When containers in a Pod communicate with entities *outside the Pod*, they must coordinate how they use the shared network resources (such as ports). #### Storage From 34ab8024ec3a0713651c4f8929f07ed497c158af Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 13:56:18 -0800 Subject: [PATCH 156/407] Remove Guide topic: Deploying Applications. --- docs/user-guide/deploying-applications.md | 94 +---------------------- 1 file changed, 2 insertions(+), 92 deletions(-) diff --git a/docs/user-guide/deploying-applications.md b/docs/user-guide/deploying-applications.md index 9bbcd34282..a1c04e1003 100644 --- a/docs/user-guide/deploying-applications.md +++ b/docs/user-guide/deploying-applications.md @@ -6,96 +6,6 @@ assignees: title: Deploying Applications --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Launching a set of replicas using a configuration file - -Kubernetes creates and manages sets of replicated containers (actually, replicated [Pods](/docs/user-guide/pods)) using [*Deployments*](/docs/user-guide/deployments). - -A Deployment simply ensures that a specified number of pod "replicas" are running at any one time. If there are too many, it will kill some. If there are too few, it will start more. It's analogous to Google Compute Engine's [Instance Group Manager](https://cloud.google.com/compute/docs/instance-groups/manager/) or AWS's [Auto-scaling Group](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AutoScalingGroup.html) (with no scaling policies). - -The Deployment created to run nginx by `kubectl run` in the [Quick start](/docs/user-guide/quick-start) could be specified using YAML as follows: - -{% include code.html language="yaml" file="run-my-nginx.yaml" ghlink="/docs/user-guide/run-my-nginx.yaml" %} - -Some differences compared to specifying just a pod are that the `kind` is `Deployment`, the number of `replicas` desired is specified, and the pod specification is under the `template` field. The names of the pods don't need to be specified explicitly because they are generated from the name of the Deployment. -View the [Deployment API -object](/docs/api-reference/extensions/v1beta1/definitions/#_v1beta1_deployment) -to view the list of supported fields. - -This Deployment can be created using `create`, just as with pods: - -```shell -$ kubectl create -f ./run-my-nginx.yaml -deployment "my-nginx" created -``` - -Unlike in the case where you directly create pods, a Deployment replaces pods that are deleted or terminated for any reason, such as in the case of node failure. For this reason, we recommend that you use a Deployment for a continuously running application even if your application requires only a single pod, in which case you can omit `replicas` and it will default to a single replica. - -## Viewing Deployment status - -You can view the Deployment you created using `get`: - -```shell -$ kubectl get deployment -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE -my-nginx 2 2 2 2 6s -``` - -This tells you that your Deployment will ensure that you have two nginx replicas (desired replicas = 2). - -You can see those replicas using `get`, just as with pods you created directly: - -```shell -$ kubectl get pods -NAME READY STATUS RESTARTS AGE -my-nginx-3800858182-9hk43 1/1 Running 0 8m -my-nginx-3800858182-e529s 1/1 Running 0 8m -``` - -## Deleting Deployments - -When you want to kill your application, delete your Deployment, as in the [Quick start](/docs/user-guide/quick-start): - -```shell -$ kubectl delete deployment/my-nginx -deployment "my-nginx" deleted -``` - -By default, this will also cause the pods managed by the Deployment to be deleted. If there were a large number of pods, this may take a while to complete. If you want to leave the pods running instead, specify `--cascade=false`. - -If you try to delete the pods before deleting the Deployments, it will just replace them, as it is supposed to do. - -## Labels - -Kubernetes uses user-defined key-value attributes called [*labels*](/docs/user-guide/labels) to categorize and identify sets of resources, such as pods and Deployments. The example above specified a single label in the pod template, with key `run` and value `my-nginx`. All pods created carry that label, which can be viewed using `-L`: - -```shell -$ kubectl get pods -L run -NAME READY STATUS RESTARTS AGE RUN -my-nginx-3800858182-1v53o 1/1 Running 0 46s my-nginx -my-nginx-3800858182-2ds1q 1/1 Running 0 46s my-nginx -``` - -The labels from the pod template are copied to the Deployment's labels by default, as well -- all resources in Kubernetes support labels: - -```shell -$ kubectl get deployment/my-nginx -L run -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE RUN -my-nginx 2 2 2 2 2m my-nginx -``` - -More importantly, the pod template's labels are used to create a [`selector`](/docs/user-guide/labels/#label-selectors) that will match pods carrying those labels. You can see this field by requesting it using the [Go template output format of `kubectl get`](/docs/user-guide/kubectl/kubectl_get): - -```shell{% raw %} -$ kubectl get deployment/my-nginx -o template --template="{{.spec.selector}}" -map[matchLabels:map[run:my-nginx]]{% endraw %} -``` - -You could also specify the `selector` explicitly, such as if you wanted to specify labels in the pod template that you didn't want to select on, but you should ensure that the selector will match the labels of the pods created from the pod template, and that it won't match pods created by other Deployments. The most straightforward way to ensure the latter is to create a unique label value for the Deployment, and to specify it in both the pod template's labels and in the selector's -matchLabels. - -## What's next? - -[Learn about exposing applications to users and clients, and connecting tiers of your application together.](/docs/user-guide/connecting-applications) +[Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) From ed4770e3d8212d5e6faf46998378776712aa61bf Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 14:10:23 -0800 Subject: [PATCH 157/407] Remove Guide topic: Secrets Walkthrough. --- docs/user-guide/secrets/walkthrough.md | 58 +------------------------- 1 file changed, 2 insertions(+), 56 deletions(-) diff --git a/docs/user-guide/secrets/walkthrough.md b/docs/user-guide/secrets/walkthrough.md index 11ca7672df..bb069c675e 100644 --- a/docs/user-guide/secrets/walkthrough.md +++ b/docs/user-guide/secrets/walkthrough.md @@ -1,61 +1,7 @@ --- -assignees: title: Secrets Walkthrough --- -Following this example, you will create a secret and a [pod](/docs/user-guide/pods/) that consumes that secret in a [volume](/docs/user-guide/volumes/). See [Secrets design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/secrets.md) for more information. +{% include user-guide-content-moved.md %} -## Step Zero: Prerequisites - -This example assumes you have a Kubernetes cluster installed and running, and that you have -installed the `kubectl` command line tool somewhere in your path. Please see the [getting -started](/docs/getting-started-guides/) for installation instructions for your platform. - -## Step One: Create the secret - -A secret contains a set of named byte arrays. - -Use the [`secret.yaml`](/docs/user-guide/secrets/secret.yaml) file to create a secret: - -```shell -$ kubectl create -f docs/user-guide/secrets/secret.yaml -``` - -You can use `kubectl` to see information about the secret: - -```shell -$ kubectl get secrets -NAME TYPE DATA -test-secret Opaque 2 - -$ kubectl describe secret test-secret -Name: test-secret -Labels: -Annotations: - -Type: Opaque - -Data -==== -data-1: 9 bytes -data-2: 11 bytes -``` - -## Step Two: Create a pod that consumes a secret - -Pods consume secrets in volumes. Now that you have created a secret, you can create a pod that -consumes it. - -Use the [`secret-pod.yaml`](/docs/user-guide/secrets/secret-pod.yaml) file to create a Pod that consumes the secret. - -```shell -$ kubectl create -f docs/user-guide/secrets/secret-pod.yaml -``` - -This pod runs a binary that displays the content of one of the pieces of secret data in the secret -volume: - -```shell -$ kubectl logs secret-test-pod -2015-04-29T21:17:24.712206409Z content of file "/etc/secret-volume/data-1": value-1 -``` \ No newline at end of file +[Distributing Credentials Securely](docs/tasks/configure-pod-container/distribute-credentials-secure/) From a70480af0047539220c3a428205a97a6e3c004db Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 8 Feb 2017 14:20:30 -0800 Subject: [PATCH 158/407] Update walkthrough.md --- docs/user-guide/secrets/walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/secrets/walkthrough.md b/docs/user-guide/secrets/walkthrough.md index bb069c675e..a58364859f 100644 --- a/docs/user-guide/secrets/walkthrough.md +++ b/docs/user-guide/secrets/walkthrough.md @@ -4,4 +4,4 @@ title: Secrets Walkthrough {% include user-guide-content-moved.md %} -[Distributing Credentials Securely](docs/tasks/configure-pod-container/distribute-credentials-secure/) +[Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) From 25a3aac7b764a078644df45c9dd6997534ac60ed Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Wed, 8 Feb 2017 16:58:11 -0800 Subject: [PATCH 159/407] Remove Guide topic: kubectl to Manage Resources. --- docs/user-guide/working-with-resources.md | 52 +---------------------- 1 file changed, 2 insertions(+), 50 deletions(-) diff --git a/docs/user-guide/working-with-resources.md b/docs/user-guide/working-with-resources.md index 07a9bd55b7..7b55db4e25 100644 --- a/docs/user-guide/working-with-resources.md +++ b/docs/user-guide/working-with-resources.md @@ -5,54 +5,6 @@ assignees: title: Using kubectl to Manage Resources --- -*This document is aimed at users who have worked through some of the examples, -and who want to learn more about using kubectl to manage resources such -as pods and services. Users who want to access the REST API directly, -and developers who want to extend the Kubernetes API should -refer to the [api conventions](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md) and -the [api document](/docs/api/).* +{% include user-guide-content-moved.md %} -## Resources are Automatically Modified - -When you create a resource such as pod, and then retrieve the created -resource, a number of the fields of the resource are added. -You can see this at work in the following example: - -```shell -$ cat > /tmp/original.yaml < /tmp/current.yaml -pods/original -$ wc -l /tmp/original.yaml /tmp/current.yaml - 51 /tmp/current.yaml - 9 /tmp/original.yaml - 60 total -``` - -The resource we posted had only 9 lines, but the one we got back had 51 lines. -If you `diff -u /tmp/original.yaml /tmp/current.yaml`, you can see the fields added to the pod. -The system adds fields in several ways: - - - Some fields are added synchronously with creation of the resource and some are set asynchronously. - - For example: `metadata.uid` is set synchronously. (Read more about [metadata](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#metadata)). - - For example, `status.hostIP` is set only after the pod has been scheduled. This often happens fast, but you may notice pods which do not have this set yet. This is called Late Initialization. (Read more about [status](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status) and [late initialization](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#late-initialization)). - - Some fields are set to default values. Some defaults vary by cluster and some are fixed for the API at a certain version. (Read more about [defaulting](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#defaulting)). - - For example, `spec.containers[0].imagePullPolicy` always defaults to `IfNotPresent` in api v1. - - For example, `spec.containers[0].resources.limits.cpu` may be defaulted to `100m` on some clusters, to some other value on others, and not defaulted at all on others. - -The API will generally not modify fields that you have set; it just sets ones which were unspecified. - -## Finding Documentation on Resource Fields - -You can browse auto-generated API documentation [here](/docs/api/). +[Kubernetes Object Management](/docs/concepts/tools/kubectl/object-management-overview/) From 554d6b7e9fc081f58a73fc1aba541e946e255e83 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Thu, 9 Feb 2017 15:49:59 +0800 Subject: [PATCH 160/407] Update ui.md I reword it base on comment. --- docs/user-guide/ui.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index 3456a7e98a..cb330f1f9b 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -98,7 +98,7 @@ track=stable - **Namespace**: Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called [namespaces](/docs/admin/namespaces/). They let you partition resources into logically named groups. Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace. The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters. - When the namespace name only contains number, such as "10", the pod can't be created in this kind of namespace, the pod will be put in default namespace. + Namespace names should not consist of only numbers. If the name is set as a number, such as 10, the pod will be put in the default namespace. In case the creation of the namespace is successful, it is selected by default. If the creation fails, the first namespace is selected. From 667c1818dc139ac0b0c382d0711ba80ec3fece63 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Thu, 9 Feb 2017 10:18:59 -0800 Subject: [PATCH 161/407] Migrate Init Containers to Concepts (#2438) * Initial commit for Init Containers migration to Concepts section * Add 1.5 beta include file * Change include to renamed user-guide-content-moved.md file * Fix Concepts/KO/Pods ToC * reformat examples to bullet points * fix formatting * Add back Detailed Behavior, Support and compatibility * Adjust formatting issues * revised based on feedback from Steve * complete sentence rewrite --- _data/concepts.yml | 8 +- _includes/1-5-beta.md | 1 + docs/concepts/abstractions/init-containers.md | 184 ++++++++++++++++++ docs/user-guide/pods/init-container.md | 165 +--------------- 4 files changed, 192 insertions(+), 166 deletions(-) create mode 100644 _includes/1-5-beta.md create mode 100644 docs/concepts/abstractions/init-containers.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 4f41c2c818..4227231cc1 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -9,12 +9,14 @@ toc: - docs/concepts/tools/kubectl/object-management-using-imperative-commands.md - docs/concepts/tools/kubectl/object-management-using-imperative-config.md - docs/concepts/tools/kubectl/object-management-using-declarative-config.md - + - title: Kubernetes Objects section: - docs/concepts/abstractions/overview.md - - docs/concepts/abstractions/pod.md - + - title: Pods + section: + - docs/concepts/abstractions/pod.md + - docs/concepts/abstractions/init-containers.md - title: Controllers section: - docs/concepts/abstractions/controllers/statefulsets.md diff --git a/_includes/1-5-beta.md b/_includes/1-5-beta.md new file mode 100644 index 0000000000..abccaa303c --- /dev/null +++ b/_includes/1-5-beta.md @@ -0,0 +1 @@ +***NOTE: This feature is beta in Kubernetes 1.5.*** \ No newline at end of file diff --git a/docs/concepts/abstractions/init-containers.md b/docs/concepts/abstractions/init-containers.md new file mode 100644 index 0000000000..1bea00b3a2 --- /dev/null +++ b/docs/concepts/abstractions/init-containers.md @@ -0,0 +1,184 @@ +--- +assignees: +- erictune +title: Init Containers +--- + +{% capture overview %} +This page provides an overview of Init Containers, which are specialized +Containers that run before app Containers and can contain utilities or setup +scripts not present in an app image. +{% endcapture %} + +{:toc} + +{% include 1-5-beta.md %} + +**Once the feature exits beta, Init Containers will be specified in the PodSpec +alongside the app `containers` array.** + +{% capture body %} +## Understanding Init Containers + +A [Pod](/docs/concepts/abstractions/pod/) can have multiple Containers running +apps within it, but it can also have one or more Init Containers, which are run +before the app Containers are started. + +Init Containers are exactly like regular Containers, except: + +* They always run to completion. +* Each one must complete successfully before the next one is started. + +If an Init Container fails for a Pod, Kubernetes restarts the Pod repeatedly until the Init +Container succeeds. However, if the Pod has a `restartPolicy` of Never, it is not restarted. + +To specify a Container as an Init Container, add the `annotations` key +`pod.beta.kubernetes.io/init-containers`. Its value should be a +JSON array of objects of type +[Container](http://kubernetes.io/docs/api-reference/v1/definitions/#_v1_container). + +The status of an Init Container is returned as another annotation, +`pod.beta.kubernetes.io/init-container-statuses`, which is an array of +container statuses similar to the `status.containerStatuses` field. + +### Differences from regular Containers + +Init Containers support all the fields and features of app Containers, +including resource limits, volumes, and security settings. However, the +resource requests and limits for an Init Container are handled slightly +differently, which are documented in [Resources](#resources) below. Also, Init Containers do not +support readiness probes because they must run to completion before the Pod can +be ready. + +If multiple Init Containers are specified for a Pod, those Containers are run +one at a time in sequential order. Each must succeed before the next can run. +When all of the Init Containers have run to completion, Kubernetes initializes +the Pod and runs the application Containers as usual. + +## What can Init Containers be used for? + +Because Init Containers have separate images from app Containers, they +have some advantages for start-up related code: + +* They can contain and run utilities that are not desirable to include in the + app Container image for security reasons. +* They can contain utilities or custom code for setup that is not present in an app + image. For example, there is no need to make an image `FROM` another image just to use a tool like + `sed`, `awk`, `python`, or `dig` during setup. +* The application image builder and deployer roles can work independently without + the need to jointly build a single app image. +* They use Linux namespaces so they have a different filesystem view from app Containers. + Consequently, they can be given access to Secrets that app Containers are not able to + access. +* They run to completion before any app Containers start, whereas app + Containers run in parallel, so Init Containers provide an easy way to block or + delay the startup of app Containers until some set of preconditions are met. + +### Examples +Here are some ideas for how to use Init Containers: + +* Wait for a service to be created with a shell command like: + + for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; exit 1 + +* Register this Pod with a remote server from the downward API with a command like: + + curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$()&ip=$()' + +* Wait for some time before starting the app Container with a command like `sleep 60`. +* Clone a git repository into a volume. +* Place values into a configuration file and run a template tool to dynamically + generate a configuration file for the the main app Container. For example, + place the POD_IP value in a configuration and generate the main app + configuration file using Jinja. + +More detailed usage examples can be found in the [StatefulSets documentation](/docs/concepts/abstractions/controllers/statefulsets/) +and the [Production Pods guide](/docs/user-guide/production-pods.md#handling-initialization). + +## Detailed behavior + +During the startup of a Pod, the Init Containers are started in order, after the +network and volumes are initialized. Each Container must exit successfully before +the next is started. If a Container fails to start due to the runtime or +exits with failure, it is retried according to the Pod `restartPolicy`. However, +if the Pod `restartPolicy` is set to Always, the Init Containers use +`RestartPolicy` OnFailure. + +A Pod cannot be `Ready` until all Init Containers have succeeded. The ports on an +Init Container are not aggregated under a service. A Pod that is initializing +is in the `Pending` state but should have a condition `Initializing` set to true. + +If the Pod is [restarted](#pod-restart-reasons), all Init Containers must +execute again. + +Changes to the Init Container spec are limited to the container image field. +Altering an Init Container image field is equivalent to restarting the Pod. + +Because Init Containers can be restarted, retried, or re-executed, Init Container +code should be idempotent. In particular, code that writes to files on `EmptyDirs` +should be prepared for the possibility that an output file already exists. + +Init Containers have all of the fields of an app Container. However, Kubernetes +prohibits `readinessProbe` from being used because Init Containers cannot +define readiness distinct from completion. This is enforced during validation. + +Use `activeDeadlineSeconds` on the Pod and `livenessProbe` on the Container to +prevent Init Containers from failing forever. The active deadline includes Init +Containers. + +The name of each app and Init Container in a Pod must be unique; a +validation error is thrown for any Container sharing a name with another. + +### Resources + +Given the ordering and execution for Init Containers, the following rules +for resource usage apply: + +* The highest of any particular resource request or limit defined on all Init + Containers is the *effective init request/limit* +* The Pod's *effective request/limit* for a resource is the higher of: + * the sum of all app Containers request/limit for a resource + * the effective init request/limit for a resource +* Scheduling is done based on effective requests/limits, which means + Init Containers can reserve resources for initialization that are not used + during the life of the Pod. +* QoS tier of the Pod's *effective QoS tier* is the QoS tier for Init Containers + and app containers alike. + +Quota and limits are applied based on the effective Pod request and +limit. + +Pod level cgroups are based on the effective Pod request and limit, the +same as the scheduler. + + +### Pod restart reasons + +A Pod can restart, causing re-execution of Init Containers, for the following +reasons: + +* A user updates the PodSpec causing the Init Container image to change. + App Container image changes only restart the app Container. +* The Pod infrastructure container is restarted. This is uncommon and would + have to be done by someone with root access to nodes. +* All containers in a Pod are terminated while `restartPolicy` is set to Always, + forcing a restart, and the Init Container completion record has been lost due + to garbage collection. + +## Support and compatibility + +A cluster with Kubelet and Apiserver version 1.4.0 or greater supports Init +Containers with the beta annotations. Support varies for other combinations of +Kubelet and Apiserver versions; see the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md) for details. + +{% endcapture %} + + +{% capture whatsnext %} + +* [Creating a Pod that has an Init Container](/docs/tasks/configure-pod-container/configure-pod-initialization/#creating-a-pod-that-has-an-init-container) + +{% endcapture %} + + +{% include templates/concept.md %} \ No newline at end of file diff --git a/docs/user-guide/pods/init-container.md b/docs/user-guide/pods/init-container.md index ce7679d40e..90a862ea02 100644 --- a/docs/user-guide/pods/init-container.md +++ b/docs/user-guide/pods/init-container.md @@ -1,168 +1,7 @@ --- -assignees: -- erictune title: Init Containers --- -* TOC -{:toc} - -In addition to having one or more main containers (or **app containers**), a -pod can also have one or more **init containers** which run before the app -containers. Init containers allow you to reduce and reorganize setup scripts -and "glue code". - -## Overview - -An init container is exactly like a regular container, except that it always -runs to completion and each init container must complete successfully before -the next one is started. If the init container fails, Kubernetes will restart -the pod until the init container succeeds. If a pod is marked as `RestartNever`, -the pod will fail if the init container fails. - -You specify a container as an init container by adding an annotation. -The annotation key is `pod.beta.kubernetes.io/init-containers`. The annotation -value is a JSON array of [objects of type `v1.Container` -](http://kubernetes.io/docs/api-reference/v1/definitions/#_v1_container) - -Once the feature exits beta, the init containers will be specified on the Pod -Spec alongside the app `containers` array. -The status of the init containers is returned as another annotation - -`pod.beta.kubernetes.io/init-container-statuses` -- as an array of the -container statuses (similar to the `status.containerStatuses` field). - -Init containers support all of the same features as normal containers, -including resource limits, volumes, and security settings. The resource -requests and limits for an init container are [handled slightly differently]( -#resources). Init containers do not support readiness probes since they will -run to completion before the pod can be ready. -An init container has all of the fields of an app container. - -If you specify multiple init containers for a pod, those containers run one at -a time in sequential order. Each must succeed before the next can run. Once all -init containers have run to completion, Kubernetes initializes the pod and runs -the application containers as usual. - -## What are Init Containers Good For? - -Because init containers have separate images from application containers, they -have some advantages for start-up related code. These include: - -* they can contain utilities that are not desirable to include in the app container - image for security reasons, -* they can contain utilities or custom code for setup that is not present in an app - image. (No need to make an image `FROM` another image just to use a tool like - `sed`, `awk`, `python`, `dig`, etc during setup). -* the application image builder and the deployer roles can work independently without - the need to jointly build a single app image. - -Because init containers have different filesystem view (Linux namespaces) from -app containers, they can be given access to Secrets that the app containers are -not able to access. - -Since init containers run to completion before any app containers start, and -since app containers run in parallel, they provide an easier way to block or -delay the startup of application containers until some precondition is met. - -Because init containers run in sequence and there can be multiple init containers, -they can be composed easily. - -Here are some ideas for how to use init containers: -- Wait for a service to be created with a shell command like: - `for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; exit 1` -- Register this pod with a remote server with a command like: - `curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$(POD_NAME)&ip=$(POD_IP)'` - using `POD_NAME` and `POD_IP` from the downward API. -- Wait for some time before starting the app container with a command like `sleep 60`. -- Clone a git repository into a volume -- Place values like a POD_IP into a configuration file, and run a template tool (e.g. jinja) - to generate a configuration file to be consumed by the main app contianer. - -Complete usage examples can be found in the [StatefulSets -documentation](/docs/concepts/abstractions/controllers/statefulsets/) and the [Production Pods -guide](/docs/user-guide/production-pods.md#handling-initialization). - - -## Detailed Behavior - -Each pod may have 0..N init containers defined along with the existing -1..M app containers. - -On startup of the pod, after the network and volumes are initialized, the init -containers are started in order. Each container must exit successfully before -the next is invoked. If a container fails to start (due to the runtime) or -exits with failure, it is retried according to the pod RestartPolicy, except -when the pod restart policy is RestartPolicyAlways, in which case just the init -containers use RestartPolicyOnFailure. - -A pod cannot be ready until all init containers have succeeded. The ports on an -init container are not aggregated under a service. A pod that is being -initialized is in the `Pending` phase but should has a condition `Initializing` -set to `true`. - -If the pod is [restarted](#pod-restart-reasons) all init containers must -execute again. - -Changes to the init container spec are limited to the container image field. -Altering an init container image field is equivalent to restarting the pod. - -Because init containers can be restarted, retried, or reexecuted, init container -code should be idempotent. In particular, code that writes to files on EmptyDirs -should be prepared for the possibility that an output file already exists. - -An init container has all of the fields of an app container. The following -fields are prohibited from being used on init containers by validation: - -* `readinessProbe` - init containers must exit for pod startup to continue, - are not included in rotation, and so cannot define readiness distinct from - completion. - -Init container authors may use `activeDeadlineSeconds` on the pod and -`livenessProbe` on the container to prevent init containers from failing -forever. The active deadline includes init containers. - -The name of each app and init container in a pod must be unique - it is a -validation error for any container to share a name. - -### Resources - -Given the ordering and execution for init containers, the following rules -for resource usage apply: - -* The highest of any particular resource request or limit defined on all init - containers is the **effective init request/limit** -* The pod's **effective request/limit** for a resource is the higher of: - * sum of all app containers request/limit for a resource - * effective init request/limit for a resource -* Scheduling is done based on effective requests/limits, which means - init containers can reserve resources for initialization that are not used - during the life of the pod. -* QoS tier of the pod's **effective QoS tier** is the QoS tier for init containers - and app containers alike. - -Quota and limits are applied based on the effective pod request and -limit. - -Pod level cGroups are based on the effective pod request and limit, the -same as the scheduler. - - -## Pod Restart Reasons - -A Pod may "restart", causing reexecution of init containers, for the following -reasons: - -* An init container image is changed by a user updating the Pod Spec. - * App container image changes only restart the app container. -* The pod infrastructure container is restarted. - * This is uncommon and would have to be done by someone with root access to nodes. -* All containers in a pod are terminated, requiring a restart (RestartPolicyAlways) AND the record of init container completion has been lost due to garbage collection. - -## Support and compatibility - -A cluster with Kubelet and Apiserver version 1.4.0 or greater supports init -containers with the beta annotations. Support varies for other combinations of -Kubelet and Apiserver version; see the [release notes -](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md) for details. - +{% include user-guide-content-moved.md %} +* [Init Containers](/docs/concepts/abstractions/init-containers/) \ No newline at end of file From 50db7c5b29684854b0a6d5f30f946c3d5ccd1cc5 Mon Sep 17 00:00:00 2001 From: scjane Date: Thu, 9 Feb 2017 16:18:48 +0800 Subject: [PATCH 162/407] Update accessing-the-cluster.md --- docs/user-guide/accessing-the-cluster.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/user-guide/accessing-the-cluster.md b/docs/user-guide/accessing-the-cluster.md index 8fb9e6b0b4..1f5a65aa61 100644 --- a/docs/user-guide/accessing-the-cluster.md +++ b/docs/user-guide/accessing-the-cluster.md @@ -1,10 +1,10 @@ ---- -assignees: -- lavalamp -- mikedanese -title: Accessing Clusters ---- - +--- +assignees: +- lavalamp +- mikedanese +title: Accessing Clusters +--- + * TOC {:toc} @@ -125,6 +125,7 @@ with future high-availability support. The Kubernetes project-supported Go client library is at [https://github.com/kubernetes/client-go](https://github.com/kubernetes/client-go). To use it, + * To get the library, run the following command: `go get k8s.io/client-go//kubernetes` See [https://github.com/kubernetes/client-go](https://github.com/kubernetes/client-go) to see which versions are supported. * Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g., `import "k8s.io/client-go/1.4/pkg/api/v1"` is correct. From efbf64d09d62805707b523aa0e8eeaf90befb8b6 Mon Sep 17 00:00:00 2001 From: Himanshu Raj Date: Thu, 9 Feb 2017 00:09:44 -0800 Subject: [PATCH 163/407] Add Contiv Information to addons.md --- docs/admin/addons.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/admin/addons.md b/docs/admin/addons.md index aeee68cc30..fc1c4ec83c 100644 --- a/docs/admin/addons.md +++ b/docs/admin/addons.md @@ -14,6 +14,7 @@ Add-ons in each section are sorted alphabetically - the ordering does not imply * [Calico](http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/) is a secure L3 networking and network policy provider. * [Canal](https://github.com/tigera/canal/tree/master/k8s-install/kubeadm) unites Flannel and Calico, providing networking and network policy. +* [Contiv](http://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](http://github.com/contiv). The [installer](http://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options. * [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml) is an overlay network provider that can be used with Kubernetes. * [Romana](http://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/user-guide/networkpolicies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize). * [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database. From 1c04cedd412b31bbcd8b708e8ac1033fac8e31a6 Mon Sep 17 00:00:00 2001 From: xilabao Date: Wed, 8 Feb 2017 20:43:18 -0600 Subject: [PATCH 164/407] add authorizationMode to kubeadm config ref to https://github.com/kubernetes/kubernetes/pull/39846 --- docs/admin/kubeadm.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 1014ee1ab6..edf9c87a7d 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -191,6 +191,7 @@ available as configuration file options. externalDNSNames: - - + authorizationMode: cloudProvider: discovery: bindPort: From 76938f5b68aeca4e2368f879851e9bf66d2f7042 Mon Sep 17 00:00:00 2001 From: Klaus Ma Date: Sat, 28 Jan 2017 21:59:13 +0800 Subject: [PATCH 165/407] Updated multiple scheduler beta docs. --- docs/admin/multiple-schedulers.md | 22 ++++++++++------------ docs/admin/multiple-schedulers/pod2.yaml | 3 +-- docs/admin/multiple-schedulers/pod3.yaml | 3 +-- 3 files changed, 12 insertions(+), 16 deletions(-) diff --git a/docs/admin/multiple-schedulers.md b/docs/admin/multiple-schedulers.md index bf7386e492..3e3b4d270b 100644 --- a/docs/admin/multiple-schedulers.md +++ b/docs/admin/multiple-schedulers.md @@ -61,7 +61,7 @@ config. Save it as `my-scheduler.yaml`: {% include code.html language="yaml" file="multiple-schedulers/my-scheduler.yaml" ghlink="/docs/admin/multiple-schedulers/my-scheduler.yaml" %} An important thing to note here is that the name of the scheduler specified as an -argument to the scheduler command in the container spec should be unique. This is the name that is matched against the value of the optional `scheduler.alpha.kubernetes.io/name` annotation on pods, to determine whether this scheduler is responsible for scheduling a particular pod. +argument to the scheduler command in the container spec should be unique. This is the name that is matched against the value of the optional `spec.schedulername` on pods, to determine whether this scheduler is responsible for scheduling a particular pod. Please see the [kube-scheduler documentation](/docs/admin/kube-scheduler/) for @@ -92,14 +92,14 @@ pod in this list. ### 4. Specify schedulers for pods Now that our second scheduler is running, let's create some pods, and direct them to be scheduled by either the default scheduler or the one we just deployed. In order to schedule a given pod using a specific scheduler, we specify the name of the -scheduler as an annotation in that pod spec. Let's look at three examples. +scheduler in that pod spec. Let's look at three examples. -1. Pod spec without any scheduler annotation +1. Pod spec without any scheduler name {% include code.html language="yaml" file="multiple-schedulers/pod1.yaml" ghlink="/docs/admin/multiple-schedulers/pod1.yaml" %} - When no scheduler annotation is supplied, the pod is automatically scheduled using the + When no scheduler name is supplied, the pod is automatically scheduled using the default-scheduler. Save this file as `pod1.yaml` and submit it to the Kubernetes cluster. @@ -108,12 +108,11 @@ scheduler as an annotation in that pod spec. Let's look at three examples. kubectl create -f pod1.yaml ``` -2. Pod spec with `default-scheduler` annotation +2. Pod spec with `default-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod2.yaml" ghlink="/docs/admin/multiple-schedulers/pod2.yaml" %} - A scheduler is specified by supplying the scheduler name as a value to the annotation - with key `scheduler.alpha.kubernetes.io/name`. In this case, we supply the name of the + A scheduler is specified by supplying the scheduler name as a value to `spec.schedulername`. In this case, we supply the name of the default scheduler which is `default-scheduler`. Save this file as `pod2.yaml` and submit it to the Kubernetes cluster. @@ -122,13 +121,12 @@ scheduler as an annotation in that pod spec. Let's look at three examples. kubectl create -f pod2.yaml ``` -3. Pod spec with `my-scheduler` annotation +3. Pod spec with `my-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod3.yaml" ghlink="/docs/admin/multiple-schedulers/pod3.yaml" %} In this case, we specify that this pod should be scheduled using the scheduler that we - deployed - `my-scheduler`. Note that the value of the annotation with key - `scheduler.alpha.kubernetes.io/name` should match the name supplied to the scheduler + deployed - `my-scheduler`. Note that the value of `spec.schedulername` should match the name supplied to the scheduler command as an argument in the deployment config for the scheduler. Save this file as `pod3.yaml` and submit it to the Kubernetes cluster. @@ -149,9 +147,9 @@ In order to make it easier to work through these examples, we did not verify tha pods were actually scheduled using the desired schedulers. We can verify that by changing the order of pod and deployment config submissions above. If we submit all the pod configs to a Kubernetes cluster before submitting the scheduler deployment config, -we see that the pod `annotation-second-scheduler` remains in "Pending" state forever +we see that the pod `second-scheduler` remains in "Pending" state forever while the other two pods get scheduled. Once we submit the scheduler deployment config -and our new scheduler starts running, the `annotation-second-scheduler` pod gets +and our new scheduler starts running, the `second-scheduler` pod gets scheduled as well. Alternatively, one could just look at the "Scheduled" entries in the event logs to diff --git a/docs/admin/multiple-schedulers/pod2.yaml b/docs/admin/multiple-schedulers/pod2.yaml index 2f9fdf9875..26b0376967 100644 --- a/docs/admin/multiple-schedulers/pod2.yaml +++ b/docs/admin/multiple-schedulers/pod2.yaml @@ -2,11 +2,10 @@ apiVersion: v1 kind: Pod metadata: name: annotation-default-scheduler - annotations: - scheduler.alpha.kubernetes.io/name: default-scheduler labels: name: multischeduler-example spec: + schedulername: default-scheduler containers: - name: pod-with-default-annotation-container image: gcr.io/google_containers/pause:2.0 \ No newline at end of file diff --git a/docs/admin/multiple-schedulers/pod3.yaml b/docs/admin/multiple-schedulers/pod3.yaml index 52276df5f6..e64070abfa 100644 --- a/docs/admin/multiple-schedulers/pod3.yaml +++ b/docs/admin/multiple-schedulers/pod3.yaml @@ -2,11 +2,10 @@ apiVersion: v1 kind: Pod metadata: name: annotation-second-scheduler - annotations: - scheduler.alpha.kubernetes.io/name: my-scheduler labels: name: multischeduler-example spec: + schedulername: my-scheduler containers: - name: pod-with-second-annotation-container image: gcr.io/google_containers/pause:2.0 \ No newline at end of file From 601fd27c56f7b9fe5c20f6f288f7bed089199559 Mon Sep 17 00:00:00 2001 From: Robert Roland Date: Fri, 3 Feb 2017 11:23:45 -0800 Subject: [PATCH 166/407] Adds documentation for new vSphere vm-uuid parameter. Related to kubernetes/kubernetes#40892 --- docs/getting-started-guides/vsphere.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/getting-started-guides/vsphere.md b/docs/getting-started-guides/vsphere.md index ba07d14890..61ea733bf3 100644 --- a/docs/getting-started-guides/vsphere.md +++ b/docs/getting-started-guides/vsphere.md @@ -69,6 +69,7 @@ Sample Config: datacenter = datastore = working-dir = + vm-uuid = [Disk] scsicontrollertype = pvscsi ``` From 380fc428863c1cd942c8efe77007b09172f1c794 Mon Sep 17 00:00:00 2001 From: cwitte4191 Date: Fri, 3 Feb 2017 13:52:34 -0600 Subject: [PATCH 167/407] Update service-accounts.md Added an automated code snippet to populate the registrykey on a serviceaccount using jq. --- docs/user-guide/service-accounts.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/service-accounts.md b/docs/user-guide/service-accounts.md index d1129a1983..ad1c0a68bf 100644 --- a/docs/user-guide/service-accounts.md +++ b/docs/user-guide/service-accounts.md @@ -147,8 +147,18 @@ NAME TYPE DATA myregistrykey kubernetes.io/.dockerconfigjson 1 ``` -Next, read/modify/write the service account for the namespace to use this secret as an imagePullSecret +Next, read/modify/write the service account for the namespace to use this secret as an imagePullSecret. +Automated version using json and the jq utility: +```shell +kubectl get serviceaccounts default -o json | + jq 'del(.metadata.resourceVersion)'| + jq 'setpath(["imagePullSecrets"];[{"name":"myregistrykey"}])' | + kubectl replace serviceaccount default -f - + +``` + +Interactive version requiring manual edit: ```shell $ kubectl get serviceaccounts default -o yaml > ./sa.yaml $ cat sa.yaml From 94aab012c02ae53ff0445513ea495f27afecab48 Mon Sep 17 00:00:00 2001 From: Phillip Wittrock Date: Wed, 8 Feb 2017 17:03:17 -0800 Subject: [PATCH 168/407] Kubectl task document for printing out containers in a cluster. --- _data/tasks.yml | 4 + .../list-all-running-container-images.md | 122 ++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 docs/tasks/kubectl/list-all-running-container-images.md diff --git a/_data/tasks.yml b/_data/tasks.yml index de066c9ed2..f301de1edc 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -3,6 +3,10 @@ abstract: "Step-by-step instructions for performing operations with Kubernetes." toc: - docs/tasks/index.md +- title: Using the Kubectl Command-Line + section: + - docs/tasks/kubectl/list-all-running-container-images.md + - title: Configuring Pods and Containers section: - docs/tasks/configure-pod-container/define-environment-variable-container.md diff --git a/docs/tasks/kubectl/list-all-running-container-images.md b/docs/tasks/kubectl/list-all-running-container-images.md new file mode 100644 index 0000000000..4fb64ee442 --- /dev/null +++ b/docs/tasks/kubectl/list-all-running-container-images.md @@ -0,0 +1,122 @@ +--- +title: Listing all Container images running in the cluster +--- + +{% capture overview %} + +This page shows how to use kubectl to list all of the Container images +for Pods running in a cluster. + +{% endcapture %} + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + +{% capture steps %} + +In this exercise you will use kubectl to fetch all of the Pods +running in a cluster, and format the output to pull out the list +of Containers for each. + +## List all Containers in all namespaces + +- Fetch all Pods in all namespaces using `kubectl get pods --all-namespaces` +- Format the output to include only the list of Container image names + using `-o jsonpath={..image}`. This will recursively parse out the + `image` field from the returned json. + - See the [jsonpath reference](/docs/user-guide/jsonpath/) + for further information on how to use jsonpath. +- Format the output using standard tools: `tr`, `sort`, `uniq` + - Use `tr` to replace spaces with newlines + - Use `sort` to sort the results + - Use `uniq` to aggregate image counts + +```sh +kubectl get pods --all-namespaces -o jsonpath="{..image}" |\ +tr -s '[[:space:]]' '\n' |\ +sort |\ +uniq -c +``` + +The above command will recursively return all fields named `image` +for all items returned. + +As an alternative, it is possible to use the absolute path to the image +field within the Pod. This ensures the correct field is retrieved +in the even the field name is repeated, +e.g. many fields are called `name` within a given item: + +```sh +kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" +``` + +The jsonpath is interpreted as follows: + +- `.items[*]`: for each returned value +- `.spec`: get the spec +- `.containers[*]`: for each container +- `.image`: get the image + +**Note:** When fetching a single Pod by name, e.g. `kubect get pod nginx`, +the `.items[*]` portion of the path should be omitted because a single +Pod is returned instead of a list of items. + +## List Containers by Pod + +The formatting can be controlled further by using the `range` operation to +iterate over elements individually. + +```sh +kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |\ +sort +``` + +## List Containers filtering by Pod label + +To target only Pods matching a specific label, use the -l flag. The +following matches only Pods with labels matching `app=nginx`. + +```sh +kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx +``` + +## List Containers filtering by Pod namespace + +To target only pods in a specific namespace, use the namespace flag. The +following matches only Pods in the `kube-system` namespace. + +```sh +kubectl get pods --namespace kube-system -o jsonpath="{..image}" +``` + +## List Containers using a go-template instead of jsonpath + +As an alternative to jsonpath, Kubectl supports using [go-templates](https://golang.org/pkg/text/template/) +for formatting the output: + +{% raw %} +```sh +kubectl get pods --all-namespaces -o go-template --template="{{range .items}}{{range .spec.containers}}{{.image}} {{end}}{{end}}" +``` +{% endraw %} + + +{% endcapture %} + +{% capture discussion %} + +{% endcapture %} + +{% capture whatsnext %} + +### Reference + +* [Jsonpath](/docs/user-guide/jsonpath/) reference guide +* [Go template](https://golang.org/pkg/text/template/) reference guide + +{% endcapture %} + +{% include templates/task.md %} From 3a993cdadc42db2ae7d38de270cd4a74aeb4cbbb Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Thu, 9 Feb 2017 14:08:54 -0800 Subject: [PATCH 169/407] New Task: Exposing Pod Info ... Env Vars --- _data/tasks.yml | 1 + .../dapi-envars-container.yaml | 45 +++++ .../dapi-envars-pod.yaml | 38 ++++ ...ronment-variable-expose-pod-information.md | 163 ++++++++++++++++++ 4 files changed, 247 insertions(+) create mode 100644 docs/tasks/configure-pod-container/dapi-envars-container.yaml create mode 100644 docs/tasks/configure-pod-container/dapi-envars-pod.yaml create mode 100644 docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md diff --git a/_data/tasks.yml b/_data/tasks.yml index f301de1edc..cb61b1a05a 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -14,6 +14,7 @@ toc: - docs/tasks/configure-pod-container/assign-cpu-ram-container.md - docs/tasks/configure-pod-container/configure-volume-storage.md - docs/tasks/configure-pod-container/configure-persistent-volume-storage.md + - docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md - docs/tasks/configure-pod-container/distribute-credentials-secure.md - docs/tasks/configure-pod-container/pull-image-private-registry.md - docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md diff --git a/docs/tasks/configure-pod-container/dapi-envars-container.yaml b/docs/tasks/configure-pod-container/dapi-envars-container.yaml new file mode 100644 index 0000000000..8b3b3a39d3 --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-envars-container.yaml @@ -0,0 +1,45 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-resourcefieldref +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox:1.24 + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_CPU_REQUEST MY_CPU_LIMIT; + printenv MY_MEM_REQUEST MY_MEM_LIMIT; + sleep 10; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + env: + - name: MY_CPU_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.cpu + - name: MY_CPU_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.cpu + - name: MY_MEM_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.memory + - name: MY_MEM_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.memory + restartPolicy: Never diff --git a/docs/tasks/configure-pod-container/dapi-envars-pod.yaml b/docs/tasks/configure-pod-container/dapi-envars-pod.yaml new file mode 100644 index 0000000000..00762373b3 --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-envars-pod.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-fieldref +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_NODE_NAME MY_POD_NAME MY_POD_NAMESPACE; + printenv MY_POD_IP MY_POD_SERVICE_ACCOUNT; + sleep 10; + done; + env: + - name: MY_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: MY_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: MY_POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: MY_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: MY_POD_SERVICE_ACCOUNT + valueFrom: + fieldRef: + fieldPath: spec.serviceAccountName + restartPolicy: Never diff --git a/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md new file mode 100644 index 0000000000..1390727000 --- /dev/null +++ b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md @@ -0,0 +1,163 @@ +--- +title: Exposing Pod Information to Containers Through Environment Variables +--- + +{% capture overview %} + +This page shows how a Pod can use environment variables to expose information +about itself to Containers running in the Pod. Environment variables can expose +Pod fields and Container fields. + +There are two ways to expose Pod and Container fields to a running Container: +environment variables and +[DownwardAPIVolumeFiles](/docs/resources-reference/v1.5/#downwardapivolumefile-v1). +Together, these two ways of exposing Pod and Container fields are called the +*Downward API*. + +{% endcapture %} + + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + + +{% capture steps %} + +## Using Pod fields as values for environment variables + +In this exercise, you create a Pod that has one Container. Here is the +configuration file for the Pod: + +{% include code.html language="yaml" file="dapi-envars-pod.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-envars-pod.yaml" %} + +In the configuration file, you can see five environment variables. The `env` +field is an array of +[EnvVars](/docs/resources-reference/v1.5/#envvar-v1). +The first element in the array specifies that the `MY_NODE_NAME` environment +variable gets its value from the Pod's `spec.nodeName` field. Similarly, the +other environment variables get their names from Pod fields. + +**Note**: The fields in this example are Pod fields. They are not fields of the +Container in the Pod. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-envars-pod.yaml +``` + +Verify that the Container in the Pod is running: + +``` +kubectl get pods +``` + +View the Container's logs: + +``` +kubectl logs dapi-envars-fieldref +``` + +The output shows the values of selected environment variables: + +``` +minikube +dapi-envars-fieldref +default +172.17.0.4 +default +``` + +To see why these values are in the log, look at the `command` and `args` fields +in the configuration file. When the Container starts, it writes the values of +five environment variables to stdout. It repeats this every ten seconds. + +Next, get a shell into the Container that is running in your Pod: + +``` +kubectl exec -it dapi-envars-fieldref -- sh +``` + +In your shell, view the environment variables: + +``` +/# printenv +``` + +The output shows that certain environment variables have been assigned the +values of Pod fields: + +``` +MY_POD_SERVICE_ACCOUNT=default +... +MY_POD_NAMESPACE=default +MY_POD_IP=172.17.0.4 +... +MY_NODE_NAME=minikube +... +MY_POD_NAME=dapi-envars-fieldref +``` + +## Using Container fields as values for environment variables + +In the preceding exercise, you used Pod fields as the values for environment +variables. In this next exercise, you use Container fields as the values for +environment variables. Here is the configuration file for a Pod that has one +container: + +{% include code.html language="yaml" file="dapi-envars-container.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-envars-container.yaml" %} + +In the configuration file, you can see four environment variables. The `env` +field is an array of +[EnvVars](/docs/resources-reference/v1.5/#envvar-v1). +The first element in the array specifies that the `MY_CPU_REQUEST` environment +variable gets its value from the `requests.cpu` field of a Container named +`test-container`. Similarly, the other environment variables get their values +from Container fields. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-envars-container.yaml +``` + +Verify that the Container in the Pod is running: + +``` +kubectl get pods +``` + +View the Container's logs: + +``` +kubectl logs dapi-envars-resourcefieldref +``` + +The output shows the values of selected environment variables: + +``` +1 +1 +33554432 +67108864 +``` + +{% endcapture %} + +{% capture whatsnext %} + +* [Defining Environment Variables for a Container](/docs/tasks/configure-pod-container/define-environment-variable-container/) +* [PodSpec](/docs/resources-reference/v1.5/#podspec-v1) +* [Container](/docs/resources-reference/v1.5/#container-v1) +* [EnvVar](/docs/resources-reference/v1.5/#envvar-v1) +* [EnvVarSource](/docs/resources-reference/v1.5/#envvarsource-v1) +* [ObjectFieldSelector](/docs/resources-reference/v1.5/#objectfieldselector-v1) +* [ResourceFieldSelector](/docs/resources-reference/v1.5/#resourcefieldselector-v1) + +{% endcapture %} + + +{% include templates/task.md %} From 10d678bfc81cecf533e981529f700a0c0859af5c Mon Sep 17 00:00:00 2001 From: Jeff Mendoza Date: Fri, 10 Feb 2017 13:15:15 -0800 Subject: [PATCH 170/407] Fix Travis test builds' vendoring workaround. --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index e527e16dbc..09a28b78cb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,6 +13,7 @@ install: - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apiserver - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/client-go +- rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/sample-apiserver - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ From b51f5a2e2c276884eec36f0a82eef4c76c9a286e Mon Sep 17 00:00:00 2001 From: Jasmine Hegman Date: Wed, 8 Feb 2017 17:22:46 -0700 Subject: [PATCH 171/407] Fixed a tiny typo Kubernets -> Kubernetes --- docs/user-guide/logging/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/logging/overview.md b/docs/user-guide/logging/overview.md index a9f00471cc..42b9087172 100644 --- a/docs/user-guide/logging/overview.md +++ b/docs/user-guide/logging/overview.md @@ -68,7 +68,7 @@ When you run [`kubectl logs`](/docs/user-guide/kubectl/kubectl_logs), as in the There are two types of system components: those that run in a container and those that do not run in a container. For example: -* The Kubernets scheduler and kube-proxy run in a container. +* The Kubernetes scheduler and kube-proxy run in a container. * The kubelet and container runtime, for example Docker, do not run in containers. On machines with systemd, the kubelet and container runtime write to journald. If From 2f351db5bbfeec8e226fc2da8a0b6a9cffd8a7df Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Wimsingues?= Date: Wed, 8 Feb 2017 11:28:26 +0100 Subject: [PATCH 172/407] redundnacy => redundancy --- docs/concepts/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 3f131c6358..93e4533caf 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -44,7 +44,7 @@ For example, when you use the Kubernetes API to create a Deployment object, you The Kubernetes master is responsible for maintaining the desired state for your cluster. When you interact with Kubernetes, such as by using the `kubectl` command-line interface, you're communicating with your cluster's Kubernetes master. -> The "master" refers to a collection of processes managing the cluster state. Typically these processes are all run on a single node in the cluster, and this node is also referred to as the master. The master can also be replicated for availability and redundnacy. +> The "master" refers to a collection of processes managing the cluster state. Typically these processes are all run on a single node in the cluster, and this node is also referred to as the master. The master can also be replicated for availability and redundancy. ### Kubernetes Nodes From 715ccb48e8f34614bc9d15cc077827bed2da53a6 Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 13 Feb 2017 11:08:33 +0800 Subject: [PATCH 173/407] Update cron-jobs.md --- docs/user-guide/cron-jobs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/cron-jobs.md b/docs/user-guide/cron-jobs.md index 55b85adf46..34402d5568 100644 --- a/docs/user-guide/cron-jobs.md +++ b/docs/user-guide/cron-jobs.md @@ -19,10 +19,10 @@ A _Cron Job_ manages time based [Jobs](/docs/user-guide/jobs/), namely: One CronJob object is like one line of a _crontab_ (cron table) file. It runs a job periodically on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) format. -**Note:**: The question mark (`?`) in the schedule has the same meaning as an asterisk `*`, +**Note:** The question mark (`?`) in the schedule has the same meaning as an asterisk `*`, that is, it stands for any of available value for a given field. -**Note:**: ScheduledJob resource was introduced in Kubernetes version 1.4, but starting +**Note:** ScheduledJob resource was introduced in Kubernetes version 1.4, but starting from version 1.5 its current name is CronJob. A typical use case is: From 822d00df62dbd39e8cab1f6265b6298e4d4d513c Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 13 Feb 2017 10:40:57 +0800 Subject: [PATCH 174/407] Update index.md --- docs/user-guide/jobs/expansions/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/expansions/index.md b/docs/user-guide/jobs/expansions/index.md index 767ac65215..f401574519 100644 --- a/docs/user-guide/jobs/expansions/index.md +++ b/docs/user-guide/jobs/expansions/index.md @@ -111,7 +111,7 @@ In the first example, each instance of the template had one parameter, and that used as a label. However label keys are limited in [what characters they can contain](/docs/user-guide/labels/#syntax-and-character-set). -This slightly more complex example uses a the jinja2 template language to generate our objects. +This slightly more complex example uses the jinja2 template language to generate our objects. We will use a one-line python script to convert the template to a file. First, copy and paste the following template of a Job object, into a file called `job.yaml.jinja2`: @@ -179,6 +179,7 @@ cat job.yaml.jinja2 | render_template | kubectl create -f - ## Alternatives If you have a large number of job objects, you may find that: + - even using labels, managing so many Job objects is cumbersome. - You exceed resource quota when creating all the Jobs at once, and do not want to wait to create them incrementally. From 58322206bd4a8ccc635e824ca6d2eb853de96371 Mon Sep 17 00:00:00 2001 From: PassionForPie Date: Sun, 12 Feb 2017 23:57:39 +0000 Subject: [PATCH 175/407] Minor grammatical fix See https://en.wiktionary.org/wiki/greengrocer's_apostrophe --- docs/tutorials/kubernetes-basics/expose-intro.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kubernetes-basics/expose-intro.html b/docs/tutorials/kubernetes-basics/expose-intro.html index 9ee7a4117a..ad4394b0e4 100644 --- a/docs/tutorials/kubernetes-basics/expose-intro.html +++ b/docs/tutorials/kubernetes-basics/expose-intro.html @@ -28,7 +28,7 @@ title: Using a Service to Expose Your App

Kubernetes Services

-

While Pods do have their own unique IP across the cluster, those IP's are not exposed outside Kubernetes. Taking into account that over time Pods may be terminated, deleted or replaced by other Pods, we need a way to let other Pods and applications automatically discover each other. Kubernetes addresses this by grouping Pods in Services. A Kubernetes Service is an abstraction layer which defines a logical set of Pods and enables external traffic exposure, load balancing and service discovery for those Pods.

+

While Pods do have their own unique IP across the cluster, those IPs are not exposed outside Kubernetes. Taking into account that over time Pods may be terminated, deleted or replaced by other Pods, we need a way to let other Pods and applications automatically discover each other. Kubernetes addresses this by grouping Pods in Services. A Kubernetes Service is an abstraction layer which defines a logical set of Pods and enables external traffic exposure, load balancing and service discovery for those Pods.

This abstraction will allow us to expose Pods to traffic originating from outside the cluster. Services have their own unique cluster-private IP address and expose a port to receive traffic. If you choose to expose the service outside the cluster, the options are:

    From 9197fb582278638eab4ebe644ca402166f595792 Mon Sep 17 00:00:00 2001 From: scjane Date: Fri, 10 Feb 2017 18:32:47 +0800 Subject: [PATCH 176/407] Update jobs.md --- docs/user-guide/jobs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index cc2f9d38c3..d48b856ae3 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -63,7 +63,7 @@ Events: To view completed pods of a job, use `kubectl get pods --show-all`. The `--show-all` will show completed pods too. -To list all the pods that belong to job in a machine readable form, you can use a command like this: +To list all the pods that belong to a job in a machine readable form, you can use a command like this: ```shell $ pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}) @@ -100,7 +100,7 @@ the same schema as a [pod](/docs/user-guide/pods), except it is nested and does In addition to required fields for a Pod, a pod template in a job must specify appropriate labels (see [pod selector](#pod-selector)) and an appropriate restart policy. -Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` are allowed. +Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` is allowed. ### Pod Selector From c3b483c49f36ee24bfeaa6a9a4d77ef775b3d572 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Wimsingues?= Date: Wed, 8 Feb 2017 11:05:16 +0100 Subject: [PATCH 177/407] Update overview.md comamnd => command --- docs/concepts/abstractions/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/abstractions/overview.md index dc2e8cf7ab..1a84d74534 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/abstractions/overview.md @@ -17,7 +17,7 @@ This page explains how Kubernetes objects are represented in the Kubernetes API, A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system will constantly work to ensure that that object exists. By creating an object, you're effectively telling the Kubernetes system what you want your cluster's workload to look like; this is your cluster's **desired state**. -To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the [Kubernetes API](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md). When you use the `kubectl` comamnd-line interface, for example, the CLI makes the necessary Kubernetes API calls for you; you can also use the Kubernetes API directly in your own programs. Kubernetes currently provides a `golang` [client library](https://github.com/kubernetes/client-go) for this purpose, and other language libraries (such as [Python](https://github.com/kubernetes-incubator/client-python)) are being developed. +To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the [Kubernetes API](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md). When you use the `kubectl` command-line interface, for example, the CLI makes the necessary Kubernetes API calls for you; you can also use the Kubernetes API directly in your own programs. Kubernetes currently provides a `golang` [client library](https://github.com/kubernetes/client-go) for this purpose, and other language libraries (such as [Python](https://github.com/kubernetes-incubator/client-python)) are being developed. ### Object Spec and Status From eccf13d47b847485de34e7c2547a8fe7549ece63 Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Fri, 10 Feb 2017 11:02:18 +0800 Subject: [PATCH 178/407] Update kubefed.md --- docs/admin/federation/kubefed.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/admin/federation/kubefed.md b/docs/admin/federation/kubefed.md index 695edf7bee..4af8db60b5 100644 --- a/docs/admin/federation/kubefed.md +++ b/docs/admin/federation/kubefed.md @@ -36,11 +36,11 @@ in your `$PATH` and set the executable permission on those binaries. Note: The URL in the curl command below downloads the binaries for Linux amd64. If you are on a different platform, please use the URL for the binaries appropriate for your platform. You can find the list -of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-3). +of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1). ```shell -curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.0/kubernetes-client-linux-amd64.tar.gz +curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.2/kubernetes-client-linux-amd64.tar.gz tar -xzvf kubernetes-client-linux-amd64.tar.gz sudo cp kubernetes/client/bin/kubefed /usr/local/bin sudo chmod +x /usr/local/bin/kubefed @@ -78,7 +78,7 @@ control plane. ## Deploying a federation control plane. -"To deploy a federation control plane on your host cluster, run +To deploy a federation control plane on your host cluster, run `kubefed init` command. When you use `kubefed init`, you must provide the following: @@ -94,7 +94,7 @@ domain suffix `example.com`: kubefed init fellowship --host-cluster-context=rivendell --dns-zone-name="example.com" ``` -The domain suffix you specify in `--dns-zone-name` must be an existing +The domain suffix specified in `--dns-zone-name` must be an existing domain that you control, and that is programmable by your DNS provider. `kubefed init` sets up the federation control plane in the host From 0123149feb11612d5b6a3806be2d373f55b490fb Mon Sep 17 00:00:00 2001 From: Ahmet Alp Balkan Date: Mon, 13 Feb 2017 09:21:14 -0800 Subject: [PATCH 179/407] Add SIG meetings calendar to /community/ Signed-off-by: Ahmet Alp Balkan --- community/index.html | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/community/index.html b/community/index.html index 201c70a776..5843b4829e 100644 --- a/community/index.html +++ b/community/index.html @@ -27,6 +27,13 @@ cid: community lists of SIGs, from AWS and Openstack to Big Data and Scalability, there's a place for you to contribute and instructions for forming a new SIG if your special interest isn't covered (yet).

    + +

    As a member of the Kubernetes community, you are welcome to join any of the SIG meetings + you are interested in. No registration required.

    +
    + +

Events

From 34ff10be2fbc78dcd8fd71385d12e16009d02809 Mon Sep 17 00:00:00 2001 From: Jason Lock Date: Fri, 30 Dec 2016 09:31:06 -0800 Subject: [PATCH 180/407] Increase font-size and line-height CSS Properties Increase the font-size and line-height CSS properties for the paragraph and list items within the documentation body content. Adjusting these properties will make the information a little easier to read. --- _sass/_base.sass | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/_sass/_base.sass b/_sass/_base.sass index 72557bb983..3149232bec 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -816,9 +816,9 @@ dd font-weight: 500 p - font-size: 14px + font-size: 16px font-weight: 300 - line-height: 1.25em + line-height: 1.75em p + p margin-top: 10px @@ -894,6 +894,7 @@ dd li margin-bottom: 0.75em + font-size: 16px table width: 100% From 896c2ddd184112186bff9a31ab86b8d05b348ea0 Mon Sep 17 00:00:00 2001 From: Jason Lock Date: Fri, 30 Dec 2016 09:58:26 -0800 Subject: [PATCH 181/407] Add line-height to List Item Add line-height to the body content list items. The same line-height as the paragraph. --- _sass/_base.sass | 1 + 1 file changed, 1 insertion(+) diff --git a/_sass/_base.sass b/_sass/_base.sass index 3149232bec..4b9e051af4 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -895,6 +895,7 @@ dd li margin-bottom: 0.75em font-size: 16px + line-height: 1.75em table width: 100% From 809c1b8ebe32b973bfa9646eb26fa0556874b5ab Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Mon, 13 Feb 2017 10:05:51 -0800 Subject: [PATCH 182/407] reimplement PR #2217 socks shop port changed --- docs/getting-started-guides/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index a4e5679ffb..a2d4e1b834 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -269,7 +269,7 @@ It takes several minutes to download and start all the containers, watch the out Then go to the IP address of your cluster's master node in your browser, and specify the given port. So for example, `http://:`. -In the example above, this was `31869`, but it is a different port for you. +In the example above, this was `30001`, but it is a different port for you. If there is a firewall, make sure it exposes this port to the internet before you try to access it. From 49dea6f23b3244e3c4d54f8e400ec066e4deb4df Mon Sep 17 00:00:00 2001 From: samiam Date: Mon, 13 Feb 2017 11:18:16 -0500 Subject: [PATCH 183/407] Update kubectl_completion.md Quote shell commands to improve format --- docs/user-guide/kubectl/kubectl_completion.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/kubectl/kubectl_completion.md b/docs/user-guide/kubectl/kubectl_completion.md index 125b7791cd..f870bbd482 100644 --- a/docs/user-guide/kubectl/kubectl_completion.md +++ b/docs/user-guide/kubectl/kubectl_completion.md @@ -11,17 +11,19 @@ Output shell completion code for the given shell (bash or zsh). This command prints shell code which must be evaluation to provide interactive completion of kubectl commands. - $ source <(kubectl completion bash) + `$ source <(kubectl completion bash)` will load the kubectl completion code for bash. Note that this depends on the bash-completion framework. It must be sourced before sourcing the kubectl completion, e.g. on the Mac: +``` $ brew install bash-completion $ source $(brew --prefix)/etc/bash_completion $ source <(kubectl completion bash) +``` If you use zsh [1], the following will load kubectl zsh completion: - $ source <(kubectl completion zsh) + `$ source <(kubectl completion zsh)` [1] zsh completions are only supported in versions of zsh >= 5.2 From 3808e4ccd786b78f4411cf61e2153113f873c974 Mon Sep 17 00:00:00 2001 From: Andreas Kohn Date: Mon, 13 Feb 2017 15:42:53 +0100 Subject: [PATCH 184/407] Improve grammar --- docs/user-guide/jsonpath.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/jsonpath.md b/docs/user-guide/jsonpath.md index 74ce7bbc77..54c2780da8 100644 --- a/docs/user-guide/jsonpath.md +++ b/docs/user-guide/jsonpath.md @@ -6,8 +6,8 @@ JSONPath template is composed of JSONPath expressions enclosed by {}. And we add three functions in addition to the original JSONPath syntax: 1. The `$` operator is optional since the expression always start from the root object by default. -2. We can use `""` to quote text inside JSONPath expression. -3. We can use `range` operator to iterate list. +2. We can use `""` to quote text inside JSONPath expressions. +3. We can use `range` operator to iterate lists. The result object is printed as its String() function. From 3137a4926a2c77d229f14a50c0e5fd40ef720d5d Mon Sep 17 00:00:00 2001 From: scjane Date: Mon, 13 Feb 2017 17:10:28 +0800 Subject: [PATCH 185/407] Update index.md --- docs/user-guide/jobs/work-queue-2/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-2/index.md b/docs/user-guide/jobs/work-queue-2/index.md index 5e2e8a03ec..4fd806d392 100644 --- a/docs/user-guide/jobs/work-queue-2/index.md +++ b/docs/user-guide/jobs/work-queue-2/index.md @@ -47,7 +47,7 @@ If you're not working from the source tree, you could also download [`redis-pod. ## Filling the Queue with tasks -Now lets fill the queue with some "tasks". In our example, our tasks are just strings to be +Now let's fill the queue with some "tasks". In our example, our tasks are just strings to be printed. Start a temporary interactive pod for running the Redis CLI From 01cea0c7139002c506f63e7f6c12ea54a088388b Mon Sep 17 00:00:00 2001 From: caiyixiang Date: Mon, 13 Feb 2017 16:54:22 +0800 Subject: [PATCH 186/407] Update source-ip.md --- docs/tutorials/services/source-ip.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index 56daa8b345..e1eac87c4e 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -20,8 +20,8 @@ of Services, and how you can toggle this behavior according to your needs. This document makes use of the following terms: * [NAT](https://en.wikipedia.org/wiki/Network_address_translation): network address translation -* [Source NAT](/docs/user-guide/services/#ips-and-vips): replacing the source IP on a packet, usually with a node's IP -* [Destination NAT](/docs/user-guide/services/#ips-and-vips): replacing the destination IP on a packet, usually with a pod IP +* [Source NAT](https://en.wikipedia.org/wiki/Network_address_translation#SNAT): replacing the source IP on a packet, usually with a node's IP +* [Destination NAT](https://en.wikipedia.org/wiki/Network_address_translation#DNAT): replacing the destination IP on a packet, usually with a pod IP * [VIP](/docs/user-guide/services/#ips-and-vips): a virtual IP, such as the one assigned to every Kubernetes Service * [Kube-proxy](/docs/user-guide/services/#virtual-ips-and-service-proxies): a network daemon that orchestrates Service VIP management on every node From a5fe6ae18b2ecda9947c2c7c04a6785c357439cc Mon Sep 17 00:00:00 2001 From: foxyriver Date: Mon, 13 Feb 2017 15:24:26 +0800 Subject: [PATCH 187/407] fix parallelism result According to job.yaml, the parallelism is 2 --- docs/user-guide/jobs/work-queue-1/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-1/index.md b/docs/user-guide/jobs/work-queue-1/index.md index 4ce1ffb667..f926f4211f 100644 --- a/docs/user-guide/jobs/work-queue-1/index.md +++ b/docs/user-guide/jobs/work-queue-1/index.md @@ -227,7 +227,7 @@ Name: job-wq-1 Namespace: default Image(s): gcr.io/causal-jigsaw-637/job-wq-1 Selector: app in (job-wq-1) -Parallelism: 4 +Parallelism: 2 Completions: 8 Labels: app=job-wq-1 Pods Statuses: 0 Running / 8 Succeeded / 0 Failed From 068818e1c68596bcfd9262fad1e6434778d53084 Mon Sep 17 00:00:00 2001 From: Christoph Blecker Date: Thu, 9 Feb 2017 17:41:08 -0800 Subject: [PATCH 188/407] Fix link to StatefulSet docs from Pods page --- docs/user-guide/pods/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user-guide/pods/index.md b/docs/user-guide/pods/index.md index 321900c948..47e2dcaf3b 100644 --- a/docs/user-guide/pods/index.md +++ b/docs/user-guide/pods/index.md @@ -1,5 +1,5 @@ --- -assignees: +assignees: title: Pods --- @@ -60,7 +60,7 @@ related thing (e.g. volume) is also destroyed and created anew. ![pod diagram](/images/docs/pod.svg){: style="max-width: 50%" } -*A multi-container pod that contains a file puller and a +*A multi-container pod that contains a file puller and a web server that uses a persistent volume for shared storage between the containers.* ## Motivation for pods @@ -150,7 +150,7 @@ Pod is exposed as a primitive in order to facilitate: * clean composition of Kubelet-level functionality with cluster-level functionality — Kubelet is effectively the "pod controller" * high-availability applications, which will expect pods to be replaced in advance of their termination and certainly in advance of deletion, such as in the case of planned evictions, image prefetching, or live pod migration [#3949](http://issue.k8s.io/3949) -There is new first-class support for stateful pods with the [StatefulSet](/docs/concepts/controllers/statefulsets/) controller (currently in beta). The feature was alpha in 1.4 and was called [PetSet](/docs/user-guide/petset/). For prior versions of Kubernetes, best practice for having stateful pods is to create a replication controller with `replicas` equal to `1` and a corresponding service, see [this MySQL deployment example](/docs/tutorials/stateful-application/run-stateful-application/). +There is new first-class support for stateful pods with the [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets/) controller (currently in beta). The feature was alpha in 1.4 and was called [PetSet](/docs/user-guide/petset/). For prior versions of Kubernetes, best practice for having stateful pods is to create a replication controller with `replicas` equal to `1` and a corresponding service, see [this MySQL deployment example](/docs/tutorials/stateful-application/run-stateful-application/). ## Termination of Pods @@ -168,7 +168,7 @@ An example flow: 6. When the grace period expires, any processes still running in the Pod are killed with SIGKILL. 7. The Kubelet will finish deleting the Pod on the API server by setting grace period 0 (immediate deletion). The Pod disappears from the API and is no longer visible from the client. -By default, all deletes are graceful within 30 seconds. The `kubectl delete` command supports the `--grace-period=` option which allows a user to override the default and specify their own value. The value `0` [force deletes](/docs/user-guide/pods/#force-termination-of-pods) the pod. In kubectl version >= 1.5, you must specify an additional flag `--force` along with `--grace-period=0` in order to perform force deletions. +By default, all deletes are graceful within 30 seconds. The `kubectl delete` command supports the `--grace-period=` option which allows a user to override the default and specify their own value. The value `0` [force deletes](/docs/user-guide/pods/#force-termination-of-pods) the pod. In kubectl version >= 1.5, you must specify an additional flag `--force` along with `--grace-period=0` in order to perform force deletions. ### Force deletion of pods From f1ffbe959903983532d13d1c0eb84c6cd310fcad Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Fri, 10 Feb 2017 11:47:39 +0800 Subject: [PATCH 189/407] replace 30 seconds with 35 seconds replace 30 seconds with 35 seconds --- .../configure-liveness-readiness-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md index 8c877ac16c..d168ceb211 100644 --- a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md +++ b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md @@ -84,7 +84,7 @@ FirstSeen LastSeen Count From SubobjectPath Type 23s 23s 1 {kubelet worker0} spec.containers{liveness} Normal Started Started container with docker id 86849c15382e ``` -After 30 seconds, view the Pod events again: +After 35 seconds, view the Pod events again: ```shell kubectl describe pod liveness-exec From a35d5d3320f88f36f0017eb6e4662a75d97496f7 Mon Sep 17 00:00:00 2001 From: Ahmet Alp Balkan Date: Fri, 10 Feb 2017 14:48:21 -0800 Subject: [PATCH 190/407] Move "User Guide" up in the sidebar So that it is above "Accessing the Cluster" article which describes more advanced topics and does not look coherent when it is read linearly. Signed-off-by: Ahmet Alp Balkan --- _data/guides.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data/guides.yml b/_data/guides.yml index 511ece0885..98ef2db801 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -5,6 +5,8 @@ toc: - docs/whatisk8s.md +- docs/user-guide/index.md + - title: Accessing the Cluster section: - docs/user-guide/prereqs.md @@ -12,8 +14,6 @@ toc: - docs/user-guide/sharing-clusters.md - docs/user-guide/kubeconfig-file.md -- docs/user-guide/index.md - - docs/user-guide/ui.md - title: Workload Deployment and Management From e79c6662f52c1862db9b4736fdb9a573d179c0bf Mon Sep 17 00:00:00 2001 From: Jesse Johnston Date: Thu, 9 Feb 2017 09:14:26 -0800 Subject: [PATCH 191/407] Update overview.md Update link to Kubernetes API Conventions to point to current document location. --- docs/concepts/abstractions/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/abstractions/overview.md index 1a84d74534..712ad0f99f 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/abstractions/overview.md @@ -25,7 +25,7 @@ Every Kubernetes object includes two nested object fields that govern the object For example, a Kubernetes Deployment is an object that can represent an application running on your cluster. When you create the Deployment, you might set the Deployment spec to specify that you want three replicas of the application to be running. The Kubernetes system reads the Deployment spec and starts three instances of your desired application--updating the status to match your spec. If any of those instances should fail (a status change), the Kubernetes system responds to the difference between spec and status by making a correction--in this case, starting a replacement instance. -For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#spec-and-status). +For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md). ### Describing a Kubernetes Object From fe9a0c14476e3872c19d6151424d801ea8792c60 Mon Sep 17 00:00:00 2001 From: Erik Bean Date: Mon, 13 Feb 2017 13:25:40 -0800 Subject: [PATCH 192/407] Fix unmatched closing paren End of line 60 had a closing parenthesis, with no opening one to match. Probably left over from a previous version of the document --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index f8b9cf2f72..5c364c5376 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -57,7 +57,7 @@ Pods do not, by themselves, self-heal. If a Pod is scheduled to a Node that fail ### Pods and Controllers -A Controller can create and manage multiple Pods for you, handling replication and rollout and providing self-healing capabilities at cluster scope. For example, if a Node fails, the Controller might automatically replace the Pod by scheduling an identical replacement on a different Node). +A Controller can create and manage multiple Pods for you, handling replication and rollout and providing self-healing capabilities at cluster scope. For example, if a Node fails, the Controller might automatically replace the Pod by scheduling an identical replacement on a different Node. Some examples of Controllers that contain one or more pods include: From ec8fd54d0fa8a91b463907236eed6a37c22f198e Mon Sep 17 00:00:00 2001 From: Albert Vaca Date: Mon, 13 Feb 2017 20:58:10 +0100 Subject: [PATCH 193/407] Updated dead links --- docs/user-guide/ingress.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ingress.md b/docs/user-guide/ingress.md index 809a025c57..009fd13f45 100644 --- a/docs/user-guide/ingress.md +++ b/docs/user-guide/ingress.md @@ -220,7 +220,7 @@ Note that there is a gap between TLS features supported by various Ingress contr An Ingress controller is bootstrapped with some loadbalancing policy settings that it applies to all Ingress, such as the loadbalancing algorithm, backend weight scheme etc. More advanced loadbalancing concepts (e.g.: persistent sessions, dynamic weights) are not yet exposed through the Ingress. You can still get these features through the [service loadbalancer](https://github.com/kubernetes/contrib/tree/master/service-loadbalancer). With time, we plan to distill loadbalancing patterns that are applicable cross platform into the Ingress resource. -It's also worth noting that even though health checks are not exposed directly through the Ingress, there exist parallel concepts in Kubernetes such as [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) which allow you to achieve the same end result. Please review the controller specific docs to see how they handle health checks ([nginx](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/README.md), [GCE](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/README.md#health-checks)). +It's also worth noting that even though health checks are not exposed directly through the Ingress, there exist parallel concepts in Kubernetes such as [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) which allow you to achieve the same end result. Please review the controller specific docs to see how they handle health checks ([nginx](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md), [GCE](https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md#health-checks)). ## Updating an Ingress From 2a597e378960b2256536feb3e8944cda4c379ced Mon Sep 17 00:00:00 2001 From: James Gregory Date: Tue, 14 Feb 2017 11:01:07 +1100 Subject: [PATCH 194/407] Update links to ingress repository Ingress moved out of contrib to kubernetes/ingress. --- docs/user-guide/ingress.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user-guide/ingress.md b/docs/user-guide/ingress.md index 009fd13f45..4628d5050d 100644 --- a/docs/user-guide/ingress.md +++ b/docs/user-guide/ingress.md @@ -81,11 +81,11 @@ __Global Parameters__: For the sake of simplicity the example Ingress has no glo ## Ingress controllers -In order for the Ingress resource to work, the cluster must have an Ingress controller running. This is unlike other types of controllers, which typically run as part of the `kube-controller-manager` binary, and which are typically started automatically as part of cluster creation. You need to choose the ingress controller implementation that is the best fit for your cluster, or implement one. Examples and instructions can be found [here](https://github.com/kubernetes/contrib/tree/master/ingress/controllers). +In order for the Ingress resource to work, the cluster must have an Ingress controller running. This is unlike other types of controllers, which typically run as part of the `kube-controller-manager` binary, and which are typically started automatically as part of cluster creation. You need to choose the ingress controller implementation that is the best fit for your cluster, or implement one. Examples and instructions can be found [here](https://github.com/kubernetes/ingress/tree/master/controllers). ## Before you begin -The following document describes a set of cross platform features exposed through the Ingress resource. Ideally, all Ingress controllers should fulfill this specification, but we're not there yet. The docs for the GCE and nginx controllers are [here](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/README.md) and [here](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/README.md) respectively. **Make sure you review controller specific docs so you understand the caveats of each one**. +The following document describes a set of cross platform features exposed through the Ingress resource. Ideally, all Ingress controllers should fulfill this specification, but we're not there yet. The docs for the GCE and nginx controllers are [here](https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md) and [here](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md) respectively. **Make sure you review controller specific docs so you understand the caveats of each one**. ## Types of Ingress @@ -214,7 +214,7 @@ spec: servicePort: 80 ``` -Note that there is a gap between TLS features supported by various Ingress controllers. Please refer to documentation on [nginx](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#https), [GCE](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/gce#tls), or any other platform specific Ingress controller to understand how TLS works in your environment. +Note that there is a gap between TLS features supported by various Ingress controllers. Please refer to documentation on [nginx](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md#https), [GCE](https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md#tls), or any other platform specific Ingress controller to understand how TLS works in your environment. ### Loadbalancing @@ -282,7 +282,7 @@ Techniques for spreading traffic across failure domains differs between cloud pr * Combining L4 and L7 Ingress * More Ingress controllers -Please track the [L7 and Ingress proposal](https://github.com/kubernetes/kubernetes/pull/12827) for more details on the evolution of the resource, and the [Ingress sub-repository](https://github.com/kubernetes/contrib/tree/master/ingress) for more details on the evolution of various Ingress controllers. +Please track the [L7 and Ingress proposal](https://github.com/kubernetes/kubernetes/pull/12827) for more details on the evolution of the resource, and the [Ingress repository](https://github.com/kubernetes/ingress/tree/master) for more details on the evolution of various Ingress controllers. ## Alternatives From 5d0f273f348ffa8e5aac00e06dd68f6013328618 Mon Sep 17 00:00:00 2001 From: Zhou Tao Date: Sat, 11 Feb 2017 12:03:38 +0800 Subject: [PATCH 195/407] doc(kubeadm.md) - change base64 decode option to '--decode' `base64 -D` will lead to error on Ubuntu: "base64: invalid option -- 'D'" --- docs/getting-started-guides/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index a2d4e1b834..5fc815ea9c 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -352,7 +352,7 @@ Please note: `kubeadm` is a work in progress and these limitations will be addre 1. There is no built-in way of fetching the token easily once the cluster is up and running, but here is a `kubectl` command you can copy and paste that will print out the token for you: ```console - # kubectl -n kube-system get secret clusterinfo -o yaml | grep token-map | awk '{print $2}' | base64 -D | sed "s|{||g;s|}||g;s|:|.|g;s/\"//g;" | xargs echo + # kubectl -n kube-system get secret clusterinfo -o yaml | grep token-map | awk '{print $2}' | base64 --decode | sed "s|{||g;s|}||g;s|:|.|g;s/\"//g;" | xargs echo ``` 1. If you are using VirtualBox (directly or via Vagrant), you will need to ensure that `hostname -i` returns a routable IP address (i.e. one on the second network interface, not the first one). From 2c9b093c37a09c877cd90ca0ee24b31347fcecf6 Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Mon, 13 Feb 2017 16:39:37 +0800 Subject: [PATCH 196/407] Update index.md --- docs/user-guide/persistent-volumes/index.md | 34 ++++++++++----------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index 3ecc8ea99f..8b1178e45d 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -172,23 +172,23 @@ In the CLI, the access modes are abbreviated to: | Volume Plugin | ReadWriteOnce| ReadOnlyMany| ReadWriteMany| | :--- | :---: | :---: | :---: | -| AWSElasticBlockStore | x | - | - | -| AzureFile | x | x | x | -| AzureDisk | x | - | - | -| CephFS | x | x | x | -| Cinder | x | - | - | -| FC | x | x | - | -| FlexVolume | x | x | - | -| Flocker | x | - | - | -| GCEPersistentDisk | x | x | - | -| Glusterfs | x | x | x | -| HostPath | x | - | - | -| iSCSI | x | x | - | -| PhotonPersistentDisk | x | - | - | -| Quobyte | x | x | x | -| NFS | x | x | x | -| RBD | x | x | - | -| VsphereVolume | x | - | - | +| AWSElasticBlockStore | ✓ | - | - | +| AzureFile | ✓ | ✓ | ✓ | +| AzureDisk | ✓ | - | - | +| CephFS | ✓ | ✓ | ✓ | +| Cinder | ✓ | - | - | +| FC | ✓ | ✓ | - | +| FlexVolume | ✓ | ✓ | - | +| Flocker | ✓ | - | - | +| GCEPersistentDisk | ✓ | ✓ | - | +| Glusterfs | ✓ | ✓ | ✓ | +| HostPath | ✓ | - | - | +| iSCSI | ✓ | ✓ | - | +| PhotonPersistentDisk | ✓ | - | - | +| Quobyte | ✓ | ✓ | ✓ | +| NFS | ✓ | ✓ | ✓ | +| RBD | ✓ | ✓ | - | +| VsphereVolume | ✓ | - | - | ### Class From 07c4b9f6f8379d2e65d82f2481ae343e188656ae Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 14 Feb 2017 14:25:31 +0800 Subject: [PATCH 197/407] fix typo --- docs/user-guide/kubeconfig-file.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubeconfig-file.md b/docs/user-guide/kubeconfig-file.md index b4d7425127..93877f8a84 100644 --- a/docs/user-guide/kubeconfig-file.md +++ b/docs/user-guide/kubeconfig-file.md @@ -305,7 +305,7 @@ $ kubectl config use-context federal-context ### Final notes for tying it all together -So, tying this all together, a quick start to creating your own kubeconfig file: +So, tying this all together, a quick start to create your own kubeconfig file: - Take a good look and understand how your api-server is being launched: You need to know YOUR security requirements and policies before you can design a kubeconfig file for convenient authentication. From a334d932e1b860c5c89ebcda85d2ee75b04c0b8c Mon Sep 17 00:00:00 2001 From: yupengzte Date: Tue, 14 Feb 2017 10:36:00 +0800 Subject: [PATCH 198/407] fix typo Signed-off-by: yupengzte --- docs/deprecation-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecation-policy.md b/docs/deprecation-policy.md index c51326bbc5..cdc0bb0102 100644 --- a/docs/deprecation-policy.md +++ b/docs/deprecation-policy.md @@ -233,7 +233,7 @@ after their announced deprecation for no less than:** * **Beta: 3 months or 1 release (whichever is longer)** * **Alpha: 0 releases** -**Rule #6: Deprecated CLI elements must emit warnings (optionally disableable) +**Rule #6: Deprecated CLI elements must emit warnings (optionally disable) when used.** ## Deprecating a feature or behavior From 5a24d6d68125254d2fbc58c67b11efb469c98fca Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Tue, 14 Feb 2017 15:18:45 +0000 Subject: [PATCH 199/407] Fix the standard storageClass for GCE As you can see here: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go#L117-L121 The default is not ssd --- docs/user-guide/persistent-volumes/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index 8b1178e45d..4508a1a25f 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -396,7 +396,7 @@ parameters: zone: us-central1-a ``` -* `type`: `pd-standard` or `pd-ssd`. Default: `pd-ssd` +* `type`: `pd-standard` or `pd-ssd`. Default: `pd-standard` * `zone`: GCE zone. If not specified, a random zone in the same region as controller-manager will be chosen. #### Glusterfs From 87f0e317abd84abbfc94880ee76eb8d55fee2cff Mon Sep 17 00:00:00 2001 From: scjane Date: Tue, 14 Feb 2017 15:03:43 +0800 Subject: [PATCH 200/407] Update multiple-schedulers.md Similar to #2518. But I think an unordered list can be used here :) --- docs/admin/multiple-schedulers.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/admin/multiple-schedulers.md b/docs/admin/multiple-schedulers.md index 3e3b4d270b..c20e129b19 100644 --- a/docs/admin/multiple-schedulers.md +++ b/docs/admin/multiple-schedulers.md @@ -95,7 +95,7 @@ Now that our second scheduler is running, let's create some pods, and direct the scheduler in that pod spec. Let's look at three examples. -1. Pod spec without any scheduler name +- Pod spec without any scheduler name {% include code.html language="yaml" file="multiple-schedulers/pod1.yaml" ghlink="/docs/admin/multiple-schedulers/pod1.yaml" %} @@ -108,7 +108,7 @@ scheduler in that pod spec. Let's look at three examples. kubectl create -f pod1.yaml ``` -2. Pod spec with `default-scheduler` +- Pod spec with `default-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod2.yaml" ghlink="/docs/admin/multiple-schedulers/pod2.yaml" %} @@ -121,7 +121,7 @@ scheduler in that pod spec. Let's look at three examples. kubectl create -f pod2.yaml ``` -3. Pod spec with `my-scheduler` +- Pod spec with `my-scheduler` {% include code.html language="yaml" file="multiple-schedulers/pod3.yaml" ghlink="/docs/admin/multiple-schedulers/pod3.yaml" %} From 886f15554f000f683c17bdc2c681c79c89da521a Mon Sep 17 00:00:00 2001 From: Fabrizio Milo Date: Tue, 14 Feb 2017 09:29:08 -0800 Subject: [PATCH 201/407] Update pod.md --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index 5c364c5376..59a7beaee5 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -27,7 +27,7 @@ The [Kubernetes Blog](http://blog.kubernetes.io) has some additional information * [The Distributed System Toolkit: Patterns for Composite Containers](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html) * [Container Design Patterns](http://blog.kubernetes.io/2016/06/container-design-patterns.html) -Each Pod is meant to run a single instance of a given application. If you want to scale your application horizontally (e.g., run muliple instances), you should use multiple Pods, one for each instance. In Kubernetes, this is generally referred to as _replication_. Replicated Pods are usually created and managed as a group by an abstraction called a Controller. See [Pods and Controllers](#pods-and-controllers) for more information. +Each Pod is meant to run a single instance of a given application. If you want to scale your application horizontally (e.g., run multiple instances), you should use multiple Pods, one for each instance. In Kubernetes, this is generally referred to as _replication_. Replicated Pods are usually created and managed as a group by an abstraction called a Controller. See [Pods and Controllers](#pods-and-controllers) for more information. ### How Pods Manage Multiple Containers From 3a5b4c74d3844475c0f80e74f4e66dd5f184e4c6 Mon Sep 17 00:00:00 2001 From: Fabrizio Milo Date: Tue, 14 Feb 2017 09:20:59 -0800 Subject: [PATCH 202/407] Update overview.md --- docs/concepts/abstractions/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/abstractions/overview.md index 712ad0f99f..4e832eaffb 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/abstractions/overview.md @@ -9,7 +9,7 @@ This page explains how Kubernetes objects are represented in the Kubernetes API, {% capture body %} ## Understanding Kubernetes Objects -*Kubernetes Objects* are persistent entities in the Kubernetes system. Kubenetes uses these entities to represent the state of your cluster. Specifically, they can describe: +*Kubernetes Objects* are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe: * What containerized applications are running (and on which nodes) * The resources available to those applications From 4e3f6ae0e3e3c748998b8c4caf7ba5ffec42fd86 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Feb 2017 11:07:33 -0800 Subject: [PATCH 203/407] fix Kubenetes typo Kubenetes -> Kubernetes --- .../configure-liveness-readiness-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md index d168ceb211..ff05756350 100644 --- a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md +++ b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md @@ -247,7 +247,7 @@ where you would set it. Suppose the Container listens on 127.0.0.1 and the Pod's If your pod relies on virtual hosts, which is probably the more common case, you should not use `host`, but rather set the `Host` header in `httpHeaders`. -In addition to command probes and HTTP probes, Kubenetes supports +In addition to command probes and HTTP probes, Kubernetes supports [TCP probes](/docs/api-reference/v1/definitions/#_v1_tcpsocketaction). {% endcapture %} From aaca71474ec9fa9a73fc94b96c867ad0c336da75 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Tue, 14 Feb 2017 16:37:49 +0800 Subject: [PATCH 204/407] column READY is missed column READY is missed --- docs/user-guide/deployments.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index 0b4ae0d85a..52d24391c4 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -62,8 +62,8 @@ This indicates that the Deployment has created all three replicas, and all repli ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-2035384211 3 3 18s +NAME DESIRED CURRENT READY AGE +nginx-deployment-2035384211 3 3 0 18s ``` You may notice that the name of the Replica Set is always `-`. @@ -180,9 +180,9 @@ We can run `kubectl get rs` to see that the Deployment updated the Pods by creat ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 3 3 6s -nginx-deployment-2035384211 0 0 36s +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 3 3 0 6s +nginx-deployment-2035384211 0 0 0 36s ``` Running `get pods` should now show only the new Pods: @@ -287,10 +287,10 @@ You will also see that both the number of old replicas (nginx-deployment-1564180 ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 2 2 25s -nginx-deployment-2035384211 0 0 36s -nginx-deployment-3066724191 2 2 6s +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 2 2 0 25s +nginx-deployment-2035384211 0 0 0 36s +nginx-deployment-3066724191 2 2 2 6s ``` Looking at the Pods created, you will see that the 2 Pods created by new Replica Set are stuck in an image pull loop. @@ -514,10 +514,10 @@ The Deployment was still in progress when we paused it, so the actions of scalin ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 2 2 1h -nginx-deployment-2035384211 2 2 1h -nginx-deployment-3066724191 0 0 1h +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 2 2 2 1h +nginx-deployment-2035384211 2 2 0 1h +nginx-deployment-3066724191 0 0 0 1h ``` In a separate terminal, watch for rollout status changes and you'll see the rollout won't continue: @@ -546,10 +546,10 @@ deployment nginx-deployment successfully rolled out ```shell $ kubectl get rs -NAME DESIRED CURRENT AGE -nginx-deployment-1564180365 3 3 1h -nginx-deployment-2035384211 0 0 1h -nginx-deployment-3066724191 0 0 1h +NAME DESIRED CURRENT READY AGE +nginx-deployment-1564180365 3 3 3 1h +nginx-deployment-2035384211 0 0 0 1h +nginx-deployment-3066724191 0 0 0 1h ``` Note: You cannot rollback a paused Deployment until you resume it. From 40399ff6365825c65f71d89e64830b8975bec7cf Mon Sep 17 00:00:00 2001 From: Arthur Miranda Date: Tue, 14 Feb 2017 15:41:28 -0300 Subject: [PATCH 205/407] Fix broken/outdated links in the ingress.md file The Ingress controllers have moved to the kubernetes/ingress repository, then some links are broken or outdated. --- docs/user-guide/ingress.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/ingress.md b/docs/user-guide/ingress.md index 4628d5050d..0a4f13da2f 100644 --- a/docs/user-guide/ingress.md +++ b/docs/user-guide/ingress.md @@ -44,9 +44,9 @@ It can be configured to give services externally-reachable urls, load balance tr Before you start using the Ingress resource, there are a few things you should understand. The Ingress is a beta resource, not available in any Kubernetes release prior to 1.1. You need an Ingress controller to satisfy an Ingress, simply creating the resource will have no effect. -GCE/GKE deploys an ingress controller on the master. You can deploy any number of custom ingress controllers in a pod. You must annotate each ingress with the appropriate class, as indicated [here](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#running-multiple-ingress-controllers) and [here](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#disabling-glbc). +GCE/GKE deploys an ingress controller on the master. You can deploy any number of custom ingress controllers in a pod. You must annotate each ingress with the appropriate class, as indicated [here](https://github.com/kubernetes/ingress/tree/master/controllers/nginx#running-multiple-ingress-controllers) and [here](https://github.com/kubernetes/ingress/blob/master/controllers/gce/BETA_LIMITATIONS.md#disabling-glbc). -Make sure you review the [beta limitations](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/gce/BETA_LIMITATIONS.md) of this controller. In environments other than GCE/GKE, you need to [deploy a controller](https://github.com/kubernetes/contrib/tree/master/ingress/controllers) as a pod. +Make sure you review the [beta limitations](https://github.com/kubernetes/ingress/blob/master/controllers/gce/BETA_LIMITATIONS.md) of this controller. In environments other than GCE/GKE, you need to [deploy a controller](https://github.com/kubernetes/ingress/tree/master/controllers) as a pod. ## The Ingress Resource @@ -71,7 +71,7 @@ spec: __Lines 1-4__: As with all other Kubernetes config, an Ingress needs `apiVersion`, `kind`, and `metadata` fields. For general information about working with config files, see [here](/docs/user-guide/deploying-applications), [here](/docs/user-guide/configuring-containers), and [here](/docs/user-guide/working-with-resources). -__Lines 5-7__: Ingress [spec](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status) has all the information needed to configure a loadbalancer or proxy server. Most importantly, it contains a list of rules matched against all incoming requests. Currently the Ingress resource only supports http rules. +__Lines 5-7__: Ingress [spec](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#spec-and-status) has all the information needed to configure a loadbalancer or proxy server. Most importantly, it contains a list of rules matched against all incoming requests. Currently the Ingress resource only supports http rules. __Lines 8-9__: Each http rule contains the following information: A host (e.g.: foo.bar.com, defaults to * in this example), a list of paths (e.g.: /testpath) each of which has an associated backend (test:80). Both the host and path must match the content of an incoming request before the loadbalancer directs traffic to the backend. From 3ac058495389eb3194fdb73f80bcb796610a4f5f Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Tue, 14 Feb 2017 10:36:08 +0800 Subject: [PATCH 206/407] add "--show-all" to kubectl get pods add "--show-all" to "kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}" ,because the pod was ended --- docs/user-guide/jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index d48b856ae3..d64ae9d730 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -66,7 +66,7 @@ To view completed pods of a job, use `kubectl get pods --show-all`. The `--show To list all the pods that belong to a job in a machine readable form, you can use a command like this: ```shell -$ pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}) +$ pods=$(kubectl get pods --show-all --selector=job-name=pi --output=jsonpath={.items..metadata.name}) echo $pods pi-aiw0a ``` From a8009333b1e831cf13384076936d87ec93133747 Mon Sep 17 00:00:00 2001 From: Robert Kubis Date: Sat, 11 Feb 2017 16:05:47 +0000 Subject: [PATCH 207/407] Update cheatsheet for multi-container handling Added how to retrieve logs for containers in multi-container pods --- docs/user-guide/kubectl-cheatsheet.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user-guide/kubectl-cheatsheet.md b/docs/user-guide/kubectl-cheatsheet.md index c147304a7d..7fc0142f04 100644 --- a/docs/user-guide/kubectl-cheatsheet.md +++ b/docs/user-guide/kubectl-cheatsheet.md @@ -197,7 +197,9 @@ $ kubectl -n my-ns delete po,svc --all # Delete all pods and servic ```console $ kubectl logs my-pod # dump pod logs (stdout) +$ kubectl logs my-pod -c my-container # dump pod container logs (stdout, multi-container case) $ kubectl logs -f my-pod # stream pod logs (stdout) +$ kubectl logs -f my-pod -c my-container # stream pod container logs (stdout, multi-container case) $ kubectl run -i --tty busybox --image=busybox -- sh # Run pod as interactive shell $ kubectl attach my-pod -i # Attach to Running Container $ kubectl port-forward my-pod 5000:6000 # Forward port 6000 of Pod to your to 5000 on your local machine From 523bb2a1fd531e92316b4cf273d7427b47c9ad4d Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Feb 2017 13:49:14 -0800 Subject: [PATCH 208/407] re: issue #1671; update to direct link Update Docker Volumes link to https://docs.docker.com/engine/tutorials/dockervolumes/ --- docs/user-guide/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 9c69683525..0d4e77a97d 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -54,7 +54,7 @@ Before running examples in the user guides, please ensure you have completed the : A service defines a set of pods and a means by which to access them, such as single stable IP address and corresponding DNS name. [**Volume**](/docs/user-guide/volumes/) -: A volume is a directory, possibly with some data in it, which is accessible to a Container as part of its filesystem. Kubernetes volumes build upon [Docker Volumes](https://docs.docker.com/userguide/dockervolumes/), adding provisioning of the volume directory and/or device. +: A volume is a directory, possibly with some data in it, which is accessible to a Container as part of its filesystem. Kubernetes volumes build upon [Docker Volumes](https://docs.docker.com/engine/tutorials/dockervolumes/), adding provisioning of the volume directory and/or device. [**Secret**](/docs/user-guide/secrets/) : A secret stores sensitive data, such as authentication tokens, which can be made available to containers upon request. From 1c634023f6ab5552d4dec0d9112e991961114d46 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Feb 2017 14:34:13 -0800 Subject: [PATCH 209/407] re: issue #2034; fix docker image link (#2532) * re: issue #2034; fix docker image link change link in line 26 to: https://gcr.io/google_containers/hpa-example * point link to local Dockerfile * specify it as the Dockerfile, not image --- docs/user-guide/horizontal-pod-autoscaling/walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md b/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md index f0a5cbc64a..616061e930 100644 --- a/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md +++ b/docs/user-guide/horizontal-pod-autoscaling/walkthrough.md @@ -23,7 +23,7 @@ heapster monitoring will be turned-on by default). ## Step One: Run & expose php-apache server To demonstrate Horizontal Pod Autoscaler we will use a custom docker image based on the php-apache image. -The image can be found [here](/docs/user-guide/horizontal-pod-autoscaling/image). +The Dockerfile can be found [here](/docs/user-guide/horizontal-pod-autoscaling/image/Dockerfile). It defines an [index.php](/docs/user-guide/horizontal-pod-autoscaling/image/index.php) page which performs some CPU intensive computations. First, we will start a deployment running the image and expose it as a service: From e1eb525f0164b99d4783e88aa43a65c15402ba83 Mon Sep 17 00:00:00 2001 From: foxish Date: Wed, 15 Feb 2017 13:27:00 -0800 Subject: [PATCH 210/407] Fix travis.yml --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 09a28b78cb..d05639d195 100644 --- a/.travis.yml +++ b/.travis.yml @@ -14,6 +14,7 @@ install: - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apiserver - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/client-go - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/sample-apiserver +- rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ From 183fc3d41cbd474c129f466aa00c3a997a3a7e08 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Tue, 14 Feb 2017 20:03:22 +0800 Subject: [PATCH 211/407] The key-value "optional:true" doesn't work The key-value "optional:true" doesn't work in 1.5.2 yet. So should we add a note here? "The key-value "optional:true" will be work after kubernetes 1.5.2." --- docs/user-guide/configmap/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 3227e416c0..e2de3a7aec 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -590,3 +590,5 @@ Kubelet only supports use of ConfigMap for pods it gets from the API server. Th created using kubectl, or indirectly via a replication controller. It does not include pods created via the Kubelet's `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) + +The key-value "optional:true" will be work after kubernetes 1.5.2. From 3e48da5f8bf976d9206df13c641f4329af33648d Mon Sep 17 00:00:00 2001 From: sallydeng Date: Wed, 15 Feb 2017 13:15:20 +0800 Subject: [PATCH 212/407] Change sentence base on reviewer's comment "NOTE: The key-value optional:true is supported for kubernetes 1.5.3 and above." --- docs/user-guide/configmap/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index e2de3a7aec..5e074e5787 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -591,4 +591,4 @@ created using kubectl, or indirectly via a replication controller. It does not via the Kubelet's `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) -The key-value "optional:true" will be work after kubernetes 1.5.2. +NOTE: The key-value optional:true is supported for kubernetes 1.5.3 and above. From 2aea9f942494fc9e7d8141e5327902abc48b82ed Mon Sep 17 00:00:00 2001 From: Michael Mrowetz Date: Wed, 15 Feb 2017 11:58:43 +0900 Subject: [PATCH 213/407] #2534 mark openstack-heat as standalone-salt-conf --- docs/admin/salt.md | 12 ++++++------ docs/getting-started-guides/openstack-heat.md | 9 +++++++-- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/admin/salt.md b/docs/admin/salt.md index 05d0ae9250..b570dc8473 100644 --- a/docs/admin/salt.md +++ b/docs/admin/salt.md @@ -10,11 +10,11 @@ The Salt scripts are shared across multiple hosting providers and depending on w ## Salt cluster setup -The **salt-master** service runs on the kubernetes-master [(except on the default GCE setup)](#standalone-salt-configuration-on-gce). +The **salt-master** service runs on the kubernetes-master [(except on the default GCE and OpenStack-Heat setup)](#standalone-salt-configuration-on-gce-and-others). The **salt-minion** service runs on the kubernetes-master and each kubernetes-node in the cluster. -Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file [(except on GCE)](#standalone-salt-configuration-on-gce). +Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file [(except on GCE and OpenStack-Heat)](#standalone-salt-configuration-on-gce-and-others). ```shell [root@kubernetes-master] $ cat /etc/salt/minion.d/master.conf @@ -25,15 +25,15 @@ The salt-master is contacted by each salt-minion and depending upon the machine If you are running the Vagrant based environment, the **salt-api** service is running on the kubernetes-master. It is configured to enable the vagrant user to introspect the salt cluster in order to find out about machines in the Vagrant environment via a REST API. -## Standalone Salt Configuration on GCE +## Standalone Salt Configuration on GCE and others -On GCE, the master and nodes are all configured as [standalone minions](http://docs.saltstack.com/en/latest/topics/tutorials/standalone_minion.html). The configuration for each VM is derived from the VM's [instance metadata](https://cloud.google.com/compute/docs/metadata) and then stored in Salt grains (`/etc/salt/minion.d/grains.conf`) and pillars (`/srv/salt-overlay/pillar/cluster-params.sls`) that local Salt uses to enforce state. +On GCE and OpenStack, using the Openstack-Heat provider, the master and nodes are all configured as [standalone minions](http://docs.saltstack.com/en/latest/topics/tutorials/standalone_minion.html). The configuration for each VM is derived from the VM's [instance metadata](https://cloud.google.com/compute/docs/metadata) and then stored in Salt grains (`/etc/salt/minion.d/grains.conf`) and pillars (`/srv/salt-overlay/pillar/cluster-params.sls`) that local Salt uses to enforce state. -All remaining sections that refer to master/minion setups should be ignored for GCE. One fallout of the GCE setup is that the Salt mine doesn't exist - there is no sharing of configuration amongst nodes. +All remaining sections that refer to master/minion setups should be ignored for GCE and OpenStack. One fallout of this setup is that the Salt mine doesn't exist - there is no sharing of configuration amongst nodes. ## Salt security -*(Not applicable on default GCE setup.)* +*(Not applicable on default GCE and OpenStack-Heat setup.)* Security is not enabled on the salt-master, and the salt-master is configured to auto-accept incoming requests from minions. It is not recommended to use this security configuration in production environments without deeper study. (In some environments this isn't as bad as it might sound if the salt master port isn't externally accessible and you trust everyone on your network.) diff --git a/docs/getting-started-guides/openstack-heat.md b/docs/getting-started-guides/openstack-heat.md index e4fb7c277d..8a2fd52d11 100644 --- a/docs/getting-started-guides/openstack-heat.md +++ b/docs/getting-started-guides/openstack-heat.md @@ -23,7 +23,7 @@ This guide assumes you have access to a working OpenStack cluster with the follo - Heat - DNS resolution of instance names -By default this provider provisions 4 m1.medium instances. If you do not have resources available, please see the [Set additional configuration values](#set-additional-configuration-values) section for information on reducing the footprint of your cluster. +By default this provider provisions 4 `m1.medium` instances. If you do not have resources available, please see the [Set additional configuration values](#set-additional-configuration-values) section for information on reducing the footprint of your cluster. ## Pre-Requisites If you already have the required versions of the OpenStack CLI tools installed and configured, you can move on to the [Starting a cluster](#starting-a-cluster) section. @@ -92,7 +92,7 @@ Please see the contents of these files for documentation regarding each variable ## Starting a cluster -Once Kubernetes version 1.3 is released, and you've installed the OpenStack CLI tools and have set your OpenStack environment variables, issue this command: +Once you've installed the OpenStack CLI tools and have set your OpenStack environment variables, issue this command: ```sh export KUBERNETES_PROVIDER=openstack-heat; curl -sS https://get.k8s.io | bash @@ -194,6 +194,11 @@ nova list --name=$STACK_NAME See the [OpenStack CLI Reference](http://docs.openstack.org/cli-reference/) for more details. +### Salt + +The OpenStack-Heat provider uses a [standalone Salt configuration](/docs/admin/salt/#standalone-salt-configuration-on-gce-and-others). +It only uses Salt for bootstraping the machines and creates no salt-master and does not auto-start the salt-minion service on the nodes. + ## SSHing to your nodes Your public key was added during the cluster turn-up, so you can easily ssh to them for troubleshooting purposes. From dec125aa533e80f9293a6b0243004202b2df4a54 Mon Sep 17 00:00:00 2001 From: Arthur Miranda Date: Wed, 15 Feb 2017 15:07:08 -0300 Subject: [PATCH 214/407] Update the links of Deployment User Guide The file 'blob/master/docs/devel/api-conventions.md' has moved to 'community/blob/master/contributors/devel/api-conventions.md' Updates the links associated with these file. --- docs/user-guide/deployments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index 52d24391c4..fb8f2cc0e0 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -615,7 +615,7 @@ the Deployment's `status.conditions`: * Status=False * Reason=ProgressDeadlineExceeded -See the [Kubernetes API conventions](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties) for more information on status conditions. +See the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#typical-status-properties) for more information on status conditions. Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with `Reason=ProgressDeadlineExceeded`. @@ -725,7 +725,7 @@ As with all other Kubernetes configs, a Deployment needs `apiVersion`, `kind`, a `metadata` fields. For general information about working with config files, see [deploying applications](/docs/user-guide/deploying-applications), [configuring containers](/docs/user-guide/configuring-containers), and [using kubectl to manage resources](/docs/user-guide/working-with-resources) documents. -A Deployment also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). +A Deployment also needs a [`.spec` section](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#spec-and-status). ### Pod Template From 0fca22e65953a6237bdf8916fcf05373083ea8b3 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Wed, 15 Feb 2017 16:15:20 +0800 Subject: [PATCH 215/407] kubectl_apply.md-change it for label key When I test this command, if the configmap has a label, it just delete the configmaps which has the same label key and that are not in the file. kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap So I advise change the description like this: "Apply the configuration in manifest.yaml and delete all the other configmaps with the same label key that are not in the file." --- docs/user-guide/kubectl/kubectl_apply.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl/kubectl_apply.md b/docs/user-guide/kubectl/kubectl_apply.md index c729263d6c..81c1a12137 100644 --- a/docs/user-guide/kubectl/kubectl_apply.md +++ b/docs/user-guide/kubectl/kubectl_apply.md @@ -32,7 +32,7 @@ kubectl apply -f FILENAME # Apply the configuration in manifest.yaml that matches label app=nginx and delete all the other resources that are not in the file and match label app=nginx. kubectl apply --prune -f manifest.yaml -l app=nginx - # Apply the configuration in manifest.yaml and delete all the other configmaps that are not in the file. + # Apply the configuration in manifest.yaml and delete all the other configmaps with the same label key that are not in the file. kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap ``` From 4e5e79f35f6a4db8dde18b4ef0160f5c083ebdba Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Feb 2017 14:40:39 -0800 Subject: [PATCH 216/407] rollback PR #2522 Rollingback PR #2522 until we can verify which version optional:true will be available in. --- docs/user-guide/configmap/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 5e074e5787..40b3fa181b 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -591,4 +591,3 @@ created using kubectl, or indirectly via a replication controller. It does not via the Kubelet's `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) -NOTE: The key-value optional:true is supported for kubernetes 1.5.3 and above. From 0ce865cc763f16336b96485c824f9a24ab215399 Mon Sep 17 00:00:00 2001 From: Michail Kargakis Date: Wed, 25 Jan 2017 10:44:31 +0100 Subject: [PATCH 217/407] Update deployment completeness documentation --- docs/user-guide/deployments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index fb8f2cc0e0..f2e854690e 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -578,6 +578,7 @@ Kubernetes marks a Deployment as _complete_ when it has the following characteri equals or exceeds the number required by the Deployment strategy. * All of the replicas associated with the Deployment have been updated to the latest version you've specified, meaning any updates you've requested have been completed. +* No old pods for the Deployment are running. You can check if a Deployment has completed by using `kubectl rollout status`. If the rollout completed successfully, `kubectl rollout status` returns a zero exit code. From 3ed3018153c3eb0c5b9ca2da8c2d0b1c51cf6f8f Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Feb 2017 14:56:39 -0800 Subject: [PATCH 218/407] Move Pod Lifecycle to Concepts. (#2420) * Move Pod Lifecycle to Concepts. * Put moved note on old page. * Update pod-lifecycle.md --- _data/concepts.yml | 6 + docs/concepts/workloads/pods/pod-lifecycle.md | 282 ++++++++++++++++++ docs/user-guide/pod-states.md | 166 +---------- 3 files changed, 290 insertions(+), 164 deletions(-) create mode 100644 docs/concepts/workloads/pods/pod-lifecycle.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 4227231cc1..432f4c86e1 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -25,6 +25,12 @@ toc: section: - docs/concepts/object-metadata/annotations.md +- title: Workloads + section: + - title: Pods + section: + - docs/concepts/workloads/pods/pod-lifecycle.md + - title: Configuration section: - docs/concepts/configuration/container-command-args.md diff --git a/docs/concepts/workloads/pods/pod-lifecycle.md b/docs/concepts/workloads/pods/pod-lifecycle.md new file mode 100644 index 0000000000..540ba506fc --- /dev/null +++ b/docs/concepts/workloads/pods/pod-lifecycle.md @@ -0,0 +1,282 @@ +--- +title: Pod Lifecycle +--- + +{% capture overview %} + +{% comment %}Updated: 4/14/2015{% endcomment %} +{% comment %}Edited and moved to Concepts section: 2/2/17{% endcomment %} + +This page describes the lifecycle of a Pod. + +{% endcapture %} + + +{% capture body %} + +## Pod phase + +A Pod's `status` field is a +[PodStatus](/docs/resources-reference/v1.5/#podstatus-v1) +object, which has a `phase` field. + +The phase of a Pod is a simple, high-level summary of where the Pod is in its +lifecycle. The phase is not intended to be a comprehensive rollup of observations +of Container or Pod state, nor is it intended to be a comprehensive state machine. + +The number and meanings of Pod phase values are tightly guarded. +Other than what is documented here, nothing should be assumed about Pods that +have a given `phase` value. + +Here are the possible values for `phase`: + +* Pending: The Pod has been accepted by the Kubernetes system, but one or more of + the Container images has not been created. This includes time before being + scheduled as well as time spent downloading images over the network, + which could take a while. + +* Running: The Pod has been bound to a node, and all of the Containers have been + created. At least one Container is still running, or is in the process of + starting or restarting. + +* Succeeded: All Containers in the Pod have terminated in success, and will not + be restarted. + +* Failed: All Containers in the Pod have terminated, and at least one Container + has terminated in failure. That is, the Container either exited with non-zero + status or was terminated by the system. + +* Unknown: For some reason the state of the Pod could not be obtained, typically + due to an error in communicating with the host of the Pod. + +## Pod conditions + +A Pod has a PodStatus, which has an array of +[PodConditions](docs/resources-reference/v1.5/#podcondition). Each element +of the PodCondition array has a `type` field and a `status` field. The `type` +field is a string, with possible values PodScheduled, Ready, Initialized, and +Unschedulable. The `status` field is a string, with possible values True, False, +and Unknown. + +## Container probes + +A [Probe](/docs/resources-reference/v1.5/#probe-v1) is a diagnostic +performed periodically by the [kubelet](/docs/admin/kubelet/) +on a Container. To perform a diagnostic, +the kublet calls a +[Handler](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#Handler) implemented by +the Container. There are three types of handlers: + +* [ExecAction](/docs/resources-reference/v1.5/#execaction-v1): + Executes a specified command inside the Container. The diagnostic + is considered successful if the command exits with a status code of 0. + +* [TCPSocketAction](/docs/resources-reference/v1.5/#tcpsocketaction-v1): + Performs a TCP check against the Container's IP address on + a specified port. The diagnostic is considered successful if the port is open. + +* [HTTPGetAction](/docs/resources-reference/v1.5/#httpgetaction-v1): + Performs an HTTP Get request against the Container's IP + address on a specified port and path. The diagnostic is considered successful + if the response has a status code greater than or equal to 200 and less than 400. + +Each probe has one of three results: + +* Success: The Container passed the diagnostic. +* Failure: The Container failed the diagnostic. +* Unknown: The diagnostic failed, so no action should be taken. + +The kubelet can optionally perform and react to two kinds of probes on running +Containers: + +* `livenessProbe`: Indicates whether the Container is running. If + the liveness probe fails, the kubelet kills the Container, and the Container + is subjected to its [restart policy](#restart-policy). If a Container does not + provide a liveness probe, the default state is `Success`. + +* `readinessProbe`: Indicates whether the Container is ready to service requests. + If the readiness probe fails, the endpoints controller removes the Pod's IP + address from the endpoints of all Services that match the Pod. The default + state of readiness before the initial delay is `Failure`. If a Container does + not provide a readiness probe, the default state is `Success`. + +### When should you use liveness or readiness probes? + +If the process in your Container is able to crash on its own whenever it +encounters an issue or becomes unhealthy, you do not necessarily need a liveness +probe; the kubelet will automatically perform the correct action in accordance +with the Pod's `restartPolicy`. + +If you'd like your Container to be killed and restarted if a probe fails, then +specify a liveness probe, and specify a `restartPolicy` of Always or OnFailure. + +If you'd like to start sending traffic to a Pod only when a probe succeeds, +specify a readiness probe. In this case, the readiness probe might be the same +as the liveness probe, but the existence of the readiness probe in the spec means +that the Pod will start without receiving any traffic and only start receiving +traffic after the probe starts succeeding. + +If you want your Container to be able to take itself down for maintenance, you +can specify a readiness probe that checks an endpoint specific to readiness that +is different from the liveness probe. + +Note that if you just want to be able to drain requests when the Pod is deleted, +you do not necessarily need a readiness probe; on deletion, the Pod automatically +puts itself into an unready state regardless of whether the readiness probe exists. +The Pod remains in the unready state while it waits for the Containers in the Pod +to stop. + +## Pod and Container status + +For detailed information about Pod Container status, see +[PodStatus](/docs/resources-reference/v1.5/#podstatus-v1) +and +[ContainerStatus](/docs/resources-reference/v1.5/#containerstatus-v1). +Note that the information reported as Pod status depends on the current +[ContainerState](/docs/resources-reference/v1.5/#containerstate-v1). + +## Restart policy + +A PodSpec has a `restartPolicy` field with possible values Always, OnFailure, +and Never. The default value is Always. +`restartPolicy` applies to all Containers in the Pod. `restartPolicy` only +refers to restarts of the Containers by the kubelet on the same node. Failed +Containers that are restarted by the kubelet are restarted with an exponential +back-off delay (10s, 20s, 40s ...) capped at five minutes, and is reset after ten +minutes of successful execution. As discussed in the +[Pods document](/docs/user-guide/pods/#durability-of-pods-or-lack-thereof), +once bound to a node, a Pod will never be rebound to another node. + + + +## Pod lifetime + +In general, Pods do not disappear until someone destroys them. This might be a +human or a controller. The only exception to +this rule is that Pods with a `phase` of Succeeded or Failed for more than some +duration (determined by the master) will expire and be automatically destroyed. + +Three types of controllers are available: + +- Use a [Job](/docs/user-guide/jobs/) for Pods that are expected to terminate, + for example, batch computations. Jobs are appropriate only for Pods with + `restartPolicy` equal to OnFailure or Never. + +- Use a [ReplicationController](/docs/user-guide/replication-controller/), + [ReplicaSet](/docs/user-guide/replicasets/), or + [Deployment](/docs/user-guide/deployments/) + for Pods that are not expected to terminate, for example, web servers. + ReplicationControllers are appropriate only for Pods with a `restartPolicy` of + Always. + +- Use a [DaemonSet](/docs/admin/daemons/) for Pods that need to run one per + machine, because they provide a machine-specific system service. + +All three types of controllers contain a PodTemplate. It +is recommended to create the appropriate controller and let +it create Pods, rather than directly create Pods yourself. That is because Pods +alone are not resilient to machine failures, but controllers are. + +If a node dies or is disconnected from the rest of the cluster, Kubernetes +applies a policy for setting the `phase` of all Pods on the lost node to Failed. + +## Examples + +### Advanced liveness probe example + +Liveness probes are executed by the kubelet, so all requests are made in the +kubelet network namespace. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + labels: + test: liveness + name: liveness-http +spec: + containers: + - args: + - /server + image: gcr.io/google_containers/liveness + livenessProbe: + httpGet: + # when "host" is not defined, "PodIP" will be used + # host: my-host + # when "scheme" is not defined, "HTTP" scheme will be used. Only "HTTP" and "HTTPS" are allowed + # scheme: HTTPS + path: /healthz + port: 8080 + httpHeaders: + - name: X-Custom-Header + value: Awesome + initialDelaySeconds: 15 + timeoutSeconds: 1 + name: liveness +``` + +### Example states + + * Pod is running and has one Container. Container exits with success. + * Log completion event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Pod `phase` becomes Succeeded. + * Never: Pod `phase` becomes Succeeded. + + * Pod is running and has one Container. Container exits with failure. + * Log failure event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Pod `phase` becomes Failed. + + * Pod is running and has two Containers. Container 1 exits with failure. + * Log failure event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Do not restart Container; Pod `phase` stays Running. + * If Container 1 is not running, and Container 2 exits: + * Log failure event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Pod `phase` becomes Failed. + + * Pod is running and has one Container. Container runs out of memory. + * Container terminates in failure. + * Log OOM event. + * If `restartPolicy` is: + * Always: Restart Container; Pod `phase` stays Running. + * OnFailure: Restart Container; Pod `phase` stays Running. + * Never: Log failure event; Pod `phase` becomes Failed. + + * Pod is running, and a disk dies. + * Kill all Containers. + * Log appropriate event. + * Pod `phase` becomes Failed. + * If running under a controller, Pod is recreated elsewhere. + + * Pod is running, and its node is segmented out. + * Node controller waits for timeout. + * Node controller sets Pod `phase` to Failed. + * If running under a controller, Pod is recreated elsewhere. + +{% endcapture %} + + +{% capture whatsnext %} + +* Get hands-on experience + [attaching handlers to Container lifecycle events](/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/). + +* Get hands-on experience + [configuring liveness and readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/). + +* [Container Lifecycle Hooks](/docs/user-guide/container-environment/) + +{% endcapture %} + +{% include templates/concept.md %} + diff --git a/docs/user-guide/pod-states.md b/docs/user-guide/pod-states.md index 462f8b65bd..a3cea728cc 100644 --- a/docs/user-guide/pod-states.md +++ b/docs/user-guide/pod-states.md @@ -4,168 +4,6 @@ assignees: title: The Lifecycle of a Pod --- -Updated: 4/14/2015 - -This document covers the lifecycle of a pod. It is not an exhaustive document, but an introduction to the topic. - -## Pod Phase - -As consistent with the overall [API convention](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties), phase is a simple, high-level summary of the phase of the lifecycle of a pod. It is not intended to be a comprehensive rollup of observations of container-level or even pod-level conditions or other state, nor is it intended to be a comprehensive state machine. - -The number and meanings of `PodPhase` values are tightly guarded. Other than what is documented here, nothing should be assumed about pods with a given `PodPhase`. - -* Pending: The pod has been accepted by the system, but one or more of the container images has not been created. This includes time before being scheduled as well as time spent downloading images over the network, which could take a while. -* Running: The pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. -* Succeeded: All containers in the pod have terminated in success, and will not be restarted. -* Failed: All containers in the pod have terminated, at least one container has terminated in failure (exited with non-zero exit status or was terminated by the system). -* Unknown: For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod. - -## Pod Conditions - -A pod containing containers that specify readiness probes will also report the Ready condition. Condition status values may be `True`, `False`, or `Unknown`. - -## Container Probes - -A [Probe](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#Probe) is a diagnostic performed periodically by the kubelet on a container. Specifically the diagnostic is one of three [Handlers](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#Handler): - -* `ExecAction`: executes a specified command inside the container expecting on success that the command exits with status code 0. -* `TCPSocketAction`: performs a tcp check against the container's IP address on a specified port expecting on success that the port is open. -* `HTTPGetAction`: performs an HTTP Get against the container's IP address on a specified port and path expecting on success that the response has a status code greater than or equal to 200 and less than 400. - -Each probe will have one of three results: - -* `Success`: indicates that the container passed the diagnostic. -* `Failure`: indicates that the container failed the diagnostic. -* `Unknown`: indicates that the diagnostic failed so no action should be taken. - -The kubelet can optionally perform and react to two kinds of probes on running containers: - -* `LivenessProbe`: indicates whether the container is *live*, i.e. running. If the LivenessProbe fails, the kubelet will kill the container and the container will be subjected to its [RestartPolicy](#restartpolicy). The default state of Liveness before the initial delay is `Success`. The state of Liveness for a container when no probe is provided is assumed to be `Success`. -* `ReadinessProbe`: indicates whether the container is *ready* to service requests. If the ReadinessProbe fails, the endpoints controller will remove the pod's IP address from the endpoints of all services that match the pod. The default state of Readiness before the initial delay is `Failure`. The state of Readiness for a container when no probe is provided is assumed to be `Success`. - -### When should I use liveness or readiness probes? - -If the process in your container is able to crash on its own whenever it encounters an issue or becomes unhealthy, you do not necessarily need a liveness probe - the kubelet will automatically perform the correct action in accordance with the RestartPolicy when the process crashes. - -If you'd like your container to be killed and restarted if a probe fails, then specify a LivenessProbe and a RestartPolicy of `Always` or `OnFailure`. - -If you'd like to start sending traffic to a pod only when a probe succeeds, specify a ReadinessProbe. In this case, the ReadinessProbe may be the same as the LivenessProbe, but the existence of the ReadinessProbe in the spec means that the pod will start without receiving any traffic and only start receiving traffic once the probe starts succeeding. - -If a container wants the ability to take itself down for maintenance, you can specify a ReadinessProbe that checks an endpoint specific to readiness which is different than the LivenessProbe. - -Note that if you just want to be able to drain requests when the pod is deleted, you do not necessarily need a ReadinessProbe - on deletion, the pod automatically puts itself into an unready state regardless of whether the ReadinessProbe exists or not while it waits for the containers in the pod to stop. - -## Container Statuses - -More detailed information about the current (and previous) container statuses can be found in [ContainerStatuses](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#PodStatus). The information reported depends on the current [ContainerState](https://godoc.org/k8s.io/kubernetes/pkg/api/v1#ContainerState), which may be Waiting, Running, or Terminated. - -## RestartPolicy - -The possible values for RestartPolicy are `Always`, `OnFailure`, or `Never`. If RestartPolicy is not set, the default value is `Always`. RestartPolicy applies to all containers in the pod. RestartPolicy only refers to restarts of the containers by the Kubelet on the same node. Failed containers that are restarted by Kubelet, are restarted with an exponential back-off delay, the delay is in multiples of sync-frequency 0, 1x, 2x, 4x, 8x ... capped at 5 minutes and is reset after 10 minutes of successful execution. As discussed in the [pods document](/docs/user-guide/pods/#durability-of-pods-or-lack-thereof), once bound to a node, a pod will never be rebound to another node. This means that some kind of controller is necessary in order for a pod to survive node failure, even if just a single pod at a time is desired. - -Three types of controllers are currently available: - -- Use a [`Job`](/docs/user-guide/jobs/) for pods which are expected to terminate (e.g. batch computations). -- Use a [`ReplicationController`](/docs/user-guide/replication-controller/) or [`Deployment`](/docs/user-guide/deployments/) - for pods which are not expected to terminate (e.g. web servers). -- Use a [`DaemonSet`](/docs/admin/daemons/): Use for pods which need to run 1 per machine because they provide a - machine-specific system service. -If you are unsure whether to use ReplicationController or Daemon, then see [Daemon Set versus -Replication Controller](/docs/admin/daemons/#daemon-set-versus-replication-controller). - -`ReplicationController` is *only* appropriate for pods with `RestartPolicy = Always`. -`Job` is *only* appropriate for pods with `RestartPolicy` equal to `OnFailure` or `Never`. - -All 3 types of controllers contain a PodTemplate, which has all the same fields as a Pod. -It is recommended to create the appropriate controller and let it create pods, rather than to -directly create pods yourself. That is because pods alone are not resilient to machine failures, -but Controllers are. - -## Pod lifetime - -In general, pods which are created do not disappear until someone destroys them. This might be a human or a `ReplicationController`, or another controller. The only exception to this rule is that pods with a `PodPhase` of `Succeeded` or `Failed` for more than some duration (determined by the master) will expire and be automatically reaped. - -If a node dies or is disconnected from the rest of the cluster, some entity within the system (call it the NodeController for now) is responsible for applying policy (e.g. a timeout) and marking any pods on the lost node as `Failed`. - -## Examples - -### Advanced livenessProbe example - -Liveness probes are executed by `kubelet`, so all requests will be made within kubelet network namespace. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - labels: - test: liveness - name: liveness-http -spec: - containers: - - args: - - /server - image: gcr.io/google_containers/liveness - livenessProbe: - httpGet: - # when "host" is not defined, "PodIP" will be used - # host: my-host - # when "scheme" is not defined, "HTTP" scheme will be used. Only "HTTP" and "HTTPS" are allowed - # scheme: HTTPS - path: /healthz - port: 8080 - httpHeaders: - - name: X-Custom-Header - value: Awesome - initialDelaySeconds: 15 - timeoutSeconds: 1 - name: liveness -``` - -### Example states - - * Pod is `Running`, 1 container, container exits success - * Log completion event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: pod becomes `Succeeded` - * Never: pod becomes `Succeeded` - - * Pod is `Running`, 1 container, container exits failure - * Log failure event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: pod becomes `Failed` - - * Pod is `Running`, 2 containers, container 1 exits failure - * Log failure event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: pod stays `Running` - * When container 2 exits... - * Log failure event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: pod becomes `Failed` - - * Pod is `Running`, container becomes OOM - * Container terminates in failure - * Log OOM event - * If RestartPolicy is: - * Always: restart container, pod stays `Running` - * OnFailure: restart container, pod stays `Running` - * Never: log failure event, pod becomes `Failed` - - * Pod is `Running`, a disk dies - * All containers are killed - * Log appropriate event - * Pod becomes `Failed` - * If running under a controller, pod will be recreated elsewhere - - * Pod is `Running`, its node is segmented out - * NodeController waits for timeout - * NodeController marks pod `Failed` - * If running under a controller, pod will be recreated elsewhere +{% include user-guide-content-moved.md %} +[Pod Lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/) From 33da5e4fb19ca91b35c1565e9548ad3ad4d3c7f0 Mon Sep 17 00:00:00 2001 From: foxish Date: Wed, 15 Feb 2017 16:06:49 -0800 Subject: [PATCH 219/407] Fix travis and add comments --- .travis.yml | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index d05639d195..b0a565803c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,9 +7,17 @@ install: - export PATH=$GOPATH/bin:$PATH - mkdir -p $HOME/gopath/src/k8s.io - mv $TRAVIS_BUILD_DIR $HOME/gopath/src/k8s.io/kubernetes.github.io + +# (1) Fetch dependencies for us to run the tests in test/examples_test.go - go get -t -v k8s.io/kubernetes.github.io/test -- git clone --depth=50 --branch=master https://github.com/kubernetes/md-check $HOME/gopath/src/k8s.io/md-check -- go get -t -v k8s.io/md-check + +# The dependencies are complicated for test/examples_test.go +# k8s.io/kubernetes/pkg is a dependency, which in turn depends on apimachinery +# but we also have apimachinery directly as one of our dependencies, which causes a conflict. +# Additionally, we get symlinks when we clone the directory. The below steps do the following: + +# (a) Replace the symlink with the actual dependencies from kubernetes/staging/src/ +# (b) copy all the vendored files to $GOPATH/src - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/apiserver - rm $GOPATH/src/k8s.io/kubernetes/vendor/k8s.io/client-go @@ -18,6 +26,12 @@ install: - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ + +# (2) Fetch md-check along with all its dependencies. +- git clone --depth=50 --branch=master https://github.com/kubernetes/md-check $HOME/gopath/src/k8s.io/md-check +- go get -t -v k8s.io/md-check + +# (3) Fetch mungedocs - go get -v k8s.io/kubernetes/cmd/mungedocs script: From e71b14fa5c89df9104648732b35cafdd8ab89a4d Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Mon, 13 Feb 2017 15:03:37 -0800 Subject: [PATCH 220/407] Move Guide toic to Tasks: kubectl exec. --- _data/tasks.yml | 1 + .../kubectl/get-shell-running-container.md | 148 ++++++++++++++++++ docs/tasks/kubectl/shell-demo.yaml | 14 ++ docs/user-guide/getting-into-containers.md | 68 +------- 4 files changed, 165 insertions(+), 66 deletions(-) create mode 100644 docs/tasks/kubectl/get-shell-running-container.md create mode 100644 docs/tasks/kubectl/shell-demo.yaml diff --git a/_data/tasks.yml b/_data/tasks.yml index cb61b1a05a..6d7cdb921e 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -6,6 +6,7 @@ toc: - title: Using the Kubectl Command-Line section: - docs/tasks/kubectl/list-all-running-container-images.md + - docs/tasks/kubectl/get-shell-running-container.md - title: Configuring Pods and Containers section: diff --git a/docs/tasks/kubectl/get-shell-running-container.md b/docs/tasks/kubectl/get-shell-running-container.md new file mode 100644 index 0000000000..a005f3e0fe --- /dev/null +++ b/docs/tasks/kubectl/get-shell-running-container.md @@ -0,0 +1,148 @@ +--- +assignees: +- caesarxuchao +- mikedanese +title: Getting a Shell to a Running Container +--- + +{% capture overview %} + +This page shows how to use `kubectl exec` to get a shell to a +running Container. + +{% endcapture %} + + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + + +{% capture steps %} + +## Getting a shell to a Container + +In this exercise, you create a Pod that has one Container. The Container +runs the nginx image. Here is the configuration file for the Pod: + +{% include code.html language="yaml" file="shell-demo.yaml" ghlink="/docs/tasks/kubectl/shell-demo.yaml" %} + +Create the Pod: + +```shell +kubectl create -f https://k8s.io/docs/tasks/kubectl/shell-demo.yaml +``` + +Verify that the Container is running: + +```shell +kubectl get pod shell-demo +``` + +Get a shell to the running Container: + +```shell +kubectl exec -it shell-demo -- /bin/bash +``` + +In your shell, list the running processes: + +```shell +root@shell-demo:/# ps aux +``` + +In your shell, list the nginx processes: + +```shell +root@shell-demo:/# ps aux | grep nginx +``` + +In your shell, experiment with other commands. Here are +some examples: + +```shell +root@shell-demo:/# ls / +root@shell-demo:/# cat /proc/mounts +root@shell-demo:/# cat /proc/1/maps +root@shell-demo:/# apt-get update +root@shell-demo:/# apt-get install tcpdump +root@shell-demo:/# tcpdump +root@shell-demo:/# apt-get install lsof +root@shell-demo:/# lsof +``` + +## Writing the root page for nginx + +Look again at the configuration file for your Pod. The Pod +has an `emptyDir` volume, and the Container mounts the volume +at `/usr/share/nginx/html`. + +In your shell, create an `index.html` file in the `/usr/share/nginx/html` +directory: + +```shell +root@shell-demo:/# echo Hello shell demo > /usr/share/nginx/html/index.html +``` + +In your shell, send a GET request to the nginx server: + +```shell +root@shell-demo:/# apt-get update +root@shell-demo:/# apt-get install curl +root@shell-demo:/# curl localhost +``` + +The output shows the text that you wrote to the `index.html` file: + +```shell +Hello shell demo +``` + +When you are finished with your shell, enter `exit`. + +## Running individual commands in a Container + +In an ordinary command window, not your shell, list the environment +variables in the running Container: + +```shell +kubectl exec shell-demo env +``` + +Experiment running other commands. Here are some examples: + +```shell +kubectl exec shell-demo ps aux +kubectl exec shell-demo ls / +kubectl exec shell-demo cat /proc/1/mounts +``` + +{% endcapture %} + +{% capture discussion %} + +## Opening a shell when a Pod has more than one Container + +If a Pod has more than one Container, use `--container` or `-c` to +specify a Container in the `kubectl exec` command. For example, +suppose you have a Pod named my-pod, and the Pod has two containers +named main-app and helper-app. The following command would open a +shell to the main-app Container. + +```shell +kubectl exec -it my-pod --container main-app -- /bin/bash +``` + +{% endcapture %} + + +{% capture whatsnext %} + +* [kubectl exec](/docs/user-guide/kubectl/v1.5/#exec) + +{% endcapture %} + + +{% include templates/task.md %} diff --git a/docs/tasks/kubectl/shell-demo.yaml b/docs/tasks/kubectl/shell-demo.yaml new file mode 100644 index 0000000000..2a7d274a64 --- /dev/null +++ b/docs/tasks/kubectl/shell-demo.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Pod +metadata: + name: shell-demo +spec: + volumes: + - name: shared-data + emptyDir: {} + containers: + - name: nginx + image: nginx + volumeMounts: + - name: shared-data + mountPath: /usr/share/nginx/html diff --git a/docs/user-guide/getting-into-containers.md b/docs/user-guide/getting-into-containers.md index bf6a5a8a2f..ff89f111f6 100644 --- a/docs/user-guide/getting-into-containers.md +++ b/docs/user-guide/getting-into-containers.md @@ -5,70 +5,6 @@ assignees: title: Running Commands in a Container with kubectl exec --- -Developers can use `kubectl exec` to run commands in a container. This guide demonstrates two use cases. +{% include user-guide-content-moved.md %} -## Using kubectl exec to check the environment variables of a container - -Kubernetes exposes [services](/docs/user-guide/services/#environment-variables) through environment variables. It is convenient to check these environment variables using `kubectl exec`. - -We first create a pod and a service, - -```shell -$ kubectl create -f examples/guestbook/redis-master-controller.yaml -$ kubectl create -f examples/guestbook/redis-master-service.yaml -``` -wait until the pod is Running and Ready, - -```shell -$ kubectl get pod -NAME READY REASON RESTARTS AGE -redis-master-ft9ex 1/1 Running 0 12s -``` - -then we can check the environment variables of the pod, - -```shell -$ kubectl exec redis-master-ft9ex env -... -REDIS_MASTER_SERVICE_PORT=6379 -REDIS_MASTER_SERVICE_HOST=10.0.0.219 -... -``` - -We can use these environment variables in applications to find the service. - - -## Using kubectl exec to check the mounted volumes - -It is convenient to use `kubectl exec` to check if the volumes are mounted as expected. -We first create a Pod with a volume mounted at /data/redis, - -```shell -kubectl create -f docs/user-guide/walkthrough/pod-redis.yaml -``` - -wait until the pod is Running and Ready, - -```shell -$ kubectl get pods -NAME READY REASON RESTARTS AGE -storage 1/1 Running 0 1m -``` - -we then use `kubectl exec` to verify that the volume is mounted at /data/redis, - -```shell -$ kubectl exec storage ls /data -redis -``` - -## Using kubectl exec to open a bash terminal in a pod - -After all, open a terminal in a pod is the most direct way to introspect the pod. Assuming the pod/storage is still running, run - -```shell -$ kubectl exec -ti storage -- bash -root@storage:/data# -``` - -This gets you a terminal. \ No newline at end of file +[Getting a Shell to a Running Container](/docs/tasks/kubectl/get-shell-running-container/) From 771a20326eb8745a426c1b62df0dc340dcf86598 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Thu, 16 Feb 2017 07:44:49 -0500 Subject: [PATCH 221/407] Let's put kubectl in ~/bin. --- docs/getting-started-guides/ubuntu/installation.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/installation.md b/docs/getting-started-guides/ubuntu/installation.md index cc93995144..9e3bdd8b2b 100644 --- a/docs/getting-started-guides/ubuntu/installation.md +++ b/docs/getting-started-guides/ubuntu/installation.md @@ -159,15 +159,17 @@ juju scp kubernetes-master/0:config ~/.kube/config Fetch a binary for the architecture you have deployed. If your client is a different architecture you will need to get the appropriate `kubectl` binary -through other means. +through other means. In this example we copy kubectl to `~/bin` for convenience, +by default this should be in your $PATH. ``` -juju scp kubernetes-master/0:kubectl ./kubectl +mkdir -p ~/bin +juju scp kubernetes-master/0:kubectl ~/bin/kubectl ``` Query the cluster: - ./kubectl cluster-info + kubectl cluster-info Output: From 8140e9c2cfacb71448a417f7500985c1399defdc Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Thu, 16 Feb 2017 16:20:08 +0800 Subject: [PATCH 222/407] remove its name from file content --- docs/admin/service-accounts-admin.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/admin/service-accounts-admin.md b/docs/admin/service-accounts-admin.md index 4a31fbeced..0c7b495a10 100644 --- a/docs/admin/service-accounts-admin.md +++ b/docs/admin/service-accounts-admin.md @@ -71,8 +71,9 @@ account. To create additional API tokens for a service account, create a secret of type `ServiceAccountToken` with an annotation referencing the service account, and the controller will update it with a generated token: -```json secret.json: + +```json { "kind": "Secret", "apiVersion": "v1", @@ -100,4 +101,4 @@ kubectl delete secret mysecretname ### Service Account Controller Service Account Controller manages ServiceAccount inside namespaces, and ensures -a ServiceAccount named "default" exists in every active namespace. \ No newline at end of file +a ServiceAccount named "default" exists in every active namespace. From 1a81cfd224d66da68c30462be9cbeca011a166b1 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Thu, 16 Feb 2017 17:17:54 +0800 Subject: [PATCH 223/407] replace kubernetes.d with kubelet.d replace kubernetes.d with kubelet.d --- docs/admin/static-pods.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 8c9e482d0d..cacf6e2010 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -26,11 +26,11 @@ For example, this is how to start a simple web server as a static pod: [joe@host ~] $ ssh my-node1 ``` -2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: +2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`: ```shell -[root@my-node1 ~] $ mkdir /etc/kubernetes.d/ -[root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml +[root@my-node1 ~] $ mkdir /etc/kubelet.d/ +[root@my-node1 ~] $ cat </etc/kubelet.d/static-web.yaml apiVersion: v1 kind: Pod metadata: @@ -114,11 +114,11 @@ CONTAINER ID IMAGE COMMAND CREATED ... Running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes pods as files appear/disappear in this directory. ```shell -[joe@my-node1 ~] $ mv /etc/kubernetes.d/static-web.yaml /tmp +[joe@my-node1 ~] $ mv /etc/kubelet.d/static-web.yaml /tmp [joe@my-node1 ~] $ sleep 20 [joe@my-node1 ~] $ docker ps // no nginx container is running -[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubernetes.d/ +[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubelet.d/ [joe@my-node1 ~] $ sleep 20 [joe@my-node1 ~] $ docker ps CONTAINER ID IMAGE COMMAND CREATED ... From b8c5b721118725c54ba1d1fe55da784ae2e24407 Mon Sep 17 00:00:00 2001 From: liangxiaoping Date: Thu, 16 Feb 2017 18:37:34 +0800 Subject: [PATCH 224/407] fix etcd disaster-recovery hyperlink --- docs/admin/etcd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/etcd.md b/docs/admin/etcd.md index 41a3e35ddb..06b0eda07c 100644 --- a/docs/admin/etcd.md +++ b/docs/admin/etcd.md @@ -20,7 +20,7 @@ Data Reliability: for reasonable safety, either etcd needs to be run as a etcd) or etcd's data directory should be located on durable storage (e.g., GCE's persistent disk). In either case, if high availability is required--as it might be in a production cluster--the data directory ought to be [backed up -periodically](https://coreos.com/etcd/docs/2.2.1/admin_guide.html#disaster-recovery), +periodically](https://coreos.com/etcd/docs/latest/op-guide/recovery.html), to reduce downtime in case of corruption. ## Default configuration From 3c262fa8d847f58e1116693228b8729a01a28b7e Mon Sep 17 00:00:00 2001 From: "architect.bian" Date: Thu, 16 Feb 2017 18:43:21 +0800 Subject: [PATCH 225/407] modify typora modify namespace-kube-system to namespace=kube-system --- docs/tasks/administer-cluster/dns-horizontal-autoscaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md b/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md index 069409058a..3d80cc77b2 100644 --- a/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md +++ b/docs/tasks/administer-cluster/dns-horizontal-autoscaling.md @@ -156,7 +156,7 @@ The output is: Verify that the replica count is zero: - kubectl get deployment --namespace-kube-system + kubectl get deployment --namespace=kube-system The output displays 0 in the DESIRED and CURRENT columns: From fe734530ba107c5f6e2f0537386adc0ecbedd7de Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 16 Feb 2017 12:59:54 -0800 Subject: [PATCH 226/407] Move Guide topic to Tasks: Downward API (#2439) --- _data/tasks.yml | 1 + .../dapi-volume-resources.yaml | 54 ++++ .../configure-pod-container/dapi-volume.yaml | 39 +++ ...nward-api-volume-expose-pod-information.md | 242 ++++++++++++++++++ ...ronment-variable-expose-pod-information.md | 12 + docs/tasks/index.md | 3 + 6 files changed, 351 insertions(+) create mode 100644 docs/tasks/configure-pod-container/dapi-volume-resources.yaml create mode 100644 docs/tasks/configure-pod-container/dapi-volume.yaml create mode 100644 docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 6d7cdb921e..5dae817f8e 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -16,6 +16,7 @@ toc: - docs/tasks/configure-pod-container/configure-volume-storage.md - docs/tasks/configure-pod-container/configure-persistent-volume-storage.md - docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md + - docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md - docs/tasks/configure-pod-container/distribute-credentials-secure.md - docs/tasks/configure-pod-container/pull-image-private-registry.md - docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md diff --git a/docs/tasks/configure-pod-container/dapi-volume-resources.yaml b/docs/tasks/configure-pod-container/dapi-volume-resources.yaml new file mode 100644 index 0000000000..65770f283f --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-volume-resources.yaml @@ -0,0 +1,54 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kubernetes-downwardapi-volume-example-2 +spec: + containers: + - name: client-container + image: gcr.io/google_containers/busybox:1.24 + command: ["sh", "-c"] + args: + - while true; do + echo -en '\n'; + if [[ -e /etc/cpu_limit ]]; then + echo -en '\n'; cat /etc/cpu_limit; fi; + if [[ -e /etc/cpu_request ]]; then + echo -en '\n'; cat /etc/cpu_request; fi; + if [[ -e /etc/mem_limit ]]; then + echo -en '\n'; cat /etc/mem_limit; fi; + if [[ -e /etc/mem_request ]]; then + echo -en '\n'; cat /etc/mem_request; fi; + sleep 5; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + volumeMounts: + - name: podinfo + mountPath: /etc + readOnly: false + volumes: + - name: podinfo + downwardAPI: + items: + - path: "cpu_limit" + resourceFieldRef: + containerName: client-container + resource: limits.cpu + - path: "cpu_request" + resourceFieldRef: + containerName: client-container + resource: requests.cpu + - path: "mem_limit" + resourceFieldRef: + containerName: client-container + resource: limits.memory + - path: "mem_request" + resourceFieldRef: + containerName: client-container + resource: requests.memory + diff --git a/docs/tasks/configure-pod-container/dapi-volume.yaml b/docs/tasks/configure-pod-container/dapi-volume.yaml new file mode 100644 index 0000000000..7126cefae5 --- /dev/null +++ b/docs/tasks/configure-pod-container/dapi-volume.yaml @@ -0,0 +1,39 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kubernetes-downwardapi-volume-example + labels: + zone: us-est-coast + cluster: test-cluster1 + rack: rack-22 + annotations: + build: two + builder: john-doe +spec: + containers: + - name: client-container + image: gcr.io/google_containers/busybox + command: ["sh", "-c"] + args: + - while true; do + if [[ -e /etc/labels ]]; then + echo -en '\n\n'; cat /etc/labels; fi; + if [[ -e /etc/annotations ]]; then + echo -en '\n\n'; cat /etc/annotations; fi; + sleep 5; + done; + volumeMounts: + - name: podinfo + mountPath: /etc + readOnly: false + volumes: + - name: podinfo + downwardAPI: + items: + - path: "labels" + fieldRef: + fieldPath: metadata.labels + - path: "annotations" + fieldRef: + fieldPath: metadata.annotations + diff --git a/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md b/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md new file mode 100644 index 0000000000..a55d22f387 --- /dev/null +++ b/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information.md @@ -0,0 +1,242 @@ +--- +title: Exposing Pod Information to Containers Using a DownwardApiVolumeFile +--- + +{% capture overview %} + +This page shows how a Pod can use a DownwardAPIVolumeFile to expose information +about itself to Containers running in the Pod. A DownwardAPIVolumeFile can expose +Pod fields and Container fields. + +{% endcapture %} + + +{% capture prerequisites %} + +{% include task-tutorial-prereqs.md %} + +{% endcapture %} + +{% capture steps %} + +## The Downward API + +There are two ways to expose Pod and Container fields to a running Container: + +* [Environment variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) +* DownwardAPIVolumeFiles + +Together, these two ways of exposing Pod and Container fields are called the +*Downward API*. + +## Storing Pod fields + +In this exercise, you create a Pod that has one Container. +Here is the configuration file for the Pod: + +{% include code.html language="yaml" file="dapi-volume.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-volume.yaml" %} + +In the configuration file, you can see that the Pod has a `downwardAPI` Volume, +and the Container mounts the Volume at `/etc`. + +Look at the `items` array under `downwardAPI`. Each element of the array is a +[DownwardAPIVolumeFile](/docs/resources-reference/v1.5/#downwardapivolumefile-v1). +The first element specifies that the value of the Pod's +`metadata.labels` field should be stored in a file named `labels`. +The second element specifies that the value of the Pod's `annotations` +field should be stored in a file named `annotations`. + +**Note**: The fields in this example are Pod fields. They are not +fields of the Container in the Pod. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-volume.yaml +``` + +Verify that Container in the Pod is running: + +```shell +kubectl get pods +``` + +View the Container's logs: + +```shell +kubectl logs kubernetes-downwardapi-volume-example +``` + +The output shows the contents of the `labels` file and the `annotations` file: + +```shell +cluster="test-cluster1" +rack="rack-22" +zone="us-est-coast" + +build="two" +builder="john-doe" +``` + +Get a shell into the Container that is running in your Pod: + +``` +kubectl exec -it kubernetes-downwardapi-volume-example -- sh +``` + +In your shell, view the `labels` file: + +```shell +/# cat /etc/labels +``` + +The output shows that all of the Pod's labels have been written +to the `labels` file: + +```shell +cluster="test-cluster1" +rack="rack-22" +zone="us-est-coast" +``` + +Similarly, view the `annotations` file: + +```shell +/# cat /etc/annotations +``` + +View the files in the `/etc` directory: + +```shell +/# ls -laR /etc +``` + +In the output, you can see that the `labels` and `annotations` files +are in a temporary subdirectory: in this example, +`..2982_06_02_21_47_53.299460680`. In the `/etc` directory, `..data` is +a symbolic link to the temporary subdirectory. Also in the `/etc` directory, +`labels` and `annotations` are symbolic links. + +``` +drwxr-xr-x ... Feb 6 21:47 ..2982_06_02_21_47_53.299460680 +lrwxrwxrwx ... Feb 6 21:47 ..data -> ..2982_06_02_21_47_53.299460680 +lrwxrwxrwx ... Feb 6 21:47 annotations -> ..data/annotations +lrwxrwxrwx ... Feb 6 21:47 labels -> ..data/labels + +/etc/..2982_06_02_21_47_53.299460680: +total 8 +-rw-r--r-- ... Feb 6 21:47 annotations +-rw-r--r-- ... Feb 6 21:47 labels +``` + +Using symbolic links enables dynamic atomic refresh of the metadata; updates are +written to a new temporary directory, and the `..data` symlink is updated +atomically using +[rename(2)](http://man7.org/linux/man-pages/man2/rename.2.html). + +Exit the shell: + +```shell +/# exit +``` + +## Storing Container fields + +The preceding exercise, you stored Pod fields in a DownwardAPIVolumeFile. +In this next exercise, you store Container fields. Here is the configuration +file for a Pod that has one Container: + +{% include code.html language="yaml" file="dapi-volume-resources.yaml" ghlink="/docs/tasks/configure-pod-container/dapi-volume-resources.yaml" %} + +In the configuration file, you can see that the Pod has a `downwardAPI` Volume, +and the Container mounts the Volume at `/etc`. + +Look at the `items` array under `downwardAPI`. Each element of the array is a +DownwardAPIVolumeFile. + +The first element specifies that in the Container named `client-container`, +the value of the `limits.cpu` field +`metadata.labels` field should be stored in a file named `cpu_limit`. + +Create the Pod: + +```shell +kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/dapi-volume-resources.yaml +``` + +Get a shell into the Container that is running in your Pod: + +``` +kubectl exec -it kubernetes-downwardapi-volume-example-2 -- sh +``` + +In your shell, view the `cpu_limit` file: + +```shell +/# cat /etc/cpu_limit +``` +You can use similar commands to view the `cpu_request`, `mem_limit` and +`mem_request` files. + +{% endcapture %} + +{% capture discussion %} + +## Capabilities of the Downward API + +The following information is available to Containers through environment +variables and DownwardAPIVolumeFiles: + +* The node’s name +* The Pod’s name +* The Pod’s namespace +* The Pod’s IP address +* The Pod’s service account name +* A Container’s CPU limit +* A container’s CPU request +* A Container’s memory limit +* A Container’s memory request + +In addition, the following information is available through +DownwardAPIVolumeFiles. + +* The Pod's labels +* The Pod's annotations + +**Note**: If CPU and memory limits are not specified for a Container, the +Downward API defaults to the node allocatable value for CPU and memory. + +## Projecting keys to specific paths and file permissions + +You can project keys to specific paths and specific permissions on a per-file +basis. For more information, see +[Secrets](/docs/user-guide/secrets/). + +## Motivation for the Downward API + +It is sometimes useful for a Container to have information about itself, without +being overly coupled to Kubernetes. The Downward API allows containers to consume +information about themselves or the cluster without using the Kubernetes client +or API server. + +An example is an existing application that assumes a particular well-known +environment variable holds a unique identifier. One possibility is to wrap the +application, but that is tedious and error prone, and it violates the goal of low +coupling. A better option would be to use the Pod's name as an identifier, and +inject the Pod's name into the well-known environment variable. + +{% endcapture %} + + +{% capture whatsnext %} + +* [PodSpec](/docs/resources-reference/v1.5/#podspec-v1) +* [Volume](/docs/resources-reference/v1.5/#volume-v1) +* [DownwardAPIVolumeSource](/docs/resources-reference/v1.5/#downwardapivolumesource-v1) +* [DownwardAPIVolumeFile](/docs/resources-reference/v1.5/#downwardapivolumefile-v1) +* [ResourceFieldSelector](/docs/resources-reference/v1.5/#resourcefieldselector-v1) + +{% endcapture %} + +{% include templates/task.md %} + diff --git a/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md index 1390727000..10ac238c5e 100644 --- a/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md +++ b/docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md @@ -26,6 +26,17 @@ Together, these two ways of exposing Pod and Container fields are called the {% capture steps %} +## The Downward API + +There are two ways to expose Pod and Container fields to a running Container: + +* Environment variables +* [DownwardAPIVolumeFiles](/docs/resources-reference/v1.5/#downwardapivolumefile-v1) + +Together, these two ways of exposing Pod and Container fields are called the +*Downward API*. + + ## Using Pod fields as values for environment variables In this exercise, you create a Pod that has one Container. Here is the @@ -161,3 +172,4 @@ The output shows the values of selected environment variables: {% include templates/task.md %} + diff --git a/docs/tasks/index.md b/docs/tasks/index.md index 39f81c6751..d490fe5532 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -12,6 +12,8 @@ single thing, typically by giving a short sequence of steps. * [Defining a Command and Arguments for a Container](/docs/tasks/configure-pod-container/define-command-argument-container/) * [Assigning CPU and RAM Resources to a Container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/) * [Configuring a Pod to Use a Volume for Storage](/docs/tasks/configure-pod-container/configure-volume-storage/) +* [Exposing Pod Information to Containers Through Environment Variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) +* [Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) * [Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) * [Pulling an Image from a Private Registry](/docs/tasks/configure-pod-container/pull-image-private-registry/) * [Configuring Liveness and Readiness Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) @@ -55,3 +57,4 @@ single thing, typically by giving a short sequence of steps. If you would like to write a task page, see [Creating a Documentation Pull Request](/docs/contribute/create-pull-request/). + From cd69b5b1e30bf2613191d0f8e80b2ac9a400b436 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 16 Feb 2017 13:20:48 -0800 Subject: [PATCH 227/407] Add note about moved content. (#2563) --- docs/user-guide/downward-api/index.md | 135 +------------------ docs/user-guide/downward-api/volume/index.md | 116 +--------------- 2 files changed, 4 insertions(+), 247 deletions(-) diff --git a/docs/user-guide/downward-api/index.md b/docs/user-guide/downward-api/index.md index 33b2b06c49..beb1cddd9b 100644 --- a/docs/user-guide/downward-api/index.md +++ b/docs/user-guide/downward-api/index.md @@ -5,137 +5,6 @@ assignees: title: Using the Downward API to Convey Pod Properties --- -It is sometimes useful for a container to have information about itself, but we -want to be careful not to over-couple containers to Kubernetes. The downward -API allows containers to consume information about themselves or the system and -expose that information how they want it, without necessarily coupling to the -Kubernetes client or REST API. +{% include user-guide-content-moved.md %} -An example of this is a "legacy" app that is already written assuming -that a particular environment variable will hold a unique identifier. While it -is often possible to "wrap" such applications, this is tedious and error prone, -and violates the goal of low coupling. Instead, the user should be able to use -the Pod's name, for example, and inject it into this well-known variable. - - -## Capabilities - -The following information is available to a `Pod` through the downward API: - -* The node's name -*   The pod's name -* The pod's namespace -* The pod's IP -* The pod's service account name -* A container's cpu limit -* A container's cpu request -* A container's memory limit -* A container's memory request - -More information will be exposed through this same API over time. - - -## Exposing pod information into a container - -Containers consume information from the downward API using environment -variables or using a volume plugin. - - -## Environment variables - -Most environment variables in the Kubernetes API use the `value` field to carry -simple values. However, the alternate `valueFrom` field allows you to specify -a `fieldRef` to select fields from the pod's definition, and a `resourceFieldRef` -to select fields from one of its container's definition. - -The `fieldRef` field is a structure that has an `apiVersion` field and a `fieldPath` -field. The `fieldPath` field is an expression designating a field of the pod. The -`apiVersion` field is the version of the API schema that the `fieldPath` is -written in terms of. If the `apiVersion` field is not specified it is -defaulted to the API version of the enclosing object. - -The `fieldRef` is evaluated and the resulting value is used as the value for -the environment variable. This allows users to publish their pod's name in any -environment variable they want. - -The `resourceFieldRef` is a structure that has a `containerName` field, a `resource` -field, and a `divisor` field. The `containerName` is the name of a container, -whose resource (cpu or memory) information is to be exposed. The `containerName` is -optional for environment variables and defaults to the current container. The -`resource` field is an expression designating a resource in a container, and the `divisor` -field specifies an output format of the resource being exposed. If the `divisor` -is not specified, it defaults to "1" for cpu and memory. The table shows possible -values for cpu and memory resources for `resource` and `divisor` settings: - - -| Setting | Cpu | Memory | -| ------------- |-------------| -----| -| resource | limits.cpu, requests.cpu| limits.memory, requests.memory| -| divisor | 1(cores), 1m(millicores) | 1(bytes), 1k(kilobytes), 1M(megabytes), 1G(gigabytes), 1T(terabytes), 1P(petabytes), 1E(exabytes), 1Ki(kibibyte), 1Mi(mebibyte), 1Gi(gibibyte), 1Ti(tebibyte), 1Pi(pebibyte), 1Ei(exbibyte)| - - -### Example - -This is an example of a pod that consumes its name and namespace via the -downward API: - -{% include code.html language="yaml" file="dapi-pod.yaml" ghlink="/docs/user-guide/downward-api/dapi-pod.yaml" %} - -This is an example of a pod that consumes its container's resources via the downward API: - -{% include code.html language="yaml" file="dapi-container-resources.yaml" ghlink="/docs/user-guide/downward-api/dapi-container-resources.yaml" %} - -## Downward API volume - -Using a similar syntax it's possible to expose pod information to containers using plain text files. -Downward API are dumped to a mounted volume. This is achieved using a `downwardAPI` -volume type and the different items represent the files to be created. `fieldPath` references the field to be exposed. -For exposing a container's resources limits and requests, `containerName` must be specified with `resourceFieldRef`. - -Downward API volume permits to store more complex data like [`metadata.labels`](/docs/user-guide/labels) and [`metadata.annotations`](/docs/user-guide/annotations). Currently key/value pair set fields are saved using `key="value"` format: - -```conf -key1="value1" -key2="value2" -``` - -In future, it will be possible to specify an output format option. - -Downward API volumes can expose: - -* The node's name -* The pod's name -* The pod's namespace -* The pod's labels -* The pod's annotations -* The pod's service account name -* A container's cpu limit -* A container's cpu request -* A container's memory limit -* A container's memory request - -The downward API volume refreshes its data in step with the kubelet refresh loop. When labels will be modifiable on the fly without respawning the pod containers will be able to detect changes through mechanisms such as [inotify](https://en.wikipedia.org/wiki/Inotify). - -In future, it will be possible to specify a specific annotation or label. - -#### Projecting keys to specific paths and file permissions - -You can project keys to specific paths and specific permissions on a per-file -basis. The [Secrets](/docs/user-guide/secrets/) user guide explains the syntax. - -### Example - -This is an example of a pod that consumes its labels and annotations via the downward API volume, labels and annotations are dumped in `/etc/labels` and in `/etc/annotations`, respectively: - -{% include code.html language="yaml" file="volume/dapi-volume.yaml" ghlink="/docs/user-guide/downward-api/volume/dapi-volume.yaml" %} - -This is an example of a pod that consumes its container's resources via the downward API volume. - -{% include code.html language="yaml" file="volume/dapi-volume-resources.yaml" ghlink="/docs/user-guide/downward-api/volume/dapi-volume-resources.yaml" %} - -For a more thorough example, see -[environment variables](/docs/user-guide/environment-guide/). - -## Default values for container resource limits - -If cpu and memory limits are not specified for a container, the downward API will default to the node allocatable value for cpu and memory. +[Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) diff --git a/docs/user-guide/downward-api/volume/index.md b/docs/user-guide/downward-api/volume/index.md index 882b2da1e9..6f359cc797 100644 --- a/docs/user-guide/downward-api/volume/index.md +++ b/docs/user-guide/downward-api/volume/index.md @@ -2,118 +2,6 @@ title: Downward API Volumes --- -Following this example, you will create a pod with a downward API volume. -A downward API volume is a k8s volume plugin with the ability to save some pod information in a plain text file. The pod information can be for example some [metadata](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#metadata) or a container's [resources](/docs/user-guide/compute-resources). +{% include user-guide-content-moved.md %} -Supported metadata fields: - -1. `metadata.annotations` -2. `metadata.namespace` -3. `metadata.name` -4. `metadata.labels` - -Supported container's resources: - -1. `limits.cpu` -2. `limits.memory` -3. `requests.cpu` -4. `requests.memory` - -### Step Zero: Prerequisites - -This example assumes you have a Kubernetes cluster installed and running, and the `kubectl` command line tool somewhere in your path. Please see the [gettingstarted](/docs/getting-started-guides/) for installation instructions for your platform. - -### Step One: Create the pod - -Use the [dapi-volume.yaml](/docs/user-guide/downward-api/volume/dapi-volume.yaml) file to create a Pod with a downward API volume which stores pod labels and pod annotations to `/etc/labels` and `/etc/annotations` respectively. - -```shell -$ kubectl create -f docs/user-guide/downward-api/volume/dapi-volume.yaml -``` - -### Step Two: Examine pod/container output - -The pod displays (every 5 seconds) the content of the dump files which can be executed via the usual `kubectl log` command - -```shell -$ kubectl logs kubernetes-downwardapi-volume-example -cluster="test-cluster1" -rack="rack-22" -zone="us-est-coast" -build="two" -builder="john-doe" -kubernetes.io/config.seen="2015-08-24T13:47:23.432459138Z" -kubernetes.io/config.source="api" -``` - -### Internals - -In pod's `/etc` directory one may find the file created by the plugin (system files elided): - -```shell -$ kubectl exec kubernetes-downwardapi-volume-example -i -t -- sh -/ # ls -laR /etc -/etc: -total 4 -drwxrwxrwt 3 0 0 120 Jun 1 19:55 . -drwxr-xr-x 17 0 0 4096 Jun 1 19:55 .. -drwxr-xr-x 2 0 0 80 Jun 1 19:55 ..6986_01_06_15_55_10.473583074 -lrwxrwxrwx 1 0 0 31 Jun 1 19:55 ..data -> ..6986_01_06_15_55_10.473583074 -lrwxrwxrwx 1 0 0 18 Jun 1 19:55 annotations -> ..data/annotations -lrwxrwxrwx 1 0 0 13 Jun 1 19:55 labels -> ..data/labels - -/etc/..6986_01_06_15_55_10.473583074: -total 8 -drwxr-xr-x 2 0 0 80 Jun 1 19:55 . -drwxrwxrwt 3 0 0 120 Jun 1 19:55 .. --rw-r--r-- 1 0 0 129 Jun 1 19:55 annotations --rw-r--r-- 1 0 0 59 Jun 1 19:55 labels -/ # -``` - -The file `labels` is stored in a temporary directory (`..6986_01_06_15_55_10.473583074` in the example above) which is symlinked to by `..data`. Symlinks for annotations and labels in `/etc` point to files containing the actual metadata through the `..data` indirection.  This structure allows for dynamic atomic refresh of the metadata: updates are written to a new temporary directory, and the `..data` symlink is updated atomically using `rename(2)`. - -## Example of downward API volume with container resources - -Use the `docs/user-guide/downward-api/volume/dapi-volume-resources.yaml` file to create a Pod with a downward API volume which stores its container's limits and requests in /etc. - -```shell -$ kubectl create -f docs/user-guide/downward-api/volume/dapi-volume-resources.yaml -``` - -### Examine pod/container output - -In pod's `/etc` directory one may find the files created by the plugin: - -```shell -$ kubectl exec kubernetes-downwardapi-volume-example -i -t -- sh -/ # ls -alR /etc -/etc: -total 4 -drwxrwxrwt 3 0 0 160 Jun 1 19:47 . -drwxr-xr-x 17 0 0 4096 Jun 1 19:48 .. -drwxr-xr-x 2 0 0 120 Jun 1 19:47 ..6986_01_06_15_47_23.076909525 -lrwxrwxrwx 1 0 0 31 Jun 1 19:47 ..data -> ..6986_01_06_15_47_23.076909525 -lrwxrwxrwx 1 0 0 16 Jun 1 19:47 cpu_limit -> ..data/cpu_limit -lrwxrwxrwx 1 0 0 18 Jun 1 19:47 cpu_request -> ..data/cpu_request -lrwxrwxrwx 1 0 0 16 Jun 1 19:47 mem_limit -> ..data/mem_limit -lrwxrwxrwx 1 0 0 18 Jun 1 19:47 mem_request -> ..data/mem_request - -/etc/..6986_01_06_15_47_23.076909525: -total 16 -drwxr-xr-x 2 0 0 120 Jun 1 19:47 . -drwxrwxrwt 3 0 0 160 Jun 1 19:47 .. --rw-r--r-- 1 0 0 1 Jun 1 19:47 cpu_limit --rw-r--r-- 1 0 0 1 Jun 1 19:47 cpu_request --rw-r--r-- 1 0 0 8 Jun 1 19:47 mem_limit --rw-r--r-- 1 0 0 8 Jun 1 19:47 mem_request - -/ # cat /etc/cpu_limit -1 -/ # cat /etc/mem_limit -67108864 -/ # cat /etc/cpu_request -1 -/ # cat /etc/mem_request -33554432 -``` +[Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) From 17daf981f7476756611414e4abc247baa8c1d736 Mon Sep 17 00:00:00 2001 From: Guillaume Breton Date: Thu, 16 Feb 2017 18:07:10 +0000 Subject: [PATCH 228/407] Fix typos in running zookeeper article --- docs/tutorials/stateful-application/zookeeper.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/stateful-application/zookeeper.md b/docs/tutorials/stateful-application/zookeeper.md index ba2bb219d6..e31d8e280b 100644 --- a/docs/tutorials/stateful-application/zookeeper.md +++ b/docs/tutorials/stateful-application/zookeeper.md @@ -580,7 +580,7 @@ env: key: purge.interval ``` -The entry point of the container invokes a bash script, `zkConfig.sh`, prior to +The entry point of the container invokes a bash script, `zkGenConfig.sh`, prior to launching the ZooKeeper server process. This bash script generates the ZooKeeper configuration files from the supplied environment variables. @@ -653,7 +653,7 @@ ZK_LOG_DIR=/var/log/zookeeper ### Configuring Logging -One of the files generated by the `zkConfigGen.sh` script controls ZooKeeper's logging. +One of the files generated by the `zkGenConfig.sh` script controls ZooKeeper's logging. ZooKeeper uses [Log4j](http://logging.apache.org/log4j/2.x/), and, by default, it uses a time and size based rolling file appender for its logging configuration. Get the logging configuration from one of Pods in the `zk` StatefulSet. From e854c082ad5e3144ffd0f023b55d2eec8e19db02 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Fri, 17 Feb 2017 16:54:23 +0800 Subject: [PATCH 229/407] fix typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1、column PORTS is missed 2、fix command `kubectl get pods`output --- docs/admin/static-pods.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index cacf6e2010..610da03da7 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -72,8 +72,8 @@ When kubelet starts, it automatically starts all pods defined in directory speci ```shell [joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES -f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c ``` If we look at our Kubernetes API server (running on host `my-master`), we see that a new mirror-pod was created there too: @@ -81,9 +81,9 @@ If we look at our Kubernetes API server (running on host `my-master`), we see th ```shell [joe@host ~] $ ssh my-master [joe@my-master ~] $ kubectl get pods -POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE -static-web-my-node1 172.17.0.3 my-node1/192.168.100.71 role=myrole Running 11 minutes - web nginx Running 11 minutes +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 2m + ``` Labels from the static pod are propagated into the mirror-pod and can be used as usual for filtering. @@ -94,8 +94,9 @@ Notice we cannot delete the pod with the API server (e.g. via [`kubectl`](/docs/ [joe@my-master ~] $ kubectl delete pod static-web-my-node1 pods/static-web-my-node1 [joe@my-master ~] $ kubectl get pods -POD IP CONTAINER(S) IMAGE(S) HOST ... -static-web-my-node1 172.17.0.3 my-node1/192.168.100.71 ... +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 12s + ``` Back to our `my-node1` host, we can try to stop the container manually and see, that kubelet automatically restarts it in a while: From 632b2ba4742ac1e8517e194f1887055cdfd6ca29 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Wed, 15 Feb 2017 16:33:38 +0800 Subject: [PATCH 230/407] modify one word examples to example --- docs/user-guide/walkthrough/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/walkthrough/index.md b/docs/user-guide/walkthrough/index.md index d7872b2260..e0400ec7ed 100644 --- a/docs/user-guide/walkthrough/index.md +++ b/docs/user-guide/walkthrough/index.md @@ -9,7 +9,7 @@ title: Kubernetes 101 For Kubernetes 101, we will cover kubectl, pods, volumes, and multiple containers -In order for the kubectl usage examples to work, make sure you have an examples directory locally, either from [a release](https://github.com/kubernetes/kubernetes/releases) or [the source](https://github.com/kubernetes/kubernetes). +In order for the kubectl usage examples to work, make sure you have an example directory locally, either from [a release](https://github.com/kubernetes/kubernetes/releases) or [the source](https://github.com/kubernetes/kubernetes). * TOC {:toc} From f8c25df15616e25140c84a2fedfe6c5f8adf05e6 Mon Sep 17 00:00:00 2001 From: Michael Fraenkel Date: Fri, 17 Feb 2017 08:34:06 -0500 Subject: [PATCH 231/407] Revert "mend" This reverts commit 324451c3378b4ba01d27e3db6117f86d0517a598. --- docs/user-guide/configmap/index.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 40b3fa181b..36d3e89394 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -317,7 +317,10 @@ spec: restartPolicy: Never ``` -When this pod is run, the output will be empty. +When this pod is run, its output will include the lines: + +```shell +``` ### Use-Case: Set command-line arguments with ConfigMap From ce82d6301da92b61818ce2d7135ee206ffe463e2 Mon Sep 17 00:00:00 2001 From: Michael Fraenkel Date: Fri, 17 Feb 2017 08:34:15 -0500 Subject: [PATCH 232/407] Revert "Document new optional support for ConfigMap and Secret" This reverts commit 7abcc6c8540a7b0a4f6712e77e0f5cb134271bed. --- docs/user-guide/configmap/index.md | 70 ++-------------------------- docs/user-guide/secrets/index.md | 74 +++--------------------------- 2 files changed, 10 insertions(+), 134 deletions(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 36d3e89394..1355e599f2 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -291,37 +291,6 @@ SPECIAL_LEVEL_KEY=very SPECIAL_TYPE_KEY=charm ``` -#### Optional ConfigMap in environment variables - -There might be situations where environment variables are not -always required. These environment variables can be marked as optional in a -pod like so: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: dapi-test-pod -spec: - containers: - - name: test-container - image: gcr.io/google_containers/busybox - command: [ "/bin/sh", "-c", "env" ] - env: - - name: SPECIAL_LEVEL_KEY - valueFrom: - configMapKeyRef: - name: a-config - key: akey - optional: true - restartPolicy: Never -``` - -When this pod is run, its output will include the lines: - -```shell -``` - ### Use-Case: Set command-line arguments with ConfigMap ConfigMaps can also be used to set the value of the command or arguments in a container. This is @@ -453,38 +422,6 @@ very You can project keys to specific paths and specific permissions on a per-file basis. The [Secrets](/docs/user-guide/secrets/) user guide explains the syntax. -#### Optional ConfigMap via volume plugin - -Volumes and files provided by a ConfigMap can be also be marked as optional. -The ConfigMap or the key specified does not have to exist. The mount path for -such items will always be created. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: dapi-test-pod -spec: - containers: - - name: test-container - image: gcr.io/google_containers/busybox - command: [ "/bin/sh", "-c", "ls /etc/config" ] - volumeMounts: - - name: config-volume - mountPath: /etc/config - volumes: - - name: config-volume - configMap: - name: no-config - optional: true - restartPolicy: Never -``` - -When this pod is run, the output will be: - -```shell -``` - ## Real World Example: Configuring Redis Let's take a look at a real-world example: configuring redis using ConfigMap. Say we want to inject @@ -580,10 +517,9 @@ $ kubectl exec -it redis redis-cli ## Restrictions -ConfigMaps must be created before they are consumed in pods unless they are -marked as optional. Controllers may be written to tolerate missing -configuration data; consult individual components configured via ConfigMap on -a case-by-case basis. +ConfigMaps must be created before they are consumed in pods. Controllers may be written to tolerate +missing configuration data; consult individual components configured via ConfigMap on a case-by-case +basis. ConfigMaps reside in a namespace. They can only be referenced by pods in the same namespace. diff --git a/docs/user-guide/secrets/index.md b/docs/user-guide/secrets/index.md index 9723e00aee..29e9044fde 100644 --- a/docs/user-guide/secrets/index.md +++ b/docs/user-guide/secrets/index.md @@ -375,41 +375,6 @@ However, it is using its local ttl-based cache for getting the current value of As a result, the total delay from the moment when the secret is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of secrets cache in kubelet. -#### Optional Secrets as Files from a Pod - -Volumes and files provided by a Secret can be also be marked as optional. -The Secret or the key within a Secret does not have to exist. The mount path for -such items will always be created. - -```json -{ - "apiVersion": "v1", - "kind": "Pod", - "metadata": { - "name": "mypod", - "namespace": "myns" - }, - "spec": { - "containers": [{ - "name": "mypod", - "image": "redis", - "volumeMounts": [{ - "name": "foo", - "mountPath": "/etc/foo" - }] - }], - "volumes": [{ - "name": "foo", - "secret": { - "secretName": "mysecret", - "defaultMode": 256, - "optional": true - } - }] - } -} -``` - #### Using Secrets as Environment Variables To use a secret in an environment variable in a pod: @@ -456,30 +421,6 @@ $ echo $SECRET_PASSWORD 1f2d1e2e67df ``` -#### Optional Secrets from Environment Variables - -You may not want to require all your secrets to exist. They can be marked as -optional as shown in the pod: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: optional-secret-env-pod -spec: - containers: - - name: mycontainer - image: redis - env: - - name: OPTIONAL_SECRET - valueFrom: - secretKeyRef: - name: mysecret - key: username - optional: true - restartPolicy: Never -``` - #### Using imagePullSecrets An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry @@ -511,8 +452,7 @@ can be automatically attached to pods based on their service account. Secret volume sources are validated to ensure that the specified object reference actually points to an object of type `Secret`. Therefore, a secret -needs to be created before any pods that depend on it, unless it is marked as -optional. +needs to be created before any pods that depend on it. Secret API objects reside in a namespace. They can only be referenced by pods in that same namespace. @@ -532,12 +472,12 @@ not common ways to create pods.) When a pod is created via the API, there is no check whether a referenced secret exists. Once a pod is scheduled, the kubelet will try to fetch the -secret value. If a required secret cannot be fetched because it does not -exist or because of a temporary lack of connection to the API server, the -kubelet will periodically retry. It will report an event about the pod -explaining the reason it is not started yet. Once the secret is fetched, the -kubelet will create and mount a volume containing it. None of the pod's -containers will start until all the pod's volumes are mounted. +secret value. If the secret cannot be fetched because it does not exist or +because of a temporary lack of connection to the API server, kubelet will +periodically retry. It will report an event about the pod explaining the +reason it is not started yet. Once the secret is fetched, the kubelet will +create and mount a volume containing it. None of the pod's containers will +start until all the pod's volumes are mounted. ## Use cases From 8e490b4251601507ead8f853a2fb0c74431aa280 Mon Sep 17 00:00:00 2001 From: Harry Zhang Date: Sat, 18 Feb 2017 03:06:01 +0800 Subject: [PATCH 233/407] Add kubelet eviction in GC doc (#2484) --- docs/admin/garbage-collection.md | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index a960172426..9717a41dd8 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -42,15 +42,34 @@ to free. Default is 80%. We also allow users to customize garbage collection policy through the following kubelet flags: 1. `minimum-container-ttl-duration`, minimum age for a finished container before it is -garbage collected. Default is 1 minute. +garbage collected. Default is 0 minute, which means any finished container will be garbaged collected. 2. `maximum-dead-containers-per-container`, maximum number of old instances to retain -per container. Default is 2. +per container. Default is 1. 3. `maximum-dead-containers`, maximum number of old instances of containers to retain globally. -Default is 100. +Default is -1, which means there is no global limit. Containers can potentially be garbage collected before their usefulness has expired. These containers can contain logs and other data that can be useful for troubleshooting. A sufficiently large value for -`maximum-dead-containers-per-container` is highly recommended to allow at least 2 dead containers to be +`maximum-dead-containers-per-container` is highly recommended to allow at least 1 dead container to be retained per expected container. A higher value for `maximum-dead-containers` is also recommended for a similar reason. See [this issue](https://github.com/kubernetes/kubernetes/issues/13287) for more details. + + +### Deprecation + +Some kubelet Garbage Collection features in this doc will be replaced by kubelet eviction in the future. + +Including: + +| Existing Flag | New Flag | Rationale | +| ------------- | -------- | --------- | +| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` | existing eviction signals can trigger image garbage collection | +| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | eviction reclaims achieve the same behavior | +| `--maximum-dead-containers` | | deprecated once old logs are stored outside of container's context | +| `--maximum-dead-containers-per-container` | | deprecated once old logs are stored outside of container's context | +| `--minimum-container-ttl-duration` | | deprecated once old logs are stored outside of container's context | +| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources | +| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources | + +See [kubelet eviction design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-eviction.md) for more details. From 7022b43bc23b959a837bb85b924bf0768231f62c Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Mon, 20 Feb 2017 10:24:13 +0800 Subject: [PATCH 234/407] dedent yaml code blocks otherwise extra redundant ````yaml` characters will be displayed because github code fencing doesn't need any indentation --- docs/admin/kubeadm.md | 80 +++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index edf9c87a7d..a43beec497 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -180,49 +180,49 @@ available as configuration file options. ### Sample Master Configuration - ```yaml - apiVersion: kubeadm.k8s.io/v1alpha1 - kind: MasterConfiguration - api: - advertiseAddresses: - - - - - bindPort: - externalDNSNames: - - - - - authorizationMode: - cloudProvider: - discovery: - bindPort: - etcd: - endpoints: - - - - - caFile: - certFile: - keyFile: - kubernetesVersion: - networking: - dnsDomain: - serviceSubnet: - podSubnet: - secrets: - givenToken: - ``` +```yaml +apiVersion: kubeadm.k8s.io/v1alpha1 +kind: MasterConfiguration +api: + advertiseAddresses: + - + - + bindPort: + externalDNSNames: + - + - +authorizationMode: +cloudProvider: +discovery: + bindPort: +etcd: + endpoints: + - + - + caFile: + certFile: + keyFile: +kubernetesVersion: +networking: + dnsDomain: + serviceSubnet: + podSubnet: +secrets: + givenToken: +``` ### Sample Node Configuration - ```yaml - apiVersion: kubeadm.k8s.io/v1alpha1 - kind: NodeConfiguration - apiPort: - discoveryPort: - masterAddresses: - - - secrets: - givenToken: - ``` +```yaml +apiVersion: kubeadm.k8s.io/v1alpha1 +kind: NodeConfiguration +apiPort: +discoveryPort: +masterAddresses: +- +secrets: + givenToken: +``` ## Automating kubeadm From dc053ed54fd3e5a8964915cd04bde4d739eb6e4e Mon Sep 17 00:00:00 2001 From: xuxinkun Date: Fri, 17 Feb 2017 18:19:33 +0800 Subject: [PATCH 235/407] Add more options for self-registration. --- docs/admin/node.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/admin/node.md b/docs/admin/node.md index 0923169256..f91f155919 100644 --- a/docs/admin/node.md +++ b/docs/admin/node.md @@ -176,6 +176,9 @@ For self-registration, the kubelet is started with the following options: - `--kubeconfig=` - Path to credentials to authenticate itself to the apiserver. - `--cloud-provider=` - How to talk to a cloud provider to read metadata about itself. - `--register-node` - Automatically register with the API server. + - `--node-ip` IP address of the node. + - `--node-labels` - Labels to add when registering the node in the cluster. + - `--node-status-update-frequency` - Specifies how often kubelet posts node status to master. Currently, any kubelet is authorized to create/modify any node resource, but in practice it only creates/modifies its own. (In the future, we plan to only allow a kubelet to modify its own node resource.) From dfb8b79b15ddea13a96fefee9dd30666ca1b7464 Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Thu, 16 Feb 2017 16:44:45 +0800 Subject: [PATCH 236/407] Parallel Jobs with a *work queue*: do not specify `.spec.completions`, default to `.spec.Parallelism`, not default to 1 --- docs/user-guide/jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index d64ae9d730..438b63dff7 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -120,7 +120,7 @@ There are three main types of jobs: - the job is complete when there is one successful pod for each value in the range 1 to `.spec.completions`. - **not implemented yet:** each pod passed a different index in the range 1 to `.spec.completions`. 1. Parallel Jobs with a *work queue*: - - do not specify `.spec.completions` + - do not specify `.spec.completions`, default to `.spec.Parallelism` - the pods must coordinate with themselves or an external service to determine what each should work on - each pod is independently capable of determining whether or not all its peers are done, thus the entire Job is done. - when _any_ pod terminates with success, no new pods are created. From b807813244847ccea3b5fdb2c61439334cf63422 Mon Sep 17 00:00:00 2001 From: Olegs Jeremejevs Date: Tue, 21 Feb 2017 17:48:51 +0200 Subject: [PATCH 237/407] Fix typo in concepts --- docs/concepts/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 93e4533caf..8e738e809e 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -17,7 +17,7 @@ Once you've set your desired state, the *Kubernetes Control Plane* works to make ## Kubernetes Objects -Kubernetes contains a number of abstractions that represent your the state of your system: deployed containerized applications and workloads, their associated network and disk resources, and other information about what your cluster is doing. These abstractions are represented by objects in the Kubernetes API; see the [Kubernetes Objects overview](/docs/concepts/abstractions/overview/) for more details. +Kubernetes contains a number of abstractions that represent the state of your system: deployed containerized applications and workloads, their associated network and disk resources, and other information about what your cluster is doing. These abstractions are represented by objects in the Kubernetes API; see the [Kubernetes Objects overview](/docs/concepts/abstractions/overview/) for more details. The basic Kubernetes objects include: From 65eacf1145151a8be3b019a0f688fac5a488f47a Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 21 Feb 2017 15:23:24 +0800 Subject: [PATCH 238/407] fix typo --- .../configure-pod-container/pull-image-private-registry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/pull-image-private-registry.md b/docs/tasks/configure-pod-container/pull-image-private-registry.md index e9b640873a..f54d881178 100644 --- a/docs/tasks/configure-pod-container/pull-image-private-registry.md +++ b/docs/tasks/configure-pod-container/pull-image-private-registry.md @@ -80,7 +80,7 @@ Copy the base64 representation of the secret data into a file named `secret64`. **Important**: Make sure there are no line breaks in your `secret64` file. -To understand what is in the `dockercfg` field, convert the secret data to a +To understand what is in the `.dockercfg` field, convert the secret data to a readable format: base64 -d secret64 From 4806f01d8f8e646bcfbb513811e5fab339c9d337 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 21 Feb 2017 11:52:18 +0800 Subject: [PATCH 239/407] add an example --- docs/concepts/configuration/container-command-args.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/concepts/configuration/container-command-args.md b/docs/concepts/configuration/container-command-args.md index 25d555f702..3f39baff57 100644 --- a/docs/concepts/configuration/container-command-args.md +++ b/docs/concepts/configuration/container-command-args.md @@ -66,6 +66,7 @@ Here are some examples: | `[/ep-1]` | `[foo bar]` | <not set> | <not set> | `[ep-1 foo bar]` | | `[/ep-1]` | `[foo bar]` | `[/ep-2]` | <not set> | `[ep-2]` | | `[/ep-1]` | `[foo bar]` | <not set> | `[zoo boo]` | `[ep-1 zoo boo]` | +| `[/ep-1]` | `[foo bar]` | `[/ep-2]` | `[zoo boo]` | `[ep-2 zoo boo]` | {% endcapture %} From 406fe2683bff73ac4fb5dc3040dc3e71ed5393ba Mon Sep 17 00:00:00 2001 From: Ben Mathews Date: Fri, 17 Feb 2017 14:39:48 -0700 Subject: [PATCH 240/407] Remove reference to port-forward to services port forwarding to services isn't supported. --- docs/user-guide/kubectl-cheatsheet.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/user-guide/kubectl-cheatsheet.md b/docs/user-guide/kubectl-cheatsheet.md index 7fc0142f04..6b67cba4a0 100644 --- a/docs/user-guide/kubectl-cheatsheet.md +++ b/docs/user-guide/kubectl-cheatsheet.md @@ -203,7 +203,6 @@ $ kubectl logs -f my-pod -c my-container # stream pod container log $ kubectl run -i --tty busybox --image=busybox -- sh # Run pod as interactive shell $ kubectl attach my-pod -i # Attach to Running Container $ kubectl port-forward my-pod 5000:6000 # Forward port 6000 of Pod to your to 5000 on your local machine -$ kubectl port-forward my-svc 6000 # Forward port to service $ kubectl exec my-pod -- ls / # Run command in existing pod (1 container case) $ kubectl exec my-pod -c my-container -- ls / # Run command in existing pod (multi-container case) $ kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers From 363db9d8afcd1cd330fc60cacac457be9eb09b10 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Mon, 20 Feb 2017 22:37:54 +0800 Subject: [PATCH 241/407] Fixed typos in persistent volume index.md --- docs/user-guide/persistent-volumes/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index 4508a1a25f..e2d2661d13 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -421,7 +421,7 @@ parameters: * `restauthenabled` : Gluster REST service authentication boolean that enables authentication to the REST server. If this value is 'true', `restuser` and `restuserkey` or `secretNamespace` + `secretName` have to be filled. This option is deprecated, authentication is enabled when any of `restuser`, `restuserkey`, `secretName` or `secretNamespace` is specified. * `restuser` : Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool. * `restuserkey` : Gluster REST service/Heketi user's password which will be used for authentication to the REST server. This parameter is deprecated in favor of `secretNamespace` + `secretName`. -* `secretNamespace` + `secretName` : Identification of Secret instance that containes user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must have type "kubernetes.io/glusterfs", e.g. created in this way: +* `secretNamespace` + `secretName` : Identification of Secret instance that contains user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must have type "kubernetes.io/glusterfs", e.g. created in this way: ``` $ kubectl create secret generic heketi-secret --type="kubernetes.io/glusterfs" --from-literal=key='opensesame' --namespace=default ``` @@ -507,7 +507,7 @@ parameters: * `quobyteAPIServer`: API Server of Quobyte in the format `http(s)://api-server:7860` * `registry`: Quobyte registry to use to mount the volume. You can specify the registry as ``:`` pair or if you want to specify multiple registries you just have to put a comma between them e.q. ``:,:,:``. The host can be an IP address or if you have a working DNS you can also provide the DNS names. * `adminSecretNamespace`: The namespace for `adminSecretName`. Default is "default". -* `adminSecretName`: secret that holds information about the Quobyte user and the password to authenticate agains the API server. The provided secret must have type "kubernetes.io/quobyte", e.g. created in this way: +* `adminSecretName`: secret that holds information about the Quobyte user and the password to authenticate against the API server. The provided secret must have type "kubernetes.io/quobyte", e.g. created in this way: ``` $ kubectl create secret generic quobyte-admin-secret --type="kubernetes.io/quobyte" --from-literal=key='opensesame' --namespace=kube-system ``` From 4731f33d4887f500da3ab8811530d2695e090c22 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Mon, 20 Feb 2017 21:29:07 +0800 Subject: [PATCH 242/407] Highlight some keyworks in garbage-collection.md. --- docs/admin/garbage-collection.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 9717a41dd8..8164fbb7d9 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -24,9 +24,9 @@ threshold has been met. ### Container Collection The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers any single -pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting 'MinAge' to zero and setting 'MaxPerPodContainer' and 'MaxContainers' respectively to less than zero. +pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting `MinAge` to zero and setting `MaxPerPodContainer` and `MaxContainers` respectively to less than zero. -Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. 'MaxPerPodContainer' and 'MaxContainer' may potentially conflict with each other in situations where retaining the maximum number of containers per pod ('MaxPerPodContainer') would go outside the allowable range of global dead containers ('MaxContainers'). 'MaxPerPodContainer' would be adjusted in this situation: A worst case scenario would be to downgrade 'MaxPerPodContainer' to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. +Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. `MaxPerPodContainer` and `MaxContainer` may potentially conflict with each other in situations where retaining the maximum number of containers per pod (`MaxPerPodContainer`) would go outside the allowable range of global dead containers (`MaxContainers`). `MaxPerPodContainer` would be adjusted in this situation: A worst case scenario would be to downgrade `MaxPerPodContainer` to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. Containers that are not managed by kubelet are not subject to container garbage collection. From 0b9dca5a36f81187fb8079cdb21211fab416259c Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Sat, 18 Feb 2017 15:14:37 +0800 Subject: [PATCH 243/407] replace argumment with argument replace argumment with argument --- docs/admin/network-plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/network-plugins.md b/docs/admin/network-plugins.md index d5e05ac7b1..f06d22fb87 100644 --- a/docs/admin/network-plugins.md +++ b/docs/admin/network-plugins.md @@ -49,7 +49,7 @@ The plugin requires a few things: * The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `network-plugin-dir` to supply additional search path. The first found match will take effect. * Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin -* Kubelet should also be run with the `--non-masquerade-cidr=` argumment to ensure traffic to IPs outside this range will use IP masquerade. +* Kubelet should also be run with the `--non-masquerade-cidr=` argument to ensure traffic to IPs outside this range will use IP masquerade. * The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=` controller-manager command-line options. ### Customizing the MTU (with kubenet) From 142bf7dcc4293678d3f14b6e844d86bfa3720787 Mon Sep 17 00:00:00 2001 From: "Ricardo M. Augusto" Date: Tue, 21 Feb 2017 13:16:27 -0300 Subject: [PATCH 244/407] Updated result of command Actually according to the command in example: kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub The keys will be available with names ssh-publickey and ssh-privatekey. --- docs/user-guide/secrets/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/secrets/index.md b/docs/user-guide/secrets/index.md index 29e9044fde..8b7c7ae2fb 100644 --- a/docs/user-guide/secrets/index.md +++ b/docs/user-guide/secrets/index.md @@ -534,8 +534,8 @@ consumes it in a volume: When the container's command runs, the pieces of the key will be available in: ```shell -/etc/secret-volume/id-rsa.pub -/etc/secret-volume/id-rsa +/etc/secret-volume/ssh-publickey +/etc/secret-volume/ssh-privatekey ``` The container is then free to use the secret data to establish an ssh connection. From b89155a35103b93aaa755c4877e9edbb1c778f83 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Tue, 21 Feb 2017 16:08:27 +0800 Subject: [PATCH 245/407] replace petset with statefulsets --- docs/user-guide/kubectl-cheatsheet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl-cheatsheet.md b/docs/user-guide/kubectl-cheatsheet.md index 6b67cba4a0..56f6d71de4 100644 --- a/docs/user-guide/kubectl-cheatsheet.md +++ b/docs/user-guide/kubectl-cheatsheet.md @@ -243,7 +243,7 @@ Resource type | Abbreviated alias `namespaces` |`ns` `networkpolicies` | `nodes` |`no` -`petset` | +`statefulsets` | `persistentvolumeclaims` |`pvc` `persistentvolumes` |`pv` `pods` |`po` From c5a79de79c19243e3b690f1ab7a49e486f8f5ddd Mon Sep 17 00:00:00 2001 From: Cao Shufeng Date: Mon, 20 Feb 2017 23:45:50 -0500 Subject: [PATCH 246/407] Remove details about where authenticator is implemented It not necessary to mention where the functions are implemented in document. Further more, the directories recorded here is out of date. Now they are in: staging/src/k8s.io/apiserver/plugin/pkg/authenticator/... --- docs/admin/authentication.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/admin/authentication.md b/docs/admin/authentication.md index 7c59c955ac..7a7d8f1bc3 100644 --- a/docs/admin/authentication.md +++ b/docs/admin/authentication.md @@ -85,9 +85,9 @@ See [APPENDIX](#appendix) for how to generate a client cert. The API server reads bearer tokens from a file when given the `--token-auth-file=SOMEFILE` option on the command line. Currently, tokens last indefinitely, and the token list cannot be changed without restarting API server. -The token file format is implemented in `plugin/pkg/auth/authenticator/token/tokenfile/...` -and is a csv file with a minimum of 3 columns: token, user name, user uid, followed by -optional group names. Note, if you have more than one group the column must be double quoted e.g. +The token file is a csv file with a minimum of 3 columns: token, user name, user uid, +followed by optional group names. Note, if you have more than one group the column must be +double quoted e.g. ```conf token,user,uid,"group1,group2,group3" @@ -115,9 +115,9 @@ and the password cannot be changed without restarting API server. Note that basi authentication is currently supported for convenience while we finish making the more secure modes described above easier to use. -The basic auth file format is implemented in `plugin/pkg/auth/authenticator/password/passwordfile/...` -and is a csv file with a minimum of 3 columns: password, user name, user id, followed by -optional group names. Note, if you have more than one group the column must be double quoted e.g. +The basic auth file is a csv file with a minimum of 3 columns: password, +user name, user id, followed by optional group names. Note, if you have more than +one group the column must be double quoted e.g. ```conf password,user,uid,"group1,group2,group3" From 82613412954fb14eb5384a912b003d8b0703c7eb Mon Sep 17 00:00:00 2001 From: Alexander Kanevskiy Date: Tue, 21 Feb 2017 20:41:52 +0200 Subject: [PATCH 247/407] Updated description for kubeadm init --kubernetes-version --- docs/admin/kubeadm.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index a43beec497..32bd70812f 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -143,10 +143,11 @@ By default, `kubeadm init` automatically generates the token used to initialise each new node. If you would like to manually specify this token, you can use the `--token` flag. The token must be of the format `<6 character string>.<16 character string>`. -- `--use-kubernetes-version` (default 'v1.5.1') the kubernetes version to initialise +- `--kubernetes-version` (default 'latest') the kubernetes version to initialise `kubeadm` was originally built for Kubernetes version **v1.4.0**, older versions are not -supported. With this flag you can try any future version, e.g. **v1.6.0-beta.1** +supported. The current version of `kubeadm` requires at least **v1.6.0-alpha.3** due to RBAC being enabled by default. +With this flag you can try any future version, e.g. **v1.6.0-beta.1** whenever it comes out (check [releases page](https://github.com/kubernetes/kubernetes/releases) for a full list of available versions). From a7192a9e9a7b5b3bebd746256bd1b8870fa627b8 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 21 Feb 2017 12:35:15 -0800 Subject: [PATCH 248/407] Revert "Updated description for kubeadm init --kubernetes-version" --- docs/admin/kubeadm.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 32bd70812f..a43beec497 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -143,11 +143,10 @@ By default, `kubeadm init` automatically generates the token used to initialise each new node. If you would like to manually specify this token, you can use the `--token` flag. The token must be of the format `<6 character string>.<16 character string>`. -- `--kubernetes-version` (default 'latest') the kubernetes version to initialise +- `--use-kubernetes-version` (default 'v1.5.1') the kubernetes version to initialise `kubeadm` was originally built for Kubernetes version **v1.4.0**, older versions are not -supported. The current version of `kubeadm` requires at least **v1.6.0-alpha.3** due to RBAC being enabled by default. -With this flag you can try any future version, e.g. **v1.6.0-beta.1** +supported. With this flag you can try any future version, e.g. **v1.6.0-beta.1** whenever it comes out (check [releases page](https://github.com/kubernetes/kubernetes/releases) for a full list of available versions). From a512e93e9d1c7f71d47acf3f7098c0ee752b2271 Mon Sep 17 00:00:00 2001 From: divyenpatel Date: Fri, 17 Feb 2017 14:58:30 -0800 Subject: [PATCH 249/407] updated vsphere getting started guide addressed review comments addressed chenopis's review comments --- docs/getting-started-guides/vsphere.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/vsphere.md b/docs/getting-started-guides/vsphere.md index 61ea733bf3..a22d38609f 100644 --- a/docs/getting-started-guides/vsphere.md +++ b/docs/getting-started-guides/vsphere.md @@ -79,8 +79,7 @@ Sample Config: #### Known issues -* [Volumes are not removed from a VM configuration if the VM is down](https://github.com/kubernetes/kubernetes/issues/33061). The workaround is to manually remove the disk from VM settings before powering it up. -* [FS groups are not supported in 1.4.7](https://github.com/kubernetes/kubernetes/issues/34039) - This issue is fixed in 1.4.8 +* [Unable to execute command on pod container using kubectl exec](https://github.com/kubernetes/kubernetes-anywhere/issues/337) ### Kube-up (Deprecated) @@ -216,7 +215,7 @@ going on (find yourself authorized with your SSH key, or use the password IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level -------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ---------------------------- -Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu))([@luomiao](https://github.com/luomiao)) +Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu)), ([@luomiao](https://github.com/luomiao)), ([@divyenpatel](https://github.com/divyenpatel)) For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart. From b61c95f0723a34d3803bf0c31d6d07a336b5b848 Mon Sep 17 00:00:00 2001 From: "Elijah C. Voigt" Date: Fri, 17 Feb 2017 17:12:13 -0800 Subject: [PATCH 250/407] concepts/abstractions: init containers example. --- docs/concepts/abstractions/init-containers.md | 94 ++++++++++++++++++- 1 file changed, 93 insertions(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/init-containers.md b/docs/concepts/abstractions/init-containers.md index 1bea00b3a2..89738eef56 100644 --- a/docs/concepts/abstractions/init-containers.md +++ b/docs/concepts/abstractions/init-containers.md @@ -95,6 +95,98 @@ Here are some ideas for how to use Init Containers: More detailed usage examples can be found in the [StatefulSets documentation](/docs/concepts/abstractions/controllers/statefulsets/) and the [Production Pods guide](/docs/user-guide/production-pods.md#handling-initialization). +### Init Containers in use + +The following yaml file outlines a simple Pod which has two Init Containers. +The first waits for `myservice` and the second waits for `mydb`. Once both +containers complete the Pod will begin. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: myapp-pod + labels: + app: myapp + annotations: + pod.beta.kubernetes.io/init-containers: '[ + { + "name": "init-myservice", + "image": "busybox", + "command": ["sh", "-c", "until nslookup myservice; do echo waiting for myservice; sleep 2; done;"] + }, + { + "name": "init-mydb", + "image": "busybox", + "command": ["sh", "-c", "until nslookup mydb; do echo waiting for mydb; sleep 2; done;"] + } + ]' +spec: + containers: + - name: myapp-container + image: busybox + command: ['sh', '-c', 'echo The app is running! && sleep 3600'] +``` + +This Pod can be started and debugged with the following commands: + +``` +$ kubectl create -f myapp.yaml +pod "myapp-pod" created +$ kubectl get -f myapp.yaml +NAME READY STATUS RESTARTS AGE +myapp-pod 0/1 Init:0/2 0 6m +$ kubectl describe -f myapp.yaml +i11:32 $ kubectl describe -f examples/init-container.yaml +Name: myapp-pod +Namespace: default +[...] +Labels: app=myapp +Status: Pending +[...] +Init Containers: + init-myservice: +[...] + State: Running +[...] + init-mydb: +[...] + State: Running +[...] +Containers: + myapp-container: +[...] + State: Waiting + Reason: PodInitializing + Ready: False +[...] +Events: + FirstSeen LastSeen Count From SubObjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 16s 16s 1 {default-scheduler } Normal Scheduled Successfully assigned myapp-pod to 172.17.4.201 + 16s 16s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulling pulling image "busybox" + 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulled Successfully pulled image "busybox" + 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container with docker id 5ced34a04634; Security:[seccomp=unconfined] + 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container with docker id 5ced34a04634 +$ kubectl logs myapp-pod -c init-myservice # Inspect the first init container +$ kubectl logs myapp-pod -c init-mydd # Inspect the second init container +``` + +Once we start the `mydb` and `myservice` Services we can see the Init Containers +complete and the `myapp-pod` is created: + +``` +$ kubectl create -f services.yaml +service "myservice" created +service "mydb" created +$ kubectl get -f myapp.yaml +NAME READY STATUS RESTARTS AGE +myapp-pod 1/1 Running 0 9m +``` + +This example is very simple but should provide some inspiration for you to +create your own Init Containers. + ## Detailed behavior During the startup of a Pod, the Init Containers are started in order, after the @@ -181,4 +273,4 @@ Kubelet and Apiserver versions; see the [release notes](https://github.com/kuber {% endcapture %} -{% include templates/concept.md %} \ No newline at end of file +{% include templates/concept.md %} From c295346b055f897cdaa797a6e0f09afa6b66b10f Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Tue, 21 Feb 2017 09:15:27 +0800 Subject: [PATCH 251/407] Update source-ip.md --- docs/tutorials/services/source-ip.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index e1eac87c4e..6366690e88 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -132,7 +132,7 @@ client_address=10.240.0.5 client_address=10.240.0.3 ``` -Note that these are not your IPs, they're cluster internal IPs. This is what happens: +Note that these are not right client IPs, they're cluster internal IPs. This is what happens: * Client sends packet to `node2:nodePort` * `node2` replaces the source IP address (SNAT) in the packet with its own IP address From 0a76c67c5b07fa9c4b070fd625c66fd97f165038 Mon Sep 17 00:00:00 2001 From: ztewyk <10110918@zte.com.cn> Date: Wed, 22 Feb 2017 08:40:09 +0800 Subject: [PATCH 252/407] Update source-ip.md --- docs/tutorials/services/source-ip.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index 6366690e88..4f5f7d884a 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -132,7 +132,7 @@ client_address=10.240.0.5 client_address=10.240.0.3 ``` -Note that these are not right client IPs, they're cluster internal IPs. This is what happens: +Note that these are not the correct client IPs, they're cluster internal IPs. This is what happens: * Client sends packet to `node2:nodePort` * `node2` replaces the source IP address (SNAT) in the packet with its own IP address From 69cfd63deffb6cfb4a893765f37ea27460043b90 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 21 Feb 2017 19:12:47 -0800 Subject: [PATCH 253/407] Move Guide topic: Garbage Collection. (#2488) --- _data/concepts.yml | 1 + .../controllers/garbage-collection.md | 110 ++++++++++++++++++ .../abstractions/controllers/my-repset.yaml | 17 +++ docs/user-guide/garbage-collection.md | 33 +----- 4 files changed, 130 insertions(+), 31 deletions(-) create mode 100644 docs/concepts/abstractions/controllers/garbage-collection.md create mode 100644 docs/concepts/abstractions/controllers/my-repset.yaml diff --git a/_data/concepts.yml b/_data/concepts.yml index 432f4c86e1..56a556a801 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -20,6 +20,7 @@ toc: - title: Controllers section: - docs/concepts/abstractions/controllers/statefulsets.md + - docs/concepts/abstractions/controllers/garbage-collection.md - title: Object Metadata section: diff --git a/docs/concepts/abstractions/controllers/garbage-collection.md b/docs/concepts/abstractions/controllers/garbage-collection.md new file mode 100644 index 0000000000..889049c1ba --- /dev/null +++ b/docs/concepts/abstractions/controllers/garbage-collection.md @@ -0,0 +1,110 @@ +--- +title: Garbage Collection +--- + +{% capture overview %} + +The role of the Kubernetes garbage collector is to delete certain objects +that once had an owner, but no longer have an owner. + +**Note**: Garbage collection is a beta feature and is enabled by default in +Kubernetes version 1.4 and later. + +{% endcapture %} + + +{% capture body %} + +## Owners and dependents + +Some Kubernetes objects are owners of other objects. For example, a ReplicaSet +is the owner of a set of Pods. The owned objects are called *dependents* of the +owner object. Every dependent object has a `metadata.ownerReferences` field that +points to the owning object. + +Sometimes, Kubernetes sets the value of `ownerReference` automatically. For +example, when you create a ReplicaSet, Kubernetes automatically sets the +`ownerReference` field of each Pod in the ReplicaSet. You can also specify +relationships between owners and dependents by manually setting the +`ownerReference` field. + +Here's a configuration file for a ReplicaSet that has three Pods: + +{% include code.html language="yaml" file="my-repset.yaml" ghlink="/docs/concepts/abstractions/controllers/my-repset.yaml" %} + +If you create the ReplicaSet and then view the Pod metadata, you can see +OwnerReferences field: + +```shell +kubectl create -f http://k8s.io/docs/concepts/abstractions/controllers/my-repset.yaml +kubectl get pods --output=yaml +``` + +The output shows that the Pod owner is a ReplicaSet named my-repset: + +```shell +apiVersion: v1 +kind: Pod +metadata: + ... + ownerReferences: + - apiVersion: extensions/v1beta1 + controller: true + kind: ReplicaSet + name: my-repset + uid: d9607e19-f88f-11e6-a518-42010a800195 + ... +``` + +## Controlling whether the garbage collector deletes dependents + +When you delete object, you can specify whether the object's dependents +are deleted automatically. Deleting dependents automatically is called +*cascading deletion*. If you delete an object without deleting its +dependents automatically, the dependents are said to be *orphaned*. + +To delete dependent objects automatically, set the `orphanDependents` query +parameter to false in your request to delete the owner object. + +To orphan the dependents of an owner object, set the `orphanDependents` query +parameter to true in your request to delete the owner object. + +The default value for `orphanDependents` is true. So unless you specify +otherwise, dependent objects are orphaned. + +Here's an example that deletes dependents automatically: + +```shell +kubectl proxy --port=8080 +curl -X DELETE localhost:8080/apis/extensions/v1beta1/namespaces/default/replicasets/my-repset?orphanDependents=false +``` + +To delete dependents automatically using kubectl, set `--cascade` to true. +To orphan dependents, set `--cascade` to false. The default value for +`--cascade` is true. + +Here's an example that orphans the dependents of a ReplicaSet: + +```shell +kubectl delete replicaset my-repset --cascade=false +``` + +## Ongoing development + +In Kubernetes version 1.5, synchronous garbage collection is under active +development. See the tracking +[issue](https://github.com/kubernetes/kubernetes/issues/29891) for more details. + +{% endcapture %} + + +{% capture whatsnext %} + +[Design Doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/garbage-collection.md) + +[Known issues](https://github.com/kubernetes/kubernetes/issues/26120) + +{% endcapture %} + + +{% include templates/concept.md %} diff --git a/docs/concepts/abstractions/controllers/my-repset.yaml b/docs/concepts/abstractions/controllers/my-repset.yaml new file mode 100644 index 0000000000..54befd8f9d --- /dev/null +++ b/docs/concepts/abstractions/controllers/my-repset.yaml @@ -0,0 +1,17 @@ +apiVersion: extensions/v1beta1 +kind: ReplicaSet +metadata: + name: my-repset +spec: + replicas: 3 + selector: + matchLabels: + pod-is-for: garbage-collection-example + template: + metadata: + labels: + pod-is-for: garbage-collection-example + spec: + containers: + - name: nginx + image: nginx diff --git a/docs/user-guide/garbage-collection.md b/docs/user-guide/garbage-collection.md index af90b4dd1a..16f9380866 100644 --- a/docs/user-guide/garbage-collection.md +++ b/docs/user-guide/garbage-collection.md @@ -4,35 +4,6 @@ assignees: title: Garbage Collection (Beta) --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Garbage Collection - -Note: the Garbage Collection is a beta feature and is enabled by default in Kubernetes version 1.4. - -### What does Garbage Collector do - -When you delete, for example, a ReplicaSet, it is often desirable for the server to automatically garbage collect all the Pods that the ReplicaSet creates. The Garbage Collector (GC) implements this. In general, when you delete an owner object, GC deletes that owner's dependent objects. - -### How to establish an owner-dependent relationship between objects - -Kubernetes 1.3 added a metadata.ownerReferences field to every Kubernetes API object. If an API object is a dependent of another object, ownerReference should point to the owning API object. - -When you create a ReplicationController or a ReplicaSet in Kubernetes 1.4, the Kubernetes control plane automatically sets the ownerReference field in each created pod to point to the owning ReplicationController or ReplicaSet. - -You can set up owner-dependent relationships among other objects by manually setting the ownerReference field on dependent objects. - -### Controlling whether Garbage Collector deletes dependents - -When deleting an object, you can request the GC to ***asynchronously*** delete its dependents by ***explicitly*** specifying `deleteOptions.orphanDependents=false` in the deletion request that you send to the API server. A 200 OK response from the API server indicates the owner is deleted. - -In Kubernetes version 1.5, synchronous garbage collection is under active development. See the tracking [issue](https://github.com/kubernetes/kubernetes/issues/29891) for more details. - -If you specify `deleteOptions.orphanDependents=true`, or leave it blank, then the GC will first reset the `ownerReferences` in the dependents, then delete the owner. Note that the deletion of the owner object is asynchronous, that is, a 200 OK response will be sent by the API server before the owner object gets deleted. - -### Other references - -[Design Doc](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/garbage-collection.md) - -[Known issues](https://github.com/kubernetes/kubernetes/issues/26120) +[Garbage Collection](/docs/concepts/abstractions/controllers/garbage-collection/) From 8f6de25c987a4a3e09e9e0c00d723f5e47c7dff5 Mon Sep 17 00:00:00 2001 From: Jared Date: Wed, 22 Feb 2017 10:54:28 -0800 Subject: [PATCH 254/407] Update review-issues.md --- docs/contribute/review-issues.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/contribute/review-issues.md b/docs/contribute/review-issues.md index f5430d0b11..1353620e81 100644 --- a/docs/contribute/review-issues.md +++ b/docs/contribute/review-issues.md @@ -12,6 +12,8 @@ This page explains how documentation issues are reviewed and prioritized for the ## Categorizing issues Issues should be sorted into different buckets of work using the following labels and definitions. If an issue doesn't have enough information to identify a problem that can be researched, reviewed, or worked on (i.e. the issue doesn't fit into any of the categories below) you should close the issue with a comment explaining why it is being closed. +### Needs Clarification +* Issues that need more information from the original submitter to make them actionable. Issues with this label that aren't followed up within a week may be closed. ### Actionable * Issues that can be worked on with current information (or may need a comment to explain what needs to be done to make it more clear) @@ -26,8 +28,9 @@ Issues should be sorted into different buckets of work using the following label * Issues that are suggestions for better processes or site improvements that require community agreement to be implemented * Topics can be brought to SIG meetings as agenda items -#### Needs UX Review -* Issues that are suggestions for improving the user interface of the site or fixing a broken UX. +### Needs UX Review +* Issues that are suggestions for improving the user interface of the site. +* Fixing broken site elements. ## Prioritizing Issues From f5e5175247234df0a3414d77d09af80ed84b4043 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Wed, 22 Feb 2017 10:11:42 +0800 Subject: [PATCH 255/407] Fixed flag for eviction-soft. --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 8164fbb7d9..ad55b773a5 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -64,7 +64,7 @@ Including: | Existing Flag | New Flag | Rationale | | ------------- | -------- | --------- | -| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` | existing eviction signals can trigger image garbage collection | +| `--image-gc-high-threshold` | `--eviction-hard` or `--eviction-soft` | existing eviction signals can trigger image garbage collection | | `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | eviction reclaims achieve the same behavior | | `--maximum-dead-containers` | | deprecated once old logs are stored outside of container's context | | `--maximum-dead-containers-per-container` | | deprecated once old logs are stored outside of container's context | From fd3b6e66d888e93be53b330077bc7d49c8ae5c59 Mon Sep 17 00:00:00 2001 From: Xiuyu Li Date: Tue, 21 Feb 2017 16:18:52 +0800 Subject: [PATCH 256/407] show kubectl_rollingupdate svg --- docs/user-guide/kubectl/kubectl_rolling-update.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl/kubectl_rolling-update.md b/docs/user-guide/kubectl/kubectl_rolling-update.md index 382e630c78..f20483e125 100644 --- a/docs/user-guide/kubectl/kubectl_rolling-update.md +++ b/docs/user-guide/kubectl/kubectl_rolling-update.md @@ -13,7 +13,7 @@ Perform a rolling update of the given ReplicationController. Replaces the specified replication controller with a new replication controller by updating one pod at a time to use the new PodTemplate. The new-controller.json must specify the same namespace as the existing replication controller and overwrite at least one (common) label in its replicaSelector. -! http://kubernetes.io/images/docs/kubectl_rollingupdate.svg +![kubectl_rollingupdate](http://kubernetes.io/images/docs/kubectl_rollingupdate.svg) ``` kubectl rolling-update OLD_CONTROLLER_NAME ([NEW_CONTROLLER_NAME] --image=NEW_CONTAINER_IMAGE | -f NEW_CONTROLLER_SPEC) From da2ebff41729fb9e979c7f48071b821df58bcad8 Mon Sep 17 00:00:00 2001 From: Andres Villarroel Date: Tue, 7 Feb 2017 09:30:13 -0800 Subject: [PATCH 257/407] Update hello-minikube.md - Proposing a different way to run curl without proxy, without touching current environment - There's a formatting problem and the whole thing is being displayed in a single line ![screenshot](http://imgur.com/download/nfk02hd) --- docs/tutorials/stateless-application/hello-minikube.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/tutorials/stateless-application/hello-minikube.md b/docs/tutorials/stateless-application/hello-minikube.md index 6f2a47b897..70fed7db16 100644 --- a/docs/tutorials/stateless-application/hello-minikube.md +++ b/docs/tutorials/stateless-application/hello-minikube.md @@ -74,10 +74,9 @@ chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl ``` Determine whether you can access sites like [https://cloud.google.com/container-registry/](https://cloud.google.com/container-registry/) directly without a proxy, by opening a new terminal and using + ```shell -export http_proxy="" -export https_proxy="" -curl https://cloud.google.com/container-registry/ +curl --proxy "" https://cloud.google.com/container-registry/ ``` If NO proxy is required, start the Minikube cluster: From 47ae58f3c476bb92cf3095bf7011e54cbf7b8fa8 Mon Sep 17 00:00:00 2001 From: tailnode Date: Thu, 23 Feb 2017 16:08:38 +0800 Subject: [PATCH 258/407] fix some wrong links --- _includes/v1.5/extensions-v1beta1-definitions.html | 2 +- _includes/v1.5/v1-definitions.html | 2 +- docs/api-reference/apps/v1beta1/definitions.html | 2 +- docs/api-reference/batch/v1/definitions.html | 2 +- docs/api-reference/extensions/v1beta1/definitions.html | 2 +- docs/api-reference/v1.5/index.html | 2 +- .../api-reference/extensions/v1beta1/definitions.html | 2 +- docs/resources-reference/v1.5/index.html | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/_includes/v1.5/extensions-v1beta1-definitions.html b/_includes/v1.5/extensions-v1beta1-definitions.html index bd86db7daa..f210a6350b 100755 --- a/_includes/v1.5/extensions-v1beta1-definitions.html +++ b/_includes/v1.5/extensions-v1beta1-definitions.html @@ -3453,7 +3453,7 @@ Populated by the system when a graceful deletion is requested. Read-only. More i
- + diff --git a/_includes/v1.5/v1-definitions.html b/_includes/v1.5/v1-definitions.html index 16858392bc..b42f1ad675 100755 --- a/_includes/v1.5/v1-definitions.html +++ b/_includes/v1.5/v1-definitions.html @@ -4172,7 +4172,7 @@ The resulting set of endpoints can be viewed as:
- + diff --git a/docs/api-reference/apps/v1beta1/definitions.html b/docs/api-reference/apps/v1beta1/definitions.html index 77ef25e10c..c62a8bf681 100755 --- a/docs/api-reference/apps/v1beta1/definitions.html +++ b/docs/api-reference/apps/v1beta1/definitions.html @@ -3620,7 +3620,7 @@ The StatefulSet guarantees that a given network identity will always map to the - + diff --git a/docs/api-reference/batch/v1/definitions.html b/docs/api-reference/batch/v1/definitions.html index 9989f4c4ca..0a17bce0a7 100755 --- a/docs/api-reference/batch/v1/definitions.html +++ b/docs/api-reference/batch/v1/definitions.html @@ -3609,7 +3609,7 @@ Populated by the system when a graceful deletion is requested. Read-only. More i - + diff --git a/docs/api-reference/extensions/v1beta1/definitions.html b/docs/api-reference/extensions/v1beta1/definitions.html index 3f1129b1b6..47dc1d4f76 100755 --- a/docs/api-reference/extensions/v1beta1/definitions.html +++ b/docs/api-reference/extensions/v1beta1/definitions.html @@ -3457,7 +3457,7 @@ Populated by the system when a graceful deletion is requested. Read-only. More i - + diff --git a/docs/api-reference/v1.5/index.html b/docs/api-reference/v1.5/index.html index 84caadbe95..ae50987334 100644 --- a/docs/api-reference/v1.5/index.html +++ b/docs/api-reference/v1.5/index.html @@ -8010,7 +8010,7 @@ Appears In PodPodTemplateSp - + diff --git a/docs/federation/api-reference/extensions/v1beta1/definitions.html b/docs/federation/api-reference/extensions/v1beta1/definitions.html index 897ab5aca8..28b2883fc2 100755 --- a/docs/federation/api-reference/extensions/v1beta1/definitions.html +++ b/docs/federation/api-reference/extensions/v1beta1/definitions.html @@ -3073,7 +3073,7 @@ Populated by the system when a graceful deletion is requested. Read-only. More i - + diff --git a/docs/resources-reference/v1.5/index.html b/docs/resources-reference/v1.5/index.html index c3902110de..8b5c05ec19 100644 --- a/docs/resources-reference/v1.5/index.html +++ b/docs/resources-reference/v1.5/index.html @@ -1062,7 +1062,7 @@ Appears In PodPodTemplateSp - + From 741292ddb65669805c684f436d11eac53f30e699 Mon Sep 17 00:00:00 2001 From: roopakparikh Date: Wed, 22 Feb 2017 11:06:01 -0800 Subject: [PATCH 259/407] Added KUBE2GO.io Adding a new Free service KUBE2GO.io that provides quick creation of K8s clusters on AWS and other public clouds. --- docs/getting-started-guides/index.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/getting-started-guides/index.md b/docs/getting-started-guides/index.md index 9ee9e01301..c58c6cf297 100644 --- a/docs/getting-started-guides/index.md +++ b/docs/getting-started-guides/index.md @@ -46,6 +46,7 @@ clusters. [AppsCode.com](https://appscode.com/products/cloud-deployment/) provides managed Kubernetes clusters for various public clouds (including AWS and Google Cloud Platform). [KCluster.io](https://kcluster.io) provides highly available and scalable managed Kubernetes clusters for AWS. +[KUBE2GO.io] (https://kube2go.io) get started with highly available Kubernetes clusters on multiple public clouds along with useful tools for development, debugging, monitoring. [Platform9](https://platform9.com/products/kubernetes/) offers managed Kubernetes on-premises or any public cloud, and provides 24/7 health monitoring and alerting. @@ -62,6 +63,7 @@ few commands, and have active community support. - [CenturyLink Cloud](/docs/getting-started-guides/clc) - [IBM SoftLayer](https://github.com/patrocinio/kubernetes-softlayer) - [Stackpoint.io](/docs/getting-started-guides/stackpoint/) +- [KUBE2GO.io](https://kube2go.io/) ### Custom Solutions @@ -131,6 +133,7 @@ GKE | | | GCE | [docs](https://clou Stackpoint.io | | multi-support | multi-support | [docs](http://www.stackpointcloud.com) | Commercial AppsCode.com | Saltstack | Debian | multi-support | [docs](https://appscode.com/products/cloud-deployment/) | Commercial KCluster.io | | multi-support | multi-support | [docs](https://kcluster.io) | Commercial +KUBE2GO.io | | multi-support | multi-support | [docs](https://kube2go.io) | Commercial Platform9 | | multi-support | multi-support | [docs](https://platform9.com/products/kubernetes/) | Commercial GCE | Saltstack | Debian | GCE | [docs](/docs/getting-started-guides/gce) | Project Azure Container Service | | Ubuntu | Azure | [docs](https://azure.microsoft.com/en-us/services/container-service/) | Commercial From fb7a756a77840e26012a0463c419168ad71b8a56 Mon Sep 17 00:00:00 2001 From: roopakparikh Date: Thu, 23 Feb 2017 14:16:47 -0800 Subject: [PATCH 260/407] Removed Extra space in the KUBE2GO commit Last pull request had an extra space and was missing a newline between the last provider and KUBE2GO --- docs/getting-started-guides/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/getting-started-guides/index.md b/docs/getting-started-guides/index.md index c58c6cf297..4ed17d4f47 100644 --- a/docs/getting-started-guides/index.md +++ b/docs/getting-started-guides/index.md @@ -46,7 +46,8 @@ clusters. [AppsCode.com](https://appscode.com/products/cloud-deployment/) provides managed Kubernetes clusters for various public clouds (including AWS and Google Cloud Platform). [KCluster.io](https://kcluster.io) provides highly available and scalable managed Kubernetes clusters for AWS. -[KUBE2GO.io] (https://kube2go.io) get started with highly available Kubernetes clusters on multiple public clouds along with useful tools for development, debugging, monitoring. + +[KUBE2GO.io](https://kube2go.io) get started with highly available Kubernetes clusters on multiple public clouds along with useful tools for development, debugging, monitoring. [Platform9](https://platform9.com/products/kubernetes/) offers managed Kubernetes on-premises or any public cloud, and provides 24/7 health monitoring and alerting. From 58867752f51235621c55a6bd0b43167767f5df9d Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Sat, 18 Feb 2017 10:46:57 +0800 Subject: [PATCH 261/407] Delete the parameter "--google-json-key string" Delete the parameter "--google-json-key string" --- docs/admin/kube-proxy.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/admin/kube-proxy.md b/docs/admin/kube-proxy.md index ea13d528e3..bf72aee787 100644 --- a/docs/admin/kube-proxy.md +++ b/docs/admin/kube-proxy.md @@ -41,7 +41,6 @@ DynamicKubeletConfig=true|false (ALPHA - default=false) DynamicVolumeProvisioning=true|false (ALPHA - default=true) ExperimentalHostUserNamespaceDefaulting=true|false (ALPHA - default=false) StreamingProxyRedirects=true|false (ALPHA - default=false) - --google-json-key string The Google Cloud Platform Service Account JSON Key to use for authentication. --healthz-bind-address ip The IP address for the health check server to serve on, defaulting to 127.0.0.1 (set to 0.0.0.0 for all interfaces) (default 127.0.0.1) --healthz-port int32 The port to bind the health check server. Use 0 to disable. (default 10249) --hostname-override string If non-empty, will use this string as identification instead of the actual hostname. From fc9b83e95d0a02f8239452f5559a9638ad79d286 Mon Sep 17 00:00:00 2001 From: Jeff Mendoza Date: Fri, 24 Feb 2017 14:36:12 -0800 Subject: [PATCH 262/407] Fix apimachinery vendored dependencies in examples_test. --- .travis.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.travis.yml b/.travis.yml index b0a565803c..691d897659 100644 --- a/.travis.yml +++ b/.travis.yml @@ -26,6 +26,8 @@ install: - cp -r $GOPATH/src/k8s.io/kubernetes/vendor/* $GOPATH/src/ - rm -rf $GOPATH/src/k8s.io/kubernetes/vendor/* - cp -r $GOPATH/src/k8s.io/kubernetes/staging/src/* $GOPATH/src/ +- cp -r $GOPATH/src/k8s.io/apimachinery/vendor/* $GOPATH/src/ +- rm -rf $GOPATH/src/k8s.io/apimachinery/vendor/* # (2) Fetch md-check along with all its dependencies. - git clone --depth=50 --branch=master https://github.com/kubernetes/md-check $HOME/gopath/src/k8s.io/md-check From 180a9b6abc026b3072b0e982764f8c5c77a7fcaa Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Fri, 24 Feb 2017 09:40:33 +0800 Subject: [PATCH 263/407] package apt-transport-https should be installed Ubuntu default install not include apt-transport-https. so if you want to download package from https repo, you need install apt-transport-https package first --- docs/getting-started-guides/kubeadm.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index 5fc815ea9c..b5a9845b44 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -69,6 +69,7 @@ For each host in turn: * SSH into the machine and become `root` if you are not already (for example, run `sudo su -`). * If the machine is running Ubuntu or HypriotOS, run: + apt-get update && apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat < /etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main From eeb1367c64bd245ba81fe4138927b9715cccd6dd Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Sat, 18 Feb 2017 12:53:35 +0800 Subject: [PATCH 264/407] doc-walkthrough-content modify --- docs/admin/namespaces/walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/namespaces/walkthrough.md b/docs/admin/namespaces/walkthrough.md index 7576d2c2e0..e3aa0c7bfc 100644 --- a/docs/admin/namespaces/walkthrough.md +++ b/docs/admin/namespaces/walkthrough.md @@ -145,7 +145,7 @@ dev At this point, all requests we make to the Kubernetes cluster from the command line are scoped to the development namespace. -Let's create some content. +Let's create some contents. ```shell $ kubectl run snowflake --image=kubernetes/serve_hostname --replicas=2 From 474ae68f2508b0db2aea8ecdce89428ad6fdbf40 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Fri, 24 Feb 2017 14:31:58 +0800 Subject: [PATCH 265/407] limitrange-update I think it's redundant that the second to ,in order to do sth and do sth!thanks! --- docs/admin/limitrange/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/limitrange/index.md b/docs/admin/limitrange/index.md index 82bed781f1..1791f14f4d 100644 --- a/docs/admin/limitrange/index.md +++ b/docs/admin/limitrange/index.md @@ -24,7 +24,7 @@ each namespace. 3. Users may create a pod which consumes resources just below the capacity of a machine. The left over space may be too small to be useful, but big enough for the waste to be costly over the entire cluster. As a result, the cluster operator may want to set limits that a pod must consume at least 20% of the memory and CPU of their -average node size in order to provide for more uniform scheduling and to limit waste. +average node size in order to provide for more uniform scheduling and limit waste. This example demonstrates how limits can be applied to a Kubernetes [namespace](/docs/admin/namespaces/walkthrough/) to control min/max resource limits per pod. In addition, this example demonstrates how you can From 4508f114f954dab50f76894ffc6bcb5ebb501b7b Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Fri, 24 Feb 2017 14:38:27 +0800 Subject: [PATCH 266/407] fix command kubectl get svc,pod output fix command kubectl get svc,pod output --- .../network-policy/walkthrough.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/getting-started-guides/network-policy/walkthrough.md b/docs/getting-started-guides/network-policy/walkthrough.md index 21a74278c5..041c4e99f8 100644 --- a/docs/getting-started-guides/network-policy/walkthrough.md +++ b/docs/getting-started-guides/network-policy/walkthrough.md @@ -31,12 +31,13 @@ This will run two nginx Pods in the default Namespace, and expose them through a ```console $ kubectl get svc,pod -NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE -kubernetes 10.100.0.1 443/TCP 46m -nginx 10.100.0.16 80/TCP 33s -NAME READY STATUS RESTARTS AGE -nginx-701339712-e0qfq 1/1 Running 0 35s -nginx-701339712-o00ef 1/1 Running 0 35s +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +svc/kubernetes 10.100.0.1 443/TCP 46m +svc/nginx 10.100.0.16 80/TCP 33s + +NAME READY STATUS RESTARTS AGE +po/nginx-701339712-e0qfq 1/1 Running 0 35s +po/nginx-701339712-o00ef 1/1 Running 0 35s ``` We should be able to access our new nginx Service from other Pods. Let's try to access it from another Pod From 30023fb5b5d4a0f48cb3c86049a883833fff3ba6 Mon Sep 17 00:00:00 2001 From: Patrick Decat Date: Fri, 24 Feb 2017 16:14:16 +0100 Subject: [PATCH 267/407] Fix typo: federation-controller-mananger => federation-controller-manager --- _data/guides.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data/guides.yml b/_data/guides.yml index 98ef2db801..30ee93078c 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -228,5 +228,5 @@ toc: - title: Federation Components section: - docs/admin/federation-apiserver.md - - title : federation-controller-mananger + - title : federation-controller-manager path: /docs/admin/federation-controller-manager From ae56b5bf072706f7df3e6e3b333b8d4078c0725b Mon Sep 17 00:00:00 2001 From: Patrick Decat Date: Fri, 24 Feb 2017 16:15:24 +0100 Subject: [PATCH 268/407] Fix typo: federation-controller-mananger => federation-controller-manager --- docs/admin/federation-controller-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/federation-controller-manager.md b/docs/admin/federation-controller-manager.md index d3dca5bf06..caa001e4c8 100644 --- a/docs/admin/federation-controller-manager.md +++ b/docs/admin/federation-controller-manager.md @@ -1,5 +1,5 @@ --- -title: federation-controller-mananger +title: federation-controller-manager notitle: true --- From 6d133c64d5d27b18e09516a54a46b96359d6982c Mon Sep 17 00:00:00 2001 From: Jordan Liggitt Date: Fri, 24 Feb 2017 11:18:24 -0500 Subject: [PATCH 269/407] Clarify minimum version of kubernetes supporting PSP authz --- docs/user-guide/pod-security-policy/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/pod-security-policy/index.md b/docs/user-guide/pod-security-policy/index.md index da1c6514d6..d3b40902ea 100644 --- a/docs/user-guide/pod-security-policy/index.md +++ b/docs/user-guide/pod-security-policy/index.md @@ -163,5 +163,5 @@ following ## Working With RBAC -Use PodSecurityPolicy to control access to privileged containers based on role and groups. +In Kubernetes 1.5 and newer, you can use PodSecurityPolicy to control access to privileged containers based on user role and groups. (see [more details](https://github.com/kubernetes/kubernetes/blob/master/examples/podsecuritypolicy/rbac/README.md)). From 29b3f2b99a720bf209c4795f3a3d6623da43e10a Mon Sep 17 00:00:00 2001 From: Stephen Gordon Date: Sat, 25 Feb 2017 16:12:07 -0500 Subject: [PATCH 270/407] Provide correct location for KUBE_ETCD_SERVERS Provide correct location for KUBE_ETCD_SERVERS configuration key. It was previously listed as being in /etc/kubernetes/config but is actually in /etc/kubernetes/apiserver. Related: https://github.com/kubernetes/kubernetes.github.io/issues/1600 --- docs/getting-started-guides/centos/centos_manual_config.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/centos/centos_manual_config.md b/docs/getting-started-guides/centos/centos_manual_config.md index 031a5bb3f5..b64af60475 100644 --- a/docs/getting-started-guides/centos/centos_manual_config.md +++ b/docs/getting-started-guides/centos/centos_manual_config.md @@ -61,9 +61,6 @@ echo "192.168.121.9 centos-master * Edit /etc/kubernetes/config which will be the same on all hosts to contain: ```shell -# Comma separated list of nodes in the etcd cluster -KUBE_ETCD_SERVERS="--etcd-servers=http://centos-master:2379" - # logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" @@ -111,6 +108,9 @@ KUBE_API_PORT="--port=8080" # Port kubelets listen on KUBELET_PORT="--kubelet-port=10250" +# Comma separated list of nodes in the etcd cluster +KUBE_ETCD_SERVERS="--etcd-servers=http://centos-master:2379" + # Address range to use for services KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" From 650d632519d1e03cf113825448fd4f2f09a9c0ad Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Mon, 27 Feb 2017 14:16:14 -0800 Subject: [PATCH 271/407] Move Compute Resources topic to Concepts. (#2410) --- _data/concepts.yml | 1 + .../manage-compute-resources-container.md | 430 ++++++++++++++++++ docs/user-guide/compute-resources.md | 366 +-------------- 3 files changed, 433 insertions(+), 364 deletions(-) create mode 100644 docs/concepts/configuration/manage-compute-resources-container.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 56a556a801..bd187c6cee 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -35,6 +35,7 @@ toc: - title: Configuration section: - docs/concepts/configuration/container-command-args.md + - docs/concepts/configuration/manage-compute-resources-container.md - title: Policies section: diff --git a/docs/concepts/configuration/manage-compute-resources-container.md b/docs/concepts/configuration/manage-compute-resources-container.md new file mode 100644 index 0000000000..2754260d65 --- /dev/null +++ b/docs/concepts/configuration/manage-compute-resources-container.md @@ -0,0 +1,430 @@ +--- +title: Managing Compute Resources for Containers +--- + +{% capture overview %} + +When you specify a [Pod](/docs/user-guide/pods), you can optionally specify how +much CPU and memory (RAM) each Container needs. When Containers have resource +requests specified, the scheduler can make better decisions about which nodes to +place Pods on. And when Containers have their limits specified, contention for +resources on a node can be handled in a specified manner. For more details about +the difference between requests and limits, see +[Resource QoS](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md). + +{% endcapture %} + + +{% capture body %} + +## Resource types + +*CPU* and *memory* are each a *resource type*. A resource type has a base unit. +CPU is specified in units of cores, and memory is specified in units of bytes. + +CPU and memory are collectively referred to as *compute resources*, or just +*resources*. Compute +resources are measurable quantities that can be requested, allocated, and +consumed. They are distinct from +[API resources](/docs/api/). API resources, such as Pods and +[Services](/docs/user-guide/services) are objects that can be read and modified +through the Kubernetes API server. + +## Resource requests and limits of Pod and Container + +Each Container of a Pod can specify one or more of the following: + +* `spec.containers[].resources.limits.cpu` +* `spec.containers[].resources.limits.memory` +* `spec.containers[].resources.requests.cpu` +* `spec.containers[].resources.requests.memory` + +Although requests and limits can only be specified on individual Containers, it +is convenient to talk about Pod resource requests and limits. A +*Pod resource request/limit* for a particular resource type is the sum of the +resource requests/limits of that type for each Container in the Pod. + +## Meaning of CPU + +Limits and requests for CPU resources are measured in *cpu* units. +One cpu, in Kubernetes, is equivalent to: + +- 1 AWS vCPU +- 1 GCP Core +- 1 Azure vCore +- 1 *Hyperthread* on a bare-metal Intel processor with Hyperthreading + +Fractional requests are allowed. A Container with +`spec.containers[].resources.requests.cpu` of `0.5` is guaranteed half as much +CPU as one that asks for 1 CPU. The expression `0.1` is equivalent to the +expression `100m`, which can be read as "one hundred millicpu". Some people say +"one hundred millicores", and this is understood to mean the same thing. A +request with a decimal point, like `0.1`, is converted to `100m` by the API, and +precision finer than `1m` is not allowed. For this reason, the form `100m` might +be preferred. + +CPU is always requested as an absolute quantity, never as a relative quantity; +0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine. + +## Meaning of memory + +Limits and requests for `memory` are measured in bytes. You can express memory as +a plain integer or as a fixed-point integer using one of these SI suffixes: +E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, +Mi, Ki. For example, the following represent roughly the same value: + +```shell +128974848, 129e6, 129M, 123Mi +``` + +Here's an example. +The following Pod has two Containers. Each Container has a request of 0.25 cpu +and 64MiB (226 bytes) of memory Each Container has a limit of 0.5 +cpu and 128MiB of memory. You can say the Pod has a request of 0.5 cpu and 128 +MiB of memory, and a limit of 1 core and 256MiB of memory. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: frontend +spec: + containers: + - name: db + image: mysql + resources: + requests: + memory: "64Mi" + cpu: "250m" + limits: + memory: "128Mi" + cpu: "500m" + - name: wp + image: wordpress + resources: + requests: + memory: "64Mi" + cpu: "250m" + limits: + memory: "128Mi" + cpu: "500m" +``` + +## How Pods with resource requests are scheduled + +When you create a Pod, the Kubernetes scheduler selects a node for the Pod to +run on. Each node has a maximum capacity for each of the resource types: the +amount of CPU and memory it can provide for Pods. The scheduler ensures that, +for each resource type, the sum of the resource requests of the scheduled +Containers is less than the capacity of the node. Note that although actual memory +or CPU resource usage on nodes is very low, the scheduler still refuses to place +a Pod on a node if the capacity check fails. This protects against a resource +shortage on a node when resource usage later increases, for example, during a +daily peak in request rate. + +## How Pods with resource limits are run + +When the kubelet starts a Container of a Pod, it passes the CPU and memory limits +to the container runtime. + +When using Docker: + +- The `spec.containers[].resources.requests.cpu` is converted to its core value, + which is potentially fractional, and multiplied by 1024. This number is used + as the value of the + [`--cpu-shares`](https://docs.docker.com/engine/reference/run/#/cpu-share-constraint) + flag in the `docker run` command. + +- The `spec.containers[].resources.limits.cpu` is converted to its millicore value, + multiplied by 100000, and then divided by 1000. This number is used as the value + of the [`--cpu-quota`](https://docs.docker.com/engine/reference/run/#/cpu-quota-constraint) + flag in the `docker run` command. he [`--cpu-period`] flag is set to 100000, + which represents the default 100ms period for measuring quota usage. The + kubelet enforces cpu limits if it is started with the + [`--cpu-cfs-quota`] flag set to true. As of Kubernetes version 1.2, this flag + defaults to true. + +- The `spec.containers[].resources.limits.memory` is converted to an integer, and + used as the value of the + [`--memory`](https://docs.docker.com/engine/reference/run/#/user-memory-constraints) + flag in the `docker run` command. + +If a Container exceeds its memory limit, it might be terminated. If it is +restartable, the kubelet will restart it, as with any other type of runtime +failure. + +If a Container exceeds its memory request, it is likely that its Pod will +be evicted whenever the node runs out of memory. + +A Container might or might not be allowed to exceed its CPU limit for extended +periods of time. However, it will not be killed for excessive CPU usage. + +To determine whether a Container cannot be scheduled or is being killed due to +resource limits, see the +[Troubleshooting](#troubleshooting) section. + +## Monitoring compute resource usage + +The resource usage of a Pod is reported as part of the Pod status. + +If [optional monitoring](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/README.md) +is configured for your cluster, then Pod resource usage can be retrieved from +the monitoring system. + +## Troubleshooting + +### My Pods are pending with event message failedScheduling + +If the scheduler cannot find any node where a Pod can fit, the Pod remains +unscheduled until a place can be found. An event is produced each time the +scheduler fails to find a place for the Pod, like this: + +```shell +$ kubectl describe pod frontend | grep -A 3 Events +Events: + FirstSeen LastSeen Count From Subobject PathReason Message + 36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others +``` + +In the preceding example, the Pod named "frontend" fails to be scheduled due to +insufficient CPU resource on the node. Similar error messages can also suggest +failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod +is pending with a message of this type, there are several things to try: + +- Add more nodes to the cluster. +- Terminate unneeded Pods to make room for pending Pods. +- Check that the Pod is not larger than all the nodes. For example, if all the + nodes have a capacity of `cpu: 1`, then a Pod with a limit of `cpu: 1.1` will + never be scheduled. + +You can check node capacities and amounts allocated with the +`kubectl describe nodes` command. For example: + +```shell +$ kubectl.sh describe nodes e2e-test-minion-group-4lw4 +Name: e2e-test-minion-group-4lw4 +[ ... lines removed for clarity ...] +Capacity: + alpha.kubernetes.io/nvidia-gpu: 0 + cpu: 2 + memory: 7679792Ki + pods: 110 +Allocatable: + alpha.kubernetes.io/nvidia-gpu: 0 + cpu: 1800m + memory: 7474992Ki + pods: 110 +[ ... lines removed for clarity ...] +Non-terminated Pods: (5 in total) + Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits + --------- ---- ------------ ---------- --------------- ------------- + kube-system fluentd-gcp-v1.38-28bv1 100m (5%) 0 (0%) 200Mi (2%) 200Mi (2%) + kube-system kube-dns-3297075139-61lj3 260m (13%) 0 (0%) 100Mi (1%) 170Mi (2%) + kube-system kube-proxy-e2e-test-... 100m (5%) 0 (0%) 0 (0%) 0 (0%) + kube-system monitoring-influxdb-grafana-v4-z1m12 200m (10%) 200m (10%) 600Mi (8%) 600Mi (8%) + kube-system node-problem-detector-v0.1-fj7m3 20m (1%) 200m (10%) 20Mi (0%) 100Mi (1%) +Allocated resources: + (Total limits may be over 100 percent, i.e., overcommitted.) + CPU Requests CPU Limits Memory Requests Memory Limits + ------------ ---------- --------------- ------------- + 680m (34%) 400m (20%) 920Mi (12%) 1070Mi (14%) +``` + +In the preceding output, you can see that if a Pod requests more than 1120m +CPUs or 6.23Gi of memory, it will not fit on the node. + +By looking at the `Pods` section, you can see which Pods are taking up space on +the node. + +The amount of resources available to Pods is less than the node capacity, because +system daemons use a portion of the available resources. The `allocatable` field +[NodeStatus](/docs/resources-reference/v1.5/#nodestatus-v1) +gives the amount of resources that are available to Pods. For more information, see +[Node Allocatable Resources](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md). + +The [resource quota](/docs/admin/resourcequota/) feature can be configured +to limit the total amount of resources that can be consumed. If used in conjunction +with namespaces, it can prevent one team from hogging all the resources. + +### My Container is terminated + +Your Container might get terminated because it is resource-starved. To check +whether a Container is being killed because it is hitting a resource limit, call +`kubectl describe pod` on the Pod of interest: + +```shell +[12:54:41] $ ./cluster/kubectl.sh describe pod simmemleak-hra99 +Name: simmemleak-hra99 +Namespace: default +Image(s): saadali/simmemleak +Node: kubernetes-node-tf0f/10.240.216.66 +Labels: name=simmemleak +Status: Running +Reason: +Message: +IP: 10.244.2.75 +Replication Controllers: simmemleak (1/1 replicas created) +Containers: + simmemleak: + Image: saadali/simmemleak + Limits: + cpu: 100m + memory: 50Mi + State: Running + Started: Tue, 07 Jul 2015 12:54:41 -0700 + Last Termination State: Terminated + Exit Code: 1 + Started: Fri, 07 Jul 2015 12:54:30 -0700 + Finished: Fri, 07 Jul 2015 12:54:33 -0700 + Ready: False + Restart Count: 5 +Conditions: + Type Status + Ready False +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {scheduler } scheduled Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD created Created with docker id 6a41280f516d + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD started Started with docker id 6a41280f516d + Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} spec.containers{simmemleak} created Created with docker id 87348f12526a +``` + +In the preceding example, the `Restart Count: 5` indicates that the `simmemleak` +Container in the Pod was terminated and restarted five times. + +You can call `get pod` with the `-o go-template=...` option to fetch the status +of previously terminated Containers: + +```shell{% raw %} +[13:59:01] $ ./cluster/kubectl.sh get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-60xbc +Container Name: simmemleak +LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]{% endraw %} +``` + +You can see that the Container was terminated because of `reason:OOM Killed`, +where `OOM` stands for Out Of Memory. + +## Opaque integer resources (Alpha feature) + +Kubernetes version 1.5 introduces Opaque integer resources. Opaque +integer resources allow cluster operators to advertise new node-level +resources that would be otherwise unknown to the system. + +Users can consume these resources in Pod specs just like CPU and memory. +The scheduler takes care of the resource accounting so that no more than the +available amount is simultaneously allocated to Pods. + +**Note:** Opaque integer resources are Alpha in Kubernetes version 1.5. +Only resource accounting is implemented; node-level isolation is still +under active development. + +Opaque integer resources are resources that begin with the prefix +`pod.alpha.kubernetes.io/opaque-int-resource-`. The API server +restricts quantities of these resources to whole numbers. Examples of +_valid_ quantities are `3`, `3000m` and `3Ki`. Examples of _invalid_ +quantities are `0.5` and `1500m`. + +There are two steps required to use opaque integer resources. First, the +cluster operator must advertise a per-node opaque resource on one or more +nodes. Second, users must request the opaque resource in Pods. + +To advertise a new opaque integer resource, the cluster operator should +submit a `PATCH` HTTP request to the API server to specify the available +quantity in the `status.capacity` for a node in the cluster. After this +operation, the node's `status.capacity` will include a new resource. The +`status.allocatable` field is updated automatically with the new resource +asynchronously by the kubelet. Note that because the scheduler uses the +node `status.allocatable` value when evaluating Pod fitness, there may +be a short delay between patching the node capacity with a new resource and the +first pod that requests the resource to be scheduled on that node. + +**Example:** + +Here is an HTTP request that advertises five "foo" resources on node `k8s-node-1`. + +```http +PATCH /api/v1/nodes/k8s-node-1/status HTTP/1.1 +Accept: application/json +Content-Type: application/json-patch+json +Host: k8s-master:8080 + +[ + { + "op": "add", + "path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-foo", + "value": "5" + } +] +``` + +**Note**: In the preceding request, `~1` is the encoding for the character `/` +in the patch path. The operation path value in JSON-Patch is interpreted as a +JSON-Pointer. For more details, see +[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3). + +To consume an opaque resource in a Pod, include the name of the opaque +resource as a key in the `spec.containers[].resources.requests` map. + +The Pod is scheduled only if all of the resource requests are +satisfied, including cpu, memory and any opaque resources. The Pod will +remain in the `PENDING` state as long as the resource request cannot be met by +any node. + +**Example:** + +The Pod below requests 2 cpus and 1 "foo" (an opaque resource.) + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: my-pod +spec: + containers: + - name: my-container + image: myimage + resources: + requests: + cpu: 2 + pod.alpha.kubernetes.io/opaque-int-resource-foo: 1 +``` + +## Planned Improvements + +Kubernetes version 1.5 only allows resource quantities to be specified on a +Container. It is planned to improve accounting for resources that are shared by +all Containers in a Pod, such as +[emptyDir volumes](/docs/user-guide/volumes/#emptydir). + +Kubernetes version 1.5 only supports Container requests and limits for CPU and +memory. It is planned to add new resource types, including a node disk space +resource, and a framework for adding custom +[resource types](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/resources.md). + +Kubernetes supports overcommitment of resources by supporting multiple levels of +[Quality of Service](http://issue.k8s.io/168). + +In Kubernetes version 1.5, one unit of CPU means different things on different +cloud providers, and on different machine types within the same cloud providers. +For example, on AWS, the capacity of a node is reported in +[ECUs](http://aws.amazon.com/ec2/faqs/), while in GCE it is reported in logical +cores. We plan to revise the definition of the cpu resource to allow for more +consistency across providers and platforms. + +{% endcapture %} + + +{% capture whatsnext %} + +* Get hands-on experience +[assigning CPU and RAM resources to a container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/). + +* [Container](/docs/api-reference/v1/definitions/#_v1_container) + +* [ResourceRequirements](/docs/resources-reference/v1.5/#resourcerequirements-v1) + +{% endcapture %} + +{% include templates/concept.md %} + diff --git a/docs/user-guide/compute-resources.md b/docs/user-guide/compute-resources.md index d2856f50aa..51fcaafa9d 100644 --- a/docs/user-guide/compute-resources.md +++ b/docs/user-guide/compute-resources.md @@ -5,368 +5,6 @@ assignees: title: Managing Compute Resources --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -When specifying a [pod](/docs/user-guide/pods), you can optionally specify how much CPU and memory (RAM) each -container needs. When containers have their resource requests specified, the scheduler is -able to make better decisions about which nodes to place pods on; and when containers have their -limits specified, contention for resources on a node can be handled in a specified manner. For -more details about the difference between requests and limits, please refer to -[Resource QoS](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/resource-qos.md). - -*CPU* and *memory* are each a *resource type*. A resource type has a base unit. CPU is specified -in units of cores. Memory is specified in units of bytes. - -CPU and RAM are collectively referred to as *compute resources*, or just *resources*. Compute -resources are measureable quantities which can be requested, allocated, and consumed. They are -distinct from [API resources](/docs/user-guide/working-with-resources). API resources, such as pods and -[services](/docs/user-guide/services) are objects that can be written to and retrieved from the Kubernetes API -server. - -## Resource Requests and Limits of Pod and Container - -Each container of a pod can optionally specify one or more of the following: - -* `spec.containers[].resources.limits.cpu` -* `spec.containers[].resources.limits.memory` -* `spec.containers[].resources.requests.cpu` -* `spec.containers[].resources.requests.memory`. - -Specifying resource requests and/or limits is optional. In some clusters, unset limits or requests -may be replaced with default values when a pod is created or updated. The default value depends on -how the cluster is configured. If the requests values are not specified, they are set to be equal -to the limits values by default. Please note that limits must always be greater than or equal to -requests. - -Although requests/limits can only be specified on individual containers, it is convenient to talk -about pod resource requests/limits. A *pod resource request/limit* for a particular resource -type is the sum of the resource requests/limits of that type for each container in the pod, with -unset values treated as zero (or equal to default values in some cluster configurations). - -### Meaning of CPU -Limits and requests for `cpu` are measured in cpus. -One cpu, in Kubernetes, is equivalent to: - -- 1 AWS vCPU -- 1 GCP Core -- 1 Azure vCore -- 1 *Hyperthread* on a bare-metal Intel processor with Hyperthreading - -Fractional requests are allowed. A container with `spec.containers[].resources.requests.cpu` of `0.5` will -be guaranteed half as much CPU as one that asks for `1`. The expression `0.1` is equivalent to the expression -`100m`, which can be read as "one hundred millicpu" (some may say "one hundred millicores", and this is understood -to mean the same thing when talking about Kubernetes). A request with a decimal point, like `0.1` is converted to -`100m` by the API, and precision finer than `1m` is not allowed. For this reason, the form `100m` may be preferred. - -CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same amount of cpu on a single -core, dual core, or 48 core machine. - -# Meaning of Memory - -Limits and requests for `memory` are measured in bytes. -Memory can be expressed a plain integer or as fixed-point integers with one of these SI suffixes (E, P, T, G, M, K) -or their power-of-two equivalents (Ei, Pi, Ti, Gi, Mi, Ki). For example, the following represent roughly the same value: -`128974848`, `129e6`, `129M` , `123Mi`. - -### Example -The following pod has two containers. Each has a request of 0.25 core of cpu and 64MiB -(226 bytes) of memory and a limit of 0.5 core of cpu and 128MiB of memory. The pod can -be said to have a request of 0.5 core and 128 MiB of memory and a limit of 1 core and 256MiB of -memory. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: frontend -spec: - containers: - - name: db - image: mysql - resources: - requests: - memory: "64Mi" - cpu: "250m" - limits: - memory: "128Mi" - cpu: "500m" - - name: wp - image: wordpress - resources: - requests: - memory: "64Mi" - cpu: "250m" - limits: - memory: "128Mi" - cpu: "500m" -``` - -## How Pods with Resource Requests are Scheduled - -When a pod is created, the Kubernetes scheduler selects a node for the pod to -run on. Each node has a maximum capacity for each of the resource types: the -amount of CPU and memory it can provide for pods. The scheduler ensures that, -for each resource type (CPU and memory), the sum of the resource requests of the -containers scheduled to the node is less than the capacity of the node. Note -that although actual memory or CPU resource usage on nodes is very low, the -scheduler will still refuse to place pods onto nodes if the capacity check -fails. This protects against a resource shortage on a node when resource usage -later increases, such as due to a daily peak in request rate. - -## How Pods with Resource Limits are Run - -When kubelet starts a container of a pod, it passes the CPU and memory limits to the container -runner (Docker or rkt). - -When using Docker: - -- The `spec.containers[].resources.requests.cpu` is converted to its core value (potentially fractional), - and multiplied by 1024, and used as the value of the [`--cpu-shares`](https://docs.docker.com/engine/reference/run/#/cpu-share-constraint) - flag to the `docker run` command. -- The `spec.containers[].resources.limits.cpu` is converted to its millicore value, - multiplied by 100000, and then divided by 1000, and used as the value of the [`--cpu-quota`]( - https://docs.docker.com/engine/reference/run/#/cpu-quota-constraint) flag to the `docker run` - command. The [`--cpu-period`] flag is set to 100000 which represents the default 100ms period - for measuring quota usage. The kubelet enforces cpu limits if it was started with the - [`--cpu-cfs-quota`] flag set to true. As of version 1.2, this flag will now default to true. -- The `spec.containers[].resources.limits.memory` is converted to an integer, and used as the value - of the [`--memory`](https://docs.docker.com/engine/reference/run/#/user-memory-constraints) flag - to the `docker run` command. - -**TODO: document behavior for rkt** - -If a container exceeds its memory limit, it may be terminated. If it is restartable, it will be -restarted by kubelet, as will any other type of runtime failure. - -A container may or may not be allowed to exceed its CPU limit for extended periods of time. -However, it will not be killed for excessive CPU usage. - -To determine if a container cannot be scheduled or is being killed due to resource limits, see the -"Troubleshooting" section below. - -## Monitoring Compute Resource Usage - -The resource usage of a pod is reported as part of the Pod status. - -If [optional monitoring](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/README.md) is configured for your cluster, -then pod resource usage can be retrieved from the monitoring system. - -## Troubleshooting - -### My pods are pending with event message failedScheduling - -If the scheduler cannot find any node where a pod can fit, then the pod will remain unscheduled -until a place can be found. An event will be produced each time the scheduler fails to find a -place for the pod, like this: - -```shell -$ kubectl describe pod frontend | grep -A 3 Events -Events: - FirstSeen LastSeen Count From Subobject PathReason Message - 36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others -``` - -In the case shown above, the pod "frontend" fails to be scheduled due to insufficient -CPU resource on the node. Similar error messages can also suggest failure due to insufficient -memory (PodExceedsFreeMemory). In general, if a pod or pods are pending with this message and -alike, then there are several things to try: - -- Add more nodes to the cluster. -- Terminate unneeded pods to make room for pending pods. -- Check that the pod is not larger than all the nodes. For example, if all the nodes -have a capacity of `cpu: 1`, then a pod with a limit of `cpu: 1.1` will never be scheduled. - -You can check node capacities and amounts allocated with the `kubectl describe nodes` command. -For example: - -```shell -$ kubectl describe nodes gke-cluster-4-386701dd-node-ww4p -Name: gke-cluster-4-386701dd-node-ww4p -[ ... lines removed for clarity ...] -Capacity: - cpu: 1 - memory: 464Mi - pods: 40 -Allocated resources (total requests): - cpu: 910m - memory: 2370Mi - pods: 4 -[ ... lines removed for clarity ...] -Pods: (4 in total) - Namespace Name CPU(milliCPU) Memory(bytes) - frontend webserver-ffj8j 500 (50% of total) 2097152000 (50% of total) - kube-system fluentd-cloud-logging-gke-cluster-4-386701dd-node-ww4p 100 (10% of total) 209715200 (5% of total) - kube-system kube-dns-v8-qopgw 310 (31% of total) 178257920 (4% of total) -TotalResourceLimits: - CPU(milliCPU): 910 (91% of total) - Memory(bytes): 2485125120 (59% of total) -[ ... lines removed for clarity ...] -``` - -Here you can see from the `Allocated resources` section that that a pod which ask for more than -90 millicpus or more than 1341MiB of memory will not be able to fit on this node. - -Looking at the `Pods` section, you can see which pods are taking up space on the node. - -The [resource quota](/docs/admin/resourcequota/) feature can be configured -to limit the total amount of resources that can be consumed. If used in conjunction -with namespaces, it can prevent one team from hogging all the resources. - -### My container is terminated - -Your container may be terminated because it's resource-starved. To check if a container is being killed because it is hitting a resource limit, call `kubectl describe pod` -on the pod you are interested in: - -```shell -[12:54:41] $ ./cluster/kubectl.sh describe pod simmemleak-hra99 -Name: simmemleak-hra99 -Namespace: default -Image(s): saadali/simmemleak -Node: kubernetes-node-tf0f/10.240.216.66 -Labels: name=simmemleak -Status: Running -Reason: -Message: -IP: 10.244.2.75 -Replication Controllers: simmemleak (1/1 replicas created) -Containers: - simmemleak: - Image: saadali/simmemleak - Limits: - cpu: 100m - memory: 50Mi - State: Running - Started: Tue, 07 Jul 2015 12:54:41 -0700 - Last Termination State: Terminated - Exit Code: 1 - Started: Fri, 07 Jul 2015 12:54:30 -0700 - Finished: Fri, 07 Jul 2015 12:54:33 -0700 - Ready: False - Restart Count: 5 -Conditions: - Type Status - Ready False -Events: - FirstSeen LastSeen Count From SubobjectPath Reason Message - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {scheduler } scheduled Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD created Created with docker id 6a41280f516d - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} implicitly required container POD started Started with docker id 6a41280f516d - Tue, 07 Jul 2015 12:53:51 -0700 Tue, 07 Jul 2015 12:53:51 -0700 1 {kubelet kubernetes-node-tf0f} spec.containers{simmemleak} created Created with docker id 87348f12526a -``` - -The `Restart Count: 5` indicates that the `simmemleak` container in this pod was terminated and restarted 5 times. - -You can call `get pod` with the `-o go-template=...` option to fetch the status of previously terminated containers: - -```shell{% raw %} -[13:59:01] $ ./cluster/kubectl.sh get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-60xbc -Container Name: simmemleak -LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]{% endraw %} -``` - -We can see that this container was terminated because `reason:OOM Killed`, where *OOM* stands for Out Of Memory. - -## Opaque Integer Resources (Alpha Feature) - -Kubernetes version 1.5 introduces Opaque integer resources. Opaque -integer resources allow cluster operators to advertise new node-level -resources that would be otherwise unknown to the system. - -Users can consume these resources in pod specs just like CPU and memory. -The scheduler takes care of the resource accounting so that no more than the -available amount is simultaneously allocated to pods. - -**Note:** Opaque integer resources are Alpha in Kubernetes version 1.5. -Only resource accounting is implemented; node-level isolation is still -under active development. - -Opaque integer resources are resources that begin with the prefix -`pod.alpha.kubernetes.io/opaque-int-resource-`. The API server -restricts quantities of these resources to whole numbers. Examples of -_valid_ quantities are `3`, `3000m` and `3Ki`. Examples of _invalid_ -quantities are `0.5` and `1500m`. - -There are two steps required to use opaque integer resources. First, the -cluster operator must advertise a per-node opaque resource on one or more -nodes. Second, users must request the opaque resource in pods. - -To advertise a new opaque integer resource, the cluster operator should -submit a `PATCH` HTTP request to the API server to specify the available -quantity in the `status.capacity` for a node in the cluster. After this -operation, the node's `status.capacity` will include a new resource. The -`status.allocatable` field is updated automatically with the new resource -asychronously by the Kubelet. Note that since the scheduler uses the -node `status.allocatable` value when evaluating pod fitness, there may -be a short delay between patching the node capacity with a new resource and the -first pod that requests the resource to be scheduled on that node. - -**Example:** - -The HTTP request below advertises 5 "foo" resources on node `k8s-node-1`. - -_NOTE: `~1` is the encoding for the character `/` in the patch path. -The operation path value in JSON-Patch is interpreted as a JSON-Pointer. -For more details, please refer to -[IETF RFC 6901, section 3](https://tools.ietf.org/html/rfc6901#section-3)._ - -```http -PATCH /api/v1/nodes/k8s-node-1/status HTTP/1.1 -Accept: application/json -Content-Type: application/json-patch+json -Host: k8s-master:8080 - -[ - { - "op": "add", - "path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-foo", - "value": "5" - } -] -``` - -To consume opaque resources in pods, include the name of the opaque -resource as a key in the `spec.containers[].resources.requests` map. - -The pod will be scheduled only if all of the resource requests are -satisfied (including cpu, memory and any opaque resources.) The pod will -remain in the `PENDING` state while the resource request cannot be met by any -node. - -**Example:** - -The pod below requests 2 cpus and 1 "foo" (an opaque resource.) - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: my-pod -spec: - containers: - - name: my-container - image: myimage - resources: - requests: - cpu: 2 - pod.alpha.kubernetes.io/opaque-int-resource-foo: 1 -``` - -## Planned Improvements - -The current system only allows resource quantities to be specified on a container. -It is planned to improve accounting for resources which are shared by all containers in a pod, -such as [EmptyDir volumes](/docs/user-guide/volumes/#emptydir). - -The current system only supports container requests and limits for CPU and Memory. -It is planned to add new resource types, including a node disk space -resource, and a framework for adding custom [resource types](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/resources.md). - -Kubernetes supports overcommitment of resources by supporting multiple levels of [Quality of Service](http://issue.k8s.io/168). - -Currently, one unit of CPU means different things on different cloud providers, and on different -machine types within the same cloud providers. For example, on AWS, the capacity of a node -is reported in [ECUs](http://aws.amazon.com/ec2/faqs/), while in GCE it is reported in logical -cores. We plan to revise the definition of the cpu resource to allow for more consistency -across providers and platforms. +[Managing Compute Resources for Containers](/docs/concepts/configuration/manage-compute-resources-container/) From 11f3e0fec14504ff2f86b02953c5275f2febfc7c Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Mon, 27 Feb 2017 15:24:11 +0800 Subject: [PATCH 272/407] Delete the parameter "--google-json-key string" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit “# kube-scheduler -help”can not find --google-json-key option # kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-23T00:04:39Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-22T23:56:57Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} --- docs/admin/kube-scheduler.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/admin/kube-scheduler.md b/docs/admin/kube-scheduler.md index 15e47d2f46..91ffc303aa 100644 --- a/docs/admin/kube-scheduler.md +++ b/docs/admin/kube-scheduler.md @@ -36,7 +36,6 @@ DynamicKubeletConfig=true|false (ALPHA - default=false) DynamicVolumeProvisioning=true|false (ALPHA - default=true) ExperimentalHostUserNamespaceDefaulting=true|false (ALPHA - default=false) StreamingProxyRedirects=true|false (ALPHA - default=false) - --google-json-key string The Google Cloud Platform Service Account JSON Key to use for authentication. --hard-pod-affinity-symmetric-weight int RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule corresponding to every RequiredDuringScheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit PreferredDuringScheduling affinity rule. (default 1) --kube-api-burst int32 Burst to use while talking with Kubernetes apiserver (default 100) --kube-api-content-type string Content type of requests sent to apiserver. (default "application/vnd.kubernetes.protobuf") From d30c4a7c7885a6a2e8e4ca5bcf732cc804112382 Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Sun, 26 Feb 2017 11:39:46 +0800 Subject: [PATCH 273/407] Add diagnose tips when you face problem. --- docs/getting-started-guides/kubeadm.md | 32 ++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index b5a9845b44..229ef993f0 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -195,6 +195,38 @@ Once a pod network has been installed, you can confirm that it is working by che And once the `kube-dns` pod is up and running, you can continue by joining your nodes. +If you see following status +``` +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system canal-node-f0lqp 2/3 RunContainerError 2 48s +``` +Or + +``` +kube-system canal-node-77d0h 2/3 CrashLoopBackOff 3 3m +kube-system kube-dns-2924299975-7q1vq 0/4 ContainerCreating 0 15m +``` +The three status ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` very common. + +You may have trouble in configure. to diagnose what happened. you can using ```kubectl describe -n kube-system po {YOUR_POD_NAME}``` to check what's in logs. do not using kubectl logs. you will got +``` +# kubectl logs -n kube-system canal-node-f0lqp +Error from server (BadRequest): the server rejected our request for an unknown reason (get pods canal-node-f0lqp) +``` +The kubectl describe will gave you more details about the logs + +``` +# kubectl describe -n kube-system po kube-dns-2924299975-1l2t7 + 2m 2m 1 {kubelet nac} spec.containers{flannel} Warning Failed Failed to start container with docker id 927e7ccdc32b with error: Error response from daemon: {"message":"chown /etc/resolv.conf: operation not permitted"} + +``` +Or +``` + 6m 1m 191 {kubelet nac} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1l2t7_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1l2t7_kube-system(dee8ef21-fbcb-11e6-ba19-38d547e0006a)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod" +``` + +Then you can do some search with google, and you will find solutions. + ### (4/4) Joining your nodes The nodes are where your workloads (containers and pods, etc) run. From 06104e35a3d89d3b00139d165a0a6a907be7e5f5 Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Tue, 28 Feb 2017 08:01:23 +0800 Subject: [PATCH 274/407] Update grammar for kubeadm.md, thanks @chenopis --- docs/getting-started-guides/kubeadm.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index 229ef993f0..aba4e1d52d 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -195,37 +195,47 @@ Once a pod network has been installed, you can confirm that it is working by che And once the `kube-dns` pod is up and running, you can continue by joining your nodes. -If you see following status +If you see the following status ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system canal-node-f0lqp 2/3 RunContainerError 2 48s ``` + Or ``` kube-system canal-node-77d0h 2/3 CrashLoopBackOff 3 3m kube-system kube-dns-2924299975-7q1vq 0/4 ContainerCreating 0 15m ``` -The three status ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` very common. +The three statuses ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` are very common. + +You may have trouble in the configuration. To help diagnose what happened, you can use the following command to check what is in the logs: + +```bash +kubectl describe -n kube-system po {YOUR_POD_NAME} +``` + +Do not using kubectl logs. you will got the following error: -You may have trouble in configure. to diagnose what happened. you can using ```kubectl describe -n kube-system po {YOUR_POD_NAME}``` to check what's in logs. do not using kubectl logs. you will got ``` # kubectl logs -n kube-system canal-node-f0lqp Error from server (BadRequest): the server rejected our request for an unknown reason (get pods canal-node-f0lqp) ``` -The kubectl describe will gave you more details about the logs + +The ```kubectl describe``` gives you more details about the logs ``` # kubectl describe -n kube-system po kube-dns-2924299975-1l2t7 2m 2m 1 {kubelet nac} spec.containers{flannel} Warning Failed Failed to start container with docker id 927e7ccdc32b with error: Error response from daemon: {"message":"chown /etc/resolv.conf: operation not permitted"} ``` + Or ``` 6m 1m 191 {kubelet nac} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1l2t7_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1l2t7_kube-system(dee8ef21-fbcb-11e6-ba19-38d547e0006a)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod" ``` -Then you can do some search with google, and you will find solutions. +You can then do some Google searches on the error messages, which may help you to find some solutions. ### (4/4) Joining your nodes From 1cbd24e888ae2f952bf8cb2db5d374f934a6f15f Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Tue, 28 Feb 2017 09:13:50 +0800 Subject: [PATCH 275/407] Merge two pieces to one words --- docs/getting-started-guides/kubeadm.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index aba4e1d52d..24c9498e44 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -195,34 +195,32 @@ Once a pod network has been installed, you can confirm that it is working by che And once the `kube-dns` pod is up and running, you can continue by joining your nodes. -If you see the following status + +You may have trouble in the configuration if you see the following statuses + ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system canal-node-f0lqp 2/3 RunContainerError 2 48s -``` - -Or - -``` kube-system canal-node-77d0h 2/3 CrashLoopBackOff 3 3m kube-system kube-dns-2924299975-7q1vq 0/4 ContainerCreating 0 15m ``` + The three statuses ```RunContainerError``` and ```CrashLoopBackOff``` and ```ContainerCreating``` are very common. -You may have trouble in the configuration. To help diagnose what happened, you can use the following command to check what is in the logs: +To help diagnose what happened, you can use the following command to check what is in the logs: ```bash kubectl describe -n kube-system po {YOUR_POD_NAME} ``` -Do not using kubectl logs. you will got the following error: +Do not using kubectl logs. You will got the following error: ``` # kubectl logs -n kube-system canal-node-f0lqp Error from server (BadRequest): the server rejected our request for an unknown reason (get pods canal-node-f0lqp) ``` -The ```kubectl describe``` gives you more details about the logs +The ```kubectl describe``` comand gives you more details about the logs ``` # kubectl describe -n kube-system po kube-dns-2924299975-1l2t7 From eb57603fcbf52f8114aa000172e482fd46e5262b Mon Sep 17 00:00:00 2001 From: xilabao Date: Sun, 26 Feb 2017 21:09:41 -0600 Subject: [PATCH 276/407] add http proxy infomation in kubeadm --- docs/admin/kubeadm.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index a43beec497..24f39217f0 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -258,6 +258,8 @@ These environment variables are a short-term solution, eventually they will be i | `KUBE_ETCD_IMAGE` | `gcr.io/google_containers/etcd-:2.2.5` | The etcd container image to use. | | `KUBE_REPO_PREFIX` | `gcr.io/google_containers` | The image prefix for all images that are used. | +If you want to use kubeadm with an http proxy, you may need to configure it to support http_proxy, https_proxy, or no_proxy. + ## Releases and release notes If you already have kubeadm installed and want to upgrade, run `apt-get update && apt-get upgrade` or `yum update` to get the latest version of kubeadm. From 3c20c4da09024eaa4874f2ed68b57fdd24e7016c Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Mon, 27 Feb 2017 17:45:20 -0800 Subject: [PATCH 277/407] Update landing pages for Tasks and Tutorials. (#2634) --- _data/tasks.yml | 2 +- docs/tasks/index.md | 6 ++++++ docs/tasks/kubectl/list-all-running-container-images.md | 2 +- docs/tutorials/index.md | 6 +++--- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/_data/tasks.yml b/_data/tasks.yml index 5dae817f8e..ab237c3474 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -3,7 +3,7 @@ abstract: "Step-by-step instructions for performing operations with Kubernetes." toc: - docs/tasks/index.md -- title: Using the Kubectl Command-Line +- title: Using the kubectl Command-Line section: - docs/tasks/kubectl/list-all-running-container-images.md - docs/tasks/kubectl/get-shell-running-container.md diff --git a/docs/tasks/index.md b/docs/tasks/index.md index d490fe5532..6bd8db2e6c 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -6,12 +6,18 @@ This section of the Kubernetes documentation contains pages that show how to do individual tasks. A task page shows how to do a single thing, typically by giving a short sequence of steps. +#### Using the kubectl Command Line + +* [Listing Alll Container Images Running in a Cluster](/docs/tasks/kubectl/list-all-running-container-images/) +* [Getting a Shell to a Running Container](/docs/tasks/kubectl/get-shell-running-container/) + #### Configuring Pods and Containers * [Defining Environment Variables for a Container](/docs/tasks/configure-pod-container/define-environment-variable-container/) * [Defining a Command and Arguments for a Container](/docs/tasks/configure-pod-container/define-command-argument-container/) * [Assigning CPU and RAM Resources to a Container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/) * [Configuring a Pod to Use a Volume for Storage](/docs/tasks/configure-pod-container/configure-volume-storage/) +* [Configuring a Pod to Use a PersistentVolume for Storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/) * [Exposing Pod Information to Containers Through Environment Variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) * [Exposing Pod Information to Containers Using a DownwardAPIVolumeFile](/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/) * [Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) diff --git a/docs/tasks/kubectl/list-all-running-container-images.md b/docs/tasks/kubectl/list-all-running-container-images.md index 4fb64ee442..070de69004 100644 --- a/docs/tasks/kubectl/list-all-running-container-images.md +++ b/docs/tasks/kubectl/list-all-running-container-images.md @@ -1,5 +1,5 @@ --- -title: Listing all Container images running in the cluster +title: Listing All Container Images Running in a Cluster --- {% capture overview %} diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 76e42570b1..25b8cca32e 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -7,14 +7,14 @@ A tutorial shows how to accomplish a goal that is larger than a single [task](/docs/tasks/). Typically a tutorial has several sections, each of which has a sequence of steps. -#### Kubernetes Basics - * [Kubernetes Basics](/docs/tutorials/kubernetes-basics/) is an in-depth interactive tutorial that helps you understand the Kubernetes system and try out some basic Kubernetes features. -#### Stateless Applications +* [Online Training Course](https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615) * [Hello Minikube](/docs/tutorials/stateless-application/hello-minikube/) +#### Stateless Applications + * [Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) * [Using a Service to Access an Application in a Cluster](/docs/tutorials/stateless-application/expose-external-ip-address-service/) From dd595ae05b225330f1e8f91de3ae78f11c631be1 Mon Sep 17 00:00:00 2001 From: Wang Jie Date: Tue, 28 Feb 2017 18:57:54 +0800 Subject: [PATCH 278/407] Update static-pods.md It works. Please review it again. --- docs/admin/static-pods.md | 65 ++++++++++++++++++++------------------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 7ac6e7c890..44e7134af5 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -20,47 +20,48 @@ The configuration files are just standard pod definition in json or yaml format For example, this is how to start a simple web server as a static pod: -1.Choose a node where we want to run the static pod. In this example, it's `my-node1`. +1. Choose a node where we want to run the static pod. In this example, it's `my-node1`. -```shell -[joe@host ~] $ ssh my-node1 -``` + ``` + [joe@host ~] $ ssh my-node1 + ``` -2.Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: +2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: -```shell -[root@my-node1 ~] $ mkdir /etc/kubernetes.d/ -[root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml -apiVersion: v1 -kind: Pod -metadata: - name: static-web - labels: - role: myrole -spec: - containers: - - name: web - image: nginx - ports: + ``` + [root@my-node1 ~] $ mkdir /etc/kubernetes.d/ + [root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml + apiVersion: v1 + kind: Pod + metadata: + name: static-web + labels: + role: myrole + spec: + containers: - name: web - containerPort: 80 - protocol: TCP -EOF -``` + image: nginx + ports: + - name: web + containerPort: 80 + protocol: TCP + EOF + ``` -3.Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. On Fedora edit `/etc/kubernetes/kubelet` to include this line: +3. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. + On Fedora edit `/etc/kubernetes/kubelet` to include this line: -```conf -KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" -``` + ```conf + KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" + ``` -Instructions for other distributions or Kubernetes installations may vary. + Instructions for other distributions or Kubernetes installations may vary. -4.Restart kubelet. On Fedora, this is: +4. Restart kubelet. On Fedora, this is: -```shell -[root@my-node1 ~] $ systemctl restart kubelet -``` + ``` + [root@my-node1 ~] $ systemctl restart kubelet + ``` ## Pods created via HTTP From 6f83f5070a1dd42c58259cda5308b44c39e943eb Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 28 Feb 2017 09:08:11 -0800 Subject: [PATCH 279/407] Move Guide Topic: Multi-container pods. (#2642) --- docs/user-guide/pods/multi-container.md | 170 +----------------------- 1 file changed, 2 insertions(+), 168 deletions(-) diff --git a/docs/user-guide/pods/multi-container.md b/docs/user-guide/pods/multi-container.md index 55e0e56f84..465379e51c 100644 --- a/docs/user-guide/pods/multi-container.md +++ b/docs/user-guide/pods/multi-container.md @@ -4,172 +4,6 @@ assignees: title: Creating Multi-Container Pods --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -A pod is a group of containers that are scheduled -onto the same host. Pods serve as units of scheduling, deployment, and -horizontal scaling/replication. Pods share fate, and share some resources, such -as storage volumes and IP addresses. - -## Creating a pod - -Multi-container pods must be created with the `create` command. Properties -are passed to the command as a YAML- or JSON-formatted configuration file. - -The `create` command can be used to create a pod directly, or it can create -a pod or pods through a `Deployment`. It is highly recommended that -you use a -[Deployment](/docs/user-guide/deployments/) -to create your pods. It watches for failed pods and will start up -new pods as required to maintain the specified number. - -If you don't want a Deployment to monitor your pod (e.g. your pod -is writing non-persistent data which won't survive a restart, or your pod is -intended to be very short-lived), you can create a pod directly with the -`create` command. - -### Using `create` - -Note: We recommend using a -[Deployment](/docs/user-guide/deployments/) -to create pods. You should use the instructions below only if you don't want -to create a Deployment. - -If your pod will contain more than one container, or if you don't want to -create a Deployment to manage your pod, use the -`kubectl create` command and pass a pod specification as a JSON- or -YAML-formatted configuration file. - -```shell -$ kubectl create -f FILE -``` - -Where: - -* `-f FILE` or `--filename FILE` is the name of a - [pod configuration file](#pod-configuration-file) in either JSON or YAML - format. - -A successful create request returns the pod name. Use the -[`kubectl get`](#viewing_a_pod) command to view status after creation. - -### Pod configuration file - -A pod configuration file specifies required information about the pod. -It can be formatted as YAML or as JSON, and supports the following fields: - -{% capture tabspec %}configfiles -JSON,json,pod-config.json,/docs/user-guide/pods/pod-config.json -YAML,yaml,pod-config.yaml,/docs/user-guide/pods/pod-config.yaml{% endcapture %} -{% include tabs.html %} - -Required fields are: - -* `kind`: Always `Pod`. -* `apiVersion`: Currently `v1`. -* `metadata`: An object containing: - * `name`: Required if `generateName` is not specified. The name of this pod. - It must be an - [RFC1035](https://www.ietf.org/rfc/rfc1035.txt) compatible value and be - unique within the namespace. - * `labels`: Optional. Labels are arbitrary key:value pairs that can be used - by - [Deployment](/docs/user-guide/deployments/) - and [services](/docs/user-guide/services/) for grouping and targeting - pods. - * `generateName`: Required if `name` is not set. A prefix to use to generate - a unique name. Has the same validation rules as `name`. - * `namespace`: Required. The namespace of the pod. - * `annotations`: Optional. A map of string keys and values that can be used - by external tooling to store and retrieve arbitrary metadata about - objects. -* `spec`: The pod specification. See [The `spec` schema](#the_spec_schema) for - details. - - -### The `spec` schema - -A full description of the `spec` schema is contained in the -[Kubernetes API reference](/docs/api-reference/v1/definitions/#_v1_podspec). - -The following fields are required or commonly used in the `spec` schema: - -{% capture tabspec %}specfiles -JSON,json,pod-spec-common.json,/docs/user-guide/pods/pod-spec-common.json -YAML,yaml,pod-spec-common.yaml,/docs/user-guide/pods/pod-spec-common.yaml{% endcapture %} -{% include tabs.html %} - -#### `containers[]` - -A list of containers belonging to the pod. Containers cannot be added or removed once the pod is created, and there must be at least one container in a pod. - -The `containers` object **must contain**: - -* `name`: Name of the container. It must be a DNS_LABEL and be unique within the pod. Cannot be updated. -* `image`: Docker image name. - -The `containers` object **commonly contains** the following optional properties: - -* `command[]`: The entrypoint array. Commands are not executed within a shell. The docker image's entrypoint is used if this is not provided. Cannot be updated. -* `args[]`: A command array containing arguments to the entrypoint. The docker image's `cmd` is used if this is not provided. Cannot be updated. -* `env[]`: A list of environment variables in key:value format to set in the container. Cannot be updated. - * `name`: The name of the environment variable; must be a `C_IDENTIFIER`. - * `value`: The value of the environment variable. Defaults to empty string. -* `imagePullPolicy`: The image pull policy. Accepted values are: - * `Always` - * `Never` - * `IfNotPresent`Defaults to `Always` if `:latest` tag is specified, or `IfNotPresent` otherwise. Cannot be updated. -* `ports[]`: A list of ports to expose from the container. Cannot be updated. - * `containerPort`: The port number to expose on the pod's IP address. - * `name`: The name for the port that can be referred to by services. Must be a `DNS_LABEL` and be unique without the pod. - * `protocol`: Protocol for the port. Must be UDP or TCP. Default is TCP. -* `resources`: The Compute resources required by this container. Contains: - * `cpu`: CPUs to reserve for each container. Default is whole CPUs; scale suffixes (e.g. `100m` for one hundred milli-CPUs) are supported. If the host does not have enough available resources, your pod will not be scheduled. - * `memory`: Memory to reserve for each container. Default is bytes; [binary scale suffixes](http://en.wikipedia.org/wiki/Binary_prefix) (e.g. `100Mi` for one hundred mebibytes) are supported. If the host does not have enough available resources, your pod will not be scheduled.Cannot be updated. - -#### `restartPolicy` - -Restart policy for all containers within the pod. Options are: - -* `Always` -* `OnFailure` -* `Never` - -#### `volumes[]` - -A list of volumes that can be mounted by containers belonging to the pod. You must specify a `name` and a source for each volume. The container must also include a `volumeMount` with matching `name`. Source is one of: - -* `emptyDir`: A temporary directory that shares a pod's lifetime. Contains: - * `medium`: The type of storage used to back the volume. Must be an empty string (default) or `Memory`. -* `hostPath`: A pre-existing host file or directory. This is generally used for privileged system daemons or other agents tied to the host. Contains: - * `path`: The path of the directory on the host. -* `secret`: Secret to populate volume. Secrets are used to hold sensitive information, such as passwords, OAuth tokens, and SSH keys. Learn more from [the docs on secrets](/docs/user-guide/secrets/). Contains: - * `secretName`: The name of a secret in the pod's namespace. - -The `name` must be a DNS_LABEL and unique within the pod. - - -### Sample file - -For example, the following configuration file creates two containers: a -`redis` key-value store image, and a `django` frontend image. - -{% capture tabspec %}samplefiles -JSON,json,pod-sample.json,/docs/user-guide/pods/pod-sample.json -YAML,yaml,pod-sample.yaml,/docs/user-guide/pods/pod-sample.yaml{% endcapture %} -{% include tabs.html %} - -## Viewing a pod - -{% include_relative _viewing-a-pod.md %} - -## Deleting a pod - -If you created your pod directly with `kubectl create`, use `kubectl delete`: - -```shell -$ kubectl delete pod NAME -``` - -A successful delete request returns the name of the deleted pod. +[Communicating Between Containers Running in the Same Pod](/docs/tasks/configure-pod-container/communicate-containers-same-pod/) From 8b97c4265f09ca4a520aaf475062192ddf467f3d Mon Sep 17 00:00:00 2001 From: EJ Date: Wed, 15 Feb 2017 19:05:45 -0800 Subject: [PATCH 280/407] fix link to go to pod-lifecycle page --- .../configure-liveness-readiness-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md index ff05756350..a5c3d8ee56 100644 --- a/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md +++ b/docs/tasks/configure-pod-container/configure-liveness-readiness-probes.md @@ -255,7 +255,7 @@ In addition to command probes and HTTP probes, Kubernetes supports {% capture whatsnext %} * Learn more about -[Container Probes](/docs/user-guide/pod-states/#container-probes). +[Container Probes](/docs/concepts/workloads/pods/pod-lifecycle/#container-probes). * Learn more about [Health Checking section](/docs/user-guide/walkthrough/k8s201/#health-checking). From 764225a7a018a7853d66690ab887516e133cde58 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Tue, 28 Feb 2017 14:58:36 +0800 Subject: [PATCH 281/407] fix the command output fix the command output --- docs/getting-started-guides/gce.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting-started-guides/gce.md b/docs/getting-started-guides/gce.md index 245de295f9..109dcb8d95 100644 --- a/docs/getting-started-guides/gce.md +++ b/docs/getting-started-guides/gce.md @@ -134,10 +134,10 @@ $ kubectl get --all-namespaces services should show a set of [services](/docs/user-guide/services) that look something like this: ```shell -NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE -default kubernetes 10.0.0.1 443/TCP 1d -kube-system kube-dns 10.0.0.2 53/TCP,53/UDP k8s-app=kube-dns 1d -kube-system kube-ui 10.0.0.3 80/TCP k8s-app=kube-ui 1d +NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) AGE +default kubernetes 10.0.0.1 443/TCP 1d +kube-system kube-dns 10.0.0.2 53/TCP,53/UDP 1d +kube-system kube-ui 10.0.0.3 80/TCP 1d ... ``` From 2ef4477df3d8a2b84baf592decbfee4c28ef6fd0 Mon Sep 17 00:00:00 2001 From: mlambert890b Date: Sat, 4 Feb 2017 00:06:16 -0800 Subject: [PATCH 282/407] mirantis_logo.png ,/images/community_logos/mirantis_logo.png updated per Mirantis request --- images/community_logos/mirantis_logo.png | Bin 11543 -> 8607 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/images/community_logos/mirantis_logo.png b/images/community_logos/mirantis_logo.png index d2c395e323e80f86b3ba57d55731ab0707023f77..4f407a37bb6a3b411f8209c789232391adb43e83 100644 GIT binary patch literal 8607 zcmb`NWl$Sm*tdhb6e#Z6;$B<}4Nh?@Qb>W~P^<(94#k7J1t{(koZ=d!KufVuq)^-; z1s?wMetPHm`rI?S=flqIIkWf9?)AIo#OP|jA;PD{2LJ#>YQWcePvg?l5r~WZbQZe1 zl6@L5y_D1paGwf{YZnawFay+HD;oIa9_IOjzc0WaDqR%U9F7_)x2{(0bImJhZmF?9>)e(_P2E-0|JWyvCf$iK^RE4V}-j!cM8u5YvD= z)Q-~`|2>5LT()4HNzIxQ#rtyu71V9A14j7g+Y9^eI3xUtH<15-35{>K(+TDp+-a0e zY)Z`e62>?(TflurVWAr5tK!GL=XQ^EJC}ZEc-eZmx)jFlYuO;OJ1_^x6oz$SiZ=WZ9Noy$8m>e_DI|9l}-@oG6?<1x+ml9Uki=0)SFf3 zT^V%a{g(92LucL-6OWKN<#$UBwI8dt6|w~AF)>gO|gl1;s&5yCdvELAHFRzK( z__APo?blTFlTG#tW-kWxaJkTPaan!)lwhv_{80<)CK0wrM}P2g#9IjS7xqLp^PG^) z5&pI{5Aic_Ksr=(z-8W=%^?R;ASHUuyWYA}gKXuZ$=*RQc`>IO&-S!_I} zHKX+jg`HA%;4(H+@y(I#{HyQ1xi%tExzBbjo~!i3i9KSYy@15vAMTiJ z6)|)13$u4`B7ac=ehB#<{rHg32@qt$6vCNvhDjAC87X%l6p%E36RRb6qdjVMk)i%4 zSpwu;cI_@S4M0S6S65lXQA?3gxBptPU~A9k7Q5;8A=4?{w^nf0!Om z9;%mK+6-+};?hVYvyfS7jiY1_aMFT3?DQ|riv~YrhPgg)pv&Q$!J5PIgh+eGJr7;x z#R=~-2HXXmr(5jQW7a>Qj#xaV-G2&ZC(i)e3%({)-Lf_xy__*esTiFoL#`nOYJ6^fXV7dDEIehIBgQk5A-k{LjPJDE9(f4x+_HLVe z$^-RhFGx z=7Dl{)8Ap+_Mls9e`x+8k9)J@5SLA@jy(idQXv?(nLJk0KH7taABI2h+jJJj4`?qO|%C0RRoV-k#I&nW;Zm{ve>FJsn`v|m4-lEx@H8PU9bQjZlnr?hq4P27+_|cc4ovzv-8F~f`jI$zop^2 z$L;?vF5-jDGiYKHwdZ_lbqg;vV7GFxuZ2|~Sub@ye5*&o*jOLcL)sEx(crKF?dKTv zqEzj6as4t@0VJ)mtNH4Fj18>>A9@zk(~>(9R;At%4gs7 zg4Kd}z*Mmv{u0Ja((g)>?cG(iI|szzhtjPT(rNo~e&=zQh?>#pen!CD zjf_&rMz6+;>iEdsxDxB%_1vWYut(j9TP$(!YSQcw4fe;P=-Vw8wxoWB8`C|`G8B2B)_>Yyv7&tTt$%U5aJ8tI zUvL5kEL?!QO{T9EptBJAJ#So{FG6)GF|guj*D`Z%L+Id*wkjI~%CDC;uBccsz^z#f zoBlTjw`cVEc{F(=ZLHj^vgC4I zZO=-!_pM*A;l^?6Kv#J_stEYk?U5=$G^7KMg~+ZD9{Gk{zLze13Z)2bKV zZqN9a@#z2(?4f2O%ug$#J{-GJW#(Yz zX1oEj7L2NUHJ}45s$^B{o>)rxAnk!J-0CT%-hr`t^<>6%$` z)3+_QCbR4Kk!eSD@!sIoiww&P#sMo+ zSbl|?5znU?*}AL4$nJ`9>L~+l-Yfv1qAM5y$4-VvfZBFxS2ikT8#m+VH#Wgx5t4_> z1Hch6fFrL_*G5z~FKYWlezPFYsCnZ6{r5lm*B-;-4b}OL?sZl1;?7g4qsiK(y$AG4 zTy3i74*}1OhM+FTl1t29dqVw{M(+M9uBe~|lU8X%p1GLnnbxtbS8Gj2aH~OiY5$KW zsm0F!1PLr;99Tq*NX+(&mVx)r=(r7^y~uOv>jbpdxO6x#Oy89mZ23)#jlNzt_o1-~ zk{oI#dA#E3zOTd^7BEVQFA*%RmV^iKK``a~f^0=b?e|+%U z0tFzFXW5I%2Htdi+VYfZr=Lpt%Td&zrEkk=Vo zh%&fj#ts2(ZJgPhH2OytzH^oJAO{-Nt0^|ww{H!6@`XBbw|dbbc5c-%-yfOZ!Jpp4 zOuq9_CeQ5^71l$78>t;bgf^Aj(4~|3kUY<2M$jiRf3DF!ARcilDaM798OkBn*3xW2 zLRg1Cqhuw+@VjSdQ9#0|KoTzipw?J#$iwR7BvLfuRZG}#yJh{{nh{%4?42nk1fyGG zU}5AZ$L0CU2o8H}`duLn%6N+MeIm6-%nNoEc;xAC+FZ%GaE41-;NDfRlD9N^3 z@wxan>yS1 z-5J-&Qu>^J8C_N!`>-u(3~jzPxKk|49apNeS2WhKGBzf+vNVwho|NIbUgJG0W}B;k zDnPL|(Ak-z?3R1mpxxb^dXEu=nM*wRUPMVXUK7*qHa+NJr8Li8e5Z2?ma$b{9XK5t z(I(0lUX6PCE(+}70#V{~q$k;Sbu?cBJD%kr9bhNJYO&fWvtj(VYjL=#mg0SeB9@j- z9pxilfj2Ib?pF@E#_^vp?~S!=Z~iKmBD`uXe+c^`Rro!+qpdlm{1i!MTYM2Kq?Ysc z-IS@^#`4U2Dy8s(f+dK_fHs3qRw{Bd(v8Q9)IFWClr85CT5q5h6bP$VKQj8m+L0Wwr`Fi0mkz ze{M*FyJkO%u=`eBf~D5Udw`X-OsJh!vJAJBcbvsh2!w8dfYJlCmKhh#IyX}nm)?jE z!{2yGlikCiysCKFyG`O^4#M^`n8PJyE~&HYrbzNJU1n8fiC{|oSQ_Fz-}vfn}E3U--$a{#qa+E739U?S;_Xv{aH_zOOm zFS86`A?MB+JWU$0b+uk`$8LxVG^YXbIt4|iX`;d=_Bn3;TBd{S7d_GTNr z==LG>Ik}`R=q5Ujd_+r{Z2?Q3_p>Rct;N>;9>ex>#3uvclp&g8^9mo5#nmm)mbYtM zrh-jdyzV;(T=Y0;o=vy-tNQbM=GbTQifAn6IuPBbegWi$h74Y)9Y_0xeXAn%!S##% z&L3R|I3f^PDo?lG0&d0O9p`1{6g;9Sqbwm5KLsedU>G2=mw?6n*)p@^ZR~evb@W#Fq<^t7($G1^k3qmxW|wyJHvH2FMKYbYobP zLk?Me-Y#PaeylDoW+k3n46Er}ld+lqGqEZN{ff^kF&x_Ta*|y_*;=Tm0>BxP$;`|I zQHHSM_LDWUE;Ap7orn{zg*LzYCJu4?wsFbZc3JQrN=vRByGc-kca`ABHKsUc*E_-p zF8k`LRu57y2IdK|Pk)vv|B6hN8KN4IN9VW1^vh(dmAj3?USn?ecjJr;l zv@5KSl~no#H3X}cuG3@aDzdlC*1oj&0)GyS_WdtuBFn*XCEMjAja50TZpSFZm-Gh* zv3mN}80q4|>u<(`R7O~o^cWJ?1gw2VdPB7b@fdIia78csBlA=#<}{W>q8;-CJUkH7 zvSsEYY2t!-Y{X#(OS4t90;Er;621R^A^AiAYWO3KBchaRB9lwXs_9$+ znVP@9TQch*RPlwufBVY&Gq<`1jz>(2NLg5!sJ8d;v+svJLQB~m31KrxM7rQcbI%s{1#+~i|G;oQpFjTcm#t0{*3UV> zd8)LqqZRq<;-o<$SAKE+tBfvzcm4ncB*yBcJL_4lP8mzPZpZtr0K(g>u74}%*vp)tT~?v#Q%BRvLo{XzTsg(f zxKDP3G_ZAhFcC{|JvW08$BQv^fB!`8a6~Cv=Nm$Snll{4RQrkq5JXZ229PFQ<-IRb z_({O7f%?vNc{{|Hjb=5Pu~>7(u0wzoDqj^7UZ>gWGHUf%axAf|ME*HD>g7CTz~Uu{ zYA7S6)*6YCD+J4X;VBnVrI?^Ysvu)(+I;eTxlD{sZ&qH?x>h0M^qyrlvZ?AHh~p_* zMH<*$^=!?>3XqD!h7A^2Dwjp^tN> zKw-M!Acj7b3YxxHsd8L~bttR<9}MzOXb?has*!tCbg1J?H(bg?K{+LEmghw zQhr99zVVyI9=P--<0El^CEK4ESYB+Ocu4OaClGASe0Yr+ssoXc?9rU515^VS(~Feq zUybx~zWre)Ve@t_l-@hf$#6+^psmdcwEQy1*30^HZVE@ljaDe0IgRK}0(bCOZZjxR zJi0M_X)cJSTHQz!gLZk*+1$&z@G|8X4k>Qk30oT754rKA<;TZTXF9BM0To`*LYdSk zzG{iZBQ;*qYd+1oc^I9prcQlMC&k6d_WNS2Jdtf`;S{u39O$Nf^G<1-a8dTx7#-jSZ`0(2`)#iHG&FGoT`0vU^ONIV%kWu8fJ4U+}Q;NCKiWGpqTORi4%eaE=uyd3%k@K`{$-9u5FF8Mu?oa!sl!0d|M7cxhuB zPqNK5aOROCZeLQaXNYZy!laQ zmGe?`TG?=@Kb&ae91UkMu@ev4U0depGPimzn0(SiRxe*M%uuD}amZxWQ^kT0c*36A zum$~7L>}+El**#RGa|G?PP50-(!K z5j<`dSmXqE-V7beS8&G~8?;Hi7}*B~w&gpsd-}{)=qu(^Se5b1npzp$YV1HXZ&+Dp zIlyr`oLywKCNhpT9%qX_Bh+3+uy>~*)`#6E!GKl=j{opuM~<<#9RtAr*v(C9(Pp>r z*o3X&W;TrlXqi1Zgg($)sSfB@q4&!J6w-PxIDNvI8B`y?CWJ?U6L}1mHK$84&o(Ej z;}?jUyEjnIn@%;=u7rOR;$QLn3vd{e+8=I^MZo`7B2vb+H_lEMD^9TT*Tb9f=0pP5@QkLKo*W}gD;ym6t z4v7Ziv0?MLfm}VBO7BM2v>y+j=#XIlTewn{tKu-vy4gG~aw(?tUmJ|*C}O|t>t!Q;tokzGxgl+orNm%w;cM=DHMY|&H z6v6Otf{$#M1FZ+OnGEhX>MC#ddOr{lv3rq6Y&3+7kB-rLagK!#(b{UYIRpdTOFPN% z2{Vi^)l;m_jw7B@X$sU*=LH4h*!L4Jm_9jP!OPg=Ys2!@Sxpy1=Jr<eXhcBDn?pe-+^%p5!YGV!9PC}RMRUIJV8im}+m9hO}>9^cqOC#6dBs8!OHh5L@?n8tC7X@_J=uQJC@X{Hk z+gXAv&33q1&IbBr7wP~jk8AitWT5M`Df5e;jI0&)0&-})`2i;?YY)-YmWO7HvB>$X zMX^-iovIc)_=B)ZTy8i#9X54O6l0ED<3ygkVwYs}|DY!P8C4>#cO9q}oG2i&)ds!a zmPFTmN^QPk$?o-9NAx^kGb5cVla$Hp{^~k>UccZ(L;$nxz&Gu58-0y5N-(2@%mt){ zs&3`&(%*Fup^FBcj+gVxq3xq%t!Bpc-!J z#O`y`_ao>g3+tZYvS;{tDy^_{v-OYi&u!9v^0XDJlNH(tKEx^=rBSv;z3W6)%PiU2 zM68A(DUV%m%`7Z(WTAkXDG}Daa#a3@ZscqH=I;}2%T_S&*#IA5lHkTYV- z;k@Q=Lk~i^IBe4bKpJxp0BiRaMK<{s@|C z*B!&3)SBKL`6&nnuWXcPuWu9oPwAN9v!+(Yce`sxZleLR%^UfXQ%MIucBmxMcUP}< ze`X2~_&oufrpSYLx9r)?X}w=Z}f#j@*_3fdDW(p7(*7qpp1vt zi9-S`sP=H>7!WYugyw~Y)U|T(f43^ACewkYG(EXsWrbt7W&)>0hK@I%Nd5gzB2BXm z(g8HpDu(mKzHVsky-jg;A>NlRfP%wr5dA@`@STG19C7lDgyf9QsN_{p*brrZSK!$0;k6)A1Di?&%-;7_ZChXOwZEv;Kq~ z1$8mseZ8_X$9f{|g1}kcs{tL08rY_5%3&J6FLc)FUUcJ!eWI)tH}lnaK}ranbTAlo zR~jx2U~@xv*XI>=Q!k% zI`inZ0tvp5t1o{fqa`KTziMa$^9?QO4@9^=OI1gI@8u?P2YsUGOaDO$bSvX|(^ya$ zABmG2%+9gl77oNU~PCbz{7uvopsanH8->g`@2g9Vu@@q3umTK}*q8tCXpuDhO6w-pNPmWGO z`07%cm;HpgnwGEr7pZAzwTtVoHLFg$oL2G*0u2e>L)yIq%&>{9=>yST`Vc zoY8l0biB7)U7=%eJciqDyewxrI`^YkOw&Q{=DQ>;AG*GXeogf}(+I*oF3@Eh;M0W^*xemWOb|zkO0cnS353)?8zP+13Ro?-ZKYyKEt2nDZo7p0~1+4w%kOuL<1A zexgp}LYVh`ObiKDls|G4798~&B5uHCV`ix5YV5Pvo=}h3s=)tH4#OQJIbJvQ@*}b7 zNEZT-QX{wqfGdoYk~L2;Uyr|+J?xh^bPH~fG^CT2`QHV3Br$OkrC)i4>U`R90Z>!X KeqF0%9r{1m>yx$s literal 11543 zcmb`tRZ|>X6E!+WaCZsr?(PnQ1$P;2kl+phhT!fH+})i)2Zvz6A!u;t!9s$byysM% zAMn+8v3l+9i(S=gUvyXZj`^gago#Fm1^@ssRg~qm|MBWS1fd}Pvpi3~CH@hro3fE7 z0DzAFKfnQU@<;#xG))INxlf<$UAV5!qe@8i+TTl=cyaot^h>l`u+1tU|LPfGxTQ7ub}yBjt>GWP3$ zQ#url-VK0*(rQEF`AP=|_!BHH&PEGE?gzm6Euy0UVCuOr?vGLL*k`iYrf`sOIG9(8 zuqq0K03aQZC|d!LR)mA(e`C}E6d?jEr>w2^0iQSkmP{c>ivURepX@+5fJr(HAzV=k zfEwR6S{`682B@3Uk5vE|asvqLR7XSrYa9S>6$3kEKvO3GHj9Ja0zkn4aBD?JumBJP z0hSYVbbf&FYyg4ct%3LrTLaE9>p!KkTg0LCA_}1-v{1_qpz%(E)QYy^TJcu@;UK$yydhpvZ*Yi3=8izdpqpU2$DMjfF?&txWE*SDl=<@->OVF zhF-9}M*!fq$8~Ut83h4iANFg`_x)PxRjG&p0I^d^bq4^<<>@(fzqd+_qW}Q%MWL)s zGUWGzBb}Y{N6s`+uv&60NmKB{wCp3( z+^{%@^t!Ml+_0ZOC=s#?{ zcyf}6igJH=w2?`~3l<;$sI|xbnItJU&J8;uLFOCz%*{DAEZZ#1j+$mY)l6H5Q9DM~ zq+f^76Z2p_*X)ak7pee56-4xvr3O*TC>yF7YOE<&QGcZ8$Nh;ago+TZHN-@lR;Aj; z*p2&lgv5%MC(2Nnoo)p$37-omFj{Uzh#oISj+Y6V46o8$rn`W*pt#_oO?S#(oy)IG zMwd4E)5fURKNStkhhz8IL1gkj^+U}n->5@ZrEsV_<6H?<1V#W>P&U2Y+s^KZx5*x9an zhO8Z|UkFpaqh$ay7&FLOxpb>*Dr&ZC(Dm;0rgW34&2@qr6!rLY_N!5sGHdl}@;{yH zkn4b}#j2uez*VlgQf0wLuQ5nQ!)n8ta%EaY%@+H;W7O!@q^*AJG)|bck=~&KBaF>6 z>tB3Wd%nF=^u$TjR2PfL@d5b^TD9-e?WB)X0FV-3y$3;xMta7A+O+9 zgYwv#I@M%mWg6Bh*J{-&o%abgTvp~+^eOZhJ%?h+j02fWnUuguvmf%0b=GyZD$y#5 z)Fnh88aQj5%E^m+b^QZRX!+T7$UV|&?9YN z%#w(Y3b?z9Q*e!M+-S*}4NJ?FgEBx_zrs_X^Wvq;iV2F?@eX6A`}F(HW6ooxj7W?$ zjH>Eq>Q_11>d*?73NbY?wegwhnUa~6Y=K+}o>kuC+~eHU+?`HKV?Sf|j;f9>Ta}gr z;|imzj$W4tV-cfrLp@`&R>k`KdWMyr`kqSMN~lJfMs2|gc)01NW3r>VleFn@4QZpH z!?_~}1hygucaAm;U#0b>1$74{yhengGEJ@B>^0BRnz5}4wgS*+UfQ5(r z+}7No%c_mWE5R#=tEkPwY5qc%0wX@pAHx%JyIP0yQs{*l+)g`n!J2ZKq2TsoIxcIj z6odNJrp|KTnn%zzvE*+_kHUS)eQ}{}-fbg`X(3qWMCX*|uxo)u#Ex7D-Mh_u!23L) z62brnMu-WO2{nO#{uA(pW*|{Ybi?*{%lf5Ed=XTJP$ubX{nx|6dK&wQi{yHm$K!9a zDYFh(5ubujR(>FZjzDq4!5B;tQ4t1_Qi*QShGg?RSpvTEn{u8z@=x-a^4qy!9!-%( z4xj+5WDTdI_)np3{%)x;PCKLNWy5?!mCvdlML1a-1YJJ&@4Q^zMeXJ9q3;J0f8&}k zbMr5r9lDRGB&z1bV;9YoR*TjUPXA8^hqa0*p5YeXmmv|&e~4lTOv^PdI3K+xXUa;c zajVR#9*}yG(ktLkKgw6meh3X`s!r}z!&KYK6)hmj@u7C+l@RUY2}?~$c9*T8QBUgt zpMj6;e5UGbBU;`@UE(^qC^L{Jom!LYJ{WPJb;wqn=o&MxvLd&DZ4K-OW@5gJWY!k2 z&0J2me&0wI%DXOnzr-8D`<2>J9;s@;L1B2*SLZ0wBJS?LfwoWhfh|%WsZP8rwu9Aq zS+n-iAgdv`k1;^qDb#%M``5vCIHz-TJ+#A!p!3buOEXPFE~A}x&{JeSxg=psYy-gt zaTa078uO;#A$gSR_krxq^tS5S5va|a-Gbfe(Kbhc307uJ)=?Ig3+DRIHq0IUw#$d# z3L9KmGt3)3VI6YN(gE?VUfasa*%>h@-7?*K-6vhatA(w0(CQzT`783k-rxKe%Nr)W zkFS!G{Tq64dg{w|%eK!d&#jMA4KX#Re=?t)&FVFr-7m)x77$*M>99Ihp-amY3lh58 zNh@sUN?7Am{cri#z8si>n0g6$V|#RN^8N>>kElD3l$fGFec}V_ejzVX`A~oYMZ|CCkGxCFiIsa~2 zZj^Kc9$U}uIAXhAcjI{dT2Oe8I~MB|n>D@d^15x=IK(&TGRSEZwQxT3Iun0OThtuy z$?M5!fSNi%uLAno1MMD9$qR&Lx9hpVRLUO4@uZ^ACj_@rXn#T z&kKYlMMUE*MGho^xgEOxGvqn);#z^9m4!>|F`);=SSsp8fjYEJ0iljyBBLd zj*Wl)CbhM)wmJY1zyJV1!T^B3@BjD|0PyAp0M0A{0MTy%0I_ST#i$|xKr5>vFQe=C z^CI83H(xK~ao}0xFtaTeQ5%vRjwfqhtc+VColc9P?gx@)8H%Q}Khl#&Mj}y^DaYLH z-4jkkV)+h3iEaZde3d;}P|>88DBBGF8T@W|KDqwr_9XFmv}yA$S$OQZ0iE-oy}Q^k zcY6J8>_qOwK}kv3uC(hK8WqKq_5b1jKm0H7f9fcO&=^mpUGjqVr2C0C5X2DnAI?cE z$t~%*;YrXi;E@m#Ad{Y0=w1l*(qc_;hUj$*jj9lT*q2>1YNK=P6!o)eTO=lHk=v#&KTH%#S}w_#nTb z`XP6r`>DkU;z)=iU@Tfk^Hs-daTO~_-MNiFxGcW~W%T$UR3QdrUAxg#U8 zG5AFB*nxpldT9HQxoU6G;^@E?;u8Q7GtCh^6})PO8y$ZGIratMqH0{@tVB@scpagO zsOw`Fm7jEySN`@Lp5~vr}TMOV&SYN{LG^A zt-%O<0dn+)kL7YM3ndiglS0`Nfwao zxLp!qA_0wri-ONOTkUUwvFzIJf_6zMaLyl;g{>wHy+atQmp64dwBpR2B(p-ojEgPk zfS?jc&x8LsZIZX$TOmN`ynQLhRL;BPDKGWqQ$hkOZpaE86%FtNA%IGZ0=I;K@jS5K zwB9h~yrdkW$^H^(kh(sJ5q9SnYQhKIJM#!4|JF~sS-cE$dV*PK`5uMAV}~?#`$tvl zZK-b!d?C8T+RYLbBOa39{k*GcQ+^8e{+g|W;DPz!Hxkw---`)W#`<=c$=Q_z#&iR* zZS6WN!HmVur;@~Fn$ck)P1TM6Kkwai(b&7pGgv`sGi*x1yob1@!6zJR8|44>X{QRl zqiw?9Shq^VET#kZTQQG{z=6kB->_$RhR02O@Y^xu_*}uS8AgrZjv;jg^zik~`6c4A zyQ0x>LuQ?48k%H*+MUb}5je9d_Zq`W2W(WG{4m`Ur#i0-l1erEp|$JH!|>B<*VG1X zV>B=kX!w0W4Ve>868(kTO)vX^3?r|CXG=Gsm~|Ady{lFC)I`S>%^;3vb31?UuWsb3 zjziAWCxlKS4*`G$0_UP%1y^`1I8>^d{4OPyh=S~$q%V?t--^El%e!cdIxXD|+Mt7% zvi)w7WUJ5s3O8=KcB&&<-SKr)d$0yJx?{p==z44wew`Xes$j#0TAdjOo5cu>`44!^ zBlQAafY)1&XE+H+WZauu`FoKK^p>|{1}1FK-;1H%x>Qn87)D$mXpJEk^EwgRFSHMf z8`ZS|1WjDDg~xc#m~Qwj)V*aR?RiC4TVRUS&x|cqy@MH7snR{|5)0n!ZT#e(3YE5l zBhlIZt3`f+yyuEnjb|Gyq9EN^>?PT)ip+LSQ0O4iQ3Z!<8K3D+cG>#3*p|gk8;?;4 z`ZDIG9iLP!1DRBB(VjxHpQ8!GxL1P7*wM>t&@Rz#r^4na4ww};ajEc~*vQcke@u4+ zIWnviZmZ+?^Xsuq4#ii?0b5eBYu~VaOqQ6**-yAtWa2Rg7q9*1^}5P6NR~F1e*7s% z&lD=Dz&4=LV_8mn-JcbCB?|7aWUW)u$C(t0<8)0n?*NN$9nSXadhAEz})@qG|5zLhjJ>i?*m1lzo3Uq1KU*ey!t_mQL`UyGz zm7|@xu*nf*BPCh3RHa{yIg`B9J)YwUvuTkk%dGlV+ZmR6vG6gq9 z^u~0>rFqdBZdG7>eOFuzeK;3$2WUJSPwsPG_0n9i419yvk~+`SZMb%ZB-Pm}nxrxa z%Ej@>7oUBO$|S;#*)b1NcSs8YSw}~RNT#~ z&)iXqL&`41jBPb9*kazTo%p$vsP`o?;q1)}{Ut$E9j{};SP#iCFVnAe5T-|=LS&*^ znE|)7G2dIB?1Y%Q;$fbFV8x-GPC~@kL8l$iWBkn};a3T84-}0^VF$)u*=he|Y{wNz z2=k6+@2d!}eGFMP!|+6=LgaLNp94@3bvHpD6y!Mas0KIP-K}kApqS0gmn#KT$9cM= z-o90PzI3VD%Z(*+b^f+ocoH<1+Rj zA*guz_(q8SuXllQOcSqnJ7qQ+mv=H6qqXiA6uzZl)&nyZF1Y1UKA3>xhgeVxE998v zlu9Ypa}~ngP4N+RO^obE{@v#VytX7sHcFA;zC(CAQtIVhxb|-BrC9C+#TXFi8>`n} zMg7uHDX&VS!P5HcGG#1 z{k<_m>_J&~AX0zd;LkY&ULCQVr>U~JN9|;V^za7Ix=ns6#Ma8UETz}5-grbLrl{FtGSSWMPV`& z*kGL!vE1zcwi0Wjwo~f()7XZ`+ta$sxXVEq@0_1rj8^*~=1hThfBH>z?2 z;dcw6o!J#P!h&mBF_ysK&~wE3&IA$Rn4&eHk*wh?J?=DYz?u{pTM?Bq@b6 z=}nsoccqWuOZ&m&(P6r!ZdM__sr$DdeW&X+E$xtnp8YoqWuJ={S~?Rg!jJ_!=+z%T zx9fIw_m%ISq>Y>hLRFl{uI^#Kq_|u)wi;D0jK@TyQq6%?T_8=w^d9w8-!wyPUZ=E@f2F&6gGjHl z?@2ZlAuM12-zn!HM$=$yhOjW%fAYNQVZT^*|L9M})D!W_?l#s82^Ytnn>OIDB*69r zxmmh?z&e*~gF2QltnP=$0=m2o6PDn@-C+Y~tDX}qJESO?=LGGBAfi9l$!ZbfYaW&m zF52o(;2(UrzrDSgvJkvXgzIUB3Q{1iC2-gb+Wa*V%JBPiM|EUoio|O<*Y%QC3K%WIBVS1<+s1zL+;yMQ5e&2_%6QZSamlly2 zR!ia;7@Ic=b(@SThQ?+@;XZS$VXAmYYVAMKrEE^Zl@n1`vl;?Si)9s`_3v?%XhpvA zd2#P_5B?5BP5gNVWVUNE~Nz{?ofPHj>%;Mo>jOrv^NFh z)!zCamxgidoI4TeGkKqX9S%l{?c;eNodL}gbXg5zZQ(YX=WiDq+ZejpFC2E;nRK!> z|JGtk%F#?RJu-_4Rx(s=-h3<6}27u1HfWG`{GkUX_@<1#%^l6108IO%&DI`5g|4%wKV*5tOZ{(dx-b^z+W zpv0W|efn?g73+xZBrfXmCj@toyvV9<@DH7NvuJvSfu&>XvN4xmHf6oP)TCkJ+Z85-KO8;D+b$=JoZiAfeXNGnf-a~MD!jJB7y)t-Y0*rBF z_ipPBex_~-=qC&ng0upmJ^k&*q4OIKQhh%hLe1zHxk8QW52tRJE5}RQj$W3}@X2(o z+A>m>y0XShE49RH>2|=zR1twLr$BQtUT)QGpb4jaG%r(zoj=m7b94)>^^4ZrKO#AtLcb(wCwuy`e(XVyPAP#T0Ij@2K?f~n_&94 zM!RZsb6%CfG&arYMdx2d_OOv?8i&9{h$@+NJjShNBa4anT@kS_7`}!3uNggEzqa^- zO&3Um_3mEiC%!cjd8;a-9aG)GpAXX7W-DWw7B4)k%n1sq?7@*$?d8CGO{MRazt2ZY zjlQ4v4=W4r$y7$Pl9gN#@X*)U=uqLuH7VUM^88SAV$;aI{$Lf2_@1O3pj-Yefq?z3 zVKUiYAmdl=Wekm6yodqvPj6eS^x@>ugUP46FD>ZPY;i9;_PbQchoSBQy>BhE|Z@b!~~iQ6$F z*p{+XVg|#uRa2@v()xi#T>8xmywR$$(Fnq}%+B@sQ&8`0%mZ^t=4|6pVJEW(sl0|H zvtw6-NWTgV-|Qk*YrXTU#6*6|cI-~Y&>j$dDvFdcCx80owq^EUGmgBJ{JmBz!{AA% z@a=||Va3r#{RS@9)e&b5WU%O&rI_6E&4MnbpX!s}B9zf)>x7*U_pE+rlFY{FmuNsU zXqZTbv((p8ee8DBFuYK3TrkBRV9Bpt;l`5zu5X0DiW{1K$jl1a%-HUrD{m)1(E3GE8I}0$nnz}yao3icvj5=f*?@xg({%q<6^D+C` z+XeP<)CQFi`E-#3NlGHZWZdjU7o%%4(ZaN+1xfdYjfmuW%{|C|4#}aFI;5PAs@#Iy zfkqBEtz;~*rk{>P9_9Ahn#_iD&N#P?tbT8+XAFi=hY0N_rFa(`;F7){(q_7UT!`^A zNI-t1?vYcmu;y^F)g+wBP4NgNaW!`S+jT666w`Y_D0g~>JrMRH>g)U z*xjKeg@-J8zA|Kh6a&8&<7ujYDh8Dp7n8?!*>gtHNnpM!Sl3{r=HHOhH#V;op`Xv& zC5IprCA$pN;>Nr>mvwndrvB;Xjq)nhlV8VoNDROTTHK+cSGslVE64#!r>b^8Mehge zc;@BPMkF=1#z3V9{&%bdy$=UjbkCl%WE!{&o>;Y&Nqwe`V-9Y& zD#IEK{U@;|hJHhpQ5B10XyOZrsKt8N^$fhi@LMmd4t}blie-Ec6x07|kQ~0pbTTgg zADm1B!2;!Z&5pTIgUKt#e^e75Fs4@t z?y~|@RBBre1Hn-d8cL9L(1-!1=rd)!O%;`%9ecAe>VtTTp;#`NO`w5t15R2p{ZgqX zd!3%gR@lUiDsvv3B+$A>2j}mXK*4J=y(kTBO@8R}XZ zu}Vgr{`<8wWJ;ICF7wUlvuUby>-(MpoPRw@c{D;&rrK@5G|uo?vwSMOF&COIHL=9# z6NQWd`g88lc)r9n8`H(w;Zq+Po$8*(`w3V&P&_WhL9~@)bZRk(Uw9b%`h^gh8J*Oc z2D)W+PS>OPskW7f)GZ9G`sDtiLvzDSX(DOGw1V|H-22TxIwkYIDmc;{ojD5&$-)#0z$R4fH z@{FQ+7n`p}i4&)k8_XwNwwAV|8Va6Gzmw2vT~{HB8FTzpmKWKKEXWo7Ol|TsUrys; zXl*ONc>ZrHwDK%&g8I;(3cV6!yq*G^@~OoDotMY9aoK8kqzyxyK#bNU)w^S z%vme=+_%(Z-fy8jd>+1irx%d4BfC((t##|GEi{V$8kf@@aeXNOC8|Ufl3bG+NR+)@ z9TR5N@0{%>PiH@0?lY~)#e+23 zrJ=}p4n*Y~Jhs|HLk>46*0hmrGcJ2X+b8p(ARpctU(r;r2dkZJb5>7RXBUOpo(Pi` zD5rFhDWbC6j0}m3W|^Nz#`;npgA2wS!(yG=?`)v4A6q_1p1R@2y%}YEl$O4UU1L}v zUu9bZY~P+KGiLev=kDOll?``Fi*A9;Jy)EVN;q64)9nhuhOg6yZF#0x8SKqgh_bHs zL%1d7!S+&R50)Bln?DnaLo}JODd@H-i@;URYXc$3d|V*Cy3k0GA^LF)=H(~fk2##w z6G6%~HcbXOjJ5nUrUlmVj5P&Wu327IycR~13~zcx>NBXLeZwJ9BxxoSjB3I>!w<^q zcxhvs>}}i9Z7cvZxWqO)Taq|n;9B;#o35wWKx47A60$rlEv}yVb1!x_IKEWzI72>z z(fq9ZJ@gWl@cM`&kVw0PQjVayId&MAhKaEXf3(?$isvC(p&;|0w`#02(+vbvHtd+v z(W!`Ko;IVX!{jFU=@P0P z$Dug5j)hbQJdmLzq(fymB9QcBedQbL=@iHs}F zh4KID{L<dNhT_N2iOKni1D3eIceC$+yHxh0w4 zF~FP;Xa5)_I)Zl#E)k+usdFz|bFFry&ea_^3{J5XPhE0K{&12M`=3ChO#}uln_`84 zjgu0QZjaM{{=N%yDe^7AnssJ36Z%*5KB_qMUKjQjzmGD!Mw!GS$36kGK=?E4$wDB} z>haIAW0No&pY$-?S=z%eb008|*U`jhkgK%pGz%ck&|57gSnOGY-ZH?rypOjJrdCx%qMM_O&!clV6KN9MAyd2R~+V_I+HjJ+?r&rh$& z%2AbH5T6USYlXFw{Ty&zzaTAqtK8%z4?EyFGO41By1!d9oAX6qz%C`wSe^3;=d;6C zZh6pA_FQBrFg*m-&1&;dxCIhYuY#A*Leo{-(p4IqCpzB9RBm)9P*O+8jzmjZkKO1a z>_f_4_MKo1$}zxl;q(i0ZMtMrnB#;%`sdE_uw-YIkb+}x4jZ-)#^0N2uE1Q#DYWJ= zl4FUfN!J&+mMC}jBtr2|NkV5SrZV}dwG78_m)g7DSBl6WSw|6N!ax0=x{i-nd}^mM zHW2+R@@4)FtnM@>|N3KBITBI}72d`j?BFq*9fWw``UIe@-g;UTx(8EHX45|M8Vv>( z#^1Yp#k8ZTj9g8NaQ!bUzuET7bg&@C2~=#KyJn-NQhT#F)BEFpzGs=?YlJM1Rfzi^ z&&jQ*@_z-!4KJTPDk^o8vDlJU-mC`Sv-kF@Fr~ZCYQ^>b_}1_HXLGgMms{(44zWyZ z_48TT509d*2uMN997|=pf*#NmrW)GtL>j7o!UkCd_6xPaDFA=i9B|9Dv?xmCL32P1{2;&%Y^YsZm{$Z zBK{Mrr$#Ohju;Iq?ar`uv&b)Z#;q^10Mp8t%?(vaghxwgp@Cy$5V+)+;(oOyf{<*A zo^R;*F(Pn1&_fS0hy^vC>~Pzja>qv!6M9w$b0-zO2kzMI-unmfqYap}{Tff?W2}X`GZ4{P#oCx*< zK;x>%X)!%yU!d?)0+Y=eY~Ce)8-Jf=`ttsQPangOiXM#P`L|Ks#an11nG6yp+A$np z*d-*9v75hb2s^T#vrX}n-!3iWeiOcBcv2P0iBe>cyF#0Pe8k+`@d_5I&$@bGw{c$v zRBFGQG#u@r<+R_n{NDoRf( zEa9t-y4XV`T|V?pwQ>m4lP-Fkeu;%b7$emSv{2=$mhOJVQsvf5;9BAH-JWKJAFOeDBc9n@XrOr|U8KZUyFz zS%sLMQ-J8aFndV-{3s^{yzonXa0)^S?mAzc+-3$m`rq*oG&9HDv*+Y}*s@xUJrYAi`SM4J?A$Xb1g z#9yR{sw1sHVjk3|6|lP^>KJeF$3sElhYE@gDRg4>4?p(@qYNKkKBNZv=~#j4=l^lb)-ziC)ZH$`SS8Lcx%dEP%iny{q~;w zaWK-S#trSPgvUv{p)MBB0@Dvaqn6QkJqfD>Gnln`!oS#pP)0Ew9*52HxzGroB zO~<)-d+Upg-+t_LP9jSNp_Vwhu(+{d(w-8y%htKiJT6JMZftKbRX~MrpV_W=>tu%kxiOswhXOAp5?W45cgJ(eMHcwM+~T||!VLEC;CJ(I zLX_x%vTv6)ASZ}!<&Op3yokR1?uJhHp??lpXYU~&mM}Z_OsmooarLNse0->|nH_Qe z*{Lanbd@aq21;21)*Nv(J+aTQp3wa>aOGs;W#yqz%2}4NX(0~%-c#U0U+E?(abB`L z50~|0eVc-sLOEwi)O#6-`OW(b=u;&IQtb*ZwBPJZENBUdR!ZKEsDFC&oUl;%*x}{0 z_!yM3_X>DFFg-^EawT`33bZZqDX^{5N>(U2H^^Fk{eJ*G4j@bb From 4bb9a3a6af602f6a26ab88104935ba862868f7c1 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Tue, 28 Feb 2017 14:34:01 +0800 Subject: [PATCH 283/407] kubeadm reference -- / set up/manage mean set up or manage ? it's better use or? --- docs/admin/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 24f39217f0..2145b75310 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -31,7 +31,7 @@ server, as well as an additional kubeconfig file for administration. controller manager and scheduler, and placing them in `/etc/kubernetes/manifests`. The kubelet watches this directory for static resources to create on startup. These are the core components of Kubernetes, and -once they are up and running we can use `kubectl` to set up/manage any +once they are up and running we can use `kubectl` to set up or manage any additional components. 1. kubeadm installs any add-on components, such as DNS or discovery, via the API From 8ab16cb669aa6f7e1f4d480d05e15eac24af10e6 Mon Sep 17 00:00:00 2001 From: steveperry-53 Date: Tue, 28 Feb 2017 10:52:54 -0800 Subject: [PATCH 284/407] Prototype for deprecating User Guide topic. --- _data/guides.yml | 1 - .../run-stateless-application-deployment.md | 3 +++ docs/user-guide/simple-nginx.md | 7 ------- robots.txt | 1 + 4 files changed, 4 insertions(+), 8 deletions(-) delete mode 100644 docs/user-guide/simple-nginx.md diff --git a/_data/guides.yml b/_data/guides.yml index 30ee93078c..df3ebad90a 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -56,7 +56,6 @@ toc: - title: Containers and Pods section: - - docs/user-guide/simple-nginx.md - docs/user-guide/pods/single-container.md - docs/user-guide/pods/multi-container.md - docs/user-guide/pods/init-container.md diff --git a/docs/tutorials/stateless-application/run-stateless-application-deployment.md b/docs/tutorials/stateless-application/run-stateless-application-deployment.md index 8e3d725752..2b9793bfa4 100644 --- a/docs/tutorials/stateless-application/run-stateless-application-deployment.md +++ b/docs/tutorials/stateless-application/run-stateless-application-deployment.md @@ -1,5 +1,8 @@ --- title: Running a Stateless Application Using a Deployment +redirect_from: +- "/docs/user-guide/simple-nginx/" +- "/docs/user-guide/simple-nginx.html" --- {% capture overview %} diff --git a/docs/user-guide/simple-nginx.md b/docs/user-guide/simple-nginx.md deleted file mode 100644 index 2a6b11133f..0000000000 --- a/docs/user-guide/simple-nginx.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Running Your First Containers ---- - -{% include user-guide-content-moved.md %} - -[Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) diff --git a/robots.txt b/robots.txt index 9bb39d8dbd..10fff656d2 100644 --- a/robots.txt +++ b/robots.txt @@ -5,5 +5,6 @@ Disallow: /v1.0/ Disallow: /v1.1/ Disallow: /404/ Disallow: 404.html +Disallow: /docs/user-guide/docs/user-guide/simple-nginx/ SITEMAP: http://kubernetes.io/sitemap.xml From 04f2e50caa59ebcfb5db461f65b6733bbf8f2a9e Mon Sep 17 00:00:00 2001 From: Philipp T Date: Tue, 28 Feb 2017 21:53:38 +0100 Subject: [PATCH 285/407] missing word The return of the OCD. --- docs/concepts/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 8e738e809e..39e44fbc97 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -8,7 +8,7 @@ The Concepts section helps you learn about the parts of the Kubernetes system an To work with Kubernetes, you use *Kubernetes API objects* to describe your cluster's *desired state*: what applications or other workloads you want to run, what container images they use, the number of replicas, what network and disk resources you want to make available, and more. You set your desired state by creating objects using the Kubernetes API, typically via the command-line interface, `kubectl`. You can also use the Kubernetes API directly to interact with the cluster and set or modify your desired state. -Once you've set your desired state, the *Kubernetes Control Plane* works to make the cluster's current state match the desired state. To do so, Kuberentes performs a variety of tasks automatically--such as starting or restarting containers, scaling the number of replicas of a given application, and more. The Kubernetes Control Plane consists of a collection processes running on your cluster: +Once you've set your desired state, the *Kubernetes Control Plane* works to make the cluster's current state match the desired state. To do so, Kuberentes performs a variety of tasks automatically--such as starting or restarting containers, scaling the number of replicas of a given application, and more. The Kubernetes Control Plane consists of a collection of processes running on your cluster: * The **Kubernetes Master** is a collection of four processes that run on a single node in your cluster, which is designated as the master node. * Each individual non-master node in your cluster runs two processes: From 1564ae52b76ea4dd090dfe5ac9b6c61bf9cb41ec Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 28 Feb 2017 19:05:31 -0800 Subject: [PATCH 286/407] Move Guide topic: Using Environment Variables. (#2645) --- docs/user-guide/environment-guide/index.md | 95 +--------------------- 1 file changed, 2 insertions(+), 93 deletions(-) diff --git a/docs/user-guide/environment-guide/index.md b/docs/user-guide/environment-guide/index.md index cbdd806ecd..9db0f41f9b 100644 --- a/docs/user-guide/environment-guide/index.md +++ b/docs/user-guide/environment-guide/index.md @@ -4,97 +4,6 @@ assignees: title: Using Environment Variables --- -This example demonstrates running pods, replication controllers, and -services. It shows two types of pods: frontend and backend, with -services on top of both. Accessing the frontend pod will return -environment information about itself, and a backend pod that it has -accessed through the service. The goal is to illuminate the -environment metadata available to running containers inside the -Kubernetes cluster. The documentation for the Kubernetes environment -is [here](/docs/user-guide/container-environment). +{% include user-guide-content-moved.md %} -![Diagram](/images/docs/diagram.png) - -## Prerequisites - -This example assumes that you have a Kubernetes cluster installed and -running, and that you have installed the `kubectl` command line tool -somewhere in your path. Please see the [getting -started](/docs/getting-started-guides/) for installation instructions -for your platform. - -## Optional: Build your own containers - -These are the configuration files for the containers: - -* [backend-rc.yaml](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/user-guide/environment-guide/backend-rc.yaml) -* [backend-srv.yaml](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/user-guide/environment-guide/backend-srv.yaml) -* [show-rc.yaml](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/user-guide/environment-guide/show-rc.yaml) -* [show-srv.yaml](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/user-guide/environment-guide/show-srv.yaml) - -## Get everything running - -```shell -kubectl create -f ./backend-rc.yaml -kubectl create -f ./backend-srv.yaml -kubectl create -f ./show-rc.yaml -kubectl create -f ./show-srv.yaml -``` - -## Query the service - -Use `kubectl describe service show-srv` to determine the public IP of -your service. - -> Note: If your platform does not support external load balancers, - you'll need to open the proper port and direct traffic to the - internal IP shown for the frontend service with the above command - -Run `curl :80` to query the service. You should get -something like this back: - -```shell -Pod Name: show-rc-xxu6i -Pod Namespace: default -USER_VAR: important information - -Kubernetes environment variables -BACKEND_SRV_SERVICE_HOST = 10.147.252.185 -BACKEND_SRV_SERVICE_PORT = 5000 -KUBERNETES_RO_SERVICE_HOST = 10.147.240.1 -KUBERNETES_RO_SERVICE_PORT = 80 -KUBERNETES_SERVICE_HOST = 10.147.240.2 -KUBERNETES_SERVICE_PORT = 443 -KUBE_DNS_SERVICE_HOST = 10.147.240.10 -KUBE_DNS_SERVICE_PORT = 53 - -Found backend ip: 10.147.252.185 port: 5000 -Response from backend -Backend Container -Backend Pod Name: backend-rc-6qiya -Backend Namespace: default -``` - -First the frontend pod's information is printed. The pod name and -[namespace](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/namespaces.md) are retrieved from the -[Downward API](/docs/user-guide/downward-api). Next, `USER_VAR` is the name of -an environment variable set in the [pod -definition](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/user-guide/environment-guide/show-rc.yaml). Then, the dynamic Kubernetes environment -variables are scanned and printed. These are used to find the backend -service, named `backend-srv`. Finally, the frontend pod queries the -backend service and prints the information returned. Again the backend -pod returns its own pod name and namespace. - -Try running the `curl` command a few times, and notice what -changes. Ex: `watch -n 1 curl -s ` Firstly, the frontend service -is directing your request to different frontend pods each time. The -frontend pods are always contacting the backend through the backend -service. This results in a different backend pod servicing each -request as well. - -## Cleanup - -```shell -kubectl delete rc,service -l type=show-type -kubectl delete rc,service -l type=backend-type -``` +[Exposing Pod Information to Containers Through Environment Variables](/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/) From b9ca35076b5afd4fb2b282167340d4c8c17908e4 Mon Sep 17 00:00:00 2001 From: tanshanshan Date: Thu, 2 Mar 2017 00:26:42 +0800 Subject: [PATCH 287/407] fix typo (#2656) fix typo --- docs/admin/namespaces/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/admin/namespaces/index.md b/docs/admin/namespaces/index.md index 887f14cd91..8da2107960 100644 --- a/docs/admin/namespaces/index.md +++ b/docs/admin/namespaces/index.md @@ -44,9 +44,9 @@ You can list the current namespaces in a cluster using: ```shell $ kubectl get namespaces -NAME LABELS STATUS -default Active -kube-system Active +NAME STATUS AGE +default Active 11d +kube-system Active 11d ``` Kubernetes starts with two initial namespaces: From a2fbc82bd08451951929ea19f792dc86530c4063 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 1 Mar 2017 08:55:15 -0800 Subject: [PATCH 288/407] Move Guide topic: Using Port Forwarding. (#2661) --- ...connecting-to-applications-port-forward.md | 44 +------------------ 1 file changed, 2 insertions(+), 42 deletions(-) diff --git a/docs/user-guide/connecting-to-applications-port-forward.md b/docs/user-guide/connecting-to-applications-port-forward.md index 4855994736..afa3559eba 100644 --- a/docs/user-guide/connecting-to-applications-port-forward.md +++ b/docs/user-guide/connecting-to-applications-port-forward.md @@ -5,46 +5,6 @@ assignees: title: Connect with Port Forwarding --- -kubectl port-forward forwards connections to a local port to a port on a pod. Its man page is available [here](/docs/user-guide/kubectl/kubectl_port-forward). Compared to [kubectl proxy](/docs/user-guide/accessing-the-cluster/#using-kubectl-proxy), `kubectl port-forward` is more generic as it can forward TCP traffic while `kubectl proxy` can only forward HTTP traffic. This guide demonstrates how to use `kubectl port-forward` to connect to a Redis database, which may be useful for database debugging. +{% include user-guide-content-moved.md %} -## Creating a Redis master - -```shell -$ kubectl create -f examples/redis/redis-master.yaml -pods/redis-master -``` - -wait until the Redis master pod is Running and Ready, - -```shell -$ kubectl get pods -NAME READY STATUS RESTARTS AGE -redis-master 2/2 Running 0 41s -``` - -## Connecting to the Redis master[a] - -The Redis master is listening on port 6379, to verify this, - -```shell{% raw %} -$ kubectl get pods redis-master --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}' -6379{% endraw %} -``` - -then we forward the port 6379 on the local workstation to the port 6379 of pod redis-master, - -```shell -$ kubectl port-forward redis-master 6379:6379 -I0710 14:43:38.274550 3655 portforward.go:225] Forwarding from 127.0.0.1:6379 -> 6379 -I0710 14:43:38.274797 3655 portforward.go:225] Forwarding from [::1]:6379 -> 6379 -``` - -To verify the connection is successful, we run a redis-cli on the local workstation, - -```shell -$ redis-cli -127.0.0.1:6379> ping -PONG -``` - -Now one can debug the database from the local workstation. +[Using Port Forwarding to Access Applications in a Cluster](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) From a642102dc8b8e83175f2918613041ca424a2c09d Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 1 Mar 2017 09:20:52 -0800 Subject: [PATCH 289/407] Move Guide topic: Bootstrapping Pet Sets. (#2662) * Move Guide topic: Bootstrapping Pet Sets. * Add TOC entry. --- _data/concepts.yml | 1 + docs/concepts/abstractions/controllers/petsets.md | 15 +++++++++++++++ docs/user-guide/petset/bootstrapping/index.md | 6 ++++-- 3 files changed, 20 insertions(+), 2 deletions(-) create mode 100644 docs/concepts/abstractions/controllers/petsets.md diff --git a/_data/concepts.yml b/_data/concepts.yml index bd187c6cee..440f9dbf2b 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -20,6 +20,7 @@ toc: - title: Controllers section: - docs/concepts/abstractions/controllers/statefulsets.md + - docs/concepts/abstractions/controllers/petsets.md - docs/concepts/abstractions/controllers/garbage-collection.md - title: Object Metadata diff --git a/docs/concepts/abstractions/controllers/petsets.md b/docs/concepts/abstractions/controllers/petsets.md new file mode 100644 index 0000000000..760e7e882a --- /dev/null +++ b/docs/concepts/abstractions/controllers/petsets.md @@ -0,0 +1,15 @@ +--- +assignees: +- bprashanth +- enisoc +- erictune +- foxish +- janetkuo +- kow3ns +- smarterclayton +title: PetSets +--- + +__Warning:__ Starting in Kubernetes version 1.5, PetSet has been renamed to [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets). To use (or continue to use) PetSet in Kubernetes 1.5, you _must_ [migrate](/docs/tasks/manage-stateful-set/upgrade-pet-set-to-stateful-set/) your existing PetSets to StatefulSets. For information on working with StatefulSet, see the tutorial on [how to run replicated stateful applications](/docs/tutorials/stateful-application/run-replicated-stateful-application). + +__This document has been deprecated__. diff --git a/docs/user-guide/petset/bootstrapping/index.md b/docs/user-guide/petset/bootstrapping/index.md index de3ccb4bd8..1988cba12a 100644 --- a/docs/user-guide/petset/bootstrapping/index.md +++ b/docs/user-guide/petset/bootstrapping/index.md @@ -10,7 +10,9 @@ assignees: title: Bootstrapping Pet Sets --- -__Warning:__ Starting in Kubernetes version 1.5, PetSet has been renamed to [StatefulSet](/docs/concepts/abstractions/controllers/statefulsets). To use (or continue to use) PetSet in Kubernetes 1.5, you _must_ [migrate](/docs/tasks/manage-stateful-set/upgrade-pet-set-to-stateful-set/) your existing PetSets to StatefulSets. For information on working with StatefulSet, see the tutorial on [how to run replicated stateful applications](/docs/tutorials/stateful-application/run-replicated-stateful-application). +{% include user-guide-content-moved.md %} + +[PetSets](/docs/concepts/abstractions/controllers/petsets/) + -__This document has been deprecated__. From 6b675d9b69e1a5f6f36307c08f818f63516805d8 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 1 Mar 2017 09:42:05 -0800 Subject: [PATCH 290/407] Move Guide topic: Connect with Proxies. (#2663) * Move Guide topic: Connect with Proxies. * Fix link. --- .../connecting-to-applications-proxy.md | 26 ++----------------- 1 file changed, 2 insertions(+), 24 deletions(-) diff --git a/docs/user-guide/connecting-to-applications-proxy.md b/docs/user-guide/connecting-to-applications-proxy.md index 559b06da9f..a95c2b06a2 100644 --- a/docs/user-guide/connecting-to-applications-proxy.md +++ b/docs/user-guide/connecting-to-applications-proxy.md @@ -5,28 +5,6 @@ assignees: title: Connect with Proxies --- -You have seen the [basics](/docs/user-guide/accessing-the-cluster) about `kubectl proxy` and `apiserver proxy`. This guide shows how to use them together to access a service([kube-ui](/docs/user-guide/ui)) running on the Kubernetes cluster from your workstation. +{% include user-guide-content-moved.md %} - -## Getting the apiserver proxy URL of kube-ui - -kube-ui is deployed as a cluster add-on. To find its apiserver proxy URL, - -```shell -$ kubectl cluster-info | grep "KubeUI" -KubeUI is running at https://173.255.119.104/api/v1/proxy/namespaces/kube-system/services/kube-ui -``` - -if this command does not find the URL, try the steps [here](/docs/user-guide/ui/#accessing-the-ui). - - -## Connecting to the kube-ui service from your local workstation - -The above proxy URL is an access to the kube-ui service provided by the apiserver. To access it, you still need to authenticate to the apiserver. `kubectl proxy` can handle the authentication. - -```shell -$ kubectl proxy --port=8001 -Starting to serve on localhost:8001 -``` - -Now you can access the kube-ui service on your local workstation at [http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kube-ui](http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kube-ui) \ No newline at end of file +[Using an HTTP Proxy to Access the Kubernetes API](/docs/tasks/access-kubernetes-api/http-proxy-access-api/) From 06416ac6e8f4ba2c0ddbf36582b4d03e1de88e33 Mon Sep 17 00:00:00 2001 From: xilabao Date: Tue, 28 Feb 2017 13:55:54 +0800 Subject: [PATCH 291/407] add DefaultTolerationSeconds admission controller --- docs/admin/admission-controllers.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/admin/admission-controllers.md b/docs/admin/admission-controllers.md index 089dce2605..29429c5b2c 100644 --- a/docs/admin/admission-controllers.md +++ b/docs/admin/admission-controllers.md @@ -252,6 +252,11 @@ This plugin ignores any `PersistentVolumeClaim` updates, it acts only on creatio See [persistent volume](/docs/user-guide/persistent-volumes) documentation about persistent volume claims and storage classes and how to mark a storage class as default. +### DefaultTolerationSeconds + +This plug-in sets the default forgiveness toleration for pods, which have no forgiveness tolerations, to tolerate +the taints `notready:NoExecute` and `unreachable:NoExecute` for 5 minutes. + ## Is there a recommended set of plug-ins to use? Yes. From c3e5ad74c51a3e43d14ff4de600f253b5a353a49 Mon Sep 17 00:00:00 2001 From: Rob Szumski Date: Thu, 23 Feb 2017 17:00:25 -0800 Subject: [PATCH 292/407] getting-started-guides: add CoreOS Tectonic --- docs/getting-started-guides/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/getting-started-guides/index.md b/docs/getting-started-guides/index.md index 4ed17d4f47..2d3d18cd57 100644 --- a/docs/getting-started-guides/index.md +++ b/docs/getting-started-guides/index.md @@ -61,6 +61,7 @@ few commands, and have active community support. - [GCE](/docs/getting-started-guides/gce) - [AWS](/docs/getting-started-guides/aws) - [Azure](/docs/getting-started-guides/azure) +- [Tectonic by CoreOS](https://coreos.com/tectonic) - [CenturyLink Cloud](/docs/getting-started-guides/clc) - [IBM SoftLayer](https://github.com/patrocinio/kubernetes-softlayer) - [Stackpoint.io](/docs/getting-started-guides/stackpoint/) From 0d7bcf5e6e31956bc05d2a0a0e8cee57d89039c0 Mon Sep 17 00:00:00 2001 From: Manjunath A Kumatagi Date: Tue, 28 Feb 2017 22:49:37 -0500 Subject: [PATCH 293/407] Correct the certificate name --- docs/admin/authentication.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/authentication.md b/docs/admin/authentication.md index 7a7d8f1bc3..63335e9d77 100644 --- a/docs/admin/authentication.md +++ b/docs/admin/authentication.md @@ -541,8 +541,8 @@ Finally, add the following parameters into API server start parameters: 1. Generate server certificate and key. (build-server-full [filename]: Generate a keypair and sign locally for a client or server) - ./easyrsa --subject-alt-name="IP:${MASTER_IP}" build-server-full kubernetes-master nopass -1. Copy `pki/ca.crt`, `pki/issued/kubernetes-master.crt`, and `pki/private/kubernetes-master.key` to your directory. + ./easyrsa --subject-alt-name="IP:${MASTER_IP}" build-server-full server nopass +1. Copy `pki/ca.crt`, `pki/issued/server.crt`, and `pki/private/server.key` to your directory. 1. Fill in and add the following parameters into the API server start parameters: --client-ca-file=/yourdirectory/ca.crt From 6dc694a16166237b6b74aa392c4dafc8f278514d Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 11:43:24 +0200 Subject: [PATCH 294/407] Update index.md --- docs/getting-started-guides/ubuntu/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/index.md b/docs/getting-started-guides/ubuntu/index.md index d54b94a88c..5635c1d553 100644 --- a/docs/getting-started-guides/ubuntu/index.md +++ b/docs/getting-started-guides/ubuntu/index.md @@ -15,7 +15,7 @@ Supports AWS, GCE, Azure, Joyent, OpenStack, Bare Metal and local workstation de ### Quick Start -[conjure-up](http://conjure-up.io/) provides quick wasy to deploy Kubernetes on multiple clouds and bare metal. It provides a user-friendly UI that prompts you for cloud credentials and configuration options: +[conjure-up](http://conjure-up.io/) provides a quick way to deploy Kubernetes on multiple clouds and bare metal. It provides a user-friendly UI that prompts you for cloud credentials and configuration options: Available for Ubuntu 16.04 and newer: From a15d5c1223e9538211213b0b97d8531e3e2d7b29 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 12:21:18 +0200 Subject: [PATCH 295/407] Update installation.md --- docs/getting-started-guides/ubuntu/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/installation.md b/docs/getting-started-guides/ubuntu/installation.md index 9e3bdd8b2b..41499b8111 100644 --- a/docs/getting-started-guides/ubuntu/installation.md +++ b/docs/getting-started-guides/ubuntu/installation.md @@ -9,7 +9,7 @@ title: Setting up Kubernetes with Juju Ubuntu 16.04 introduced the [Canonical Distribution of Kubernetes](https://www.ubuntu.com/cloud/kubernetes), a pure upstream distribution of Kubernetes designed for production usage. This page shows you how to deploy a cluster. {% endcapture %} -Out of the box it comes with the following components on 9 machines: +Out of the box Juju Kubernetes comes with the following components on 9 machines: - Kubernetes (automated deployment, operations, and scaling) - Three node Kubernetes cluster with one master and two worker nodes. From 5e5d91e3c2fe9d454ed92a46e227fee87007d2c5 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 12:32:55 +0200 Subject: [PATCH 296/407] Update validation.md --- docs/getting-started-guides/ubuntu/validation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/validation.md b/docs/getting-started-guides/ubuntu/validation.md index 51b265a722..e881ab9fa6 100644 --- a/docs/getting-started-guides/ubuntu/validation.md +++ b/docs/getting-started-guides/ubuntu/validation.md @@ -31,7 +31,7 @@ users do, when unit and integration tests are insufficient. To deploy the end-to-end test suite, you need to relate the `kubernetes-e2e` charm to your existing kubernetes-master nodes and easyrsa: ``` -juju deploy kubernetes-e2e +juju deploy cs:~containers/kubernetes-e2e juju add-relation kubernetes-e2e kubernetes-master juju add-relation kubernetes-e2e easyrsa ``` @@ -83,7 +83,7 @@ As an example, you can run a more limited set of tests for rapid validation of a deployed cluster. The following example will skip the `Flaky`, `Slow`, and `Feature` labeled tests: - juju run-action kubernetes-e2e/0 skip='\[(Flaky|Slow|Feature:.*)\]' + juju run-action kubernetes-e2e/0 test skip='\[(Flaky|Slow|Feature:.*)\]' > Note: the escaping of the regex due to how bash handles brackets. @@ -120,13 +120,13 @@ Output: Action queued with id: 4ceed33a-d96d-465a-8f31-20d63442e51b -Copy output to your local machine +Copy output to your local machine: juju scp kubernetes-e2e/0:4ceed33a-d96d-465a-8f31-20d63442e51b.log . ##### Action result output -Or you can just show the output inline:: +Or you can just show the output inline: juju run-action kubernetes-e2e/0 test From 923aeb8bda6984a03aa301e9e313e2dd6f43af61 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 12:45:43 +0200 Subject: [PATCH 297/407] Update backups.md --- docs/getting-started-guides/ubuntu/backups.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/backups.md b/docs/getting-started-guides/ubuntu/backups.md index b839cafee4..88bb83aabe 100644 --- a/docs/getting-started-guides/ubuntu/backups.md +++ b/docs/getting-started-guides/ubuntu/backups.md @@ -10,6 +10,8 @@ This page shows you how to backup and restore data from the different deployed s This page assumes you have a working Juju deployed cluster. {% endcapture %} +{% capture steps %} + ## Exporting cluster data Exporting of cluster data is not supported at this time. @@ -18,7 +20,6 @@ Exporting of cluster data is not supported at this time. Importing of cluster data is not supported at this time. -{% capture steps %} ## Exporting etcd data Migrating etcd is a fairly easy task. From dbedb28b51eeb81bb30057f99fdc03e801005043 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 12:57:29 +0200 Subject: [PATCH 298/407] Update backups.md --- docs/getting-started-guides/ubuntu/backups.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/backups.md b/docs/getting-started-guides/ubuntu/backups.md index 88bb83aabe..8617c557ad 100644 --- a/docs/getting-started-guides/ubuntu/backups.md +++ b/docs/getting-started-guides/ubuntu/backups.md @@ -11,7 +11,6 @@ This page assumes you have a working Juju deployed cluster. {% endcapture %} {% capture steps %} - ## Exporting cluster data Exporting of cluster data is not supported at this time. From f2e481abd8874cf6863e692175856dfb0bf64866 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 13:37:30 +0200 Subject: [PATCH 299/407] Spell fixes --- docs/getting-started-guides/ubuntu/calico.md | 2 +- docs/getting-started-guides/ubuntu/decommissioning.md | 2 +- docs/getting-started-guides/ubuntu/glossary.md | 4 ++-- docs/getting-started-guides/ubuntu/local.md | 2 +- docs/getting-started-guides/ubuntu/monitoring.md | 6 +++--- docs/getting-started-guides/ubuntu/networking.md | 2 +- docs/getting-started-guides/ubuntu/storage.md | 4 ++-- docs/getting-started-guides/ubuntu/troubleshooting.md | 4 ++-- 8 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/calico.md b/docs/getting-started-guides/ubuntu/calico.md index 631d2cdaac..ccd66093a4 100644 --- a/docs/getting-started-guides/ubuntu/calico.md +++ b/docs/getting-started-guides/ubuntu/calico.md @@ -80,7 +80,7 @@ The master requires the root CA public key, `ca.pem`; the apiserver certificate, Calico needs its own etcd cluster to store its state. In this guide we install a single-node cluster on the master server. -> Note: In a production deployment we recommend running a distributed etcd cluster for redundancy. In this guide, we use a single etcd for simplicitly. +> Note: In a production deployment we recommend running a distributed etcd cluster for redundancy. In this guide, we use a single etcd for simplicity. 1. Download the template manifest file: diff --git a/docs/getting-started-guides/ubuntu/decommissioning.md b/docs/getting-started-guides/ubuntu/decommissioning.md index 0da845236e..2d1b42c7e4 100644 --- a/docs/getting-started-guides/ubuntu/decommissioning.md +++ b/docs/getting-started-guides/ubuntu/decommissioning.md @@ -13,7 +13,7 @@ This page assumes you have a working Juju deployed cluster. {% endcapture %} {% capture steps %} -It is recommended to deploy individual Kubernetes clusters in their own models, so that there is a clean seperation between environments. To remove a cluster first find out which model it's in with `juju list-models`. The controller reserves an `admin` model for itself. If you have chosen to not name your model it might show up as `default`. +It is recommended to deploy individual Kubernetes clusters in their own models, so that there is a clean separation between environments. To remove a cluster first find out which model it's in with `juju list-models`. The controller reserves an `admin` model for itself. If you have chosen to not name your model it might show up as `default`. ``` $ juju list-models diff --git a/docs/getting-started-guides/ubuntu/glossary.md b/docs/getting-started-guides/ubuntu/glossary.md index 1c8b2ec99b..2afe083a82 100644 --- a/docs/getting-started-guides/ubuntu/glossary.md +++ b/docs/getting-started-guides/ubuntu/glossary.md @@ -16,7 +16,7 @@ This page assumes you have a working Juju deployed cluster. controller - The management node of a cloud environment. Typically you have one controller per cloud region, or more in HA environments. The controller is responsible for managing all subsequent models in a given environment. It contains the Juju API server and its underlying database. -model - A collection of charms and their relationships that define a deployment. This includes machines and units. A controller can host multiple models. It is recommended to seperate Kubernetes clusters into individual models for management and isolation reasons. +model - A collection of charms and their relationships that define a deployment. This includes machines and units. A controller can host multiple models. It is recommended to separate Kubernetes clusters into individual models for management and isolation reasons. charm - The definition of a service, including its metadata, dependencies with other services, required packages, and application management logic. It contains all the operational knowledge of deploying a Kubernetes cluster. Included charm examples are `kubernetes-core`, `easy-rsa`, `kibana`, and `etcd`. @@ -25,4 +25,4 @@ unit - A given instance of a service. These may or may not use up a whole machin machine - A physical node, these can either be bare metal nodes, or virtual machines provided by a cloud. {% endcapture %} -{% include templates/task.md %} \ No newline at end of file +{% include templates/task.md %} diff --git a/docs/getting-started-guides/ubuntu/local.md b/docs/getting-started-guides/ubuntu/local.md index 854a34fa5d..7285b5915a 100644 --- a/docs/getting-started-guides/ubuntu/local.md +++ b/docs/getting-started-guides/ubuntu/local.md @@ -49,7 +49,7 @@ Now you're ready to install conjure-up and deploy Kubernetes. ``` -Note: During this set up phase cojure-up will ask you to "Setup an ipv6 subnet" with LXD, ensure you answer NO. ipv6 with Juju/LXD is currently unsupported. +Note: During this set up phase conjure-up will ask you to "Setup an ipv6 subnet" with LXD, ensure you answer NO. ipv6 with Juju/LXD is currently unsupported. ### Walkthrough diff --git a/docs/getting-started-guides/ubuntu/monitoring.md b/docs/getting-started-guides/ubuntu/monitoring.md index ee6b102f43..7ac97ea4e9 100644 --- a/docs/getting-started-guides/ubuntu/monitoring.md +++ b/docs/getting-started-guides/ubuntu/monitoring.md @@ -28,7 +28,7 @@ Configure Datadog with your api-key, found in the [Datadog dashboard](). Replace juju configure datadog api-key=XXXX ``` -Finally, attach `datadog` to all applications you wish to montior. For example, kubernetes-master, kubernetes-worker, and etcd: +Finally, attach `datadog` to all applications you wish to monitor. For example, kubernetes-master, kubernetes-worker, and etcd: ``` juju add-relation datadog kubernetes-worker @@ -74,7 +74,7 @@ juju add-relation kubernetes-worker filebeat ### Existing ElasticSearch cluster -In the event an ElasticSearch cluster already exists, the following can be used to connect and leverage it instead of creating a new, seprate, cluster. First deploy the two beats, filebeat and topbeat +In the event an ElasticSearch cluster already exists, the following can be used to connect and leverage it instead of creating a new, separate, cluster. First deploy the two beats, filebeat and topbeat ``` juju deploy filebeat @@ -122,7 +122,7 @@ juju add-relation nrpe kubeapi-load-balancer ### Existing install of Nagios -If you already have an exisiting Nagios installation, the `nrpe-external-master` charm can be used instead. This will allow you to supply configuration options that map your exisiting external Nagios installation to NRPE. Replace `255.255.255.255` with the IP address of the nagios instance. +If you already have an existing Nagios installation, the `nrpe-external-master` charm can be used instead. This will allow you to supply configuration options that map your existing external Nagios installation to NRPE. Replace `255.255.255.255` with the IP address of the nagios instance. ``` juju deploy nrpe-external-master diff --git a/docs/getting-started-guides/ubuntu/networking.md b/docs/getting-started-guides/ubuntu/networking.md index 54ca15c6ad..bb16491e2b 100644 --- a/docs/getting-started-guides/ubuntu/networking.md +++ b/docs/getting-started-guides/ubuntu/networking.md @@ -45,7 +45,7 @@ $ route | grep default | head -n 1 | awk {'print $8'} establishing networking setup with etcd. Ensure this network range is not active on layers 2/3 you're deploying to, as it will cause collisions and odd behavior if care is not taken when selecting a good CIDR range to assign to flannel. It's -also good practice to ensure you alot yourself a large enough IP range to support +also good practice to ensure you allot yourself a large enough IP range to support how large your cluster will potentially scale. Class A IP ranges with /24 are a good option. {% endcapture %} diff --git a/docs/getting-started-guides/ubuntu/storage.md b/docs/getting-started-guides/ubuntu/storage.md index 104dd588a4..9d193fa81b 100644 --- a/docs/getting-started-guides/ubuntu/storage.md +++ b/docs/getting-started-guides/ubuntu/storage.md @@ -83,8 +83,8 @@ test 50M RWO Available 10s ``` To consume these Persistent Volumes, your pods will need an associated -Persistant Volume Claim with them, and is outside the scope of this README. See the -[Persistant Volumes](http://kubernetes.io/docs/user-guide/persistent-volumes/) +Persistent Volume Claim with them, and is outside the scope of this README. See the +[Persistent Volumes](http://kubernetes.io/docs/user-guide/persistent-volumes/) documentation for more information. {% endcapture %} diff --git a/docs/getting-started-guides/ubuntu/troubleshooting.md b/docs/getting-started-guides/ubuntu/troubleshooting.md index 44b3a34346..b371e1af0c 100644 --- a/docs/getting-started-guides/ubuntu/troubleshooting.md +++ b/docs/getting-started-guides/ubuntu/troubleshooting.md @@ -42,9 +42,9 @@ Machine State DNS Inst id Series AZ In this example we can glean some information. The `Workload` column will show the status of a given service. The `Message` section will show you the health of a given service in the cluster. During deployment and maintenance these workload statuses will update to reflect what a given node is doing. For example the workload my say `maintenance` while message will describe this maintenance as `Installing docker`. -During normal oprtation the Workload should read `active`, the Agent column (which reflects what the Juju agent is doing) should read `idle`, and the messages will either say `Ready` or another descriptive term. `juju status --color` will also return all green results when a cluster's deployment is healthy. +During normal operation the Workload should read `active`, the Agent column (which reflects what the Juju agent is doing) should read `idle`, and the messages will either say `Ready` or another descriptive term. `juju status --color` will also return all green results when a cluster's deployment is healthy. -Status can become unweildly for large clusters, it is then recommended to check status on individual services, for example to check the status on the workers only: +Status can become unwieldy for large clusters, it is then recommended to check status on individual services, for example to check the status on the workers only: juju status kubernetes-workers From af34e80f7e60ebf4449b15329c90f17ca50a2940 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Wed, 1 Mar 2017 14:58:06 +0200 Subject: [PATCH 300/407] Using it vs Juju Kubernetes --- docs/getting-started-guides/ubuntu/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/installation.md b/docs/getting-started-guides/ubuntu/installation.md index 41499b8111..9e3bdd8b2b 100644 --- a/docs/getting-started-guides/ubuntu/installation.md +++ b/docs/getting-started-guides/ubuntu/installation.md @@ -9,7 +9,7 @@ title: Setting up Kubernetes with Juju Ubuntu 16.04 introduced the [Canonical Distribution of Kubernetes](https://www.ubuntu.com/cloud/kubernetes), a pure upstream distribution of Kubernetes designed for production usage. This page shows you how to deploy a cluster. {% endcapture %} -Out of the box Juju Kubernetes comes with the following components on 9 machines: +Out of the box it comes with the following components on 9 machines: - Kubernetes (automated deployment, operations, and scaling) - Three node Kubernetes cluster with one master and two worker nodes. From 94f02ff83d94bd5123bd91717d602f66f19050e8 Mon Sep 17 00:00:00 2001 From: mlambert890b Date: Tue, 28 Feb 2017 11:01:00 -0800 Subject: [PATCH 301/407] Q1 update to k8s tech and service partner list New partners registered to K8s Partner Ecosystem sign-up form. Logo updates companion to this edit still WIP --- _includes/partner-script.js | 110 +++++++++++++++++++++++++++++++++++- 1 file changed, 108 insertions(+), 2 deletions(-) diff --git a/_includes/partner-script.js b/_includes/partner-script.js index 4d0a117620..ad8b4ea61a 100644 --- a/_includes/partner-script.js +++ b/_includes/partner-script.js @@ -279,14 +279,120 @@ logo: 'harbur', link: 'https://harbur.io/', blurb: 'Based in Barcelona, Harbur is a consulting firm that helps companies deploy self-healing solutions empowered by Container technologies' - }, + }, { type: 1, name: 'Endocode', logo: 'endocode', link: 'https://endocode.com/kubernetes/', blurb: 'Endocode practices and teaches the open source way. Kernel to cluster - Dev to Ops. We offer Kubernetes trainings, services and support.' - } + }, + { + type: 0, + name: 'Spotinst', + logo: 'spotinst', + link: 'http://blog.spotinst.com/2016/08/04/elastigroup-kubernetes-minions-steroids/', + blurb: 'Spotinst uses a prediction algorithm in the Amazon EC2 Spot allowing k8s clusters to increase performance and lower the infrastructure costs' + }, + { + type: 1, + name: 'inwinSTACK', + logo: 'inwinstack', + link: 'http://www.inwinstack.com/index.php/en/solutions-en/', + blurb: 'Our container service leverages OpenStack-based infrastructure and its container orchestration engine Magnum to manage Kubernetes clusters.' + }, + { + type: 1, + name: 'Semantix', + logo: 'semantix', + link: 'http://www.semantix.com.br/', + blurb: 'Semantix is a company that works with data analytics and distributed systems. Kubernetes is used to orchestrate services for our customers.' + }, + { + type: 0, + name: 'ASM Technologies Limited', + logo: 'asmtech', + link: 'http://www.asmtech.com/', + blurb: 'http://www.asmtech.com/audience-channels/software_publisher/' + }, + { + type: 1, + name: 'InfraCloud Technologies', + logo: 'infracloud', + link: 'http://blog.infracloud.io/state-of-kubernetes/', + blurb: 'InfraCloud Technologies is software consultancy which provides services in Containers, Cloud and DevOps.' + }, + { + type: 0, + name: 'SignalFx', + logo: 'signalfx', + link: 'https://github.com/signalfx/integrations/tree/master/kubernetes', + blurb: 'Gain real-time visibility across metrics & the most intelligent alerts for todays architectures, including deep integration with Kubernetes' + }, + { + type: 0, + name: 'NATS', + logo: 'nats', + link: 'https://github.com/pires/kubernetes-nats-cluster', + blurb: 'NATS is a simple, secure, and scalable cloud native messaging system.' + }, + { + type: 1, + name: 'RX-M', + logo: 'rxm', + link: 'http://rx-m.com/training/kubernetes-training/', + blurb: 'Market neutral Kubernetes Dev, DevOps and Production training and consulting services' + }, + { + type: 1, + name: 'Emerging Technology Advisors', + logo: 'eta', + link: 'https://www.emergingtechnologyadvisors.com/services/kubernetes.html', + blurb: 'ETA helps companies architect, implement, and manage scalable applications using Kubernetes on on public or private cloud.' + }, + { + type: 0, + name: 'CloudPlex.io', + logo: 'cloudplex', + link: 'http://www.cloudplex.io', + blurb: 'CloudPlex enables operations teams to visually deploy, orchestrate, manage, and monitor infrastructure, applications, and services in public or private cloud.' + }, + { + type: 1, + name: 'Kumina', + logo: 'kumina', + link: 'https://www.kumina.nl/managed_kubernetes', + blurb: 'Kumina creates Kubernetes solutions on your choice of infrastructure with around-the-clock management and unlimited support.' + }, + { + type: 0, + name: 'CA Technologies', + logo: 'ca', + link: 'https://www.ca.com/us/products/application-deployment.html', + blurb: 'The RA CDE Kubernetes plugin enables an automated process for pushing changes to production by applying standard Kubernetes YAML files' + }, + { + type: 0, + name: 'CoScale', + logo: 'coscale', + link: 'http://www.coscale.com/blog/how-to-monitor-your-kubernetes-cluster', + blurb: 'Full stack monitoring of containers and microservices orchestrated by Kubernetes. Powered by anomaly detection to find problems faster.' + }, + { + type: 0, + name: 'Supergiant.io', + logo: 'supergiant', + link: 'https://supergiant.io/blog/supergiant-packing-algorithm-unique-save-money', + blurb: 'Supergiant autoscales hardware for Kubernetes. Open-source, it makes HA, distributed, stateful apps easy to deploy, manage, and scale.' + }, + { + type: 0, + name: 'Avi Networks', + logo: 'avinetworks', + link: 'https://kb.avinetworks.com/avi-vantage-openshift-installation-guide/', + blurb: 'Avis elastic application services fabric provides scalable, feature rich & integrated L4-7 networking for K8S environments.' + } + ] var isvContainer = document.getElementById('isvContainer') From 742af56ffe83f18af64ebcdb93fe6cc83e860564 Mon Sep 17 00:00:00 2001 From: mlambert890b Date: Wed, 1 Mar 2017 08:06:47 -0800 Subject: [PATCH 302/407] Logo file updates for new registrants Logos for new registrants --- images/square-logos/asm.png | Bin 0 -> 2716 bytes images/square-logos/avinetworks.png | Bin 0 -> 21259 bytes images/square-logos/ca.png | Bin 0 -> 17790 bytes images/square-logos/cloudplex.png | Bin 0 -> 16389 bytes images/square-logos/coscale.png | Bin 0 -> 74625 bytes images/square-logos/eta.png | Bin 0 -> 1237 bytes images/square-logos/infracloud.png | Bin 0 -> 11242 bytes images/square-logos/inwinstack.png | Bin 0 -> 12665 bytes images/square-logos/kumina.png | Bin 0 -> 25027 bytes images/square-logos/nats.png | Bin 0 -> 5985 bytes images/square-logos/puppet.png | Bin 11473 -> 2764 bytes images/square-logos/rxm.png | Bin 0 -> 11212 bytes images/square-logos/semantix.png | Bin 0 -> 8616 bytes images/square-logos/signalfx.png | Bin 0 -> 18629 bytes images/square-logos/spotinst.png | Bin 0 -> 86339 bytes images/square-logos/supergiant.png | Bin 0 -> 13850 bytes 16 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 images/square-logos/asm.png create mode 100644 images/square-logos/avinetworks.png create mode 100644 images/square-logos/ca.png create mode 100644 images/square-logos/cloudplex.png create mode 100644 images/square-logos/coscale.png create mode 100644 images/square-logos/eta.png create mode 100644 images/square-logos/infracloud.png create mode 100644 images/square-logos/inwinstack.png create mode 100644 images/square-logos/kumina.png create mode 100644 images/square-logos/nats.png create mode 100644 images/square-logos/rxm.png create mode 100644 images/square-logos/semantix.png create mode 100644 images/square-logos/signalfx.png create mode 100644 images/square-logos/spotinst.png create mode 100644 images/square-logos/supergiant.png diff --git a/images/square-logos/asm.png b/images/square-logos/asm.png new file mode 100644 index 0000000000000000000000000000000000000000..6e383ccb9937fa32625c5e5594a7f68a80a64c28 GIT binary patch literal 2716 zcmb7Gc{r4N8=kV$NhuXl(-g@zW|(1yA!Zn(OqLO8Va&XkGiEW1K}BRqC(C`Feo z4hJ1sLO4W{M6#u#lVwcV%Q&M`ecvDFeAo58*Y&=?-}7A0eLwfJ{P8BZ5b-}LYbt|4 zpq~hiI5+9(u<|OdmVPDs`JK{3gJb7%SU1Y`~XvZb>805E|;4glN$GBqr?8L$F@ zWb|kxFP@k4el&%}Feb0W7>6?0QZxu;WfRIKQ|JH>><0wUm>7uYK@9{BU}s1O@#uvI8pO27bkWN;`Wh{;8VVjy36(bD^h8wvq`4dKx- zkiSKFIlF+dEDiuh7$c1+FjEV#Im#G@Kv}@e48d>{7!qoNg2K#VRhTP~@Efduk+Y%~-q5D1I~rp7Ez02GEop`a#kC>(AijWFVdF?r-rBPMs} zcLp55rEq9$9*xBWuP~DRSbQD^B31f#2@LiRS|;~fo1_hchLYJ(n6b%9N#6sVo&R?z zgYhGp%X0(%qxT<)xuh^Q0CfYnEIx-K9i0Epl~imrmIIJ^EDnjq3i{qfmp~Se#SLV! z!C34{*UZ6s&SVOWx#HaO73J)VCNQ}?GLr%ja2SY`$CyT=qTz59*32AXhC`z4U@&{6 z9TH(~XNo|=aWJH*t%dD(ERIFtGXN&zYAGCa{!}MRVP^lQ`%6R{4+}~Z&9a_2mSYD~|WBmc9 zbmuwJA>-|m|E27?rCGBve;-xE$XCW6=RipC`?s@XyuWfT5CnKoccgb#nb*NSubnD!sMS$pUI8iRfTUAUg6iT8dN)nDi`lBh=r*>%?k`D zdAcR$mMDa4L>yp{;hL3ZnUz;s`ctw#O*TXw@4t#7{&GWZkB3bPX<&QD8b{`80ybK1 zJh$wgQqN)fc~-Iwt5hX5G@QZGB-iz6X6O!1HiLIunKRjz&C6n*zh>pba@O4%;lZ7TNJUrr6+YUk=b-t6!KWQ@%Q(Hu?qEBSVeWln)iUo4-v&7x zrFTyeVrJI9L>s6R6y~EacMN9_If;F_Q$sI35c4wilEn)=r~Vhi^1j7=*OSTOCY8aLu~;NhN;=xZ()lKRS;2e`(U2L=#9JQW-xr>wNCIJIfb zeCX9Z-6y6>`ag^ud|`RxVA(mXc8y`O%&7{Ap<2{$fY^giKT+efuc zX9G<2^XE42IO3~Mi6qsIR;aaT{&KQl_pxB>1v|$~lEsTnT(yPn(l+z{1TeL!Ay7wOA_b>X{qgIt-2TMy%N`ep++y7p&=>kAQ!JDo={JMHBP%HO^YqsyDk!?uO_@uTfy;_W3pzYa-~ zca2U;7!D6C^cZx`d1YFiUuM}SuPH}EOZ_3?W&Os>ZARPnZYCvY9=?FASOC=RZl}yC zdsR%&%eYu!O&63L-i*Du>#AW#6~-X91uwzQTh{%M=oE>DOls1@%owy6auviTXRLPWq)THE9^`y9`~2(|KDL8J%9+!;`F2 z%*wtWWhpx~vY47Sa=d1HyM@Q+sn?r-E-RszuPd4K5SJ(3^D(msSU4-V zHLG(a1HX&1!KF1Y+*>JKxVhn8RRcpJIVAey*(I`*R=WbuI_l-q;Z0uI^Pnx7Pw|&N zbU@<*go&(K#mDsg$72jRT3uOI2Lrr=SMOsy6@!elS-OEEtG~W}ZriEq&PJsyGo_q< z{_K^I7wgn-V)Z|qq4pk+OB+zJ9@Mz_vCHetaLb%nYo_|SHvm&Umlva|KAgC)fl)o2 z+t~rt?)x+rpj|$(F92#AwWo=yen?g0hN% zosI%kQ0{a~QXNnglvNDubQGw9a;IC8>VT@CtYToNqd*muJKd602UG=R6$3jR1*)Lj z>6WBApeiV<7})73P&p_~;+q2mC;%wP{{t>u{nH7jtv-<}aDn0nJB%=D%;s{-49Wgn z&5Kjs35H%IrTj~c)6rCyLE6_w12*$gt2AtvMH~wn*18W)yvjhyw{mWruv>tSJbV;1 zJ)&^nAOctZQ(r6)x$64_CI7wySC@d~{{jRuR>eqaksJfopuk)zAaG=?0bCTqZ|%_G zE8Y&_=*e%Swg6L;TfW38X*vt}<==UIXH0)#h_Kv5HZI9!!=5Az2W+o-ne`tYQ1WdN zF0KzBo*qn0mW~997*6H_hgVenYX^eTtQ7j*>25%21R)tpgn=fxleIvp*J22`45cCeVN1zDKq+I{#PU}LN$Yu;;0S>W)|AmZtJurKQMDE}~ixkjP4cDmHUP=qo`` zxuoR#j0j2~TEqgMADVO8q%fe>k~=Ho0-~WNwHUpU%yUB%(GG?)&0=cptOjSRL#49I z+8aaQ)QWE5AQWwYsHexJq=wALSQ+;G410@apky)=t!N}eyP4znH2@+8lu8{N2_8U6 zL`Q%c(Pk5+rh=2vW1k-U1j_8??a_OnKF;O06Q9Jz-uX4 zl+b20e78~Y1$YieS3s-`{VD_{5wBJ$E(6;bZAu~kB2`S#Tk)p;!S88QPU4}UnTjddf2S+4{ zR>>kwsAvdFT0crEs)ZbAtL?ZsSTrUe$z+_Eu_|0HzOUDWpoj=4LvdQkz->Fx2t=(J zWgr7=G!4p4-ebX|GMiF`fsAd!r7A9(#U(Z%!*?qiu1LK~s57*CxqhKZDWIfyq-_*c zW20f|q;*t*M2(Agt2}dkr!4R=$NIK#I#i!z7$w2AqcJe-s>4B6e8@1Iad?KRyKR-H z9lxR-phJH)D8scLTvb-2zLk#CR0so6nxbM%y6|%zB5_)rp*g0$AawG8s0P3mF2rzt zjWnZWNNC2%)?NW{69Wh8WB^C&ctnxUyNJ1jWdZO!fOD&4VJn!B0;P|fpGR$^Rt+hy zNa`1W2!~k;K0Q`H!*wytH^f00M==6;3^hdnHwfV6@`%Ob@IAn#UrPZ_TDgu4DwFk4 z7mEsCJIbyYa8(9MBZ_6>Tu|-4$ltZN)?&S@KaGa^8Wl#*7Z9PUq2{j4s2L(P)osde zQBp)nKNp3Z11I80q(~N32Aoq4o5R3#r0Au|oOj#?0703~r;vyx#pdO@J5eC-0Wpm? zH(yjpB<5RVWgt_q@uj>!;7simQ~eFZ*S+1|N45TPX$0O|fL8tM~5?|dOZ#C3`x zw=xV}IVicN#)jbnbD@li;6MXq#+b`1cu2TW(VHYd%NG!jaV$PO&qFenKu{>aA%m1$m4h;L^R3`s z$^L60pMo3XXemhK; z?oz>^9X5QSeF2$F7H(~gmik%5;lq_dGd5TQKb?o$5DUX_gsu~)P>|Yk9O_lHRb@BU zjUwauh(;;5l?e6nig|sVGoZ*I{W39>+cr>JYuJ+v>Q&F~pQp>X$sBzX!Rayn|j z*;tC8!Vzud?IwKS0=}@+F9aLaYWstR4}c=35R<)EB}zPGTcRv0_k~Ab3>UW(+@v z+gX%4*=}T>{X1Uy+i%g|-GPIS{UAns^z%S;6#Rs;F?S>=^O||$+-6L70bVCwx&3CW zU$YX29D4$qPxw3#Z$dtBL8~Jo)_H)O`06UuzC%Gt5~s6p8e@nf1*A3r3tz$G5Bwa> z&0}!z*G~l|?+(Oc$Vog|*&78vs!WuMh_67&_4fK5M3YhEkwc8*SoE5 z`CVYI{g5prk#Lg;@&TMAH4^XNI3c8BP%2hnSVHY^1SUdHM<&+~EL((UzJDb~^{+hG`j&<{sx|y`#ns^1xPzl)E#{c_W0^I z;MXMK$7*DgL(CG&&(1yLnn_TyB_~M?Q_`G??z)GjrPDrcE9&v z`uUdw$yOD=@Tqp_+PVl_dgH@zS@ArzpKfou+v#am_L6ZZDR3l2nT4D0!kRm8M$a=( zVPwunS2Th4sr%r_%dY~GHSn21zYSm#+++_iTlEj)FE(_@Vo#?rV$0c z0%|45G|>wQ>#AV5N^mI#X(|egd_m%?#+Du$_0(OV@GvQH$N=xWhQ*i6#Du;MnahgWW=xxN6EN*i zz)|5AwS^l<_UG_}(!{Cq8N^~q6+ZEEKz-qUcU{qcxU9f4l)5 zW2a!>i>?70b_E*Z$UA8*XO-cr(hDdx7(;a6_@14KEQ zN-m9T_TF_!k01xUvsi($e?8M|v7&Ya_MUSyu+M>jOJ;CH@Q{Rzr=zcQukPKaKG&clI5uX&Q-rZ@U*r zi~=H*I1CDNVF^lWlVB*rC<&=?ho%&ioU4(P@$mZurHbQ6aw%b`S1!L0BUUa%9rEa} zX~M{(jz|61&jFIn2q+yH@{;I*%1{?UU#XY2Q-YGv@O?zM^+sob6wP+S``-ug>TiCI zF@0IMiCQ!sb2Mtkj>4)x{2pFU4^njrG#qpkCeFMB883=teKUNI(S8j|F0fNT;HVrO z$@?=4qQ$@YWXQZO`kUJv9)dUxZV$+H14~}Pf*Y>FsP1+o;bTK>BPM?B3?z^F7?2#R z0@T8PI;L5{-gjt)$UoG4W$H3lt-=XDJxNL)^##kr>gxoaeH^dccN@lf-N^eMmPhJw z%yqW`llB1;P4K+}+?cD)A`da+=$K~>g`tf8W~6Utk(A0NQ;-o_($>lM0O|F3^U8Bj zw{|6>xd=KE4cPk%ItvNBarM=RcdUUE_pv6C#1WU@2<)~$5NTFn3fsbvHKCoQ>`9Rz zdKG=Wcq&PCsF&HhS9LeOUqCFH2QnRaYwjh8y|n-hxgPWsTy&0_h6As^6NrpJzP1)> zGso;j6H+lDP$%676x#6m4VR;J$wHt%i?;f4*ypVCfg_Gm zDrdnUB~YMoFcdCSW|wP#Sucks3d9 z4ILB$S2mnv``%9jop0m$>n=gcg8qxEk44{)Ewm93pNjI$9=S&v|zM4I|(q z0pCg;lti83jIHjBpft`S$rh#RUic5Za`QFVHIqihk7IfBE;!=mJAv9pfE)5X8}R%U z=V8LCg{UcHu&#Clc0cK4BtCvR3T`84n#w+^rd-kHQigp-tMVkjBwwrylpHrh6ErPj zuA`+-20%CZAN?&>{_dw}>g+&GK7sbeQJ8Y&HNcep0KXO>SSwiJA}E8kE!yp<|;!$#l6u zHkFH6b5YK{w;lQ4J%|;LKZ5$y8ssB!^tA4aX;qOod78_MwrhoH zLOUZU!SnBGi-2+e8kw=-u%BzxmMq__aT}GJf#i9qZOOpq> z+u=QUA67m2zi@Nw(dz{0AH4?-z2YXIVG>ZUw@HUAZmfz(2Fo>rNMKp3@R;7^A_k?3 z$OU449uWr1^WAv$!qbsh^#(?`4$^fkh`sL<82gnoBuLGz#fU^Wc_-tP#qgCe5lSlw zw|Xi&B`7(lToy>J$EyGNF}fdp2rU87=8Q%2VgH3)&NyGTxN?39ak{KbwaN9N@1Y-I z<==jf(fMBVI|+18ITX|9%mo_ikwQXMF9!ugv^^APFI?;|BjH3+R*+{mW7rpiTC z>jWyj0-S6LSo|`cyY3Q9jP=9KrO{KU#a^F2894D|1ob0_p~P0J8dF?j z9q44sHyt2a$f&%6=N`u^_gpVRpdr$Xw)$4=dG$@eq$vPubl6CPQbIrml*PiaO2d>p zo%byRBDMSs(6<^-U34lYd+qRZ1-u)ZfcIT~E3nJn2ol_($*bfAvS@#wgpFRh^c+m; zUJb|R_ebNPD`>AxMJCGZAqFiRxH>@ROVC$2e|G_za=lWb+C6#~OugktK~BmKIt z@Y*@3U-1eWJP&=TI@BI?3`WnMgKVM>36mhfQeakUN^YgjO&GRn#$vg}P7@MR`R#ci z-w!yRXGwnD`KYI?4Y(y#u5RC^Y1bHNLOrm2wOdnpm<9fvY^EWhm zdGth^QFFlIm~ife$T=~@l4`G3@`9tz)&hDR!He z0_H!91vg%f-GUSfxeV4fwqTF%Tnrra9w5?aBI!dMMQD^4@0WdHRV~Tf$AYIyG$cDI z*NX^ErPt@WD9YubZc?syt{z$laVjP5RKQs5r@W=XYjH+SD4UsB}5etH;W~r$V1>r zF@dr!b168PF1&W(85pr{8RGdq^d#%BVXuR6=%v@o7M{dchkU77$l8XX27*#MC=3E zq{4SPP!f<-aWDJ>uikM9TC-h1vI&_9Q?cuu8-T_!$Yt}0*VH1H%_5#)&YgpT=fRC7 zfOI?7+;A?OH|B$B=$-Lqj6dmQG=1tzlKCAZYIK~Au(OEFdKtTO#Sj6ZIy|TtXe_V2 z_=qa~B`Cukq`nQnUw(llzq=o!gEacR03EG+VE-%U0V5{@iMk<+2On=$wKv~d7D~Hm?c!1jbD-O8WZA`u^8i6^JU3mSDxu|>f@2E-lqbu5q zDWCc(@X0U2kJrkcILj*|EH4Om7PQps?*RV(8!UYAZj32-@Z2O;)QraASKTDivJhiR zsveoyt0iYLsnyC`9;&1i>qCOn0iqtw3bn1t{bx_d_Qd<|>jCoX@#-Zr(c1AYkj6Y^(e|4M(UMIg?YLM!avb)*{~Kv!t%tu0S^vo!r!i=ql6xvvmKR#LoF34oKLlhX&RdI`^7buM;eawp<=H#rf9 zFitjlGEmDzE0d9~$_FSFLJvBh2D;nu%oXROb@d{Qj5_E>19tn`w}In6hM=Yy1*y>E zT>%QdhbS|ZU8){cE>}UA&^C#*ZKT3Z%8b=%AS{Dm8DpjsZ4cdp-oO4Dtw9RiPJp!| z#^SJf_X8s)06xtU)=OxtvDF^aaHIesDq2%ni3~gWs`rlA$BZB-$xrEQTqc@3%qnF~ zi`q$GIbuJMU5oiwT!1ml-a?`f!K#Ka*z@C`MdE}LRcUCf7KI=pG-4W|5=V1{{z>f?YT!$pXx)t zwgt(fPeRL=&s6DBQW!+n3pU z)wijvpJ=1_Gp^WZK5|qiQfTEC1LAx(%L`V#ju$Wf2FB-mrSfidya6*#y9m)E{tJlJ zAy0|usJ&oH8j+6!8Y&GPmB#TlU{bcKdY1&2xx>V4R95JvJJ5N@4d`3&4BY+>WD>RL znz%0xW+H9FSOl?zE}tVd*^P`+QS~O;NPUN35gnlR_65)e*t#{Lqx)q_xkeWjUp))4 zH(x_jrWbv*Wsg1qyPa~jl7Kb!2;77eh%sux4YcBJn39sFurIVShE6(dQYcA0tRW%V zl*O&QlAU^yoUF`O2=xOX>o};TyiE6?>yd}B`jP)ei=Rcm8;8I9o|wu4p&G{Tn7FI$ zAeW%zVn@K;*f7p0As1$;cVqpnmm#<0C3v|M`V-A?ryYsCzB5ZnT(R)!v}X|n9-`cH z4q_^OQ=3~up{)pOlX|BmoGI8B%^~9_FqsYOscOd;|Am+Cz6Gtl>k*H~u`)>Fknhe0 z4m%!xydD9=Em}>h8Wi1L+w!bKx@cQcC1iM-E;+b?aPZDNEBmv(|sDxl2Na*OBQrjj=!D=+c?+w);Hw51n>N^MsHLi}% zK#{?l{79VyD?Mo_Eou8i5G~LaT#1F(UXJFsUq`JI!SZ+=4nF@vVETdZ6DrjEI)jf`EdWk96I=od1G@ZH@T@ap&HV)UxF5b0WvloLhs;m4ri%u6KN;5*8z zH7R}M8F490l7LAFw#)?qrEHrL{G^F#WkJ)$MKqHF{50Oa`xbc5KaNDI3x&pd_`6QU zl-W0^x|iApKf3G8YGbfBM_2^LR-c)&5u>n&hR2Z2+GuY zJahGhs9(7dqhbkkWfB)QXCiYsE(_ zTTCTR(BhLKYXna{j)ix9ALBeqEFbG?8!`2w>w$d_2i!W9UZ=-JeikoUTH~qEmhR~v zR>+EtZ9u8pTq}c|4hgfbvcRgh@!YI)uxp_cUOt1hbqzT9^qIh6{{CSv6>NK}v8b+P9AMPc0GR~}lOt4xp|UF}8OV>I>ai6F8_l;j5VNm9tb0Pc&|d4fRA#FH5>ls-cM;8&vkZ+4Iy* zc%a(aP@Vg@tJ)pk$H1D8NDi2{*2`J<~lX{P6)S{nGuCJM7l3#(GEzOQq8bbxhiQ^J3{(WA>K?lDgHi|2=`C_`94#>z!VpIw$Rp-B z@C!bo^elSXvFPS&(D>GT)O%@kI59LHc|2OrnvHz4K|3KuZ(cI|<5)dnFC0u`zGejS$)sRB@B2s+#JZAl=*mY)C#GN+ zuQ$|WS}kQA5A23c$xmNIN^x@@uy!GynsX*5_P&dz`g*K#TQK#cFC%f{$v~{xN_^x* zbwY$Lm8C3`u$s0#C157j5w!plTy#2+utN{>|9Jv$+;#)TWqQHxW!){~u;)#80wX6O zh|~x@!=CIVwy+X7MZq05-Y<2$hDyY3n~wAm001BWNklfQntEOL~4jSOtYN$cX zQ5Z764q(|Ul0G&j-6uhTw%R7_cggpFDbtm-l#ajpqIa)Pl@Ji=Qyd6NNhq^y+VJ2b z*3U1WkEgG@9Fsa%BI-JLr(v|Jm*4GR1PQ~Wa<*0x%tCjw5Vm_@nPz%emH0SHh;4&7 zWe7^$!#i-e31(m1VDtb6wmVki)$`B6*iH!;2 z$|Ed`MCNo?!8qhCWk4vw<6j<^qS_(ckL?Xx^^*&5H^Se?- zs9Ka*XUJ%WPadyBT49~HWbdf2_h&PxO+?`2y0HGfAE5V{C(zW>hEyVf=8-iOL>n0K^#E$!q}r z{ujLY(|ggB&mvP>gLTcLaOm}S0ErPmqDJTH>BNQtmz2>a%d3|st43ICw#3VnJ5V~y zzuCzl-HF%deh2Y2Z=l9Yp~H`&_1Mp1%*m$%v3g0D@nUhcyHrV*i?xU$C`0w98tv0? zA?!nw#Hd6+tEwKKc@i(&eiQbN0sZ+5-l%WDdvCf0Xr725K0=>1VGL*!GbBLFF@VOi zFM=}fr4VD{=-OqNf64jSrLPN-Yzj*Q7YE*W2e8KsAljtQDAK;9NJ{-IjWpAIDg>?J zT&pIjg1R^Lv~3|N!)-<#0uiHJH3LzJ1pF7v0@CZS^oA?Zu>4J_4_O(DVV}=_1+kBO zUFA%A83n*d6n;>U?zilqrkUT$6=ZroFYXB!bqN2-G`9rfJf2D#e&yl&Ffz44k}}%k zqjAWUw*uq#lxC)WQgx=SD50PPlpIZ%M9gO4LUhGEB3YOMrn`j|*q+(T0I3eVG50dm zFMSQQj)#>oVAm5rh5FB*qKeCyKgyeh>ovREr)8K9RrGIs#$x$0w%5yG!(V=f_6P67 zs8kR70AzOE3wvF03(z`Ah2(WB=WvTs14N=6vWm)~RB_NEfaG)S$nKP&3>Xr!O069^ zNKzLXu@5+>pGixwe9@0?LrvE@MC;;MRacKAuDlr-wI71oT5!ih_NQG|WV5NLf+Ir} zp%E@M%?gxGL4h)G1LV9s5;WMQw+*{&^Z^}f@Q>MNWB2qbG{Hw(I>6M=oC19M+p@=7 zQNY*M^_np#DT|<_J*GkFB~&PjCm@JqW}YKwn1GXI)_php$NqpN_uqlBK@PcCfOqSf zaM-+`03&u$U5n*@)v`Mscu`7`+iPkt{qzff z15XqoK&0gY)hwS2Ql`m-vMeSArk1EoWR3~*& zaA*l6l1If#q_ma^92wTa6wvm|`?3Bn4`8g9f)k5lUDHJDebdi?x-n86StzIu4@N)~ z10TiB0AYYl(cFN8Y^X!3%ABA*ru%KO=>Vk|b8U4y1g6l%BDG4X1ZjyQzWu#fXl!4J zc#y`LL@o9@=ThL%;{Y%-oqO2IED{iCn^lQm6?f&%sX8z!FP-18^bNf9{i`vqzYR!4 z@z#h|9CPD6Kx8ufd<2n32gTMkW{!%MCYwvOu__}eb+sf7tV)i-+h*!J@DwNmHy7sM zP&LSD-CP&e->iERug{r@*35cj-5k~>TX4|j_W~2yJSlFSCvO5hN_A+a9AIP@8#pN& zir%NZ&9oJZY4ahS0oJ^Og|okn3HeU=u8+?87EJow>8N=x8%?#SwstzdBiza}s#b#d zN(v0ff-(RNPqEP6V^Mg8j*bD1U?)WOz$5+SJDDz|e{wszo_-8qR^ohZ8XOTjdr=354sx$#!GORkKmMbP+~o(pcq0(RlKy z7<1y6704x=>8UelWOi1%?>#@FNGG@%a2N*eiAWgPX3vfPEYQg{-i$Ox6Q-bN8>#^dd zImj-07WGaB>nTh2J^}|^avhLpl@OFUim(Vu6D%eu%hfz7TRH6T>*ALQo%(^r zui&}w&Bnx3I}!;d-!@^!>E{6lzZXGbq_j?9!c{^$As$kQusN(3LBxsbqy418^>U`8}9;ICQE6As$*b=JmU}MgoUBbqQd*O1t^)i7O>HgOZ`a$ zhpA6Imj$Tjo>licymaBWFrs5QVvzzmn_98o*UttH{fJ6zHGA7bm>cI~Tq?CAMlqrj zKSZVX0k1!g|J*PKyA}H3I4+h&ns6{x<9*;%qV%kz`>>7!C>ur02J!f)ph= z>z#VmVA(@I0-pIB>hgVP_ki*5{|H8$HVcR~6|>uI$7#3&;#)(JZ2?M_L$C{MR4if1 z$`wh;#M&fVtUU5_EPnWB7=dnNBQ91pPQ=l&?Kc{T*6Pfmt(isfy8=o*DGA=o{x5Af z4WB|vx@kOl(OH<>wiGQc(B{@7J7quYd&LbX#2Vm4R1!ilslfu9Jf>J%WZoR^HOn6= z+2J54b8smFd79CyFJsY77o#=T4KM1Tt#LFC_}=XZS|$LsBUB;^LtYZ4Fd0fJxg?rr zx?mP)e;ZHFo{2sEE~L_3QgU(lCD#CBrvvq^YBz}*jm@m0W{0MdISg&>2B?eSPV!~& zoO`(m5{d~IlCr}q0J_#oHd(=nla<%yMX5LfTI#?2V~BN&Wes*Ticpu z`Ba`^WuOcbItqE!HLY_U-ni;KG_)^AEZ>JVCyue7`U+Y;{Z)~2z8lvyH_8t43wbFW zW=bnlDv<66o_GYye*Qz$^{zv%sU92lIRrD#ze0|9k#l`$hkJRrdYvlk&siZLHB7`O z8P2AuvZKTDO}>EV-iWJQklGnM|0EXOITvI6KJ*73+8f5=;2ZA;nkNW&^K2nfrn~=^ zPSt=?DO)DIkr(Q9s7NbeW*2p@#_Q*P9V0td!72FYsBc2z{U>2E3(UBk7t!TyChi!< zA9=u{jxlP98P;{lW^^ulk~iqDXaO(S@g=xqUejg$&l$HDQ-i&q2+5P7vm!XH%f@ z&1cY&3#j2L4CxGI;-y3wZctKvO8`~x2m}$f1xW*|SKyud?}W4PDKte3SXYQ(^idy1 z^EWO6>PJf4jti2gAkJlMN=ZmST4l(fRwcjMA)3qvbliCj`d)ex_1RwZxe=_W8;_%} zz6BVw8xXG#z5M}uz?@V>RVbR;l!3wG%0S8IG8`6@%!*z)ehwWSc;Si*F~0pxY2?2) zmc;Z^&H>){8I`C(%a`U7C8Rl5(W%K4Mn1AgBnBZjOgsydsqp{yDKLKi5lv(4t z$a)z>V%*1a;Ym8(|A(No3&%X>O}Z=!>jhqa75};NBJAq)qA%T#o|cK&f955?^!Fo3 z)*`QW1WZE3P#Dg5uPTG~8@zhhV7g+KFUb^v8jL&z$%Vv;in20G~ zI7Lcvx!7TzCVe#{DJcmfDvg(U<6Ow$1f>~255TOYhGEx_-W1mU_CfSN_6xM+y5J>~ zSl>Jm({A{wYIRtk>P;x0Qk%}wY}QZXt=q0e&7!A}$feNc)?%Npd>c6S)2gmqd+m}f zFRGFe&+yMfA^-5)vn?DQdf!<&nw!aZSCg$wQX11+nN|$H;}txA{Txh8twq-Nu&QYi-owh!G1KG- zyJCxDmHSzYF$@p3hWwOm1xl0kRPfS>NAkd`xA5wv-^GMfx17z*(%F4^7}F8&1Kb*E zr^0*W4E73omZ};V9&79%!uXrne^PBa_8v}R$Jp#_B_K8W!rpDFQJ zv5O@p$$i;k&bq1f88V=T3OisqhnEW74n#8)m+*+V>_=)fbfHEv9d|KNv~o#l8^v-Ov!_ZCydL1CC-LH4w_?ot)riFsSdkcs zLvH>dFlra1-58<~x`^Bz$^w1uST_57#NJ*kS?X-)TWr)SMkdAxsO8|zH;lWF*D1Db?6S`t3B)S*4?)vx!_sE zQ+>!Zv|_>sPekOir;3fpG8313tA$>y`0YbTJ^X8o^E{+$YtS)zGNxVmeV}1Xcuuxz zpc<$H0Qg?J$Y-eIACCLB1t?9Hki;Z1oxo#%z{@|s4&8Q4TXm)lC&?NDb|lFnreB?;o2Y!^z1sOsp;j=@V`!^^i`jk+=;lOKtsH!swDms*2C!W9ZVoZEz1)OwF4kkF@dp82ROhqnHCkZZU zq&iK{$~j{m!LXM%Y%MLe1t`5-5b#h|slE1lvEtfGkzf2W>ViDFqe(;#{UG)@^=zPd zEV_N5Hp=CtZKH$Fm7Gmzg4~SM>DGQ`4_n1BRn|9?Y{e1Ev71H)?GN39HGlpwCf2y< z^P}i&9E}-QGmUN+b!a?uJ>-lSx*sarM>4&QWFz235>xtYe2VH;txL#A%7#^$L>?UO z@^q}m{5ju6W9KTw{T#ZJjo9ZK=L1I_4=}9br+~Mg$Aa%)iYO67AeF4f7&7z=+i^y@Cylt(t4OFK-J_7P3JgO}%)a(Xv;u`1;ux)zg8-nmWAU zM=<^7yT!H%q77l)ln{r#bwmWInAwmn6Fq195}n`2MNpw(N-=T^)h-)jsBo1q!_oy< zboES(?q`Zd6zk#**!6R#p!Q>*17a;A5M_^9N&n*j>_STO4hG`78s`dkW`nmVi>Hx&oWzD|`P&=DiZX)C*`tRb{1rD-TYziL=O zsSC@p?3m&zfCo7qo=WZ03$!i6({txwbmwZ+MI%@pL~$?+?e;qw*wBVWm!5*=?sW(f zE$AM(3--B@XAZIG%t7E#ONcQ7FvZQ73zBfmgbqS=2ZJ)KS7XKmg+?A&z8H%yory_( z>xGF1oMOvnxKek`9jQ&7V#`P@ZJW$0{QJVQFtx82L4O7v$&uLm8>a%ten4e<2daf~ zSZOB&(%l?}Y*Sh(`tna5_OOE8OxTci-;&zK)N~e?bpr1$!Be+fh28RN#R1w78;u#K zoC_TGVL555Um21TlvfMObmJHp@#3YDb}LigO3*7cs@l3RR!`g2*qCg3x&fDoycckD^1|H>)w5;e%nPE8;OmYPk4 zFgVGqb1BtTO1Xe{sZPI8%~N%cav^pHFyX7;l#^$D@CXrgVpTD>Tp4JJONXewh2R>5f_9~9CF?}qrBeI( zCN6lC`;Igj3OUJWTKC99=zQdUjLY>Sm-W%xJQllOd#iMgpv|Vv-&FlERWnq{#ZgX^ zgtBB`YKU?oOmWwxu_pCS7u=ctp4_eRI`QJUr(smbS~T$JI<`}sxI0#?T!|*;-ZwNM ze)NYh^2?`*qgIHM-e!Etbt<-)*7Olj@{?`QU|=gK!kDnMu9h7bovFQM30^w?YuM9C ziKJW}ZNiMJehBQozw+Cn37L{t2oTjrvDmM4g4ButPyYk+@46A=3VFoqTd^eP;P5+{ z4Kxxy%ULO5bevDPL#pM}GDoEHdbmKTJrNB`S}EEDlu1W;^a<+;xUdXGtyzY}mz|FZ zsV?NZJCGYO0;%bTV2|(40TN?WmbO$)F{zfGvG!n%$2?r1Bt^w8Fr9K$rKL*YRn1PZ zY$2Y%>;mlNI*6rH$TB(=i6WErvA(_uGcKM3Og}=E&K9D&q&KI#BTz%BnkLy94R@o= z{#6C^-wFVh`>Hx!Lzho7v4ky`-&u(Hmz|FB{q0J9*N?^lvu^?RIYMQy#MPOW5|Yy; z*aJcN&I`dy^RC1AwzrXQZ%4FcGUEFmhLLAo0E}o=wi#U2wnPV>RV5r3fp8@#E0i{& z#5AB(;!}ZA5!~R2<@ber1}-@)-Gh!hZbsjWPh*7Nhpt>2Z`Cy7Sk|JC+gIB1g+XR1 zP}A+R1wQ8vcy1TXwKc;frlVK1*AwGn!Ub(nVcMZjV2 zMLNy~oGR%nX--+yCXT8~WMF%kKq=JO7?eg@hN;H|m5@xxBGm&dc@6)XdpRbf50SWw z70DVL{ij_0-?+?nA#o zQ^y+AB|P*wE>=4uamabIfrAf$&qfP=Eh0pDp6Te!i&I%nO5~S-((LqCVAEF)O67== zf0!dGlV0;C{&m)uF$F0(DfcxmhWFk2Q(%{U!X;Kg*YXK5PVmZKgMZ(3C&sj`N8mch z?70v2n{^q`I1U*%Mn*06?6UhBN~p!O0~osc5SbXNQ*&IhA{E(zpvzaP0s1LCGDt~^n9Khq_SrrNQRg(=3) z6+M}Sef$(0b@_D?F(|l6xQ-6}u(KPh z%w$(eHFgLUiXE`$A9pk;B?r$HQzVv%Akw`KuUv61>f2U}r0i@NjYF=!4Sw@jcsbp( zbTgU%PSCS7$UONZUjN@m5v!}izBA7OMo*9y(UgTr6B{$A4hf?BXjihMW2xnVbu&eG zilGB`JyYtu*laYH%cGW@$u76;EAZ?+^N{W9z^MKA#XevBCg6^MA8Q(@?g&cVMqA2r zhhaTKelpA((nlwphM-~dE}k&40e^VKIYF*X zmDtLmKRmZrHFVEH_0A3t|6#?v1V>G0-r>j+vil|yb_R?^P#W*pOu>8-sz&B7G64d; zY}*F7all~+t5M1aOb!O8FTKZ2PF$$ID-$#WK*_}Uuuygo~K`^_xweQ>%WDGDe`;q?3=JY-VA)0 zBz;9w=gqPnD(=94(%L8eOV?J(6ZJh+lcn=Q%SbfGfHI*w@S1bArE{-H7t1yhA}brB zUnxT#bzVt8Xd}TFj?tEgwvOBcD2Q@6kL<@X-r9$+FJdK^f1FQX;sc46Qrg$rzD8z)IC+M zYumocq>_7#q%~o=;b>4*DJlh&G*8s}m;(Wf#xUNb5gKYAinuY2zBV3!6EGpe97dp6 zUKvJ0H74sTTQgB&&lo~g)F0BsC1qUdJ+@F(BEbyW^!_p5)qmGmEjGH+VyLuSqvSao zA7{Mvt7AxFwn9}vNM$%lPjiuux@?jEcDb9a<|x~yMhGx}Srihx)G75Q;*;N%!%_84 zW*9yz=n@G6b>}FtWuAdj-&#nOm?=uY6=hzrV{#c#s-u;au9w(fs6!Nosy6Xak%-}E z<~iJ^WBe6?FfY`_f@^^l%!DmwTgcH!7M3BLlFithdZff*zFY60j%oU zp`f%)650B+1c)tA!oOP)1Y7!);K%Xg7(5b6gHj}n$^<9RswL)u##TMG3`Tp&3~T6f zytiJ~l$;$?vZZLCT70frP%R5}Izo{Ugq%qkQJv7a@Y*JpT5<(Sm11U{Fg8@C;$Y0? zypTs-tlsbDyM{KUcz~s8000uYNkliM*4|(XGo~esaLM#BE!<`_9uVatTQ<$M67g)?a=xsI3hKZ2a%je;)UP$f5Or zbbx7ApuztMEaWiLf0Rh@pPM4yHx_TM%df4golp?uiDV8+xTNodet2Q8n4rNMzvadx zI!**-!RhnRS*r19GF5f9*745)^zw}(J8@`pCCyK)P*&;SB0VaC$>78Hb>uf~hCD=~ zw3xsLd_-L9NQ(A0PY7qqm!UCc#PuRwlT^bmsw9w01Kt^$dF+0PMOZ_aUQ@)XKRtp$ zIvbD;#b7oZ;)(7KLmCKjK?I>?FcshXGXEG}iR7%`$Go-obQZwId-Tc>`}ONr0YUu5 zz8nPr2KO|>NlgyGHslI$R;yoc{SnhG@JoOYLt*Pfl2X!WQ*?sn(Byt3bPyy9YZQtx z)~Tikb~|B*?xHx|oeiT49*+mk&1~$eJtGHQaq|`xMnY+gBA7tc4tf)IYsgh)6SC+j zJUrP=%M{7zckSy_>s*aShf$_fX(}WjUM$`M;)u?wXy*CDPPoI@73KCpvrJJsN;^pt2+#Fokvjm;@ z6UnwZ`89w|A{P_Q-2c4R{Kxecc6Z!zGADLEum7T*e96S`QHA?Ww(~&j9eFE}by0bj z1<#Jg!nnyS^J;CM1vk~$FvX*tzkfc?){U5UpE0{}S|vXp&EAS2ib&*Cg{y|3QTF17 z69HgVFms<2jgrM61-G}iIeK}OS_ObWpr2*s-Sth}r5g8o6&>)O^^8$ zG=;X4iue@29)DjU@yGaYu$6aoz%+KoU3B6? zf`gO8nG>X_cvj)u12`D+N^bJRrA2L(a18j*_GTlRYMON3Y+*0Z|2Q6<_xk=0R}MRb zi$(2C7w0w7fO>Z&A^U;-8PadUKQt|zPr0CJ)2nYzYuo)*%o=!71Hr8@duJz@y$-0A zqSfex<4}eVREkG695E00x?P(Gi$x(9nT$mkzu(IgoBIvk`R`Os znhE&|@kCv>kN0ah_+gCdsf5^P3&ku>g!17}T2w;4*N++4(SDlKl#%0KLB*C8mC0ZV z!Y@1`;#ifop&_*J_qtHW-ieFhOT}L|lt-m4VB3{u)2#H3KSB>zmSnH;b&|uD!Ad zWA4p<-XVK!G9L4<{`a5mEJP?4t4 zNSR5uh290}7Ysa%c^Y|fF`BV~=YN^uUW|xW9?qs;+Ve29INX@9H;$djb3x(G-Lt=R z`xDk5(vqLC$y&GO8_$frq7&-creb+-u74#;ni#EmNroX37UM;IIvtPx-YZ0I_FtCj zi&JXb5^QH}g;`DNzNxUyOn3(K_}Yh-wJU$ZSW_PC(6JmYn@g3E+IS|;V9Vl!9WI;Q z4j28H#T&p{^EdVNPuM~bKF<0*O2}#6F6B96O4(eO`Q~tc0mwP_Mlb4EPSvB^!cn9O znhC_A;kQD)r~v?#kfLoBhY^+EC-_F4p|u|n*?iv~!8MxD| z^^ll^B372axO?-+OfUcl+|wu!$-P;Fk?dH`emX`ox@KH<)uYK4T|R*OPvDe7>=L8;D$S5$|SI z*%x+g`0#85_23m7Etho!u?d{67C+j<&$knvn+%zhO6Mmu3~w!ubXO-_5R!74eDcGg zOQM%jKjsFpYiMb)`<^gd5WMZvvZ~hEt0Y8M;4&PZs=;5(Y+COq$3)o{o{Ej!oa?tc zuHRjK_E9S)u*M~AXK%IsnwO4Vs!VM9nyDK46ra|XxdpYM(Xb`v_cSYgZ2+P`5^q95PRSJr`?kNfsQ-k&E~D z9&=q|HIzl|_}9NdLfG@x^mh+xT_=wp_N7ZcXNRb%NMr4q97+PN?y=fxvL{w|Jj+41 z-iRw>IB7T|7g#j-G-E*k$IK@oW$ymat?*8&17~Dd=BO#+2v1(ilq|c~k(`L{im$?J zgO)A7C8|@AwJE9eYyQ@nM6ZZt2C7YM=6LxCJju$^0c{{M19WV+k1_b@FrM2X0%Ww|qx#D>*79=un%Zz~k7%E_p||haRZ*0|s8s)Z zRr17y)7%{C&dGMZZPqKEDrXn{S@n~6=W*O4qc$RjF(>=x;{J$DI=1GXr_6QuVqqYW z4PntN!&&X%;BZ;K|9)!)!4Td6QGR^WfQAkhUp4XsklTWS5vkY41NRgj#e>5(z6j}b zg1mz}*}1q&(i?V=m0A^51kDt>OS>d4@Xg=(Tt~iHls#MR_MPhW4k~k}QN=%;de>{q zwsGt7OA+@g*2=v%7fScMRbp1Zjtlx)y+*F%xK6C-Ue7{Rj>=ra&}oIT2xamofJt3_ zJOm^#AHP`x%=&QjMmo6p@=#5@36mnrYh>Ws$D`kvyRUBEWOZcn7THnWRYsgGWO zMxQ7g?dv(0-vR#a1-mUJEzJiZ8V>}6JF=)Hj>c2v7i=B!GRWvL^h-h`Qhen*v7UW*J0lAB+)Uso{@Exm;Cden|q z5dp|SV1~dC%Q!A^1#-*+;E*#GUOi~jCi5SHvc{ks?oySUZ=PAMR=>MaU4(lU7uUDn z`>lq{x%9v2C?*{vbPOYy`Ec%gD7E+7CB2OMXGgHVNov6a;eit$R!_Y7JEMx=ud!b$ z*@7U|@trm6s)~Tds~~96(v77`3yYL&OFw=zlfv@wOc)3>T%qFBxavks-~CeV<+a}P!7{uGDgH661s&SL7P^anByeu=(QkLsGS{tS)J9sp- zgPy)V`|X=)iKo=#5rfyjzQ<^VS3v7pz|zP8mZ0yK6&?p3-}fQswW`TJH5B@q@{}AK z=w_T(23Q^dorVv@sII~#D_5R?UVkE7Xm{z3KlMq{Rji1>j(6A9bV>N7@)YmEL3N4g z_g}ww@~WW`N2-fTpfc&|p#cRgbFt)Hota3L3;0>^QPbk9tL#DaD9v-#K`R+%1`)~A zX#^2kujy1*W;N$$I}T$Pn)#D-FOATE>y6dWg9%ddQ^x`}@0B-}{m#9(-OVsSfEC9E z4YSuHE5+?4%6R7A#4dsl>N%~K$XSguHZYYZo(N6u(qq&eRzP?=MAc#^qki8kDN@@kH)?Nb66YQ8hYamP($cB1}|-m(=H7CxRuY z_{SN_`ynpB@*d|xOTcu`YUMl+!^#p554fJ9p}!NQJRYKU;tD+km6m%!r4C`L*=31p zDndfVg=4bT>V#(koqEx#g5E~fCHt@TANV}(<>MbG0kcDcZR!xY<7d>aR0k=dZ9fnH zY}FmILF25$78l`eIv*o9q@LPaJut%786+x4tH1b+4So+q5ZbWDZa2Ns@a`xR z9?lYJgtd3tASex(2e_b)zkgZYNcR3ojRW@hz|b^*Zce+ibfhR75`{PraXMA@pAp{1 zb1{$FScd9L=_p)$XxIwe@J57Z{c`kCPALVm-l-@#pmBE{mu^8R}*={FmSY9Z!1oHhob9KE4~^P?^4FLyc-)de-f zHuIqsI|I^Ze3RM+YbcsG7(YRY1mguN+BSypErwqGCp^V=_6A}MuFAFOk!vRGKC3Zw z9Uub$cnlc)2oM}aQd0&!^djCHnedM+4*~*uI z;i{~pK?Bek7|;z#qzwK1+SOW8c-T>S4HT_3rpvjG?3nndD0|Ha@7(UX{Uc)4d|pbjJozu zwPd#b?ZU~KpO|+#TNTy*Q=X-QWy7|gKl|t8;NT=mbn^v{t73;pN4XHa_i&k5W`RZ0}BZj#{W z&kdZTw6+Tv7y|lV2RK+(4h|R?)Qpw7maCS6Jgc>i4gWo9NL`J0QYjR2XJf-;GigR>b42NOFJ3z;B12?+_m zv#B|+s<`BTg#Wn{AhUFJb>wAc_VDmv@?c|faJFCu@bK_3v#>I=vNHbhV07`acLjPf z+Pjed9poQ4;$|);&Q^}DRu1+gf8hd+9o$?6$jJUm^v~CS)M@AV&qVev|H0}HKW0y$ zBQt=Bh53I8Qc(E6huPWvkC85}s%HOU_CF?cQTK8*V^%eDad2}sG5h0A{y&xAYGwYv z;QaOEf34iqftJz6pY;E0KT%G^2l8C9X@iYHdw|^=97lMKUueh1BmAjd#q_czF-$R*nf-r7wq5o|7Vha>(qbmYyVK- zUm5>M!esh~3x^mRD+_>=Q-VVj002m^iSn>;i*bmsv2nAAit=!a{1f+Iq5qcj@50Lb zOP{}Bt^Nhe@&AJTx6prs{j2}*Dmh#I5eN9!2oePFGygZ+zxkV*@VZ*L+M4|vcZGrX{0%V?yre@|qH(OUSK@t<7p{;|3gP}7&^IrhAR{uHv z@7Dhe%g_8@EBL$DzfA&v1Npo4&lDpF|3}waDfCG zxgXV4ofcFF1TxaFBl1dA6a)~Xq+}7~F{9t$A*ze|>)`1;YY*x=@@P7Vsy(;Pix><% zNsbLsQG^Lq#MOm+No4lrsYn5dFIh)*RUZd0RR-nd<t*#UR><7a(e>!^{(fz@-)GABg zZ%4#0PAl9c^TV{eGt+M82m%-B6B83Be7wBZvS5hxLhwX$^^Qx$MzO<_*<3D*vA8j+ zK|W_D!KW6}_?F0F0)Egl z=`>eVa^?!gTu&ks#?Cm!lOg#!4gyk85UCLTr1&M-QJAS+#l*VNt2V4W0k1R>u{=o# zlENwn?0aMEbKHpi`7D-MnW6Q=sUoV>7u3=Im~ znP_2z@G{UnH!ATEgv^ijL_>Hw8Z3v7kd;p5Go(55Nl;UT3_%=OM2VH1onIL?RC>_c z1xr@38F%sKKw@#eO5mQ945QLF^HcqIjK)h8C4ATyGt{<^wWgMXMOj3x%*=gq_$>Y& zvU8(gCpc8pswkqR_8V5v2tLWNc)B@5$;ciNb0l#L6s=BjiFP!mbR~G?oT{a8M7*kh z9H#!t!s0^+P1IH3U^pra3X);gL^qhl9WR+-q}zN3q87Qu7WP@1zfX&25_dH>LKz!8 z672x`uc*RzFzVO#1pxt%;izzRitH<+{A$vSnQL{_pGj$HH`5C+s(BvPU%6D!$)8OV z(Lc3RDpq>MA&#~SAYKz68+6_$06u@}Xvg*&LY-getqw?(y%oJf*b%I;ql%$l5Wg}W zTOL*u$G#0WoZP*+DBO6Wi29MBU~)q+f78bzn_5Y3X=j(KBy?^)|ICXE?#EA`6H+No zD;`dz)4)76*xQ2c%3*M$77Ya-ZEIu0H-7czk+{QHQcKS|6fz+ue$;K~>0C(x)JYB3 zJOl_i8e(%D6>0FFa38YFNNO8(*Oua)4{rN%RFgo??Hc?qC#qWLEdlC(;=y&J+6%kbvpRQAf!<;sL^tG(y- z2sI2oKE8XXh^OFD4MVCwE2O^M;=r|UMzxc`Lx0)xGc~8f%Kb}*rA-b@91Q(WYxDx= zSP>?>NrM;?!SSv#_KbH&H^IZ;!0@-A!UXu;9pFBjb-Ss(+D;da`9@l;?w4_^3d4jv zX0V{ykvVyHfAIpkgsanOe*-bG0ZC7l9_tf%y7m3LYhJ((x{9C# zCB)DG3)z%O#@yB>r2^7j&+l(`D>|U@5CtTpTfwe_3cHeK8Hbcc@mOGg#w2rSe&<$s zwha&bj`+4Kx_4E}WsS(JsfEL>>U#B|kd8T~hoz`##_dS3|At%s;>yZP29A}@$a|pd z!!`*lN1Pc|D+&_y8t8fAG1}mH{OmUKrJ%E*GitvuE1WbWA?q~lwA|`jtq2_`YfsZK z+?D4t!fQ%_vB>4V)|RtVIIIlvBW}Lko;E{Q_o+&0s!a}d?s!G5@LL|SKffgPbM;yC zsH(xlzFIgoYvwCwRwv4!LQ4ieO>Ekpzi-fzNrs929V{1Oy6tky-9B}z!^VvAMq3{4 zWI9%%Ap4`)JT2iCf4oiB+T-EAoi1@$M96GUO_~b4gD~;+plhZ>=$9jDp@^0U3~aF= zt6wa=0|YDFoj|tmrNIE`=+O3jjX;rw& z=w&daTpD|)$fFqD=6FWCb;_sX5O_^s&Va=N7w4Vfq|3^G1Uv?I5rtG<^!pg~_Yly(Q?dNML9E>{!A)Z&S_}*M2I{ zN@|G;eEP=h>}+)Uqx30JY;}fkScR61dc?CB?{JBuhCrgj*H(3)!xw=5+gmQi+uP5R zHpwvtWJLwP)2bmnvU=P`y9Kw?QQM!Lp5?^S;Wf)`5hSy|h%B0&Hgc4_nv zXdEHQh&0_f*q*Qm_MsZ6LS8=hQSCt8dU+@|}Bqvkr5I z{NTMsmd_M5p1dLb;*d9W=vQA zg9ajScc)t?%Nct@E9+7-ftx)8_7QHkTngBF$!(A!x;akQdVg}BzvOng)eIaRR#%I> znji%(NuI`V4L3@zTjJsf-mcrDW?e2dcbUiwJ7ug{6^Kqw>br1tf!jFn@?Hbl!TU_K zxtdIaFv03KRmBp3;|&4uPU7Zx{zpcu!i*#}dJ(jqr*sgn3>ww z>(_pv*l3MX*cOD4W8Q%LYE0tY$f#Vm^>oDyeajAi{rB%~JW7(9+b;HfoAZdHk&$7u ze$5`}k8BVDLB)Oy(lC6^(@dSezABcpHFG1?2=gYTlT5-4JcSY?8z#v#emxO%3T&Z5 z6t8!%yn7A}g5oepym}gv^X?PSv*1=QPU=O<0vaKr?Ku=?C~e_=IE1#MN0?EN3Az!@sHh%y9eGs94L+$f;G!3&vDN@Knjo63jZ9B!HCc@vBg0tr?!o?#RmnhWI=SF*OpL49gPCh zY)U<1yoD-KYX5lOp1LEGOKxURSo+qa5z{f|`p5yb0i}gDD=pmI(9YgJql^fNPKRF``Kaj03P5 zwkx*=T4ti6Chvfgr%ucD5Ufo?e}&SD!zwF}Vf8!< z>Hp55UIMctm}jX3F<);A@jDy8M0E;76q>R6+W#_JqE@?0dB0f=z)NtG_oNjk1ENa+ z$lKS&9IIm@O4`q2U>&b{Yxt+?BdrCy)5H$bNEGBxn4eU9$IfPYRbG`@!=;49&3$Wi zuhU&*4gr0{#U)lq`vV zf@D=_`nm6Lp*MG9qeL|2nEjyCIbopD^v+ev=dZ4ZCWREFr>aA7>9H6Dde~B< z^c3LGts)p58!EV#BG=MCb@9fPOyqFfY@lR(*uaCvbeiU7pF_ zz&)U$9%elqhwK8GAwI0~MLs^Rf<}d@qEhm9A2IgB-5PE>$tQ|WWn`rTulkn_VbENS z`?P5?esL$}CdSDe8B1U=(9oJ8ABA+Da;GW*HTsjLHRRIrf{pT{d4Euo9WEwwsdo(G zqcD|e>%N6zLVSiC2q$(Ee`A58w%X%mL*cVU$I~M8*h&6jI2P)!J1#d1kD0sve0Sop zvqQ`-DRUM#CVO4tTqA$VBl9}o7&nzD`lVC!r?UKqEzbSbCuuPAJEZ6?0Z9$a!iog= z?YuU`D^*|V8Q$oDdk@0WMIU}{1FHOl$SW#;>x2$`YZE@C>m_|EgJ~%8np;3|3l4=#u!_~aUF?mrMEHz(b zvz(<;-E5=C#5@LkuDhqio!Zz~@ay7{WUQ>KlJlqb@ICKqW&F!c^1)N72ppk7b|Cuv4&E^@FB(f6_^@H+g{!m19pvPCVTB7qxL(z*kciX%qgAIy8z2 zF+gT<>Dp{TQx_J|UMqL>lnNIeBFs$KtN^7jG^b`{C|ZXUo{0J|4Wk0zTZ%SzrB5{? ze)_HUW3^kXcr}9zuk?4eJ2YxEB-WPgzLaxDJ115b=kIDP2m)A`Fhs|TnyAHlVxqCx zg$)bp`^V}to{8v)kNypuREn}ZDSdbao-~}S17D=ZwKWs$29#ggrtWw>&HLEdj88TN~Q zz6yr<&^!@z82CkZzMl-znDlwGKY^sK1aX*$1aO{_DvoTsf=n@L zeQL+giQ$cSr9jT~k#VOt;r<#dmfNyIXHhHyi~@Dsut*taQj_^U<44$CxBWD&Fmvj(%VzRe7p(z^Pc&yyF;@9m3J=dw!YPse+s?hjw-eL}I$( z!v2&+nn=wPA?1g!wp=aAP9!`8a-%tdxaw-pP~XEh2%R+WwsNj?ffyg$^p@lTME?$JtBCC#&rDVI;jGVxzYFkRXi zSPV>U1T2ff6d_E}*JU+KQ*NpEKEJ2+;*?0vDKdTFdNb5(mQ%i;C+uFT-$!fMA&udx zNJz>c3|iwg7jfD+$K#A$=~!u*CQeKYhw^+>voBEZ;KqgcM&p~5UgLRmkH*frHRB#` zK!`(UP6$RkdmyY}PQ*#h4zVdVdobK<-T!tyc0#c>UPw?tQxCs9&QPuxr{UP*oT1B9 zZp`S96*(TWL~OG+Q}i+4e0fIIFk`Kb^=~(2OBZ*?%>|%ZMoo4sq?Y9-5xZouhUS`x z7P5k$0LP%fz6ojhE&dea5LqSKDoy|v-nre36}Bm=|J+nzt> zRsn=en?r8{;=jGdM6prf<)_{!hRVi$n{vcoT%vQfmwM9F)?la#)FB#KOkxpCb!5xd zWm2_j8pE7AtSR=} zvCne=>2*B_u&!|CX{2JfK~5+!H7BDR7>Ca#sk7O+tzp|qC42v=n~rT%^#0hu(!aZd z5^=yDv^%^1YwWPE3lT-;L_Zbo)!1Yn{zZVG$5c1V0fzP=KJSsfy@<(MVaBspRr_## znlCoS6q2g{4Y9CIwRjcv!J$f`Vcqx#nf5AP5Jtf_Sxj^(YG+n>`#tWN~0a{2{++#LhX)<@ptORk2PRX<9-r82Ffi4s2<-z_YN;&pRUikTSeQ4m96Dz4d~ zZHz{0bALE$l(zLSGj5EPYJ@mdfQhHm+ofzbsh2qaom!@^WLa$kBmlpBee|Aq@%Hxi zT~M*^daycHc-SB5>ti{CG^`1@noZ(*c%x|*JLBHCg-u2YQZWI~kX+V-zTN?c3cZnq z@hD|Ig*@>d`wFh$NbGN$4sRdkg3L6LQJg)@TGA>AJRX<#?9fI$GYSeWBia7n78+1A z5^&~czrR=0cGMi#)7CSkc0`(g$6enowY|v{@AH(tNnJ{nXU-is0dWP-;}u_iOn47E z_t%co+~C%_%%@{lK{w7!nU)We?}*kEY8K0zVB7+kYeg>@tiLUzWg{6*%^?3cn_=qS z7Tb$op${&|`c_GG{;i|tCymWoS7+b7C%*!ZePLQDqFGQ++NX0yC*SX(CV$42c|uXn zj){2^-53r9rY+hW4!%dJ6m4b)Ks%wGrU0=s+K_WihxTH;Rbr`_PaK?*Z(VA4{O|scWL~pgw=g()qf?YmG+Y9*8uXMN{enmtD z(mBuG>F$x9P6TP8eHeuapzf0wIoO;s3+a)-<)K9Clu1J0CJV5fg^IQ$iJz;-?0EI1 zjp<4{f+LY8v08MTqY+&8t6{Pq7;>bvcF5$0{X+*i)en_XDHhVIm(bWhn86$-8v5yx zBcBIug*4$lLFUNr`WDP0o%yx{1#g=ei&aDxMtM*HGDBZR#n~GPxP{XBD##8)6o2#I zW$4e9T{S9UEu7p0r@>q5$m9&DI{cb)HqBTs{1&OTnDTUFJAjlRFx7=p` zGMkn*78b!APkF_!CoVPAHRb7=ABwiSb|f7>C*jb^D8)4r=ma5`#-I(|;WF=f^QDqW zs)z=TCpOflXDgMJEI9Q`GEq;3GrEB(&H2KZu8s}|IAK7xQ_uJrwQ6%rgtaM#Xr`BO##c`x+ff1K z2}fw7)4j)g{(fO&Oaqz&$cv3%c{tOxJuV#hh0s7fTlOQ|{DPlsT(P~z?2&$yG73Q8Qkd-& z7_0iWxyN>VY=FA+wPxtk+01cK0b_O-lxlsXZpGE1qu#AL{f#FK;#Uj#atHH=$bdbz z=$M(X{wi7;lMEVPaH`uLPY_qj_hx@c-L7VWP#3fJ+e$E0G2Xw$Ig4a zp03M{jCvDubOc=DotCsb!}^~b^Spx0NsQ`KE2x|x=xk}LHOO{Ym@#2yZwbO-O`AokU=p9 z3afnc*&0q;Un&kP1lxK+*19SsywGIuA_%PR$v4-BLbsZR34#f;7?r;`%RxVBJTTU! z5)Jyu2whi5NaQdU=gVib(h35y*;N!KJFB(nKR*ua;UsfVc}+%EkvNZEiEbf3lK6|Y}@qfO3oJV^2u{B`D~|( zVW!UYe!45Z05Zh3oH$CPJ1C;Ir=KJNO~{GAyqroJRVGlyloz&ckVdNnD5y>$mLa`l zQSjSf(6-SywvVnWGHtp>ICwIxREj~r1QyDP*{dxYv4!T!lbO;ahalUmb3JXA7*S95_s7nzfgNdu1IT`YUj#FFnpookc4z=f{EOa$zEdg~P?vCGcT`&J>A zp~x@C5tM%(;&QXhu{*%dpI``oCgUBivnlxspi)4gc|btdHwelo$#NiN=g>ZZ5qUPJ zm>M%1B8h9nlFi}AW<|A~dBt#*z`%}&=;q^nB`4+KN%@jMRav8q;FGROb>N6R^)W| zHzIqLe8WDRdOY$`-;$ZD9o>zVo5@WGnF?%9e~m+tr<6FS=8%-QItGy6WHD=9X_K)> z;ecp4`?HBkl8jU3sIg{?kNu^!%6z-|M6u<)jn0|k3eOCY!R=uMlxgD5NKX&z2|u%v z;i53lv#A)b?bAK)s7hZwDg_c+VjrK$MIJo_wP$Eo&MZx){YnXB!ah3oUS3ZMlBz^5 z_VeQB76H7j?9W#v z;7fi}oN<++LhmXiYd{XPPkw!K_}L;6lXgvUnjKvn@{XyA;hP+RI+mLgko*La3rj7|lC(0^m-e{76TpFf>ah@S zpOl+P;p0zV|FZLAt`BPNb|zuG1X5~!XaNzT+SM=tXg|$%J)-$FwxMEmI_G*|lQXZn zr^(-lM1tHtZq~aAYM1V+;n9rdCpw$BTtFymJxc%lz_>|Qx+wGM`H2@g)^A+I{S873 z6)q!rWsTxk+OPOZZuR;vQq5{U_A0ozyz*kNGg$ol_wVMTvy4tObYRN%phG$Bh0e;I1$8bj z0&rasM#t5OEv^;Al%zsv6yOmw*wQpRqiUWT6Qt8t_Zunj(5@BR%^(vqq`DA7TSmdp9S$Po*p7>eDCOZ3#kCP>aI?-vz@Kj$ zF@xFkZYA3bIy}^HUB@{QT(YZ3T4g&ojmPcp`x0sR3cs?LmGq@$2rL<>eVaxwq|GZ{ zU217Nvs&d_^Yh#If%g7+fI$4nRUj6Kl~T-(4kamhoj84s>QgMFlx?o!9(`AQj0`t! z-eU4CkY>X_05V}sOg3-<@)0u~hC-TRz9n(1d}h1alWQ}CiuRlhRuvPegnFFDBySf} z!ZHx(9~vai=U4P%q=+aQqtX_8V?;|Rg@L>sg~gi}(1|4xX^(j0!{08L-Dc1B7q+kx$uNTnT(Y!Xr9W;o{dtzedr8bLLcbnSmS(xL?#dN7h9% zonOQ-HzL1)8Yl6WFy+_>Pkua^>lL zIldJ$f83PLpyNydY+5R=q>m;>M+$G`-U!X72*kKCY-e0)S?B63eAlh;-~%$!bjkb* zDwJ(g^={mb`J_0XA+m=>OedSFmO!}3()N^-tZ2BrFz0eocDBNSYiA`b)b97vGVupu zb$Yr19#Kl;OjIWW{TaNd7@hdf| z$|4eMdGEI1bUSgD-qPvF{l?uFrZ|F`Hr z3&?djE7O-a^S-gEN*fm^hCV`1l)7>emxQQ8)tiu5EYVO!BBA*84YphT?N3=*>BM5T!HhCm*9%d@aCNPRNotr=F_1aj?Ee^@+#1-s_ zEDB`orsHh+hukJ53~I`IYP}wQm6nKX+ouvMu`zUdUmVwm88rexIc6c=J#E zXp>y%*rWMb`ezxM!Aj?fq-UpQq9wlcc7p}!(P~x$DlI&kl{!ZT9vi3BMx3X?<)CqX z`T+I%;itBO1SFP*P3UACCaAGhEA%VNXM}*p_=#m$9EmG_e`b|zSl^7Bslx=hK$K-? z9+bk67gbccQb4NoMZp1UrVGtOs(OjVXF+CX5WtTtGMFs#hrVpK47lAq2UbYi32?8x zQYCp(mFV$DarqIO@doK zv%>=Ka5=?;E}ZA*(mt{|l;-deyjAoy;iYOO6@Fw{@GPU#7#1Th*(EFWYyolDgAucg z$LE8A={-I9L~7*HdZVB8;hiigaj`^PN&ZNYFP7^>?)PQHhpW@=Bip~)b7`yNe@@qY zqV)`5iJfE+E-Y91H6nhk=+%Y;b;AS8q(zlJjTNrDX~vdnN1j?2Av-JMu%{G+{o42w zRfcb#jn)V)$W&R;V0)1}UDgtFb_8DY$k-bv1P8>&kYLG zq3O}8mQSFz0K#f2?UK_;WVLgStLGnYwaM~Xgko|`sa`_(-wY+~r1#jq>j=M6ArHb) zfs(`r&tC{@RjkbvjJoAFy*|ugEsm)3g&tYt>WOyNSvpbdH0o^58Af1U-{D) z>9fRf=vNQo5VSE_6RgklJN4tvIwn9|*J$zSWV><=ho1lnpTsI5;>e4{`+T&Rqb_E? zi7j^Tau`2{;BM_XgNQ9S<@ZD#TtoN2Fa24lRcN#5*TwJ3#(V`X^y~u68xuy=+>Jca5q;NEStr2}xB+cOTMLG}CwVS{2!+Z-?k zDf7?}pKvOJ)g=PLROOf?4`^3bki_zav+=&NxRve6XTH`zFh@uMIlD42k$tB2A`5&6 zA0@MuU{f`{kh2KjTLskBwDK7GI^z33$A!nw@@uyAThxb*1#(_)2Zt=Cgo7$Es4HSyotsy_VN%ni1z@P| zKL|_GnidLLoJOqU+AcUbf7ThF>&*DVkS{3{vewnbm8{vXVs~|CuCGt!Ldwk4WFp#@ z5$-M!elr)|?h;VKJe^dgUb(*>t4eqPF`K7&qL)zV=2dq?Z*T#M<2Rn&JmAMDLK%Mk zoi{0C##A_NvB~Vw5#*zLkd=4wRSN>}dI)0A_7mKIv_W1Y*?B(V>v-Jx8bRt2BCay zhjVYUg?A$C$pKAE?+)T3I}LD(p2UP9?wi zQE#&c1_E06#%c}Wl=*GnhVR9Dg9wJUzEFd8EA0Ce+orCvEd9gAdNMx9_wD(vbLEK} zJzsNDDwVSKi_-T}ju*_Q^rD*V+X)G}%IAGjL(0xzuTYz)Uj$?#pECR6d~(xey2#TS zEtMgzoAne|zh90=(oUiug~nm@XZJda@TTSsTL!e^KMs4;>92&5w`Y2^8tfIknZv=f_j zoI`f(Df4_T)JuUR6(G6v>XbffEU<+Za|{(6#v+9NVqURU-A^$3I3W`7X7SI)8q)w zZh+c(&QcLO-IR?{cv5|Dq=BrUijCJgdENDCm}qya)05RGm!v#iY9<>#MBDj_@^=k8BBV6LbnXY21#Zc@+XXx;F8P=K z@tRM2sHfK0s6mI9P2=U^bq|oZC57Zel~9uTc|MJM3P9n=^E|jbeA-TTSV9&45R{9J zt&87}_kq;a1z*wdl5Zd!V4&{F%HESg{cM#N#2)GJCtI8@`Y+?L#6p3S@7Rw_Yo!eV z0uOoJGkuX4tKZlB>KnXV%SN6l4C-Zuz+7;lE=^94)WmDay6zKJWBPLS(-m#L9w&**K6;+4wiWd{T5xFu=wH;4ms@}Q%;J% zJ2Mp0?-_L4oN@m%KtUedK2Po-Y@4!t5VY2wV#m3wTdxE;b4uof+v(-(0cm<(BTd|IX9LcKL{a z&6n}Iz1|yJZU(c23HgMU0Rmpmxa!;S@y&FTCyzA}y0;}&;7w{pcIx?=ehc++KlKtE z%tK6IHNWyidONz|TfFT?S~nE;c9YwEGmA?bni2keX(QbXx-+VNsK+uq2cJ;)wD;^+tCFZpg6mo#c(iu3mx$@-bQQtY4t zVa1KkW>zxijIc?7_|A$bpd@6|$EZ`&!iN9NogsZ9$%8l*Vzg7X-Lz=xIiK~6h=HYL^d(DA2F z0@@sPB}j&;I0c`lwr?%3gp|v2cLhnaCx}Wt?IwbIZQ4?TY&~v5h`+RfW=&3h3CL7? za|C+_>by{KrTdsjt@@Iy4v>74c6oz2bDNu>Dl%1QY&uiai{a0^q6l5scWwv1qA?FV zQ}zdZjHc3mw-S*x3)Y52fSgZJH_1s~Q#K&k0XrOsNGLzCMy$QoquCL0^wYjd0$&Se zYi^1_8?$z0La?tm52|5`0~|RWVeh!oI<6y>=`tv)=Hd>Pe&6a|?uXvZ^$A%`z^Fsf zxUDYace6VkI)5niQkU zj`x$z1J-x`@~B;yM~9rK9|pp(OP8wMxC>i)Jg&3nx*a`;_;KL!hEsRLonj6Z$@0E!!D=1^ zHc@Kj&FACoA79R!JX$#U-6$j2DOuhL#|axps+ryJE6B7r3^t_ba2UzXEV-k|bX|D? zX@2n_MOR7_*-uG-S50^ zEKF;N=HagxarVETjyAy#F1{Y^CD9-4e`mq(&vOl-!n`t-pU40V96GC5lyAtOT9wBP z;YgaUQB7xByFlnOKW}eOEsxm#I8w*p7&{F-zQSZbaD1M9X6kspYjU?>jp1-cDCrtHMYs;kch>=?NIe3$;DjRhsP6v;wYeCo(cR^}Vcd>O^CL@1l_0M~C9 zUz~a^4UH@mZ3}@!zD&c$Ac>!BO)RZuA=0hk&E#MryM56OYY9Sn3A-2}vznu?0tJ$bZ?&p@QM0AW zbFVjLuE+Zhf;!hqy-`#cyOK|l!1V`qWF2;9CLim_GqS}JD1MRRY6CMK zBGSvS$s)D9x1J{eFvZhJ9F7Arr9+xqhGAu`)Jj5uNF0h-`3yE;$>PTZ4hSqsW zt88XJ&9(ay+I^xqkIQzO_R2TY+KqmiOi~qjb*B59ecASpeO9by_M@7K1 zucD5DZ<5v4+sW$qns1CKWpuj~_@J|3Q>W0J2YCSb(Q>}to&TeajAFnu#`tI@qO1ws z3uG|lyaApL7kz`BYx&|UsDUQ-khP8iIBS9M-+&ajrB~=tr);NX_8VN!?c@OS%|GXn zb(5PG#GwlGQ!#0?c960lq?`2QS-{Cy<~Nzjd$1;9hnM#&@d^)>In(|5ougO!@iY1_ zF&SwIJiJf$PSj2Vgw(ma=j$zu#Jkrsp|+t-Op=VG_~`q=Wm-zcu}+RQk~`LvuC<$_ zo4*$95L~>F8W343tmByxMvGJP!{a{o#~{gl^c=rn9k@~pnA&W|Ik16cgXBI`CjByl z-Cb8-uVfaP_Swx6mvUl>R9=E8kt21ejn=F*%L+#|Nb0Pvb^^Xyvh2(ZTH~d^|6L0e z^@54deZA3omSB6ZCbQZ6`tFXUv`l$O8F81q7eMM@HMg|1 z&=^y^eWpwbQ%F9qiljw)HO-MfdJ}kmr5kjzeA%Vi?@T|&Q||r@gt{IrFOS0|)8Zl) zK_x`o^S;?499fMxgPE0johkQpSKdv0BJ zJoi*@{Mx}B*Hgah;d^ZSjS#BYA=R*dCiz;>whSu-V(svj-Uh#^ik)XK9dcoQfyI~+ zERLC^OJH4|E_T6G^_^yBCrt}#;V}G*eU8tT(i{ssuier1pdI0*z0<5WNcwXluzCs) ztpK};<@!tI%>j6r6z*P1g26xA{!J3xJj4Z7F*JO#WB3~KtkicP5ZHl zcPGW#%H#+vOdK$s>9y7grA}=emCxdn_y@%%e@yjl>f*4w+MLIhZQ?a5VIOfnOJ9_` zu1Nis0?vM+T)2zEU}ghTZzkSUdCwiooYs{izVt3re5aHz=eqhj{6+Ldf`(Hncbo;+ zpD&Nj+M3YrSjS*BW9Pxk_EOQ&TC^K*tGZ}%dLeeG+n8U?l4zlmGrqV=!As&0lrr%p zcn>PR@Km?*@_~G#=lF<}?e_@|&4<}l5+w>|?RGR`Qz+LK`XT8EJM)c&H^fy5=;dO% zF_p3SVp~k%hUQNH(hVFh>0NO@Xmsr6>Gf89bG2{%c35PG=g=qpB0N8(*f`LIR3@FS zD=@lgQjcJ>9F!7n5#2CUEL)_p$Zq+@od6>=H2-5r0|8rbLc2$B>b4gfP&60$g(wSv z#rU{o&N>fZ<09;b`C0KaVFuLCy&|rDepv*DE}GO38+D6_8iYGL>@5FA2C1MwKtM}n z**SiM*r+W6HRsUF+8cqgT3}^07|ffCyUWL$qJ>ynl>BN-3+Unati^dhv(8 zVL}%;s8$C8bHw{Q^@NthaI<~`1s6vyBORJf%1UU4BxD>D6bC_~x?Q;a&_Kx+l7)9Tu-P9ctuJ`Y-rVGKWYx zH1w2va4^mdTaNbB`a&D8r$oyQU~gF|bOJ;3!4J~=$ay&|JCWC0rtcedzwyQOoq)Nj z;tmwENZ&bFpn9~8E}+xG_hcQvIGdJK0=|=tQWFY9BX%l0m3Blzo>_(RniL~gh;i$! zOG&5iLv7;_J^ReHSGLL)3tb`oe1Y2f38>0I{r%(n5j9#(#C-yrmTzSql-0FX+wsBp z6Ak(GN?)#u;5+MRIro~>&;A6Kum1ERWxliWF!_QZHR^0BEl|f8 zQ=Fr5*NjQ{3&W22TM2E{OU^@GkxY(nG}|0cc4d~ZW6GWPqNAzDeP3S64&7~y_BA`5<-Q{UNk7fGtFrhj%l%#anqO`#NU+0g-woF9_0dxCcUgIpL`M= z)=x9%<{xnwZ;mKbK6*XrCh5zka<29ivNdKy%`Dl2?&Je4j+VD5EPkZdDiQLo3&>DNXvcJ!La&Ldk%i&kKA$xxX{7cRxKcAAomAK>;E>*X7VgYz&=!GdR)Ue(8kX!Y66Y@p8zWY)ct~X59hr=cbmRKo6jFhGQ1s+iG8nEgDl|#3iK9V zgZn7@x0<2~p?xQ`|ABbXT)@lqenAPx3dO{Xlvj9_t{|kMgbEiF8)Os`5k68r8IbZA zqp}$e;#oZED949rEyw$T$QAF#=*lE4#L1}WxBZqh&p`QFa`%t%qr4&IGMRlTTKKwf zQE48on>phYJEAXxzc{E8{T1jdMigHUp{q>%kRa2a8xnHh8!15qh;R!MWU+E|FQr)K zMoMSH?b+rl#V_3Dlw3@Ma43H<>@R+Qal&z+c$KANw6vDuETvWPR)*zpD$dZv;)`JK zu3dS&po~qy?_yxVilw?ogk%?S8834N1Gx8!LRvFU;bXBu3U_JQi4S|SG=DEv}SOFb?(Aqwi73v$`I72G$UV{ z`0bc)d$hF97@9R8l1)C4Tb+VKPy+RhjrU@Rhe7&DYQph`HH>&?Fsf^9yWpnT zCi-O<>WrS^F2M_+1ha4i63h03b?Bi7Eekmj1iaV4(ip`$nYJ000tzw3x7}hyInXyNljZ z>Zkiuby8jzBlhNxO%_9zvS%We5H#yB3^w{?xiFArAVsn%RnI$Bawdd`2=s3>?EsW2 zc`hi87;ok#MlHSitvj6^!DoIw0Wk@GtF!pG<-4SZ?aq_S=I8c{wUf0Q-yO+^C~lS* zZbPvG0*QqPDO4n}f|vl7OKy}vr`1RbG8&*+NTAVJAWX{8cM4QtR2p&w1ipXo5V(;= z!bE{^jSN>ENq|QU=+a_}5R)+ALvi4!G~u~YWI<$#5;!<+1Q8&fOEEN5j6k-*{)Ph( z#H0~*QbYkv3My0>AvBpH5?`Vigm^&=%m1a20eg^Ntxbi}@L!TsVbJ{q8K;4tqUPW* zeT05p972EN6Q3VN2S|uNRuF~gP2PG6$1fzG02u!J&P)T?R!B4D{vU@vf5Cdv_M^*M zb_5^(mVQ7`gi{@5oom(axV@wF2~A`NK<%iyAWugbx_(nkyF?lJ&cNkS3eB7Pb&e6ss+b(!U)gbcmd%FC2SFLI%PE+TyGJPkPfDA2m1 z^9(Ie7u=L{2@=RRK+^0eKLsTwgkxt`xIEFp7&w3dkjWqR zRxJ;4AJy=f%3O*y~U_~YbHW?7H-~t3b=J@&I(X@UJzeFTuT80wy8vrR; z(E=1`Q=v+Uf7Q=2ah# zDs{q&bg)P;GCydCfLF!0#2xmrjQ~Q35@Y4vxX(m0^dtBdL6k75fI&LvNHQGH(5=L# z@kQ;U{UF4PLZft4O{N9x2sINWtxUjT%QRa$k?KNNLMmL%&zcJ;87mu?YDCK;6q4Y{ zqE+VU6Xw7PHQMWu?N(oisylo zcqt>_H&w>1Q)}{%{nt(?I~CI=olhMqB5E{6|7?gLNUV~D>%Ij1(HToIx)X`gXR84p z(Z*|9r>BiZ4N!LGa+u1g$ai(_Obh^{L{#3EpF!_1{6T9vvL<-nA54G~SUWepw3`=P=> zX@E8g5hl?XJNrBmXpIzV&bMlBAg4VjB6M`6Vr;e+;+r2a&_TmU=Z~i9O9UcO2CPIG9w< zK`deD1!bi9#*9)ntJKa%tNJX?l#Garf_@sdRz}Eik^@+`!rEuhn(IVVI>R$S%c$>g zB6y}LYCaueJnNIXCnVB1K`fEKtV?^XWkU%|zm%9)=QvNVl@%BqrAp`gsKy;)hpK9z zK06tVSBRD=xO_H8qG>QS#w^ncgkofPtzvYom{x%!CFp=sVfvm8gI4`hdraNjqaBU*mqc>x`ljW#iF9&VVkazer>A5M1mX$rl7!PygxJyV<=dNlqtvYhS~L~^t^P|M)~rBF_Gy&ZlDS7hd3 zZIQbDEXO`&*F%m3!!Nju6jlrnSR)g_RLpbFWXemy8+_g3YWMy`@1083@Vs1A~GsZip&wD`-F1sJ}cBwt%6}QEzlO4tf}mLiE~#)_3fF zLbuVP#YJ1i2N?^Nw4&lRSIf`|{+i3oZoEA;#|9E`kb z5BuKXxn=jN+yEqC<6ah?rKQ61XIwNut`*owT%nFagT{WyXJg-cF+^noGa9FFc&~n` z)q|y@UN`zF(7neaj*|vk-30u7Ob{07Y)&fZjHjp)Xq;q*n|g-Z3;kPBXns5RN;a+7 zYy=P0zHV?Z@gxT;Oqipdv8A^F;XlWgj;_xVN-IzhN2@SC-u+h~=0@CzQgs&2}dH zP8#hRusM{5>7Cd-#Au~J@IW-k#&ke~f0(Flhv0`ELY)P9Vojiuki=l};O9q#N}&W1 z1chWl0EQfeDxI1G*c5=_7-2ZQ76~?LBf(||iFqefYy?KLl^skqEwJfS)`|A4TUv2J zsvFH62FKfZylyAvh?#M;i)%-;hCbe!Dcc!^UG9({Tc9oe;qaFG&t$Zs|2`^+460@; zPH6Ma*R-qY<^ZcK#~XO8=SVuoEuzIe>bO`DK)_R2Fz~QM3+;7(s{e|r-uJ-kR<9eo znfFn_!a%`*ZZ{uz;whzOmJq+jy^F>2IVZ~UHRznkZA!Np0!fY)Od1(nKiDepL-X$o zyXNEK4$89+h7DTB@iM+40WWmKSSoA@zZ284;61{&_pXs<4-ba2qpsTrmy+WoY?a6Q zf$_wkWWyh3g~FzGfcm2H8A8(Gb_?))6pS<+rLStzBN_=S8s{fQ6P>j^8Dey;_b@5vz0+ifJ*qJMM zk8FeM!P`v^1eLrHUPYb&fCe5erT1(Kmj(S5f=j%4b3XX#h}h`%V3wr27I$JB7?a0e zrGkmJ!u=ggoh%aEk7%@mD(CId1~s2gev^Covj1acCoEA5nWrye0y`&WZABk!(R?^o zHZnMB;w9pwU-&Yeip9%o{(W8NQ-HV)SJc~`%Q_3M zvA!99I|47~VPuNsuH_(5j6iBlnY<aohMl5W0FZj$Sc~y@x5lod-MCWHc(b zS{5C2L2{B!vx=o`a=~(O_?qGN%c?(us$T}s=?i*OZmIEvkF3&G<%|>Et_i!)lHz6h zX^iYc9bIP&@704etZc=>ITN79g~}oU@NPH0?&>o>ETQdxX$Fg7$j$AhYPKDnY~#g& zkuh|UI}C;3{}i59+O^x)pQtfb)0MS;4hMdw-IA*eYP}OK6lx2$ao|0nKi;bM# z?mxkM3yPeyr7LDuf}A$3KrJEKt|pxPrnHKEh2D5}U%oAkO$d$m$0(lKCQUe;Fez2!{!Xp=xy*;s|sXIhKo? zd%(AD60%sZ(zkEE8LkV{9c#D2!rOn6N*o2~3q`ZSfA25+^1)P8fgCGiM1X5;NgHIr zBf~bIANu6l9u0@jm)JOOt!PGbAcWTEjbm_KX;V^DgKXX<_bvdp(ta)q4#5@0#zCR5dfdRq6?eQYKMg#W^r(l0SVf4(xhNS-yG z$aOEqBU8WPJ*FKivdoO|q!RSnHLCGFHCi-_h%Puo+D00H3u_?+7zZ@tzqTqA7Mma|7srE^ z4qpQ_kauDSA}a5&UMFAJMkyM8P`cInBUGvaU4JuyDXV}ri%@DCDc)UX%hD^dT(|@V zf!l6|xfA;lY;0>s!Ik~us7R%a$zexi|1Lmuc6KnYGG0Wc7wWs=R`d$OFC%uvFfNv) zk*HaV4BhD>blnm^tRa(N<|exDz4I)nLhBtth=EP6qvLkiDheU_vp%EgaxLQ0+l|)R zOBpFsepTqUZ7U~mq{$&Kgri=CW@m#P^`_Hs-m1}fF_bFhhJo6?Z+S})!*7cf*ZD3o z9G?ZwJkYTKcyc!y>4`TQ`(01I_qLku=D4KL3qPF+DP%sx&XD>IOTgCiFD}FD2eH!C zg7T>2JR=v7nizc8mgL%Kspr{f%OsxkfP0*JpWAET)iTE&BO$7|-IsXzXAi{2j%KKs z1gCt@L!Wn}~dD28t>V8KXMT6lFGCc!H_000w2To`57fy!nNY z|K+!VNeXfJ^~-6Ke9lcL{QN^FB!~)%BG`@D5m|li4lUdUuL0)VmCqr*W_sc0AS7rSqA53NZcc@v>%i~9ZZjZC4Ct)~{i{Mk*YU(3JQJzl6_UeV zCwP67JTb!s;awBG$*F4Sv?)Ty6o(j0pMI_tN#Mc&WG@&#`kD?_%edP1Y8I341vBS@ zwEGyCBwbpC+N`*ualN?284cFfjd*4MNsuWAb^KtnOIhAuQffN$O}F~NH*!m!k9*j( z$(*IM_}a)R9o&iWbnOO8+grWB=uovpQXep!1yp1+G^Bh6yl>eXUyrI))ZxS~4x0o_ zGpH5f>{iU*d=z-p#rNU-z>f0R>%!mru^y$0YUxTC!*k(6;JvjHpemDkHB&^8C%q*n z=cD*gTEru6IAp%*dhi2h=PYsq$NkeP<0F(lAWZuuZ9N0xH(LJhalZavST&zl0;|0* z0z7dS(rf)Ow;j97_F&|Asx>lYGSfrA%u+a0!k(Ai^uEDuBINe+=oJZuJtX zHFDoo;vDYvD8BbNNlR>3C3^F*o2<8h(3{8;A>>^m=LIT~ngw)!h(|X&sp9 zzB!)}3Tmx}goL-0&f@-fx|s9WZ=}1S2uzzqBkUvTp`KPfdhu9d~PKN zZACzG`Jp$OAZwK7e0XEh7-)uMz+cwK>_vTbAaeU)<=|k_mo+UFMawG)1RwSFL92}b zc@H?5y3V{Z%JV9kPJ-f>kQs)LCG$XR2B5Ywv@tzb9y}&){0hZ_2LWOzo7gwr#^ZU( zhW)jZx!u1G`&j1+yl|3w{*Vn8s882xLDQ>4i5U(QyQ<)BrMzEhgT1`RG3?WhPXPLB zag7Z(Re2$K-SgwedZYb^P8Ny0>dkh))bpJht-Ikgb@|C>Hhhz z&mSF#h=UCaT3pfl402Oi|LjVu)J@=~e756QyWL30_ah)xo;#GefivWsCiLunSm};7 zYU&A5vwK=+v9>F7PzHo*TL z$?BTdkiy|tpC9Ho3qs@)hFuVT+SUiaO{iTHtt~BzyY7AOD)8;NM7YmnVC=~vZv?up z55LkF^@w;yVwRpn@qJllo>KN;bA&pi6H+UY7NA-<@(y=Zh_guzm%>%1u%c zL#TN4?aIyvQDxkBj7+52j1b73SGWYdxZ~XHL?Ukvwu2KeyVS|)$_l6rg5e{QU&*YO zLCx=W4{cY%#|iKwbhi2*WK=UecsPzwJ~6bOd1%~6+-lztp-=z$$7Rn8zpY1t2n4ZJ*e0=^Ukyl?7S$v7}+fB5sNB40KFezjVzy$Ieh&fu_LXck0<4Fj|_ z3e1k^X-pYn1U-lpJyR$|cO6^io+gIuhn}0{Doe$N4pcYp=icwCJ?<^OkYnS*bB+!Q zR1^lM?cdpR!n}9_j7Iba5h{kr;wc0*wS!lGy-3#h{<)68 zM#Ml%&2V3%#PO&6mb@l`FW7M+eCg+gx*r#q=0c$HG3FjPl<5mCt>NI(exG@A^GWD0 zQgylu$|`uLQih-$iYto7d}bVO3(p79T^G=+gdWs&0?CEEvEDr#IH*PB2d3X{cKnH% zzi#a?+U!NQ9TGE~Sp@HwW`SxmmfLr%$|i9lj}&lK5@tICT8)QkG$e7~GhSm)^*?ZO z^1d$;I5jTLy4Cp*nOICiO5`HVDi$m@pqU|l8XIC9&Xs;=YD|gtFJm}z(Vmjp%6?E+F_ zCUBdUF|3P+L}i-aq1loc0G91XMY*=48n*8{tu~`l6vF0zX{{3XVbl696w>|7`eE-GQxVP3vuEmkq{+s)} zDVpU^6?`BZyyaDYxVun5uD>t1;|fUDj6efD#o3%jags#_4c7 zvmTwqDPTKba z|IQEfOw@cgr(Pdun$_rkAsG>$6=vPV=l`#>^;_gBxN~P~(Gnj$Gl_4caVaL&h%BWv z2i5_fYCicS%6u%g9{J~h32K@;;?3Vvn6c26Zur5Rf=K* zmVN_DWErOgPc&u_KJ*tT5C0Wd8QcSUUhosw&c99nv&kY`w>(usrU3ZZrXRbaLHTr@ zpQrfkdJKa{gT^`IRA+xztjB{+dtu~=^U%;LWVsR6)^mh5Y!%{J$vl+;;cho^&7ev9 zJ4)>I1p;!X(zlL7g)c-^%0$fMs!4(cB1{5>V?@qPFHkSG!nOl>3?iSR6-7h+?(R=} zAIwZ=1*jw_hXU}aKYlpfr@aE;Z}i3;e}(KcM(#tbG1A!2en!_S2*sDWSM7UBE(}w0 zec&5Y6P8aq4GgS9V;OIV`eVtisp!QP*LPvGV*e(qwj-ylPh{oC_IW1czZd#KtZqg0 z910frL5Jn@Tn+NW)OPocsbW1q$@jt8-jmJybGQJmv5WwRv(;)m!3zL2o1EAXjBHgNH97eRL_ zZCZ_yLq?0Xx&&xc=^50_)+rywx-ZV1J(k93Z@hA@p_DE+cqj1ScA_Mzz;f&IA^{%7 z-&5`m?>_8{mQokcWO&B}33&<7aoerD=!-R#h4wKID9&>D%hd@T2L0&aEprH;rEY?Qa@DLnE0>vs|o!?m*)p@VHWKZUs zoeOru!NkeUwjMG$BDx+de)^UNp~Ik371CLU=KWmpYDYo=LqRiHm`dNB_+oA|Knenv z4pMx!+Zc1Z<2n9xvxO+lqKFS`tkK||Un1WhVeT)~dbY;Ugt<~;@HF?k zF_>MW?4;?0Dc({sG8EnkfR(ek!y#*>&}7S|BA*2_qraFp52{p_3mSt)%^>a?noz6@ zQHJf*oWSFrx?(D$|Im4EW8c{So)e?#N@^5_DUSw|Lcu60N)rev6j+llVfJAfljT8U zoLanhdl=>_&+1)M*+f_if2tMwhti$I4lg3U4+TdO?_w7(Gikh(sy^-l+G*!pD%?ia z0=Ia>7EAL1iRg2JnKkHy|5CNOc61Rveyjz47HqV!a!~1qxs!0#h%>QNgI+YSaA_mX z+pYt(H4b@uGuvaCHe<@zO;jQjFr86SIj@L__cLFcMP7*2*WX!?nG9ytS=LjV(XIan^LlzsqDK{d@?fga7b=Tu zkoC9&fsex%`ReEJeM584X$9a{?~|-0_eD^z9XZq&aowo5fmNLs68W+yyLVWXaAhUh zI0yc_AlS@n*w_@c?qV=%WbcJD$`36vic0lc4772qF-ZkQ*xnF5hLMb)BVhFrOPh=T zm!0@(t0WogJ%2wzc#uXcj;yL}MicJ+-Outb)M6axZSzYT$Njn}8PLDJ6%0*0miMgQ zKaQ0^ZfV0_kuL~of}9)oSK1_DyhQPIvhWc^?M8Si^1DPj!-pY0(}$5V(fQylLdFEbbYiT-CD3+C%v0|i` zLH*5J-w!hSx(gduo6=*P^2ap)m#!SIpWxgajO|(+CUb9Ct39Kz*>Z5ERDT99k%j+- zK_G0gn%hYe=>e;)_KOCEw|D8ctDENUW2Xn-lkJM*a&_i@`hjO@)QZ2g)ku~mvVD8IWR>?wy-CoK8D>oM=i=a`0VlTJmhoa6{V%t`T^p*N)2icOZx6z9rK9m7+`<$GTh4 z%d3%@x<&t12`%V_PCG4df-v2?Pv)A`duQ@~yU}jaVlgm{WmHC&P*X3?mrc*$n%Xzk zqQqr^^sAt?Dsq5bHzO{Zkojj+`yVfuvt@nNY|gt^{yToMS~=huhL79h+#CCp{WT~m zEmBYR7J}EBj>UbzGsEwd6117Ns$u;HWAf?Bn4z&FSV#*UU^X<7Da@KRX z777T#6io!lXmjsXuNac?`LJQqbcstFrfuERvbY91;%|GyMn?3HsCKkvEJJ1L9V|GrQ|Y@aTo&FGZlAaM7&3c#N0%2 z&E@_;HHGrTfzxo*gD%FHji*zF0?O`7oTZ#lc^Utxt&;@< zR74rsRA#|1S*ut@c-vt%N#+yL@wMfG!i}`&yr_E`I3}0~vQxZRG@%@I_I`Mv)xL2R zIZab!_#usKRV!xVf=}<_zP*r0wC-XOWU#h%rf#iKBghsYBq4P^d^ebtcRiWATYl4g zdY`h?@5RF15mVk5#->uB$?zRRe8>PP(8Uh@5&(Cu>ycq~8=_)(AMhvpy&cV=*WEJH z{4o68PPU3(&k=rkdQH>e4H2XQxvG*}X6|y*@OGBoaCmEB)b2(~`N&`ppi+{sKbP6EHqgB9aKVP-zR;NAeuW(v)j~H5G2)hN(NP!u!Q2=3O!NltwQMg^oEvsh zp%Rzp!>nAHl4Hu97D)t;ED$|Pz^Di&5kx2h z)ntbiwg#`#HFMzb1OGdJn4>fD*n7P%ctr)o!XjGGooCCRcbvwq%<;LT7n8j=Rt!%W zb3B>~w4W|aN|j!Tk~yqYU3-^0fo<{mq6epr{T=M8eA!I3ESkkei4Y-2*+nuBG=NG& zRq4(4kH_vg#at1u(MqRb+LJkD8Cbk-Sh>zSRG*>HmZV!5DH$!ttjLu1Tsqn-q0c4M-E3e-Acx>0wQ0yJg3}aauScRTTye~g;jWd(@jf~c)_6mc!(0=1i3tnLmxII!v-4fgP3+#NjtGb>j zo@GhBwCVnEpmABA^FhvKMbGqfc6@5@MdC1{`04UamIPs^gQREAFwi~AA8&i_*C8Mv zh0AHXlAb3c?Pd#qWrx(*g>XI=JN80;V)!1^9AM4qTXB7uXqao1q9YcyzMLw#(kUd-2V@7cNcbi-PAm@ljUxLo zN(mL=zjIeAEW$RDS*5zT6f|1hG+Uy!V9sto4hE)e0@0XB zfIvk>MbhOw%Nf#Hs>kRr<>2uBsHN3TCq&-1D~XNmo`rSsy>A?`!y#+zi99Zx4s$dw z6B#}$qMI7((MRQo{{EQWSd8N0+o=G9uC^cGQ1t}Ex|(XKmRU|X62xrqI9o8PtlM;h ztR@Y{usHh6b8QGS_(CFA6=zZn3Jdj2lrqW}rq{)~;HtFN1N*#!#y{g*ugP8P{E4Ki z<)cs_N~pg&AcH1FHX~&jJ9ZnPmjYy2bkr)0I_BmA_9rz0jPadVMVY>CA*AZ&U}=K6 zK|oe$eTJ|gBT}2P#2Wngu6gL8?*tIoR*{U1>b71wjW$~wQWV8HLsvo#rle z261sBSSDrDL0asnc?Mv=<)loY2^C+bZU{AGM7}#*sit}a6$vnUItWSY!J6)>Lv-=& z9c$jhe0QU}0CdY>rmIB7tItciJ%7)T%ZY-du)G`bU3G}|)C>%bRI@N#R3=$~3@AF7 zs1;(xlyj;$6K`ug+I4dw>8_WbWIJ;yr68-Q1FNb|!I#Q7r8`DdpjcF+aEpUzY#B=P z+mXz1FYQRv>>-<53;grMIgC!Vf0-^V#f`z<0L*#CD+6_N>y*cMIiP9J#4&Lf1pHZkm_N(fp{6T)>mPA|bz1vr1H; zFKa0P1;KH`LSVI63OPP%qUypM1h zhTIzvARUV&Mx*OX_{}^}{^2r;w0S?T`a4UBt?)s`FRclxv<~jjO@rhgyt|3%Cs;-RbQCJ(QqzXGhmh%KA)H)*cMsX4y+sj1=*4O;N~y{VM-emGD3BdH~~bT}SQyT~A!ecJ$4i?ikgw5U%bs61`k~XHPXy~ZUH@M`A{Bh~7qoF-P z3m)Vz&P%vtw5E)1$-{BbF^r+c4hzD*PA@r|9M(BZET~548N5ynkuE!NlA=nc!BVRk@qFb2n@QJ??vpj* z1ZW5;LR_5Az9IzHoT2809jmUY`}%wN30AL=U%h9WNGZVH6!JXycoFeTz#LP??!9)# zJ>Xcgggl{c6Q3N)UJFc9`<2-vk9uaat)ib%d_BnC-+%3==4Bu75T8(O{rzzIOu5g# zk9P9RyPNw9taP8CV3`2m*}!LIze9nuCQV>)WgYlFAlbR{$@%@$biPSqghS zp(iUGDPIg+NJjw59bb#-ba0NCTm^P1a9m78Z7@n`2S+$-5|mKvy_0od znD^ep+hU2JM!l}Orh_gy!X?Y@H^r9J{brwd5%O45iOypiM%A4V)Q*_`#|zIEGk6;s1A2&~qSWy1)KL-!14NG>s6!Is-X=5lh^j;P{IBa+eg%wS z%c?UJOp9x4wAitekw?uKKt|~oaW0WPR4JqM<>h56y9{S*vcaeV{}c8j1Y{je`DQ5k zG3S}d$# zTkrJxc06x;JKoQ5O|krX+jO4YyK3LTV##y<#lwA3(=3xUHgmxpG5vm89DlPHPQ0IG zQ%74nffx#j9>?SVwz0E+UR^Jtr=deltYYj3<<9N- zei|%4NecZ%?S0$2PCI?S&;MfJAoPE~^|+{QpWEDwfoJL{z#zWZ)xQ=%IBbZfM|S0q z_wv!?;pl6%fDNbK6dJRJTzWg%+4=f>zD>@57b~Os%JLILY{ z5X*JbTJI}Ay6OM*K}G`?wsv@0vpCpt-$~_( z<2|QoY{=>E?$&vld9|^z39YBssKXGH%i&fUoyQ~Ob^x7ro)$%$`fmyaem(qFIEdr> zlYkOF=yfvyEnZ;ocOhMD*XOHT=!3S+euwEIIlSe&uFc2aM>cG$!?#1?GxDBI_5pFC zB*8oP&-Yg{_+PG@j$^QMJkYN^-cPpDG;V)3BkzCl)l*;xU9#uDB0OF1#3Kv7fSyG3 z=NCS9p@sTq{hrv@_+rPh?R#nYefL7K#i(LqV+U;(W>|BwGxl~=krt#keco@!Ry!QY z4bVkC!)G@S2sPTSJfwm)U? z=_HNMX}`^ob`Njx0`$9$Hh>BAiANW_pJsnKV)l=G`_ruuvEJ=o-yx+9@$3FyX6bT% zpCfP_M!iGW8jEJ`r4cFR6WR+wpWEJMBH5R)9H{7ARE5jMI{ugIm|!skJj(rEw3#3w zfT`Dtg5Tu|W0dK-K^FPG>Z2+F7^^=$@5N{<#eDpKHE0Ve7P z1sIYcTOo{R&-#DfHxccWqjDdmsECo`^rNXeEz%{gm--2a);-JA^7w4-1pTp zCv{XDltiqGg9k;&hZzOH=dvSsx?Cf-NnTNfxC<#LcH;k+tb-PJWnEoehqoOTlW8); z`MTLekIOptjGhb=^e~eS(~?&t_kEv_^;1w4!iye3r~h&c`>JqR=%ydMi3PjpSm^y` zsAyo5(r`~=DznPn5?Xs9-n{h1&bvaR~5h(11N7L z)vQ#6*p=tnf(Qdr=ei`#iP(s@f2G#@i2Eszm*gXqx)$$NlpLZz%R_zraqmMHFEO;5 zsO(oA&SbZM^M~Q3yR0nFqYF6dt%$*>c_PTs>GlfpS#PldzFhlWdy}c+fiEZ=6+n_J za1*ca9>lPVCHCIuUPtUNOx=o`m5{l*x<(RMy28@YF^cBf`yAktQBQ`ZCFMISK)mjJ zK5=>G6(Wl=kEW$CVg`B_s4e#&annJV$@!^_uIPK$dLT~5;uMuq`uW{w??fmaRHtM> z?W`+Ti7Ok%3p{1O#17A&D>ayInx*Se8J0-QZ6?9C)L_s=l9Adh*Pw=}9Kf2hh1I)W zEJu#2aJK}RQgF$JC6Z#02uqdC%Q%o@(3r?{d37-ZbFY7gF3{EK_h^HLSPbWj#5*1q zWJBG8lCqr9j051f(8$sPw2x z=Q#R;u#K$tX%sitvv9Pi_@Vb9^q3QJOg%1Kg8RE9Sn^Jb1i{cNEyL9^3_smJW9juI zpBYYyMG@>JemT#Zq8?UJn;R|%^!8Zq&-jzte*#{cFYZXNq9!F6iZV5}i5&DoP}`7( zjhPEK>e)$6hiO%Qbfj~Fh+>kCo-83jPYOM1$*@V`#)>ah`F2kFY zgopWAuE3EYHR~bK4=y`D7Vn!(4LKvZ!2?cV2`Ng|cc_Z+8P zUEOT&lDi8Tg$dsa6>uh^y2i5_8r_m5Os^kyu7pbIVJoazeTnbV9T+ z7IuH1@9-$UmLHkpXqg?Q>`fiCOX3-ABJ-_I_ zEK0A~O^uztKK?vvI(+Lgr(-qq{7}$%aBWP%hr#ab1bGI!kU)iVX@58ML{vx*8FWsr zJfaL4NUO9p ziA^VTHm@$#Em^y?NicR!R)s~egUp_EVO5r*Gwm-1e}Ws)X;fX1S!#u(Bk7w{ap7vj zJGf3Dc!pDvU|>f%1u5qBvOul-mtUJ*$TrmVD%tzsr2!aj)410`k09~k{qhiTVS{^S zBK*Vl#ofae$7Rp+aouP|8|kmp0ov!=@4w_NH7yIFjidz(7;t{8uGTZhVGoUoNVd?{ z(=U?C4V$Jm)v8Pmj4=)P@b>!IZVOm~zn2Z^+ZXo8dh@9H)v^6t!GWp8?Ld^2n9Eh@~cH5@%GZlCrl@vovzF$t=rH$C(T|yIGcEi;1gB_4wTaecMWBmGv^-& zhiN?VPYy@wYO}@p^y!&-&*-dXJkCS?1mhg(_6@q-E;+bvRz`zE%OW=ZJo85G%6MNTHNT_SN@DW&8;= zEfhfY4>LKvc{>~dZSfv~@W<(pO?g_)R*Yb*mcqz=Vk(}MnXXy-T;%YfZ2FiA6*gQR z{DuPy5C5^wg@7WO0ne1}#jQ{y|A6XXV-*fd8lHb5O($XPH=6&b zYK-h6VbO_yqUgkxWv%dkXm!CV1|pu!f1-??qzz)!KfW4zq|99eEgnb8e`3mJ+%g`!zOv+lS1Vb-km-Zj(R2IU+|0Ra-q5}07XfU!-3v%vwUHnzdE z1J0?PLp$e$jVeh9k#mkBC{@lm*Zn=GfJVZvZ5ksmaC}^xDxI*;{_iJ0``J&9Z>jz7 z|LuSLw=1r=;=k9|RbR!wAMyXI|JQ%zGv|||&+^azarad%6Zx0z_K*M17x?G@o?kcZ zt}Cwif3B1N|I4SNKDp_NEB?PL>Z?ajy6@l>#~%KVN3KX--Y%scxQzFugmyA{8Sl$@ zUs^qIX}#@b`o4_!rPTwM*4tik^c_zo;>lzpnT#Z&;bbV13?&mWWtX3iZ^>vP8Hy*v z{DCtsrtOl$I~q^KVu>V3Mw7rAk0&CDcqE<(ClY*%B^eL=7mp`HF$Hf%!Ns&)a(D}% zLdo%DJe-V%60t~3K<7Vxi6vv~3i%6V$D@fDf4GeIAGrtOkwh{Qk{vn+&p5ck#^dru z83s{+PeCq(&i+S!+o5g0R$2>=q-UA7LPcm#x#{7wR6betoJ;`J+ z2}>90jrhK(B={wWx5)2A!sj)7XIeee(OGTnY8Y5`RsWi6EnDw$?t9eH|5d=THtIhd z57`s3u<&&Oo&Ob1AOw^8C3W-%8xEHo-pN=Z7LA1yes9m`j<#a!ifpqk*P_j}YV&QH zJexY-uFf;-3Qeo)Z98vw9eKrPY)kkKCL%ttkBLeTyAO%5N9hv@Apib>wo4B0L@a~~ z7D@&^{a+YY78teJ&J`;AiXxDksABpJq#6E-0-KYSVQ3%LhS(J;hf{v+R7H7ct<*RIX7>vOE_*{1evgFe%$$*`+) z83B%;ial*Ak2>F>{fn{vPrV)arVVqw`(BPYwkE<}$UKk;hmwJKGIAlm?U1%za`Y`) zSQK?U80`7frpq^X*h1EruRm(Wdp3BFy<^*YyP>Pr)SmBF7kihFHm|6#=nBj$(@gE@ zW=)Y%UvArcQ^c??5%$FsQ5>}Rb`od{2|Tt@(eHlsk;8bxWxR)WHaZ#R8x|LllQ6g; z`~&uYvBR_HQA_7&n3XO|jUW~V)k2sItFXKJ5^C3r!w`2^DIyPK19*sca;%JvR3VS5( z-sm~-g1NKViUy}CvFeJ=nCzNjo36m9DRZou8gq9gBOb1fTyWz1Qp{x&)1BwThp z5GHOjWmkTJ8H?CO4o3kTfa3x)(Qyf%#$rKeJ?4s9bb1Y!pae z##J@G-mme^z}GohSOvFJ0&-Xd!-nqZ54z;=rsIWnagN$L+}jt~_dnu4@K*TPmr;W* z>f93Z?~g@Xafzn|p=`WpqOTD_LH$x#fT-KT6(Vu~&4=v~o6frZ_5oe7b47_)lk3EF z)1GhC zslCLsvc%Mm_q5Qe1*AgTihMu5cS87{X8``rt<~tNgL_k_#CdoC9 zT@KYE%VE?q24dkpbMF^MeUU|-W6~G&>+|fI3}>6lwfX9V>p&oZJW!az@V@e?PrKyk zTjA$^lxv@9vt`G_u1$B?H%zg0HQ3rmIn@R3wj2k+P)(Lomu1(H5Rq-rrJGh}Tif&d zwOM`ZruceaM=g&hyd*z}k(LN5!5BodWIBpIAG7+8ylCjCw(GO4dNhL)hvrI~uFA3J zo@mI7!!D8Vo(jM)A765KE9w#wHxZ5a;-Q|n*BJHmMqJxM)}KO0U-ay}&(v9G(5psNFLmsFEbcxAMWgzOOpp7-$X0?diinQJLMHElSIxRK zmo`VTX!Kb&ZT>)Kv9ITSJU3Ao!};4TJ-n6RmE@l%V`y7pl9S~?h!Y--?wkk@gv@RB zJr7#eHQC!MJenf6I@6-b6uGX3SW6sjMV7Tw!sd>6Gys7UAc3VT20Kv})OPW@CjwT_ z&bv*T0tC6D9H14JTRNL!)~yJG^9T2(=U)R+QM(jw42edg0ledQxv{2+lthzW;FoAg z$Q83~a2|McV9glgihKfM_LU>7?IWyOm0MG4ZcDSQuZ;|R7mo%RD@r1j_!I&c z+LLdhd2;}O;(0wDzUEiYQQv_V23A)OXftf;T&FtAq|0`IWt+;>Im*}f4}3M4U1Yk& z$#{?9n+v1WM;J+L<~633)do$bxg*P}AK^kXXfFx%d?Ak!if7F54+CxZwu_&)i*imr zg0iK$nf?{TPkhqfsToGlDLzz$@}TY~*cJ8ojo+KM%(m#Vy(OzAK`;FYUZ0qV~ zfB*Zjs3VlXDaVacURXt@^USczedwv)m1TI}$eVCzOU&9UJ*#KLeHNZC&q62&j~Tx0 z;>SDHm(ssVKDwkBgSE0BKNh-%)Y$Q*vJb{v#s*_1lnCs1?OxR1US)+>>x-%e1Pq?Q2`3)-ADk2u)Cggop^7=c)K=C}LV+?5Ogz6<*XP7r<8roIls`=yWi2f+y_B zM2PTL_~0An_G-LxR(+8{5?zRe=i0YiA9EjwCUB^ssskc7WPBxwsl}sS*S%H z`?_21ntR*5_dM{}-=BWv&G$Z5cdYI)T44f%dst8t#>*{4xk`k>&hX*)?7A{ink?!< zqo&lVE41ly-Mg2><9>Pf( z*yx)s#+?<7qMV^Z#rN(z^ym}Msm3;Cl-7(atxGR&N>eqkXOuUwb1<@`F1@5KQ`Nwp zR$QA=TAyAru6RuI{QDl>uz5!)NPY{qlXMqlldKof{E+#9ZqM$AEbZA&E%cp>>TcE+ znK#@T^Bxk>AR3{t1343QEEWx0I*qGpUD^z*IuG})iFByl58$Si2&))?!?ay|c*9G@ z!ghKC-~XgsE`FET|ltHrt{pJi4YfVBJXkWr+MZr^sJyc!Lex)b8;=`s7Q6Rm~Zt z;98$vTA!h6NH1;3P&H+h*JqX0WtH(Mpk|c|WwWP%=W}FP<49P2Ic%Ude{}O*4?Vv3 z&{5H$g=FRS9PtlR=&!W@RBY~Sh(-=!3*d5rlgniiJcZpP zL&i`0wG}d_Nk{a(xUW6i-~C!5!pyYc+b%Y|p%t6Uz4Yl9^C}wBN*mJ4Thc(TtUj%@ zHnY4gv#d5#Rhv;#lRv7tpsFRitiGVCH5f)QGcwr4|Pt&@bOK^HW-W?I@0s{`=8ZLxP~(` zRE<}bHjPv@BM7dVI(y^Bjhvfu*ZfS&LO$!RznOG@vNNq*t#Yb!ZS567!;gwv$E&2L z;(LKJ_(_HG418tPRatadjvqOh$!R%ZcT4jyipd_zjj zPD$m=@}~5%Ce^q}k3RW=%Lk9;^y3>hRF1$!W|CqthvIRQ)ARUq|0u1Vps@G)>XvD1 zH*V!Lxc7E09g!4*hQQ6N`hn%+R@ldqSLYM7`R+x+pD-xK9d;mswG`02RtQ#@pLY`osp zR^Zg8_3256%IaHNp9q+hX9dKkF2j;|!-ubxk&+)n6h)2)f1`SjbKqhJ2; zc*6`O8ktD$>poW5Fa@$cX=rhrU)lWB^RL-lekBHqT@(}rTiNJ;{uQ6Pg1txs(Z$3K z8j3_WZQuRz7vFfie#XVguK!t<&h4`(An>jSp2qDKI+rD!iGsOvcDi( zJafYDPuZv*@8hB@Gc?(LAE0ZdCo`jhnWKJU-`}ux!c&L`{05p13DdSn_S!LVM?!uw|8k_h9}U z3%a-7Xr&yKIN|ZQWxKXN6pz~R(O@O=J_^@ASB?I|<4yON?XEeuEflwTd2?1}t1xp_ z!;Cq196L6EE`F|h?&N(Hl}vIMT?Ym_ik#%MX!4!fVu!vYVEi6!kx=-_bLAi&a__ju ztSNAj^sX(!598c=XCi7s@F=qkMaMgL+pyxzI9EcylFmaPq!eGIFN%toK%yV<7d&C; zEb=IQgY4y)f=3I@a#8PC(XJE{XDb0IitK0;?689E`#Yi9(M2k`+zpvtd z(*TR$9FEap#am3>s3+k!5-^Cs!NSf&#EhjXQcLVr1R*}^0Dh2g+@FZ?*>O`qfH{YL0pNc@y79(C=1>By>Li++?vuj*T=>fL-#JbaXqN%~Zb^3CV6 z&BKa!GVTucyk}c`tz+FarnOhux@K5+J{|M-k*2`NW-#!=2r=S{Vb?y>rkU2Y(;Qtb zmbK$;Yi5OQtEDvRX?W8+Jp;zNiL-Jln#3Wh*kj;**8_{iOgfi+cPie(jYB3=J9rZZ zqxu8M$T_ZKU#DCui8yvyJ1TtITv9wyWew_zpjktnH)jp% z3gzq`gQ;=K^%!Gfj}ZwittG}a?~eOiof+sZnxnkl<^mX1FgwIc)l@6ju#*(zKUGuxfBZ@E{T zYXW&~Rz)N9ea#KG_VycwjlK~{aqmHMdoI~+=;u~-0rL*M{XfFCP6rP!8+r_?JI-E2 zn7DhD*H_#!Lq_!JSD)1{!;5!3={x);J%XMhb;xzamUUBNPEvY7S=3X}*CdV+4f_w8 z*EN|nDx;P%$CS;;_3eKff)E-Qv;lY2;ql%1wAVFjU=pV7C$7d-?SCKJ#gbEq5W1$EI=2fg6${AGCY zW_n)#muBjpw1HxVkiu$J$gxc_z9jm{3Q`=Rz=dN$qjB48yC%zwqN^{mtIIt*Zj;$s zgL{Lnq2ygS5_#d(H*(5qMyQ(7E1T2FLoKb(Q`LU>`M0>Y6lGteh4^JqtA4FJ6$L_y z?y>!*&W6P>4C68Xz^DBy|6-rh7h;Oi_E7)-XmoHzT zDIYgJ4YzrvWTN3{%Pp&$d;7uwS%jh;$cNZ^?7snTAdirkq;S^M=P&j?B+6Mq$DtSd z+y6{)KIoe~2-Aj1anFI2r256nhxL!ciZ}EUc5gN4M>#ai&&VW;n=}D`{}+-Ff*lRs z(mj5TP=1_8UNP%RZ0)&5O`2VsVO%*b;odt0ElvkXl%DRRJq;6QjD)3EkZxQHeUqA4 zFsi<5(>6?6VUo(JzeV+BJb)j`&0#94iBj>acnIDU-8FArVA5VmlA2AEi*MYp?XSEo zFZHX}59bNPino|k-ecC)6L2?L^tg^m_Q>-bdR!uxqOHPh1#d(i6;c8NKN{N0t@>=E zCd;AAFn1J(4Bsg-eo+62vV*%L8ol#@#Uo0_Wst=t$!Mr-O$5qre_*lA;pT>Ncd9JG zT!##qK{t~x3Wq_PiifGOi4XwGU)*GHg1BdsW$k3UCf%$rq!gct;xKz3SJp&QN^$?< z9ftFd!;N>`Z}x4y0b|UnCxs6^HrKp$4q_G4Q?diZ;KFR?D3);THLtFw*qKR-NtSC} znQ7mah6?j2aQxZwClyJ~&GnDH59)i~1AN*YF%HfB~f7mjUt{mqYj!LX=J!b3z& z8hnSL-xxmJ?%?5LZ@lwA89Ffs1wA&Wp<700VL(6^i`{KDz+-gw7Ca*Hv@NoY$` zHI5{5Rx+-#cEXp-)qw~TQsCN5<~j}s|D=vm5Hmc4*b~%IO>EnDc+RbNWfzZnVDXdg z0A4-@CT$m;nv<1LDh-sKrYQKkcHB>mF56GUz;YaIg#&BK6V7hLl+s$p)P=JB9QNn1 zZIbchKl+(VQ(*4+lZEgB)sA$d!j2uuq!%76bWchMRWYUW-HyZWQWD1@B9b|KBMmy0 zUtAi1CUzWhH!jw!pvET}|H_ps$F@#KVb31ba%J(j%!($NV5AyX)imjoufF$qd4!;o zIzM$GYzF8UX{kH-1A%Yax>}~sBfgqZGL9VfrO&=*vbmxWCXGpA9C*vQ@{b@d+Bm}8 zvEAC)Ks(KC<+hG&-p;h+#zYubDONXgz~zY-$TrM)Lzv;XZL77j%BssU!{GIlI?8i& zzeyI5=vYvZkWvUb;@_xao2k9bq0Mqjg`6v`%%|TtHxYKR-Y`r#y_vaJ^q|OdVa`IekWM0Ox9_-# zg=DxA2J$)vd)8lY^m>S-F_}FJ({jlJH#Z}GIvnrb^bFi`=L6ZQ8U#6+FX<(~)|62q?pGpe z*`?!hRJH8+6%B=zjo9ruWkSdJWHIAqI%#PgDHH@+nNm2Sg!yGHi~jzc*N?BCr?Js8 zisbrHHpN30wYU#HL;ViBF2mHGi&fz!deZ$0YuL)1Z&^S|^wEv5XjkOIk;o*MUqq~ggq+ZaS0?-I3W@X8XOyD*z}pbxv;PWl1gAsf! zZ(;5*G>sXrKrUG`@P?eK)<>Rt*=%MBT1KkKZwYh|-@|%S1WLrQy02?INe?#7pNt*3 z{n|2@CNCnjc%kkvFB?y~f1BHGm`1{|;?0{Ym4tl}=YV1JWG97?aI-Q+UY%!K)e;X| zAs>kmiUBQ7X8IK6%wu4+$V#{7e;Kq?FQo=%ws}=k+_VK^CLku~rHCBpr6M+NSQPPi zugS9L@n?$0wvH@ok!eH6XNbxi;Z*i|Dxn!yjZ6)emLyqZW!vn~2EyXH!hzSOmM@!-=htZ&1LH(e()d!!hG z*S_}=hc27kPSQiDhhWfE1TAV|n4BZ}Hl~jpQ|k;H-m&>+E6W0E3eD@v>1+sDbZZ|6ZX?Qa;FpP!~^tlf`Z|uyo6W?hsF}JZoX{K>~ zW7si#+2#MnP#RXefe}d1bP#@F%NkQx8RZErD75Ng6O}D@-W(4OKu-#Q{3}xou$NJQ zqvnkRU9EQ3hHNWEYjd;2L`R`(&y(?>g_D7V@C9=;d4Gr-rof27@Crs{8VTI^jdwn{ z?#2aWb(6rAS)n6K#-X~WD>I0h0nPyvOmjyy=2kY;jGuAGJ%4NOTIcl#L`-16F|%dx z=0od@ghIU^n>%Z$lkAqNYv>j^11k$r%PG$_`1A|w+pyvdgdl?t3gSzK9HwozAo*C@ zm8h7bJ>A$@6|t;fmKgC;Wm2`2z9M!EaQJ1NhhG>_XHpo=sS&>x9_zl2n$WR-CL=Dz z-v-9`PldE`tbn+P2K>1Zil)iT5rf69?Ogl(8z0=Va8b*&>x)J=WR;G|R*lK8tgUI8 zI&<#)2bV1S;PdY`Zre>+7cNvTM?nK@xzDK^={R9yhiUbAvK&~D*hWUOy41v4Wm|5G z2Rz}_;)bVQa3LKJE8dFX>K7Ws3Qr{cM!l;o&!8)2CcRU4rA=LE-h5{w+=KlBYsTY- zERR#}h7Lw2^E*6S=E_WZZN3q=tUljBoT9VFfB5BO$b=A=;Hoh4l#VYnFVra>F%iH* z^q?1s_<})~*K2dRY!0W@;j}wlF1N?)^9O>=9p!XIjELMhp$49YR>bg`W&Jf49feb~ zDZ7Pl5X z^gd`Slu8o>`4l3@4~~BYWN9E2jUtYVU($^@gMr7gMFKxU!WFc2+Sj%8>$A*y{0BMs z4~#gD2|kU?d0`lnbzMB@$Mt-N1{;T z!V-K^7vfHd89I%t>nwU!ex@W&DYKd{U}dGZ`?+Mu#(UGpaDOhcmqf{9txRP^{phuEzi0l*P_pNk-gth z5ir2jVdk=>KuH}yLb7=4v2Sd8JR5lE(oA}ms>tzbvyEi_ubt}dc_$X=R_dGzbNDxU zU!iGYbj_RFKDTYY-O^q~<`Sv1 zc3CPui^Me+J3^3~)>iuaUXF)N;!T%(DS97*M;d(@`$4@s=nJrlCLRw52ebqFDzkXk zGe{X?ff*Kj^Xy&Zi#iht7w$}{tRsH3^ ztUHm3q%i14+cw|kKl)O@{A1X?C&HSMG1O@?70GhM>?72PhR2eDwwQOH&!Ttrf8pJA zo4KQw*&|N!taSN?_ELkEmCQ06+B_zPx>giZmn&p$hY#T^7j%c}8-rf(Li=)_c{{cm z0W?C5$)MN0XNd`otgXPV{S$bbG-C!6OtO@Fwxu&yDTJC$l7*?G+Prp(ZOaYj9rxPyEOH)L;^}_Ed+13A zVrBaR^Okw$4bzQX4gDR|z0V^0?$Bi0bxas4V(lu1j-P6{E3yVU$JqDZ5%+COhCMU| zGfc+c1x?RG$J_bFTZju+tN1*kf&IQ6w@9rsU5Q1FH!ag4vv#Vy`(I3kJOo|vYz(oq zgi-T9RO;N}-uHy5y_W1q(qnAuBBt8gkrcRW^{iWvX=WL^_BCRZRZ(pW+#wNplCiU=IM=FI{VkO_G$8*no65Go874&>FB8NA9*(x_9mhPqzCI_ z!sBIy5mA#vrjY4-&-Po5E5{f#=??v0JemyGPt-#zv+2rgtmfWE92O@}7AsPj)mf|? z$b4R#uE?Tci2?#MWKc=n29}?+w6jbhCLd#tVjLy*u50{#-$nuhqEQg5k$FNOtte|^ z_+Q5RH}?P*GwxGFcq%Of!tq;wGOTN|t-udcV9;I3OclvjCX4ORvyp&F47NcV5Qv8a z2^^JKd5j%j$hOvc=vm{2s}1@pD_IFB^O_viAhM}}v(Vn2ZEL5lZ5Ffe%^FxeE)->z zQE}ts5u3NPmC0-&l)*IK;{L9?(<3TOtssvCJHh{4A zN26|PYs6jqSzX3=_$Aw}`)u28vTmMZ**M#@ex_yPO!KB016!^|iE|xzCEVW@w`_@p zx)V{GtY09b3AhSL9hHO>>pkrfpqd@kOK@I zm^Hb?Sm#KJfU=m4xP2}nxG%6m=@4NgK?PhY-GTrfkzmmBqhZr*leW^XM(u*D7ocXl zRN(3#%A*0yDCuOQar3?iT_~LSdb{Pmlx+&TfCu#y0Y$%ln zFSig0A)cMoC=u~}5p5S1eZy@jC9e=$%AXyUDV+B0uB~%z9V(pIR^3R#&qO3;4vLh$ zD>7`H=j;DG8tRY6$zBz4tLzF<3aI?zMUYE!8{{rI>_V}^-%%9gWWkqR85qhik=X*c zca+m)U|go};T-P(0gSu*QOhdLw$0N$$KHsz_DHd( z6sYiPL6z7v<3N_BfvgqwEkP|3DcF_24(H39|7TrT^nH4NKh;tvsRRiX}N7gGIFH$Ro6t25 zFJ^2nF|MBE*z=gb@8ghtbsRlB=3;&tNfAucmh@Mdks>z~Nv+IIx_I#&`eqk8-n3U# zH<=2Ii7iv}<6&>uvfjS)R^uuvNEDbTQ>@K&wC9?1IlX!ojA5BSba?#aIOcL4tE!yq zW;?gv7Io~12fO1DJ8Bu`+9kDJ_;@QJX4o-I7^0NCWArW(%z|Ot zChP8n)>UIn)Th>DIH?iTUSOf@TYDZ<&g3qqHs6g2r~QkAd55w#80ucWp|c!6+5jH) z?SD%I4~Z-GQrdnC-XHF@w4BHVIKO=(o)giR6=fQM%z?s|CBHNobVUuT>^mPGSXE}x z{mDc6QCq2XMG5P;quH^_gn>w_rqrP=qJBA(5uMs}J`w!36-AzH_a*&yksO!O_FJ&O zaQg%3CGnZ59-hA7(XY&#Z?mqcBm^!I7hQ&(87;aDV|xa#Yd~6r)>z#v^{ju@Ji9)zk*u6Md1Ie&CWbN`D`KN8m1BT8f7Vag{(W)(x1-sR` zR0DMDa?Io(GUZrZ=-hToEM%1mFqhKy$HrT6n4!|a{uG+bE0joz5Lcf&YzQBB|XWaG3b@;JZ>mt`N zwj^pMa|q-&m_7D)BoT;5ZLz@7uxn?)wlQS>(PvTz`c?$Zx>%4&FlU~)e#Wi5fG+sM z)3=gyBK}gLZh<_d&hbTlAnJyf91ftkT14$JmmE*<2|H9F7mIJ@xI6b9`bsZ@gCA47x~xU6Yo^EfY?PTJsAaAF_2TA z^4HV0)Q=}W31H&>U{^e*a`5Y0N-a-4bnp{?_KBjwixU+mI!RHaz=)168Bj_B%+Iw^ znlR{A@+^6()L;}N;OE=OC8>=kv7J1LA5R`1Kajg|0)I(0468!g^-KHETm07VzboFU zmp}1|K}0E>Uw+lpU!^~k9^s8SP%_yFkp1UkwCGtfEu5`}vgVTCOs4Ugk<9#X@NctjOAa5?C{#Q0}p<`8Z z6*q$~2%N_OIC*gDEu?YM!LmsM1vf!lG24m6;ch#A`N^wJ-iJ##mfN3zIrW++&pdgc zoF?~Co+DSCEEcFd_vDGn0gX@+Dy1>0R!Mt)X&-$38GQRa@#amakR*^De~Z+JJ^m>^ zuhSZqVgRY$Iq}IINKcmW&cSI{4Eli^rZ}-=pbFl?6;c~_7Zcg0)j z0$yKo6=3F)_bcvBit>&hWFkY~|ZCmsY= zsa{LnmXlBKTo6*^g#ac!d?LX~(wqeHpLt~JZmC<*NO`F8f`K6I7xtmo54qhhemzUPkH2c_t!Svl?(AB-@x@o)nsehF z6Q^D~rhfd?S#$2Z|KYd(`RR_mhr9tY2H9i~!@0Pap5gs%cGvn1TVH(TwVUTJoG|_R zs`?4jZ@A^2hyMQQw`~Ux_k`$&Y=Fv z`Ib!f^!D*~k3GI@@zSTBec|OJM~^A5kvd(u=EMh2rAIXSm5(lcYRTizJi2t*!%Lri zcPnS{S(`>tCTJUqH~YGzSQddax- zQp#P|rmNz>GMFTDAI(dOh4WIBoTkclQzNp;K4-8ao&SWwkSZPW~v6h)^R zeil_VOX|k7Uc2b==XdWpK>Jd|OX!>bVu`g|x8HQv{i<=T6i!bssn05JVNqJDaAr_C zSv77<^Yka5eYJnUC~AaUDap(VMN9T0^!Mg_AE#H;r+BcuSw` z+;_01Wtwu^hEWYu)^DcnzjE5?7%61%^h>Yv7E%V33a=H^24xNOTBTHb4b|GI{(N$u zGW)jE@9`UdIV-&B6kd#6Ggg%N;@jmcBPGjz3e1f(m!^-TfGrCR$=(PIxucq=-EiBk zeFvr3IQ5%?;deg$vby=|%;LJNQhJ=Xt!4QF%AID9Y8_F`3sdX5j~#S0eKWtY`nJ$s(b?Ku@}j$zDagE!qpV@A*t+QYIvDn>JlEwX;@@qN;@=L>I< zzf?Mkf3I$@tQ()98b{v=%*8dV89|R{mDch}$!hu$GNN8}(U|d{fBhrVxR`1C<=5XA zR=2YHP==~`WDx{SS?lW5szj7Gr@l2s?cozyP+nox+}rLs(r-XVQ!Y(Bv~*c|QB`gw zE7>%$mQhwk!-(QBxfMdAd}VbLN@Y__vv>3d2im+r77O%B;-t+BC`#v${lO zL-V9-UwQLGedmUa+xC3&_43&_E~spn3~ORFiY3cl^!S+okl3(wXZ=;vl{HA3vQ+G~ z6KBqO`qei&H*DRwYyT(T{xom?eX6l7(7kZZvij`O8WyB*dOSSTVt1=*T4YRsTuJrq z4=m~2u(G3WSlM4D8%@sBXdxo|0bHJhG%AcT`i; z)Y;2ke!XkMwhdc$fB4z=H!t{GY0U(9OipFfgeCI$D2NEop!@BufF^0J0Jh!-A`V7_v2UJ{_xeeKRUU8 z{_Rhv1Nj?&IZM1{HqgR{7H5^!KvHnjf{Oav?_RX;;1Ll?yt;_b1S;G2>W-!<*UrE9 zL6^r3skr_A1&fxXms6mcrDGcM%A0Px=aHiW1E|W(&nj=tEUlY+#{-9Z`^El&cx6Z=e1Y&c%QdZ6Uw`*~kC-iXDc=~CIi#W(^XFLX z(BYoEiu#ddO_}92MPnzVl{Tf9H;*i?TY5~Hv8L4*9BTw>U z&#_~yJTs!K8K#|IR@2tL#vf$b7>@)!DdH9k2VE|Y*m>MEk=%2jyJq}M6bLHdL&3A= z&EI$MC^*ZMU}>S?&Bo%T2M%^ytv00+V_n^!Vw2 z%|iO1nW}1^d+l8r8H&qDUis|gH+bLoz@u)DPwY%yfYc31$wz-sMwFsWNDKe1?X2)d zef{j)AE_=55^2SaGSWIS% z!DJh-IQz{Gr<;7Np}jP;&HwrLv&1|8__Hq}xxqWLtZ_`!wC-blj7s_qsuublvMEk% zsP66uAImLA-@&kHtZAL@4N^iIjDF@l7~ZeH|0(NAiWUtwE^9n;%oq;Ere1evW?3tW zy{uXT^%sw;sGT(Rx?30Azw`&qYJP{2Q4iD1eRNJ-T_?k$yA$Muwk4 z;zQba+Shm_8V22hswSQe@y(cf8}{L$qkZGAzBa9>8Wj&-_0{()WVFgH&lYdA@sirf ztut?$c{}VQ7nuw!3VdA%$h zi#}MpqiNs5qXyUoO1kG!j*@ASfw_im)97 zxOCv=o%^y>HH`Y4@`e#5)%sOk(MZq}3TryoHMC61EtlW3%Nw& z4Yx0R$mzx7mb_*D{VXU5KZigYC(V>4?L^kXe*YzN20iwaFOVw@K4SUf{tG$>G=bMxIm zf{B?~-q1SrI^hOXT#+0ltFRF$dRo(0Feu}U1p(Rgemy7s^KlcaylHuxPOa-`({-{?U1h^moC;W#x#cxW zo_G!|gx4H`Hx8VNhAB87<$Y4mlcQ5MMFuQ;=`~qYM`1#_qnpd?rc~BV9#waBRsFPz z`YCv-xJgz0#OGdq12Q}1;xjuuJG|r3=U;t0XViqW^46@%hLXzq9or8mYNW!}fLWn! z!C90Us_q0OG7?KZx$LzZ*l!v3YhB@(ru~PH(NT(@6m}rYP`(MRuoyyQ$ulniHVlnr zV)ILDt#+UIy7&vZ2U2@8>Gy~BA3UO4xq9)FFN|)w`bspxN+1`*{rY+HZ8q!6?|eL> zq(+8UNljIC>zegj!m(&59Aq*o&62hX3^bD3Uo=ku) zh?%HtLv|4*rM07*rkJfvPGi&#;^*+;vc{?FHt$qMJ$ywO$fv?2Od+xCMZp_V<`LK4 zbia1xx(%DQuG_d}^~SBMH*Q|LY0H|8Th?sYOkdHgQ?EXwPAj^j z_Ky3Pc*3mn!OYGOD;!8yqKQQ$;AnR`tPXpKxd6#z$EuBakPGX!meuD~Ha@iU8LvNx zxeGlhSpaesl!kX}^UmFM6Q`%A@Wzyyb35z-XiPI`!2n4pmZH zS$EaPUoNNOChsSb5vGr!lt`M*_FL|Hkbaa{h^j8Hvi_5=zQq)#B-{S(-aGGq^y4po zaQj@)I#(zrm=s6s=v-YON9ZvrHYasTgk%)6P?^WB z=tiePct*!(g*QD$L%;b>T~qW zbww4;iynKHH-Q&_^u@QOMAFKeG2GGBOGmdp_VjC;cI-W7F!h?OI7h$yPIJxdn{x;a z$tscf#p+jeCCZK@zR$y7LK{<@6W!zb@#r0lj-Pysk^6txvrB~PF`7UUPa?$ zPrhikQJ*{6zGkyTt1#Tl#4ohu=~o|l{JF(X4chN|@G+cyf>T8!VK>dHwL|d6IgdN} zvu}Rff9S~m?jw7dGV^$na6#;2ZqR(LDgSHj`&JoxCcG*O6}vLS_%>YVc0iu%b{Uw`M!o90iLIXAbG z{(?A~a#RykqsPDT_Pc?g-|TeFzi%t4idcz$H?tkpA2NqACyFg+yT*WG) zrFHi|x(sUymw?CPy?4>$$eoM|B2IM~B~5tdD^b>OzMV$4OrKLQrgda7%M-SwtC|bT zYu|ePZG^?$4?n@(I3M|a+g*zgI3j06+Wc#`xx6#yE+BTmf_E9E)ekIPHUw`Tnpx3M z)i9~Hb!zqa>0>9%8aH9asr%dSzRzmc$ZuSA61Gt_6pX#Px^-4v%k+i`Gpbv!sUAP0 z=Bk<19A7oFs%6@Zcm3^D2+!#Fza0Bt|8e*XK`(vkr@Kz%0t|kbnbb8$|~Wt zW5%`a*tr*$q{x_L_u->=EVw7XqB*k^5sa;XP$t4x2mSM0Zbds8is_{-lcvwxwr%IO zox5tUnlVCJ(pWUQ`TMpGMd}X~rriI7Prl45t)V5fwxD{F!C)k8zhmFQG0oE?$im}T z;uZ-YJ6ZuZOdTKpeEP*V0^!KA7ym(EY^17wBtbjrG&xOFLuF6BBcI%7;&}^*Oa6|; zP!PZUM*EVdUdBoyf+dEA2tIars#weE#pCiT8|U19@7BEsQD})mp_-5e(PuO*d;ZnZ z(M_nsLT_Axmql6anJ@=S{AQMon|1S@E7xxF%OY|}ZItS0AR1e)?VNezoq3hAt~Z2@ zR)8Fo&x+$Uz*T7;W1V%Jkm!wWx3PfF)!bcA=NEY30@=g89?v9{{fw{ecyqa z@ze0l@KT~X{^$Es&%Xv!dgi6qa>|I*i)r!;C-bMtvu{1U4Zrc1GoP*rd%TcNQv~AHxO96 zdE4KXJW)Sk+USOf730Q_s=KPZre#dyvG9iB4luY9d;xq58#lwa6id+YsRD8kC9)r}Jg;*V|q zmGkG_v2ZXBXYlQ;@D>|N(X1t!$l@^M<0%QyBS1Mz@RGD94spAn{t-hG zdgZU!eu&h~JN6KT;o5Dx_qlu&RORooD;xiCGEV?_c{sq)9IM^Ee)F!CYqoT*+uXH& z^WmO;mLB8`G9!^2p-6bZXgS(zKGJ9HA#UOFDi_PJ8QQywY`1&7-TkJ+eTJib{U)=S z5r7-CXTXF$3S&6dZ~BGZ?eWtg7MlY;akSU)3umVOWH2isy`k;+P2bNBZ|FxTkAW{X zviL(}?RKzdVONPo39na-T{>D^uGpfWARQzDEx}T9#*|ILN4isPDGY!gKGkJ#6jN>gm9dB+$`i@wM!s%l8Z@EIkt+b8%)4X5a<tO z&JOPsNP-N#Kri7WrK3_EshIEklc%MwlwUwMWy#BplvYU#IZ+0bT*DJ+EmpBK zTt1Iup*;3qA6zcHN|=$nBW4We{p*PWy1ep#y32F;FVzo9^M2vv)Sv$K@H0F*JG{^E zVV6%o?>%rC@AD49ur}Z_-ox76=lx-q@jmYm3~K`}<2|h1ecm5-8SnEB!LT;qGTy`5 z-RJ#bm+?OD5DaSrF5^9{-F@C4b{X&U4#BWC;4-;=le_Up;!#8C`fG ko_rbm3lWub*QS^93f^7vKmSMP6>Rl2E!AI-nZ5Y`2gx69q5uE@ literal 0 HcmV?d00001 diff --git a/images/square-logos/eta.png b/images/square-logos/eta.png new file mode 100644 index 0000000000000000000000000000000000000000..ec9ab9d5cb3a8f5bd0b6363fa548c40c19cde760 GIT binary patch literal 1237 zcmeAS@N?(olHy`uVBq!ia0vp^OM!SA2OE&wf7Ih9km4-xh%9Dc;1&X5#!GkW{xC4G zw0pWZhE&{odv{~uV>gMmhqk>%E15pTFwWEvlQrB^$gbxo`y;0DE#DKqDO$?OY9Xv@ z0S8wyUgXSoJFKXx)4)3I!DqKw*0LQ3_x<^wFMiG#!CTAtY`MaBUzBg$HAnTNMQQ{Q7bKZT;F+MLYlR zN#%HG*PQ*!KXfYp@8VxlPCwQk-B`DBRnhIQocnC==Q?vtR<8eg`)bnPB|&}tXLrwk zuG??wUD9`2xZdvkL#9baZyUDDKC#riIPKGdyYpsGxxD@P?&zStEv4c|pL}!4>rUKd z@3*~qUqrq$$NHUN-a*f9@6zAaI@##HC!glUdGk*Pr|Ug^uyfLsyBn=eOupRLXYFhB zq5G@a^%&1nd!oPlpVVK~8lP^oU^Tzq1^1l$fBy(L{aEw%j1xz_tIYrV0*8{Di(gg+ zJm+2SFRQtK;-@1=SFZY0ecERG%D}=!OYKi|IzK(&So9R=dgtGAj1wzAou9RT`jdHe z>p(UJf3N#`+VS*sZO@rKyP9`e&X9R}?T}|>XVA5i2c4gL1@$%W`kDVZ@2TTDmDEk1 zes3LX%vblHoGSjYMpd&t^RB+T)-s?g52{-}S9q$h(`fq9&y|5b``rJ{J}&Zp{g*lF z&Y@FJcewsl35wIbHX9f$*98up?L1$z%SH3OG|SIaAGy<-{&Fk(QdfL;_~0AplQwbl zw;8X0>auHYeg7}I+@k1p5YS?wq<2qRj3;gS4zwy!-EuzXMYn%1VkC_s*Q=zOR~D5e zO$rDuEiu1S^FBW@WL&-K&qv3pyLJHeJXE(VQq^Ri`e|vSz+TUp-s@DJ zeyIEuQ}lGr`s`V!A61^&ZQ#phc-!V@UvBgHS|63C3*NtdqLg~MJkd=#HFWCi6XhMh z>XKG$&lTJH{@ayli;s!tPknLvg6F!L>V2=5M#UAc3Z3^yO37^Ft^7B)Lch)yJNl$- z|C}r7-0_o&Y}Q)4a?fcu;rHg5rM>li-BO$B=jBbpCrz4i^Ql+Rw<+qv6H|h(`iF6O ztn?}~|NVYN>*PsOri0=mzV(!9_`FF*mFv!=dcOF+t@>$5=*{S-uPTDCY+JG>X0`C~ z`Ry&Qr|$HyJehE5!X~Su@Am(6e=<28@ZQ~$qi)hA7-P3-~yPcEJ^ zHM{Z=_au|>Q&awW?DYBbd-18MOMK?5pJw~MJb9_lT?yyk2>#r>xoPbmV!@^Rs5-{A@FPr%fKl45i7vROs1 U>~nkwETW?(XjHHpt*kfZz_n-O0^ezU^DzMOQD* zuDZ^t>iSNksFqB50)~ zlVmZ;;IYOF3s2qDOW$f+ZS(6K)Q)?-pY*mj3c=)jqw*l1!~YBmnF&eCa^A?Zp__dyo;U8ahrIl#-_Y zJqKB}>z6uAJWDa_^hsr4rD$b=jucFmxgff(u<3Bf1y1%TKD+%PDU5k&aGDJKi^9<2 zXE#`>Tl>~XZL6CzLP$%qQbrDRV?hFAoHgWxtDM;eMT$`NSsjF|KLDY~pY}2ZvfIA`JYWo9R?Vu3ndHKB`GM*Mng#Df zz;S4p1V%!r&{5w^f#TR_gm#c;O&Zd0%~J36DGM+N}zGsU75WawO}x!;Z`K zA(WqKznz201vr$j$Gd)u4q%EbzH10ri$Q;xDz(K@!JB4E1tPtprj)m6!U@={ePc~i zDnR1{AJXguaZRIDm;F3hMW#nmI$-IA?plaQ0%^l!P0zQV1y*AZeql|stJ+@$V2%P( z(%|7FsF}-^it|JCPJ38pui!>0k9pA0i-x65aiy+*XkaH`&nZH%JYv+uwUo4sH{@dw zN>CB$Nkb$!GdNy!$OO2ByUl2JuxF40=9pJnyDtq<2E;M(VG5Y$GO6I@Lr1@0&)G4I zIMrxQJ16rV{D6P|LV(FQ$ogp`Tgoj7rfz85bi>pYem&0My)U3GJU9v^X)g%21KPvM z;~fJHv^=(z6$OmHvOB-zOS7gilO!~d>7)g!0wdVJv!S(l!YG>_#qb6-@LZS6!Op>o z(gF>eQ*_`2pas|}u^r4%o;J6`jUVElQR5(s>K(v+VAM98q{iG-j8lY4dbq-2m ztiTvD5KaVCp@GM&1qB*PguDqhV(LAJ^sD|fa?Y1!AX3Th4xfXRjP&ejQX#7Jn36sy z6)Fce0f>lxG1TYSW`2dST_k;B!2Kftp{P6`$zlbrg_JS{7O}7;n%?(I(k1J6rR`1l zvfFLsL~LXtNL(}^qu>fD7=E#v@Wd$&9G z%xQ;WTU;TfSdTX99JL@|({4CcJ%tXHGAEW;Nw4@dlPnN{1td$3ITlnLy1?(LYlh3> zNDcE;U0CxaZW}|miO)GI;J>fEkKL2knmmW{Hknmak8BDWOwObXsFdJJO8Nbc8(u8& z5)gQEwewu*{B2i%SU#%otdl*5{kPE^QTfC9bmv5}e&0Z0Z8eq>9+keQ$q%Z7TxWF7 z?t6GnTi9K{)j-9#6~=}}I}|_!CWJ#0%90r29&=<5WUs->Js?uK$wi2Is}g%x07{gn z!YxxniAt2uj0kaej?w0WMr*SdN11bOD!NienG^@rSKqi>>Q`FA4i@l@xa?no;JVg7(K_{dw5UVIT#v7{#03c$0Z~Wi?@znViS>J3+S1dyAl*hW=sjdnWOi)6 zQ)rEGI8JIczK}0ZdPOd@>$j4#INK_k&>@(>jQvO~d4_wW5M@zN>R|4PJ!@|3g@r_@bm(cc>a zh|EO%WA!>sOOohw&SAK%w{Zf044FF($G!LS`eP$2rweoT+tJ}@BRx#3q$^xe- zi_FIt*b$JR!a|*Z8Tj0V@m7pf@?RvCHSq%>StSc;?VM95^^yyc+|)G`j9;b(V6boa zVEGG>P?SZBWADZBf7NoS67Vo_+kXK4j}p+ZaN=j^RK(1iNV99v(O9rlI0?P66Q`X6 zsbu2?8@YXWIr5BQ=*B_x`G>sJ3(nl9;&q2l%FjtjoCil%cMoF%^6|TP1S)WknBsN! zUIM!ERmpRK_aaSo4n7I@!z@B&(@iEdkwY&VAQn-ZwY@H86iAD;c+|Vn+;H89Yva## zJ?l!B6b41fv%I*RAef9BY6GVSchTNvXd&3L3sDsH3^U3!pagqTAW*6LK#tM_ZuNr> zy~Y?_$S9=5W^lYZye90YXe=u%$VrJZ0VM>s#@Z!zVH%)M&pX78xe%(5PQsWzF!08L z130l!kS4UqllP}QFxsWG%<}8$VP(uAN!Go_4-SgkCvwyWB%aUx{%Ycjg#`0A4H0?H;5%uz#ovq zFJ#UN&7o&0T(OA+(}V)Zw&Mf7V!%}Kgqd5ayO{NQ-T&UC04Ad)PqeRcB@dM=>$s$;<&Y z1A{Nj(NfL4!yCUSQOF|MBV3!fys`O)o5$p_OU8-eLf;smaMM%nI@gFabrKA8uB8Ry z>M|8G=Z3&tWr~QAIxwY&r433XyViuJLNV4>-1ep3*qxNIblBe_$FeM>iZ9i19c8rO zU+A(WG($!ekM+YMi@^IpMpo(RES@#{MIA0KcpV~8U134Lg2H5`w565ghJTXQT26eN zKD{!jNMu;+kCK;f%Yeybxs>6F4E1>-E^$-7vQUd}f#>hvLW-#n7_e|?=;+~8oSV)M zBh46+g(F6;Ki(V*oHntCaY`%JYbm0deiZ9WO7HdsS9t7UvUcy%Q`YAJ)X!Z=w{GaD zP0?Pavtj4IQOaRyw3!^-B$tpE*s(R(z{mNuBOwWbq&Mr~W=`N3nvv9f5NtUV;kFPE zwuRtgNxqIoV9|tN8ZHX&CPwmI_1w)B%U~oF(}*JH(ezxW*{`#|G(tEIW(GSN6w2MM zn=RjXzj1$>h7@Olu?5koeIkP4VRWSVz%Ue?c7PfFV^X`V9-`#(Z{v_eGeCr$At#rb zo?g(#hJl;)Hu=xj#ZqVg5`OvdwtLi1j_l(j>wd zkV4o8#naTR+%5UMS2) zMOiF)QMp(zYd!MNnbCnEjrle$Eqy;2LwKM_+gQTJhEe~$sfm-@{x|M%o-eCI*C{(nH%IDYPzrf1=aKBb6WmKuU=@=V-FIn0wOA9(jN65aAxuZ_*W<%EBuO7c z4pHsCGFHv`k3`?|TvUuNumU_ST1Y{nv7cVDl_lbZB?h&aURIT7U_RO=;#k;~jVG)W9yFI+T z-uWCvsY?^7FZrx%WpecBM*66pRw{LrplqqG0Dfm9fCf_|| z_YLO;BNm^H5ESb+Tj60SUk77<-CAV3(MLOABXYAv(Vg;MLaCb~SYJ*1@?U5N<10m!7b z=@jhKerYJme4C|f<7&OUd5j$fZGi8&7u9GSS+tU|{iqay#|QTxkYGB3h5j3ec1ZJf{yBE;$B$WuuCcg?GHuN+ zRxJ9U3cO_R zl`M_+rL%fAa10UG^>7N~$*&7>Wup0jrX&mK`ujfyyS+@A?M$+w)Ga*W((z z^>*1qeNfW#`@RPKe(P7G<2OU+wZot^)gQ7L?3PqaN^jOp|CBZSg==yS=JlZySIGBq zcr5XWoh8ZOH>U>ke)$p0K+g+G!)NpC+J7=b#Om};PZ7S?i=X~pXVW9>8uU$I0bev4 zn_@n#Fi(+psKtMzJ^Xmx-$jI!of4K-b1%B&P|D04uC_k&5BI;~GRVuTcCRe22Alv} zB(YSDV%lP}cgu6P2b6Ke6z~hRmV(m01xi0u528XvsSWz%59xy=S1|xI3Hn#!`=1=eYv3@$pu8r~a(?Zg z(9=eAx{*Ipx925TVxN5YygXn0cqaGDO-qQW`Q$JAj*GT+rxiob-BFUd?<%{S8Q;T$ zbeet8W(1Dy4vs9M9E1!_EB`Iax;-F_SCE zRXbLI3NHoaP%tu>YH2zWg@-y7J{k)FLHm0Hxr+imSS6k=H=VDpO>HDZ-(oC~eQ0Bd zT#wEFv=FVfc#!?jnmwXLmD(DTi5q_E_SoC>{G7QhdmDE^?D`IF;qd(YHkHps;Mo7c zOUzpnFUv>U1Te5tEQ1mZ%#F^61gdOHSvlZOkhG7+kCZ-3OiJ^$OZZaegdTF0Q77<) zecOb?X25bzsxbbFG6*=7uB4I(CP$^*p~YIZOG%{%Igw*df$WL+Di4btR{ly_L4-ll z4#L5LVG+jj)&WQz%hLiA5e@0h8x_MATxvRTD(r z5ox<8Sl{!E+`ag6mnT+^BH%J0u@7WX*}I)&gJk}+jy_!^hXj7eI1iz%6iU2#c-bvT zh79=#go;D2n`>C@?8gfaG@~39JMS+pkm@n4R76e%EB={84H08Z1%+AulhEO@w}$73 zmAM`poRC#i*NMTy-zi1dIH$`}(Kk~;FJ8N$1WtOvKAh&s4mD-wvSD;|bRCbsZJqUo zMl<+b>0Q@l`64bieTy8o`v~2yoUj0$p=eo+C0CWrM|+*e1bd5?ObN-EkpTEQbauNX zYrl_a^q#lN?g#`pAfqmL&yh8o9VFb!UvQB|Q|0pZ%`5etoOwSF@ z1lyhU8G^gNKbXvxQ^}H!^ydF6^d^! z!~3UIv8Dn#uQ!}++$xH>aSjBYN3w4GookI4JEiGH;?-%>8&H8o_BceZ63gX7VB*?Ptvtbk#DkE9vXpbO&f~ z7&yZ9Uw3_NB&PiI%1;^TTW!{JROm5Ob=+X?j;197jL?(9LMPuac0kc5!HT6wZtBrR zk!!QU;Wmr;;54-Cglrf?-#3=T*TNtSZORY_#l6}{xXX?Y<)hyy z`NjiApl|VMB{O$k9QUHgh}*^?j#@oP41}Vy^ZS9sXrWRxNg95rlMJyEt;2GX_OJ|- zNq|*3ivm&>eeRIbHCcXourAiB4-|v^M1hoxYOfm8!BSP^UhSExU3kLJt^9<=KE!SZ zX!#FqF*%dY!?b=*V7XHE#Ng~-?>n-0`#o9%_vvLqVKX>h$F(SZ$3FOE1{LY6yvW^ zpuXKfqF6fC>FuJO<8QSQOL>xS|6zTw)6N`L&)b&%p6i4VE`;4}%!II#HqZZ2*FvKi z)Qv{Oi!JEsA3jpw8`g8KmNF9AM9J>KT2nECX^niGu}9I z{t=U3p<{P@2IP6&9ZgyG&KK|uHIkA!QyPUBIXNuOILlWr||`!kfB`7A(9D8!>|6sH4v}`}5V>Y~Lr(M!m!s zaU_sH<+ESkT_`EfmdrUw zFNV4MZPjF|xQ5i85l>7cXWZN~`@No)dzlXUE!AlhtCvQoi*qun-~`u||C>n-d}JXI zCgH)8`Qn=e1-2rkNh^m3{RI697$>p$+>BXPKOz(Y*5Po90FX5hZc&zs3){*Gd-YVb zhyoHV|G$-UXOZ*uil|LOZgF&Al2zhrtCz#<1T#&lOI=E@l}s^T2(iAe32@c>;ki_) zHsF_6F&8mUb2I()cOYCL4Nzqg^#gAn*=11=7%>DvSySh={Y9!lo~y%QiVAUaqXlUa zJ@!0tqSyG7Fht=Hvi~5TE2tggS5F>o-1M@!9RiTQA|2R?Ra>4Vi8zn*?<879il_>$ad^iDIzCTM=YYNb zRF-2Rso_N=b(NAn^xpj1o__(?J)ZCNOD{oHi#II9~hin*Yl zaO`@)Z%axcE7%~@PN-0qwA5Jy2g1J1)t-fBa2__^V^?@?>4)85#17HyQ-Cc$-Uej# zA|xd{6TQ+ac#*Wp<|Ix6ql)oowXbzj-m_s0oE`}L6v`MT!0EIfxld)v!@UrPa_ zfCQY8QdDMhVyDl3Q86$@Z+;DQSz6^-y}9`6b(QNWI!(T2AdIV()&W%8c}fci;V7mU zSX7xEmNI1BIW{LNRUATsXQsr1v`6O;4!TEUO^rjOo`f(CzV7tky64z-1=HyB?pM<} z?hm)GW5wo6w$%+5RLgK-qJ$@|HE;+R-^geRHrMX(5$QJUalqk(H$J7x6s*&8&i;x1 zk;@7j#P0-zdWBb=&`DvQ&-L6IcW@hg?>rr5y$~_djM^=hHhp^}i%5sCUyN6t1k8s5 z!r>~7+D%~*(7J+ItmE@Gu@2s|oN6~tFk`!*@CT7^z&HC8gEhL%bC2gD08j@baTB&g zIQr1#W_Je70&U?*ve(I+>CjC(guYlMe+w1v$aNgIsR_J6c6XE=eX|sE>h<|@JYV&n z>yeMe_^02)1Mr&#>6tkYzBymSMC1t;kN}}+{3^>oHOHf&gb&jLR1>@a0PzNF#xjwj=KZYbmL#+|`wfowW+m?A9!xk#Xh zu{)1|$9aSdMeZVHajA1pNe6Di%OzriEOULG?$ku9=p~navy>$?i#kO z*Wt9xEE>1r3iB^qC>QJPGFl$E+gGoNwI-bS`xA;+6C6K#pRp1^0{_+M1NX+2*6WWu3o=R!dd-i18x4@UbL$ql_HCI(Jgy zg1s2sSOSFM3{Ml z0px*5>5lfHLUNhM8wLk=axg;BZ z=`RV8%Z7qNDg9qB017J~8`x8icXp-=41t z3|QR=#X&9l$MUH=WyjyLI+R9zsQ(lotzIH&zpdK|Gg1;JlW1+@m(^O4r2NZ<&%>YF zVF8zWzU}Xa6RCi3@rV&KGqd05hz{CD-gLj)1j88;Nhq5~SeTjj?zCzR#`lYmiElP! z`KCVV4ZGY1E7XcHBSS(?Wcjnv;`C zcfLN-$~if}uQ`4{%3>&FUeegEZ;A+1{SKi1eB~oMe)@RDDoG^MA&$rS(Tk+F%iLJI0~+vdUCUclKCNhhk>CBc)vR zy~TMGrS_bNb?{6#sp{yAiJUE0YtO%(*u0i3iKbnWq2Mjb#-ld+fBqC0(KIlKd3`u5&ramOpmp`W2AC1am7$Kvdnnfke7aunldUn z>;3-hOe*XWVkzN6{i@M#BxLbD(h6Povvk`AI{`64iKQtrD~Jynw-P#-9_~MF7S<^=?qP^Erh+rtHx!yGMy=$BeZPL8O9~3U?@P;Iv<6O* zveCZDZ??W@a$vLOKNrs6tGc~Jk8V|EAAHhgCny9~gsH4N#HkR5L>(tqvLdj!PDITF ztmXc7Rn0;#jp;~0-%FEuGT#?;G&$~C29)iQve3a$wfu3xn(6ewImp}zv#$T=U7uR%uUgCUu$oDK!w3IfpmVu5y8u~3+kyLJ-P!J~^t z!~v&F(j9ON#{;i`@p7J3zOQUH`2-squBTn{ohrOrNj#JT3QowDs{Mt7=9El>jChyZ zZY1`dhqW;8w|?^xR0g$frZ7-q%DW|k8GmNf@+q-txgY`^wURA~amx#~a`pHjOm4W(jaio*cGp$7WyblWVwckYbI4 z{^CA|AF1ZX=v{U8tM-h=r!C;yZ^vBZvp@#z^)WfHWz*7%^lAl8=9<}lw3LUMqJ_jB zOOYZpcw(SC=tQ$Hr58W1a==k%vKMDs|GxR=cCtoNDBvEL^9?HtzH(Nz1hIl*asA`H zndncW=$8L2sZAj5YLAvYu+=4*-Lu#iq=`3dG&f<1y>Q|C(sSRNHZ;ml*cNAluf-xFyA=`-o zQ`cGLxPPEchfMUXf-F9*TsDgiA_>Wz(D!;^_zi1gI5%rN!DJ*Y8l87e>5AnbT5i3n z$myu)@P^E^?FC=z`*WucVm2h+Djhq2)|H(wUj(mGkF(yk6YaSrKHeu3p-!$4U-U{; z>%2=*AM~rJwV`SlqqCh|N+H&SPAQ}~g%-aPiV?R{Hl{WLVu*cAEliDfB*3eF~8s9ci(wIR-*0UGK*&F7^@^Q*jN&c+?45vM+ z6+uqwuWz0ERX+e#>V$0RPZLvb-By{-;f6@l_%NE3Q>>w`KGeITXMGp}w^V%wy`+uc zVxX{5L{5%hi_Kz~;R1%pJLvXS{bU~1Vx=nh(XhLis`Jl#7N~Zp1Xo#<)x^RBY>tUi zAPd5aZhv<5e9Zm9>n(PzkP{1ouZCCI^B?FUi_qOdB*%g(nR;zrveh@ zRp)>Fjeet5D+!nR&{W!vK0v@lc`QAS{E3az*uh$}=-w)7&0LjWz@tQhCg1gL>x$aO!s&H}6*G4xEg+IBWHv^1LIsSLu4FwH$L-IMSDw|6 zO>`sA`JEVi>i`FaB?o`r1nnJZC6`=4XI#{iTGPW}b_Ysea~T&}AO7~`Vd=7|ZJYS5{~`c>qN-{cH2{w?l$>f_gVu5wVJ74h;6RPv zV_Q@()9YMR8}*9QsS))pP#>+}6}7f)-z~l+#^5a(C1r#OVe#kZI?UzIqrn4%zdt+I zE7dv6duG@+5Ayp4L_`E?r{wtHDD;TP+>B2l<*vQud+Q@5nKaMXC$tp@wUYa|7ln>{ z&i%oVUy`56fv3ESH{$L$Q8vGqvi<&gTju^5`w5?*KK*wI@W~*id)a4{Wg_l`vqPP% zohY*BNW4#v*q8%SH0vTgEQQNzNNl2Ys%r1lZDHR&tPR4&oLYYLmD~ACZ2OvT5i zm)@n@{m)S_LPhxkBW2W zgyq|9FYjfPz$}Y>cosP=Q0hm7Ve3?W^z(S@KtI2h`@mmE_0y`m6{n9kP||OSgCnAn z=jH6WXK&%t_9tzFq1q}Ggq>9I)y)#}R$m%ult=oX^YB>~202AoJ&ZUi0R&Sj9&|-p zAbKj8S^>?d`UUk_`5)M)5lNf?Qwj$CAN2p3jlb}LSqoYu{aG{rJiJhHQp%Eb;wC}= E1EjDuO#lD@ literal 0 HcmV?d00001 diff --git a/images/square-logos/inwinstack.png b/images/square-logos/inwinstack.png new file mode 100644 index 0000000000000000000000000000000000000000..5a849db4da14abd9e3da40c7b21fde6da87f0b2f GIT binary patch literal 12665 zcmai5WmFqov<+SY6e#XciWe_lT!QoAEl%;`?poa4U5Z0-hm_*Qy|@-9cyQkI{dj*L zYh?!J&bjBDefQpXk}M+rQ-)xHNx>ix2un^@N)0%d0I$?nsKBwOU-l1hLbecB5(k0m z;xHbKQGmaxO=Q)SKp-!A5GWuR1iA+<1?+=Bt{foHp%Dlqm<|FFIb^r1egpRqmpMT_nur zT_oDTA3*^Q#3tkKl~c)0+Y+%t$CT<3-h^fR9SL~3o^@8m8w*g%Cgs4Dk75P^_y2o3TIIkDGD;Uk214>46~rV+=iv=L>qptl7H$fAnU7|0$89>LF!!;ki(G@u#wy_kM_9*E2v2D@=-?VDCP%4=a1OLbz=+ItMQ+u5 zZ=65txSqH-%sks4XSv#qe=y=c`(vb#U zrTi*=$$+8rvs z$q1#@K~P*d{po*7(s_5V3y+B;wcFt#d53mRyjq$WR{r9l{|=gzt%f~$C#y1VWy+VW zn9p+=%u-$6zUr0w&i94hK$_pyl*8dwp|t105D9X3QKp9+kE2d`mNhc~A0arNx-`&< zBk_nMku!N&IuvxuMm>#g^yhw9kRtS#ex{-*Rxs9$M4c{nT>tSjkLGEgbS#h@Up-@ObTjayEyX_(m{Tx$Hw3{z!Vnb{4ji$~&-J#VilnhWLhjUGSc~e*BjWeVXo*rl<*& zOp=WFe2e;8$V>Bt|8A}eUb>0fSe0z8RO?gvFfmux>$Fx23Fa=@7vi(3uUlSoQkrrK zYRJd)v?V~75zuSs^!W-yhz`+Fp6jNH$i+15!GHX#2`>u zKLGpm?iz9(@pqD~h;W#5bdxlLQjp2@OxX3-eQ>xrWNBr&U0JI@qBvr0x;W?NvbI~D z!V}vJ>KZ|X#x0yoi`GQ`_!Qe7mYf&PDMrog$}03wii~fbo-j7$?B+meT@XrrWN-9e zuoTE4VcfiUCT6>GfOGF{SvF!q&14SaE6hZ>DoMJvK9{6dMUmZdUWdhB(y0Cwi7ghE zoWse4MU9W>!SGqGm-Ex<@{l9a>gX{t$!w2f)D`lO)v}^&VwEhrc|L@fowXw83U%ct zkRr-h15rfmoT@=&qL!g(l^(C-SQa4L`v7~SG|mVu#1dq1Iv#PKW@70UhDV7L8^h^L zU&88!-PBwCTQGGg-`8R0|Cpj@$ry_;2DM{YZlKiPsQddO$^*dAK3Cq&f%O0lwC`s$ z(HU#ks~#siY;yA6CtE2{O=x1d*LO9C-q%%xSaov`SCI71SpX!3eM!hq?U>+A?NevY0oIfkFRzP{OZ9Pq_l0Xt!xI)S*dB^Vp|>tG@K((l7&kV zJ+ZR3y~zY9;fJ^vdN-DEpGLV88`x|QyLY2Emcs!zi+RdQu!fV`5_zL9YV`%ARMW;+Q%jo0jHwoWBGoRTqJ_vf_YKcJ zaL3CQjmMm$xs1@LxJA^p*JSl=K`Zh}Z1rfzw#!DNS)7|8n%?DTbns0hBwx1JmfWux*%51a@#0-1gf|Ei-^ter3I6-IdQ*N5w`-aGM$;CBI5 zVqk_p(PV~UF5AT0xN{N#%iJHt|24*h-j5y7K4mAV@?Nyzc!-*r(Ao%jJel7Cpu9X+WPxjVOpNi1Pq`T^}-FG*s&H|5mBC$US?} zn>jBFn+D(SAl?acZ>%PN|Crl!DKG$gM?=}L>rdkq&5HQw&QYs5yB+=Ob3#lSVbvTZ zA@hDQK{38?>chdym&bL?_OLvT)kT&%=ec|8v#okahdqeRA83e)b3i5He8r&S)^HN_ zY*+ofb7~{)HF#y*uV;D9HZrg{-jAv`Ibi8|tnF1M8j_FusRFs++8?;b=0Ug}BHy17 zd;)5)t2@}YV}I(!UgylLko*0q?C9pc{5Qww_@YgUjYtcv(fdE|nh`mM|JNENu7>zi zgp@H2RX4xLYP&K#+IhFgEd+ePj3~p*AO7_9!9c4RO-xymiAa@^CI-0-VNnoDxBKOmA=AH8SArH~*0nmgtqPB3umw7{mN+$Z4f6GFY?>ZPc>^1P|zgL)>U-481 zT-Hy)W$s>Bo9#pSDrR^mxj07V&=Uj;`j0wPyO`&nWtm^+nEyvd!z17-gv;;vr)ZV0 z05U$-?JCg;3h-DITZ)N>1_6=ln^c>vFu{1GEy7X<)-Q$>%=^K#xW$z4qlmA6 zs!aq97h!Kjy7sZgW&Kexf1=M|J${*fl@WCr;QE{Lq`P&Igvhq|aeVvXgl~qZhwY$3 zrF=NEiBN}W^KFQlFnV4`-EVwhMAWqkCo+}QbGvR-CkTb<}H)%0+y^&)QJFY5AWmH73Er&{@4VluhX{WubVTNch z+++_T@vLuNo|eaJkgM{T(Qdmr;E(d4q`dK4#S@K=$$^8s@o8a&5wK(yZn8HiRdFg4 z@5l`Y*eU2r|2w@a*^W32xRlpL$?_`M##*l!#NsA0e6Lb# zD-=Y3S%JL~Y2V265pf02`Tca%L=DaV)NmBE!B~d#BhT=Mz&1a;*`+w5Ny6Sxr7l;s zF=0BH*{pDX>~=IT`r!CL#q8#4P#QXv&a_}uFV(&mqpFMcR8hTL%ua)uhoGaO;-huK zhwoPozs$Jt%tefidWmH(Y^6*WW=a`}JVa0$z`{`8`i z^drtN-oX)O2XoqAd@uLT2N{vtY--+>(D#NSo~@>LUlGOBaKOdUsur62#` zv#M}B@9K?R(%4gIdUzQndJP7eqC#ujUN%PV0Q;8pCnO)d*0M4z#-ECTGe}6@R8Xp~ zH5&@2gVZITw9Je+n!IQ&CA6{-d-}Rstf3{o+*dal>EEf#ilU>+ZB!!VWWldqX&38T z*(qvN82o}A6NGnYnvXc>F>!s}YeJ>L7uifyoz`C`*vK(ZFQ#DqZWf4#yNR_#w&mwV zF)Tu0_5Fc-MS^xTaO1!QfoO2SfyL+-vgrO!%Id*Oyz zl=*Xh$fsWn6rd(gD9rAdw_i*`wnsoj)mB#8yH`b_chLX-;g3aBTn7|Z{JxJ-({I0v8Y740RC zV)VbM$p;RhD&xx!1V;bF=prCj&DiliVS4=#h<*P~L8SclH(JsPFAg;|f$NOR>^%Q_ z2(rqaol3D`Xe<^5p2Xqx$5ZO7F_EB-pfo&rN_N?wQ!}m|$r2juXXuZJ@HN&aHrt%Z zb^EPLbY|xC$RcU(>-f{mc!-OsKy=EAM_~?%T0TX zG6e$hV|8zlSv>@;HY#pOPi1P^?|05Z5cM3|#T35{66sA0A1ZLe(qY`;e` zCzB8sj(1x7dK6r}`iAD+&nj%wuV9|t0_)KKh$SKr200$KXI6xLL@e0-d#gF7WtMn? zw%^OxFvG3gWhsF25cmtnRehL7Q|>#-_RLOD=_T_S?3^b%(yeL!NaEnguIG9A1N^s? z50R7iivbRPHb=vULP193T$LYmMcEw-_HLdQ4eydTUm<@)@%id4I~?TGB&%J|a+2n| zR@N5APQw!2Sdg60e&Eb^2t_v@I*&R$Z5*~nLWq4u-Cj3#Euae~zQ*k(vx_e3sKg5RL%vUvOXwvadu&NC`Ly~WPL5rzV8Mancs>3jQr8ZQ2Trf-Q)7#Qr*g~UpZ+Z}(Ztmi z`kyIlFl0B~13b!Jgv@?}T7F~E+Q>faSoss4E=Iob_HTP;0PR>kH)7p?G=u`kR5;o9 z0UJM})H=0T3H^)_uQ7AS;rIT-hb{!`zAl?8Z1mqm1ze@7;N@ zoR$iioT2j#568*fd}0-A9op7&=(}C|^Pm>|kPVVToyFSXaCN=Dtji(gLHP7EKv`VT zT;Bqi{D;MCRTuNCa2^5&e8nV=3H&ORwsCdaEPcTiS@Kfi!uAYJ1#Ctc0flY{n~q-D zkocOR^glNw9f6qT`|@|;SoZ?mZ-+}W^$xph8I&$8AxSI^c=oF(KGbO}4k-~oL!cHy zu{Aw5YwC4*Fbx!SO$<&Hmy$cp5ml;mtNJ{{$mZy`r^=K94TWBHhViPc?(Qx^jqcz# z=t%*F8#BvR$IOmC9r#RxNKDix;9MAbJ$3wMMa?_ND#}(KqGSBGMadYp3XBkHIQOMD zlIRhC`?E7t`Z722$yd+wF65L|UM!D=GOmDQ#CG8-csl3V&~Qi)(Yv{Krz7QMeH3&h@poZ`8}=?>!2u9K+>CsPbZ`W_~hf z##;IpqZZl{wI4jTPQ?dKBMSfFF$I;adHSH=nhs7~uV=zFj{OIGTcoh>n2>WwT=()T zQ^zOYc5xT&uc9a99>pBEr#M^arWKkrP$zJ7LebqcxyaH; zX=GWFSV``dm8|$u-MC#53cf2e>*k~r%st$p1UT>Avnn6Y5sL2-;^i@+M>}&)vjwfj zCNEP%#olrUnObD#j*GvEz#j>lpOF{hKn}1TZmo4CcitFm$#EU5TL=uS*~mAkSxzQ& zw;MABl@73T5Wku8Ch6_+R`9=1(AFh5TE_ej^{3G$Kt<|ExslZOF4vCki)RJxb%NSrZe!xy4R{nr!MI;w#fph~V_fGR(BXSArG& zm4R|GcaOL4w_oiom|N&cM*kTlA3rcTN2aK9x{eS{3Av{r_hQp0D#l6L`@~0Ph!9}j z>L2jEi+!^lDak^gEHPUC*!uO*)^D?c>i(o)S}8vNP2OZfn#4%SwNhXaX0z$aI=x<) zMqZ(B(z_f^1p@wrFrugw)hU?~GG(vbWlP_RCVbY!kWC_4?f8Ags9dFQ@4WCxpi}qe z5%G=*gQHmjd(A{WX9I_Eng)|_P%*)~kIv!+NTpIyR|wjZbp+&t`Gvh5J*L4flez+a zKR%HVUMsCSOtVx5B2Cr>(|wE2m38O5=VKVlRYGDGj#s7)sdTMY6QhY+&DxN()oVX| zo9nYh?yw-PcChXt)sArMi&(V7JJ=B@sKwfqsHwo+KjJdIHi?2wMQ5Qd4;o$HC;XSaA(!_6R(pv)}3yyP2O2^1Z~Oo5&#LvH~Ez z@rsg!VV|L5dz164lPm-O`x_-b)|XLoBK}ZLdhHQSJHZ2anotI_-V}{=Hi*cn=_2!v z$Bk)xC@4hnwQ00VuOk-H!WkKLpT~SH0a}fO;w8;ZPVD)uR_x>U%XztQ-u7!af=UYtyoWyaSm7YGta($3NJaYA8oRVc0ssga;OJ$>zW#HBO|(Ln+U5*na!n zj;2pkK<3zJj!6;#m3BGQB@ZuqbMRGq3BduFOT}$+fg=g-yUm^-v_W=$t+*?Dhv~pn znfh#2ky-Rk=LJ}VEAcD8J;<&hSwymgnEO)aVXjKgy=?&T1M?yIlW-A8qX*>Ig|J05+lp;Yw3BQy(oh6uv{BTYqX|+ zQ`wVCXr?Ow4fzbt&cxmGMfa2S<&8N$1ofc$Z2wKE zeG!3+Xl@E3GlH4KouO9>dPGJ3>Jla2&9XP5#Rf1l-)Ik0xG|(ND$2?HOg$C%YVM=i zR>S3RjHhcj_x2zcuJo%(4q^06p2TA4nnv-t0}#Jw@_IQMygvvF!>v@MjbO2-Q# zr?6SDQ#`sAnq=Vlnt#(8o7{u&Y?_tD(4L(0{GF~Nrh2{@eY-l>T%IGd#T|Dbb59=* zYDSJxoEFN;PY)g2Qw7rVT>S4eWoJX5zDW3Cw`OW8peg7?k{@#cvkoK~>gykJ1+;t^ zWLIUAdHEZxh-I3+@!o#`GhI-*4kv*PnOMT?4T*(O`u|~~XoXIhRFI;PAK3SRy zdF)n9p+DImBl=iNW=zt|2kGVRTXxbZ2U_E=T6;x9wTJLGHe75eG;}-g-X^(W`9sAb zr~5-aJdj%BH5FhiR!lPD8ObPLvSeXOzi0EekuC1H2xbt4yh1*z3Qy)a@!3Zt$(kz8 zFu0?rp5%*CISW^&rVhnu61!=uDL;iC?nGS`12V}TA$g8w-%paIaZHgfCAV_$t2`>Dc@s zCCX^zdPHc$E7YNhM`CYko@9uDlu3;w;07oJTemgSLpY8M==ZJJmX1;B_R)Vn+uvb68yhhX zsXn+1w~KsJUsBd1Ehf1weD3`cMk^vk#D{XmL)mhnncm%_<$H)?o!5*{`}$|7Cz`0x zA6FKXwWi|Hw^|LB0$u_GJ&N<(-?NipPMwu~aECH`{TDcpm!VK{n>V@rqgix}yD(D& zaM>T@hp2;o`ERv>eUFLwH~HPJzV+R%Pi9aWsJ()IE-qkNi{8V7-t`yP)klUkz&z+1 z>SE62zmvkM#oA{>80prMibo595Mkg3Y%#Jmca*m)DCBiMzPg>WC`@!~*9|ZU0I5dy zGs%tU^EKVzKk(lHO+LQMc?zqqI}k^TMgs)h@eOFN%I!y`*Nn zCX$N@<6mGM6q}f4vPewL4wcQIyZdBbZNWktl99aqZ?J8jqrR~W=2F@S;oliWC75ZaYTZxXyfT%w=R=rDy-a&2Z ztv}|tKk?}!sB_6cTRkxbQi=$~G(nFH^z&?LEAPb1e`^fH#zR+adU!6`myW)gr?wj< z1s?^52n-T6FU)oJ%!V+Vc+2K{rC-9d)J{c5+kEBq%eb1uq-sUS`;&k~_G$kj9@}s7 z(>hb@FldCP>ZYiI4bgnM+S^>=E17Me#)Ub(py!ogt&K#W&`&^084|N(RT+})Z$QDE zfzlWhbY;7L>5VSu5V}e%O*-?d-F^$5hnc4SM?-=Y>w9}17|_O_;uc$wzjc|{{S zhacMT#|sOhGks&ou`-Lz7?R&%4*O1r8LC4XMV&p>=3Q8kX6m<23sH>6@z_Na*qBvF zbw4yL&}`p5w_|@2bXrGy^WtU2dR3~aSNmvKN9#7o;Yo?%L-@ow13C+)sPi{2Z0YV0 z*rPLu;?0S^3hGA9KI$s=u>l%wBWSGwASo6md;ybhJoPkm)%?xp40j|u!>!#w`A-fb-Qnmvj?4dRIg$0qF@Np9};J8vGkxdF+{c`0vFe%K>5RlUeDx91@8c zPysP@;P2sIbU%9MyDcSsX@Y|bVszcj$8P_2TA)o)VLt6J!m}+xsv(ee%aH02D9e5v zadgv9yVJ|%i<%Q^&*~pd$tUP{kpQ+a@O)E)dx}0QZ~d4}O$X_<61%HtU-h*+-#~wW ztnW#`pXoY3_k`5-t9R2GRT1&VyOUGxuB~ly{ewS@LbN>X>abk|=7Vw;2Ed4ol{D8M zjV-gbeonpK{Ag{|lfA9nh};=?vVIogV*lIWY=SDO2SH()0;0rCq^RcmV{@c^$*IYX2YJ`gSq zXf0+U`MlrL$R?jNQB(t-m^Z18M?=5b?hwYB3Cvey=WDg1>MWZ7R_0VD-$YhF+vN4)0|H-fUQ9XDu0 zl6{N6j|s*8)&3K^xJ>UK5}i!6OX1I%7T;C|2pw3iJL2{QQwSo)BCBk~e2$Z&5*tle zIg9})q#5LmNG(trBg~#XF>e4`aLpr<*g4GVY3iNNn*u-Qxb;y^{C%0Zh_xp1?A1jW2$0U3FTsFo?+pm{k9mi*~c}$K_{%-Mg*xGVdJWv_<-P16}(T(cMU_ z=Z*_#E*NMxcpwsF%I_S7@DVsn=r2F-y-injw7L0F)WymiJk%oL^Ga!iuj+?u1YZX$ z@5S|1Rshk3-@@OL;av%~K;7f;rw8=ZJ>x577nOS7L6~GE!Sa}i44oMOIHgUGxTQj1 z$WYwyXtn7ncf=QCi&V?L^9!2HY4Ae)!rE^KcUW^k!+;0ZT}SNi6Hf{3qk1eaB?y`l+1D7f+!q zIfQid$0@LGn{z5BnSPyc1o(+q<9R=M8N)fI%o`y1F^`PFwdbJofUKw}%;TdW{&LMP z{7;4%X}?|y4N%$MRE&CF(6Pr+`1e;OCl#tYj1LGqS zO`qH`KVpX^-=3f69v`;=aV*C(=jq*>rSQBQ$ZSy#qa3W%sRq6AzL-jY1sDD=j-1}7 zhqhKO*ezLpzq=p%2;DbRZ(KvojbA~YgWbl{@6k?)Y4Z>^MgfkBdJIAeXuV}5u^qgC zB3aZvv%M+6OB>eiZ6?j?@-pfUHkO3q*IAst>JB1e#>^?yYRr%-QB$VZyf;**0 zCS`+gwZ%S$^_}EOjn}b`F!M&Ca|xj@adXXg$W*q|3G2_l`UmL;`B9;T{1)x7PPZ2zG)22I%?Y?-ZX3O1KQzw0m$bY2U+G3H+hpKq#n z9g7JP#BL3yK!4Y{{@urZfRP_VSjXJZVVTj&dD)Pw)Ax$^%GA8o2?p?gUAhMT)g^dR zE72w%7H-x=dlOgvySgwwVsW3i+SJc)QxiTg62E37%Wl5~dunK&W?^+vx z;Hpi#Se{wG*C))s2Y+41uu@UdKM7baZ#gW2$Tj~3Az}v*fM3YGdN1lJVjgXoNk_mT zWRs^;tZpBi;O~UDUo{eJNehwQdE;IzqwO~pC7f~|x>9?U4@J3snC6nn^+rl@`*4v; z0HzAS@y%6gu#5-Qg)YrZ2K!KSy|xhifvl9D0$kYpg4MMAryV#B=A-`!}RFsX#M|O1j*v-v*7C%VQ1wYJso${czij;MwjcNn5;jIO^hBa3YT~ z%y4}%k!SM?so;}jH5EVm{D0m)z9zpNysusUuHd2BU7SV?kba%0u($avEaB;oMhpO! zxKP$GImJfRB(aG$b<;o~mO>G;vQ^TCGqgB0geAbfExTbD;Oh3FegGWuWXZ0UMF6e^ zAXO5u5=GWSCAWdl)_Uw2AenvD_&Yn5aM! zL3jkFc%Bf_547-al-R_ZN~Bvz@dW*tij^6^nI=c>v@^wAPR8)Kl0HmPR8EBN)mR*3ZA`c(g|304!t zBcg^h76qW2D8SawpVt@b$6Cp(%k;xvQ{Jk+ZgVjo^FAI+ldnPStkM5KlB7^ z_A_Rp$p`phE2%K)myq5bDfZGDV42W9M$8V!Kk*(W9KGrs`&IQ{E26Lwi58=epchIB z$gV|VvugQ5pwZWzL~#7dztbCzY?p>v-RjNT(g^4x3xI1CIRE`5d9n_uqwTkJ!*8*y@SQ$I5gEyDjI08>F~X{FtVN z;_a)RSXf$}p$L)66*B0;m!^iYJSTGcV-vRRxf(yZ3!a5j#ORqrXa|u7J!l8p3^KDi zfwyQZ_`F?L{|GaYPdDrTXDS>(B`!bb)m}ct=69?Nz8BL^Lx4k8S38$BwaV5c;Q#JK z9KKP+_)rKln-%r1qf#QW0W3zklo?L=fv^u?;yd+q-Zdb`V89W)3@KzWq7ou)ODf=> zpP4FxdDWi6y=$Cf+RmbCtEBqRQyEV}lzZ>LXfmX|dGmMV~%U&0;P z&Yyhg=KssWTJfz5kQo~MA>M{~&8q_&MKyRtS{1@ZVrwo+@T=`yUArdTV0CGnN}F=b zsx9z7x!d2Z@e2+5STBxxXK6B+J}x=LMmWQ)RMWIsYmhuiNSD*9bN)5SqHvV$|F1 z>S*b@=mvcNs0f+-t!kG9D5bW@dr%r6f-PD2JYTb(>iVM7Tb*|j<63OMpzcS+Y{1qj z?DXNaojvPh-*M~Ny@dKmzxwYyEl|>n`Vq(65Dg0a6OpSq&_NlAumv6XZ}2 zznHdsg3ANekGs)QlyIDx{8?a^XOebYL162IC-e)>&5-B0bwCop47Cq+apy9*Y>h&A zEcnkqfc<1ZK@@}H;F5bo+P-*PIz-#A>c7P^daqKXo~tEu6#(J2SeY6zyh(BQ_^NxJ zZ!|3%@TtNVj?k3^-9SRv^WIcN!jUjHU+bQbC53M@V4@jVjkm3Ecr_ps6aXbWKq?kM z|9O(|(aS)Os)Cztz`V6eg;*ah07az%&Mb;!T_(&r|KGzqFWO?3fLI_B21Nb&Y|9JX zlmzA|>Rd+y?&T|j)XxDJTLZttyWGzF@+K_8C}OIl;H{#7aR_TMb`nw;J698cCm>O7 zyCq!+!dacb-sB&;7RMNJ$~frRu0L*mM6tWZe}*AA0~!tkG|VBX@M)R&P--;*SRIl^=}mK$IPcfDffO+=1#>a*&jn}w zQ1`%S?Hd&&dk?%)b`S)(q%!GcFdtcm;E@L)5=&_&nzc|ofiw*cp^~UX*GAd{7C8&p zjJX`r2*B%rRjR@~9UDb5V$Al>4g)A+K1OJ13ko+OxmY~0v*35H9T88*%Aj^Vw8TsT zQ8->2N`V${fc?x9x2rFIp>lTHlJ4o%hAw~&UxCF&DG1^X?~KvX;PZ^RCUQ{`A-xp` z_CubvYjqNx=lBy%+$+lq_r@T<*Gv0;ZdCPap{l*!$+J zvjNS&%T2m*ru^e)0-=2Y#N-I@j8mt0-5$`DB`v_=G|R!CzkhwaB)n3*CW79%eMsy` z?I5f!TgUeKHFE@*m75T$050n|JD`4hj|1((Z!~A&4+QK_F$T+KkY6yby*#m_0sn@g z1krr7Z(+cU+`tBn&#`~-GprB-!(N(_onL#*AP=Gp$yMB)uf(E^j0tcQE*bYpq;c`h z{u+D&7%O$kRY$WK%{hlv07}*kEIqL-j_27j+xZC&Nh@Hm`?ojKfnKN{HT&9BWop>| Q{TGCsw6au literal 0 HcmV?d00001 diff --git a/images/square-logos/kumina.png b/images/square-logos/kumina.png new file mode 100644 index 0000000000000000000000000000000000000000..75f31ca8ef53c4b12c94b351652151d883c42fdd GIT binary patch literal 25027 zcmeI4byQSe_wYw75b2PRE@_5tq(MSb8X0DW?rsnX>5xvPr9tWX(hUMqlG2TUNXvWi zTRuGBhxM%A`+L{>53?4|?6c3_``*vl`S7AzfsqRd z-s5-T1ss^e?I5I1=4KYQyiNk-KmGCopD&u3$w_~9u`?AQ7rqEcsxGHMDr#v1C*@$` zWQ2mjU{Wp~CNKv#8!HC`DJu(@gPDbg8O*}S0_Fv?@UpUy{_!Fgya#;Zw}Bb(Dv3$_ z5e|?9$c^mmtazE39UUE+9NCyGZ48;gJUl$iEUe6|tc*YpMq6hKJBSmbg)PPJPX6>G z2DgRU7+cvHTUwA__=V_O+S>_`lV1e->-EQZnOpr8$insyb^t|YCx{g@n2Ck?pNwG8 zzih1RZOnci8Vt$|H-nqQE$nOoJMcf+Ss7W{S=t&|{)do%ssCeOU`*uX{)+vN$7^o> zkD+bt#2o+xe<1xwOIsCZD>$KM`4>+Yn*Vt0a3|yc zV0NMT-Rx&7{60ebz`62@+Q1=pmNqJumS%##XPLsE2uVdne`XcwBQ;|Sn5CmFEkE;R zlfTFNZ*Oohh#g!I$VW~_79JoORakgo>ma6+Jr`e1f_Ms6+^0{{dEj2#SR=V0ez{|Qk}?pO1Fb1h~GwZEtq!1eDF z3(OJU`~B@MguN(E=DjI&dJEb&B_KugRru&KzJa)9P?+OU!470pr&xb?rb{;4oC{y9TgL2PW{7xgDV{$~OHHKqMEV_n$%tYcmX z^kQxkgkF>f9LCT5FRgzc)?b!?m>K{3aQ=4)e)s>2`G1b#Xau*os=R)8^|RN1c4BL3 zVCM+2feRY~8S>v${_kG@)$Fo+e&&m{z|8pn!9dtT9RBZy;lHb@|G_Z)ry1)1-!S~0 z#ZV)Ng&`a!$oyvl|Jm^0jok0!`LC7rXSM!&CGuY^PP{-B0~rPRSv|n)@o#N^NB+IP z`b9OG+5ha%1x!0EEWj29c>Q7etNOC3#?_|3sxO=VZK-NuY$wS2N3=`LzgS(YpubP= z%D`q1F38Ej!N$SQe7X6ug@UmY+)Q1}7?_7`FXj_4=buKGHUDU-@rxw~faGVmpO(L9 z{;>RguzuN?{j9>@TY+sKu;XO@Ys>j>1^U-K_P_bz?^F8U^m@tHb&yLyt`@IxUCrl` z_8QkEAXkgmxUS}NNqdd!5|FFKYg||Jxum_ubqUDT;x(?T`CQUouNriwAZ*U0l8Yd#&tEHOWJE(mw;R?UgNr&&n4|Ou1i3!7O!z#&F7N# z8rLNtSBuxUuI6(|dyVT7kgLUOTvzkCq`k&<3CPvrHLk1qT+&|Sx&-8E@fz3Fd@gCP zaa{s(wRnx|YCe~=*SIbLxmvu&bv2($+G|{wfLtwJKKrU1ukk?BP=;$1HUj>2eSwNsQJrIaD4g`8&8Tq_b0tBLYDlI0g;`DhV z*)dLK_^4gzKz<^AeA@Nxfp`YRu%F(Iod>PzY2Y!AHq6^t93|8-Vy}%b70pV-#f#tA zMNdx*LdM~)$;rts-A8BEhW=BeQrHxo)BSrr@A=r4M~?bj`%Y30YAKM&Zc$TbqnA1( z%gQPy#KontgT;;Y^#cc6urmV!XrEvyM^RH}J(ByW1(X+BKzX4Blo#4Rm4NBdcGJCw z2$7)SS|aLfFXBW>G9Lp?+1JX5I(G(Nt2 zk-)xkB{aCb$DTo48b8M?Tg^^+>(YkmX!yi>521eG?k7^JzP`gBt78SM+lAb+rXOW_ z4%H~h#E8O8rUS9kTeD~P^L$i^Jf*1e5zU?<`S*&rM83RLvFUSAwi)i4u#-gwl z^8Ra(u|PxyMLW7TYX1X^!bXz_ycwY>>`dCO^!DHrdbbR!&z*9P0~;vsx2@=L>~E;Z zwVOmlP$KD!BIO~d4gv1nASlltXVy~%w-=!=e_(+q+?fv=Ix;fH=`J6!n#pwUcAkg*3!2^X;6=@T{zp_4CT13txsv}b=wZ?)n?K9rC9HM=NP&E2TiK{(qQqJ;x5cmO0sTDTxpS^+p7VnyMx-= z2R*#`to&c+1t<4*M^;KmYD!5=4>(J$WKclOE|j)!J{ty>vZb4Nckv2M*+7x<2!J7S zyMZDdBNhDqOJIaGUqNLuuKq(-hs4`8s+n|LrU;s2GGenY{Xsxk)#o%V_1?*k#rE1<~DP3 z`|{N)Y}b#b#uTS9lW#Ua^wk{HoQ`0kI3WB0)bFZg327U_hc5|A?^|R??`jIkF_MWu zh+>-qeFg;Dxfv>x{T?$u6ZOLFL|m5gTMDN{Vi9{p)K2c$9}1mLxQ9A=Qf1?I!*)~! z^pR?)H)opmbaChm^sOs|27N?U(c#%LHPy5*X@=sn&E|DA>oLX^5mJzjSJrAy!sDUa zB6nZ-SRAt65n=TXrSW3A)oG3Dao$MWx039g^)eTK)$Fx=zy5eRuLIF{%x!i9hc$jO z<)~z&qUT@>F9z#3Y0=b0n42aXQ?G>R0z;YK49+Pd`mkfGzA-`23FE(oXuVK`j^Q5Y zz#C%pjP#=qDNSoI|5Hx#_vQ5OEsx!JXUenm6g{-$;<=zkSqplFOuEFKXl>7G4b^f~XJX zNTKA(h~?uVc|$^QY6v={t(Be`aW%991lzXbSK9zV^Tj_E@2rOIb=;7N?)I{Kkz3E#xpQblFcT1X5aN8DyCx$Blte4_;5NB zP9)`vjJL=ya`AP`u~)r0!whu(Bx48>z_Ix_KggXr`$Y$+pNb;fE8*IVCS z`KsE#NW+Wa%wKzyyVHReceCKwR}4l<)~c4T!1iBpV53`=dU>doRbtdaZvkMdVB zXrv^ED5k4l1#?aCh*zT1U8-4Zb40M_*OCjaV8+Uii-O?Hk ziO0?pm(a6oeime=&aM#zJ4Pp($A`3W_E#qlVd2ZrMfeloWLfmw5wEtKe+!MK^6@Iv zC~>RIiVTum02RADy~$KVMR4z)A4lZ;M%E)fx*XMDNM1xdX_;KoW9GAHa-fv=)zKG$DAILB(6VBEbaH8lBROH7}8iATa2b@ z9|zWk1SYF77;7X^OS|Tde;KMy1*No9lhiE}wrF!jf`X~q3W9hGv-jbFOyp_UJ{&|f zss^5_^e?NaZKlS>gAW%mQ>wG@yiRl{{D$eUp5gT2$K7}gSN2k@h+IH&c3!L`&n|XGAf}tOeMscgomD7VJzE zzjp;?XYsrm)`!JDFsK&htesY%w7|E(%CZr2+f4Pxo zlQaFU)}x)pKXi|4jYt5VXEu4Xla#`6DlDZJJ#|C z321b3w7gF6n|BcA*G20t73Iy*7~LKe64$reUN z+r-fx%?bVpMSSB_WRA(b$cLueJtKqIRF74a!^O2sD)r57oaAc0+rOn~A9wVblR&Ha zTSut)@MlKTSBPv>ee$ESTmcq*@zP)k=?DHeI*7ocUO%bWzb1eJtSsXAC65yNw(c<- zqDmQlFL)V274QJ}q26N%0)=a4@m_Iku$W{vN`j}M;_;bBS65eBKk~hMR=mmQWsdq2 zWN2EboVKl%2*3vCods&-O8L&*@?Bu7!?B8lI*3Iyz%Qi{iAYD#D|lN2p=S7z8;%OQ zn=pl~zyO?ru~Arkc>MX)liKdK5UiSu0{rCKT)QuhpMH|1q#f6hJB0$HYPI`5_iRQk zD(C^@+~oUyYBI7Lsr~)C%nf>DU%#9WJ3{vw4a!MOz64qyePZ_?Q`{nBoE8Ss%T`UT z*uxWx$v^trE&mAYhZ>oW#2=E7DkVJ@QM`>ZjUF=15>UcPof6YIrR)l69-NszA=P_xMq@%aZLVmt~qY5jF1^qO8e)`>+%zyIWjcHR4=S>EP*l zq*QHp*GV4TO3dQY#tVEEj@#pQZvyf&@R&%F+o>pQcJ5$zt>0APx}9To4^xF*Nq-YY z&9fSVBU2&EwHg4=!uXPftja?tcbp<7ftC(kT3KF!Aio&&iEb@W0C#N2fcw;dde#&*79li*+A?bR zNTa$S=1p2e>e5mI8*(~X&J>RBu98lYtER!XxB@{Et{E$e=k7;Tt=TBB?=s90QBknK z!39Lr#GY0Pn=!TFuC}vYGr=+0{v__U?{YWaRMWl!uCowLTX;~w<%g<3wIqs@GY^ZO zOBL)D=f2lByqQ@h2ozzG5M|IBAhyx%0S_ zgIkJHMJDTqAKi|NNs+J1j%Q4)h?}bD+@Y0nR78KLHa&|nkM{0Ug^!U53YH4H2n6~i zu8{D{+we4Cg~0nn6_FHWi5h)o*LHu&a~Fy4fWV&KvU#EZKQ;F@jAfGp!r!^6OGN=0?M z=B%YH+@k@YZNPwwjE|2mbZCDI;YLD2LU9Kxeti6lvnO%b-G-SkhVGWu_W5JtU7{b* zw8>IuwsW^_fw}ql9Vh-c1jz zl$EVi_l1**#_>Dvf+v2=lUNoz_UqnaDRv@N&XQmK`Xd_|m&I~Ro73{Mn4BsHqOgr7 z!Og(CR@?3e1L=}sMN`||wz^BN!vrDQDN{}f2@$aPjiiTtcx(yJ&yPndwcDRJyB~gq zEY+T}mNKN95UADl=(LH*Ib zWZ=K&A(zI~;2FP92==~9?BwO+FDVcnjk*tEmV{W3hKXA zVbR#khj69vmmrd@wN&Wsq=`3?X^C4%WWBPjZu!h%8anP^C>>zMqcfnI>s7fu?JE8V zd+p;^LY^n4yATcEiBCTjGYw7kRHo4Jfb|3_5kKYU?@gVr(~CxjhXXoa(@4(PYMPU` z>Em=-Bazc`PCGGDOPEbozH9#RtWcdcFeoU@%n;pK7Zwu{h05N3D!HL0CLVYTF^kY zBZ;EgK`)7`vhr(g%efZRW&3^cFQ3QGKU=OmsrbB&HcHkDoom1x-|#L-nk| zB%Y_>cv^|3s9p$7LovGQD59_E{gtWM$1ELkT+`@NqluRFogIo4<5lK-oz`t`_S)JH zd8(?|o%vf@T3FsVl5ko~w=_36A670e+VKY1R(1CGH9|^=M@yeXid9%lNd-K7znLJE z#1~7Jo!Zy7zb-8&b?Y~xb!sl5)Mu$^iMfu!^} zVB;b^=HKn0XrEkW-ma1ak&j?d^x<)=`Sim#@Y8)KEYve6ENeC^&9mj|)YU_V_2yP* zglJ7nO~uX3@~zbKf$_x(*x8v&soBOg9m&T)K|#@P4aCuJDTa`bEaK|y?)u_L`!qvJ z%#LhsyvnHLoYr0kfU#Y_7uwO^i4p(Kn0$H>x32ntJJ-;tKUXzR zEUBAux3tu^SszCxmVw7Wt6_%u&UX^h$oc!oKgPZLTVfcsZjoY6l^~$|-o8B-K$C*5 zRbhc+Z*M=B9wvw>Q0;*APP&tvN>F%VdEs7IyKN7&menv6B-WqC)MQZ>Xqe)BY$`P4 zx>c{mozFRtXUcWsm71={5uaAqwrbRTTPPb-evoyOBU8% z^NJI7Ur+ZC(2q36$$12YH>S(hGXhy@nRGL$jL1fG^yd5 zcG3Z*J>q&=Nd(sKT^nxaZwncn`S{kAiJqR%VB*Eu-B%gv^WDz#`=h{a#KSVQk7_6j zAkFIHxY^ft=2~^UmUlF@?dchKIN}}r%}((7SNw#uG{oe0%GtX=PPRSG*S(uWc5U~J zjEsPJ)R$}K@NPoZ7i52@!!Q2v_Q8e4PgCp1D4{;s_k4ja{8egXw`iq1-PkP!bJJ## zZyzlR9DF^*dg^v|^ld9>V(Zl>p~0`^#3yRKZNWs%^*2xOIcz9z1_p(#<1H>ND3rR_ zeDN`!sgpmE#>yPof>~58;!eAspXHD;*Ulm=^~Bpg@pZ#ir6MDHapwP|>NqzqZm)~J z-Hh=5osSgwOCpF#@-)Ny_O$>+ZO>+ucHEFcXc?XmE>$OMBt!DIbp;NT*zVRZYjnh} z8{n%I*HO;+4A5GbXO0rbQ zx4jUfWqHr|d+x!FCp8xJ0 zWId)Jf2KGkWSvVvNl8S;mQ7<^V?)4c_EY>oiDto}rn^rQB&iRNAb~(y>Xi`H?eUbD zm>A149Vm33)}BaCPHtxX$8vvaKd^X=S3+*`ImRexC`#u{={F_?_q;yV(0bZdVSlit z!=PIqJon;e)lp31>S#KRtN}n=cRqY98Xep|htebRpb}SgO8L)i)FoehN^!31I?}?1ec%?BG?K@hPk&y%w zUL+??er?CzGN4Ggb>#0M2US{iVf&$xw9S%v6BC4Bnf=XH6A1Fu7yS2f@(bRQ z^4%@!Z@;@INXf-@P$}5jx!@|@!M$B8J?XgZ{_qi>*k}{EiHT`E=l-1KWT81qS>d{R z!`sRixw*1&EHCO@50q5Y4~3L7=T-DfVQ{#)Ib${&v75HANIUlyE&UTONfOs6Hi;1Z zlP#TZ>*GOG-7z-(Q;w-0%N=TnQV+(I_3ok(!^S>B^3_YcqYiH&ya1KLs0UIR5N^zK zwZk>2Pf;|4(9XYC084y8k5t=mXHL{^okt`|PqK`**+cl^qFpER?7>sPPCm;9_nz;x z5k51YKCp9IjyF}?n}(UNon(&RC*h4MNaRGY8faC<=W}2vGZ=Lrl#*e)#{?YkdxI%W zU^io>k~nNS!eY^*?ylS6ly%ueDPRSm0dL?)cL6qA4DaCZ7afN(mEC$K^6R& zE+O#H51Nx3c+A*G;}Sfd)c+NOBF)Y1I&8I>B8V56B&H7Xs`0aKRL@U3Q z3Dty7PCnd9=6QXi?pVR2GHG=v`@PG)iIS4ikM}XG`ME2PRo#hha0I74oEUJLZM@5~ zyzzcq*~N=HJE(NmrY(#xsC}L%cQa{JH(f>%L(4H{cE4<{csvA5fv$ja`h~@D+WrB( zT44+E&Y?hWua6H|@yL7z^aeW@7s=7NY3TVrG2W-q;h|acRhFe=YLU0-1PZEZGLKc$ z$a!Nr5T3+!eNO1;PtIvSihL|8dXw>n5J<3L+ow-bQPsOz0ZD9>mA3v>Sy2dii(B3*M2+HE* zYL zVgN%oYo|o8r}iiJ9ysU zG2>K_n_Ftr`o#M!X(Z)9o*MJ+?ryzxji28gPP0ibhR0Hv&3-6+2Ju2+?aiOdOe1`N zmX~+Y1Pz8cbVfDGY^VeqNRU>h)7!xh;>)r#GkrNsNAPZk)qlY=uiYQfV$i6K+bv~< z1^K2mk^9{z>dM%oc|EfAJ})nCzG^uIdNi1+&Av$PVZE#~^X#=jukCB)P#3mPeVZ=G zzkjt0eDCWvqfYGhwb2zc>bd)7qQbI$ecvIcvfN^dchZ6j43?04la)m+c)nrlvcHC7 zZu@ms_z7B!gefRGCT4JL&mK{#KbbE!De2+E(9pbf-2{O|cz2AVuC9C-u|Tmp!=by! zQtwGP&hvsd8w&V!u7tt|Wu_y=dW{GUF;Ogfzj@8;cGDNtHbu6|Sn)#B;#om_j#~@q zM|t-3i}2!8aPa;7 zXY`DWj1TD;si-g!)l{Pm&krS^cWbM4p^Bjb%wp$Xry$T`Z+HKukoU~s~2IcFPCHQ;p5hf%i z7@3#^_6eM?{WuvfH>a10W%|N}CV=f;S5_8{%v2j*G3~}yJ=4}E5yPz2cgAL@sG;~` zbp3WPc4?+e45%%z8EL-#y~V?tA8##e2r2jjN$k^b57z6c^2uqDSIsBp9_fi(LW zCDM4FyIOaCFrJIg4#R%taxmrC8jnT&iqHUW--P(bX7k)?c1mFVdC$<(Aek&VtOnP) z0Mz|}!uqw`B7rcW(=TXqz));|)r7+{D_7eR)A~s~4j~r~Mb2Z*Q$2QR!I0C#l|6wP z$DkW$$3eMWPttMwQMe>#2-&UPpo_ff7dnycN#NPb**je27)2!WG1_-&eKYq$WHcoG zZ2#y%3(gSpgD{~VaUaTF$_ggP|K+<=nfJoL7oZ5>%D)F7x2YtKl zAO#QXR;hp^?o_7>HeHeh+Z-a6xE7=zLn+npA8G#%D`wFMRhdfT^Ijw-Y|?zZd7c z?arW8-owq6haxwWQ&epVq1eA*mj#& zlsazL=iuCW1YE+<(YBbB=~oa+KOa_Aopdse6ATA@T9g#;LeiMT8FXv8kF^!C%Pq#AH2gRVA*9#J$Wb(RHDJqHW0AwZkn!hJp$BY)o}Xm7^V~8J6_|q*wIY~HqYr( z3h1Hv1VE04mM7lok(jYDtzj^6uk*5;ep|mmcycn4IiJmf)JUP0q`91&-Piza^>(6V zwRHaT2HlG*Un?GcA9N95$CDa(w6`2&#m&$bf=|yW{$%%Tv&a)?`6(;i){mvP{w<7L zGYmfH`txle21XoJlYg`U|IX48jlzD>TtNo!#-nSw)H{K*~xeNv8DMiAzTVS6s)4YcnV$D{W2r> zjy)PV%-pnBT)Lh*vQ?!&5cUuJ$h11j!Ivw!P~VtUJ4sm#fu`r{|E$*h^p z_2=7>GA%;7o}xgHa`TB6uODZv&@e$d-FP<1Z(HLsJsvyaWIpJ_s>Axb8PaK0RT_w7 zV(0bl;$ciW=~D+c(fCQH-S*_Hd!Krcdz|GxKPR0!K$~+nguxWlRm2n(F|ep*-C6`@^$Px^RI7=X)HqM%k0Vc{FOX$s29FDV;= z>x)+b0c61Ca%riJ4|-1$$DXO}O&z8Ou{sX;z|EMf>f8ii4)>avu#X>)u(fL^mWK6p zb%_A`7c{lmgA;taV^ZLf+S&Ptl9~z(TT$5HBor6dk&=13e7e`)zG{a_Lh=-dw}XV# zc26<;_109i@<%3L;HICy$Jud2c$}0AYywyY6rMlt@2!|@qKehV++0Eq3he0aZZ>!i z{?fxn`-oxT!;rja47{tm8@MK!)mF>YN`j^MU{`J{0 z;Nh{?%EZ!$Z{z3Kd3-+>Jmv|%m9uola`d7EK`xRz-u{K0Y!CRKo-zBMJf7xZ^}`dC Y>EBYnfRUaJJcb977MB+*6w&kkKkjznG5`Po literal 0 HcmV?d00001 diff --git a/images/square-logos/nats.png b/images/square-logos/nats.png new file mode 100644 index 0000000000000000000000000000000000000000..4dcb1116831ef11d1b7189e2bff24188844dd3ff GIT binary patch literal 5985 zcmchbS5Ono-o_&+s0c{!<)umQHI#tTL243ukPgy>(5oUfC`d6%FH%DXrPr4#U62xb zj|2z=L_h)@&gFOU-JYHO?{9YZd1h~Rc4wXhBLgk+hs+NF006m;wz|nJi*E~gpX7Gz z>eoi!;;xI@b2R{7Z@$8~_O60{~t}0|0+-mtOw@0Q|)P0PG6@0QeaI zV0e|?YOHj-L1M3`rGCrXcJZt&@)i$XXn`Pv^M?={gZB1p__e zVLv$lKyRd@t_BWTI9Lk(z-AV{hCe*a#&srVq-aiQb1zCZI*l~)5I$_5UEYcP{ICZ- zL?-b;Sl)vAsd4mk?E`3xgoo$-Z|Jou@V}|NI!>k<_PnaYs!=q;i!TVAQRn~Ecsg=} zc=oE8#x_?LmobDt$(?88IX|>d9E&`13_t1y;tmFPr@K_bVat#2s{JqfD^zN5Rh8h9 z;)LhuV5zY>?dvyp;8Rn^`>S%G2gJlr#0*JPs*JY2cLQP#DFMWZIrp0bZzY6wIT^ht zVw;9ze%ZQi3XrL~^MqAGcP@5FdxgaU6aI`|gj-)nCU90g;QUuon`~0PU{=?09yF*< z$8qLTvytCTqS!o~J1tcccUj}d;rOE4wqZLG{O^iddjudSq+op7&IE$%}ImH;9USY1u#D@nlvGOWgYMjUZZi z?UtmVt;1!+H_#FghDu>KtulP+V7D58c6A${$x2{x-sLGsTcRqGC{NswV$t_E5 z>iX>%yko3wgJa+oL2aD;)Otv+nRqDcg)Xx~12;WKg$l45efK#@ddlK0IjM{w0z}kQ z+(9uUgAYcXCmXG%P$tyIZxQs1`9CZTz{bXX`5X5vd5h&f-+vTk-#7Uvb+d?#KcPn- z#kGa~{xgx2GtJ&K<-w`|36-iMLgpEsO|zT75DKo87&`@9R&C^1^TOPV*05Ki#Y zoCng{KGXvY&(w>wpV%XWMWMzT>fl6BC1ai7*jI)wf8jVmepm-5tjD<0e7^#2!Tr@l z9J4qR1yd#|E;)*1GoRcLeo9dZSfPI4(2sxqjy~g6s_Nuiw&Yzwey6oj=ZLXZ=&R9| zdm1`@$04vy|2&7R1RwrA&jc8aJBmGnbxLxvBat~jw}PV7LMkE|J)qwdJnE~bNAJ#8 zjK5njnGwlVR&dwe2Z$SnM4R6Ek28U8JTVGA)_k@e3x}g6;0<%18P? zI6Ups2z0shGHF%(B=it8FK1qgRM2-R8O>KH;73wB;O?260hjSJmHf6EX6*>wuTk^q zDb6bxl`rc_CL@$msRzzHp$RDj<^)FfV%g=4zEBQK}Te)Ls(%6W2(=lPUq4Lae{sy z)!7#m3L{VP?&#x&xhdD8_>8|>&qY5bhw2$2LMsHjZu-6PwoL~cB_HXqXnmF@L`w7z zg3hey1-Wchgxr-f#^`xSynYmwVm~uGhApE)3P~YkQgc-Pyugx#Al54w@kLWAoVsiu;;_pKdtz8f_XL2J3xiJ0IZR|qeC zZ|#IR!^exhnJ!fs0pbty_|TQi6-XFv_{`@t@cK1N8#HaKrE^lLI<~YT^+O+$r&<^s zKx||nywz#rJI+H{na=dRtCz0gI=U$@a-t5BE^t3~h>?rNG$GX7SUCr7W<8v_LUsX~ z@GKGvQPf>f!d_U_=6ie-v~Dn2??A+Vtyms^EA#4%cVst-{1B}&)K90U6Wu!J9O~9y zkame@vBdd5^?xC%9-YdVTgJAQ!U7t|T#3jIG&B zu@}V|0|C!m>{Um{q5^5Y9{P>F4huPBoJqB_v1P*0xku+MPL6>3W1B$!(PY#GBZ#Ds zeo3F@*#iF+WsHeQ2n4wh?fE72B9CFI=RL*-n$A36GQr$_(<64o!mu=u!>}4<){po_ zTGRZtl%jKVm9k?vwmBiW($ z^6;BS$eu#bp@eP$Js7~X@znSg>7q59#8&QZYS)o!oTnCnJ+fQE-9jXGH$}gLN87t z&y|_6wf091IvKR&U#t6$_jB#8clvwBfCerVdJ0X{_~SWBpR+bm&+vQQZ+d&fc9UU_ z2c9EA#W+Uqb)C8o-6gyac-`fVFUwEDzBWG2@|8;Hw}f&b`xrjB=lrZ1a7RoVEBUu4 zNoIn^*MBRvD?M^+z_yn7vQ-G0nbgf~%6Qt+mi|1roVVuxD4 zI-==?f+p{lg43(}hDPK9s4Nui+H^eDkG;kZD{A@tHEIM3bN1x2^NdxehP6-oe3|30 zZ2Xs5s#IDlfOjAzjJyU@@iiO#uO6$DfZe3 zIUr~M4gwhjUkSkRrewpjRpY@i&CRWHDq*iVV5@^AQYij4I#K10VX~$_dx{(i-KOO( za1VJb#wO-2Vlg23`MfLvXxnl8tkO1|Vmy?~K|vbW_au)XH2Xx4(O1CiW61gHW#0Bc zq1HCytz)|9YeZT^*2y~ikZ&Gbz6 z>7LSuKiS-Hb6yh-Nx%*FjSlh1V>?sEH-&iz$;BU8!!~z4(ocp* z&ex_R$4qNf;sE2Jch7%8ZlApw$o|;N^!GNO+3kUG1_}m#-?w-T*ZSOEQjFvW>Fx-3 z;*Y)YS2mG@Z&K!TY&@a6(Db47=B?jW&Fk+bZm%PvKzi3VqphDQi>@sA7yUYc(d&Fg zturUs%eLvRJEB`DG&rFv+zP}I?g+9~&XmeiphIbMX(l(5m?4qJ3GR)w{!c}isWL_` ziuG1{l(SZ33P81V@jA4`8i3H#-~MOE7w$Oc6Hs_AOX2ms1AcbH-+my2rU{yxY9Ndb zb(!OMP2PEm?ftzE8$k^p2n*E|EAD<_7U8XfaIx6&@=5aAr#(hT>n=U>cH%8A)2&GH zgRMGuwq0_y%OXnt5?R+K%G&`3z~zWnZs_BrCzL1OexBsM?-=U_L(lrcc4s3(*q05t z^DcWHJWgCk|FvUpJEf_F)fcjT0DNrjYYBIEZmik?e#&Xd73(@RP$4YMx%$(f@9he* z8=e@NQKhwTY?R2@=BLcfE1XZTR)faJ*TquF>GUkzC>9Kdh-+1Qdo7j^w?BzpN$DE@ zz%XZg?yKaQ1I9Yu8F<4W!CxRh*(z9f{}V(w(2HYBOlpEu)Fw&hDefcPO#G!6Zg3PY z92^6qIhmY?bvCbKn*U_$A;RU}R>*{G8otudlVBsJ>WS~Euvc%stPiZe>NY$-F8I_e zxobCyAv3$l2m~8&JmNx*Do?bG$W5=pWzAd;?V2Ovd%wvkSTFV(g^Ts>s$efTGdSG5 zAQJWoW7NX|4=}%BIv$B@G|`%{QXGuC&fPgZHpBcV-y>&lHX-}7zS><$5sF=e=m=?^ z@~}4cV(4SI`dGKMr6@C#L88!{VRp4oRJahtA-GIssB6z=M9eEhtB&okZs4dJo_yRb zWLVe!Q$wrQ0g+|~?q<$BDR*BTVd|(DvNjOVJ>PW;fC|Nh9@b>-aP>>rmp{=}cxe}{TaaA#wz2qXF7wBP8%g0s!Kyp`-fG;T3 zzDgEY{1$|j((g?WS&DuHh$*w^(-^ZL(rNfHg-+x(c^67aY!+w{aTQ~ykQp3tlKGK2 z>(5WUZyIlki(Qq34AM_i7&XJ*3h6Er)KV7PEF|PxdyK+*i)Ob1V zrzKZWD&eE&uJ)=5 z8Wz+IE~`|3I9MGkXY;Lf=L3)U{j_M`D&*|3iE)++eW07kPF2n#wY{Rt4LcE3ZnzBj z+csC%?wc9#32B;y;%R&*0e#PSx2M9ah8-h?z4)9YomA_+MF#^C9@iu-l|n+FHMkuD z`CP{*+TYgCh_XVO#U;YrymM~qUA)^)!ajEYO|>gh#ut8z{hBz}Yv2U$Kgy8X`h)BFdV zOy(x>uMI=#92k0p$8>2B2p}UhN7@>_{(7JwJXR!SH$gP5T(=27!Sz%G?t6-Tcy_&Q z3M6blsU@N?XJ)C!)fi2L?}KmjTmbbq5L8re3MuLKyWPKc!Eb#b|gja0TmT z=)p*7Kvj%O97yrB_1P>a(}yvXpj69IJoXEeQeQ0>`g5Hp1;oZ;okHJgykH)1kb9|l z@Vg@pxQl`6$X&osR({ld#j-Ber`0=7H549zAn;#7*;?vgejT1|dyy394{n#%{D?VL zY4>|;5)Dz#)60=cCl7V0PMh21pSr!rBpKSeV{neB*up)%M6xdb+@cvs8oV8DLcqNW z<}q2SOm!NO9^;)WBtq`?kzDSxx@dq-3)wMMh%Ihf8~zOlgy55;mfXgzYycgLGCkzZ z`3%M&FhR$=?7TmAf+nJA4Wtq-Zd3zW!N+B%XF>hu5J{5PZ$brD zTA})4X(CoQNftR#FCZXbfBqk|+Vx9p z@sR=Ble2@Szy$X{7RMXN29@ntifd8rK*ul6Hi0AG$Me>B=7i=x7+IS3i9b^&fQJcZmQ1 literal 0 HcmV?d00001 diff --git a/images/square-logos/puppet.png b/images/square-logos/puppet.png index b07f03831565464f7128a93b323c948a91fde3ba..48f18705455fe7019cee6acc42d314c155056213 100644 GIT binary patch literal 2764 zcmbVO_fr%26a6GXTI3Wws?;F8D@~*uBnY90jv`e+nnVEul2eb97L%`Fp`(BhI+!3x zAbKJw?I=MYp-B})P>ewk_~!nF+nu*>_RY@zuroXJ(w!Y`g#@Jp0RTdFD5NWw%ei>K z&&#cyU8o5z@dn!4BEiAGSweeupQ{MOqP*h)fFJu85Rg|O!4>(ic4!;EIVc=xA5oz^ z@#RV~;qG48voTRoA<r<&$ zay=#aHaA+M!b{H*=Nej$-Y6z(TK2_C+`44^HW=CLAXwRlj>Rj$&Sj;~O2C;!p`a(k zMjNA+@T`$PjE?HTqpix~mHQ4+v!aX>qwTx3h`fJLsLxrrU-KP(y-W^Ej$o-idq2jKZib6^0*K)IiZQ3Qrfn?rD?tjR}h2fsisTh=#S^{J{RXm z&eFSQv}*LWB{MU4wNE5SSk`2tqo&HkIq8s{}Rk3-pFqp+I%fdwfD(ZX zv<4ogbnR=a=J{1hNnhCff?V~@sH~rkmI1PwRhfgA;#@oVl)#DWo_p%QpX8L{IPKG* z7m^cpCe$9fks1-#p4sI@6Un&{V18Wi@uHd8N-K|Ymap^ykt>f4sdY_jQ!{^+s)>oAR5a9?;VFLm zt^}rkQvWDv5WIlV*un@g@LTB1#v=n0cY{NT@e&0`=C)Q%~V1fI1|Z ze^!SqrK3A&1lbnBH(n^ZSGBs|p<3Lo9E`R3yS$vv2O^*zF`xlvX-5qzvM;&D`uL*X zm0#|VpZ+3QN`yoKhENIdhBaGo^C>yiEX?y3OURnqZ(C5r(fl?)xRLdO1-MuHfJ6Q)b-^sGy?f7Ornu@ z*m%Kugztwp7v0MZr3ON%P9k(^K*CpJ&&u%xE1!yI)k)J`gdbOsZTWT{L}g77yO)^A z;4*Ub`?%dW!ppQs<_O~CIgE}m^xHFVlep~EZjU6~u`bmqFphgt;_}!P%xDE1@S**a zeMv@h(~op7y>6(esCZ?V%qHHI_F!1BU7Bl8SB{C2a?SXAs-w><-#-qyB3Y?Wnq8s2 zzYU6>djK?WguEbrdaa?MfvdWEs}^o4ZK;Tl?=fxpo?wPLxaFygzOXwvVVZ$NM5aKz zKT#BJ|2j^;4<3N=1&YgUQ?Yd9>**;9V+~*Qj(BhBdDH;7{OQxN!u1O6j%)If!kE3r zddo1#cK*Dzulote&pNWBz)E^IBV}kN=;oW_4Nc)Oc%Cco>TXS=KQDh-mIr-Jx##HD?4Y@uT zWw9z`bcpf5;=A@FQ#yB43WGKtra|SU8X~Ts(f!x{#A4h0O+&4c|1lqlj*Rs6@%Bzi zpp!Q(mwJ8j`-acNe2wUB!u#CJAj|e9Z@piYjOlr|;XAFh9ZA?5OYh0kr2YIt@mD{5b!gK+X<8}+ z{Z5`X9v#ctIdHrDIwT|{x9-~VU~vlugQ>IFNUY-f>$nFhiEpZ-Yxi)dl}ZWa(J4PS zw;;g!T&YI+=Xe;ps}6vfvqMD`v_>! zl0?vT1%u^R7=b?CSvi<2$w7J9KwsbQb2vIlZO}=B1*7fFr3g#PIU6h1P{K}*?jLzp z$p3*SyMAnJ%<)J0aS@UkF!v1TFv{qGcBfDAZ~(ibKx#-yDH|aohgw1@*#C(juJY5H z!8zas3y4TdS)~QEM`+;vAxxRyN#da9g$Yp`;Rf12p%UWhP0jqWBP=Zh8~mnVkiI?k z%GB|+Xcc76&u^=;uC%oD6`U|fSno!#!-RHU$AWu{FV>Ht(deJo{|UtvU-{qIWNQa? zEn4_An5E*^A40s;5&qLtmHKCQe}2MuIfZUdwImjoP7Db&*kNV_y!J1XmUp8tdoFSn zsW?qBctl#iEr-)=(ZR$OHag@^u)H+IunLPNRlz-ci^`a(k1Ipwkl9$-@I4xRn)B7v zF!dA(wJ@?qBZgfUt6se*vpBpsr50$m{dmTm0V7~_lKRsXBy;9R()0?V96_@`vT*R3Dg^6ccqV)DM?U{Hz( z-4&c_f4ePOulS9-Xp1TU+pbAXeCvZ#7t1s#D`($&IA)-IvBM9y^F5rcJgg1Vg;yC_ zGa;(@+vGM4H*=bJ_aw}t3qzSav+qz=K}K7x4qbw~x(Pic??fLN>`JaLJS{+1+3ZYz zZwo$S7-_O)t#8)Gvuq>$ej@P4z>g!7C7K_T!w$c<9v)Cq{R_I!FbH%O;aVLlyEi3V g6CnKG!)0(!T&Vlhbxpit|G%cv&c+d0XB}|s|Bp}z;s5{u literal 11473 zcmc(l)n63c*TsjDksRs)2?<4pMga-Q0R-t9x)czQ?(P!AL696`2&ti@yBnlLx?`we z=+2klU-4d?b8h$NTR^LR`c=Adpt?NCy#eqkHRbfzy$ap zE3NI7j!E})GoH*iX!At`q?MI5S`Nf>>F@V4%wuU0_ewGSfMD~IVq-yk-gy74+hpbM z`eOm<{H08uLc|@#|1x_THw4SPy#+|E?-DoJlO#F0H z`$DfGhwMHGOOZVya%~&K&{MK$!xYxO#SZ{Pl3uWgW@2yApAl1;DYtJ;ydIN;!PaGF zoO(3D1U*wi^L%3TeR9;?A!VE^>7gKOsFoD4V|B`;khW(=k6hCA_r1tSIq@`onDq3$kyoUlw*FGG>-nsT;Ku6B zTTvLP>MTvZESTN@fFPOUyRhL&vG zRZ0NeC{Jr>V?%yPsZzn#AwZ_In$?rkEL=(gXF(M_6gCpcrwGOg(&mW7xoI*>(PB{q z~n0N23iT#prwj9zZy`@Y_%bd!Zs0PdEGM0BmxW-#9bgv`{3NwNQ}aNf`#s19;E!S96p^}H ze7Z72z4Hwj-_}~Qo$6BlE!}1&VamRbtnW-Qmt#_%C4U+Ds#FL$5%@neC_^NB@m_gH zM@7qrA(ASS>P-K=xK(osp|qbwE?Zpx zxnUA4=SR5-V7M)?BJaN-vTu341uC2(D>&B(y5Gi+UZ{4CU=d9_Ri>ik}yx-fq#-^ zk^VySg)4RmNW46|zFV}?%Y9ffCia(&MwFwEJo@l> zf89J$n>D6k0T0ljHi;cZ{_Cd*y~rFX)W2XuT@j~9dmovdJwp^u{wG!Iq!P(tcqfr7 zkvM34*?z~%E9_I!_BOG9=j;q6GdUS&AJBFrjP-29lEI=1Hnqs_^TBhvp9Knasl~wg zt#s_{?T|jHCqF~86Lw2WTj*fVF$k+|ga3kB&7?6&dR#Sm1&(&B>us~=T}MY2-D5TX zhU-jDLw9XhBi}kKxSXUp(%(y&9_Z=wpi=;520lI=A3^!PGp_BrBQIkUl{`a%uXdy=lugNIbYDC4jY!IQeik2%!ar8~Z6LUHt zqt=bcg!q|_Ee#|3l%sz?-|f#-tZz5ow;t_odkx|Txpc@C_9vH>vhOrnFY2qA2J}=S zgkkUg&A)TNEc)M8jf&UZ$dq-}SaTVdfGMMqpPYns&Rd!gcRGA#tn}04xRIic(ijo| zB8A{8g*;nZ5Os$sR|jGNnS4gBd&@t^tFq=%!n2Q!dh=qOd-$ocjGRVWGv0q7A?`c; zhG@fXTNYbP>0HdnMFgGtxi_-}f}N-JJH3&~NKge#`TZX;_MSx^nT1 z(Z=?m?Ud8Bb&a#gV&BW^;_FdbZ%PdsR$Fq_qVb+FtfwpF0@NG;XnU6aI*qbGI8AbNW-jL9jOPM+ls)YHRBm zis1u!(Wa|oKUabG!7j?5)6m>>)U~nO!k|MRyw#@h{Tu%(+kxo;xSnd7r=f^TBBF}d! z0wheLdx3eU(I*^<=1a+sxFY;Ae-9iNY4Lj;UrTjr@<8i*CdhbE+864tR_DU&@Rbnk z%)pb#g_pfpy~_<0R3I4ouiLBKU3df=V!?ahAfV}>v%7XZ3(fVl6 zxPJ0Xq;2`Mh@fY+%U`ezgB{lH_-ug>I`2+XW3Q|0Mhr);eJ{n-j7rvRZO$LmyCowz zV53&Q?gA*_5VG&Da2)MyGG6?+kQW?&f>&2n+aVog^W`h6=5;8k-Sxk}dBmen1T157 z)K7{EJp8V*5%vbnz5P!@Vz}sX0cCm|JDFnd5+ZcJktm%I zOpt}91%~y1rvIfy!i9KCCQH%W)J_30v*vKaWYlHbAjh;P$F-><$~xT}MB|kU8_TjQ zMa2ws{-f2_=4cd+u$AwlkXD*rnIURhxSXTX^>f;386Jj~h5_dVh_>NRkf7`$j(6Vn z07@oGb?renmC$_ejtC@&vXm5PbT}(ax%p4d?@{B{lB3c@t5F|^rA~NVe3aegk$mmq z=`kn%$Sd4#0d2eG0E-^^@*61|8=K#RVP2hT%_U*^?AGbfN!O{hYw8hkt9^7FdEoJ+ zE=lRbaF?TzbK$zk6o_~x!AJ|7=YWkY0#wkzr1NKG2{&c5-A zp!=v$+3UHv13R%2m1kAut$r(FDq7ZJnNyj84x^LBI>SB|cb4B6kwxsYi3%20cDGly zb#l1_EU$1m2BUi&?oZ_&$f9szu}q{5SJ!$D3) zSVMnZg%jG3RCxQ8X46F9MzWloej3#oxN~$_#D&aE;5xV>FDalHF?+`nl?g6ebo^rWoT=P zdHhsi4?8net1b1K(KkN^EGqouCZfvyFv!si;ny>t;G#rY^WI%-(;md9Fv*IhyxjMB zxb~4gR>UN|NGNnxQ(wQ?@_kFK@QRQl4V@b0>ttpc0SYva2zvUDv*BC&InvEm#M%Jk zrg(Q!AT4D$1Q;w+Dz0GVniVGHlsQcxwdB>-(*6tbOO^c-MPPl<+|m@IhBg5b3<<&pF%=BQ`Iv8jG7rhTI`wHJ zu>j?jYi}apJ9&4Ha}fcHWyLpotf`u>xPc(wlIT)JT$Mc-O+XW&N~L&rvDg;H*L$+Q zmPOD3C;bzkJ-NNP`#3{JKEi1^24$WiOb`CK<6&c`zfQ^D#1C3p8sC-W$mJji2aGKk z`4?8d9ILe3QT}6B(mj}{qSxAdq+TdIhSQO4&C%QUQ*OSlHraK=c~y=ObeI;<=WHWA z&(D+K#f2(I!pzR#vrLp`2gS&X{aZN!6g!T33)2S;b~l+v^%X|=!w=Ajr&=yb$opU{ zzzP7a*$8+2_14$_jp`ql=&Kgku%nK74FY~@D$7MiTYY}|LlI@YN(u480U1f9^UQ@5 zP_;0V16U9zoJI(#b|pG8mi`r~KYn*-U4^CGJ%1t;XWoXc*Nd8mFdYMk0LceRBt(;5o={IkYrnytwp1u^Ay^L~M|#NtL12zyw&zi=nvGr%8Oz_Swmusb zYdP^)9CJx`mYeuz^d|9p46K*a%@_}_U6^gaO=sZqD+24xaVFd)EheTgAiz3BAFE57 z?3?8hn)4y_RxV5vN+q}=Wc3A^!_Q0!jL{5tvlR_TM>qFSWHA{{Q1-ps6~JZ&wu>xj zWIbuNhXe&-hXetcKUHXJI5Gx*{GwoGCk!KmLZKt|kf;=pRLvpegkKt{#?=2#;N;is zFA&qG_?1n(;0;XU3O0RKV-Y95p)X%x05KQh$>ydY2bFA)+8$@KhD>`EQ<&z#7i?M} z^K8tE+W?$&SKbgdrjYb>qI6$o7Ul%1FULql=R@_2g}{cBqrhB_)sSFyypV>F4HuZr zwy06^mcK5PlKjNPpYEz1mVPW61}ZG$k*){dn3&x8u01M3q~f?o{Q>-7@orCd?Z|?U z^0?q(pKdyUL8?xO0naj~7#A1TMTu-D^}_0zCxf(q%=GpvX!fGj_Ig5$ouICmZA*)b zkGrJ~#u8_jSsJa&;6&hI<%Pp39Ea zRHDsR#EQssq2hja5P3)f7$c_Y?l$W=Jv{2ViQKgMTRU4>v$dP*Iv9v!c8l3cGok3H zna=!Pkh$;_u`G-ce9BC*dG;?b{hGgA=XkCLFwBN&amfA|AiK)HGow|T?(lKWx;KMuR4FSS^#rsy==78V<^TYh%fy(TlEkXijvuAKaeKMF+2-hnbk4W~%i4r38_zwrH{qLoo3I~3ZX#Jk@Xa8ewZM8?b)cpsB( zDB*up2V~X&N=;A!XiS|0?r`uuPSg2C6F4M(7aZJPapnhpnz}t^%-HwKW+WL^V-J@~U{|m&5U;%~Tgp;#Kx038-grEckSXa8WI>uBOem#oF2W)9!)?8s4MPKrEgtFo8Pu z%a%Di6ZawIgPN9aP)&8cPPbFqwdItsN6M3^B|`)BWAE5?H*jW5dycify+TIj#IV+58O-$W!xSMv zU6a236s~y?uI@94AGSG(k3|6r&PvQlkb>^WdTRXc{}hcxmUX?VE{;2>cYel(he;V0 z{Yr@sMLsG%(aw*O1E_SQkcp(`*kjT5joK@FBCxcy30U%+4Y6J&{cK z=z+?w?6}JXYc4HqucWe}XL$-ddVDfq?}Frw2}pVuvJi;w$if@4Vxm9q;H!oQ-&eev z9`@ApW}SNpNPvc2b0K(yFRJ${U>&m+)?X|wErI;IgM8-i*^$LB3JbvyzU>M5h_%U{ zo&Zxm>&`!QxeYNIBUjxg&PEQusytqA(dTD(9q(-1RrB66=gVr*_A(;
gAYCP6? z-v)AJjO3KO{NwPjkS7UzQc8jEd6vl#qjwhL{ENI{A0E-)dg=A*8Aq+b$l};7BBQTw zUzye3ZqaQo|9UoWG(qRdQi^pIIkr17O0TN%S*&;?aq!9qD?=w3Bt8P*#PA;fx_9%x z;Zg^JCx0x!D?=zd{74dy@&n8>`@`c@Y*5(r2>3ez0Go?@QftyhorUw$QU0Nm0 zWWPn%u5KdH{*rHXWy#B! zx^wq`0q7C=gC!@_U(>K?KmQQI4VN;>jFItogXw}L<{ z!jbtLH1zbm{>MiQh<69g-g)NcQO6412!ravdpv-+7PFcbOTvK;;#FAC>;1s{*%nh_ zxB6l|Anboj+}s-LS*#0Beu37xJ(4dK$LVqqLNyU6WUJ8=dZMe` z=CQ@e@J%!pmc%eK>x-TnO1mI~X#P}j7mEuYO0kk4oM(eZHcR3yUVa-!IM|gmR@cu+ z(h}UPEl>@@?B#XnB_-7V0~#o>RuyvbD!vuarq2n>szeGW32pu6!r`_5qCfN3Wo;R% z^6^Dig%ej|aCVV8lq$y8;IFMf>Z#Om)N>W){?82S<9)TIzSlb$Xup828n)M`Dw##Jb?tF=S9cj{#KKR|eX(M-$DW%{CT?C8-A(Z^i_*lX3uKJvyg z_QMnnR8!7 zuguC-gp+p2`pAp+j5ske%}Q5tRrn8HP{j3d*&uRFuP-V)TA_l`V?^VTpZKA^@$NM- z(W=0UM;sgw=V$S%w}MYmJJ%Plc(OG2=I5Pzv$QsbX~DZGQIh{-W=^aZM1N?n9GeL`K_nYZGNb)ETbmt zb^zJ9*2m@^`*A1U$IUY~C)eC5$J;%}h`sr`3Ojmfsp9WCLulTkKtU38h}46Cm&!oq zABt8-tFdJP6~A6qGXj!It?iy(|1lD^r3*k{L#O$K=?TfP)x$f-1nSEOjkToNv8+sX zzi$5APxozv1F=l_(AWwEuKgAaiJ-zpWG6M>k3kEv?*&dh5vWqeW}hqZ=$wr(Cu|ct zf0s2a0*f~!>FG+q;39Rr0402LdjGrJ6ccFt{Vg@`gTqU5>&F(Wx=S&QI4SpUF&C7a zznqiXiNR6QCHtOtrYB~s2-vt7+LFax#pR8#R$GJh6xTjn&Se-0~($Xa3i^OHYGqQ zS!ri~7bzq@>4Q3y>YO`Q?8DGWJYUF1DkA!~jujO{!QU^Pe=UZaIE^ePX;WWW{s)JT z_Ps7IDJRtaU5t%H#xZUwmrcJ!%*u|EdxzjNEBtT&(!o>z4rNV3)R(j_!M8 zl|DYt`qwa=26t*i%}E!d;aU5F_r$}u*SzH`_4M%U+Q8KzI*A%5jYT%o`Gv;T(o)~^ zcC(Th_m=c88ga2u9V*hPsU-Kw$wY=muf?3@Yrpq{w%@gw!a&sJhNoRk`;F568`F*^ z%K^VUKfgt{9|*m!YW-v}dz^Yh2~Dk(6V;kfD{7r6c!~R{?VIm?=2g;69Q2jhI|fnT z#dVDuPM7h1(9wSl*c@qlI7@E5xua`)h8`8f_}yI+K=TUIpp`b52>pk#0JFn%tNUUU zqr>$ezKM~ybk_y!IlY1V+P3xV$h4}_@9jOft(8FHt&@u@Z#Hg1CcJ#;;!V(N62;Yk z%eg8n|GNZg-`0-x6R(J@C$nXSrH?h60VEBZUtB9mzaT9I71=$SV&evil_ zfKM)yMC(p3I_;}Hxz4G{&5cqJi~Gx-R|qDJ6%x>{C1knom49=1BhpF6=%ymak{XNz z8&4ZG9UG!~9?`K>pxgTgv+C{<5oA7xnD&NJVghbo=W2VFP2so~SB54foL4{E?-iPigDc5%w%EhI7;e`rX?IHIVh0Fo3WX_Fbxl;)f)?zKE5^oySa zk{wjPf9<-TYNGYmWi;0G@}lV=;QBmXfp5RAI+u00ql~o}T{h_s&wO2MQD9!xkspq7 zbO>m~tovtF5EcxBeFBblpT(Yez?PGS8cu7r<{Vr1VN`9p&^d$pM=-_HwQWt_2lIWE zzCkg}Vk)Mq4dpTsJ+V=NK5xE*DQVAt4w!bo3j@tpp3XelN%GLdTbGZH@y@-{V3K8@ z`Nnz2t5dm;9pMfJKFufH3g_ELSY!nH>e`Op_Dp6=hLqT#hQ1vbSd^Ul(!e(4eenKPJ4S-})8q zOke&b3yhmU&IaN$R8oYL*3?)6BkyRQeJHHOGzPj;W$5GU^WE&%On&k*1V(1SMWxGH~+Z;RT4YtZ~1fUzqS+9hV7JDdoN+Ja;^3u(|R+GwqUnoxf5QrS)^8yDB@~byrxq z9IitI`KknrLP=h^ebYV9b=11{UWS4oz0#`J$Z)C@7DP!Vm=0Uv<(vM`#@0w3n&$wl zF7>0SOcK_&NuAACf)ra@3-73{#28;bee|kKSC@d4yg#(&V5b^GV5{29ZaVPUES@%M*-hm2$3&~u)SbCoU*%E4Fz48~ z`KwHt4js;yU(v{yslPb^?bKsSqFa@lrf!}w0TUGs$AeXv2lM^EyKHyTci;U;x!%~& z{lsA+&1C#}k;M@C(T*!h;U%EYxxkP0SB^Tp&A}soIlNJ!6m|PS8la2_mWlLP;e)~9 z{l9{g+fR2^N|+Yd&{<0&m$J1VvMI;)2;iG%SyPkHtipbLz_x@$z*X19-epI;AioDx zmGoiBl8ZG3;3^{98=eCO^AH07z48;ovEskclG0xflG?;QV>O;_JKMp6mLb`DQeU_J z%Oiiwgj*5>^I}n1AOG{0f*};$`(Eqw?|!R`^OzAVrweVdE_O`8A8H0b)^_lXX=_Wd ztuO>&)EICy^o?3b?6!OTT^VOFk&Tf5b;xKvY`($GMy9=z2&8D1!tuD5I8CIC({yxB z3ND)lE`~g^eLwRRo3DG|P41$`I`5CeB+R%G_-m+g@#%Y3Km-UogfP7Y`7}5yxtOz> z->b$k&ar{qtoECKQ>?icc%R!g*xDX>L$VL!IjCrxkhHr!wzxMnuN#jO>=h{)&R5{S3=xNI(W zRcbc)-)mx4MW|DG6=H*sBs1$Q`m0 z_~iBSwe4bK)$@}Z;^Q%yw1@T9o40o_69?a7oA5_CXo4mkkFvRiQg9^OP|au1i?_TS zfgmnAW{2uH&}+YYs;UEGykP+{?4~+5yKPL2i5xqYIrg_?fh@2^Fu7D7ZURJVBUcZk zW~GV!3fI6u)yx|FLIhU+tb?JdJ$U*QVWN z(0D;d)&x+tC7)%?glwon2oJ1x_YSOwn zz++Y`Y$X8lHMcN~ZM~trHaYe|MGXEF_1B>Wr7N#%GUnunI?Y+W!nxivHWZbV6iY}a z#tMi0w&9PiD{Zlxk`xq#-ZxgcT70&Y^WxJaPl*>?Zl}CX@Lv9UyMHDA;Ciwtqm%;D ztNm4Rf+Q`iPi{!JZA=<>6HHQ_U#D0 z64{9Rx6kssSN&7A);ojp?9B919wZvoa%mE+q_sLN%$v3W_p?$N(aG})v%yM}OzTWH|0^xUBYFCUApb$N-@7^b0 zmMSbl$XyOi+g!UI9y!9%Uxlo;!Wl`V&ER&?)c6UrpfTPSTc0#A%jU~|?K_^kbM8k6 zJNIwAx|N2rnE;J|JUEYH^=N*&i+U+Chn%NgU9CDcJ7&GFH@8au z(0?dZ84pNWS!)W;@~$rIpC;otu4o&a5`+YGbj%$0J)iaP=lGfZXe>_D8Q=1^EzgvK zapM%)9jw92%RaxikZYeF{3RdCd5usLnD&_Cy%d^*TkL=RZ>4usQ>fIeZ4rWU$l^@d z|Jnv&cUzL!eqlP94@w&?Ttfx+2L8@nFVrOJw_wUF@}6yvQvXi_s*M!)d!3?EX#M@Q zE7n)C5$7gIlzoc9wSuoCzsh(0B#BjiDwmXpa;Fisq9!#TaLWXeB^DR^RR}Xhg!85& zYufg{1j^}Ca#DoJKgtlZ{;(Z>aYpyQ0k;?i1MeQ2d1bbQ-%#q2+hjRT+kJMPBlk@g zhk+qAhWqofX;B^rU)z<={41(9awA}%)E2cV|3z88w`JRNx`7ej{p6cg$r%QO^Tx`U zh%qL2IE>tHyO-6}25x$Pc&X5s1fWcKnL!`965_phphf*!HA?LRdoZZe=Qges;eB=% zdKLzrahi=bF4hrr#3fs}>3j);7Jn8W!tE|w%$ODuQuFoqsTt>7x=v%Xs`Wd)Y2c=o z>}t%s&8J!Rl?Zwck{CQ+z8Ynj)j+=WzfJ>j;W+U{cJM)K&Gd)Gzn>H^RHe4B`r}kx zJW3+GAl9%T%N=!S&!oG1KVNlWj(VE&h-$u*Y%7tl`$tt~R)MP*^~sga#t{(Tr46*& zefm?!9+ptF3!jeg{lAiFcIDG$0+#xfB(-;D*Z*3=;$uGAu;ouiUU0%Ye@3ON0Sn89 z{y7yn+AaOv{hs~1#?i<4GS8`~?Wc2@7DQlC+r?o-ZfktJD&m^_gu6|mByG+CVs_*I zB!qeS1Hvmh<&Dbh(p2DT?{`B9Wn5`*xj5G@74!jd_?h(qk`ezhkz4v#2%9Ny&?|-W?=QZJpYZV#>25hNQ z0?dA6UuRDykK7N$O&v9@t?OH!oUl0{Gobg1?)HaInZv-il3G{4@8>{PdeSLs3odeb zToRcg58tU_iwnzCr8vwl$wsF%uyeqQ_ut+!(@LtQDqX)c&0UPDsoiR+ki6EF;Rc`re1Ny&uo^Q9NB&{nP2otG3>9i_j*^`!m&`F zD$ymb#?g3eNq~dg$)QYwPPRG&drG>#;Lh^f>ub}O!JnU~|A?>3%uAOn4*O3K@|w2c zQ__ni2}mfv=T95z3_cUQHlZM{ZVeJg9^p zFTP3x2z<}yA22#|ZSzC>I%4*W1w21Mi$5Y?G{j(BupBegTA^ARKu?AwS9=nb+@}-& z?4~z$?6`8@@|N6Ic$`$$-tW)Mb@|@#EH*T(r^f}$-f;)|j=VLMS|;@pF&lz6zFKA{ z8$7t1=l7Jj?>={hD)L#WVh-IAMi1ujyGc#51G}~NP*~6w-446L9+S*<3b&Md87}2o zHy+~>{Dt=L-}oMpwQ>w)k;Ct!J1dQre*6*WfnK`N)>~AA>^QMc7sp2K*64n zX|kH=-wZJ@+nJ;+91C}p@Ax55B(Let-WtyGHAsByXNpI?WtR7T{23N zisGXB(Bmfc7$;lK{T;-LyD{1Q>ffkOeY z$dz`V4P6e`Zy^l9TFDv|R$l=2=1(15DQSZ3Gf8^GddtU zZuhprtoOI?gE=$kbIUljg&tZ|y6&P5m0}rh*-Xdwp3$jti{DPQ$3$%Yow9LX%@i^f zJ?FhM9@kH-Sxq{M(_Xx^p3^lTQ=m$|j(E_%5MqT~&5jzMw}j&R2JBYd;xOJFRwg^G zz<6ia0^U&<=qtSptLu#Vs>LDo(?htXOTPIo@MnBu7{G{$3&2G91Bt=`o1Ah8Dgxpj z-V1gBq{sg0g{0pt%kzzDKc6IGwY?Jftk1z&O7vXvAbOTmtjhV|!C`jpBr%%f!AW@S z;EyO*iJ^h&gmqbt=s(}n6?N3Taf_~xI6&$V6UyiswQV!1C`pQrQJ6M6+FpoSS{+Qr zJf%#=?;UCSbnYMg!+EB7Qce635I=^+BpxQqjnB$cOK30R>Q+cvq0*cqOC&E+&ho2; z*^Mp!<_N!;1)!*P$;$8F#8L9>>HgTbtq5n$P0$1ilw2! z7h~{xqZY06!z@_}m^w2Gp4y|fd?{or8X~tDv_j)xqV24!eJOa^7nNZz&fH%<%=E(IL+7{7G z_i%u*j*BTV3bHH;@*sxIpg}1Y3J({9@V>hjju+e$x@c3N&P0#AV0Wx*ljot{QRR6G z4|s*YmUerg*=Corn3N!+Dk4f_j3Geemx!1`2+|nvQPmL{n9rIHqvctSY!w|6k}8EU7NV`R^y8l*>i-n~fuj{&mKJmG)qC$V zy>>)#uj8zDj00$JsXp?H~w|A{_{6FbWS1^D<~4PgxXFSahFw=FcIGoyi{V zusEfuwr7~W4!5s@lBu48LcXOD&j^)hPs07KAzcV4fhJJ`|nSmwxq9kBJ@aw zet=J(aNijtus;O3mYyIKNv7w*rY0-V7WlYID~0Qj6PA4gx+hfc|6HE*itnwh#gfmE zq_t<=MK3HfOT!XB`rEH*D;Sn~JWb{}qJigOPJtTRAt{D_1`{?b#&@SER*TMqj+?^5 zUuhMG)4F6(;Qo3)SJI_=76{(8kt>%{kmx^bP7EZcOWVE}qZUC!8%PRE1Ca;{RWX`O zPvnP+gs)3E0OL|hk{#2B5gPv-Ge_rm{RhV!j8SN^`UBat0gp{-y;QgY(r4beCAh`A z%d_J}mEkGS+4D#LVJZ!4ZF-~<@sgE5NlCZ05^j9oQ9s9BL+(W`5$Y>Qvzd`Hnydgw!?6d(~Nu6|Dmo_q|89>nO%y>Gs+r6 z2dt|4MDVnD-Kwn`*xO7#?AjXeG;!9raq77u9vgTQcxV1=*EgRil2qy;Xa8dRetP** zlGhD5o=qu%sU;TU_5BND{%`qwRFEEEU@CL{0d2UsBy) zH}tj-EzZRr+Biyj74mV-l4{rBL7@E0SPjOe_n5<|;rQN>82y03TlA2?dAs)OUeupk zRLd7c+n-q0y@ z1T|)<5eI2~W2f2e2>ix%@mLU_kkHYROiDknQTvML~oIk-$v-R1GlSl3J`RUhF(^*S1I zXbCK+V~ikW*8cK)Rn6Ub7YO3B5l-35N))mVKvYC$bRYG_s$h+g7p>&UNZX&O5UZ#5 zX3Kvn>?W+y!|fqDvVr?nD@ht<#hoq#>#xQUAT^(s4&6H%CoqHrx8>KWeBb3Ep>t6o z?ezp=FN(Un^g1l@sO1XH@=tE0GJQJBHtNOW6wt7)t4G+r=#P^*AsxcBUgo8b|G3%q z+7TUsMOXyauu~e(P$taz#z=&_lj<2>^`1YH=h>aELV}**hmKVOG9`j4ix9zydCB;y zrecHVR>IqO%YstVc>tBNQm%OILbz~=F1^~a2AQLR9WGFhjF)X;nw?i1@C5{6LQcio z7AR^EP#x;2x=}`GJl0+hdWHKQ{D>NYeSkmL8wZ16^60^KKjQ@;^A+0swh($W^`M#6 z44&$b?lF>~cyw5^%SqKov}a;&-6bL0r+3x|F5U zcqG+@y|Eui?jt)dYv+Mwi`{1)YeE)k<_Y|>Ys#Posr@oZc*V=Z@V^KHmnvIqy7xmz z;=SNTLYTi>ajwntS~r+0`{dR7aN`_^S^6lt0ZSle|2%(7`)12<<571pAEr=64#J2O zIB(yl5`Vt;j7N@83ItXX%t1f57KIRkF&cP*v zbW@KFnH>d*n$f*kmb^nN?Iv|>PklV>678q{8Xk6GB>;7%|GX1kb@_$k1jheG_(|;V z_rG<&aNWN7+{`N6CU|xI5~sEwm=4Dv?C6k|l*fK{?2#l9dwsq>do_7|T)b^*?Vem~ zW#Q_5i(i90re2nc7+mnY!}K^;@UlZ?Eisr4AOO(uZN%|bd2^2-;fZPFIQPJW7LeNf zhomw&j4J;fmyDdxox5)rvc>)G$PyDJexVAkxu^4i5$?VW$`c|$(%hH3Yal? z6+Jbvjp1|zniLgB2$CMO*b^Vj?XMLTc4rNEHxvm-D-G34>(9j`?~-L}Xm&Q}NINFi z^c;T7zO1AO;_t&_{>4UHz+3NT_TJVqneV(%h3|1fXONZgtBhl)#Vo!+E&dnPg|D5} zZCBs{B^CFXu4?zYGHX%BGkVJ5q~3=?PWQ17WkT&7`gMKqlloyQ^;&$~g8rhQrn{Hp%VhAkghlrDS z28>a^TtwAui(wD`Pz^MmD6?@xO%E_wCAvm&KHlK{*g{TXN! zF2R8PSe$i#ZT-+pxd_myUiT5O=5;6~(9kG|`ToN&B85t*As1v+IhgWrp^9puZ2GAt zG&xIIRE__bFpm&aTpy||r<7E#h4Z##x&q>@8%5cfGxfaK%_9g2Ng4g@QRrUsF)`Zt zWsJ_~Cc27GGp8a&Rd;zx%kABh(7<@^NiH^2K23nwZs85D`xHZ?(URC#4r!3#M@_m}#n7gPOj@$QK$0pb_Rr#X^(cFFvT3W7!UGNSj_rFUUz(3g)WU6CNmE{ZNc7q34 z%3-qckM!|z9C~gt3YxdHYp^O_U>2QRdd9UdYi0XmTx>@LdNRU+Vp;dDwfMK!??>sh zC3zaIWk0yUA123>q_r|x#dYcA;v<%W)=Ne1dEF?qOVj5KFc|4g5xIzy-v_y4F(*pC zcpC-6c113K6XwO4$p#D9Oi`d6d37Hfoxxda+GL`re9J9#RPU1&s1Bu~b8V?Lo?sK1 zW8bSf&aIG2UW1>`I;VvPGm@4`Q%bRtA?*&jeF-6xkVpcyf`2X(#G7^3sbV%BvtS352Lx$reZ zNinr044cYa5mCl3!SEvz zjxPs*NR!qTr_MthXYXG^glGF$)s!GSlVN3Z5Fzij055{%bibv+QK(H{B{Hz;>itu! zwGk*qQ8otBV%+#N_4qFXy)W^A4n#uAx9N6==1V$GZETpS=+5&wR~h1zf3aOSt5A!; zb2`rNZ1^iWnp1E~hBPZ)LXEq}YaX}Hsq`4kq+L@CvXl$U5ts!JLm{w+_S@j4i}UCK zo2lGG%Mh?>hhyv7LUgbWg>{ycO&E@V%l!c8llW~K62-l_E9fLRR{(*Hv(F*+ihS*8OQ!;`$j;g&)}aNn)^R6W+KCJ4dM2!LAC*Ob5lV8>)g_wZCAAx_9i=BUB_G){ z?dbenfxe-a3GO=*uX^JJxs`Q|$>WKof4@N&42#^RM1lMH9wPn}MY$JFB$oW`^l3(DL zD%UUXMCtzmpSx!6F0W$-jNjfueFDq+S0yfbrXS8c$ibh{nlHFOz%iS5g>gPGRz;UR z!2^k*U+rzpOt1fzcfj>f=WSm>My2={J#m_M---C5JO+aN!qnjh*d_4}dg{5wm7S2p zn@!G=``&FH6jp0yTQ#lnQpgc+cH5|okuqgNwaLp>rqL^8S?V2NA^anL#L`gW4G9Jq78>G#UAr zylP)Jf4y{jeBO7`6l(~{KJ+DRgv#?d9`AM>)ezF(Wuv5_Tkspq&Sy*oy>@r(FQDlp zma)c@SL8r50oGgsjSIi5zZ*S;4D1?x!oG zwa1Wk)4`iX!-!*)dLl{G1%JcBU%?Z2$g0KUaWHRc#>Sh30)mFZVdSIz-V4%8H#Rlo zi?&S58elSd_(l42aa&dfzu`uKj8d|1bMr+><=l9Agsl{L%;R zZm`q92eaYWX~XKXKA97zHRpXs2&(#ctZ(&U#21$XF)-;5XUW_&HTd_8kKh+%%?>$ z4E+LH8+hB+oYfP3D5$vn+uy`(l7A{w_Rq~+r9@X0wNlC|W90eDW1k@JPA6tpW)|$S zkT2r%q@canHQZNQ=Ik@{?PC7?63GO#f7qe;*WF8K24{Zp?J+O(^P38-Dn&x#qp$){ z$$Ry6oje+?YR%-Wn0!MT%Fc}-W_@mLX^2MktPU%gZri#{!1-d?*F$Z{C0d{ybas-0 z*+4Q=sPu~%>X@DGEUlnwQBD({GzO63*_U8Xzzkk2(j)piMCCUma`@&6jE%`@jCKOdXC z`ey#*>$^9MT0EY6$E^5q)pNzsi9UD)qEN?pVjq}%?rmt;jq6t{)KG721?9sb>VQgF z)H)UDB8@dcG|C>!r(hFbCM7Hh170?$X5;nXs-u{psTg*xgb{=ViyTkk9XnOm^kb7? zF0u@c*AsZh*+x*UC5(^X?qdu@bMP)Ph2qAq8RQWSOWUPegqT^CM z)2H$q^F0`VL(vLB0J>Wb{M6YCrg+sG9L*3ql3@PCcVC(`2iE0%p2n%rHTWdm{L@SvoEUT@OX&{r<*@BB z|KUE9BT;7E)ol0KBu`x>{RffVBGp2=U|z+f$$NHV8cMgm-O5CVYG26cLOTk-$5Cw4 z26xWEs+f<{UJfQsvxAoFTMEF@C{w}7x0JY8>IS;tq~3o;tgha8a%fdo`t1F7dCM@sH&LUAwwZk?B3@ zQk0-8(4olvmatR3rUp6hMsOTZZzRSwWuxTF_ku(d-0L?K1HS+D03Hw}CSkfhFSN9- zo`3C_hzVskmH6s%UaK=cX~Jxk5vD5)Xyfh7)B- zZ$;1Ae)3YfFMwtNGxn8mJ9W;Fq=vM7$YZ=dc!I&bZ`ew>mxtyx)EkRy5Y|X{@+V!G zhu*EdgZeAq{k9C4V5Ha_E}O#dsFaF=N&}mZv2OliSs!aieK;iXDtnx?dhNu|sKP|`tR{YGT4=|mgkVUaPXk5~fgyDs2iTmS)dU@x9kwCWXap3a zrNx&uZ_MV1Xe`Tk?jVQdxek#Ux#9c)kli|k>LRKW8=myOzBqw5{*}GCw({!8BJ(|@ zqNVwjc=Z6uhk`CNZ7mq>Yusx4T{@uDLOE@m4iiLBa?R7*U#o4HHn~5+r>ub;C!IyQ zVIIrADs=%ov2wiQDQ!N@QGFt(P9b%67kR;b%{}xJ#$SzT`JRJtIf`es%@J=5 z{dTm_endKz<@cgtE3yk=e5)!Z5zW3~=l;s5AuvY7MiN#9uE7k%O}KH(c4O}={M-r9 z@)zqBX{dNdff70``nk@sah&`6@Yf8oBHi%`6xA-YO^4D{PxHSa+(M2?X=S~8SmY0W zE4?=ml{^}Z36q0{l2;Ar`Q1|=EUKyW5PIi*{U~zQhICW^>KZHZlMsldjnXWe=dN>G5$iFB zr><=gbB)<_fp8d}`{8)Rj;OF{xplO|L}Y?3!yqki2~!ehK1aW`g`MR*{*AGE@3*}1 zIP45k%yI-;fDK@W;nL`Fw-fvOT^2@gi>ato0<{Hiavv%$l2AL^YvewHQ$E@Vs9Weh zqvnb!2{yj_q0+XB(Uc5gh|X&&uI}9=?9Ok97k6$uF;)(-Hq_*VOpsSWT&UEkX29 zu-B+?>T4k?mPpF3a-=+;&AR8~gU!9h7sn6SWWJC8xP0)x)VL&ni#(*37G7c4Az;6b zo#!_yeN=ZQxbiH_p?Ga|G(J%4Zm>m>Im8wfGR4?b5Wu}S@})L#e38xZz!3OoU1$u< zonhH1}`N`;42Gdo_M4YrXCkRtgHBvVY|wYUv;id1Ixmep7{=MyB*G z#|N!$XZ7oNB^mm3#uS1#iyATaa!Nb>R$QzUWh<13cg*@(k=#KDxZ=>6j#vVhCQVxT zt%uD=#xY|+LIwV~E}&hhp!EgXz*PMHu2bjl#Zcuic{v0_rwRyF)@_4WDtJKca3a8Q zwYXJ4lj3VhzwA44%xAnRV)YsA;!NuxIcRdVjz{ykKDS>c*(du~AK4AJymO_j+50uM zo^2!%k7@(XV3_109jgk;9V`E`AN()Ph|@s4|>R;L1AMZMh+WBG4{6M_+?uh>j<+=p|GW`1oPsVqw@dU=*aQ4OlVD@G17>naESzR4<&LtSi>9s0gXJ&Z((6}E!Oh?UU3?mTWD>VaB9eQU03CajXrtMRNN5CU7gMNB(BV19CN z`OPlFC_L5RFVVAg1^aI_ZT!2)&-mYkhnYi$nW)ns)N@kzWN?gCK1@$<8^9L z>1ADRQ2n74b+QHBi%MF%+Oyb~K}W}+*wG8mV@YtGrQReLrJM2)He9@It!bTG!4iOy zMr5~PQP`B0#7&81P)B2li$3oFD+oa~xTpHFsn_RAI*CU$+3~Ok5<=#oacS5MJqRl# z?Og?Dod3GJmUP4^cf&EpE#09Nm>YRc<>9*EG7XlSnHR0XE=P|`LwX}9vniJ*=)B%_ zMi*k7YBBxOPP%e4g`t&7ADCI>c7zPNt-{TvZVGII z@9|PyochKQlsAR}`Z_ohnVW*mKnSkDM>vC>JDFq(#RtvST_FQ}bJh~}PowY`I^A`f zBrdkFdSuDPzb4B)){<8@YLUl+ss~eZqf>j_Qp>O;goz~-2J+#aYCC)FR@Vt`E3$GT z4wpi)i||93R>N~~dWQS&v&Hge%MOuo3$^T--+sWcJx)Xp|8aU<9X}nx*MorU0Vh3A ziGe@$P2UgvrgoREudG6pO7@5?Pl9fA^87Aomu`tLBEy^i6h zp98-KPoNN^86@=sau32N_^#e3FsKLs09aA~%>wYt?}f3vp&88E60YMGDI$VpBOpTV@MZuC6Yxm$EV9(sTRthtUuvzhSVM9p}?L+NbB1mxoj{ zDC7fXYBn8Lg!#e}-TuG3#E-w>B zX`iJlHR8Z?BLpMl(Wi4gTpXe4S~Nf*3SJvx*i%L0XkF-_(rVHxOX7c{+dA07ayR)q zCd>?lka{lY^*0-(4@4a%7%dyKwGV{qths$u*w6DRvCC6SYdNaav#*58h`$c-+JQ+} zDmdmena+RkPHh<5$#O6Zj@n6+*^PdL)IXV&9vui^5(q5166k{4E{mhlm0yU>{I^ew!pG^YPm&sIDHASn!QE_ZHxK)j=aO^N6d9Mn-sF)ZPJO=cCL zcLE%M>~0j=Nro5{OTeR8dZCzaNb&TOjMD2EvBz|cC75`T?6oP2;rHF=0S1wS>PAS_ zLS-2-_32x186T6N(hK8MRVYd$rXm~uk0L2(Uta{xA@g1aLTcjg#PCbq^y~wVlz_sm z4H|t5-Ie9orKgJ|JI?FB6}tV{S~ne)A7(w69EF4RPHxvn{uOH_yY*uCsDO*7E< zX6Bv?^@h`TzM~f)*b%$5gHZiRE!#{RBqzaZdIyBg&)AExO=F!=wW*&k+QPqSc zt~HP!$c~m*f-AqOPz%Ax)`B*8W8@d+!iVTvd6P$aDcOKruF+1tPyTAT`9Va9lt<+B zzK^tr2S_hvmDTJ}bU~NkM2HzfJVUnM{N3Ykxu|y7K#%hfrao8VO>{JFSifwWB0d0& zHiI6X#2@!vau}~eh6H2~EWdzkE{!L92{QhZOqo>2+<+<@jMp01j1F>mm)c^m*>8?_ zW@h!q!|nOYAJWaO#;twxqbz2fBU9b{eW_?I*DpSGAt~(jTq{RcF%lZSVHgrWw>7%C z85U{S?6%z(aMeTOvuGbup1V5&#Nd60oCaaE@4Do+h5mr3`)oN7dmB+0LAIi$46g)C zq8rlz{1~sJ{$uJYk8sWc2<66Od}IOMTA-|dZ+U8^aC`Ny#0`Va^KPHOfzNlRxCk}| zEyXKN&pCncZ3XCOO`tfqlAtsu)ePmfMPS__RQ)OAFp4Zp^D{OCZ)8hKItN}w40#e*n{rhZ^+k)1qBt30;&6b9$4rYQU zsHw?k^%2n3#M{4A?r3fO*cs@8{bTchY95rHIP%S@q8+jR z0aoP4x2*4Pk>q-RgM*Bm;VaLhz-p=1wD1BJeIo5;%fjg!B!tKNONVPw_D5=qeGS0GuJnC~ar=>PI-n2(M&;~Zm5@Lz6=iY~VnXZ@qU2k+Q z3J=%w5oHz?pzhnh60$;ZunK3!;AiIB*@SsZ6&w@ihL|~T$7F0CZu5&qypi#%R=)Fb z9Mha*YpcQa{RiJ$?is_dGRIY3QFwyQmc4ma7BZJG{Z}*xkJgX+N>gJleDi{zPs4CK zr_SAb0fQoEVR`o_5>-17_cawyeIB}A9uh?)A&(Z9Pa_+o=2x}IqapijKA?!zV*e4M zzkXYI9Y1xi=Da!s#@U1RVy!pu<>&I$D}5&eKB#*FSg7Z6Up*2*mMl^*g#Pywq02kW z11I4Ea=8X+3g%)Y08!5gL~i0>m%XI;H>6w~K}uOF9s5rMlUD}$4FzB`X;G`+tCZC4 z-B9WchAaEC1*RbWOAM5^}FE+Gjf1m9xDOahgy)%witm-}$QaO(s~Mc%5w z{l&DBJiHX@lchv%Usk2(jfqym&z`|Pk&_aIO}}(T4Xv$)HC)py9h7_{pBYzT4T|;_ zHm&!A_f>=`30qbuan!)=_9p>?(FC~HOs;D>6G~A0Z;N>&36-9)(qCi>co!P2B4iKV z{4mh)81CIsAa_@?1PdjpJ*h>-eWB{ z2a=B1&K<_yrggWAS!DAgY#r5HO4~jeV;5#jwu+T;m*Um&-`Z5;b@Tq2Q0L^D%x%68B;AXqw zYYwyWGA8Tf2{ZAZNu#O!Z0ztt(5~%|pQG7DO+S=hOxL5gH6zSC=S`T%bi0>zq`u|v znVNX18WJa2?HlgxJ}7Rjs6V6xuM>$$1p8*z!wPOcgMp(RUIR~TUx{l95)Y4uO?G#W z2W$Se=BTcnvl{4qevazy-RDAl3XMF|i;sIPDogKCqYG2tLySue*{DHau@}4Tr5xxC zIyqLm`fAOPWj#IWE$VlBw@ z!HT%``v(n7$rz>wwT5|uT_I@lR?ycFfT|PN2BHfATY0}3gh-*GVY1oj8^eq>HGq~- zCqD2W9X>B7mwPlCnv|@U3)s>D0t38;*w{HsGaa^dF#+tXq?wFFH3c+X6d|^DDn70d zJ)h_LmOc)a5>`yIG5{$r;5~s81O^6pIXODJ0llP|{^AAR%YUZ%nE-#Oz#OER{^69d zrZzwk>Iwmf@`>?U3JMAW#3lFyMV|@_iShu11O!F-1tj7^Mr!XJX)rMbCLPCOHK!{&Ri1%KD*Uj4*2KM50 zc4PjV0R(ZgbhUGV*+HEFe;C29q3$qgrh8BS5rUJ;zi6G^{+XuxgzMX@9}0E+ zJB!-3P#DzB7U}{}RQxMz01hKNXDg_u8|PmbO--Pxvl|TTYza{XNi*HE@Y&f}0mVd= zpDHMc$cu;x2nY%)iAjiyi-JH3iXZ`b5kXOT;lHsUsHM9T#2NNC*6QC_h5w5EQw~lp z_m)8rS33`gm9i_;3Gi3gK)e6!i=c#{ve19!`zO}wKl}0rEBs%v{P)iA|5@Dswb*}8 z-B-{b@n5BTKloSiL!9p`-u1q;an2V+(a`9rRYCIlUh{je@ck&2&TBGM*^-i;u#onW z5eEn1xhj$B92-MAj6JcnT%`sZry>K% z*IWT%eTLbKE(UYi7XRb85<)t$Dlt}xPnpEkr47N;7s+Y| zvcV<`nZ^FQ%N3(=l%ShyJ82EM)UOzJx7~m20d)yF1hW$Wzo?R}`a>v9XYjwvOSk(m7H0*i$|h&RVqfA=`-2{r~}cMCv{ zsf(}SccyL@GwvSoCPMT{idbbgxY;*?_}2vly#4&fR($aDIjJkYAz&oH$O{$b}8X*!e?VP;AUp4%WR975pQadDQQ8 z*;eAk)i=fGc&W`sX=5uyzXD%3EjdHy6!wkBH@vI_zV16OHdr6y*4r$(E4_sY2Iskx zT7(!plHU*y5%avhsc-GgH!YVxspVeMZTRG8(g$b6ls%l_BMD`RrEa!l@}e&9tZA}B zEYwyY&sM4-?l12Xh+(X7?{r3k_X}*-hMF43$~vTCs@~j0Jp5PvrlPCxf+K@O)n%UW z;19{K=85kbhh>&gX9b?u+|7?8hl+>suoQywkDe5FGq;MSAbwIdU3d4O#J5Jr59dz8 z4E&>gJeba;hV2g)UKt{16UEJUO#Cmk1DB3-rXW}$-!P}0f>>@|J8`#|QZ zDOi4>mfa~=pu1H~bY?#$D$jn`^hEX+HqonO4`5&A}9Qb&eZb;{kAPlw<7ESgK4Ez+R5!owJS|Ul zk#?czRnd8H=N~jBrslxGTioy(nao*NQZ7e|-CKuFj$o~C2$${3*VkY%-YmqqdoOm! zP`B0aWD_ydKa+(pocMXoh>2S?Pj?<-u`Mb#12vI3JjygIVuRd6^bYpv%=Atc-C2Zv74{rDbU^DTZI)c)f^scN{$0M$E%&A^tv_1dotMWDb zhg$0FMvt4^guZCc5|*~BwaeR_1w((-6@hdGH}Pp4+;-a3BC1N@>V_lWDw*W%!7DZM z2_slwP`h^Yxcq`1ZE-s@Vot9${kH8Lax~5J1ppp_7L9u1o2ha{z|=Df*FBw*?8-S>Xt|Ea4pg0ch&N&s?E70{tKbxvV9^@K^3)-CD*)e zgfPgCKZ?Q{Ppjc2P1g!iLXYhedpddnNC2=T1maY+w=yw56j!xxa&6wPl3K#yG_m(6 z`jVUhxXDq?S~)tn2JkV#mnKuyA#zUiU+qP55nW|9O;a7%7)f~XEK-sbuK@3h-JFlz zP)0+$tv{b(RMmY!T1!7qbM-u^)R$*~sN*f1wk(9Hd!;vPTlOn6onbS|haI<4`qj4M zAbg8_30~v*g&hZX3Zq(4io9;8H;d;?|bK?jBS!dO}| zRY--Ea;07KO1KVa!)MmSDi#3EKzQL|lbasgiB-ghZ; zW^d)QOao$ZIprJ8R1p5H=a~Y-Uy7jLBZqH3-Avq-QJmyytpVTA9kqdqK1d{Q?+Wbp zwHuCTe#z6r-6Z1B`Er4_GW=;Wwp))bGlohycQrn?JSu(2xV(LAKb;y*&EQ*6*|Tv@ zThQM5;ia#9QlEFa*O5orc>xMLTZ_c$wmtOr;*tIaAf-)0C1b)xKzm+gyP9~iUu0TC zI;ZVxSO7tzrrr{1VP20u#)f>1%2sB9j(G`cn*zVpo#knY3aR3jrF3KDS}fYWSXLb? zaDOaTT2i9*tkPw653280^2}t99cVz!za6;t9>u4(Y%AlBnJtt#weR%_Vfxw6?>!LO z*t52++lwqqzq)9tc}}`th+?|NT=bzYI!VoQu7GBsuU&E%6QqpSRT642hM9aL242W- z60uBDTT8k_PR6hxJf{oKhfXM+*mYh-7CP3}f9k+clBQKe1OQ361#cNud~eJ|e-~ac z#k`$)Bv5WjzL7au`e8c`VokdLS#d7X=E$wHl;GF*Y>q1@+CW>MqsY=wDsl@Ehc|gi zcMQe2ez7ZZcBHCC()x=2)oLp5RU05J9V{%sls<(1bDOCId%CeA=Ido%I=3BvENT2D z$?Y0bufQ1tMdWw2a*Baa`b7+%Bi!|%O9855)>%LD_N?M!-l50(OdmwiU}8^=@4~0N zh9qMTeh%a9gcXUrovM~`52$RH2atct@%ab~g&4>d`cddKL#YK)xo!viH5ed%l4&nS z8C*`+2~!{D7QrMoZ#cF13EfB25m>TgS$;cZ>rK7gdc0|I=R1RQu?9qwzRs)XFUjS7 zC0aTH=!!ak$eaAJgyd9hnTGz)PYJVdUiV9#E2!NS?e3}1wTSoD(L<1!&Lyrp?9#uB zevU-+f)lVjd7Z35Z~DB9w-YHZ^KSvF!FUc`Aa~&8Pt{=|$ZYfH&umQ7QKPCXHd<;x z1wVke7WL2Dg+@&y8oQd)c4nTFfUwvV`MK4^8C-fG{zO%$6YFG(`D=NC=F z+)_60xl_F9*()qXWE%uTS&2|2vMDhpe@DQay`y|vbVQXT6qcZzhx6hDsPbu809WOi zc=;Bewx!or?Bsc-0zx$B)S$$nwj)5s+O}5GKu=@LdjHGznMLmm=A^}HMa1o|@eFU0 zlw-|!hul2XAedQ6aHl5@0k#*7d7)IA(nR2R;i%NIe@1C|W*46T*(x)S*fUIx=@4m` zND}`*hhWC(qlgJWA3re@zcrt2a;Ce+6k(lAm=0%Ts@rH z?N{x2MmW3157h%ce7t!eXIn4%zN}GWLoXxwNcY3&^Y6-LwknC@NOP}GcPieMgx|eL z`??Z-U8!$%v`m3gZ&nf|kbtD#FWC80JH>oj;fG@p zmEW-pb$zk$J=4~dX&851id3AzwR@C{W`oR^)yeG&AHZxwSruYEkzheYM{zEn}B7* z_K8MbMm5mRu7;ectA>p{E>v=8)~_t#f}sf4m|PLVFH%f)YiUJiDXDja1b;`d$8h5t=NXK6ouEgPn`L9bsvhCj_X3^;FB87l(9&W8j0<-|2TVFt%wo}p6NQsEF)Qn?Jg0m6e(J|ZXF?W# zTC^gW5oK3|zFTFGwrwLYD~qgK`Dy`gfuKe(aC?aW@f)W128SLrjiWMd^Vk}GOE8{T z>gB1}P$I!hZb3rTbJYRtHhUvMwgTmt-5oeoHpGurk&HPztMI!Wu5dl^4|2e=j!uP;#X#Nk!(- zu%xlt{P^kJt-zuU3f)7XL;75g3u;C=jyCQob{PBd60%a7xtJgEc;sA;=xSC+_ zi3nHRLgA2%kK(?0TS0hn7v;^t!^+*l$d(lHMJk{A2BN}S9yU{4B~k7dnyY zS_rkK3+gXe%QYg*elp_@-@ZlA3={VTYY$#f@xJAmrSlmJdzMG`OVwI+UOrC(-&4ee z&+Ndcz>z;4y8*odd^`}a$3bjB%_TU0r$k?d#Kz$H(BH*S;-_9%y?}?VY=0M8cxb`Y zB??4_#y}Ee1?RTxuF4LLg^c!J{1DP{2iz1Fi@2%s#JsWju7OY7nkLNHln-k~0#{xq zYeZ`hm7|_Cft`ZMvUGLv&KO_WL}r24APz;TCEP0v51(<~*ctCewig7Kc*yl~G+dH7 zc&IU}%kzkFxs+`hHPpwEW>mWtERKRtXeLKvRSf@(5oF;&5^F%n|W1rmMk77UU?i z$OHiJcR)3GEvg(??%On&V|1;ZO|zk2zv$l(*G(_XWM8?}<4mz0_2^r=eym3z4Ob_+ z+qi<}S!C$_=YY&PKOAP9Kvsvmflsc+t9s zG19W)Uc}H)Ze&Zd%@z}}?1`4QM{Q%*bK{zZz%{Cx;{64LyE(y>UE@X9(~d#Usrpy# zT-SdqW{gz`%>*ye}?CP>E zKBQ+<_+aenM*^V4kZIc(0lnK;d#`xB2)3;cDwcW5-uIFEq^WdDuw~)0roVtDZO+6+ z;<4Y(5pKPu?6(F`TfHEC6GsbwPOZW-wK&SiSrS6(J6XOBZI|2s?aDQ@VwCdiNfcUe2$+)0U$ z*pxe^vu-dZXX~uwu>N57w!!vEjbSWH8X_jwx9Ze$kuIv8Q8)e-9c_aF}!mvi1^~cXgC&*nwDh@PIvNP zjI|N@JG9UrcP@gmj7XDB#Q->_X7mPkDFgm_<>B5^x!8Dae(U>AW!94PM@AxFQp!vR z$JTSvGO*?*)p54XsFSt&UuyufR72a1$S`Afn}0=-1Z0gzJbtW0nyQ@7(15T}VogvB zg6h#~ZFtH(^Wz{e+wOY16(Sds3m{0JGuqQ*tuLj(H>=D>IBD30Io?gJyipAz{C>Nd z_x`b>6-f-5=p#Z9^Y8F>%ZTExUD(EjM6r6*Ws~Z#@wa7oIa6=kGQ9@{T_CqU3tc__ zgr|K{)|U(1wTQs2 zl377dRG_sBM*EA2f#(bvif#azVE;<2{NRBv+M*MJwGWZ?sL)V}<(I2-Y2TCN2^bTzI@mMpY^NwekZBD+|o|7l=kZaplOpJlcWZh zY9RFu5k@_WoRr9=^76yJ&Y~Dfg#TiLM1(*daq`?KX+s7tfxq$$_Er*uV}P)Cvj($* zy(bS@9(EyJ|FF0*AG&=ZKlKE){6@3VGe2TA_(eyR#zioaIu)a%z@l*es)uK;XLKc^ zW6{dP1wn#|v!L0^sk!uowC}cP^yhMgBu=Ao{el@+Z8jrpwlSysCE&F_yC(7*Mz+u@ zVRa0Ktfk?vItwg3vD^DiF(%pGeIrtZFg?)j$5b@EmRC=jV2ty^GicMw^}cK0{i$P} zIqcNP+K|$AmJ;2HTDoOS@G_ApB+j}^y44!@TH7vKFor}ps+WY$y3s%_d-=n8EPDfJ zbLJ5lGEnM}o?<<28?+wSp0t{WP=1Z8tQhn->N592$D48gXYbg@I>FvVH23d~uz&?= z?uWE>lVS0~Cb19w=vF3eW6JAX-ctrzOj(=yre^O) z_#bRI*%LCYU7mk1{U#E36#Ia5JudCl1P0T4vBEo?y_T!s*uwsYBO&X;Yw6%OCqIUm zx`dk9U2P7RxMl6?cTX-9)dHig$hk2#4Eh^z{dlF1qIvc-nBD2tl~&~P+x6EJmi+-Q z4MLEXd5)}56Xpi*eVaNfI!zP#92xHChbJaWhY5l;k59fW)WJM8qo@Oq6ehn(=RV=^ z$fF_83w&)VP?!E0;<(V&}+1G!RHo6ww#6cqz7+xRCQI^Hc8lWHPsorZPy{C*^ArPoc7qs4p?Ak3z5uM2{*2_{q3|B{VgjGxaum;e z8cb^T6LWXtYN$KIVJ`Yq5sDN14c89uBVF6>o>)qC%uNeXBmA!LGTbfVhao@{RM!a@ zgBqQ9N@dmjzA!ZUdVUmmbih@(#b|bLI^G`rJ(w)+S-N!_Be|(2KC`GxPEUyBouOvL zEd1v&NOt@8#J89Vt1*q%K{h8D$>F0s_i}9hX(8vw-#L%THNk^+UX(dyB&;g_q~5|A zsG7poW#WW0`P7G0UFqsSYi&V^`C*7ymzb&4na{)iUG^P8%m)~^M02&9GdPMwG7Mq3 zq5BWiEn)jf)rmX5`{D#62vfJdM!rm)!3{LhOsD}5NeCwH3iyAH zRDmbV7JOQI788nzFn;(#i%1LOG@bmx+0mF#Bcj@_V)x5j`AowMb7cojm_an7KZu0j z<2C7yAE0@-^vNK$JnPn!#A5|muEHi=6m7rmTO8`TTCqB{hst5)PufR>Q^b=EWdlBf z7nG6_tn=jIPfmzWddYlUq|~oz0dz~{=3Z@7YQaQ2;Puep@JiBn4;s2qEU9Q!qTM!q zHxsHlE7xNCOhAy1MhiWW5t}A0d{t)GTf~EG1@ob0Yi=x9V%q!kPM@OkI;G}t`B6&1$Lv1=F?id;tRCyrVpY#?iZG?ADvdz_5s+{JpsG-I*NOXanX$AY%^61`SIMde$2 zZn9>YXi;`D{4n$EN5a z?tDLHR^rkFf$ZO?emwr_J_FVy+jO_?Uj#o@G5?JJxin>e+#F9Q16m-otSgJwO%yth z$sDIm1dhU;%o$U?77)eqDqY3F?rJ)lRJj&x9p`<5+@)z6&DSy=3CvOm(wFhmZ-mIC zAv6eonsKKlcI4DLpvk7Stw*5w54eQR9h@f93a(44JdU~upWL9Z)RFx>Rre~Tf@5}q zwhx!>k&XV##Ys7yxM|Gnue!618bR;yk3}%QIO>|&oR%v;N+o?Iz>E zF7B2L>)Rdsj!|7eUl~5|me5z2%$$YUkv;m9z3wBtJ9HECfdOA``mna zNxU3Y)ta-RnRt-W%$v^oRay%;!)JSEdI|tg6ery9C0WwG7xu!(tGd%*8bVc+F7w*C z4O_Jl6Q0otL0??U7yJnl8nfOo+aIR01vQ_zcv*kRElYxe9;q3T-?zU^us{(@8F-rP ziblF2I}5y8KlafNkL{ScO^Uej@B2i&#B7S1LOf!HBkgwPy- zphnK>`wMm&Wu*vdNt;SBM8FLOKRZS?>nl|NDaLyf0h-rQ^q3sfA0O-%ssDq&A3KVWjQ4!XWl(^-uh<7RbS%C277wn`nkWar@>&cY5ct5VST<-= zMyhw)>M@}2R3EF179w03jR)8HhmL9J0BNVAEWUCCaogxpvu|!rC`&T=Zz?tMzv82c zA{693U}95z`h?+ZQA_>Jw^P0m_em#yHjF2nODgw%IyM4?KDW;c`Uvq1jF$b1Pm{^Q zJK{npV+50~l3;)$@Cbe6L;7SoP2{(RXl$C>6k8MIjxgEe?z^kD`*3v%y!(FOp8+Wb j<^NlJ`us>19S2QU{uB3e4$k;L|4yhXJqOh&ScLo!#PH&S literal 0 HcmV?d00001 diff --git a/images/square-logos/signalfx.png b/images/square-logos/signalfx.png new file mode 100644 index 0000000000000000000000000000000000000000..590a59904f98155bf6d3705e31bb7100fcb175e5 GIT binary patch literal 18629 zcmd3uMOz$9l!X&4xVuZ^?(Xi;(81l^8;795rEzyBxCVE32<}es;5y$d=0D7$s#bSZ zRp)u1b5B*Inu;t6A|c|JFJDmP<)ncBttJ1BRCu`m=AI$B)h}PjzsO68YkC=8$O zpM!1!|HPS74bI}H9_04Nf5m3`|Bs9DO-^KgeVKYkX0d+kc<1%(|@&ysbHJ5QQJF*6=0_4rjtx8zWp$) z)rO4wT6%v>JPH1IZ{-bdNN};d>`cl!oz!P!;K#iq z16wp?uZXMl&t*!@b$|g}(a$bzcIR>yYrxx#|oae_<66s9E(71#o$12|2`eqkm%Nrtp+hQ&-4H#os&O$SxvNq6+_6AKlSi$ zqo3|;fEwL8Lf5x5w?ckjh~abBV5dCv*?X@QOssX#6@cG(Z_5v_1dO{y-R3v&vahZo z5BHhlpIhtEOf#}VWa03l79zIpaNUu+m1fsaey(ET1eq+PJoK1AiKyBc%PDnf8Tf8D>dpIMO<=CU(AB8@?@ho-+XG~(I>zo4)qNu60nPmYbZF{~U^XO{frMf?66Z{TE`W|n4CU*Q( zo#8@F5_@^;-c{UP!>>zJ-;vFHpCr6q2Sm%jj|!9Sq`daSD)`9?)Y+Pt%m3UnWxsO1 zR6$ZHsi~rOB*}ppP`>RN7uoM&AD($g2ov>M>e2 zH8vR^2=?x`kB2Wt+Nr_x;6n4w65kpIj9BkcfP7p&LSLJtooW{?kgv5FcAA&xy$OI3C zL{%=f7ZSb{>?Eg2 zh$`=%k8uVKf`tOy6O=+DbGA9GuEoUQsOn$I7yeuSdaCAH^e5kZ`}gzLF+P%Xln+to z6lYY^Pb?@-O(?OMpC2Z&GgdHAc<(wrDL=Z%B457BULfp9?`Z3qK8u8!R=^S{truc* zRXrYXQkF^4jg#q_uC#<*`8t?={CG@_*Awsvu@bMGO(Qe>cP)#efDk8BNeM|0M;9&( zo69r^`Ce>m&oS7ean*Xc2%VmY>KqCAf$?QOb}K;Ja!@B?EJ&BUi9kr9fn;YmW*}q@ z9jnip5=8KNtxZ7S%(tW~ zC0&|aJUWDmOPkEl(mx47Z5XLAP0iVL$j*~6rgG^8KN)+ms{^vdDlV2YU|62Z*xSatBLr4+OQUw2i=(1cmLtI(aWCXi$((47{ zBS;lun=^FB`yy*U*^+Lz7#~$Xq+XRva_M<&9^gtS_9}DC@YV|p=rfOck$n^Ux~j zi*9$~M&tSsl%l2$(bVRz1m|qAC7plK{nQVsg{lyYG}|38wJCB-u;}CXhRJxa z=&j^ciw5(CT!4flZ15Jj&>_il(JVMce7mEK1oKyHKE=Ko;>~+OuTP7It#vJ63dODT z;mix9>m(gNT4i*v2G`L_l9Q@|3Ma-o=T3kL{m^WIY-2rGnA!~Kdqa7Xb^+k;;mJ-H zwXGmWB}h!C%IyZ`(K3qkJ3p6_V8hki`D*wgC zLwxgg--k4v)_dxxfS}pB8-01tc9_UiI%m9*;HI>4(8%X^98yIEb`#21EH*eVsYcuK!>T_jl!S4-mmZB%&zn|ON|>0l$i6JtL%N8za<%3+1EWD6YtYlJS` z&P&67et_|+cvJ?e%Zt#D(I3oO&zLYo7h9Nfnic}iQBHkRd)83Wb0LB~-V~=_7Kkgc zm)}N9aj4jfrYP>}-nfeton@YjKb^R=nvNQw2Y`VMSvKQ7?w9)V6>0#c%<(<}_Z4UG z#g>Y}J6LIAs7_KLx0nFU4_AK&n6+08)4-oKt{Bl zdeZAxi}GQC?9gACn?QcLW_)a~%3k+k(~w#9d}I`|ndNS;c~5M*vY4K@n*o;NuoR8w z^@i@WeHaO>1t74OzE}y2mvN*22wA2n4ERpT9V2li%~LIU9V9 zGpWy>Ql|A59{hgEQzEDEwS-kr|DnZV@AuahhuNw<_-_Bc0s9Tm#G-f4!IVQIU5T!o zrbKsfBfmI8^gZdLnUR3USer*VC31ba!6~3dgZTttXp<^*ZFH&+vmx2kyqM`r?uy;>$ zU;932B#RGV0VF>p8tZZ!jt$Gx)Y*sl6@=zs^8!@at^4Zy7=UV%r`4ej0&U&0JKEYc z@|6iRCe)*A-U{D?X6N;mnlQ1Wr6I0l9->Hsatt`6a1c^O$jouQLtSXLJ@$>VcLe%2 zcGd`OQt~gSv$diaMf4YdK~^Y8o`vhiUQ4|Ra~@^xXA?QOl`$N`BzZ;pTxoTJ$!h*WvLlm9d^C{Qrb>~6%;35xMXLdlunHgA&&5*+pk%>eby0&P+}RP%l!-n@C3t$u zDgNT^AhGA`_?3;$A8eb%L6}UU6&J+%`rgLt61UYHl7?fnCK3WB`^O~J`AFaHg)_z9 z!|6d(Sav$7urr|fO7OZ5kG-MHoRg;Q02<)Uq$1Ro-krbvm`&zUFp^{)d^`SUh)Y9O z%uv=J2B}+~!�fB|U&TftOfq~!l00F>> zvoa>O4mK&F+~7H18!lybs%vZt>(r9%e*gt&LxQ_0C@i4$koO_7{c=0_pcOFpH!>e= z(CMXSCbz8?k$fIxJ-9X^_N(2WU#9=2QD7n|b2WP16%=q_VdcVYFiUKwv%H{bQQAZC z3^S%VCj!N{RxyHr|2UV9ap)pR-RbVp%pWM~Ai}iu3n-j@2O(^K z2bZG!Z|67`(x+5chkE^YrB%mh|-=+ik5eCSG4 zb&s!miQ(H6ZP4-0@wi3>T!Pk2(u_VOVmJPrs&DfZ8b$(oIrU=|_7T#$4*mo^ z=Bh1R)n)?^S{a{fntv{|WAZ9x5f!>H0aFT-rkz9W1qx#pO%$ z;r^cp9Mrdtxy6b7*TVJ%HP1UyIeBc!*i$%&YaQ`i<@VvBX7sU#S*DcDt5Y)MI_=ib z1bth8ZIWGm3k^q`jGuWnYWc}b!3y~i5lb5?ho~Kt^ia_;2C-Ql_#1!WMZCAPPWs7H zlIZ3K|2j>Z421K_@zbMugl%#B!gR{=MvPZ#}wTV87$`@IL> z&#>y&q_6LBmg5=AvsGv4l_%lRGY_fEoxgn$1i?<0y+6cb2m_D}P9?u;-4ZitKQg)D zZYgS*Fcxzt97eU&Dbi1q`NHo1%vFULilt;m`#1@%dv^Dgdf3rhI{fy$7}*JuJY6?a zYsyjN0#3Zoyk$1)9YgL3%d3?zlY#hQm-@~}9@$(^OqQLuTnnl1?Bge1S1=`Cq{5Qy z7207yDn=^&mr&Eh-mo+VY;r)!W=ygvu8LyQABqxx*@YgiS8mtSyUr2x8LOI9hB4iS zLDp$gl%@kXRt$R|Oi7bWgS!&+vc!>zAJx&CE#G zoF3RiVt-DxOa2Whslnh#%e27zXVoPX$R76T`)^tCH@)D4k6QmNB0b)`ln$k55C`dy zFE2_d|K( z(onPJJn2O3&ZsyoyTzLYA3*!<3}~4CNL(-%Y;#2a?2Wuodh((P(o`F#A@D!I9**UY zSGmg%%MlWIO=I%+pjpZx-ms6jc-ZUbthG2#KoIKK!?3LGj1E@dJo{7Rl9=~Hda;#{ zzah>vFj|-(b)HWgCpFK+dV9|ABbcnd-3AG5<_(1i++7IB!H9yv!*mjl(Lux(vwy2A z<|=DQQYQJ=L_MeppQ2$72ldGiC5@5t)lwoIxu$_TML+s_QRI%==^QV)6^Skj55xRy zRMEec#2ciptFua-2z`-`ga2$RgG@YwQWh9gS{Ecs+LOgCc(iXpXv6e}QMOZ}KN?of z%LX$*>&0CF(m3-wJ^YO{H$8w%`q%+6jOlwtxQnb|FLjaVOoMW?m;IdTD#$BZGhy&n zz@>i|kMm)L{*DclR-l?K3yHORsT2uRZ;Doa|!zTV;6#jCpcb8o*iE zqKI_EiYZU@kq3<9>H65@bo-E70s!vu9ie}enpw$}`pkFC7-te?R@83Gbk`A3otl8w z7@S!Mxh+=GVUZLF1hj~yj%zk;ZawQt98qeLG|t_n(c>Ur*!eutjYab}aHdh5IzC6P zP!(>90n^mrsBiR~(v&Ad2m`D;*>iY}<`e&fg!cmw&-)wQeDr7r?O5mLVQdadt&$72 z+e@*S=dXM1zD^7=!|?^s4wJf2XvMZimi)!N9epF4Ra^V9Ps3OI>5}dHa(rrmV!XR{ zI%)WX-GTU*o-h_)9sv~QHQw5o$X0ro0mupEVD_2lS80n=_or1~o|V{E5zc6RpuY?_ z#dVGRC0F%`g(m#7YS=$XY9(a3wYu1|$FX|jclv4Lhl1OOvP6}9$raQX4lRm7G$l=p z$u10!OhI@-&WP)UbY58|GTPJ=Ni>g8aGs62OpkjYyaH}&hUPL2$2>|Q!ZubGa!NDe z1-5r3nI>g-4NU*_?jhu9h5djLPlG^t6%|eu_y&Z7ex5Pt#`j*w8!NO6v_0 zY)~d%(UvZ70Ps(RO`cB}Nsaj79MGu-+v~t7u7*soap=|UGAC?rx6yB8^u|!-Ij!{t zvqCfHOzLxMAj5ec_D%_vItxX+%ZI+!$t$WATn(rW$}V?!-sd;p2q0!RCzg#(AKVd< z^b+(6<87JR&(H}}7g9nXi3EVubNJjXpMC{yPs}Cq9g+mTvo|v77k!)52+}L+{mbizWIIU9aZ@drgr;QQWrUkO#dW33BxBJ0;B-!YO0MsYigF# z;S#x@NCRtHbr#ezf=Pk4SsI64nqRf8g$V#fsTwS(jyu1;< zX(rC%qN#u@73omJZe2a#`y2H8A>v_kS+=j}BEI*kr?x0G;-ee9xjQ{+@1euS(5YER zC*nzM?vq-l*@VT5lYdd7y{umx&c}MsMsmv_4>@OYyL99k^(Dt@o zx6j`)UdN#mDn^=JpAfXdLX=?5p_h5#{ba+Ogb+xjWNIrpSwODy5&zmv@06=oZ)wnG zUPNM7FWj3^kNzzOW+}YW>8s~0cVW@tjP99!rO@FqYe$eF*MoZv18>VD*%g1T>%nfVBJ3r9ZLW1)FX|sqPFD$Op z&uU64OvdZL1h|?C#zi$N*fj3`c31JB?GYTXs%ej;Fc@L;oKniozSr=xijT@t$#5pw z|K^`bz*x%eFjPAxnB^fSCq3b@QUd}i5H!Kecx?8Vrs)BT@pKh{0UqN<O^6p=NC~YtFBZlb#?1N`=OWEt!QLoh@e$amg!MeT5T`Qu zC2gmi>;Oa~)vdHdk;cUE7Q$yXNsI5cf^9p=z|+KJ`cf^svZbcz=7Pq;j4kO9?;WIS z;ej#)iR*X{QmyPSt(c<{=Qfkc9!M$bna*mi{dXNLN|R-0tl2ZB?>~LKqPx}?qWT`U zc9ce^dp{1R;h|gtz-<9;QyL=Er^hGJF_=M@rj=S*0`hv!eeaB6{w+@)^*xJH5;@Ww zYB_s--stmx&{XDJR@)ekl99cuo^`gm{K_%qs@!|yGoxTA7tYgIOxV>jeg$QVJ`1Rj zc+t<)`h}OJen7FdLl5AU&GsS?2hR?xc$C_;etlBzkLJRWm)Og!$Np)T3-Hj3uBTAZbq2 zw5>-;hUjC#br7r|UkYTh=Dk|kX0_bzD7H4Vd{_wS6;fut*}=dea1PqF5UkysPv^Nq zL)*rgSlKXoinwwrKkH(eK;&r+CyOzb)yXFs8=b7?{stB$&!HO8@6q8~s=!WCV;6Jk zV65`lw$#PHoZq_Nz(4myRei+1xSO_*cTJEyFP? z){kWSmJE8c&lC-`ywSHcD=(ivMl1WZeMv8;*xmn^1&E10_gf{VKHW=H!xSe_*u#lB zh@>oTs32yIUC!Q_;N2L%o&@05xgVmyfi;RU#}500{Z~?FWc*X5LkmG(7|KLcL3g<#WFQ7bCo(pK1{}J+=WI}8DovK+` z&&&EDUZ=;3w0pCdCy!d(iN8?1yT}-F*TH$4f1}TF!>BnQNl^7M`L8k8Vzifv@2Z;d z!~tbDWvzYVanQ5A^6%(;Ox2N+X{SiI&Z4dov)YC!M`b&vuFI<>Oh-SS=6dSd(Gbij zZd8zMA_lvQlAr4BEL@G;Z248Rm%b%^$nKJHcn-nhu`}u!cNA8fNSNbyv=mrKj1pCc zc9#S5`rp<%n{{2D?svlQB{7zcpPH273_9Y9bH;b03(tc$+s2>^MAD3RGA>?Tw*l|Z zzH1RGFOz%!No{9Vp&uv6MPsGp%(M-WcYfwA`* z<2gOnf7iiR3+GP5;3|H{lT&@aGp!21K>X+Hg1xP7c-{j76B>U5u%y4UCA%9@hR(H! z$ywc^3*hMUhhB>_;F@|;pF5;1ZBu1zOL_%pZvBWs%;_wI72geCt{Dc6iaHgLTMbDF zcQ@1=`q+cVfsSlg1H=?V4l2xXzzk#O%cO^iX+4tkI^PVnV8; z7<3-KcsT~Rva6tm}zdJZvo!4m6$|nG&px*l1S!pmP zNeKg&1as(8-)A_D-|4u4g;uL;EaYIpnv0ED&U9eQAJEKeIWe5?8>XYD9w5MEt z=1hTTH~JZy6}`;2JgseQ_l#KDLMS$6Zr&IlZ{G@E#{{yj0hc)AZA!#FA9g~*o^VPq$D+enPX zK#_+7deG(;5wkRHB~|5Yc%Fj`pmxEa_T3wr&+Z(g#CJz&$>bYOtM!&xI45D{MsAhs zPG5#BEQz7_{X8I}q{@(TxijfVU)XzLz!eA>NFJYw#-raDn`N%VOtgJ*eh95PUB0(A z3@~Ge?|vic9CSpk#X44Pqk%fq%+Eal^swiJDH?<6y{+=zIrwVA*`J;_t>q^89?a61V*mu& znIfdX@#qQC-kLGUlA;&`YA zmcy!}it(#R?ODuO)pgvOzg1KuVa~=x@+k~Sof=@($jg*>=M9DF9A{W6RQ^;I8KaU0 zKqIHep^>-CO;@@K>bj!a+jV2R`=qD##{(s)o-@)^J6ui9lUv-SCn9T&Pw1)JRmrNM zVgdt@qu6U}5W)4gngU2n!C8CWLjskq(!fk{Kua`M<7{v!No1<6PpiRs)9*6#o;f&5 zCMI~O{G;qw0Pp8y1eiMzN{Gje@T-(%%A0RDKHSN(7Q*t<6i=kvfY z3Zm+iMz?Y_knuRx!-fCIg_1?!lqRJTd&J|p)T+(%KS~iGLJ!ufghJ$3{-Zvv3?RW# zWV9{#^b~Jr&c@sN=#5{QzjY;|1M#xf3rjEw&Tjk;MW^po;d`q2S=5wnDZO6#6(Y$) zmY|?z3e%s#KFZ6hZnCPmvPr=0&apqNue3bK28PK>@47>qG`i!mVH|TQn%*l+>K?jX z8xj)w#LaZ|$o60CaruLhc5QosnVb163-*b=)_~n8SB`*}a`B=RB!FV)V}?3RXeUhm z&uqFs-WIu+4A&$c2ySj{x7zX?U0Yu)jz9NgSdQrdQ7;BumsM5pe#a_?YG&ue?7(P9 zJPW0D(Nzvx7}V>!A4+spFle{Gja9;TCY@o;8eW*V98@WJ9&F#X(w^Gw>^w8Eob>;( zIV^J?Gk6E_3XmUiw>XDv@rD&1%L&KPaYrK-Az2!re7bHj-&F~v^4nQnpJeN$bXm0v zw;R8eWe6Q$S~B@(mKk;B=$mqe50bCrkTz`_U~bi zC5{5H?P^I!p?9uK+>UY*^Ki(}TGAG7f zsH#AtDn`K0Q>5X={c{14nvVp+1TN(qah-_mQGdRbwzIrlX&`G9?ul#C8N&Fce9(~8 zQ*z?G{k%zSo)e4hpcPb>o~B)lS}IO0>l0Ic0qJ1-O!4|={b75R7nf; zFGcaQy7aYHb*n$q!R@-katsd&;uzLIfdnop9FL@;E*Yd0m+DdoD@##S-$&h1%zOWu zb0L}0Xh|49*`Ctt*iUzV6nw+W=6#YfsecQj+Vx`R8mVg_P|KCQ#pWxoF!4PuPR%~~ zA7x(?dbm4(Zq{^H+S>1_4c@D)&Q&&5e&Sn9Mcr;i(5CDforC5Mmlk{5gmq2sEs0Ml zYCri%4E5v9NwCr05*LIQTA_hTYQ1K^4 zCjCb~w3$y-{y;&n+5koft=m{zR?6!8`?jIPA6?b-`sDOc9KRHxCcAunAWUpSEe^dF z(ckbq3aq=G?4+8#NfWJ`xA_X8`g1M!)3u~5);C5$w~n^g45zy80My87x|S66CiY1f z*bXscxMBbbU07xu0E+>>fs~*J>E|QiNV7(FsI=-)gfjotH|*8eQ_tQ1)Y0m8w#wV_ zWB2I7SZP5lSzxDU?NC|LQp)(ASWU+P4V3C4nxAFnVo>+8t@2mYny;Gp21GVWw1BW&-Avny#Cvsx_RI5h?mzxD~_H#?qh<;e%tMxQxX@GWBa>) z`IdAJxfCTxLDRac$TW8xAG1Sd6#h4G9a~M|`t86jo$Elsk|k#ti}52#sWdaIw#^9k z@yhE?@!Vajq05CfLjlJ?!zdT;_jfvbpHcICwS)*+YB0OJrxm)E(vy(vohU&bSL1i# zX^rSWrLC2q!^fTA$Yr3=40TQA$wOO&K*jn74nuDi%7=2|!( zV?D36KQ=IS#P&?SY4LC|4PujtN3OxN2a9OYn`AIZ>@Jny2M#vt3th@Qgfe$Fzc|3T zlPdM(a#v3Y&(vN$vjlj=`K^S~n*1i81vnFgV8%2$A3`U(yy^|EY*o5cx1@-2an7U={x zUO;3Br|djs%(FR8G_S~jWMj9iTOH28$#{7mR-65XRx*muUy?La&8yO}DU)I;&{MKP zkZH^nSu&+`5@3UnCF{;cg}|ABl__g%+wd8sLNiJ=?>P^-l0Swv^)2WacPn1K9*Z_Z zqf&b1y*sHF+b1mz#B`bJ_akj$w#yVhDZVH9KRb7@5GY>>Ye#mxb=)JD&SLyqnBVE` zKT2^WVC{M!j%R*@<)=N;XelC;iHn8nHi&j%9nEl_)qX8C;_` zrV9U1pA#EPkdk*d_p`GJz3W*7l|$y#0k@+YD6hQF=^uOki`<>RNSjWMo!{XM=IJ9n7rqCw;^yn!lgE*A zM@CZBekWaz8S1gP+_Sogp&TJ|q0x$erm1z*o!)eFG--YP&-o=k1s@+!;JNHR*(Gb# zRXn5Q=6LYh@iNnDBo^*)CwcrGD<-zZ1es^uwsnt1?3(d^octqLJ%nkw+*0bUA2pGT zGyPaQ75ccHu{I^DNjw@Yz-l)=*SZO@40st)og=m%RK;Jhp1&*5! z+uI;y$wyufVL2TverR3yj}r*hA6O;7VOqt!AzQ_?E5*LnSX~(pJcMCKWz{D_zR|DC zX)rxU)tM5aIwn|(Z4clF%pxce2apD!@8KeIWT;FA{-Y<&@#QRbOU~?Y*$O2}=eqlk zS-hicssd#DaMdo2fR(?UmTIKSD4;?AiZSDmPko4u_>Vo$R~kdU9R;MtDn~p2q6~vH zC9>o2_96S(i^_*BCykla?j2yHRS-YrS#3;aIWZTwRf<}vzQD-do%Ezp${f({Gk#LxD*-m`j!k9*O3cRLWMDt-dw zg{6@Tu;K->{%f~t09JBJE;>J6s_fUhBR6BSY^Im?e7ya5#RWDqSaN*l8%M-sh{0Un zJP9l2gOUCS4P=)D7V@`#rA;}S{n&52OwPO2y~q;c`}_{}aZ-a|jE?c>jXo3GLS zv>4b&J}^9tQmreTx2f#ZU^#%ORPW|SZF^1#_duZVDLy~|Ucqp0_+4M`Lc*$)GoCmQ zZ@!UO#%Xo$CM102c`#{U4xP69EoYQgwM)kWkXtw97z`PQU4Lm9JTV1*9zko-y)k2XXSsVeFW#PL>-kPM#q#ua5HJQ_$ z)T;GV9YCA{d5(c3^H54-Ya)ZAh3}JY7kr4&4rKEDf}I8y?%&Z%bw?E!C~f}X1_6h@ z6kP|KLp7k_rhtu92>fwJ;snCJPOITOLhkk+`f1Jxc`!mHMm(IW6?`7?8%ga zrPH6n9+&I3lyRiqgjQsaI?{r#d#Ys#rv=slFYBZDdMW>(IsEr4|1vmkbuD|?bF6ST z#kbb#(~9l4H9_}HF}7XO{u9c2OReVa)4!8i9~W;|HjMr|oAlHjiNQpb<&tIYcs`)m z{J`VH7=BmGF3zr1osH4jk2`~djskdH$Vh@h7A_(e`&F|JQ=kRHsvlDBoYZs`Nc@lKmgO75 zvFo1*MlZW%LejkVZx7h5i6d$tuTrm9VEu~mT-j>@G5eW$9KSu&<_XqPGraaeZ#R<& zv;BCj+1WX*c8?i%t`_Af$aQWtZ!bL{DgED(+oZl9t;bO8%6G_rg1aB3@nOI(-t+sh zD!E!9^B<7Nyy8rHCe1zTyQe?~IVC4>TOGwGlkh*Y0cYf1Bx$v_YjDy6Z4a))848WI zn*2^+q3n@D-|v!paW+hW0d#04-)GQLQ+qq8e30>cyHLH|mTvDTn3`vCp1#2U9HDwU z{;=%rKFm7yt)qSb5$-`KIhr;Qg3|1CbC>-}ml^AjA45!xBZ;S~zobv#mCKjIfL}}< zfIi#v55v!FoUGhoZF0ZS$xP!N^WK1(>^I0`zpl?L-S42ezeba48>42wV8qeNZb8O1 zh%b~hJZEU;=OQ0;2G^0oFmgM77QmaQe${9!=h2^N`6flHHe1HU5#WQl3c>tL5q*^# zZ{NL;xCK$SWKtKe4NIfZZnQ%1AaYp@zvw7=sbIOJT@Fe5AouCJU)c_$`XY7B`a0tMF5p-{(Q6-0Ri769neLh?w$8GM=<5B_+3|;@Uv* zmtvSnY2$XD0wZzCbBWdiu{eAeOySpuvYG(A^=-oey}w!zl)u;KOfmw!j0#%yxf@So zwH$4?)|S;i*~hq>GAD~6jG;+{!G0*<@qTXGn`)+l;>BV@wfhYWwkT3xTDtv&Xte{W zA^A=qYXA7EFWrlWW?8dvE*s}T16-uJ)kbw+fj`=Dorx?&X=w+RrM9ajeN%aTb+Zk+ zOrpW+ex>E0cgRe|c-?9$#6u^UGF%V3Rrr^NPt1LI3*PYvs9^~3)B=aE=2mVFA;b% zPF=?=efok8Lvj_(pT(ipt`ha?k(iaLME|WpCrv3uStGg(e@!KQ5W`pY#6(3Q(YXYa z8&kW}LMNZ5pji$KCERE0e#y8he*piNEn}Y8_DPl+ADD^mtfWCNg>cQff~&wG46YcQ zy}Hq-+#NDgc7xFNuol?!32ey+Tfz$ity29mE`|g}#Gw^Xe`w^jkl&xA;BO>~?n*RY z>9sVsHC-Ri0vL>0-GK8%(&_RA@gb$@kuWdDWt>enuLl`}k?PqxO9|raG3%Lg3jSTm z?&V*6SF{m0J;q+;ZfIR{`y=)Jk7ZD^dko?KypZ|OG z6e1b}urLhaVBncYVVR%%BGn&haKM03Eskx$rzxOKs-jUweNY*PZ&UPm8vYt(vGAaw z9MJK{6GIf_YZWv-6IFa0`s^UCJfqNZX{~PC1uSkinLmI_PZb|e)h;QEXr@$*LZKk& zVLi@yfmPvq)8xTPyE#A#G<~l*Jcv?um-p;rh9^JZf+;wY%R1_chn_1}E3*GK-r`7h zft0@n_V4UfpB1K5oam=(=|ZF67ECR1!7Zn2mv}!R5ito2_GJO~F0Qi9dww0NRvL@t2MC__Ni_n+8$~ zrywWISQ2b zgecR}J=ahcDX?owQwFKKv*|*JMVy83R9^@U<_$8sUsOt2yb^-<=O9jn;}{y`EQ&Y? z1ruWX%Z3*&g@)+@UR!^*bQPOMWw2&=5St@@l1erH`rQ|rN@){8Z~h7{ZfQE2Ln^_m z^MB-^f~bWTh!e!i7FWv=P$0J2r26X)bfQ@^bS+-yLMWomuA)@)UkIe{3DkX>Jpz&q z$|VJ(qzcEW_et+{iCgadq6AFfS;eD({l#a-mrKmX8u{f<;g9>V9iX!mX%kt}8@*F2_$b70Hw~UxLJc8>P$WDJ2uuD@P zO&^X*I{kA0mp)RzJW(bG1#sXMo2X~?`~~IXuk~(WvHjBs zT%)Em=4d*9cjBDxBrs{^mOknDuY(8*EWUrq6ZIh~Z#OmDbtBqgRu}yo&{2j6j79OR zw5-btW_gG{i^P*pST)9X`5H^U|HaIhS!=ta>wZU8D^(^46uPm29NRX7Tb;(S6+J+3 zgv_}g)jacNbA{(3C49w>BkC<_ceKEI18RC@h03as#Zn}PVJa>p1yD&!!S_@Ff{9nzP+dXtW<8Bej6GcH2?r{EdXw_(=mIT>knc(j>ILh0z2}T zuH%VQM-Ss`ZKmpF=EYc#wjPw;ggFpuG7ae4W9gf3XB6W2-8qi)oJkDuWjOCEM_S#x ztjpIW3pB~EEGy(d#yBKC9t5NL;#36OwoK3r;To#K(P`7BmwU9igvT?9i-Mq0@l?KNg>{v~| zmn>yA?p0E10^rq#^ING3i?HM9X*9WbV81UZRRfdgF@Nhz@aUau(aBeek&@X~N*4=# zM;^yV8&8e1_k?{JZCzKr)+f%vPtG*i`VF1ZPHtd9HNZ_+z4w&?qhgLOCM~_UZkf0_ z&zXCSlHHV`R@mJ@&y13|C)qJwMxHOcAsRXa<6G?EhMDM$cuLXr=p6c`<-VTN!ir{X zE`vi^xl85gf*mnOpicw8)?`=mhx(GFDMSKV~1#&7KYFSj9`HeKUZq!6A z3E`4CE#jiAFJ;U2-d*Y5U#TC9%ux5l<1#N|`5T_lQlSVGdeNLf_oh}RA~~gsV%AWb zdiJC<0)w)0o={-NI7KWw`g*y8L}iN3R!yO5%3@2jqxHlS2?WUTUu*N?1ilZ%Nn~2z z@mSGZPi)hO=YAqXT=2AnIQ||0bB@`t2 zN;Y#d5?i2GnC62XxjZ&eA{CxoDZKi9WngcKwcc-^bCIW0fxO*vB47IsXNL3YSRPQE zk|!xHQy}R2 zTtimiWsnteWHcb`m1SCnHS?YVQrKy6LmlFPsNZ=BX&`ca&$?)k1^RL~yeP*X+gG_~ z&v&cpB}$Sio^q&?5~ELm2{ z#x&Jev@Nv9SqBP~&WH>u=pIf;dWsF=yM#MaFc_BkCXBD}W z=#WUwmRoCXb8KQ9jHc0Tr_T#0I?b~6v15Iy?)ka;au|cg|``+J)s6$+irb*xSEy= zJ*(>cSN4XaNlbf3shZyv-+9iI&n#8A*zW3KnKYF8%UNo<=@PSD3W>OvqmH|IJ)Gq|68TR+;>4qY11%%M#=021h9q?%VqSj!}Z>r`y|w=c)4J|tVtfpD&4~I2XDr`8SjeuOGELqe5Zlm>Isrm14NvairMA!=RNC- z$7ot~hRVf$6NVRJ(xu5e`w>cpLibApNkwJqQLj$FlVYL18d{o}%heN-u zMIEo0OHCuVxH!J))~F(anhbqnJX+pT(y}X8jAc}AmB+hh)>)0A$+CT_G)-xZdRr`k z1FQ=5W-q%OT}6x$pMirR@3z5=TPDRtpfjYjNT^^m#zK-KQG;llNVdg}iC zEi(+>%P*{^tAqvVlN1{re!o)paRckt1TJtZ40qqMw-Mgxq-^}f%$Hg&l)0jo@U7!j zqV5eg_Y$Dy5Mi5Wvc0mruB+|%;B2OG`z+$ztYrN1at3JC-GY+s7L@g>DbaJ!-Wm+9z6-mjTRIc?9U z{QG!Hc2pnm$^(eNP&;Aa8h6asj#bv-@B+x%+Vp2m^{Sy&REnK}jt91=-3rUbh5XwoA0eY8z?E-Mux|LFIsnhpE8U$F0@1*Be6;29~g&2qF$`k6KU3)`~we# zXyxo5u^7;<(rRz&N4JXnmSV{+BYs20OLlAw%0XkYKD5*x8$|tW17Z~Ug{VKjEUxWI z&tP}PI?dE?oETQn<^ifSO$OxaUz|Tla&HWSbF|_VfBbn54j>itzMhRFxEncnXYh$% z&lK<;oj&SBUowTRDs?rL6Cm9dIrzJqJ7W@^n2=uZ3n#wfvUzSbjOk1TyRsncy(4hq z;XKQ$ScsvEz+T$M>zg7ZAFY=IG9Ag8hV}^Pgx?>OJmV*QbYZj^sNwW{1TxI&%%B+T zxRVc{{@A-Wq-2Ena(h!?4m5Pdj~1r$2zF76A8=8)zS8f@JzFKOI7WJH4+?_yH)lxG zjpzTM-Mwl~Z6bFo(;4kKQ0Sa`38(Z<=MHPlg71rdeDF0spR=I)S@`{C-A|tGgwofj z+Ofd?Hs2%~es3~WfQ3Hon(#!Br8XXsYQ;Bi_;{UwoqW3c(hU7Z-_G>iihW!|;N9v_ zbBlw7(SO`D;?@lpOj+wIeJMSn_yzh8I*s{}=c$_(89ryoicCgZ=ROf>#qUWdB@tq! zC1`YnS_;g{jv&TNB4RCJAvP5W4G){O{uOFY^ih4^HoCaFVJ!&GP7Op|hGnhoDWBpX z#a-vrrL30|yg;D{>96?J=e>va1|>0Cb4}NFSD;B70~XN>P{#rQWj}qI&~FVbaR#1U zxq69+^wfLb4)K58!zfn}&WVt7E4|V^-7NBIh68we?0Wv4KG(kPM-?vbEX|J8ah$Xg zdv|ATuh;QtyCZa4(|kb#;@@TCI0?Ln@ZFp9PcGUxEfDtS?Un;4^DIz>1y0xfsjo9L z%@9b`bYp#U%BR{yazB(4I$g7)!YA9Ft4+@4;p%M-<~=S-_gh32w;mgn!P{EHqDP`c zQGQ_*Q}49bAB$|DJ!usr?R|FA@6H$!)vrB~z5B^YO`u`BWdc&J`dgRteP$lu2|xK)f=YWo4`X dpYp!4g0-pTvB-&VnV#r?*Cju3aPWScCh>?CBH>L5#tvA3WkM`S&beIzt3 zXhTOrDLTX?YqsyYXUvTA{tMr^f4}d1d7gW@_Vs?|jW9Aew0zn6Wf+DnCu;9MjA1Kq z7`EihQeJqn^jfbm{69VqZ3}M<;}=5z$Mw1+cpAetV#NJ>P5u5J=y<7b#kW4R`%U_L zE#4Dm?PqyU`1_v4B$8JyJ-T@u3j9Zh|LB1I=P#H-@E;xiqr*aT@Si;VClCM0!+&8jO9uYGgv}%Fs{v(=)40sf z)e|GfDJ2>cTW1HI-D9oxd^%J%ojp`=Og?w|A&V>+)-@G8`D-*=7{k(@5)9mx!-HuX zrd&VHDbA|u44*Iz{;rVxx$g4wB|~|wh7_y3ged1Z3C`+mD{+f3YKq^=6`~JP@w;rQ zMIA%zY7z@0gv4yjDvpK7?dN2-q+UJ&E{t?Y;PaV^;HidI-xmMvzNJ@JsI2@HO*|<2Fx)Q^RpAZbfi66htF6#ce={DKJ*LBg8{!#DvUFsFvmUAZO>64aXn1Tx~ zGyMfuw`bL5KEWK5FOuW0y1$N|erWaKI!gPQgPgb#@@cZuaMz~pue&0h|Cp|P`L6b* z_xDfYtSK1Qv={AXND7MgSS=*laA04`NBzO{+UJkX!sGctH55XqZ;s%5ze$CdtGpyU zans?Cr+s(+t)_DsOf${ zqk^!6y%)8AH872Se}AYE(S^gy{KaTx#ZF^!I=KzRZhmydF9m`1TXkn&*OE)iIS1KX zLnLt89{q;6Ka~2})vnes`^%?i8(8Z=bPN$SV}Gmvwv$Hunop;0q8gUbf?WQESEe}N z)kOq#i0j$rZ8}lEiK;wF=~1Iio-Tc~eF4xKQfQ(W>_+f5^1I$+9)&3kR3~fI@|$pu zR7*iX9bz6yx!QsiQ?0*dhov5QUiWYpXIgMHnkJ?c{l<7OPq?aJX?}*f)0O24FzvyD zX|o!LeVu?}_>0zMotE5f;=V+ACEM*4N(%r>MG8F;3j|j_6q+l%XsXMb@)H!<4He#) zgB*ZZ91P;(33N!`PTq4Su}AHvS*`N|Vfl{8?|{pUg|k8)@)~_Y$muRX?g5&R5$3u3 zS>OZuHrn|EJuVNetZr~jOxt!KE%|r698cTsg(vr-lh0ntUrFj; zxaBP*Zf#tT_mOXH-p0|vVk9@DC_a9!vnkCS*>5|BHs6XiS6#SyKia&QiPmO#X+B5W zu>Rw94pe@wjD`6t_?sALe`KI7bsNNKe$cNY(&e4RW!AokiO64>)@1$lDgDJ7)_aWV zjJ!(41)rmSWx=5BM&BiA?@yQRTtI$01?_<>e(Vc#{?JfNZA60h4fJLkhedQD#vG6< zLEyDY3)ZEoF15>It4QlT_EA_$ETG?&48~=L_tA;Q^_bdY_hf~CW3&Qcj@Yq%nr-l1mNBgV;Wlrb;0o{)IEmBF&TW6DmDAN@((6PtsH}l8ayX@V`AI{@^ zbQ+gkqU~NSIV4jfyRvf8URlD!MSIO|-FUWJ`O%`-Q&RVi(1m2XM7E6(KgT%4EP6WY zpCS`Ca6OA}#jc>d`@g!F2$bJJf~w`%(fH=UTJ>07EWjf<_oSZ$NBz1EZ2?jaXj+sU zA|4?cWN=Y8`^7rEH?4ah&87i4{4hwy1earnsHBCY|CZ&L=zGbI>)kpFIHDyD+o%~? zltkfsi7W1MQCFAgnQyBQ;!xW|Y#DH5+_dGpF~y8xatdqS>`sxpasQDpXAXVO8jzkU z6N{2?P>Ko)FSh3)vE;;~1>4XK8B??@N{Y^8hc>}Un3bf(HZ#x73y^YV%sF0u?2*o} z)p|^zx&NL_f*l(ntZIJ^FhllP*&f|OJ+F<+G2Q{KvhB4G7vS84=rXMjboq9K-w^0> zse)MOa{tp24$_*GX}Luh+3;DJwdADivrsI$={a5nh<$+L3h-+IAqw8HBjv8;R<#6P zY&Bh-_V_wS*!6BK*1|}`#o3`D2TNCyzxjze1DWOr!5@~BX@$p@V#J(vY#pLHSO7yL zX{W`p>5vvUV1AJ9$N|u?b`Mu#RkVd?Tta6k<*Z#q)hIl{acisXfDE`j*`XMH;}GSh zg)&>~$v+A~a-xq}lsFMvjOBUAQ|DrO?&q{8b0|c*i4IE#ejcYp;WJ-@(-#=<0R^nQHGph(TCRqJm1j6O-r@NJCx~Cw(a6vM$Q-mWE z0;w0oF~Twq1qV8&&Nzpa++iJqk-L9c@L<&Dc%OiTo+fFLMO1tLl>deH5rX!j89-b1 z?r|?bEA{h#p!Gx0?v?2u!_y8>oX`y4WQULc(gf1Z;I&*BVVh%zeC+hPFZ-~*s5cCb z?`QJ>t38e|iY{~$jLo5_CGuh;E&rB2@~jDPkN0*?dn{h009dQ#@xSM!Riks7eF15S zNo9~M!kSE?Td^(2{9+? zkR{TqpTbAolW|+1D~4KV&S4-fX!LR6R3g z92bB5WK-+`6JhfOg3yo&Qmd9+=$1&dAudXJE1CN2yvn30Z^g4rp>BN>VHNJ6#y}eb zwZDTYbMjkLer+d9g{*ja4Y}aUQ*U(tIWQpba51)&^ z*?*&-oZ9swbJM4SKS)FWLv=bsj1GaI8B?O9`>FEQ_hgsbe0j61+ss6`hcZSFzW*j&wtg8hEN5|7pQbY6KKFX%M+ z7uio{o!_;7ymFHYdtsXf$U>KKoKLy7m*|&$6QZq?rg*DEF&)7nzT{8#K07?dwgvCv(J(y@uR@ zgnrDR|4uKeG8ubweIY)uiRST8GzdFLC>aM6Vdb(8s^UG^ab!iR$soQ+5AX zRc@d(F2LpxSq#b7z<%WA+$Cz3T4%~O=7I+kTEcx!zUfW;^4+0=JFx1o-z%^x+PE6;U((jEJl6JyMfdyfaldc3mE*$vaEPVhz*3#z zeL?~s!d-Mj!E#=@7^6`l_%q98r=izIVcx<2A@UR=vZ#Zl(XOjr;zLbV&#w($H z005}vXPiTqZ6s7FpGbjNO=fw@Q^^j>ZDqC;VkoE!JXm^7YDAnSV_iT$vVrjH=T~W( z5>1VEe!riC4kv)f&w#mjWBDGMRi>*VZ zPi)~tJ%n9gtO$Fbg$^X!iRWJ}>X(iT8>Jx(u+lG5`8(RB!KX|@%V|A(QaaM})y6H~i4-3S_o@8N zk)fJmjU{NifQ8^+K;V6U#6gJ1 zYVj(-+y<+AmU326i*SAZkF>o8)q0lvk7ubrh~}TPl($00S8zn!y1+wX842I}>HS#x z$){Qisk|FeS?O8mbeeuyM&3_w(REzW1JPYdRH!YRr1kp{2zPVZ&2p4HLlTJz`er!?xby}-SlzXB-E)n1 z{4U2pP^G~(Q?;c|hmsmddO#|6Mwv!->gUutalI|aqqq$SM zL_b2DT_LyBVMorhA;N7y4F9vx&5)8~ ze>@^&Or5sx2Upi#r|&Yff^8E^7O-3Z$-hAJS&MIy&q4#+!_U+`h8!fg@53J5l9PQO zzF7W`nbjIDT8fd57Jqr{6SrZaScPa1$IlgPWg^^5G@w}NTy*yXnbqNhfC3Q+l%zs8 z1+c+9ww|}#9@4@62kZ(LAvUJ>HMP#XHmzGYADJqFr)PzGL-7-l%n&Yr{O1ILCbMiV zup{L9{YML2Z4lBD?gDGQbFzF@)YIbBH#cAY1lBu;O?ODrUc6_+q1g$Rn2NdZ3smb~ zP{=>Fl>V2~U)!~+zG_io{}lpi+QLN5CU7GZhu{0Pr&S;|nc1BOz}{+|o~dxV|7kya zQ*7j2(-`qz=f)&ydZjDVG_K!2DzS^`wPm@wx7ELhF%l2N=qo66%cE@2 zz3@Pv0_Miq7#sODt*$$K13&R152BAKHXF%R1%Te3ht}nwVd>+C?)Psru}!NJmOYrfi(DwZii<>jnCjc>)yqTv|?h5x$ye_gZcpmtOyX zhMreUkLmi@heRdke?^#T!a#ICJ4lKt)8Y}dzws7FxAG(n>>(xh#N#FVwaP|6El5b{ zYf#>!cB`o+D(PPTnT$o%+}P@OrgciRkk`VE$th061tDcu7P1m=j!ErWS&OgKMF$^T zgS{!M)prV`x*Hc9Ax!DZ#;u~4HpU+C5pTM_pJKQH^!-`zZt!1?f>(tBNv zQUNxb$PITv0t^Aw1C$a<(oKEH9KjZKrBFo%`40UnG$A)2&Fu0-lIwQs(y3Ad^YupZ zJY6@6WlQFC%y-)FUO~y>q(fADlwM!`6Abtng)3sZ_J`*ceY`?tgG8x^d(zE+hr;9r z5>Q>V)jy8s-dA%-JVhasyMZsl58YMV0t~vf5NgI$A~Q%;Tz3NxHoWByMp`m{$&q4C zNqW2IlY^xaJHaMbB00YbSjL{|57ny;$1B7BXETrh*uEBS!c||yi*N#YQYW&9P0aLY z{LaqLzPH>w`@pCS95xZo#urJiGpPG2i%ssj1s6Z?|H!G5<@l2pU0VJKec0x8PW(8}wv8!hqb~42*y@cr1!_Tn?>>j|JegLfwEW3UN5fkkX4;@;%ZuSfD+8L@0e+!B>ziWoJ=fdo~AU_8~fUZ*yZ` zKz6WdF}A1d;tTt|{>yx?S)e44J)b0kg?J>@Y zu|yF6RgQeK895OMHVB88sC^YsW!=nGrQ z(yy5l!V-D)+hKM~!JM5;tY<~793lQocqRXB*pMaR2Qv5+J zC=}gs*tu`+)csGNB0@4xO#Y?gN9lQcHKRoSXI7ve*z03>)>^{vfCT*(OA2pWEyd=7 zS}j0i<#rIS055{>cHXim6Y6egDiEvbf{glin*-n2X~{u-P^c;>_J)63yJy8+#bqE% z^e?JSGCE=QKdQ1KCq;t~U7@^qEzIX+EP0c1HKW$2>~(j%HtTrO1W0R_s^9xyQ$U+q z=_E5dlm6U0^}*`KVlmrCHHU7vJP#wb99h947byuX5OcA;KdU8_PG^<~yKeZV)}F}_ ze}1NRT%1M(@{U(b1yA3MycSUQL0w}FXZ1$3`nO_*Hd_gQ$;&kXt)_MR1TcDsZ<-H? zQ5Te;oH{2gLaGcVtgHsw94S#}LThC`S(0LXVn`H70hw2uR(E*mzajUHaQRnbTt-NB zqDTw7gVgm69c2zj8F@8=8R&*`jJF#b?>=OpbL6eW%P@N)lC+mQ&A83gPEJ23v z;@MjcEsWx5SNPCDoiM8Xxm3eYt#N(8*WkqqOdY31%A>*=wb$tXsTKP zVsm{V>gs@02N1qcX%+}w-Sd-kV{hu%(&2W=RxCKr|fe_=c<)wr3b}?T*6k%N%kbYoSSD;D-mNvvEG>@*A0Yvl{|^ zvgo9+nXmb~8`eWtd-K`CB3+th&vboz{{T7Ta0InT&cPNfm`|5X4C+?!1AAQVG@^ zHT71FHblXAKWUYj)y8x7zX#RpI# zVtbPDyz{j^^2d$`$Ve~3rS;=`XH!@@M12c_La;zZ@(Rk6E(qK0w(w$o-+%9i1PJ#2 zbS4k6kZv2#IQ@7`m0 ze_1}#{*MwCP(tM3Y7Yf`@|N8qHC=LGGe}2KLgBv|CfeX9Ap-ZYC{bo>EP)qo9`{{0 zEkxVSXg*L87@PHp?>Q<&T?*Ve#%`sOojv61Gn&O5{GqVmY1t;!3gKtM<16^aM4UZiyicp0(G_$qVBKIMeQw!py$U( zJlK;qE*?YG2*sHR*8!*lTYx$d+Irj8|ju>LM% z{hpU}|Kr`M!gx-;L0^M7a-xMle4foMuepZ}a3?2g~NLD-+nKr(Ih}41t$>EJN z{BN-{D<~CP`=OLB^gCElfQfX_2HEGvnDR zugn`L%*1R*B|Bs)s?AxpfL@D8Ea_w{KTsZ;Na7;9L)5Uz+Mk!K5Rm+Y)1y>HN@tvJ zvvcVX1bmRqOf;5vn4vb@ge0dSI=vF{A$0U{a62C9{)A}zo#=9IszwpJt4S3!Ch(Bx z#r!Hxl)i_d2bN=9>L9q8mR}cjwL>=zG#JV^^TPvK*yx%~E&NjhK_=1(4-HI6gl*p&LO~%(rQ?y>Vu-5eAiW`HkheSYY*Md`N;%{}7EeF)%T8aB%Q0 zMkPUG1^LR#QL+9|jJjelFdMIc=kid1%65ApyHXt(13U|J+j8M!nW;PvcD6^ZLrDqi z9NI>M_A>`q`gQZp(EqwZuMj?xl1-FWbXuo7zHYj(Y7@L|O?H z+CBiO+Ty<}4QJpn(H=*O66oECvYW7E)aoW^qZIw7&YeVPXyjjkq@;FJ$6+OG-1jEx zq?WRM>I9S?$+wSPqBikjNeSQg6)GrUevdBVs}p_tEm$I}N$Ht4H_~vD#H6lJ<;cCP zNM%LHx~c?iGAutW>gtBHNN1~xj2NKPqI7GPaAu`nZlgKiFqfZ|zy$%v2s zP)NB9d}x4B$e-d�FmcyFL0Q4e{Zq!a%CN+!8Kq@DgSZCUG1YL{{oWS~Q;oRf4fK zrc~o!saAu=3UOofC%$!2_$CRieD5L3O{Vi_f=l*<0s_@Y?m)9@EBJU#L#PxV;U$&W z;1RUaY-l~9(wR11tf15=@k-Tc6V#B>P~D)Hg0)3`MxE=FON-_SKnetZg1g{P;Z__$ z4!8xB;T+UOmMCHab$cP)u$^VmilBt(dOkxL#0hZfTg8rZl1Zw6u_cUUaN34&ao zdyvy^L9r)}JcBN{+Y<~Mfgm^=-wF_9mp}^BI_QX8kKOwa@W+*cI3 z^xT(=5YBap@==|x#O9RPu4J2R45Yu=3HTnL0elw+p^~Jj5;3*N&v;Oi1@u9U-M$Pc zRuEH&cT_-4AWy|`*I!5J*io~nmbXQ@R)`_~6T|`dxf8-`8 zVUT@gzDAD7BFz3cy!|H4?!b&Bp|{E+d)?&@U|95d*j_4=UA1pELE9fhMVlu4D}Dq* zL4h5VdgItF$8*Sbs2A9@iJ`;o?dn#}iUs&y{ZM~C*LC96+n%11MP3wxP>b_jIm&PR z%_D5ie-p+XJHJbIrRlp#BMYZR<{Op1Kj~T)X2n&FHOEYEj?&OfFZZ-IlTR&hYWz`p z;H2i-l;GPU(VP?OsVm7g$Yb$_1$Cc;izG*Kaf=30VpxpOyov)xh)5AvIP`@k!z~hA zsom1%-wv{Qx#u|Iw)boPom&e#3qAt~AXDHi527+PVRJ^*Twto_O_E1Ulh)%5IlW#hK-DhTc zxgBv`L)<2t*=jxr9LYCv?68sW&w66e0cKB{8_N$LI*>_Jmo89lom7*w!mW1u;yk@G zwIR%;Ge9wJ$k?7EhwIS!CuWYfcE+`pLK*di1BQ)_?14N^R${kj^1C#Tn0G#pEs4A5 z&S|_jeRATP;6<%p|47oF46SBSp?%zt4{N) zv40RXD#jZ=^4^$oz@lUU?BBgW!f%0xl2NCYIYY#}Gag*;&+&}sIrVaCP1Z*)K90&c z7^+L;9Y40S@$^Z*!r7naWd_=OOoUgjW1U{y0784~Xo$zCip1GLTlIlXVf}BfAI`zOIZ2jPr+!@rgomyHlkhWL&%n9PRce5zd+@}3o$&CN z{<^kn))NI;k)`aC zLRZl^)>-+-dBdzm6mE^T`$QJ|q2EE2Z~h=4_X`%8;FXJVHL#dl8`Zj5SMWos?D>+j6*Fbiv0tBH3` zx_Mcrj+igE&933KcJiUokFXtUuO~1%x76m5XVnjIl+3#T0liT64ty6VI2%$y9G1*> z4$Z6&bulp!jvD1Qn0+`~=Kb!PzTdY1%Xu)!FFjxvgLog+tIvhY&vcm9?dQeFc~!Q* z|E&A|JSSJUaYm=;bL)Es3|TeGGapPt+4p3r%E)qo-EoiI0BWJ}OLG@SRp~5hRL%{z zt9X9;=p%ZzZj+7ko1CyP(S5bv(~vZUviW9c4}5cK5fhzBd(!wpVi(kZ06FrjYInGc z5rZl_uk&Q~guMJ+D!H!tTIO7^3KS=G*yuMXgR4PR7^z8O#4UeFJ+okI!E-}pdS2V8 zAtU8*3$$_1(F|STgBE^RMyHm0q}&bXTJo&91D0{d$^ zN7v37PD|15W-T)>CVA(zN8w{f`}u>Ro&@T|9Apg;SW-J85@NqibxE1Q@DQZDfN(r+ z2Xs5z;Sb8RYWwee%GWv@0M|G+uHy*T>&W?DiEE4KAI*Sldxw2&>*66w1(Xz_Srh+P zzBTy3?PiL^;e(ee&*pb`K4&dISOeDF^}0?zgG@V|X6Rwhmrv)F=_$e5dopKccc(b| zyrdt#TsaC(Q5U+-4G0j48{br7j; z{-D*Z>VK`k@`Vk9F4g)_Lcr|Hi;vWbvWZxUQW0rpfC)JLc&yHDBkk^1}!jY}G5B*cc&{@Q5 zbwcN$ZY|GoG#@=soAPYl3fVlJK23bTLvP)@cVUGDpvV6OJ0htGgBOR#r zlN~;QTX&@i(r-f!<<0};euIld`$KP`7#N&8D5j3r&7NbIU+q%pA49k?25A66i07{D z0P(}HwwTcalcy-U+Yf%(xPYFFNq4h&Zk1cwzJPV4L$ICeeY@*D0OBxwULbl8p@&BF zbvXJ)kCtzTr8fN83|8o-G*F^Xy;7ap{d=>06bJ#n8oxi7t??chGE?WJo;i-q|3JQu- znWO$GXtKT{Vc_GwMZm4uV^;-2w7oLxyJkQ}QGCT1)HW0-Z{ZD_huj@35I(v7sfWN< zbUI_$gPW(`tB6bU<_8;%{elZ4*#J=uP(PCcksVAbiudV1cYNFp+%8gBg22JBbVW8Z zajQXfchpex0~q^+)jjoCa|J|#?JjZr|DFISt!Z3nMRDB!o^Wc9lA4)a{tC)X8Us@J zPc)K(8zfTOgL3#>1A!-C4jzpdY-*R^Xr;*FSUnY}<75$%G^iP2fGXb@t_u1Eib@-x z0np3FO3Mkb@-DS$t7M4sXOhFtlX(k$7)}WwN^zFi+ndYSMLn&&p|1GoOy1`US9z8KFp$uRhg?94%{AAwi9X1&Y~?R!@C382gNY@lW6HUMbVxQhR;=JhBZ~sje-7(z%ulO`cFiL7J89D@!MY z2V(?*g~DchWf6S@J!JN}SIRH2z-jIbP_lRsyL3P+Fwu#b&%a5WdoHZST#u#s!j!a5 zH>6fww;_;_C^fy4Xn??Cj9;iKlbNL5oScD#L5-s8KXV35TLjCrX{I6>T(d;z%vT-6 z2GMk|TX^@lz6=v&#?T)pv(rtupCIQUQTGngydAJ22^hxk6U z=)|5QH@3WTUP7!g9Ns~jgH<4naIeY!)cg{%tDk2`NZuk$@JP#pxletKzM!cN@RC&9 z*{<-3YoFGcTMjAj9BVUJ)d{L3)TYqteA^R30e7i?kMoXj}u^Rug51Eh`}ayXFf`m zK0WQycl!W34F*+ZMhi(1MxH6GIx}t%@oG|?){cj?qPIVs-3tpuR{E-^^j;@Tu&h4qo>tC;24C^tZ9T`kgaAz zT_k>e*JQ&2z2bUYDbNQ})?(2gr@{bK~_0qU*?}tR6H@9gYzFd=#+)_b{19jXKVt zErGqo@ql@mIt()Px=-6gSiuw~lMZ#x**a|BcrhHR3uRRo5E^I*T!L;O8iC)e7Kwf{ zqjZA!Ak}A?M1gQ72e7%!tjGftX9>Rxt%~f^_Ra&`b=69-XP5f&kk3aljpEiYg;w^^)F5G%T>BX1%^T-{@a=Syuw-w5Z zG|TPqoHX=8f~0FBWdXPU;BG5(ob6>+BBp7-_=bBeg4&|otf7*ELN&5F+$~7%(dH}e zlcYfc#Lq^?5)uTu6cltMEb1R2IX$|EytMe6G%Xz3WG#Y~tj%|$SaA-}g!}by+W<;W z=ncN~8u?0KszYh;1X4vbn$I(Hl!4OG*ls6c@Zoe;Q4-U%MOVq*P4Jg_CHLe&dc}f* z6aYq`&D*iVAl9mm0mZ~_bAfMvXXC}-it7WhD~p+XV=mz4sVQKRJc5F{v?!aArjzp^ zpHAs9;5*xuh&S&KcVKbyH*nSY;ovBU@C8F9`pARIU269+AUap2^e@eI}}0j^@T(l++GVr85n!cOJ+mi;63zlM^R2nPb1xm z%{}an2xQo`qAXh`6P4GZWlPmhwh3-sfimVlC_C7>Z2Rp>ZED&K&2JCg=D%E+kqh`1~ zp0qI-#Q9E6yA~UnCYukPo!JSxWDAiXwy5!bciy%55%t{l1z)P~! z_`>;f$*6Un#kxf@0FfS9lo9L7WDBaH4Mr}@!sm8-DT1JQENrM%AVhso*ENgO3pQi!!eHKcj;Z!}xH(Y4q)!`DrwlyAaOarBet^wEZkP zj)DGZ3g)Cn2FJfMd5tg$a}ej+3=QCFz(j{dn>L792HJjI)QQSbf5elz0Y&XpO*uU2 z?nf|KQ1E9AvLcYM?KYCjip!qjA!p^#3oVG zJ1@(JD*&z>WGQkqI9}?CoQBZ6cyz!)I{n4YzaTWhRt?~z1-%d9RVZZYs;MUIBYUza?^ z(K4Gvp?s-m_Uq;FdaJU0PlOE--cqSRfU6tulSjc(uw+93LK^DYAVqmKBP40K(hQz~ z+no54@j3w~Qg}Y^widFmxNEX}Z6<7#1QO74crU&hi69Qz9|CaYn)4pIdeksvc^ClZ zr^-u^uC8NsF$p^X&ENi-#a@NO7VE$h01n%9`P#fW8$_ZFax`|19O2v1XT%`bi7j00 zMkrwe+Ja>DG=a%vz>$pSto{R5D>IL8_hhh_yC@Euy&8kag+?XV5V`vcv(EAa9G z4_%7$9uZ>mI+51}5y(smyol$aCxS2ru>q7^~jNn4xGJhD|=jUh>ZeCi` zM8NHfqAxd~za{c|tJqFp80n-H;g393?Lq?U!?834G>x=J-U{!Z#rwM|stu6OL(^!M z(|fK2Wuq^LquL4>z<^1Y$2~6j9iN;RGRBG`RiT1IckNY_cksO;0h^G>gLtz0x#V{e zTAQ`tZ#2$+O%=idq4Dq&pBGOGzk-Knq?6b0`!$om5`3PX?cN&eGy_%U%Q4VR0Tafg+t-73bOHSjt)wqOJPtIyAc}6zFbbf4 z6{GKJ4ee`BMX&f)A_jfS=gjGtra6$XDqmDY{*%^;kp_K=w_zSccI1~9S65_Cs_hKS zTDHeC(E(Z<@hn`~(_&-> znyo1jsmN2K(Tr61VWAu9KGv+LXo4nd<{Mxv^-7RuH{Aic)MkBcKA`mOEN@D>0nkw( zkTV=U#yh?e;-OU;h60q&3 zE0jXVT7Gep%!w=Bk}eJwt`|=EO&D~PI(u|3=bK!1O?S*D7r-|wb^AQN`OA63s5THl z5onuI)bdcB!cEN&ImLY`xi$Z}HkBh_1W_}18oP1t2H(zs~yqy)v&ng`I$pD4k zf>`LmZ_%Xj+8#fObbo|$TU^j1(E&;lhgohn<&&5Yw%K>p=Y6w(J5u0wLDG7-lY(JE zZy61y5MPtBqk6!O@zI-zzedtEGT>S?e3WMf!(Tt@6fq0x(X1Yj4OGYXu8|A zz2L{Y7~gRtO~C9S|C6IxP`I~0{fEnPwFWnGZZNtOXT`Y)`-;Hb46A1gn!OCnM>yh; zFc7am_INQz{#Xjj5K!%*au5v{m!ycZ*-t)}6TLv^_n`$OF;Kvlr>(x$SpKWMtp*N< zupTu>EF^as*}Hd0?6TFU6hkW*qUf+zV{^NcFYyX|o)tfj3E6!zoQHbp#pjDAQv*u0 zCye}~%?Y(OlJWj|^&fX@e_~XLU>8gf{>k`F8vJCQP{22NluW5_+q6UcF(kl@|BZmf zFpC~K+VtY<52M_m(Vyo$SX{J)%u)bu^@kG^e(cQ(8d1;o?0&$oU?Ih`@87G@#!nC2 zD#ycjatVj0L&gThcDc}kQS;XklKk}g)75>p>G0b3 z>XyetZnn6FurD)d41a|?dW%Vve~RIoP+?o^P7&tr6QcLynTualhjK!!OKK@0WXXGV-FYa;A)kTZ#PYnHIjx9FdX@A2hYYK#kmok# zLzy$SQ<%;a^aH_0gASEYk{1(zR{q|>9W;wp)8CK~V>X5zW6$x4KPp@Y-D2g+?r~u0 zoZiGm*xT>-{?wql(^TP`gNR+^)`tPSMNh|)1%CQm`(DIInc(rnE&ed--GQomv)MvL zV{Bk*v)hAj^@PMFJ=B=W7eU9ts0N>(bX`#BMT={(g35yo^bFT9`2I!6iK(10|KVs9 z&cPseg5t#*m(2K6>?KZobEd4eH9(dXXWMEDU(3H&FFGE4#kc9$JdPXP;fk>-7HM8G zvx%07Uvm0IV*|d|+N6VFT8*xheHV;{&?CEk;De-vpX7W+1Y#PhY2YQ#?hTbg*H3>r zAc+nhKBMpRX8St}N_Cc^D$v|}p5~+`V9y#LFnk9ZpA_HjlDF1|xqDUFcXQ%w?vsJ( zQs?`ZS?+BL3XY^@fy4kCAnj#_XT4`XF4x=iUcLRex>~^ewr$?LVVbXiJBD64Ph0}^ zyqEVNDAtFfZHn>(>Wf~CT|BG@SEbq3 zaV(ZUOf%-6Vq5%Vh|Y`04Y^rI7{2HQh=~aC;}%8!xmVlY#j`Q2Tdb94DkAja0wav3 zMt@J42{g4^+aiRHfH5Ej*7nm|*J{1tY@yik#M~^Vqe{|96ma_7WNqTgau97|5N1pK z+E~-Gj$y(FR>2JWt@GTa219Ks@^Hxvd@v&vdjdXr0Ou5bc605xj^5v=I>s1nLv-L6 zRJe?;KjdM}X(+pQOlmSpDE6#4YpYrCb&(9Y+AJaYoT0qrfs|FH4m+(6`LyS;0TKkZ ztOB+~_lKG=Poxf=O%%|OQM6#xtr^E%gySNA;+H7tQ5Yn^=I%M)H;Wd?e8wvNllS~! zS^(4X{!nv8!p%*e_o1bOW+gKRppB&0=tGIk&2qbo9>Pdc0Fntkve~IG#*~?fb%_E} z{nG7DCCmxfizJZRKdlsQ=3FfosYMUF_Wo1ciQi>rcNip%-EU#+R8VTOa}Sf{YIPQr zJSDwnoQ_P=b}$Uv04^JYr@szbQaLKN_SI zaFPXpEMC5fKWxNlk9*@Qd)lIEeV5lsC49ub21Vq#Q8hqLn91#LAk_3TvC!P5s zeQ)%*jN36cEX&QxE&P4m;i`u`%vsn^$Y}&lj5|8F@;~=24!F&Lju*^}BrQbNxZ+LO zcXUIEF24C)AQLd!ze=&yFX6XOU%F<|n^;@ui{CzcSz8%E7;52F3L zC3HZwz48&mB&GLX|3{ZHcjV4V*?F#F_nLrq7ZfZKzXyTmV8l|=#8y)=O5ou;M;B^M z(hi`?Qdz#pQLuzem$c?|e9w`vJ0a>!xZrni&WqFUgJS7=-Q6GK_IdF}^kP}d_ILUe zWr;|iV2=kQ>jucv1g=u&p8?o~N#K)Emh%iT8-m8XDVB@If8N#;RisemqcTjnljB@0gxM z51jQ4e2B@NolY#Xs&^HzSPzSPy}=nWnY+3l->S6cOPbph+dRL4)V(4sp65X6s50X` z190#}uiX8f92+mNHe|T~(pPnSX~JzB zs95ULc{UJGQl{%wnZKcg*vMq*AYc&TbUed?xNuI?pY5{*$h#wd+;+Y%%Mj1DCDtbJ zE0j6=6$7K^8!n_xV098dg~oV&keuAeP?LZYTlXqn0a;{S8cPdAM=BEJ8sHA63WKuw zhGqBuMol=yBP{yIfjamVqmZJ)?9S3wIEeu=HRYJDk$_mnyapy=Knw{#Q%XxMNI;u`N7_dk%ohbJr^6NT3Pit>ywzg%tEp z7-?1jNi3LE7tt5j6I0$%Z)|)j|J+VI&VI?Me_@bDZg!r)6OSPpgudeGJ`% z9hJc9GA^`I0k&>C?5LzC5$n;90;JCBF*5ckBUc}soJPdxtuv_=d=Q4e- zEesl~JzK(e(8qKEzJ+K`K^)0c1k4Cia0S6?OVS<+r4RquJ-^dq)~qVHhzP^5D1l_E z%Vsa=shSS33GP$kUV{C~bwHEIQl)xcvb84@*kT9855JrYp7gS7=slJKAm8w--*N9^ z$Dskn6QB-cdi0~Zd!Z11no`;9`7@^ z!Hjr;pNa`W(O;RI@>>Tqa!^%r*dWD52?|tx(SXvU@7JF_XVmuTh4OgZOEwdffOLWo z`E5e5JYphla|D^hR-I@&HpuG$mVGy5kP-mGr2=7$fH0mw z7!ihzfJxef_0WW64Dup$(ig|8o)Z@se5;^zG*ni}C`K|0t7y(=D7cFLbM%6-l{RBZ z-aq>;7*G5;A7yKD0Su`Sy<|O>&h@o5V&oV$O@JKTSd}p!W1a!xvc|>*Ng3H3yqOs# zPe-7s;X88*Ru83LwI>lk5+<^|1u>*GDXhEx(a^E+TdLqMFa*L4pBloTZUlmbIyKF& z-wZ&a;PN3YJ`}nLub8uNP+zAj+t6_hOymfEUgo*TD9WPuBxDJi zfV(!Wro3V32|=!@=b%x^4D0?l`3De7qv#sRBNdO{laEjKwVpI!{2TtnAstn=8N_`& z$WSI+of97~22DCN9@}!^-u-=;3QttwSyr^)SYGbeo=akL>vcKW;8FxBqXCjR)h^KAl@CkP_vZ{ zB4i34V%)Xeu1o}sUGcaNY^n1E^u{f;34ivpwUII5HV*?B5CzbpL2tR5w0Qs?%>%Ia z7E?5UWGg`W%RwFfBs{vzSZ4GG5E-02a|>Jqpt+f!k9Yh4gGSO9oP26`GkFd)Lcm`I z7QA(BHML=ce`F;WJnjf2JM$Sb76y`H#LG-4q{DlCYE@Y#kmSc&&7{Ay7I-rv$nu|z zeh2~pEuOxVcR=vD$SXv!G`W=b%bzQsVgg5Ah4||Q_*LXE;C1}*!tE%d6M`m0!8Wz3WRAc?dz@TvkF5XEpGMxssM!+aRc4%mT82W1)h zkPO~fp6`r0S+7(A4l&~!}Y&(A-c^5hn1dBB9 ze!TpHRdPaET(1T0RrP9LsIek47C1s!DoJ}FbkTy%bvY=Ud$PdwW?e-jzlW8x2f)V5 z2t0evSZq}M8z1lI$!*M&sZ;>V{dXOLw1N}@TJwD%-eo7F9g8Lifa4}v*%%;6As0?+ z*ezGtyij6fl|I3}?ZJXtQoR^BtAUpp7nby*9_nG=lNL zy%b@fLlW8{G(AYjQ(ZLj;s2<5?szKy@1Ki|th8)W*GyKjXGSR_DW&WYO(Hv736)Tn z>=i{Kv$DA;LZW0PD?;|jUcd8PZlCY({^8}`=XuWiyw850^M0Ra@78nXDqBe*?+XQG zoBAv#+t1mPCP=)&T)5)@p#}%@hV`}(00tPLX%HhUBuup(^Hw8u@f>S|o9l9Q(7pj{ z=Dk`07}=S6hV2&h@TKe36({mIxgl2Lx&=led21as zo-OU_>MDd(WOVgx3edf{{#sgS3&wa_w|`ZBw{qH{jT1#Wj?oh%(|Ollvny|TnGS9c z?g}w6j?WM-Mo4Tw5(3CTWjBmm(6=1v$>65X^671kb+)O zN|mx)_xfCW@r3ekpV|Nsc2uxn^$r=XwXCfS7t4KO!4G~u3X>A*g2P)!m$IE=Cz_{| zAKqR*N|V=}Z7QB<-yX>w=Dr06XaD`?CQqG@;7oEkpDZl5w-1sDKn7rYpmBf_ zH7Jp_zdWJ~3#%xRZCqy750ZF^g{2a<^F!hEF;49AiSC8rt#JrT|Hd(&TDLHEOR6zf za?yFWWuuWAlAc0u`J*Bnt;1Mbu!F=rCW2J2Zs{c;v2eb7Ap^Vl_;&wA-LM-#+U@-Y zsNdMakzo2Gn07Pk!$=6}uxW~$ldEgmHn$MZm$z2)v72X-1t(MZKt-Gs8Gj6srf z>sDN6S1_%#xJcY1SD;bD){og>Lboq_LB;%HfY$Q$?uD(?2t52xRLt8z@%h~|K449N z2itY<(vU56-2xzX8WKTaYp*}N@Y2Z}j~ONA#$gAP(LX6b@8ISuI%hcxr|WHqPvIi} zqnt4ut9)Xyed~#rRG-a1F&C2ur1>G6XDH%wVasjdm|=;qH8vp=(y1r&3@u$4 z@W$Ju5rqS?+fyrAelo!lyzG7%VzsR`nH%d?2nRQOuDNx0!SJC`$&;cX54D~5=eP#9hd(?H?6DItJs_aFr9M|nk1iUSTq#Wkofd0I)XDvnH}L-CMGd5KftDl9q7$C{{JA>8}1%28~C>vz@eu#w*YR}Wl||_ zlkkI$tXr3G>2jwX50Hy*y&wh0|DS-oeq6uQsM@`-m1%%3m;Y=~RNPil;GVLW%oY|W z{=?$YD;yxt$gr=#zw^}Lp@&9+_kw3+f7tuQ6BL4{n z^7$)qYYT!nR45YzFGynq0OZaP9^JgM;yd_3^F%ht?$y|nIWYQ(a}5x8^WcWN*4Hhg zC+3Q(!Qn*oL89$+1r5QNq@|Bv-C8}5&%d!L=95HCD5I-!d%P7ThN9l>y-x|8h%#it zxsxP~rDKwDZn(tf@lH&iGY1imX$P<(_)MtolAHTLxJ{%D-G`hljtewSPBC*^UZl>6 zB+ZZ#%$^p$p>X@G*4|{)zB12PsN>4!gSG`a8%)Z1ywBF2RY(V@cQ0_`*A77iz_XFK z{~a{r0)&TX>0yhdyhP8K;cwpxTd}sr?t8{UEW#iNZzmX1t&mpafELM5_Wj<<>m$gH z;>S?*@F8jEbQQ=;=vEm7IrF5umPzj!(icj?-t-Kb;n;Y+h3J__;9q|YJV5S}_9PP; z(6q**X(N;nxEHrptCv=HAjKmLxj^qMMxxRa;@@ z+Bz9-mBjQ?I-eve9Gdch$FU&$h5wB$^As`PmB`!20~yM*s1SD8qAo!8EIdzQu7UxY z-K9E((EU0^#zokLF60QcOc*!F6vDqS?O`}{Cl)2Tdm3RU>I7i6B>UgRQ-~NOY`H9n zfjO#M$|RpMeRo5JQS1q%5QX}P%}M)IQPdMjy6;vF;_0^{gjrl7F2SH9(M2-E8xEcj z2ku*5^W`$X$WBwZl@K75LtF~Sow!Kmu#XkU{~AG;vOl+s(g83OeX1nVCDd|{U;+xB z;<%T7kSc_*C?cql<}UbMJqtm$_5vt09Vf$CIO{lv4TkN$Tf3z=TKJa~um^_3%sEJw zi-nw~nf649Ijy$gjUo7vfGIS=7XK76?7-zj((1DVT)imdlrjGesQ{_Z@(o%sJX*oI zd-k64Rz~H3R^AiX{QHkG#V~$~`;cH8j4_nnX4k*PBMVQ%od(7ZFovE{1W zjze#fl*wtB@FyP~mHOtCYZ7Zz;SP4QxrbCfoMAP10w`a4E5Zs#9f&p@FA~F~-uUHVxt?gxMA*TY>sf zNw&Ypi3?N>JsIlRvL8{oPcjT|JqaCJ0&v@EGiNM-QL)l~mQ~WVkof z+eS&D=_CyjQz;lwCUu^uo%VF$2=3;8msMyGHk;QJWpfrvTq4AS;(hG8aQgG`KWtM} z8JiF2ijTj1NYuTxzGu6U9C{L_!?G91a5Y7zW;Iwn`n@=XP~Qo7K5JxxgXSbWBOc|k zrIV@50g_ExS?81I%B}YicT@LWCg(-WFO$aVVu$%0ZLF=niq{kRdbLH4xZRmh5C=Q; z+bFinf$!4*p&NU6M>0d_W|{?`q4Zy52d;PR-i{4A5^y{!egH-TuY4{!jBkrYNxp#h z5fa`Lz!J$`$vS?Uk?`K3f&MoVD4exd0vRn>TOkbL{lDSPgts23dw2?w?7FM4#*xgjG|n&aOT;byz1%WiJZ22)PG z{jvFl#5cF^zzR2pfA|55-Zt6bMeCbS`M^}JZ!uSJofb*Z*j9aWc@U8!N4XgzwfKa- z%5i)3qH{xMWtBXd1E_O$g1Golbn;(N>0 zX7W4L0>=Y%=0kyI21J^@88JhqZ;&^Rgo6zTA6+c+9g|f?WK>0s7Y(U5mbXGt@8~ozyTexcem7c=-R_Jqd#OO79n(>W{tlbL=!8)jLAEKYHux`S4bMXahh?l8aD*`fD$C zBAGC+rnwo1m_g;RyJwze!-2S0*HJ@H*nsKf?5PkyeGGg}p*)2U76G6{-7AJna9_l1FgDA<9)b7Yn*g|lvq8$ZnOaCVfIxOH#_KqA;?GqcIJnyA5-Ql(scw5D6eLr=y0G8K%R#riiIGt1cO%o6%*bfq54q? z;$vYE8_$$@i?k#_E(*%4QN(I5gtuP(A>T`(QvDCPBqcaGoM3~pYu}gC4OIc$VZsN` zq06Y|bGo7w=C&j$;7V0xBPcH+6r&8zUdP?Xqbcs6iLp1MUf6x(&Ph!!CiQDQFC~w z2xQF5+<7|qc4I|S*-Yhq$I^;a-7xmFDpAqCumB@U4wB}d18U?T41Ck~DLO+BU=gXf z!*IhadYlve^qtT!iszHvt_max5*-9mg53t-P(jPPbt9W5=x031KourZDiVKhmP8## zBxz=o{M!#o2BUl+nFI%AdJp35Nmot5LHrMRr)}^t$pRU^Aa~bEa@Q+ZUk}QKepFh( zQW4dI_*MB^60t6X#1K&;S*NA1U>ea6E&wb8Qt~edNCCS~{LmkTb%~1(DPB9lydAzv zYUmrLX?;0IhBF}n>mWIstp_cg)R6@+EW~d_UrGFrO%rTQ*{*H#xYP#XihghbV8+Zv zNi->csme(Fok$bUf3kVxA5FeI1K$1zCVCrR9e5)h>b}`YLU4&_eu%KPh9;TILWm9i zXNW^2?%9G_KtDvO&XFe3)>U$ZsI9ovuN8>7Jqgjt((U36Q4FxMP*qqUxwr#b`i3r> zJBZL?t^nBaFp{OJ|9^B}C5cYXFN&=oD(NaJ$%jdkc=nGSfF{9|WE?Xg9wL71qK<+@ zC2k{y>p8p?fuYw_{Px-$gZkM5iH8S@h$~Id77KI?dD0lxFG%x7qbx5HSDQ(KQ%e(o zjK^c zBwqiGCygNqhM;d3X>#Qx%!s1wM-qV*UB9i@lG{B_&eRPGYt^TGDt>x(InSm%ma3$7aqgj zt%(DYAz-Ob%y4gRELNPyn6EN#uHK`L6`g0kI>j(zHVcRHogDke+V_W(HQ$@@c+#rA z?(vxG_k+qx#Y=&^n2WmZtL#CN%BCX5W)2@=Es*Nvk(RDDYUA$V_D${aOVjFkt&T1v zHqU_VND|1#0Nlznb!jHT?8$%sR`Fha$dBBeIjx%9LB_UJ%O$v}Hk52x>;?i73c z+UwnoM5|TiE`52?jdIx~STl0CNonxbtsBv;f)8kw>5*Wz#B%OP&G}7h&Z^LIGx@^K z&7$*MZf%No{oXAzR&h>pKB^{pUNS@^(?zLx`w0vR-G z^GjZ-LV647TtUo6I~am%5mD|)%K4H(0ax~xj<})UQz1ciO603$KD{*x=2oc0stos3 zN)0zsiU+zRIx|9Vf#5@!1tldfWhr{gFAnXKm*oq;Z~C5?(y8KX1bw*!E4}8lz)Af{ zzI0J-O3A9Bx>F4u(Tq6m+Nt?lxG~0o#YwM|j$zgoySe+)tnhoBLZuClNcV8tLwIa! zgE4h8Yv;HshU$df{A$|)*D(oAqsEj@;aPVnwLsc2K_o#@%By!E*CmHl>5B8O{)6`d zbD+Z^b~){oWCV4%g-EuW%Ny7tQ z7pI=>S3TAs9h6ZjO?J6w_=MJm;lm*`y1*HhS-lzlswp1oLd~5Q!?;I=!p-f*rBJOg z@%oPMBHT`{qx}axRQ`@_9MhWB<&9Yt7J+$3!`LsieCGrAV2kU1>Ek9t2K=y*^MSWZ zjoQ-KdtlY|(h=eGN&88+_nKtEYT0cj{yDPH-bMbeN%oS&>Hd|6pwj(q!^+&l*Bq^8 zMO`wE=^QGIwJ+!SYQ~g8k5P3Go-6HLj$N1IBbWElD||IY6Up?;E+|y%iq@6m8!8R8NsTRzqNN5d37p-&t2Ot= zu=eMYizw(U(iTh8g_<3YT0GS3HfWn_nU3npTv0sTZl9L5>1<{jK4`4LHfCk$WN~zi zAIA`?73;+{fWK`&mqTX7ZBkf!B-y7MavLgXAUHSO?M$b{2CWZyuONx`YJp_C8jC_u zBGbHo>%KGCM7l&gLGR$XjvYR&?StBaxc!fn*&2T?ei3k>Ugu)pFJ9YWM;4IN;+b$o z?B-%`LN~j!UUZGt&Ktw0%O~b8{_W1XZfcg+4MFt79G>QQ@vB~rS?-Dnv*YQE>X)Zn z69?{3^MPxV13zsy>o@zm<^!(oTEg`*wcdjn@sA=-IljH_0dB~}NvG-HjgHjoK2HXP zVwz0mUzYP_n}n&5Cn}zpUpjQ=lhsDONlNGO!y`>H&&9uiY41y*VyB>s1#C14Pa>DY zTjTOtv|>Hu8S`QcN*70UA9nF!rl6UFv`=svK_~i9^rUO=9Mxi{~i})lTS6%c$K23WW7(aRDWk5 zwN=vrSDhh!$r$&})nCCgbaKj9)=u-rzyw`9dRypHC;nm8?Vf1;+N>*TY*iwoW#^mL zk@AYX(cG9#vKXoT^p5Sr;#|3+WPy^)N-~e;pXq>~Au4V7Ltib5T{0orv?r~;5opD3 z&R@VErXSHX9%e3*e{&`$rPpfA*|72IfKblVZQC-f;6<&4`M4tQ3x6|dCO8l4=3fez ze5EW>nk|`#;~D>J`ifP%i-tvNEI}d|X0>?PFt$BfPigI#;-_nGYMw2=Fujy0FuFn> zqPX8k{*yQjhhyeyug0Xue!rE0XZ4bhw?S2zAkOtzZ%hlbhdfGhr2U;T5NLHCvM+H$ z)%sae^M$vPgLKUDcCua5CNJ?$H4fa_bnZTR#o;e`a@`K;9L=9*ES}iM&J1xre@=0d z(L40}|B`9F%vF!Lu;v)&92L?H{Pkrub44~wwQtuz^MT|3lB-2`o}O3!R3ac+)*C<6 zh4jDL$ZVo-#HyT{0v-n5J!2|Lc1%krgNsF)k6t^qVq8fKMK2BG)sed{l%eIb$4o)9 zd6(OtD5s(srK|kTF|(g#gK~H{JQP?40h~shCiNSvx!d$=8pt@mc}J#ay(C${coTGb zfb~-9Ue6-$_UlrqU)%dFxbdHj0|{KU&oij^eJjtM;mh#bnW^#Xx#!OVu9zz4S7$cK zF1a-^i@lfv7vQBVWA=Ry`Tl#~c9@+Fq#x7rdx^i{tjwZ8cv5YW5e6Bj6o4H;RYbd= zt*dk4$6iT__wJE#EcSL73$>0W7O_tSB`b1AIOWs{(|wEQ8_j>{5^{_d!>>}%byoOL zOHceQ=CnRm>#}d|hC+8*=1}S1JGjUYwK00ESq3HNH1B+IW#-E9nok?uEt<BLc4LbaFOK^=3$r~M$p-C{=ElH6eN2z0Z6SK~F z7&I!=JRKWcVQXUT%+@d9x=!2F+-y%4@E7DhCx~i7OHsD3pKVg6d%D*1qX1a|M^7X+ z1sIpH_p({OAN77m7Va=(BLT{?yHas1wj*^V{lXbJ{byxImb}(iO}uv0`g+o^aIJB@ z`Claroa3$YU;M*wmKfIYA*k9o63Faj)hwyIcuz-H`oYD^qGT86a6qT!Zf;u}J``})Pcgsqu` zx0)I4pbrjfo)EusHdMUl(4Ur){w}QjSnjPWd)O{{bNAArTtK%wuR7q*`H`KASQmCEf%`mYy$!>l#^E=r)@XjTk z%)9m1(iWrnB0^Ew z;2-9)c^spp&%0!hG)&&KDDG-eeff9CDA(K0io#Dq+OK>t*!Bv)$VHukvCxh?E_7@keyDZ?>J?pe!N zS-n|TeM28?^iC?ZjGBn~gK1il?tOyLyVRtD*7Q18ewyuu>Ll6yvD8xkSFjGg9B19y z8~t~wTQ}!PD)jX+(=d{~y|-AJlHPLSz<7(%q=$yq?1f3Pz=n`sGf+H+WFHK2;x`3K z9?AB!pZ@gBoQ2tsVTB&jAQ<4Obi?g(cx&3v_!}KHjnP!F-o@3d z`kX>;Xzw!SHELs=&&b#mDRmD1WVo64 z)&;u#2bzT=D;YiA_WOg5V@4tyJl4jlFDk#@%(5+?xbr!7W!Tg57YBn}s&<)j)z?hF zfD6ve&^w>Y+YsF@=5Ynz%5mQIds>rmZ7F;A!h3-XINb_@01iD$-0obc>hNoruT14G zk5@`{BXYVDWJ!sdimYR{M)4xN#wRbt^BaAazecvoZV-2oy78AftR;|@@k|D_w>>q* z!9F#m?CTB*gYOyG$ZEO$t{BaG--l8I4oeo$EI+T_eG@PJ8l$2Y>><`B8dkdSCY4GT zUwXLy8lO2vo?DNSz0rm?E-BiouRZBGn%ZtYl=6Oa$7t|e1Q2~gR%_OlGiyi4QA(4K z9*}qK_lu~2@mP?CQkM-y|LVl3%!;eBhL7s`{a#ptQ<3OFW zWW@6$FytF%K3(`xTMGsmDb&Bb*{5M-Vts7!N;jp~Vrn^;NJ(4%4l7_sv!SloeXduo zT*bjOWXMH!=Np-3iI)V7PbS9m7rn`*GGEOde5kw_iF1bv|bm$ufVJmFkr6?jf-yUk7=cFvsnoY~) z!mr3J<=?_%^tko-jBcKqYjY!e)eNPiPpu8ScT^$3r68 z={rswa@R5kF=`_MY5U0tDVm%tV<&E3<53vAsc)DMiKb^{T%4yxMs=?C*f_7SHgiHZ zy$kw=1_L|*kyG+oHMiryhgVL$5^l{}cvq=V3UvAo#z5jbr_1ba?6)9C+_bOjiDlNV zT*XV52Twy^BN4y|tsK;2YIcvL7ijST72!MkZvdHeh^>2Au>RVULB8XhIl%=`NP>V+ zU9j(?RkJVynqqlBNTD{ei=P!k2#eoq(^~w*V%^qM{QyP6s6t&{aJ&5YZQZY0Eglf6-JKesrV9`UbHy~t}gFMn0)`wqK zA6|oiH{22OD6hC)$oLfxeV43O-Ye+=oKnS6hDUkuHn-t2CXX@L4YJSX*!i6R`_g3Q z)z}eG-b>-F!b81%FHLP)+nFS`XrTf^>^M(Rc{5H4s-V|22svD1kY3M>}k1HM;wNM<^@%Y0P(G{)rx|1$L*(s@9b&EKe!w6RQsMK=+!3CqVz=zG0m3q z$8&7rtf9vCbqQzDO;@Nh`<2;!h_n|V!=8EkL&=KdnUMzh0gW=A#xCk~@?NaLrHw`#MFRLT#$pWOv>CxH&3qa z%%@%69DZ40rAH~jM3l@1?N%xIUQ!QR0pGA8)4y&zOG5ps#$zPDUBd|KD!SqK7(#9` zYh7Uyrbw6+&S;_AJSh>67DXX@i`y^rLlf^TI?PAk`+2vVC8q=j2+rl-NmGwiq0Cec z2I$g@gaa4V`=E^XQoqEc?fEV%MUI;5{H)(9U2Eq56jb>WoJt>Z_5&(*Pa$J@VrjGV zq$7JJgAA6R^?13F9#Pv|bP2mISz6(L9M?xW;%g+IgDiZ0&1#+*Sj{By;_y_EUvlPt zE$<8Tef|(&lTl5~K{C4TnqT(Omw|qJBwn;jRr$B=foo`n|HzrC;(`ZS z*N~2P`f4?x56b!19%BM|txtCKGK*T(-Z5Y70eOXb<2fQBbG|;`2px4FbUnpz0&Wt$ zN%8Xyy3=a`a=UC<2?v(s`KZBfdmV3|15$?ne8mB+k*TsJhDS9f*ajylAUgFet zi`@?ps=vmzhXQe+PC4{=j#(=L1qf85pL!IT`-(=UR9eI6i0}`vX-U4Ct4lg#?u8m~ zKs7*9u4xpjHh2Ec+3x(eM&e*a#6ZebKyxlbCB%<~(=Euxcdcm#f8H0K)d`E^~!eSzWtu+YJ@= zP8qze7z5x}3;9HPi+f<4u}cLv3F4ZPa4(fpQ`!mG7h|2G68v7yh|zbiOLjZZI)0KM1zrDx zM|m+HlJyb0dl%%n2B?@s?u!^v@k_qwT$E1^v-gz>edg2VWCB|XwMNKeeOs1*40BXl2Ve<`#s-n>m_|hox<10(Wg8<1pt?u|}iJUao z4>tE{eQ&uPL0{oRV0xS|EBfcRihU{Sdg&ye`azuz#|HvT-W8QeiNyTY;AgpZQq6u1 zjQTr?oIqFg0jg$RV36aiC?F9ixzbr-F2dk3T(h=UXTThl=X9V1Zd5BH769P*JX{E8 z)Yv?Rb;|MuO8;`sd|SfY>Bzty4?vkwpW?+;#xe3T413fjNsvPuev^cJB}d!Ax@ZYR zw!_!Q3kCiMfY}$ouxFa(irkZ}2r;ZB2bDsj8^i_*erJ zz<`FO3WCUE4cGMdL4EnMzrW;*=k2=qpVZpZpBJmaLknU4-hTq*MgnKm5>JRyeY)22 z$ydU`S~?ekM8poVyfV1t%q)DVAq`?z`gQ+6Nr0q7#QCGU-n%QusTb}V;pRbOHwKHh z%ZwhfDihd@uz@}SFps*kr@%YoAj5ZGF@rzT=K?(M&mV6N{_}<#>>oJC>h^0%{GkI+ zv)R~_et*mOYmS-XoGwgB#Z`JG^?F7r|ps^Cs=^F77Z= zV=Juf0)?n2=+%S*?=fG(KF1M>kzkAkAbzP3Duny0rProhM{9IHK&}_Y`1mq0cq|&I z?tDw=w9?_FzuK2;>tjK~#j4E5BtmiVa-y}}PYPW15)=pje$pyqjeEyNC|i6$?Ropa z{MztmH&Btc{(umaI$N;W(8BN1@Gsw7FE5E)O2i=%7L0EQivzXTDu1MRj|2cRf+;9E zg$-)6;;2y3c`4z5v>P@0fqcBZ2XIu&4)vf0&9J!8_6D;EMZ&N1WnZZ?Yl(B>I*86_ zV$O)h4pYX_XM1JnMMs+4hm}hxX{Epdac1x*T@k`KWHX*pp?y5b+Qpn(a$s809?ZrW z%F#qWn7Ugy@OQlT&2D5;5Sb1R!y4F_b3?=ht5OhAe@H1WW`)HGk`v0tZ=7Z?U71W| z212~I8tzKSsP+%)IKtT{_$BP8068)qENf&Q8F30xIZ>Kxf%TLK+5j18J`PD>H3-?D z8`W-vF)#nfP1Y9C(WZQAX0k5%(0Kb*r2ye%ceW$Z zT+#(HGS&*8gN0W)OwCdi@|{2Y^Et9UeOlfmUXjrqU!)k2gfShcYTJzb_UzXIRwoR+ zL+9kL(~C1%O`<^Ve8})P>h6}CoJpCB5hSPqibx}*vPzsdgHB?+Yb^Kw_$36MzM*lH zz^Zu0Cz}}x`Q&|_rpz|S^x?(K@laQjtvm4yaj|<2&>k4jbv`o!YgpJ0wYt}22r*r% z;wABL7a2kCdyo-IDA*y9LvEXEa5eEKi!$sHiht4{MEwLrz<)O)ep~*dz0v7;E#r)beSI&S2#~%y(7;2Q{7Ia6{J#pP~ zT6sA+V@9_?qNZrbF`i=ZiZ{7IY#pdu>ez}cHLmsuXaky(Hibf)D*iACNHKBxo-o*n zFJrsdx;^>hpo0&*SdqtiNdc;83LiWLb529SJ;|!)`xv3G4y5NJUpO>JgDR<=6h_UB zRel#nq3IkAG7^0__|GlDdT1RY{|w%=IF~up>sAaTaY%$MWkCjXzTMph5xLP=r2FW3 zvbE2!>*zQQlGF)?GJ2Hrf%kbD+{9G{@@6%DrY9rcR{IXH9RSM&(ad3+j#01GlvBtw zE*%J5o^}@)CpY-6@u2(be)PPR;6-7>pXT5~S5OOCpmtOfWvg!ovYc?hen5J_%K;~9N zB#$~TOfGu&onM}AS#$`Ms}+@d6FV~g0W0NA?qaRPWqQJa6?g0QBOYZiZjyD;kT#gO z1XF{WO}~38;Oz)3<*Tz7!l;DCwd$O>^lGDL{q=4LAMoctO`j9=UddXQtpg_X`QEl< zWv{@LII~vTAjPoVQ{hEIV(D3+48*ow9WN*mw$p@N@6RQg+EE?Eal!_{mWxZ&Diy{w zC*IDQOiR>(Igxoi*U`OD*TkOti5*AvoxNgRjTgt+ZJ@-@>b^mdq!wrS$to-F2OT|b zxc3AGN4cUwV}9L$*IZAD*QjT1EHUgS{7p$*^Y){z{dFo1&8Zf{Q7f&rKevwycPmPh zJ8(*-?1|!PRI6O&l#%>N=3Z`}phhoD((R3b-p!2yAwmm34c}Kg8&9**$hF_D+%UZ$ z@=dx>sf`)Qx{r!xX6ggk^rAmDzkXdE|2_z|bv6+-=XAk7O0tdDZvQO0hDg?sGO@~> zwO4YZ8JopS;=69lTokXnwui$D6Ej-D*sV99u^6t3kBvHkNn-zdduhYSmhJkb4pW16 z28Pklsfzb5Im_riU45#_bZFOiS01x?!H$8SrOGAs6rqVrM2e5%R@hH8QJ}M2-F;N> z_S`4hpI1g4-p(vb#@rj9vb}RjkL%|L>wQSKbVC!oy<;g$ag$8)i52|MY`acnA0{08-l$&6 zC&6RxiRvVSNXOcZ4V+2g<6v)0l0Iy?)$_3^)}Bo}mD7KE{a*ejp_7FpoyWvKUEAGn zIgro!nBm1($=&un`R%#LZro%(txtZU7T@hhiYj62uwU@VsQZs7sksn(N&h``?24w^ z`Ip45CnWNTyMF8OtQW|bo{heSG@muY)5W~al{w+{9<~sdoSN=dw9MUzg8gxdC47{* zF4Gs~oT?1VG#!|D)ClYQ*i;F-5;hK$JQyg{f;w>C51n4V9q;e{Dfu)inK1vsx28X2 z-iApRG7ylB?)Jfq2!$CmdNUZQ?|FP>CEVJ`$1B#_$+zK`f5oE5V&}U?&+uW#m{0fS z(n4?USH%P$Fz;BJv3$~6XLaw7TaQ&_)AN=nD@;=ShASxJVRBehSgr5Lc;}b@SL{sT z$9EE+gdAiZz4KfBR&hG<&8v*y3^vktmnE_kf~cMT?bdeePO=!VWx5$PgNJ+c!Y^^VzzXoQNBN1oeI(c}^^U?Bt{Sr( z8;-(Gol7o&y#y0U?jyq{YJNZ8uyeTcyDc|aGn0$Wmo3N*Jx3NV$gqK}{oywzeEf*l zoM6UB!}jC;o<*s;(tLD$raKGT+X3CCFE;Bl8Hp4l-wR8$IpbdW+?*T3F0cLMM5K@J zsuNm+1om_4uy&4hw&u87KabWrpi=VX-^+}r~NbuEjhZQ|um8Thh__d$mv{SKc6HBDl#yOgWwVtAroC4$3w+fs`9Gq+~1zRpuJ|r}iNZBTM z`l=l*=zKV1wQgad4Ev=zCOqqhN`85S$*gnPWic8xUOm#D7}`DD*)e{tZ~4C0#`tBs zpJT(%RPjFP>h^2TT74;cR2f`@kCc+LL{2UjMPZ|1gCQX92P=g{%O<>XQ(lIZ z$HT}2D{4|HCmOwTO2wZN(SnTs6KD&vs*M(M2 zlgMlm%4~FOtW28R)8L|+JGIE&uiJ5~hd7~S*ew)Xr^ z_@Mo>ks&n|McJG)JKK%ejqcC=N-Sx~bSdHdeT&L?Pu29DMEp@_dx8vI7>YOK2Xpt&gK^;IIw)@9?OpZfgI*`VvZq}M4vdbROKHM z<=uNb?z-EPK`AK?bI(L5j&Lt7+*Q(Y+}C*Qy;UxA>>~cEuCgm)GR#oV$$^)So1# z$)-<@4b3A-oK0S1wxjmP8Rj*`{O+>v&Kude)TwCG?P`42RnLejy4zqRW06UybShz{ zC_yc%b-gbm%IDW@4;jU?6=80sBN4rLXiGS+DR zmbau(ddPNhwXa`;;?YP=p?g^ChvHpJ4UhYf7a*y#C|D=S)>u+{B2(`e z$H(;3H*7D3a*rEz=fu3G&~q~@3Ng7EEB0O_%(&I}dB(+&j{{kd?JB&5NeXMa->~K; z*SQ#5Nq^9d8fy5n7ZYvg$}PY0-?Z3M*R60}8y1sau1(gZ9=mH_hGAP@GHvmm3Nw!8 zzqQbDIs6%Rg% zcZ`)dJB>cZdF_7j;zW{5fcBjbA^k&Q$H)p!n0rtp6Z^>)9V0Uk#J6x%@}8L9RQ0RP z6Rz8>;X=;dWbpFhAX&-RvfOs>{3CgXIVC9V!!qN@Hb$ms#)>CI^NvUq4bSK$wB4-u zTm183kO2S8)q+l?JLS7Rj!eFhULMmF)lgF!a#-rR_>c^_GQ&D^n!!&#_?S^&Lytk* zvggy~$+E#ouaCZtkL#S@Uyo<_`w>9@r3=a52qJS=Q=1)VHzx~x-=BRf6eL6Rr2w1 zL1!6T{iWY6y@EQzaW!T0>&g8>b)jeb$(5mtgSa*{3m#gIzP;YFNca#V?6A2QrqdIf zbg7^$kIBj*ej-1CR$pQC`sN)0wuIXI)_An_QrYAqSD|}aB|3SMjjm_uASKM=yUB#P z8-B0r0?fJ%mnabVIjOou6NBiG_xvUX@TR*<*k$*naWd|?yE6OYPl#EU;3>Fg zuH!Y@&mI0h#wcgmOX3QiU zN^^atd+GYgox7khg!b;R!M{An(%UhC{UN8?&t+h1_3!t!d{Lja`x3_Sz<04!XLc$p z{mWijelPp(J38;QXRD_>sqll)UsQ?$*2!nc0#8nMPTv?Vxj`%Zx&3IMdITJ{9l!?S zqp-iL{x~+m>kw3+q5YSDl&fb&;-uQ?`)*G?6%91F(f*4ia`DkFQL7jpA6GdJTopOn zDL#g0{p+iPK^6vWnj9a0nu+Egnb{PgryQP4hdF6?IBytNUkb9Sfa9cWsjQbgW29Fb z>fi0H#CAWUlf{k@s0CQ6dX116E*L)lBr+}2gEfK{ej<-X7{M*%>hHdYVc5k)s8x@HC>3Ifm_Z)p53D@{$W^I@*BEep%^hc+l;P zZD)k`wBjt)VqND?vv%>uC;eP-Uiml-fM^qL!n>$B4(|`?a`1|{O=#udG`cR-{OoEA#t3e9Kwag$$zj>b;z+nrc>)$q2}e-$UMMv*O_ z+nw(mKQWZ)q~Cn3^-5tyc|^?(cQk!#uALRO2ETn9TDc01LV$TTnevTwPlu&HipQDD z^|(z!#+&hUk7$Uo4p)D`)LT}I658GO7|_~U2!x8HSj)Jk9V|?QGu?QOQjw91kp9Obai}K3nX_5McfC^%QDs#+zfYv}1;IGd)h6 z*L8niew0MLz@S^5PW|XN?&da;6!%q*lug83I}%`NghiH0r8^^N_vP7Y*EuXCFz#7i zBCCE_sjOvbcne-;eXP}Aps-dEUSL(F=wisABMyU@Q{5u+ye_Av=`oBPKz@+_mRsr_ zfrno`p?RGq-ABip7-T`aau*(YfrM9KGu6+$4W>qv{=B4W`pLS%p!>_>uW|e%!XlN@ zsMMn+h0lsEAZ?2~0e?3ZR|8s64RN#TnBURZNALIgUjL-K{XDV&a`Zk#L(C(UdH)w0 z1BBZ-c5oF;qDJM3$=C*z2I{f-fTd-6{A1oBhW&TGAdC5!^Lv|z!2(Nh&v7fq0kY6NJ@%bZsmx#e9C_Qij;*f`AMD7hXXMW-AiXNc9J4x$hu;= zkO`yWJrOf{$L$mKrk%`#{&(0bAJs;iobLTm5Ma&=@*F=OPB(1H3+Z#DywVvzRh*h9 zRcA{-rlWj1`6RT+Rc5OGOGkA<;uiD4DEg*#sE(eje9{RW(gpp$mVkO*#3DnJKt5Z` zzUy!@9^TB0y)w|e7}exm;Z#4a5GTl-mjZ+APHY;wZLhcw)D~HUyj|`)q4IO!%-m89 zUcaQ~yUKHq2u{V89e4gs&7t?2k%ut4@9OG+Tp%;jy#^lO$q$&M;<0C9^+vF@S=C( z2TLr3mKc)(VFnM=$sCoe;nCbL!awzA>oj zBFOp+bLzV1)S6>Is@8-%U(tN7G%0HK3Zar-Gl@TCxj+2P#@pShGrdf@-O{XeUM>Sl zV+kNO1n5lf4&BwG1>8=8o8vEvP}h$*7{w469Ncr6>GNdyt1~;1oD(|(K3@U{be83q z8H&*U1KE{SE(cOCm^zVy~AKha}3#7;QH z)9O#31gbqMvunxQRr$(YO0u`f6fB$XJsv09{N_Rsp*lrRyyQ21zP17 zM64;+=p!t6Anp*l_tk^{US~0QV1l;*i-apJ6G6ZTOLoeQ`!Yk=fVr@)#+kRj2QQ```g+E%LkLI?U21ARk#m^8&Pp-Mawzz@k7)H=LIB zr~zG&bPx?R~#E^`m2vQ!;5H&QBNh_f!cSNBhxc^Y-_^+z57M z4GYmBMAcrSBWHE|K@edxd(rE!F?TjkO`|s@{A3_TQSqG~^dga0Y<4LB{L-sri~N;m4{Aj2@V`5f=eRcBq+u-b*)p z^9Y&whn|mnkMP}!vUEH0@Ru;{MtA8{#IQESUhiy0*KB7|)Y}$@LbE{2n^#984&v-B zr}|3Y0OfKn?+hSBL9}Y7_NU))H-WmFPnI?H8{s?1Q=Fg0H?xPj#y!cUoJwnr18I|k>UAcu$#pNQpgvr119AQbZf!6opXPV0lpH~nT)4zE?hb3+p zN8wuyqlY-zk(kk{viAZf+FlbWlxJ)C1I1_d9L3x*j+J%I-N3dKU|To%O&1(30P2U?3+gI}q9(u*cIG zyt|@P^OKJ7E^`Abs|tmlE8|PMgei;G~f)L zG+mZ~dQcH*`4+h6ErvOSpN;G_aJ(Wq6@0M05`<(xPcRZG?u{Y#0-)Nj_c>*qWWsti z@7IJQNc<_7RCF#Vn`rymNQ>o1g=X-NzWFxITgqQt5TvF-bn&=rd6vXOrjb#9U9IK{f4CiZys7_eWwK8zOH+jG+1i!b5H zJ^RJ{!7Ad2rVBy}uiC0-{z!0Rc{@+*FZU%eem8~R$>bW;UFcz(dfnx#$Q*zL=c>%# zV>~iktS*Ku#)L>b36sJeg4ocFU)RTn_U%~T3+1S^plKkM=fPi@b}rdsIU-@*1}_I* z!#IQ!;OQ7^)!l}sEn&Ees6R%Yo{O=>34`rsL8v3387+LED~q6N8&DGU^JSL(6@}pf zsr~0(?gj-Z_t#rePQ!yO#$)Oh+xjAE+BI=PBQFo_h2ilh)TE)E>zl6q+8j90UIqTN zgA_Wj11m!9R{)!5LVui-tv-{f2a4LhkZOmsy;6nufD8P}4;y9`DZ=2Hl|J9Ab9r)I zSwelK!W>cy)ARozz>e^rxv)p#M{{uEKQpVEH%bNM;61It|xz zrZIrGKKa>gl`h0XHyM`^11Yq9qI?{NVii7y<%_p?b(Yirz}>2caTz$rc_TQX3L0VL z6okcxni44rRe%?J4<*r0tdG~!#grTb*QIUaE+G|oZeBU6>k7Dw>aKvp3fxy-rhh2+ z>1|oAm1U;wOufzGY5_x$_}mN-ng|E(t6;D&uuC?I_pTdY1O2(#)Vx5Z#Ui4LjAn}G zKg;ZEs^BE8`Np_Y3kYe_zOMjRv4WCE+NIG%>R*`~^!UCUvMU3P{hRyU}z{hi%0PB{94oF_l%S#FMT z>-4A}taHt(tOnItv?-G1!fKGJuuj^@iiYjXFb z%NeZ}<6V@P13Y#K2+hb-D&ZnibQTi|qG5(%6-zGa53ee~7%Fh9_eHKo^~D24dxj2^ zT$|SzxB8}VMae#z+f7k{J|M=>Et+5o_+eN1_?SFp4%BA%^$i0!WMDV;mNOb3CROF7 zz)b5)Ri*Dor=IfYKIfHDC`yJ4>3Y%hNhrV};Vt8n&6qTZKY@!%i~kBdQJ*52$eSA> zQEoJJ0G}86vc8O%QoTuk0#3EwBP8UM7hIG7QXVL23g92^x=-sN9Xi2*3whZeVgdK3 z!W1&co?g^w(rWzZJiosh-2dY|9yh=SW%|m(OsE17Amw*eU%kJqVT2N*n#Q+k|sbV|8xZnN4XnX`l5BT>JatNS<0{^WDp#uG6 z4C_E4`kp^f#9pIG66PbLyQpwb1sQLtq473X0oo4u3PHmHs|Yt!Ix1=@ z+U%GUB!#{;G-KVoo-*Z8nKbnP`bt&+M4od?IG32`RD^pJ*^#ye?60)_T02HM$Gm;v zwOR`@x%yEaap080{3C{76L1$|2r5}uIl;sD?|a)b(U?E}-cu5@xcie*my48lYOZ{od9R%+e{-gjF3OUeotXh>+W! zFzMLqVIvJP3LulzLydM~>Wl{JNKQ7iI}$4A&ai!p1+#D1^DV!T?Ki6(}`Ga8iSb?sND#6{uzb^TZm$+AWi%sk_8w1CuTT zH@3L^L5}L7jNQZ6U{*_8M*@tKTD1Al*K?edm}@_o6D`RP!jhzbzDD4GqEpa%5RTE6 zSw_3JfeAHaMvMmTb4Gg{EXq8w?C>|QDrbHj;lCPSY;)RSeqa zi$#Ya-C-a~lWkfB6R1L2N7oeu_35JfAmXkmC-BRPFJYCaYbtw0U<-m(%~43QSB;cx zNSt$!*Sp;+($+ySQ!ogFP{mxY|6V z_up6@Li&p%@)VV2?|RZ@eFX33ZeVt&&C|DV_&~*250e*ybS4h02fbL=?5wM+1=cnx)U5GPa z44L4Q0Sgj+noF*$stHftvB=LCeh4(Dtr7N)sb~|$4rZu%a$d2Y?d=9n7jLKWVOB)q z$AKb_ZCZ&DoY4gWOj>&z*O}$Npq7(_v5~FV&a%hZi~HiM#;rL#1~DHu-fxzU#_kh3Tf_tO?Gy7?wizmFue*(Dhk(|8TC5 z&>M4EdhqWDe%DD8REnV7PBi6qxLa#vXF@e_gfS7$TiRIk9_2wW(R?p@F7-Zz7~WD$ zfiTnb9hawF{ut1%V8h>?BU(D(+1IQ50bBOFWpebvu44sdMH@sWA72xf6HqnnEV9|X zQU}-P10d+#%7+w}f%1%_FNLmXU0~-IsCT`u0$BqE^ppaEH75&^FX6BUIF_dv=G649 zKMRrw>5STeQvlCF5o1s-XsO>zvr+_`{33!gCm&LGU+{;kzqa7l7{DrL#u|RR;zZsT z4r;swY=GTW^?M(L^m~r9l)5d!75gZ_NaAI}tD&G%$Mli)-O%|bpSd+Zqi77B2(scY z(RqT}0o+uj$Ht&rjw#Fn95LGAC|pQerV|+;p6PpL1bT;D+Y=!zHuGyEM0%Zaz?|1_ zDxFgn0!HXKDS0DUbqk)6xj5o#VjbUJt7tX^e-pTAA9yAklmDY#wi__(3H2JUzeZd4Spz#6P2G>{!kVG`@7D*wAE#U`J4h!u7c)HN7EPlv~%!wwcMu7 z{Q*9)G^GSXeLXNEzFOnHVhp_q&`-z0rw2Muv3lI_=VlM(;bwORBLpii@NVH7=)TBW zU~yR5qm#j8$H_wa<#oW=sNuN=a?LA_;8|h05|Ij#`7%ty(^!fMZXN8P)8Jw*m{R%V za2aH_ag6#ylaX3P5~1~5E9M6JQ~Cay@QuUM;#y!@Nb8UzUrUz(y`h!jjI#xPhe0aZ zClKtlT>>7aI-^N?y>Oxio)n1Jjo53Sp1;H(v?aO(G?F-Afi}UMoEg%yfKC;DkTj_U-}5zQ z4#N~b?qYO^6_KYDaw4^HXKttpZ|>u>H^hNxCW-qmfs-I|&N)15#7dhS%DgrS#+LBo zrgRu3Xdi<)5d3T;dBjA9DSmeCW#oURokoN0+U09pVgfOLyz`K5>f{^upfvP+w|u(S z+;62#jqC*dGxt+=;Yb}?rt?7{5@?uW(*tiHTgg-ivrPYMPqZVtfhh!)A+-(ZNrgb3 zy=uZnLX__wW`uqeT_?59BmrjY_Y-V0__hCGaoP;bfeB1x!pt$If*^p>M)QyqR+%cb ziw4D>9%jBn_`ouhBycsMgCGb?5Qlv}AFw!}V33P=fG8fi0DX=!`nF!!R$$plT!pp+ zpagTj!SA!Sw$w0Q1P2Hj6dooLYHntrmK)U)cVQV*KyDV0tR%>v;X!ynDWO;%dO(^Q zz}}JWq)KT0r>~EGGD)zGVOpGZAr1r7LC0U(58tMlpmaDsf^wW1I4H2Ty8wIhGotbH zM&_s?f*&3Qo&ZR}+BAagmts3i9@0ahpN@Y0RtYP>#%`v2n43m-Q%o5=8}STyV<1i8 z8sl9=60$t!h{;5Pc}2fwIar_p004w|P|PO)kwAB08K&&tTA{+c$=?t{5IAL=&C*Cp zYzYI-AWZ<|7ce(4OxfNK#+pUX0OU7^S)XK-#hl-Uy$BEucI@J(tw`_YT+Abi^Y9B9Y_@0FVnVH1wexCjBCN~8jD3>3%@418iA0fR5 zhI%;m0)syTSiTctJNMMVg)oCtMmvPL5_#-z&I)sPC@~yMx|0p?JCrJc_F905JX6Ag z76k$zNrys^EFUg75Ts9`S;8FGQf3_LQ^EI3c?n_G-~`7_6RriR*PSx@DbG;cj5^Y1 z#gL;fVQ$b!_()6ybIPl5OdXhA6iI=5*w%)|4%k~tr?96V;4d7D{GLV3jl+C5UzPT&xI3@sE2)N+| z`4$u5C2*KFC~GYWZ2p*b`=Y@!h(BNjyFtCA-`rF!1TP*LGB98QT@l?kLHLvfSc5&l zn1-b?a?*8Sm`5Cw{2K8HyLl>miZl@#Xo--x*KNzEfvHd?ULCIFK zY8Gr zv^ast5SM@}wBS84l*}Ozt3YJk{#y{C8SO<@%EASq$!MA)us=G(CaAYK{st|cQUQ;X zMHa#rXsiMX_6Ya-HUTuW0^w!hGD)aTWETSU0YMSiU&ImJKL~#smdb6bNr1m3gGPM+ zzgZ%|u&@bSZ9Eg3Sy(s*^*}^`@W;6#u}ok}oG$GT-oq>30B#QaDO4Kfx%p5XQ5AUs z6SxoliaHA@f{MK#CS_!>89&7Y3dn8K+Y(2#2AikI1?2!}b_IP1zjiM2DJQHd!kfhy zR~hhDjDW=*01kifdki6A(wO*_DzmK1oFt)&(4=2$4tp4K3xVZ2Xt#r_J?*8$o1z6V zFoHRRuC_oEG{A_X>JK?jc|5pIp`8^%jA#<^p&rwWeL)@USto%R!5@TgHAD*D ztBn`DhyTe(tUdz~Kl~V2!KCW#l>gNNFy8LThpYP;icOz*=^n~@%b;ly{4fmh4b>L+ z49udzg4scmlJM~2e^kJW!AjZ|m)m#2N@_rejd-BONnZ3A(~#!F$ttkH%n6Iu6sQ5& ze3s$;F{zNu=CcaZ^xqoKmiopPIKJmTkLDK^N_3Qoof8 z3>oCh{`dLMXeg85HE{VpJUFb{i!@bhE&ME7GhSqANBQO2lgD zyD5+&24tibR`>tf@R#x0oM6v=2u2gdzmd3D=-igU* zh_E0ul9AA+8Y6^lipJjlXCtonNMn#~Kr|u!d|-aap}x+s5XosehKjKf*f>|T9s&R_ zXa<;nmm8h07IvuxcFEEKJc)!bskVKM1SaCEqy!jA}9*i{rb5Wl{;xBZ;_7lwx7h>YIZ@XxEIL1{K+D!k=@B~4)()JfT zCGa_JO71v=zb@D`Ay0{biA4cq3qHe;y|+c$G8+?^?lZT|HGuuBc^E&JU1lsP;g{_v z4*`8Hd~5~?rYt*r5RxAolX+Kr4t!G;V&o@+@K7`_UvC!BMOiIgRhLbTp)3Y=vRu&>`0CB~w&Fd$-1 z>jRpc)6Jl)RAA=8+%R)J2ovpyj*LNN>=D=W01z47OO-Vq6}H)WAqW+q&-xU^s5d(N z=CY*N3W7wSH7InZA4m(6qGf@k|9E^2WC(T+bK?cGCV(QSIh`gQgZLXbvHpt8Juxph z8N-McKfPKG$_fYW`7n;nZyZZG1)AX&&^sPj7%E8~fo_|>C|w{34zVs2D#W{?zIpfi zU-u`OQWQ3kFJ2A6Vlyc0n(%^sx=_+B#DaK8LT(oAn-2P-3j;SG?O*9{x!JPx$tuVT<5Hkk(a!ErAj)jEQVm^E@=1;Z{XxCsFMB zHRx|>z5&073J3UZXl(;L&>BS=ME0)^WeTFkw>dnO z(dqYq0Z;?T3C~g&SaG)=%_V)};WO~}2bX1lmtBFKvO&W@ciJvY3}~!+r1ncOv|9p( z(s4LYypbH`7^X0tGUv+}Bry-XHmJ!VuI>XfJNKMHh)QFRKpF_;ZAH;l zO-id3maua19!MHw$O?s`3wpbp*HFSpO$A-ClkO)g%IK@4X;}R5bcJ)(=4-N}d$`$Eh}92yR`JM{tr{p5kgX0l*(^=ywK&M0uTk3p!=jHwQ;-uG6^yH9Ou0>+AjcL+hYFDH$S~PCuu9 zSI`+aZyGGblw0O~RL4D;sjYXK7QdWv9wHOS5)Ta83sIg;W^cx=r;nkYoc-|(jeDE~ z4ytWK+5W~Y2wjJCpn!55B0!_hvYLUwRtZGRv4ol|F$Qb3#P)y2Qfp0wgbd=deh`V z=_Q;$SgS*2FMaI%5upz_1JE%+Csh|OdQCoV5t5tZK)k=Px}oC`9`0<36k*?v6?B#c zzWxrOTbd=HHQ5RyB1CtH18bTO_}J7%*E3WMm`K`UF#KTmc z2{kiwH!-@+=ZLw=&CgTuoVKpP%i4^9IrO~zhj1)?Xw8NNsh$gk^sEz*ACw*-*G=;O za{CyW?|D$@3KP{;339c&bUFCnPvB@5M_VCmjLrw^2}^-qNYKa$Uvn$fKKVx{O~?K8uY}2Fe%dE zzbAWX6M($PAJXrFU9*A=bJKo9T(865C)c)AdCD`>ZP8C7O_VHV*`H8dL5hN|0w!FI}+3ixsvpF`ND5 ziazu;AnAMTrGK}d= zkF^qTeS71U?>Qs1nTjN_Iz}*hl8-GC8Msv8=Cf~Cr02>g9+UcZ3}OlTjf-}U380E${d=MM`Y)3!F-yKx=;nTf6A>Xd_N=ND8k?{BAI zz-B{@_T7okzEL6RX_yl_Nw#Uv_ZYs$Qe9(1yyF>2PPD>rj{0n^Vo+1P^~y$8O}JQ# zA|(|p>!l-b7tzt6W1lX-f5Q-=8auOpy4PIOo(-YfL*G_@@s(T3!I+xE+nLInuwNPc z=@AOi8OYvf6o`B;N(HE0{(`iXO8WWAYM+v~Zvwg~~uOqW0m;3~yVLWerWcmY8dLV%>a;YCYtN3-X+urOc@58> zVXZ0q*)x|t>40<+r3vZgBu$>CBaCq*-VAC6^g#wlXOXXHSH1y6W6t0?KeG6qv;Oy- z>g(6v(A!CMPDUs_j)uIqd904;RNR_Qv~r|pC{1xo)l*WzhWpP~<-(hK*3{gx@l{^* z1tLxj;`}cfO0DMD5O)PE>tt%>USS3cXjA&tQ}5=c!EiUs#;~yyA7FPfT|0|r`3bJB zB%#lYkqQpM1j>Rq;uC^JY%-HH z%08w`zamyxsC_$M{bxUVA}aSP&-xr%RaoSk53mqGkbBJfgrO?${hu%t`j6Rh&&sL5 z3O88?gt{*Wh?FNh|2FtwO)e*w-)IEq%^REW4Dovj8}xNnJXkP@Bd|Ofl-f4n-S9Jv zg}M;ylbEnNqi1nNU#1;QtOqyY>q07P1Z8#e~y1_oDfFeHd6b> zn29kHy;rXtY%CiUxp*sdBlbWtyuvLmK@9hduDohHP18|LVh+$9?fm%)+r(hC0{`m? zVF{2JTf30Pj0lM`^Q0crm%u|ESuM$69-G0fvT7Ufh_zcNXhCn&Wf3FRgliFQNg)JS zw<`7mNgI`r)96}T<2FK~v0A5@^WXxoqYU+UR1U`0w%Qy!_8x_~_S-+!YJhlXJ1hi# z4@C^~)3;1UE!Ip|U}84|jRnTAvD;OEW#qA}qEZQL!Sx2fvoAi_W6z6ktO>!rbLJIp zpv3xBx7Y6$?PqMo{^j4+W@0?<;3get<^blM5s0{(H{T*0K>p zQBl2sJuWDK(C)qDyqT|J$ah4$!j3cvgd~ntU~b-!VT^(N&B=>4R9k5tI^~!MWFUP2 z_NJ=*e3eqB5^FLjDFW4Ut52=fn$5?sto!QZg1r76I!tIzi^gviYG?5F$J4;u-RC?X zk5%<(s;hPWZz-Ld3~`Xc|Eb+W5z>StaSH*SV|LME=XLhdc}a&3Zf!cq*o>R#W`G(q zE%f(L6qDuNTuA<_*Wa`#7)p=}PiEc$WB}duH#g_oD(2038}4zEdNjQ(;meS>hKC^a z&+BiBX$=Pkd7@0cnGsP9;Obxt(5t@Z;4T?Kxoy0I2LupQ>)>pGDgT}y^Z2o6swR16 zjU`~IC+Nxl-IB=XdC`B;-6MHY##s@pI~Q2uGLZLPpR)zaLFKa2$nGQZyp1h$F)JM9 ze%Q@Ex+IH|L7uwPUNZOVy;2k*uMC8ifN^5n)O2bSnsx()j}iq1cpCnHuc^IbXoUw& zyJ6o60v1pK9{%N=Cn`x4)6WP&@5~sMM?`z&mZi>qI=^VPSFhIT`7_&U{Lfi7UPRqZ zXc>Q@%HLxrFIVKG6-?I_rW^3Ca&HHK+V@#T2C#tdMH=XVv7ss#)A;GG;_tr1(~>rG z;z_Md4T4&2XIaAwHZv=T<0mzFw9W*-8#sZ6@ja(V3{D<9zB3#vz(kt-p`*sLukJ9% zK|v{@M~-VNNgJ4DY8eGPV%2G9Filu+YFGVAp!H<3=Kf~M*$))lZVka&S{#{ z5->NXA~=!b#^}^;uCMI-?4irl&DFFCMHvac>|}@VjN$fMk`_I(~tu=V$THYlrWGIV{8EsZznu zx-XcJ^iZ`7OrXp?y>gHzZ9-jhAhBDgW5_~_E$^YKI{1|E{61Z!b7TkAZl+SZO^^>P zmM^$@lAqyDSPr(x*j`Wei%Vyf)B1HmuROeb&N8CiWZK4N`)jXWI0>755XtHiQs-&D zwmX!cEt2d3zr0|xo!gw0SrYT}1W~jtg@Lqww<=OP^;cs`f;et`p54`UWscf%qN5gg z)mAx;A7^`t1<6_w5^X4`aSIJmOl~(;+m$iS^5g2U(6-{##A45FNaYl;nrfnSR8qVb z_6knefly~Qpvz-9-ixk*z;~pjxh7l}tYjxqvCXB(8Cr^i9X?85E91l;j4S$q%+GoC zK7ox&J=qggxi_ykH4Cnm06B3wea)n;UrYDoYiRZAHF}SP?hKLxQFqwtqr&R< zH}8rB927-*h}mvO)NB)5uef*IzU9JwaTv$EIc@c|l=jQ@h4MWK#hAkm2*Kp>Jv>&q zzoGi6)$++fXMC2AiWs0glCp8aYOb}~1mL6IB{Xy5R)78J2t@T4uY%u9m<0hI{i8*x zYjHXt$X1mTBZc@_^y~d7!LW}G3`fNs0kkE{J_$NENt7U7sr_!|9C5fvL9JRox&0;I zCm4?;SaVlef6+ihA0dR8;W<}xK0MIA0MP14vp2PX=hg0fo2!+-3;7xz(oK5ct%GU$ znb=w8;HkhR@2Jw6lk={`(6Yuz?_Es58iW%^qG7NTpb-giFf~k?w7wwQ^w}!C)fxwjN;IScjn*t%R=9lyDLv4MN|n|F819)( zt6#Yyax@B5mV0_K5Ja@+58Vi^tm3_XxNr|HpktoyU}{^M-j8o{oWoG|f%F*LY;Tgs zPxL8A(iKcXGsc?TqIvUMor+-v_gK-H2B}Bu>-O>w9{^5(nGW2PxIWpdoi_Qh4S)5nP@GEniFqJUL1K7+!HK>emD}#(4F%HIm{| zFL5A~SfH9szHcHvxlgccdjS*ac1gNo#%Tw$^DOICO2O6PHz%Ilz5Ot*rqipA@9ghz zKDm$Vl#gLTJ{oiJ+%|1DB*nc2lgE)u zvbP#4g27+?spZ8v-*=#$3LfW0kuD47BQ`-B^U6M4j-)A4Y8up+_#^`%BUoO zhd=zisKN|N0}$kLaomCt&-BL#&R`wn=wTp@&WRJds_(=lTiit=ZzzY=@6dlrZfKrP z7ySFmTCEsl#(sb;AX>>!b@hqJvXxu8a&miWI495o|L%%3o>2lgDG!?m+e5H<`TE-t z%6-g{Rtz7B@WWR9Wb*Z>x~CRxs)`44hxhOI7T{?@NsOUBSoAa_FQA1h-RTbcuBfWR zO7KNrb_Op3$Ao01foaL~zD!0mbMEpYe2y@&3$Y_Ay~5@pG*AAC^aJ3%h=!Mz+tH5! ziAytLI_ay{!I#{Y;=OvUlUuLY&%lTX`y621xJDA6i^)eD$m@{;V8ONTWtsebz)b@M zc9kN0UsEdOm4XMt=POa%hh_E_5RF0D_m`r~65kjJ68tF@jqgUf>}B>l=^ZwEF$TQS z)3>E)G8O^_`m$RpT_HbO|2PKkq1@jQ;^;ZM$KNhY$9zCn9dJiVC&C9t;vC=b;0ju` zRwuLEneIToXNuB-T@M5f5#0fsm#pPKk3~pWaP2T$Vv$!>bL!ksZ*OS1hr>`re<1fq z0^kzy{c}BY-mpl}aMtkCf4wDOnBR{}oct*EA%i9V;}6$3LnIq2PEVH6 z3RjGt@`JWHJ#W3O$%bGvc@kx{hhyErifT`iTbnMTAnkW7IlSkotAGz6A_(Vi?pD(? z-Pl+hGhxJ>inU616f5h$|F5Oax?bP~<+PWgqFEK3?B8qXP9&=Zc@^U7lQFyRMHfKW zH)Y{>lL0Gs?_p_#;0e+i+-|@+{oxdA`eh=5$G12wC;|b#Gz^PpD){)VYTy61{`6DL z*2yP}wEJ}Jze$;L%c^M(||t1&CLGNZod|6~y;c)XtTvgCuWF2(QAgwFKH+p9*nR=n z{%#`!!ra$?tYZ+oI&2apN0XJjP@5P1DX!o5+?%d^6QzMpm!RRW+O!j%8y^)75TDiF zC7aHTouNXzWS31V+&WsF4p=Y*j3ZZrDYg|te<&inogWCYds#ZnHbk`}U>jeIAvT!Tkn|wc99wtb|f}t$GYJxD3?$Rqi zZ!s(%a52yL4);fp|7QOI2BNV?A26*JB`Kdwt=7)&yzrw$AY9XZ9yjNWx#^``lPodoXFpNYqid2B>N9ODytvp z!70Ksw|M@lt!Etf+}J&blSXbj=!Dgc#sJfRkd3!7-GjjjE)CV#pZyA_n{L8ErVk*v z!wy8GiL79NkUfNRaeQkFPL2=9j`M_jCwjU)a6a1i%xbuQjbimmk#SNe}mAo!kNW4LUdYlNViAMOwuetXMKHEldsp^Je+{Z$Edcg zJm92eX>n+o1tHBn!#m=NG<+st?B$wGdmoqwd8mQ;vnBM9NIkv=7~-YqWP#yI z@CU5!C2B#nziyavq+MufTTTxRwF)Spa9}u_Cy8$i5W3oSChS?@hn{_g8tE^>r3Qv$ zQmnFr9H)+L3!gZ!9#*;cJWg3g)vE4M>v4`fmFnvmS1kS^(X2}yxjn3!Hk!K{Y5lgC zG6Ql}|66!N?V8Vn_5>PUrNyauD~q_N z#9%_?Y>U;JapgOnRh@~fZI#cJj;5S$NMW8%)Odg5b0e)D##TjPQ}KC*v=vwg3ixz@ zNpV#rtcZfkk1V#2$IgEG#UU|KBCXH&)>_Tk^V*nbn<32?^;4{I4h0z}L`*9WEDrhc zogdIOvYbg_78?SWUDhGspbd}Y#z5dDw*VXEG{XuvllSNXQd~?(kBkjIYKTay-*pxE z{pi+35*}7CtFI?oYf8RCL?q^X;m3_6DL{;m+M7;9&d8=ICk6LgvxYCeQhF7sui2W@ zB|v_+mB}YM@coHvRicmc10H8%@)K4^O$tYhoUfxCBbN%4Juj`!8Kf5U##-&G3V3yZ zaH80}OTqT2yvAbdN+WCw4$l~Txj__-Fs#x@xp7s~KE;O0;F(bLxkrcvp!YFXj{Sd znaG$4;v}5zq*6jGXwV|6%aJ~7Za%qNW&Ozb%@g0Zj?NvVoa z!1Vjqs5$_^{SGW>A*=*z#@A!N@69jYN8_wC4~RkOKPPQ1=TBe__(viSaHP>TlArwk z%M*PLYldorf}*u_P_m7*-OisHJ$TLw@9te#0#Dm|Tehgas6^+PwZ&tfzIK^2`F;Oi zEx@Ehxrj=!#vLlk!B=B=wb7%ESj>=j@1vKTM!QwImYy30H5UK!ryts0pU`eta99Np z_v(MRmZ989$W;?kX^&=)HYCtT#n50EJN_yNBK?;&tJ^(X*eslLECr9X%1gL0!g2m- z9@psa(fsW;JW+Gk#vY}BfJ7aJk1@29%BS6M6>?ERWi z0kL6sPw?;Gt|M9$4xHE>5IpF%(s_jE5)9&;}4LGC* zJR~zP33$mX(vz6nt@Y~ze&^5pCe^@(lpGLnOX~X8^u6NOBRR&HC+H2o%7t!E99hST z)Z3lPB-5WOeDZ0TgIRj$B;e=8kg9+>2lUsuPAv$67{gUH^ zG|cU%EaezazlCLEppVOrv~0iX^m}0(u{9_avMV@`k6vUAr+r@eeLf@J(%h%Y6!bG} zkhyF3bjImt-ViIWpA{<|Em4l#oO<^-Og-%q6&pufEw)xOaULxe5{Vm1#RYd}yIrqd zYY%AEZXO!Nh?rZet5u2|(^WE{y;B9tzt3@KsG1%E{wj$e7DG7kI3_D$T6~&4N`@cq zGazk+eVKKYW|nxRefuQ6!5wn1d_TJx=xo7A=$+$uF~Cv-&ykctjD{e2vBc)$(CLN% zZo2?y8z+mzmhv`3?B0Fg0eEw8YF(>Ndl%Kax^Mj2U=UD&>iMRGh6k#`j^=^Z>;F?^ zt!AN$+=5tTxr)k7t&>&vx@DK$D__2Dz%YpoO<`V_y7G-m9FXgE)B4xYVZcXaKxx;f z`(;g+bi_lUMiiC5-{{>I5m$7II+zKyX#AUMr)4u zwkMT~^lyq3Md72c#(vfKnl7u45^Gk}zmhVU%d~1>To*t=Kb98M>56FXiW8 zAf%7rC!{W)n_afih|zyk1qBQpR^Y3kv6okQTtPm%x;jQgqlKA{53ez<9Lx3VVA=5f z{Dfv)NRol`U_jvQj+;rB!5*eg`OqA{7rSw#`w(~I^1x5KYfpzH2tX<=t{`+)g>f|t zlNZ(hsvbVV9uKFD!hgF~oz(rym*mgIy^)F8NXtJi7tjH+jlX+Pj+)jN{^)J9Pj`~W z7n2Jy_y@v(hR=)9a|`Vsx8GcA4O-u)TedM2g;}WFs})MhSS^FSN{J@i&|-aj@~2MP zrL2Bn?`*(aGwgbin&ZP>RW?*xz8|A2xb;L=xV(dXHiMd`74-&lw) zwXMZ)&)rNN%e?;UNC5|mjrzjzE6g`Bya}wNgh4;cBLax|2LPm&v*(S=}+S@0R-6q-XE(hcm(8|=u-nQnseFkfe zC&@}vEuy)DoTS1`ypDb=sB!1llz_&Kny!B~+GZCi=_R9l1$)qaG5eP^A-pw@|WB9&Wc=c4Ptnm^HU z+sJ|O>*g8(i#wwxdKDGv>$jFWo*v@jUTKt7LU}@{Pow-CDBc(bF#3qoN%jnQXzY0V z2chXUMS`1q?#L-Goyfcg9-N_qOi28b-RcMQKJJ5iT(Yz}b3S@xwCA{VFO4)eIrsWB zYaG&<=RKm}+p3thC8%4;ZvF@b9BRmv=BSq+7@d7r6JmEdJiq+S>u*Lp?*{6cM;GxK z37F#A?K{dG2egjNFV5TKqhZJ+x2_u#jb41}^r!pHUY|*i+Uj_duEJtuUM>MsG`Bg3$6mOP3FYVN3G+@m3zmvC0HPv6?9~biKJ50 zOV(K^37h$wKODMY!xj5S#`&@F7cKsU^v0@BcAB}U2AW%q$NlWEISn}JbeK{_^f9u& zO)D37;6JveLVUlEWHd61(&*(ZYQr|~bh-F@8`s7zs2SezpWD~;!@ly`oWkq=pqL;o z?eY`7(}!c}Uvx8^`hq%7DF)R&8ZUGnt^fIINkt)cz3;+%rUx~<{C~ds_@i|}d@x8P z{m+jb1=&-gx*8tBrGP&jLw2VfzdhpE^W@Nw>-YsVIh~Zv6kYWRQi1Rt$6z2be*Kwu|m45Gcn%gy=?E3i|yruN?O)V76FsD?yn43~kf6F*lHg`#Fg||D`u+uf@o!?ERcE`>p$Ezk-bb`c%l;)p%qIUal{H?rO@wV{i z^xwwxxru{=+PijjN}%7u!1gh0;vX``K>7dLdv6R=`%e zx%bbesJFv%>b2ul1@&@u(_(tGn%1=I4vgDgu@Loe|8~=Lbfy?1_v!6}<_gM{huk_P zUZy+V!mZeWnpgavX4_vns!Mm*cv9HB+S-w~J>rhm`Veo3YMiRoxN0Aa#M?MIF$&#eKSzoabht@oS6e@L;rne(ZhruyW{{#w`X?{1xZ z*ISvXJoQRlSYIyxZQzs0<`8SA1D~o2KK`DZ`g1|YZ{I=t4#%4O*WZ9@wO^faAqlLe zHDCKZ{hi1^vzz)87AKMn0+_|0FMYl>j-lvc^wc6gO!SKJdDKqKjbJDn$zvOfsyr=K zm3yx^1bi-0ZNHN6>e#WxAMPVh@BbxN=8Ba;zg!(ZS)Y%rRlk$t9D9W%kKB#68Juui zB(FJZc7N7dbkmAlw(RCwbI1&vie1+p7I(|EtBUk|+^$ohGqYo3i0Np7I23x&N;nz~ z70dq&rgGwc!`TT{O8oaCbGLvEPqT+C{XPlptYYHlEgPRHX?e6S2<5xi>0eKN-v5nl zbT)Zd)3m7f_m4iuT|@l?4VL91-c#~v9k=uQl@GpNTq$gcQ*!_b-)I@%TDD=a8V)M6 zCNJESoAbuTw1+<5cPE_-EGtG)SxN1<^xAx*g+c;tv6Jua=+^bmp3@!Lh9c*vTj`lq z;+8iuT?T@B9`&2}@oWUHsHfG0^C3SE2XU;}Z)b}P4z8^lobay-DQvki<4}^lvi44V z*jc-FLZv8GggN6Rhkny4^TGRkk@3Ht=0(30We>g8R@}>a!@h19$~6LHBTej_W!crA z_dAf!sl}Y?&VwX(`*9(*<8Eon)U^tJt=22W@e}ycb+?T8gpJpQ!eT=^TyKQBtzPc$ z?tSmAG+fSavumPaVrBksN9WrZOlDQ{fUxnc*^R|Ixn*60BLN$ZtIN~c^u^S<(gz=l zC2#pHQ#8|ELpJD%qo%V8Y8qX1dbciLxrg0|0No&MEnvE8c5ouK#Fdh9|3uNO#;WV+ z1+^7+8%dUOdiOb10sR|!RU0S6f^@pxE_DBi`69BuVt1)7;E{cMe!%hDN@l>Uk{@!+ z@k^pyJW;o7o?Q06c0VG7zioQ`W?J)QG4Wla3!Zu|hlo?O3+vWncnhv7j|-NKg`zj< zMyvd|g62ZHbebSXH;b6_oI8?!+?>DaPxx%a!03>VMASW`{|`h!6CMsEu&7ci)UccbRS%9BR=uG5n|kS$haWUMEVv zsLyB`>i68+DyCnEU+`O~?P~67Gi> z$Ipa`9dyaro3degI?0{S3Ougn0al7*)(`l*Fe9C|EJJ+=4ln^+`I4N724g{?r_bzh5NQG{`HiJG>+O#^FaY zlBS8a6OLuEZ$+esei(SW_N_1Yx!oGy`E-vvvF$xcyLT&xTfaOPPk+UuR5}JO+OdHE zPk++%z?Pq(H;1)cLdDjR&_5N=GOO5hxfeP}aoVQym2eVg<4GA^(tDpgR_tr>vLY9E z)HSKFp$mg?HKc6R`@I+LGV*JEias2g%sn#H-FB=t6u@6XHLJvuwzshBmIC9s;@%Y9mWgh^5P#uavi7_3 zih2)*T;b~rQYzoj`g`$lGsp>eQS{LdGq)udtaj~um4*@QyjN4D24$a0TY6d+UaeiE zo#f)#u|V!Q$i*W$wLdN4<$05)sX^`!ifQeL8jio8p=c*p4{5|xu0fOaq2l@7O;d0p z_!M(i@zB?Eo_pCP)NAvfku?g$bs=x9Mp_vwcKa&uMus$NEtA@Pu27V+(0wqiy)~mA ze|Shnqa@Jl>|BZ3$|p8B75aJfn0}C2#+bg&toXlQigbsUaVvSc3$?ldbx>I*qOX$Q zJ zzq*GlDDr;PP96#ke16W&oukyzL;OSEpF71pA6Tmv;Y9yT{qbD=^b-(h^$aroHO*G_ z=LiJ{i3v9h$Ks@X+P7*8dZzi!z?E3%IlM|nt22}o{kGQOD`TVn?s|2rmLp@l8Bo4f zGKmZhV8&1RwD!{cK9SFty-$7GOEJ{+q&}}+GvyI06J#k?_q^&h1SM&Zb{A_qC#1>K z{^nL)pcEIueK50K8Wg|AwWqq~b)PqfctFufFGco% zl#Rb@^x+pIE4yboxOIhlHRY#@I+~x6OSZgt8=5JH%kd9+qO4^bYKiYjoAQ&0nL_k` z*UUzQi$x{c#30%1BY+NRT%A`Svmf(6u+!hnN5-!yxb-3 zD-~T|+toK6wbJRA_abn*wedHc9d8p1}CTeNX7`<7)-Y+pQ0x4%t-fpu1LfjoKtn>XHtF2kFS0)+>!kcdqAJ|^X@+S&a{rKH1`;uAn(e`Z^tIzNursr4;<=< zVWifVTXepV@`X^_iOwJn-L*~}5po*h0g)M(EfiCr8qlxpZ6;4spifW4^_yQL#!I%b z1*b9VXU)wjpREZdo&H<4zRz;Gjg`DdJBIC$-_-wW?>nQKT)Kas01`^19ZKj$1oS9H z6p&s7#R8&IlqyK^P^A}vAX20Y9ueuFq9RRtk93qKHI&c=qy+>Liol(RbIyBs@BMt& zy7#}FFIg*jvS*$>duDd|?L9(LAi0r$_srvEmXYBiU;X^I$b3`Bh*D4&gJl4?rz6hG zjXz-Z-TOvhCiBV)SeE^4r8ZG?k)+ik^7al zD2ppSfX4!SEx5tRh$H_T2)QjHYg7aHv9k&!j8=4#YycHRH=}0E;vx}o(J)3l@@P4< zv)(Z&BbR^aM>ZORtl+@T;DS8Z=YryaU(LgJ^qmpdB~7-<($+w<(#7r?yG++ka{+pe z-v{|U?j|Fu<7BI?1eVyeq1aZa!Up^;P}+bZ?jALSbcwqs6uf$0n%VFz0vG+I;$%eW zVqH1dybLzB;vMO)=ulDYuDXB_3U^JG{FTniin>tw=&h^c;xL3ploFHdWvbtJ85L-; zOIDKCvu4-D0=y_`V>2Ku9GC!aB=s`IA@Ws!Qed_4f(c;(+>zfF1mi(mYDMXV+qZgKQwfEQD16;yEL=v89j?;0<;aY>3?-72!uz?4M#HKXfSp zhb_`+x*TC9+`$zpZ$r$)5)aNr^!OK;xYBu;?Xw?Gp9(DMyjQ`CSRCKGqu&?=%GlDb zV_z(@OY-us_g7_-nDMmf{#ve#UHHCF)yY`y0y8xoQU zO%&8BoTOip96a0l=UJ!(Hj`P+&FH5#Wm|nfY_E5AIVet0%=dAZp>a|;D0#E%%Dgs= z4OmvP*LQJ{9^hty=2R517|w$4T<6?jac?O7lgS~w)8Ugo}ZMm_J4Kz*DY&2@z5 zOAjr)WoKIfL@J(NtNaj6@4W<&QOV^|Kq3IRG5Ux+8&tLraWw!tQ_mb76hvhv>}NMy ziqQtXUyNV*lixrh;&nd>ozD0oVJa;E;u}8!{Ld4~@xIX$v{TYf^n~F+_3h7_G`rFp zm5||&%M=)JncrwdScjp-M+QV$jae$p$J#cxYh;)}&%8vYv2IrOl*`~AMbRl{7dmTx zvT@Gs(WUjEXa;Y`q4cpv7rAw#a=SM-AiD9EUF3MHm4XsQIMRmnq$ZwXu>rWIaAV}D z009N4ztWgSXLtA>Z4$5rOJO6p_51l0PGh<{AuQ<}8?8k#Xc>=VaHTRZr zlrLA8PYONM$q4PPmB3yO4bVR>;Q%0wi9VR{k%lMvkkeZU+zE_4KwM!$Y6%>+=T~sm zT0c~nuxwR?Eg4I4F;w%Xyjl02zNgV-Bihi2KZvlOY&Akn!7lxsE%IRx=8%}W)B_$~ z$AP_-Gf=Dns|Hp-sQ<8{D?R8cdbYp&2*^M*&7a>QD6KNqnO*%!hkY*p8)E!6S(-jq zuZ#om?{vTs{#&Sg5K@3D2*y54M*)aMi9OIO3>c3(_o!BmhtYl?Dyt5?i3#Xj*zrlH z;}_@pjl^tr#Uu<`IfdxrtTnj8UJcZlJ`CRA*_S32P-ZU}>T-+M)M^BUs!2fAlayGw z6^246UU|T8{B4VVv)2-iu0q_aZ3KW28F@+t@l5}(~ z-RrO61L?AB>8EJ|bAY38g-sX?>8*Ma8S3XoO_xVlfSHWzPvCJyM<)sjo2Q*)LoD`! zb0Z-%pa3}EA-vlk^~a$_yyeyILUJ&Sy_EwH+}xa}FSju&K#8-ibzJNrfCOX#Z<`G} zls2q&R%l!gWD?*Up~?~lgas`B72oZRS$OmiX0^GoSFd#!{thVpXsQ%daAE~Sh};GE zGIB`#LIoE&_W}VEtmd>6|&ZepOGD6Zu2@4!pw|g5F`ay2V|at)tb+ zrw7BBI^~)o8mt_93ii%}62QhnUteJI62c+(`4+dP7eM*KEO(u!hUnD3lKW(}FrdHs zcga1Tq8C9v#WK+Y1^Sa@!KEPt?(PnoCfvNVf`1$QRSnR2S2-LGl z)om>{HFPQ1(m=}>F10;rtK&rP0>tQi=rbJFrx&ASg~YIi+17aqcchbG6p#Hi80a{&K;@nsW*ye9mLqJY8%K(`5- zo1FsO!`?L#TJ55Sc3zjobl8TYgbNs;^)F(qpV-TKWI$+O5-{m+60wu2CEspQ=v`;X5$uXvaJb%X3Ph!eo#=Dne8J; z@%Ru6j=II(HIW%hLAZOr6QUHD-3rYgfndsjxGz+k)H0WV=7{K>WQpgDk?m`9;wX!> zzqnB8%&vaYZcIMWP0q{%_P7lan6Mhp3E}b5T zDzMcNuanq6qMBQw8a`d0hMDw7^1V`02_kAZ!0SuS)GKOGFj>JppJr<74L`gER(uKA@y`WfZwTuPw z@w*O|*J4$+^_`%Z!Bnyb0DD7F$)yMUJ_EV4y8_g>cR;NJN(w)`fGY1kf+*ht{JzMU z$GP5y`i*VD*EP-xC6`)szdnoMj?Qat(%)P(sA`$P(3+uv>H(VB6~B)jF>3F!&3SgH z%m!Q~?!gElEHQx(gSdH*DtZ0Q(j27)ciiY@a@@y6K_#^5M530(Q=Z^SBY5@zlsFN@ zeo~xlgbPZKx_<{al^OE`6atnw(m;WTcqE`U*NfXR9#)`ik`*#8J)*3Hj>nwdin-VA ze)&j^1f6gKdtf>V7~Td4SrHiFeE)zBn^`$+L9noZGye%14QKo0b4zQ!5u!4a2OEl} zYc@u${HpVaNw}HO5HlIK^-VOziUDlX+K0cv?9_602qLQAyoK?In5;4Y`%c@Xb=3gq zaX}?bGIy)x71dx3uvX;pmQr}`s|?tdBF?-laANHcMH5@Mibck?r@AY0!|tDAf}(x$ zcELGRE;6oTvEmhL3J8PzGMI2PEEcY}zHNawJ@X0UxktqUQj>xzDpGN1{n%YmbUNY{NI zJZw&Y?7mpWPL-&MpnyvU@~lc_mVru#TG8CUn|b7Z{QjYc;mC-sQH2fdc;q$gKvO>f z^tE9^Yv?Ohbcuc!U>IPM^lov}0C}#ZuA+SvsJtojCH(Dom{EVi6w8LmkV`oMO84dT zs#>iC7;!12!KA%c*%a~)cHQj^o4vy_w;A0n{}>4}fzjOHrIBTAZ?FxYbk{X}aC4`K zMV~B21ido(k6M5b9K@p$K)#}D?@@gJY74gHCje_aq2}=H#=T19cUm}8!CIHwjJG8` zUNMu_+}kJrNia#6h4=DtghQo%!p#hiA;Qwxez9?Whw#sA8D=P04tGyDLCdd`V-f_n z_>OuOeVId}hIvwFH1`}HKKi=-I+wD|eomXDz3z~MKG9GbJF3v3{ZRsPnoVF3#> ze~Vmo@6C$Dk@H}i8!x*fq zYd?e%1Enx~+cDDwAe)idT{#6(VWx#7&fxyBdpIrfBfd#+KYvcu}B$y0udRmIi^VCB;PB+XRr zRab?>64O=DYL(N-9~IG`ETFc)7ST8?!_2z}tX#;#qq5afl*TM|h%72jP^p3Xdc>Du z_r-H6-INsZCBo5RRFT(inG-xa&Lti6X68l50-6>X1Z%L)%%>zB~f^5 zHp)-?uM=pB$lv;Imv+T^X}j@3xgA{A{4t{ZK4AxB%*2B%JS>`*X$;J=juDk>kV9~U z+2SrZBrW3Bz}=mHfCwNcSN`=Y1Kl?wGroLM72US!{c<{zq)d5Mi`(&C*-Au#fW}gka(I!6vM^?GxN%iI6-u%X@=qHk#I5)5Lqb%C1#{(d!?MPD|pXK6y7Tj zom1|`PUoCOhYqWTO6p*pMqjgV)Qyf=jve3f0m|4BM)8&cx2N}Bwl}fPWzuWbI5xlA zc)su7>9Mkt`|uvG0*-G+<6DfN_ zWEy)bE>G`07X;7_O=qUl$#+9Xk0%h!0%@_LDcZSlNg!k--+DWi?M8ZtqTqSANnVt9 z)t%!B&%aoKz6tR+0&*(uryl7%<&G6;>I0@87hK@zIHULE$<`0Pq9c991wa9+wDZ8v zj9dh$CFPyb^2;BAiBxDuJetxlfr~S^JB7d%fJ=p~@~uyA%>`uJ(a_rC9zTfGFAWMVlqo`{d{&i?@VHv7 zq>jon)R)Z`!uugXYdZEWx&)mrXIaFZdJ{pB3vHJK~U z{j8i?76d-mKdWi;QwtvGiY89k&Di-rqe8yPXDC^$_1oX|IDsYK6(cUsmOK^wLcY=? zbNvo&n9oG+k($C6*YtxuDZE`4~Z*{ic*zU>&VpQTtgaMsWj9I4t&+gnE0uSSNcn9{@mso z+9rDJdECTj@fTwOvz?WDbcvBPVNUK09p6Pi7Y4dmBPXUe@~LJAWTF%2%ax>=oeX5r zb#k}F1P1!_6*S+o9liQ&{H#MJisdCA8T+h2b}0H*KxD!eQdr1Ac8s%7E(OD+l{ylRi^cWB2EpF6{T=X{wvacylrfSIs4(&)l=bCs=I${+Sl9cY2qz zj8{xbe|0bSw#$A4m}ai;$;A%0#B(08sz2M|GmWIY^vtZUpxnN&+Ts=wjL^KCC^ zs9ECY_d)tBa4B<0>M8qd&2HVtoMGe2%Ng>J^Xo6<*zmRJ>_5`2IArfQc|HolrZ4nI zC;Dbz@qRO!@I8H4WFXk}cM7`OYNd>;_Ek9MK1q%1snPH0E#1B)6RM3-1TJ|nl5ytu zq)xXBM!IaGtOd4uX1o-mv!b8f(#sXtY}|H#ZwhZGiHNo8uWh~!X)5%$l^H(^ZcN{d zdoy!XJySuJB zzUQN^(T(FaTv!;7ac(@!w`$L@P}!t+m8c?W(}5%X{HR5?NB2T#==`FYuR}hS5C4i} z1&gY*?byRAoKZ-jZPnPdXFE$SYKSahGpej)8T(9G8@7e_PuL*{u(4P zS08OQH3W>b?Z}mt^VIe2vqmmpC-jd|FzO_8yILV+jR{7vorctZs$DUXS5)LUVnZFE z##9-3#?vR~3;C0jtwPTo`-P-I=QCAvbF)i6R2)BCd}Z#@th;mw+FjzLW+AiBNsyZN z(yS)R621gEI2!5u7-$8@x2Dod1*R)GJdeBkHQidg%b|AgH}P8X&eZ;0U$z1{X%VMO zLL5xWwF`;6v?eaURTfUA?01pLqE*9hg>?6?u0PN4{%XdoidYZ7&X4^*gI)`Lq0h-w zn9u&H=Z<{oCGw)OVEjPmdktLv+nV8r&wZEK7ZbR2pOwL2ElS8Ga_CuV*1 zjNcS>S{sp60S_vE)KrFr>h6y9?fI?!O!H;jzVMdeN3S&Z(fjX7_`D?U5F%J9?%MBD zkaWlbT25BR)kcx85~Y2!gLM$rYlbyDROJ2n##4_a&oD(n6-Lc|kB?@1ObfWJ2)OicBOlJu`ufELbe?JPbIes-y5>4x)p|Q)++3Ikil*R*Bi&2aCUmvQ#rYU z&hb`Ue9PcGP;Aw;V))Fqin!(C8H)ER{&NCDtnhsJ@GMQ&-M)a&-d~mD)rKx*sd{d0 zDTIeAMO-L7C0rk#;!C%nb)7DWXjpwI?pw@>7#D|gp~fEd0u*yzsg)x1wni5lKF>&X zWy;>ZFxaD9kTK_^u`fCeL_` zZ}n`=ezKHba+0ChIaU;Xoqwh94)OHo-SDwU4T4M_CHJd9epjQZHyPd^L8rK`I>f9A z+5kvIkvGq3=5M#og}ewBz3#hujPLGK%1bwTN_5<8-;YLyby0swv+9Xv&JZW#gl+v>$=54`83wZ`|;j{#vzt`?bDS} zp$qT*eDi7+O>zBwu|txtsNNpZ9kq74l^7 zSC_co{Q$<$Ld>m-G)?6$(ymL#H#ra*y-_Qp+vmbVZ};`(@7>57>fCxT$B0mCsdRE; z*Cr1M?CJ~HJx6b8vT%c{tFpXaHYNCIxCni+pM-3VGwB0Q1&gPiBnNUyr8C^Rl|ejk!R9 z3Q)V+8zxd8(td^Zfir>jZHRy=wh5>>`p!N)RT-QU{ZG_R9}wJ(o?p)(T* z4(GR^sNd_!)~oiICm#KbpI7m1C7fI|g*vIwWV0y^M>A0D4Yah>jC8LTvzHec`d;Pj z`Q?!_6%n_jX%Y1ujNLk=rG?qq`p(Q%Gy2?`LlAS0|zJB4Ai?(Vnp% zgIFp{*@qemRpQN5eNVr4nT0e^<;$B54yiSJ<vHTBlVMMr{=tFXbJ8Yn zo5`3UuEITqJg?6Eo#L_T=>;A_lHO;Zc`wKfxFnu)?)$l{ zkiW5j+>Z*~FLD6qJXGyH%iFL#*)-GH*-5*eBWaHJ5}*8D)AoLQd65s>Y}>Zx8(u;% z?Jb^YK2m#kpA3{#R^#iLR9$*KwzTHI6BEbi``MmT6ezii2Og}bsoQ0K z4F0BBcxi+Em!?N%xLf9%XKhN7P}UmoY@a9YTrHvM0}gCD4=!JiK`Wz~Zt?B8!< zVlW3zh5vjbD0sLF@bxbe|KZIqUi{07|LMG#eQ^KKQiEi&!ke2fW&)R|;8f$x837HD zRN0WWy%qlSdr?TJ!NrD?0ws(EV3P)0@xM%RjRdO;i-=@P6!#2JttnBMG9q1_wQbSk^BwZ^yWEp`$hipjm^OpCqcLPdbelBbU}w7iUS8b)D6_h z%uw_Nc0R7Ne*FnCE|rm?J0g|b8dE#&=h+>O9+b@aYnV<{VDnJU)8((LI5uJV%$ORP zevBXQanslIVuO@`UIp)=aXJF?5ON*$EI5&YV(1?WLblF}3glX_$wqW-z@GiTnz#B{yFp%SQC~2~QS|5t7RJhO}FOcgh zbr2yrcos7PKc<)cD5z(CA&xO1u;NK5uzZZzVTa(~tp(neZE}FJO8ZGwCa6}x{C(!3 z4RBfj47;eqvw&T_!R^9BDGsL_`@qv+9sDnEA!nFvcPy)LSC38lM^YrUU%T)SxEDyl zhhBgC8r)4JY;uSCvs24dR*I+v+o~O+vU?ov>5&tEQHR;Q!s$JK;^=Nei%ywJxYjFI8aD8yKP-hu(3H7nqFWc1_{e7#aQ1)HcOp zj1Mf){}2G>!TkSHDZKWjRdnuRp!6V^9{0I6OYrRCo-R$<-hbL;63cWxZNpUvB`Sfj%@Hb5iiz zKBELmhQgt@S>gk;o78O$DE-$yk~R#latsO0en?mjX#B$m`18^w_@$n#n%_L!6=VZ* z_Ha9R?(i2(Oh7abHOynUyF-nS{}+jW*zt=O|8N3={(s^HN17C|l{18m>-%2N%>AzJ`ZLiwryi#+qP})Boo`1m=o^Ub|%Th&Lk7-y#LQ|>fD`+)$69K zSJkSnuI{eq7ptl)gMx&Q1OWkoA}1@U{@-2x-^oOP`_J}|$ZkMDkV42wifQ^7T^Yjv z&>GnYQt5SMP`DwDlO2u{yX=jp!uO9Yk;X{JHIKzllvd&zDWIi(jMKS&QekW|0~dvb z>4@=PHeanSeV*j=xj$WE1sk&ETDm9Abx-x4Oi#ZY=RYyr&m>^Oh=n}IQO$=_7L62L ziuI!LNXe4ISpp)6X+m*9LJbOk;6{oP*yax?zlArCdCF}U;ey!U{PJMTR3AH;V+kjL7BX856jQCj2;=?;b~F9J>^ECSLWaYd|l@0(vx7Ym9; zQktHns^;$bGrHh%2mV2yYQDg*4z2_D60vmH4|ExQthnp2i$?20v)uPF1Anwfs;fp| z_^@(8vnqCmCWRe4(m)34%u)I*OWwCNo8a6!hh{X5>_Bwm=Ia-K|D2B+e(94ghTs=b*rib2ZK zTp@M(qKVm{r}hZgX`{$Br*51X=YWLQy2uu>r*nBrap)UtfIZFZ$9_{4n!Mj z`GU|il{b~>a@}!w*mYavdMjZ3I~Hy;xMJP@+*I}&ySb7n-vv@EkvamE#Rb;gz37aU zYO9tcUs1jS#H<3Cyq4I`s-pAYkDw53yp=&W6EEf%N;wxO?tSU%j0VYBLql8*M@mh9 zfeoMcigj@E;&Gc=D3eD6@uvHA zu7FTKw!0{>+|k%VN%|r&zoC}lI3JWgIS<4tImP>(jv9;;(uUd>hSL!IO)DN5*BThM zs~(Y527&wcdS6{k$}qGj4dZOfnOeH)xtnM~aWzq2q{x z05g)Fvs#&G6q(5<3NpVKho9APAUbO&5jOc(L~%I)y&iv2noE!-q?}3>5CE#GK|u*c zBfT}yDPUQysVG+k2vT0bY|1m4!l*?#RQVy39+dA5h-!!EM6sTISl5&1a8Pkpw5$hA zN)q}Enu?OJn?aO`U+7c{P}myg%r3HGA6tHlO7ipg^5}IX;MRzW(@ZPie1s+4h%Vlo zrGVc7B~ajk3fxSfqo`LN(D%ykoyG6nZ%i~*!zJURB~8TFmRtg$#5Yq48U`!7t;j-A zr%+5_s5hZW+N6b=jcOrOlq!&}Hncm3e#jP;;nCt#uXa=URKN=4i!rYyMOd1yOH<6V zdQrkT@B@xb2+I9%V&xu0zvjuzaVIRU?kua;wVnr5(TZ&ZTP{9!DGE=uiWs-zCX7QvFQa;DJ8>Si3753HlDo2^}iCPtO~{dl^)t6OVySWrKu8dyzt zfhI`8AT2$UeX132_?bNY%uo2bMPYGkI1Lx8H-KG@y-t6PAKpyU90*UFF@lw&zyID$ z1K?e|H$87p5NU^uXAZ=@zwyN;@RcZP>lrUb*it6FC9-JQp zZvm4a$&~UGQWx#u&!G)pZ&N=(hFfnQe?Toer^qWpsKQ=BPaQLv{$$LZT}Jb`)4kOi zTI*~nI62_!VI@H=2vMMqZd5nA#|^mwvqutt`;Tf#wPRaikxi|3t*Fd@yvG;q8_%}^ zx}(8}|*}oVP6g~FHQE`U4S4=o(K_~)}P8?N-F^;c$TJ3s0*xd!y^y*RW zEr)oqEhTIv1I{Oq!eS} z6h67`ywqm7&0|3J>H;c`$#)VyrZqPB0-SBLxGwwj`{k(+_@@uL*PwHARn!mcZ*oPpXj|-x&0uG2uK8(04^$!J=$G(=iUsoW45CBZ zjAX)ZKLSw>)~_?`+r=ps3a#`|Yn$e?DAUvUy4d-%#*ZeAks1dIND$cQ^hS zDY~^{=SSUO+SjObjea>_?kI%;0SxSjLQc2o50uh6HfUu() zp^Ce=;~PU(-GGdO@JQS$Hs_K1JPJW|4d*M$b$slb;&7Ejw7m1LOuIa%?`pp)si_yi zMX%p4-wR&X;eeCldF|9f4shMRv7}u;U@DbtCrDp%A&L3Uad0;+ibmIp{PWD^5U{T7 zd^^R^R;;iH*TjI#i&iM|!h6rY%ecgwt>l@x#IMPCiSlWMO`y?uJrf!+0X# zE4E45?H>w`rm|{8KA$Qgo#gO}OL+ffSGDp9-&p)9_tXI$|B=xjU5mXN+j2wPBcP zN3qbjc_oDk1l+$dchsHu%t!6in5M7aZxGN?e;&?~O(rH+ZUF?j4%|Qtpb&zK-tF@N zs_EbuvTAN!)Jh$h+V}vyMbQg4)DC7>-ZZz}-_38N&?-C;aV$Sz57U@_1pJe{^ca~n z8>O+v+8>a@QfI}DyfN&;U9{NTLoH?@&}NqwKM5D2N@M$RBr1`*Axr|rL+l61ZC5w) zjRv-3IU=C|20(NLte2#B(62fZ+*GxE5X(In=)j<*L`I16UH|)Usp&Wghc);&ycxCC z^szac(=Off?I)g(^JkBEQ_}W1!TR|g-2lzh?XWq=yrx6w7ENdq{3VPrN2A-USzVAl zY|UCl${joHU}z#HLK{dIWu|3;5HmG8Debyw#wgY?=@str6HF|c_^SFqQN`%(7DDYwoYfJR8dG%n0(o3H;S;I(% z|3yw5Y{TDN`@Pg6vvEny-*i~l9~vb!ulpplXRZfw)Q)JP2uW%zy7*%JmFh>8^kR)f zi-vgZ@3_^?4Bn$jbGF0IprH-A%K4=kv}y?=`-o2^^6N)49-9hHj=06WlYhn2*lDMl z{pH~t?q9nD1z8d$Kf=yzsyMoYiy;gT@QIGY&W0#fIg$G&8d3Mwg}sCibXu8x;D7e= z66mgt%r{Xq&Zv$3NxZljzRRdptzu&hn3fH=k14$!t@k$Hpqbj^_5GoRA9ekUt-bG2 z_N)v>3at|JSQ6XCs&2R*MYppw3@;;PPa5 zaQI_1{?MfVP+SEg;XKek3%I){HW8Adc!eaU%iT79JfYNLhJzC)_M01!!T zogFdLO;wjZN-T?%ZXuF_wy8RccC1t$q2agz%a8=ZEH$yZEh(`E^b@(>w5^pUr-v4A ztzV}Erk*W@z*p=*GM$tN>n4U}jl~tARc^cDH9Dg^YNM?n;v^}Q=Y^)KK-Lck$!y0w z@&VZ+@fYBBnwuu+hN=`OF03Qh0tZ{G2O#$Rce@aRyC39#l%93^vr0WS%NxauZCrVL z`eI3A_(?SySeDd^@}pfqk?g);9~bzuz;RZZtfL?Vp!DTRi9-VzLZ%#YNfWj7IN(S) z@#~F%Th87H$J>HLiSggrKuJvPB*~w-$z$nHvyy$M2+8mImHDBgFH?Vi>N<5r} z$j*A&c`z!pnVv6YK}kS%MCo;qivKO+ZxLO?e?Czm9-&zh5i1dbMnOFFgNiJQIj-@l zBb<+`brM3w1#I7a$lU(ya{jYim>-5X28h(35gTt4Vs_X`N)PY{kb|Q@j`I7r5gdYH zl8Uy$SRs|O{Uozj_pFrP!yN{ST9PPFTEfE`r#5#O*}b4guR2g_f(3M8zK-*KS%iap z$WV+DtX0wmybs zF<;gANk-DSzJKcc=eE{ceAO)lAqy&t2H8(;UzwUW&gz=ZBd%s!5g3p&d#R}2ZL3CW)mga{Ajg{(5?h$q(t zA7GMkfUu#z-+VAo64{rJ~=6-gv`5NFMHQoTWo73kAOcj>t} zyZ%LoI)Axp)RYIg^9*QNZoVge)!I z_@VH?PV9*%eYG1`CNuSf&@iVV6+E4MLtt=KB*v$gaAU@TEi9GRlZ3V6m`sbXcy-a> zH046@!6)yT%r0CXq6wPI6zr4v-!LpzC=y@Hra87~lMa$3SUA^(Y}B8JgR(mNet&~2 z1A6)|QF7@hxXbj$>d7Qr*KlQ%}p1Rwu*aK2Dez|#Hf*5pXl~O@J=Aj)dxfjw%>YHrGe?y+;ws)-k(1+u+7OZ>fES)8zDxVuSeRNUff$MW3Soi*j?QGAQJ-(1D-CZk zw%_M=#cB}`8{q7ubpM;4EAQv(U5ONRwRbh=b%}J440@MfaW_^&Hr1u8e4pJCy$9Xq zb-oY$wJwaraq2|AQpizDZ;(k}i$fDg96H7r-CrOemTc391#>mE*M!suw<1T~a3cl_ zospN}GnH-AQ?@0H!?<;Rj-9j?Z*x>dpl@XC zH-bw-=u={Ot^uRn9UO)%mXYnVpT+H(D06M|nY6{cM&MMQAmg$DA@7GPpRarsNzv_Q z#=|(+R0FZr^@_x9$Ol|@CtB`ZQ)Ei+3uFg;C$|(h{6dEz^WZ>CrhENVM|!cTvJj6D z$^7v2x%{~9-dd^8ldtze`ElDA4fAV%ftsoem(u{vm*mDBB~%tR#l+ggExQu`g`m9J9otQpn&`hBO7R=TMgM z!vgVl-v^53NsW|Gp9PFfEsM$NaPj)*d$<%J?YB^tcITD-ype-UjWH9t7MExnS-Sh# zhYNWh(RiQHz^}@ay~u19F>iL(Kmua#rK+Wc^`BV&gdNC_yr%Z4^8~*j-#;spi+zK7 zbablSX##*UFBGVm0N0zyoP>QGH&Xv~SWwmIgO^v=_CMoqBpe&0Dc{bc zqOAI1yA;s|AxI9qH_&sH3S!R50z*x6w%6k#JGxv}F>lZOa0K&IMK zXSPI8p>(Eww?2$%?J?)ep{!&sT(h zy8=p0kCk~EAfuHo$LboTB0A1;f-S?NuwAvyU|$pnWcT8sQp(2 zMjWct>0*jT$9=MsUxZLg5+=Um{!;5!;to1TN1-4iSdsY@WcgbM5?G;luU-`McA|Kp zzBZnyYd#%jpg!=I+Z?%|+sT2;mg1=qN&r}ftqJapxC7%+#GZj|hxWRS2vbrRwB01( z5VR8CfRaNe5W47nL>|}p%Rx6UZF_$+bjPpp zoWq6a`58k z{;5a-tOFFM&=(zX@IO~Zqi4D#zBAl}O@Be8cy05|{|0*2(iC@-%G#8NINEag zDk81q8ShWHs6>(^rFLSvwM`daXO0wqwgC-}f3eA#)zIaM?|1llbp( z0R@bMyl=&AooC5wfASRO20Fp7h^laA8%%AIzb^DthRSRdWP~k;GXYs4u6PlJ9K|fq zpUoFHY8?ssXzOnE?b$>0J&3#cFXn=QUl=|Y*WcL;2oPF*WNo<#rzA>OVXRZ4gjCj1 z&6T&+Yi8S0>_AW9|M`-yr8tbMsR={Fb?6hA8~BXDZ14WW?@! zB}>pIkmErCA9#fU@2nufW2hr)mBeA)JBObc48I{7US{%jUMJKC++DVoaHjC`?XIV^ zBAgf~+Q&Sn{J3nwwRe#++zQ=&*pVLKf5RfZm&w^72ei$zTI(Ek&TM%r%aqiPR8PVe2jK()evpa*G zJr`2yq^e0DHNz`y$-TPR7b1j|cagUWu$!qF@=>;f?*Vm2z3L7=Ifh$N4(0^tK}hZC zqN=Q>7q&K0OshMrObr&k>mGlse*9Xqa0nnhNKgB~7TOJyaqN8E3P*)Ik={u9T%sU+~ zWRC9-yH}XaC!x>=9v&3bI!QQ50UH56eW+RfRr(b_U>UoC)1XA{4x-R2^W|fT#sf$J zvveYOZ(jcR(5FJDF>};+7U7(JbX$O^-p?p~oA68+y{j}G^uKEzb5RPnZgTuHnQ74J z*_Xfm?D@_W`IOfkNmE<-n#-%O#5DpRs%A-M-P4Wk29UphVjw&F;Z(2EADRxdGx<0X z=%FiB?UqB*7X3iWulMq$Vv2oK+QOPo~qps0Ut72y# z7m!NbHZ5an&AOY{pq5VBK`0$d9OT||PCCroH#*Vm05+axXg!aa&;9AuRVeDv60?^p zVrE0wC;{C3&!P-o>cO8C#H!XCR9;m(6DKWnb!JNao7(U>&L>;1pq{~EeDI9mn z{XIZRYx|x0aCOSg-*z)tA(Y^$2YYr1qHR@lV4sbO$|9U_n|`2n#bqEi+U%h8bpl24 zeqhp<;r0xk#O!5gWW0W&bpEh}Z7ZQ&5!=fXDZ2oDe;edic~Zft7*R`s(nirr2aIK>R{ZAeM9M|~hVFStWrzZnI^4X|ALH+H zqxZqtu=Yw==*sG4N}A<0^J#N~DyKGzZKTGzV~=mn(Q^JyiJusY=mx(KT(}&^=Rlhv z{~USv2*n@pt9^wdV>iKZzRuHQl{d`bNmM-(9FD4$H!yuU+F{pCd8`*0Ct4T}_{3-3 zPQx?DzK>65p~2v?EwyDj#t}8e`t2`HLBu`bFP{8qU#Sz9vy3GQND%S* zs<*uMD5fQW5~dhJPL#{#+ZhQ_pXjL5WkIFQtQ(nG!QBZwc!uMoR=&NZ-Tk&9VI)V~ z4z?8!mo9ZGU74cDuaRoni@CBgm=UX8NeIPYGWeV~HG|ac3t&X1KqmKKNy|8*jj{IV zGqR|+wk2|%yiGwD5hRtn>|{VjHE7sBJ~KiG{TAH2lV}hPR4}HD$3tTP{f2g52&8Ol zb#d^JkWph9;OY7>hM3#q7bgLBd_y2~v07U8sfj?`VM_s|HMVjBb~6bckcl%#VLCwj ze~^`Uj%J5KsUOQJIo!S{LxFX=6T=FlwT_D0obVdD?=k;v5!33KrQ5ki#3~jq4-RCG z>geR9G!-Dc@(BKhl&`RK|7c%s+qRP7t!N-XN937Xi>R@1a6JoWV7>a)U)ha+8lvHtMF-6fm?F@S^}od^XOCTFD)0ctH?Gc4P{1*c2g~ z!SD=NbEjVvouGoAlE_up<}!$`jXXTFf$7PdZ{&Q25b3<8RphVO%`+8g;e&)Wxd&@Wc=wL#8flgnOH%-2hM==LOQ{0%yZLp2OF!>xT&bRr*qyvUZh%{z)<^1a9;uQ)8hUOU58jYhCr{R@ zT%28V%;@%#&Iy8H*5@*R4GvWvR?+E<#;dxFXCNI%+@7JjV%YaGkJpFn+gn|1YdhiX zG8W69a`@j7{hW!M_`|c9axN|Rkb(B!XGP4YbnHEGA4$_)J<3bzt_yo1vBfPgg>8{grqZ77@`fepY zup%KN?)VuRBRDsax+mg2j<*B`)f_oPj2?E5H6HZtn?{H-$JQ9NfRY%O7My{ilgV79 zfjax$5%V>n()LEvQEBnA_ucD@T-_4b!_O#*OBz5OR4SzN%v8F9vw^jFLTI>-I<{ZU8pZFLlV zKak4^uHeRHi(w+ffX3;Zwws-kW$tBlL|D>7img`ihQH_m_d3N1kG4BOCWp4G72feV zC-Ij=V?~NeYcW?ZZjd9#Z0{kXB0n!#{u!WB@%}EQ(k-w0&mo&DBg@bAx9jYf_nLb3 zfiilUL}o~*&RbJPpTFkI-GM28?M*WQy!(7q9#*`l$|sR@?MX77c7OK#*h^G}_BZxn zXzHcYaPgK`plm8a$veCjJayU=xjxwe8^uUjhVHyIopd#_u{7-pdlMS4KVi0Hvo>^W zF7Ad1Jt^$~lRq~Vg64fdAYRd%l+7I zFj)Xu4Jp5RvmRnJlK0a>CLh1o`yX$U(X@5IltWkFAh@7{hIq176MI3mp;om!&clxq zorb+i15h1fS|F4|?}g4Kza=|&uEV6Sr*=aHi_mxCDVkGNB3o(Dta=ktK1X$)j5Ut23f@@Mgs7 zsB~*vJH_0+OC1b5*TX4C6-kscI}EhU(_p8g9h*t*w}=(DPyP#^M)ZH1foxkKRtbox zVv^a{M)jO#hp6sh80C$M_ZsJ&`@*-M+PjJf$9&)JrZ-^n!jdAG=00anLM`;7NFxg= zP#scg42gD2Gu1QIFx0a*C%6>78CiO%X2Fi@zMLdr{&wZ)SnisCzfr<5V)A*YfYz{O zzNDYKO}`Dxt{$2%`td>PzV64_Ukg_mO_ZrTc*+`a_a`HN$2|lB{u&PORt~ljWnayb z*cn9$3&bcvjzyoi8|ly9b@suhcV(9-1CjZTXlD}vz4l{}jLIwNbR%x?T+pK0SxJfi zE^L!Vi!LqA)zyN5 zNey^akL~Yx9GX)>41BzdDgYQn>jw*K8R=?Au1f?lC(h-roxk|`4<+RVk1peO`+P|k zc7|ii7M&MSP?DhE%i;SoLki3|70W+g2N04q5KVE}Jutm2-O2vUgSVqnQH387`Bh; zt5?HpWfJ_k4WgEmoZ9OZpV9(VUMC8F-=LgJ_l>HH4{iZ0fyQv|ci(f5svhIG1WJfZ zZ}pps7ysD#i$O(MJ~ulUH8a2T_0e<$XkQ)6OZV|yxD!$b{$S#=gN@q`TS2 zULl%ONH^Wtk$I&pvS95S{4amy z%_C!EEtgJ<0t8Va&V(Hg6@XOrP84WL#h5v$5E^Y6Dao&s%7HN{<>*v8tuNId~!rQ}4(o0V-ndzM$$nra2Q`_(tPW8}YV^ z6qBGas6F=U*m-1c0kSNjZn@jc{!{x{rgez{e&@uKa*M>W*MU1;k(xq>VX|G#_vqet z&?%$F)sRTP+Al*dgF2iOZI$YYHdHfACN#OMsexOwuhviawcwnn8%o;qW^D+RaqeyC zRjy8g`AtBgSM*3$1I6CiSw>4M1H!vJQKLGpWZ-C5a{`7txFk$(ZQL+xV~~Ayq~sD{ zX(qWjY5FHDNS;dQ>;2y6AdzQ&eLd#$vg9 z#3u8xKaF4@gE#k74wpT)QP7Jeq35z1F29FOwqb^-TNhzN&ow_JFA`{4Ltmd^a?^E; zNMHr0jqIlH&56;lyA`0fUlj4U>(p}TUp z&TbN@xqKPJkN5RFCK6%rJPY&CzU34j__!gno|FMCI&qd9{9fu)c@uCVV7y-G-2cH% z9B?bP@G11Pea{iz)SLuQ>?4|I8gw^7Q)=8Fbo5lt=+u2iO(wFc;E35qwCjH(bgSF! z#;ca_-i)5TIH z>gQuef1n!4pI`U0VEdoF)SOe(*&J&VcSr`Hm7Aa^0l~k+7`U7^-zt=HwY?&A78Ybk zn9h6dX9cFvklIt{gg&1S!erNjEQ4q@;=Zi(WfJkJ3ciwSoJTl&Zu$hccRV!M7N4PN zwd;q7ytm2tTZ@Z}(@kEWG@GJ{{b>Ia+i;{{mCzp)_+O(HmEE>)#@TKyhN^ouVW9Hv zK@9Tte?I&iIUxzS4ek8O_x_C*W~EP$_ouF9cIcPy{q4GWaxCy&@QwSN5KYx0ks?`8 zofXniy>IK^*^=vjsklD?jXJCnbcdn#9rK5UD-dc`CM4hYI2&sCl5xe?|EBlJI**Qv z_a$fI->NMfgRsH9kDTvhcseSr7jDSi>_j7YpE-ygaKc#LkHtzP&H}`o(CB zj_?*eP0*{!8}lLU+yD7`(fKEi!DX6l41u)8AWd!gtkwgw-^C-6CxA&&E=6^Kz_B&e zgoiz@d)2b#Z}~Go*YTph1G`iu(iTD33t#KlIq04MQvS$;woTqY-DKS~n#koS<9nv= z<9;;BXd=(=7>-$92?!%ZsnBGi$5nl2)6oRHu!R*Y*mU*6I9=h-XLH4Tel3U#8Ie~t z++2|2z^wHfZ&gVF$JX0%uJfNUkx#bv4J!gR9S~s&aR6sUP2cz57eQ}V(Szae2mi&x zn!DR6T|n{_xd=K;m2m;1@F0wq@gH-j88jdj~7aILJ5quMY64qwwR> zdfoRVe@e_vSVfyu7MoFr6kb zphg4K0=^l4%rbC`wLx~2o3ws7UpeAE98aZyCAc$t^GR-YpO{`*SqY;n)ss{br}cl` z3rEFfe_ty6j1c*HnJZBOZT%-xnxt=ERQq+=JSF;O44DmmiPQfKORba>fuSOZ*1BOE zr!~r@C9n=B{556#slNVUp%A2v$M}`J36ih6jIb^Y)J7;^k=(la-v64>h`dscFTZtO z6b?k^rh>)is+s_`+CEIB8xPC z#6tbA)Ux?MUfwC9!ULwr!7a)ovh&G&O zU5dK7YVSQyzL2o@f)oxSe9|nWKng(76YolKB!tk#0SDfCS>d>+Moj)b#*O^8nrodJ zDguZaT6rDs_W?t@!UNBu#rYO`GX^Po^7n1`s}C2Z)my7y8*cW6Z$%_jG4n{Eis5|! zCqvj@%OCE&+0D)9+!1h}yI-H)DN3g>s`;w73)z^$9aGFb2+1#~!O!6kjp)QOg0RY& zq{y_^l89Z^)rh;Vs~!JII}E%iSFi_FR!d|EQRZ%)GnF2c+9rW&uA*4}b%3m@r;>WV zML8P89-r+_*CBuF6Aac%11~Gfs#xf|;%H)YQIYG;1Ify!&yjc>WIR<%BcJ_9@rQ-3 zxEwUQw2*WE=iB`uP80o7dYz3`#@tk3jcJWRAGD+5s_Xp>kNLmT19DX4-NpGj`Uzvex9ryZ_B_pY+>0jJ;;yy>1Ob4No&;qNCA5ipG%vi;z`V zG~?}c&kYhyWL;&04P*MWoVL0CIvNZaULtAn>xnfCoydtviEpuk5Z0K5KcZj`Oq|Qw7LdE3NTz?kvphgii}ns^X?ma3<>({f zT1r-f(7 z*IU@HHNZL&G>;ulq{He|QS)uuoEm$sXo-JFRnD!u@Ka-cTVc&kh9V0&nayX=6c8&y zr(Jh}QiuIr-EIqNF$WPxSV0(rnFJ9EtCguh9+Y*1^) z!GV<)*efq0@B65cfr_IV95!nHAn4S)az%!GVs@&0iYOo4oRVM9M$7CTFnHlqYAz<) z{55vk31qZ66qN8lMk7X<2`)P;53C;-DDqr!Hih%N6?Ub2Rc;9Co7^x&EL@W%dz2XS zLB$6Wkj?IA85&%)4`>fMx#Ln$bR3#}37R)XxsZNoOYW|1cn&2&~;rq5t7LLT*X+XJP+QKVD)w kf9wCh+yDKC?0rE1Q=kqetxhcdW4<8dq?9Ea#7)Bf58^|Y Date: Thu, 2 Mar 2017 09:23:54 +0800 Subject: [PATCH 303/407] The attribute [allowfullscree] must have a value Signed-off-by: yupengzte --- case-studies/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/case-studies/index.html b/case-studies/index.html index e4e15111c5..cf96a049d5 100644 --- a/case-studies/index.html +++ b/case-studies/index.html @@ -94,6 +94,6 @@ cid: caseStudies
- +
From 8f30034cc256d74cf647f2c8f036cb13a8dc1639 Mon Sep 17 00:00:00 2001 From: Viet Hung Nguyen Date: Mon, 27 Feb 2017 12:09:27 +0700 Subject: [PATCH 304/407] Remove newline before passing to base64 --- .../distribute-credentials-secure.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/tasks/configure-pod-container/distribute-credentials-secure.md b/docs/tasks/configure-pod-container/distribute-credentials-secure.md index 69d5d37890..923d7df6c6 100644 --- a/docs/tasks/configure-pod-container/distribute-credentials-secure.md +++ b/docs/tasks/configure-pod-container/distribute-credentials-secure.md @@ -22,11 +22,11 @@ Suppose you want to have two pieces of secret data: a username `my-app` and a pa convert your username and password to a base-64 representation. Here's a Linux example: - echo 'my-app' | base64 - echo '39528$vdg7Jb' | base64 + echo -n 'my-app' | base64 + echo -n '39528$vdg7Jb' | base64 -The output shows that the base-64 representation of your username is `bXktYXBwCg==`, -and the base-64 representation of your password is `Mzk1MjgkdmRnN0piCg==`. +The output shows that the base-64 representation of your username is `bXktYXBw`, +and the base-64 representation of your password is `Mzk1MjgkdmRnN0pi`. ## Creating a Secret From e1212ea63baaa73b27523a8f5d7007f512fa92c0 Mon Sep 17 00:00:00 2001 From: Hung Nguyen Viet Date: Mon, 27 Feb 2017 15:33:56 +0700 Subject: [PATCH 305/407] Use single quote to avoid string interpolation, update output --- .../distribute-credentials-secure.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tasks/configure-pod-container/distribute-credentials-secure.md b/docs/tasks/configure-pod-container/distribute-credentials-secure.md index 923d7df6c6..1ccb48e344 100644 --- a/docs/tasks/configure-pod-container/distribute-credentials-secure.md +++ b/docs/tasks/configure-pod-container/distribute-credentials-secure.md @@ -42,7 +42,7 @@ username and password: **Note:** If you want to skip the Base64 encoding step, you can create a Secret by using the `kubectl create secret` command: - kubectl create secret generic test-secret --from-literal=username="my-app",password="39528$vdg7Jb" + kubectl create secret generic test-secret --from-literal=username='my-app',password='39528$vdg7Jb' 1. View information about the Secret: @@ -69,8 +69,8 @@ username and password: Data ==== - password: 13 bytes - username: 7 bytes + password: 12 bytes + username: 6 bytes ## Creating a Pod that has access to the secret data through a Volume From f6bb5e6d44f6ec79976915d86407f03fdaf38726 Mon Sep 17 00:00:00 2001 From: Hung Nguyen Viet Date: Mon, 27 Feb 2017 15:34:42 +0700 Subject: [PATCH 306/407] Update command as files contain no newline --- .../configure-pod-container/distribute-credentials-secure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/distribute-credentials-secure.md b/docs/tasks/configure-pod-container/distribute-credentials-secure.md index 1ccb48e344..2deaaa5f97 100644 --- a/docs/tasks/configure-pod-container/distribute-credentials-secure.md +++ b/docs/tasks/configure-pod-container/distribute-credentials-secure.md @@ -112,7 +112,7 @@ is exposed: 1. In your shell, display the contents of the `username` and `password` files: - root@secret-test-pod:/etc/secret-volume# cat username password + root@secret-test-pod:/etc/secret-volume# cat username; echo; cat password; echo The output is your username and password: From e6ec869ca74d96eff79b8a4f3507c16268032ce0 Mon Sep 17 00:00:00 2001 From: Hung Nguyen Viet Date: Mon, 27 Feb 2017 15:38:02 +0700 Subject: [PATCH 307/407] Remove all dead links, use just filename As user can guess that it is file copied from the shown content --- .../distribute-credentials-secure.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tasks/configure-pod-container/distribute-credentials-secure.md b/docs/tasks/configure-pod-container/distribute-credentials-secure.md index 2deaaa5f97..1a0e25e46e 100644 --- a/docs/tasks/configure-pod-container/distribute-credentials-secure.md +++ b/docs/tasks/configure-pod-container/distribute-credentials-secure.md @@ -37,7 +37,7 @@ username and password: 1. Create the Secret - kubectl create -f http://k8s.io/docs/tasks/administer-cluster/secret.yaml + kubectl create -f secret.yaml **Note:** If you want to skip the Base64 encoding step, you can create a Secret by using the `kubectl create secret` command: @@ -80,7 +80,7 @@ Here is a configuration file you can use to create a Pod: 1. Create the Pod: - kubectl create -f http://k8s.io/docs/tasks/administer-cluster/secret-pod.yaml + kubectl create -f secret-pod.yaml 1. Verify that your Pod is running: @@ -127,7 +127,7 @@ Here is a configuration file you can use to create a Pod: 1. Create the Pod: - kubectl create -f http://k8s.io/docs/tasks/administer-cluster/secret-envars-pod.yaml + kubectl create -f secret-envars-pod.yaml 1. Verify that your Pod is running: From 0c6b5c6c3f8b7986308802add717882145cc14cf Mon Sep 17 00:00:00 2001 From: Jeffrey Regan Date: Mon, 27 Feb 2017 13:57:50 -0800 Subject: [PATCH 308/407] Fix link to dev guide. --- docs/user-guide/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 0d4e77a97d..1f045cb7e0 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -8,7 +8,7 @@ The Kubernetes **Guides** can help you work with various aspects of the Kubernet * The Kubernetes [User Guide](#user-guide-internal) can help you run programs and services on an existing Kubernetes cluster. * The [Cluster Admin Guide](/docs/admin/) can help you set up and administrate your own Kubernetes cluster. -* The [Developer Guide](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel) can help you either write code to directly access the Kubernetes API, or to contribute directly to the Kubernetes project. +* The [Developer Guide] can help you either write code to directly access the Kubernetes API, or to contribute directly to the Kubernetes project. ## Kubernetes User Guide @@ -86,3 +86,5 @@ Pods and containers * [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/) * [Assign pods to selected nodes](/docs/user-guide/node-selection/) * [Perform a rolling update on a running group of pods](/docs/user-guide/update-demo/) + +[Developer Guide]: https://github.com/kubernetes/community/blob/master/contributors/devel/README.md From b86ebf19915265be4322ae98ba17bcea78311d41 Mon Sep 17 00:00:00 2001 From: houjun Date: Fri, 24 Feb 2017 14:53:02 +0800 Subject: [PATCH 309/407] Add --leader-elect=false Second scheduler could not start without disabling leader-elect. --- docs/admin/multiple-schedulers/my-scheduler.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/multiple-schedulers/my-scheduler.yaml b/docs/admin/multiple-schedulers/my-scheduler.yaml index 0a140deead..aa7cc1baef 100644 --- a/docs/admin/multiple-schedulers/my-scheduler.yaml +++ b/docs/admin/multiple-schedulers/my-scheduler.yaml @@ -17,7 +17,7 @@ spec: spec: containers: - command: [/usr/local/bin/kube-scheduler, --address=0.0.0.0, - --scheduler-name=my-scheduler] + --scheduler-name=my-scheduler, --leader-elect=false] image: gcr.io/my-gcp-project/my-kube-scheduler:1.0 livenessProbe: httpGet: From 7dcfaf0b87be75b8c9c138df475e15d662fbdce6 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Mon, 27 Feb 2017 16:33:50 +0800 Subject: [PATCH 310/407] cluster-components-could be running could be doing sth --- docs/admin/cluster-components.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/cluster-components.md b/docs/admin/cluster-components.md index a3d4f37779..1f3404d6e4 100644 --- a/docs/admin/cluster-components.md +++ b/docs/admin/cluster-components.md @@ -15,7 +15,7 @@ cluster (e.g., scheduling), and detecting and responding to cluster events (e.g., starting up a new pod when a replication controller's 'replicas' field is unsatisfied). -Master components could in theory be run on any node in the cluster. However, +Master components could in theory be running on any node in the cluster. However, for simplicity, current set up scripts typically start all master components on the same VM, and does not run user containers on this VM. See [high-availability.md](/docs/admin/high-availability) for an example multi-master-VM setup. From 40bf97accf52ea183e3cae2264152d0bd64e7090 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Tue, 28 Feb 2017 09:50:01 +0800 Subject: [PATCH 311/407] update cluster-components.md In theory,Master components can be run.... --- docs/admin/cluster-components.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/cluster-components.md b/docs/admin/cluster-components.md index 1f3404d6e4..878cb443a2 100644 --- a/docs/admin/cluster-components.md +++ b/docs/admin/cluster-components.md @@ -15,7 +15,7 @@ cluster (e.g., scheduling), and detecting and responding to cluster events (e.g., starting up a new pod when a replication controller's 'replicas' field is unsatisfied). -Master components could in theory be running on any node in the cluster. However, +In theory,Master components can be run on any node in the cluster. However, for simplicity, current set up scripts typically start all master components on the same VM, and does not run user containers on this VM. See [high-availability.md](/docs/admin/high-availability) for an example multi-master-VM setup. From 1abe7577cc47e72ae181799a5a1fcf64b34a25cc Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Wed, 1 Mar 2017 10:21:57 +0800 Subject: [PATCH 312/407] update cluster-components--add a space add a space after the comma: "In theory, Master components..." --- docs/admin/cluster-components.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/cluster-components.md b/docs/admin/cluster-components.md index 878cb443a2..280b9b1f2c 100644 --- a/docs/admin/cluster-components.md +++ b/docs/admin/cluster-components.md @@ -15,7 +15,7 @@ cluster (e.g., scheduling), and detecting and responding to cluster events (e.g., starting up a new pod when a replication controller's 'replicas' field is unsatisfied). -In theory,Master components can be run on any node in the cluster. However, +In theory, Master components can be run on any node in the cluster. However, for simplicity, current set up scripts typically start all master components on the same VM, and does not run user containers on this VM. See [high-availability.md](/docs/admin/high-availability) for an example multi-master-VM setup. From a380fec339ec4c25a6738425d04a73bdcd25584d Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Thu, 2 Mar 2017 16:07:41 +0800 Subject: [PATCH 313/407] Highlighted command --kubectl describe In web page ,there is no space ,need some empty space,thanks! --- docs/user-guide/kubectl/kubectl_describe.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/kubectl/kubectl_describe.md b/docs/user-guide/kubectl/kubectl_describe.md index a5fd703d73..6db3c2d9b7 100644 --- a/docs/user-guide/kubectl/kubectl_describe.md +++ b/docs/user-guide/kubectl/kubectl_describe.md @@ -10,9 +10,9 @@ Show details of a specific resource or group of resources Show details of a specific resource or group of resources. This command joins many API calls together to form a detailed description of a given resource or group of resources. - +``` $ kubectl describe TYPE NAME_PREFIX - +``` will first check for an exact match on TYPE and NAME PREFIX. If no such resource exists, it will output details for every resource that has a name prefixed with NAME PREFIX. Valid resource types include: From abe3171d0a07f83bb7d6369c77a2d5898e08c874 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Thu, 2 Mar 2017 16:08:40 +0800 Subject: [PATCH 314/407] Revert "Highlighted command --kubectl describe" This reverts commit a70d0a3e3537c3f91b197b23f949e1e506c951d1. --- docs/user-guide/kubectl/kubectl_describe.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/kubectl/kubectl_describe.md b/docs/user-guide/kubectl/kubectl_describe.md index 6db3c2d9b7..a5fd703d73 100644 --- a/docs/user-guide/kubectl/kubectl_describe.md +++ b/docs/user-guide/kubectl/kubectl_describe.md @@ -10,9 +10,9 @@ Show details of a specific resource or group of resources Show details of a specific resource or group of resources. This command joins many API calls together to form a detailed description of a given resource or group of resources. -``` + $ kubectl describe TYPE NAME_PREFIX -``` + will first check for an exact match on TYPE and NAME PREFIX. If no such resource exists, it will output details for every resource that has a name prefixed with NAME PREFIX. Valid resource types include: From 6a35189e21bc70e0b012f6fafb084039f5fc0157 Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Tue, 28 Feb 2017 17:29:55 +0800 Subject: [PATCH 315/407] Add example show how to configure proxy for kube --- docs/admin/kubeadm.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 2145b75310..fe24c3e8a8 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -260,6 +260,22 @@ These environment variables are a short-term solution, eventually they will be i If you want to use kubeadm with an http proxy, you may need to configure it to support http_proxy, https_proxy, or no_proxy. +For example, your kube master node ip is 10.18.17.16, and you have a proxy support both http/https on 10.18.17.16 port 8080 + +You can using following command + +```bash +export PROXY_PORT=8080 +export PROXY_IP=10.18.17.16 +export http_proxy=http://$PROXY_IP:$PROXY_PORT +export HTTP_PROXY=$http_proxy +export https_proxy=$http_proxy +export HTTPS_PROXY=$http_proxy +export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com,example.com,10.18.17.16" +``` + +Remember change ```proxy_ip``` and add kube master node ip to ```no_proxy``` + ## Releases and release notes If you already have kubeadm installed and want to upgrade, run `apt-get update && apt-get upgrade` or `yum update` to get the latest version of kubeadm. From 4cba37750b1ec9389b787249a75eb7450f311b6a Mon Sep 17 00:00:00 2001 From: huzhifeng Date: Fri, 3 Mar 2017 12:25:31 +0800 Subject: [PATCH 316/407] Revision as the suggestion describe --- docs/admin/kubeadm.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index fe24c3e8a8..787b59bdbe 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -260,7 +260,7 @@ These environment variables are a short-term solution, eventually they will be i If you want to use kubeadm with an http proxy, you may need to configure it to support http_proxy, https_proxy, or no_proxy. -For example, your kube master node ip is 10.18.17.16, and you have a proxy support both http/https on 10.18.17.16 port 8080 +For example, if your kube master node IP address is 10.18.17.16 and you have proxy support both http/https on 10.18.17.16 port 8080, you can use the following command: You can using following command @@ -274,7 +274,7 @@ export HTTPS_PROXY=$http_proxy export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com,example.com,10.18.17.16" ``` -Remember change ```proxy_ip``` and add kube master node ip to ```no_proxy``` +Remember to change ```proxy_ip``` and add a kube master node IP address to ```no_proxy```. ## Releases and release notes From c62eb411cc0112a7dac1bc92e67cd0585d39955a Mon Sep 17 00:00:00 2001 From: Xiuyu Li Date: Wed, 1 Mar 2017 17:43:00 +0800 Subject: [PATCH 317/407] move ha-master-gce.png to images/docs Signed-off-by: Xiuyu Li --- docs/admin/ha-master-gce.md | 2 +- {docs/admin => images/docs}/ha-master-gce.png | Bin 2 files changed, 1 insertion(+), 1 deletion(-) rename {docs/admin => images/docs}/ha-master-gce.png (100%) diff --git a/docs/admin/ha-master-gce.md b/docs/admin/ha-master-gce.md index 025035cca7..22e177980f 100644 --- a/docs/admin/ha-master-gce.md +++ b/docs/admin/ha-master-gce.md @@ -107,7 +107,7 @@ This operation may be sped up by migrating etcd data directory, as described [he ## Implementation notes -![](ha-master-gce.png) +![ha-master-gce](http://kubernetes.io/images/docs/ha-master-gce.png) ### Overview diff --git a/docs/admin/ha-master-gce.png b/images/docs/ha-master-gce.png similarity index 100% rename from docs/admin/ha-master-gce.png rename to images/docs/ha-master-gce.png From e610c89cc478736201b2a6c3c9a2ed5a4164554f Mon Sep 17 00:00:00 2001 From: Xiuyu Li Date: Fri, 3 Mar 2017 11:03:23 +0800 Subject: [PATCH 318/407] use relative path for ha-master-gce image Signed-off-by: Xiuyu Li --- docs/admin/ha-master-gce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/ha-master-gce.md b/docs/admin/ha-master-gce.md index 22e177980f..27c51c9d25 100644 --- a/docs/admin/ha-master-gce.md +++ b/docs/admin/ha-master-gce.md @@ -107,7 +107,7 @@ This operation may be sped up by migrating etcd data directory, as described [he ## Implementation notes -![ha-master-gce](http://kubernetes.io/images/docs/ha-master-gce.png) +![ha-master-gce](/images/docs/ha-master-gce.png) ### Overview From f88f28bcedbb6deb7a475cde78a94c1cc53373c6 Mon Sep 17 00:00:00 2001 From: yupengzte Date: Thu, 2 Mar 2017 10:39:39 +0800 Subject: [PATCH 319/407] The attribute [calendarWrapper] must be unique Signed-off-by: yupengzte --- _layouts/docwithnav.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_layouts/docwithnav.html b/_layouts/docwithnav.html index 073e1c9344..3d12017366 100755 --- a/_layouts/docwithnav.html +++ b/_layouts/docwithnav.html @@ -33,7 +33,7 @@
  • SUPPORT
    • From 5f22fabf80dc07adf98da2e9bc60de660a9cdec3 Mon Sep 17 00:00:00 2001 From: Bruce Auyeung Date: Thu, 2 Mar 2017 11:25:15 +0800 Subject: [PATCH 320/407] add required images in kubeadm init step --- docs/getting-started-guides/kubeadm.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index 24c9498e44..03cf546eaa 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -103,7 +103,15 @@ Note: Disabling SELinux by running `setenforce 0` is required in order to allow ### (2/4) Initializing your master The master is the machine where the "control plane" components run, including `etcd` (the cluster database) and the API server (which the `kubectl` CLI communicates with). -All of these components run in pods started by `kubelet`. +All of these components run in pods started by `kubelet` and the following images are required and will be automatically pulled by `kubelet` if they are absent while `kubeadm init` is initializing your master: + + gcr.io/google_containers/kube-proxy-amd64 v1.5.3 + gcr.io/google_containers/kube-controller-manager-amd64 v1.5.3 + gcr.io/google_containers/kube-scheduler-amd64 v1.5.3 + gcr.io/google_containers/kube-apiserver-amd64 v1.5.3 + gcr.io/google_containers/etcd-amd64 3.0.14-kubeadm + gcr.io/google_containers/kube-discovery-amd64 1.0 + gcr.io/google_containers/pause-amd64 3.0 Right now you can't run `kubeadm init` twice without tearing down the cluster in between, see [Tear down](#tear-down). From d360cee99270e23de0d61c76a29cc05b1f200e41 Mon Sep 17 00:00:00 2001 From: Rob Paisley Date: Thu, 2 Mar 2017 19:41:28 -0500 Subject: [PATCH 321/407] Add links to Docker/rtk in cluster-intro.html --- docs/tutorials/kubernetes-basics/cluster-intro.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kubernetes-basics/cluster-intro.html b/docs/tutorials/kubernetes-basics/cluster-intro.html index 830b651594..a7ca857705 100644 --- a/docs/tutorials/kubernetes-basics/cluster-intro.html +++ b/docs/tutorials/kubernetes-basics/cluster-intro.html @@ -74,7 +74,7 @@ title: Using Minikube to Create a Cluster

    The Master is responsible for managing the cluster. The master coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates.

    -

    A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master. The node should also have tools for handling container operations, such as Docker or rkt. A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

    +

    A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master. The node should also have tools for handling container operations, such as Docker or rtk. A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

    From 94432910664737f4dc2833b2eed3d5a6eca33e7a Mon Sep 17 00:00:00 2001 From: Chris Schmich Date: Thu, 2 Mar 2017 19:11:11 -0600 Subject: [PATCH 322/407] s/acomplish/accomplish/ --- docs/user-guide/replicasets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/replicasets.md b/docs/user-guide/replicasets.md index ea3e7bde14..2b6a767be2 100644 --- a/docs/user-guide/replicasets.md +++ b/docs/user-guide/replicasets.md @@ -94,7 +94,7 @@ of the replicated pods. kubectl create -f hpa-rs.yaml ``` -Alternatively, you can just use the `kubectl autoscale` command to acomplish the same +Alternatively, you can just use the `kubectl autoscale` command to accomplish the same (and it's easier!) ```shell From e06617a5a146fe69e0632755c78ea75c25f81f87 Mon Sep 17 00:00:00 2001 From: houjun Date: Fri, 3 Mar 2017 09:59:55 +0800 Subject: [PATCH 323/407] Modify the link URL of [kubelet eviction design doc] [kubelet eviction design doc] should be linked to kubernetes.github.io instead of community/design-proposal. --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index ad55b773a5..f34b7e3156 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -72,4 +72,4 @@ Including: | `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources | | `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources | -See [kubelet eviction design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-eviction.md) for more details. +See [kubelet eviction design doc](https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/admin/out-of-resource.md) for more details. From 34cb50be94ed451464ae0ef228c0f988169f8a30 Mon Sep 17 00:00:00 2001 From: foxyriver Date: Fri, 3 Mar 2017 14:25:56 +0800 Subject: [PATCH 324/407] fix CronJob object name fix CronJob object name --- docs/user-guide/cron-jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/cron-jobs.md b/docs/user-guide/cron-jobs.md index 34402d5568..db3153688a 100644 --- a/docs/user-guide/cron-jobs.md +++ b/docs/user-guide/cron-jobs.md @@ -32,7 +32,7 @@ A typical use case is: ### Prerequisites -You need a working Kubernetes cluster at version >= 1.4 (for ScheduledJob), >= 1.5 (for CronJobs), +You need a working Kubernetes cluster at version >= 1.4 (for ScheduledJob), >= 1.5 (for CronJob), with batch/v2alpha1 API turned on by passing `--runtime-config=batch/v2alpha1` while bringing up the API server (see [Turn on or off an API version for your cluster](/docs/admin/cluster-management/#turn-on-or-off-an-api-version-for-your-cluster) for more). You cannot use Cron Jobs on a hosted Kubernetes provider that has disabled alpha resources. From 76f9a1b7853c5d38f3182773cef48844385259b3 Mon Sep 17 00:00:00 2001 From: Xiuyu Li Date: Fri, 3 Mar 2017 14:40:34 +0800 Subject: [PATCH 325/407] remove redundant a Signed-off-by: Xiuyu Li --- docs/user-guide/deployments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index f2e854690e..d99909916c 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -621,7 +621,7 @@ See the [Kubernetes API conventions](https://github.com/kubernetes/community/blo Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with `Reason=ProgressDeadlineExceeded`. -**Note:** If you pause a Deployment, Kubernetes does not check progress against your specified deadline. You can safely pause a Deployment in the middle of a rollout and resume without triggering a the condition for exceeding the deadline. +**Note:** If you pause a Deployment, Kubernetes does not check progress against your specified deadline. You can safely pause a Deployment in the middle of a rollout and resume without triggering the condition for exceeding the deadline. You may experience transient errors with your Deployments, either due to a low timeout that you have set or due to any other kind of error that can be treated as transient. For example, let's suppose you have insufficient quota. If you describe the Deployment From a06df4d14ae43b7ef9f7d948894e1dae7c747ad8 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Fri, 3 Mar 2017 16:18:17 +0800 Subject: [PATCH 326/407] kubeadm reference--change any to some (#2683) * modify one word examples to example * doc-walkthrough-content modify * limitrange-update I think it's redundant that the second to ,in order to do sth and do sth!thanks! * cluster-components-could be running could be doing sth * update cluster-components.md In theory,Master components can be run.... * update cluster-components--add a space add a space after the comma: "In theory, Master components..." * Highlighted command --kubectl describe In web page ,there is no space ,need some empty space,thanks! * Revert "Highlighted command --kubectl describe" This reverts commit a70d0a3e3537c3f91b197b23f949e1e506c951d1. * kubeadm reference--change any to some kubeadm can install any add-on ? --- docs/admin/kubeadm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/kubeadm.md b/docs/admin/kubeadm.md index 787b59bdbe..ce9a83fc29 100644 --- a/docs/admin/kubeadm.md +++ b/docs/admin/kubeadm.md @@ -34,7 +34,7 @@ resources to create on startup. These are the core components of Kubernetes, and once they are up and running we can use `kubectl` to set up or manage any additional components. -1. kubeadm installs any add-on components, such as DNS or discovery, via the API +1. kubeadm installs some add-on components, such as DNS or discovery, via the API server. Running `kubeadm join` on each node in the cluster consists of the following steps: From ec03295bf0d13461aa17030f1087a8089484fbde Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 11:21:01 -0800 Subject: [PATCH 327/407] Move Guide topics: Logging (#2687) --- _data/concepts.yml | 4 + _data/tasks.yml | 5 +- .../clusters}/counter-pod.yaml | 0 .../clusters}/fluentd-sidecar-config.yaml | 0 docs/concepts/clusters/logging.md | 223 ++++++++++++++++++ .../two-files-counter-pod-agent-sidecar.yaml | 0 ...o-files-counter-pod-streaming-sidecar.yaml | 0 .../clusters}/two-files-counter-pod.yaml | 0 .../counter-pod.yaml | 10 + .../debug-init-containers.md | 3 + .../logging-elasticsearch-kibana.md | 104 ++++++++ .../logging-stackdriver.md | 148 ++++++++++++ docs/user-guide/logging/elasticsearch.md | 97 +------- docs/user-guide/logging/overview.md | 217 +---------------- docs/user-guide/logging/stackdriver.md | 142 +---------- 15 files changed, 502 insertions(+), 451 deletions(-) rename docs/{user-guide/logging/examples => concepts/clusters}/counter-pod.yaml (100%) rename docs/{user-guide/logging/examples => concepts/clusters}/fluentd-sidecar-config.yaml (100%) create mode 100644 docs/concepts/clusters/logging.md rename docs/{user-guide/logging/examples => concepts/clusters}/two-files-counter-pod-agent-sidecar.yaml (100%) rename docs/{user-guide/logging/examples => concepts/clusters}/two-files-counter-pod-streaming-sidecar.yaml (100%) rename docs/{user-guide/logging/examples => concepts/clusters}/two-files-counter-pod.yaml (100%) create mode 100644 docs/tasks/debug-application-cluster/counter-pod.yaml rename docs/tasks/{troubleshoot => debug-application-cluster}/debug-init-containers.md (96%) create mode 100644 docs/tasks/debug-application-cluster/logging-elasticsearch-kibana.md create mode 100644 docs/tasks/debug-application-cluster/logging-stackdriver.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 440f9dbf2b..32af951748 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -33,6 +33,10 @@ toc: section: - docs/concepts/workloads/pods/pod-lifecycle.md +- title: Clusters + section: + - docs/concepts/clusters/logging.md + - title: Configuration section: - docs/concepts/configuration/container-command-args.md diff --git a/_data/tasks.yml b/_data/tasks.yml index ab237c3474..8863559c77 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -29,9 +29,12 @@ toc: - docs/tasks/access-application-cluster/port-forward-access-application-cluster.md - docs/tasks/access-application-cluster/load-balance-access-application-cluster.md -- title: Debugging Applications in a Cluster +- title: Monitoring, Logging, and Debugging section: - docs/tasks/debug-application-cluster/determine-reason-pod-failure.md + - docs/tasks/debug-application-cluster/debug-init-containers.md + - docs/tasks/debug-application-cluster/logging-stackdriver.md + - docs/tasks/debug-application-cluster/logging-elasticsearch-kibana.md - title: Accessing the Kubernetes API section: diff --git a/docs/user-guide/logging/examples/counter-pod.yaml b/docs/concepts/clusters/counter-pod.yaml similarity index 100% rename from docs/user-guide/logging/examples/counter-pod.yaml rename to docs/concepts/clusters/counter-pod.yaml diff --git a/docs/user-guide/logging/examples/fluentd-sidecar-config.yaml b/docs/concepts/clusters/fluentd-sidecar-config.yaml similarity index 100% rename from docs/user-guide/logging/examples/fluentd-sidecar-config.yaml rename to docs/concepts/clusters/fluentd-sidecar-config.yaml diff --git a/docs/concepts/clusters/logging.md b/docs/concepts/clusters/logging.md new file mode 100644 index 0000000000..465a2d324e --- /dev/null +++ b/docs/concepts/clusters/logging.md @@ -0,0 +1,223 @@ +--- +assignees: +- crassirostris +- piosz +title: Logging and Monitoring Cluster Activity +--- + +Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism; as such, most container engines are likewise designed to support some kind of logging. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams. + +However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution. For example, if a container crashes, a pod is evicted, or a node dies, you'll usually still want to access your application's logs. As such, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called _cluster-level-logging_. Cluster-level logging requires a separate backend to store, analyze, and query logs. Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster. + +This document includes: + +* A basic demonstration of logging in Kubernetes using the standard output stream +* A detailed description of the node logging architecture in Kubernetes +* Guidance for implementing cluster-level logging in Kubernetes + +The guidance for cluster-level logging assumes that a logging backend is present inside or outside of your cluster. If you're not interested in having cluster-level logging, you might still find the description of how logs are stored and handled on the node to be useful. + +## Basic logging in Kubernetes + +In this section, you can see an example of basic logging in Kubernetes that +outputs data to the standard output stream. This demonstration uses +a [pod specification](/docs/concepts/clusters/counter-pod.yaml) with +a container that writes some text to standard output once per second. + +{% include code.html language="yaml" file="counter-pod.yaml" ghlink="/docs/tasks/debug-application-cluster/counter-pod.yaml" %} + +To run this pod, use the following command: + +```shell +$ kubectl create -f http://k8s.io/docs/tasks/debug-application-cluster/counter-pod.yaml +pod "counter" created +``` + +To fetch the logs, use the `kubectl logs` command, as follows + +```shell +$ kubectl logs counter +0: Mon Jan 1 00:00:00 UTC 2001 +1: Mon Jan 1 00:00:01 UTC 2001 +2: Mon Jan 1 00:00:02 UTC 2001 +... +``` + +You can use `kubectl logs` to retrieve logs from a previous instantiation of a container with `--previous` flag, in case the container has crashed. If your pod has multiple containers, you should specify which container's logs you want to access by appending a container name to the command. See the [`kubectl logs` documentation](/docs/user-guide/kubectl/kubectl_logs/) for more details. + +## Logging at the node level + +![Node level logging](/images/docs/user-guide/logging/logging-node-level.png) + +Everything a containerized application writes to `stdout` and `stderr` is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to [a logging driver](https://docs.docker.com/engine/admin/logging/overview), which is configured in Kubernetes to write to a file in json format. + +**Note:** The Docker json logging driver treats each line as a separate message. When using the Docker logging driver, there is no direct support for multi-line messages. You need to handle multi-line messages at the logging agent level or higher. + +By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs. + +An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes uses the [`logrotate`](http://www.linuxcommand.org/man_pages/logrotate8.html) tool to implement log rotation. + +Kubernetes performs log rotation daily, or if the log file grows beyond 10MB in size. Each rotation belongs to a single container; if the container repeatedly fails or the pod is evicted, all previous rotations for the container are lost. By default, Kubernetes keeps up to five logging rotations per container. + +The Kubernetes logging configuration differs depending on the node type. For example, you can find detailed information for GCI in the corresponding [configure helper](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh#L96). + +When you run [`kubectl logs`](/docs/user-guide/kubectl/kubectl_logs), as in the basic logging example, the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response. Note that `kubectl logs` **only returns the last rotation**; you must manually extract prior rotations, if desired and cluster-level logging is not enabled. + +### System component logs + +There are two types of system components: those that run in a container and those +that do not run in a container. For example: + +* The Kubernetes scheduler and kube-proxy run in a container. +* The kubelet and container runtime, for example Docker, do not run in containers. + +On machines with systemd, the kubelet and container runtime write to journald. If +systemd is not present, they write to `.log` files in the `/var/log` directory. +System components inside containers always write to the `/var/log` directory, +bypassing the default logging mechanism. They use the [glog](https://godoc.org/github.com/golang/glog) +logging library. You can find the conventions for logging severity for those +components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md). + +Similarly to the container logs, system component logs in the `/var/log` +directory are rotated daily and based on the log size. However, +system component logs have a higher size retention: by default, +they can store up to 100MB. + +## Cluster-level logging architectures + +While Kubernetes does not provide a native solution for cluster-level logging, there are several common approaches you can consider. Here are some options: + +* Use a node-level logging agent that runs on every node. +* Include a dedicated sidecar container for logging in an application pod. +* Push logs directly to a backend from within an application. + +### Using a node logging agent + +![Using a node level logging agent](/images/docs/user-guide/logging/logging-with-node-agent.png) + +You can implement cluster-level logging by including a _node-level logging agent_ on each node. The logging agent is a dedicated tool that exposes logs or pushes logs to a backend. Commonly, the logging agent is a container that has access to a directory with log files from all of the application containers on that node. + +Because the logging agent must run on every node, it's common to implement it as either a DaemonSet replica, a manifest pod, or a dedicated native process on the node. However the latter two approaches are deprecated and highly discouraged. + +Using a node-level logging agent is the most common and encouraged approach for a Kubernetes cluster, because it creates only one agent per node, and it doesn't require any changes to the applications running on the node. However, node-level logging _only works for applications' standard output and standard error_. + +Kubernetes doesn't specify a logging agent, but two optional logging agents are packaged with the Kubernetes release: [Stackdriver Logging](/docs/user-guide/logging/stackdriver) for use with Google Cloud Platform, and [Elasticsearch](/docs/user-guide/logging/elasticsearch). You can find more information and instructions in the dedicated documents. Both use [fluentd](http://www.fluentd.org/) with custom configuration as an agent on the node. + +### Using a sidecar container with the logging agent + +You can use a sidecar container in one of the following ways: + +* The sidecar container streams application logs to its own `stdout`. +* The sidecar container runs a logging agent, which is configured to pick up logs from an application container. + +#### Streaming sidecar container + +![Sidecar container with a streaming container](/images/docs/user-guide/logging/logging-with-streaming-sidecar.png) + +By having your sidecar containers stream to their own `stdout` and `stderr` +streams, you can take advantage of the kubelet and the logging agent that +already run on each node. The sidecar containers read logs from a file, a socket, +or the journald. Each individual sidecar container prints log to its own `stdout` +or `stderr` stream. + +This approach allows you to separate several log streams from different +parts of your application, some of which can lack support +for writing to `stdout` or `stderr`. The logic behind redirecting logs +is minimal, so it's hardly a significant overhead. Additionally, because +`stdout` and `stderr` are handled by the kubelet, you can use built-in tools +like `kubectl logs`. + +Consider the following example. A pod runs a single container, and the container +writes to two different log files, using two different formats. Here's a +configuration file for the Pod: + +{% include code.html language="yaml" file="two-files-counter-pod.yaml" ghlink="/docs/concepts/clusters/two-files-counter-pod.yaml" %} + +It would be a mess to have log entries of different formats in the same log +stream, even if you managed to redirect both components to the `stdout` stream of +the container. Instead, you could introduce two sidecar containers. Each sidecar +container could tail a particular log file from a shared volume and then redirect +the logs to its own `stdout` stream. + +Here's a configuration file for a pod that has two sidecar containers: + +{% include code.html language="yaml" file="two-files-counter-pod-streaming-sidecar.yaml" ghlink="/docs/concepts/clusters/two-files-counter-pod-streaming-sidecar.yaml" %} + +Now when you run this pod, you can access each log stream separately by +running the following commands: + +```shell +$ kubectl logs counter count-log-1 +0: Mon Jan 1 00:00:00 UTC 2001 +1: Mon Jan 1 00:00:01 UTC 2001 +2: Mon Jan 1 00:00:02 UTC 2001 +... +``` + +```shell +$ kubectl logs counter count-log-2 +Mon Jan 1 00:00:00 UTC 2001 INFO 0 +Mon Jan 1 00:00:01 UTC 2001 INFO 1 +Mon Jan 1 00:00:02 UTC 2001 INFO 2 +... +``` + +The node-level agent installed in your cluster picks up those log streams +automatically without any further configuration. If you like, you can configure +the agent to parse log lines depending on the source container. + +Note, that despite low CPU and memory usage (order of couple of millicores +for cpu and order of several megabytes for memory), writing logs to a file and +then streaming them to `stdout` can double disk usage. If you have +an application that writes to a single file, it's generally better to set +`/dev/stdout` as destination rather than implementing the streaming sidecar +container approach. + +Sidecar containers can also be used to rotate log files that cannot be +rotated by the application itself. [An example](https://github.com/samsung-cnct/logrotate) +of this approach is a small container running logrotate periodically. +However, it's recommended to use `stdout` and `stderr` directly and leave rotation +and retention policies to the kubelet. + +#### Sidecar container with a logging agent + +![Sidecar container with a logging agent](/images/docs/user-guide/logging/logging-with-sidecar-agent.png) + +If the node-level logging agent is not flexible enough for your situation, you +can create a sidecar container with a separate logging agent that you have +configured specifically to run with your application. + +**Note**: Using a logging agent in a sidecar container can lead +to significant resource consumption. Moreover, you won't be able to access +those logs using `kubectl logs` command, because they are not controlled +by the kubelet. + +As an example, you could use [Stackdriver](/docs/user-guide/logging/stackdriver/), +which uses fluentd as a logging agent. Here are two configuration files that +you can use to implement this approach. The first file contains +a [ConfigMap](/docs/user-guide/configmap/) to configure fluentd. + +{% include code.html language="yaml" file="fluentd-sidecar-config.yaml" ghlink="/docs/concepts/clusters/fluentd-sidecar-config.yaml" %} + +**Note**: The configuration of fluentd is beyond the scope of this article. For +information about configuring fluentd, see the +[official fluentd documentation](http://docs.fluentd.org/). + +The second file describes a pod that has a sidecar container running fluentd. +The pod mounts a volume where fluentd can pick up its configuration data. + +{% include code.html language="yaml" file="two-files-counter-pod-agent-sidecar.yaml" ghlink="/docs/concepts/clusters/two-files-counter-pod-agent-sidecar.yaml" %} + +After some time you can find log messages in the Stackdriver interface. + +Remember, that this is just an example and you can actually replace fluentd +with any logging agent, reading from any source inside an application +container. + +### Exposing logs directly from the application + +![Exposing logs directly from the application](/images/docs/user-guide/logging/logging-from-application.png) + +You can implement cluster-level logging by exposing or pushing logs directly from +every application; however, the implementation for such a logging mechanism +is outside the scope of Kubernetes. diff --git a/docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml b/docs/concepts/clusters/two-files-counter-pod-agent-sidecar.yaml similarity index 100% rename from docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml rename to docs/concepts/clusters/two-files-counter-pod-agent-sidecar.yaml diff --git a/docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml b/docs/concepts/clusters/two-files-counter-pod-streaming-sidecar.yaml similarity index 100% rename from docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml rename to docs/concepts/clusters/two-files-counter-pod-streaming-sidecar.yaml diff --git a/docs/user-guide/logging/examples/two-files-counter-pod.yaml b/docs/concepts/clusters/two-files-counter-pod.yaml similarity index 100% rename from docs/user-guide/logging/examples/two-files-counter-pod.yaml rename to docs/concepts/clusters/two-files-counter-pod.yaml diff --git a/docs/tasks/debug-application-cluster/counter-pod.yaml b/docs/tasks/debug-application-cluster/counter-pod.yaml new file mode 100644 index 0000000000..f997886386 --- /dev/null +++ b/docs/tasks/debug-application-cluster/counter-pod.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Pod +metadata: + name: counter +spec: + containers: + - name: count + image: busybox + args: [/bin/sh, -c, + 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'] diff --git a/docs/tasks/troubleshoot/debug-init-containers.md b/docs/tasks/debug-application-cluster/debug-init-containers.md similarity index 96% rename from docs/tasks/troubleshoot/debug-init-containers.md rename to docs/tasks/debug-application-cluster/debug-init-containers.md index 3c362c5072..77abef0a84 100644 --- a/docs/tasks/troubleshoot/debug-init-containers.md +++ b/docs/tasks/debug-application-cluster/debug-init-containers.md @@ -8,6 +8,9 @@ assignees: - kow3ns - smarterclayton title: Debugging Init Containers +redirect_from: +- "/docs/tasks/troubleshoot/debug-init-containers/" +- "/docs/tasks/troubleshoot/debug-init-containers.html" --- {% capture overview %} diff --git a/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana.md b/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana.md new file mode 100644 index 0000000000..4441067d60 --- /dev/null +++ b/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana.md @@ -0,0 +1,104 @@ +--- +assignees: +- crassirostris +- piosz +title: Logging Using Elasticsearch and Kibana +--- + +On the Google Compute Engine (GCE) platform, the default logging support targets +[Stackdriver Logging](https://cloud.google.com/logging/), which is described in detail +in the [Logging With Stackdriver Logging](/docs/user-guide/logging/stackdriver). + +This article describes how to set up a cluster to ingest logs into +[Elasticsearch](https://www.elastic.co/products/elasticsearch), and view +them using [Kibana](https://www.elastic.co/products/kibana), as an alternative to +Stackdriver Logging when running on GCE. Note that Elasticsearch and Kibana do not work with Kubernetes clusters hosted on Google Container Engine. + +To use Elasticsearch and Kibana for cluster logging, you should set the +following environment variable as shown below when creating your cluster with +kube-up.sh: + +```shell +KUBE_LOGGING_DESTINATION=elasticsearch +``` + +You should also ensure that `KUBE_ENABLE_NODE_LOGGING=true` (which is the default for the GCE platform). + +Now, when you create a cluster, a message will indicate that the Fluentd log +collection daemons that run on each node will target Elasticsearch: + +```shell +$ cluster/kube-up.sh +... +Project: kubernetes-satnam +Zone: us-central1-b +... calling kube-up +Project: kubernetes-satnam +Zone: us-central1-b ++++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e81793/devel ++++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = 6987c098277871b6d69623141276924ab687f89d) ++++ kubernetes-salt.tar.gz uploaded (sha1 = bdfc83ed6b60fa9e3bff9004b542cfc643464cd0) +Looking for already existing resources +Starting master and configuring firewalls +Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/kubernetes-master-pd]. +NAME ZONE SIZE_GB TYPE STATUS +kubernetes-master-pd us-central1-b 20 pd-ssd READY +Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/kubernetes-master-ip]. ++++ Logging using Fluentd to elasticsearch +``` + +The per-node Fluentd pods, the Elasticsearch pods, and the Kibana pods should +all be running in the kube-system namespace soon after the cluster comes to +life. + +```shell +$ kubectl get pods --namespace=kube-system +NAME READY REASON RESTARTS AGE +elasticsearch-logging-v1-78nog 1/1 Running 0 2h +elasticsearch-logging-v1-nj2nb 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-node-5oq0 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-node-6896 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-node-l1ds 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-node-lz9j 1/1 Running 0 2h +kibana-logging-v1-bhpo8 1/1 Running 0 2h +kube-dns-v3-7r1l9 3/3 Running 0 2h +monitoring-heapster-v4-yl332 1/1 Running 1 2h +monitoring-influx-grafana-v1-o79xf 2/2 Running 0 2h +``` + +The `fluentd-elasticsearch` pods gather logs from each node and send them to +the `elasticsearch-logging` pods, which are part of a +[service](/docs/user-guide/services/) named `elasticsearch-logging`. These +Elasticsearch pods store the logs and expose them via a REST API. +The `kibana-logging` pod provides a web UI for reading the logs stored in +Elasticsearch, and is part of a service named `kibana-logging`. + +The Elasticsearch and Kibana services are both in the `kube-system` namespace +and are not directly exposed via a publicly reachable IP address. To reach them, +follow the instructions for [Accessing services running in a cluster](/docs/user-guide/accessing-the-cluster/#accessing-services-running-on-the-cluster). + +If you try accessing the `elasticsearch-logging` service in your browser, you'll +see a status page that looks something like this: + +![Elasticsearch Status](/images/docs/es-browser.png) + +You can now type Elasticsearch queries directly into the browser, if you'd +like. See [Elasticsearch's documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html) +for more details on how to do so. + +Alternatively, you can view your cluster's logs using Kibana (again using the +[instructions for accessing a service running in the cluster](/docs/user-guide/accessing-the-cluster/#accessing-services-running-on-the-cluster)). +The first time you visit the Kibana URL you will be presented with a page that +asks you to configure your view of the ingested logs. Select the option for +timeseries values and select `@timestamp`. On the following page select the +`Discover` tab and then you should be able to see the ingested logs. +You can set the refresh interval to 5 seconds to have the logs +regularly refreshed. + +Here is a typical view of ingested logs from the Kibana viewer: + +![Kibana logs](/images/docs/kibana-logs.png) + +Kibana opens up all sorts of powerful options for exploring your logs! For some +ideas on how to dig into it, check out [Kibana's documentation](https://www.elastic.co/guide/en/kibana/current/discover.html). + diff --git a/docs/tasks/debug-application-cluster/logging-stackdriver.md b/docs/tasks/debug-application-cluster/logging-stackdriver.md new file mode 100644 index 0000000000..b382d0faf7 --- /dev/null +++ b/docs/tasks/debug-application-cluster/logging-stackdriver.md @@ -0,0 +1,148 @@ +--- +assignees: +- crassirostris +- piosz +title: Logging Using Stackdriver +--- + +Before reading this page, it's highly recommended to familiasrize yourself with the [overview of logging in Kubernetes](/docs/user-guide/logging/overview). + +This article assumes that you have created a Kubernetes cluster with cluster-level logging support for sending logs to Stackdriver Logging. You can do this either by selecting the **Enable Stackdriver Logging** checkbox in the create cluster dialogue in [GKE](https://cloud.google.com/container-engine/), or by setting the `KUBE_LOGGING_DESTINATION` flag to `gcp` when manually starting a cluster using `kube-up.sh`. + +The following guide describes gathering a container's standard output and standard error. To gather logs written by an application to a file, you can use [a sidecar approach](https://github.com/kubernetes/contrib/blob/master/logging/fluentd-sidecar-gcp/README.md). + +## Overview + +After creation, you can discover logging agent pods in the `kube-system` namespace, +one per node, by running the following command: + +```shell +$ kubectl get pods --namespace=kube-system +NAME READY STATUS RESTARTS AGE +... +fluentd-gcp-v1.30-50gnc 1/1 Running 0 5d +fluentd-gcp-v1.30-v255c 1/1 Running 0 5d +fluentd-gcp-v1.30-f02l5 1/1 Running 0 5d +... +``` + +To understand how logging with Stackdriver works, consider the following +synthetic log generator pod specification [counter-pod.yaml](/docs/tasks/debug-application-cluster/counter-pod.yaml): + +{% include code.html language="yaml" file="counter-pod.yaml" ghlink="/docs/tasks/debug-application-cluster/counter-pod.yaml" %} + +This pod specification has one container that runs a bash script +that writes out the value of a counter and the date once per +second, and runs indefinitely. Let's create this pod in the default namespace. + +```shell +$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml +pod "counter" created +``` + +You can observe the running pod: + +```shell +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +counter 1/1 Running 0 5m +``` + +For a short period of time you can observe the 'Pending' pod status, because the kubelet +has to download the container image first. When the pod status changes to `Running` +you can use the `kubectl logs` command to view the output of this counter pod. + +```shell +$ kubectl logs counter +0: Mon Jan 1 00:00:00 UTC 2001 +1: Mon Jan 1 00:00:01 UTC 2001 +2: Mon Jan 1 00:00:02 UTC 2001 +... +``` + +As described in the logging overview, this command fetches log entries +from the container log file. If the container is killed and then restarted by +Kubernetes, you can still access logs from the previous container. However, +if the pod is evicted from the node, log files are lost. Let's demonstrate this +by deleting the currently running counter container: + +```shell +$ kubectl delete pod counter +pod "counter" deleted +``` + +and then recreating it: + +```shell +$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml +pod "counter" created +``` + +After some time, you can access logs from the counter pod again: + +```shell +$ kubectl logs counter +0: Mon Jan 1 00:01:00 UTC 2001 +1: Mon Jan 1 00:01:01 UTC 2001 +2: Mon Jan 1 00:01:02 UTC 2001 +... +``` + +As expected, only recent log lines are present. However, for a real-world +application you will likely want to be able to access logs from all containers, +especially for the debug purposes. This is exactly when the previously enabled +Stackdriver Logging can help. + +## Viewing logs + +Stackdriver Logging agent attaches metadata to each log entry, for you to use later +in queries to select only the messages you're interested in: for example, +the messages from a particular pod. + +The most important pieces of metadata are the resource type and log name. +The resource type of a container log is `container`, which is named +`GKE Containers` in the UI (even if the Kubernetes cluster is not on GKE). +The log name is the name of the container, so that if you have a pod with +two containers, named `container_1` and `container_2` in the spec, their logs +will have log names `container_1` and `container_2` respectively. + +System components have resource type `compute`, which is named +`GCE VM Instance` in the interface. Log names for system components are fixed. +For a GKE node, every log entry from a system component has one the following +log names: + +* docker +* kubelet +* kube-proxy + +You can learn more about viewing logs on [the dedicated Stackdriver page](https://cloud.google.com/logging/docs/view/logs_viewer). + +One of the possible ways to view logs is using the +[`gcloud logging`](https://cloud.google.com/logging/docs/api/gcloud-logging) +command line interface from the [Google Cloud SDK](https://cloud.google.com/sdk/). +It uses Stackdriver Logging [filtering syntax](https://cloud.google.com/logging/docs/view/advanced_filters) +to query specific logs. For example, you can run the following command: + +```shell +$ gcloud beta logging read 'logName="projects/$YOUR_PROJECT_ID/logs/count"' --format json | jq '.[].textPayload' +... +"2: Mon Jan 1 00:01:02 UTC 2001\n" +"1: Mon Jan 1 00:01:01 UTC 2001\n" +"0: Mon Jan 1 00:01:00 UTC 2001\n" +... +"2: Mon Jan 1 00:00:02 UTC 2001\n" +"1: Mon Jan 1 00:00:01 UTC 2001\n" +"0: Mon Jan 1 00:00:00 UTC 2001\n" +``` + +As you can see, it outputs messages for the count container from both +the first and second runs, despite the fact that the kubelet already deleted +the logs for the first container. + +### Exporting logs + +You can export logs to [Google Cloud Storage](https://cloud.google.com/storage/) +or to [BigQuery](https://cloud.google.com/bigquery/) to run further +analysis. Stackdriver Logging offers the concept of sinks, where you can +specify the destination of log entries. More information is available on +the Stackdriver [Exporting Logs page](https://cloud.google.com/logging/docs/export/configure_export_v2). diff --git a/docs/user-guide/logging/elasticsearch.md b/docs/user-guide/logging/elasticsearch.md index c61cfd7cd1..e55de7f30f 100644 --- a/docs/user-guide/logging/elasticsearch.md +++ b/docs/user-guide/logging/elasticsearch.md @@ -5,99 +5,6 @@ assignees: title: Logging with Elasticsearch and Kibana --- -On the Google Compute Engine (GCE) platform, the default logging support targets -[Stackdriver Logging](https://cloud.google.com/logging/), which is described in detail -in the [Logging With Stackdriver Logging](/docs/user-guide/logging/stackdriver). +{% include user-guide-content-moved.md %} -This article describes how to set up a cluster to ingest logs into -[Elasticsearch](https://www.elastic.co/products/elasticsearch), and view -them using [Kibana](https://www.elastic.co/products/kibana), as an alternative to -Stackdriver Logging when running on GCE. Note that Elasticsearch and Kibana do not work with Kubernetes clusters hosted on Google Container Engine. - -To use Elasticsearch and Kibana for cluster logging, you should set the -following environment variable as shown below when creating your cluster with -kube-up.sh: - -```shell -KUBE_LOGGING_DESTINATION=elasticsearch -``` - -You should also ensure that `KUBE_ENABLE_NODE_LOGGING=true` (which is the default for the GCE platform). - -Now, when you create a cluster, a message will indicate that the Fluentd log -collection daemons that run on each node will target Elasticsearch: - -```shell -$ cluster/kube-up.sh -... -Project: kubernetes-satnam -Zone: us-central1-b -... calling kube-up -Project: kubernetes-satnam -Zone: us-central1-b -+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e81793/devel -+++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = 6987c098277871b6d69623141276924ab687f89d) -+++ kubernetes-salt.tar.gz uploaded (sha1 = bdfc83ed6b60fa9e3bff9004b542cfc643464cd0) -Looking for already existing resources -Starting master and configuring firewalls -Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/kubernetes-master-pd]. -NAME ZONE SIZE_GB TYPE STATUS -kubernetes-master-pd us-central1-b 20 pd-ssd READY -Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/kubernetes-master-ip]. -+++ Logging using Fluentd to elasticsearch -``` - -The per-node Fluentd pods, the Elasticsearch pods, and the Kibana pods should -all be running in the kube-system namespace soon after the cluster comes to -life. - -```shell -$ kubectl get pods --namespace=kube-system -NAME READY REASON RESTARTS AGE -elasticsearch-logging-v1-78nog 1/1 Running 0 2h -elasticsearch-logging-v1-nj2nb 1/1 Running 0 2h -fluentd-elasticsearch-kubernetes-node-5oq0 1/1 Running 0 2h -fluentd-elasticsearch-kubernetes-node-6896 1/1 Running 0 2h -fluentd-elasticsearch-kubernetes-node-l1ds 1/1 Running 0 2h -fluentd-elasticsearch-kubernetes-node-lz9j 1/1 Running 0 2h -kibana-logging-v1-bhpo8 1/1 Running 0 2h -kube-dns-v3-7r1l9 3/3 Running 0 2h -monitoring-heapster-v4-yl332 1/1 Running 1 2h -monitoring-influx-grafana-v1-o79xf 2/2 Running 0 2h -``` - -The `fluentd-elasticsearch` pods gather logs from each node and send them to -the `elasticsearch-logging` pods, which are part of a -[service](/docs/user-guide/services/) named `elasticsearch-logging`. These -Elasticsearch pods store the logs and expose them via a REST API. -The `kibana-logging` pod provides a web UI for reading the logs stored in -Elasticsearch, and is part of a service named `kibana-logging`. - -The Elasticsearch and Kibana services are both in the `kube-system` namespace -and are not directly exposed via a publicly reachable IP address. To reach them, -follow the instructions for [Accessing services running in a cluster](/docs/user-guide/accessing-the-cluster/#accessing-services-running-on-the-cluster). - -If you try accessing the `elasticsearch-logging` service in your browser, you'll -see a status page that looks something like this: - -![Elasticsearch Status](/images/docs/es-browser.png) - -You can now type Elasticsearch queries directly into the browser, if you'd -like. See [Elasticsearch's documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html) -for more details on how to do so. - -Alternatively, you can view your cluster's logs using Kibana (again using the -[instructions for accessing a service running in the cluster](/docs/user-guide/accessing-the-cluster/#accessing-services-running-on-the-cluster)). -The first time you visit the Kibana URL you will be presented with a page that -asks you to configure your view of the ingested logs. Select the option for -timeseries values and select `@timestamp`. On the following page select the -`Discover` tab and then you should be able to see the ingested logs. -You can set the refresh interval to 5 seconds to have the logs -regularly refreshed. - -Here is a typical view of ingested logs from the Kibana viewer: - -![Kibana logs](/images/docs/kibana-logs.png) - -Kibana opens up all sorts of powerful options for exploring your logs! For some -ideas on how to dig into it, check out [Kibana's documentation](https://www.elastic.co/guide/en/kibana/current/discover.html). +[Logging Using ElasticSearch and Kibana](/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) diff --git a/docs/user-guide/logging/overview.md b/docs/user-guide/logging/overview.md index 42b9087172..e5d2e24ae4 100644 --- a/docs/user-guide/logging/overview.md +++ b/docs/user-guide/logging/overview.md @@ -5,219 +5,6 @@ assignees: title: Logging Overview --- -Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism; as such, most container engines are likewise designed to support some kind of logging. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams. +{% include user-guide-content-moved.md %} -However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution. For example, if a container crashes, a pod is evicted, or a node dies, you'll usually still want to access your application's logs. As such, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called _cluster-level-logging_. Cluster-level logging requires a separate backend to store, analyze, and query logs. Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster. - -This document includes: - -* A basic demonstration of logging in Kubernetes using the standard output stream -* A detailed description of the node logging architecture in Kubernetes -* Guidance for implementing cluster-level logging in Kubernetes - -The guidance for cluster-level logging assumes that a logging backend is present inside or outside of your cluster. If you're not interested in having cluster-level logging, you might still find the description of how logs are stored and handled on the node to be useful. - -## Basic logging in Kubernetes - -In this section, you can see an example of basic logging in Kubernetes that -outputs data to the standard output stream. This demonstration uses -a [pod specification](/docs/user-guide/logging/examples/counter-pod.yaml) with -a container that writes some text to standard output once per second. - -{% include code.html language="yaml" file="examples/counter-pod.yaml" ghlink="/docs/user-guide/logging/examples/counter-pod.yaml" %} - -To run this pod, use the following command: - -```shell -$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml -pod "counter" created -``` - -To fetch the logs, use the `kubectl logs` command, as follows - -```shell -$ kubectl logs counter -0: Mon Jan 1 00:00:00 UTC 2001 -1: Mon Jan 1 00:00:01 UTC 2001 -2: Mon Jan 1 00:00:02 UTC 2001 -... -``` - -You can use `kubectl logs` to retrieve logs from a previous instantiation of a container with `--previous` flag, in case the container has crashed. If your pod has multiple containers, you should specify which container's logs you want to access by appending a container name to the command. See the [`kubectl logs` documentation](/docs/user-guide/kubectl/kubectl_logs/) for more details. - -## Logging at the node level - -![Node level logging](/images/docs/user-guide/logging/logging-node-level.png) - -Everything a containerized application writes to `stdout` and `stderr` is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to [a logging driver](https://docs.docker.com/engine/admin/logging/overview), which is configured in Kubernetes to write to a file in json format. - -**Note:** The Docker json logging driver treats each line as a separate message. When using the Docker logging driver, there is no direct support for multi-line messages. You need to handle multi-line messages at the logging agent level or higher. - -By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs. - -An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes uses the [`logrotate`](http://www.linuxcommand.org/man_pages/logrotate8.html) tool to implement log rotation. - -Kubernetes performs log rotation daily, or if the log file grows beyond 10MB in size. Each rotation belongs to a single container; if the container repeatedly fails or the pod is evicted, all previous rotations for the container are lost. By default, Kubernetes keeps up to five logging rotations per container. - -The Kubernetes logging configuration differs depending on the node type. For example, you can find detailed information for GCI in the corresponding [configure helper](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh#L96). - -When you run [`kubectl logs`](/docs/user-guide/kubectl/kubectl_logs), as in the basic logging example, the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response. Note that `kubectl logs` **only returns the last rotation**; you must manually extract prior rotations, if desired and cluster-level logging is not enabled. - -### System component logs - -There are two types of system components: those that run in a container and those -that do not run in a container. For example: - -* The Kubernetes scheduler and kube-proxy run in a container. -* The kubelet and container runtime, for example Docker, do not run in containers. - -On machines with systemd, the kubelet and container runtime write to journald. If -systemd is not present, they write to `.log` files in the `/var/log` directory. -System components inside containers always write to the `/var/log` directory, -bypassing the default logging mechanism. They use the [glog](https://godoc.org/github.com/golang/glog) -logging library. You can find the conventions for logging severity for those -components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md). - -Similarly to the container logs, system component logs in the `/var/log` -directory are rotated daily and based on the log size. However, -system component logs have a higher size retention: by default, -they can store up to 100MB. - -## Cluster-level logging architectures - -While Kubernetes does not provide a native solution for cluster-level logging, there are several common approaches you can consider. Here are some options: - -* Use a node-level logging agent that runs on every node. -* Include a dedicated sidecar container for logging in an application pod. -* Push logs directly to a backend from within an application. - -### Using a node logging agent - -![Using a node level logging agent](/images/docs/user-guide/logging/logging-with-node-agent.png) - -You can implement cluster-level logging by including a _node-level logging agent_ on each node. The logging agent is a dedicated tool that exposes logs or pushes logs to a backend. Commonly, the logging agent is a container that has access to a directory with log files from all of the application containers on that node. - -Because the logging agent must run on every node, it's common to implement it as either a DaemonSet replica, a manifest pod, or a dedicated native process on the node. However the latter two approaches are deprecated and highly discouraged. - -Using a node-level logging agent is the most common and encouraged approach for a Kubernetes cluster, because it creates only one agent per node, and it doesn't require any changes to the applications running on the node. However, node-level logging _only works for applications' standard output and standard error_. - -Kubernetes doesn't specify a logging agent, but two optional logging agents are packaged with the Kubernetes release: [Stackdriver Logging](/docs/user-guide/logging/stackdriver) for use with Google Cloud Platform, and [Elasticsearch](/docs/user-guide/logging/elasticsearch). You can find more information and instructions in the dedicated documents. Both use [fluentd](http://www.fluentd.org/) with custom configuration as an agent on the node. - -### Using a sidecar container with the logging agent - -You can use a sidecar container in one of the following ways: - -* The sidecar container streams application logs to its own `stdout`. -* The sidecar container runs a logging agent, which is configured to pick up logs from an application container. - -#### Streaming sidecar container - -![Sidecar container with a streaming container](/images/docs/user-guide/logging/logging-with-streaming-sidecar.png) - -By having your sidecar containers stream to their own `stdout` and `stderr` -streams, you can take advantage of the kubelet and the logging agent that -already run on each node. The sidecar containers read logs from a file, a socket, -or the journald. Each individual sidecar container prints log to its own `stdout` -or `stderr` stream. - -This approach allows you to separate several log streams from different -parts of your application, some of which can lack support -for writing to `stdout` or `stderr`. The logic behind redirecting logs -is minimal, so it's hardly a significant overhead. Additionally, because -`stdout` and `stderr` are handled by the kubelet, you can use built-in tools -like `kubectl logs`. - -Consider the following example. A pod runs a single container, and the container -writes to two different log files, using two different formats. Here's a -configuration file for the Pod: - -{% include code.html language="yaml" file="examples/two-files-counter-pod.yaml" ghlink="/docs/user-guide/logging/examples/two-files-counter-pod.yaml" %} - -It would be a mess to have log entries of different formats in the same log -stream, even if you managed to redirect both components to the `stdout` stream of -the container. Instead, you could introduce two sidecar containers. Each sidecar -container could tail a particular log file from a shared volume and then redirect -the logs to its own `stdout` stream. - -Here's a configuration file for a pod that has two sidecar containers: - -{% include code.html language="yaml" file="examples/two-files-counter-pod-streaming-sidecar.yaml" ghlink="/docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml" %} - -Now when you run this pod, you can access each log stream separately by -running the following commands: - -```shell -$ kubectl logs counter count-log-1 -0: Mon Jan 1 00:00:00 UTC 2001 -1: Mon Jan 1 00:00:01 UTC 2001 -2: Mon Jan 1 00:00:02 UTC 2001 -... -``` - -```shell -$ kubectl logs counter count-log-2 -Mon Jan 1 00:00:00 UTC 2001 INFO 0 -Mon Jan 1 00:00:01 UTC 2001 INFO 1 -Mon Jan 1 00:00:02 UTC 2001 INFO 2 -... -``` - -The node-level agent installed in your cluster picks up those log streams -automatically without any further configuration. If you like, you can configure -the agent to parse log lines depending on the source container. - -Note, that despite low CPU and memory usage (order of couple of millicores -for cpu and order of several megabytes for memory), writing logs to a file and -then streaming them to `stdout` can double disk usage. If you have -an application that writes to a single file, it's generally better to set -`/dev/stdout` as destination rather than implementing the streaming sidecar -container approach. - -Sidecar containers can also be used to rotate log files that cannot be -rotated by the application itself. [An example](https://github.com/samsung-cnct/logrotate) -of this approach is a small container running logrotate periodically. -However, it's recommended to use `stdout` and `stderr` directly and leave rotation -and retention policies to the kubelet. - -#### Sidecar container with a logging agent - -![Sidecar container with a logging agent](/images/docs/user-guide/logging/logging-with-sidecar-agent.png) - -If the node-level logging agent is not flexible enough for your situation, you -can create a sidecar container with a separate logging agent that you have -configured specifically to run with your application. - -**Note**: Using a logging agent in a sidecar container can lead -to significant resource consumption. Moreover, you won't be able to access -those logs using `kubectl logs` command, because they are not controlled -by the kubelet. - -As an example, you could use [Stackdriver](/docs/user-guide/logging/stackdriver/), -which uses fluentd as a logging agent. Here are two configuration files that -you can use to implement this approach. The first file contains -a [ConfigMap](/docs/user-guide/configmap/) to configure fluentd. - -{% include code.html language="yaml" file="examples/fluentd-sidecar-config.yaml" ghlink="/docs/user-guide/logging/examples/fluentd-sidecar-config.yaml" %} - -**Note**: The configuration of fluentd is beyond the scope of this article. For -information about configuring fluentd, see the -[official fluentd documentation](http://docs.fluentd.org/). - -The second file describes a pod that has a sidecar container running fluentd. -The pod mounts a volume where fluentd can pick up its configuration data. - -{% include code.html language="yaml" file="examples/two-files-counter-pod-agent-sidecar.yaml" ghlink="/docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml" %} - -After some time you can find log messages in the Stackdriver interface. - -Remember, that this is just an example and you can actually replace fluentd -with any logging agent, reading from any source inside an application -container. - -### Exposing logs directly from the application - -![Exposing logs directly from the application](/images/docs/user-guide/logging/logging-from-application.png) - -You can implement cluster-level logging by exposing or pushing logs directly from -every application; however, the implementation for such a logging mechanism -is outside the scope of Kubernetes. +[Logging and Monitoring Cluster Activity](/docs/concepts/clusters/logging/) diff --git a/docs/user-guide/logging/stackdriver.md b/docs/user-guide/logging/stackdriver.md index b71947ee1f..5664357ee5 100644 --- a/docs/user-guide/logging/stackdriver.md +++ b/docs/user-guide/logging/stackdriver.md @@ -5,144 +5,6 @@ assignees: title: Logging with Stackdriver Logging --- -Before reading this page, it's highly recommended to familiarize yourself with the [overview of logging in Kubernetes](/docs/user-guide/logging/overview). +{% include user-guide-content-moved.md %} -This article assumes that you have created a Kubernetes cluster with cluster-level logging support for sending logs to Stackdriver Logging. You can do this either by selecting the **Enable Stackdriver Logging** checkbox in the create cluster dialogue in [GKE](https://cloud.google.com/container-engine/), or by setting the `KUBE_LOGGING_DESTINATION` flag to `gcp` when manually starting a cluster using `kube-up.sh`. - -The following guide describes gathering a container's standard output and standard error. To gather logs written by an application to a file, you can use [a sidecar approach](https://github.com/kubernetes/contrib/blob/master/logging/fluentd-sidecar-gcp/README.md). - -## Overview - -After creation, you can discover logging agent pods in the `kube-system` namespace, -one per node, by running the following command: - -```shell -$ kubectl get pods --namespace=kube-system -NAME READY STATUS RESTARTS AGE -... -fluentd-gcp-v1.30-50gnc 1/1 Running 0 5d -fluentd-gcp-v1.30-v255c 1/1 Running 0 5d -fluentd-gcp-v1.30-f02l5 1/1 Running 0 5d -... -``` - -To understand how logging with Stackdriver works, consider the following -synthetic log generator pod specification [counter-pod.yaml](/docs/user-guide/logging/examples/counter-pod.yaml): - -{% include code.html language="yaml" file="examples/counter-pod.yaml" ghlink="/docs/user-guide/logging/examples/counter-pod.yaml" %} - -This pod specification has one container that runs a bash script -that writes out the value of a counter and the date once per -second, and runs indefinitely. Let's create this pod in the default namespace. - -```shell -$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml -pod "counter" created -``` - -You can observe the running pod: - -```shell -$ kubectl get pods -NAME READY STATUS RESTARTS AGE -counter 1/1 Running 0 5m -``` - -For a short period of time you can observe the 'Pending' pod status, because the kubelet -has to download the container image first. When the pod status changes to `Running` -you can use the `kubectl logs` command to view the output of this counter pod. - -```shell -$ kubectl logs counter -0: Mon Jan 1 00:00:00 UTC 2001 -1: Mon Jan 1 00:00:01 UTC 2001 -2: Mon Jan 1 00:00:02 UTC 2001 -... -``` - -As described in the logging overview, this command fetches log entries -from the container log file. If the container is killed and then restarted by -Kubernetes, you can still access logs from the previous container. However, -if the pod is evicted from the node, log files are lost. Let's demonstrate this -by deleting the currently running counter container: - -```shell -$ kubectl delete pod counter -pod "counter" deleted -``` - -and then recreating it: - -```shell -$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml -pod "counter" created -``` - -After some time, you can access logs from the counter pod again: - -```shell -$ kubectl logs counter -0: Mon Jan 1 00:01:00 UTC 2001 -1: Mon Jan 1 00:01:01 UTC 2001 -2: Mon Jan 1 00:01:02 UTC 2001 -... -``` - -As expected, only recent log lines are present. However, for a real-world -application you will likely want to be able to access logs from all containers, -especially for the debug purposes. This is exactly when the previously enabled -Stackdriver Logging can help. - -## Viewing logs - -Stackdriver Logging agent attaches metadata to each log entry, for you to use later -in queries to select only the messages you're interested in: for example, -the messages from a particular pod. - -The most important pieces of metadata are the resource type and log name. -The resource type of a container log is `container`, which is named -`GKE Containers` in the UI (even if the Kubernetes cluster is not on GKE). -The log name is the name of the container, so that if you have a pod with -two containers, named `container_1` and `container_2` in the spec, their logs -will have log names `container_1` and `container_2` respectively. - -System components have resource type `compute`, which is named -`GCE VM Instance` in the interface. Log names for system components are fixed. -For a GKE node, every log entry from a system component has one the following -log names: - -* docker -* kubelet -* kube-proxy - -You can learn more about viewing logs on [the dedicated Stackdriver page](https://cloud.google.com/logging/docs/view/logs_viewer). - -One of the possible ways to view logs is using the -[`gcloud logging`](https://cloud.google.com/logging/docs/api/gcloud-logging) -command line interface from the [Google Cloud SDK](https://cloud.google.com/sdk/). -It uses Stackdriver Logging [filtering syntax](https://cloud.google.com/logging/docs/view/advanced_filters) -to query specific logs. For example, you can run the following command: - -```shell -$ gcloud beta logging read 'logName="projects/$YOUR_PROJECT_ID/logs/count"' --format json | jq '.[].textPayload' -... -"2: Mon Jan 1 00:01:02 UTC 2001\n" -"1: Mon Jan 1 00:01:01 UTC 2001\n" -"0: Mon Jan 1 00:01:00 UTC 2001\n" -... -"2: Mon Jan 1 00:00:02 UTC 2001\n" -"1: Mon Jan 1 00:00:01 UTC 2001\n" -"0: Mon Jan 1 00:00:00 UTC 2001\n" -``` - -As you can see, it outputs messages for the count container from both -the first and second runs, despite the fact that the kubelet already deleted -the logs for the first container. - -### Exporting logs - -You can export logs to [Google Cloud Storage](https://cloud.google.com/storage/) -or to [BigQuery](https://cloud.google.com/bigquery/) to run further -analysis. Stackdriver Logging offers the concept of sinks, where you can -specify the destination of log entries. More information is available on -the Stackdriver [Exporting Logs page](https://cloud.google.com/logging/docs/export/configure_export_v2). +[Logging Using Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/) From adee25b76890b5980b51610c7fffc88649af1c67 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 12:10:47 -0800 Subject: [PATCH 328/407] Disallow indexing for liveness/index (#2689) --- robots.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/robots.txt b/robots.txt index 10fff656d2..313b796af8 100644 --- a/robots.txt +++ b/robots.txt @@ -5,6 +5,7 @@ Disallow: /v1.0/ Disallow: /v1.1/ Disallow: /404/ Disallow: 404.html -Disallow: /docs/user-guide/docs/user-guide/simple-nginx/ +Disallow: /docs/user-guide/simple-nginx +Disallow: /docs/user-guide/liveness/index SITEMAP: http://kubernetes.io/sitemap.xml From 4d7aecf5de8b1c5e76515899a4292335f42f298b Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 13:32:41 -0800 Subject: [PATCH 329/407] Deprecate Guide topics. (#2691) --- _data/guides.yml | 4 ---- .../configuration/container-command-args.md | 3 +++ docs/tasks/index.md | 5 ++++ .../run-stateless-application-deployment.md | 2 ++ docs/user-guide/configuring-containers.md | 7 ------ docs/user-guide/containers.md | 10 -------- docs/user-guide/pods/single-container.md | 7 ------ docs/user-guide/production-pods.md | 23 ------------------- robots.txt | 8 ++++++- 9 files changed, 17 insertions(+), 52 deletions(-) delete mode 100644 docs/user-guide/configuring-containers.md delete mode 100644 docs/user-guide/containers.md delete mode 100644 docs/user-guide/pods/single-container.md delete mode 100644 docs/user-guide/production-pods.md diff --git a/_data/guides.yml b/_data/guides.yml index df3ebad90a..87ca562c6f 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -56,13 +56,9 @@ toc: - title: Containers and Pods section: - - docs/user-guide/pods/single-container.md - docs/user-guide/pods/multi-container.md - docs/user-guide/pods/init-container.md - - docs/user-guide/configuring-containers.md - docs/user-guide/pod-templates.md - - docs/user-guide/production-pods.md - - docs/user-guide/containers.md - docs/user-guide/environment-guide/index.md - docs/user-guide/compute-resources.md - docs/user-guide/pod-states.md diff --git a/docs/concepts/configuration/container-command-args.md b/docs/concepts/configuration/container-command-args.md index 3f39baff57..c2dafaadfa 100644 --- a/docs/concepts/configuration/container-command-args.md +++ b/docs/concepts/configuration/container-command-args.md @@ -1,5 +1,8 @@ --- title: Container Command and Arguments +redirect_from: +- "/docs/user-guide/containers/" +- "/docs/user-guide/containers.html" --- {% capture overview %} diff --git a/docs/tasks/index.md b/docs/tasks/index.md index 6bd8db2e6c..1b532b9232 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -1,5 +1,10 @@ --- title: Tasks +redirect_from: +- "/docs/user-guide/configuring-containers/" +- "/docs/user-guide/configuring-containers.html" +- "/docs/user-guide/production-pods/" +- "/docs/user-guide/production-pods.html" --- This section of the Kubernetes documentation contains pages that diff --git a/docs/tutorials/stateless-application/run-stateless-application-deployment.md b/docs/tutorials/stateless-application/run-stateless-application-deployment.md index 2b9793bfa4..c81bcdae18 100644 --- a/docs/tutorials/stateless-application/run-stateless-application-deployment.md +++ b/docs/tutorials/stateless-application/run-stateless-application-deployment.md @@ -3,6 +3,8 @@ title: Running a Stateless Application Using a Deployment redirect_from: - "/docs/user-guide/simple-nginx/" - "/docs/user-guide/simple-nginx.html" +- "/docs/user-guide/pods/single-container/" +- "/docs/user-guide/pods/single-container.html" --- {% capture overview %} diff --git a/docs/user-guide/configuring-containers.md b/docs/user-guide/configuring-containers.md deleted file mode 100644 index 8e09d7aaa4..0000000000 --- a/docs/user-guide/configuring-containers.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Configuring Containers ---- - -{% include user-guide-content-moved.md %} - -[Tasks](/docs/tasks/) diff --git a/docs/user-guide/containers.md b/docs/user-guide/containers.md deleted file mode 100644 index dfe48ade49..0000000000 --- a/docs/user-guide/containers.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -assignees: -- mikedanese -title: Commands and Capabilities ---- - -{% include user-guide-content-moved.md %} - -* [Container Command and Arguments](/docs/concepts/configuration/container-command-args/) -* [Container Capabilities](/docs/concepts/policy/container-capabilities/) diff --git a/docs/user-guide/pods/single-container.md b/docs/user-guide/pods/single-container.md deleted file mode 100644 index fd7a44ba66..0000000000 --- a/docs/user-guide/pods/single-container.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Creating Single-Container Pods ---- - -{% include user-guide-content-moved.md %} - -[Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) diff --git a/docs/user-guide/production-pods.md b/docs/user-guide/production-pods.md deleted file mode 100644 index 5ea3a0edd5..0000000000 --- a/docs/user-guide/production-pods.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: Working with Containers in Production ---- - -{% include user-guide-content-moved.md %} - -* [Configuring a Pod to Use a Volume for Storage](/docs/tasks/configure-pod-container/configure-volume-storage/) - -* [Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) - -* [Pulling an Image from a Private Registry](/docs/tasks/configure-pod-container/pull-image-private-registry/) - -* [Communicating Between Containers Running in the Same Pod](/docs/tasks/configure-pod-container/communicate-containers-same-pod/) - -* [Assigning CPU and RAM Resources to a Container](/docs/tasks/configure-pod-container/assign-cpu-ram-container/) - -* [Configuring Liveness and Readiness Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) - -* [Configuring Pod Initialization](/docs/tasks/configure-pod-container/configure-pod-initialization/) - -* [Attaching Handlers to Container Lifecycle Events](/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/) - -* [Determining the Reason for Pod Failure](/docs/tasks/debug-application-cluster/determine-reason-pod-failure/) diff --git a/robots.txt b/robots.txt index 313b796af8..34417b3527 100644 --- a/robots.txt +++ b/robots.txt @@ -5,7 +5,13 @@ Disallow: /v1.0/ Disallow: /v1.1/ Disallow: /404/ Disallow: 404.html -Disallow: /docs/user-guide/simple-nginx + +Disallow: /docs/user-guide/configuring-containers +Disallow: /docs/user-guide/containers Disallow: /docs/user-guide/liveness/index +Disallow: /docs/user-guide/simple-nginx +Disallow: /docs/user-guide/production-pods + +Disallow: /docs/user-guide/pods/single-container SITEMAP: http://kubernetes.io/sitemap.xml From 50471b3395fa74452634a0d832f05bf2df93340f Mon Sep 17 00:00:00 2001 From: Eduardo Baitello Date: Fri, 3 Mar 2017 16:21:22 -0300 Subject: [PATCH 330/407] Wrong label and kubectl get completed pods The label `app=jobexample` should be `jobgroup=jobexample`. Also, for get completed pods the flag `--show-all` is necessary. --- docs/user-guide/jobs/expansions/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user-guide/jobs/expansions/index.md b/docs/user-guide/jobs/expansions/index.md index f401574519..8d5cb87bd4 100644 --- a/docs/user-guide/jobs/expansions/index.md +++ b/docs/user-guide/jobs/expansions/index.md @@ -71,7 +71,7 @@ job "process-item-cherry" created Now, check on the jobs: ```shell -$ kubectl get jobs -l app=jobexample +$ kubectl get jobs -l jobgroup=jobexample JOB CONTAINER(S) IMAGE(S) SELECTOR SUCCESSFUL process-item-apple c busybox app in (jobexample),item in (apple) 1 process-item-banana c busybox app in (jobexample),item in (banana) 1 @@ -85,7 +85,7 @@ do not care to see.) We can check on the pods as well using the same label selector: ```shell -$ kubectl get pods -l app=jobexample +$ kubectl get pods -l jobgroup=jobexample --show-all NAME READY STATUS RESTARTS AGE process-item-apple-kixwv 0/1 Completed 0 4m process-item-banana-wrsf7 0/1 Completed 0 4m @@ -96,7 +96,7 @@ There is not a single command to check on the output of all jobs at once, but looping over all the pods is pretty easy: ```shell -$ for p in $(kubectl get pods -l app=jobexample -o name) +$ for p in $(kubectl get pods -l jobgroup=jobexample -o name) do kubectl logs $p done From c235b99649b94cec2bc15dc01d063515022efd4e Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Fri, 3 Mar 2017 17:39:53 +0800 Subject: [PATCH 331/407] Update garbage-collection.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit change “any” to "every" --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index f34b7e3156..6ee78b5079 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -23,7 +23,7 @@ threshold has been met. ### Container Collection -The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers any single +The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers every single pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting `MinAge` to zero and setting `MaxPerPodContainer` and `MaxContainers` respectively to less than zero. Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. `MaxPerPodContainer` and `MaxContainer` may potentially conflict with each other in situations where retaining the maximum number of containers per pod (`MaxPerPodContainer`) would go outside the allowable range of global dead containers (`MaxContainers`). `MaxPerPodContainer` would be adjusted in this situation: A worst case scenario would be to downgrade `MaxPerPodContainer` to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. From a238ed8ced30d24e48fe656a742fb383df1ff51b Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 14:02:35 -0800 Subject: [PATCH 332/407] Deprecate Guide topic: Persistent Volume Walkthrough. (#2692) --- _data/guides.yml | 1 - .../configure-persistent-volume-storage.md | 3 +++ docs/user-guide/persistent-volumes/walkthrough.md | 10 ---------- robots.txt | 1 + 4 files changed, 4 insertions(+), 11 deletions(-) delete mode 100644 docs/user-guide/persistent-volumes/walkthrough.md diff --git a/_data/guides.yml b/_data/guides.yml index 87ca562c6f..318f965319 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -67,7 +67,6 @@ toc: - docs/user-guide/node-selection/index.md - docs/user-guide/downward-api/index.md - docs/user-guide/downward-api/volume/index.md - - docs/user-guide/persistent-volumes/walkthrough.md - docs/user-guide/petset/bootstrapping/index.md - title: Monitoring, Logging, and Debugging Containers diff --git a/docs/tasks/configure-pod-container/configure-persistent-volume-storage.md b/docs/tasks/configure-pod-container/configure-persistent-volume-storage.md index 27b5249138..f5be254f2e 100644 --- a/docs/tasks/configure-pod-container/configure-persistent-volume-storage.md +++ b/docs/tasks/configure-pod-container/configure-persistent-volume-storage.md @@ -1,5 +1,8 @@ --- title: Configuring a Pod to Use a PersistentVolume for Storage +redirect_from: +- "/docs/user-guide/persistent-volumes/walkthrough/" +- "/docs/user-guide/persistent-volumes/walkthrough.html" --- {% capture overview %} diff --git a/docs/user-guide/persistent-volumes/walkthrough.md b/docs/user-guide/persistent-volumes/walkthrough.md deleted file mode 100644 index 2feee3f7d9..0000000000 --- a/docs/user-guide/persistent-volumes/walkthrough.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -assignees: -- jsafrane -- saad-ali -title: Persistent Volumes Walkthrough ---- - -{% include user-guide-content-moved.md %} - -[Configuring a Pod to Use a Persistent Volume for Storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/) diff --git a/robots.txt b/robots.txt index 34417b3527..8e77127e0b 100644 --- a/robots.txt +++ b/robots.txt @@ -12,6 +12,7 @@ Disallow: /docs/user-guide/liveness/index Disallow: /docs/user-guide/simple-nginx Disallow: /docs/user-guide/production-pods +Disallow: /docs/user-guide/persistent-volumes/walkthrough Disallow: /docs/user-guide/pods/single-container SITEMAP: http://kubernetes.io/sitemap.xml From 1407f52478aab79b36c0b9b4bf5a03db12163232 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Fri, 3 Mar 2017 14:18:52 -0800 Subject: [PATCH 333/407] Reimplement PR #2525 Fixes typo by removing the word "controller" --- docs/admin/admission-controllers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/admission-controllers.md b/docs/admin/admission-controllers.md index 29429c5b2c..fada47fd50 100644 --- a/docs/admin/admission-controllers.md +++ b/docs/admin/admission-controllers.md @@ -87,7 +87,7 @@ The ImagePolicyWebhook plug-in allows a backend webhook to make admission decisi ``` #### Configuration File Format -ImagePolicyWebhook uses the admission controller config file (`--admission-controller-config-file`) to set configuration options for the behavior of the backend. This file may be json or yaml and has the following format: +ImagePolicyWebhook uses the admission config file `--admission-controller-config-file` to set configuration options for the behavior of the backend. This file may be json or yaml and has the following format: ```javascript { From b50ffef3fb6244817448422d6ba1f335b04b05cf Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Fri, 3 Mar 2017 14:19:34 -0800 Subject: [PATCH 334/407] remove extra space --- docs/admin/admission-controllers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/admission-controllers.md b/docs/admin/admission-controllers.md index fada47fd50..e3d839f0d5 100644 --- a/docs/admin/admission-controllers.md +++ b/docs/admin/admission-controllers.md @@ -87,7 +87,7 @@ The ImagePolicyWebhook plug-in allows a backend webhook to make admission decisi ``` #### Configuration File Format -ImagePolicyWebhook uses the admission config file `--admission-controller-config-file` to set configuration options for the behavior of the backend. This file may be json or yaml and has the following format: +ImagePolicyWebhook uses the admission config file `--admission-controller-config-file` to set configuration options for the behavior of the backend. This file may be json or yaml and has the following format: ```javascript { From 3456093031f3661e5a81e42b4e836b3c9cd87997 Mon Sep 17 00:00:00 2001 From: Anirudh Ramanathan Date: Fri, 3 Mar 2017 15:28:17 -0700 Subject: [PATCH 335/407] Update multiple-schedulers doc (#2063) * Update tutorial * Fix md formatting. * Remove extraneous space --- docs/admin/multiple-schedulers.md | 16 ++++++++-------- docs/admin/multiple-schedulers/my-scheduler.yaml | 9 ++++++--- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/docs/admin/multiple-schedulers.md b/docs/admin/multiple-schedulers.md index c20e129b19..e4f1b4a9f1 100644 --- a/docs/admin/multiple-schedulers.md +++ b/docs/admin/multiple-schedulers.md @@ -34,7 +34,7 @@ to build the image: ```docker FROM busybox -ADD _output/local/go/bin/kube-scheduler /usr/local/bin/kube-scheduler +ADD ./_output/dockerized/bin/linux/amd64/kube-scheduler /usr/local/bin/kube-scheduler ``` Save the file as `Dockerfile`, build the image and push it to a registry. This example @@ -45,7 +45,7 @@ For more details, please read the GCR ```shell docker build -t my-kube-scheduler:1.0 . -gcloud docker push gcr.io/my-gcp-project/my-kube-scheduler:1.0 +gcloud docker -- push gcr.io/my-gcp-project/my-kube-scheduler:1.0 ``` ### 2. Define a Kubernetes Deployment for the scheduler @@ -131,15 +131,15 @@ scheduler in that pod spec. Let's look at three examples. Save this file as `pod3.yaml` and submit it to the Kubernetes cluster. - ```shell - kubectl create -f pod3.yaml - ``` +```shell +kubectl create -f pod3.yaml +``` Verify that all three pods are running. - ```shell - kubectl get pods - ``` +```shell +kubectl get pods +``` ### Verifying that the pods were scheduled using the desired schedulers diff --git a/docs/admin/multiple-schedulers/my-scheduler.yaml b/docs/admin/multiple-schedulers/my-scheduler.yaml index aa7cc1baef..5e97f46434 100644 --- a/docs/admin/multiple-schedulers/my-scheduler.yaml +++ b/docs/admin/multiple-schedulers/my-scheduler.yaml @@ -16,8 +16,11 @@ spec: version: second spec: containers: - - command: [/usr/local/bin/kube-scheduler, --address=0.0.0.0, - --scheduler-name=my-scheduler, --leader-elect=false] + - command: + - /usr/local/bin/kube-scheduler + - --address=0.0.0.0 + - --leader-elect=false + - --scheduler-name=my-scheduler image: gcr.io/my-gcp-project/my-kube-scheduler:1.0 livenessProbe: httpGet: @@ -37,4 +40,4 @@ spec: volumeMounts: [] hostNetwork: false hostPID: false - volumes: [] \ No newline at end of file + volumes: [] From 172900f9aae66d6ba2ff762d6277f6aaacc59b5d Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 14:54:05 -0800 Subject: [PATCH 336/407] Deprecate Guide topic: Secrets Walkthrough. (#2695) --- _data/guides.yml | 1 - .../distribute-credentials-secure.md | 3 +++ docs/user-guide/secrets/walkthrough.md | 7 ------- robots.txt | 2 ++ 4 files changed, 5 insertions(+), 8 deletions(-) delete mode 100644 docs/user-guide/secrets/walkthrough.md diff --git a/_data/guides.yml b/_data/guides.yml index 318f965319..bf188b9037 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -25,7 +25,6 @@ toc: - docs/user-guide/resizing-a-replication-controller.md - docs/user-guide/rolling-updates.md - docs/user-guide/update-demo/index.md - - docs/user-guide/secrets/walkthrough.md - docs/user-guide/configmap/index.md - docs/user-guide/horizontal-pod-autoscaling/walkthrough.md - docs/user-guide/config-best-practices.md diff --git a/docs/tasks/configure-pod-container/distribute-credentials-secure.md b/docs/tasks/configure-pod-container/distribute-credentials-secure.md index 1a0e25e46e..6dcd642262 100644 --- a/docs/tasks/configure-pod-container/distribute-credentials-secure.md +++ b/docs/tasks/configure-pod-container/distribute-credentials-secure.md @@ -1,5 +1,8 @@ --- title: Distributing Credentials Securely +redirect_from: +- "/docs/user-guide/secrets/walkthrough/" +- "/docs/user-guide/secrets/walkthrough.html" --- {% capture overview %} diff --git a/docs/user-guide/secrets/walkthrough.md b/docs/user-guide/secrets/walkthrough.md deleted file mode 100644 index a58364859f..0000000000 --- a/docs/user-guide/secrets/walkthrough.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Secrets Walkthrough ---- - -{% include user-guide-content-moved.md %} - -[Distributing Credentials Securely](/docs/tasks/configure-pod-container/distribute-credentials-secure/) diff --git a/robots.txt b/robots.txt index 8e77127e0b..f5e996388b 100644 --- a/robots.txt +++ b/robots.txt @@ -15,4 +15,6 @@ Disallow: /docs/user-guide/production-pods Disallow: /docs/user-guide/persistent-volumes/walkthrough Disallow: /docs/user-guide/pods/single-container +Disallow: /docs/user-guide/secrets/walkthrough + SITEMAP: http://kubernetes.io/sitemap.xml From 5e4a1ed081149d5e62491744a2b94463eb6151cf Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 15:18:37 -0800 Subject: [PATCH 337/407] Deprecate Guide topics: quick-start ... (#2696) --- _data/guides.yml | 2 -- .../expose-external-ip-address-service.md | 3 +++ .../run-stateless-application-deployment.md | 2 ++ docs/user-guide/deploying-applications.md | 11 ----------- docs/user-guide/quick-start.md | 10 ---------- robots.txt | 9 +++++++++ 6 files changed, 14 insertions(+), 23 deletions(-) delete mode 100644 docs/user-guide/deploying-applications.md delete mode 100644 docs/user-guide/quick-start.md diff --git a/_data/guides.yml b/_data/guides.yml index bf188b9037..31be2a277e 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -18,8 +18,6 @@ toc: - title: Workload Deployment and Management section: - - docs/user-guide/quick-start.md - - docs/user-guide/deploying-applications.md - docs/user-guide/managing-deployments.md - docs/user-guide/replication-controller/operations.md - docs/user-guide/resizing-a-replication-controller.md diff --git a/docs/tutorials/stateless-application/expose-external-ip-address-service.md b/docs/tutorials/stateless-application/expose-external-ip-address-service.md index f4a63b1e31..8564ed7d72 100644 --- a/docs/tutorials/stateless-application/expose-external-ip-address-service.md +++ b/docs/tutorials/stateless-application/expose-external-ip-address-service.md @@ -1,5 +1,8 @@ --- title: Using a Service to Access an Application in a Cluster +redirect_from: +- "/docs/user-guide/quick-start/" +- "/docs/user-guide/quick-start.html" --- {% capture overview %} diff --git a/docs/tutorials/stateless-application/run-stateless-application-deployment.md b/docs/tutorials/stateless-application/run-stateless-application-deployment.md index c81bcdae18..831ed6bb45 100644 --- a/docs/tutorials/stateless-application/run-stateless-application-deployment.md +++ b/docs/tutorials/stateless-application/run-stateless-application-deployment.md @@ -5,6 +5,8 @@ redirect_from: - "/docs/user-guide/simple-nginx.html" - "/docs/user-guide/pods/single-container/" - "/docs/user-guide/pods/single-container.html" +- "/docs/user-guide/deploying-applications/" +- "/docs/user-guide/deploying-applications.html" --- {% capture overview %} diff --git a/docs/user-guide/deploying-applications.md b/docs/user-guide/deploying-applications.md deleted file mode 100644 index a1c04e1003..0000000000 --- a/docs/user-guide/deploying-applications.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -assignees: -- bgrant0607 -- caesarxuchao -- thockin -title: Deploying Applications ---- - -{% include user-guide-content-moved.md %} - -[Running a Stateless Application Using a Deployment](/docs/tutorials/stateless-application/run-stateless-application-deployment/) diff --git a/docs/user-guide/quick-start.md b/docs/user-guide/quick-start.md deleted file mode 100644 index 67dfb1dc77..0000000000 --- a/docs/user-guide/quick-start.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -assignees: -- bgrant0607 -- janetkuo -title: Launching, Exposing, and Killing Applications ---- - -{% include user-guide-content-moved.md %} - -[Using a Service to Access an Application in a Cluster](https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address-service/) diff --git a/robots.txt b/robots.txt index f5e996388b..3fbd70eb24 100644 --- a/robots.txt +++ b/robots.txt @@ -8,9 +8,11 @@ Disallow: 404.html Disallow: /docs/user-guide/configuring-containers Disallow: /docs/user-guide/containers +Disallow: /docs/user-guide/deploying-applications Disallow: /docs/user-guide/liveness/index Disallow: /docs/user-guide/simple-nginx Disallow: /docs/user-guide/production-pods +Disallow: /docs/user-guide/quick-start Disallow: /docs/user-guide/persistent-volumes/walkthrough Disallow: /docs/user-guide/pods/single-container @@ -18,3 +20,10 @@ Disallow: /docs/user-guide/pods/single-container Disallow: /docs/user-guide/secrets/walkthrough SITEMAP: http://kubernetes.io/sitemap.xml + + + + + + + From 1bc38a5bab60ac69fc31b88546fb635b32a82e0e Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Fri, 3 Mar 2017 16:00:04 -0800 Subject: [PATCH 338/407] Update Tasks landing page. (#2697) --- _data/tasks.yml | 5 ----- docs/tasks/index.md | 9 ++++----- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/_data/tasks.yml b/_data/tasks.yml index 8863559c77..bfc5c9426f 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -54,8 +54,3 @@ toc: - docs/tasks/manage-stateful-set/deleting-a-statefulset.md - docs/tasks/manage-stateful-set/debugging-a-statefulset.md - docs/tasks/manage-stateful-set/delete-pods.md - -- title: Troubleshooting - section: - - docs/tasks/troubleshoot/debug-init-containers.md - - docs/tasks/administer-cluster/access-control-identity-management/ diff --git a/docs/tasks/index.md b/docs/tasks/index.md index 1b532b9232..4572ca077b 100644 --- a/docs/tasks/index.md +++ b/docs/tasks/index.md @@ -37,9 +37,12 @@ single thing, typically by giving a short sequence of steps. * [Using Port Forwarding to Access Applications in a Cluster](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) * [Providing Load-Balanced Access to an Application in a Cluster](/docs/tasks/access-application-cluster/load-balance-access-application-cluster/) -#### Debugging Applications in a Cluster +#### Monitoring, Logging, and Debugging * [Determining the Reason for Pod Failure](/docs/tasks/debug-application-cluster/determine-reason-pod-failure/) +* [Debugging Init Containers](/docs/tasks/debug-application-cluster/debug-init-containers/) +* [Logging Using Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/) +* [Logging Using ElasticSearch and Kibana](/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) #### Accessing the Kubernetes API @@ -60,10 +63,6 @@ single thing, typically by giving a short sequence of steps. * [Debugging a StatefulSet](/docs/tasks/manage-stateful-set/debugging-a-statefulset/) * [Force Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/) -#### Troubleshooting - -* [Debugging Init Containers](/docs/tasks/troubleshoot/debug-init-containers/) - ### What's next If you would like to write a task page, see From 1cbcd2b596cc88d23fde75b02120456e74efb076 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Thu, 2 Mar 2017 07:28:36 +0800 Subject: [PATCH 339/407] Highlighted command for `kubectl proxy`. --- docs/user-guide/kubectl/kubectl_proxy.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/kubectl/kubectl_proxy.md b/docs/user-guide/kubectl/kubectl_proxy.md index 9616d6add6..f2e4b9f58a 100644 --- a/docs/user-guide/kubectl/kubectl_proxy.md +++ b/docs/user-guide/kubectl/kubectl_proxy.md @@ -11,19 +11,26 @@ Run a proxy to the Kubernetes API server To proxy all of the Kubernetes api and nothing else, use: +``` $ kubectl proxy --api-prefix=/ +``` To proxy only part of the Kubernetes api and also some static files: +``` $ kubectl proxy --www=/my/files --www-prefix=/static/ --api-prefix=/api/ +``` The above lets you 'curl localhost:8001/api/v1/pods'. To proxy the entire Kubernetes api at a different root, use: +``` $ kubectl proxy --api-prefix=/custom/ +``` + +The above lets you `curl localhost:8001/custom/api/v1/pods` -The above lets you 'curl localhost:8001/custom/api/v1/pods' ``` kubectl proxy [--port=PORT] [--www=static-dir] [--www-prefix=prefix] [--api-prefix=prefix] From 458f80b77a35dff4b7e310e9a28acb3a1fe47a18 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Tue, 28 Feb 2017 11:23:10 -0500 Subject: [PATCH 340/407] Update ubuntu install instructions. --- docs/getting-started-guides/ubuntu/index.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/index.md b/docs/getting-started-guides/ubuntu/index.md index 5635c1d553..e5a5125f2f 100644 --- a/docs/getting-started-guides/ubuntu/index.md +++ b/docs/getting-started-guides/ubuntu/index.md @@ -20,11 +20,8 @@ Supports AWS, GCE, Azure, Joyent, OpenStack, Bare Metal and local workstation de Available for Ubuntu 16.04 and newer: ``` -sudo apt-add-repository ppa:juju/stable -sudo apt-add-repository ppa:conjure-up/next -sudo apt update -sudo apt install conjure-up -conjure-up +sudo snap install conjure-up --classic +conjure-up kubernetes ``` ### Operational Guides From 1c5e5248adcd17e52f5abac36d3cc18bb4578643 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Tue, 28 Feb 2017 12:08:17 -0500 Subject: [PATCH 341/407] Update local instructions to new method. --- docs/getting-started-guides/ubuntu/local.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/local.md b/docs/getting-started-guides/ubuntu/local.md index 7285b5915a..767e867680 100644 --- a/docs/getting-started-guides/ubuntu/local.md +++ b/docs/getting-started-guides/ubuntu/local.md @@ -42,11 +42,8 @@ Next, apply those kernel parameters (you should see the above options echoed bac Now you're ready to install conjure-up and deploy Kubernetes. ``` - sudo apt-add-repository ppa:juju/stable - sudo apt-add-repository ppa:conjure-up/next - sudo apt update - sudo apt install conjure-up - + sudo snap install conjure-up --classic + conjure-up kubernetes ``` Note: During this set up phase conjure-up will ask you to "Setup an ipv6 subnet" with LXD, ensure you answer NO. ipv6 with Juju/LXD is currently unsupported. From deef42e84988e8fa52c4fe69994038a2cde8f630 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Sat, 4 Mar 2017 15:22:54 +0800 Subject: [PATCH 342/407] update init-containers.md add one word. --- docs/concepts/abstractions/init-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/init-containers.md b/docs/concepts/abstractions/init-containers.md index 89738eef56..4be8890e48 100644 --- a/docs/concepts/abstractions/init-containers.md +++ b/docs/concepts/abstractions/init-containers.md @@ -67,7 +67,7 @@ have some advantages for start-up related code: `sed`, `awk`, `python`, or `dig` during setup. * The application image builder and deployer roles can work independently without the need to jointly build a single app image. -* They use Linux namespaces so they have a different filesystem view from app Containers. +* They use Linux namespaces so that they have a different filesystem view from app Containers. Consequently, they can be given access to Secrets that app Containers are not able to access. * They run to completion before any app Containers start, whereas app From ac31bb82704961108fd224a8c98e65a580ae50f1 Mon Sep 17 00:00:00 2001 From: Craig Sutherland Date: Sat, 4 Mar 2017 19:34:27 -0500 Subject: [PATCH 343/407] Minor spelling correction -- "rtk" to "rkt" --- docs/tutorials/kubernetes-basics/cluster-intro.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kubernetes-basics/cluster-intro.html b/docs/tutorials/kubernetes-basics/cluster-intro.html index a7ca857705..1041f1af36 100644 --- a/docs/tutorials/kubernetes-basics/cluster-intro.html +++ b/docs/tutorials/kubernetes-basics/cluster-intro.html @@ -74,7 +74,7 @@ title: Using Minikube to Create a Cluster

    The Master is responsible for managing the cluster. The master coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates.

    -

    A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master. The node should also have tools for handling container operations, such as Docker or rtk. A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

    +

    A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master. The node should also have tools for handling container operations, such as Docker or rkt. A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

    From 133af46f89f8c9bf56f0fa6df049659aeeea0d71 Mon Sep 17 00:00:00 2001 From: Phanindra Pydisetty Date: Sun, 5 Mar 2017 10:24:43 +0530 Subject: [PATCH 344/407] fixing typo --- docs/tutorials/services/source-ip.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/services/source-ip.md b/docs/tutorials/services/source-ip.md index 4f5f7d884a..0df6c11f58 100644 --- a/docs/tutorials/services/source-ip.md +++ b/docs/tutorials/services/source-ip.md @@ -174,7 +174,7 @@ service "nodeport" annotated Now, re-run the test: ```console -$ for node in $NODES; do curl --connect-timeout 1 -s $node:$NODEPORT | grep -i client_address; do +$ for node in $NODES; do curl --connect-timeout 1 -s $node:$NODEPORT | grep -i client_address; done client_address=104.132.1.79 ``` From d8595aa4ea62638ccdbd6e7c4108d1e1da2a9686 Mon Sep 17 00:00:00 2001 From: Justin Nauman Date: Sun, 5 Mar 2017 15:53:32 -0600 Subject: [PATCH 345/407] Doc formatting update Corrected indentation for the `nonResourcePath` and got a bit zealous with restructuring that section... --- docs/admin/authorization.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/admin/authorization.md b/docs/admin/authorization.md index e5c091b772..2c3e29338b 100644 --- a/docs/admin/authorization.md +++ b/docs/admin/authorization.md @@ -87,15 +87,25 @@ properties: - Subject-matching properties: - `user`, type string; the user-string from `--token-auth-file`. If you specify `user`, it must match the username of the authenticated user. - `group`, type string; if you specify `group`, it must match one of the groups of the authenticated user. `system:authenticated` matches all authenticated requests. `system:unauthenticated` matches all unauthenticated requests. - - `readonly`, type boolean, when true, means that the policy only applies to get, list, and watch operations. - Resource-matching properties: - - `apiGroup`, type string; an API group, such as `extensions`. `*` matches all API groups. - - `namespace`, type string; a namespace string. `*` matches all resource requests. - - `resource`, type string; a resource, such as `pods`. `*` matches all resource requests. + - `apiGroup`, type string; an API group. + - Ex: `extensions` + - Wildard: `*` matches all API groups. + - `namespace`, type string; a namespace. + - Ex: `kube-system` + - Wildard: `*` matches all resource requests. + - `resource`, type string; a resource type + - Ex: `pods` + - Wildcard: `*` matches all resource requests. - Non-resource-matching properties: - - `nonResourcePath`, type string; matches the non-resource request paths (like `/version` and `/apis`). `*` matches all non-resource requests. `/foo/*` matches `/foo/` and all of its subpaths. + - `nonResourcePath`, type string; non-resource request paths. + - Ex: `/version` or `/apis` + - Wildcard: + - `*` matches all non-resource requests. + - `/foo/*` matches `/foo/` and all of its subpaths. + - `readonly`, type boolean, when true, means that the policy only applies to get, list, and watch operations. -An unset property is the same as a property set to the zero value for its type +**NOTES:** An unset property is the same as a property set to the zero value for its type (e.g. empty string, 0, false). However, unset should be preferred for readability. From 3abc2a301d70a726805b85e55b24eb5ab5ac99c0 Mon Sep 17 00:00:00 2001 From: mlambert890b Date: Sun, 5 Mar 2017 20:44:47 -0800 Subject: [PATCH 346/407] Repair Spotinst logo Corrected size --- images/square-logos/spotinst.png | Bin 86339 -> 5239 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/images/square-logos/spotinst.png b/images/square-logos/spotinst.png index b933c77546138512c3a36c907f37cced6bdfac76..645abbed45c5cd16f3196e27a0e37a718129b19d 100644 GIT binary patch literal 5239 zcmZ`-cQ71a^FEGKq6DXtqau zB)UWiLDbLp=Wpiw?K`{AyU)%u@666Svoo7)WS~t=$wmnP0H_gg81e?+-ncnA=q5V) zJSW@$iL<)CIsj0caqHqK=}pdM4@c?)0HOQC z=eHUw-4sA}y4oO|P6W<~)Zb&kJguW)(cOWSP1rJq_ zY7hXx;Dmswn+DJS#)jo%rbGI=>g&z387)w)P#m2N{2^EbVdSU&vp4=Mc`9kf?vV$Y z|0Cj={wv@PIlQr2L^AXkP#LWqt%*;CqeJb|)l9J%ChphKE%=xC^O%H4 zra9_GvWr{$l`BO88wB--2v@p4BrBEp=lF6;#x9yu+5HGxNU&Wx*fi}%w*im?mcf*= zNo9!#Q6tSnV$-fG-X+cjjVd_4X32Y~XG1 z4ERu-KxuO=K1|<|nYhpLWIJXsh5{~jI?Q!asLg2P^WV2QRub{!BRnIZOd&H3Ur9MCaxwm12Jczd!QC#`-qiC zV4YzOsmN;3agOk4mj7u{QHk^ySFPT@@EtRzQ@VXdTlS+uFq8Z!pffT17~lmo=k8iR zP-7oUGL0L?KU+OnWNKgOm%lKgMm|m;*tLc4lB%ToMABl&nIu(7{E_ALyP8?J9@i%{ ztj2DcU5B14e1}y68xk5laMvVXlKo}8=+2A?!ML*(4fLWr^mWjad1?8@3AXttlMK(` zmy0XyUc-_NHK?3$ee()T# zNB~PYcE<6pJ(_vn6up6(-!EKW-oQE&=erij-%^rH1PX7JDbd{jc9yR`_;X3!f~gQ# z09(ij;tHD7DS2?r?HqpV59JV0Tr+H!Xdbl|Dd}q&Ld7jyWWu&p2RNnuGZ7>+KWzEL z1NmxC6-ehTB=s!idjwuC^X_zS2Oc7MpP`=z^b&&+Fe#^LCwIT|RzsbTG&G|A%> zLJ<4B5TyijGhp-A=C3~qqqpB66&%@uGyFW{`P-}JKlGcVz-KPg2xb37b^?L>uHeVo z+Riz%l8FEy|6A4uE3n#A`%7C{hyK{(FX-XW-GCJ>%9C%)iTT$#Q05$gbuN1~g}1C# z;ECZY&Rlm-#~9sgQI{^~a`s%@m3<7EF}^m9YM*%D5N`*D{m3~|SL@60pS%X16 zet`@sRWnrKbci9dE{r+EF>AuAPL(>G6)^~~T*)Cs2RoFr=a6VwsiZHOJHHMT`y@IG z5-PF3q8?jGwzN5#<&(aJK)lXuLrJ$#2m6(?oAFEGnGR1VDcjB?R@_#xzhnd-A2Q=V z%)F!x#CUA_`v=N!3HgcLn_d z;vW#6k&4YNgRa;%Ifa0Zs6#)zDw?)u5Jxys)Yf^F$bM$OnXg@tiNno3VpWfiCq_h( z?KcEe(jW%j<$L`REX1S{^zD8|uE_BuXAE$hs%i{q9Z(X)6Q*p46$7B@xKYi32wCCklDoA{8mc zJ8O&U2NMS@1#>XUV6ABm9l@M<@exobBC^a)`j3S>@s&q8yPVM%#Ko1~$2xdO(&)OR zV5}hgVKexvRqr{3ul=u8s^~w6QFENXL=h-gT?J$Zq;wHtUwV(?$l*blr;-*fgHPQ|*DTL(1nQksR_zEA%) zJp@Qd2iPcUK0V7PdgG4d|6BQzP%u;DJjmMayJK1O!d`MCP;^ztGGFp!Hm@skHtgo# z*PS(5U9LH}s&M}}u#S?YqR*NTYG?g7pAMt>s{(F4=C6Y<*BSt4iW?%TBT)}1DwYBV zRH#t6;r6y$Yu9sQ<8v=XrUlEYm`p}UOhzBCLKfqBr`xbhlXFC+3HAf^kMQZs%$Uk#BG>ulSVy!o@Agt|_OEgs74p z-FKA5;g%VO&mqa1O>QwF`8%V<#SBFK{KXQgCq_2W9(V{+P<|BHdFQ~LhBP6Zl<(kG z3_3Z2$|FP;ZTRD_@PKbnldD&SPsCkHI~6jD_;DAFzue15uSajS zTId4aX}VrkOHp4c9bDB&;tStkvk;P_(%k6$7sMfZ+Q}4V7GF5x=-Ui3go+dP)$4Jz zo>0R9kcpP{ey>?Kee?8@F1Qq+Dtl7HCyI?#2qz~U86xSNAD?SfbfPcQhg^sqBv>;b zZDM~xS;365%k9jG#S}Z=mEj-2n#i4V-jzV<+lb){TUSN3gKwGtT-ttTVG4NbGi$wH zbcE+#Hx&_1c9I23Qpa74c473&@2=MHHrY~O@nP|zT}D6JG>zErPDK>1#Y8=^k?E&C zoq$uCwGgJ*$gPYGer@6|NAy}74c5Z{Z7=0Yl+gzbu~$9Jz@P7(k@%XJC{3g%{e-H! zU7=URQZ$<7msbH77t2K0gSi*%H?~fjFac-JPe^k`RP7o31T8~c@5ny zGpW1}0FR)nnJ-&n_R-UEple;1nLWADJ-M_i-FxJ+{KS4QFzy-(C2qA#i7#TO!Ss%b z=*4B<>au8ELSO1JbTCR_e9QSb@BDaK<(x=Y>bH8AHOhaZbad*T@MAwt=%-oZgbX$1 z`LrpnS=K@gh7r!4KuYseE`g5MS7FT0g}}dkk6)b|_f8Q_26;jlO`M}0&{EHIfAhjj z!?Yar6B}}a3V^=pm3%r7n>440j7ip*>Qg0!>ae`Y@p`(JQU+OtK<=#ufdNjtfsW4@ z`OmEZR95I2i_Tu~c&NW{GGt6x07bLjpt_CCk)&7X4(=z;C_+4A&yET3%<7X-ax)U{ z(*(C-1EFC*ic2AnGg!+6OKv8A&A`%$SvU0Lv*6}m25`Y_2>IpAFsaL&9`X&tHngK2A?^1Y6tm8@w(xm@ne2aU5eVrNQp=7nKV~G{FL)a>Wa(;j7 zrS$6dZ@Rr2Ud!};bMmYR;k6Azh=Jf%zO02#uyY0LRy{5HIwWsyJ+$|NTDrP{7oF(~ zL5Dtq?xty6eW*n7oN<=`KR#?O2-8da>Y_0afuZ%j#Ruv|XYa1MuC4GpBCFJU5mUA4 zpT~V8fNYatsHci-?+TxbSJZ6zI3UD$B0s#&H1K7p`(a>c<`0DpU$HgtYRa#j2rQnY z{dqGpp$b|1lpw0p;7#2VrQ%Ihw-gl(T@}a{549CU=-g|TZ}13KsY4g?bFi~;Jzo@P zBFYk$uH1%hhXPuxiz01AT{{aCa(&78=5~y_U%1htvkTR#D|3tswcZeUXfCO}G)JRL z{6RrH$j}X9h&IiRK+>@?7Id+sCgRc=rSx>;H_ht1q=83MS`_v@m@wX_KwHHFAcN(| zr`&_tioKd~ivr!H*_Ia{2I5=Dv1xAUsBVWqGEq zsQvH(efJQ|Gdgh`y|~@%b&QttURoh?k+t^q{{unMb)R{6x8;~}4@*z@j^4%oJB!jK8>OI|Bn+q{goyuJv4i)lSWxNip zT0JuB*+JP)2aYfCMT$yhUZMu-q+d5P6r@Mp*RO2r4PTQywYA<5O#WY`RrPWB{Q^5V@RAxQ0%XiHhfy#LJ92-=HG=9Ixsl zdW;XtzcpT(Z!VcnXxJRHUp;n^{7Aoaf>q0<5-(7v+xJN}+S69#43xT(9^XK!iF^qK z_gu0{3mTW#ZU>2`U|y*|)v#iYX()U_Ri}xeIV#u8`XR_G7PY$Q1v5S}H!2rt+ti2= zRi>LQHt+4Ki}Hnv*%2Kof1GhyT>2MIY8*uuO+qP(*E>U( z_o3RPIEV2s9mJRD{8TQOS4i@ili*MaM=~$xF|ib-N>Wi=x>miIx6%nzqdgsVuA!H2 zufcaA2L9YcK3uXAZa~Ru;5L30Dv;pr^NIYXZ|9l!PM?DNBW10s`*J@Kpv4YADLB*X zDBVjfgS*f*&^?JsQ~?vmx>a(?mfdRKc}s<1W|+|Ki}bXP(vAfodc7YjGTp69DSB8H6rKg@ECkwF^rf zoP+d>;@ott6gQE7w;L~4?XQnXXnW&d48zXGEO!2+!uJ-E!<`t0gG$D#$C>=@eq4n9 z_0RyBh*%ZHvh;hl4M`0uQ;A{U!`VC^^cvez$q?t=Y-`RH6xw2+jOx<|fH}@|vKH4S zQi%1p2{%+*XPYcFo7@{-CDmx!x(>?OX^yNrD`QIgZk)8zMkmzt zN;dn~4vgDs?Q}6EUy5W)*ANLGa1;c48p^Mky&HL{#O~3CL>CNKlhF6d_RcgSFHI}z zHy>uOOEW5Sw8*K%QgNMg^ee0ugiK#VEt3;izDokAwC1yg2(?N;F@{FElRenrR)pXR z&%siCAK|lYrmt`c+q=7e^pt&5^W%=#3>Tlhel%Rhr3 zQ42aKX`VZYAW$4;5H-2*nTIZX!j$%ZTgLbFYcd67{u@!^gqoWr9G}0Yg}=j7e@6v- zU&k8&NJ&V_iAmfOlaMi$lvj|DSGXs0b61d%;Jp2b_WuYxyc}GeLjHe&yM9RhjR1hq LG=NoW*v9?`NY|+G literal 86339 zcmeD^i9eKU+mA?zQc9*yhtya?3@R0&v>3aPWScCh>?CBH>L5#tvA3WkM`S&beIzt3 zXhTOrDLTX?YqsyYXUvTA{tMr^f4}d1d7gW@_Vs?|jW9Aew0zn6Wf+DnCu;9MjA1Kq z7`EihQeJqn^jfbm{69VqZ3}M<;}=5z$Mw1+cpAetV#NJ>P5u5J=y<7b#kW4R`%U_L zE#4Dm?PqyU`1_v4B$8JyJ-T@u3j9Zh|LB1I=P#H-@E;xiqr*aT@Si;VClCM0!+&8jO9uYGgv}%Fs{v(=)40sf z)e|GfDJ2>cTW1HI-D9oxd^%J%ojp`=Og?w|A&V>+)-@G8`D-*=7{k(@5)9mx!-HuX zrd&VHDbA|u44*Iz{;rVxx$g4wB|~|wh7_y3ged1Z3C`+mD{+f3YKq^=6`~JP@w;rQ zMIA%zY7z@0gv4yjDvpK7?dN2-q+UJ&E{t?Y;PaV^;HidI-xmMvzNJ@JsI2@HO*|<2Fx)Q^RpAZbfi66htF6#ce={DKJ*LBg8{!#DvUFsFvmUAZO>64aXn1Tx~ zGyMfuw`bL5KEWK5FOuW0y1$N|erWaKI!gPQgPgb#@@cZuaMz~pue&0h|Cp|P`L6b* z_xDfYtSK1Qv={AXND7MgSS=*laA04`NBzO{+UJkX!sGctH55XqZ;s%5ze$CdtGpyU zans?Cr+s(+t)_DsOf${ zqk^!6y%)8AH872Se}AYE(S^gy{KaTx#ZF^!I=KzRZhmydF9m`1TXkn&*OE)iIS1KX zLnLt89{q;6Ka~2})vnes`^%?i8(8Z=bPN$SV}Gmvwv$Hunop;0q8gUbf?WQESEe}N z)kOq#i0j$rZ8}lEiK;wF=~1Iio-Tc~eF4xKQfQ(W>_+f5^1I$+9)&3kR3~fI@|$pu zR7*iX9bz6yx!QsiQ?0*dhov5QUiWYpXIgMHnkJ?c{l<7OPq?aJX?}*f)0O24FzvyD zX|o!LeVu?}_>0zMotE5f;=V+ACEM*4N(%r>MG8F;3j|j_6q+l%XsXMb@)H!<4He#) zgB*ZZ91P;(33N!`PTq4Su}AHvS*`N|Vfl{8?|{pUg|k8)@)~_Y$muRX?g5&R5$3u3 zS>OZuHrn|EJuVNetZr~jOxt!KE%|r698cTsg(vr-lh0ntUrFj; zxaBP*Zf#tT_mOXH-p0|vVk9@DC_a9!vnkCS*>5|BHs6XiS6#SyKia&QiPmO#X+B5W zu>Rw94pe@wjD`6t_?sALe`KI7bsNNKe$cNY(&e4RW!AokiO64>)@1$lDgDJ7)_aWV zjJ!(41)rmSWx=5BM&BiA?@yQRTtI$01?_<>e(Vc#{?JfNZA60h4fJLkhedQD#vG6< zLEyDY3)ZEoF15>It4QlT_EA_$ETG?&48~=L_tA;Q^_bdY_hf~CW3&Qcj@Yq%nr-l1mNBgV;Wlrb;0o{)IEmBF&TW6DmDAN@((6PtsH}l8ayX@V`AI{@^ zbQ+gkqU~NSIV4jfyRvf8URlD!MSIO|-FUWJ`O%`-Q&RVi(1m2XM7E6(KgT%4EP6WY zpCS`Ca6OA}#jc>d`@g!F2$bJJf~w`%(fH=UTJ>07EWjf<_oSZ$NBz1EZ2?jaXj+sU zA|4?cWN=Y8`^7rEH?4ah&87i4{4hwy1earnsHBCY|CZ&L=zGbI>)kpFIHDyD+o%~? zltkfsi7W1MQCFAgnQyBQ;!xW|Y#DH5+_dGpF~y8xatdqS>`sxpasQDpXAXVO8jzkU z6N{2?P>Ko)FSh3)vE;;~1>4XK8B??@N{Y^8hc>}Un3bf(HZ#x73y^YV%sF0u?2*o} z)p|^zx&NL_f*l(ntZIJ^FhllP*&f|OJ+F<+G2Q{KvhB4G7vS84=rXMjboq9K-w^0> zse)MOa{tp24$_*GX}Luh+3;DJwdADivrsI$={a5nh<$+L3h-+IAqw8HBjv8;R<#6P zY&Bh-_V_wS*!6BK*1|}`#o3`D2TNCyzxjze1DWOr!5@~BX@$p@V#J(vY#pLHSO7yL zX{W`p>5vvUV1AJ9$N|u?b`Mu#RkVd?Tta6k<*Z#q)hIl{acisXfDE`j*`XMH;}GSh zg)&>~$v+A~a-xq}lsFMvjOBUAQ|DrO?&q{8b0|c*i4IE#ejcYp;WJ-@(-#=<0R^nQHGph(TCRqJm1j6O-r@NJCx~Cw(a6vM$Q-mWE z0;w0oF~Twq1qV8&&Nzpa++iJqk-L9c@L<&Dc%OiTo+fFLMO1tLl>deH5rX!j89-b1 z?r|?bEA{h#p!Gx0?v?2u!_y8>oX`y4WQULc(gf1Z;I&*BVVh%zeC+hPFZ-~*s5cCb z?`QJ>t38e|iY{~$jLo5_CGuh;E&rB2@~jDPkN0*?dn{h009dQ#@xSM!Riks7eF15S zNo9~M!kSE?Td^(2{9+? zkR{TqpTbAolW|+1D~4KV&S4-fX!LR6R3g z92bB5WK-+`6JhfOg3yo&Qmd9+=$1&dAudXJE1CN2yvn30Z^g4rp>BN>VHNJ6#y}eb zwZDTYbMjkLer+d9g{*ja4Y}aUQ*U(tIWQpba51)&^ z*?*&-oZ9swbJM4SKS)FWLv=bsj1GaI8B?O9`>FEQ_hgsbe0j61+ss6`hcZSFzW*j&wtg8hEN5|7pQbY6KKFX%M+ z7uio{o!_;7ymFHYdtsXf$U>KKoKLy7m*|&$6QZq?rg*DEF&)7nzT{8#K07?dwgvCv(J(y@uR@ zgnrDR|4uKeG8ubweIY)uiRST8GzdFLC>aM6Vdb(8s^UG^ab!iR$soQ+5AX zRc@d(F2LpxSq#b7z<%WA+$Cz3T4%~O=7I+kTEcx!zUfW;^4+0=JFx1o-z%^x+PE6;U((jEJl6JyMfdyfaldc3mE*$vaEPVhz*3#z zeL?~s!d-Mj!E#=@7^6`l_%q98r=izIVcx<2A@UR=vZ#Zl(XOjr;zLbV&#w($H z005}vXPiTqZ6s7FpGbjNO=fw@Q^^j>ZDqC;VkoE!JXm^7YDAnSV_iT$vVrjH=T~W( z5>1VEe!riC4kv)f&w#mjWBDGMRi>*VZ zPi)~tJ%n9gtO$Fbg$^X!iRWJ}>X(iT8>Jx(u+lG5`8(RB!KX|@%V|A(QaaM})y6H~i4-3S_o@8N zk)fJmjU{NifQ8^+K;V6U#6gJ1 zYVj(-+y<+AmU326i*SAZkF>o8)q0lvk7ubrh~}TPl($00S8zn!y1+wX842I}>HS#x z$){Qisk|FeS?O8mbeeuyM&3_w(REzW1JPYdRH!YRr1kp{2zPVZ&2p4HLlTJz`er!?xby}-SlzXB-E)n1 z{4U2pP^G~(Q?;c|hmsmddO#|6Mwv!->gUutalI|aqqq$SM zL_b2DT_LyBVMorhA;N7y4F9vx&5)8~ ze>@^&Or5sx2Upi#r|&Yff^8E^7O-3Z$-hAJS&MIy&q4#+!_U+`h8!fg@53J5l9PQO zzF7W`nbjIDT8fd57Jqr{6SrZaScPa1$IlgPWg^^5G@w}NTy*yXnbqNhfC3Q+l%zs8 z1+c+9ww|}#9@4@62kZ(LAvUJ>HMP#XHmzGYADJqFr)PzGL-7-l%n&Yr{O1ILCbMiV zup{L9{YML2Z4lBD?gDGQbFzF@)YIbBH#cAY1lBu;O?ODrUc6_+q1g$Rn2NdZ3smb~ zP{=>Fl>V2~U)!~+zG_io{}lpi+QLN5CU7GZhu{0Pr&S;|nc1BOz}{+|o~dxV|7kya zQ*7j2(-`qz=f)&ydZjDVG_K!2DzS^`wPm@wx7ELhF%l2N=qo66%cE@2 zz3@Pv0_Miq7#sODt*$$K13&R152BAKHXF%R1%Te3ht}nwVd>+C?)Psru}!NJmOYrfi(DwZii<>jnCjc>)yqTv|?h5x$ye_gZcpmtOyX zhMreUkLmi@heRdke?^#T!a#ICJ4lKt)8Y}dzws7FxAG(n>>(xh#N#FVwaP|6El5b{ zYf#>!cB`o+D(PPTnT$o%+}P@OrgciRkk`VE$th061tDcu7P1m=j!ErWS&OgKMF$^T zgS{!M)prV`x*Hc9Ax!DZ#;u~4HpU+C5pTM_pJKQH^!-`zZt!1?f>(tBNv zQUNxb$PITv0t^Aw1C$a<(oKEH9KjZKrBFo%`40UnG$A)2&Fu0-lIwQs(y3Ad^YupZ zJY6@6WlQFC%y-)FUO~y>q(fADlwM!`6Abtng)3sZ_J`*ceY`?tgG8x^d(zE+hr;9r z5>Q>V)jy8s-dA%-JVhasyMZsl58YMV0t~vf5NgI$A~Q%;Tz3NxHoWByMp`m{$&q4C zNqW2IlY^xaJHaMbB00YbSjL{|57ny;$1B7BXETrh*uEBS!c||yi*N#YQYW&9P0aLY z{LaqLzPH>w`@pCS95xZo#urJiGpPG2i%ssj1s6Z?|H!G5<@l2pU0VJKec0x8PW(8}wv8!hqb~42*y@cr1!_Tn?>>j|JegLfwEW3UN5fkkX4;@;%ZuSfD+8L@0e+!B>ziWoJ=fdo~AU_8~fUZ*yZ` zKz6WdF}A1d;tTt|{>yx?S)e44J)b0kg?J>@Y zu|yF6RgQeK895OMHVB88sC^YsW!=nGrQ z(yy5l!V-D)+hKM~!JM5;tY<~793lQocqRXB*pMaR2Qv5+J zC=}gs*tu`+)csGNB0@4xO#Y?gN9lQcHKRoSXI7ve*z03>)>^{vfCT*(OA2pWEyd=7 zS}j0i<#rIS055{>cHXim6Y6egDiEvbf{glin*-n2X~{u-P^c;>_J)63yJy8+#bqE% z^e?JSGCE=QKdQ1KCq;t~U7@^qEzIX+EP0c1HKW$2>~(j%HtTrO1W0R_s^9xyQ$U+q z=_E5dlm6U0^}*`KVlmrCHHU7vJP#wb99h947byuX5OcA;KdU8_PG^<~yKeZV)}F}_ ze}1NRT%1M(@{U(b1yA3MycSUQL0w}FXZ1$3`nO_*Hd_gQ$;&kXt)_MR1TcDsZ<-H? zQ5Te;oH{2gLaGcVtgHsw94S#}LThC`S(0LXVn`H70hw2uR(E*mzajUHaQRnbTt-NB zqDTw7gVgm69c2zj8F@8=8R&*`jJF#b?>=OpbL6eW%P@N)lC+mQ&A83gPEJ23v z;@MjcEsWx5SNPCDoiM8Xxm3eYt#N(8*WkqqOdY31%A>*=wb$tXsTKP zVsm{V>gs@02N1qcX%+}w-Sd-kV{hu%(&2W=RxCKr|fe_=c<)wr3b}?T*6k%N%kbYoSSD;D-mNvvEG>@*A0Yvl{|^ zvgo9+nXmb~8`eWtd-K`CB3+th&vboz{{T7Ta0InT&cPNfm`|5X4C+?!1AAQVG@^ zHT71FHblXAKWUYj)y8x7zX#RpI# zVtbPDyz{j^^2d$`$Ve~3rS;=`XH!@@M12c_La;zZ@(Rk6E(qK0w(w$o-+%9i1PJ#2 zbS4k6kZv2#IQ@7`m0 ze_1}#{*MwCP(tM3Y7Yf`@|N8qHC=LGGe}2KLgBv|CfeX9Ap-ZYC{bo>EP)qo9`{{0 zEkxVSXg*L87@PHp?>Q<&T?*Ve#%`sOojv61Gn&O5{GqVmY1t;!3gKtM<16^aM4UZiyicp0(G_$qVBKIMeQw!py$U( zJlK;qE*?YG2*sHR*8!*lTYx$d+Irj8|ju>LM% z{hpU}|Kr`M!gx-;L0^M7a-xMle4foMuepZ}a3?2g~NLD-+nKr(Ih}41t$>EJN z{BN-{D<~CP`=OLB^gCElfQfX_2HEGvnDR zugn`L%*1R*B|Bs)s?AxpfL@D8Ea_w{KTsZ;Na7;9L)5Uz+Mk!K5Rm+Y)1y>HN@tvJ zvvcVX1bmRqOf;5vn4vb@ge0dSI=vF{A$0U{a62C9{)A}zo#=9IszwpJt4S3!Ch(Bx z#r!Hxl)i_d2bN=9>L9q8mR}cjwL>=zG#JV^^TPvK*yx%~E&NjhK_=1(4-HI6gl*p&LO~%(rQ?y>Vu-5eAiW`HkheSYY*Md`N;%{}7EeF)%T8aB%Q0 zMkPUG1^LR#QL+9|jJjelFdMIc=kid1%65ApyHXt(13U|J+j8M!nW;PvcD6^ZLrDqi z9NI>M_A>`q`gQZp(EqwZuMj?xl1-FWbXuo7zHYj(Y7@L|O?H z+CBiO+Ty<}4QJpn(H=*O66oECvYW7E)aoW^qZIw7&YeVPXyjjkq@;FJ$6+OG-1jEx zq?WRM>I9S?$+wSPqBikjNeSQg6)GrUevdBVs}p_tEm$I}N$Ht4H_~vD#H6lJ<;cCP zNM%LHx~c?iGAutW>gtBHNN1~xj2NKPqI7GPaAu`nZlgKiFqfZ|zy$%v2s zP)NB9d}x4B$e-d�FmcyFL0Q4e{Zq!a%CN+!8Kq@DgSZCUG1YL{{oWS~Q;oRf4fK zrc~o!saAu=3UOofC%$!2_$CRieD5L3O{Vi_f=l*<0s_@Y?m)9@EBJU#L#PxV;U$&W z;1RUaY-l~9(wR11tf15=@k-Tc6V#B>P~D)Hg0)3`MxE=FON-_SKnetZg1g{P;Z__$ z4!8xB;T+UOmMCHab$cP)u$^VmilBt(dOkxL#0hZfTg8rZl1Zw6u_cUUaN34&ao zdyvy^L9r)}JcBN{+Y<~Mfgm^=-wF_9mp}^BI_QX8kKOwa@W+*cI3 z^xT(=5YBap@==|x#O9RPu4J2R45Yu=3HTnL0elw+p^~Jj5;3*N&v;Oi1@u9U-M$Pc zRuEH&cT_-4AWy|`*I!5J*io~nmbXQ@R)`_~6T|`dxf8-`8 zVUT@gzDAD7BFz3cy!|H4?!b&Bp|{E+d)?&@U|95d*j_4=UA1pELE9fhMVlu4D}Dq* zL4h5VdgItF$8*Sbs2A9@iJ`;o?dn#}iUs&y{ZM~C*LC96+n%11MP3wxP>b_jIm&PR z%_D5ie-p+XJHJbIrRlp#BMYZR<{Op1Kj~T)X2n&FHOEYEj?&OfFZZ-IlTR&hYWz`p z;H2i-l;GPU(VP?OsVm7g$Yb$_1$Cc;izG*Kaf=30VpxpOyov)xh)5AvIP`@k!z~hA zsom1%-wv{Qx#u|Iw)boPom&e#3qAt~AXDHi527+PVRJ^*Twto_O_E1Ulh)%5IlW#hK-DhTc zxgBv`L)<2t*=jxr9LYCv?68sW&w66e0cKB{8_N$LI*>_Jmo89lom7*w!mW1u;yk@G zwIR%;Ge9wJ$k?7EhwIS!CuWYfcE+`pLK*di1BQ)_?14N^R${kj^1C#Tn0G#pEs4A5 z&S|_jeRATP;6<%p|47oF46SBSp?%zt4{N) zv40RXD#jZ=^4^$oz@lUU?BBgW!f%0xl2NCYIYY#}Gag*;&+&}sIrVaCP1Z*)K90&c z7^+L;9Y40S@$^Z*!r7naWd_=OOoUgjW1U{y0784~Xo$zCip1GLTlIlXVf}BfAI`zOIZ2jPr+!@rgomyHlkhWL&%n9PRce5zd+@}3o$&CN z{<^kn))NI;k)`aC zLRZl^)>-+-dBdzm6mE^T`$QJ|q2EE2Z~h=4_X`%8;FXJVHL#dl8`Zj5SMWos?D>+j6*Fbiv0tBH3` zx_Mcrj+igE&933KcJiUokFXtUuO~1%x76m5XVnjIl+3#T0liT64ty6VI2%$y9G1*> z4$Z6&bulp!jvD1Qn0+`~=Kb!PzTdY1%Xu)!FFjxvgLog+tIvhY&vcm9?dQeFc~!Q* z|E&A|JSSJUaYm=;bL)Es3|TeGGapPt+4p3r%E)qo-EoiI0BWJ}OLG@SRp~5hRL%{z zt9X9;=p%ZzZj+7ko1CyP(S5bv(~vZUviW9c4}5cK5fhzBd(!wpVi(kZ06FrjYInGc z5rZl_uk&Q~guMJ+D!H!tTIO7^3KS=G*yuMXgR4PR7^z8O#4UeFJ+okI!E-}pdS2V8 zAtU8*3$$_1(F|STgBE^RMyHm0q}&bXTJo&91D0{d$^ zN7v37PD|15W-T)>CVA(zN8w{f`}u>Ro&@T|9Apg;SW-J85@NqibxE1Q@DQZDfN(r+ z2Xs5z;Sb8RYWwee%GWv@0M|G+uHy*T>&W?DiEE4KAI*Sldxw2&>*66w1(Xz_Srh+P zzBTy3?PiL^;e(ee&*pb`K4&dISOeDF^}0?zgG@V|X6Rwhmrv)F=_$e5dopKccc(b| zyrdt#TsaC(Q5U+-4G0j48{br7j; z{-D*Z>VK`k@`Vk9F4g)_Lcr|Hi;vWbvWZxUQW0rpfC)JLc&yHDBkk^1}!jY}G5B*cc&{@Q5 zbwcN$ZY|GoG#@=soAPYl3fVlJK23bTLvP)@cVUGDpvV6OJ0htGgBOR#r zlN~;QTX&@i(r-f!<<0};euIld`$KP`7#N&8D5j3r&7NbIU+q%pA49k?25A66i07{D z0P(}HwwTcalcy-U+Yf%(xPYFFNq4h&Zk1cwzJPV4L$ICeeY@*D0OBxwULbl8p@&BF zbvXJ)kCtzTr8fN83|8o-G*F^Xy;7ap{d=>06bJ#n8oxi7t??chGE?WJo;i-q|3JQu- znWO$GXtKT{Vc_GwMZm4uV^;-2w7oLxyJkQ}QGCT1)HW0-Z{ZD_huj@35I(v7sfWN< zbUI_$gPW(`tB6bU<_8;%{elZ4*#J=uP(PCcksVAbiudV1cYNFp+%8gBg22JBbVW8Z zajQXfchpex0~q^+)jjoCa|J|#?JjZr|DFISt!Z3nMRDB!o^Wc9lA4)a{tC)X8Us@J zPc)K(8zfTOgL3#>1A!-C4jzpdY-*R^Xr;*FSUnY}<75$%G^iP2fGXb@t_u1Eib@-x z0np3FO3Mkb@-DS$t7M4sXOhFtlX(k$7)}WwN^zFi+ndYSMLn&&p|1GoOy1`US9z8KFp$uRhg?94%{AAwi9X1&Y~?R!@C382gNY@lW6HUMbVxQhR;=JhBZ~sje-7(z%ulO`cFiL7J89D@!MY z2V(?*g~DchWf6S@J!JN}SIRH2z-jIbP_lRsyL3P+Fwu#b&%a5WdoHZST#u#s!j!a5 zH>6fww;_;_C^fy4Xn??Cj9;iKlbNL5oScD#L5-s8KXV35TLjCrX{I6>T(d;z%vT-6 z2GMk|TX^@lz6=v&#?T)pv(rtupCIQUQTGngydAJ22^hxk6U z=)|5QH@3WTUP7!g9Ns~jgH<4naIeY!)cg{%tDk2`NZuk$@JP#pxletKzM!cN@RC&9 z*{<-3YoFGcTMjAj9BVUJ)d{L3)TYqteA^R30e7i?kMoXj}u^Rug51Eh`}ayXFf`m zK0WQycl!W34F*+ZMhi(1MxH6GIx}t%@oG|?){cj?qPIVs-3tpuR{E-^^j;@Tu&h4qo>tC;24C^tZ9T`kgaAz zT_k>e*JQ&2z2bUYDbNQ})?(2gr@{bK~_0qU*?}tR6H@9gYzFd=#+)_b{19jXKVt zErGqo@ql@mIt()Px=-6gSiuw~lMZ#x**a|BcrhHR3uRRo5E^I*T!L;O8iC)e7Kwf{ zqjZA!Ak}A?M1gQ72e7%!tjGftX9>Rxt%~f^_Ra&`b=69-XP5f&kk3aljpEiYg;w^^)F5G%T>BX1%^T-{@a=Syuw-w5Z zG|TPqoHX=8f~0FBWdXPU;BG5(ob6>+BBp7-_=bBeg4&|otf7*ELN&5F+$~7%(dH}e zlcYfc#Lq^?5)uTu6cltMEb1R2IX$|EytMe6G%Xz3WG#Y~tj%|$SaA-}g!}by+W<;W z=ncN~8u?0KszYh;1X4vbn$I(Hl!4OG*ls6c@Zoe;Q4-U%MOVq*P4Jg_CHLe&dc}f* z6aYq`&D*iVAl9mm0mZ~_bAfMvXXC}-it7WhD~p+XV=mz4sVQKRJc5F{v?!aArjzp^ zpHAs9;5*xuh&S&KcVKbyH*nSY;ovBU@C8F9`pARIU269+AUap2^e@eI}}0j^@T(l++GVr85n!cOJ+mi;63zlM^R2nPb1xm z%{}an2xQo`qAXh`6P4GZWlPmhwh3-sfimVlC_C7>Z2Rp>ZED&K&2JCg=D%E+kqh`1~ zp0qI-#Q9E6yA~UnCYukPo!JSxWDAiXwy5!bciy%55%t{l1z)P~! z_`>;f$*6Un#kxf@0FfS9lo9L7WDBaH4Mr}@!sm8-DT1JQENrM%AVhso*ENgO3pQi!!eHKcj;Z!}xH(Y4q)!`DrwlyAaOarBet^wEZkP zj)DGZ3g)Cn2FJfMd5tg$a}ej+3=QCFz(j{dn>L792HJjI)QQSbf5elz0Y&XpO*uU2 z?nf|KQ1E9AvLcYM?KYCjip!qjA!p^#3oVG zJ1@(JD*&z>WGQkqI9}?CoQBZ6cyz!)I{n4YzaTWhRt?~z1-%d9RVZZYs;MUIBYUza?^ z(K4Gvp?s-m_Uq;FdaJU0PlOE--cqSRfU6tulSjc(uw+93LK^DYAVqmKBP40K(hQz~ z+no54@j3w~Qg}Y^widFmxNEX}Z6<7#1QO74crU&hi69Qz9|CaYn)4pIdeksvc^ClZ zr^-u^uC8NsF$p^X&ENi-#a@NO7VE$h01n%9`P#fW8$_ZFax`|19O2v1XT%`bi7j00 zMkrwe+Ja>DG=a%vz>$pSto{R5D>IL8_hhh_yC@Euy&8kag+?XV5V`vcv(EAa9G z4_%7$9uZ>mI+51}5y(smyol$aCxS2ru>q7^~jNn4xGJhD|=jUh>ZeCi` zM8NHfqAxd~za{c|tJqFp80n-H;g393?Lq?U!?834G>x=J-U{!Z#rwM|stu6OL(^!M z(|fK2Wuq^LquL4>z<^1Y$2~6j9iN;RGRBG`RiT1IckNY_cksO;0h^G>gLtz0x#V{e zTAQ`tZ#2$+O%=idq4Dq&pBGOGzk-Knq?6b0`!$om5`3PX?cN&eGy_%U%Q4VR0Tafg+t-73bOHSjt)wqOJPtIyAc}6zFbbf4 z6{GKJ4ee`BMX&f)A_jfS=gjGtra6$XDqmDY{*%^;kp_K=w_zSccI1~9S65_Cs_hKS zTDHeC(E(Z<@hn`~(_&-> znyo1jsmN2K(Tr61VWAu9KGv+LXo4nd<{Mxv^-7RuH{Aic)MkBcKA`mOEN@D>0nkw( zkTV=U#yh?e;-OU;h60q&3 zE0jXVT7Gep%!w=Bk}eJwt`|=EO&D~PI(u|3=bK!1O?S*D7r-|wb^AQN`OA63s5THl z5onuI)bdcB!cEN&ImLY`xi$Z}HkBh_1W_}18oP1t2H(zs~yqy)v&ng`I$pD4k zf>`LmZ_%Xj+8#fObbo|$TU^j1(E&;lhgohn<&&5Yw%K>p=Y6w(J5u0wLDG7-lY(JE zZy61y5MPtBqk6!O@zI-zzedtEGT>S?e3WMf!(Tt@6fq0x(X1Yj4OGYXu8|A zz2L{Y7~gRtO~C9S|C6IxP`I~0{fEnPwFWnGZZNtOXT`Y)`-;Hb46A1gn!OCnM>yh; zFc7am_INQz{#Xjj5K!%*au5v{m!ycZ*-t)}6TLv^_n`$OF;Kvlr>(x$SpKWMtp*N< zupTu>EF^as*}Hd0?6TFU6hkW*qUf+zV{^NcFYyX|o)tfj3E6!zoQHbp#pjDAQv*u0 zCye}~%?Y(OlJWj|^&fX@e_~XLU>8gf{>k`F8vJCQP{22NluW5_+q6UcF(kl@|BZmf zFpC~K+VtY<52M_m(Vyo$SX{J)%u)bu^@kG^e(cQ(8d1;o?0&$oU?Ih`@87G@#!nC2 zD#ycjatVj0L&gThcDc}kQS;XklKk}g)75>p>G0b3 z>XyetZnn6FurD)d41a|?dW%Vve~RIoP+?o^P7&tr6QcLynTualhjK!!OKK@0WXXGV-FYa;A)kTZ#PYnHIjx9FdX@A2hYYK#kmok# zLzy$SQ<%;a^aH_0gASEYk{1(zR{q|>9W;wp)8CK~V>X5zW6$x4KPp@Y-D2g+?r~u0 zoZiGm*xT>-{?wql(^TP`gNR+^)`tPSMNh|)1%CQm`(DIInc(rnE&ed--GQomv)MvL zV{Bk*v)hAj^@PMFJ=B=W7eU9ts0N>(bX`#BMT={(g35yo^bFT9`2I!6iK(10|KVs9 z&cPseg5t#*m(2K6>?KZobEd4eH9(dXXWMEDU(3H&FFGE4#kc9$JdPXP;fk>-7HM8G zvx%07Uvm0IV*|d|+N6VFT8*xheHV;{&?CEk;De-vpX7W+1Y#PhY2YQ#?hTbg*H3>r zAc+nhKBMpRX8St}N_Cc^D$v|}p5~+`V9y#LFnk9ZpA_HjlDF1|xqDUFcXQ%w?vsJ( zQs?`ZS?+BL3XY^@fy4kCAnj#_XT4`XF4x=iUcLRex>~^ewr$?LVVbXiJBD64Ph0}^ zyqEVNDAtFfZHn>(>Wf~CT|BG@SEbq3 zaV(ZUOf%-6Vq5%Vh|Y`04Y^rI7{2HQh=~aC;}%8!xmVlY#j`Q2Tdb94DkAja0wav3 zMt@J42{g4^+aiRHfH5Ej*7nm|*J{1tY@yik#M~^Vqe{|96ma_7WNqTgau97|5N1pK z+E~-Gj$y(FR>2JWt@GTa219Ks@^Hxvd@v&vdjdXr0Ou5bc605xj^5v=I>s1nLv-L6 zRJe?;KjdM}X(+pQOlmSpDE6#4YpYrCb&(9Y+AJaYoT0qrfs|FH4m+(6`LyS;0TKkZ ztOB+~_lKG=Poxf=O%%|OQM6#xtr^E%gySNA;+H7tQ5Yn^=I%M)H;Wd?e8wvNllS~! zS^(4X{!nv8!p%*e_o1bOW+gKRppB&0=tGIk&2qbo9>Pdc0Fntkve~IG#*~?fb%_E} z{nG7DCCmxfizJZRKdlsQ=3FfosYMUF_Wo1ciQi>rcNip%-EU#+R8VTOa}Sf{YIPQr zJSDwnoQ_P=b}$Uv04^JYr@szbQaLKN_SI zaFPXpEMC5fKWxNlk9*@Qd)lIEeV5lsC49ub21Vq#Q8hqLn91#LAk_3TvC!P5s zeQ)%*jN36cEX&QxE&P4m;i`u`%vsn^$Y}&lj5|8F@;~=24!F&Lju*^}BrQbNxZ+LO zcXUIEF24C)AQLd!ze=&yFX6XOU%F<|n^;@ui{CzcSz8%E7;52F3L zC3HZwz48&mB&GLX|3{ZHcjV4V*?F#F_nLrq7ZfZKzXyTmV8l|=#8y)=O5ou;M;B^M z(hi`?Qdz#pQLuzem$c?|e9w`vJ0a>!xZrni&WqFUgJS7=-Q6GK_IdF}^kP}d_ILUe zWr;|iV2=kQ>jucv1g=u&p8?o~N#K)Emh%iT8-m8XDVB@If8N#;RisemqcTjnljB@0gxM z51jQ4e2B@NolY#Xs&^HzSPzSPy}=nWnY+3l->S6cOPbph+dRL4)V(4sp65X6s50X` z190#}uiX8f92+mNHe|T~(pPnSX~JzB zs95ULc{UJGQl{%wnZKcg*vMq*AYc&TbUed?xNuI?pY5{*$h#wd+;+Y%%Mj1DCDtbJ zE0j6=6$7K^8!n_xV098dg~oV&keuAeP?LZYTlXqn0a;{S8cPdAM=BEJ8sHA63WKuw zhGqBuMol=yBP{yIfjamVqmZJ)?9S3wIEeu=HRYJDk$_mnyapy=Knw{#Q%XxMNI;u`N7_dk%ohbJr^6NT3Pit>ywzg%tEp z7-?1jNi3LE7tt5j6I0$%Z)|)j|J+VI&VI?Me_@bDZg!r)6OSPpgudeGJ`% z9hJc9GA^`I0k&>C?5LzC5$n;90;JCBF*5ckBUc}soJPdxtuv_=d=Q4e- zEesl~JzK(e(8qKEzJ+K`K^)0c1k4Cia0S6?OVS<+r4RquJ-^dq)~qVHhzP^5D1l_E z%Vsa=shSS33GP$kUV{C~bwHEIQl)xcvb84@*kT9855JrYp7gS7=slJKAm8w--*N9^ z$Dskn6QB-cdi0~Zd!Z11no`;9`7@^ z!Hjr;pNa`W(O;RI@>>Tqa!^%r*dWD52?|tx(SXvU@7JF_XVmuTh4OgZOEwdffOLWo z`E5e5JYphla|D^hR-I@&HpuG$mVGy5kP-mGr2=7$fH0mw z7!ihzfJxef_0WW64Dup$(ig|8o)Z@se5;^zG*ni}C`K|0t7y(=D7cFLbM%6-l{RBZ z-aq>;7*G5;A7yKD0Su`Sy<|O>&h@o5V&oV$O@JKTSd}p!W1a!xvc|>*Ng3H3yqOs# zPe-7s;X88*Ru83LwI>lk5+<^|1u>*GDXhEx(a^E+TdLqMFa*L4pBloTZUlmbIyKF& z-wZ&a;PN3YJ`}nLub8uNP+zAj+t6_hOymfEUgo*TD9WPuBxDJi zfV(!Wro3V32|=!@=b%x^4D0?l`3De7qv#sRBNdO{laEjKwVpI!{2TtnAstn=8N_`& z$WSI+of97~22DCN9@}!^-u-=;3QttwSyr^)SYGbeo=akL>vcKW;8FxBqXCjR)h^KAl@CkP_vZ{ zB4i34V%)Xeu1o}sUGcaNY^n1E^u{f;34ivpwUII5HV*?B5CzbpL2tR5w0Qs?%>%Ia z7E?5UWGg`W%RwFfBs{vzSZ4GG5E-02a|>Jqpt+f!k9Yh4gGSO9oP26`GkFd)Lcm`I z7QA(BHML=ce`F;WJnjf2JM$Sb76y`H#LG-4q{DlCYE@Y#kmSc&&7{Ay7I-rv$nu|z zeh2~pEuOxVcR=vD$SXv!G`W=b%bzQsVgg5Ah4||Q_*LXE;C1}*!tE%d6M`m0!8Wz3WRAc?dz@TvkF5XEpGMxssM!+aRc4%mT82W1)h zkPO~fp6`r0S+7(A4l&~!}Y&(A-c^5hn1dBB9 ze!TpHRdPaET(1T0RrP9LsIek47C1s!DoJ}FbkTy%bvY=Ud$PdwW?e-jzlW8x2f)V5 z2t0evSZq}M8z1lI$!*M&sZ;>V{dXOLw1N}@TJwD%-eo7F9g8Lifa4}v*%%;6As0?+ z*ezGtyij6fl|I3}?ZJXtQoR^BtAUpp7nby*9_nG=lNL zy%b@fLlW8{G(AYjQ(ZLj;s2<5?szKy@1Ki|th8)W*GyKjXGSR_DW&WYO(Hv736)Tn z>=i{Kv$DA;LZW0PD?;|jUcd8PZlCY({^8}`=XuWiyw850^M0Ra@78nXDqBe*?+XQG zoBAv#+t1mPCP=)&T)5)@p#}%@hV`}(00tPLX%HhUBuup(^Hw8u@f>S|o9l9Q(7pj{ z=Dk`07}=S6hV2&h@TKe36({mIxgl2Lx&=led21as zo-OU_>MDd(WOVgx3edf{{#sgS3&wa_w|`ZBw{qH{jT1#Wj?oh%(|Ollvny|TnGS9c z?g}w6j?WM-Mo4Tw5(3CTWjBmm(6=1v$>65X^671kb+)O zN|mx)_xfCW@r3ekpV|Nsc2uxn^$r=XwXCfS7t4KO!4G~u3X>A*g2P)!m$IE=Cz_{| zAKqR*N|V=}Z7QB<-yX>w=Dr06XaD`?CQqG@;7oEkpDZl5w-1sDKn7rYpmBf_ zH7Jp_zdWJ~3#%xRZCqy750ZF^g{2a<^F!hEF;49AiSC8rt#JrT|Hd(&TDLHEOR6zf za?yFWWuuWAlAc0u`J*Bnt;1Mbu!F=rCW2J2Zs{c;v2eb7Ap^Vl_;&wA-LM-#+U@-Y zsNdMakzo2Gn07Pk!$=6}uxW~$ldEgmHn$MZm$z2)v72X-1t(MZKt-Gs8Gj6srf z>sDN6S1_%#xJcY1SD;bD){og>Lboq_LB;%HfY$Q$?uD(?2t52xRLt8z@%h~|K449N z2itY<(vU56-2xzX8WKTaYp*}N@Y2Z}j~ONA#$gAP(LX6b@8ISuI%hcxr|WHqPvIi} zqnt4ut9)Xyed~#rRG-a1F&C2ur1>G6XDH%wVasjdm|=;qH8vp=(y1r&3@u$4 z@W$Ju5rqS?+fyrAelo!lyzG7%VzsR`nH%d?2nRQOuDNx0!SJC`$&;cX54D~5=eP#9hd(?H?6DItJs_aFr9M|nk1iUSTq#Wkofd0I)XDvnH}L-CMGd5KftDl9q7$C{{JA>8}1%28~C>vz@eu#w*YR}Wl||_ zlkkI$tXr3G>2jwX50Hy*y&wh0|DS-oeq6uQsM@`-m1%%3m;Y=~RNPil;GVLW%oY|W z{=?$YD;yxt$gr=#zw^}Lp@&9+_kw3+f7tuQ6BL4{n z^7$)qYYT!nR45YzFGynq0OZaP9^JgM;yd_3^F%ht?$y|nIWYQ(a}5x8^WcWN*4Hhg zC+3Q(!Qn*oL89$+1r5QNq@|Bv-C8}5&%d!L=95HCD5I-!d%P7ThN9l>y-x|8h%#it zxsxP~rDKwDZn(tf@lH&iGY1imX$P<(_)MtolAHTLxJ{%D-G`hljtewSPBC*^UZl>6 zB+ZZ#%$^p$p>X@G*4|{)zB12PsN>4!gSG`a8%)Z1ywBF2RY(V@cQ0_`*A77iz_XFK z{~a{r0)&TX>0yhdyhP8K;cwpxTd}sr?t8{UEW#iNZzmX1t&mpafELM5_Wj<<>m$gH z;>S?*@F8jEbQQ=;=vEm7IrF5umPzj!(icj?-t-Kb;n;Y+h3J__;9q|YJV5S}_9PP; z(6q**X(N;nxEHrptCv=HAjKmLxj^qMMxxRa;@@ z+Bz9-mBjQ?I-eve9Gdch$FU&$h5wB$^As`PmB`!20~yM*s1SD8qAo!8EIdzQu7UxY z-K9E((EU0^#zokLF60QcOc*!F6vDqS?O`}{Cl)2Tdm3RU>I7i6B>UgRQ-~NOY`H9n zfjO#M$|RpMeRo5JQS1q%5QX}P%}M)IQPdMjy6;vF;_0^{gjrl7F2SH9(M2-E8xEcj z2ku*5^W`$X$WBwZl@K75LtF~Sow!Kmu#XkU{~AG;vOl+s(g83OeX1nVCDd|{U;+xB z;<%T7kSc_*C?cql<}UbMJqtm$_5vt09Vf$CIO{lv4TkN$Tf3z=TKJa~um^_3%sEJw zi-nw~nf649Ijy$gjUo7vfGIS=7XK76?7-zj((1DVT)imdlrjGesQ{_Z@(o%sJX*oI zd-k64Rz~H3R^AiX{QHkG#V~$~`;cH8j4_nnX4k*PBMVQ%od(7ZFovE{1W zjze#fl*wtB@FyP~mHOtCYZ7Zz;SP4QxrbCfoMAP10w`a4E5Zs#9f&p@FA~F~-uUHVxt?gxMA*TY>sf zNw&Ypi3?N>JsIlRvL8{oPcjT|JqaCJ0&v@EGiNM-QL)l~mQ~WVkof z+eS&D=_CyjQz;lwCUu^uo%VF$2=3;8msMyGHk;QJWpfrvTq4AS;(hG8aQgG`KWtM} z8JiF2ijTj1NYuTxzGu6U9C{L_!?G91a5Y7zW;Iwn`n@=XP~Qo7K5JxxgXSbWBOc|k zrIV@50g_ExS?81I%B}YicT@LWCg(-WFO$aVVu$%0ZLF=niq{kRdbLH4xZRmh5C=Q; z+bFinf$!4*p&NU6M>0d_W|{?`q4Zy52d;PR-i{4A5^y{!egH-TuY4{!jBkrYNxp#h z5fa`Lz!J$`$vS?Uk?`K3f&MoVD4exd0vRn>TOkbL{lDSPgts23dw2?w?7FM4#*xgjG|n&aOT;byz1%WiJZ22)PG z{jvFl#5cF^zzR2pfA|55-Zt6bMeCbS`M^}JZ!uSJofb*Z*j9aWc@U8!N4XgzwfKa- z%5i)3qH{xMWtBXd1E_O$g1Golbn;(N>0 zX7W4L0>=Y%=0kyI21J^@88JhqZ;&^Rgo6zTA6+c+9g|f?WK>0s7Y(U5mbXGt@8~ozyTexcem7c=-R_Jqd#OO79n(>W{tlbL=!8)jLAEKYHux`S4bMXahh?l8aD*`fD$C zBAGC+rnwo1m_g;RyJwze!-2S0*HJ@H*nsKf?5PkyeGGg}p*)2U76G6{-7AJna9_l1FgDA<9)b7Yn*g|lvq8$ZnOaCVfIxOH#_KqA;?GqcIJnyA5-Ql(scw5D6eLr=y0G8K%R#riiIGt1cO%o6%*bfq54q? z;$vYE8_$$@i?k#_E(*%4QN(I5gtuP(A>T`(QvDCPBqcaGoM3~pYu}gC4OIc$VZsN` zq06Y|bGo7w=C&j$;7V0xBPcH+6r&8zUdP?Xqbcs6iLp1MUf6x(&Ph!!CiQDQFC~w z2xQF5+<7|qc4I|S*-Yhq$I^;a-7xmFDpAqCumB@U4wB}d18U?T41Ck~DLO+BU=gXf z!*IhadYlve^qtT!iszHvt_max5*-9mg53t-P(jPPbt9W5=x031KourZDiVKhmP8## zBxz=o{M!#o2BUl+nFI%AdJp35Nmot5LHrMRr)}^t$pRU^Aa~bEa@Q+ZUk}QKepFh( zQW4dI_*MB^60t6X#1K&;S*NA1U>ea6E&wb8Qt~edNCCS~{LmkTb%~1(DPB9lydAzv zYUmrLX?;0IhBF}n>mWIstp_cg)R6@+EW~d_UrGFrO%rTQ*{*H#xYP#XihghbV8+Zv zNi->csme(Fok$bUf3kVxA5FeI1K$1zCVCrR9e5)h>b}`YLU4&_eu%KPh9;TILWm9i zXNW^2?%9G_KtDvO&XFe3)>U$ZsI9ovuN8>7Jqgjt((U36Q4FxMP*qqUxwr#b`i3r> zJBZL?t^nBaFp{OJ|9^B}C5cYXFN&=oD(NaJ$%jdkc=nGSfF{9|WE?Xg9wL71qK<+@ zC2k{y>p8p?fuYw_{Px-$gZkM5iH8S@h$~Id77KI?dD0lxFG%x7qbx5HSDQ(KQ%e(o zjK^c zBwqiGCygNqhM;d3X>#Qx%!s1wM-qV*UB9i@lG{B_&eRPGYt^TGDt>x(InSm%ma3$7aqgj zt%(DYAz-Ob%y4gRELNPyn6EN#uHK`L6`g0kI>j(zHVcRHogDke+V_W(HQ$@@c+#rA z?(vxG_k+qx#Y=&^n2WmZtL#CN%BCX5W)2@=Es*Nvk(RDDYUA$V_D${aOVjFkt&T1v zHqU_VND|1#0Nlznb!jHT?8$%sR`Fha$dBBeIjx%9LB_UJ%O$v}Hk52x>;?i73c z+UwnoM5|TiE`52?jdIx~STl0CNonxbtsBv;f)8kw>5*Wz#B%OP&G}7h&Z^LIGx@^K z&7$*MZf%No{oXAzR&h>pKB^{pUNS@^(?zLx`w0vR-G z^GjZ-LV647TtUo6I~am%5mD|)%K4H(0ax~xj<})UQz1ciO603$KD{*x=2oc0stos3 zN)0zsiU+zRIx|9Vf#5@!1tldfWhr{gFAnXKm*oq;Z~C5?(y8KX1bw*!E4}8lz)Af{ zzI0J-O3A9Bx>F4u(Tq6m+Nt?lxG~0o#YwM|j$zgoySe+)tnhoBLZuClNcV8tLwIa! zgE4h8Yv;HshU$df{A$|)*D(oAqsEj@;aPVnwLsc2K_o#@%By!E*CmHl>5B8O{)6`d zbD+Z^b~){oWCV4%g-EuW%Ny7tQ z7pI=>S3TAs9h6ZjO?J6w_=MJm;lm*`y1*HhS-lzlswp1oLd~5Q!?;I=!p-f*rBJOg z@%oPMBHT`{qx}axRQ`@_9MhWB<&9Yt7J+$3!`LsieCGrAV2kU1>Ek9t2K=y*^MSWZ zjoQ-KdtlY|(h=eGN&88+_nKtEYT0cj{yDPH-bMbeN%oS&>Hd|6pwj(q!^+&l*Bq^8 zMO`wE=^QGIwJ+!SYQ~g8k5P3Go-6HLj$N1IBbWElD||IY6Up?;E+|y%iq@6m8!8R8NsTRzqNN5d37p-&t2Ot= zu=eMYizw(U(iTh8g_<3YT0GS3HfWn_nU3npTv0sTZl9L5>1<{jK4`4LHfCk$WN~zi zAIA`?73;+{fWK`&mqTX7ZBkf!B-y7MavLgXAUHSO?M$b{2CWZyuONx`YJp_C8jC_u zBGbHo>%KGCM7l&gLGR$XjvYR&?StBaxc!fn*&2T?ei3k>Ugu)pFJ9YWM;4IN;+b$o z?B-%`LN~j!UUZGt&Ktw0%O~b8{_W1XZfcg+4MFt79G>QQ@vB~rS?-Dnv*YQE>X)Zn z69?{3^MPxV13zsy>o@zm<^!(oTEg`*wcdjn@sA=-IljH_0dB~}NvG-HjgHjoK2HXP zVwz0mUzYP_n}n&5Cn}zpUpjQ=lhsDONlNGO!y`>H&&9uiY41y*VyB>s1#C14Pa>DY zTjTOtv|>Hu8S`QcN*70UA9nF!rl6UFv`=svK_~i9^rUO=9Mxi{~i})lTS6%c$K23WW7(aRDWk5 zwN=vrSDhh!$r$&})nCCgbaKj9)=u-rzyw`9dRypHC;nm8?Vf1;+N>*TY*iwoW#^mL zk@AYX(cG9#vKXoT^p5Sr;#|3+WPy^)N-~e;pXq>~Au4V7Ltib5T{0orv?r~;5opD3 z&R@VErXSHX9%e3*e{&`$rPpfA*|72IfKblVZQC-f;6<&4`M4tQ3x6|dCO8l4=3fez ze5EW>nk|`#;~D>J`ifP%i-tvNEI}d|X0>?PFt$BfPigI#;-_nGYMw2=Fujy0FuFn> zqPX8k{*yQjhhyeyug0Xue!rE0XZ4bhw?S2zAkOtzZ%hlbhdfGhr2U;T5NLHCvM+H$ z)%sae^M$vPgLKUDcCua5CNJ?$H4fa_bnZTR#o;e`a@`K;9L=9*ES}iM&J1xre@=0d z(L40}|B`9F%vF!Lu;v)&92L?H{Pkrub44~wwQtuz^MT|3lB-2`o}O3!R3ac+)*C<6 zh4jDL$ZVo-#HyT{0v-n5J!2|Lc1%krgNsF)k6t^qVq8fKMK2BG)sed{l%eIb$4o)9 zd6(OtD5s(srK|kTF|(g#gK~H{JQP?40h~shCiNSvx!d$=8pt@mc}J#ay(C${coTGb zfb~-9Ue6-$_UlrqU)%dFxbdHj0|{KU&oij^eJjtM;mh#bnW^#Xx#!OVu9zz4S7$cK zF1a-^i@lfv7vQBVWA=Ry`Tl#~c9@+Fq#x7rdx^i{tjwZ8cv5YW5e6Bj6o4H;RYbd= zt*dk4$6iT__wJE#EcSL73$>0W7O_tSB`b1AIOWs{(|wEQ8_j>{5^{_d!>>}%byoOL zOHceQ=CnRm>#}d|hC+8*=1}S1JGjUYwK00ESq3HNH1B+IW#-E9nok?uEt<BLc4LbaFOK^=3$r~M$p-C{=ElH6eN2z0Z6SK~F z7&I!=JRKWcVQXUT%+@d9x=!2F+-y%4@E7DhCx~i7OHsD3pKVg6d%D*1qX1a|M^7X+ z1sIpH_p({OAN77m7Va=(BLT{?yHas1wj*^V{lXbJ{byxImb}(iO}uv0`g+o^aIJB@ z`Claroa3$YU;M*wmKfIYA*k9o63Faj)hwyIcuz-H`oYD^qGT86a6qT!Zf;u}J``})Pcgsqu` zx0)I4pbrjfo)EusHdMUl(4Ur){w}QjSnjPWd)O{{bNAArTtK%wuR7q*`H`KASQmCEf%`mYy$!>l#^E=r)@XjTk z%)9m1(iWrnB0^Ew z;2-9)c^spp&%0!hG)&&KDDG-eeff9CDA(K0io#Dq+OK>t*!Bv)$VHukvCxh?E_7@keyDZ?>J?pe!N zS-n|TeM28?^iC?ZjGBn~gK1il?tOyLyVRtD*7Q18ewyuu>Ll6yvD8xkSFjGg9B19y z8~t~wTQ}!PD)jX+(=d{~y|-AJlHPLSz<7(%q=$yq?1f3Pz=n`sGf+H+WFHK2;x`3K z9?AB!pZ@gBoQ2tsVTB&jAQ<4Obi?g(cx&3v_!}KHjnP!F-o@3d z`kX>;Xzw!SHELs=&&b#mDRmD1WVo64 z)&;u#2bzT=D;YiA_WOg5V@4tyJl4jlFDk#@%(5+?xbr!7W!Tg57YBn}s&<)j)z?hF zfD6ve&^w>Y+YsF@=5Ynz%5mQIds>rmZ7F;A!h3-XINb_@01iD$-0obc>hNoruT14G zk5@`{BXYVDWJ!sdimYR{M)4xN#wRbt^BaAazecvoZV-2oy78AftR;|@@k|D_w>>q* z!9F#m?CTB*gYOyG$ZEO$t{BaG--l8I4oeo$EI+T_eG@PJ8l$2Y>><`B8dkdSCY4GT zUwXLy8lO2vo?DNSz0rm?E-BiouRZBGn%ZtYl=6Oa$7t|e1Q2~gR%_OlGiyi4QA(4K z9*}qK_lu~2@mP?CQkM-y|LVl3%!;eBhL7s`{a#ptQ<3OFW zWW@6$FytF%K3(`xTMGsmDb&Bb*{5M-Vts7!N;jp~Vrn^;NJ(4%4l7_sv!SloeXduo zT*bjOWXMH!=Np-3iI)V7PbS9m7rn`*GGEOde5kw_iF1bv|bm$ufVJmFkr6?jf-yUk7=cFvsnoY~) z!mr3J<=?_%^tko-jBcKqYjY!e)eNPiPpu8ScT^$3r68 z={rswa@R5kF=`_MY5U0tDVm%tV<&E3<53vAsc)DMiKb^{T%4yxMs=?C*f_7SHgiHZ zy$kw=1_L|*kyG+oHMiryhgVL$5^l{}cvq=V3UvAo#z5jbr_1ba?6)9C+_bOjiDlNV zT*XV52Twy^BN4y|tsK;2YIcvL7ijST72!MkZvdHeh^>2Au>RVULB8XhIl%=`NP>V+ zU9j(?RkJVynqqlBNTD{ei=P!k2#eoq(^~w*V%^qM{QyP6s6t&{aJ&5YZQZY0Eglf6-JKesrV9`UbHy~t}gFMn0)`wqK zA6|oiH{22OD6hC)$oLfxeV43O-Ye+=oKnS6hDUkuHn-t2CXX@L4YJSX*!i6R`_g3Q z)z}eG-b>-F!b81%FHLP)+nFS`XrTf^>^M(Rc{5H4s-V|22svD1kY3M>}k1HM;wNM<^@%Y0P(G{)rx|1$L*(s@9b&EKe!w6RQsMK=+!3CqVz=zG0m3q z$8&7rtf9vCbqQzDO;@Nh`<2;!h_n|V!=8EkL&=KdnUMzh0gW=A#xCk~@?NaLrHw`#MFRLT#$pWOv>CxH&3qa z%%@%69DZ40rAH~jM3l@1?N%xIUQ!QR0pGA8)4y&zOG5ps#$zPDUBd|KD!SqK7(#9` zYh7Uyrbw6+&S;_AJSh>67DXX@i`y^rLlf^TI?PAk`+2vVC8q=j2+rl-NmGwiq0Cec z2I$g@gaa4V`=E^XQoqEc?fEV%MUI;5{H)(9U2Eq56jb>WoJt>Z_5&(*Pa$J@VrjGV zq$7JJgAA6R^?13F9#Pv|bP2mISz6(L9M?xW;%g+IgDiZ0&1#+*Sj{By;_y_EUvlPt zE$<8Tef|(&lTl5~K{C4TnqT(Omw|qJBwn;jRr$B=foo`n|HzrC;(`ZS z*N~2P`f4?x56b!19%BM|txtCKGK*T(-Z5Y70eOXb<2fQBbG|;`2px4FbUnpz0&Wt$ zN%8Xyy3=a`a=UC<2?v(s`KZBfdmV3|15$?ne8mB+k*TsJhDS9f*ajylAUgFet zi`@?ps=vmzhXQe+PC4{=j#(=L1qf85pL!IT`-(=UR9eI6i0}`vX-U4Ct4lg#?u8m~ zKs7*9u4xpjHh2Ec+3x(eM&e*a#6ZebKyxlbCB%<~(=Euxcdcm#f8H0K)d`E^~!eSzWtu+YJ@= zP8qze7z5x}3;9HPi+f<4u}cLv3F4ZPa4(fpQ`!mG7h|2G68v7yh|zbiOLjZZI)0KM1zrDx zM|m+HlJyb0dl%%n2B?@s?u!^v@k_qwT$E1^v-gz>edg2VWCB|XwMNKeeOs1*40BXl2Ve<`#s-n>m_|hox<10(Wg8<1pt?u|}iJUao z4>tE{eQ&uPL0{oRV0xS|EBfcRihU{Sdg&ye`azuz#|HvT-W8QeiNyTY;AgpZQq6u1 zjQTr?oIqFg0jg$RV36aiC?F9ixzbr-F2dk3T(h=UXTThl=X9V1Zd5BH769P*JX{E8 z)Yv?Rb;|MuO8;`sd|SfY>Bzty4?vkwpW?+;#xe3T413fjNsvPuev^cJB}d!Ax@ZYR zw!_!Q3kCiMfY}$ouxFa(irkZ}2r;ZB2bDsj8^i_*erJ zz<`FO3WCUE4cGMdL4EnMzrW;*=k2=qpVZpZpBJmaLknU4-hTq*MgnKm5>JRyeY)22 z$ydU`S~?ekM8poVyfV1t%q)DVAq`?z`gQ+6Nr0q7#QCGU-n%QusTb}V;pRbOHwKHh z%ZwhfDihd@uz@}SFps*kr@%YoAj5ZGF@rzT=K?(M&mV6N{_}<#>>oJC>h^0%{GkI+ zv)R~_et*mOYmS-XoGwgB#Z`JG^?F7r|ps^Cs=^F77Z= zV=Juf0)?n2=+%S*?=fG(KF1M>kzkAkAbzP3Duny0rProhM{9IHK&}_Y`1mq0cq|&I z?tDw=w9?_FzuK2;>tjK~#j4E5BtmiVa-y}}PYPW15)=pje$pyqjeEyNC|i6$?Ropa z{MztmH&Btc{(umaI$N;W(8BN1@Gsw7FE5E)O2i=%7L0EQivzXTDu1MRj|2cRf+;9E zg$-)6;;2y3c`4z5v>P@0fqcBZ2XIu&4)vf0&9J!8_6D;EMZ&N1WnZZ?Yl(B>I*86_ zV$O)h4pYX_XM1JnMMs+4hm}hxX{Epdac1x*T@k`KWHX*pp?y5b+Qpn(a$s809?ZrW z%F#qWn7Ugy@OQlT&2D5;5Sb1R!y4F_b3?=ht5OhAe@H1WW`)HGk`v0tZ=7Z?U71W| z212~I8tzKSsP+%)IKtT{_$BP8068)qENf&Q8F30xIZ>Kxf%TLK+5j18J`PD>H3-?D z8`W-vF)#nfP1Y9C(WZQAX0k5%(0Kb*r2ye%ceW$Z zT+#(HGS&*8gN0W)OwCdi@|{2Y^Et9UeOlfmUXjrqU!)k2gfShcYTJzb_UzXIRwoR+ zL+9kL(~C1%O`<^Ve8})P>h6}CoJpCB5hSPqibx}*vPzsdgHB?+Yb^Kw_$36MzM*lH zz^Zu0Cz}}x`Q&|_rpz|S^x?(K@laQjtvm4yaj|<2&>k4jbv`o!YgpJ0wYt}22r*r% z;wABL7a2kCdyo-IDA*y9LvEXEa5eEKi!$sHiht4{MEwLrz<)O)ep~*dz0v7;E#r)beSI&S2#~%y(7;2Q{7Ia6{J#pP~ zT6sA+V@9_?qNZrbF`i=ZiZ{7IY#pdu>ez}cHLmsuXaky(Hibf)D*iACNHKBxo-o*n zFJrsdx;^>hpo0&*SdqtiNdc;83LiWLb529SJ;|!)`xv3G4y5NJUpO>JgDR<=6h_UB zRel#nq3IkAG7^0__|GlDdT1RY{|w%=IF~up>sAaTaY%$MWkCjXzTMph5xLP=r2FW3 zvbE2!>*zQQlGF)?GJ2Hrf%kbD+{9G{@@6%DrY9rcR{IXH9RSM&(ad3+j#01GlvBtw zE*%J5o^}@)CpY-6@u2(be)PPR;6-7>pXT5~S5OOCpmtOfWvg!ovYc?hen5J_%K;~9N zB#$~TOfGu&onM}AS#$`Ms}+@d6FV~g0W0NA?qaRPWqQJa6?g0QBOYZiZjyD;kT#gO z1XF{WO}~38;Oz)3<*Tz7!l;DCwd$O>^lGDL{q=4LAMoctO`j9=UddXQtpg_X`QEl< zWv{@LII~vTAjPoVQ{hEIV(D3+48*ow9WN*mw$p@N@6RQg+EE?Eal!_{mWxZ&Diy{w zC*IDQOiR>(Igxoi*U`OD*TkOti5*AvoxNgRjTgt+ZJ@-@>b^mdq!wrS$to-F2OT|b zxc3AGN4cUwV}9L$*IZAD*QjT1EHUgS{7p$*^Y){z{dFo1&8Zf{Q7f&rKevwycPmPh zJ8(*-?1|!PRI6O&l#%>N=3Z`}phhoD((R3b-p!2yAwmm34c}Kg8&9**$hF_D+%UZ$ z@=dx>sf`)Qx{r!xX6ggk^rAmDzkXdE|2_z|bv6+-=XAk7O0tdDZvQO0hDg?sGO@~> zwO4YZ8JopS;=69lTokXnwui$D6Ej-D*sV99u^6t3kBvHkNn-zdduhYSmhJkb4pW16 z28Pklsfzb5Im_riU45#_bZFOiS01x?!H$8SrOGAs6rqVrM2e5%R@hH8QJ}M2-F;N> z_S`4hpI1g4-p(vb#@rj9vb}RjkL%|L>wQSKbVC!oy<;g$ag$8)i52|MY`acnA0{08-l$&6 zC&6RxiRvVSNXOcZ4V+2g<6v)0l0Iy?)$_3^)}Bo}mD7KE{a*ejp_7FpoyWvKUEAGn zIgro!nBm1($=&un`R%#LZro%(txtZU7T@hhiYj62uwU@VsQZs7sksn(N&h``?24w^ z`Ip45CnWNTyMF8OtQW|bo{heSG@muY)5W~al{w+{9<~sdoSN=dw9MUzg8gxdC47{* zF4Gs~oT?1VG#!|D)ClYQ*i;F-5;hK$JQyg{f;w>C51n4V9q;e{Dfu)inK1vsx28X2 z-iApRG7ylB?)Jfq2!$CmdNUZQ?|FP>CEVJ`$1B#_$+zK`f5oE5V&}U?&+uW#m{0fS z(n4?USH%P$Fz;BJv3$~6XLaw7TaQ&_)AN=nD@;=ShASxJVRBehSgr5Lc;}b@SL{sT z$9EE+gdAiZz4KfBR&hG<&8v*y3^vktmnE_kf~cMT?bdeePO=!VWx5$PgNJ+c!Y^^VzzXoQNBN1oeI(c}^^U?Bt{Sr( z8;-(Gol7o&y#y0U?jyq{YJNZ8uyeTcyDc|aGn0$Wmo3N*Jx3NV$gqK}{oywzeEf*l zoM6UB!}jC;o<*s;(tLD$raKGT+X3CCFE;Bl8Hp4l-wR8$IpbdW+?*T3F0cLMM5K@J zsuNm+1om_4uy&4hw&u87KabWrpi=VX-^+}r~NbuEjhZQ|um8Thh__d$mv{SKc6HBDl#yOgWwVtAroC4$3w+fs`9Gq+~1zRpuJ|r}iNZBTM z`l=l*=zKV1wQgad4Ev=zCOqqhN`85S$*gnPWic8xUOm#D7}`DD*)e{tZ~4C0#`tBs zpJT(%RPjFP>h^2TT74;cR2f`@kCc+LL{2UjMPZ|1gCQX92P=g{%O<>XQ(lIZ z$HT}2D{4|HCmOwTO2wZN(SnTs6KD&vs*M(M2 zlgMlm%4~FOtW28R)8L|+JGIE&uiJ5~hd7~S*ew)Xr^ z_@Mo>ks&n|McJG)JKK%ejqcC=N-Sx~bSdHdeT&L?Pu29DMEp@_dx8vI7>YOK2Xpt&gK^;IIw)@9?OpZfgI*`VvZq}M4vdbROKHM z<=uNb?z-EPK`AK?bI(L5j&Lt7+*Q(Y+}C*Qy;UxA>>~cEuCgm)GR#oV$$^)So1# z$)-<@4b3A-oK0S1wxjmP8Rj*`{O+>v&Kude)TwCG?P`42RnLejy4zqRW06UybShz{ zC_yc%b-gbm%IDW@4;jU?6=80sBN4rLXiGS+DR zmbau(ddPNhwXa`;;?YP=p?g^ChvHpJ4UhYf7a*y#C|D=S)>u+{B2(`e z$H(;3H*7D3a*rEz=fu3G&~q~@3Ng7EEB0O_%(&I}dB(+&j{{kd?JB&5NeXMa->~K; z*SQ#5Nq^9d8fy5n7ZYvg$}PY0-?Z3M*R60}8y1sau1(gZ9=mH_hGAP@GHvmm3Nw!8 zzqQbDIs6%Rg% zcZ`)dJB>cZdF_7j;zW{5fcBjbA^k&Q$H)p!n0rtp6Z^>)9V0Uk#J6x%@}8L9RQ0RP z6Rz8>;X=;dWbpFhAX&-RvfOs>{3CgXIVC9V!!qN@Hb$ms#)>CI^NvUq4bSK$wB4-u zTm183kO2S8)q+l?JLS7Rj!eFhULMmF)lgF!a#-rR_>c^_GQ&D^n!!&#_?S^&Lytk* zvggy~$+E#ouaCZtkL#S@Uyo<_`w>9@r3=a52qJS=Q=1)VHzx~x-=BRf6eL6Rr2w1 zL1!6T{iWY6y@EQzaW!T0>&g8>b)jeb$(5mtgSa*{3m#gIzP;YFNca#V?6A2QrqdIf zbg7^$kIBj*ej-1CR$pQC`sN)0wuIXI)_An_QrYAqSD|}aB|3SMjjm_uASKM=yUB#P z8-B0r0?fJ%mnabVIjOou6NBiG_xvUX@TR*<*k$*naWd|?yE6OYPl#EU;3>Fg zuH!Y@&mI0h#wcgmOX3QiU zN^^atd+GYgox7khg!b;R!M{An(%UhC{UN8?&t+h1_3!t!d{Lja`x3_Sz<04!XLc$p z{mWijelPp(J38;QXRD_>sqll)UsQ?$*2!nc0#8nMPTv?Vxj`%Zx&3IMdITJ{9l!?S zqp-iL{x~+m>kw3+q5YSDl&fb&;-uQ?`)*G?6%91F(f*4ia`DkFQL7jpA6GdJTopOn zDL#g0{p+iPK^6vWnj9a0nu+Egnb{PgryQP4hdF6?IBytNUkb9Sfa9cWsjQbgW29Fb z>fi0H#CAWUlf{k@s0CQ6dX116E*L)lBr+}2gEfK{ej<-X7{M*%>hHdYVc5k)s8x@HC>3Ifm_Z)p53D@{$W^I@*BEep%^hc+l;P zZD)k`wBjt)VqND?vv%>uC;eP-Uiml-fM^qL!n>$B4(|`?a`1|{O=#udG`cR-{OoEA#t3e9Kwag$$zj>b;z+nrc>)$q2}e-$UMMv*O_ z+nw(mKQWZ)q~Cn3^-5tyc|^?(cQk!#uALRO2ETn9TDc01LV$TTnevTwPlu&HipQDD z^|(z!#+&hUk7$Uo4p)D`)LT}I658GO7|_~U2!x8HSj)Jk9V|?QGu?QOQjw91kp9Obai}K3nX_5McfC^%QDs#+zfYv}1;IGd)h6 z*L8niew0MLz@S^5PW|XN?&da;6!%q*lug83I}%`NghiH0r8^^N_vP7Y*EuXCFz#7i zBCCE_sjOvbcne-;eXP}Aps-dEUSL(F=wisABMyU@Q{5u+ye_Av=`oBPKz@+_mRsr_ zfrno`p?RGq-ABip7-T`aau*(YfrM9KGu6+$4W>qv{=B4W`pLS%p!>_>uW|e%!XlN@ zsMMn+h0lsEAZ?2~0e?3ZR|8s64RN#TnBURZNALIgUjL-K{XDV&a`Zk#L(C(UdH)w0 z1BBZ-c5oF;qDJM3$=C*z2I{f-fTd-6{A1oBhW&TGAdC5!^Lv|z!2(Nh&v7fq0kY6NJ@%bZsmx#e9C_Qij;*f`AMD7hXXMW-AiXNc9J4x$hu;= zkO`yWJrOf{$L$mKrk%`#{&(0bAJs;iobLTm5Ma&=@*F=OPB(1H3+Z#DywVvzRh*h9 zRcA{-rlWj1`6RT+Rc5OGOGkA<;uiD4DEg*#sE(eje9{RW(gpp$mVkO*#3DnJKt5Z` zzUy!@9^TB0y)w|e7}exm;Z#4a5GTl-mjZ+APHY;wZLhcw)D~HUyj|`)q4IO!%-m89 zUcaQ~yUKHq2u{V89e4gs&7t?2k%ut4@9OG+Tp%;jy#^lO$q$&M;<0C9^+vF@S=C( z2TLr3mKc)(VFnM=$sCoe;nCbL!awzA>oj zBFOp+bLzV1)S6>Is@8-%U(tN7G%0HK3Zar-Gl@TCxj+2P#@pShGrdf@-O{XeUM>Sl zV+kNO1n5lf4&BwG1>8=8o8vEvP}h$*7{w469Ncr6>GNdyt1~;1oD(|(K3@U{be83q z8H&*U1KE{SE(cOCm^zVy~AKha}3#7;QH z)9O#31gbqMvunxQRr$(YO0u`f6fB$XJsv09{N_Rsp*lrRyyQ21zP17 zM64;+=p!t6Anp*l_tk^{US~0QV1l;*i-apJ6G6ZTOLoeQ`!Yk=fVr@)#+kRj2QQ```g+E%LkLI?U21ARk#m^8&Pp-Mawzz@k7)H=LIB zr~zG&bPx?R~#E^`m2vQ!;5H&QBNh_f!cSNBhxc^Y-_^+z57M z4GYmBMAcrSBWHE|K@edxd(rE!F?TjkO`|s@{A3_TQSqG~^dga0Y<4LB{L-sri~N;m4{Aj2@V`5f=eRcBq+u-b*)p z^9Y&whn|mnkMP}!vUEH0@Ru;{MtA8{#IQESUhiy0*KB7|)Y}$@LbE{2n^#984&v-B zr}|3Y0OfKn?+hSBL9}Y7_NU))H-WmFPnI?H8{s?1Q=Fg0H?xPj#y!cUoJwnr18I|k>UAcu$#pNQpgvr119AQbZf!6opXPV0lpH~nT)4zE?hb3+p zN8wuyqlY-zk(kk{viAZf+FlbWlxJ)C1I1_d9L3x*j+J%I-N3dKU|To%O&1(30P2U?3+gI}q9(u*cIG zyt|@P^OKJ7E^`Abs|tmlE8|PMgei;G~f)L zG+mZ~dQcH*`4+h6ErvOSpN;G_aJ(Wq6@0M05`<(xPcRZG?u{Y#0-)Nj_c>*qWWsti z@7IJQNc<_7RCF#Vn`rymNQ>o1g=X-NzWFxITgqQt5TvF-bn&=rd6vXOrjb#9U9IK{f4CiZys7_eWwK8zOH+jG+1i!b5H zJ^RJ{!7Ad2rVBy}uiC0-{z!0Rc{@+*FZU%eem8~R$>bW;UFcz(dfnx#$Q*zL=c>%# zV>~iktS*Ku#)L>b36sJeg4ocFU)RTn_U%~T3+1S^plKkM=fPi@b}rdsIU-@*1}_I* z!#IQ!;OQ7^)!l}sEn&Ees6R%Yo{O=>34`rsL8v3387+LED~q6N8&DGU^JSL(6@}pf zsr~0(?gj-Z_t#rePQ!yO#$)Oh+xjAE+BI=PBQFo_h2ilh)TE)E>zl6q+8j90UIqTN zgA_Wj11m!9R{)!5LVui-tv-{f2a4LhkZOmsy;6nufD8P}4;y9`DZ=2Hl|J9Ab9r)I zSwelK!W>cy)ARozz>e^rxv)p#M{{uEKQpVEH%bNM;61It|xz zrZIrGKKa>gl`h0XHyM`^11Yq9qI?{NVii7y<%_p?b(Yirz}>2caTz$rc_TQX3L0VL z6okcxni44rRe%?J4<*r0tdG~!#grTb*QIUaE+G|oZeBU6>k7Dw>aKvp3fxy-rhh2+ z>1|oAm1U;wOufzGY5_x$_}mN-ng|E(t6;D&uuC?I_pTdY1O2(#)Vx5Z#Ui4LjAn}G zKg;ZEs^BE8`Np_Y3kYe_zOMjRv4WCE+NIG%>R*`~^!UCUvMU3P{hRyU}z{hi%0PB{94oF_l%S#FMT z>-4A}taHt(tOnItv?-G1!fKGJuuj^@iiYjXFb z%NeZ}<6V@P13Y#K2+hb-D&ZnibQTi|qG5(%6-zGa53ee~7%Fh9_eHKo^~D24dxj2^ zT$|SzxB8}VMae#z+f7k{J|M=>Et+5o_+eN1_?SFp4%BA%^$i0!WMDV;mNOb3CROF7 zz)b5)Ri*Dor=IfYKIfHDC`yJ4>3Y%hNhrV};Vt8n&6qTZKY@!%i~kBdQJ*52$eSA> zQEoJJ0G}86vc8O%QoTuk0#3EwBP8UM7hIG7QXVL23g92^x=-sN9Xi2*3whZeVgdK3 z!W1&co?g^w(rWzZJiosh-2dY|9yh=SW%|m(OsE17Amw*eU%kJqVT2N*n#Q+k|sbV|8xZnN4XnX`l5BT>JatNS<0{^WDp#uG6 z4C_E4`kp^f#9pIG66PbLyQpwb1sQLtq473X0oo4u3PHmHs|Yt!Ix1=@ z+U%GUB!#{;G-KVoo-*Z8nKbnP`bt&+M4od?IG32`RD^pJ*^#ye?60)_T02HM$Gm;v zwOR`@x%yEaap080{3C{76L1$|2r5}uIl;sD?|a)b(U?E}-cu5@xcie*my48lYOZ{od9R%+e{-gjF3OUeotXh>+W! zFzMLqVIvJP3LulzLydM~>Wl{JNKQ7iI}$4A&ai!p1+#D1^DV!T?Ki6(}`Ga8iSb?sND#6{uzb^TZm$+AWi%sk_8w1CuTT zH@3L^L5}L7jNQZ6U{*_8M*@tKTD1Al*K?edm}@_o6D`RP!jhzbzDD4GqEpa%5RTE6 zSw_3JfeAHaMvMmTb4Gg{EXq8w?C>|QDrbHj;lCPSY;)RSeqa zi$#Ya-C-a~lWkfB6R1L2N7oeu_35JfAmXkmC-BRPFJYCaYbtw0U<-m(%~43QSB;cx zNSt$!*Sp;+($+ySQ!ogFP{mxY|6V z_up6@Li&p%@)VV2?|RZ@eFX33ZeVt&&C|DV_&~*250e*ybS4h02fbL=?5wM+1=cnx)U5GPa z44L4Q0Sgj+noF*$stHftvB=LCeh4(Dtr7N)sb~|$4rZu%a$d2Y?d=9n7jLKWVOB)q z$AKb_ZCZ&DoY4gWOj>&z*O}$Npq7(_v5~FV&a%hZi~HiM#;rL#1~DHu-fxzU#_kh3Tf_tO?Gy7?wizmFue*(Dhk(|8TC5 z&>M4EdhqWDe%DD8REnV7PBi6qxLa#vXF@e_gfS7$TiRIk9_2wW(R?p@F7-Zz7~WD$ zfiTnb9hawF{ut1%V8h>?BU(D(+1IQ50bBOFWpebvu44sdMH@sWA72xf6HqnnEV9|X zQU}-P10d+#%7+w}f%1%_FNLmXU0~-IsCT`u0$BqE^ppaEH75&^FX6BUIF_dv=G649 zKMRrw>5STeQvlCF5o1s-XsO>zvr+_`{33!gCm&LGU+{;kzqa7l7{DrL#u|RR;zZsT z4r;swY=GTW^?M(L^m~r9l)5d!75gZ_NaAI}tD&G%$Mli)-O%|bpSd+Zqi77B2(scY z(RqT}0o+uj$Ht&rjw#Fn95LGAC|pQerV|+;p6PpL1bT;D+Y=!zHuGyEM0%Zaz?|1_ zDxFgn0!HXKDS0DUbqk)6xj5o#VjbUJt7tX^e-pTAA9yAklmDY#wi__(3H2JUzeZd4Spz#6P2G>{!kVG`@7D*wAE#U`J4h!u7c)HN7EPlv~%!wwcMu7 z{Q*9)G^GSXeLXNEzFOnHVhp_q&`-z0rw2Muv3lI_=VlM(;bwORBLpii@NVH7=)TBW zU~yR5qm#j8$H_wa<#oW=sNuN=a?LA_;8|h05|Ij#`7%ty(^!fMZXN8P)8Jw*m{R%V za2aH_ag6#ylaX3P5~1~5E9M6JQ~Cay@QuUM;#y!@Nb8UzUrUz(y`h!jjI#xPhe0aZ zClKtlT>>7aI-^N?y>Oxio)n1Jjo53Sp1;H(v?aO(G?F-Afi}UMoEg%yfKC;DkTj_U-}5zQ z4#N~b?qYO^6_KYDaw4^HXKttpZ|>u>H^hNxCW-qmfs-I|&N)15#7dhS%DgrS#+LBo zrgRu3Xdi<)5d3T;dBjA9DSmeCW#oURokoN0+U09pVgfOLyz`K5>f{^upfvP+w|u(S z+;62#jqC*dGxt+=;Yb}?rt?7{5@?uW(*tiHTgg-ivrPYMPqZVtfhh!)A+-(ZNrgb3 zy=uZnLX__wW`uqeT_?59BmrjY_Y-V0__hCGaoP;bfeB1x!pt$If*^p>M)QyqR+%cb ziw4D>9%jBn_`ouhBycsMgCGb?5Qlv}AFw!}V33P=fG8fi0DX=!`nF!!R$$plT!pp+ zpagTj!SA!Sw$w0Q1P2Hj6dooLYHntrmK)U)cVQV*KyDV0tR%>v;X!ynDWO;%dO(^Q zz}}JWq)KT0r>~EGGD)zGVOpGZAr1r7LC0U(58tMlpmaDsf^wW1I4H2Ty8wIhGotbH zM&_s?f*&3Qo&ZR}+BAagmts3i9@0ahpN@Y0RtYP>#%`v2n43m-Q%o5=8}STyV<1i8 z8sl9=60$t!h{;5Pc}2fwIar_p004w|P|PO)kwAB08K&&tTA{+c$=?t{5IAL=&C*Cp zYzYI-AWZ<|7ce(4OxfNK#+pUX0OU7^S)XK-#hl-Uy$BEucI@J(tw`_YT+Abi^Y9B9Y_@0FVnVH1wexCjBCN~8jD3>3%@418iA0fR5 zhI%;m0)syTSiTctJNMMVg)oCtMmvPL5_#-z&I)sPC@~yMx|0p?JCrJc_F905JX6Ag z76k$zNrys^EFUg75Ts9`S;8FGQf3_LQ^EI3c?n_G-~`7_6RriR*PSx@DbG;cj5^Y1 z#gL;fVQ$b!_()6ybIPl5OdXhA6iI=5*w%)|4%k~tr?96V;4d7D{GLV3jl+C5UzPT&xI3@sE2)N+| z`4$u5C2*KFC~GYWZ2p*b`=Y@!h(BNjyFtCA-`rF!1TP*LGB98QT@l?kLHLvfSc5&l zn1-b?a?*8Sm`5Cw{2K8HyLl>miZl@#Xo--x*KNzEfvHd?ULCIFK zY8Gr zv^ast5SM@}wBS84l*}Ozt3YJk{#y{C8SO<@%EASq$!MA)us=G(CaAYK{st|cQUQ;X zMHa#rXsiMX_6Ya-HUTuW0^w!hGD)aTWETSU0YMSiU&ImJKL~#smdb6bNr1m3gGPM+ zzgZ%|u&@bSZ9Eg3Sy(s*^*}^`@W;6#u}ok}oG$GT-oq>30B#QaDO4Kfx%p5XQ5AUs z6SxoliaHA@f{MK#CS_!>89&7Y3dn8K+Y(2#2AikI1?2!}b_IP1zjiM2DJQHd!kfhy zR~hhDjDW=*01kifdki6A(wO*_DzmK1oFt)&(4=2$4tp4K3xVZ2Xt#r_J?*8$o1z6V zFoHRRuC_oEG{A_X>JK?jc|5pIp`8^%jA#<^p&rwWeL)@USto%R!5@TgHAD*D ztBn`DhyTe(tUdz~Kl~V2!KCW#l>gNNFy8LThpYP;icOz*=^n~@%b;ly{4fmh4b>L+ z49udzg4scmlJM~2e^kJW!AjZ|m)m#2N@_rejd-BONnZ3A(~#!F$ttkH%n6Iu6sQ5& ze3s$;F{zNu=CcaZ^xqoKmiopPIKJmTkLDK^N_3Qoof8 z3>oCh{`dLMXeg85HE{VpJUFb{i!@bhE&ME7GhSqANBQO2lgD zyD5+&24tibR`>tf@R#x0oM6v=2u2gdzmd3D=-igU* zh_E0ul9AA+8Y6^lipJjlXCtonNMn#~Kr|u!d|-aap}x+s5XosehKjKf*f>|T9s&R_ zXa<;nmm8h07IvuxcFEEKJc)!bskVKM1SaCEqy!jA}9*i{rb5Wl{;xBZ;_7lwx7h>YIZ@XxEIL1{K+D!k=@B~4)()JfT zCGa_JO71v=zb@D`Ay0{biA4cq3qHe;y|+c$G8+?^?lZT|HGuuBc^E&JU1lsP;g{_v z4*`8Hd~5~?rYt*r5RxAolX+Kr4t!G;V&o@+@K7`_UvC!BMOiIgRhLbTp)3Y=vRu&>`0CB~w&Fd$-1 z>jRpc)6Jl)RAA=8+%R)J2ovpyj*LNN>=D=W01z47OO-Vq6}H)WAqW+q&-xU^s5d(N z=CY*N3W7wSH7InZA4m(6qGf@k|9E^2WC(T+bK?cGCV(QSIh`gQgZLXbvHpt8Juxph z8N-McKfPKG$_fYW`7n;nZyZZG1)AX&&^sPj7%E8~fo_|>C|w{34zVs2D#W{?zIpfi zU-u`OQWQ3kFJ2A6Vlyc0n(%^sx=_+B#DaK8LT(oAn-2P-3j;SG?O*9{x!JPx$tuVT<5Hkk(a!ErAj)jEQVm^E@=1;Z{XxCsFMB zHRx|>z5&073J3UZXl(;L&>BS=ME0)^WeTFkw>dnO z(dqYq0Z;?T3C~g&SaG)=%_V)};WO~}2bX1lmtBFKvO&W@ciJvY3}~!+r1ncOv|9p( z(s4LYypbH`7^X0tGUv+}Bry-XHmJ!VuI>XfJNKMHh)QFRKpF_;ZAH;l zO-id3maua19!MHw$O?s`3wpbp*HFSpO$A-ClkO)g%IK@4X;}R5bcJ)(=4-N}d$`$Eh}92yR`JM{tr{p5kgX0l*(^=ywK&M0uTk3p!=jHwQ;-uG6^yH9Ou0>+AjcL+hYFDH$S~PCuu9 zSI`+aZyGGblw0O~RL4D;sjYXK7QdWv9wHOS5)Ta83sIg;W^cx=r;nkYoc-|(jeDE~ z4ytWK+5W~Y2wjJCpn!55B0!_hvYLUwRtZGRv4ol|F$Qb3#P)y2Qfp0wgbd=deh`V z=_Q;$SgS*2FMaI%5upz_1JE%+Csh|OdQCoV5t5tZK)k=Px}oC`9`0<36k*?v6?B#c zzWxrOTbd=HHQ5RyB1CtH18bTO_}J7%*E3WMm`K`UF#KTmc z2{kiwH!-@+=ZLw=&CgTuoVKpP%i4^9IrO~zhj1)?Xw8NNsh$gk^sEz*ACw*-*G=;O za{CyW?|D$@3KP{;339c&bUFCnPvB@5M_VCmjLrw^2}^-qNYKa$Uvn$fKKVx{O~?K8uY}2Fe%dE zzbAWX6M($PAJXrFU9*A=bJKo9T(865C)c)AdCD`>ZP8C7O_VHV*`H8dL5hN|0w!FI}+3ixsvpF`ND5 ziazu;AnAMTrGK}d= zkF^qTeS71U?>Qs1nTjN_Iz}*hl8-GC8Msv8=Cf~Cr02>g9+UcZ3}OlTjf-}U380E${d=MM`Y)3!F-yKx=;nTf6A>Xd_N=ND8k?{BAI zz-B{@_T7okzEL6RX_yl_Nw#Uv_ZYs$Qe9(1yyF>2PPD>rj{0n^Vo+1P^~y$8O}JQ# zA|(|p>!l-b7tzt6W1lX-f5Q-=8auOpy4PIOo(-YfL*G_@@s(T3!I+xE+nLInuwNPc z=@AOi8OYvf6o`B;N(HE0{(`iXO8WWAYM+v~Zvwg~~uOqW0m;3~yVLWerWcmY8dLV%>a;YCYtN3-X+urOc@58> zVXZ0q*)x|t>40<+r3vZgBu$>CBaCq*-VAC6^g#wlXOXXHSH1y6W6t0?KeG6qv;Oy- z>g(6v(A!CMPDUs_j)uIqd904;RNR_Qv~r|pC{1xo)l*WzhWpP~<-(hK*3{gx@l{^* z1tLxj;`}cfO0DMD5O)PE>tt%>USS3cXjA&tQ}5=c!EiUs#;~yyA7FPfT|0|r`3bJB zB%#lYkqQpM1j>Rq;uC^JY%-HH z%08w`zamyxsC_$M{bxUVA}aSP&-xr%RaoSk53mqGkbBJfgrO?${hu%t`j6Rh&&sL5 z3O88?gt{*Wh?FNh|2FtwO)e*w-)IEq%^REW4Dovj8}xNnJXkP@Bd|Ofl-f4n-S9Jv zg}M;ylbEnNqi1nNU#1;QtOqyY>q07P1Z8#e~y1_oDfFeHd6b> zn29kHy;rXtY%CiUxp*sdBlbWtyuvLmK@9hduDohHP18|LVh+$9?fm%)+r(hC0{`m? zVF{2JTf30Pj0lM`^Q0crm%u|ESuM$69-G0fvT7Ufh_zcNXhCn&Wf3FRgliFQNg)JS zw<`7mNgI`r)96}T<2FK~v0A5@^WXxoqYU+UR1U`0w%Qy!_8x_~_S-+!YJhlXJ1hi# z4@C^~)3;1UE!Ip|U}84|jRnTAvD;OEW#qA}qEZQL!Sx2fvoAi_W6z6ktO>!rbLJIp zpv3xBx7Y6$?PqMo{^j4+W@0?<;3get<^blM5s0{(H{T*0K>p zQBl2sJuWDK(C)qDyqT|J$ah4$!j3cvgd~ntU~b-!VT^(N&B=>4R9k5tI^~!MWFUP2 z_NJ=*e3eqB5^FLjDFW4Ut52=fn$5?sto!QZg1r76I!tIzi^gviYG?5F$J4;u-RC?X zk5%<(s;hPWZz-Ld3~`Xc|Eb+W5z>StaSH*SV|LME=XLhdc}a&3Zf!cq*o>R#W`G(q zE%f(L6qDuNTuA<_*Wa`#7)p=}PiEc$WB}duH#g_oD(2038}4zEdNjQ(;meS>hKC^a z&+BiBX$=Pkd7@0cnGsP9;Obxt(5t@Z;4T?Kxoy0I2LupQ>)>pGDgT}y^Z2o6swR16 zjU`~IC+Nxl-IB=XdC`B;-6MHY##s@pI~Q2uGLZLPpR)zaLFKa2$nGQZyp1h$F)JM9 ze%Q@Ex+IH|L7uwPUNZOVy;2k*uMC8ifN^5n)O2bSnsx()j}iq1cpCnHuc^IbXoUw& zyJ6o60v1pK9{%N=Cn`x4)6WP&@5~sMM?`z&mZi>qI=^VPSFhIT`7_&U{Lfi7UPRqZ zXc>Q@%HLxrFIVKG6-?I_rW^3Ca&HHK+V@#T2C#tdMH=XVv7ss#)A;GG;_tr1(~>rG z;z_Md4T4&2XIaAwHZv=T<0mzFw9W*-8#sZ6@ja(V3{D<9zB3#vz(kt-p`*sLukJ9% zK|v{@M~-VNNgJ4DY8eGPV%2G9Filu+YFGVAp!H<3=Kf~M*$))lZVka&S{#{ z5->NXA~=!b#^}^;uCMI-?4irl&DFFCMHvac>|}@VjN$fMk`_I(~tu=V$THYlrWGIV{8EsZznu zx-XcJ^iZ`7OrXp?y>gHzZ9-jhAhBDgW5_~_E$^YKI{1|E{61Z!b7TkAZl+SZO^^>P zmM^$@lAqyDSPr(x*j`Wei%Vyf)B1HmuROeb&N8CiWZK4N`)jXWI0>755XtHiQs-&D zwmX!cEt2d3zr0|xo!gw0SrYT}1W~jtg@Lqww<=OP^;cs`f;et`p54`UWscf%qN5gg z)mAx;A7^`t1<6_w5^X4`aSIJmOl~(;+m$iS^5g2U(6-{##A45FNaYl;nrfnSR8qVb z_6knefly~Qpvz-9-ixk*z;~pjxh7l}tYjxqvCXB(8Cr^i9X?85E91l;j4S$q%+GoC zK7ox&J=qggxi_ykH4Cnm06B3wea)n;UrYDoYiRZAHF}SP?hKLxQFqwtqr&R< zH}8rB927-*h}mvO)NB)5uef*IzU9JwaTv$EIc@c|l=jQ@h4MWK#hAkm2*Kp>Jv>&q zzoGi6)$++fXMC2AiWs0glCp8aYOb}~1mL6IB{Xy5R)78J2t@T4uY%u9m<0hI{i8*x zYjHXt$X1mTBZc@_^y~d7!LW}G3`fNs0kkE{J_$NENt7U7sr_!|9C5fvL9JRox&0;I zCm4?;SaVlef6+ihA0dR8;W<}xK0MIA0MP14vp2PX=hg0fo2!+-3;7xz(oK5ct%GU$ znb=w8;HkhR@2Jw6lk={`(6Yuz?_Es58iW%^qG7NTpb-giFf~k?w7wwQ^w}!C)fxwjN;IScjn*t%R=9lyDLv4MN|n|F819)( zt6#Yyax@B5mV0_K5Ja@+58Vi^tm3_XxNr|HpktoyU}{^M-j8o{oWoG|f%F*LY;Tgs zPxL8A(iKcXGsc?TqIvUMor+-v_gK-H2B}Bu>-O>w9{^5(nGW2PxIWpdoi_Qh4S)5nP@GEniFqJUL1K7+!HK>emD}#(4F%HIm{| zFL5A~SfH9szHcHvxlgccdjS*ac1gNo#%Tw$^DOICO2O6PHz%Ilz5Ot*rqipA@9ghz zKDm$Vl#gLTJ{oiJ+%|1DB*nc2lgE)u zvbP#4g27+?spZ8v-*=#$3LfW0kuD47BQ`-B^U6M4j-)A4Y8up+_#^`%BUoO zhd=zisKN|N0}$kLaomCt&-BL#&R`wn=wTp@&WRJds_(=lTiit=ZzzY=@6dlrZfKrP z7ySFmTCEsl#(sb;AX>>!b@hqJvXxu8a&miWI495o|L%%3o>2lgDG!?m+e5H<`TE-t z%6-g{Rtz7B@WWR9Wb*Z>x~CRxs)`44hxhOI7T{?@NsOUBSoAa_FQA1h-RTbcuBfWR zO7KNrb_Op3$Ao01foaL~zD!0mbMEpYe2y@&3$Y_Ay~5@pG*AAC^aJ3%h=!Mz+tH5! ziAytLI_ay{!I#{Y;=OvUlUuLY&%lTX`y621xJDA6i^)eD$m@{;V8ONTWtsebz)b@M zc9kN0UsEdOm4XMt=POa%hh_E_5RF0D_m`r~65kjJ68tF@jqgUf>}B>l=^ZwEF$TQS z)3>E)G8O^_`m$RpT_HbO|2PKkq1@jQ;^;ZM$KNhY$9zCn9dJiVC&C9t;vC=b;0ju` zRwuLEneIToXNuB-T@M5f5#0fsm#pPKk3~pWaP2T$Vv$!>bL!ksZ*OS1hr>`re<1fq z0^kzy{c}BY-mpl}aMtkCf4wDOnBR{}oct*EA%i9V;}6$3LnIq2PEVH6 z3RjGt@`JWHJ#W3O$%bGvc@kx{hhyErifT`iTbnMTAnkW7IlSkotAGz6A_(Vi?pD(? z-Pl+hGhxJ>inU616f5h$|F5Oax?bP~<+PWgqFEK3?B8qXP9&=Zc@^U7lQFyRMHfKW zH)Y{>lL0Gs?_p_#;0e+i+-|@+{oxdA`eh=5$G12wC;|b#Gz^PpD){)VYTy61{`6DL z*2yP}wEJ}Jze$;L%c^M(||t1&CLGNZod|6~y;c)XtTvgCuWF2(QAgwFKH+p9*nR=n z{%#`!!ra$?tYZ+oI&2apN0XJjP@5P1DX!o5+?%d^6QzMpm!RRW+O!j%8y^)75TDiF zC7aHTouNXzWS31V+&WsF4p=Y*j3ZZrDYg|te<&inogWCYds#ZnHbk`}U>jeIAvT!Tkn|wc99wtb|f}t$GYJxD3?$Rqi zZ!s(%a52yL4);fp|7QOI2BNV?A26*JB`Kdwt=7)&yzrw$AY9XZ9yjNWx#^``lPodoXFpNYqid2B>N9ODytvp z!70Ksw|M@lt!Etf+}J&blSXbj=!Dgc#sJfRkd3!7-GjjjE)CV#pZyA_n{L8ErVk*v z!wy8GiL79NkUfNRaeQkFPL2=9j`M_jCwjU)a6a1i%xbuQjbimmk#SNe}mAo!kNW4LUdYlNViAMOwuetXMKHEldsp^Je+{Z$Edcg zJm92eX>n+o1tHBn!#m=NG<+st?B$wGdmoqwd8mQ;vnBM9NIkv=7~-YqWP#yI z@CU5!C2B#nziyavq+MufTTTxRwF)Spa9}u_Cy8$i5W3oSChS?@hn{_g8tE^>r3Qv$ zQmnFr9H)+L3!gZ!9#*;cJWg3g)vE4M>v4`fmFnvmS1kS^(X2}yxjn3!Hk!K{Y5lgC zG6Ql}|66!N?V8Vn_5>PUrNyauD~q_N z#9%_?Y>U;JapgOnRh@~fZI#cJj;5S$NMW8%)Odg5b0e)D##TjPQ}KC*v=vwg3ixz@ zNpV#rtcZfkk1V#2$IgEG#UU|KBCXH&)>_Tk^V*nbn<32?^;4{I4h0z}L`*9WEDrhc zogdIOvYbg_78?SWUDhGspbd}Y#z5dDw*VXEG{XuvllSNXQd~?(kBkjIYKTay-*pxE z{pi+35*}7CtFI?oYf8RCL?q^X;m3_6DL{;m+M7;9&d8=ICk6LgvxYCeQhF7sui2W@ zB|v_+mB}YM@coHvRicmc10H8%@)K4^O$tYhoUfxCBbN%4Juj`!8Kf5U##-&G3V3yZ zaH80}OTqT2yvAbdN+WCw4$l~Txj__-Fs#x@xp7s~KE;O0;F(bLxkrcvp!YFXj{Sd znaG$4;v}5zq*6jGXwV|6%aJ~7Za%qNW&Ozb%@g0Zj?NvVoa z!1Vjqs5$_^{SGW>A*=*z#@A!N@69jYN8_wC4~RkOKPPQ1=TBe__(viSaHP>TlArwk z%M*PLYldorf}*u_P_m7*-OisHJ$TLw@9te#0#Dm|Tehgas6^+PwZ&tfzIK^2`F;Oi zEx@Ehxrj=!#vLlk!B=B=wb7%ESj>=j@1vKTM!QwImYy30H5UK!ryts0pU`eta99Np z_v(MRmZ989$W;?kX^&=)HYCtT#n50EJN_yNBK?;&tJ^(X*eslLECr9X%1gL0!g2m- z9@psa(fsW;JW+Gk#vY}BfJ7aJk1@29%BS6M6>?ERWi z0kL6sPw?;Gt|M9$4xHE>5IpF%(s_jE5)9&;}4LGC* zJR~zP33$mX(vz6nt@Y~ze&^5pCe^@(lpGLnOX~X8^u6NOBRR&HC+H2o%7t!E99hST z)Z3lPB-5WOeDZ0TgIRj$B;e=8kg9+>2lUsuPAv$67{gUH^ zG|cU%EaezazlCLEppVOrv~0iX^m}0(u{9_avMV@`k6vUAr+r@eeLf@J(%h%Y6!bG} zkhyF3bjImt-ViIWpA{<|Em4l#oO<^-Og-%q6&pufEw)xOaULxe5{Vm1#RYd}yIrqd zYY%AEZXO!Nh?rZet5u2|(^WE{y;B9tzt3@KsG1%E{wj$e7DG7kI3_D$T6~&4N`@cq zGazk+eVKKYW|nxRefuQ6!5wn1d_TJx=xo7A=$+$uF~Cv-&ykctjD{e2vBc)$(CLN% zZo2?y8z+mzmhv`3?B0Fg0eEw8YF(>Ndl%Kax^Mj2U=UD&>iMRGh6k#`j^=^Z>;F?^ zt!AN$+=5tTxr)k7t&>&vx@DK$D__2Dz%YpoO<`V_y7G-m9FXgE)B4xYVZcXaKxx;f z`(;g+bi_lUMiiC5-{{>I5m$7II+zKyX#AUMr)4u zwkMT~^lyq3Md72c#(vfKnl7u45^Gk}zmhVU%d~1>To*t=Kb98M>56FXiW8 zAf%7rC!{W)n_afih|zyk1qBQpR^Y3kv6okQTtPm%x;jQgqlKA{53ez<9Lx3VVA=5f z{Dfv)NRol`U_jvQj+;rB!5*eg`OqA{7rSw#`w(~I^1x5KYfpzH2tX<=t{`+)g>f|t zlNZ(hsvbVV9uKFD!hgF~oz(rym*mgIy^)F8NXtJi7tjH+jlX+Pj+)jN{^)J9Pj`~W z7n2Jy_y@v(hR=)9a|`Vsx8GcA4O-u)TedM2g;}WFs})MhSS^FSN{J@i&|-aj@~2MP zrL2Bn?`*(aGwgbin&ZP>RW?*xz8|A2xb;L=xV(dXHiMd`74-&lw) zwXMZ)&)rNN%e?;UNC5|mjrzjzE6g`Bya}wNgh4;cBLax|2LPm&v*(S=}+S@0R-6q-XE(hcm(8|=u-nQnseFkfe zC&@}vEuy)DoTS1`ypDb=sB!1llz_&Kny!B~+GZCi=_R9l1$)qaG5eP^A-pw@|WB9&Wc=c4Ptnm^HU z+sJ|O>*g8(i#wwxdKDGv>$jFWo*v@jUTKt7LU}@{Pow-CDBc(bF#3qoN%jnQXzY0V z2chXUMS`1q?#L-Goyfcg9-N_qOi28b-RcMQKJJ5iT(Yz}b3S@xwCA{VFO4)eIrsWB zYaG&<=RKm}+p3thC8%4;ZvF@b9BRmv=BSq+7@d7r6JmEdJiq+S>u*Lp?*{6cM;GxK z37F#A?K{dG2egjNFV5TKqhZJ+x2_u#jb41}^r!pHUY|*i+Uj_duEJtuUM>MsG`Bg3$6mOP3FYVN3G+@m3zmvC0HPv6?9~biKJ50 zOV(K^37h$wKODMY!xj5S#`&@F7cKsU^v0@BcAB}U2AW%q$NlWEISn}JbeK{_^f9u& zO)D37;6JveLVUlEWHd61(&*(ZYQr|~bh-F@8`s7zs2SezpWD~;!@ly`oWkq=pqL;o z?eY`7(}!c}Uvx8^`hq%7DF)R&8ZUGnt^fIINkt)cz3;+%rUx~<{C~ds_@i|}d@x8P z{m+jb1=&-gx*8tBrGP&jLw2VfzdhpE^W@Nw>-YsVIh~Zv6kYWRQi1Rt$6z2be*Kwu|m45Gcn%gy=?E3i|yruN?O)V76FsD?yn43~kf6F*lHg`#Fg||D`u+uf@o!?ERcE`>p$Ezk-bb`c%l;)p%qIUal{H?rO@wV{i z^xwwxxru{=+PijjN}%7u!1gh0;vX``K>7dLdv6R=`%e zx%bbesJFv%>b2ul1@&@u(_(tGn%1=I4vgDgu@Loe|8~=Lbfy?1_v!6}<_gM{huk_P zUZy+V!mZeWnpgavX4_vns!Mm*cv9HB+S-w~J>rhm`Veo3YMiRoxN0Aa#M?MIF$&#eKSzoabht@oS6e@L;rne(ZhruyW{{#w`X?{1xZ z*ISvXJoQRlSYIyxZQzs0<`8SA1D~o2KK`DZ`g1|YZ{I=t4#%4O*WZ9@wO^faAqlLe zHDCKZ{hi1^vzz)87AKMn0+_|0FMYl>j-lvc^wc6gO!SKJdDKqKjbJDn$zvOfsyr=K zm3yx^1bi-0ZNHN6>e#WxAMPVh@BbxN=8Ba;zg!(ZS)Y%rRlk$t9D9W%kKB#68Juui zB(FJZc7N7dbkmAlw(RCwbI1&vie1+p7I(|EtBUk|+^$ohGqYo3i0Np7I23x&N;nz~ z70dq&rgGwc!`TT{O8oaCbGLvEPqT+C{XPlptYYHlEgPRHX?e6S2<5xi>0eKN-v5nl zbT)Zd)3m7f_m4iuT|@l?4VL91-c#~v9k=uQl@GpNTq$gcQ*!_b-)I@%TDD=a8V)M6 zCNJESoAbuTw1+<5cPE_-EGtG)SxN1<^xAx*g+c;tv6Jua=+^bmp3@!Lh9c*vTj`lq z;+8iuT?T@B9`&2}@oWUHsHfG0^C3SE2XU;}Z)b}P4z8^lobay-DQvki<4}^lvi44V z*jc-FLZv8GggN6Rhkny4^TGRkk@3Ht=0(30We>g8R@}>a!@h19$~6LHBTej_W!crA z_dAf!sl}Y?&VwX(`*9(*<8Eon)U^tJt=22W@e}ycb+?T8gpJpQ!eT=^TyKQBtzPc$ z?tSmAG+fSavumPaVrBksN9WrZOlDQ{fUxnc*^R|Ixn*60BLN$ZtIN~c^u^S<(gz=l zC2#pHQ#8|ELpJD%qo%V8Y8qX1dbciLxrg0|0No&MEnvE8c5ouK#Fdh9|3uNO#;WV+ z1+^7+8%dUOdiOb10sR|!RU0S6f^@pxE_DBi`69BuVt1)7;E{cMe!%hDN@l>Uk{@!+ z@k^pyJW;o7o?Q06c0VG7zioQ`W?J)QG4Wla3!Zu|hlo?O3+vWncnhv7j|-NKg`zj< zMyvd|g62ZHbebSXH;b6_oI8?!+?>DaPxx%a!03>VMASW`{|`h!6CMsEu&7ci)UccbRS%9BR=uG5n|kS$haWUMEVv zsLyB`>i68+DyCnEU+`O~?P~67Gi> z$Ipa`9dyaro3degI?0{S3Ougn0al7*)(`l*Fe9C|EJJ+=4ln^+`I4N724g{?r_bzh5NQG{`HiJG>+O#^FaY zlBS8a6OLuEZ$+esei(SW_N_1Yx!oGy`E-vvvF$xcyLT&xTfaOPPk+UuR5}JO+OdHE zPk++%z?Pq(H;1)cLdDjR&_5N=GOO5hxfeP}aoVQym2eVg<4GA^(tDpgR_tr>vLY9E z)HSKFp$mg?HKc6R`@I+LGV*JEias2g%sn#H-FB=t6u@6XHLJvuwzshBmIC9s;@%Y9mWgh^5P#uavi7_3 zih2)*T;b~rQYzoj`g`$lGsp>eQS{LdGq)udtaj~um4*@QyjN4D24$a0TY6d+UaeiE zo#f)#u|V!Q$i*W$wLdN4<$05)sX^`!ifQeL8jio8p=c*p4{5|xu0fOaq2l@7O;d0p z_!M(i@zB?Eo_pCP)NAvfku?g$bs=x9Mp_vwcKa&uMus$NEtA@Pu27V+(0wqiy)~mA ze|Shnqa@Jl>|BZ3$|p8B75aJfn0}C2#+bg&toXlQigbsUaVvSc3$?ldbx>I*qOX$Q zJ zzq*GlDDr;PP96#ke16W&oukyzL;OSEpF71pA6Tmv;Y9yT{qbD=^b-(h^$aroHO*G_ z=LiJ{i3v9h$Ks@X+P7*8dZzi!z?E3%IlM|nt22}o{kGQOD`TVn?s|2rmLp@l8Bo4f zGKmZhV8&1RwD!{cK9SFty-$7GOEJ{+q&}}+GvyI06J#k?_q^&h1SM&Zb{A_qC#1>K z{^nL)pcEIueK50K8Wg|AwWqq~b)PqfctFufFGco% zl#Rb@^x+pIE4yboxOIhlHRY#@I+~x6OSZgt8=5JH%kd9+qO4^bYKiYjoAQ&0nL_k` z*UUzQi$x{c#30%1BY+NRT%A`Svmf(6u+!hnN5-!yxb-3 zD-~T|+toK6wbJRA_abn*wedHc9d8p1}CTeNX7`<7)-Y+pQ0x4%t-fpu1LfjoKtn>XHtF2kFS0)+>!kcdqAJ|^X@+S&a{rKH1`;uAn(e`Z^tIzNursr4;<=< zVWifVTXepV@`X^_iOwJn-L*~}5po*h0g)M(EfiCr8qlxpZ6;4spifW4^_yQL#!I%b z1*b9VXU)wjpREZdo&H<4zRz;Gjg`DdJBIC$-_-wW?>nQKT)Kas01`^19ZKj$1oS9H z6p&s7#R8&IlqyK^P^A}vAX20Y9ueuFq9RRtk93qKHI&c=qy+>Liol(RbIyBs@BMt& zy7#}FFIg*jvS*$>duDd|?L9(LAi0r$_srvEmXYBiU;X^I$b3`Bh*D4&gJl4?rz6hG zjXz-Z-TOvhCiBV)SeE^4r8ZG?k)+ik^7al zD2ppSfX4!SEx5tRh$H_T2)QjHYg7aHv9k&!j8=4#YycHRH=}0E;vx}o(J)3l@@P4< zv)(Z&BbR^aM>ZORtl+@T;DS8Z=YryaU(LgJ^qmpdB~7-<($+w<(#7r?yG++ka{+pe z-v{|U?j|Fu<7BI?1eVyeq1aZa!Up^;P}+bZ?jALSbcwqs6uf$0n%VFz0vG+I;$%eW zVqH1dybLzB;vMO)=ulDYuDXB_3U^JG{FTniin>tw=&h^c;xL3ploFHdWvbtJ85L-; zOIDKCvu4-D0=y_`V>2Ku9GC!aB=s`IA@Ws!Qed_4f(c;(+>zfF1mi(mYDMXV+qZgKQwfEQD16;yEL=v89j?;0<;aY>3?-72!uz?4M#HKXfSp zhb_`+x*TC9+`$zpZ$r$)5)aNr^!OK;xYBu;?Xw?Gp9(DMyjQ`CSRCKGqu&?=%GlDb zV_z(@OY-us_g7_-nDMmf{#ve#UHHCF)yY`y0y8xoQU zO%&8BoTOip96a0l=UJ!(Hj`P+&FH5#Wm|nfY_E5AIVet0%=dAZp>a|;D0#E%%Dgs= z4OmvP*LQJ{9^hty=2R517|w$4T<6?jac?O7lgS~w)8Ugo}ZMm_J4Kz*DY&2@z5 zOAjr)WoKIfL@J(NtNaj6@4W<&QOV^|Kq3IRG5Ux+8&tLraWw!tQ_mb76hvhv>}NMy ziqQtXUyNV*lixrh;&nd>ozD0oVJa;E;u}8!{Ld4~@xIX$v{TYf^n~F+_3h7_G`rFp zm5||&%M=)JncrwdScjp-M+QV$jae$p$J#cxYh;)}&%8vYv2IrOl*`~AMbRl{7dmTx zvT@Gs(WUjEXa;Y`q4cpv7rAw#a=SM-AiD9EUF3MHm4XsQIMRmnq$ZwXu>rWIaAV}D z009N4ztWgSXLtA>Z4$5rOJO6p_51l0PGh<{AuQ<}8?8k#Xc>=VaHTRZr zlrLA8PYONM$q4PPmB3yO4bVR>;Q%0wi9VR{k%lMvkkeZU+zE_4KwM!$Y6%>+=T~sm zT0c~nuxwR?Eg4I4F;w%Xyjl02zNgV-Bihi2KZvlOY&Akn!7lxsE%IRx=8%}W)B_$~ z$AP_-Gf=Dns|Hp-sQ<8{D?R8cdbYp&2*^M*&7a>QD6KNqnO*%!hkY*p8)E!6S(-jq zuZ#om?{vTs{#&Sg5K@3D2*y54M*)aMi9OIO3>c3(_o!BmhtYl?Dyt5?i3#Xj*zrlH z;}_@pjl^tr#Uu<`IfdxrtTnj8UJcZlJ`CRA*_S32P-ZU}>T-+M)M^BUs!2fAlayGw z6^246UU|T8{B4VVv)2-iu0q_aZ3KW28F@+t@l5}(~ z-RrO61L?AB>8EJ|bAY38g-sX?>8*Ma8S3XoO_xVlfSHWzPvCJyM<)sjo2Q*)LoD`! zb0Z-%pa3}EA-vlk^~a$_yyeyILUJ&Sy_EwH+}xa}FSju&K#8-ibzJNrfCOX#Z<`G} zls2q&R%l!gWD?*Up~?~lgas`B72oZRS$OmiX0^GoSFd#!{thVpXsQ%daAE~Sh};GE zGIB`#LIoE&_W}VEtmd>6|&ZepOGD6Zu2@4!pw|g5F`ay2V|at)tb+ zrw7BBI^~)o8mt_93ii%}62QhnUteJI62c+(`4+dP7eM*KEO(u!hUnD3lKW(}FrdHs zcga1Tq8C9v#WK+Y1^Sa@!KEPt?(PnoCfvNVf`1$QRSnR2S2-LGl z)om>{HFPQ1(m=}>F10;rtK&rP0>tQi=rbJFrx&ASg~YIi+17aqcchbG6p#Hi80a{&K;@nsW*ye9mLqJY8%K(`5- zo1FsO!`?L#TJ55Sc3zjobl8TYgbNs;^)F(qpV-TKWI$+O5-{m+60wu2CEspQ=v`;X5$uXvaJb%X3Ph!eo#=Dne8J; z@%Ru6j=II(HIW%hLAZOr6QUHD-3rYgfndsjxGz+k)H0WV=7{K>WQpgDk?m`9;wX!> zzqnB8%&vaYZcIMWP0q{%_P7lan6Mhp3E}b5T zDzMcNuanq6qMBQw8a`d0hMDw7^1V`02_kAZ!0SuS)GKOGFj>JppJr<74L`gER(uKA@y`WfZwTuPw z@w*O|*J4$+^_`%Z!Bnyb0DD7F$)yMUJ_EV4y8_g>cR;NJN(w)`fGY1kf+*ht{JzMU z$GP5y`i*VD*EP-xC6`)szdnoMj?Qat(%)P(sA`$P(3+uv>H(VB6~B)jF>3F!&3SgH z%m!Q~?!gElEHQx(gSdH*DtZ0Q(j27)ciiY@a@@y6K_#^5M530(Q=Z^SBY5@zlsFN@ zeo~xlgbPZKx_<{al^OE`6atnw(m;WTcqE`U*NfXR9#)`ik`*#8J)*3Hj>nwdin-VA ze)&j^1f6gKdtf>V7~Td4SrHiFeE)zBn^`$+L9noZGye%14QKo0b4zQ!5u!4a2OEl} zYc@u${HpVaNw}HO5HlIK^-VOziUDlX+K0cv?9_602qLQAyoK?In5;4Y`%c@Xb=3gq zaX}?bGIy)x71dx3uvX;pmQr}`s|?tdBF?-laANHcMH5@Mibck?r@AY0!|tDAf}(x$ zcELGRE;6oTvEmhL3J8PzGMI2PEEcY}zHNawJ@X0UxktqUQj>xzDpGN1{n%YmbUNY{NI zJZw&Y?7mpWPL-&MpnyvU@~lc_mVru#TG8CUn|b7Z{QjYc;mC-sQH2fdc;q$gKvO>f z^tE9^Yv?Ohbcuc!U>IPM^lov}0C}#ZuA+SvsJtojCH(Dom{EVi6w8LmkV`oMO84dT zs#>iC7;!12!KA%c*%a~)cHQj^o4vy_w;A0n{}>4}fzjOHrIBTAZ?FxYbk{X}aC4`K zMV~B21ido(k6M5b9K@p$K)#}D?@@gJY74gHCje_aq2}=H#=T19cUm}8!CIHwjJG8` zUNMu_+}kJrNia#6h4=DtghQo%!p#hiA;Qwxez9?Whw#sA8D=P04tGyDLCdd`V-f_n z_>OuOeVId}hIvwFH1`}HKKi=-I+wD|eomXDz3z~MKG9GbJF3v3{ZRsPnoVF3#> ze~Vmo@6C$Dk@H}i8!x*fq zYd?e%1Enx~+cDDwAe)idT{#6(VWx#7&fxyBdpIrfBfd#+KYvcu}B$y0udRmIi^VCB;PB+XRr zRab?>64O=DYL(N-9~IG`ETFc)7ST8?!_2z}tX#;#qq5afl*TM|h%72jP^p3Xdc>Du z_r-H6-INsZCBo5RRFT(inG-xa&Lti6X68l50-6>X1Z%L)%%>zB~f^5 zHp)-?uM=pB$lv;Imv+T^X}j@3xgA{A{4t{ZK4AxB%*2B%JS>`*X$;J=juDk>kV9~U z+2SrZBrW3Bz}=mHfCwNcSN`=Y1Kl?wGroLM72US!{c<{zq)d5Mi`(&C*-Au#fW}gka(I!6vM^?GxN%iI6-u%X@=qHk#I5)5Lqb%C1#{(d!?MPD|pXK6y7Tj zom1|`PUoCOhYqWTO6p*pMqjgV)Qyf=jve3f0m|4BM)8&cx2N}Bwl}fPWzuWbI5xlA zc)su7>9Mkt`|uvG0*-G+<6DfN_ zWEy)bE>G`07X;7_O=qUl$#+9Xk0%h!0%@_LDcZSlNg!k--+DWi?M8ZtqTqSANnVt9 z)t%!B&%aoKz6tR+0&*(uryl7%<&G6;>I0@87hK@zIHULE$<`0Pq9c991wa9+wDZ8v zj9dh$CFPyb^2;BAiBxDuJetxlfr~S^JB7d%fJ=p~@~uyA%>`uJ(a_rC9zTfGFAWMVlqo`{d{&i?@VHv7 zq>jon)R)Z`!uugXYdZEWx&)mrXIaFZdJ{pB3vHJK~U z{j8i?76d-mKdWi;QwtvGiY89k&Di-rqe8yPXDC^$_1oX|IDsYK6(cUsmOK^wLcY=? zbNvo&n9oG+k($C6*YtxuDZE`4~Z*{ic*zU>&VpQTtgaMsWj9I4t&+gnE0uSSNcn9{@mso z+9rDJdECTj@fTwOvz?WDbcvBPVNUK09p6Pi7Y4dmBPXUe@~LJAWTF%2%ax>=oeX5r zb#k}F1P1!_6*S+o9liQ&{H#MJisdCA8T+h2b}0H*KxD!eQdr1Ac8s%7E(OD+l{ylRi^cWB2EpF6{T=X{wvacylrfSIs4(&)l=bCs=I${+Sl9cY2qz zj8{xbe|0bSw#$A4m}ai;$;A%0#B(08sz2M|GmWIY^vtZUpxnN&+Ts=wjL^KCC^ zs9ECY_d)tBa4B<0>M8qd&2HVtoMGe2%Ng>J^Xo6<*zmRJ>_5`2IArfQc|HolrZ4nI zC;Dbz@qRO!@I8H4WFXk}cM7`OYNd>;_Ek9MK1q%1snPH0E#1B)6RM3-1TJ|nl5ytu zq)xXBM!IaGtOd4uX1o-mv!b8f(#sXtY}|H#ZwhZGiHNo8uWh~!X)5%$l^H(^ZcN{d zdoy!XJySuJB zzUQN^(T(FaTv!;7ac(@!w`$L@P}!t+m8c?W(}5%X{HR5?NB2T#==`FYuR}hS5C4i} z1&gY*?byRAoKZ-jZPnPdXFE$SYKSahGpej)8T(9G8@7e_PuL*{u(4P zS08OQH3W>b?Z}mt^VIe2vqmmpC-jd|FzO_8yILV+jR{7vorctZs$DUXS5)LUVnZFE z##9-3#?vR~3;C0jtwPTo`-P-I=QCAvbF)i6R2)BCd}Z#@th;mw+FjzLW+AiBNsyZN z(yS)R621gEI2!5u7-$8@x2Dod1*R)GJdeBkHQidg%b|AgH}P8X&eZ;0U$z1{X%VMO zLL5xWwF`;6v?eaURTfUA?01pLqE*9hg>?6?u0PN4{%XdoidYZ7&X4^*gI)`Lq0h-w zn9u&H=Z<{oCGw)OVEjPmdktLv+nV8r&wZEK7ZbR2pOwL2ElS8Ga_CuV*1 zjNcS>S{sp60S_vE)KrFr>h6y9?fI?!O!H;jzVMdeN3S&Z(fjX7_`D?U5F%J9?%MBD zkaWlbT25BR)kcx85~Y2!gLM$rYlbyDROJ2n##4_a&oD(n6-Lc|kB?@1ObfWJ2)OicBOlJu`ufELbe?JPbIes-y5>4x)p|Q)++3Ikil*R*Bi&2aCUmvQ#rYU z&hb`Ue9PcGP;Aw;V))Fqin!(C8H)ER{&NCDtnhsJ@GMQ&-M)a&-d~mD)rKx*sd{d0 zDTIeAMO-L7C0rk#;!C%nb)7DWXjpwI?pw@>7#D|gp~fEd0u*yzsg)x1wni5lKF>&X zWy;>ZFxaD9kTK_^u`fCeL_` zZ}n`=ezKHba+0ChIaU;Xoqwh94)OHo-SDwU4T4M_CHJd9epjQZHyPd^L8rK`I>f9A z+5kvIkvGq3=5M#og}ewBz3#hujPLGK%1bwTN_5<8-;YLyby0swv+9Xv&JZW#gl+v>$=54`83wZ`|;j{#vzt`?bDS} zp$qT*eDi7+O>zBwu|txtsNNpZ9kq74l^7 zSC_co{Q$<$Ld>m-G)?6$(ymL#H#ra*y-_Qp+vmbVZ};`(@7>57>fCxT$B0mCsdRE; z*Cr1M?CJ~HJx6b8vT%c{tFpXaHYNCIxCni+pM-3VGwB0Q1&gPiBnNUyr8C^Rl|ejk!R9 z3Q)V+8zxd8(td^Zfir>jZHRy=wh5>>`p!N)RT-QU{ZG_R9}wJ(o?p)(T* z4(GR^sNd_!)~oiICm#KbpI7m1C7fI|g*vIwWV0y^M>A0D4Yah>jC8LTvzHec`d;Pj z`Q?!_6%n_jX%Y1ujNLk=rG?qq`p(Q%Gy2?`LlAS0|zJB4Ai?(Vnp% zgIFp{*@qemRpQN5eNVr4nT0e^<;$B54yiSJ<vHTBlVMMr{=tFXbJ8Yn zo5`3UuEITqJg?6Eo#L_T=>;A_lHO;Zc`wKfxFnu)?)$l{ zkiW5j+>Z*~FLD6qJXGyH%iFL#*)-GH*-5*eBWaHJ5}*8D)AoLQd65s>Y}>Zx8(u;% z?Jb^YK2m#kpA3{#R^#iLR9$*KwzTHI6BEbi``MmT6ezii2Og}bsoQ0K z4F0BBcxi+Em!?N%xLf9%XKhN7P}UmoY@a9YTrHvM0}gCD4=!JiK`Wz~Zt?B8!< zVlW3zh5vjbD0sLF@bxbe|KZIqUi{07|LMG#eQ^KKQiEi&!ke2fW&)R|;8f$x837HD zRN0WWy%qlSdr?TJ!NrD?0ws(EV3P)0@xM%RjRdO;i-=@P6!#2JttnBMG9q1_wQbSk^BwZ^yWEp`$hipjm^OpCqcLPdbelBbU}w7iUS8b)D6_h z%uw_Nc0R7Ne*FnCE|rm?J0g|b8dE#&=h+>O9+b@aYnV<{VDnJU)8((LI5uJV%$ORP zevBXQanslIVuO@`UIp)=aXJF?5ON*$EI5&YV(1?WLblF}3glX_$wqW-z@GiTnz#B{yFp%SQC~2~QS|5t7RJhO}FOcgh zbr2yrcos7PKc<)cD5z(CA&xO1u;NK5uzZZzVTa(~tp(neZE}FJO8ZGwCa6}x{C(!3 z4RBfj47;eqvw&T_!R^9BDGsL_`@qv+9sDnEA!nFvcPy)LSC38lM^YrUU%T)SxEDyl zhhBgC8r)4JY;uSCvs24dR*I+v+o~O+vU?ov>5&tEQHR;Q!s$JK;^=Nei%ywJxYjFI8aD8yKP-hu(3H7nqFWc1_{e7#aQ1)HcOp zj1Mf){}2G>!TkSHDZKWjRdnuRp!6V^9{0I6OYrRCo-R$<-hbL;63cWxZNpUvB`Sfj%@Hb5iiz zKBELmhQgt@S>gk;o78O$DE-$yk~R#latsO0en?mjX#B$m`18^w_@$n#n%_L!6=VZ* z_Ha9R?(i2(Oh7abHOynUyF-nS{}+jW*zt=O|8N3={(s^HN17C|l{18m>-%2N%>AzJ Date: Sun, 5 Mar 2017 20:56:00 -0800 Subject: [PATCH 347/407] fixed ASM blurb --- _includes/partner-script.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_includes/partner-script.js b/_includes/partner-script.js index ad8b4ea61a..da3cf15ee0 100644 --- a/_includes/partner-script.js +++ b/_includes/partner-script.js @@ -313,7 +313,7 @@ name: 'ASM Technologies Limited', logo: 'asmtech', link: 'http://www.asmtech.com/', - blurb: 'http://www.asmtech.com/audience-channels/software_publisher/' + blurb: 'Our technology supply chain portfolio enables your software products to be accessible, viable and available more effectively.' }, { type: 1, From e1fcf04cc185abf3597da24ef492dae0939139d5 Mon Sep 17 00:00:00 2001 From: mlambert890b Date: Sun, 5 Mar 2017 20:58:01 -0800 Subject: [PATCH 348/407] Correct ASM logo filename --- _includes/partner-script.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_includes/partner-script.js b/_includes/partner-script.js index da3cf15ee0..43352b0f03 100644 --- a/_includes/partner-script.js +++ b/_includes/partner-script.js @@ -311,7 +311,7 @@ { type: 0, name: 'ASM Technologies Limited', - logo: 'asmtech', + logo: 'asm', link: 'http://www.asmtech.com/', blurb: 'Our technology supply chain portfolio enables your software products to be accessible, viable and available more effectively.' }, From b71c9927e4f21e92800e024b3d317babdbffb2c1 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Mon, 6 Mar 2017 18:22:51 +0800 Subject: [PATCH 349/407] Update garbage-collection.md change "any" to "every" --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 6ee78b5079..ffd8ed132a 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -42,7 +42,7 @@ to free. Default is 80%. We also allow users to customize garbage collection policy through the following kubelet flags: 1. `minimum-container-ttl-duration`, minimum age for a finished container before it is -garbage collected. Default is 0 minute, which means any finished container will be garbaged collected. +garbage collected. Default is 0 minute, which means every finished container will be garbaged collected. 2. `maximum-dead-containers-per-container`, maximum number of old instances to retain per container. Default is 1. 3. `maximum-dead-containers`, maximum number of old instances of containers to retain globally. From 740e0798495680f94c81237349364f2e82a013db Mon Sep 17 00:00:00 2001 From: Marc CARRE Date: Mon, 6 Mar 2017 11:54:45 +0000 Subject: [PATCH 350/407] Update Weave Net's title. --- docs/getting-started-guides/network-policy/weave.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/network-policy/weave.md b/docs/getting-started-guides/network-policy/weave.md index d26634e8d9..02224b7425 100644 --- a/docs/getting-started-guides/network-policy/weave.md +++ b/docs/getting-started-guides/network-policy/weave.md @@ -1,7 +1,7 @@ --- assignees: - bboreham -title: Weave Net Addon +title: Using Weave Net for NetworkPolicy --- The [Weave Net Addon](https://www.weave.works/docs/net/latest/kube-addon/) for Kubernetes comes with a Network Policy Controller. From 68d5fe503fe4b5c3d5f7522ea41487edb347f6cb Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Mon, 6 Mar 2017 14:27:04 +0800 Subject: [PATCH 351/407] fix unsupported parameter of wget command --- docs/getting-started-guides/network-policy/walkthrough.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/network-policy/walkthrough.md b/docs/getting-started-guides/network-policy/walkthrough.md index 041c4e99f8..ef722f9ddc 100644 --- a/docs/getting-started-guides/network-policy/walkthrough.md +++ b/docs/getting-started-guides/network-policy/walkthrough.md @@ -50,7 +50,7 @@ Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending Hit enter for command prompt -/ # wget -s --timeout=1 nginx +/ # wget --spider --timeout=1 nginx Connecting to nginx (10.100.0.16:80) / # ``` @@ -96,7 +96,7 @@ Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending Hit enter for command prompt -/ # wget -s --timeout=1 nginx +/ # wget --spider --timeout=1 nginx Connecting to nginx (10.100.0.16:80) wget: download timed out / # @@ -110,7 +110,7 @@ Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending Hit enter for command prompt -/ # wget -s --timeout=1 nginx +/ # wget --spider --timeout=1 nginx Connecting to nginx (10.100.0.16:80) / # ``` From 491b701c9130f280929be563ec0a30babb57c6b6 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Tue, 7 Mar 2017 06:36:00 +0800 Subject: [PATCH 352/407] update to be retained change "retain" to "to be retained". --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index ffd8ed132a..ece46c7e07 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -43,7 +43,7 @@ We also allow users to customize garbage collection policy through the following 1. `minimum-container-ttl-duration`, minimum age for a finished container before it is garbage collected. Default is 0 minute, which means every finished container will be garbaged collected. -2. `maximum-dead-containers-per-container`, maximum number of old instances to retain +2. `maximum-dead-containers-per-container`, maximum number of old instances to be retained per container. Default is 1. 3. `maximum-dead-containers`, maximum number of old instances of containers to retain globally. Default is -1, which means there is no global limit. From b1993b2135f44c00e6da589e2864fe3d20aec5e1 Mon Sep 17 00:00:00 2001 From: Mark Joseph Ronquillo Date: Tue, 7 Mar 2017 18:27:31 +0800 Subject: [PATCH 353/407] Update pod.md --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index 59a7beaee5..f3ff457b97 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -49,7 +49,7 @@ A Pod can specify a set of shared storage *volumes*. All containers in the Pod c ## Working with Pods -You'll rarely create individual Pods directly in Kubernetes--even singleton Pods. This is because Pods are designed as relatively ephemeral, disposable entities. When a Pod gets created (directly by you, or indirectly by a Controller), it is scheduled to run on a Node in your your cluster. The Pod remains on that Node until the process is terminated, the pod object is deleted, or the pod is *evicted* for lack of resources, or the Node fails. +You'll rarely create individual Pods directly in Kubernetes--even singleton Pods. This is because Pods are designed as relatively ephemeral, disposable entities. When a Pod gets created (directly by you, or indirectly by a Controller), it is scheduled to run on a Node in your cluster. The Pod remains on that Node until the process is terminated, the pod object is deleted, or the pod is *evicted* for lack of resources, or the Node fails. > Note: Restarting a container in a Pod should not be confused with restarting the Pod. The Pod itself does not run, but is an environment the containers run in and persists until it is deleted. From 2958b303141521214547179588e13eb056ac46e2 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Tue, 7 Mar 2017 10:48:16 +0800 Subject: [PATCH 354/407] update init-containers.md change "a different filesystem view" to "different filesystem views". different init container should have different filesystem. --- docs/concepts/abstractions/init-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/init-containers.md b/docs/concepts/abstractions/init-containers.md index 4be8890e48..2d647a834a 100644 --- a/docs/concepts/abstractions/init-containers.md +++ b/docs/concepts/abstractions/init-containers.md @@ -67,7 +67,7 @@ have some advantages for start-up related code: `sed`, `awk`, `python`, or `dig` during setup. * The application image builder and deployer roles can work independently without the need to jointly build a single app image. -* They use Linux namespaces so that they have a different filesystem view from app Containers. +* They use Linux namespaces so that they have different filesystem views from app Containers. Consequently, they can be given access to Secrets that app Containers are not able to access. * They run to completion before any app Containers start, whereas app From 940e5e3a3647adcb24fbddc6d0d3641bda52660d Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Tue, 7 Mar 2017 15:06:22 +0800 Subject: [PATCH 355/407] Highlighted command --kubectl describe In web page ,there is no space ,need some empty space,thanks! --- docs/user-guide/kubectl/kubectl_describe.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/kubectl/kubectl_describe.md b/docs/user-guide/kubectl/kubectl_describe.md index a5fd703d73..6db3c2d9b7 100644 --- a/docs/user-guide/kubectl/kubectl_describe.md +++ b/docs/user-guide/kubectl/kubectl_describe.md @@ -10,9 +10,9 @@ Show details of a specific resource or group of resources Show details of a specific resource or group of resources. This command joins many API calls together to form a detailed description of a given resource or group of resources. - +``` $ kubectl describe TYPE NAME_PREFIX - +``` will first check for an exact match on TYPE and NAME PREFIX. If no such resource exists, it will output details for every resource that has a name prefixed with NAME PREFIX. Valid resource types include: From 886771cf248c3d3e66b7982fd9f230092bfb2b12 Mon Sep 17 00:00:00 2001 From: jianglingxia Date: Tue, 7 Mar 2017 15:00:00 +0800 Subject: [PATCH 356/407] cluster-management--3 change 3 to three is better or not ?thanks! --- docs/admin/cluster-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/cluster-management.md b/docs/admin/cluster-management.md index 310128fd77..24c5a2dfe9 100644 --- a/docs/admin/cluster-management.md +++ b/docs/admin/cluster-management.md @@ -92,7 +92,7 @@ an extended period of time (10min but it may change in the future). Cluster autoscaler is configured per instance group (GCE) or node pool (GKE). If you are using GCE then you can either enable it while creating a cluster with kube-up.sh script. -To configure cluster autoscaler you have to set 3 environment variables: +To configure cluster autoscaler you have to set three environment variables: * `KUBE_ENABLE_CLUSTER_AUTOSCALER` - it enables cluster autoscaler if set to true. * `KUBE_AUTOSCALER_MIN_NODES` - minimum number of nodes in the cluster. From c507a52d176d65349df04f960ad9b5e6109d730c Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Mon, 27 Feb 2017 15:18:39 +0800 Subject: [PATCH 357/407] fix typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1、Parameters “--allow-verification-with-non-compliant-keys” is missed. 2、Delete the parameter "--google-json-key string"; see the "# kube-controller-manager -help" # kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-23T00:04:39Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-22T23:56:57Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} --- docs/admin/kube-controller-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/kube-controller-manager.md b/docs/admin/kube-controller-manager.md index 82dd43cbcd..f3978033cd 100644 --- a/docs/admin/kube-controller-manager.md +++ b/docs/admin/kube-controller-manager.md @@ -28,6 +28,7 @@ kube-controller-manager ``` --address ip The IP address to serve on (set to 0.0.0.0 for all interfaces) (default 0.0.0.0) --allocate-node-cidrs Should CIDRs for Pods be allocated and set on the cloud provider. + --allow-verification-with-non-compliant-keys Allow a SignatureVerifier to use keys which are technically non-compliant with RFC6962. --cloud-config string The path to the cloud provider configuration file. Empty string for no configuration file. --cloud-provider string The provider for cloud services. Empty string for no provider. --cluster-cidr string CIDR Range for Pods in cluster. @@ -59,7 +60,6 @@ DynamicVolumeProvisioning=true|false (ALPHA - default=true) ExperimentalHostUserNamespaceDefaulting=true|false (ALPHA - default=false) StreamingProxyRedirects=true|false (ALPHA - default=false) --flex-volume-plugin-dir string Full path of the directory in which the flex volume plugin should search for additional third party volume plugins. (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/") - --google-json-key string The Google Cloud Platform Service Account JSON Key to use for authentication. --horizontal-pod-autoscaler-sync-period duration The period for syncing the number of pods in horizontal pod autoscaler. (default 30s) --insecure-experimental-approve-all-kubelet-csrs-for-group string The group for which the controller-manager will auto approve all CSRs for kubelet client certificates. --kube-api-burst int32 Burst to use while talking with Kubernetes apiserver (default 30) From f251dd790eba05146edb45cf2b00248e400e4972 Mon Sep 17 00:00:00 2001 From: chenhuan12 Date: Mon, 27 Feb 2017 14:52:51 +0800 Subject: [PATCH 358/407] fix typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1、Delete the parameter "--google-json-key string"; 2、Parameters "--ir-data-source string"、"--ir-dbname string"、"--ir-hawkular string"、"--ir-influxdb-host string"、"--ir-namespace-only"、"--ir-password string"、"--ir-percentile int"、"--ir-user string" is missed. see the "# kube-prxoy -help" # kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-23T00:04:39Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"f5ef9802914a47c848fd84c287333f8b4d28bbc1", GitTreeState:"dirty", BuildDate:"2017-01-22T23:56:57Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64", USEEVersion:"V1.02.01_alpha", USEEPublishDate:"2017-1-10 00:00:00"} --- docs/admin/kube-apiserver.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/admin/kube-apiserver.md b/docs/admin/kube-apiserver.md index 1e2c8a602e..51dbcac500 100644 --- a/docs/admin/kube-apiserver.md +++ b/docs/admin/kube-apiserver.md @@ -71,10 +71,17 @@ DynamicKubeletConfig=true|false (ALPHA - default=false) DynamicVolumeProvisioning=true|false (ALPHA - default=true) ExperimentalHostUserNamespaceDefaulting=true|false (ALPHA - default=false) StreamingProxyRedirects=true|false (ALPHA - default=false) - --google-json-key string The Google Cloud Platform Service Account JSON Key to use for authentication. --insecure-allow-any-token username/group1,group2 If set, your server will be INSECURE. Any token will be allowed and user information will be parsed from the token as username/group1,group2 --insecure-bind-address ip The IP address on which to serve the --insecure-port (set to 0.0.0.0 for all interfaces). Defaults to localhost. (default 127.0.0.1) --insecure-port int The port on which to serve unsecured, unauthenticated access. Default 8080. It is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. This is performed by nginx in the default setup. (default 8080) + --ir-data-source string Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb") +      --ir-dbname string                                       InfluxDB database name which contains metrics required by InitialResources. (default "k8s") + --ir-hawkular string Hawkular configuration URL. + --ir-influxdb-host string Address of InfluxDB which contains metrics required by InitialResources. (default "localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:api") + --ir-namespace-only Whether the estimation should be made only based on data from the same namespace. + --ir-password string Password used for connecting to InfluxDB. (default "root") + --ir-percentile int Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90) + --ir-user string User used for connecting to InfluxDB. (default "root") --kubelet-certificate-authority string Path to a cert file for the certificate authority. --kubelet-client-certificate string Path to a client cert file for TLS. --kubelet-client-key string Path to a client key file for TLS. From dd77deabedfb70e371d0dfcafd42be55b42ea790 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Tue, 7 Mar 2017 19:06:29 +0800 Subject: [PATCH 359/407] Update garbage-collection.md modify the url and link --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index ece46c7e07..3124de50b8 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -72,4 +72,4 @@ Including: | `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources | | `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources | -See [kubelet eviction design doc](https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/admin/out-of-resource.md) for more details. +See [Configuring Out Of Resource Handling](https://kubernetes.io/docs/admin/out-of-resource/) for more details. From 0e0440367cb01e16ec70e1d79bba18199fc5a4d0 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Wed, 8 Mar 2017 05:49:37 +0800 Subject: [PATCH 360/407] update garbage-collection.md change the url to relative path. --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 3124de50b8..ad7eabad9d 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -72,4 +72,4 @@ Including: | `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources | | `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources | -See [Configuring Out Of Resource Handling](https://kubernetes.io/docs/admin/out-of-resource/) for more details. +See [Configuring Out Of Resource Handling](/docs/admin/out-of-resource/) for more details. From 04d12196a12737e8366fbdea64a43f2b3c0982ab Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Tue, 7 Mar 2017 16:57:01 +0800 Subject: [PATCH 361/407] update out-of-resource.md change "in cases when" to "in case that" --- docs/admin/out-of-resource.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/out-of-resource.md b/docs/admin/out-of-resource.md index 30b8744624..acbc5e7a8b 100644 --- a/docs/admin/out-of-resource.md +++ b/docs/admin/out-of-resource.md @@ -18,7 +18,7 @@ If either resource is exhausted, the node would become unstable. ## Eviction Policy The `kubelet` can pro-actively monitor for and prevent against total starvation of a compute resource. In -cases where it could appear to occur, the `kubelet` can pro-actively fail one or more pods in order to reclaim +case that it appears to occur, the `kubelet` can pro-actively fail one or more pods in order to reclaim the starved resource. When the `kubelet` fails a pod, it terminates all containers in the pod, and the `PodPhase` is transitioned to `Failed`. From 582ef6dc4bee43a20b09b0a67d80c840f29883c5 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Wed, 8 Mar 2017 05:54:37 +0800 Subject: [PATCH 362/407] update out-of-resource.md use a shorter and simpler expression. --- docs/admin/out-of-resource.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/admin/out-of-resource.md b/docs/admin/out-of-resource.md index acbc5e7a8b..974454bd6d 100644 --- a/docs/admin/out-of-resource.md +++ b/docs/admin/out-of-resource.md @@ -17,8 +17,7 @@ If either resource is exhausted, the node would become unstable. ## Eviction Policy -The `kubelet` can pro-actively monitor for and prevent against total starvation of a compute resource. In -case that it appears to occur, the `kubelet` can pro-actively fail one or more pods in order to reclaim +The `kubelet` can pro-actively monitor for and prevent against total starvation of a compute resource. In those cases, the `kubelet` can pro-actively fail one or more pods in order to reclaim the starved resource. When the `kubelet` fails a pod, it terminates all containers in the pod, and the `PodPhase` is transitioned to `Failed`. From b3110014de4258313f82efdf4cfafd5a22f51c09 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang Date: Wed, 8 Mar 2017 09:45:54 +0800 Subject: [PATCH 363/407] update out-of-resource.md change "support" to "supports" --- docs/admin/out-of-resource.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/out-of-resource.md b/docs/admin/out-of-resource.md index 974454bd6d..76e7125e7e 100644 --- a/docs/admin/out-of-resource.md +++ b/docs/admin/out-of-resource.md @@ -35,7 +35,7 @@ summary API. | `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available` | | `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree` | -Each of the above signals support either a literal or percentage based value. The percentage based value +Each of the above signals supports either a literal or percentage based value. The percentage based value is calculated relative to the total capacity associated with each signal. `kubelet` supports only two filesystem partitions. From f7ad8644e9580c64ccc4d447d9dcb39c44cef45c Mon Sep 17 00:00:00 2001 From: Xin Zhang Date: Mon, 6 Mar 2017 17:10:14 +0800 Subject: [PATCH 364/407] Create a top-level CN directory to hold future md files for the Chinese kubernetes site --- cn/docs/.gitkeep | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 cn/docs/.gitkeep diff --git a/cn/docs/.gitkeep b/cn/docs/.gitkeep new file mode 100644 index 0000000000..e69de29bb2 From a551d2673e8ff16e1cf95a217b8c9ebd4b3bf2ef Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Wed, 8 Mar 2017 09:52:28 +0800 Subject: [PATCH 365/407] Removed `=` at the end of the flags. --- docs/admin/node.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/admin/node.md b/docs/admin/node.md index f91f155919..5e591ea99a 100644 --- a/docs/admin/node.md +++ b/docs/admin/node.md @@ -172,9 +172,9 @@ register itself with the API server. This is the preferred pattern, used by mos For self-registration, the kubelet is started with the following options: - - `--api-servers=` - Location of the apiservers. - - `--kubeconfig=` - Path to credentials to authenticate itself to the apiserver. - - `--cloud-provider=` - How to talk to a cloud provider to read metadata about itself. + - `--api-servers` - Location of the apiservers. + - `--kubeconfig` - Path to credentials to authenticate itself to the apiserver. + - `--cloud-provider` - How to talk to a cloud provider to read metadata about itself. - `--register-node` - Automatically register with the API server. - `--node-ip` IP address of the node. - `--node-labels` - Labels to add when registering the node in the cluster. From 6d30fc54daa8bdf6016df3cf11c5a406b9588a99 Mon Sep 17 00:00:00 2001 From: Oleg Shalygin Date: Sun, 12 Mar 2017 19:11:52 -0700 Subject: [PATCH 366/407] Remove autogenerate todo - This should be captured via a GitHub issue and not a TODO in the README documentation which leads to confusion. --- docs/user-guide/kubectl-overview.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/user-guide/kubectl-overview.md b/docs/user-guide/kubectl-overview.md index 987e154703..cc88f1b64d 100644 --- a/docs/user-guide/kubectl-overview.md +++ b/docs/user-guide/kubectl-overview.md @@ -7,8 +7,6 @@ title: kubectl Overview `kubectl` is a command line interface for running commands against Kubernetes clusters. This overview covers `kubectl` syntax, describes the command operations, and provides common examples. For details about each command, including all the supported flags and subcommands, see the [kubectl](/docs/user-guide/kubectl) reference documentation. For installation instructions see [prerequisites](/docs/user-guide/prereqs). -TODO: Auto-generate this file to ensure it's always in sync with any `kubectl` changes, see [#14177](http://pr.k8s.io/14177). - ## Syntax Use the following syntax to run `kubectl` commands from your terminal window: From 4dce53334c0b6a83d0b2737a3168b659e31e5536 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Mon, 13 Mar 2017 21:32:17 -0700 Subject: [PATCH 367/407] Move Guide topics: Federation Tasks. (#2799) --- _data/tasks.yml | 11 + docs/tasks/administer-federation/configmap.md | 87 +++++ docs/tasks/administer-federation/daemonset.md | 83 ++++ .../tasks/administer-federation/deployment.md | 108 ++++++ docs/tasks/administer-federation/events.md | 40 ++ docs/tasks/administer-federation/ingress.md | 356 ++++++++++++++++++ .../tasks/administer-federation/namespaces.md | 90 +++++ .../tasks/administer-federation/replicaset.md | 105 ++++++ docs/tasks/administer-federation/secret.md | 87 +++++ docs/user-guide/federation/configmap.md | 84 +---- docs/user-guide/federation/daemonsets.md | 80 +--- docs/user-guide/federation/deployment.md | 105 +----- docs/user-guide/federation/events.md | 37 +- .../federation/federated-ingress.md | 353 +---------------- docs/user-guide/federation/namespaces.md | 87 +---- docs/user-guide/federation/replicasets.md | 102 +---- docs/user-guide/federation/secrets.md | 84 +---- 17 files changed, 983 insertions(+), 916 deletions(-) create mode 100644 docs/tasks/administer-federation/configmap.md create mode 100644 docs/tasks/administer-federation/daemonset.md create mode 100644 docs/tasks/administer-federation/deployment.md create mode 100644 docs/tasks/administer-federation/events.md create mode 100644 docs/tasks/administer-federation/ingress.md create mode 100644 docs/tasks/administer-federation/namespaces.md create mode 100644 docs/tasks/administer-federation/replicaset.md create mode 100644 docs/tasks/administer-federation/secret.md diff --git a/_data/tasks.yml b/_data/tasks.yml index bfc5c9426f..b32c862449 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -47,6 +47,17 @@ toc: - docs/tasks/administer-cluster/safely-drain-node.md - docs/tasks/administer-cluster/change-pv-reclaim-policy.md +- title: Administering Federation + section: + - docs/tasks/administer-federation/configmap.md + - docs/tasks/administer-federation/daemonset.md + - docs/tasks/administer-federation/deployment.md + - docs/tasks/administer-federation/events.md + - docs/tasks/administer-federation/ingress.md + - docs/tasks/administer-federation/namespaces.md + - docs/tasks/administer-federation/replicaset.md + - docs/tasks/administer-federation/secret.md + - title: Managing Stateful Applications section: - docs/tasks/manage-stateful-set/upgrade-pet-set-to-stateful-set.md diff --git a/docs/tasks/administer-federation/configmap.md b/docs/tasks/administer-federation/configmap.md new file mode 100644 index 0000000000..9d8aa18cac --- /dev/null +++ b/docs/tasks/administer-federation/configmap.md @@ -0,0 +1,87 @@ +--- +title: Federated ConfigMap +--- + +This guide explains how to use ConfigMaps in a Federation control plane. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). +Other tutorials, such as Kelsey Hightower's +[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation), +might also help you create a Federated Kubernetes cluster. + +You should also have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general and [ConfigMaps](/docs/user-guide/ConfigMaps/) in particular. + +## Overview + +Federated ConfigMaps are very similar to the traditional [Kubernetes +ConfigMaps](/docs/user-guide/configmap/) and provide the same functionality. +Creating them in the federation control plane ensures that they are synchronized +across all the clusters in federation. + + +## Creating a Federated ConfigMap + +The API for Federated ConfigMap is 100% compatible with the +API for traditional Kubernetes ConfigMap. You can create a ConfigMap by sending +a request to the federation apiserver. + +You can do that using [kubectl](/docs/user-guide/kubectl/) by running: + +``` shell +kubectl --context=federation-cluster create -f myconfigmap.yaml +``` + +The `--context=federation-cluster` flag tells kubectl to submit the +request to the Federation apiserver instead of sending it to a Kubernetes +cluster. + +Once a Federated ConfigMap is created, the federation control plane will create +a matching ConfigMap in all underlying Kubernetes clusters. +You can verify this by checking each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get configmap myconfigmap +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. + +These ConfigMaps in underlying clusters will match the Federated ConfigMap. + + +## Updating a Federated ConfigMap + +You can update a Federated ConfigMap as you would update a Kubernetes +ConfigMap; however, for a Federated ConfigMap, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. +The federation control plane ensures that whenever the Federated ConfigMap is +updated, it updates the corresponding ConfigMaps in all underlying clusters to +match it. + +## Deleting a Federated ConfigMap + +You can delete a Federated ConfigMap as you would delete a Kubernetes +ConfigMap; however, for a Federated ConfigMap, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. + +For example, you can do that using kubectl by running: + +```shell +kubectl --context=federation-cluster delete configmap +``` + +Note that at this point, deleting a Federated ConfigMap will not delete the +corresponding ConfigMaps from underlying clusters. +You must delete the underlying ConfigMaps manually. +We intend to fix this in the future. diff --git a/docs/tasks/administer-federation/daemonset.md b/docs/tasks/administer-federation/daemonset.md new file mode 100644 index 0000000000..e5a558c844 --- /dev/null +++ b/docs/tasks/administer-federation/daemonset.md @@ -0,0 +1,83 @@ +--- +title: Federated DaemonSet +--- + +This guide explains how to use DaemonSets in a federation control plane. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). +Other tutorials, such as Kelsey Hightower's +[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation), +might also help you create a Federated Kubernetes cluster. + +You should also have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general and DaemonSets in particular. + +## Overview + +DaemonSets in federation control plane ("Federated Daemonsets" in +this guide) are very similar to the traditional [Kubernetes +DaemonSets](/docs/user-guide/DaemonSets/) and provide the same functionality. +Creating them in the federation control plane ensures that they are synchronized +across all the clusters in federation. + + +## Creating a Federated Daemonset + +The API for Federated Daemonset is 100% compatible with the +API for traditional Kubernetes DaemonSet. You can create a DaemonSet by sending +a request to the federation apiserver. + +You can do that using [kubectl](/docs/user-guide/kubectl/) by running: + +``` shell +kubectl --context=federation-cluster create -f mydaemonset.yaml +``` + +The `--context=federation-cluster` flag tells kubectl to submit the +request to the Federation apiserver instead of sending it to a Kubernetes +cluster. + +Once a Federated Daemonset is created, the federation control plane will create +a matching DaemonSet in all underlying Kubernetes clusters. +You can verify this by checking each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get daemonset mydaemonset +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. + +These DaemonSets in underlying clusters will match the Federated Daemonset. + + +## Updating a Federated Daemonset + +You can update a Federated Daemonset as you would update a Kubernetes +DaemonSet; however, for a Federated Daemonset, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. +The federation control plane ensures that whenever the Federated Daemonset is +updated, it updates the corresponding DaemonSets in all underlying clusters to +match it. + +## Deleting a Federated Daemonset + +You can delete a Federated Daemonset as you would delete a Kubernetes +DaemonSet; however, for a Federated Daemonset, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. + +For example, you can do that using kubectl by running: + +```shell +kubectl --context=federation-cluster delete daemonset mydaemonset +``` diff --git a/docs/tasks/administer-federation/deployment.md b/docs/tasks/administer-federation/deployment.md new file mode 100644 index 0000000000..51b092971d --- /dev/null +++ b/docs/tasks/administer-federation/deployment.md @@ -0,0 +1,108 @@ +--- +title: Federated Deployment +--- + +This guide explains how to use Deployments in the Federation control plane. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). +Other tutorials, such as Kelsey Hightower's +[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation), +might also help you create a Federated Kubernetes cluster. + +You should also have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general and [Deployment](/docs/user-guide/deployments) in particular. + +## Overview + +Deployments in federation control plane (referred to as "Federated Deployments" in +this guide) are very similar to the traditional [Kubernetes +Deployment](/docs/user-guide/deployments/), and provide the same functionality. +Creating them in the federation control plane ensures that the desired number of +replicas exist across the registered clusters. + +**As of Kubernetes version 1.5, Federated Deployment is an Alpha feature. The core +functionality of Deployment is present, but some features +(such as full rollout compatibility) are still in development.** + +## Creating a Federated Deployment + +The API for Federated Deployment is compatible with the +API for traditional Kubernetes Deployment. You can create a Deployment by sending +a request to the federation apiserver. + +You can do that using [kubectl](/docs/user-guide/kubectl/) by running: + +``` shell +kubectl --context=federation-cluster create -f mydeployment.yaml +``` + +The '--context=federation-cluster' flag tells kubectl to submit the +request to the Federation apiserver instead of sending it to a Kubernetes +cluster. + +Once a Federated Deployment is created, the federation control plane will create +a Deployment in all underlying Kubernetes clusters. +You can verify this by checking each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get deployment mydep +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. + +These Deployments in underlying clusters will match the federation Deployment +_except_ in the number of replicas and revision-related annotations. +Federation control plane ensures that the +sum of replicas in each cluster combined matches the desired number of replicas in the +Federated Deployment. + +### Spreading Replicas in Underlying Clusters + +By default, replicas are spread equally in all the underlying clusters. For ex: +if you have 3 registered clusters and you create a Federated Deployment with +`spec.replicas = 9`, then each Deployment in the 3 clusters will have +`spec.replicas=3`. +To modify the number of replicas in each cluster, you can specify +[FederatedReplicaSetPreference](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go) +as an annotation with key `federation.kubernetes.io/deployment-preferences` +on Federated Deployment. + + +## Updating a Federated Deployment + +You can update a Federated Deployment as you would update a Kubernetes +Deployment; however, for a Federated Deployment, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. +The federation control plane ensures that whenever the Federated Deployment is +updated, it updates the corresponding Deployments in all underlying clusters to +match it. So if the rolling update strategy was chosen then the underlying +cluster will do the rolling update independently and `maxSurge` and `maxUnavailable` +will apply only to individual clusters. This behavior may change in the future. + +If your update includes a change in number of replicas, the federation +control plane will change the number of replicas in underlying clusters to +ensure that their sum remains equal to the number of desired replicas in +Federated Deployment. + +## Deleting a Federated Deployment + +You can delete a Federated Deployment as you would delete a Kubernetes +Deployment; however, for a Federated Deployment, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. + +For example, you can do that using kubectl by running: + +```shell +kubectl --context=federation-cluster delete deployment mydep +``` diff --git a/docs/tasks/administer-federation/events.md b/docs/tasks/administer-federation/events.md new file mode 100644 index 0000000000..1d9f72ea0e --- /dev/null +++ b/docs/tasks/administer-federation/events.md @@ -0,0 +1,40 @@ +--- +title: Federated Events +--- + +This guide explains how to use events in federation control plane to help in debugging. + + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). Other tutorials, for example +[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) +by Kelsey Hightower, are also available to help you. + +You are also expected to have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general. + +## Overview + +Events in federation control plane (referred to as "federation events" in +this guide) are very similar to the traditional Kubernetes +Events providing the same functionality. +Federation Events are stored only in federation control plane and are not passed on to the underlying Kubernetes clusters. + +Federation controllers create events as they process API resources to surface to the +user, the state that they are in. +You can get all events from federation apiserver by running: + +```shell +kubectl --context=federation-cluster get events +``` + +The standard kubectl get, update, delete commands will all work. diff --git a/docs/tasks/administer-federation/ingress.md b/docs/tasks/administer-federation/ingress.md new file mode 100644 index 0000000000..0daa41ed49 --- /dev/null +++ b/docs/tasks/administer-federation/ingress.md @@ -0,0 +1,356 @@ +--- +title: Federated Ingress +--- + +This guide explains how to use Kubernetes Federated Ingress to deploy +a common HTTP(S) virtual IP load balancer across a federated service running in +multiple Kubernetes clusters. As of v1.4, clusters hosted in Google +Cloud (both GKE and GCE, or both) are supported. This makes it +easy to deploy a service that reliably serves HTTP(S) traffic +originating from web clients around the globe on a single, static IP +address. Low +network latency, high fault tolerance and easy administration are +ensured through intelligent request routing and automatic replica +relocation (using [Federated ReplicaSets](docs/user-guide/federation/federated-replicaset.md)). +Clients are automatically routed, via the shortest network path, to +the cluster closest to them with available capacity (despite the fact +that all clients use exactly the same static IP address). The load balancer +automatically checks the health of the pods comprising the service, +and avoids sending requests to unresponsive or slow pods (or entire +unresponsive clusters). + +Federated Ingress is released as an alpha feature, and supports Google Cloud Platform (GKE, +GCE and hybrid scenarios involving both) in Kubernetes v1.4. Work is under way to support other cloud +providers such as AWS, and other hybrid cloud scenarios (e.g. services +spanning private on-premise as well as public cloud Kubernetes +clusters). We welcome your feedback. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). Other tutorials, for example +[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) +by Kelsey Hightower, are also available to help you. + +You are also expected to have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general, and [Ingress](/docs/user-guide/ingress/) in particular. + +## Overview + +Federated Ingresses are created in much that same way as traditional +[Kubernetes Ingresses](/docs/user-guide/ingress/): by making an API +call which specifies the desired properties of your logical ingress point. In the +case of Federated Ingress, this API call is directed to the +Federation API endpoint, rather than a Kubernetes cluster API +endpoint. The API for Federated Ingress is 100% compatible with the +API for traditional Kubernetes Services. + +Once created, the Federated Ingress automatically: + +1. creates matching Kubernetes Ingress objects in every cluster +underlying your Cluster Federation, +2. ensures that all of these in-cluster ingress objects share the same + logical global L7 (i.e. HTTP(S)) load balancer and IP address. +3. monitors the health and capacity of the service "shards" (i.e. your + pods) behind this ingress in each cluster +4. ensures that all client connections are routed to an appropriate +healthy backend service endpoint at all times, even in the event of +pod, cluster, +availability zone or regional outages. + +Note that in the case of Google Cloud, the logical L7 load balancer is +not a single physical device (which would present both a single point +of failure, and a single global network routing choke point), but +rather a +[truly global, highly available load balancing managed service](https://cloud.google.com/load-balancing/), +globally reachable via a single, static IP address. + +Clients inside your federated Kubernetes clusters (i.e. Pods) will be +automatically routed to the cluster-local shard of the Federated Service +backing the Ingress in their +cluster if it exists and is healthy, or the closest healthy shard in a +different cluster if it does not. Note that this involves a network +trip to the HTTP(s) load balancer, which resides outside your local +Kubernetes cluster but inside the same GCP region. + +## Creating a federated ingress + +You can create a federated ingress in any of the usual ways, for example using kubectl: + +``` shell +kubectl --context=federation-cluster create -f myingress.yaml +``` +For example ingress YAML configurations, see the [Ingress User Guide](/docs/user-guide/ingress/) +The '--context=federation-cluster' flag tells kubectl to submit the +request to the Federation API endpoint, with the appropriate +credentials. If you have not yet configured such a context, visit the +[federation admin guide](/docs/admin/federation/) or one of the +[administration tutorials](https://github.com/kelseyhightower/kubernetes-cluster-federation) +to find out how to do so. + +As described above, the Federated Ingress will automatically create +and maintain matching Kubernetes ingresses in all of the clusters +underlying your federation. These cluster-specific ingresses (and +their associated ingress controllers) configure and manage the load +balancing and health checking infrastructure that ensures that traffic +is load balanced to each cluster appropriately. + +You can verify this by checking in each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get ingress myingress +NAME HOSTS ADDRESS PORTS AGE +myingress * 130.211.5.194 80, 443 1m +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. The name and +namespace of the underlying ingress will automatically match those of +the Federated Ingress that you created above (and if you happen to +have had ingresses of the same name and namespace already existing in +any of those clusters, they will be automatically adopted by the +Federation and updated to conform with the specification of your +Federated Ingress - either way, the end result will be the same). + +The status of your Federated Ingress will automatically reflect the +real-time status of the underlying Kubernetes ingresses, for example: + +``` shell +$kubectl --context=federation-cluster describe ingress myingress + +Name: myingress +Namespace: default +Address: 130.211.5.194 +TLS: + tls-secret terminates +Rules: + Host Path Backends + ---- ---- -------- + * * echoheaders-https:80 (10.152.1.3:8080,10.152.2.4:8080) +Annotations: + https-target-proxy: k8s-tps-default-myingress--ff1107f83ed600c0 + target-proxy: k8s-tp-default-myingress--ff1107f83ed600c0 + url-map: k8s-um-default-myingress--ff1107f83ed600c0 + backends: {"k8s-be-30301--ff1107f83ed600c0":"Unknown"} + forwarding-rule: k8s-fw-default-myingress--ff1107f83ed600c0 + https-forwarding-rule: k8s-fws-default-myingress--ff1107f83ed600c0 +Events: + FirstSeen LastSeen Count From SubobjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 3m 3m 1 {loadbalancer-controller } Normal ADD default/myingress + 2m 2m 1 {loadbalancer-controller } Normal CREATE ip: 130.211.5.194 +``` + +Note that: + +1. the address of your Federated Ingress +corresponds with the address of all of the +underlying Kubernetes ingresses (once these have been allocated - this +may take up to a few minutes). +2. we have not yet provisioned any backend Pods to receive +the network traffic directed to this ingress (i.e. 'Service +Endpoints' behind the service backing the Ingress), so the Federated Ingress does not yet consider these to +be healthy shards and will not direct traffic to any of these clusters. +3. the federation control system will +automatically reconfigure the load balancer controllers in all of the +clusters in your federation to make them consistent, and allow +them to share global load balancers. But this reconfiguration can +only complete successfully if there are no pre-existing Ingresses in +those clusters (this is a safety feature to prevent accidental +breakage of existing ingresses). So to ensure that your federated +ingresses function correctly, either start with new, empty clusters, or make +sure that you delete (and recreate if necessary) all pre-existing +Ingresses in the clusters comprising your federation. + +#Adding backend services and pods + +To render the underlying ingress shards healthy, we need to add +backend Pods behind the service upon which the Ingress is based. There are several ways to achieve this, but +the easiest is to create a [Federated Service](federated-services.md) and +[Federated Replicaset](federated-replicasets.md). Details of how those +work are covered in the aforementioned user guides - here we'll simply use them, to +create appropriately labelled pods and services in the 13 underlying clusters of +our federation: + +``` shell +kubectl --context=federation-cluster create -f services/nginx.yaml +``` + +``` shell + kubectl --context=federation-cluster create -f myreplicaset.yaml +``` + +Note that in order for your federated ingress to work correctly on +Google Cloud, the node ports of all of the underlying cluster-local +services need to be identical. If you're using a federated service +this is easy to do. Simply pick a node port that is not already +being used in any of your clusters, and add that to the spec of your +federated service. If you do not specify a node port for your +federated service, each cluster will choose it's own node port for +its cluster-local shard of the service, and these will probably end +up being different, which is not what you want. + +You can verify this by checking in each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get services nginx +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +nginx 10.63.250.98 104.199.136.89 80/TCP 9m +``` + + +## Hybrid cloud capabilities + +Federations of Kubernetes Clusters can include clusters running in +different cloud providers (e.g. Google Cloud, AWS), and on-premises +(e.g. on OpenStack). However, in Kubernetes v1.4, Federated Ingress is only +supported across Google Cloud clusters. In future versions we intend +to support hybrid cloud Ingress-based deployments. + +## Discovering a federated ingress + +Ingress objects (in both plain Kubernets clusters, and in federations +of clusters) expose one or more IP addresses (via +the Status.Loadbalancer.Ingress field) that remains static for the lifetime +of the Ingress object (in future, automatically managed DNS names +might also be added). All clients (whether internal to your cluster, +or on the external network or internet) should connect to one of these IP +or DNS addresses. As mentioned above, all client requests are automatically +routed, via the shortest network path, to a healthy pod in the +closest cluster to the origin of the request. So for example, HTTP(S) +requests from internet +users in Europe will be routed directly to the closest cluster in +Europe that has available capacity. If there are no such clusters in +Europe, the request will be routed to the next closest cluster +(typically in the U.S.). + +## Handling failures of backend pods and whole clusters + +Ingresses are backed by Services, which are typically (but not always) +backed by one or more ReplicaSets. For Federated Ingresses, it is +common practise to use the federated variants of Services and +ReplicaSets (see [Federated Services](federated-services.md) and +[Federated ReplicaSets](federated-replicasets.md)) for this purpose, as +described above. + +In particular, Federated ReplicaSets ensure that the desired number of +pods are kept running in each cluster, even in the event of node +failures. In the event of entire cluster or availability zone +failures, Federated ReplicaSets automatically place additional +replacas in the other available clusters in the federation to accommodate the +traffic which was previously being served by the now unavailable +cluster. While the Federated ReplicaSet ensures that sufficient replicas are +kept running, the Federated Ingress ensures that user traffic is +automatically redirected away from the failed cluster to other +available clusters. + +## Known issue + +GCE L7 load balancer back-ends and health checks are known to "flap"; this is due +to conflicting firewall rules in the federation's underlying clusters, which might override one another. To work around this problem, you can +install the firewall rules manually to expose the targets of all the +underlying clusters in your federation for each Federated Ingress +object. This way, the health checks can consistently pass and the GCE L7 load balancer +can remain stable. You install the rules using the +[`gcloud`](https://cloud.google.com/sdk/gcloud/) command line tool, +[Google Cloud Console](https://console.cloud.google.com) or the +[Google Compute Engine APIs](https://cloud.google.com/compute/docs/reference/latest/). + +You can install these rules using +[`gcloud`](https://cloud.google.com/sdk/gcloud/) as follows: + +```shell +gcloud compute firewall-rules create \ + --source-ranges 130.211.0.0/22 --allow [] \ + --target-tags [] \ + --network +``` + +where: + +1. `firewall-rule-name` can be any name. +2. `[]` is the comma separated list of node ports corresponding to the services that back the Federated Ingress. +3. [] is the comma separated list of the target tags assigned to the nodes in a Kubernetes cluster. +4. is the name of the network where the firewall rule must be installed. + +Example: +```shell +gcloud compute firewall-rules create my-federated-ingress-firewall-rule \ + --source-ranges 130.211.0.0/22 --allow tcp:30301, tcp:30061, tcp:34564 \ + --target-tags my-cluster-1-minion, my-cluster-2-minion \ + --network default +``` + + +## Troubleshooting + +#### I cannot connect to my cluster federation API +Check that your + +1. Client (typically kubectl) is correctly configured (including API endpoints and login credentials), and +2. Cluster Federation API server is running and network-reachable. + +See the [federation admin guide](/docs/admin/federation/) to learn +how to bring up a cluster federation correctly (or have your cluster administrator do this for you), and how to correctly configure your client. + +#### I can create a federated ingress/service/replicaset successfully against the cluster federation API, but no matching ingresses/services/replicasets are created in my underlying clusters + +Check that: + +1. Your clusters are correctly registered in the Cluster Federation API (`kubectl describe clusters`) +2. Your clusters are all 'Active'. This means that the cluster + Federation system was able to connect and authenticate against the + clusters' endpoints. If not, consult the event logs of the federation-controller-manager pod to ascertain what the failure might be. (`kubectl --namespace=federation logs $(kubectl get pods --namespace=federation -l module=federation-controller-manager -oname`) +3. That the login credentials provided to the Cluster Federation API + for the clusters have the correct authorization and quota to create + ingresses/services/replicasets in the relevant namespace in the + clusters. Again you should see associated error messages providing + more detail in the above event log file if this is not the case. +4. Whether any other error is preventing the service creation + operation from succeeding (look for `ingress-controller`, + `service-controller` or `replicaset-controller`, + errors in the output of `kubectl logs federation-controller-manager --namespace federation`). + +#### I can create a federated ingress successfully, but request load is not correctly distributed across the underlying clusters + +Check that: + +1. the services underlying your federated ingress in each cluster have + identical node ports. See [above](#creating_a_federated_ingress) for further explanation. +2. the load balancer controllers in each of your clusters are of the + correct type ("GLBC") and have been correctly reconfigured by the + federation control plane to share a global GCE load balancer (this + should happen automatically). If they of the correct type, and + have been correctly reconfigured, the UID data item in the GLBC + configmap in each cluster will be identical across all clusters. + See + [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) + for further details. + If this is not the case, check the logs of your federation + controller manager to determine why this automated reconfiguration + might be failing. +3. no ingresses have been manually created in any of your clusters before the above + reconfiguration of the load balancer controller completed + successfully. Ingresses created before the reconfiguration of + your GLBC will interfere with the behavior of your federated + ingresses created after the reconfiguration (see + [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) + for further information. To remedy this, + delete any ingresses created before the cluster joined the + federation (and had it's GLBC reconfigured), and recreate them if + necessary. + +#### This troubleshooting guide did not help me solve my problem + +Please use one of our [support channels](http://kubernetes.io/docs/troubleshooting/) to seek assistance. + +## For more information + + * [Federation proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/federation.md) details use cases that motivated this work. diff --git a/docs/tasks/administer-federation/namespaces.md b/docs/tasks/administer-federation/namespaces.md new file mode 100644 index 0000000000..5ed3ffafec --- /dev/null +++ b/docs/tasks/administer-federation/namespaces.md @@ -0,0 +1,90 @@ +--- +title: Federated Namespaces +--- + +This guide explains how to use namespaces in Federation control plane. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). Other tutorials, for example +[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) +by Kelsey Hightower, are also available to help you. + +You are also expected to have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general and [Namespaces](/docs/user-guide/namespaces/) in particular. + +## Overview + +Namespaces in federation control plane (referred to as "federated namespaces" in +this guide) are very similar to the traditional [Kubernetes +Namespaces](/docs/user-guide/namespaces/) providing the same functionality. +Creating them in the federation control plane ensures that they are synchronized +across all the clusters in federation. + + +## Creating a Federated Namespace + +The API for Federated Namespaces is 100% compatible with the +API for traditional Kubernetes Namespaces. You can create a namespace by sending +a request to the federation apiserver. + +You can do that using kubectl by running: + +``` shell +kubectl --context=federation-cluster create -f myns.yaml +``` + +The '--context=federation-cluster' flag tells kubectl to submit the +request to the Federation apiserver instead of sending it to a Kubernetes +cluster. + +Once a federated namespace is created, the federation control plane will create +a matching namespace in all underlying Kubernetes clusters. +You can verify this by checking each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get namespaces myns +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. The name and +spec of the underlying namespace will match those of +the Federated Namespace that you created above. + + +## Updating a Federated Namespace + +You can update a federated namespace as you would update a Kubernetes +namespace, just send the request to federation apiserver instead of sending it +to a specific Kubernetes cluster. +Federation control plan will ensure that whenever the federated namespace is +updated, it updates the corresponding namespaces in all underlying clusters to +match it. + +## Deleting a Federated Namespace + +You can delete a federated namespace as you would delete a Kubernetes +namespace, just send the request to federation apiserver instead of sending it +to a specific Kubernetes cluster. + +For example, you can do that using kubectl by running: + +```shell +kubectl --context=federation-cluster delete ns myns +``` + +As in Kubernetes, deleting a federated namespace will delete all resources in that +namespace from the federation control plane. + +Note that at this point, deleting a federated namespace will not delete the +corresponding namespaces and resources in those namespaces from underlying clusters. +Users are expected to delete them manually. +We intend to fix this in the future. diff --git a/docs/tasks/administer-federation/replicaset.md b/docs/tasks/administer-federation/replicaset.md new file mode 100644 index 0000000000..c173a38996 --- /dev/null +++ b/docs/tasks/administer-federation/replicaset.md @@ -0,0 +1,105 @@ +--- +title: Federated ReplicaSets +--- + +This guide explains how to use replica sets in the Federation control plane. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). Other tutorials, for example +[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) +by Kelsey Hightower, are also available to help you. + +You are also expected to have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general and [ReplicaSets](/docs/user-guide/replicasets/) in particular. + +## Overview + +Replica Sets in federation control plane (referred to as "federated replica sets" in +this guide) are very similar to the traditional [Kubernetes +ReplicaSets](/docs/user-guide/replicasets/), and provide the same functionality. +Creating them in the federation control plane ensures that the desired number of +replicas exist across the registered clusters. + + +## Creating a Federated Replica Set + +The API for Federated Replica Set is 100% compatible with the +API for traditional Kubernetes Replica Set. You can create a replica set by sending +a request to the federation apiserver. + +You can do that using [kubectl](/docs/user-guide/kubectl/) by running: + +``` shell +kubectl --context=federation-cluster create -f myrs.yaml +``` + +The '--context=federation-cluster' flag tells kubectl to submit the +request to the Federation apiserver instead of sending it to a Kubernetes +cluster. + +Once a federated replica set is created, the federation control plane will create +a replica set in all underlying Kubernetes clusters. +You can verify this by checking each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get rs myrs +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. + +These replica sets in underlying clusters will match the federation replica set +except in the number of replicas. Federation control plane will ensure that the +sum of replicas in each cluster match the desired number of replicas in the +federation replica set. + +### Spreading Replicas in Underlying Clusters + +By default, replicas are spread equally in all the underlying clusters. For ex: +if you have 3 registered clusters and you create a federated replica set with +`spec.replicas = 9`, then each replica set in the 3 clusters will have +`spec.replicas=3`. +To modify the number of replicas in each cluster, you can specify +[FederatedReplicaSetPreference](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go) +as an annotation with key `federation.kubernetes.io/replica-set-preferences` +on federated replica set. + + +## Updating a Federated Replica Set + +You can update a federated replica set as you would update a Kubernetes +replica set; however, for a federated replica set, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. +The Federation control plan ensures that whenever the federated replica set is +updated, it updates the corresponding replica sets in all underlying clusters to +match it. +If your update includes a change in number of replicas, the federation +control plane will change the number of replicas in underlying clusters to +ensure that their sum remains equal to the number of desired replicas in +federated replica set. + +## Deleting a Federated Replica Set + +You can delete a federated replica set as you would delete a Kubernetes +replica set; however, for a federated replica set, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. + +For example, you can do that using kubectl by running: + +```shell +kubectl --context=federation-cluster delete rs myrs +``` + +Note that at this point, deleting a federated replica set will not delete the +corresponding replica sets from underlying clusters. +You must delete the underlying Replica Sets manually. +We intend to fix this in the future. diff --git a/docs/tasks/administer-federation/secret.md b/docs/tasks/administer-federation/secret.md new file mode 100644 index 0000000000..2c5eac6dba --- /dev/null +++ b/docs/tasks/administer-federation/secret.md @@ -0,0 +1,87 @@ +--- +title: Federated Secrets +--- + +This guide explains how to use secrets in Federation control plane. + +* TOC +{:toc} + +## Prerequisites + +This guide assumes that you have a running Kubernetes Cluster +Federation installation. If not, then head over to the +[federation admin guide](/docs/admin/federation/) to learn how to +bring up a cluster federation (or have your cluster administrator do +this for you). Other tutorials, for example +[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) +by Kelsey Hightower, are also available to help you. + +You are also expected to have a basic +[working knowledge of Kubernetes](/docs/getting-started-guides/) in +general and [Secrets](/docs/user-guide/secrets/) in particular. + +## Overview + +Secrets in federation control plane (referred to as "federated secrets" in +this guide) are very similar to the traditional [Kubernetes +Secrets](/docs/user-guide/secrets/) providing the same functionality. +Creating them in the federation control plane ensures that they are synchronized +across all the clusters in federation. + + +## Creating a Federated Secret + +The API for Federated Secret is 100% compatible with the +API for traditional Kubernetes Secret. You can create a secret by sending +a request to the federation apiserver. + +You can do that using [kubectl](/docs/user-guide/kubectl/) by running: + +``` shell +kubectl --context=federation-cluster create -f mysecret.yaml +``` + +The '--context=federation-cluster' flag tells kubectl to submit the +request to the Federation apiserver instead of sending it to a Kubernetes +cluster. + +Once a federated secret is created, the federation control plane will create +a matching secret in all underlying Kubernetes clusters. +You can verify this by checking each of the underlying clusters, for example: + +``` shell +kubectl --context=gce-asia-east1a get secret mysecret +``` + +The above assumes that you have a context named 'gce-asia-east1a' +configured in your client for your cluster in that zone. + +These secrets in underlying clusters will match the federated secret. + + +## Updating a Federated Secret + +You can update a federated secret as you would update a Kubernetes +secret; however, for a federated secret, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. +The Federation control plan ensures that whenever the federated secret is +updated, it updates the corresponding secrets in all underlying clusters to +match it. + +## Deleting a Federated Secret + +You can delete a federated secret as you would delete a Kubernetes +secret; however, for a federated secret, you must send the request to +the federation apiserver instead of sending it to a specific Kubernetes cluster. + +For example, you can do that using kubectl by running: + +```shell +kubectl --context=federation-cluster delete secret mysecret +``` + +Note that at this point, deleting a federated secret will not delete the +corresponding secrets from underlying clusters. +You must delete the underlying secrets manually. +We intend to fix this in the future. diff --git a/docs/user-guide/federation/configmap.md b/docs/user-guide/federation/configmap.md index 9d8aa18cac..6929073cf8 100644 --- a/docs/user-guide/federation/configmap.md +++ b/docs/user-guide/federation/configmap.md @@ -2,86 +2,6 @@ title: Federated ConfigMap --- -This guide explains how to use ConfigMaps in a Federation control plane. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). -Other tutorials, such as Kelsey Hightower's -[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation), -might also help you create a Federated Kubernetes cluster. - -You should also have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and [ConfigMaps](/docs/user-guide/ConfigMaps/) in particular. - -## Overview - -Federated ConfigMaps are very similar to the traditional [Kubernetes -ConfigMaps](/docs/user-guide/configmap/) and provide the same functionality. -Creating them in the federation control plane ensures that they are synchronized -across all the clusters in federation. - - -## Creating a Federated ConfigMap - -The API for Federated ConfigMap is 100% compatible with the -API for traditional Kubernetes ConfigMap. You can create a ConfigMap by sending -a request to the federation apiserver. - -You can do that using [kubectl](/docs/user-guide/kubectl/) by running: - -``` shell -kubectl --context=federation-cluster create -f myconfigmap.yaml -``` - -The `--context=federation-cluster` flag tells kubectl to submit the -request to the Federation apiserver instead of sending it to a Kubernetes -cluster. - -Once a Federated ConfigMap is created, the federation control plane will create -a matching ConfigMap in all underlying Kubernetes clusters. -You can verify this by checking each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get configmap myconfigmap -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. - -These ConfigMaps in underlying clusters will match the Federated ConfigMap. - - -## Updating a Federated ConfigMap - -You can update a Federated ConfigMap as you would update a Kubernetes -ConfigMap; however, for a Federated ConfigMap, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. -The federation control plane ensures that whenever the Federated ConfigMap is -updated, it updates the corresponding ConfigMaps in all underlying clusters to -match it. - -## Deleting a Federated ConfigMap - -You can delete a Federated ConfigMap as you would delete a Kubernetes -ConfigMap; however, for a Federated ConfigMap, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. - -For example, you can do that using kubectl by running: - -```shell -kubectl --context=federation-cluster delete configmap -``` - -Note that at this point, deleting a Federated ConfigMap will not delete the -corresponding ConfigMaps from underlying clusters. -You must delete the underlying ConfigMaps manually. -We intend to fix this in the future. +[Federated ConfigMap](/docs/tasks/administer-federation/configmap/) diff --git a/docs/user-guide/federation/daemonsets.md b/docs/user-guide/federation/daemonsets.md index 7699ead4e3..ce1ae99b9e 100644 --- a/docs/user-guide/federation/daemonsets.md +++ b/docs/user-guide/federation/daemonsets.md @@ -2,82 +2,6 @@ title: Federated DaemonSet --- -This guide explains how to use DaemonSets in a federation control plane. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). -Other tutorials, such as Kelsey Hightower's -[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation), -might also help you create a Federated Kubernetes cluster. - -You should also have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and DaemonSets in particular. - -## Overview - -DaemonSets in federation control plane ("Federated Daemonsets" in -this guide) are very similar to the traditional [Kubernetes -DaemonSets](/docs/user-guide/DaemonSets/) and provide the same functionality. -Creating them in the federation control plane ensures that they are synchronized -across all the clusters in federation. - - -## Creating a Federated Daemonset - -The API for Federated Daemonset is 100% compatible with the -API for traditional Kubernetes DaemonSet. You can create a DaemonSet by sending -a request to the federation apiserver. - -You can do that using [kubectl](/docs/user-guide/kubectl/) by running: - -``` shell -kubectl --context=federation-cluster create -f mydaemonset.yaml -``` - -The `--context=federation-cluster` flag tells kubectl to submit the -request to the Federation apiserver instead of sending it to a Kubernetes -cluster. - -Once a Federated Daemonset is created, the federation control plane will create -a matching DaemonSet in all underlying Kubernetes clusters. -You can verify this by checking each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get daemonset mydaemonset -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. - -These DaemonSets in underlying clusters will match the Federated Daemonset. - - -## Updating a Federated Daemonset - -You can update a Federated Daemonset as you would update a Kubernetes -DaemonSet; however, for a Federated Daemonset, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. -The federation control plane ensures that whenever the Federated Daemonset is -updated, it updates the corresponding DaemonSets in all underlying clusters to -match it. - -## Deleting a Federated Daemonset - -You can delete a Federated Daemonset as you would delete a Kubernetes -DaemonSet; however, for a Federated Daemonset, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. - -For example, you can do that using kubectl by running: - -```shell -kubectl --context=federation-cluster delete daemonset mydaemonset -``` \ No newline at end of file +[Federated DaemonSet](/docs/tasks/administer-federation/daemonset/) diff --git a/docs/user-guide/federation/deployment.md b/docs/user-guide/federation/deployment.md index 51b092971d..df8456d709 100644 --- a/docs/user-guide/federation/deployment.md +++ b/docs/user-guide/federation/deployment.md @@ -2,107 +2,6 @@ title: Federated Deployment --- -This guide explains how to use Deployments in the Federation control plane. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). -Other tutorials, such as Kelsey Hightower's -[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation), -might also help you create a Federated Kubernetes cluster. - -You should also have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and [Deployment](/docs/user-guide/deployments) in particular. - -## Overview - -Deployments in federation control plane (referred to as "Federated Deployments" in -this guide) are very similar to the traditional [Kubernetes -Deployment](/docs/user-guide/deployments/), and provide the same functionality. -Creating them in the federation control plane ensures that the desired number of -replicas exist across the registered clusters. - -**As of Kubernetes version 1.5, Federated Deployment is an Alpha feature. The core -functionality of Deployment is present, but some features -(such as full rollout compatibility) are still in development.** - -## Creating a Federated Deployment - -The API for Federated Deployment is compatible with the -API for traditional Kubernetes Deployment. You can create a Deployment by sending -a request to the federation apiserver. - -You can do that using [kubectl](/docs/user-guide/kubectl/) by running: - -``` shell -kubectl --context=federation-cluster create -f mydeployment.yaml -``` - -The '--context=federation-cluster' flag tells kubectl to submit the -request to the Federation apiserver instead of sending it to a Kubernetes -cluster. - -Once a Federated Deployment is created, the federation control plane will create -a Deployment in all underlying Kubernetes clusters. -You can verify this by checking each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get deployment mydep -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. - -These Deployments in underlying clusters will match the federation Deployment -_except_ in the number of replicas and revision-related annotations. -Federation control plane ensures that the -sum of replicas in each cluster combined matches the desired number of replicas in the -Federated Deployment. - -### Spreading Replicas in Underlying Clusters - -By default, replicas are spread equally in all the underlying clusters. For ex: -if you have 3 registered clusters and you create a Federated Deployment with -`spec.replicas = 9`, then each Deployment in the 3 clusters will have -`spec.replicas=3`. -To modify the number of replicas in each cluster, you can specify -[FederatedReplicaSetPreference](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go) -as an annotation with key `federation.kubernetes.io/deployment-preferences` -on Federated Deployment. - - -## Updating a Federated Deployment - -You can update a Federated Deployment as you would update a Kubernetes -Deployment; however, for a Federated Deployment, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. -The federation control plane ensures that whenever the Federated Deployment is -updated, it updates the corresponding Deployments in all underlying clusters to -match it. So if the rolling update strategy was chosen then the underlying -cluster will do the rolling update independently and `maxSurge` and `maxUnavailable` -will apply only to individual clusters. This behavior may change in the future. - -If your update includes a change in number of replicas, the federation -control plane will change the number of replicas in underlying clusters to -ensure that their sum remains equal to the number of desired replicas in -Federated Deployment. - -## Deleting a Federated Deployment - -You can delete a Federated Deployment as you would delete a Kubernetes -Deployment; however, for a Federated Deployment, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. - -For example, you can do that using kubectl by running: - -```shell -kubectl --context=federation-cluster delete deployment mydep -``` +[Federated Deployment](/docs/tasks/administer-federation/deployment/) diff --git a/docs/user-guide/federation/events.md b/docs/user-guide/federation/events.md index 1d9f72ea0e..f148806bca 100644 --- a/docs/user-guide/federation/events.md +++ b/docs/user-guide/federation/events.md @@ -2,39 +2,6 @@ title: Federated Events --- -This guide explains how to use events in federation control plane to help in debugging. +{% include user-guide-content-moved.md %} - -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). Other tutorials, for example -[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) -by Kelsey Hightower, are also available to help you. - -You are also expected to have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general. - -## Overview - -Events in federation control plane (referred to as "federation events" in -this guide) are very similar to the traditional Kubernetes -Events providing the same functionality. -Federation Events are stored only in federation control plane and are not passed on to the underlying Kubernetes clusters. - -Federation controllers create events as they process API resources to surface to the -user, the state that they are in. -You can get all events from federation apiserver by running: - -```shell -kubectl --context=federation-cluster get events -``` - -The standard kubectl get, update, delete commands will all work. +[Federated Evemts](/docs/tasks/administer-federation/events/) diff --git a/docs/user-guide/federation/federated-ingress.md b/docs/user-guide/federation/federated-ingress.md index a137ace1f2..b9a3dc2f51 100644 --- a/docs/user-guide/federation/federated-ingress.md +++ b/docs/user-guide/federation/federated-ingress.md @@ -2,355 +2,6 @@ title: Federated Ingress --- -This guide explains how to use Kubernetes Federated Ingress to deploy -a common HTTP(S) virtual IP load balancer across a federated service running in -multiple Kubernetes clusters. As of v1.4, clusters hosted in Google -Cloud (both GKE and GCE, or both) are supported. This makes it -easy to deploy a service that reliably serves HTTP(S) traffic -originating from web clients around the globe on a single, static IP -address. Low -network latency, high fault tolerance and easy administration are -ensured through intelligent request routing and automatic replica -relocation (using [Federated ReplicaSets](docs/user-guide/federation/federated-replicaset.md)). -Clients are automatically routed, via the shortest network path, to -the cluster closest to them with available capacity (despite the fact -that all clients use exactly the same static IP address). The load balancer -automatically checks the health of the pods comprising the service, -and avoids sending requests to unresponsive or slow pods (or entire -unresponsive clusters). +{% include user-guide-content-moved.md %} -Federated Ingress is released as an alpha feature, and supports Google Cloud Platform (GKE, -GCE and hybrid scenarios involving both) in Kubernetes v1.4. Work is under way to support other cloud -providers such as AWS, and other hybrid cloud scenarios (e.g. services -spanning private on-premise as well as public cloud Kubernetes -clusters). We welcome your feedback. - -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). Other tutorials, for example -[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) -by Kelsey Hightower, are also available to help you. - -You are also expected to have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general, and [Ingress](/docs/user-guide/ingress/) in particular. - -## Overview - -Federated Ingresses are created in much that same way as traditional -[Kubernetes Ingresses](/docs/user-guide/ingress/): by making an API -call which specifies the desired properties of your logical ingress point. In the -case of Federated Ingress, this API call is directed to the -Federation API endpoint, rather than a Kubernetes cluster API -endpoint. The API for Federated Ingress is 100% compatible with the -API for traditional Kubernetes Services. - -Once created, the Federated Ingress automatically: - -1. creates matching Kubernetes Ingress objects in every cluster -underlying your Cluster Federation, -2. ensures that all of these in-cluster ingress objects share the same - logical global L7 (i.e. HTTP(S)) load balancer and IP address. -3. monitors the health and capacity of the service "shards" (i.e. your - pods) behind this ingress in each cluster -4. ensures that all client connections are routed to an appropriate -healthy backend service endpoint at all times, even in the event of -pod, cluster, -availability zone or regional outages. - -Note that in the case of Google Cloud, the logical L7 load balancer is -not a single physical device (which would present both a single point -of failure, and a single global network routing choke point), but -rather a -[truly global, highly available load balancing managed service](https://cloud.google.com/load-balancing/), -globally reachable via a single, static IP address. - -Clients inside your federated Kubernetes clusters (i.e. Pods) will be -automatically routed to the cluster-local shard of the Federated Service -backing the Ingress in their -cluster if it exists and is healthy, or the closest healthy shard in a -different cluster if it does not. Note that this involves a network -trip to the HTTP(s) load balancer, which resides outside your local -Kubernetes cluster but inside the same GCP region. - -## Creating a federated ingress - -You can create a federated ingress in any of the usual ways, for example using kubectl: - -``` shell -kubectl --context=federation-cluster create -f myingress.yaml -``` -For example ingress YAML configurations, see the [Ingress User Guide](/docs/user-guide/ingress/) -The '--context=federation-cluster' flag tells kubectl to submit the -request to the Federation API endpoint, with the appropriate -credentials. If you have not yet configured such a context, visit the -[federation admin guide](/docs/admin/federation/) or one of the -[administration tutorials](https://github.com/kelseyhightower/kubernetes-cluster-federation) -to find out how to do so. - -As described above, the Federated Ingress will automatically create -and maintain matching Kubernetes ingresses in all of the clusters -underlying your federation. These cluster-specific ingresses (and -their associated ingress controllers) configure and manage the load -balancing and health checking infrastructure that ensures that traffic -is load balanced to each cluster appropriately. - -You can verify this by checking in each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get ingress myingress -NAME HOSTS ADDRESS PORTS AGE -myingress * 130.211.5.194 80, 443 1m -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. The name and -namespace of the underlying ingress will automatically match those of -the Federated Ingress that you created above (and if you happen to -have had ingresses of the same name and namespace already existing in -any of those clusters, they will be automatically adopted by the -Federation and updated to conform with the specification of your -Federated Ingress - either way, the end result will be the same). - -The status of your Federated Ingress will automatically reflect the -real-time status of the underlying Kubernetes ingresses, for example: - -``` shell -$kubectl --context=federation-cluster describe ingress myingress - -Name: myingress -Namespace: default -Address: 130.211.5.194 -TLS: - tls-secret terminates -Rules: - Host Path Backends - ---- ---- -------- - * * echoheaders-https:80 (10.152.1.3:8080,10.152.2.4:8080) -Annotations: - https-target-proxy: k8s-tps-default-myingress--ff1107f83ed600c0 - target-proxy: k8s-tp-default-myingress--ff1107f83ed600c0 - url-map: k8s-um-default-myingress--ff1107f83ed600c0 - backends: {"k8s-be-30301--ff1107f83ed600c0":"Unknown"} - forwarding-rule: k8s-fw-default-myingress--ff1107f83ed600c0 - https-forwarding-rule: k8s-fws-default-myingress--ff1107f83ed600c0 -Events: - FirstSeen LastSeen Count From SubobjectPath Type Reason Message - --------- -------- ----- ---- ------------- -------- ------ ------- - 3m 3m 1 {loadbalancer-controller } Normal ADD default/myingress - 2m 2m 1 {loadbalancer-controller } Normal CREATE ip: 130.211.5.194 -``` - -Note that: - -1. the address of your Federated Ingress -corresponds with the address of all of the -underlying Kubernetes ingresses (once these have been allocated - this -may take up to a few minutes). -2. we have not yet provisioned any backend Pods to receive -the network traffic directed to this ingress (i.e. 'Service -Endpoints' behind the service backing the Ingress), so the Federated Ingress does not yet consider these to -be healthy shards and will not direct traffic to any of these clusters. -3. the federation control system will -automatically reconfigure the load balancer controllers in all of the -clusters in your federation to make them consistent, and allow -them to share global load balancers. But this reconfiguration can -only complete successfully if there are no pre-existing Ingresses in -those clusters (this is a safety feature to prevent accidental -breakage of existing ingresses). So to ensure that your federated -ingresses function correctly, either start with new, empty clusters, or make -sure that you delete (and recreate if necessary) all pre-existing -Ingresses in the clusters comprising your federation. - -#Adding backend services and pods - -To render the underlying ingress shards healthy, we need to add -backend Pods behind the service upon which the Ingress is based. There are several ways to achieve this, but -the easiest is to create a [Federated Service](federated-services.md) and -[Federated Replicaset](federated-replicasets.md). Details of how those -work are covered in the aforementioned user guides - here we'll simply use them, to -create appropriately labelled pods and services in the 13 underlying clusters of -our federation: - -``` shell -kubectl --context=federation-cluster create -f services/nginx.yaml -``` - -``` shell - kubectl --context=federation-cluster create -f myreplicaset.yaml -``` - -Note that in order for your federated ingress to work correctly on -Google Cloud, the node ports of all of the underlying cluster-local -services need to be identical. If you're using a federated service -this is easy to do. Simply pick a node port that is not already -being used in any of your clusters, and add that to the spec of your -federated service. If you do not specify a node port for your -federated service, each cluster will choose it's own node port for -its cluster-local shard of the service, and these will probably end -up being different, which is not what you want. - -You can verify this by checking in each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get services nginx -NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE -nginx 10.63.250.98 104.199.136.89 80/TCP 9m -``` - - -## Hybrid cloud capabilities - -Federations of Kubernetes Clusters can include clusters running in -different cloud providers (e.g. Google Cloud, AWS), and on-premises -(e.g. on OpenStack). However, in Kubernetes v1.4, Federated Ingress is only -supported across Google Cloud clusters. In future versions we intend -to support hybrid cloud Ingress-based deployments. - -## Discovering a federated ingress - -Ingress objects (in both plain Kubernets clusters, and in federations -of clusters) expose one or more IP addresses (via -the Status.Loadbalancer.Ingress field) that remains static for the lifetime -of the Ingress object (in future, automatically managed DNS names -might also be added). All clients (whether internal to your cluster, -or on the external network or internet) should connect to one of these IP -or DNS addresses. As mentioned above, all client requests are automatically -routed, via the shortest network path, to a healthy pod in the -closest cluster to the origin of the request. So for example, HTTP(S) -requests from internet -users in Europe will be routed directly to the closest cluster in -Europe that has available capacity. If there are no such clusters in -Europe, the request will be routed to the next closest cluster -(typically in the U.S.). - -## Handling failures of backend pods and whole clusters - -Ingresses are backed by Services, which are typically (but not always) -backed by one or more ReplicaSets. For Federated Ingresses, it is -common practise to use the federated variants of Services and -ReplicaSets (see [Federated Services](federated-services.md) and -[Federated ReplicaSets](federated-replicasets.md)) for this purpose, as -described above. - -In particular, Federated ReplicaSets ensure that the desired number of -pods are kept running in each cluster, even in the event of node -failures. In the event of entire cluster or availability zone -failures, Federated ReplicaSets automatically place additional -replacas in the other available clusters in the federation to accommodate the -traffic which was previously being served by the now unavailable -cluster. While the Federated ReplicaSet ensures that sufficient replicas are -kept running, the Federated Ingress ensures that user traffic is -automatically redirected away from the failed cluster to other -available clusters. - -## Known issue - -GCE L7 load balancer back-ends and health checks are known to "flap"; this is due -to conflicting firewall rules in the federation's underlying clusters, which might override one another. To work around this problem, you can -install the firewall rules manually to expose the targets of all the -underlying clusters in your federation for each Federated Ingress -object. This way, the health checks can consistently pass and the GCE L7 load balancer -can remain stable. You install the rules using the -[`gcloud`](https://cloud.google.com/sdk/gcloud/) command line tool, -[Google Cloud Console](https://console.cloud.google.com) or the -[Google Compute Engine APIs](https://cloud.google.com/compute/docs/reference/latest/). - -You can install these rules using -[`gcloud`](https://cloud.google.com/sdk/gcloud/) as follows: - -```shell -gcloud compute firewall-rules create \ - --source-ranges 130.211.0.0/22 --allow [] \ - --target-tags [] \ - --network -``` - -where: - -1. `firewall-rule-name` can be any name. -2. `[]` is the comma separated list of node ports corresponding to the services that back the Federated Ingress. -3. [] is the comma separated list of the target tags assigned to the nodes in a Kubernetes cluster. -4. is the name of the network where the firewall rule must be installed. - -Example: -```shell -gcloud compute firewall-rules create my-federated-ingress-firewall-rule \ - --source-ranges 130.211.0.0/22 --allow tcp:30301, tcp:30061, tcp:34564 \ - --target-tags my-cluster-1-minion, my-cluster-2-minion \ - --network default -``` - - -## Troubleshooting - -#### I cannot connect to my cluster federation API -Check that your - -1. Client (typically kubectl) is correctly configured (including API endpoints and login credentials), and -2. Cluster Federation API server is running and network-reachable. - -See the [federation admin guide](/docs/admin/federation/) to learn -how to bring up a cluster federation correctly (or have your cluster administrator do this for you), and how to correctly configure your client. - -#### I can create a federated ingress/service/replicaset successfully against the cluster federation API, but no matching ingresses/services/replicasets are created in my underlying clusters - -Check that: - -1. Your clusters are correctly registered in the Cluster Federation API (`kubectl describe clusters`) -2. Your clusters are all 'Active'. This means that the cluster - Federation system was able to connect and authenticate against the - clusters' endpoints. If not, consult the event logs of the federation-controller-manager pod to ascertain what the failure might be. (`kubectl --namespace=federation logs $(kubectl get pods --namespace=federation -l module=federation-controller-manager -oname`) -3. That the login credentials provided to the Cluster Federation API - for the clusters have the correct authorization and quota to create - ingresses/services/replicasets in the relevant namespace in the - clusters. Again you should see associated error messages providing - more detail in the above event log file if this is not the case. -4. Whether any other error is preventing the service creation - operation from succeeding (look for `ingress-controller`, - `service-controller` or `replicaset-controller`, - errors in the output of `kubectl logs federation-controller-manager --namespace federation`). - -#### I can create a federated ingress successfully, but request load is not correctly distributed across the underlying clusters - -Check that: - -1. the services underlying your federated ingress in each cluster have - identical node ports. See [above](#creating_a_federated_ingress) for further explanation. -2. the load balancer controllers in each of your clusters are of the - correct type ("GLBC") and have been correctly reconfigured by the - federation control plane to share a global GCE load balancer (this - should happen automatically). If they of the correct type, and - have been correctly reconfigured, the UID data item in the GLBC - configmap in each cluster will be identical across all clusters. - See - [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) - for further details. - If this is not the case, check the logs of your federation - controller manager to determine why this automated reconfiguration - might be failing. -3. no ingresses have been manually created in any of your clusters before the above - reconfiguration of the load balancer controller completed - successfully. Ingresses created before the reconfiguration of - your GLBC will interfere with the behavior of your federated - ingresses created after the reconfiguration (see - [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) - for further information. To remedy this, - delete any ingresses created before the cluster joined the - federation (and had it's GLBC reconfigured), and recreate them if - necessary. - -#### This troubleshooting guide did not help me solve my problem - -Please use one of our [support channels](http://kubernetes.io/docs/troubleshooting/) to seek assistance. - -## For more information - - * [Federation proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/federation.md) details use cases that motivated this work. +[Federated Ingress](/docs/tasks/administer-federation/ingress/) diff --git a/docs/user-guide/federation/namespaces.md b/docs/user-guide/federation/namespaces.md index 5ed3ffafec..76b722c2b8 100644 --- a/docs/user-guide/federation/namespaces.md +++ b/docs/user-guide/federation/namespaces.md @@ -2,89 +2,6 @@ title: Federated Namespaces --- -This guide explains how to use namespaces in Federation control plane. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). Other tutorials, for example -[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) -by Kelsey Hightower, are also available to help you. - -You are also expected to have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and [Namespaces](/docs/user-guide/namespaces/) in particular. - -## Overview - -Namespaces in federation control plane (referred to as "federated namespaces" in -this guide) are very similar to the traditional [Kubernetes -Namespaces](/docs/user-guide/namespaces/) providing the same functionality. -Creating them in the federation control plane ensures that they are synchronized -across all the clusters in federation. - - -## Creating a Federated Namespace - -The API for Federated Namespaces is 100% compatible with the -API for traditional Kubernetes Namespaces. You can create a namespace by sending -a request to the federation apiserver. - -You can do that using kubectl by running: - -``` shell -kubectl --context=federation-cluster create -f myns.yaml -``` - -The '--context=federation-cluster' flag tells kubectl to submit the -request to the Federation apiserver instead of sending it to a Kubernetes -cluster. - -Once a federated namespace is created, the federation control plane will create -a matching namespace in all underlying Kubernetes clusters. -You can verify this by checking each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get namespaces myns -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. The name and -spec of the underlying namespace will match those of -the Federated Namespace that you created above. - - -## Updating a Federated Namespace - -You can update a federated namespace as you would update a Kubernetes -namespace, just send the request to federation apiserver instead of sending it -to a specific Kubernetes cluster. -Federation control plan will ensure that whenever the federated namespace is -updated, it updates the corresponding namespaces in all underlying clusters to -match it. - -## Deleting a Federated Namespace - -You can delete a federated namespace as you would delete a Kubernetes -namespace, just send the request to federation apiserver instead of sending it -to a specific Kubernetes cluster. - -For example, you can do that using kubectl by running: - -```shell -kubectl --context=federation-cluster delete ns myns -``` - -As in Kubernetes, deleting a federated namespace will delete all resources in that -namespace from the federation control plane. - -Note that at this point, deleting a federated namespace will not delete the -corresponding namespaces and resources in those namespaces from underlying clusters. -Users are expected to delete them manually. -We intend to fix this in the future. +[Federated Namespaces](/docs/tasks/administer-federation/namespaces/) diff --git a/docs/user-guide/federation/replicasets.md b/docs/user-guide/federation/replicasets.md index c173a38996..5c597d74ec 100644 --- a/docs/user-guide/federation/replicasets.md +++ b/docs/user-guide/federation/replicasets.md @@ -2,104 +2,6 @@ title: Federated ReplicaSets --- -This guide explains how to use replica sets in the Federation control plane. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). Other tutorials, for example -[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) -by Kelsey Hightower, are also available to help you. - -You are also expected to have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and [ReplicaSets](/docs/user-guide/replicasets/) in particular. - -## Overview - -Replica Sets in federation control plane (referred to as "federated replica sets" in -this guide) are very similar to the traditional [Kubernetes -ReplicaSets](/docs/user-guide/replicasets/), and provide the same functionality. -Creating them in the federation control plane ensures that the desired number of -replicas exist across the registered clusters. - - -## Creating a Federated Replica Set - -The API for Federated Replica Set is 100% compatible with the -API for traditional Kubernetes Replica Set. You can create a replica set by sending -a request to the federation apiserver. - -You can do that using [kubectl](/docs/user-guide/kubectl/) by running: - -``` shell -kubectl --context=federation-cluster create -f myrs.yaml -``` - -The '--context=federation-cluster' flag tells kubectl to submit the -request to the Federation apiserver instead of sending it to a Kubernetes -cluster. - -Once a federated replica set is created, the federation control plane will create -a replica set in all underlying Kubernetes clusters. -You can verify this by checking each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get rs myrs -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. - -These replica sets in underlying clusters will match the federation replica set -except in the number of replicas. Federation control plane will ensure that the -sum of replicas in each cluster match the desired number of replicas in the -federation replica set. - -### Spreading Replicas in Underlying Clusters - -By default, replicas are spread equally in all the underlying clusters. For ex: -if you have 3 registered clusters and you create a federated replica set with -`spec.replicas = 9`, then each replica set in the 3 clusters will have -`spec.replicas=3`. -To modify the number of replicas in each cluster, you can specify -[FederatedReplicaSetPreference](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go) -as an annotation with key `federation.kubernetes.io/replica-set-preferences` -on federated replica set. - - -## Updating a Federated Replica Set - -You can update a federated replica set as you would update a Kubernetes -replica set; however, for a federated replica set, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. -The Federation control plan ensures that whenever the federated replica set is -updated, it updates the corresponding replica sets in all underlying clusters to -match it. -If your update includes a change in number of replicas, the federation -control plane will change the number of replicas in underlying clusters to -ensure that their sum remains equal to the number of desired replicas in -federated replica set. - -## Deleting a Federated Replica Set - -You can delete a federated replica set as you would delete a Kubernetes -replica set; however, for a federated replica set, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. - -For example, you can do that using kubectl by running: - -```shell -kubectl --context=federation-cluster delete rs myrs -``` - -Note that at this point, deleting a federated replica set will not delete the -corresponding replica sets from underlying clusters. -You must delete the underlying Replica Sets manually. -We intend to fix this in the future. +[Federated ReplicaSets](/docs/tasks/administer-federation/replicaset/) diff --git a/docs/user-guide/federation/secrets.md b/docs/user-guide/federation/secrets.md index 2c5eac6dba..bbd503e506 100644 --- a/docs/user-guide/federation/secrets.md +++ b/docs/user-guide/federation/secrets.md @@ -2,86 +2,6 @@ title: Federated Secrets --- -This guide explains how to use secrets in Federation control plane. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## Prerequisites - -This guide assumes that you have a running Kubernetes Cluster -Federation installation. If not, then head over to the -[federation admin guide](/docs/admin/federation/) to learn how to -bring up a cluster federation (or have your cluster administrator do -this for you). Other tutorials, for example -[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation) -by Kelsey Hightower, are also available to help you. - -You are also expected to have a basic -[working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and [Secrets](/docs/user-guide/secrets/) in particular. - -## Overview - -Secrets in federation control plane (referred to as "federated secrets" in -this guide) are very similar to the traditional [Kubernetes -Secrets](/docs/user-guide/secrets/) providing the same functionality. -Creating them in the federation control plane ensures that they are synchronized -across all the clusters in federation. - - -## Creating a Federated Secret - -The API for Federated Secret is 100% compatible with the -API for traditional Kubernetes Secret. You can create a secret by sending -a request to the federation apiserver. - -You can do that using [kubectl](/docs/user-guide/kubectl/) by running: - -``` shell -kubectl --context=federation-cluster create -f mysecret.yaml -``` - -The '--context=federation-cluster' flag tells kubectl to submit the -request to the Federation apiserver instead of sending it to a Kubernetes -cluster. - -Once a federated secret is created, the federation control plane will create -a matching secret in all underlying Kubernetes clusters. -You can verify this by checking each of the underlying clusters, for example: - -``` shell -kubectl --context=gce-asia-east1a get secret mysecret -``` - -The above assumes that you have a context named 'gce-asia-east1a' -configured in your client for your cluster in that zone. - -These secrets in underlying clusters will match the federated secret. - - -## Updating a Federated Secret - -You can update a federated secret as you would update a Kubernetes -secret; however, for a federated secret, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. -The Federation control plan ensures that whenever the federated secret is -updated, it updates the corresponding secrets in all underlying clusters to -match it. - -## Deleting a Federated Secret - -You can delete a federated secret as you would delete a Kubernetes -secret; however, for a federated secret, you must send the request to -the federation apiserver instead of sending it to a specific Kubernetes cluster. - -For example, you can do that using kubectl by running: - -```shell -kubectl --context=federation-cluster delete secret mysecret -``` - -Note that at this point, deleting a federated secret will not delete the -corresponding secrets from underlying clusters. -You must delete the underlying secrets manually. -We intend to fix this in the future. +[Federated Secrets](/docs/tasks/administer-federation/secret/) From ee23982f7b7d52738fd80f81dca00c3a8e97592f Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Mon, 13 Mar 2017 23:43:45 -0700 Subject: [PATCH 368/407] Move Guide topics: Federation tutorial and concept. (#2802) * Move Guide topics: Federation tutorial and concept. * Add title. * Fix link. --- _data/concepts.yml | 3 +- _data/tutorials.yml | 3 + docs/admin/federation/kubefed.md | 203 +---------------- .../cluster-administration/federation.md | 137 ++++++++++++ .../set-up-cluster-federation-kubefed.md | 208 ++++++++++++++++++ docs/user-guide/federation/index.md | 134 +---------- 6 files changed, 354 insertions(+), 334 deletions(-) create mode 100644 docs/concepts/cluster-administration/federation.md create mode 100644 docs/tutorials/federation/set-up-cluster-federation-kubefed.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 32af951748..9750451b23 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -33,9 +33,10 @@ toc: section: - docs/concepts/workloads/pods/pod-lifecycle.md -- title: Clusters +- title: Cluster Administration section: - docs/concepts/clusters/logging.md + - docs/concepts/cluster-administration/federation.md - title: Configuration section: diff --git a/_data/tutorials.yml b/_data/tutorials.yml index 695d2b8805..33353cb9ff 100644 --- a/_data/tutorials.yml +++ b/_data/tutorials.yml @@ -49,3 +49,6 @@ toc: - title: Services section: - docs/tutorials/services/source-ip.md +- title: Federated Cluster Administration + section: + - docs/tutorials/federation/set-up-cluster-federation-kubefed.md diff --git a/docs/admin/federation/kubefed.md b/docs/admin/federation/kubefed.md index 4af8db60b5..7b1b8f499a 100644 --- a/docs/admin/federation/kubefed.md +++ b/docs/admin/federation/kubefed.md @@ -4,205 +4,6 @@ assignees: title: Setting up Cluster Federation with Kubefed --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -Kubernetes version 1.5 includes a new command line tool called -`kubefed` to help you administrate your federated clusters. -`kubefed` helps you to deploy a new Kubernetes cluster federation -control plane, and to add clusters to or remove clusters from an -existing federation control plane. - -This guide explains how to administer a Kubernetes Cluster Federation -using `kubefed`. - -> Note: `kubefed` is an alpha feature in Kubernetes 1.5. - -## Prerequisites - -This guide assumes that you have a running Kubernetes cluster. Please -see one of the [getting started](/docs/getting-started-guides/) guides -for installation instructions for your platform. - - -## Getting `kubefed` - -Download the client tarball corresponding to Kubernetes version 1.5 -or later -[from the release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md), -extract the binaries in the tarball to one of the directories -in your `$PATH` and set the executable permission on those binaries. - -Note: The URL in the curl command below downloads the binaries for -Linux amd64. If you are on a different platform, please use the URL -for the binaries appropriate for your platform. You can find the list -of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1). - - -```shell -curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.2/kubernetes-client-linux-amd64.tar.gz -tar -xzvf kubernetes-client-linux-amd64.tar.gz -sudo cp kubernetes/client/bin/kubefed /usr/local/bin -sudo chmod +x /usr/local/bin/kubefed -sudo cp kubernetes/client/bin/kubectl /usr/local/bin -sudo chmod +x /usr/local/bin/kubectl -``` - - -## Choosing a host cluster. - -You'll need to choose one of your Kubernetes clusters to be the -*host cluster*. The host cluster hosts the components that make up -your federation control plane. Ensure that you have a `kubeconfig` -entry in your local `kubeconfig` that corresponds to the host cluster. -You can verify that you have the required `kubeconfig` entry by -running: - -```shell -kubectl config get-contexts -``` - -The output should contain an entry corresponding to your host cluster, -similar to the following: - -``` -CURRENT NAME CLUSTER AUTHINFO NAMESPACE - gke_myproject_asia-east1-b_gce-asia-east1 gke_myproject_asia-east1-b_gce-asia-east1 gke_myproject_asia-east1-b_gce-asia-east1 -``` - - -You'll need to provide the `kubeconfig` context (called name in the -entry above) for your host cluster when you deploy your federation -control plane. - - -## Deploying a federation control plane. - -To deploy a federation control plane on your host cluster, run -`kubefed init` command. When you use `kubefed init`, you must provide -the following: - -* Federation name -* `--host-cluster-context`, the `kubeconfig` context for the host cluster -* `--dns-zone-name`, a domain name suffix for your federated services - -The following example command deploys a federation control plane with -the name `fellowship`, a host cluster context `rivendell`, and the -domain suffix `example.com`: - -```shell -kubefed init fellowship --host-cluster-context=rivendell --dns-zone-name="example.com" -``` - -The domain suffix specified in `--dns-zone-name` must be an existing -domain that you control, and that is programmable by your DNS provider. - -`kubefed init` sets up the federation control plane in the host -cluster and also adds an entry for the federation API server in your -local kubeconfig. Note that in the alpha release in Kubernetes 1.5, -`kubefed init` does not automatically set the current context to the -newly deployed federation. You can set the current context manually by -running: - -```shell -kubectl config use-context fellowship -``` - -where `fellowship` is the name of your federation. - - -## Adding a cluster to a federation - -Once you've deployed a federation control plane, you'll need to make -that control plane aware of the clusters it should manage. You can add -a cluster to your federation by using the `kubefed join` command. - -To use `kubefed join`, you'll need to provide the name of the cluster -you want to add to the federation, and the `--host-cluster-context` -for the federation control plane's host cluster. - -The following example command adds the cluster `gondor` to the -federation with host cluster `rivendell`: - -``` -kubefed join gondor --host-cluster-context=rivendell -``` - -> Note: Kubernetes requires that you manually join clusters to a -federation because the federation control plane manages only those -clusters that it is responsible for managing. Adding a cluster tells -the federation control plane that it is responsible for managing that -cluster. - -### Naming rules and customization - -The cluster name you supply to `kubefed join` must be a valid RFC 1035 -label. - -Furthermore, federation control plane requires credentials of the -joined clusters to operate on them. These credentials are obtained -from the local kubeconfig. `kubefed join` uses the cluster name -specified as the argument to look for the cluster's context in the -local kubeconfig. If it fails to find a matching context, it exits -with an error. - -This might cause issues in cases where context names for each cluster -in the federation don't follow -[RFC 1035](https://www.ietf.org/rfc/rfc1035.txt) label naming rules. -In such cases, you can specify a cluster name that conforms to the -[RFC 1035](https://www.ietf.org/rfc/rfc1035.txt) label naming rules -and specify the cluster context using the `--cluster-context` flag. -For example, if context of the cluster your are joining is -`gondor_needs-no_king`, then you can join the cluster by running: - -```shell -kubefed join gondor --host-cluster-context=rivendell --cluster-context=gondor_needs-no_king -``` - -#### Secret name - -Cluster credentials required by the federation control plane as -described above are stored as a secret in the host cluster. The name -of the secret is also derived from the cluster name. - -However, the name of a secret object in Kubernetes should conform -to the DNS subdomain name specification described in -[RFC 1123](https://tools.ietf.org/html/rfc1123). If this isn't the -case, you can pass the secret name to `kubefed join` using the -`--secret-name` flag. For example, if the cluster name is `noldor` and -the secret name is `11kingdom`, you can join the cluster by -running: - -```shell -kubefed join noldor --host-cluster-context=rivendell --secret-name=11kingdom -``` - -Note: If your cluster name does not conform to the DNS subdomain name -specification, all you need to do is supply the secret name via the -`--secret-name` flag. `kubefed join` automatically creates the secret -for you. - - -## Removing a cluster from a federation - -To remove a cluster from a federation, run the `kubefed unjoin` -command with the cluster name and the federation's -`--host-cluster-context`: - -``` -kubefed unjoin gondor --host-cluster-context=rivendell -``` - - -## Turning down the federation control plane: - -Proper cleanup of federation control plane is not fully implemented in -this alpha release of `kubefed`. However, for the time being, deleting -the federation system namespace should remove all the resources except -the persistent storage volume dynamically provisioned for the -federation control plane's etcd. You can delete the federation -namespace by running the following command: - -``` -$ kubectl delete ns federation-system -``` +[Setting up Cluster Federation with kubefed](/docs/tutorials/federation/set-up-cluster-federation-kubefed/) diff --git a/docs/concepts/cluster-administration/federation.md b/docs/concepts/cluster-administration/federation.md new file mode 100644 index 0000000000..8d151df3e9 --- /dev/null +++ b/docs/concepts/cluster-administration/federation.md @@ -0,0 +1,137 @@ +--- +title: Federation +--- + +This guide explains why and how to manage multiple Kubernetes clusters using +federation. + + +* TOC +{:toc} + + +## Why federation + +Federation makes it easy to manage multiple clusters. It does so by providing 2 +major building blocks: + + * Sync resources across clusters: Federation provides the ability to keep + resources in multiple clusters in sync. This can be used, for example, to + ensure that the same deployment exists in multiple clusters. + * Cross cluster discovery: It provides the ability to auto-configure DNS + servers and load balancers with backends from all clusters. This can be used, + for example, to ensure that a global VIP or DNS record can be used to access + backends from multiple clusters. + +Some other use cases that federation enables are: + +* High Availability: By spreading load across clusters and auto configuring DNS + servers and load balancers, federation minimises the impact of cluster + failure. +* Avoiding provider lock-in: By making it easier to migrate applications across + clusters, federation prevents cluster provider lock-in. + + +Federation is not helpful unless you have multiple clusters. Some of the reasons +why you might want multiple clusters are: + +* Low latency: Having clusters in multiple regions minimises latency by serving + users from the cluster that is closest to them. +* Fault isolation: It might be better to have multiple small clusters rather + than a single large cluster for fault isolation (for example: multiple + clusters in different availability zones of a cloud provider). + [Multi cluster guide](/docs/admin/multi-cluster) has more details on this. +* Scalability: There are scalability limits to a single kubernetes cluster (this + should not be the case for most users. For more details: + [Kubernetes Scaling and Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md)). +* Hybrid cloud: You can have multiple clusters on different cloud providers or + on-premises data centers. + + +### Caveats + +While there are a lot of attractive use cases for federation, there are also +some caveats. + +* Increased network bandwidth and cost: The federation control plane watches all + clusters to ensure that the current state is as expected. This can lead to + significant network cost if the clusters are running in different regions on + a cloud provider or on different cloud providers. +* Reduced cross cluster isolation: A bug in the federation control plane can + impact all clusters. This is mitigated by keeping the logic in federation + control plane to a minimum. It mostly delegates to the control plane in + kubernetes clusters whenever it can. The design and implementation also errs + on the side of safety and avoiding multicluster outage. +* Maturity: The federation project is relatively new and is not very mature. + Not all resources are available and many are still alpha. [Issue + 38893](https://github.com/kubernetes/kubernetes/issues/38893) ennumerates + known issues with the system that the team is busy solving. + +## Setup + +To be able to federate multiple clusters, we first need to setup a federation +control plane. +Follow the [setup guide](/docs/admin/federation/) to setup the +federation control plane. + +## Hybrid cloud capabilities + +Federations of Kubernetes Clusters can include clusters running in +different cloud providers (e.g. Google Cloud, AWS), and on-premises +(e.g. on OpenStack). Simply create all of the clusters that you +require, in the appropriate cloud providers and/or locations, and +register each cluster's API endpoint and credentials with your +Federation API Server (See the +[federation admin guide](/docs/admin/federation/) for details). + +Thereafter, your API resources can span different clusters +and cloud providers. + +## API resources + +Once we have the control plane setup, we can start creating federation API +resources. +The following guides explain some of the resources in detail: + +* [ConfigMap](https://kubernetes.io/docs/user-guide/federation/configmap/) +* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) +* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) +* [Events](https://kubernetes.io/docs/user-guide/federation/events/) +* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) +* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) +* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) +* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) +* [Services](https://kubernetes.io/docs/user-guide/federation/federated-services/) + +[API reference docs](/docs/federation/api-reference/) lists all the +resources supported by federation apiserver. + +## Cascading deletion + +Kubernetes version 1.5 includes support for cascading deletion of federated +resources. With cascading deletion, when you delete a resource from the +federation control plane, the corresponding resources in all underlying clusters +are also deleted. + +To enable cascading deletion, set the option +`DeleteOptions.orphanDependents=false` when you delete a resource from the +federation control plane. + +The following Federated resources are affected by cascading deletion: + +* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) +* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) +* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) +* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) +* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) +* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) + +Note: By default, deleting a resource from federation control plane does not +delete the corresponding resources from underlying clusters. + + +## For more information + +* [Federation + proposal](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/federation.md) +* [Kubecon2016 talk on federation](https://www.youtube.com/watch?v=pq9lbkmxpS8) diff --git a/docs/tutorials/federation/set-up-cluster-federation-kubefed.md b/docs/tutorials/federation/set-up-cluster-federation-kubefed.md new file mode 100644 index 0000000000..4af8db60b5 --- /dev/null +++ b/docs/tutorials/federation/set-up-cluster-federation-kubefed.md @@ -0,0 +1,208 @@ +--- +assignees: +- madhusudancs +title: Setting up Cluster Federation with Kubefed +--- + +* TOC +{:toc} + +Kubernetes version 1.5 includes a new command line tool called +`kubefed` to help you administrate your federated clusters. +`kubefed` helps you to deploy a new Kubernetes cluster federation +control plane, and to add clusters to or remove clusters from an +existing federation control plane. + +This guide explains how to administer a Kubernetes Cluster Federation +using `kubefed`. + +> Note: `kubefed` is an alpha feature in Kubernetes 1.5. + +## Prerequisites + +This guide assumes that you have a running Kubernetes cluster. Please +see one of the [getting started](/docs/getting-started-guides/) guides +for installation instructions for your platform. + + +## Getting `kubefed` + +Download the client tarball corresponding to Kubernetes version 1.5 +or later +[from the release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md), +extract the binaries in the tarball to one of the directories +in your `$PATH` and set the executable permission on those binaries. + +Note: The URL in the curl command below downloads the binaries for +Linux amd64. If you are on a different platform, please use the URL +for the binaries appropriate for your platform. You can find the list +of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1). + + +```shell +curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.2/kubernetes-client-linux-amd64.tar.gz +tar -xzvf kubernetes-client-linux-amd64.tar.gz +sudo cp kubernetes/client/bin/kubefed /usr/local/bin +sudo chmod +x /usr/local/bin/kubefed +sudo cp kubernetes/client/bin/kubectl /usr/local/bin +sudo chmod +x /usr/local/bin/kubectl +``` + + +## Choosing a host cluster. + +You'll need to choose one of your Kubernetes clusters to be the +*host cluster*. The host cluster hosts the components that make up +your federation control plane. Ensure that you have a `kubeconfig` +entry in your local `kubeconfig` that corresponds to the host cluster. +You can verify that you have the required `kubeconfig` entry by +running: + +```shell +kubectl config get-contexts +``` + +The output should contain an entry corresponding to your host cluster, +similar to the following: + +``` +CURRENT NAME CLUSTER AUTHINFO NAMESPACE + gke_myproject_asia-east1-b_gce-asia-east1 gke_myproject_asia-east1-b_gce-asia-east1 gke_myproject_asia-east1-b_gce-asia-east1 +``` + + +You'll need to provide the `kubeconfig` context (called name in the +entry above) for your host cluster when you deploy your federation +control plane. + + +## Deploying a federation control plane. + +To deploy a federation control plane on your host cluster, run +`kubefed init` command. When you use `kubefed init`, you must provide +the following: + +* Federation name +* `--host-cluster-context`, the `kubeconfig` context for the host cluster +* `--dns-zone-name`, a domain name suffix for your federated services + +The following example command deploys a federation control plane with +the name `fellowship`, a host cluster context `rivendell`, and the +domain suffix `example.com`: + +```shell +kubefed init fellowship --host-cluster-context=rivendell --dns-zone-name="example.com" +``` + +The domain suffix specified in `--dns-zone-name` must be an existing +domain that you control, and that is programmable by your DNS provider. + +`kubefed init` sets up the federation control plane in the host +cluster and also adds an entry for the federation API server in your +local kubeconfig. Note that in the alpha release in Kubernetes 1.5, +`kubefed init` does not automatically set the current context to the +newly deployed federation. You can set the current context manually by +running: + +```shell +kubectl config use-context fellowship +``` + +where `fellowship` is the name of your federation. + + +## Adding a cluster to a federation + +Once you've deployed a federation control plane, you'll need to make +that control plane aware of the clusters it should manage. You can add +a cluster to your federation by using the `kubefed join` command. + +To use `kubefed join`, you'll need to provide the name of the cluster +you want to add to the federation, and the `--host-cluster-context` +for the federation control plane's host cluster. + +The following example command adds the cluster `gondor` to the +federation with host cluster `rivendell`: + +``` +kubefed join gondor --host-cluster-context=rivendell +``` + +> Note: Kubernetes requires that you manually join clusters to a +federation because the federation control plane manages only those +clusters that it is responsible for managing. Adding a cluster tells +the federation control plane that it is responsible for managing that +cluster. + +### Naming rules and customization + +The cluster name you supply to `kubefed join` must be a valid RFC 1035 +label. + +Furthermore, federation control plane requires credentials of the +joined clusters to operate on them. These credentials are obtained +from the local kubeconfig. `kubefed join` uses the cluster name +specified as the argument to look for the cluster's context in the +local kubeconfig. If it fails to find a matching context, it exits +with an error. + +This might cause issues in cases where context names for each cluster +in the federation don't follow +[RFC 1035](https://www.ietf.org/rfc/rfc1035.txt) label naming rules. +In such cases, you can specify a cluster name that conforms to the +[RFC 1035](https://www.ietf.org/rfc/rfc1035.txt) label naming rules +and specify the cluster context using the `--cluster-context` flag. +For example, if context of the cluster your are joining is +`gondor_needs-no_king`, then you can join the cluster by running: + +```shell +kubefed join gondor --host-cluster-context=rivendell --cluster-context=gondor_needs-no_king +``` + +#### Secret name + +Cluster credentials required by the federation control plane as +described above are stored as a secret in the host cluster. The name +of the secret is also derived from the cluster name. + +However, the name of a secret object in Kubernetes should conform +to the DNS subdomain name specification described in +[RFC 1123](https://tools.ietf.org/html/rfc1123). If this isn't the +case, you can pass the secret name to `kubefed join` using the +`--secret-name` flag. For example, if the cluster name is `noldor` and +the secret name is `11kingdom`, you can join the cluster by +running: + +```shell +kubefed join noldor --host-cluster-context=rivendell --secret-name=11kingdom +``` + +Note: If your cluster name does not conform to the DNS subdomain name +specification, all you need to do is supply the secret name via the +`--secret-name` flag. `kubefed join` automatically creates the secret +for you. + + +## Removing a cluster from a federation + +To remove a cluster from a federation, run the `kubefed unjoin` +command with the cluster name and the federation's +`--host-cluster-context`: + +``` +kubefed unjoin gondor --host-cluster-context=rivendell +``` + + +## Turning down the federation control plane: + +Proper cleanup of federation control plane is not fully implemented in +this alpha release of `kubefed`. However, for the time being, deleting +the federation system namespace should remove all the resources except +the persistent storage volume dynamically provisioned for the +federation control plane's etcd. You can delete the federation +namespace by running the following command: + +``` +$ kubectl delete ns federation-system +``` diff --git a/docs/user-guide/federation/index.md b/docs/user-guide/federation/index.md index 10cea0ad6e..6e97d9429b 100644 --- a/docs/user-guide/federation/index.md +++ b/docs/user-guide/federation/index.md @@ -2,136 +2,6 @@ title: Federation User Guide --- -This guide explains why and how to manage multiple Kubernetes clusters using -federation. +{% include user-guide-content-moved.md %} - -* TOC -{:toc} - - -## Why federation - -Federation makes it easy to manage multiple clusters. It does so by providing 2 -major building blocks: - - * Sync resources across clusters: Federation provides the ability to keep - resources in multiple clusters in sync. This can be used, for example, to - ensure that the same deployment exists in multiple clusters. - * Cross cluster discovery: It provides the ability to auto-configure DNS - servers and load balancers with backends from all clusters. This can be used, - for example, to ensure that a global VIP or DNS record can be used to access - backends from multiple clusters. - -Some other use cases that federation enables are: - -* High Availability: By spreading load across clusters and auto configuring DNS - servers and load balancers, federation minimises the impact of cluster - failure. -* Avoiding provider lock-in: By making it easier to migrate applications across - clusters, federation prevents cluster provider lock-in. - - -Federation is not helpful unless you have multiple clusters. Some of the reasons -why you might want multiple clusters are: - -* Low latency: Having clusters in multiple regions minimises latency by serving - users from the cluster that is closest to them. -* Fault isolation: It might be better to have multiple small clusters rather - than a single large cluster for fault isolation (for example: multiple - clusters in different availability zones of a cloud provider). - [Multi cluster guide](/docs/admin/multi-cluster) has more details on this. -* Scalability: There are scalability limits to a single kubernetes cluster (this - should not be the case for most users. For more details: - [Kubernetes Scaling and Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md)). -* Hybrid cloud: You can have multiple clusters on different cloud providers or - on-premises data centers. - - -### Caveats - -While there are a lot of attractive use cases for federation, there are also -some caveats. - -* Increased network bandwidth and cost: The federation control plane watches all - clusters to ensure that the current state is as expected. This can lead to - significant network cost if the clusters are running in different regions on - a cloud provider or on different cloud providers. -* Reduced cross cluster isolation: A bug in the federation control plane can - impact all clusters. This is mitigated by keeping the logic in federation - control plane to a minimum. It mostly delegates to the control plane in - kubernetes clusters whenever it can. The design and implementation also errs - on the side of safety and avoiding multicluster outage. -* Maturity: The federation project is relatively new and is not very mature. - Not all resources are available and many are still alpha. [Issue - 38893](https://github.com/kubernetes/kubernetes/issues/38893) ennumerates - known issues with the system that the team is busy solving. - -## Setup - -To be able to federate multiple clusters, we first need to setup a federation -control plane. -Follow the [setup guide](/docs/admin/federation/) to setup the -federation control plane. - -## Hybrid cloud capabilities - -Federations of Kubernetes Clusters can include clusters running in -different cloud providers (e.g. Google Cloud, AWS), and on-premises -(e.g. on OpenStack). Simply create all of the clusters that you -require, in the appropriate cloud providers and/or locations, and -register each cluster's API endpoint and credentials with your -Federation API Server (See the -[federation admin guide](/docs/admin/federation/) for details). - -Thereafter, your API resources can span different clusters -and cloud providers. - -## API resources - -Once we have the control plane setup, we can start creating federation API -resources. -The following guides explain some of the resources in detail: - -* [ConfigMap](https://kubernetes.io/docs/user-guide/federation/configmap/) -* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) -* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) -* [Events](https://kubernetes.io/docs/user-guide/federation/events/) -* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) -* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) -* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) -* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) -* [Services](https://kubernetes.io/docs/user-guide/federation/federated-services/) - -[API reference docs](/docs/federation/api-reference/) lists all the -resources supported by federation apiserver. - -## Cascading deletion - -Kubernetes version 1.5 includes support for cascading deletion of federated -resources. With cascading deletion, when you delete a resource from the -federation control plane, the corresponding resources in all underlying clusters -are also deleted. - -To enable cascading deletion, set the option -`DeleteOptions.orphanDependents=false` when you delete a resource from the -federation control plane. - -The following Federated resources are affected by cascading deletion: - -* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/) -* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/) -* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/) -* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/) -* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/) -* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/) - -Note: By default, deleting a resource from federation control plane does not -delete the corresponding resources from underlying clusters. - - -## For more information - -* [Federation - proposal](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/federation.md) -* [Kubecon2016 talk on federation](https://www.youtube.com/watch?v=pq9lbkmxpS8) +[Federation](/docs/concepts/cluster-administration/federation.md) From bdaa82e79a4dbef1877bb13b7b8fe5039e90fc81 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 10:06:54 -0700 Subject: [PATCH 369/407] Move kubectl Concept topics to Tutorials. (#2804) * Move kubectl Concept topics to Tutorials. * Add redirects and update links. --- _data/concepts.yml | 7 ------- _data/tutorials.yml | 6 ++++++ .../declarative-object-management-configuration.md} | 7 +++++-- .../imperative-object-management-command.md} | 7 +++++-- .../imperative-object-management-configuration.md} | 7 +++++-- .../object-management-kubectl/object-management.md} | 9 ++++++--- .../object-management-kubectl}/simple_deployment.yaml | 0 .../object-management-kubectl}/update_deployment.yaml | 0 8 files changed, 27 insertions(+), 16 deletions(-) rename docs/{concepts/tools/kubectl/object-management-using-declarative-config.md => tutorials/object-management-kubectl/declarative-object-management-configuration.md} (99%) rename docs/{concepts/tools/kubectl/object-management-using-imperative-commands.md => tutorials/object-management-kubectl/imperative-object-management-command.md} (94%) rename docs/{concepts/tools/kubectl/object-management-using-imperative-config.md => tutorials/object-management-kubectl/imperative-object-management-configuration.md} (92%) rename docs/{concepts/tools/kubectl/object-management-overview.md => tutorials/object-management-kubectl/object-management.md} (93%) rename docs/{concepts/tools/kubectl => tutorials/object-management-kubectl}/simple_deployment.yaml (100%) rename docs/{concepts/tools/kubectl => tutorials/object-management-kubectl}/update_deployment.yaml (100%) diff --git a/_data/concepts.yml b/_data/concepts.yml index 9750451b23..56eee7200e 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -3,13 +3,6 @@ abstract: "Detailed explanations of Kubernetes system concepts and abstractions. toc: - docs/concepts/index.md -- title: Kubectl Command Line - section: - - docs/concepts/tools/kubectl/object-management-overview.md - - docs/concepts/tools/kubectl/object-management-using-imperative-commands.md - - docs/concepts/tools/kubectl/object-management-using-imperative-config.md - - docs/concepts/tools/kubectl/object-management-using-declarative-config.md - - title: Kubernetes Objects section: - docs/concepts/abstractions/overview.md diff --git a/_data/tutorials.yml b/_data/tutorials.yml index 33353cb9ff..c4e8d5f799 100644 --- a/_data/tutorials.yml +++ b/_data/tutorials.yml @@ -32,6 +32,12 @@ toc: - title: Online Training Course path: https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615 - docs/tutorials/stateless-application/hello-minikube.md +- title: Object Management Using kubectl + section: + - docs/tutorials/object-management-kubectl/object-management.md + - docs/tutorials/object-management-kubectl/imperative-object-management-command.md + - docs/tutorials/object-management-kubectl/imperative-object-management-configuration.md + - docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md - title: Stateless Applications section: - docs/tutorials/stateless-application/run-stateless-application-deployment.md diff --git a/docs/concepts/tools/kubectl/object-management-using-declarative-config.md b/docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md similarity index 99% rename from docs/concepts/tools/kubectl/object-management-using-declarative-config.md rename to docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md index 6c8dfbd250..4ce295c041 100644 --- a/docs/concepts/tools/kubectl/object-management-using-declarative-config.md +++ b/docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md @@ -1,5 +1,8 @@ --- title: Declarative Management of Kubernetes Objects Using Configuration Files +redirect_from: +- "/docs/concepts/tools/kubectl/object-management-using-declarative-config/" +- "/docs/concepts/tools/kubectl/object-management-using-declarative-config.html" --- {% capture overview %} @@ -949,8 +952,8 @@ The recommended approach for ThirdPartyResources is to use [imperative object co {% endcapture %} {% capture whatsnext %} -- [Managing Kubernetes Objects Using Imperative Commands](/docs/concepts/tools/kubectl/object-management-using-imperative-commands/) -- [Imperative Management of Kubernetes Objects Using Configuration Files](/docs/concepts/tools/kubectl/object-management-using-imperative-config/) +- [Managing Kubernetes Objects Using Imperative Commands](/docs/tutorials/object-management-kubectl/imperative-object-management-command/) +- [Imperative Management of Kubernetes Objects Using Configuration Files](docs/tutorials/object-management-kubectl/imperative-object-management-configuration/) - [Kubectl Command Reference](/docs/user-guide/kubectl/v1.5/) - [Kubernetes Object Schema Reference](/docs/resources-reference/v1.5/) {% endcapture %} diff --git a/docs/concepts/tools/kubectl/object-management-using-imperative-commands.md b/docs/tutorials/object-management-kubectl/imperative-object-management-command.md similarity index 94% rename from docs/concepts/tools/kubectl/object-management-using-imperative-commands.md rename to docs/tutorials/object-management-kubectl/imperative-object-management-command.md index 372e5c0584..e658b0dcda 100644 --- a/docs/concepts/tools/kubectl/object-management-using-imperative-commands.md +++ b/docs/tutorials/object-management-kubectl/imperative-object-management-command.md @@ -1,5 +1,8 @@ --- title: Managing Kubernetes Objects Using Imperative Commands +redirect_from: +- "/docs/concepts/tools/kubectl/object-management-using-imperative-commands/" +- "/docs/concepts/tools/kubectl/object-management-using-imperative-commands.html" --- {% capture overview %} @@ -150,8 +153,8 @@ kubectl create --edit -f /tmp/srv.yaml {% endcapture %} {% capture whatsnext %} -- [Managing Kubernetes Objects Using Object Configuration (Imperative)](/docs/concepts/tools/kubectl/object-management-using-imperative-config/) -- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/concepts/tools/kubectl/object-management-using-declarative-config/) +- [Managing Kubernetes Objects Using Object Configuration (Imperative)](/docs/tutorials/object-management-kubectl/imperative-object-management-configuration/) +- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/tutorials/object-management-kubectl/declarative-object-management-configuration/) - [Kubectl Command Reference](/docs/user-guide/kubectl/v1.5/) - [Kubernetes Object Schema Reference](/docs/resources-reference/v1.5/) {% endcapture %} diff --git a/docs/concepts/tools/kubectl/object-management-using-imperative-config.md b/docs/tutorials/object-management-kubectl/imperative-object-management-configuration.md similarity index 92% rename from docs/concepts/tools/kubectl/object-management-using-imperative-config.md rename to docs/tutorials/object-management-kubectl/imperative-object-management-configuration.md index 065e27c6dd..196d26fa8a 100644 --- a/docs/concepts/tools/kubectl/object-management-using-imperative-config.md +++ b/docs/tutorials/object-management-kubectl/imperative-object-management-configuration.md @@ -1,5 +1,8 @@ --- title: Imperative Management of Kubernetes Objects Using Configuration Files +redirect_from: +- "/docs/concepts/tools/kubectl/object-management-using-imperative-config/" +- "/docs/concepts/tools/kubectl/object-management-using-imperative-config.html" --- {% capture overview %} @@ -120,8 +123,8 @@ template: {% endcapture %} {% capture whatsnext %} -- [Managing Kubernetes Objects Using Imperative Commands](/docs/concepts/tools/kubectl/object-management-using-imperative-commands/) -- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/concepts/tools/kubectl/object-management-using-declarative-config/) +- [Managing Kubernetes Objects Using Imperative Commands](/docs/tutorials/object-management-kubectl/imperative-object-management-command/) +- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/tutorials/object-management-kubectl/declarative-object-management-configuration/) - [Kubectl Command Reference](/docs/user-guide/kubectl/v1.5/) - [Kubernetes Object Schema Reference](/docs/resources-reference/v1.5/) {% endcapture %} diff --git a/docs/concepts/tools/kubectl/object-management-overview.md b/docs/tutorials/object-management-kubectl/object-management.md similarity index 93% rename from docs/concepts/tools/kubectl/object-management-overview.md rename to docs/tutorials/object-management-kubectl/object-management.md index 2604a50468..96804ef109 100644 --- a/docs/concepts/tools/kubectl/object-management-overview.md +++ b/docs/tutorials/object-management-kubectl/object-management.md @@ -1,5 +1,8 @@ --- title: Kubernetes Object Management +redirect_from: +- "/docs/concepts/tools/kubectl/object-management-overview/" +- "/docs/concepts/tools/kubectl/object-management-overview.html" --- {% capture overview %} @@ -162,9 +165,9 @@ Disadvantages compared to imperative object configuration: {% endcapture %} {% capture whatsnext %} -- [Managing Kubernetes Objects Using Imperative Commands](/docs/concepts/tools/kubectl/object-management-using-imperative-commands/) -- [Managing Kubernetes Objects Using Object Configuration (Imperative)](/docs/concepts/tools/kubectl/object-management-using-imperative-config/) -- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/concepts/tools/kubectl/object-management-using-declarative-config/) +- [Managing Kubernetes Objects Using Imperative Commands](/docs/tutorials/object-management-kubectl/imperative-object-management-command/) +- [Managing Kubernetes Objects Using Object Configuration (Imperative)](/docs/tutorials/object-management-kubectl/imperative-object-management-configuration/) +- [Managing Kubernetes Objects Using Object Configuration (Declarative)](/docs/tutorials/object-management-kubectl/declarative-object-management-configuration/) - [Kubectl Command Reference](/docs/user-guide/kubectl/v1.5/) - [Kubernetes Object Schema Reference](/docs/resources-reference/v1.5/) diff --git a/docs/concepts/tools/kubectl/simple_deployment.yaml b/docs/tutorials/object-management-kubectl/simple_deployment.yaml similarity index 100% rename from docs/concepts/tools/kubectl/simple_deployment.yaml rename to docs/tutorials/object-management-kubectl/simple_deployment.yaml diff --git a/docs/concepts/tools/kubectl/update_deployment.yaml b/docs/tutorials/object-management-kubectl/update_deployment.yaml similarity index 100% rename from docs/concepts/tools/kubectl/update_deployment.yaml rename to docs/tutorials/object-management-kubectl/update_deployment.yaml From 99426f4c55f907a05ab6f5981530015261584bbf Mon Sep 17 00:00:00 2001 From: yupengzte Date: Wed, 8 Mar 2017 17:17:48 +0800 Subject: [PATCH 370/407] The calendarWrapper attribute should be unique Signed-off-by: yupengzte --- _sass/_base.sass | 10 +++++++++- community/index.html | 4 ++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/_sass/_base.sass b/_sass/_base.sass index 4b9e051af4..97aacc41be 100644 --- a/_sass/_base.sass +++ b/_sass/_base.sass @@ -1280,7 +1280,7 @@ $feature-box-div-margin-bottom: 40px background-color: $white box-shadow: 0 5px 5px rgba(0,0,0,.24),0 0 5px rgba(0,0,0,.12) - #calendarWrapper + #calendarMeetings position: relative width: 80vw height: 60vw @@ -1288,6 +1288,14 @@ $feature-box-div-margin-bottom: 40px max-height: 900px margin: 20px auto + #calendarEvents + position: relative + width: 80vw + height: 30vw + max-width: 1200px + max-height: 450px + margin: 20px auto + iframe position: absolute border: 0 diff --git a/community/index.html b/community/index.html index 5843b4829e..f967dc23cc 100644 --- a/community/index.html +++ b/community/index.html @@ -30,14 +30,14 @@ cid: community

    As a member of the Kubernetes community, you are welcome to join any of the SIG meetings you are interested in. No registration required.

    -
    +

    Events

    -
    +
    From 5e581ccf3e91ecb3afbde83f803dba233e3dcf9b Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 11:12:24 -0700 Subject: [PATCH 371/407] Fix links. (#2808) --- docs/tasks/administer-federation/configmap.md | 2 +- docs/tasks/administer-federation/daemonset.md | 4 ++-- docs/tasks/administer-federation/ingress.md | 13 ++++++------- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/tasks/administer-federation/configmap.md b/docs/tasks/administer-federation/configmap.md index 9d8aa18cac..b36cf7d8d6 100644 --- a/docs/tasks/administer-federation/configmap.md +++ b/docs/tasks/administer-federation/configmap.md @@ -20,7 +20,7 @@ might also help you create a Federated Kubernetes cluster. You should also have a basic [working knowledge of Kubernetes](/docs/getting-started-guides/) in -general and [ConfigMaps](/docs/user-guide/ConfigMaps/) in particular. +general and [ConfigMaps](/docs/user-guide/configmap/) in particular. ## Overview diff --git a/docs/tasks/administer-federation/daemonset.md b/docs/tasks/administer-federation/daemonset.md index e5a558c844..a6f1d92e42 100644 --- a/docs/tasks/administer-federation/daemonset.md +++ b/docs/tasks/administer-federation/daemonset.md @@ -25,8 +25,8 @@ general and DaemonSets in particular. ## Overview DaemonSets in federation control plane ("Federated Daemonsets" in -this guide) are very similar to the traditional [Kubernetes -DaemonSets](/docs/user-guide/DaemonSets/) and provide the same functionality. +this guide) are very similar to the traditional Kubernetes +DaemonSets and provide the same functionality. Creating them in the federation control plane ensures that they are synchronized across all the clusters in federation. diff --git a/docs/tasks/administer-federation/ingress.md b/docs/tasks/administer-federation/ingress.md index 0daa41ed49..410f49d541 100644 --- a/docs/tasks/administer-federation/ingress.md +++ b/docs/tasks/administer-federation/ingress.md @@ -11,7 +11,7 @@ originating from web clients around the globe on a single, static IP address. Low network latency, high fault tolerance and easy administration are ensured through intelligent request routing and automatic replica -relocation (using [Federated ReplicaSets](docs/user-guide/federation/federated-replicaset.md)). +relocation (using [Federated ReplicaSets](/docs/tasks/administer-federation/replicaset/). Clients are automatically routed, via the shortest network path, to the cluster closest to them with available capacity (despite the fact that all clients use exactly the same static IP address). The load balancer @@ -173,8 +173,8 @@ Ingresses in the clusters comprising your federation. To render the underlying ingress shards healthy, we need to add backend Pods behind the service upon which the Ingress is based. There are several ways to achieve this, but -the easiest is to create a [Federated Service](federated-services.md) and -[Federated Replicaset](federated-replicasets.md). Details of how those +the easiest is to create a Federated Service and +Federated Replicaset. Details of how those work are covered in the aforementioned user guides - here we'll simply use them, to create appropriately labelled pods and services in the 13 underlying clusters of our federation: @@ -236,8 +236,7 @@ Europe, the request will be routed to the next closest cluster Ingresses are backed by Services, which are typically (but not always) backed by one or more ReplicaSets. For Federated Ingresses, it is common practise to use the federated variants of Services and -ReplicaSets (see [Federated Services](federated-services.md) and -[Federated ReplicaSets](federated-replicasets.md)) for this purpose, as +ReplicaSets for this purpose, as described above. In particular, Federated ReplicaSets ensure that the desired number of @@ -331,7 +330,7 @@ Check that: have been correctly reconfigured, the UID data item in the GLBC configmap in each cluster will be identical across all clusters. See - [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) + [the GLBC docs](https://github.com/kubernetes/ingress/blob/7dcb4ae17d5def23d3e9c878f3146ac6df61b09d/controllers/gce/README.md) for further details. If this is not the case, check the logs of your federation controller manager to determine why this automated reconfiguration @@ -341,7 +340,7 @@ Check that: successfully. Ingresses created before the reconfiguration of your GLBC will interfere with the behavior of your federated ingresses created after the reconfiguration (see - [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) + [the GLBC docs](https://github.com/kubernetes/ingress/blob/7dcb4ae17d5def23d3e9c878f3146ac6df61b09d/controllers/gce/README.md) for further information. To remedy this, delete any ingresses created before the cluster joined the federation (and had it's GLBC reconfigured), and recreate them if From 44dcf9b7dc7662d06a280d60041f8a0e72700fbb Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 11:15:19 -0700 Subject: [PATCH 372/407] Fix link. (#2806) --- .../declarative-object-management-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md b/docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md index 4ce295c041..f0e1fea561 100644 --- a/docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md +++ b/docs/tutorials/object-management-kubectl/declarative-object-management-configuration.md @@ -953,7 +953,7 @@ The recommended approach for ThirdPartyResources is to use [imperative object co {% capture whatsnext %} - [Managing Kubernetes Objects Using Imperative Commands](/docs/tutorials/object-management-kubectl/imperative-object-management-command/) -- [Imperative Management of Kubernetes Objects Using Configuration Files](docs/tutorials/object-management-kubectl/imperative-object-management-configuration/) +- [Imperative Management of Kubernetes Objects Using Configuration Files](/docs/tutorials/object-management-kubectl/imperative-object-management-configuration/) - [Kubectl Command Reference](/docs/user-guide/kubectl/v1.5/) - [Kubernetes Object Schema Reference](/docs/resources-reference/v1.5/) {% endcapture %} From f1e7d0bda98483156ba72af19ed5484c6f9bc59f Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 12:59:13 -0700 Subject: [PATCH 373/407] Move topic from clusters to cluster-administration. (#2812) --- _data/concepts.yml | 2 +- .../counter-pod.yaml | 0 .../fluentd-sidecar-config.yaml | 0 .../{clusters => cluster-administration}/logging.md | 13 ++++++++----- .../two-files-counter-pod-agent-sidecar.yaml | 0 .../two-files-counter-pod-streaming-sidecar.yaml | 0 .../two-files-counter-pod.yaml | 0 7 files changed, 9 insertions(+), 6 deletions(-) rename docs/concepts/{clusters => cluster-administration}/counter-pod.yaml (100%) rename docs/concepts/{clusters => cluster-administration}/fluentd-sidecar-config.yaml (100%) rename docs/concepts/{clusters => cluster-administration}/logging.md (95%) rename docs/concepts/{clusters => cluster-administration}/two-files-counter-pod-agent-sidecar.yaml (100%) rename docs/concepts/{clusters => cluster-administration}/two-files-counter-pod-streaming-sidecar.yaml (100%) rename docs/concepts/{clusters => cluster-administration}/two-files-counter-pod.yaml (100%) diff --git a/_data/concepts.yml b/_data/concepts.yml index 56eee7200e..c15ad31a62 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -28,7 +28,7 @@ toc: - title: Cluster Administration section: - - docs/concepts/clusters/logging.md + - docs/concepts/cluster-administration/logging.md - docs/concepts/cluster-administration/federation.md - title: Configuration diff --git a/docs/concepts/clusters/counter-pod.yaml b/docs/concepts/cluster-administration/counter-pod.yaml similarity index 100% rename from docs/concepts/clusters/counter-pod.yaml rename to docs/concepts/cluster-administration/counter-pod.yaml diff --git a/docs/concepts/clusters/fluentd-sidecar-config.yaml b/docs/concepts/cluster-administration/fluentd-sidecar-config.yaml similarity index 100% rename from docs/concepts/clusters/fluentd-sidecar-config.yaml rename to docs/concepts/cluster-administration/fluentd-sidecar-config.yaml diff --git a/docs/concepts/clusters/logging.md b/docs/concepts/cluster-administration/logging.md similarity index 95% rename from docs/concepts/clusters/logging.md rename to docs/concepts/cluster-administration/logging.md index 465a2d324e..e9ceec0a71 100644 --- a/docs/concepts/clusters/logging.md +++ b/docs/concepts/cluster-administration/logging.md @@ -3,6 +3,9 @@ assignees: - crassirostris - piosz title: Logging and Monitoring Cluster Activity +redirect_from: +- "/docs/concepts/clusters/logging/" +- "/docs/concepts/clusters/logging.html" --- Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism; as such, most container engines are likewise designed to support some kind of logging. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams. @@ -21,7 +24,7 @@ The guidance for cluster-level logging assumes that a logging backend is present In this section, you can see an example of basic logging in Kubernetes that outputs data to the standard output stream. This demonstration uses -a [pod specification](/docs/concepts/clusters/counter-pod.yaml) with +a [pod specification](/docs/concepts/cluster-administration/counter-pod.yaml) with a container that writes some text to standard output once per second. {% include code.html language="yaml" file="counter-pod.yaml" ghlink="/docs/tasks/debug-application-cluster/counter-pod.yaml" %} @@ -131,7 +134,7 @@ Consider the following example. A pod runs a single container, and the container writes to two different log files, using two different formats. Here's a configuration file for the Pod: -{% include code.html language="yaml" file="two-files-counter-pod.yaml" ghlink="/docs/concepts/clusters/two-files-counter-pod.yaml" %} +{% include code.html language="yaml" file="two-files-counter-pod.yaml" ghlink="/docs/concepts/cluster-administration/two-files-counter-pod.yaml" %} It would be a mess to have log entries of different formats in the same log stream, even if you managed to redirect both components to the `stdout` stream of @@ -141,7 +144,7 @@ the logs to its own `stdout` stream. Here's a configuration file for a pod that has two sidecar containers: -{% include code.html language="yaml" file="two-files-counter-pod-streaming-sidecar.yaml" ghlink="/docs/concepts/clusters/two-files-counter-pod-streaming-sidecar.yaml" %} +{% include code.html language="yaml" file="two-files-counter-pod-streaming-sidecar.yaml" ghlink="/docs/concepts/cluster-administration/two-files-counter-pod-streaming-sidecar.yaml" %} Now when you run this pod, you can access each log stream separately by running the following commands: @@ -197,7 +200,7 @@ which uses fluentd as a logging agent. Here are two configuration files that you can use to implement this approach. The first file contains a [ConfigMap](/docs/user-guide/configmap/) to configure fluentd. -{% include code.html language="yaml" file="fluentd-sidecar-config.yaml" ghlink="/docs/concepts/clusters/fluentd-sidecar-config.yaml" %} +{% include code.html language="yaml" file="fluentd-sidecar-config.yaml" ghlink="/docs/concepts/cluster-administration/fluentd-sidecar-config.yaml" %} **Note**: The configuration of fluentd is beyond the scope of this article. For information about configuring fluentd, see the @@ -206,7 +209,7 @@ information about configuring fluentd, see the The second file describes a pod that has a sidecar container running fluentd. The pod mounts a volume where fluentd can pick up its configuration data. -{% include code.html language="yaml" file="two-files-counter-pod-agent-sidecar.yaml" ghlink="/docs/concepts/clusters/two-files-counter-pod-agent-sidecar.yaml" %} +{% include code.html language="yaml" file="two-files-counter-pod-agent-sidecar.yaml" ghlink="/docs/concepts/cluster-administration/two-files-counter-pod-agent-sidecar.yaml" %} After some time you can find log messages in the Stackdriver interface. diff --git a/docs/concepts/clusters/two-files-counter-pod-agent-sidecar.yaml b/docs/concepts/cluster-administration/two-files-counter-pod-agent-sidecar.yaml similarity index 100% rename from docs/concepts/clusters/two-files-counter-pod-agent-sidecar.yaml rename to docs/concepts/cluster-administration/two-files-counter-pod-agent-sidecar.yaml diff --git a/docs/concepts/clusters/two-files-counter-pod-streaming-sidecar.yaml b/docs/concepts/cluster-administration/two-files-counter-pod-streaming-sidecar.yaml similarity index 100% rename from docs/concepts/clusters/two-files-counter-pod-streaming-sidecar.yaml rename to docs/concepts/cluster-administration/two-files-counter-pod-streaming-sidecar.yaml diff --git a/docs/concepts/clusters/two-files-counter-pod.yaml b/docs/concepts/cluster-administration/two-files-counter-pod.yaml similarity index 100% rename from docs/concepts/clusters/two-files-counter-pod.yaml rename to docs/concepts/cluster-administration/two-files-counter-pod.yaml From 7be7d6b51960881e17055e03776ad329d1e646a4 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 14:09:54 -0700 Subject: [PATCH 374/407] Move a batch of cluster admin topics. (#2813) --- _data/concepts.yml | 7 + docs/admin/cluster-components.md | 131 +---------------- docs/admin/multi-cluster.md | 61 +------- docs/admin/rescheduler.md | 50 +------ docs/admin/sysctls.md | 117 +-------------- ...aranteed-scheduling-critical-addon-pods.md | 57 ++++++++ .../multiple-clusters.md | 66 +++++++++ .../cluster-administration/sysctl-cluster.md | 122 ++++++++++++++++ docs/concepts/overview/components.md | 136 ++++++++++++++++++ 9 files changed, 396 insertions(+), 351 deletions(-) create mode 100644 docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md create mode 100644 docs/concepts/cluster-administration/multiple-clusters.md create mode 100644 docs/concepts/cluster-administration/sysctl-cluster.md create mode 100644 docs/concepts/overview/components.md diff --git a/_data/concepts.yml b/_data/concepts.yml index c15ad31a62..93c06c68a6 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -3,6 +3,10 @@ abstract: "Detailed explanations of Kubernetes system concepts and abstractions. toc: - docs/concepts/index.md +- title: Overview + section: + - docs/concepts/overview/components.md + - title: Kubernetes Objects section: - docs/concepts/abstractions/overview.md @@ -29,7 +33,10 @@ toc: - title: Cluster Administration section: - docs/concepts/cluster-administration/logging.md + - docs/concepts/cluster-administration/multiple-clusters.md - docs/concepts/cluster-administration/federation.md + - docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md + - docs/concepts/cluster-administration/sysctl-cluster.md - title: Configuration section: diff --git a/docs/admin/cluster-components.md b/docs/admin/cluster-components.md index 280b9b1f2c..49154b1750 100644 --- a/docs/admin/cluster-components.md +++ b/docs/admin/cluster-components.md @@ -4,133 +4,6 @@ assignees: title: Kubernetes Components --- -This document outlines the various binary components that need to run to -deliver a functioning Kubernetes cluster. +{% include user-guide-content-moved.md %} -## Master Components - -Master components are those that provide the cluster's control plane. For -example, master components are responsible for making global decisions about the -cluster (e.g., scheduling), and detecting and responding to cluster events -(e.g., starting up a new pod when a replication controller's 'replicas' field is -unsatisfied). - -In theory, Master components can be run on any node in the cluster. However, -for simplicity, current set up scripts typically start all master components on -the same VM, and does not run user containers on this VM. See -[high-availability.md](/docs/admin/high-availability) for an example multi-master-VM setup. - -Even in the future, when Kubernetes is fully self-hosting, it will probably be -wise to only allow master components to schedule on a subset of nodes, to limit -co-running with user-run pods, reducing the possible scope of a -node-compromising security exploit. - -### kube-apiserver - -[kube-apiserver](/docs/admin/kube-apiserver) exposes the Kubernetes API; it is the front-end for the -Kubernetes control plane. It is designed to scale horizontally (i.e., one scales -it by running more of them-- [high-availability.md](/docs/admin/high-availability)). - -### etcd - -[etcd](/docs/admin/etcd) is used as Kubernetes' backing store. All cluster data is stored here. -Proper administration of a Kubernetes cluster includes a backup plan for etcd's -data. - -### kube-controller-manager - -[kube-controller-manager](/docs/admin/kube-controller-manager) is a binary that runs controllers, which are the -background threads that handle routine tasks in the cluster. Logically, each -controller is a separate process, but to reduce the number of moving pieces in -the system, they are all compiled into a single binary and run in a single -process. - -These controllers include: - -* Node Controller: Responsible for noticing & responding when nodes go down. -* Replication Controller: Responsible for maintaining the correct number of pods for every replication - controller object in the system. -* Endpoints Controller: Populates the Endpoints object (i.e., join Services & Pods). -* Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces. -* ... and others. - -### kube-scheduler - -[kube-scheduler](/docs/admin/kube-scheduler) watches newly created pods that have no node assigned, and -selects a node for them to run on. - -### addons - -Addons are pods and services that implement cluster features. The pods may be managed -by Deployments, ReplicationContollers, etc. Namespaced addon objects are created in -the "kube-system" namespace. - -Addon manager takes the responsibility for creating and maintaining addon resources. -See [here](http://releases.k8s.io/HEAD/cluster/addons) for more details. - -#### DNS - -While the other addons are not strictly required, all Kubernetes -clusters should have [cluster DNS](/docs/admin/dns/), as many examples rely on it. - -Cluster DNS is a DNS server, in addition to the other DNS server(s) in your -environment, which serves DNS records for Kubernetes services. - -Containers started by Kubernetes automatically include this DNS server -in their DNS searches. - -#### User interface - -The kube-ui provides a read-only overview of the cluster state. Access -[the UI using kubectl proxy](/docs/user-guide/connecting-to-applications-proxy/#connecting-to-the-kube-ui-service-from-your-local-workstation) - -#### Container Resource Monitoring - -[Container Resource Monitoring](/docs/user-guide/monitoring) records generic time-series metrics -about containers in a central database, and provides a UI for browsing that data. - -#### Cluster-level Logging - -A [Cluster-level logging](/docs/user-guide/logging/overview) mechanism is responsible for -saving container logs to a central log store with search/browsing interface. - -## Node components - -Node components run on every node, maintaining running pods and providing them -the Kubernetes runtime environment. - -### kubelet - -[kubelet](/docs/admin/kubelet) is the primary node agent. It: - -* Watches for pods that have been assigned to its node (either by apiserver - or via local configuration file) and: -* Mounts the pod's required volumes -* Downloads the pod's secrets -* Runs the pod's containers via docker (or, experimentally, rkt). -* Periodically executes any requested container liveness probes. -* Reports the status of the pod back to the rest of the system, by creating a - "mirror pod" if necessary. -* Reports the status of the node back to the rest of the system. - -### kube-proxy - -[kube-proxy](/docs/admin/kube-proxy) enables the Kubernetes service abstraction by maintaining -network rules on the host and performing connection forwarding. - -### docker - -`docker` is of course used for actually running containers. - -### rkt - -`rkt` is supported experimentally as an alternative to docker. - -### supervisord - -`supervisord` is a lightweight process babysitting system for keeping kubelet and docker -running. - -### fluentd - -`fluentd` is a daemon which helps provide [cluster-level logging](#cluster-level-logging). +[Kubernetes Components](/docs/concepts/overview/components/) diff --git a/docs/admin/multi-cluster.md b/docs/admin/multi-cluster.md index 085a9afa9f..4fe4e8b6ae 100644 --- a/docs/admin/multi-cluster.md +++ b/docs/admin/multi-cluster.md @@ -4,63 +4,6 @@ assignees: title: Using Multiple Clusters --- -You may want to set up multiple Kubernetes clusters, both to -have clusters in different regions to be nearer to your users, and to tolerate failures and/or invasive maintenance. -This document describes some of the issues to consider when making a decision about doing so. +{% include user-guide-content-moved.md %} -If you decide to have multiple clusters, Kubernetes provides a way to [federate them](/docs/admin/federation/). - -## Scope of a single cluster - -On IaaS providers such as Google Compute Engine or Amazon Web Services, a VM exists in a -[zone](https://cloud.google.com/compute/docs/zones) or [availability -zone](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). -We suggest that all the VMs in a Kubernetes cluster should be in the same availability zone, because: - - - compared to having a single global Kubernetes cluster, there are fewer single-points of failure - - compared to a cluster that spans availability zones, it is easier to reason about the availability properties of a - single-zone cluster. - - when the Kubernetes developers are designing the system (e.g. making assumptions about latency, bandwidth, or - correlated failures) they are assuming all the machines are in a single data center, or otherwise closely connected. - -It is okay to have multiple clusters per availability zone, though on balance we think fewer is better. -Reasons to prefer fewer clusters are: - - - improved bin packing of Pods in some cases with more nodes in one cluster (less resource fragmentation) - - reduced operational overhead (though the advantage is diminished as ops tooling and processes matures) - - reduced costs for per-cluster fixed resource costs, e.g. apiserver VMs (but small as a percentage - of overall cluster cost for medium to large clusters). - -Reasons to have multiple clusters include: - - - strict security policies requiring isolation of one class of work from another (but, see Partitioning Clusters - below). - - test clusters to canary new Kubernetes releases or other cluster software. - -## Selecting the right number of clusters - -The selection of the number of Kubernetes clusters may be a relatively static choice, only revisited occasionally. -By contrast, the number of nodes in a cluster and the number of pods in a service may change frequently according to -load and growth. - -To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run -on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not -be considered). Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions. -Call the number of regions to be in `R`. - -Second, decide how many clusters should be able to be unavailable at the same time, while still being available. Call -the number that can be unavailable `U`. If you are not sure, then 1 is a fine choice. - -If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then -you need at least the larger of `R` or `U + 1` clusters. If it is not (e.g. you want to ensure low latency for all -users in the event of a cluster failure), then you need to have `R * (U + 1)` clusters -(`U + 1` in each of `R` regions). In any case, try to put each cluster in a different zone. - -Finally, if any of your clusters would need more than the maximum recommended number of nodes for a Kubernetes cluster, then -you may need even more clusters. Kubernetes v1.3 supports clusters up to 1000 nodes in size. - -## Working with multiple clusters - -When you have multiple clusters, you would typically create services with the same config in each cluster and put each of those -service instances behind a load balancer (AWS Elastic Load Balancer, GCE Forwarding Rule or HTTP Load Balancer) spanning all of them, so that -failures of a single cluster are not visible to end users. +[Using Multiple Clusters](/docs/concepts/cluster-administration/multiple-clusters/) diff --git a/docs/admin/rescheduler.md b/docs/admin/rescheduler.md index 9e3fc61c39..d8d418ac2b 100644 --- a/docs/admin/rescheduler.md +++ b/docs/admin/rescheduler.md @@ -6,52 +6,6 @@ assignees: title: Guaranteed Scheduling For Critical Add-On Pods --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Overview - -In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a master machine -there are a number of add-ons which, for various reasons, must run on a regular cluster node (rather than the Kubernetes master). -Some of these add-ons are critical to a fully functional cluster, such as Heapster, DNS, and UI. -A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade) -and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space -vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason). - -## Rescheduler: guaranteed scheduling of critical add-ons - -Rescheduler ensures that critical add-ons are always scheduled -(assuming the cluster has enough resources to run the critical add-on pods in the absence of regular pods). -If the scheduler determines that no node has enough free resources to run the critical add-on pod -given the pods that are already running in the cluster -(indicated by critical add-on pod's pod condition PodScheduled set to false, the reason set to Unschedulable) -the rescheduler tries to free up space for the add-on by evicting some pods; then the scheduler will schedule the add-on pod. - -To avoid situation when another pod is scheduled into the space prepared for the critical add-on, -the chosen node gets a temporary taint "CriticalAddonsOnly" before the eviction(s) -(see [more details](https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md)). -Each critical add-on has to tolerate it, -while the other pods shouldn't tolerate the taint. The taint is removed once the add-on is successfully scheduled. - -*Warning:* currently there is no guarantee which node is chosen and which pods are being killed -in order to schedule critical pods, so if rescheduler is enabled your pods might be occasionally -killed for this purpose. - -## Config - -Rescheduler doesn't have any user facing configuration (component config) or API. -It's enabled by default. It can be disabled: - -* during cluster setup by setting `ENABLE_RESCHEDULER` flag to `false` -* on running cluster by deleting its manifest from master node -(default path `/etc/kubernetes/manifests/rescheduler.manifest`) - -### Marking add-on as critical - -To be critical an add-on has to run in `kube-system` namespace (configurable via flag) -and have the following annotations specified: - -* `scheduler.alpha.kubernetes.io/critical-pod` set to empty string -* `scheduler.alpha.kubernetes.io/tolerations` set to `[{"key":"CriticalAddonsOnly", "operator":"Exists"}]` - -The first one marks a pod a critical. The second one is required by Rescheduler algorithm. +[Guaranteed Scheduling for Critical Add-On Pods](/docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods/) diff --git a/docs/admin/sysctls.md b/docs/admin/sysctls.md index aa75c4df2a..4931a8d6bf 100644 --- a/docs/admin/sysctls.md +++ b/docs/admin/sysctls.md @@ -4,119 +4,6 @@ assignees: title: Using Sysctls in a Kubernetes Cluster --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -This document describes how sysctls are used within a Kubernetes cluster. - -## What is a Sysctl? - -In Linux, the sysctl interface allows an administrator to modify kernel -parameters at runtime. Parameters are available via the `/proc/sys/` virtual -process file system. The parameters cover various subsystems such as: - -- kernel (common prefix: `kernel.`) -- networking (common prefix: `net.`) -- virtual memory (common prefix: `vm.`) -- MDADM (common prefix: `dev.`) -- More subsystems are described in [Kernel docs](https://www.kernel.org/doc/Documentation/sysctl/README). - -To get a list of all parameters, you can run - -``` -$ sudo sysctl -a -``` - -## Namespaced vs. Node-Level Sysctls - -A number of sysctls are _namespaced_ in today's Linux kernels. This means that -they can be set independently for each pod on a node. Being namespaced is a -requirement for sysctls to be accessible in a pod context within Kubernetes. - -The following sysctls are known to be _namespaced_: - -- `kernel.shm*`, -- `kernel.msg*`, -- `kernel.sem`, -- `fs.mqueue.*`, -- `net.*`. - -Sysctls which are not namespaced are called _node-level_ and must be set -manually by the cluster admin, either by means of the underlying Linux -distribution of the nodes (e.g. via `/etc/sysctls.conf`) or using a DaemonSet -with privileged containers. - -**Note**: it is good practice to consider nodes with special sysctl settings as -_tainted_ within a cluster, and only schedule pods onto them which need those -sysctl settings. It is suggested to use the Kubernetes [_taints and toleration_ -feature](/docs/user-guide/kubectl/kubectl_taint.md) to implement this. - -## Safe vs. Unsafe Sysctls - -Sysctls are grouped into _safe_ and _unsafe_ sysctls. In addition to proper -namespacing a _safe_ sysctl must be properly _isolated_ between pods on the same -node. This means that setting a _safe_ sysctl for one pod - -- must not have any influence on any other pod on the node -- must not allow to harm the node's health -- must not allow to gain CPU or memory resources outside of the resource limits - of a pod. - -By far, most of the _namespaced_ sysctls are not necessarily considered _safe_. - -For Kubernetes 1.4, the following sysctls are supported in the _safe_ set: - -- `kernel.shm_rmid_forced`, -- `net.ipv4.ip_local_port_range`, -- `net.ipv4.tcp_syncookies`. - -This list will be extended in future Kubernetes versions when the kubelet -supports better isolation mechanisms. - -All _safe_ sysctls are enabled by default. - -All _unsafe_ sysctls are disabled by default and must be allowed manually by the -cluster admin on a per-node basis. Pods with disabled unsafe sysctls will be -scheduled, but will fail to launch. - -**Warning**: Due to their nature of being _unsafe_, the use of _unsafe_ sysctls -is at-your-own-risk and can lead to severe problems like wrong behavior of -containers, resource shortage or complete breakage of a node. - -## Enabling Unsafe Sysctls - -With the warning above in mind, the cluster admin can allow certain _unsafe_ -sysctls for very special situations like e.g. high-performance or real-time -application tuning. _Unsafe_ sysctls are enabled on a node-by-node basis with a -flag of the kubelet, e.g.: - -```shell -$ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.ipv4.route.min_pmtu' ... -``` - -Only _namespaced_ sysctls can be enabled this way. - -## Setting Sysctls for a Pod - -The sysctl feature is an alpha API in Kubernetes 1.4. Therefore, sysctls are set -using annotations on pods. They apply to all containers in the same pod. - -Here is an example, with different annotations for _safe_ and _unsafe_ sysctls: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: sysctl-example - annotations: - security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1 - security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.route.min_pmtu=1000,kernel.msgmax=1 2 3 -spec: - ... -``` - -**Note**: a pod with the _unsafe_ sysctls specified above will fail to launch on -any node which has not enabled those two _unsafe_ sysctls explicitly. As with -_node-level_ sysctls it is recommended to use [_taints and toleration_ -feature](/docs/user-guide/kubectl/kubectl_taint.md) or [labels on nodes](/docs -/user-guide/labels.md) to schedule those pods onto the right nodes. +[Using Sysctls in a Kubernetes Cluster](/docs/concepts/cluster-administration/sysctl-cluster/) diff --git a/docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md b/docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md new file mode 100644 index 0000000000..9e3fc61c39 --- /dev/null +++ b/docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md @@ -0,0 +1,57 @@ +--- +assignees: +- davidopp +- filipg +- piosz +title: Guaranteed Scheduling For Critical Add-On Pods +--- + +* TOC +{:toc} + +## Overview + +In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a master machine +there are a number of add-ons which, for various reasons, must run on a regular cluster node (rather than the Kubernetes master). +Some of these add-ons are critical to a fully functional cluster, such as Heapster, DNS, and UI. +A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade) +and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space +vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason). + +## Rescheduler: guaranteed scheduling of critical add-ons + +Rescheduler ensures that critical add-ons are always scheduled +(assuming the cluster has enough resources to run the critical add-on pods in the absence of regular pods). +If the scheduler determines that no node has enough free resources to run the critical add-on pod +given the pods that are already running in the cluster +(indicated by critical add-on pod's pod condition PodScheduled set to false, the reason set to Unschedulable) +the rescheduler tries to free up space for the add-on by evicting some pods; then the scheduler will schedule the add-on pod. + +To avoid situation when another pod is scheduled into the space prepared for the critical add-on, +the chosen node gets a temporary taint "CriticalAddonsOnly" before the eviction(s) +(see [more details](https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md)). +Each critical add-on has to tolerate it, +while the other pods shouldn't tolerate the taint. The taint is removed once the add-on is successfully scheduled. + +*Warning:* currently there is no guarantee which node is chosen and which pods are being killed +in order to schedule critical pods, so if rescheduler is enabled your pods might be occasionally +killed for this purpose. + +## Config + +Rescheduler doesn't have any user facing configuration (component config) or API. +It's enabled by default. It can be disabled: + +* during cluster setup by setting `ENABLE_RESCHEDULER` flag to `false` +* on running cluster by deleting its manifest from master node +(default path `/etc/kubernetes/manifests/rescheduler.manifest`) + +### Marking add-on as critical + +To be critical an add-on has to run in `kube-system` namespace (configurable via flag) +and have the following annotations specified: + +* `scheduler.alpha.kubernetes.io/critical-pod` set to empty string +* `scheduler.alpha.kubernetes.io/tolerations` set to `[{"key":"CriticalAddonsOnly", "operator":"Exists"}]` + +The first one marks a pod a critical. The second one is required by Rescheduler algorithm. diff --git a/docs/concepts/cluster-administration/multiple-clusters.md b/docs/concepts/cluster-administration/multiple-clusters.md new file mode 100644 index 0000000000..085a9afa9f --- /dev/null +++ b/docs/concepts/cluster-administration/multiple-clusters.md @@ -0,0 +1,66 @@ +--- +assignees: +- davidopp +title: Using Multiple Clusters +--- + +You may want to set up multiple Kubernetes clusters, both to +have clusters in different regions to be nearer to your users, and to tolerate failures and/or invasive maintenance. +This document describes some of the issues to consider when making a decision about doing so. + +If you decide to have multiple clusters, Kubernetes provides a way to [federate them](/docs/admin/federation/). + +## Scope of a single cluster + +On IaaS providers such as Google Compute Engine or Amazon Web Services, a VM exists in a +[zone](https://cloud.google.com/compute/docs/zones) or [availability +zone](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). +We suggest that all the VMs in a Kubernetes cluster should be in the same availability zone, because: + + - compared to having a single global Kubernetes cluster, there are fewer single-points of failure + - compared to a cluster that spans availability zones, it is easier to reason about the availability properties of a + single-zone cluster. + - when the Kubernetes developers are designing the system (e.g. making assumptions about latency, bandwidth, or + correlated failures) they are assuming all the machines are in a single data center, or otherwise closely connected. + +It is okay to have multiple clusters per availability zone, though on balance we think fewer is better. +Reasons to prefer fewer clusters are: + + - improved bin packing of Pods in some cases with more nodes in one cluster (less resource fragmentation) + - reduced operational overhead (though the advantage is diminished as ops tooling and processes matures) + - reduced costs for per-cluster fixed resource costs, e.g. apiserver VMs (but small as a percentage + of overall cluster cost for medium to large clusters). + +Reasons to have multiple clusters include: + + - strict security policies requiring isolation of one class of work from another (but, see Partitioning Clusters + below). + - test clusters to canary new Kubernetes releases or other cluster software. + +## Selecting the right number of clusters + +The selection of the number of Kubernetes clusters may be a relatively static choice, only revisited occasionally. +By contrast, the number of nodes in a cluster and the number of pods in a service may change frequently according to +load and growth. + +To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run +on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not +be considered). Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions. +Call the number of regions to be in `R`. + +Second, decide how many clusters should be able to be unavailable at the same time, while still being available. Call +the number that can be unavailable `U`. If you are not sure, then 1 is a fine choice. + +If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then +you need at least the larger of `R` or `U + 1` clusters. If it is not (e.g. you want to ensure low latency for all +users in the event of a cluster failure), then you need to have `R * (U + 1)` clusters +(`U + 1` in each of `R` regions). In any case, try to put each cluster in a different zone. + +Finally, if any of your clusters would need more than the maximum recommended number of nodes for a Kubernetes cluster, then +you may need even more clusters. Kubernetes v1.3 supports clusters up to 1000 nodes in size. + +## Working with multiple clusters + +When you have multiple clusters, you would typically create services with the same config in each cluster and put each of those +service instances behind a load balancer (AWS Elastic Load Balancer, GCE Forwarding Rule or HTTP Load Balancer) spanning all of them, so that +failures of a single cluster are not visible to end users. diff --git a/docs/concepts/cluster-administration/sysctl-cluster.md b/docs/concepts/cluster-administration/sysctl-cluster.md new file mode 100644 index 0000000000..aa75c4df2a --- /dev/null +++ b/docs/concepts/cluster-administration/sysctl-cluster.md @@ -0,0 +1,122 @@ +--- +assignees: +- sttts +title: Using Sysctls in a Kubernetes Cluster +--- + +* TOC +{:toc} + +This document describes how sysctls are used within a Kubernetes cluster. + +## What is a Sysctl? + +In Linux, the sysctl interface allows an administrator to modify kernel +parameters at runtime. Parameters are available via the `/proc/sys/` virtual +process file system. The parameters cover various subsystems such as: + +- kernel (common prefix: `kernel.`) +- networking (common prefix: `net.`) +- virtual memory (common prefix: `vm.`) +- MDADM (common prefix: `dev.`) +- More subsystems are described in [Kernel docs](https://www.kernel.org/doc/Documentation/sysctl/README). + +To get a list of all parameters, you can run + +``` +$ sudo sysctl -a +``` + +## Namespaced vs. Node-Level Sysctls + +A number of sysctls are _namespaced_ in today's Linux kernels. This means that +they can be set independently for each pod on a node. Being namespaced is a +requirement for sysctls to be accessible in a pod context within Kubernetes. + +The following sysctls are known to be _namespaced_: + +- `kernel.shm*`, +- `kernel.msg*`, +- `kernel.sem`, +- `fs.mqueue.*`, +- `net.*`. + +Sysctls which are not namespaced are called _node-level_ and must be set +manually by the cluster admin, either by means of the underlying Linux +distribution of the nodes (e.g. via `/etc/sysctls.conf`) or using a DaemonSet +with privileged containers. + +**Note**: it is good practice to consider nodes with special sysctl settings as +_tainted_ within a cluster, and only schedule pods onto them which need those +sysctl settings. It is suggested to use the Kubernetes [_taints and toleration_ +feature](/docs/user-guide/kubectl/kubectl_taint.md) to implement this. + +## Safe vs. Unsafe Sysctls + +Sysctls are grouped into _safe_ and _unsafe_ sysctls. In addition to proper +namespacing a _safe_ sysctl must be properly _isolated_ between pods on the same +node. This means that setting a _safe_ sysctl for one pod + +- must not have any influence on any other pod on the node +- must not allow to harm the node's health +- must not allow to gain CPU or memory resources outside of the resource limits + of a pod. + +By far, most of the _namespaced_ sysctls are not necessarily considered _safe_. + +For Kubernetes 1.4, the following sysctls are supported in the _safe_ set: + +- `kernel.shm_rmid_forced`, +- `net.ipv4.ip_local_port_range`, +- `net.ipv4.tcp_syncookies`. + +This list will be extended in future Kubernetes versions when the kubelet +supports better isolation mechanisms. + +All _safe_ sysctls are enabled by default. + +All _unsafe_ sysctls are disabled by default and must be allowed manually by the +cluster admin on a per-node basis. Pods with disabled unsafe sysctls will be +scheduled, but will fail to launch. + +**Warning**: Due to their nature of being _unsafe_, the use of _unsafe_ sysctls +is at-your-own-risk and can lead to severe problems like wrong behavior of +containers, resource shortage or complete breakage of a node. + +## Enabling Unsafe Sysctls + +With the warning above in mind, the cluster admin can allow certain _unsafe_ +sysctls for very special situations like e.g. high-performance or real-time +application tuning. _Unsafe_ sysctls are enabled on a node-by-node basis with a +flag of the kubelet, e.g.: + +```shell +$ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.ipv4.route.min_pmtu' ... +``` + +Only _namespaced_ sysctls can be enabled this way. + +## Setting Sysctls for a Pod + +The sysctl feature is an alpha API in Kubernetes 1.4. Therefore, sysctls are set +using annotations on pods. They apply to all containers in the same pod. + +Here is an example, with different annotations for _safe_ and _unsafe_ sysctls: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sysctl-example + annotations: + security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1 + security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.route.min_pmtu=1000,kernel.msgmax=1 2 3 +spec: + ... +``` + +**Note**: a pod with the _unsafe_ sysctls specified above will fail to launch on +any node which has not enabled those two _unsafe_ sysctls explicitly. As with +_node-level_ sysctls it is recommended to use [_taints and toleration_ +feature](/docs/user-guide/kubectl/kubectl_taint.md) or [labels on nodes](/docs +/user-guide/labels.md) to schedule those pods onto the right nodes. diff --git a/docs/concepts/overview/components.md b/docs/concepts/overview/components.md new file mode 100644 index 0000000000..280b9b1f2c --- /dev/null +++ b/docs/concepts/overview/components.md @@ -0,0 +1,136 @@ +--- +assignees: +- lavalamp +title: Kubernetes Components +--- + +This document outlines the various binary components that need to run to +deliver a functioning Kubernetes cluster. + +## Master Components + +Master components are those that provide the cluster's control plane. For +example, master components are responsible for making global decisions about the +cluster (e.g., scheduling), and detecting and responding to cluster events +(e.g., starting up a new pod when a replication controller's 'replicas' field is +unsatisfied). + +In theory, Master components can be run on any node in the cluster. However, +for simplicity, current set up scripts typically start all master components on +the same VM, and does not run user containers on this VM. See +[high-availability.md](/docs/admin/high-availability) for an example multi-master-VM setup. + +Even in the future, when Kubernetes is fully self-hosting, it will probably be +wise to only allow master components to schedule on a subset of nodes, to limit +co-running with user-run pods, reducing the possible scope of a +node-compromising security exploit. + +### kube-apiserver + +[kube-apiserver](/docs/admin/kube-apiserver) exposes the Kubernetes API; it is the front-end for the +Kubernetes control plane. It is designed to scale horizontally (i.e., one scales +it by running more of them-- [high-availability.md](/docs/admin/high-availability)). + +### etcd + +[etcd](/docs/admin/etcd) is used as Kubernetes' backing store. All cluster data is stored here. +Proper administration of a Kubernetes cluster includes a backup plan for etcd's +data. + +### kube-controller-manager + +[kube-controller-manager](/docs/admin/kube-controller-manager) is a binary that runs controllers, which are the +background threads that handle routine tasks in the cluster. Logically, each +controller is a separate process, but to reduce the number of moving pieces in +the system, they are all compiled into a single binary and run in a single +process. + +These controllers include: + +* Node Controller: Responsible for noticing & responding when nodes go down. +* Replication Controller: Responsible for maintaining the correct number of pods for every replication + controller object in the system. +* Endpoints Controller: Populates the Endpoints object (i.e., join Services & Pods). +* Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces. +* ... and others. + +### kube-scheduler + +[kube-scheduler](/docs/admin/kube-scheduler) watches newly created pods that have no node assigned, and +selects a node for them to run on. + +### addons + +Addons are pods and services that implement cluster features. The pods may be managed +by Deployments, ReplicationContollers, etc. Namespaced addon objects are created in +the "kube-system" namespace. + +Addon manager takes the responsibility for creating and maintaining addon resources. +See [here](http://releases.k8s.io/HEAD/cluster/addons) for more details. + +#### DNS + +While the other addons are not strictly required, all Kubernetes +clusters should have [cluster DNS](/docs/admin/dns/), as many examples rely on it. + +Cluster DNS is a DNS server, in addition to the other DNS server(s) in your +environment, which serves DNS records for Kubernetes services. + +Containers started by Kubernetes automatically include this DNS server +in their DNS searches. + +#### User interface + +The kube-ui provides a read-only overview of the cluster state. Access +[the UI using kubectl proxy](/docs/user-guide/connecting-to-applications-proxy/#connecting-to-the-kube-ui-service-from-your-local-workstation) + +#### Container Resource Monitoring + +[Container Resource Monitoring](/docs/user-guide/monitoring) records generic time-series metrics +about containers in a central database, and provides a UI for browsing that data. + +#### Cluster-level Logging + +A [Cluster-level logging](/docs/user-guide/logging/overview) mechanism is responsible for +saving container logs to a central log store with search/browsing interface. + +## Node components + +Node components run on every node, maintaining running pods and providing them +the Kubernetes runtime environment. + +### kubelet + +[kubelet](/docs/admin/kubelet) is the primary node agent. It: + +* Watches for pods that have been assigned to its node (either by apiserver + or via local configuration file) and: +* Mounts the pod's required volumes +* Downloads the pod's secrets +* Runs the pod's containers via docker (or, experimentally, rkt). +* Periodically executes any requested container liveness probes. +* Reports the status of the pod back to the rest of the system, by creating a + "mirror pod" if necessary. +* Reports the status of the node back to the rest of the system. + +### kube-proxy + +[kube-proxy](/docs/admin/kube-proxy) enables the Kubernetes service abstraction by maintaining +network rules on the host and performing connection forwarding. + +### docker + +`docker` is of course used for actually running containers. + +### rkt + +`rkt` is supported experimentally as an alternative to docker. + +### supervisord + +`supervisord` is a lightweight process babysitting system for keeping kubelet and docker +running. + +### fluentd + +`fluentd` is a daemon which helps provide [cluster-level logging](#cluster-level-logging). From 1fdc1ebe55926b1a0669998949f24d04c160f776 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 14:41:51 -0700 Subject: [PATCH 375/407] Move Guide topic: Limit storage consumption. (#2814) * Move Guide topic: Limit storage consumption. * Add title. --- _data/tasks.yml | 1 + .../resourcequota/limitstorageconsumption.md | 71 +---------------- .../limit-storage-consumption.md | 76 +++++++++++++++++++ 3 files changed, 79 insertions(+), 69 deletions(-) create mode 100644 docs/tasks/administer-cluster/limit-storage-consumption.md diff --git a/_data/tasks.yml b/_data/tasks.yml index b32c862449..121090ea79 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -46,6 +46,7 @@ toc: - docs/tasks/administer-cluster/dns-horizontal-autoscaling.md - docs/tasks/administer-cluster/safely-drain-node.md - docs/tasks/administer-cluster/change-pv-reclaim-policy.md + - docs/tasks/administer-cluster/limit-storage-consumption.md - title: Administering Federation section: diff --git a/docs/admin/resourcequota/limitstorageconsumption.md b/docs/admin/resourcequota/limitstorageconsumption.md index 1c88088d8f..d1f3116cb0 100644 --- a/docs/admin/resourcequota/limitstorageconsumption.md +++ b/docs/admin/resourcequota/limitstorageconsumption.md @@ -4,75 +4,8 @@ assignees: - janetkuo title: Limiting Storage Consumption --- -This example demonstrates an easy way to limit the amount of storage consumed in a namespace. -The following resources are used in the demonstration: +{% include user-guide-content-moved.md %} -* [Resource Quota](/docs/admin/resourcequota/) -* [Limit Range](/docs/admin/limitrange/) -* [Persistent Volume Claim](/docs/user-guide/persistent-volumes/) +[Limiting Storage Consumption](/docs/tasks/administer-cluster/limit-storage-consumption/) -This example assumes you have a functional Kubernetes setup. - -## Limiting Storage Consumption - -The cluster-admin is operating a cluster on behalf of a user population and the admin wants to control -how much storage a single namespace can consume in order to control cost. - -The admin would like to limit: - -1. The number of persistent volume claims in a namespace -2. The amount of storage each claim can request -3. The amount of cumulative storage the namespace can have - - -## LimitRange to limit requests for storage - -Adding a `LimitRange` to a namespace enforces storage request sizes to a minimum and maximum. Storage is requested -via `PersistentVolumeClaim`. The admission controller that enforces limit ranges will reject any PVC that is above or below -the values set by the admin. - -In this example, a PVC requesting 10Gi of storage would be rejected because it exceeds the 2Gi max. - -``` -apiVersion: v1 -kind: LimitRange -metadata: - name: storagelimits -spec: - limits: - - type: PersistentVolumeClaim - max: - storage: 2Gi - min: - storage: 1Gi -``` - -Minimum storage requests are used when the underlying storage provider requires certain minimums. For example, -AWS EBS volumes have a 1Gi minimum requirement. - -## StorageQuota to limit PVC count and cumulative storage capacity - -Admins can limit the number of PVCs in a namespace as well as the cumulative capacity of those PVCs. New PVCs that exceed -either maximum value will be rejected. - -In this example, a 6th PVC in the namespace would be rejected because it exceeds the maximum count of 5. Alternatively, -a 5Gi maximum quota when combined with the 2Gi max limit above, cannot have 3 PVCs where each has 2Gi. That would be 6Gi requested - for a namespace capped at 5Gi. - -``` -apiVersion: v1 -kind: ResourceQuota -metadata: - name: storagequota -spec: - hard: - persistentvolumeclaims: "5" - requests.storage: "5Gi" -``` - -## Summary - -A limit range can put a ceiling on how much storage is requested while a resource quota can effectively cap the storage -consumed by a namespace through claim counts and cumulative storage capacity. The allows a cluster-admin to plan their -cluster's storage budget without risk of any one project going over their allotment. diff --git a/docs/tasks/administer-cluster/limit-storage-consumption.md b/docs/tasks/administer-cluster/limit-storage-consumption.md new file mode 100644 index 0000000000..bde00eb2e7 --- /dev/null +++ b/docs/tasks/administer-cluster/limit-storage-consumption.md @@ -0,0 +1,76 @@ +--- +title: Limiting Storage Consumption +--- + +This example demonstrates an easy way to limit the amount of storage consumed in a namespace. + +The following resources are used in the demonstration: + +* [Resource Quota](/docs/admin/resourcequota/) +* [Limit Range](/docs/admin/limitrange/) +* [Persistent Volume Claim](/docs/user-guide/persistent-volumes/) + +This example assumes you have a functional Kubernetes setup. + +## Limiting Storage Consumption + +The cluster-admin is operating a cluster on behalf of a user population and the admin wants to control +how much storage a single namespace can consume in order to control cost. + +The admin would like to limit: + +1. The number of persistent volume claims in a namespace +2. The amount of storage each claim can request +3. The amount of cumulative storage the namespace can have + + +## LimitRange to limit requests for storage + +Adding a `LimitRange` to a namespace enforces storage request sizes to a minimum and maximum. Storage is requested +via `PersistentVolumeClaim`. The admission controller that enforces limit ranges will reject any PVC that is above or below +the values set by the admin. + +In this example, a PVC requesting 10Gi of storage would be rejected because it exceeds the 2Gi max. + +``` +apiVersion: v1 +kind: LimitRange +metadata: + name: storagelimits +spec: + limits: + - type: PersistentVolumeClaim + max: + storage: 2Gi + min: + storage: 1Gi +``` + +Minimum storage requests are used when the underlying storage provider requires certain minimums. For example, +AWS EBS volumes have a 1Gi minimum requirement. + +## StorageQuota to limit PVC count and cumulative storage capacity + +Admins can limit the number of PVCs in a namespace as well as the cumulative capacity of those PVCs. New PVCs that exceed +either maximum value will be rejected. + +In this example, a 6th PVC in the namespace would be rejected because it exceeds the maximum count of 5. Alternatively, +a 5Gi maximum quota when combined with the 2Gi max limit above, cannot have 3 PVCs where each has 2Gi. That would be 6Gi requested + for a namespace capped at 5Gi. + +``` +apiVersion: v1 +kind: ResourceQuota +metadata: + name: storagequota +spec: + hard: + persistentvolumeclaims: "5" + requests.storage: "5Gi" +``` + +## Summary + +A limit range can put a ceiling on how much storage is requested while a resource quota can effectively cap the storage +consumed by a namespace through claim counts and cumulative storage capacity. The allows a cluster-admin to plan their +cluster's storage budget without risk of any one project going over their allotment. From fa668f520d128723689995e412406fa2da00bee8 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 15:24:28 -0700 Subject: [PATCH 376/407] Move Guide topic: Networking. (#2816) --- _data/concepts.yml | 1 + docs/admin/networking.md | 210 +---------------- .../cluster-administration/networking.md | 215 ++++++++++++++++++ 3 files changed, 218 insertions(+), 208 deletions(-) create mode 100644 docs/concepts/cluster-administration/networking.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 93c06c68a6..982ba71163 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -32,6 +32,7 @@ toc: - title: Cluster Administration section: + - docs/concepts/cluster-administration/networking.md - docs/concepts/cluster-administration/logging.md - docs/concepts/cluster-administration/multiple-clusters.md - docs/concepts/cluster-administration/federation.md diff --git a/docs/admin/networking.md b/docs/admin/networking.md index a2b7d37110..7041b1b7e7 100644 --- a/docs/admin/networking.md +++ b/docs/admin/networking.md @@ -4,212 +4,6 @@ assignees: title: Networking in Kubernetes --- -Kubernetes approaches networking somewhat differently than Docker does by -default. There are 4 distinct networking problems to solve: +{% include user-guide-content-moved.md %} -1. Highly-coupled container-to-container communications: this is solved by - [pods](/docs/user-guide/pods/) and `localhost` communications. -2. Pod-to-Pod communications: this is the primary focus of this document. -3. Pod-to-Service communications: this is covered by [services](/docs/user-guide/services/). -4. External-to-Service communications: this is covered by [services](/docs/user-guide/services/). - -* TOC -{:toc} - - -## Summary - -Kubernetes assumes that pods can communicate with other pods, regardless of -which host they land on. We give every pod its own IP address so you do not -need to explicitly create links between pods and you almost never need to deal -with mapping container ports to host ports. This creates a clean, -backwards-compatible model where pods can be treated much like VMs or physical -hosts from the perspectives of port allocation, naming, service discovery, load -balancing, application configuration, and migration. - -To achieve this we must impose some requirements on how you set up your cluster -networking. - -## Docker model - -Before discussing the Kubernetes approach to networking, it is worthwhile to -review the "normal" way that networking works with Docker. By default, Docker -uses host-private networking. It creates a virtual bridge, called `docker0` by -default, and allocates a subnet from one of the private address blocks defined -in [RFC1918](https://tools.ietf.org/html/rfc1918) for that bridge. For each -container that Docker creates, it allocates a virtual ethernet device (called -`veth`) which is attached to the bridge. The veth is mapped to appear as `eth0` -in the container, using Linux namespaces. The in-container `eth0` interface is -given an IP address from the bridge's address range. - -The result is that Docker containers can talk to other containers only if they -are on the same machine (and thus the same virtual bridge). Containers on -different machines can not reach each other - in fact they may end up with the -exact same network ranges and IP addresses. - -In order for Docker containers to communicate across nodes, they must be -allocated ports on the machine's own IP address, which are then forwarded or -proxied to the containers. This obviously means that containers must either -coordinate which ports they use very carefully or else be allocated ports -dynamically. - -## Kubernetes model - -Coordinating ports across multiple developers is very difficult to do at -scale and exposes users to cluster-level issues outside of their control. -Dynamic port allocation brings a lot of complications to the system - every -application has to take ports as flags, the API servers have to know how to -insert dynamic port numbers into configuration blocks, services have to know -how to find each other, etc. Rather than deal with this, Kubernetes takes a -different approach. - -Kubernetes imposes the following fundamental requirements on any networking -implementation (barring any intentional network segmentation policies): - - * all containers can communicate with all other containers without NAT - * all nodes can communicate with all containers (and vice-versa) without NAT - * the IP that a container sees itself as is the same IP that others see it as - -What this means in practice is that you can not just take two computers -running Docker and expect Kubernetes to work. You must ensure that the -fundamental requirements are met. - -This model is not only less complex overall, but it is principally compatible -with the desire for Kubernetes to enable low-friction porting of apps from VMs -to containers. If your job previously ran in a VM, your VM had an IP and could -talk to other VMs in your project. This is the same basic model. - -Until now this document has talked about containers. In reality, Kubernetes -applies IP addresses at the `Pod` scope - containers within a `Pod` share their -network namespaces - including their IP address. This means that containers -within a `Pod` can all reach each other's ports on `localhost`. This does imply -that containers within a `Pod` must coordinate port usage, but this is no -different than processes in a VM. We call this the "IP-per-pod" model. This -is implemented in Docker as a "pod container" which holds the network namespace -open while "app containers" (the things the user specified) join that namespace -with Docker's `--net=container:` function. - -As with Docker, it is possible to request host ports, but this is reduced to a -very niche operation. In this case a port will be allocated on the host `Node` -and traffic will be forwarded to the `Pod`. The `Pod` itself is blind to the -existence or non-existence of host ports. - -## How to achieve this - -There are a number of ways that this network model can be implemented. This -document is not an exhaustive study of the various methods, but hopefully serves -as an introduction to various technologies and serves as a jumping-off point. - -The following networking options are sorted alphabetically - the order does not -imply any preferential status. - -### Contiv - -[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](http://contiv.io) is all open sourced. - -### Flannel - -[Flannel](https://github.com/coreos/flannel#flannel) is a very simple overlay -network that satisfies the Kubernetes requirements. Many -people have reported success with Flannel and Kubernetes. - -### Google Compute Engine (GCE) - -For the Google Compute Engine cluster configuration scripts, we use [advanced -routing](https://cloud.google.com/compute/docs/networking#routing) to -assign each VM a subnet (default is `/24` - 254 IPs). Any traffic bound for that -subnet will be routed directly to the VM by the GCE network fabric. This is in -addition to the "main" IP address assigned to the VM, which is NAT'ed for -outbound internet access. A linux bridge (called `cbr0`) is configured to exist -on that subnet, and is passed to docker's `--bridge` flag. - -We start Docker with: - -```shell -DOCKER_OPTS="--bridge=cbr0 --iptables=false --ip-masq=false" -``` - -This bridge is created by Kubelet (controlled by the `--network-plugin=kubenet` -flag) according to the `Node`'s `spec.podCIDR`. - -Docker will now allocate IPs from the `cbr-cidr` block. Containers can reach -each other and `Nodes` over the `cbr0` bridge. Those IPs are all routable -within the GCE project network. - -GCE itself does not know anything about these IPs, though, so it will not NAT -them for outbound internet traffic. To achieve that we use an iptables rule to -masquerade (aka SNAT - to make it seem as if packets came from the `Node` -itself) traffic that is bound for IPs outside the GCE project network -(10.0.0.0/8). - -```shell -iptables -t nat -A POSTROUTING ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE -``` - -Lastly we enable IP forwarding in the kernel (so the kernel will process -packets for bridged containers): - -```shell -sysctl net.ipv4.ip_forward=1 -``` - -The result of all this is that all `Pods` can reach each other and can egress -traffic to the internet. - -### L2 networks and linux bridging - -If you have a "dumb" L2 network, such as a simple switch in a "bare-metal" -environment, you should be able to do something similar to the above GCE setup. -Note that these instructions have only been tried very casually - it seems to -work, but has not been thoroughly tested. If you use this technique and -perfect the process, please let us know. - -Follow the "With Linux Bridge devices" section of [this very nice -tutorial](http://blog.oddbit.com/2014/08/11/four-ways-to-connect-a-docker/) from -Lars Kellogg-Stedman. - -### Nuage Networks VCS (Virtualized Cloud Services) - -[Nuage](http://www.nuagenetworks.net) provides a highly scalable policy-based Software-Defined Networking (SDN) platform. Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards. - -The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage's policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications.The platform's real-time analytics engine enables visibility and security monitoring for Kubernetes applications. - -### OpenVSwitch - -[OpenVSwitch](/docs/admin/ovs-networking) is a somewhat more mature but also -complicated way to build an overlay network. This is endorsed by several of the -"Big Shops" for networking. - -### OVN (Open Virtual Networking) - -OVN is an opensource network virtualization solution developed by the -Open vSwitch community. It lets one create logical switches, logical routers, -stateful ACLs, load-balancers etc to build different virtual networking -topologies. The project has a specific Kubernetes plugin and documentation -at [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes). - -### Project Calico - -[Project Calico](http://docs.projectcalico.org/) is an open source container networking provider and network policy engine. - -Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet. Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking. Calico also provides fine-grained, intent based network security policy for Kubernetes pods via its distributed firewall. - -Calico can also be run in policy enforcement mode in conjunction with other networking solutions such as Flannel, aka [canal](https://github.com/tigera/canal), or native GCE networking. - -### Romana - -[Romana](http://romana.io) is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network. Romana supports Kubernetes [Network Policy](/docs/user-guide/networkpolicies/) to provide isolation across network namespaces. - -### Weave Net from Weaveworks - -[Weave Net](https://www.weave.works/products/weave-net/) is a -resilient and simple to use network for Kubernetes and its hosted applications. -Weave Net runs as a [CNI plug-in](https://www.weave.works/docs/net/latest/cni-plugin/) -or stand-alone. In either version, it doesn't require any configuration or extra code -to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes. - -## Other reading - -The early design of the networking model and its rationale, and some future -plans are described in more detail in the [networking design -document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/networking.md). +[Cluster Networking](/docs/concepts/cluster-administration/networking/) diff --git a/docs/concepts/cluster-administration/networking.md b/docs/concepts/cluster-administration/networking.md new file mode 100644 index 0000000000..533a9ee874 --- /dev/null +++ b/docs/concepts/cluster-administration/networking.md @@ -0,0 +1,215 @@ +--- +assignees: +- thockin +title: Cluster Networking +--- + +Kubernetes approaches networking somewhat differently than Docker does by +default. There are 4 distinct networking problems to solve: + +1. Highly-coupled container-to-container communications: this is solved by + [pods](/docs/user-guide/pods/) and `localhost` communications. +2. Pod-to-Pod communications: this is the primary focus of this document. +3. Pod-to-Service communications: this is covered by [services](/docs/user-guide/services/). +4. External-to-Service communications: this is covered by [services](/docs/user-guide/services/). + +* TOC +{:toc} + + +## Summary + +Kubernetes assumes that pods can communicate with other pods, regardless of +which host they land on. We give every pod its own IP address so you do not +need to explicitly create links between pods and you almost never need to deal +with mapping container ports to host ports. This creates a clean, +backwards-compatible model where pods can be treated much like VMs or physical +hosts from the perspectives of port allocation, naming, service discovery, load +balancing, application configuration, and migration. + +To achieve this we must impose some requirements on how you set up your cluster +networking. + +## Docker model + +Before discussing the Kubernetes approach to networking, it is worthwhile to +review the "normal" way that networking works with Docker. By default, Docker +uses host-private networking. It creates a virtual bridge, called `docker0` by +default, and allocates a subnet from one of the private address blocks defined +in [RFC1918](https://tools.ietf.org/html/rfc1918) for that bridge. For each +container that Docker creates, it allocates a virtual ethernet device (called +`veth`) which is attached to the bridge. The veth is mapped to appear as `eth0` +in the container, using Linux namespaces. The in-container `eth0` interface is +given an IP address from the bridge's address range. + +The result is that Docker containers can talk to other containers only if they +are on the same machine (and thus the same virtual bridge). Containers on +different machines can not reach each other - in fact they may end up with the +exact same network ranges and IP addresses. + +In order for Docker containers to communicate across nodes, they must be +allocated ports on the machine's own IP address, which are then forwarded or +proxied to the containers. This obviously means that containers must either +coordinate which ports they use very carefully or else be allocated ports +dynamically. + +## Kubernetes model + +Coordinating ports across multiple developers is very difficult to do at +scale and exposes users to cluster-level issues outside of their control. +Dynamic port allocation brings a lot of complications to the system - every +application has to take ports as flags, the API servers have to know how to +insert dynamic port numbers into configuration blocks, services have to know +how to find each other, etc. Rather than deal with this, Kubernetes takes a +different approach. + +Kubernetes imposes the following fundamental requirements on any networking +implementation (barring any intentional network segmentation policies): + + * all containers can communicate with all other containers without NAT + * all nodes can communicate with all containers (and vice-versa) without NAT + * the IP that a container sees itself as is the same IP that others see it as + +What this means in practice is that you can not just take two computers +running Docker and expect Kubernetes to work. You must ensure that the +fundamental requirements are met. + +This model is not only less complex overall, but it is principally compatible +with the desire for Kubernetes to enable low-friction porting of apps from VMs +to containers. If your job previously ran in a VM, your VM had an IP and could +talk to other VMs in your project. This is the same basic model. + +Until now this document has talked about containers. In reality, Kubernetes +applies IP addresses at the `Pod` scope - containers within a `Pod` share their +network namespaces - including their IP address. This means that containers +within a `Pod` can all reach each other's ports on `localhost`. This does imply +that containers within a `Pod` must coordinate port usage, but this is no +different than processes in a VM. We call this the "IP-per-pod" model. This +is implemented in Docker as a "pod container" which holds the network namespace +open while "app containers" (the things the user specified) join that namespace +with Docker's `--net=container:` function. + +As with Docker, it is possible to request host ports, but this is reduced to a +very niche operation. In this case a port will be allocated on the host `Node` +and traffic will be forwarded to the `Pod`. The `Pod` itself is blind to the +existence or non-existence of host ports. + +## How to achieve this + +There are a number of ways that this network model can be implemented. This +document is not an exhaustive study of the various methods, but hopefully serves +as an introduction to various technologies and serves as a jumping-off point. + +The following networking options are sorted alphabetically - the order does not +imply any preferential status. + +### Contiv + +[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](http://contiv.io) is all open sourced. + +### Flannel + +[Flannel](https://github.com/coreos/flannel#flannel) is a very simple overlay +network that satisfies the Kubernetes requirements. Many +people have reported success with Flannel and Kubernetes. + +### Google Compute Engine (GCE) + +For the Google Compute Engine cluster configuration scripts, we use [advanced +routing](https://cloud.google.com/compute/docs/networking#routing) to +assign each VM a subnet (default is `/24` - 254 IPs). Any traffic bound for that +subnet will be routed directly to the VM by the GCE network fabric. This is in +addition to the "main" IP address assigned to the VM, which is NAT'ed for +outbound internet access. A linux bridge (called `cbr0`) is configured to exist +on that subnet, and is passed to docker's `--bridge` flag. + +We start Docker with: + +```shell +DOCKER_OPTS="--bridge=cbr0 --iptables=false --ip-masq=false" +``` + +This bridge is created by Kubelet (controlled by the `--network-plugin=kubenet` +flag) according to the `Node`'s `spec.podCIDR`. + +Docker will now allocate IPs from the `cbr-cidr` block. Containers can reach +each other and `Nodes` over the `cbr0` bridge. Those IPs are all routable +within the GCE project network. + +GCE itself does not know anything about these IPs, though, so it will not NAT +them for outbound internet traffic. To achieve that we use an iptables rule to +masquerade (aka SNAT - to make it seem as if packets came from the `Node` +itself) traffic that is bound for IPs outside the GCE project network +(10.0.0.0/8). + +```shell +iptables -t nat -A POSTROUTING ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE +``` + +Lastly we enable IP forwarding in the kernel (so the kernel will process +packets for bridged containers): + +```shell +sysctl net.ipv4.ip_forward=1 +``` + +The result of all this is that all `Pods` can reach each other and can egress +traffic to the internet. + +### L2 networks and linux bridging + +If you have a "dumb" L2 network, such as a simple switch in a "bare-metal" +environment, you should be able to do something similar to the above GCE setup. +Note that these instructions have only been tried very casually - it seems to +work, but has not been thoroughly tested. If you use this technique and +perfect the process, please let us know. + +Follow the "With Linux Bridge devices" section of [this very nice +tutorial](http://blog.oddbit.com/2014/08/11/four-ways-to-connect-a-docker/) from +Lars Kellogg-Stedman. + +### Nuage Networks VCS (Virtualized Cloud Services) + +[Nuage](http://www.nuagenetworks.net) provides a highly scalable policy-based Software-Defined Networking (SDN) platform. Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards. + +The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage's policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications.The platform's real-time analytics engine enables visibility and security monitoring for Kubernetes applications. + +### OpenVSwitch + +[OpenVSwitch](/docs/admin/ovs-networking) is a somewhat more mature but also +complicated way to build an overlay network. This is endorsed by several of the +"Big Shops" for networking. + +### OVN (Open Virtual Networking) + +OVN is an opensource network virtualization solution developed by the +Open vSwitch community. It lets one create logical switches, logical routers, +stateful ACLs, load-balancers etc to build different virtual networking +topologies. The project has a specific Kubernetes plugin and documentation +at [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes). + +### Project Calico + +[Project Calico](http://docs.projectcalico.org/) is an open source container networking provider and network policy engine. + +Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet. Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking. Calico also provides fine-grained, intent based network security policy for Kubernetes pods via its distributed firewall. + +Calico can also be run in policy enforcement mode in conjunction with other networking solutions such as Flannel, aka [canal](https://github.com/tigera/canal), or native GCE networking. + +### Romana + +[Romana](http://romana.io) is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network. Romana supports Kubernetes [Network Policy](/docs/user-guide/networkpolicies/) to provide isolation across network namespaces. + +### Weave Net from Weaveworks + +[Weave Net](https://www.weave.works/products/weave-net/) is a +resilient and simple to use network for Kubernetes and its hosted applications. +Weave Net runs as a [CNI plug-in](https://www.weave.works/docs/net/latest/cni-plugin/) +or stand-alone. In either version, it doesn't require any configuration or extra code +to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes. + +## Other reading + +The early design of the networking model and its rationale, and some future +plans are described in more detail in the [networking design +document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/networking.md). From 458548ace1f972c748c60c6cc05da15eefa711f7 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 15:45:26 -0700 Subject: [PATCH 377/407] Move Guide topic: Network Plugins. (#2819) * Move Guide topic: Network Plugins. * Fix link. --- _data/concepts.yml | 1 + docs/admin/network-plugins.md | 66 +---------------- .../cluster-administration/network-plugins.md | 73 +++++++++++++++++++ 3 files changed, 76 insertions(+), 64 deletions(-) create mode 100644 docs/concepts/cluster-administration/network-plugins.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 982ba71163..5a82b9b7a1 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -33,6 +33,7 @@ toc: - title: Cluster Administration section: - docs/concepts/cluster-administration/networking.md + - docs/concepts/cluster-administration/network-plugins.md - docs/concepts/cluster-administration/logging.md - docs/concepts/cluster-administration/multiple-clusters.md - docs/concepts/cluster-administration/federation.md diff --git a/docs/admin/network-plugins.md b/docs/admin/network-plugins.md index f06d22fb87..bd22f4eb9f 100644 --- a/docs/admin/network-plugins.md +++ b/docs/admin/network-plugins.md @@ -6,68 +6,6 @@ assignees: title: Network Plugins --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -__Disclaimer__: Network plugins are in alpha. Its contents will change rapidly. - -Network plugins in Kubernetes come in a few flavors: - -* CNI plugins: adhere to the appc/CNI specification, designed for interoperability. -* Kubenet plugin: implements basic `cbr0` using the `bridge` and `host-local` CNI plugins - -## Installation - -The kubelet has a single default network plugin, and a default network common to the entire cluster. It probes for plugins when it starts up, remembers what it found, and executes the selected plugin at appropriate times in the pod lifecycle (this is only true for docker, as rkt manages its own CNI plugins). There are two Kubelet command line parameters to keep in mind when using plugins: - -* `network-plugin-dir`: Kubelet probes this directory for plugins on startup -* `network-plugin`: The network plugin to use from `network-plugin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is simply "cni". - -## Network Plugin Requirements - -Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{page.version}}/pkg/kubelet/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy. - -By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like docker with a bridge) work correctly with the iptables proxy. - -### CNI - -The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`). - -If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used. - -In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/cni/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0 - -Limitation: Due to [#31307](https://github.com/kubernetes/kubernetes/issues/31307), `HostPort` won't work with CNI networking plugin at the moment. That means all `hostPort` attribute in pod would be simply ignored. - -### kubenet - -Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments. - -Kubenet creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host. - -The plugin requires a few things: - -* The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `network-plugin-dir` to supply additional search path. The first found match will take effect. -* Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin -* Kubelet should also be run with the `--non-masquerade-cidr=` argument to ensure traffic to IPs outside this range will use IP masquerade. -* The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=` controller-manager command-line options. - -### Customizing the MTU (with kubenet) - -The MTU should always be configured correctly to get the best networking performance. Network plugins will usually try -to infer a sensible MTU, but sometimes the logic will not result in an optimal MTU. For example, if the -Docker bridge or another interface has a small MTU, kubenet will currently select that MTU. Or if you are -using IPSEC encapsulation, the MTU must be reduced, and this calculation is out-of-scope for -most network plugins. - -Where needed, you can specify the MTU explicitly with the `network-plugin-mtu` kubelet option. For example, -on AWS the `eth0` MTU is typically 9001, so you might specify `--network-plugin-mtu=9001`. If you're using IPSEC you -might reduce it to allow for encapsulation overhead e.g. `--network-plugin-mtu=8873`. - -This option is provided to the network-plugin; currently **only kubenet supports `network-plugin-mtu`**. - -## Usage Summary - -* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`). -* `--network-plugin=kubenet` specifies that we use the `kubenet` network plugin with CNI `bridge` and `host-local` plugins placed in `/opt/cni/bin` or `network-plugin-dir`. -* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin. +[Network Plugins](/docs/concepts/cluster-administration/network-plugins/) diff --git a/docs/concepts/cluster-administration/network-plugins.md b/docs/concepts/cluster-administration/network-plugins.md new file mode 100644 index 0000000000..f06d22fb87 --- /dev/null +++ b/docs/concepts/cluster-administration/network-plugins.md @@ -0,0 +1,73 @@ +--- +assignees: +- dcbw +- freehan +- thockin +title: Network Plugins +--- + +* TOC +{:toc} + +__Disclaimer__: Network plugins are in alpha. Its contents will change rapidly. + +Network plugins in Kubernetes come in a few flavors: + +* CNI plugins: adhere to the appc/CNI specification, designed for interoperability. +* Kubenet plugin: implements basic `cbr0` using the `bridge` and `host-local` CNI plugins + +## Installation + +The kubelet has a single default network plugin, and a default network common to the entire cluster. It probes for plugins when it starts up, remembers what it found, and executes the selected plugin at appropriate times in the pod lifecycle (this is only true for docker, as rkt manages its own CNI plugins). There are two Kubelet command line parameters to keep in mind when using plugins: + +* `network-plugin-dir`: Kubelet probes this directory for plugins on startup +* `network-plugin`: The network plugin to use from `network-plugin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is simply "cni". + +## Network Plugin Requirements + +Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{page.version}}/pkg/kubelet/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy. + +By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like docker with a bridge) work correctly with the iptables proxy. + +### CNI + +The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`). + +If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used. + +In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/cni/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0 + +Limitation: Due to [#31307](https://github.com/kubernetes/kubernetes/issues/31307), `HostPort` won't work with CNI networking plugin at the moment. That means all `hostPort` attribute in pod would be simply ignored. + +### kubenet + +Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments. + +Kubenet creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host. + +The plugin requires a few things: + +* The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `network-plugin-dir` to supply additional search path. The first found match will take effect. +* Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin +* Kubelet should also be run with the `--non-masquerade-cidr=` argument to ensure traffic to IPs outside this range will use IP masquerade. +* The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=` controller-manager command-line options. + +### Customizing the MTU (with kubenet) + +The MTU should always be configured correctly to get the best networking performance. Network plugins will usually try +to infer a sensible MTU, but sometimes the logic will not result in an optimal MTU. For example, if the +Docker bridge or another interface has a small MTU, kubenet will currently select that MTU. Or if you are +using IPSEC encapsulation, the MTU must be reduced, and this calculation is out-of-scope for +most network plugins. + +Where needed, you can specify the MTU explicitly with the `network-plugin-mtu` kubelet option. For example, +on AWS the `eth0` MTU is typically 9001, so you might specify `--network-plugin-mtu=9001`. If you're using IPSEC you +might reduce it to allow for encapsulation overhead e.g. `--network-plugin-mtu=8873`. + +This option is provided to the network-plugin; currently **only kubenet supports `network-plugin-mtu`**. + +## Usage Summary + +* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`). +* `--network-plugin=kubenet` specifies that we use the `kubenet` network plugin with CNI `bridge` and `host-local` plugins placed in `/opt/cni/bin` or `network-plugin-dir`. +* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin. From d0d6f4ca488ed4b8b9f2016ca0194eb259833617 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 16:28:06 -0700 Subject: [PATCH 378/407] Move Guide topic: Static Pods. (#2820) --- _data/concepts.yml | 1 + docs/admin/static-pods.md | 123 +---------------- .../cluster-administration/static-pod.md | 128 ++++++++++++++++++ 3 files changed, 131 insertions(+), 121 deletions(-) create mode 100644 docs/concepts/cluster-administration/static-pod.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 5a82b9b7a1..e7cce8c70d 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -38,6 +38,7 @@ toc: - docs/concepts/cluster-administration/multiple-clusters.md - docs/concepts/cluster-administration/federation.md - docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md + - docs/concepts/cluster-administration/static-pod.md - docs/concepts/cluster-administration/sysctl-cluster.md - title: Configuration diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 88aa78a212..d924319fd9 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -4,125 +4,6 @@ assignees: title: Static Pods --- -**If you are running clustered Kubernetes and are using static pods to run a pod on every node, you should probably be using a [DaemonSet](/docs/admin/daemons/)!** +{% include user-guide-content-moved.md %} -*Static pods* are managed directly by kubelet daemon on a specific node, without API server observing it. It does not have associated any replication controller, kubelet daemon itself watches it and restarts it when it crashes. There is no health check though. Static pods are always bound to one kubelet daemon and always run on the same node with it. - -Kubelet automatically creates so-called *mirror pod* on Kubernetes API server for each static pod, so the pods are visible there, but they cannot be controlled from the API server. - -## Static pod creation - -Static pod can be created in two ways: either by using configuration file(s) or by HTTP. - -### Configuration files - -The configuration files are just standard pod definition in json or yaml format in specific directory. Use `kubelet --pod-manifest-path=` to start kubelet daemon, which periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there. - -For example, this is how to start a simple web server as a static pod: - -1. Choose a node where we want to run the static pod. In this example, it's `my-node1`. - - ``` - [joe@host ~] $ ssh my-node1 - ``` - -2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`: - - ``` - [root@my-node1 ~] $ mkdir /etc/kubernetes.d/ - [root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml - apiVersion: v1 - kind: Pod - metadata: - name: static-web - labels: - role: myrole - spec: - containers: - - name: web - image: nginx - ports: - - name: web - containerPort: 80 - protocol: TCP - EOF - ``` - -3. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. - On Fedora edit `/etc/kubernetes/kubelet` to include this line: - - ``` - KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" - ``` - - Instructions for other distributions or Kubernetes installations may vary. - -4. Restart kubelet. On Fedora, this is: - - ``` - [root@my-node1 ~] $ systemctl restart kubelet - ``` - -## Pods created via HTTP - -Kubelet periodically downloads a file specified by `--manifest-url=` argument and interprets it as a json/yaml file with a pod definition. It works the same as `--pod-manifest-path=`, i.e. it's reloaded every now and then and changes are applied to running static pods (see below). - -## Behavior of static pods - -When kubelet starts, it automatically starts all pods defined in directory specified in `--pod-manifest-path=` or `--manifest-url=` arguments, i.e. our static-web. (It may take some time to pull nginx image, be patient…): - -```shell -[joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c -``` - -If we look at our Kubernetes API server (running on host `my-master`), we see that a new mirror-pod was created there too: - -```shell -[joe@host ~] $ ssh my-master -[joe@my-master ~] $ kubectl get pods -NAME READY STATUS RESTARTS AGE -static-web-my-node1 1/1 Running 0 2m - -``` - -Labels from the static pod are propagated into the mirror-pod and can be used as usual for filtering. - -Notice we cannot delete the pod with the API server (e.g. via [`kubectl`](/docs/user-guide/kubectl/) command), kubelet simply won't remove it. - -```shell -[joe@my-master ~] $ kubectl delete pod static-web-my-node1 -pods/static-web-my-node1 -[joe@my-master ~] $ kubectl get pods -NAME READY STATUS RESTARTS AGE -static-web-my-node1 1/1 Running 0 12s - -``` - -Back to our `my-node1` host, we can try to stop the container manually and see, that kubelet automatically restarts it in a while: - -```shell -[joe@host ~] $ ssh my-node1 -[joe@my-node1 ~] $ docker stop f6d05272b57e -[joe@my-node1 ~] $ sleep 20 -[joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED ... -5b920cbaf8b1 nginx:latest "nginx -g 'daemon of 2 seconds ago ... -``` - -## Dynamic addition and removal of static pods - -Running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes pods as files appear/disappear in this directory. - -```shell -[joe@my-node1 ~] $ mv /etc/kubelet.d/static-web.yaml /tmp -[joe@my-node1 ~] $ sleep 20 -[joe@my-node1 ~] $ docker ps -// no nginx container is running -[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubelet.d/ -[joe@my-node1 ~] $ sleep 20 -[joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED ... -e7a62e3427f1 nginx:latest "nginx -g 'daemon of 27 seconds ago -``` +[Static Pods](/docs/concepts/cluster-administration/static-pod/) diff --git a/docs/concepts/cluster-administration/static-pod.md b/docs/concepts/cluster-administration/static-pod.md new file mode 100644 index 0000000000..88aa78a212 --- /dev/null +++ b/docs/concepts/cluster-administration/static-pod.md @@ -0,0 +1,128 @@ +--- +assignees: +- jsafrane +title: Static Pods +--- + +**If you are running clustered Kubernetes and are using static pods to run a pod on every node, you should probably be using a [DaemonSet](/docs/admin/daemons/)!** + +*Static pods* are managed directly by kubelet daemon on a specific node, without API server observing it. It does not have associated any replication controller, kubelet daemon itself watches it and restarts it when it crashes. There is no health check though. Static pods are always bound to one kubelet daemon and always run on the same node with it. + +Kubelet automatically creates so-called *mirror pod* on Kubernetes API server for each static pod, so the pods are visible there, but they cannot be controlled from the API server. + +## Static pod creation + +Static pod can be created in two ways: either by using configuration file(s) or by HTTP. + +### Configuration files + +The configuration files are just standard pod definition in json or yaml format in specific directory. Use `kubelet --pod-manifest-path=` to start kubelet daemon, which periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there. + +For example, this is how to start a simple web server as a static pod: + +1. Choose a node where we want to run the static pod. In this example, it's `my-node1`. + + ``` + [joe@host ~] $ ssh my-node1 + ``` + +2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`: + + ``` + [root@my-node1 ~] $ mkdir /etc/kubernetes.d/ + [root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml + apiVersion: v1 + kind: Pod + metadata: + name: static-web + labels: + role: myrole + spec: + containers: + - name: web + image: nginx + ports: + - name: web + containerPort: 80 + protocol: TCP + EOF + ``` + +3. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. + On Fedora edit `/etc/kubernetes/kubelet` to include this line: + + ``` + KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" + ``` + + Instructions for other distributions or Kubernetes installations may vary. + +4. Restart kubelet. On Fedora, this is: + + ``` + [root@my-node1 ~] $ systemctl restart kubelet + ``` + +## Pods created via HTTP + +Kubelet periodically downloads a file specified by `--manifest-url=` argument and interprets it as a json/yaml file with a pod definition. It works the same as `--pod-manifest-path=`, i.e. it's reloaded every now and then and changes are applied to running static pods (see below). + +## Behavior of static pods + +When kubelet starts, it automatically starts all pods defined in directory specified in `--pod-manifest-path=` or `--manifest-url=` arguments, i.e. our static-web. (It may take some time to pull nginx image, be patient…): + +```shell +[joe@my-node1 ~] $ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c +``` + +If we look at our Kubernetes API server (running on host `my-master`), we see that a new mirror-pod was created there too: + +```shell +[joe@host ~] $ ssh my-master +[joe@my-master ~] $ kubectl get pods +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 2m + +``` + +Labels from the static pod are propagated into the mirror-pod and can be used as usual for filtering. + +Notice we cannot delete the pod with the API server (e.g. via [`kubectl`](/docs/user-guide/kubectl/) command), kubelet simply won't remove it. + +```shell +[joe@my-master ~] $ kubectl delete pod static-web-my-node1 +pods/static-web-my-node1 +[joe@my-master ~] $ kubectl get pods +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 12s + +``` + +Back to our `my-node1` host, we can try to stop the container manually and see, that kubelet automatically restarts it in a while: + +```shell +[joe@host ~] $ ssh my-node1 +[joe@my-node1 ~] $ docker stop f6d05272b57e +[joe@my-node1 ~] $ sleep 20 +[joe@my-node1 ~] $ docker ps +CONTAINER ID IMAGE COMMAND CREATED ... +5b920cbaf8b1 nginx:latest "nginx -g 'daemon of 2 seconds ago ... +``` + +## Dynamic addition and removal of static pods + +Running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes pods as files appear/disappear in this directory. + +```shell +[joe@my-node1 ~] $ mv /etc/kubelet.d/static-web.yaml /tmp +[joe@my-node1 ~] $ sleep 20 +[joe@my-node1 ~] $ docker ps +// no nginx container is running +[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubelet.d/ +[joe@my-node1 ~] $ sleep 20 +[joe@my-node1 ~] $ docker ps +CONTAINER ID IMAGE COMMAND CREATED ... +e7a62e3427f1 nginx:latest "nginx -g 'daemon of 27 seconds ago +``` From 60c692157602c22d5997140f8522325a82deef76 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 14 Mar 2017 16:39:25 -0700 Subject: [PATCH 379/407] User Guide content migration: post notice (#2818) * Add User Guide content migration notice. * Fix formatting * Tweek formatting to block highlight text in light gray. * Try table instead of code block * remove extra lines * try table format * fix links * incorporate @ddonnelly feedback --- _includes/user-guide-migration-notice.md | 12 ++++++++++++ docs/user-guide/index.md | 2 ++ 2 files changed, 14 insertions(+) create mode 100644 _includes/user-guide-migration-notice.md diff --git a/_includes/user-guide-migration-notice.md b/_includes/user-guide-migration-notice.md new file mode 100644 index 0000000000..7e64abe8d0 --- /dev/null +++ b/_includes/user-guide-migration-notice.md @@ -0,0 +1,12 @@ + + +

    unschedulable

    Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info: http://releases.k8s.io/HEAD/docs/admin/node.md#manual-node-administration"

    Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info: http://releases.k8s.io/HEAD/docs/admin/node.md#manual-node-administration

    false

    boolean

    false

    nodeSelector

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/README

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection

    false

    object

    nodeSelector

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/README

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection

    false

    object

    nodeSelector

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/README

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection

    false

    object

    nodeSelector

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/README

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection

    false

    object

    nodeSelector

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/README

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection

    false

    object
    nodeSelector
    object    		NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/READMENodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection
    restartPolicy
    string

    nodeSelector

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/README

    NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection

    false

    object
    nodeSelector
    object    		NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection/READMENodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://kubernetes.io/docs/user-guide/node-selection
    restartPolicy
    string
    + + + +
    +

    NOTICE

    +

    As of March 14, 2017, the @kubernetes/sig-docs-maintainers have begun migration of the User Guide content as announced previously to the SIG Docs community through the kubernetes-sig-docs group and kubernetes.slack.com #sig-docs channel.

    +

    The user guides within this section are being refactored into topics within Tutorials, Tasks, and Concepts. Anything that has been moved will have a notice placed in its previous location as well as a link to its new location. The reorganization implements the table of contents (ToC) outlined in the kubernetes-docs-toc document and should improve the documentation's findability and readability for a wider range of audiences.

    +

    For any questions, please contact: kubernetes-sig-docs@googlegroups.com

    +
    diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 1f045cb7e0..01a4b5538f 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -4,6 +4,8 @@ assignees: title: User Guide --- +{% include user-guide-migration-notice.md %} + The Kubernetes **Guides** can help you work with various aspects of the Kubernetes system. * The Kubernetes [User Guide](#user-guide-internal) can help you run programs and services on an existing Kubernetes cluster. From 2dcf7a36bb640f155a18dbb4130f72b9f626a9e7 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 16:54:21 -0700 Subject: [PATCH 380/407] Move Guide topic: Out of Resource. (#2821) --- _data/concepts.yml | 1 + docs/admin/out-of-resource.md | 361 +---------------- .../cluster-administration/out-of-resource.md | 368 ++++++++++++++++++ 3 files changed, 371 insertions(+), 359 deletions(-) create mode 100644 docs/concepts/cluster-administration/out-of-resource.md diff --git a/_data/concepts.yml b/_data/concepts.yml index e7cce8c70d..9301f10917 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -35,6 +35,7 @@ toc: - docs/concepts/cluster-administration/networking.md - docs/concepts/cluster-administration/network-plugins.md - docs/concepts/cluster-administration/logging.md + - docs/concepts/cluster-administration/out-of-resource.md - docs/concepts/cluster-administration/multiple-clusters.md - docs/concepts/cluster-administration/federation.md - docs/concepts/cluster-administration/guaranteed-scheduling-critical-addon-pods.md diff --git a/docs/admin/out-of-resource.md b/docs/admin/out-of-resource.md index 76e7125e7e..00f92cecf7 100644 --- a/docs/admin/out-of-resource.md +++ b/docs/admin/out-of-resource.md @@ -6,363 +6,6 @@ assignees: title: Configuring Out Of Resource Handling --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -The `kubelet` needs to preserve node stability when available compute resources are low. - -This is especially important when dealing with incompressible resources such as memory or disk. - -If either resource is exhausted, the node would become unstable. - -## Eviction Policy - -The `kubelet` can pro-actively monitor for and prevent against total starvation of a compute resource. In those cases, the `kubelet` can pro-actively fail one or more pods in order to reclaim -the starved resource. When the `kubelet` fails a pod, it terminates all containers in the pod, and the `PodPhase` -is transitioned to `Failed`. - -### Eviction Signals - -The `kubelet` can support the ability to trigger eviction decisions on the signals described in the -table below. The value of each signal is described in the description column based on the `kubelet` -summary API. - -| Eviction Signal | Description | -|----------------------------|-----------------------------------------------------------------------| -| `memory.available` | `memory.available` := `node.status.capacity[memory]` - `node.stats.memory.workingSet` | -| `nodefs.available` | `nodefs.available` := `node.stats.fs.available` | -| `nodefs.inodesFree` | `nodefs.inodesFree` := `node.stats.fs.inodesFree` | -| `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available` | -| `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree` | - -Each of the above signals supports either a literal or percentage based value. The percentage based value -is calculated relative to the total capacity associated with each signal. - -`kubelet` supports only two filesystem partitions. - -1. The `nodefs` filesystem that kubelet uses for volumes, daemon logs, etc. -1. The `imagefs` filesystem that container runtimes uses for storing images and container writable layers. - -`imagefs` is optional. `kubelet` auto-discovers these filesystems using cAdvisor. `kubelet` does not care about any -other filesystems. Any other types of configurations are not currently supported by the kubelet. For example, it is -*not OK* to store volumes and logs in a dedicated `filesystem`. - -In future releases, the `kubelet` will deprecate the existing [garbage collection](/docs/admin/garbage-collection/) -support in favor of eviction in response to disk pressure. - -### Eviction Thresholds - -The `kubelet` supports the ability to specify eviction thresholds that trigger the `kubelet` to reclaim resources. - -Each threshold is of the following form: - -`` - -* valid `eviction-signal` tokens as defined above. -* valid `operator` tokens are `<` -* valid `quantity` tokens must match the quantity representation used by Kubernetes -* an eviction threshold can be expressed as a percentage if ends with `%` token. - -For example, if a node has `10Gi` of memory, and the desire is to induce eviction -if available memory falls below `1Gi`, an eviction threshold can be specified as either -of the following (but not both). - -* `memory.available<10%` -* `memory.available<1Gi` - -#### Soft Eviction Thresholds - -A soft eviction threshold pairs an eviction threshold with a required -administrator specified grace period. No action is taken by the `kubelet` -to reclaim resources associated with the eviction signal until that grace -period has been exceeded. If no grace period is provided, the `kubelet` will -error on startup. - -In addition, if a soft eviction threshold has been met, an operator can -specify a maximum allowed pod termination grace period to use when evicting -pods from the node. If specified, the `kubelet` will use the lesser value among -the `pod.Spec.TerminationGracePeriodSeconds` and the max allowed grace period. -If not specified, the `kubelet` will kill pods immediately with no graceful -termination. - -To configure soft eviction thresholds, the following flags are supported: - -* `eviction-soft` describes a set of eviction thresholds (e.g. `memory.available<1.5Gi`) that if met over a -corresponding grace period would trigger a pod eviction. -* `eviction-soft-grace-period` describes a set of eviction grace periods (e.g. `memory.available=1m30s`) that -correspond to how long a soft eviction threshold must hold before triggering a pod eviction. -* `eviction-max-pod-grace-period` describes the maximum allowed grace period (in seconds) to use when terminating -pods in response to a soft eviction threshold being met. - -#### Hard Eviction Thresholds - -A hard eviction threshold has no grace period, and if observed, the `kubelet` -will take immediate action to reclaim the associated starved resource. If a -hard eviction threshold is met, the `kubelet` will kill the pod immediately -with no graceful termination. - -To configure hard eviction thresholds, the following flag is supported: - -* `eviction-hard` describes a set of eviction thresholds (e.g. `memory.available<1Gi`) that if met -would trigger a pod eviction. - -The `kubelet` has the following default hard eviction thresholds: - -* `--eviction-hard=memory.available<100Mi` - -### Eviction Monitoring Interval - -The `kubelet` evaluates eviction thresholds per its configured housekeeping interval. - -* `housekeeping-interval` is the interval between container housekeepings. - -### Node Conditions - -The `kubelet` will map one or more eviction signals to a corresponding node condition. - -If a hard eviction threshold has been met, or a soft eviction threshold has been met -independent of its associated grace period, the `kubelet` will report a condition that -reflects the node is under pressure. - -The following node conditions are defined that correspond to the specified eviction signal. - -| Node Condition | Eviction Signal | Description | -|-------------------------|-------------------------------|--------------------------------------------| -| `MemoryPressure` | `memory.available` | Available memory on the node has satisfied an eviction threshold | -| `DiskPressure` | `nodefs.available`, `nodefs.inodesFree`, `imagefs.available`, or `imagefs.inodesFree` | Available disk space and inodes on either the node's root filesytem or image filesystem has satisfied an eviction threshold | - -The `kubelet` will continue to report node status updates at the frequency specified by -`--node-status-update-frequency` which defaults to `10s`. - -### Oscillation of node conditions - -If a node is oscillating above and below a soft eviction threshold, but not exceeding -its associated grace period, it would cause the corresponding node condition to -constantly oscillate between true and false, and could cause poor scheduling decisions -as a consequence. - -To protect against this oscillation, the following flag is defined to control how -long the `kubelet` must wait before transitioning out of a pressure condition. - -* `eviction-pressure-transition-period` is the duration for which the `kubelet` has -to wait before transitioning out of an eviction pressure condition. - -The `kubelet` would ensure that it has not observed an eviction threshold being met -for the specified pressure condition for the period specified before toggling the -condition back to `false`. - -### Reclaiming node level resources - -If an eviction threshold has been met and the grace period has passed, -the `kubelet` will initiate the process of reclaiming the pressured resource -until it has observed the signal has gone below its defined threshold. - -The `kubelet` attempts to reclaim node level resources prior to evicting end-user pods. If -disk pressure is observed, the `kubelet` reclaims node level resources differently if the -machine has a dedicated `imagefs` configured for the container runtime. - -#### With Imagefs - -If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: - -1. Delete dead pods/containers - -If `imagefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: - -1. Delete all unused images - -#### Without Imagefs - -If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: - -1. Delete dead pods/containers -1. Delete all unused images - -### Evicting end-user pods - -If the `kubelet` is unable to reclaim sufficient resource on the node, -it will begin evicting pods. - -The `kubelet` ranks pods for eviction as follows: - -* by their quality of service -* by the consumption of the starved compute resource relative to the pods scheduling request. - -As a result, pod eviction occurs in the following order: - -* `BestEffort` pods that consume the most of the starved resource are failed -first. -* `Burstable` pods that consume the greatest amount of the starved resource -relative to their request for that resource are killed first. If no pod -has exceeded its request, the strategy targets the largest consumer of the -starved resource. -* `Guaranteed` pods that consume the greatest amount of the starved resource -relative to their request are killed first. If no pod has exceeded its request, -the strategy targets the largest consumer of the starved resource. - -A `Guaranteed` pod is guaranteed to never be evicted because of another pod's -resource consumption. If a system daemon (i.e. `kubelet`, `docker`, `journald`, etc.) -is consuming more resources than were reserved via `system-reserved` or `kube-reserved` allocations, -and the node only has `Guaranteed` pod(s) remaining, then the node must choose to evict a -`Guaranteed` pod in order to preserve node stability, and to limit the impact -of the unexpected consumption to other `Guaranteed` pod(s). - -Local disk is a `BestEffort` resource. If necessary, `kubelet` will evict pods one at a time to reclaim -disk when `DiskPressure` is encountered. The `kubelet` will rank pods by quality of service. If the `kubelet` -is responding to `inode` starvation, it will reclaim `inodes` by evicting pods with the lowest quality of service -first. If the `kubelet` is responding to lack of available disk, it will rank pods within a quality of service -that consumes the largest amount of disk and kill those first. - -#### With Imagefs - -If `nodefs` is triggering evictions, `kubelet` will sort pods based on the usage on `nodefs` -- local volumes + logs of all its containers. - -If `imagefs` is triggering evictions, `kubelet` will sort pods based on the writable layer usage of all its containers. - -#### Without Imagefs - -If `nodefs` is triggering evictions, `kubelet` will sort pods based on their total disk usage -- local volumes + logs & writable layer of all its containers. - -### Minimum eviction reclaim - -In certain scenarios, eviction of pods could result in reclamation of small amount of resources. This can result in -`kubelet` hitting eviction thresholds in repeated successions. In addition to that, eviction of resources like `disk`, - is time consuming. - -To mitigate these issues, `kubelet` can have a per-resource `minimum-reclaim`. Whenever `kubelet` observes -resource pressure, `kubelet` will attempt to reclaim at least `minimum-reclaim` amount of resource below -the configured eviction threshold. - -For example, with the following configuration: - -``` ---eviction-hard=memory.available<500Mi,nodefs.available<1Gi,imagefs.available<100Gi ---eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"` -``` - -If an eviction threshold is triggered for `memory.available`, the `kubelet` will work to ensure -that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` will work -to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it will -work to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure -on their associated resources. - -The default `eviction-minimum-reclaim` is `0` for all resources. - -### Scheduler - -The node will report a condition when a compute resource is under pressure. The -scheduler views that condition as a signal to dissuade placing additional -pods on the node. - -| Node Condition | Scheduler Behavior | -| ---------------- | ------------------------------------------------ | -| `MemoryPressure` | No new `BestEffort` pods are scheduled to the node. | -| `DiskPressure` | No new pods are scheduled to the node. | - -## Node OOM Behavior - -If the node experiences a system OOM (out of memory) event prior to the `kubelet` is able to reclaim memory, -the node depends on the [oom_killer](https://lwn.net/Articles/391222/) to respond. - -The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the pod. - -| Quality of Service | oom_score_adj | -|----------------------------|-----------------------------------------------------------------------| -| `Guaranteed` | -998 | -| `BestEffort` | 1000 | -| `Burstable` | min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999) | - -If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` will calculate -an `oom_score` based on the percentage of memory its using on the node, and then add the `oom_score_adj` to get an -effective `oom_score` for the container, and then kills the container with the highest score. - -The intended behavior should be that containers with the lowest quality of service that -are consuming the largest amount of memory relative to the scheduling request should be killed first in order -to reclaim memory. - -Unlike pod eviction, if a pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`. - -## Best Practices - -### Schedulable resources and eviction policies - -Let's imagine the following scenario: - -* Node memory capacity: `10Gi` -* Operator wants to reserve 10% of memory capacity for system daemons (kernel, `kubelet`, etc.) -* Operator wants to evict pods at 95% memory utilization to reduce thrashing and incidence of system OOM. - -To facilitate this scenario, the `kubelet` would be launched as follows: - -``` ---eviction-hard=memory.available<500Mi ---system-reserved=memory=1.5Gi -``` - -Implicit in this configuration is the understanding that "System reserved" should include the amount of memory -covered by the eviction threshold. - -To reach that capacity, either some pod is using more than its request, or the system is using more than `500Mi`. - -This configuration will ensure that the scheduler does not place pods on a node that immediately induce memory pressure -and trigger eviction assuming those pods use less than their configured request. - -### DaemonSet - -It is never desired for a `kubelet` to evict a pod that was derived from -a `DaemonSet` since the pod will immediately be recreated and rescheduled -back to the same node. - -At the moment, the `kubelet` has no ability to distinguish a pod created -from `DaemonSet` versus any other object. If/when that information is -available, the `kubelet` could pro-actively filter those pods from the -candidate set of pods provided to the eviction strategy. - -In general, it is strongly recommended that `DaemonSet` not -create `BestEffort` pods to avoid being identified as a candidate pod -for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` pods. - -## Deprecation of existing feature flags to reclaim disk - -`kubelet` has been freeing up disk space on demand to keep the node stable. - -As disk based eviction matures, the following `kubelet` flags will be marked for deprecation -in favor of the simpler configuration supported around eviction. - -| Existing Flag | New Flag | -| ------------- | -------- | -| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` | -| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | -| `--maximum-dead-containers` | deprecated | -| `--maximum-dead-containers-per-container` | deprecated | -| `--minimum-container-ttl-duration` | deprecated | -| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | -| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | - -## Known issues - -### kubelet may not observe memory pressure right away - -The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage -increases within that window rapidly, the `kubelet` may not observe `MemoryPressure` fast enough, and the `OOMKiller` -will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this -latency, and instead have the kernel tell us when a threshold has been crossed immediately. - -If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for -this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature -to prevent system OOMs, and promote eviction of workloads so cluster state can rebalance. - -### kubelet may evict more pods than needed - -The pod eviction may evict more pods than needed due to stats collection timing gap. This can be mitigated by adding -the ability to get root container stats on an on-demand basis (https://github.com/google/cadvisor/issues/1247) in the future. - -### How kubelet ranks pods for eviction in response to inode exhaustion - -At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes -inode exhaustion, it will evict pods by ranking them by quality of service. The following issue has been opened in cadvisor -to track per container inode consumption (https://github.com/google/cadvisor/issues/1422) which would allow us to rank pods -by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict -that pod over others. +[Configuring Out of Resource Handling](/docs/concepts/cluster-administration/out-of-resource/) diff --git a/docs/concepts/cluster-administration/out-of-resource.md b/docs/concepts/cluster-administration/out-of-resource.md new file mode 100644 index 0000000000..76e7125e7e --- /dev/null +++ b/docs/concepts/cluster-administration/out-of-resource.md @@ -0,0 +1,368 @@ +--- +assignees: +- derekwaynecarr +- vishh +- timstclair +title: Configuring Out Of Resource Handling +--- + +* TOC +{:toc} + +The `kubelet` needs to preserve node stability when available compute resources are low. + +This is especially important when dealing with incompressible resources such as memory or disk. + +If either resource is exhausted, the node would become unstable. + +## Eviction Policy + +The `kubelet` can pro-actively monitor for and prevent against total starvation of a compute resource. In those cases, the `kubelet` can pro-actively fail one or more pods in order to reclaim +the starved resource. When the `kubelet` fails a pod, it terminates all containers in the pod, and the `PodPhase` +is transitioned to `Failed`. + +### Eviction Signals + +The `kubelet` can support the ability to trigger eviction decisions on the signals described in the +table below. The value of each signal is described in the description column based on the `kubelet` +summary API. + +| Eviction Signal | Description | +|----------------------------|-----------------------------------------------------------------------| +| `memory.available` | `memory.available` := `node.status.capacity[memory]` - `node.stats.memory.workingSet` | +| `nodefs.available` | `nodefs.available` := `node.stats.fs.available` | +| `nodefs.inodesFree` | `nodefs.inodesFree` := `node.stats.fs.inodesFree` | +| `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available` | +| `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree` | + +Each of the above signals supports either a literal or percentage based value. The percentage based value +is calculated relative to the total capacity associated with each signal. + +`kubelet` supports only two filesystem partitions. + +1. The `nodefs` filesystem that kubelet uses for volumes, daemon logs, etc. +1. The `imagefs` filesystem that container runtimes uses for storing images and container writable layers. + +`imagefs` is optional. `kubelet` auto-discovers these filesystems using cAdvisor. `kubelet` does not care about any +other filesystems. Any other types of configurations are not currently supported by the kubelet. For example, it is +*not OK* to store volumes and logs in a dedicated `filesystem`. + +In future releases, the `kubelet` will deprecate the existing [garbage collection](/docs/admin/garbage-collection/) +support in favor of eviction in response to disk pressure. + +### Eviction Thresholds + +The `kubelet` supports the ability to specify eviction thresholds that trigger the `kubelet` to reclaim resources. + +Each threshold is of the following form: + +`` + +* valid `eviction-signal` tokens as defined above. +* valid `operator` tokens are `<` +* valid `quantity` tokens must match the quantity representation used by Kubernetes +* an eviction threshold can be expressed as a percentage if ends with `%` token. + +For example, if a node has `10Gi` of memory, and the desire is to induce eviction +if available memory falls below `1Gi`, an eviction threshold can be specified as either +of the following (but not both). + +* `memory.available<10%` +* `memory.available<1Gi` + +#### Soft Eviction Thresholds + +A soft eviction threshold pairs an eviction threshold with a required +administrator specified grace period. No action is taken by the `kubelet` +to reclaim resources associated with the eviction signal until that grace +period has been exceeded. If no grace period is provided, the `kubelet` will +error on startup. + +In addition, if a soft eviction threshold has been met, an operator can +specify a maximum allowed pod termination grace period to use when evicting +pods from the node. If specified, the `kubelet` will use the lesser value among +the `pod.Spec.TerminationGracePeriodSeconds` and the max allowed grace period. +If not specified, the `kubelet` will kill pods immediately with no graceful +termination. + +To configure soft eviction thresholds, the following flags are supported: + +* `eviction-soft` describes a set of eviction thresholds (e.g. `memory.available<1.5Gi`) that if met over a +corresponding grace period would trigger a pod eviction. +* `eviction-soft-grace-period` describes a set of eviction grace periods (e.g. `memory.available=1m30s`) that +correspond to how long a soft eviction threshold must hold before triggering a pod eviction. +* `eviction-max-pod-grace-period` describes the maximum allowed grace period (in seconds) to use when terminating +pods in response to a soft eviction threshold being met. + +#### Hard Eviction Thresholds + +A hard eviction threshold has no grace period, and if observed, the `kubelet` +will take immediate action to reclaim the associated starved resource. If a +hard eviction threshold is met, the `kubelet` will kill the pod immediately +with no graceful termination. + +To configure hard eviction thresholds, the following flag is supported: + +* `eviction-hard` describes a set of eviction thresholds (e.g. `memory.available<1Gi`) that if met +would trigger a pod eviction. + +The `kubelet` has the following default hard eviction thresholds: + +* `--eviction-hard=memory.available<100Mi` + +### Eviction Monitoring Interval + +The `kubelet` evaluates eviction thresholds per its configured housekeeping interval. + +* `housekeeping-interval` is the interval between container housekeepings. + +### Node Conditions + +The `kubelet` will map one or more eviction signals to a corresponding node condition. + +If a hard eviction threshold has been met, or a soft eviction threshold has been met +independent of its associated grace period, the `kubelet` will report a condition that +reflects the node is under pressure. + +The following node conditions are defined that correspond to the specified eviction signal. + +| Node Condition | Eviction Signal | Description | +|-------------------------|-------------------------------|--------------------------------------------| +| `MemoryPressure` | `memory.available` | Available memory on the node has satisfied an eviction threshold | +| `DiskPressure` | `nodefs.available`, `nodefs.inodesFree`, `imagefs.available`, or `imagefs.inodesFree` | Available disk space and inodes on either the node's root filesytem or image filesystem has satisfied an eviction threshold | + +The `kubelet` will continue to report node status updates at the frequency specified by +`--node-status-update-frequency` which defaults to `10s`. + +### Oscillation of node conditions + +If a node is oscillating above and below a soft eviction threshold, but not exceeding +its associated grace period, it would cause the corresponding node condition to +constantly oscillate between true and false, and could cause poor scheduling decisions +as a consequence. + +To protect against this oscillation, the following flag is defined to control how +long the `kubelet` must wait before transitioning out of a pressure condition. + +* `eviction-pressure-transition-period` is the duration for which the `kubelet` has +to wait before transitioning out of an eviction pressure condition. + +The `kubelet` would ensure that it has not observed an eviction threshold being met +for the specified pressure condition for the period specified before toggling the +condition back to `false`. + +### Reclaiming node level resources + +If an eviction threshold has been met and the grace period has passed, +the `kubelet` will initiate the process of reclaiming the pressured resource +until it has observed the signal has gone below its defined threshold. + +The `kubelet` attempts to reclaim node level resources prior to evicting end-user pods. If +disk pressure is observed, the `kubelet` reclaims node level resources differently if the +machine has a dedicated `imagefs` configured for the container runtime. + +#### With Imagefs + +If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: + +1. Delete dead pods/containers + +If `imagefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: + +1. Delete all unused images + +#### Without Imagefs + +If `nodefs` filesystem has met eviction thresholds, `kubelet` will free up disk space in the following order: + +1. Delete dead pods/containers +1. Delete all unused images + +### Evicting end-user pods + +If the `kubelet` is unable to reclaim sufficient resource on the node, +it will begin evicting pods. + +The `kubelet` ranks pods for eviction as follows: + +* by their quality of service +* by the consumption of the starved compute resource relative to the pods scheduling request. + +As a result, pod eviction occurs in the following order: + +* `BestEffort` pods that consume the most of the starved resource are failed +first. +* `Burstable` pods that consume the greatest amount of the starved resource +relative to their request for that resource are killed first. If no pod +has exceeded its request, the strategy targets the largest consumer of the +starved resource. +* `Guaranteed` pods that consume the greatest amount of the starved resource +relative to their request are killed first. If no pod has exceeded its request, +the strategy targets the largest consumer of the starved resource. + +A `Guaranteed` pod is guaranteed to never be evicted because of another pod's +resource consumption. If a system daemon (i.e. `kubelet`, `docker`, `journald`, etc.) +is consuming more resources than were reserved via `system-reserved` or `kube-reserved` allocations, +and the node only has `Guaranteed` pod(s) remaining, then the node must choose to evict a +`Guaranteed` pod in order to preserve node stability, and to limit the impact +of the unexpected consumption to other `Guaranteed` pod(s). + +Local disk is a `BestEffort` resource. If necessary, `kubelet` will evict pods one at a time to reclaim +disk when `DiskPressure` is encountered. The `kubelet` will rank pods by quality of service. If the `kubelet` +is responding to `inode` starvation, it will reclaim `inodes` by evicting pods with the lowest quality of service +first. If the `kubelet` is responding to lack of available disk, it will rank pods within a quality of service +that consumes the largest amount of disk and kill those first. + +#### With Imagefs + +If `nodefs` is triggering evictions, `kubelet` will sort pods based on the usage on `nodefs` +- local volumes + logs of all its containers. + +If `imagefs` is triggering evictions, `kubelet` will sort pods based on the writable layer usage of all its containers. + +#### Without Imagefs + +If `nodefs` is triggering evictions, `kubelet` will sort pods based on their total disk usage +- local volumes + logs & writable layer of all its containers. + +### Minimum eviction reclaim + +In certain scenarios, eviction of pods could result in reclamation of small amount of resources. This can result in +`kubelet` hitting eviction thresholds in repeated successions. In addition to that, eviction of resources like `disk`, + is time consuming. + +To mitigate these issues, `kubelet` can have a per-resource `minimum-reclaim`. Whenever `kubelet` observes +resource pressure, `kubelet` will attempt to reclaim at least `minimum-reclaim` amount of resource below +the configured eviction threshold. + +For example, with the following configuration: + +``` +--eviction-hard=memory.available<500Mi,nodefs.available<1Gi,imagefs.available<100Gi +--eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"` +``` + +If an eviction threshold is triggered for `memory.available`, the `kubelet` will work to ensure +that `memory.available` is at least `500Mi`. For `nodefs.available`, the `kubelet` will work +to ensure that `nodefs.available` is at least `1.5Gi`, and for `imagefs.available` it will +work to ensure that `imagefs.available` is at least `102Gi` before no longer reporting pressure +on their associated resources. + +The default `eviction-minimum-reclaim` is `0` for all resources. + +### Scheduler + +The node will report a condition when a compute resource is under pressure. The +scheduler views that condition as a signal to dissuade placing additional +pods on the node. + +| Node Condition | Scheduler Behavior | +| ---------------- | ------------------------------------------------ | +| `MemoryPressure` | No new `BestEffort` pods are scheduled to the node. | +| `DiskPressure` | No new pods are scheduled to the node. | + +## Node OOM Behavior + +If the node experiences a system OOM (out of memory) event prior to the `kubelet` is able to reclaim memory, +the node depends on the [oom_killer](https://lwn.net/Articles/391222/) to respond. + +The `kubelet` sets a `oom_score_adj` value for each container based on the quality of service for the pod. + +| Quality of Service | oom_score_adj | +|----------------------------|-----------------------------------------------------------------------| +| `Guaranteed` | -998 | +| `BestEffort` | 1000 | +| `Burstable` | min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999) | + +If the `kubelet` is unable to reclaim memory prior to a node experiencing system OOM, the `oom_killer` will calculate +an `oom_score` based on the percentage of memory its using on the node, and then add the `oom_score_adj` to get an +effective `oom_score` for the container, and then kills the container with the highest score. + +The intended behavior should be that containers with the lowest quality of service that +are consuming the largest amount of memory relative to the scheduling request should be killed first in order +to reclaim memory. + +Unlike pod eviction, if a pod container is OOM killed, it may be restarted by the `kubelet` based on its `RestartPolicy`. + +## Best Practices + +### Schedulable resources and eviction policies + +Let's imagine the following scenario: + +* Node memory capacity: `10Gi` +* Operator wants to reserve 10% of memory capacity for system daemons (kernel, `kubelet`, etc.) +* Operator wants to evict pods at 95% memory utilization to reduce thrashing and incidence of system OOM. + +To facilitate this scenario, the `kubelet` would be launched as follows: + +``` +--eviction-hard=memory.available<500Mi +--system-reserved=memory=1.5Gi +``` + +Implicit in this configuration is the understanding that "System reserved" should include the amount of memory +covered by the eviction threshold. + +To reach that capacity, either some pod is using more than its request, or the system is using more than `500Mi`. + +This configuration will ensure that the scheduler does not place pods on a node that immediately induce memory pressure +and trigger eviction assuming those pods use less than their configured request. + +### DaemonSet + +It is never desired for a `kubelet` to evict a pod that was derived from +a `DaemonSet` since the pod will immediately be recreated and rescheduled +back to the same node. + +At the moment, the `kubelet` has no ability to distinguish a pod created +from `DaemonSet` versus any other object. If/when that information is +available, the `kubelet` could pro-actively filter those pods from the +candidate set of pods provided to the eviction strategy. + +In general, it is strongly recommended that `DaemonSet` not +create `BestEffort` pods to avoid being identified as a candidate pod +for eviction. Instead `DaemonSet` should ideally launch `Guaranteed` pods. + +## Deprecation of existing feature flags to reclaim disk + +`kubelet` has been freeing up disk space on demand to keep the node stable. + +As disk based eviction matures, the following `kubelet` flags will be marked for deprecation +in favor of the simpler configuration supported around eviction. + +| Existing Flag | New Flag | +| ------------- | -------- | +| `--image-gc-high-threshold` | `--eviction-hard` or `eviction-soft` | +| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | +| `--maximum-dead-containers` | deprecated | +| `--maximum-dead-containers-per-container` | deprecated | +| `--minimum-container-ttl-duration` | deprecated | +| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | +| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | + +## Known issues + +### kubelet may not observe memory pressure right away + +The `kubelet` currently polls `cAdvisor` to collect memory usage stats at a regular interval. If memory usage +increases within that window rapidly, the `kubelet` may not observe `MemoryPressure` fast enough, and the `OOMKiller` +will still be invoked. We intend to integrate with the `memcg` notification API in a future release to reduce this +latency, and instead have the kernel tell us when a threshold has been crossed immediately. + +If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for +this issue is to set eviction thresholds at approximately 75% capacity. This increases the ability of this feature +to prevent system OOMs, and promote eviction of workloads so cluster state can rebalance. + +### kubelet may evict more pods than needed + +The pod eviction may evict more pods than needed due to stats collection timing gap. This can be mitigated by adding +the ability to get root container stats on an on-demand basis (https://github.com/google/cadvisor/issues/1247) in the future. + +### How kubelet ranks pods for eviction in response to inode exhaustion + +At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes +inode exhaustion, it will evict pods by ranking them by quality of service. The following issue has been opened in cadvisor +to track per container inode consumption (https://github.com/google/cadvisor/issues/1422) which would allow us to rank pods +by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict +that pod over others. From c2d59f0c83fe7fe322236be6c5047e9cce40795e Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 17:25:29 -0700 Subject: [PATCH 381/407] Move Guide topic: Monitoring Node Health. (#2822) * Move Guide topic: Monitoring Node Health. * Move to Tasks. --- _data/tasks.yml | 1 + docs/admin/node-problem.md | 242 +---------------- .../monitor-node-health.md | 248 ++++++++++++++++++ 3 files changed, 251 insertions(+), 240 deletions(-) create mode 100644 docs/tasks/debug-application-cluster/monitor-node-health.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 121090ea79..fa0f64927a 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -34,6 +34,7 @@ toc: - docs/tasks/debug-application-cluster/determine-reason-pod-failure.md - docs/tasks/debug-application-cluster/debug-init-containers.md - docs/tasks/debug-application-cluster/logging-stackdriver.md + - docs/tasks/debug-application-cluster/monitor-node-health.md - docs/tasks/debug-application-cluster/logging-elasticsearch-kibana.md - title: Accessing the Kubernetes API diff --git a/docs/admin/node-problem.md b/docs/admin/node-problem.md index 08ca3208dd..dd882f7e5e 100644 --- a/docs/admin/node-problem.md +++ b/docs/admin/node-problem.md @@ -5,244 +5,6 @@ assignees: title: Monitoring Node Health --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Node Problem Detector - -*Node problem detector* is a [DaemonSet](/docs/admin/daemons/) monitoring the -node health. It collects node problems from various daemons and reports them -to the apiserver as [NodeCondition](/docs/admin/node/#node-condition) and -[Event](/docs/api-reference/v1/definitions/#_v1_event). - -It supports some known kernel issue detection now, and will detect more and -more node problems over time. - -Currently Kubernetes won't take any action on the node conditions and events -generated by node problem detector. In the future, a remedy system could be -introduced to deal with node problems. - -See more information -[here](https://github.com/kubernetes/node-problem-detector). - -## Limitations - -* The kernel issue detection of node problem detector only supports file based -kernel log now. It doesn't support log tools like journald. - -* The kernel issue detection of node problem detector has assumption on kernel -log format, and now it only works on Ubuntu and Debian. However, it is easy to extend -it to [support other log format](/docs/admin/node-problem/#support-other-log-format). - -## Enable/Disable in GCE cluster - -Node problem detector is [running as a cluster addon](cluster-large.md/#addon-resources) enabled by default in the -gce cluster. - -You can enable/disable it by setting the environment variable -`KUBE_ENABLE_NODE_PROBLEM_DETECTOR` before `kube-up.sh`. - -## Use in Other Environment - -To enable node problem detector in other environment outside of GCE, you can use -either `kubectl` or addon pod. - -### Kubectl - -This is the recommended way to start node problem detector outside of GCE. It -provides more flexible management, such as overwriting the default -configuration to fit it into your environment or detect -customized node problems. - -* **Step 1:** Create `node-problem-detector.yaml`: - -```yaml -apiVersion: extensions/v1beta1 -kind: DaemonSet -metadata: - name: node-problem-detector-v0.1 - namespace: kube-system - labels: - k8s-app: node-problem-detector - version: v0.1 - kubernetes.io/cluster-service: "true" -spec: - template: - metadata: - labels: - k8s-app: node-problem-detector - version: v0.1 - kubernetes.io/cluster-service: "true" - spec: - hostNetwork: true - containers: - - name: node-problem-detector - image: gcr.io/google_containers/node-problem-detector:v0.1 - securityContext: - privileged: true - resources: - limits: - cpu: "200m" - memory: "100Mi" - requests: - cpu: "20m" - memory: "20Mi" - volumeMounts: - - name: log - mountPath: /log - readOnly: true - volumes: - - name: log - hostPath: - path: /var/log/ -``` - -***Notice that you should make sure the system log directory is right for your -OS distro.*** - -* **Step 2:** Start node problem detector with `kubectl`: - -```shell -kubectl create -f node-problem-detector.yaml -``` - -### Addon Pod - -This is for those who have their own cluster bootstrap solution, and don't need -to overwrite the default configuration. They could leverage the addon pod to -further automate the deployment. - -Just create `node-problem-detector.yaml`, and put it under the addon pods directory -`/etc/kubernetes/addons/node-problem-detector` on master node. - -## Overwrite the Configuration - -The [default configuration](https://github.com/kubernetes/node-problem-detector/tree/v0.1/config) -is embedded when building the docker image of node problem detector. - -However, you can use [ConfigMap](/docs/user-guide/configmap/) to overwrite it -following the steps: - -* **Step 1:** Change the config files in `config/`. -* **Step 2:** Create the ConfigMap `node-problem-detector-config` with `kubectl create configmap -node-problem-detector-config --from-file=config/`. -* **Step 3:** Change the `node-problem-detector.yaml` to use the ConfigMap: - -```yaml -apiVersion: extensions/v1beta1 -kind: DaemonSet -metadata: - name: node-problem-detector-v0.1 - namespace: kube-system - labels: - k8s-app: node-problem-detector - version: v0.1 - kubernetes.io/cluster-service: "true" -spec: - template: - metadata: - labels: - k8s-app: node-problem-detector - version: v0.1 - kubernetes.io/cluster-service: "true" - spec: - hostNetwork: true - containers: - - name: node-problem-detector - image: gcr.io/google_containers/node-problem-detector:v0.1 - securityContext: - privileged: true - resources: - limits: - cpu: "200m" - memory: "100Mi" - requests: - cpu: "20m" - memory: "20Mi" - volumeMounts: - - name: log - mountPath: /log - readOnly: true - - name: config # Overwrite the config/ directory with ConfigMap volume - mountPath: /config - readOnly: true - volumes: - - name: log - hostPath: - path: /var/log/ - - name: config # Define ConfigMap volume - configMap: - name: node-problem-detector-config -``` - -* **Step 4:** Re-create the node problem detector with the new yaml file: - -```shell -kubectl delete -f node-problem-detector.yaml # If you have a node-problem-detector running -kubectl create -f node-problem-detector.yaml -``` - -***Notice that this approach only applies to node problem detector started with `kubectl`.*** - -For node problem detector running as cluster addon, because addon manager doesn't support -ConfigMap, configuration overwriting is not supported now. - -## Kernel Monitor - -*Kernel Monitor* is a problem daemon in node problem detector. It monitors kernel log -and detects known kernel issues following predefined rules. - -The Kernel Monitor matches kernel issues according to a set of predefined rule list in -[`config/kernel-monitor.json`](https://github.com/kubernetes/node-problem-detector/blob/v0.1/config/kernel-monitor.json). -The rule list is extensible, and you can always extend it by [overwriting the -configuration](/docs/admin/node-problem/#overwrite-the-configuration). - -### Add New NodeConditions - -To support new node conditions, you can extend the `conditions` field in -`config/kernel-monitor.json` with new condition definition: - -```json -{ - "type": "NodeConditionType", - "reason": "CamelCaseDefaultNodeConditionReason", - "message": "arbitrary default node condition message" -} -``` - -### Detect New Problems - -To detect new problems, you can extend the `rules` field in `config/kernel-monitor.json` -with new rule definition: - -```json -{ - "type": "temporary/permanent", - "condition": "NodeConditionOfPermanentIssue", - "reason": "CamelCaseShortReason", - "message": "regexp matching the issue in the kernel log" -} -``` - -### Change Log Path - -Kernel log in different OS distros may locate in different path. The `log` -field in `config/kernel-monitor.json` is the log path inside the container. -You can always configure it to match your OS distro. - -### Support Other Log Format - -Kernel monitor uses [`Translator`](https://github.com/kubernetes/node-problem-detector/blob/v0.1/pkg/kernelmonitor/translator/translator.go) -plugin to translate kernel log the internal data structure. It is easy to -implement a new translator for a new log format. - -## Caveats - -It is recommended to run the node problem detector in your cluster to monitor -the node health. However, you should be aware that this will introduce extra -resource overhead on each node. Usually this is fine, because: - -* The kernel log is generated relatively slowly. -* Resource limit is set for node problem detector. -* Even under high load, the resource usage is acceptable. -(see [benchmark result](https://github.com/kubernetes/node-problem-detector/issues/2#issuecomment-220255629)) +[Monitoring Node Health](/docs/tasks/debug-application-cluster/monitor-node-health/) diff --git a/docs/tasks/debug-application-cluster/monitor-node-health.md b/docs/tasks/debug-application-cluster/monitor-node-health.md new file mode 100644 index 0000000000..08ca3208dd --- /dev/null +++ b/docs/tasks/debug-application-cluster/monitor-node-health.md @@ -0,0 +1,248 @@ +--- +assignees: +- Random-Liu +- dchen1107 +title: Monitoring Node Health +--- + +* TOC +{:toc} + +## Node Problem Detector + +*Node problem detector* is a [DaemonSet](/docs/admin/daemons/) monitoring the +node health. It collects node problems from various daemons and reports them +to the apiserver as [NodeCondition](/docs/admin/node/#node-condition) and +[Event](/docs/api-reference/v1/definitions/#_v1_event). + +It supports some known kernel issue detection now, and will detect more and +more node problems over time. + +Currently Kubernetes won't take any action on the node conditions and events +generated by node problem detector. In the future, a remedy system could be +introduced to deal with node problems. + +See more information +[here](https://github.com/kubernetes/node-problem-detector). + +## Limitations + +* The kernel issue detection of node problem detector only supports file based +kernel log now. It doesn't support log tools like journald. + +* The kernel issue detection of node problem detector has assumption on kernel +log format, and now it only works on Ubuntu and Debian. However, it is easy to extend +it to [support other log format](/docs/admin/node-problem/#support-other-log-format). + +## Enable/Disable in GCE cluster + +Node problem detector is [running as a cluster addon](cluster-large.md/#addon-resources) enabled by default in the +gce cluster. + +You can enable/disable it by setting the environment variable +`KUBE_ENABLE_NODE_PROBLEM_DETECTOR` before `kube-up.sh`. + +## Use in Other Environment + +To enable node problem detector in other environment outside of GCE, you can use +either `kubectl` or addon pod. + +### Kubectl + +This is the recommended way to start node problem detector outside of GCE. It +provides more flexible management, such as overwriting the default +configuration to fit it into your environment or detect +customized node problems. + +* **Step 1:** Create `node-problem-detector.yaml`: + +```yaml +apiVersion: extensions/v1beta1 +kind: DaemonSet +metadata: + name: node-problem-detector-v0.1 + namespace: kube-system + labels: + k8s-app: node-problem-detector + version: v0.1 + kubernetes.io/cluster-service: "true" +spec: + template: + metadata: + labels: + k8s-app: node-problem-detector + version: v0.1 + kubernetes.io/cluster-service: "true" + spec: + hostNetwork: true + containers: + - name: node-problem-detector + image: gcr.io/google_containers/node-problem-detector:v0.1 + securityContext: + privileged: true + resources: + limits: + cpu: "200m" + memory: "100Mi" + requests: + cpu: "20m" + memory: "20Mi" + volumeMounts: + - name: log + mountPath: /log + readOnly: true + volumes: + - name: log + hostPath: + path: /var/log/ +``` + +***Notice that you should make sure the system log directory is right for your +OS distro.*** + +* **Step 2:** Start node problem detector with `kubectl`: + +```shell +kubectl create -f node-problem-detector.yaml +``` + +### Addon Pod + +This is for those who have their own cluster bootstrap solution, and don't need +to overwrite the default configuration. They could leverage the addon pod to +further automate the deployment. + +Just create `node-problem-detector.yaml`, and put it under the addon pods directory +`/etc/kubernetes/addons/node-problem-detector` on master node. + +## Overwrite the Configuration + +The [default configuration](https://github.com/kubernetes/node-problem-detector/tree/v0.1/config) +is embedded when building the docker image of node problem detector. + +However, you can use [ConfigMap](/docs/user-guide/configmap/) to overwrite it +following the steps: + +* **Step 1:** Change the config files in `config/`. +* **Step 2:** Create the ConfigMap `node-problem-detector-config` with `kubectl create configmap +node-problem-detector-config --from-file=config/`. +* **Step 3:** Change the `node-problem-detector.yaml` to use the ConfigMap: + +```yaml +apiVersion: extensions/v1beta1 +kind: DaemonSet +metadata: + name: node-problem-detector-v0.1 + namespace: kube-system + labels: + k8s-app: node-problem-detector + version: v0.1 + kubernetes.io/cluster-service: "true" +spec: + template: + metadata: + labels: + k8s-app: node-problem-detector + version: v0.1 + kubernetes.io/cluster-service: "true" + spec: + hostNetwork: true + containers: + - name: node-problem-detector + image: gcr.io/google_containers/node-problem-detector:v0.1 + securityContext: + privileged: true + resources: + limits: + cpu: "200m" + memory: "100Mi" + requests: + cpu: "20m" + memory: "20Mi" + volumeMounts: + - name: log + mountPath: /log + readOnly: true + - name: config # Overwrite the config/ directory with ConfigMap volume + mountPath: /config + readOnly: true + volumes: + - name: log + hostPath: + path: /var/log/ + - name: config # Define ConfigMap volume + configMap: + name: node-problem-detector-config +``` + +* **Step 4:** Re-create the node problem detector with the new yaml file: + +```shell +kubectl delete -f node-problem-detector.yaml # If you have a node-problem-detector running +kubectl create -f node-problem-detector.yaml +``` + +***Notice that this approach only applies to node problem detector started with `kubectl`.*** + +For node problem detector running as cluster addon, because addon manager doesn't support +ConfigMap, configuration overwriting is not supported now. + +## Kernel Monitor + +*Kernel Monitor* is a problem daemon in node problem detector. It monitors kernel log +and detects known kernel issues following predefined rules. + +The Kernel Monitor matches kernel issues according to a set of predefined rule list in +[`config/kernel-monitor.json`](https://github.com/kubernetes/node-problem-detector/blob/v0.1/config/kernel-monitor.json). +The rule list is extensible, and you can always extend it by [overwriting the +configuration](/docs/admin/node-problem/#overwrite-the-configuration). + +### Add New NodeConditions + +To support new node conditions, you can extend the `conditions` field in +`config/kernel-monitor.json` with new condition definition: + +```json +{ + "type": "NodeConditionType", + "reason": "CamelCaseDefaultNodeConditionReason", + "message": "arbitrary default node condition message" +} +``` + +### Detect New Problems + +To detect new problems, you can extend the `rules` field in `config/kernel-monitor.json` +with new rule definition: + +```json +{ + "type": "temporary/permanent", + "condition": "NodeConditionOfPermanentIssue", + "reason": "CamelCaseShortReason", + "message": "regexp matching the issue in the kernel log" +} +``` + +### Change Log Path + +Kernel log in different OS distros may locate in different path. The `log` +field in `config/kernel-monitor.json` is the log path inside the container. +You can always configure it to match your OS distro. + +### Support Other Log Format + +Kernel monitor uses [`Translator`](https://github.com/kubernetes/node-problem-detector/blob/v0.1/pkg/kernelmonitor/translator/translator.go) +plugin to translate kernel log the internal data structure. It is easy to +implement a new translator for a new log format. + +## Caveats + +It is recommended to run the node problem detector in your cluster to monitor +the node health. However, you should be aware that this will introduce extra +resource overhead on each node. Usually this is fine, because: + +* The kernel log is generated relatively slowly. +* Resource limit is set for node problem detector. +* Even under high load, the resource usage is acceptable. +(see [benchmark result](https://github.com/kubernetes/node-problem-detector/issues/2#issuecomment-220255629)) From c96bb862d5a6be3513cd97719a0d86ee62775720 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 18:01:32 -0700 Subject: [PATCH 382/407] Move Guide topic: AppArmor. (#2823) --- _data/tutorials.yml | 3 + docs/admin/apparmor/index.md | 387 +-------------------------- docs/tutorials/clusters/apparmor.md | 392 ++++++++++++++++++++++++++++ 3 files changed, 397 insertions(+), 385 deletions(-) create mode 100644 docs/tutorials/clusters/apparmor.md diff --git a/_data/tutorials.yml b/_data/tutorials.yml index c4e8d5f799..8ae5f09a3b 100644 --- a/_data/tutorials.yml +++ b/_data/tutorials.yml @@ -52,6 +52,9 @@ toc: - title: Connecting Applications section: - docs/tutorials/connecting-apps/connecting-frontend-backend.md +- title: Clusters + section: + - docs/tutorials/clusters/apparmor.md - title: Services section: - docs/tutorials/services/source-ip.md diff --git a/docs/admin/apparmor/index.md b/docs/admin/apparmor/index.md index 224f0bbdeb..60835fd5bf 100644 --- a/docs/admin/apparmor/index.md +++ b/docs/admin/apparmor/index.md @@ -4,389 +4,6 @@ assignees: title: AppArmor --- -AppArmor is a Linux kernel enhancement that can reduce the potential attack surface of an -application and provide greater defense in depth for Applications. Beta support for AppArmor was -added in Kubernetes v1.4. +{% include user-guide-content-moved.md %} -* TOC -{:toc} - -## What is AppArmor - -AppArmor is a Linux kernel security module that supplements the standard Linux user and group based -permissions to confine programs to a limited set of resources. AppArmor can be configured for any -application to reduce its potential attack surface and provide greater defense in depth. It is -configured through profiles tuned to whitelist the access needed by a specific program or container, -such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either -enforcing mode, which blocks access to disallowed resources, or complain mode, which only reports -violations. - -AppArmor can help you to run a more secure deployment by restricting what containers are allowed to -do, and /or providing better auditing through system logs. However, it is important to keep in mind -that AppArmor is not a silver bullet, and can only do so much to protect against exploits in your -application code. It is important to provide good, restrictive profiles, and harden your -applications and cluster from other angles as well. - -AppArmor support in Kubernetes is currently in beta. - -## Prerequisites - -1. **Kubernetes version is at least v1.4**. Kubernetes support for AppArmor was added in - v1.4. Kubernetes components older than v1.4 are not aware of the new AppArmor annotations, and - will **silently ignore** any AppArmor settings that are provided. To ensure that your Pods are - receiving the expected protections, it is important to verify the Kubelet version of your nodes: - - $ kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.kubeletVersion}\n{end}' - gke-test-default-pool-239f5d02-gyn2: v1.4.0 - gke-test-default-pool-239f5d02-x1kf: v1.4.0 - gke-test-default-pool-239f5d02-xwux: v1.4.0 - -2. **AppArmor kernel module is enabled**. For the Linux kernel to enforce an AppArmor profile, the - AppArmor kernel module must be installed and enabled. Several distributions enable the module by - default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the - module is enabled, check the `/sys/module/apparmor/parameters/enabled` file: - - $ cat /sys/module/apparmor/parameters/enabled - Y - - If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor - options if the kernel module is not enabled. - - *Note: Ubuntu carries many AppArmor patches that have not been merged into the upstream Linux - kernel, including patches that add additional hooks and features. Kubernetes has only been - tested with the upstream version, and does not promise support for other features.* - -3. **Container runtime is Docker**. Currently the only Kubernetes-supported container runtime that - also supports AppArmor is Docker. As more runtimes add AppArmor support, the options will be - expanded. You can verify that your nodes are running docker with: - - $ kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.containerRuntimeVersion}\n{end}' - gke-test-default-pool-239f5d02-gyn2: docker://1.11.2 - gke-test-default-pool-239f5d02-x1kf: docker://1.11.2 - gke-test-default-pool-239f5d02-xwux: docker://1.11.2 - - If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor - options if the runtime is not Docker. - -4. **Profile is loaded**. AppArmor is applied to a Pod by specifying an AppArmor profile that each - container should be run with. If any of the specified profiles is not already loaded in the - kernel, the Kubelet (>= v1.4) will reject the Pod. You can view which profiles are loaded on a - node by checking the `/sys/kernel/security/apparmor/profiles` file. For example: - - $ ssh gke-test-default-pool-239f5d02-gyn2 "sudo cat /sys/kernel/security/apparmor/profiles | sort" - apparmor-test-deny-write (enforce) - apparmor-test-audit-write (enforce) - docker-default (enforce) - k8s-nginx (enforce) - - For more details on loading profiles on nodes, see - [Setting up nodes with profiles](#setting-up-nodes-with-profiles). - -As long as the Kubelet version includes AppArmor support (>= v1.4), the Kubelet will reject a Pod -with AppArmor options if any of the prerequisites are not met. You can also verify AppArmor support -on nodes by checking the node ready condition message (though this is likely to be removed in a -later release): - - $ kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}\n{end}' - gke-test-default-pool-239f5d02-gyn2: kubelet is posting ready status. AppArmor enabled - gke-test-default-pool-239f5d02-x1kf: kubelet is posting ready status. AppArmor enabled - gke-test-default-pool-239f5d02-xwux: kubelet is posting ready status. AppArmor enabled - -## Securing a Pod - -*Note: AppArmor is currently in beta, so options are specified as annotations. Once support graduates to -general availability, the annotations will be replaced with first-class fields (more details in -[Upgrade path to GA](#upgrade-path-to-general-availability)).* - -AppArmor profiles are specified *per-container*. To specify the AppArmor profile to run a Pod -container with, add an annotation to the Pod's metadata: - - container.apparmor.security.beta.kubernetes.io/: - -Where `` is the name of the container to apply the profile to, and `` -specifies the profile to apply. The `profile_ref` can be one of: - -- `runtime/default` to apply the runtime's default profile. -- `localhost/` to apply the profile loaded on the host with the name `` - -See the [API Reference](#api-reference) for the full details on the annotation and profile name formats. - -The Kubernetes AppArmor enforcement works by first checking that all the prerequisites have been -met, and then forwarding the profile selection to the container runtime for enforcement. If the -prerequisites have not been met, the Pod will be rejected, and will not run. - -To verify that the profile was applied, you can expect to see the AppArmor security option listed in the container created event: - - $ kubectl get events | grep Created - 22s 22s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet e2e-test-stclair-minion-group-31nt} Created container with docker id 269a53b202d3; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] - -You can also verify directly that the container's root process is running with the correct profile by checking its proc attr: - - $ kubectl exec cat /proc/1/attr/current - k8s-apparmor-example-deny-write (enforce) - -## Example - -In this example you'll see: - -- One way to load a profile on a node -- How to enforce the profile on a Pod -- How to check that the profile is loaded -- What happens when a profile is violated -- What happens when a profile cannot be loaded - -*This example assumes you have already set up a cluster with AppArmor support.* - -First, we need to load the profile we want to use onto our nodes. The profile we'll use simply -denies all file writes: - -{% include code.html language="text" file="deny-write.profile" ghlink="/docs/admin/apparmor/deny-write.profile" %} - -Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our -nodes. For this example we'll just use SSH to install the profiles, but other approaches are -discussed in [Setting up nodes with profiles](#setting-up-nodes-with-profiles). - - $ NODES=( - # The SSH-accessible domain names of your nodes - gke-test-default-pool-239f5d02-gyn2.us-central1-a.my-k8s - gke-test-default-pool-239f5d02-x1kf.us-central1-a.my-k8s - gke-test-default-pool-239f5d02-xwux.us-central1-a.my-k8s) - $ for NODE in ${NODES[*]}; do ssh $NODE 'sudo apparmor_parser -q < - - profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { - #include - - file, - - # Deny all file writes. - deny /** w, - } - EOF' - done - -Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: - -{% include code.html language="yaml" file="hello-apparmor-pod.yaml" ghlink="/docs/admin/apparmor/hello-apparmor-pod.yaml" %} - - $ kubectl create -f /dev/stdin < - Status: Failed - Reason: AppArmor - Message: Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded - IP: - Controllers: - Containers: - hello: - Image: busybox - Port: - Command: - sh - -c - echo 'Hello AppArmor!' && sleep 1h - Requests: - cpu: 100m - Environment Variables: - Volumes: - default-token-dnz7v: - Type: Secret (a volume populated by a Secret) - SecretName: default-token-dnz7v - QoS Tier: Burstable - Events: - FirstSeen LastSeen Count From SubobjectPath Type Reason Message - --------- -------- ----- ---- ------------- -------- ------ ------- - 23s 23s 1 {default-scheduler } Normal Scheduled Successfully assigned hello-apparmor-2 to e2e-test-stclair-minion-group-t1f5 - 23s 23s 1 {kubelet e2e-test-stclair-minion-group-t1f5} Warning AppArmor Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded - -Note the pod status is Failed, with a helpful error message: `Pod Cannot enforce AppArmor: profile -"k8s-apparmor-example-allow-write" is not loaded`. An event was also recorded with the same message. - -## Administration - -### Setting up nodes with profiles - -Kubernetes does not currently provide any native mechanisms for loading AppArmor profiles onto -nodes. There are lots of ways to setup the profiles though, such as: - -- Through a [DaemonSet](../daemons/) that runs a Pod on each node to - ensure the correct profiles are loaded. An example implementation can be found - [here](https://github.com/kubernetes/contrib/tree/master/apparmor/loader). -- At node initialization time, using your node initialization scripts (e.g. Salt, Ansible, etc.) or - image. -- By copying the profiles to each node and loading them through SSH, as demonstrated in the - [Example](#example). - -The scheduler is not aware of which profiles are loaded onto which node, so the full set of profiles -must be loaded onto every node. An alternative approach is to add a node label for each profile (or -class of profiles) on the node, and use a -[node selector](../../user-guide/node-selection/) to ensure the Pod is run on a -node with the required profile. - -### Restricting profiles with the PodSecurityPolicy - -If the PodSecurityPolicy extension is enabled, cluster-wide AppArmor restrictions can be applied. To -enable the PodSecurityPolicy, two flags must be set on the `apiserver`: - - --admission-control=PodSecurityPolicy[,others...] - --runtime-config=extensions/v1beta1/podsecuritypolicy[,others...] - -With the extension enabled, the AppArmor options can be specified as annotations on the PodSecurityPolicy: - - apparmor.security.beta.kubernetes.io/defaultProfileName: - apparmor.security.beta.kubernetes.io/allowedProfileNames: [,others...] - -The default profile name option specifies the profile to apply to containers by default when none is -specified. The allowed profile names option specifies a list of profiles that Pod containers are -allowed to be run with. If both options are provided, the default must be allowed. The profiles are -specified in the same format as on containers. See the [API Reference](#api-reference) for the full -specification. - -### Disabling AppArmor - -If you do not want AppArmor to be available on your cluster, it can be disabled by a command-line flag: - - --feature-gates=AppArmor=false - -When disabled, any Pod that includes an AppArmor profile will fail validation with a "Forbidden" -error. Note that by default docker always enables the "docker-default" profile on non-privileged -pods (if the AppArmor kernel module is enabled), and will continue to do so even if the feature-gate -is disabled. The option to disable AppArmor will be removed when AppArmor graduates to general -availability (GA). - -### Upgrading to Kubernetes v1.4 with AppArmor - -No action is required with respect to AppArmor to upgrade your cluster to v1.4. However, if any -existing pods had an AppArmor annotation, they will not go through validation (or PodSecurityPolicy -admission). If permissive profiles are loaded on the nodes, a malicious user could pre-apply a -permissive profile to escalate the pod privileges above the docker-default. If this is a concern, it -is recommended to scrub the cluster of any pods containing an annotation with -`apparmor.security.beta.kubernetes.io`. - -### Upgrade path to General Availability - -When AppArmor is ready to be graduated to general availability (GA), the options currently specified -through annotations will be converted to fields. Supporting all the upgrade and downgrade paths -through the transition is very nuanced, and will be explained in detail when the transition -occurs. We will commit to supporting both fields and annotations for at least 2 releases, and will -explicitly reject the annotations for at least 2 releases after that. - -## Authoring Profiles - -Getting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some -tools to help with that: - -- `aa-genprof` and `aa-logprof` generate profile rules by monitoring an application's activity and - logs, and admitting the actions it takes. Further instructions are provided by the - [AppArmor documentation](http://wiki.apparmor.net/index.php/Profiling_with_tools). -- [bane](https://github.com/jfrazelle/bane) is an AppArmor profile generator for Docker that uses a - simplified profile language. - -It is recommended to run your application through Docker on a development workstation to generate -the profiles, but there is nothing preventing running the tools on the Kubernetes node where your -Pod is running. - -To debug problems with AppArmor, you can check the system logs to see what, specifically, was -denied. AppArmor logs verbose messages to `dmesg`, and errors can usually be found in the system -logs or through `journalctl`. More information is provided in -[AppArmor failures](http://wiki.apparmor.net/index.php/AppArmor_Failures). - -Additional resources: - -- [Quick guide to the AppArmor profile language](http://wiki.apparmor.net/index.php/QuickProfileLanguage) -- [AppArmor core policy reference](http://wiki.apparmor.net/index.php/ProfileLanguage) - -## API Reference - -**Pod Annotation**: - -Specifying the profile a container will run with: - -- **key**: `container.apparmor.security.beta.kubernetes.io/` - Where `` matches the name of a container in the Pod. - A separate profile can be specified for each container in the Pod. -- **value**: a profile reference, described below - -**Profile Reference**: - -- `runtime/default`: Refers to the default runtime profile. - - Equivalent to not specifying a profile (without a PodSecurityPolicy default), except it still - requires AppArmor to be enabled. - - For Docker, this resolves to the - [`docker-default`](https://docs.docker.com/engine/security/apparmor/) profile for non-privileged - containers, and unconfined (no profile) for privileged containers. -- `localhost/`: Refers to a profile loaded on the node (localhost) by name. - - The possible profile names are detailed in the - [core policy reference](http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference#Profile_names_and_attachment_specifications) - -Any other profile reference format is invalid. - -**PodSecurityPolicy Annotations** - -Specifying the default profile to apply to containers when none is provided: - -- **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` -- **value**: a profile reference, described above - -Specifying the list of profiles Pod containers is allowed to specify: - -- **key**: `apparmor.security.beta.kubernetes.io/allowedProfileNames` -- **value**: a comma-separated list of profile references (described above) - - Although an escaped comma is a legal character in a profile name, it cannot be explicitly - allowed here +[AppArmor](/docs/tutorials/clusters/apparmor/) diff --git a/docs/tutorials/clusters/apparmor.md b/docs/tutorials/clusters/apparmor.md new file mode 100644 index 0000000000..224f0bbdeb --- /dev/null +++ b/docs/tutorials/clusters/apparmor.md @@ -0,0 +1,392 @@ +--- +assignees: +- stclair +title: AppArmor +--- + +AppArmor is a Linux kernel enhancement that can reduce the potential attack surface of an +application and provide greater defense in depth for Applications. Beta support for AppArmor was +added in Kubernetes v1.4. + +* TOC +{:toc} + +## What is AppArmor + +AppArmor is a Linux kernel security module that supplements the standard Linux user and group based +permissions to confine programs to a limited set of resources. AppArmor can be configured for any +application to reduce its potential attack surface and provide greater defense in depth. It is +configured through profiles tuned to whitelist the access needed by a specific program or container, +such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either +enforcing mode, which blocks access to disallowed resources, or complain mode, which only reports +violations. + +AppArmor can help you to run a more secure deployment by restricting what containers are allowed to +do, and /or providing better auditing through system logs. However, it is important to keep in mind +that AppArmor is not a silver bullet, and can only do so much to protect against exploits in your +application code. It is important to provide good, restrictive profiles, and harden your +applications and cluster from other angles as well. + +AppArmor support in Kubernetes is currently in beta. + +## Prerequisites + +1. **Kubernetes version is at least v1.4**. Kubernetes support for AppArmor was added in + v1.4. Kubernetes components older than v1.4 are not aware of the new AppArmor annotations, and + will **silently ignore** any AppArmor settings that are provided. To ensure that your Pods are + receiving the expected protections, it is important to verify the Kubelet version of your nodes: + + $ kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.kubeletVersion}\n{end}' + gke-test-default-pool-239f5d02-gyn2: v1.4.0 + gke-test-default-pool-239f5d02-x1kf: v1.4.0 + gke-test-default-pool-239f5d02-xwux: v1.4.0 + +2. **AppArmor kernel module is enabled**. For the Linux kernel to enforce an AppArmor profile, the + AppArmor kernel module must be installed and enabled. Several distributions enable the module by + default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the + module is enabled, check the `/sys/module/apparmor/parameters/enabled` file: + + $ cat /sys/module/apparmor/parameters/enabled + Y + + If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor + options if the kernel module is not enabled. + + *Note: Ubuntu carries many AppArmor patches that have not been merged into the upstream Linux + kernel, including patches that add additional hooks and features. Kubernetes has only been + tested with the upstream version, and does not promise support for other features.* + +3. **Container runtime is Docker**. Currently the only Kubernetes-supported container runtime that + also supports AppArmor is Docker. As more runtimes add AppArmor support, the options will be + expanded. You can verify that your nodes are running docker with: + + $ kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.containerRuntimeVersion}\n{end}' + gke-test-default-pool-239f5d02-gyn2: docker://1.11.2 + gke-test-default-pool-239f5d02-x1kf: docker://1.11.2 + gke-test-default-pool-239f5d02-xwux: docker://1.11.2 + + If the Kubelet contains AppArmor support (>= v1.4), it will refuse to run a Pod with AppArmor + options if the runtime is not Docker. + +4. **Profile is loaded**. AppArmor is applied to a Pod by specifying an AppArmor profile that each + container should be run with. If any of the specified profiles is not already loaded in the + kernel, the Kubelet (>= v1.4) will reject the Pod. You can view which profiles are loaded on a + node by checking the `/sys/kernel/security/apparmor/profiles` file. For example: + + $ ssh gke-test-default-pool-239f5d02-gyn2 "sudo cat /sys/kernel/security/apparmor/profiles | sort" + apparmor-test-deny-write (enforce) + apparmor-test-audit-write (enforce) + docker-default (enforce) + k8s-nginx (enforce) + + For more details on loading profiles on nodes, see + [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + +As long as the Kubelet version includes AppArmor support (>= v1.4), the Kubelet will reject a Pod +with AppArmor options if any of the prerequisites are not met. You can also verify AppArmor support +on nodes by checking the node ready condition message (though this is likely to be removed in a +later release): + + $ kubectl get nodes -o=jsonpath=$'{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}\n{end}' + gke-test-default-pool-239f5d02-gyn2: kubelet is posting ready status. AppArmor enabled + gke-test-default-pool-239f5d02-x1kf: kubelet is posting ready status. AppArmor enabled + gke-test-default-pool-239f5d02-xwux: kubelet is posting ready status. AppArmor enabled + +## Securing a Pod + +*Note: AppArmor is currently in beta, so options are specified as annotations. Once support graduates to +general availability, the annotations will be replaced with first-class fields (more details in +[Upgrade path to GA](#upgrade-path-to-general-availability)).* + +AppArmor profiles are specified *per-container*. To specify the AppArmor profile to run a Pod +container with, add an annotation to the Pod's metadata: + + container.apparmor.security.beta.kubernetes.io/: + +Where `` is the name of the container to apply the profile to, and `` +specifies the profile to apply. The `profile_ref` can be one of: + +- `runtime/default` to apply the runtime's default profile. +- `localhost/` to apply the profile loaded on the host with the name `` + +See the [API Reference](#api-reference) for the full details on the annotation and profile name formats. + +The Kubernetes AppArmor enforcement works by first checking that all the prerequisites have been +met, and then forwarding the profile selection to the container runtime for enforcement. If the +prerequisites have not been met, the Pod will be rejected, and will not run. + +To verify that the profile was applied, you can expect to see the AppArmor security option listed in the container created event: + + $ kubectl get events | grep Created + 22s 22s 1 hello-apparmor Pod spec.containers{hello} Normal Created {kubelet e2e-test-stclair-minion-group-31nt} Created container with docker id 269a53b202d3; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write] + +You can also verify directly that the container's root process is running with the correct profile by checking its proc attr: + + $ kubectl exec cat /proc/1/attr/current + k8s-apparmor-example-deny-write (enforce) + +## Example + +In this example you'll see: + +- One way to load a profile on a node +- How to enforce the profile on a Pod +- How to check that the profile is loaded +- What happens when a profile is violated +- What happens when a profile cannot be loaded + +*This example assumes you have already set up a cluster with AppArmor support.* + +First, we need to load the profile we want to use onto our nodes. The profile we'll use simply +denies all file writes: + +{% include code.html language="text" file="deny-write.profile" ghlink="/docs/admin/apparmor/deny-write.profile" %} + +Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our +nodes. For this example we'll just use SSH to install the profiles, but other approaches are +discussed in [Setting up nodes with profiles](#setting-up-nodes-with-profiles). + + $ NODES=( + # The SSH-accessible domain names of your nodes + gke-test-default-pool-239f5d02-gyn2.us-central1-a.my-k8s + gke-test-default-pool-239f5d02-x1kf.us-central1-a.my-k8s + gke-test-default-pool-239f5d02-xwux.us-central1-a.my-k8s) + $ for NODE in ${NODES[*]}; do ssh $NODE 'sudo apparmor_parser -q < + + profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { + #include + + file, + + # Deny all file writes. + deny /** w, + } + EOF' + done + +Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: + +{% include code.html language="yaml" file="hello-apparmor-pod.yaml" ghlink="/docs/admin/apparmor/hello-apparmor-pod.yaml" %} + + $ kubectl create -f /dev/stdin < + Status: Failed + Reason: AppArmor + Message: Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded + IP: + Controllers: + Containers: + hello: + Image: busybox + Port: + Command: + sh + -c + echo 'Hello AppArmor!' && sleep 1h + Requests: + cpu: 100m + Environment Variables: + Volumes: + default-token-dnz7v: + Type: Secret (a volume populated by a Secret) + SecretName: default-token-dnz7v + QoS Tier: Burstable + Events: + FirstSeen LastSeen Count From SubobjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 23s 23s 1 {default-scheduler } Normal Scheduled Successfully assigned hello-apparmor-2 to e2e-test-stclair-minion-group-t1f5 + 23s 23s 1 {kubelet e2e-test-stclair-minion-group-t1f5} Warning AppArmor Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded + +Note the pod status is Failed, with a helpful error message: `Pod Cannot enforce AppArmor: profile +"k8s-apparmor-example-allow-write" is not loaded`. An event was also recorded with the same message. + +## Administration + +### Setting up nodes with profiles + +Kubernetes does not currently provide any native mechanisms for loading AppArmor profiles onto +nodes. There are lots of ways to setup the profiles though, such as: + +- Through a [DaemonSet](../daemons/) that runs a Pod on each node to + ensure the correct profiles are loaded. An example implementation can be found + [here](https://github.com/kubernetes/contrib/tree/master/apparmor/loader). +- At node initialization time, using your node initialization scripts (e.g. Salt, Ansible, etc.) or + image. +- By copying the profiles to each node and loading them through SSH, as demonstrated in the + [Example](#example). + +The scheduler is not aware of which profiles are loaded onto which node, so the full set of profiles +must be loaded onto every node. An alternative approach is to add a node label for each profile (or +class of profiles) on the node, and use a +[node selector](../../user-guide/node-selection/) to ensure the Pod is run on a +node with the required profile. + +### Restricting profiles with the PodSecurityPolicy + +If the PodSecurityPolicy extension is enabled, cluster-wide AppArmor restrictions can be applied. To +enable the PodSecurityPolicy, two flags must be set on the `apiserver`: + + --admission-control=PodSecurityPolicy[,others...] + --runtime-config=extensions/v1beta1/podsecuritypolicy[,others...] + +With the extension enabled, the AppArmor options can be specified as annotations on the PodSecurityPolicy: + + apparmor.security.beta.kubernetes.io/defaultProfileName: + apparmor.security.beta.kubernetes.io/allowedProfileNames: [,others...] + +The default profile name option specifies the profile to apply to containers by default when none is +specified. The allowed profile names option specifies a list of profiles that Pod containers are +allowed to be run with. If both options are provided, the default must be allowed. The profiles are +specified in the same format as on containers. See the [API Reference](#api-reference) for the full +specification. + +### Disabling AppArmor + +If you do not want AppArmor to be available on your cluster, it can be disabled by a command-line flag: + + --feature-gates=AppArmor=false + +When disabled, any Pod that includes an AppArmor profile will fail validation with a "Forbidden" +error. Note that by default docker always enables the "docker-default" profile on non-privileged +pods (if the AppArmor kernel module is enabled), and will continue to do so even if the feature-gate +is disabled. The option to disable AppArmor will be removed when AppArmor graduates to general +availability (GA). + +### Upgrading to Kubernetes v1.4 with AppArmor + +No action is required with respect to AppArmor to upgrade your cluster to v1.4. However, if any +existing pods had an AppArmor annotation, they will not go through validation (or PodSecurityPolicy +admission). If permissive profiles are loaded on the nodes, a malicious user could pre-apply a +permissive profile to escalate the pod privileges above the docker-default. If this is a concern, it +is recommended to scrub the cluster of any pods containing an annotation with +`apparmor.security.beta.kubernetes.io`. + +### Upgrade path to General Availability + +When AppArmor is ready to be graduated to general availability (GA), the options currently specified +through annotations will be converted to fields. Supporting all the upgrade and downgrade paths +through the transition is very nuanced, and will be explained in detail when the transition +occurs. We will commit to supporting both fields and annotations for at least 2 releases, and will +explicitly reject the annotations for at least 2 releases after that. + +## Authoring Profiles + +Getting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some +tools to help with that: + +- `aa-genprof` and `aa-logprof` generate profile rules by monitoring an application's activity and + logs, and admitting the actions it takes. Further instructions are provided by the + [AppArmor documentation](http://wiki.apparmor.net/index.php/Profiling_with_tools). +- [bane](https://github.com/jfrazelle/bane) is an AppArmor profile generator for Docker that uses a + simplified profile language. + +It is recommended to run your application through Docker on a development workstation to generate +the profiles, but there is nothing preventing running the tools on the Kubernetes node where your +Pod is running. + +To debug problems with AppArmor, you can check the system logs to see what, specifically, was +denied. AppArmor logs verbose messages to `dmesg`, and errors can usually be found in the system +logs or through `journalctl`. More information is provided in +[AppArmor failures](http://wiki.apparmor.net/index.php/AppArmor_Failures). + +Additional resources: + +- [Quick guide to the AppArmor profile language](http://wiki.apparmor.net/index.php/QuickProfileLanguage) +- [AppArmor core policy reference](http://wiki.apparmor.net/index.php/ProfileLanguage) + +## API Reference + +**Pod Annotation**: + +Specifying the profile a container will run with: + +- **key**: `container.apparmor.security.beta.kubernetes.io/` + Where `` matches the name of a container in the Pod. + A separate profile can be specified for each container in the Pod. +- **value**: a profile reference, described below + +**Profile Reference**: + +- `runtime/default`: Refers to the default runtime profile. + - Equivalent to not specifying a profile (without a PodSecurityPolicy default), except it still + requires AppArmor to be enabled. + - For Docker, this resolves to the + [`docker-default`](https://docs.docker.com/engine/security/apparmor/) profile for non-privileged + containers, and unconfined (no profile) for privileged containers. +- `localhost/`: Refers to a profile loaded on the node (localhost) by name. + - The possible profile names are detailed in the + [core policy reference](http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference#Profile_names_and_attachment_specifications) + +Any other profile reference format is invalid. + +**PodSecurityPolicy Annotations** + +Specifying the default profile to apply to containers when none is provided: + +- **key**: `apparmor.security.beta.kubernetes.io/defaultProfileName` +- **value**: a profile reference, described above + +Specifying the list of profiles Pod containers is allowed to specify: + +- **key**: `apparmor.security.beta.kubernetes.io/allowedProfileNames` +- **value**: a comma-separated list of profile references (described above) + - Although an escaped comma is a legal character in a profile name, it cannot be explicitly + allowed here From 567c32ef783448ab011fdfe10b5b929f131eac8d Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 19:09:08 -0700 Subject: [PATCH 383/407] Apparmor (#2825) * Move Guide topic: AppArmor. * Add included files. --- docs/tutorials/clusters/apparmor.md | 5 +++-- docs/tutorials/clusters/deny-write.profile | 10 ++++++++++ docs/tutorials/clusters/hello-apparmor-pod.yaml | 13 +++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) create mode 100644 docs/tutorials/clusters/deny-write.profile create mode 100644 docs/tutorials/clusters/hello-apparmor-pod.yaml diff --git a/docs/tutorials/clusters/apparmor.md b/docs/tutorials/clusters/apparmor.md index 224f0bbdeb..881c2e72a6 100644 --- a/docs/tutorials/clusters/apparmor.md +++ b/docs/tutorials/clusters/apparmor.md @@ -140,7 +140,8 @@ In this example you'll see: First, we need to load the profile we want to use onto our nodes. The profile we'll use simply denies all file writes: -{% include code.html language="text" file="deny-write.profile" ghlink="/docs/admin/apparmor/deny-write.profile" %} + +{% include code.html language="text" file="deny-write.profile" ghlink="/docs/tutorials/clusters/deny-write.profile" %} Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our nodes. For this example we'll just use SSH to install the profiles, but other approaches are @@ -167,7 +168,7 @@ discussed in [Setting up nodes with profiles](#setting-up-nodes-with-profiles). Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile: -{% include code.html language="yaml" file="hello-apparmor-pod.yaml" ghlink="/docs/admin/apparmor/hello-apparmor-pod.yaml" %} +{% include code.html language="yaml" file="hello-apparmor-pod.yaml" ghlink="/docs/tutorials/clusters/hello-apparmor-pod.yaml" %} $ kubectl create -f /dev/stdin < + +profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { + #include + + file, + + # Deny all file writes. + deny /** w, +} diff --git a/docs/tutorials/clusters/hello-apparmor-pod.yaml b/docs/tutorials/clusters/hello-apparmor-pod.yaml new file mode 100644 index 0000000000..3e9b3b2a9c --- /dev/null +++ b/docs/tutorials/clusters/hello-apparmor-pod.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Pod +metadata: + name: hello-apparmor + annotations: + # Tell Kubernetes to apply the AppArmor profile "k8s-apparmor-example-deny-write". + # Note that this is ignored if the Kubernetes node is not running version 1.4 or greater. + container.apparmor.security.beta.kubernetes.io/hello: localhost/k8s-apparmor-example-deny-write +spec: + containers: + - name: hello + image: busybox + command: [ "sh", "-c", "echo 'Hello AppArmor!' && sleep 1h" ] From fcdbb3a920182e5b6b1b6ff8eb117d7fdff6c785 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Tue, 14 Mar 2017 19:13:33 -0700 Subject: [PATCH 384/407] Move Guide topic: Audit. (#2824) --- _data/concepts.yml | 1 + docs/admin/audit.md | 61 +---------------- docs/concepts/cluster-administration/audit.md | 67 +++++++++++++++++++ 3 files changed, 70 insertions(+), 59 deletions(-) create mode 100644 docs/concepts/cluster-administration/audit.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 9301f10917..d625c01753 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -35,6 +35,7 @@ toc: - docs/concepts/cluster-administration/networking.md - docs/concepts/cluster-administration/network-plugins.md - docs/concepts/cluster-administration/logging.md + - docs/concepts/cluster-administration/audit.md - docs/concepts/cluster-administration/out-of-resource.md - docs/concepts/cluster-administration/multiple-clusters.md - docs/concepts/cluster-administration/federation.md diff --git a/docs/admin/audit.md b/docs/admin/audit.md index ec4f7b2e95..33a0f1f722 100644 --- a/docs/admin/audit.md +++ b/docs/admin/audit.md @@ -5,63 +5,6 @@ assignees: title: Audit in Kubernetes --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -Kubernetes Audit provides a security-relevant chronological set of records documenting -the sequence of activities that have affected system by individual users, administrators -or other components of the system. It allows cluster administrator to -answer the following questions: - - what happened? - - when did it happen? - - who initiated it? - - on what did it happen? - - where was it observed? - - from where was it initiated? - - to where was it going? - -NOTE: Currently, Kubernetes provides only basic audit capabilities, there is still a lot -of work going on to provide fully featured auditing capabilities (see [this issue](https://github.com/kubernetes/features/issues/22)). - -Kubernetes audit is part of [kube-apiserver](/docs/admin/kube-apiserver) logging all requests -coming to the server. Each audit log contains two entries: - -1. The request line containing: - - unique id allowing to match the response line (see 2) - - source ip of the request - - HTTP method being invoked - - original user invoking the operation - - impersonated user for the operation - - namespace of the request or - - URI as requested -2. The response line containing: - - the unique id from 1 - - response code - -Example output for user `admin` asking for a list of pods: - -``` -2016-09-07T13:03:57.400333046Z AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" ip="127.0.0.1" method="GET" user="admin" as="" namespace="default" uri="/api/v1/namespaces/default/pods" -2016-09-07T13:03:57.400710987Z AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" response="200" -``` - -NOTE: The audit capabilities are available *only* for the secured endpoint of the API server. - -## Configuration - -[Kube-apiserver](/docs/admin/kube-apiserver) provides following options which are responsible -for configuring where and how audit logs are handled: - -- `audit-log-path` - enables the audit log pointing to a file where the requests are being logged to. -- `audit-log-maxage` - specifies maximum number of days to retain old audit log files based on the timestamp encoded in their filename. -- `audit-log-maxbackup` - specifies maximum number of old audit log files to retain. -- `audit-log-maxsize` - specifies maximum size in megabytes of the audit log file before it gets rotated. Defaults to 100MB - -If an audit log file already exists, Kubernetes appends new audit logs to that file. -Otherwise, Kubernetes creates an audit log file at the location you specified in -`audit-log-path`. If the audit log file exceeds the size you specify in `audit-log-maxsize`, -Kubernetes will rename the current log file by appending the current timestamp on -the file name (before the file extension) and create a new audit log file. -Kubernetes may delete old log files when creating a new log file; you can configure -how many files are retained and how old they can be by specifying the `audit-log-maxbackup` -and `audit-log-maxage` options. +[Auditing](/docs/concepts/cluster-administration/audit/) diff --git a/docs/concepts/cluster-administration/audit.md b/docs/concepts/cluster-administration/audit.md new file mode 100644 index 0000000000..e60ad728b9 --- /dev/null +++ b/docs/concepts/cluster-administration/audit.md @@ -0,0 +1,67 @@ +--- +assignees: +- soltysh +- sttts +title: Auditing +--- + +* TOC +{:toc} + +Kubernetes Audit provides a security-relevant chronological set of records documenting +the sequence of activities that have affected system by individual users, administrators +or other components of the system. It allows cluster administrator to +answer the following questions: + - what happened? + - when did it happen? + - who initiated it? + - on what did it happen? + - where was it observed? + - from where was it initiated? + - to where was it going? + +NOTE: Currently, Kubernetes provides only basic audit capabilities, there is still a lot +of work going on to provide fully featured auditing capabilities (see [this issue](https://github.com/kubernetes/features/issues/22)). + +Kubernetes audit is part of [kube-apiserver](/docs/admin/kube-apiserver) logging all requests +coming to the server. Each audit log contains two entries: + +1. The request line containing: + - unique id allowing to match the response line (see 2) + - source ip of the request + - HTTP method being invoked + - original user invoking the operation + - impersonated user for the operation + - namespace of the request or + - URI as requested +2. The response line containing: + - the unique id from 1 + - response code + +Example output for user `admin` asking for a list of pods: + +``` +2016-09-07T13:03:57.400333046Z AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" ip="127.0.0.1" method="GET" user="admin" as="" namespace="default" uri="/api/v1/namespaces/default/pods" +2016-09-07T13:03:57.400710987Z AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" response="200" +``` + +NOTE: The audit capabilities are available *only* for the secured endpoint of the API server. + +## Configuration + +[Kube-apiserver](/docs/admin/kube-apiserver) provides following options which are responsible +for configuring where and how audit logs are handled: + +- `audit-log-path` - enables the audit log pointing to a file where the requests are being logged to. +- `audit-log-maxage` - specifies maximum number of days to retain old audit log files based on the timestamp encoded in their filename. +- `audit-log-maxbackup` - specifies maximum number of old audit log files to retain. +- `audit-log-maxsize` - specifies maximum size in megabytes of the audit log file before it gets rotated. Defaults to 100MB + +If an audit log file already exists, Kubernetes appends new audit logs to that file. +Otherwise, Kubernetes creates an audit log file at the location you specified in +`audit-log-path`. If the audit log file exceeds the size you specify in `audit-log-maxsize`, +Kubernetes will rename the current log file by appending the current timestamp on +the file name (before the file extension) and create a new audit log file. +Kubernetes may delete old log files when creating a new log file; you can configure +how many files are retained and how old they can be by specifying the `audit-log-maxbackup` +and `audit-log-maxage` options. From 47d2aa9bfdbb45ce5e8ef7db5eab42405ff9af75 Mon Sep 17 00:00:00 2001 From: "Timothy St. Clair" Date: Fri, 10 Mar 2017 10:02:59 -0600 Subject: [PATCH 385/407] Added 1.6 Upgrade notes to outline changes for etcd 3. --- _data/guides.yml | 1 + docs/admin/cluster-management.md | 12 +++++++++--- docs/admin/upgrade-1-6.md | 26 ++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 3 deletions(-) create mode 100644 docs/admin/upgrade-1-6.md diff --git a/_data/guides.yml b/_data/guides.yml index 31be2a277e..0a06052579 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -169,6 +169,7 @@ toc: section: - docs/admin/index.md - docs/admin/cluster-management.md + - docs/admin/upgrade-1-6.md - docs/admin/kubeadm.md - docs/admin/addons.md - docs/admin/audit.md diff --git a/docs/admin/cluster-management.md b/docs/admin/cluster-management.md index 24c5a2dfe9..ebe5687f2e 100644 --- a/docs/admin/cluster-management.md +++ b/docs/admin/cluster-management.md @@ -19,7 +19,9 @@ To install Kubernetes on a set of machines, consult one of the existing [Getting ## Upgrading a cluster -The current state of cluster upgrades is provider dependent. +The current state of cluster upgrades is provider dependent, and some releases may require special care when upgrading. It is recommended that administrators consult both the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md), as well as the version specific upgrade notes prior to upgrading their clusters. + +* [Upgrading to 1.6](/docs/admin/upgrade) ### Upgrading Google Compute Engine clusters @@ -56,8 +58,12 @@ The node upgrade process is user-initiated and is described in the [GKE document ### Upgrading clusters on other platforms -The `cluster/kube-push.sh` script will do a rudimentary update. This process is still quite experimental, we -recommend testing the upgrade on an experimental cluster before performing the update on a production cluster. +Different providers, and tools, will manage upgrades differently. It is recommended that you consult their main documentation regarding upgrades. + +* [kops](https://github.com/kubernetes/kops) +* [kargo](https://github.com/kubernetes-incubator/kargo) +* [CoreOS Tectonic](https://coreos.com/tectonic/docs/latest/admin/upgrade.html) +* ... ## Resizing a cluster diff --git a/docs/admin/upgrade-1-6.md b/docs/admin/upgrade-1-6.md new file mode 100644 index 0000000000..ca73f04e26 --- /dev/null +++ b/docs/admin/upgrade-1-6.md @@ -0,0 +1,26 @@ +--- +assignees: +- mml +title: Cluster Management Guide +--- + +* TOC +{:toc} + +This document outlines the potentially disruptive changes that exist in the 1.6 release cycle. Operators, administrators, and developers should +take note of the changes below in order to maintain continuity across their upgrade process. + +## Cluster defaults set to etcd 3 + +In the 1.6 release cycle, the default backend storage layer has been upgraded to fully leverage [etcd 3 capabilities](https://coreos.com/blog/etcd3-a-new-etcd.html) by default. +For new clusters, there is nothing an operator will need to do, it should "just work". However, if you are upgrading from a 1.5 cluster, care should be taken to ensure +continuity. + +It is possible to maintain v2 compatibility mode while running etcd 3 for an interim period of time. To do this, you will simply need to update an argument passed to your apiserver during +startup: + +``` +$ kube-apiserver --storage-backend='etcd2' $(EXISTING_ARGS) +``` + +However, for long-term maintenance of the cluster, we recommend that the operator plan an outage window in order to perform a [v2->v3 data upgrade](https://coreos.com/etcd/docs/latest/upgrades/upgrade_3_0.html). \ No newline at end of file From a804035cffd0caaed24b48208293d193b470649d Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 10:19:27 -0700 Subject: [PATCH 386/407] Move Guide topic: Disruption Budget. (#2834) --- _data/tasks.yml | 1 + docs/admin/disruptions.md | 90 +----------------- .../configure-pod-disruption-budget.md | 95 +++++++++++++++++++ 3 files changed, 98 insertions(+), 88 deletions(-) create mode 100644 docs/tasks/configure-pod-container/configure-pod-disruption-budget.md diff --git a/_data/tasks.yml b/_data/tasks.yml index fa0f64927a..64b610c479 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -23,6 +23,7 @@ toc: - docs/tasks/configure-pod-container/communicate-containers-same-pod.md - docs/tasks/configure-pod-container/configure-pod-initialization.md - docs/tasks/configure-pod-container/attach-handler-lifecycle-event.md + - docs/tasks/configure-pod-container/configure-pod-disruption-budget.md - title: Accessing Applications in a Cluster section: diff --git a/docs/admin/disruptions.md b/docs/admin/disruptions.md index 6c4ee7df4a..d5ec02f8d5 100644 --- a/docs/admin/disruptions.md +++ b/docs/admin/disruptions.md @@ -3,93 +3,7 @@ assignees: - davidopp title: Pod Disruption Budget --- -This guide is for anyone wishing to specify safety constraints on pods or anyone -wishing to write software (typically automation software) that respects those -constraints. -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## Rationale - -Various cluster management operations may voluntarily evict pods. "Voluntary" -means an eviction can be safely delayed for a reasonable period of time. The -principal examples today are draining a node for maintenance or upgrade -(`kubectl drain`), and cluster autoscaling down. In the future the -[rescheduler](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/rescheduling.md) -may also perform voluntary evictions. By contrast, something like evicting pods -because a node has become unreachable or reports `NotReady`, is not "voluntary." - -For voluntary evictions, it can be useful for applications to be able to limit -the number of pods that are down simultaneously. For example, a quorum-based application would -like to ensure that the number of replicas running is never brought below the -number needed for a quorum, even temporarily. Or a web front end might want to -ensure that the number of replicas serving load never falls below a certain -percentage of the total, even briefly. `PodDisruptionBudget` is an API object -that specifies the minimum number or percentage of replicas of a collection that -must be up at a time. Components that wish to evict a pod subject to disruption -budget use the `/eviction` subresource; unlike a regular pod deletion, this -operation may be rejected by the API server if the eviction would cause a -disruption budget to be violated. - -## Specifying a PodDisruptionBudget - -A `PodDisruptionBudget` has two components: a label selector `selector` to specify the set of -pods to which it applies, and `minAvailable` which is a description of the number of pods from that -set that must still be available after the eviction, i.e. even in the absence -of the evicted pod. `minAvailable` can be either an absolute number or a percentage. -So for example, 100% means no voluntary evictions from the set are permitted. In -typical usage, a single budget would be used for a collection of pods managed by -a controller—for example, the pods in a single ReplicaSet. - -Note that a disruption budget does not truly guarantee that the specified -number/percentage of pods will always be up. For example, a node that hosts a -pod from the collection may fail when the collection is at the minimum size -specified in the budget, thus bringing the number of available pods from the -collection below the specified size. The budget can only protect against -voluntary evictions, not all causes of unavailability. - -## Requesting an eviction - -If you are writing infrastructure software that wants to produce these voluntary -evictions, you will need to use the eviction API. The eviction subresource of a -pod can be thought of as a kind of policy-controlled DELETE operation on the pod -itself. To attempt an eviction (perhaps more REST-precisely, to attempt to -*create* an eviction), you POST an attempted operation. Here's an example: - -```json -{ - "apiVersion": "policy/v1beta1", - "kind": "Eviction", - "metadata": { - "name": "quux", - "namespace": "default" - } -} -``` - -You can attempt an eviction using `curl`: - -```bash -$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json -``` - -The API can respond in one of three ways. - - 1. If the eviction is granted, then the pod is deleted just as if you had sent - a `DELETE` request to the pod's URL and you get back `200 OK`. - 2. If the current state of affairs wouldn't allow an eviction by the rules set - forth in the budget, you get back `429 Too Many Requests`. This is - typically used for generic rate limiting of *any* requests, but here we mean - that this request isn't allowed *right now* but it may be allowed later. - Currently, callers do not get any `Retry-After` advice, but they may in - future versions. - 3. If there is some kind of misconfiguration, like multiple budgets pointing at - the same pod, you will get `500 Internal Server Error`. - -For a given eviction request, there are two cases. - - 1. There is no budget that matches this pod. In this case, the server always - returns `200 OK`. - 2. There is at least one budget. In this case, any of the three above responses may - apply. +[Configuring a Pod Disruption Budget](/docs/tasks/configure-pod-container/configure-pod-disruption-budget/) diff --git a/docs/tasks/configure-pod-container/configure-pod-disruption-budget.md b/docs/tasks/configure-pod-container/configure-pod-disruption-budget.md new file mode 100644 index 0000000000..489f5e0f4c --- /dev/null +++ b/docs/tasks/configure-pod-container/configure-pod-disruption-budget.md @@ -0,0 +1,95 @@ +--- +assignees: +- davidopp +title: Configuring a Pod Disruption Budget +--- +This guide is for anyone wishing to specify safety constraints on pods or anyone +wishing to write software (typically automation software) that respects those +constraints. + +* TOC +{:toc} + +## Rationale + +Various cluster management operations may voluntarily evict pods. "Voluntary" +means an eviction can be safely delayed for a reasonable period of time. The +principal examples today are draining a node for maintenance or upgrade +(`kubectl drain`), and cluster autoscaling down. In the future the +[rescheduler](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/rescheduling.md) +may also perform voluntary evictions. By contrast, something like evicting pods +because a node has become unreachable or reports `NotReady`, is not "voluntary." + +For voluntary evictions, it can be useful for applications to be able to limit +the number of pods that are down simultaneously. For example, a quorum-based application would +like to ensure that the number of replicas running is never brought below the +number needed for a quorum, even temporarily. Or a web front end might want to +ensure that the number of replicas serving load never falls below a certain +percentage of the total, even briefly. `PodDisruptionBudget` is an API object +that specifies the minimum number or percentage of replicas of a collection that +must be up at a time. Components that wish to evict a pod subject to disruption +budget use the `/eviction` subresource; unlike a regular pod deletion, this +operation may be rejected by the API server if the eviction would cause a +disruption budget to be violated. + +## Specifying a PodDisruptionBudget + +A `PodDisruptionBudget` has two components: a label selector `selector` to specify the set of +pods to which it applies, and `minAvailable` which is a description of the number of pods from that +set that must still be available after the eviction, i.e. even in the absence +of the evicted pod. `minAvailable` can be either an absolute number or a percentage. +So for example, 100% means no voluntary evictions from the set are permitted. In +typical usage, a single budget would be used for a collection of pods managed by +a controller—for example, the pods in a single ReplicaSet. + +Note that a disruption budget does not truly guarantee that the specified +number/percentage of pods will always be up. For example, a node that hosts a +pod from the collection may fail when the collection is at the minimum size +specified in the budget, thus bringing the number of available pods from the +collection below the specified size. The budget can only protect against +voluntary evictions, not all causes of unavailability. + +## Requesting an eviction + +If you are writing infrastructure software that wants to produce these voluntary +evictions, you will need to use the eviction API. The eviction subresource of a +pod can be thought of as a kind of policy-controlled DELETE operation on the pod +itself. To attempt an eviction (perhaps more REST-precisely, to attempt to +*create* an eviction), you POST an attempted operation. Here's an example: + +```json +{ + "apiVersion": "policy/v1beta1", + "kind": "Eviction", + "metadata": { + "name": "quux", + "namespace": "default" + } +} +``` + +You can attempt an eviction using `curl`: + +```bash +$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json +``` + +The API can respond in one of three ways. + + 1. If the eviction is granted, then the pod is deleted just as if you had sent + a `DELETE` request to the pod's URL and you get back `200 OK`. + 2. If the current state of affairs wouldn't allow an eviction by the rules set + forth in the budget, you get back `429 Too Many Requests`. This is + typically used for generic rate limiting of *any* requests, but here we mean + that this request isn't allowed *right now* but it may be allowed later. + Currently, callers do not get any `Retry-After` advice, but they may in + future versions. + 3. If there is some kind of misconfiguration, like multiple budgets pointing at + the same pod, you will get `500 Internal Server Error`. + +For a given eviction request, there are two cases. + + 1. There is no budget that matches this pod. In this case, the server always + returns `200 OK`. + 2. There is at least one budget. In this case, any of the three above responses may + apply. From 9b5e645d922e60ccb9b846311c3d67d08feb710b Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 10:43:56 -0700 Subject: [PATCH 387/407] Move Guide topic: Limit range. (#2835) --- _data/tasks.yml | 1 + docs/admin/limitrange/index.md | 208 +---------------- .../configure-pod-container/limit-range.md | 214 ++++++++++++++++++ 3 files changed, 217 insertions(+), 206 deletions(-) create mode 100644 docs/tasks/configure-pod-container/limit-range.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 64b610c479..606f595e27 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -13,6 +13,7 @@ toc: - docs/tasks/configure-pod-container/define-environment-variable-container.md - docs/tasks/configure-pod-container/define-command-argument-container.md - docs/tasks/configure-pod-container/assign-cpu-ram-container.md + - docs/tasks/configure-pod-container/limit-range.md - docs/tasks/configure-pod-container/configure-volume-storage.md - docs/tasks/configure-pod-container/configure-persistent-volume-storage.md - docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md diff --git a/docs/admin/limitrange/index.md b/docs/admin/limitrange/index.md index 1791f14f4d..6d982a9aae 100644 --- a/docs/admin/limitrange/index.md +++ b/docs/admin/limitrange/index.md @@ -5,210 +5,6 @@ assignees: title: Setting Pod CPU and Memory Limits --- -By default, pods run with unbounded CPU and memory limits. This means that any pod in the -system will be able to consume as much CPU and memory on the node that executes the pod. +{% include user-guide-content-moved.md %} -Users may want to impose restrictions on the amount of resources a single pod in the system may consume -for a variety of reasons. - -For example: - -1. Each node in the cluster has 2GB of memory. The cluster operator does not want to accept pods -that require more than 2GB of memory since no node in the cluster can support the requirement. To prevent a -pod from being permanently unscheduled to a node, the operator instead chooses to reject pods that exceed 2GB -of memory as part of admission control. -2. A cluster is shared by two communities in an organization that runs production and development workloads -respectively. Production workloads may consume up to 8GB of memory, but development workloads may consume up -to 512MB of memory. The cluster operator creates a separate namespace for each workload, and applies limits to -each namespace. -3. Users may create a pod which consumes resources just below the capacity of a machine. The left over space -may be too small to be useful, but big enough for the waste to be costly over the entire cluster. As a result, -the cluster operator may want to set limits that a pod must consume at least 20% of the memory and CPU of their -average node size in order to provide for more uniform scheduling and limit waste. - -This example demonstrates how limits can be applied to a Kubernetes [namespace](/docs/admin/namespaces/walkthrough/) to control -min/max resource limits per pod. In addition, this example demonstrates how you can -apply default resource limits to pods in the absence of an end-user specified value. - -See [LimitRange design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/) - -## Step 0: Prerequisites - -This example requires a running Kubernetes cluster. See the [Getting Started guides](/docs/getting-started-guides/) for how to get started. - -Change to the `` directory if you're not already there. - -## Step 1: Create a namespace - -This example will work in a custom namespace to demonstrate the concepts involved. - -Let's create a new namespace called limit-example: - -```shell -$ kubectl create namespace limit-example -namespace "limit-example" created -``` - -Note that `kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands: - -```shell -$ kubectl get namespaces -NAME STATUS AGE -default Active 51s -limit-example Active 45s -``` - -## Step 2: Apply a limit to the namespace - -Let's create a simple limit in our namespace. - -```shell -$ kubectl create -f docs/admin/limitrange/limits.yaml --namespace=limit-example -limitrange "mylimits" created -``` - -Let's describe the limits that we have imposed in our namespace. - -```shell -$ kubectl describe limits mylimits --namespace=limit-example -Name: mylimits -Namespace: limit-example -Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ----- -------- --- --- --------------- ------------- ----------------------- -Pod cpu 200m 2 - - - -Pod memory 6Mi 1Gi - - - -Container cpu 100m 2 200m 300m - -Container memory 3Mi 1Gi 100Mi 200Mi - -``` - -In this scenario, we have said the following: - -1. If a max constraint is specified for a resource (2 CPU and 1Gi memory in this case), then a limit -must be specified for that resource across all containers. Failure to specify a limit will result in -a validation error when attempting to create the pod. Note that a default value of limit is set by -*default* in file `limits.yaml` (300m CPU and 200Mi memory). -2. If a min constraint is specified for a resource (100m CPU and 3Mi memory in this case), then a -request must be specified for that resource across all containers. Failure to specify a request will -result in a validation error when attempting to create the pod. Note that a default value of request is -set by *defaultRequest* in file `limits.yaml` (200m CPU and 100Mi memory). -3. For any pod, the sum of all containers memory requests must be >= 6Mi and the sum of all containers -memory limits must be <= 1Gi; the sum of all containers CPU requests must be >= 200m and the sum of all -containers CPU limits must be <= 2. - -## Step 3: Enforcing limits at point of creation - -The limits enumerated in a namespace are only enforced when a pod is created or updated in -the cluster. If you change the limits to a different value range, it does not affect pods that -were previously created in a namespace. - -If a resource (CPU or memory) is being restricted by a limit, the user will get an error at time -of creation explaining why. - -Let's first spin up a [Deployment](/docs/user-guide/deployments) that creates a single container Pod to demonstrate -how default values are applied to each pod. - -```shell -$ kubectl run nginx --image=nginx --replicas=1 --namespace=limit-example -deployment "nginx" created -``` - -Note that `kubectl run` creates a Deployment named "nginx" on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead. -If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/user-guide/kubectl/kubectl_run/) for more details. -The Deployment manages 1 replica of single container Pod. Let's take a look at the Pod it manages. First, find the name of the Pod: - -```shell -$ kubectl get pods --namespace=limit-example -NAME READY STATUS RESTARTS AGE -nginx-2040093540-s8vzu 1/1 Running 0 11s -``` - -Let's print this Pod with yaml output format (using `-o yaml` flag), and then `grep` the `resources` field. Note that your pod name will be different. - -```shell -$ kubectl get pods nginx-2040093540-s8vzu --namespace=limit-example -o yaml | grep resources -C 8 - resourceVersion: "57" - selfLink: /api/v1/namespaces/limit-example/pods/nginx-2040093540-ivimu - uid: 67b20741-f53b-11e5-b066-64510658e388 -spec: - containers: - - image: nginx - imagePullPolicy: Always - name: nginx - resources: - limits: - cpu: 300m - memory: 200Mi - requests: - cpu: 200m - memory: 100Mi - terminationMessagePath: /dev/termination-log - volumeMounts: -``` - -Note that our nginx container has picked up the namespace default CPU and memory resource *limits* and *requests*. - -Let's create a pod that exceeds our allowed limits by having it have a container that requests 3 CPU cores. - -```shell -$ kubectl create -f docs/admin/limitrange/invalid-pod.yaml --namespace=limit-example -Error from server: error when creating "docs/admin/limitrange/invalid-pod.yaml": Pod "invalid-pod" is forbidden: [Maximum cpu usage per Pod is 2, but limit is 3., Maximum cpu usage per Container is 2, but limit is 3.] -``` - -Let's create a pod that falls within the allowed limit boundaries. - -```shell -$ kubectl create -f docs/admin/limitrange/valid-pod.yaml --namespace=limit-example -pod "valid-pod" created -``` - -Now look at the Pod's resources field: - -```shell -$ kubectl get pods valid-pod --namespace=limit-example -o yaml | grep -C 6 resources - uid: 3b1bfd7a-f53c-11e5-b066-64510658e388 -spec: - containers: - - image: gcr.io/google_containers/serve_hostname - imagePullPolicy: Always - name: kubernetes-serve-hostname - resources: - limits: - cpu: "1" - memory: 512Mi - requests: - cpu: "1" - memory: 512Mi -``` - -Note that this pod specifies explicit resource *limits* and *requests* so it did not pick up the namespace -default values. - -Note: The *limits* for CPU resource are enforced in the default Kubernetes setup on the physical node -that runs the container unless the administrator deploys the kubelet with the following flag: - -```shell -$ kubelet --help -Usage of kubelet -.... - --cpu-cfs-quota[=true]: Enable CPU CFS quota enforcement for containers that specify CPU limits -$ kubelet --cpu-cfs-quota=false ... -``` - -## Step 4: Cleanup - -To remove the resources used by this example, you can just delete the limit-example namespace. - -```shell -$ kubectl delete namespace limit-example -namespace "limit-example" deleted -$ kubectl get namespaces -NAME STATUS AGE -default Active 12m -``` - -## Summary - -Cluster operators that want to restrict the amount of resources a single container or pod may consume -are able to define allowable ranges per Kubernetes namespace. In the absence of any explicit assignments, -the Kubernetes system is able to apply default resource *limits* and *requests* if desired in order to -constrain the amount of resource a pod consumes on a node. +[Setting Pod CPU and Memory Limits](/docs/tasks/configure-pod-container/limit-range/) diff --git a/docs/tasks/configure-pod-container/limit-range.md b/docs/tasks/configure-pod-container/limit-range.md new file mode 100644 index 0000000000..1791f14f4d --- /dev/null +++ b/docs/tasks/configure-pod-container/limit-range.md @@ -0,0 +1,214 @@ +--- +assignees: +- derekwaynecarr +- janetkuo +title: Setting Pod CPU and Memory Limits +--- + +By default, pods run with unbounded CPU and memory limits. This means that any pod in the +system will be able to consume as much CPU and memory on the node that executes the pod. + +Users may want to impose restrictions on the amount of resources a single pod in the system may consume +for a variety of reasons. + +For example: + +1. Each node in the cluster has 2GB of memory. The cluster operator does not want to accept pods +that require more than 2GB of memory since no node in the cluster can support the requirement. To prevent a +pod from being permanently unscheduled to a node, the operator instead chooses to reject pods that exceed 2GB +of memory as part of admission control. +2. A cluster is shared by two communities in an organization that runs production and development workloads +respectively. Production workloads may consume up to 8GB of memory, but development workloads may consume up +to 512MB of memory. The cluster operator creates a separate namespace for each workload, and applies limits to +each namespace. +3. Users may create a pod which consumes resources just below the capacity of a machine. The left over space +may be too small to be useful, but big enough for the waste to be costly over the entire cluster. As a result, +the cluster operator may want to set limits that a pod must consume at least 20% of the memory and CPU of their +average node size in order to provide for more uniform scheduling and limit waste. + +This example demonstrates how limits can be applied to a Kubernetes [namespace](/docs/admin/namespaces/walkthrough/) to control +min/max resource limits per pod. In addition, this example demonstrates how you can +apply default resource limits to pods in the absence of an end-user specified value. + +See [LimitRange design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md) for more information. For a detailed description of the Kubernetes resource model, see [Resources](/docs/user-guide/compute-resources/) + +## Step 0: Prerequisites + +This example requires a running Kubernetes cluster. See the [Getting Started guides](/docs/getting-started-guides/) for how to get started. + +Change to the `` directory if you're not already there. + +## Step 1: Create a namespace + +This example will work in a custom namespace to demonstrate the concepts involved. + +Let's create a new namespace called limit-example: + +```shell +$ kubectl create namespace limit-example +namespace "limit-example" created +``` + +Note that `kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands: + +```shell +$ kubectl get namespaces +NAME STATUS AGE +default Active 51s +limit-example Active 45s +``` + +## Step 2: Apply a limit to the namespace + +Let's create a simple limit in our namespace. + +```shell +$ kubectl create -f docs/admin/limitrange/limits.yaml --namespace=limit-example +limitrange "mylimits" created +``` + +Let's describe the limits that we have imposed in our namespace. + +```shell +$ kubectl describe limits mylimits --namespace=limit-example +Name: mylimits +Namespace: limit-example +Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio +---- -------- --- --- --------------- ------------- ----------------------- +Pod cpu 200m 2 - - - +Pod memory 6Mi 1Gi - - - +Container cpu 100m 2 200m 300m - +Container memory 3Mi 1Gi 100Mi 200Mi - +``` + +In this scenario, we have said the following: + +1. If a max constraint is specified for a resource (2 CPU and 1Gi memory in this case), then a limit +must be specified for that resource across all containers. Failure to specify a limit will result in +a validation error when attempting to create the pod. Note that a default value of limit is set by +*default* in file `limits.yaml` (300m CPU and 200Mi memory). +2. If a min constraint is specified for a resource (100m CPU and 3Mi memory in this case), then a +request must be specified for that resource across all containers. Failure to specify a request will +result in a validation error when attempting to create the pod. Note that a default value of request is +set by *defaultRequest* in file `limits.yaml` (200m CPU and 100Mi memory). +3. For any pod, the sum of all containers memory requests must be >= 6Mi and the sum of all containers +memory limits must be <= 1Gi; the sum of all containers CPU requests must be >= 200m and the sum of all +containers CPU limits must be <= 2. + +## Step 3: Enforcing limits at point of creation + +The limits enumerated in a namespace are only enforced when a pod is created or updated in +the cluster. If you change the limits to a different value range, it does not affect pods that +were previously created in a namespace. + +If a resource (CPU or memory) is being restricted by a limit, the user will get an error at time +of creation explaining why. + +Let's first spin up a [Deployment](/docs/user-guide/deployments) that creates a single container Pod to demonstrate +how default values are applied to each pod. + +```shell +$ kubectl run nginx --image=nginx --replicas=1 --namespace=limit-example +deployment "nginx" created +``` + +Note that `kubectl run` creates a Deployment named "nginx" on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead. +If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/user-guide/kubectl/kubectl_run/) for more details. +The Deployment manages 1 replica of single container Pod. Let's take a look at the Pod it manages. First, find the name of the Pod: + +```shell +$ kubectl get pods --namespace=limit-example +NAME READY STATUS RESTARTS AGE +nginx-2040093540-s8vzu 1/1 Running 0 11s +``` + +Let's print this Pod with yaml output format (using `-o yaml` flag), and then `grep` the `resources` field. Note that your pod name will be different. + +```shell +$ kubectl get pods nginx-2040093540-s8vzu --namespace=limit-example -o yaml | grep resources -C 8 + resourceVersion: "57" + selfLink: /api/v1/namespaces/limit-example/pods/nginx-2040093540-ivimu + uid: 67b20741-f53b-11e5-b066-64510658e388 +spec: + containers: + - image: nginx + imagePullPolicy: Always + name: nginx + resources: + limits: + cpu: 300m + memory: 200Mi + requests: + cpu: 200m + memory: 100Mi + terminationMessagePath: /dev/termination-log + volumeMounts: +``` + +Note that our nginx container has picked up the namespace default CPU and memory resource *limits* and *requests*. + +Let's create a pod that exceeds our allowed limits by having it have a container that requests 3 CPU cores. + +```shell +$ kubectl create -f docs/admin/limitrange/invalid-pod.yaml --namespace=limit-example +Error from server: error when creating "docs/admin/limitrange/invalid-pod.yaml": Pod "invalid-pod" is forbidden: [Maximum cpu usage per Pod is 2, but limit is 3., Maximum cpu usage per Container is 2, but limit is 3.] +``` + +Let's create a pod that falls within the allowed limit boundaries. + +```shell +$ kubectl create -f docs/admin/limitrange/valid-pod.yaml --namespace=limit-example +pod "valid-pod" created +``` + +Now look at the Pod's resources field: + +```shell +$ kubectl get pods valid-pod --namespace=limit-example -o yaml | grep -C 6 resources + uid: 3b1bfd7a-f53c-11e5-b066-64510658e388 +spec: + containers: + - image: gcr.io/google_containers/serve_hostname + imagePullPolicy: Always + name: kubernetes-serve-hostname + resources: + limits: + cpu: "1" + memory: 512Mi + requests: + cpu: "1" + memory: 512Mi +``` + +Note that this pod specifies explicit resource *limits* and *requests* so it did not pick up the namespace +default values. + +Note: The *limits* for CPU resource are enforced in the default Kubernetes setup on the physical node +that runs the container unless the administrator deploys the kubelet with the following flag: + +```shell +$ kubelet --help +Usage of kubelet +.... + --cpu-cfs-quota[=true]: Enable CPU CFS quota enforcement for containers that specify CPU limits +$ kubelet --cpu-cfs-quota=false ... +``` + +## Step 4: Cleanup + +To remove the resources used by this example, you can just delete the limit-example namespace. + +```shell +$ kubectl delete namespace limit-example +namespace "limit-example" deleted +$ kubectl get namespaces +NAME STATUS AGE +default Active 12m +``` + +## Summary + +Cluster operators that want to restrict the amount of resources a single container or pod may consume +are able to define allowable ranges per Kubernetes namespace. In the absence of any explicit assignments, +the Kubernetes system is able to apply default resource *limits* and *requests* if desired in order to +constrain the amount of resource a pod consumes on a node. From 3175e9a9b677ea9e5a47903a24aa52f716beb416 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 11:09:50 -0700 Subject: [PATCH 388/407] Quota (#2836) * Move Guide topic: Limit range. * Move Guide topic: Resource Quota and Limits. --- _data/tasks.yml | 1 + docs/admin/resourcequota/walkthrough.md | 360 +---------------- .../apply-resource-quota-limit.md | 366 ++++++++++++++++++ 3 files changed, 369 insertions(+), 358 deletions(-) create mode 100644 docs/tasks/configure-pod-container/apply-resource-quota-limit.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 606f595e27..309eb0673c 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -14,6 +14,7 @@ toc: - docs/tasks/configure-pod-container/define-command-argument-container.md - docs/tasks/configure-pod-container/assign-cpu-ram-container.md - docs/tasks/configure-pod-container/limit-range.md + - docs/tasks/configure-pod-container/apply-resource-quota-limit.md - docs/tasks/configure-pod-container/configure-volume-storage.md - docs/tasks/configure-pod-container/configure-persistent-volume-storage.md - docs/tasks/configure-pod-container/environment-variable-expose-pod-information.md diff --git a/docs/admin/resourcequota/walkthrough.md b/docs/admin/resourcequota/walkthrough.md index 1120e7550d..cf90328c96 100644 --- a/docs/admin/resourcequota/walkthrough.md +++ b/docs/admin/resourcequota/walkthrough.md @@ -5,362 +5,6 @@ assignees: title: Applying Resource Quotas and Limits --- -This example demonstrates a typical setup to control for resource usage in a namespace. +{% include user-guide-content-moved.md %} -It demonstrates using the following resources: - -* [Namespace](/docs/admin/namespaces) -* [Resource Quota](/docs/admin/resourcequota/) -* [Limit Range](/docs/admin/limitrange/) - -This example assumes you have a functional Kubernetes setup. - -## Scenario - -The cluster-admin is operating a cluster on behalf of a user population and the cluster-admin -wants to control the amount of resources that can be consumed in a particular namespace to promote -fair sharing of the cluster and control cost. - -The cluster-admin has the following goals: - -* Limit the amount of compute resource for running pods -* Limit the number of persistent volume claims to control access to storage -* Limit the number of load balancers to control cost -* Prevent the use of node ports to preserve scarce resources -* Provide default compute resource requests to enable better scheduling decisions - -## Step 1: Create a namespace - -This example will work in a custom namespace to demonstrate the concepts involved. - -Let's create a new namespace called quota-example: - -```shell -$ kubectl create -f docs/admin/resourcequota/namespace.yaml -namespace "quota-example" created -$ kubectl get namespaces -NAME STATUS AGE -default Active 2m -kube-system Active 2m -quota-example Active 39s -``` - -## Step 2: Apply an object-count quota to the namespace - -The cluster-admin wants to control the following resources: - -* persistent volume claims -* load balancers -* node ports - -Let's create a simple quota that controls object counts for those resource types in this namespace. - -```shell -$ kubectl create -f docs/admin/resourcequota/object-counts.yaml --namespace=quota-example -resourcequota "object-counts" created -``` - -The quota system will observe that a quota has been created, and will calculate consumption -in the namespace in response. This should happen quickly. - -Let's describe the quota to see what is currently being consumed in this namespace: - -```shell -$ kubectl describe quota object-counts --namespace=quota-example -Name: object-counts -Namespace: quota-example -Resource Used Hard --------- ---- ---- -persistentvolumeclaims 0 2 -services.loadbalancers 0 2 -services.nodeports 0 0 -``` - -The quota system will now prevent users from creating more than the specified amount for each resource. - - -## Step 3: Apply a compute-resource quota to the namespace - -To limit the amount of compute resource that can be consumed in this namespace, -let's create a quota that tracks compute resources. - -```shell -$ kubectl create -f docs/admin/resourcequota/compute-resources.yaml --namespace=quota-example -resourcequota "compute-resources" created -``` - -Let's describe the quota to see what is currently being consumed in this namespace: - -```shell -$ kubectl describe quota compute-resources --namespace=quota-example -Name: compute-resources -Namespace: quota-example -Resource Used Hard --------- ---- ---- -limits.cpu 0 2 -limits.memory 0 2Gi -pods 0 4 -requests.cpu 0 1 -requests.memory 0 1Gi -``` - -The quota system will now prevent the namespace from having more than 4 non-terminal pods. In -addition, it will enforce that each container in a pod makes a `request` and defines a `limit` for -`cpu` and `memory`. - -## Step 4: Applying default resource requests and limits - -Pod authors rarely specify resource requests and limits for their pods. - -Since we applied a quota to our project, let's see what happens when an end-user creates a pod that has unbounded -cpu and memory by creating an nginx container. - -To demonstrate, lets create a deployment that runs nginx: - -```shell -$ kubectl run nginx --image=nginx --replicas=1 --namespace=quota-example -deployment "nginx" created -``` - -Now let's look at the pods that were created. - -```shell -$ kubectl get pods --namespace=quota-example -``` - -What happened? I have no pods! Let's describe the deployment to get a view of what is happening. - -```shell -$ kubectl describe deployment nginx --namespace=quota-example -Name: nginx -Namespace: quota-example -CreationTimestamp: Mon, 06 Jun 2016 16:11:37 -0400 -Labels: run=nginx -Selector: run=nginx -Replicas: 0 updated | 1 total | 0 available | 1 unavailable -StrategyType: RollingUpdate -MinReadySeconds: 0 -RollingUpdateStrategy: 1 max unavailable, 1 max surge -OldReplicaSets: -NewReplicaSet: nginx-3137573019 (0/1 replicas created) -... -``` - -A deployment created a corresponding replica set and attempted to size it to create a single pod. - -Let's look at the replica set to get more detail. - -```shell -$ kubectl describe rs nginx-3137573019 --namespace=quota-example -Name: nginx-3137573019 -Namespace: quota-example -Image(s): nginx -Selector: pod-template-hash=3137573019,run=nginx -Labels: pod-template-hash=3137573019 - run=nginx -Replicas: 0 current / 1 desired -Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed -No volumes. -Events: - FirstSeen LastSeen Count From SubobjectPath Type Reason Message - --------- -------- ----- ---- ------------- -------- ------ ------- - 4m 7s 11 {replicaset-controller } Warning FailedCreate Error creating: pods "nginx-3137573019-" is forbidden: Failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory -``` - -The Kubernetes API server is rejecting the replica set requests to create a pod because our pods -do not specify `requests` or `limits` for `cpu` and `memory`. - -So let's set some default values for the amount of `cpu` and `memory` a pod can consume: - -```shell -$ kubectl create -f docs/admin/resourcequota/limits.yaml --namespace=quota-example -limitrange "limits" created -$ kubectl describe limits limits --namespace=quota-example -Name: limits -Namespace: quota-example -Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ----- -------- --- --- --------------- ------------- ----------------------- -Container memory - - 256Mi 512Mi - -Container cpu - - 100m 200m - -``` - -If the Kubernetes API server observes a request to create a pod in this namespace, and the containers -in that pod do not make any compute resource requests, a default request and default limit will be applied -as part of admission control. - -In this example, each pod created will have compute resources equivalent to the following: - -```shell -$ kubectl run nginx \ - --image=nginx \ - --replicas=1 \ - --requests=cpu=100m,memory=256Mi \ - --limits=cpu=200m,memory=512Mi \ - --namespace=quota-example -``` - -Now that we have applied default compute resources for our namespace, our replica set should be able to create -its pods. - -```shell -$ kubectl get pods --namespace=quota-example -NAME READY STATUS RESTARTS AGE -nginx-3137573019-fvrig 1/1 Running 0 6m -``` - -And if we print out our quota usage in the namespace: - -```shell -$ kubectl describe quota --namespace=quota-example -Name: compute-resources -Namespace: quota-example -Resource Used Hard --------- ---- ---- -limits.cpu 200m 2 -limits.memory 512Mi 2Gi -pods 1 4 -requests.cpu 100m 1 -requests.memory 256Mi 1Gi - - -Name: object-counts -Namespace: quota-example -Resource Used Hard --------- ---- ---- -persistentvolumeclaims 0 2 -services.loadbalancers 0 2 -services.nodeports 0 0 -``` - -As you can see, the pod that was created is consuming explicit amounts of compute resources, and the usage is being -tracked by Kubernetes properly. - -## Step 5: Advanced quota scopes - -Let's imagine you did not want to specify default compute resource consumption in your namespace. - -Instead, you want to let users run a specific number of `BestEffort` pods in their namespace to take -advantage of slack compute resources, and then require that users make an explicit resource request for -pods that require a higher quality of service. - -Let's create a new namespace with two quotas to demonstrate this behavior: - -```shell -$ kubectl create namespace quota-scopes -namespace "quota-scopes" created -$ kubectl create -f docs/admin/resourcequota/best-effort.yaml --namespace=quota-scopes -resourcequota "best-effort" created -$ kubectl create -f docs/admin/resourcequota/not-best-effort.yaml --namespace=quota-scopes -resourcequota "not-best-effort" created -$ kubectl describe quota --namespace=quota-scopes -Name: best-effort -Namespace: quota-scopes -Scopes: BestEffort - * Matches all pods that have best effort quality of service. -Resource Used Hard --------- ---- ---- -pods 0 10 - - -Name: not-best-effort -Namespace: quota-scopes -Scopes: NotBestEffort - * Matches all pods that do not have best effort quality of service. -Resource Used Hard --------- ---- ---- -limits.cpu 0 2 -limits.memory 0 2Gi -pods 0 4 -requests.cpu 0 1 -requests.memory 0 1Gi -``` - -In this scenario, a pod that makes no compute resource requests will be tracked by the `best-effort` quota. - -A pod that does make compute resource requests will be tracked by the `not-best-effort` quota. - -Let's demonstrate this by creating two deployments: - -```shell -$ kubectl run best-effort-nginx --image=nginx --replicas=8 --namespace=quota-scopes -deployment "best-effort-nginx" created -$ kubectl run not-best-effort-nginx \ - --image=nginx \ - --replicas=2 \ - --requests=cpu=100m,memory=256Mi \ - --limits=cpu=200m,memory=512Mi \ - --namespace=quota-scopes -deployment "not-best-effort-nginx" created -``` - -Even though no default limits were specified, the `best-effort-nginx` deployment will create -all 8 pods. This is because it is tracked by the `best-effort` quota, and the `not-best-effort` -quota will just ignore it. The `not-best-effort` quota will track the `not-best-effort-nginx` -deployment since it creates pods with `Burstable` quality of service. - -Let's list the pods in the namespace: - -```shell -$ kubectl get pods --namespace=quota-scopes -NAME READY STATUS RESTARTS AGE -best-effort-nginx-3488455095-2qb41 1/1 Running 0 51s -best-effort-nginx-3488455095-3go7n 1/1 Running 0 51s -best-effort-nginx-3488455095-9o2xg 1/1 Running 0 51s -best-effort-nginx-3488455095-eyg40 1/1 Running 0 51s -best-effort-nginx-3488455095-gcs3v 1/1 Running 0 51s -best-effort-nginx-3488455095-rq8p1 1/1 Running 0 51s -best-effort-nginx-3488455095-udhhd 1/1 Running 0 51s -best-effort-nginx-3488455095-zmk12 1/1 Running 0 51s -not-best-effort-nginx-2204666826-7sl61 1/1 Running 0 23s -not-best-effort-nginx-2204666826-ke746 1/1 Running 0 23s -``` - -As you can see, all 10 pods have been allowed to be created. - -Let's describe current quota usage in the namespace: - -```shell -$ kubectl describe quota --namespace=quota-scopes -Name: best-effort -Namespace: quota-scopes -Scopes: BestEffort - * Matches all pods that have best effort quality of service. -Resource Used Hard --------- ---- ---- -pods 8 10 - - -Name: not-best-effort -Namespace: quota-scopes -Scopes: NotBestEffort - * Matches all pods that do not have best effort quality of service. -Resource Used Hard --------- ---- ---- -limits.cpu 400m 2 -limits.memory 1Gi 2Gi -pods 2 4 -requests.cpu 200m 1 -requests.memory 512Mi 1Gi -``` - -As you can see, the `best-effort` quota has tracked the usage for the 8 pods we created in -the `best-effort-nginx` deployment, and the `not-best-effort` quota has tracked the usage for -the 2 pods we created in the `not-best-effort-nginx` quota. - -Scopes provide a mechanism to subdivide the set of resources that are tracked by -any quota document to allow greater flexibility in how operators deploy and track resource -consumption. - -In addition to `BestEffort` and `NotBestEffort` scopes, there are scopes to restrict -long-running versus time-bound pods. The `Terminating` scope will match any pod -where `spec.activeDeadlineSeconds is not nil`. The `NotTerminating` scope will match any pod -where `spec.activeDeadlineSeconds is nil`. These scopes allow you to quota pods based on their -anticipated permanence on a node in your cluster. - -## Summary - -Actions that consume node resources for cpu and memory can be subject to hard quota limits defined by the namespace quota. - -Any action that consumes those resources can be tweaked, or can pick up namespace level defaults to meet your end goal. - -Quota can be apportioned based on quality of service and anticipated permanence on a node in your cluster. +[Applying Resource Quotas and Limits](/docs/tasks/configure-pod-container/apply-resource-quota-limit/) diff --git a/docs/tasks/configure-pod-container/apply-resource-quota-limit.md b/docs/tasks/configure-pod-container/apply-resource-quota-limit.md new file mode 100644 index 0000000000..1120e7550d --- /dev/null +++ b/docs/tasks/configure-pod-container/apply-resource-quota-limit.md @@ -0,0 +1,366 @@ +--- +assignees: +- derekwaynecarr +- janetkuo +title: Applying Resource Quotas and Limits +--- + +This example demonstrates a typical setup to control for resource usage in a namespace. + +It demonstrates using the following resources: + +* [Namespace](/docs/admin/namespaces) +* [Resource Quota](/docs/admin/resourcequota/) +* [Limit Range](/docs/admin/limitrange/) + +This example assumes you have a functional Kubernetes setup. + +## Scenario + +The cluster-admin is operating a cluster on behalf of a user population and the cluster-admin +wants to control the amount of resources that can be consumed in a particular namespace to promote +fair sharing of the cluster and control cost. + +The cluster-admin has the following goals: + +* Limit the amount of compute resource for running pods +* Limit the number of persistent volume claims to control access to storage +* Limit the number of load balancers to control cost +* Prevent the use of node ports to preserve scarce resources +* Provide default compute resource requests to enable better scheduling decisions + +## Step 1: Create a namespace + +This example will work in a custom namespace to demonstrate the concepts involved. + +Let's create a new namespace called quota-example: + +```shell +$ kubectl create -f docs/admin/resourcequota/namespace.yaml +namespace "quota-example" created +$ kubectl get namespaces +NAME STATUS AGE +default Active 2m +kube-system Active 2m +quota-example Active 39s +``` + +## Step 2: Apply an object-count quota to the namespace + +The cluster-admin wants to control the following resources: + +* persistent volume claims +* load balancers +* node ports + +Let's create a simple quota that controls object counts for those resource types in this namespace. + +```shell +$ kubectl create -f docs/admin/resourcequota/object-counts.yaml --namespace=quota-example +resourcequota "object-counts" created +``` + +The quota system will observe that a quota has been created, and will calculate consumption +in the namespace in response. This should happen quickly. + +Let's describe the quota to see what is currently being consumed in this namespace: + +```shell +$ kubectl describe quota object-counts --namespace=quota-example +Name: object-counts +Namespace: quota-example +Resource Used Hard +-------- ---- ---- +persistentvolumeclaims 0 2 +services.loadbalancers 0 2 +services.nodeports 0 0 +``` + +The quota system will now prevent users from creating more than the specified amount for each resource. + + +## Step 3: Apply a compute-resource quota to the namespace + +To limit the amount of compute resource that can be consumed in this namespace, +let's create a quota that tracks compute resources. + +```shell +$ kubectl create -f docs/admin/resourcequota/compute-resources.yaml --namespace=quota-example +resourcequota "compute-resources" created +``` + +Let's describe the quota to see what is currently being consumed in this namespace: + +```shell +$ kubectl describe quota compute-resources --namespace=quota-example +Name: compute-resources +Namespace: quota-example +Resource Used Hard +-------- ---- ---- +limits.cpu 0 2 +limits.memory 0 2Gi +pods 0 4 +requests.cpu 0 1 +requests.memory 0 1Gi +``` + +The quota system will now prevent the namespace from having more than 4 non-terminal pods. In +addition, it will enforce that each container in a pod makes a `request` and defines a `limit` for +`cpu` and `memory`. + +## Step 4: Applying default resource requests and limits + +Pod authors rarely specify resource requests and limits for their pods. + +Since we applied a quota to our project, let's see what happens when an end-user creates a pod that has unbounded +cpu and memory by creating an nginx container. + +To demonstrate, lets create a deployment that runs nginx: + +```shell +$ kubectl run nginx --image=nginx --replicas=1 --namespace=quota-example +deployment "nginx" created +``` + +Now let's look at the pods that were created. + +```shell +$ kubectl get pods --namespace=quota-example +``` + +What happened? I have no pods! Let's describe the deployment to get a view of what is happening. + +```shell +$ kubectl describe deployment nginx --namespace=quota-example +Name: nginx +Namespace: quota-example +CreationTimestamp: Mon, 06 Jun 2016 16:11:37 -0400 +Labels: run=nginx +Selector: run=nginx +Replicas: 0 updated | 1 total | 0 available | 1 unavailable +StrategyType: RollingUpdate +MinReadySeconds: 0 +RollingUpdateStrategy: 1 max unavailable, 1 max surge +OldReplicaSets: +NewReplicaSet: nginx-3137573019 (0/1 replicas created) +... +``` + +A deployment created a corresponding replica set and attempted to size it to create a single pod. + +Let's look at the replica set to get more detail. + +```shell +$ kubectl describe rs nginx-3137573019 --namespace=quota-example +Name: nginx-3137573019 +Namespace: quota-example +Image(s): nginx +Selector: pod-template-hash=3137573019,run=nginx +Labels: pod-template-hash=3137573019 + run=nginx +Replicas: 0 current / 1 desired +Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed +No volumes. +Events: + FirstSeen LastSeen Count From SubobjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 4m 7s 11 {replicaset-controller } Warning FailedCreate Error creating: pods "nginx-3137573019-" is forbidden: Failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory +``` + +The Kubernetes API server is rejecting the replica set requests to create a pod because our pods +do not specify `requests` or `limits` for `cpu` and `memory`. + +So let's set some default values for the amount of `cpu` and `memory` a pod can consume: + +```shell +$ kubectl create -f docs/admin/resourcequota/limits.yaml --namespace=quota-example +limitrange "limits" created +$ kubectl describe limits limits --namespace=quota-example +Name: limits +Namespace: quota-example +Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio +---- -------- --- --- --------------- ------------- ----------------------- +Container memory - - 256Mi 512Mi - +Container cpu - - 100m 200m - +``` + +If the Kubernetes API server observes a request to create a pod in this namespace, and the containers +in that pod do not make any compute resource requests, a default request and default limit will be applied +as part of admission control. + +In this example, each pod created will have compute resources equivalent to the following: + +```shell +$ kubectl run nginx \ + --image=nginx \ + --replicas=1 \ + --requests=cpu=100m,memory=256Mi \ + --limits=cpu=200m,memory=512Mi \ + --namespace=quota-example +``` + +Now that we have applied default compute resources for our namespace, our replica set should be able to create +its pods. + +```shell +$ kubectl get pods --namespace=quota-example +NAME READY STATUS RESTARTS AGE +nginx-3137573019-fvrig 1/1 Running 0 6m +``` + +And if we print out our quota usage in the namespace: + +```shell +$ kubectl describe quota --namespace=quota-example +Name: compute-resources +Namespace: quota-example +Resource Used Hard +-------- ---- ---- +limits.cpu 200m 2 +limits.memory 512Mi 2Gi +pods 1 4 +requests.cpu 100m 1 +requests.memory 256Mi 1Gi + + +Name: object-counts +Namespace: quota-example +Resource Used Hard +-------- ---- ---- +persistentvolumeclaims 0 2 +services.loadbalancers 0 2 +services.nodeports 0 0 +``` + +As you can see, the pod that was created is consuming explicit amounts of compute resources, and the usage is being +tracked by Kubernetes properly. + +## Step 5: Advanced quota scopes + +Let's imagine you did not want to specify default compute resource consumption in your namespace. + +Instead, you want to let users run a specific number of `BestEffort` pods in their namespace to take +advantage of slack compute resources, and then require that users make an explicit resource request for +pods that require a higher quality of service. + +Let's create a new namespace with two quotas to demonstrate this behavior: + +```shell +$ kubectl create namespace quota-scopes +namespace "quota-scopes" created +$ kubectl create -f docs/admin/resourcequota/best-effort.yaml --namespace=quota-scopes +resourcequota "best-effort" created +$ kubectl create -f docs/admin/resourcequota/not-best-effort.yaml --namespace=quota-scopes +resourcequota "not-best-effort" created +$ kubectl describe quota --namespace=quota-scopes +Name: best-effort +Namespace: quota-scopes +Scopes: BestEffort + * Matches all pods that have best effort quality of service. +Resource Used Hard +-------- ---- ---- +pods 0 10 + + +Name: not-best-effort +Namespace: quota-scopes +Scopes: NotBestEffort + * Matches all pods that do not have best effort quality of service. +Resource Used Hard +-------- ---- ---- +limits.cpu 0 2 +limits.memory 0 2Gi +pods 0 4 +requests.cpu 0 1 +requests.memory 0 1Gi +``` + +In this scenario, a pod that makes no compute resource requests will be tracked by the `best-effort` quota. + +A pod that does make compute resource requests will be tracked by the `not-best-effort` quota. + +Let's demonstrate this by creating two deployments: + +```shell +$ kubectl run best-effort-nginx --image=nginx --replicas=8 --namespace=quota-scopes +deployment "best-effort-nginx" created +$ kubectl run not-best-effort-nginx \ + --image=nginx \ + --replicas=2 \ + --requests=cpu=100m,memory=256Mi \ + --limits=cpu=200m,memory=512Mi \ + --namespace=quota-scopes +deployment "not-best-effort-nginx" created +``` + +Even though no default limits were specified, the `best-effort-nginx` deployment will create +all 8 pods. This is because it is tracked by the `best-effort` quota, and the `not-best-effort` +quota will just ignore it. The `not-best-effort` quota will track the `not-best-effort-nginx` +deployment since it creates pods with `Burstable` quality of service. + +Let's list the pods in the namespace: + +```shell +$ kubectl get pods --namespace=quota-scopes +NAME READY STATUS RESTARTS AGE +best-effort-nginx-3488455095-2qb41 1/1 Running 0 51s +best-effort-nginx-3488455095-3go7n 1/1 Running 0 51s +best-effort-nginx-3488455095-9o2xg 1/1 Running 0 51s +best-effort-nginx-3488455095-eyg40 1/1 Running 0 51s +best-effort-nginx-3488455095-gcs3v 1/1 Running 0 51s +best-effort-nginx-3488455095-rq8p1 1/1 Running 0 51s +best-effort-nginx-3488455095-udhhd 1/1 Running 0 51s +best-effort-nginx-3488455095-zmk12 1/1 Running 0 51s +not-best-effort-nginx-2204666826-7sl61 1/1 Running 0 23s +not-best-effort-nginx-2204666826-ke746 1/1 Running 0 23s +``` + +As you can see, all 10 pods have been allowed to be created. + +Let's describe current quota usage in the namespace: + +```shell +$ kubectl describe quota --namespace=quota-scopes +Name: best-effort +Namespace: quota-scopes +Scopes: BestEffort + * Matches all pods that have best effort quality of service. +Resource Used Hard +-------- ---- ---- +pods 8 10 + + +Name: not-best-effort +Namespace: quota-scopes +Scopes: NotBestEffort + * Matches all pods that do not have best effort quality of service. +Resource Used Hard +-------- ---- ---- +limits.cpu 400m 2 +limits.memory 1Gi 2Gi +pods 2 4 +requests.cpu 200m 1 +requests.memory 512Mi 1Gi +``` + +As you can see, the `best-effort` quota has tracked the usage for the 8 pods we created in +the `best-effort-nginx` deployment, and the `not-best-effort` quota has tracked the usage for +the 2 pods we created in the `not-best-effort-nginx` quota. + +Scopes provide a mechanism to subdivide the set of resources that are tracked by +any quota document to allow greater flexibility in how operators deploy and track resource +consumption. + +In addition to `BestEffort` and `NotBestEffort` scopes, there are scopes to restrict +long-running versus time-bound pods. The `Terminating` scope will match any pod +where `spec.activeDeadlineSeconds is not nil`. The `NotTerminating` scope will match any pod +where `spec.activeDeadlineSeconds is nil`. These scopes allow you to quota pods based on their +anticipated permanence on a node in your cluster. + +## Summary + +Actions that consume node resources for cpu and memory can be subject to hard quota limits defined by the namespace quota. + +Any action that consumes those resources can be tweaked, or can pick up namespace level defaults to meet your end goal. + +Quota can be apportioned based on quality of service and anticipated permanence on a node in your cluster. From 8e3132bbe46363377f43ece03eedc4771e23568b Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 11:27:58 -0700 Subject: [PATCH 389/407] Quota2 (#2838) * Move Guide topic: Limit range. * Move Guide topic: Resource Quota concept. --- _data/concepts.yml | 1 + docs/admin/resourcequota/index.md | 235 +---------------------- docs/concepts/policy/resource-quotas.md | 240 ++++++++++++++++++++++++ 3 files changed, 243 insertions(+), 233 deletions(-) create mode 100644 docs/concepts/policy/resource-quotas.md diff --git a/_data/concepts.yml b/_data/concepts.yml index d625c01753..136d9520e7 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -51,3 +51,4 @@ toc: - title: Policies section: - docs/concepts/policy/container-capabilities.md + - docs/concepts/policy/resource-quotas.md diff --git a/docs/admin/resourcequota/index.md b/docs/admin/resourcequota/index.md index 24d7a09246..f90049dfdc 100644 --- a/docs/admin/resourcequota/index.md +++ b/docs/admin/resourcequota/index.md @@ -4,237 +4,6 @@ assignees: title: Resource Quotas --- -When several users or teams share a cluster with a fixed number of nodes, -there is a concern that one team could use more than its fair share of resources. +{% include user-guide-content-moved.md %} -Resource quotas are a tool for administrators to address this concern. - -A resource quota, defined by a `ResourceQuota` object, provides constraints that limit -aggregate resource consumption per namespace. It can limit the quantity of objects that can -be created in a namespace by type, as well as the total amount of compute resources that may -be consumed by resources in that project. - -Resource quotas work like this: - -- Different teams work in different namespaces. Currently this is voluntary, but - support for making this mandatory via ACLs is planned. -- The administrator creates one or more Resource Quota objects for each namespace. -- Users create resources (pods, services, etc.) in the namespace, and the quota system - tracks usage to ensure it does not exceed hard resource limits defined in a Resource Quota. -- If creating or updating a resource violates a quota constraint, the request will fail with HTTP - status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated. -- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify - requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use - the LimitRange admission controller to force defaults for pods that make no compute resource requirements. - See the [walkthrough](/docs/admin/resourcequota/walkthrough/) for an example to avoid this problem. - -Examples of policies that could be created using namespaces and quotas are: - -- In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 Gib and 10 cores, - let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation. -- Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace - use any amount. - -In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, -there may be contention for resources. This is handled on a first-come-first-served basis. - -Neither contention nor changes to quota will affect already created resources. - -## Enabling Resource Quota - -Resource Quota support is enabled by default for many Kubernetes distributions. It is -enabled when the apiserver `--admission-control=` flag has `ResourceQuota` as -one of its arguments. - -Resource Quota is enforced in a particular namespace when there is a -`ResourceQuota` object in that namespace. There should be at most one -`ResourceQuota` object in a namespace. - -## Compute Resource Quota - -You can limit the total sum of [compute resources](/docs/user-guide/compute-resources) that can be requested in a given namespace. - -The following resource types are supported: - -| Resource Name | Description | -| --------------------- | ----------------------------------------------------------- | -| `cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | -| `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. | -| `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. | -| `memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | -| `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | -| `requests.memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | - -## Storage Resource Quota - -You can limit the total sum of [storage resources](/docs/user-guide/persistent-volumes) that can be requested in a given namespace. - -In addition, you can limit consumption of storage resources based on associated storage-class. - -| Resource Name | Description | -| --------------------- | ----------------------------------------------------------- | -| `requests.storage` | Across all persistent volume claims, the sum of storage requests cannot exceed this value. | -| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | -| `.storageclass.storage.k8s.io/requests.storage` | Across all persistent volume claims associated with the storage-class-name, the sum of storage requests cannot exceed this value. | -| `.storageclass.storage.k8s.io/persistentvolumeclaims` | Across all persistent volume claims associated with the storage-class-name, the total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | - -For example, if an operator wants to quota storage with `gold` storage class separate from `bronze` storage class, the operator can -define a quota as follows: - -* `gold.storageclass.storage.k8s.io/requests.storage: 500Gi` -* `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi` - -## Object Count Quota - -The number of objects of a given type can be restricted. The following types -are supported: - -| Resource Name | Description | -| ------------------------------- | ------------------------------------------------- | -| `configmaps` | The total number of config maps that can exist in the namespace. | -| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | -| `pods` | The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `status.phase in (Failed, Succeeded)` is true. | -| `replicationcontrollers` | The total number of replication controllers that can exist in the namespace. | -| `resourcequotas` | The total number of [resource quotas](/docs/admin/admission-controllers/#resourcequota) that can exist in the namespace. | -| `services` | The total number of services that can exist in the namespace. | -| `services.loadbalancers` | The total number of services of type load balancer that can exist in the namespace. | -| `services.nodeports` | The total number of services of type node port that can exist in the namespace. | -| `secrets` | The total number of secrets that can exist in the namespace. | - -For example, `pods` quota counts and enforces a maximum on the number of `pods` -created in a single namespace. - -You might want to set a pods quota on a namespace -to avoid the case where a user creates many small pods and exhausts the cluster's -supply of Pod IPs. - -## Quota Scopes - -Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches -the intersection of enumerated scopes. - -When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. -Resources specified on the quota outside of the allowed set results in a validation error. - -| Scope | Description | -| ----- | ----------- | -| `Terminating` | Match pods where `spec.activeDeadlineSeconds >= 0` | -| `NotTerminating` | Match pods where `spec.activeDeadlineSeconds is nil` | -| `BestEffort` | Match pods that have best effort quality of service. | -| `NotBestEffort` | Match pods that do not have best effort quality of service. | - -The `BestEffort` scope restricts a quota to tracking the following resource: `pods` - -The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota to tracking the following resources: - -* `cpu` -* `limits.cpu` -* `limits.memory` -* `memory` -* `pods` -* `requests.cpu` -* `requests.memory` - -## Requests vs Limits - -When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. -The quota can be configured to quota either value. - -If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming -container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`, -then it requires that every incoming container specifies an explicit limit for those resources. - -## Viewing and Setting Quotas - -Kubectl supports creating, updating, and viewing quotas: - -```shell -$ kubectl create namespace myspace - -$ cat < compute-resources.yaml -apiVersion: v1 -kind: ResourceQuota -metadata: - name: compute-resources -spec: - hard: - pods: "4" - requests.cpu: "1" - requests.memory: 1Gi - limits.cpu: "2" - limits.memory: 2Gi -EOF -$ kubectl create -f ./compute-resources.yaml --namespace=myspace - -$ cat < object-counts.yaml -apiVersion: v1 -kind: ResourceQuota -metadata: - name: object-counts -spec: - hard: - configmaps: "10" - persistentvolumeclaims: "4" - replicationcontrollers: "20" - secrets: "10" - services: "10" - services.loadbalancers: "2" -EOF -$ kubectl create -f ./object-counts.yaml --namespace=myspace - -$ kubectl get quota --namespace=myspace -NAME AGE -compute-resources 30s -object-counts 32s - -$ kubectl describe quota compute-resources --namespace=myspace -Name: compute-resources -Namespace: myspace -Resource Used Hard --------- ---- ---- -limits.cpu 0 2 -limits.memory 0 2Gi -pods 0 4 -requests.cpu 0 1 -requests.memory 0 1Gi - -$ kubectl describe quota object-counts --namespace=myspace -Name: object-counts -Namespace: myspace -Resource Used Hard --------- ---- ---- -configmaps 0 10 -persistentvolumeclaims 0 4 -replicationcontrollers 0 20 -secrets 1 10 -services 0 10 -services.loadbalancers 0 2 -``` - -## Quota and Cluster Capacity - -Resource Quota objects are independent of the Cluster Capacity. They are -expressed in absolute units. So, if you add nodes to your cluster, this does *not* -automatically give each namespace the ability to consume more resources. - -Sometimes more complex policies may be desired, such as: - - - proportionally divide total cluster resources among several teams. - - allow each tenant to grow resource usage as needed, but have a generous - limit to prevent accidental resource exhaustion. - - detect demand from one namespace, add nodes, and increase quota. - -Such policies could be implemented using ResourceQuota as a building-block, by -writing a 'controller' which watches the quota usage and adjusts the quota -hard limits of each namespace according to other signals. - -Note that resource quota divides up aggregate cluster resources, but it creates no -restrictions around nodes: pods from several namespaces may run on the same node. - -## Example - -See a [detailed example for how to use resource quota](/docs/admin/resourcequota/walkthrough/). - -## Read More - -See [ResourceQuota design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/admission_control_resource_quota.md) for more information. +[Resource Quotas](/docs/concepts/policy/resource-quotas/) diff --git a/docs/concepts/policy/resource-quotas.md b/docs/concepts/policy/resource-quotas.md new file mode 100644 index 0000000000..24d7a09246 --- /dev/null +++ b/docs/concepts/policy/resource-quotas.md @@ -0,0 +1,240 @@ +--- +assignees: +- derekwaynecarr +title: Resource Quotas +--- + +When several users or teams share a cluster with a fixed number of nodes, +there is a concern that one team could use more than its fair share of resources. + +Resource quotas are a tool for administrators to address this concern. + +A resource quota, defined by a `ResourceQuota` object, provides constraints that limit +aggregate resource consumption per namespace. It can limit the quantity of objects that can +be created in a namespace by type, as well as the total amount of compute resources that may +be consumed by resources in that project. + +Resource quotas work like this: + +- Different teams work in different namespaces. Currently this is voluntary, but + support for making this mandatory via ACLs is planned. +- The administrator creates one or more Resource Quota objects for each namespace. +- Users create resources (pods, services, etc.) in the namespace, and the quota system + tracks usage to ensure it does not exceed hard resource limits defined in a Resource Quota. +- If creating or updating a resource violates a quota constraint, the request will fail with HTTP + status code `403 FORBIDDEN` with a message explaining the constraint that would have been violated. +- If quota is enabled in a namespace for compute resources like `cpu` and `memory`, users must specify + requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use + the LimitRange admission controller to force defaults for pods that make no compute resource requirements. + See the [walkthrough](/docs/admin/resourcequota/walkthrough/) for an example to avoid this problem. + +Examples of policies that could be created using namespaces and quotas are: + +- In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 Gib and 10 cores, + let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation. +- Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace + use any amount. + +In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, +there may be contention for resources. This is handled on a first-come-first-served basis. + +Neither contention nor changes to quota will affect already created resources. + +## Enabling Resource Quota + +Resource Quota support is enabled by default for many Kubernetes distributions. It is +enabled when the apiserver `--admission-control=` flag has `ResourceQuota` as +one of its arguments. + +Resource Quota is enforced in a particular namespace when there is a +`ResourceQuota` object in that namespace. There should be at most one +`ResourceQuota` object in a namespace. + +## Compute Resource Quota + +You can limit the total sum of [compute resources](/docs/user-guide/compute-resources) that can be requested in a given namespace. + +The following resource types are supported: + +| Resource Name | Description | +| --------------------- | ----------------------------------------------------------- | +| `cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | +| `limits.cpu` | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. | +| `limits.memory` | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. | +| `memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | +| `requests.cpu` | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value. | +| `requests.memory` | Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value. | + +## Storage Resource Quota + +You can limit the total sum of [storage resources](/docs/user-guide/persistent-volumes) that can be requested in a given namespace. + +In addition, you can limit consumption of storage resources based on associated storage-class. + +| Resource Name | Description | +| --------------------- | ----------------------------------------------------------- | +| `requests.storage` | Across all persistent volume claims, the sum of storage requests cannot exceed this value. | +| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | +| `.storageclass.storage.k8s.io/requests.storage` | Across all persistent volume claims associated with the storage-class-name, the sum of storage requests cannot exceed this value. | +| `.storageclass.storage.k8s.io/persistentvolumeclaims` | Across all persistent volume claims associated with the storage-class-name, the total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | + +For example, if an operator wants to quota storage with `gold` storage class separate from `bronze` storage class, the operator can +define a quota as follows: + +* `gold.storageclass.storage.k8s.io/requests.storage: 500Gi` +* `bronze.storageclass.storage.k8s.io/requests.storage: 100Gi` + +## Object Count Quota + +The number of objects of a given type can be restricted. The following types +are supported: + +| Resource Name | Description | +| ------------------------------- | ------------------------------------------------- | +| `configmaps` | The total number of config maps that can exist in the namespace. | +| `persistentvolumeclaims` | The total number of [persistent volume claims](/docs/user-guide/persistent-volumes/#persistentvolumeclaims) that can exist in the namespace. | +| `pods` | The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `status.phase in (Failed, Succeeded)` is true. | +| `replicationcontrollers` | The total number of replication controllers that can exist in the namespace. | +| `resourcequotas` | The total number of [resource quotas](/docs/admin/admission-controllers/#resourcequota) that can exist in the namespace. | +| `services` | The total number of services that can exist in the namespace. | +| `services.loadbalancers` | The total number of services of type load balancer that can exist in the namespace. | +| `services.nodeports` | The total number of services of type node port that can exist in the namespace. | +| `secrets` | The total number of secrets that can exist in the namespace. | + +For example, `pods` quota counts and enforces a maximum on the number of `pods` +created in a single namespace. + +You might want to set a pods quota on a namespace +to avoid the case where a user creates many small pods and exhausts the cluster's +supply of Pod IPs. + +## Quota Scopes + +Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches +the intersection of enumerated scopes. + +When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. +Resources specified on the quota outside of the allowed set results in a validation error. + +| Scope | Description | +| ----- | ----------- | +| `Terminating` | Match pods where `spec.activeDeadlineSeconds >= 0` | +| `NotTerminating` | Match pods where `spec.activeDeadlineSeconds is nil` | +| `BestEffort` | Match pods that have best effort quality of service. | +| `NotBestEffort` | Match pods that do not have best effort quality of service. | + +The `BestEffort` scope restricts a quota to tracking the following resource: `pods` + +The `Terminating`, `NotTerminating`, and `NotBestEffort` scopes restrict a quota to tracking the following resources: + +* `cpu` +* `limits.cpu` +* `limits.memory` +* `memory` +* `pods` +* `requests.cpu` +* `requests.memory` + +## Requests vs Limits + +When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. +The quota can be configured to quota either value. + +If the quota has a value specified for `requests.cpu` or `requests.memory`, then it requires that every incoming +container makes an explicit request for those resources. If the quota has a value specified for `limits.cpu` or `limits.memory`, +then it requires that every incoming container specifies an explicit limit for those resources. + +## Viewing and Setting Quotas + +Kubectl supports creating, updating, and viewing quotas: + +```shell +$ kubectl create namespace myspace + +$ cat < compute-resources.yaml +apiVersion: v1 +kind: ResourceQuota +metadata: + name: compute-resources +spec: + hard: + pods: "4" + requests.cpu: "1" + requests.memory: 1Gi + limits.cpu: "2" + limits.memory: 2Gi +EOF +$ kubectl create -f ./compute-resources.yaml --namespace=myspace + +$ cat < object-counts.yaml +apiVersion: v1 +kind: ResourceQuota +metadata: + name: object-counts +spec: + hard: + configmaps: "10" + persistentvolumeclaims: "4" + replicationcontrollers: "20" + secrets: "10" + services: "10" + services.loadbalancers: "2" +EOF +$ kubectl create -f ./object-counts.yaml --namespace=myspace + +$ kubectl get quota --namespace=myspace +NAME AGE +compute-resources 30s +object-counts 32s + +$ kubectl describe quota compute-resources --namespace=myspace +Name: compute-resources +Namespace: myspace +Resource Used Hard +-------- ---- ---- +limits.cpu 0 2 +limits.memory 0 2Gi +pods 0 4 +requests.cpu 0 1 +requests.memory 0 1Gi + +$ kubectl describe quota object-counts --namespace=myspace +Name: object-counts +Namespace: myspace +Resource Used Hard +-------- ---- ---- +configmaps 0 10 +persistentvolumeclaims 0 4 +replicationcontrollers 0 20 +secrets 1 10 +services 0 10 +services.loadbalancers 0 2 +``` + +## Quota and Cluster Capacity + +Resource Quota objects are independent of the Cluster Capacity. They are +expressed in absolute units. So, if you add nodes to your cluster, this does *not* +automatically give each namespace the ability to consume more resources. + +Sometimes more complex policies may be desired, such as: + + - proportionally divide total cluster resources among several teams. + - allow each tenant to grow resource usage as needed, but have a generous + limit to prevent accidental resource exhaustion. + - detect demand from one namespace, add nodes, and increase quota. + +Such policies could be implemented using ResourceQuota as a building-block, by +writing a 'controller' which watches the quota usage and adjusts the quota +hard limits of each namespace according to other signals. + +Note that resource quota divides up aggregate cluster resources, but it creates no +restrictions around nodes: pods from several namespaces may run on the same node. + +## Example + +See a [detailed example for how to use resource quota](/docs/admin/resourcequota/walkthrough/). + +## Read More + +See [ResourceQuota design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/admission_control_resource_quota.md) for more information. From e803967e16594b5c17851ff949a6ce17838dec2e Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 11:50:04 -0700 Subject: [PATCH 390/407] Dns (#2839) * Move Guide topic: Limit range. * Move Guide topic: DNS. --- _data/concepts.yml | 4 + docs/admin/dns.md | 383 +---------------- .../services-networking/dns-pod-service.md | 389 ++++++++++++++++++ 3 files changed, 395 insertions(+), 381 deletions(-) create mode 100644 docs/concepts/services-networking/dns-pod-service.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 136d9520e7..21562255db 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -43,6 +43,10 @@ toc: - docs/concepts/cluster-administration/static-pod.md - docs/concepts/cluster-administration/sysctl-cluster.md +- title: Services, Load Balancing, and Networking + section: + - docs/concepts/services-networking/dns-pod-service.md + - title: Configuration section: - docs/concepts/configuration/container-command-args.md diff --git a/docs/admin/dns.md b/docs/admin/dns.md index 7a845c9f3f..84e97ce2bc 100644 --- a/docs/admin/dns.md +++ b/docs/admin/dns.md @@ -5,385 +5,6 @@ assignees: title: Using DNS Pods and Services --- -## Introduction +{% include user-guide-content-moved.md %} -As of Kubernetes 1.3, DNS is a built-in service launched automatically using the addon manager [cluster add-on](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/README.md). - -Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures -the kubelets to tell individual containers to use the DNS Service's IP to -resolve DNS names. - -## What things get DNS names? - -Every Service defined in the cluster (including the DNS server itself) is -assigned a DNS name. By default, a client Pod's DNS search list will -include the Pod's own namespace and the cluster's default domain. This is best -illustrated by example: - -Assume a Service named `foo` in the Kubernetes namespace `bar`. A Pod running -in namespace `bar` can look up this service by simply doing a DNS query for -`foo`. A Pod running in namespace `quux` can look up this service by doing a -DNS query for `foo.bar`. - -## Supported DNS schema - -The following sections detail the supported record types and layout that is -supported. Any other layout or names or queries that happen to work are -considered implementation details and are subject to change without warning. - -### Services - -#### A records - -"Normal" (not headless) Services are assigned a DNS A record for a name of the -form `my-svc.my-namespace.svc.cluster.local`. This resolves to the cluster IP -of the Service. - -"Headless" (without a cluster IP) Services are also assigned a DNS A record for -a name of the form `my-svc.my-namespace.svc.cluster.local`. Unlike normal -Services, this resolves to the set of IPs of the pods selected by the Service. -Clients are expected to consume the set or else use standard round-robin -selection from the set. - -### SRV records - -SRV Records are created for named ports that are part of normal or [Headless -Services](http://releases.k8s.io/docs/user-guide/services/#headless-services). -For each named port, the SRV record would have the form -`_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local`. -For a regular service, this resolves to the port number and the CNAME: -`my-svc.my-namespace.svc.cluster.local`. -For a headless service, this resolves to multiple answers, one for each pod -that is backing the service, and contains the port number and a CNAME of the pod -of the form `auto-generated-name.my-svc.my-namespace.svc.cluster.local`. - -### Backwards compatibility - -Previous versions of kube-dns made names of the form -`my-svc.my-namespace.cluster.local` (the 'svc' level was added later). This -is no longer supported. - -### Pods - -#### A Records - -When enabled, pods are assigned a DNS A record in the form of `pod-ip-address.my-namespace.pod.cluster.local`. - -For example, a pod with IP `1.2.3.4` in the namespace `default` with a DNS name of `cluster.local` would have an entry: `1-2-3-4.default.pod.cluster.local`. - -#### A Records and hostname based on Pod's hostname and subdomain fields - -Currently when a pod is created, its hostname is the Pod's `metadata.name` value. - -With v1.2, users can specify a Pod annotation, `pod.beta.kubernetes.io/hostname`, to specify what the Pod's hostname should be. -The Pod annotation, if specified, takes precedence over the Pod's name, to be the hostname of the pod. -For example, given a Pod with annotation `pod.beta.kubernetes.io/hostname: my-pod-name`, the Pod will have its hostname set to "my-pod-name". - -With v1.3, the PodSpec has a `hostname` field, which can be used to specify the Pod's hostname. This field value takes precedence over the -`pod.beta.kubernetes.io/hostname` annotation value. - -v1.2 introduces a beta feature where the user can specify a Pod annotation, `pod.beta.kubernetes.io/subdomain`, to specify the Pod's subdomain. -The final domain will be "...svc.". -For example, a Pod with the hostname annotation set to "foo", and the subdomain annotation set to "bar", in namespace "my-namespace", will have the FQDN "foo.bar.my-namespace.svc.cluster.local" - -With v1.3, the PodSpec has a `subdomain` field, which can be used to specify the Pod's subdomain. This field value takes precedence over the -`pod.beta.kubernetes.io/subdomain` annotation value. - -Example: - -```yaml -apiVersion: v1 -kind: Service -metadata: - name: default-subdomain -spec: - selector: - name: busybox - clusterIP: None - ports: - - name: foo # Actually, no port is needed. - port: 1234 - targetPort: 1234 ---- -apiVersion: v1 -kind: Pod -metadata: - name: busybox1 - labels: - name: busybox -spec: - hostname: busybox-1 - subdomain: default-subdomain - containers: - - image: busybox - command: - - sleep - - "3600" - name: busybox ---- -apiVersion: v1 -kind: Pod -metadata: - name: busybox2 - labels: - name: busybox -spec: - hostname: busybox-2 - subdomain: default-subdomain - containers: - - image: busybox - command: - - sleep - - "3600" - name: busybox -``` - -If there exists a headless service in the same namespace as the pod and with the same name as the subdomain, the cluster's KubeDNS Server also returns an A record for the Pod's fully qualified hostname. -Given a Pod with the hostname set to "busybox-1" and the subdomain set to "default-subdomain", and a headless Service named "default-subdomain" in the same namespace, the pod will see it's own FQDN as "busybox-1.default-subdomain.my-namespace.svc.cluster.local". DNS serves an A record at that name, pointing to the Pod's IP. Both pods "busybox1" and "busybox2" can have their distinct A records. - -As of Kubernetes v1.2, the Endpoints object also has the annotation `endpoints.beta.kubernetes.io/hostnames-map`. Its value is the json representation of map[string(IP)][endpoints.HostRecord], for example: '{"10.245.1.6":{HostName: "my-webserver"}}'. -If the Endpoints are for a headless service, an A record is created with the format ...svc. -For the example json, if endpoints are for a headless service named "bar", and one of the endpoints has IP "10.245.1.6", an A record is created with the name "my-webserver.bar.my-namespace.svc.cluster.local" and the A record lookup would return "10.245.1.6". -This endpoints annotation generally does not need to be specified by end-users, but can used by the internal service controller to deliver the aforementioned feature. - -With v1.3, The Endpoints object can specify the `hostname` for any endpoint, along with its IP. The hostname field takes precedence over the hostname value -that might have been specified via the `endpoints.beta.kubernetes.io/hostnames-map` annotation. - -With v1.3, the following annotations are deprecated: `pod.beta.kubernetes.io/hostname`, `pod.beta.kubernetes.io/subdomain`, `endpoints.beta.kubernetes.io/hostnames-map` - -## How do I test if it is working? - -### Create a simple Pod to use as a test environment - -Create a file named busybox.yaml with the -following contents: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: busybox - namespace: default -spec: - containers: - - image: busybox - command: - - sleep - - "3600" - imagePullPolicy: IfNotPresent - name: busybox - restartPolicy: Always -``` - -Then create a pod using this file: - -``` -kubectl create -f busybox.yaml -``` - -### Wait for this pod to go into the running state - -You can get its status with: -``` -kubectl get pods busybox -``` - -You should see: - -``` -NAME READY STATUS RESTARTS AGE -busybox 1/1 Running 0 -``` - -### Validate that DNS is working - -Once that pod is running, you can exec nslookup in that environment: - -``` -kubectl exec -ti busybox -- nslookup kubernetes.default -``` - -You should see something like: - -``` -Server: 10.0.0.10 -Address 1: 10.0.0.10 - -Name: kubernetes.default -Address 1: 10.0.0.1 -``` - -If you see that, DNS is working correctly. - -### Troubleshooting Tips - -If the nslookup command fails, check the following: - -#### Check the local DNS configuration first -Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" below for more information) - -``` -kubectl exec busybox cat /etc/resolv.conf -``` - -Verify that the search path and name server are set up like the following (note that search path may vary for different cloud providers): - -``` -search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal -nameserver 10.0.0.10 -options ndots:5 -``` - -#### Quick diagnosis - -Errors such as the following indicate a problem with the kube-dns add-on or associated Services: - -``` -$ kubectl exec -ti busybox -- nslookup kubernetes.default -Server: 10.0.0.10 -Address 1: 10.0.0.10 - -nslookup: can't resolve 'kubernetes.default' -``` - -or - -``` -$ kubectl exec -ti busybox -- nslookup kubernetes.default -Server: 10.0.0.10 -Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local - -nslookup: can't resolve 'kubernetes.default' -``` - -#### Check if the DNS pod is running - -Use the kubectl get pods command to verify that the DNS pod is running. - -``` -kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -``` - -You should see something like: - -``` -NAME READY STATUS RESTARTS AGE -... -kube-dns-v19-ezo1y 3/3 Running 0 1h -... -``` - -If you see that no pod is running or that the pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually. - -#### Check for Errors in the DNS pod - -Use `kubectl logs` command to see logs for the DNS daemons. - -``` -kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns -kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq -kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz -``` - -See if there is any suspicious log. W, E, F letter at the beginning represent Warning, Error and Failure. Please search for entries that have these as the logging level and use [kubernetes issues](https://github.com/kubernetes/kubernetes/issues) to report unexpected errors. - -#### Is DNS service up? - -Verify that the DNS service is up by using the `kubectl get service` command. - -``` -kubectl get svc --namespace=kube-system -``` - -You should see: - -``` -NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE -... -kube-dns 10.0.0.10 53/UDP,53/TCP 1h -... -``` - -If you have created the service or in the case it should be created by default but it does not appear, see this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information. - -#### Are DNS endpoints exposed? - -You can verify that DNS endpoints are exposed by using the `kubectl get endpoints` command. - -``` -kubectl get ep kube-dns --namespace=kube-system -``` - -You should see something like: -``` -NAME ENDPOINTS AGE -kube-dns 10.180.3.17:53,10.180.3.17:53 1h -``` - -If you do not see the endpoints, see endpoints section in the [debugging services documentation](http://kubernetes.io/docs/user-guide/debugging-services/). - -For additional Kubernetes DNS examples, see the [cluster-dns examples](https://github.com/kubernetes/kubernetes/tree/master/examples/cluster-dns) in the Kubernetes GitHub repository. - -## Kubernetes Federation (Multiple Zone support) - -Release 1.3 introduced Cluster Federation support for multi-site -Kubernetes installations. This required some minor -(backward-compatible) changes to the way -the Kubernetes cluster DNS server processes DNS queries, to facilitate -the lookup of federated services (which span multiple Kubernetes clusters). -See the [Cluster Federation Administrators' Guide](/docs/admin/federation) for more -details on Cluster Federation and multi-site support. - -## How it Works - -The running Kubernetes DNS pod holds 3 containers - kubedns, dnsmasq and a health check called healthz. -The kubedns process watches the Kubernetes master for changes in Services and Endpoints, and maintains -in-memory lookup structures to service DNS requests. The dnsmasq container adds DNS caching to improve -performance. The healthz container provides a single health check endpoint while performing dual healthchecks -(for dnsmasq and kubedns). - -The DNS pod is exposed as a Kubernetes Service with a static IP. Once assigned the -kubelet passes DNS configured using the `--cluster-dns=10.0.0.10` flag to each -container. - -DNS names also need domains. The local domain is configurable, in the kubelet using -the flag `--cluster-domain=` - -The Kubernetes cluster DNS server (based off the [SkyDNS](https://github.com/skynetservices/skydns) library) -supports forward lookups (A records), service lookups (SRV records) and reverse IP address lookups (PTR records). - -## Inheriting DNS from the node -When running a pod, kubelet will prepend the cluster DNS server and search -paths to the node's own DNS settings. If the node is able to resolve DNS names -specific to the larger environment, pods should be able to, also. See "Known -issues" below for a caveat. - -If you don't want this, or if you want a different DNS config for pods, you can -use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will -not inherit DNS. Setting it to a valid file path means that kubelet will use -this file instead of `/etc/resolv.conf` for DNS inheritance. - -## Known issues -Kubernetes installs do not configure the nodes' resolv.conf files to use the -cluster DNS by default, because that process is inherently distro-specific. -This should probably be implemented eventually. - -Linux's libc is impossibly stuck ([see this bug from -2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just -3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to -consume 1 `nameserver` record and 3 `search` records. This means that if a -local installation already uses 3 `nameserver`s or uses more than 3 `search`es, -some of those settings will be lost. As a partial workaround, the node can run -`dnsmasq` which will provide more `nameserver` entries, but not more `search` -entries. You can also use kubelet's `--resolv-conf` flag. - -If you are using Alpine version 3.3 or earlier as your base image, DNS may not -work properly owing to a known issue with Alpine. Check [here](https://github.com/kubernetes/kubernetes/issues/30215) -for more information. - -## References - -- [Docs for the DNS cluster addon](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/dns/README.md) - -## What's next -- [Autoscaling the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/). +[DNS Pods and Services](/docs/concepts/services-networking/dns-pod-service/) diff --git a/docs/concepts/services-networking/dns-pod-service.md b/docs/concepts/services-networking/dns-pod-service.md new file mode 100644 index 0000000000..ada734830d --- /dev/null +++ b/docs/concepts/services-networking/dns-pod-service.md @@ -0,0 +1,389 @@ +--- +assignees: +- davidopp +- thockin +title: DNS Pods and Services +--- + +## Introduction + +As of Kubernetes 1.3, DNS is a built-in service launched automatically using the addon manager [cluster add-on](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/README.md). + +Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures +the kubelets to tell individual containers to use the DNS Service's IP to +resolve DNS names. + +## What things get DNS names? + +Every Service defined in the cluster (including the DNS server itself) is +assigned a DNS name. By default, a client Pod's DNS search list will +include the Pod's own namespace and the cluster's default domain. This is best +illustrated by example: + +Assume a Service named `foo` in the Kubernetes namespace `bar`. A Pod running +in namespace `bar` can look up this service by simply doing a DNS query for +`foo`. A Pod running in namespace `quux` can look up this service by doing a +DNS query for `foo.bar`. + +## Supported DNS schema + +The following sections detail the supported record types and layout that is +supported. Any other layout or names or queries that happen to work are +considered implementation details and are subject to change without warning. + +### Services + +#### A records + +"Normal" (not headless) Services are assigned a DNS A record for a name of the +form `my-svc.my-namespace.svc.cluster.local`. This resolves to the cluster IP +of the Service. + +"Headless" (without a cluster IP) Services are also assigned a DNS A record for +a name of the form `my-svc.my-namespace.svc.cluster.local`. Unlike normal +Services, this resolves to the set of IPs of the pods selected by the Service. +Clients are expected to consume the set or else use standard round-robin +selection from the set. + +### SRV records + +SRV Records are created for named ports that are part of normal or [Headless +Services](http://releases.k8s.io/docs/user-guide/services/#headless-services). +For each named port, the SRV record would have the form +`_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local`. +For a regular service, this resolves to the port number and the CNAME: +`my-svc.my-namespace.svc.cluster.local`. +For a headless service, this resolves to multiple answers, one for each pod +that is backing the service, and contains the port number and a CNAME of the pod +of the form `auto-generated-name.my-svc.my-namespace.svc.cluster.local`. + +### Backwards compatibility + +Previous versions of kube-dns made names of the form +`my-svc.my-namespace.cluster.local` (the 'svc' level was added later). This +is no longer supported. + +### Pods + +#### A Records + +When enabled, pods are assigned a DNS A record in the form of `pod-ip-address.my-namespace.pod.cluster.local`. + +For example, a pod with IP `1.2.3.4` in the namespace `default` with a DNS name of `cluster.local` would have an entry: `1-2-3-4.default.pod.cluster.local`. + +#### A Records and hostname based on Pod's hostname and subdomain fields + +Currently when a pod is created, its hostname is the Pod's `metadata.name` value. + +With v1.2, users can specify a Pod annotation, `pod.beta.kubernetes.io/hostname`, to specify what the Pod's hostname should be. +The Pod annotation, if specified, takes precedence over the Pod's name, to be the hostname of the pod. +For example, given a Pod with annotation `pod.beta.kubernetes.io/hostname: my-pod-name`, the Pod will have its hostname set to "my-pod-name". + +With v1.3, the PodSpec has a `hostname` field, which can be used to specify the Pod's hostname. This field value takes precedence over the +`pod.beta.kubernetes.io/hostname` annotation value. + +v1.2 introduces a beta feature where the user can specify a Pod annotation, `pod.beta.kubernetes.io/subdomain`, to specify the Pod's subdomain. +The final domain will be "...svc.". +For example, a Pod with the hostname annotation set to "foo", and the subdomain annotation set to "bar", in namespace "my-namespace", will have the FQDN "foo.bar.my-namespace.svc.cluster.local" + +With v1.3, the PodSpec has a `subdomain` field, which can be used to specify the Pod's subdomain. This field value takes precedence over the +`pod.beta.kubernetes.io/subdomain` annotation value. + +Example: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: default-subdomain +spec: + selector: + name: busybox + clusterIP: None + ports: + - name: foo # Actually, no port is needed. + port: 1234 + targetPort: 1234 +--- +apiVersion: v1 +kind: Pod +metadata: + name: busybox1 + labels: + name: busybox +spec: + hostname: busybox-1 + subdomain: default-subdomain + containers: + - image: busybox + command: + - sleep + - "3600" + name: busybox +--- +apiVersion: v1 +kind: Pod +metadata: + name: busybox2 + labels: + name: busybox +spec: + hostname: busybox-2 + subdomain: default-subdomain + containers: + - image: busybox + command: + - sleep + - "3600" + name: busybox +``` + +If there exists a headless service in the same namespace as the pod and with the same name as the subdomain, the cluster's KubeDNS Server also returns an A record for the Pod's fully qualified hostname. +Given a Pod with the hostname set to "busybox-1" and the subdomain set to "default-subdomain", and a headless Service named "default-subdomain" in the same namespace, the pod will see it's own FQDN as "busybox-1.default-subdomain.my-namespace.svc.cluster.local". DNS serves an A record at that name, pointing to the Pod's IP. Both pods "busybox1" and "busybox2" can have their distinct A records. + +As of Kubernetes v1.2, the Endpoints object also has the annotation `endpoints.beta.kubernetes.io/hostnames-map`. Its value is the json representation of map[string(IP)][endpoints.HostRecord], for example: '{"10.245.1.6":{HostName: "my-webserver"}}'. +If the Endpoints are for a headless service, an A record is created with the format ...svc. +For the example json, if endpoints are for a headless service named "bar", and one of the endpoints has IP "10.245.1.6", an A record is created with the name "my-webserver.bar.my-namespace.svc.cluster.local" and the A record lookup would return "10.245.1.6". +This endpoints annotation generally does not need to be specified by end-users, but can used by the internal service controller to deliver the aforementioned feature. + +With v1.3, The Endpoints object can specify the `hostname` for any endpoint, along with its IP. The hostname field takes precedence over the hostname value +that might have been specified via the `endpoints.beta.kubernetes.io/hostnames-map` annotation. + +With v1.3, the following annotations are deprecated: `pod.beta.kubernetes.io/hostname`, `pod.beta.kubernetes.io/subdomain`, `endpoints.beta.kubernetes.io/hostnames-map` + +## How do I test if it is working? + +### Create a simple Pod to use as a test environment + +Create a file named busybox.yaml with the +following contents: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: busybox + namespace: default +spec: + containers: + - image: busybox + command: + - sleep + - "3600" + imagePullPolicy: IfNotPresent + name: busybox + restartPolicy: Always +``` + +Then create a pod using this file: + +``` +kubectl create -f busybox.yaml +``` + +### Wait for this pod to go into the running state + +You can get its status with: +``` +kubectl get pods busybox +``` + +You should see: + +``` +NAME READY STATUS RESTARTS AGE +busybox 1/1 Running 0 +``` + +### Validate that DNS is working + +Once that pod is running, you can exec nslookup in that environment: + +``` +kubectl exec -ti busybox -- nslookup kubernetes.default +``` + +You should see something like: + +``` +Server: 10.0.0.10 +Address 1: 10.0.0.10 + +Name: kubernetes.default +Address 1: 10.0.0.1 +``` + +If you see that, DNS is working correctly. + +### Troubleshooting Tips + +If the nslookup command fails, check the following: + +#### Check the local DNS configuration first +Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" below for more information) + +``` +kubectl exec busybox cat /etc/resolv.conf +``` + +Verify that the search path and name server are set up like the following (note that search path may vary for different cloud providers): + +``` +search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal +nameserver 10.0.0.10 +options ndots:5 +``` + +#### Quick diagnosis + +Errors such as the following indicate a problem with the kube-dns add-on or associated Services: + +``` +$ kubectl exec -ti busybox -- nslookup kubernetes.default +Server: 10.0.0.10 +Address 1: 10.0.0.10 + +nslookup: can't resolve 'kubernetes.default' +``` + +or + +``` +$ kubectl exec -ti busybox -- nslookup kubernetes.default +Server: 10.0.0.10 +Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local + +nslookup: can't resolve 'kubernetes.default' +``` + +#### Check if the DNS pod is running + +Use the kubectl get pods command to verify that the DNS pod is running. + +``` +kubectl get pods --namespace=kube-system -l k8s-app=kube-dns +``` + +You should see something like: + +``` +NAME READY STATUS RESTARTS AGE +... +kube-dns-v19-ezo1y 3/3 Running 0 1h +... +``` + +If you see that no pod is running or that the pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually. + +#### Check for Errors in the DNS pod + +Use `kubectl logs` command to see logs for the DNS daemons. + +``` +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz +``` + +See if there is any suspicious log. W, E, F letter at the beginning represent Warning, Error and Failure. Please search for entries that have these as the logging level and use [kubernetes issues](https://github.com/kubernetes/kubernetes/issues) to report unexpected errors. + +#### Is DNS service up? + +Verify that the DNS service is up by using the `kubectl get service` command. + +``` +kubectl get svc --namespace=kube-system +``` + +You should see: + +``` +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +... +kube-dns 10.0.0.10 53/UDP,53/TCP 1h +... +``` + +If you have created the service or in the case it should be created by default but it does not appear, see this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information. + +#### Are DNS endpoints exposed? + +You can verify that DNS endpoints are exposed by using the `kubectl get endpoints` command. + +``` +kubectl get ep kube-dns --namespace=kube-system +``` + +You should see something like: +``` +NAME ENDPOINTS AGE +kube-dns 10.180.3.17:53,10.180.3.17:53 1h +``` + +If you do not see the endpoints, see endpoints section in the [debugging services documentation](http://kubernetes.io/docs/user-guide/debugging-services/). + +For additional Kubernetes DNS examples, see the [cluster-dns examples](https://github.com/kubernetes/kubernetes/tree/master/examples/cluster-dns) in the Kubernetes GitHub repository. + +## Kubernetes Federation (Multiple Zone support) + +Release 1.3 introduced Cluster Federation support for multi-site +Kubernetes installations. This required some minor +(backward-compatible) changes to the way +the Kubernetes cluster DNS server processes DNS queries, to facilitate +the lookup of federated services (which span multiple Kubernetes clusters). +See the [Cluster Federation Administrators' Guide](/docs/admin/federation) for more +details on Cluster Federation and multi-site support. + +## How it Works + +The running Kubernetes DNS pod holds 3 containers - kubedns, dnsmasq and a health check called healthz. +The kubedns process watches the Kubernetes master for changes in Services and Endpoints, and maintains +in-memory lookup structures to service DNS requests. The dnsmasq container adds DNS caching to improve +performance. The healthz container provides a single health check endpoint while performing dual healthchecks +(for dnsmasq and kubedns). + +The DNS pod is exposed as a Kubernetes Service with a static IP. Once assigned the +kubelet passes DNS configured using the `--cluster-dns=10.0.0.10` flag to each +container. + +DNS names also need domains. The local domain is configurable, in the kubelet using +the flag `--cluster-domain=` + +The Kubernetes cluster DNS server (based off the [SkyDNS](https://github.com/skynetservices/skydns) library) +supports forward lookups (A records), service lookups (SRV records) and reverse IP address lookups (PTR records). + +## Inheriting DNS from the node +When running a pod, kubelet will prepend the cluster DNS server and search +paths to the node's own DNS settings. If the node is able to resolve DNS names +specific to the larger environment, pods should be able to, also. See "Known +issues" below for a caveat. + +If you don't want this, or if you want a different DNS config for pods, you can +use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will +not inherit DNS. Setting it to a valid file path means that kubelet will use +this file instead of `/etc/resolv.conf` for DNS inheritance. + +## Known issues +Kubernetes installs do not configure the nodes' resolv.conf files to use the +cluster DNS by default, because that process is inherently distro-specific. +This should probably be implemented eventually. + +Linux's libc is impossibly stuck ([see this bug from +2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just +3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to +consume 1 `nameserver` record and 3 `search` records. This means that if a +local installation already uses 3 `nameserver`s or uses more than 3 `search`es, +some of those settings will be lost. As a partial workaround, the node can run +`dnsmasq` which will provide more `nameserver` entries, but not more `search` +entries. You can also use kubelet's `--resolv-conf` flag. + +If you are using Alpine version 3.3 or earlier as your base image, DNS may not +work properly owing to a known issue with Alpine. Check [here](https://github.com/kubernetes/kubernetes/issues/30215) +for more information. + +## References + +- [Docs for the DNS cluster addon](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/dns/README.md) + +## What's next +- [Autoscaling the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/). From 1ee7bf7281d8619865e85e66d96ebf64f67a914c Mon Sep 17 00:00:00 2001 From: Tim Hockin Date: Wed, 15 Mar 2017 13:46:16 -0700 Subject: [PATCH 391/407] Delete CNAME --- CNAME | 1 - 1 file changed, 1 deletion(-) delete mode 100644 CNAME diff --git a/CNAME b/CNAME deleted file mode 100644 index b0d9e9ad80..0000000000 --- a/CNAME +++ /dev/null @@ -1 +0,0 @@ -kubernetes.io \ No newline at end of file From 1ec090cc7c351f9774b29d0e527466161f49030d Mon Sep 17 00:00:00 2001 From: Tim Hockin Date: Wed, 15 Mar 2017 14:29:07 -0700 Subject: [PATCH 392/407] Create CNAME --- CNAME | 1 + 1 file changed, 1 insertion(+) create mode 100644 CNAME diff --git a/CNAME b/CNAME new file mode 100644 index 0000000000..b0d9e9ad80 --- /dev/null +++ b/CNAME @@ -0,0 +1 @@ +kubernetes.io \ No newline at end of file From f3ae692daba49156ad3ea000661b738ed635459e Mon Sep 17 00:00:00 2001 From: Tim Hockin Date: Wed, 15 Mar 2017 14:31:30 -0700 Subject: [PATCH 393/407] Delete CNAME --- CNAME | 1 - 1 file changed, 1 deletion(-) delete mode 100644 CNAME diff --git a/CNAME b/CNAME deleted file mode 100644 index b0d9e9ad80..0000000000 --- a/CNAME +++ /dev/null @@ -1 +0,0 @@ -kubernetes.io \ No newline at end of file From 6ab05fbf1c1236b28ed1f633b8003b7e3732715f Mon Sep 17 00:00:00 2001 From: Tim Hockin Date: Wed, 15 Mar 2017 14:42:51 -0700 Subject: [PATCH 394/407] Create CNAME --- CNAME | 1 + 1 file changed, 1 insertion(+) create mode 100644 CNAME diff --git a/CNAME b/CNAME new file mode 100644 index 0000000000..b0d9e9ad80 --- /dev/null +++ b/CNAME @@ -0,0 +1 @@ +kubernetes.io \ No newline at end of file From 7071b5e7534a724af773f577353458ec5fd2785a Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 15:13:57 -0700 Subject: [PATCH 395/407] Move docs/user-guide/managing-deployments.md to /docs/concepts/cluster-administration/manage-deployment.md --- _data/concepts.yml | 1 + .../manage-deployment.md | 438 ++++++++++++++++++ .../manage-stateful-set/scale-stateful-set.md | 2 +- docs/user-guide/config-best-practices.md | 2 +- docs/user-guide/deployments.md | 2 +- docs/user-guide/index.md | 2 +- docs/user-guide/kubectl-conventions.md | 2 +- docs/user-guide/managing-deployments.md | 435 +---------------- docs/user-guide/update-demo/index.md | 2 +- 9 files changed, 447 insertions(+), 439 deletions(-) create mode 100644 docs/concepts/cluster-administration/manage-deployment.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 21562255db..b885c96bf9 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -32,6 +32,7 @@ toc: - title: Cluster Administration section: + - docs/concepts/cluster-administration/manage-deployment.md - docs/concepts/cluster-administration/networking.md - docs/concepts/cluster-administration/network-plugins.md - docs/concepts/cluster-administration/logging.md diff --git a/docs/concepts/cluster-administration/manage-deployment.md b/docs/concepts/cluster-administration/manage-deployment.md new file mode 100644 index 0000000000..fc32a5bb46 --- /dev/null +++ b/docs/concepts/cluster-administration/manage-deployment.md @@ -0,0 +1,438 @@ +--- +assignees: +- bgrant0607 +- janetkuo +- mikedanese +title: Managing Resources +--- + +You've deployed your application and exposed it via a service. Now what? Kubernetes provides a number of tools to help you manage your application deployment, including scaling and updating. Among the features we'll discuss in more depth are [configuration files](/docs/user-guide/configuring-containers/#configuration-in-kubernetes) and [labels](/docs/user-guide/deploying-applications/#labels). + +You can find all the files for this example [in our docs +repo here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/). + +* TOC +{:toc} + +## Organizing resource configurations + +Many applications require multiple resources to be created, such as a Deployment and a Service. Management of multiple resources can be simplified by grouping them together in the same file (separated by `---` in YAML). For example: + +{% include code.html language="yaml" file="nginx-app.yaml" ghlink="/docs/user-guide/nginx-app.yaml" %} + +Multiple resources can be created the same way as a single resource: + +```shell +$ kubectl create -f docs/user-guide/nginx-app.yaml +service "my-nginx-svc" created +deployment "my-nginx" created +``` + +The resources will be created in the order they appear in the file. Therefore, it's best to specify the service first, since that will ensure the scheduler can spread the pods associated with the service as they are created by the controller(s), such as Deployment. + +`kubectl create` also accepts multiple `-f` arguments: + +```shell +$ kubectl create -f docs/user-guide/nginx/nginx-svc.yaml -f docs/user-guide/nginx/nginx-deployment.yaml +``` + +And a directory can be specified rather than or in addition to individual files: + +```shell +$ kubectl create -f docs/user-guide/nginx/ +``` + +`kubectl` will read any files with suffixes `.yaml`, `.yml`, or `.json`. + +It is a recommended practice to put resources related to the same microservice or application tier into the same file, and to group all of the files associated with your application in the same directory. If the tiers of your application bind to each other using DNS, then you can then simply deploy all of the components of your stack en masse. + +A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files checked into github: + +```shell +$ kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/user-guide/nginx-deployment.yaml +deployment "nginx-deployment" created +``` + +## Bulk operations in kubectl + +Resource creation isn't the only operation that `kubectl` can perform in bulk. It can also extract resource names from configuration files in order to perform other operations, in particular to delete the same resources you created: + +```shell +$ kubectl delete -f docs/user-guide/nginx/ +deployment "my-nginx" deleted +service "my-nginx-svc" deleted +``` + +In the case of just two resources, it's also easy to specify both on the command line using the resource/name syntax: + +```shell +$ kubectl delete deployments/my-nginx services/my-nginx-svc +``` + +For larger numbers of resources, you'll find it easier to specify the selector (label query) specified using `-l` or `--selector`, to filter resources by their labels: + +```shell +$ kubectl delete deployment,services -l app=nginx +deployment "my-nginx" deleted +service "my-nginx-svc" deleted +``` + +Because `kubectl` outputs resource names in the same syntax it accepts, it's easy to chain operations using `$()` or `xargs`: + +```shell +$ kubectl get $(kubectl create -f docs/user-guide/nginx/ -o name | grep service) +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +my-nginx-svc 10.0.0.208 80/TCP 0s +``` + +With the above commands, we first create resources under docs/user-guide/nginx/ and print the resources created with `-o name` output format +(print each resource as resource/name). Then we `grep` only the "service", and then print it with `kubectl get`. + +If you happen to organize your resources across several subdirectories within a particular directory, you can recursively perform the operations on the subdirectories also, by specifying `--recursive` or `-R` alongside the `--filename,-f` flag. + +For instance, assume there is a directory `project/k8s/development` that holds all of the manifests needed for the development environment, organized by resource type: + +``` +project/k8s/development +├── configmap +│   └── my-configmap.yaml +├── deployment +│   └── my-deployment.yaml +└── pvc + └── my-pvc.yaml +``` + +By default, performing a bulk operation on `project/k8s/development` will stop at the first level of the directory, not processing any subdirectories. If we tried to create the resources in this directory using the following command, we'd encounter an error: + +```shell +$ kubectl create -f project/k8s/development +error: you must provide one or more resources by argument or filename (.json|.yaml|.yml|stdin) +``` + +Instead, specify the `--recursive` or `-R` flag with the `--filename,-f` flag as such: + +```shell +$ kubectl create -f project/k8s/development --recursive +configmap "my-config" created +deployment "my-deployment" created +persistentvolumeclaim "my-pvc" created +``` + +The `--recursive` flag works with any operation that accepts the `--filename,-f` flag such as: `kubectl {create,get,delete,describe,rollout} etc.` + +The `--recursive` flag also works when multiple `-f` arguments are provided: + +```shell +$ kubectl create -f project/k8s/namespaces -f project/k8s/development --recursive +namespace "development" created +namespace "staging" created +configmap "my-config" created +deployment "my-deployment" created +persistentvolumeclaim "my-pvc" created +``` + +If you're interested in learning more about `kubectl`, go ahead and read [kubectl Overview](/docs/user-guide/kubectl-overview). + +## Using labels effectively + +The examples we've used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another. + +For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels: + +```yaml + labels: + app: guestbook + tier: frontend +``` + +while the Redis master and slave would have different `tier` labels, and perhaps even an additional `role` label: + +```yaml + labels: + app: guestbook + tier: backend + role: master +``` + +and + +```yaml + labels: + app: guestbook + tier: backend + role: slave +``` + +The labels allow us to slice and dice our resources along any dimension specified by a label: + +```shell +$ kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml +$ kubectl get pods -Lapp -Ltier -Lrole +NAME READY STATUS RESTARTS AGE APP TIER ROLE +guestbook-fe-4nlpb 1/1 Running 0 1m guestbook frontend +guestbook-fe-ght6d 1/1 Running 0 1m guestbook frontend +guestbook-fe-jpy62 1/1 Running 0 1m guestbook frontend +guestbook-redis-master-5pg3b 1/1 Running 0 1m guestbook backend master +guestbook-redis-slave-2q2yf 1/1 Running 0 1m guestbook backend slave +guestbook-redis-slave-qgazl 1/1 Running 0 1m guestbook backend slave +my-nginx-divi2 1/1 Running 0 29m nginx +my-nginx-o0ef1 1/1 Running 0 29m nginx +$ kubectl get pods -lapp=guestbook,role=slave +NAME READY STATUS RESTARTS AGE +guestbook-redis-slave-2q2yf 1/1 Running 0 3m +guestbook-redis-slave-qgazl 1/1 Running 0 3m +``` + +## Canary deployments + +Another scenario where multiple labels are needed is to distinguish deployments of different releases or configurations of the same component. It is common practice to deploy a *canary* of a new application release (specified via image tag in the pod template) side by side with the previous release so that the new release can receive live production traffic before fully rolling it out. + +For instance, you can use a `track` label to differentiate different releases. + +The primary, stable release would have a `track` label with value as `stable`: + +```yaml + name: frontend + replicas: 3 + ... + labels: + app: guestbook + tier: frontend + track: stable + ... + image: gb-frontend:v3 +``` + +and then you can create a new release of the guestbook frontend that carries the `track` label with different value (i.e. `canary`), so that two sets of pods would not overlap: + +```yaml + name: frontend-canary + replicas: 1 + ... + labels: + app: guestbook + tier: frontend + track: canary + ... + image: gb-frontend:v4 +``` + + +The frontend service would span both sets of replicas by selecting the common subset of their labels (i.e. omitting the `track` label), so that the traffic will be redirected to both applications: + +```yaml + selector: + app: guestbook + tier: frontend +``` + +You can tweak the number of replicas of the stable and canary releases to determine the ratio of each release that will receive live production traffic (in this case, 3:1). +Once you're confident, you can update the stable track to the new application release and remove the canary one. + +For a more concrete example, check the [tutorial of deploying Ghost](https://github.com/kelseyhightower/talks/tree/master/kubecon-eu-2016/demo#deploy-a-canary). + +## Updating labels + +Sometimes existing pods and other resources need to be relabeled before creating new resources. This can be done with `kubectl label`. +For example, if you want to label all your nginx pods as frontend tier, simply run: + +```shell +$ kubectl label pods -l app=nginx tier=fe +pod "my-nginx-2035384211-j5fhi" labeled +pod "my-nginx-2035384211-u2c7e" labeled +pod "my-nginx-2035384211-u3t6x" labeled +``` + +This first filters all pods with the label "app=nginx", and then labels them with the "tier=fe". +To see the pods you just labeled, run: + +```shell +$ kubectl get pods -l app=nginx -L tier +NAME READY STATUS RESTARTS AGE TIER +my-nginx-2035384211-j5fhi 1/1 Running 0 23m fe +my-nginx-2035384211-u2c7e 1/1 Running 0 23m fe +my-nginx-2035384211-u3t6x 1/1 Running 0 23m fe +``` + +This outputs all "app=nginx" pods, with an additional label column of pods' tier (specified with `-L` or `--label-columns`). + +For more information, please see [labels](/docs/user-guide/labels/) and [kubectl label](/docs/user-guide/kubectl/kubectl_label/) document. + +## Updating annotations + +Sometimes you would want to attach annotations to resources. Annotations are arbitrary non-identifying metadata for retrieval by API clients such as tools, libraries, etc. This can be done with `kubectl annotate`. For example: + +```shell +$ kubectl annotate pods my-nginx-v4-9gw19 description='my frontend running nginx' +$ kubectl get pods my-nginx-v4-9gw19 -o yaml +apiversion: v1 +kind: pod +metadata: + annotations: + description: my frontend running nginx +... +``` + +For more information, please see [annotations](/docs/user-guide/annotations/) and [kubectl annotate](/docs/user-guide/kubectl/kubectl_annotate/) document. + +## Scaling your application + +When load on your application grows or shrinks, it's easy to scale with `kubectl`. For instance, to decrease the number of nginx replicas from 3 to 1, do: + +```shell +$ kubectl scale deployment/my-nginx --replicas=1 +deployment "my-nginx" scaled +``` + +Now you only have one pod managed by the deployment. + +```shell +$ kubectl get pods -l app=nginx +NAME READY STATUS RESTARTS AGE +my-nginx-2035384211-j5fhi 1/1 Running 0 30m +``` + +To have the system automatically choose the number of nginx replicas as needed, ranging from 1 to 3, do: + +```shell +$ kubectl autoscale deployment/my-nginx --min=1 --max=3 +deployment "my-nginx" autoscaled +``` + +Now your nginx replicas will be scaled up and down as needed, automatically. + +For more information, please see [kubectl scale](/docs/user-guide/kubectl/kubectl_scale/), [kubectl autoscale](/docs/user-guide/kubectl/kubectl_autoscale/) and [horizontal pod autoscaler](/docs/user-guide/horizontal-pod-autoscaler/) document. + + +## In-place updates of resources + +Sometimes it's necessary to make narrow, non-disruptive updates to resources you've created. + +### kubectl apply + +It is suggested to maintain a set of configuration files in source control (see [configuration as code](http://martinfowler.com/bliki/InfrastructureAsCode.html)), +so that they can be maintained and versioned along with the code for the resources they configure. +Then, you can use [`kubectl apply`](/docs/user-guide/kubectl/kubectl_apply/) to push your configuration changes to the cluster. + +This command will compare the version of the configuration that you're pushing with the previous version and apply the changes you've made, without overwriting any automated changes to properties you haven't specified. + +```shell +$ kubectl apply -f docs/user-guide/nginx/nginx-deployment.yaml +deployment "my-nginx" configured +``` + +Note that `kubectl apply` attaches an annotation to the resource in order to determine the changes to the configuration since the previous invocation. When it's invoked, `kubectl apply` does a three-way diff between the previous configuration, the provided input and the current configuration of the resource, in order to determine how to modify the resource. + +Currently, resources are created without this annotation, so the first invocation of `kubectl apply` will fall back to a two-way diff between the provided input and the current configuration of the resource. During this first invocation, it cannot detect the deletion of properties set when the resource was created. For this reason, it will not remove them. + +All subsequent calls to `kubectl apply`, and other commands that modify the configuration, such as `kubectl replace` and `kubectl edit`, will update the annotation, allowing subsequent calls to `kubectl apply` to detect and perform deletions using a three-way diff. + +**Note:** To use apply, always create resource initially with either `kubectl apply` or `kubectl create --save-config`. + +### kubectl edit + +Alternatively, you may also update resources with `kubectl edit`: + +```shell +$ kubectl edit deployment/my-nginx +``` + +This is equivalent to first `get` the resource, edit it in text editor, and then `apply` the resource with the updated version: + +```shell +$ kubectl get deployment my-nginx -o yaml > /tmp/nginx.yaml +$ vi /tmp/nginx.yaml +# do some edit, and then save the file +$ kubectl apply -f /tmp/nginx.yaml +deployment "my-nginx" configured +$ rm /tmp/nginx.yaml +``` + +This allows you to do more significant changes more easily. Note that you can specify the editor with your `EDITOR` or `KUBE_EDITOR` environment variables. + +For more information, please see [kubectl edit](/docs/user-guide/kubectl/kubectl_edit/) document. + +### kubectl patch + +Suppose you want to fix a typo of the container's image of a Deployment. One way to do that is with `kubectl patch`: + +```shell +# Suppose you have a Deployment with a container named "nginx" and its image "nignx" (typo), +# use container name "nginx" as a key to update the image from "nignx" (typo) to "nginx" +$ kubectl get deployment my-nginx -o yaml +``` + +```yaml +apiVersion: extensions/v1beta1 +kind: Deployment +... +spec: + template: + spec: + containers: + - image: nignx + name: nginx +... +``` + +```shell +$ kubectl patch deployment my-nginx -p'{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx"}]}}}}' +"my-nginx" patched +$ kubectl get pod my-nginx-1jgkf -o yaml +``` + +```yaml +apiVersion: extensions/v1beta1 +kind: Deployment +... +spec: + template: + spec: + containers: + - image: nginx + name: nginx +... +``` + +The patch is specified using json. + +The system ensures that you don't clobber changes made by other users or components by confirming that the `resourceVersion` doesn't differ from the version you edited. If you want to update regardless of other changes, remove the `resourceVersion` field when you edit the resource. However, if you do this, don't use your original configuration file as the source since additional fields most likely were set in the live state. + +For more information, please see [kubectl patch](/docs/user-guide/kubectl/kubectl_patch/) document. + +## Disruptive updates + +In some cases, you may need to update resource fields that cannot be updated once initialized, or you may just want to make a recursive change immediately, such as to fix broken pods created by a Deployment. To change such fields, use `replace --force`, which deletes and re-creates the resource. In this case, you can simply modify your original configuration file: + +```shell +$ kubectl replace -f docs/user-guide/nginx/nginx-deployment.yaml --force +deployment "my-nginx" deleted +deployment "my-nginx" replaced +``` + +## Updating your application without a service outage + +At some point, you'll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios. + +We'll guide you through how to create and update applications with Deployments. If your deployed application is managed by Replication Controllers, +you should read [how to use `kubectl rolling-update`](/docs/user-guide/rolling-updates/) instead. + +Let's say you were running version 1.7.9 of nginx: + +```shell +$ kubectl run my-nginx --image=nginx:1.7.9 --replicas=3 +deployment "my-nginx" created +``` + +To update to version 1.9.1, simply change `.spec.template.spec.containers[0].image` from `nginx:1.7.9` to `nginx:1.9.1`, with the kubectl commands we learned above. + +```shell +$ kubectl edit deployment/my-nginx +``` + +That's it! The Deployment will declaratively update the deployed nginx application progressively behind the scene. It ensures that only a certain number of old replicas may be down while they are being updated, and only a certain number of new replicas may be created above the desired number of pods. To learn more details about it, visit [Deployment page](/docs/user-guide/deployments/). + +## What's next? + +- [Learn about how to use `kubectl` for application introspection and debugging.](/docs/user-guide/introspection-and-debugging/) +- [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/) diff --git a/docs/tasks/manage-stateful-set/scale-stateful-set.md b/docs/tasks/manage-stateful-set/scale-stateful-set.md index eda728efdd..b79f188e6f 100644 --- a/docs/tasks/manage-stateful-set/scale-stateful-set.md +++ b/docs/tasks/manage-stateful-set/scale-stateful-set.md @@ -47,7 +47,7 @@ kubectl scale statefulsets --replicas= ### Alternative: `kubectl apply` / `kubectl edit` / `kubectl patch` -Alternatively, you can do [in-place updates](/docs/user-guide/managing-deployments/#in-place-updates-of-resources) on your StatefulSets. +Alternatively, you can do [in-place updates](/docs/concepts/cluster-administration/manage-deployment/#in-place-updates-of-resources) on your StatefulSets. If your StatefulSet was initially created with `kubectl apply` or `kubectl create --save-config`, update `.spec.replicas` of the StatefulSet manifests, and then do a `kubectl apply`: diff --git a/docs/user-guide/config-best-practices.md b/docs/user-guide/config-best-practices.md index 2811d84ed0..a7081a3519 100644 --- a/docs/user-guide/config-best-practices.md +++ b/docs/user-guide/config-best-practices.md @@ -112,7 +112,7 @@ This document is meant to highlight and consolidate in one place configuration b - Use `kubectl delete` rather than `stop`. `Delete` has a superset of the functionality of `stop`, and `stop` is deprecated. -- Use kubectl bulk operations (via files and/or labels) for get and delete. See [label selectors](/docs/user-guide/labels/#label-selectors) and [using labels effectively](/docs/user-guide/managing-deployments/#using-labels-effectively). +- Use kubectl bulk operations (via files and/or labels) for get and delete. See [label selectors](/docs/user-guide/labels/#label-selectors) and [using labels effectively](/docs/concepts/cluster-administration/manage-deployment/#using-labels-effectively). - Use `kubectl run` and `expose` to quickly create and expose single container Deployments. See the [quick start guide](/docs/user-guide/quick-start/) for an example. diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index d99909916c..acb65b69d5 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -718,7 +718,7 @@ to a previous revision, or even pause it if you need to apply multiple tweaks in ### Canary Deployment If you want to roll out releases to a subset of users or servers using the Deployment, you can create multiple Deployments, one for each release, -following the canary pattern described in [managing resources](/docs/user-guide/managing-deployments/#canary-deployments). +following the canary pattern described in [managing resources](/docs/concepts/cluster-administration/manage-deployment#canary-deployments). ## Writing a Deployment Spec diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 01a4b5538f..d7d7891142 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -21,7 +21,7 @@ The following topics in the Kubernetes User Guide can help you run applications 1. [Deploying continuously running applications](/docs/user-guide/deploying-applications/) 1. [Connecting applications: exposing applications to clients and users](/docs/user-guide/connecting-applications/) 1. [Working with containers in production](/docs/user-guide/production-pods/) -1. [Managing deployments](/docs/user-guide/managing-deployments/) +1. [Managing deployments](/docs/concepts/cluster-administration/manage-deployment/) 1. [Application introspection and debugging](/docs/user-guide/introspection-and-debugging/) 1. [Using the Kubernetes web user interface](/docs/user-guide/ui/) 1. [Logging](/docs/user-guide/logging/overview/) diff --git a/docs/user-guide/kubectl-conventions.md b/docs/user-guide/kubectl-conventions.md index fb0c467321..6f9be8d5bf 100644 --- a/docs/user-guide/kubectl-conventions.md +++ b/docs/user-guide/kubectl-conventions.md @@ -71,4 +71,4 @@ flag, which will provide the object to be submitted to the cluster. ### `kubectl apply` -* To use `kubectl apply` to update resources, always create resources initially with `kubectl apply` or with `--save-config`. See [managing resources with kubectl apply](/docs/user-guide/managing-deployments/#kubectl-apply) for the reason behind it. +* To use `kubectl apply` to update resources, always create resources initially with `kubectl apply` or with `--save-config`. See [managing resources with kubectl apply](/docs/concepts/cluster-administration/manage-deployment/#kubectl-apply) for the reason behind it. diff --git a/docs/user-guide/managing-deployments.md b/docs/user-guide/managing-deployments.md index fc32a5bb46..4df81aa90f 100644 --- a/docs/user-guide/managing-deployments.md +++ b/docs/user-guide/managing-deployments.md @@ -1,438 +1,7 @@ --- -assignees: -- bgrant0607 -- janetkuo -- mikedanese title: Managing Resources --- -You've deployed your application and exposed it via a service. Now what? Kubernetes provides a number of tools to help you manage your application deployment, including scaling and updating. Among the features we'll discuss in more depth are [configuration files](/docs/user-guide/configuring-containers/#configuration-in-kubernetes) and [labels](/docs/user-guide/deploying-applications/#labels). +{% include user-guide-content-moved.md %} +[Managing Resources](docs/concepts/cluster-administration/manage-deployment/) -You can find all the files for this example [in our docs -repo here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/). - -* TOC -{:toc} - -## Organizing resource configurations - -Many applications require multiple resources to be created, such as a Deployment and a Service. Management of multiple resources can be simplified by grouping them together in the same file (separated by `---` in YAML). For example: - -{% include code.html language="yaml" file="nginx-app.yaml" ghlink="/docs/user-guide/nginx-app.yaml" %} - -Multiple resources can be created the same way as a single resource: - -```shell -$ kubectl create -f docs/user-guide/nginx-app.yaml -service "my-nginx-svc" created -deployment "my-nginx" created -``` - -The resources will be created in the order they appear in the file. Therefore, it's best to specify the service first, since that will ensure the scheduler can spread the pods associated with the service as they are created by the controller(s), such as Deployment. - -`kubectl create` also accepts multiple `-f` arguments: - -```shell -$ kubectl create -f docs/user-guide/nginx/nginx-svc.yaml -f docs/user-guide/nginx/nginx-deployment.yaml -``` - -And a directory can be specified rather than or in addition to individual files: - -```shell -$ kubectl create -f docs/user-guide/nginx/ -``` - -`kubectl` will read any files with suffixes `.yaml`, `.yml`, or `.json`. - -It is a recommended practice to put resources related to the same microservice or application tier into the same file, and to group all of the files associated with your application in the same directory. If the tiers of your application bind to each other using DNS, then you can then simply deploy all of the components of your stack en masse. - -A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files checked into github: - -```shell -$ kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/user-guide/nginx-deployment.yaml -deployment "nginx-deployment" created -``` - -## Bulk operations in kubectl - -Resource creation isn't the only operation that `kubectl` can perform in bulk. It can also extract resource names from configuration files in order to perform other operations, in particular to delete the same resources you created: - -```shell -$ kubectl delete -f docs/user-guide/nginx/ -deployment "my-nginx" deleted -service "my-nginx-svc" deleted -``` - -In the case of just two resources, it's also easy to specify both on the command line using the resource/name syntax: - -```shell -$ kubectl delete deployments/my-nginx services/my-nginx-svc -``` - -For larger numbers of resources, you'll find it easier to specify the selector (label query) specified using `-l` or `--selector`, to filter resources by their labels: - -```shell -$ kubectl delete deployment,services -l app=nginx -deployment "my-nginx" deleted -service "my-nginx-svc" deleted -``` - -Because `kubectl` outputs resource names in the same syntax it accepts, it's easy to chain operations using `$()` or `xargs`: - -```shell -$ kubectl get $(kubectl create -f docs/user-guide/nginx/ -o name | grep service) -NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE -my-nginx-svc 10.0.0.208 80/TCP 0s -``` - -With the above commands, we first create resources under docs/user-guide/nginx/ and print the resources created with `-o name` output format -(print each resource as resource/name). Then we `grep` only the "service", and then print it with `kubectl get`. - -If you happen to organize your resources across several subdirectories within a particular directory, you can recursively perform the operations on the subdirectories also, by specifying `--recursive` or `-R` alongside the `--filename,-f` flag. - -For instance, assume there is a directory `project/k8s/development` that holds all of the manifests needed for the development environment, organized by resource type: - -``` -project/k8s/development -├── configmap -│   └── my-configmap.yaml -├── deployment -│   └── my-deployment.yaml -└── pvc - └── my-pvc.yaml -``` - -By default, performing a bulk operation on `project/k8s/development` will stop at the first level of the directory, not processing any subdirectories. If we tried to create the resources in this directory using the following command, we'd encounter an error: - -```shell -$ kubectl create -f project/k8s/development -error: you must provide one or more resources by argument or filename (.json|.yaml|.yml|stdin) -``` - -Instead, specify the `--recursive` or `-R` flag with the `--filename,-f` flag as such: - -```shell -$ kubectl create -f project/k8s/development --recursive -configmap "my-config" created -deployment "my-deployment" created -persistentvolumeclaim "my-pvc" created -``` - -The `--recursive` flag works with any operation that accepts the `--filename,-f` flag such as: `kubectl {create,get,delete,describe,rollout} etc.` - -The `--recursive` flag also works when multiple `-f` arguments are provided: - -```shell -$ kubectl create -f project/k8s/namespaces -f project/k8s/development --recursive -namespace "development" created -namespace "staging" created -configmap "my-config" created -deployment "my-deployment" created -persistentvolumeclaim "my-pvc" created -``` - -If you're interested in learning more about `kubectl`, go ahead and read [kubectl Overview](/docs/user-guide/kubectl-overview). - -## Using labels effectively - -The examples we've used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another. - -For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels: - -```yaml - labels: - app: guestbook - tier: frontend -``` - -while the Redis master and slave would have different `tier` labels, and perhaps even an additional `role` label: - -```yaml - labels: - app: guestbook - tier: backend - role: master -``` - -and - -```yaml - labels: - app: guestbook - tier: backend - role: slave -``` - -The labels allow us to slice and dice our resources along any dimension specified by a label: - -```shell -$ kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml -$ kubectl get pods -Lapp -Ltier -Lrole -NAME READY STATUS RESTARTS AGE APP TIER ROLE -guestbook-fe-4nlpb 1/1 Running 0 1m guestbook frontend -guestbook-fe-ght6d 1/1 Running 0 1m guestbook frontend -guestbook-fe-jpy62 1/1 Running 0 1m guestbook frontend -guestbook-redis-master-5pg3b 1/1 Running 0 1m guestbook backend master -guestbook-redis-slave-2q2yf 1/1 Running 0 1m guestbook backend slave -guestbook-redis-slave-qgazl 1/1 Running 0 1m guestbook backend slave -my-nginx-divi2 1/1 Running 0 29m nginx -my-nginx-o0ef1 1/1 Running 0 29m nginx -$ kubectl get pods -lapp=guestbook,role=slave -NAME READY STATUS RESTARTS AGE -guestbook-redis-slave-2q2yf 1/1 Running 0 3m -guestbook-redis-slave-qgazl 1/1 Running 0 3m -``` - -## Canary deployments - -Another scenario where multiple labels are needed is to distinguish deployments of different releases or configurations of the same component. It is common practice to deploy a *canary* of a new application release (specified via image tag in the pod template) side by side with the previous release so that the new release can receive live production traffic before fully rolling it out. - -For instance, you can use a `track` label to differentiate different releases. - -The primary, stable release would have a `track` label with value as `stable`: - -```yaml - name: frontend - replicas: 3 - ... - labels: - app: guestbook - tier: frontend - track: stable - ... - image: gb-frontend:v3 -``` - -and then you can create a new release of the guestbook frontend that carries the `track` label with different value (i.e. `canary`), so that two sets of pods would not overlap: - -```yaml - name: frontend-canary - replicas: 1 - ... - labels: - app: guestbook - tier: frontend - track: canary - ... - image: gb-frontend:v4 -``` - - -The frontend service would span both sets of replicas by selecting the common subset of their labels (i.e. omitting the `track` label), so that the traffic will be redirected to both applications: - -```yaml - selector: - app: guestbook - tier: frontend -``` - -You can tweak the number of replicas of the stable and canary releases to determine the ratio of each release that will receive live production traffic (in this case, 3:1). -Once you're confident, you can update the stable track to the new application release and remove the canary one. - -For a more concrete example, check the [tutorial of deploying Ghost](https://github.com/kelseyhightower/talks/tree/master/kubecon-eu-2016/demo#deploy-a-canary). - -## Updating labels - -Sometimes existing pods and other resources need to be relabeled before creating new resources. This can be done with `kubectl label`. -For example, if you want to label all your nginx pods as frontend tier, simply run: - -```shell -$ kubectl label pods -l app=nginx tier=fe -pod "my-nginx-2035384211-j5fhi" labeled -pod "my-nginx-2035384211-u2c7e" labeled -pod "my-nginx-2035384211-u3t6x" labeled -``` - -This first filters all pods with the label "app=nginx", and then labels them with the "tier=fe". -To see the pods you just labeled, run: - -```shell -$ kubectl get pods -l app=nginx -L tier -NAME READY STATUS RESTARTS AGE TIER -my-nginx-2035384211-j5fhi 1/1 Running 0 23m fe -my-nginx-2035384211-u2c7e 1/1 Running 0 23m fe -my-nginx-2035384211-u3t6x 1/1 Running 0 23m fe -``` - -This outputs all "app=nginx" pods, with an additional label column of pods' tier (specified with `-L` or `--label-columns`). - -For more information, please see [labels](/docs/user-guide/labels/) and [kubectl label](/docs/user-guide/kubectl/kubectl_label/) document. - -## Updating annotations - -Sometimes you would want to attach annotations to resources. Annotations are arbitrary non-identifying metadata for retrieval by API clients such as tools, libraries, etc. This can be done with `kubectl annotate`. For example: - -```shell -$ kubectl annotate pods my-nginx-v4-9gw19 description='my frontend running nginx' -$ kubectl get pods my-nginx-v4-9gw19 -o yaml -apiversion: v1 -kind: pod -metadata: - annotations: - description: my frontend running nginx -... -``` - -For more information, please see [annotations](/docs/user-guide/annotations/) and [kubectl annotate](/docs/user-guide/kubectl/kubectl_annotate/) document. - -## Scaling your application - -When load on your application grows or shrinks, it's easy to scale with `kubectl`. For instance, to decrease the number of nginx replicas from 3 to 1, do: - -```shell -$ kubectl scale deployment/my-nginx --replicas=1 -deployment "my-nginx" scaled -``` - -Now you only have one pod managed by the deployment. - -```shell -$ kubectl get pods -l app=nginx -NAME READY STATUS RESTARTS AGE -my-nginx-2035384211-j5fhi 1/1 Running 0 30m -``` - -To have the system automatically choose the number of nginx replicas as needed, ranging from 1 to 3, do: - -```shell -$ kubectl autoscale deployment/my-nginx --min=1 --max=3 -deployment "my-nginx" autoscaled -``` - -Now your nginx replicas will be scaled up and down as needed, automatically. - -For more information, please see [kubectl scale](/docs/user-guide/kubectl/kubectl_scale/), [kubectl autoscale](/docs/user-guide/kubectl/kubectl_autoscale/) and [horizontal pod autoscaler](/docs/user-guide/horizontal-pod-autoscaler/) document. - - -## In-place updates of resources - -Sometimes it's necessary to make narrow, non-disruptive updates to resources you've created. - -### kubectl apply - -It is suggested to maintain a set of configuration files in source control (see [configuration as code](http://martinfowler.com/bliki/InfrastructureAsCode.html)), -so that they can be maintained and versioned along with the code for the resources they configure. -Then, you can use [`kubectl apply`](/docs/user-guide/kubectl/kubectl_apply/) to push your configuration changes to the cluster. - -This command will compare the version of the configuration that you're pushing with the previous version and apply the changes you've made, without overwriting any automated changes to properties you haven't specified. - -```shell -$ kubectl apply -f docs/user-guide/nginx/nginx-deployment.yaml -deployment "my-nginx" configured -``` - -Note that `kubectl apply` attaches an annotation to the resource in order to determine the changes to the configuration since the previous invocation. When it's invoked, `kubectl apply` does a three-way diff between the previous configuration, the provided input and the current configuration of the resource, in order to determine how to modify the resource. - -Currently, resources are created without this annotation, so the first invocation of `kubectl apply` will fall back to a two-way diff between the provided input and the current configuration of the resource. During this first invocation, it cannot detect the deletion of properties set when the resource was created. For this reason, it will not remove them. - -All subsequent calls to `kubectl apply`, and other commands that modify the configuration, such as `kubectl replace` and `kubectl edit`, will update the annotation, allowing subsequent calls to `kubectl apply` to detect and perform deletions using a three-way diff. - -**Note:** To use apply, always create resource initially with either `kubectl apply` or `kubectl create --save-config`. - -### kubectl edit - -Alternatively, you may also update resources with `kubectl edit`: - -```shell -$ kubectl edit deployment/my-nginx -``` - -This is equivalent to first `get` the resource, edit it in text editor, and then `apply` the resource with the updated version: - -```shell -$ kubectl get deployment my-nginx -o yaml > /tmp/nginx.yaml -$ vi /tmp/nginx.yaml -# do some edit, and then save the file -$ kubectl apply -f /tmp/nginx.yaml -deployment "my-nginx" configured -$ rm /tmp/nginx.yaml -``` - -This allows you to do more significant changes more easily. Note that you can specify the editor with your `EDITOR` or `KUBE_EDITOR` environment variables. - -For more information, please see [kubectl edit](/docs/user-guide/kubectl/kubectl_edit/) document. - -### kubectl patch - -Suppose you want to fix a typo of the container's image of a Deployment. One way to do that is with `kubectl patch`: - -```shell -# Suppose you have a Deployment with a container named "nginx" and its image "nignx" (typo), -# use container name "nginx" as a key to update the image from "nignx" (typo) to "nginx" -$ kubectl get deployment my-nginx -o yaml -``` - -```yaml -apiVersion: extensions/v1beta1 -kind: Deployment -... -spec: - template: - spec: - containers: - - image: nignx - name: nginx -... -``` - -```shell -$ kubectl patch deployment my-nginx -p'{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx"}]}}}}' -"my-nginx" patched -$ kubectl get pod my-nginx-1jgkf -o yaml -``` - -```yaml -apiVersion: extensions/v1beta1 -kind: Deployment -... -spec: - template: - spec: - containers: - - image: nginx - name: nginx -... -``` - -The patch is specified using json. - -The system ensures that you don't clobber changes made by other users or components by confirming that the `resourceVersion` doesn't differ from the version you edited. If you want to update regardless of other changes, remove the `resourceVersion` field when you edit the resource. However, if you do this, don't use your original configuration file as the source since additional fields most likely were set in the live state. - -For more information, please see [kubectl patch](/docs/user-guide/kubectl/kubectl_patch/) document. - -## Disruptive updates - -In some cases, you may need to update resource fields that cannot be updated once initialized, or you may just want to make a recursive change immediately, such as to fix broken pods created by a Deployment. To change such fields, use `replace --force`, which deletes and re-creates the resource. In this case, you can simply modify your original configuration file: - -```shell -$ kubectl replace -f docs/user-guide/nginx/nginx-deployment.yaml --force -deployment "my-nginx" deleted -deployment "my-nginx" replaced -``` - -## Updating your application without a service outage - -At some point, you'll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios. - -We'll guide you through how to create and update applications with Deployments. If your deployed application is managed by Replication Controllers, -you should read [how to use `kubectl rolling-update`](/docs/user-guide/rolling-updates/) instead. - -Let's say you were running version 1.7.9 of nginx: - -```shell -$ kubectl run my-nginx --image=nginx:1.7.9 --replicas=3 -deployment "my-nginx" created -``` - -To update to version 1.9.1, simply change `.spec.template.spec.containers[0].image` from `nginx:1.7.9` to `nginx:1.9.1`, with the kubectl commands we learned above. - -```shell -$ kubectl edit deployment/my-nginx -``` - -That's it! The Deployment will declaratively update the deployed nginx application progressively behind the scene. It ensures that only a certain number of old replicas may be down while they are being updated, and only a certain number of new replicas may be created above the desired number of pods. To learn more details about it, visit [Deployment page](/docs/user-guide/deployments/). - -## What's next? - -- [Learn about how to use `kubectl` for application introspection and debugging.](/docs/user-guide/introspection-and-debugging/) -- [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/) diff --git a/docs/user-guide/update-demo/index.md b/docs/user-guide/update-demo/index.md index f2e3f8a0f2..14bf74ee1f 100644 --- a/docs/user-guide/update-demo/index.md +++ b/docs/user-guide/update-demo/index.md @@ -4,7 +4,7 @@ assignees: title: Rolling Update Demo --- -This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/user-guide/managing-deployments/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information. +This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/concepts/cluster-administration/manage-deployment/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information. The files for this example are viewable in [our docs repo here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/update-demo). From 630cce61de27fa1747cc7a93d47632eb9187c864 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 15:18:41 -0700 Subject: [PATCH 396/407] add nginx-app.yaml file --- .../cluster-administration/nginx-app.yaml | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 docs/concepts/cluster-administration/nginx-app.yaml diff --git a/docs/concepts/cluster-administration/nginx-app.yaml b/docs/concepts/cluster-administration/nginx-app.yaml new file mode 100644 index 0000000000..8ae449d806 --- /dev/null +++ b/docs/concepts/cluster-administration/nginx-app.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Service +metadata: + name: my-nginx-svc + labels: + app: nginx +spec: + type: LoadBalancer + ports: + - port: 80 + selector: + app: nginx +--- +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: my-nginx +spec: + replicas: 3 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.7.9 + ports: + - containerPort: 80 From bbc6835fc5a4d56f2bd87eb1cad65aa46e9954f8 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 15:25:44 -0700 Subject: [PATCH 397/407] add back missing / --- docs/user-guide/deployments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index acb65b69d5..a55d1dd255 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -718,7 +718,7 @@ to a previous revision, or even pause it if you need to apply multiple tweaks in ### Canary Deployment If you want to roll out releases to a subset of users or servers using the Deployment, you can create multiple Deployments, one for each release, -following the canary pattern described in [managing resources](/docs/concepts/cluster-administration/manage-deployment#canary-deployments). +following the canary pattern described in [managing resources](/docs/concepts/cluster-administration/manage-deployment/#canary-deployments). ## Writing a Deployment Spec From 4d9c7e19909f8c1b55e764046ee26405490377f4 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 15:34:15 -0700 Subject: [PATCH 398/407] fix link --- docs/user-guide/managing-deployments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/managing-deployments.md b/docs/user-guide/managing-deployments.md index 4df81aa90f..84215a44bf 100644 --- a/docs/user-guide/managing-deployments.md +++ b/docs/user-guide/managing-deployments.md @@ -3,5 +3,5 @@ title: Managing Resources --- {% include user-guide-content-moved.md %} -[Managing Resources](docs/concepts/cluster-administration/manage-deployment/) +[Managing Resources](/docs/concepts/cluster-administration/manage-deployment/) From 3ced6dbc2eaf8305e4ce39ebb0ca5d3d1cfcca30 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 15:54:00 -0700 Subject: [PATCH 399/407] Move Guide topic: Replication Controller Operations --- _data/tutorials.yml | 1 + ...run-stateless-ap-replication-controller.md | 230 ++++++++++++++++++ .../replication-controller/index.md | 2 +- .../replication-controller/operations.md | 228 +---------------- docs/user-guide/rolling-updates.md | 2 +- 5 files changed, 235 insertions(+), 228 deletions(-) create mode 100644 docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md diff --git a/_data/tutorials.yml b/_data/tutorials.yml index 8ae5f09a3b..b0b173a82b 100644 --- a/_data/tutorials.yml +++ b/_data/tutorials.yml @@ -43,6 +43,7 @@ toc: - docs/tutorials/stateless-application/run-stateless-application-deployment.md - docs/tutorials/stateless-application/expose-external-ip-address-service.md - docs/tutorials/stateless-application/expose-external-ip-address.md + - docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md - title: Stateful Applications section: - docs/tutorials/stateful-application/basic-stateful-set.md diff --git a/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md b/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md new file mode 100644 index 0000000000..53b0bce3fc --- /dev/null +++ b/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md @@ -0,0 +1,230 @@ +--- +assignees: +- bprashanth +title: Run Stateless AP Replication Controller +--- + +* TOC +{:toc} + +A replication controller ensures that a specified number of pod "replicas" are +running at any one time. If there are too many, it will kill some. If there are +too few, it will start more. + +## Creating a replication controller + +Replication controllers are created with `kubectl create`: + +```shell +$ kubectl create -f FILE +``` + +Where: + +* `-f FILE` or `--filename FILE` is a relative path to a + [configuration file](#replication_controller_configuration_file) in + either JSON or YAML format. + +You can use the [sample file](#sample_file) below to try a create request. + +A successful create request returns the name of the replication controller. To +view more details about the controller, see +[Viewing replication controllers](#viewing_replication_controllers) below. + +### Replication controller configuration file + +When creating a replication controller, you must point to a configuration file +as the value of the `-f` flag. The configuration +file can be formatted as YAML or as JSON, and supports the following fields: + +```json +{ + "apiVersion": "v1", + "kind": "ReplicationController", + "metadata": { + "name": "", + "labels": "", + "namespace": "" + }, + "spec": { + "replicas": int, + "selector": { + "":"" + }, + "template": { + "metadata": { + "labels": { + "":"" + } + }, + "spec": { + // See 'The spec schema' below + } + } + } +} +``` + +Required fields are: + +* `kind`: Always `ReplicationController`. +* `apiVersion`: Currently `v1`. +* `metadata`: An object containing: + * `name`: Required if `generateName` is not specified. The name of this + replication controller. It must be an + [RFC1035](https://www.ietf.org/rfc/rfc1035.txt) compatible value and be + unique within the namespace. + * `labels`: Optional. Labels are arbitrary key:value pairs that can be used + for grouping and targeting by other resources and services. + * `generateName`: Required if `name` is not set. A prefix to use to generate + a unique name. Has the same validation rules as `name`. + * `namespace`: Optional. The namespace of the replication controller. + * `annotations`: Optional. A map of string keys and values that can be used + by external tooling to store and retrieve arbitrary metadata about + objects. +* `spec`: The configuration for this replication controller. It must + contain: + * `replicas`: The number of pods to create and maintain. + * `selector`: A map of key:value pairs assigned to the set of pods that + this replication controller is responsible for managing. **This must** + **match the key:value pairs in the `template`'s `labels` field**. + * `template` contains: + * A `metadata` object with `labels` for the pod. + * The [`spec` schema](#the_spec_schema) that defines the pod + configuration. + +### The `spec` schema + +The `spec` schema (that is a child of `template`) is described in the locations +below: + +* The [`spec` schema](/docs/user-guide/pods/multi-container/#the_spec_schema) + section of the Creating Multi-Container Pods page covers required and + frequently-used fields. +* The entire `spec` schema is documented in the + [Kubernetes API reference](/docs/api-reference/v1/definitions/#_v1_podspec). + +### Sample file + +The following sample file creates 2 pods, each containing a single container +using the `redis` image. Port 80 on each container is opened. The replication +controller is tagged with the `serving` label. The pods are given the label +`frontend` and the `selector` is set to `frontend`, to indicate that the +controller should manage pods with the `frontend` label. + +```json +{ + "kind": "ReplicationController", + "apiVersion": "v1", + "metadata": { + "name": "frontend-controller", + "labels": { + "state": "serving" + } + }, + "spec": { + "replicas": 2, + "selector": { + "app": "frontend" + }, + "template": { + "metadata": { + "labels": { + "app": "frontend" + } + }, + "spec": { + "volumes": null, + "containers": [ + { + "name": "php-redis", + "image": "redis", + "ports": [ + { + "containerPort": 80, + "protocol": "TCP" + } + ], + "imagePullPolicy": "IfNotPresent" + } + ], + "restartPolicy": "Always", + "dnsPolicy": "ClusterFirst" + } + } + } +} +``` + +## Updating replication controller pods + +See [Rolling Updates](/docs/user-guide/rolling-updates/). + +## Resizing a replication controller + +See +[Resizing a replication controller](/docs/user-guide/resizing-a-replication-controller/). + +## Viewing replication controllers + +To list replication controllers on a cluster, use the `kubectl get` command: + +```shell +$ kubectl get rc +``` + +A successful get command returns all replication controllers on the cluster in +the specified or default namespace. For example: + +```shell +CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS +frontend php-redis redis name=frontend 2 +``` + +You can also use `get rc NAME` to return information about a specific +replication controller. + +To view detailed information about a specific replication controller, use the +`kubectl describe` command: + +```shell +$ kubectl describe rc NAME +``` + +A successful describe request returns details about the replication controller +including number and status of pods managed, and recent events: + +```conf +Name: frontend +Namespace: default +Image(s): gcr.io/google_samples/gb-frontend:v3 +Selector: name=frontend +Labels: name=frontend +Replicas: 2 current / 2 desired +Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Fri, 06 Nov 2015 16:52:50 -0800 Fri, 06 Nov 2015 16:52:50 -0800 1 {replication-controller } SuccessfulCreate Created pod: frontend-gyx2h + Fri, 06 Nov 2015 16:52:50 -0800 Fri, 06 Nov 2015 16:52:50 -0800 1 {replication-controller } SuccessfulCreate Created pod: frontend-vc9w4 +``` + +## Deleting replication controllers + +To delete a replication controller as well as the pods that it controls, use +`kubectl delete`: + +```shell +$ kubectl delete rc NAME +``` + +By default, `kubectl delete rc` will resize the controller to zero (effectively +deleting all pods) before deleting it. + +To delete a replication controller without deleting its pods, use +`kubectl delete` and specify `--cascade=false`: + +```shell +$ kubectl delete rc NAME --cascade=false +``` + +A successful delete request returns the name of the deleted resource. diff --git a/docs/user-guide/replication-controller/index.md b/docs/user-guide/replication-controller/index.md index 0fee281cdb..813f4a96b7 100644 --- a/docs/user-guide/replication-controller/index.md +++ b/docs/user-guide/replication-controller/index.md @@ -261,4 +261,4 @@ safe to terminate when the machine is otherwise ready to be rebooted/shutdown. ## For more information -Read [ReplicationController Operations](/docs/user-guide/replication-controller/operations/). +Read [Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/). diff --git a/docs/user-guide/replication-controller/operations.md b/docs/user-guide/replication-controller/operations.md index b3bd6115cc..d3d4697400 100644 --- a/docs/user-guide/replication-controller/operations.md +++ b/docs/user-guide/replication-controller/operations.md @@ -1,230 +1,6 @@ --- -assignees: -- bprashanth title: Replication Controller Operations --- -* TOC -{:toc} - -A replication controller ensures that a specified number of pod "replicas" are -running at any one time. If there are too many, it will kill some. If there are -too few, it will start more. - -## Creating a replication controller - -Replication controllers are created with `kubectl create`: - -```shell -$ kubectl create -f FILE -``` - -Where: - -* `-f FILE` or `--filename FILE` is a relative path to a - [configuration file](#replication_controller_configuration_file) in - either JSON or YAML format. - -You can use the [sample file](#sample_file) below to try a create request. - -A successful create request returns the name of the replication controller. To -view more details about the controller, see -[Viewing replication controllers](#viewing_replication_controllers) below. - -### Replication controller configuration file - -When creating a replication controller, you must point to a configuration file -as the value of the `-f` flag. The configuration -file can be formatted as YAML or as JSON, and supports the following fields: - -```json -{ - "apiVersion": "v1", - "kind": "ReplicationController", - "metadata": { - "name": "", - "labels": "", - "namespace": "" - }, - "spec": { - "replicas": int, - "selector": { - "":"" - }, - "template": { - "metadata": { - "labels": { - "":"" - } - }, - "spec": { - // See 'The spec schema' below - } - } - } -} -``` - -Required fields are: - -* `kind`: Always `ReplicationController`. -* `apiVersion`: Currently `v1`. -* `metadata`: An object containing: - * `name`: Required if `generateName` is not specified. The name of this - replication controller. It must be an - [RFC1035](https://www.ietf.org/rfc/rfc1035.txt) compatible value and be - unique within the namespace. - * `labels`: Optional. Labels are arbitrary key:value pairs that can be used - for grouping and targeting by other resources and services. - * `generateName`: Required if `name` is not set. A prefix to use to generate - a unique name. Has the same validation rules as `name`. - * `namespace`: Optional. The namespace of the replication controller. - * `annotations`: Optional. A map of string keys and values that can be used - by external tooling to store and retrieve arbitrary metadata about - objects. -* `spec`: The configuration for this replication controller. It must - contain: - * `replicas`: The number of pods to create and maintain. - * `selector`: A map of key:value pairs assigned to the set of pods that - this replication controller is responsible for managing. **This must** - **match the key:value pairs in the `template`'s `labels` field**. - * `template` contains: - * A `metadata` object with `labels` for the pod. - * The [`spec` schema](#the_spec_schema) that defines the pod - configuration. - -### The `spec` schema - -The `spec` schema (that is a child of `template`) is described in the locations -below: - -* The [`spec` schema](/docs/user-guide/pods/multi-container/#the_spec_schema) - section of the Creating Multi-Container Pods page covers required and - frequently-used fields. -* The entire `spec` schema is documented in the - [Kubernetes API reference](/docs/api-reference/v1/definitions/#_v1_podspec). - -### Sample file - -The following sample file creates 2 pods, each containing a single container -using the `redis` image. Port 80 on each container is opened. The replication -controller is tagged with the `serving` label. The pods are given the label -`frontend` and the `selector` is set to `frontend`, to indicate that the -controller should manage pods with the `frontend` label. - -```json -{ - "kind": "ReplicationController", - "apiVersion": "v1", - "metadata": { - "name": "frontend-controller", - "labels": { - "state": "serving" - } - }, - "spec": { - "replicas": 2, - "selector": { - "app": "frontend" - }, - "template": { - "metadata": { - "labels": { - "app": "frontend" - } - }, - "spec": { - "volumes": null, - "containers": [ - { - "name": "php-redis", - "image": "redis", - "ports": [ - { - "containerPort": 80, - "protocol": "TCP" - } - ], - "imagePullPolicy": "IfNotPresent" - } - ], - "restartPolicy": "Always", - "dnsPolicy": "ClusterFirst" - } - } - } -} -``` - -## Updating replication controller pods - -See [Rolling Updates](/docs/user-guide/rolling-updates/). - -## Resizing a replication controller - -See -[Resizing a replication controller](/docs/user-guide/resizing-a-replication-controller/). - -## Viewing replication controllers - -To list replication controllers on a cluster, use the `kubectl get` command: - -```shell -$ kubectl get rc -``` - -A successful get command returns all replication controllers on the cluster in -the specified or default namespace. For example: - -```shell -CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS -frontend php-redis redis name=frontend 2 -``` - -You can also use `get rc NAME` to return information about a specific -replication controller. - -To view detailed information about a specific replication controller, use the -`kubectl describe` command: - -```shell -$ kubectl describe rc NAME -``` - -A successful describe request returns details about the replication controller -including number and status of pods managed, and recent events: - -```conf -Name: frontend -Namespace: default -Image(s): gcr.io/google_samples/gb-frontend:v3 -Selector: name=frontend -Labels: name=frontend -Replicas: 2 current / 2 desired -Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed -Events: - FirstSeen LastSeen Count From SubobjectPath Reason Message - Fri, 06 Nov 2015 16:52:50 -0800 Fri, 06 Nov 2015 16:52:50 -0800 1 {replication-controller } SuccessfulCreate Created pod: frontend-gyx2h - Fri, 06 Nov 2015 16:52:50 -0800 Fri, 06 Nov 2015 16:52:50 -0800 1 {replication-controller } SuccessfulCreate Created pod: frontend-vc9w4 -``` - -## Deleting replication controllers - -To delete a replication controller as well as the pods that it controls, use -`kubectl delete`: - -```shell -$ kubectl delete rc NAME -``` - -By default, `kubectl delete rc` will resize the controller to zero (effectively -deleting all pods) before deleting it. - -To delete a replication controller without deleting its pods, use -`kubectl delete` and specify `--cascade=false`: - -```shell -$ kubectl delete rc NAME --cascade=false -``` - -A successful delete request returns the name of the deleted resource. +{% include user-guide-content-moved.md %} +[Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/) diff --git a/docs/user-guide/rolling-updates.md b/docs/user-guide/rolling-updates.md index 6bd3c469c8..d350f41719 100644 --- a/docs/user-guide/rolling-updates.md +++ b/docs/user-guide/rolling-updates.md @@ -99,7 +99,7 @@ as well as either: YAML format. The configuration file must specify a new top-level `id` value and include at least one of the existing `spec.selector` key:value pairs. See the - [Replication Controller Operations](/docs/user-guide/replication-controller/operations#replication-controller-configuration-file) + [Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/#replication-controller-configuration-file) page for details.

    From 7a41f3feb6c960e2db178b8efbd91e504172b0fd Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 16:17:09 -0700 Subject: [PATCH 400/407] Move Guide topic: Resizing a replication controller --- ...run-stateless-ap-replication-controller.md | 32 +++++++++++++++++-- .../resizing-a-replication-controller.md | 32 ++----------------- 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md b/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md index 53b0bce3fc..d5df3ede4a 100644 --- a/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md +++ b/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md @@ -162,8 +162,36 @@ See [Rolling Updates](/docs/user-guide/rolling-updates/). ## Resizing a replication controller -See -[Resizing a replication controller](/docs/user-guide/resizing-a-replication-controller/). +To increase or decrease the number of pods under a replication controller's +control, use the `kubectl scale` command: + + $ kubectl scale rc NAME --replicas=COUNT \ + [--current-replicas=COUNT] \ + [--resource-version=VERSION] + +Tip: You can use the `rc` alias in your commands in place of +`replicationcontroller`. + +Required fields are: + +* `NAME`: The name of the replication controller to update. +* `--replicas=COUNT`: The desired number of replicas. + +Optional fields are: + +* `--current-replicas=COUNT`: A precondition for current size. If specified, + the resize will only take place if the current number of replicas matches + this value. +* `--resource-version=VERSION`: A precondition for resource version. If + specified, the resize will only take place if the current replication + controller version matches this value. Versions are specified in the + `labels` field of the replication controller's configuration file, as a + key:value pair with a key of `version`. For example, + `--resource-version test` matches: + + "labels": { + "version": "test" + } ## Viewing replication controllers diff --git a/docs/user-guide/resizing-a-replication-controller.md b/docs/user-guide/resizing-a-replication-controller.md index c60ce8e60b..156841d586 100644 --- a/docs/user-guide/resizing-a-replication-controller.md +++ b/docs/user-guide/resizing-a-replication-controller.md @@ -4,33 +4,5 @@ assignees: title: Resizing a Replication Controller --- -To increase or decrease the number of pods under a replication controller's -control, use the `kubectl scale` command: - - $ kubectl scale rc NAME --replicas=COUNT \ - [--current-replicas=COUNT] \ - [--resource-version=VERSION] - -Tip: You can use the `rc` alias in your commands in place of -`replicationcontroller`. - -Required fields are: - -* `NAME`: The name of the replication controller to update. -* `--replicas=COUNT`: The desired number of replicas. - -Optional fields are: - -* `--current-replicas=COUNT`: A precondition for current size. If specified, - the resize will only take place if the current number of replicas matches - this value. -* `--resource-version=VERSION`: A precondition for resource version. If - specified, the resize will only take place if the current replication - controller version matches this value. Versions are specified in the - `labels` field of the replication controller's configuration file, as a - key:value pair with a key of `version`. For example, - `--resource-version test` matches: - - "labels": { - "version": "test" - } +{% include user-guide-content-moved.md %} +[Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/#resizing-a-replication-controller) From 088cfc2c6ff2b9a9ac7807bec56e20e01415a2ca Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 16:49:02 -0700 Subject: [PATCH 401/407] Concepts toc (#2840) * Move Guide topic: Limit range. * Adjust Concepts Overview TOC. --- _data/concepts.yml | 10 +++++----- .../working-with-objects}/annotations.md | 0 .../working-with-objects/kubernetes-objects.md} | 9 +++++++-- .../overview/working-with-objects}/labels.md | 3 +++ .../working-with-objects/nginx-deployment.yaml | 16 ++++++++++++++++ 5 files changed, 31 insertions(+), 7 deletions(-) rename docs/concepts/{object-metadata => overview/working-with-objects}/annotations.md (100%) rename docs/concepts/{abstractions/overview.md => overview/working-with-objects/kubernetes-objects.md} (95%) rename docs/{user-guide => concepts/overview/working-with-objects}/labels.md (99%) create mode 100644 docs/concepts/overview/working-with-objects/nginx-deployment.yaml diff --git a/_data/concepts.yml b/_data/concepts.yml index b885c96bf9..b7d07f1910 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -6,10 +6,14 @@ toc: - title: Overview section: - docs/concepts/overview/components.md + - title: Working with Kubernetes Objects + section: + - docs/concepts/overview/working-with-objects/kubernetes-objects.md + - docs/concepts/overview/working-with-objects/labels.md + - docs/concepts/overview/working-with-objects/annotations.md - title: Kubernetes Objects section: - - docs/concepts/abstractions/overview.md - title: Pods section: - docs/concepts/abstractions/pod.md @@ -20,10 +24,6 @@ toc: - docs/concepts/abstractions/controllers/petsets.md - docs/concepts/abstractions/controllers/garbage-collection.md -- title: Object Metadata - section: - - docs/concepts/object-metadata/annotations.md - - title: Workloads section: - title: Pods diff --git a/docs/concepts/object-metadata/annotations.md b/docs/concepts/overview/working-with-objects/annotations.md similarity index 100% rename from docs/concepts/object-metadata/annotations.md rename to docs/concepts/overview/working-with-objects/annotations.md diff --git a/docs/concepts/abstractions/overview.md b/docs/concepts/overview/working-with-objects/kubernetes-objects.md similarity index 95% rename from docs/concepts/abstractions/overview.md rename to docs/concepts/overview/working-with-objects/kubernetes-objects.md index 4e832eaffb..3e24b810f4 100644 --- a/docs/concepts/abstractions/overview.md +++ b/docs/concepts/overview/working-with-objects/kubernetes-objects.md @@ -1,5 +1,9 @@ --- -title: Kubernetes Objects +title: Understanding Kubernetes Objects + +redirect_from: +- "/docs/concepts/abstractions/overview/" +- "/docs/concepts/abstractions/overview.html" --- {% capture overview %} @@ -23,6 +27,7 @@ To work with Kubernetes objects--whether to create, modify, or delete them--you' Every Kubernetes object includes two nested object fields that govern the object's configuration: the object *spec* and the object *status*. The *spec*, which you must provide, describes your *desired state* for the object--the characteristics that you want the object to have. The *status* describes the *actual state* for the object, and is supplied and updated by the Kubernetes system. At any given time, the Kubernetes Control Plane actively manages an object's actual state to match the desired state you supplied. + For example, a Kubernetes Deployment is an object that can represent an application running on your cluster. When you create the Deployment, you might set the Deployment spec to specify that you want three replicas of the application to be running. The Kubernetes system reads the Deployment spec and starts three instances of your desired application--updating the status to match your spec. If any of those instances should fail (a status change), the Kubernetes system responds to the difference between spec and status by making a correction--in this case, starting a replacement instance. For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md). @@ -33,7 +38,7 @@ When you create an object in Kubernetes, you must provide the object spec that d Here's an example `.yaml` file that shows the required fields and object spec for a Kubernetes Deployment: -{% include code.html language="yaml" file="nginx-deployment.yaml" ghlink="/docs/concepts/abstractions/nginx-deployment.yaml" %} +{% include code.html language="yaml" file="nginx-deployment.yaml" ghlink="/docs/concepts/overview/working-with-objects/nginx-deployment.yaml" %} One way to create a Deployment using a `.yaml` file like the one above is to use the []`kubectl create`]() command in the `kubectl` command-line interface, passing the `.yaml` file as an argument. Here's an example: diff --git a/docs/user-guide/labels.md b/docs/concepts/overview/working-with-objects/labels.md similarity index 99% rename from docs/user-guide/labels.md rename to docs/concepts/overview/working-with-objects/labels.md index a13e160089..af99f0c364 100644 --- a/docs/user-guide/labels.md +++ b/docs/concepts/overview/working-with-objects/labels.md @@ -2,6 +2,9 @@ assignees: - mikedanese title: Labels and Selectors +redirect_from: +- "/docs/user-guide/labels/" +- "/docs/user-guide/labels.html" --- _Labels_ are key/value pairs that are attached to objects, such as pods. diff --git a/docs/concepts/overview/working-with-objects/nginx-deployment.yaml b/docs/concepts/overview/working-with-objects/nginx-deployment.yaml new file mode 100644 index 0000000000..7a0dd431ef --- /dev/null +++ b/docs/concepts/overview/working-with-objects/nginx-deployment.yaml @@ -0,0 +1,16 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + replicas: 3 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.7.9 + ports: + - containerPort: 80 From b66a83d5a65c4662ab27f26e259845262fd014a7 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 16:53:37 -0700 Subject: [PATCH 402/407] Move Guide topic: Rolling Updates --- _data/tasks.yml | 4 + .../manage-deployment.md | 2 +- .../rolling-update-replication-controller.md | 256 ++++++++++++++++++ ...run-stateless-ap-replication-controller.md | 2 +- .../horizontal-pod-autoscaling/index.md | 2 +- .../replication-controller/index.md | 2 +- docs/user-guide/rolling-updates.md | 254 +---------------- 7 files changed, 266 insertions(+), 256 deletions(-) create mode 100644 docs/tasks/run-application/rolling-update-replication-controller.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 309eb0673c..c113a1fa87 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -27,6 +27,10 @@ toc: - docs/tasks/configure-pod-container/attach-handler-lifecycle-event.md - docs/tasks/configure-pod-container/configure-pod-disruption-budget.md +- title: Running Applications + section: + - docs/tasks/run-application/rolling-update-replication-controller.md + - title: Accessing Applications in a Cluster section: - docs/tasks/access-application-cluster/port-forward-access-application-cluster.md diff --git a/docs/concepts/cluster-administration/manage-deployment.md b/docs/concepts/cluster-administration/manage-deployment.md index fc32a5bb46..c1da7f3d5f 100644 --- a/docs/concepts/cluster-administration/manage-deployment.md +++ b/docs/concepts/cluster-administration/manage-deployment.md @@ -415,7 +415,7 @@ deployment "my-nginx" replaced At some point, you'll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios. We'll guide you through how to create and update applications with Deployments. If your deployed application is managed by Replication Controllers, -you should read [how to use `kubectl rolling-update`](/docs/user-guide/rolling-updates/) instead. +you should read [how to use `kubectl rolling-update`](/docs/tasks/run-application/rolling-update-replication-controller/) instead. Let's say you were running version 1.7.9 of nginx: diff --git a/docs/tasks/run-application/rolling-update-replication-controller.md b/docs/tasks/run-application/rolling-update-replication-controller.md new file mode 100644 index 0000000000..19aadec01a --- /dev/null +++ b/docs/tasks/run-application/rolling-update-replication-controller.md @@ -0,0 +1,256 @@ +--- +assignees: +- janetkuo +title: Rolling Update Replication Controller +--- + +* TOC +{:toc} + +## Overview + +To update a service without an outage, `kubectl` supports what is called ['rolling update'](/docs/user-guide/kubectl/kubectl_rolling-update), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) and the [example of rolling update](/docs/user-guide/update-demo/) for more information. + +Note that `kubectl rolling-update` only supports Replication Controllers. However, if you deploy applications with Replication Controllers, +consider switching them to [Deployments](/docs/user-guide/deployments/). A Deployment is a higher-level controller that automates rolling updates +of applications declaratively, and therefore is recommended. If you still want to keep your Replication Controllers and use `kubectl rolling-update`, keep reading: + +A rolling update applies changes to the configuration of pods being managed by +a replication controller. The changes can be passed as a new replication +controller configuration file; or, if only updating the image, a new container +image can be specified directly. + +A rolling update works by: + +1. Creating a new replication controller with the updated configuration. +2. Increasing/decreasing the replica count on the new and old controllers until + the correct number of replicas is reached. +3. Deleting the original replication controller. + +Rolling updates are initiated with the `kubectl rolling-update` command: + + $ kubectl rolling-update NAME \ + ([NEW_NAME] --image=IMAGE | -f FILE) + +## Passing a configuration file + +To initiate a rolling update using a configuration file, pass the new file to +`kubectl rolling-update`: + + $ kubectl rolling-update NAME -f FILE + +The configuration file must: + +* Specify a different `metadata.name` value. + +* Overwrite at least one common label in its `spec.selector` field. + +* Use the same `metadata.namespace`. + +Replication controller configuration files are described in +[Creating Replication Controllers](/docs/user-guide/replication-controller/operations/). + +### Examples + + // Update pods of frontend-v1 using new replication controller data in frontend-v2.json. + $ kubectl rolling-update frontend-v1 -f frontend-v2.json + + // Update pods of frontend-v1 using JSON data passed into stdin. + $ cat frontend-v2.json | kubectl rolling-update frontend-v1 -f - + +## Updating the container image + +To update only the container image, pass a new image name and tag with the +`--image` flag and (optionally) a new controller name: + + $ kubectl rolling-update NAME [NEW_NAME] --image=IMAGE:TAG + +The `--image` flag is only supported for single-container pods. Specifying +`--image` with multi-container pods returns an error. + +If no `NEW_NAME` is specified, a new replication controller is created with +a temporary name. Once the rollout is complete, the old controller is deleted, +and the new controller is updated to use the original name. + +The update will fail if `IMAGE:TAG` is identical to the +current value. For this reason, we recommend the use of versioned tags as +opposed to values such as `:latest`. Doing a rolling update from `image:latest` +to a new `image:latest` will fail, even if the image at that tag has changed. +Moreover, the use of `:latest` is not recommended, see +[Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information. + +### Examples + + // Update the pods of frontend-v1 to frontend-v2 + $ kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2 + + // Update the pods of frontend, keeping the replication controller name + $ kubectl rolling-update frontend --image=image:v2 + +## Required and optional fields + +Required fields are: + +* `NAME`: The name of the replication controller to update. + +as well as either: + +* `-f FILE`: A replication controller configuration file, in either JSON or + YAML format. The configuration file must specify a new top-level `id` value + and include at least one of the existing `spec.selector` key:value pairs. + See the + [Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/#replication-controller-configuration-file) + page for details. +
    +
    + or: +
    +
    +* `--image IMAGE:TAG`: The name and tag of the image to update to. Must be + different than the current image:tag currently specified. + +Optional fields are: + +* `NEW_NAME`: Only used in conjunction with `--image` (not with `-f FILE`). The + name to assign to the new replication controller. +* `--poll-interval DURATION`: The time between polling the controller status + after update. Valid units are `ns` (nanoseconds), `us` or `µs` (microseconds), + `ms` (milliseconds), `s` (seconds), `m` (minutes), or `h` (hours). Units can + be combined (e.g. `1m30s`). The default is `3s`. +* `--timeout DURATION`: The maximum time to wait for the controller to update a + pod before exiting. Default is `5m0s`. Valid units are as described for + `--poll-interval` above. +* `--update-period DURATION`: The time to wait between updating pods. Default + is `1m0s`. Valid units are as described for `--poll-interval` above. + +Additional information about the `kubectl rolling-update` command is available +from the [`kubectl` reference](/docs/user-guide/kubectl/kubectl_rolling-update/). + +## Walkthrough + +Let's say you were running version 1.7.9 of nginx: + +```yaml +apiVersion: v1 +kind: ReplicationController +metadata: + name: my-nginx +spec: + replicas: 5 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.7.9 + ports: + - containerPort: 80 +``` + +To update to version 1.9.1, you can use [`kubectl rolling-update --image`](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) to specify the new image: + +```shell +$ kubectl rolling-update my-nginx --image=nginx:1.9.1 +Created my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 +``` + +In another window, you can see that `kubectl` added a `deployment` label to the pods, whose value is a hash of the configuration, to distinguish the new pods from the old: + +```shell +$ kubectl get pods -l app=nginx -L deployment +NAME READY STATUS RESTARTS AGE DEPLOYMENT +my-nginx-ccba8fbd8cc8160970f63f9a2696fc46-k156z 1/1 Running 0 1m ccba8fbd8cc8160970f63f9a2696fc46 +my-nginx-ccba8fbd8cc8160970f63f9a2696fc46-v95yh 1/1 Running 0 35s ccba8fbd8cc8160970f63f9a2696fc46 +my-nginx-divi2 1/1 Running 0 2h 2d1d7a8f682934a254002b56404b813e +my-nginx-o0ef1 1/1 Running 0 2h 2d1d7a8f682934a254002b56404b813e +my-nginx-q6all 1/1 Running 0 8m 2d1d7a8f682934a254002b56404b813e +``` + +`kubectl rolling-update` reports progress as it progresses: + +``` +Scaling up my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 from 0 to 3, scaling down my-nginx from 3 to 0 (keep 3 pods available, don't exceed 4 pods) +Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 1 +Scaling my-nginx down to 2 +Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 2 +Scaling my-nginx down to 1 +Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 3 +Scaling my-nginx down to 0 +Update succeeded. Deleting old controller: my-nginx +Renaming my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 to my-nginx +replicationcontroller "my-nginx" rolling updated +``` + +If you encounter a problem, you can stop the rolling update midway and revert to the previous version using `--rollback`: + +```shell +$ kubectl rolling-update my-nginx --rollback +Setting "my-nginx" replicas to 1 +Continuing update with existing controller my-nginx. +Scaling up nginx from 1 to 1, scaling down my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 from 1 to 0 (keep 1 pods available, don't exceed 2 pods) +Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 down to 0 +Update succeeded. Deleting my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 +replicationcontroller "my-nginx" rolling updated +``` + +This is one example where the immutability of containers is a huge asset. + +If you need to update more than just the image (e.g., command arguments, environment variables), you can create a new replication controller, with a new name and distinguishing label value, such as: + +```yaml +apiVersion: v1 +kind: ReplicationController +metadata: + name: my-nginx-v4 +spec: + replicas: 5 + selector: + app: nginx + deployment: v4 + template: + metadata: + labels: + app: nginx + deployment: v4 + spec: + containers: + - name: nginx + image: nginx:1.9.2 + args: ["nginx", "-T"] + ports: + - containerPort: 80 +``` + +and roll it out: + +```shell +$ kubectl rolling-update my-nginx -f ./nginx-rc.yaml +Created my-nginx-v4 +Scaling up my-nginx-v4 from 0 to 5, scaling down my-nginx from 4 to 0 (keep 4 pods available, don't exceed 5 pods) +Scaling my-nginx-v4 up to 1 +Scaling my-nginx down to 3 +Scaling my-nginx-v4 up to 2 +Scaling my-nginx down to 2 +Scaling my-nginx-v4 up to 3 +Scaling my-nginx down to 1 +Scaling my-nginx-v4 up to 4 +Scaling my-nginx down to 0 +Scaling my-nginx-v4 up to 5 +Update succeeded. Deleting old controller: my-nginx +replicationcontroller "my-nginx-v4" rolling updated +``` + +You can also run the [update demo](/docs/user-guide/update-demo/) to see a visual representation of the rolling update process. + +## Troubleshooting + +If the `timeout` duration is reached during a rolling update, the operation will +fail with some pods belonging to the new replication controller, and some to the +original controller. + +To continue the update from where it failed, retry using the same command. + +To roll back to the original state before the attempted update, append the +`--rollback=true` flag to the original command. This will revert all changes. diff --git a/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md b/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md index d5df3ede4a..6e33fce6e1 100644 --- a/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md +++ b/docs/tutorials/stateless-application/run-stateless-ap-replication-controller.md @@ -158,7 +158,7 @@ controller should manage pods with the `frontend` label. ## Updating replication controller pods -See [Rolling Updates](/docs/user-guide/rolling-updates/). +See [Rolling Updates](/docs/tasks/run-application/rolling-update-replication-controller/). ## Resizing a replication controller diff --git a/docs/user-guide/horizontal-pod-autoscaling/index.md b/docs/user-guide/horizontal-pod-autoscaling/index.md index 44ad3440e2..b88e5f520a 100644 --- a/docs/user-guide/horizontal-pod-autoscaling/index.md +++ b/docs/user-guide/horizontal-pod-autoscaling/index.md @@ -69,7 +69,7 @@ The detailed documentation of `kubectl autoscale` can be found [here](/docs/user ## Autoscaling during rolling update -Currently in Kubernetes, it is possible to perform a [rolling update](/docs/user-guide/rolling-updates/) by managing replication controllers directly, +Currently in Kubernetes, it is possible to perform a [rolling update](/docs/tasks/run-application/rolling-update-replication-controller/) by managing replication controllers directly, or by using the deployment object, which manages the underlying replication controllers for you. Horizontal Pod Autoscaler only supports the latter approach: the Horizontal Pod Autoscaler is bound to the deployment object, it sets the size for the deployment object, and the deployment is responsible for setting sizes of underlying replication controllers. diff --git a/docs/user-guide/replication-controller/index.md b/docs/user-guide/replication-controller/index.md index 813f4a96b7..824ab21841 100644 --- a/docs/user-guide/replication-controller/index.md +++ b/docs/user-guide/replication-controller/index.md @@ -194,7 +194,7 @@ Ideally, the rolling update controller would take application readiness into acc The two ReplicationControllers would need to create pods with at least one differentiating label, such as the image tag of the primary container of the pod, since it is typically image updates that motivate rolling updates. Rolling update is implemented in the client tool -[`kubectl rolling-update`](/docs/user-guide/kubectl/kubectl_rolling-update). Visit [`kubectl rolling-update` tutorial](/docs/user-guide/rolling-updates/) for more concrete examples. +[`kubectl rolling-update`](/docs/user-guide/kubectl/kubectl_rolling-update). Visit [`kubectl rolling-update` task](/docs/tasks/run-application/rolling-update-replication-controller/) for more concrete examples. ### Multiple release tracks diff --git a/docs/user-guide/rolling-updates.md b/docs/user-guide/rolling-updates.md index d350f41719..996f613c9f 100644 --- a/docs/user-guide/rolling-updates.md +++ b/docs/user-guide/rolling-updates.md @@ -1,256 +1,6 @@ --- -assignees: -- janetkuo title: Rolling Updates --- -* TOC -{:toc} - -## Overview - -To update a service without an outage, `kubectl` supports what is called ['rolling update'](/docs/user-guide/kubectl/kubectl_rolling-update), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) and the [example of rolling update](/docs/user-guide/update-demo/) for more information. - -Note that `kubectl rolling-update` only supports Replication Controllers. However, if you deploy applications with Replication Controllers, -consider switching them to [Deployments](/docs/user-guide/deployments/). A Deployment is a higher-level controller that automates rolling updates -of applications declaratively, and therefore is recommended. If you still want to keep your Replication Controllers and use `kubectl rolling-update`, keep reading: - -A rolling update applies changes to the configuration of pods being managed by -a replication controller. The changes can be passed as a new replication -controller configuration file; or, if only updating the image, a new container -image can be specified directly. - -A rolling update works by: - -1. Creating a new replication controller with the updated configuration. -2. Increasing/decreasing the replica count on the new and old controllers until - the correct number of replicas is reached. -3. Deleting the original replication controller. - -Rolling updates are initiated with the `kubectl rolling-update` command: - - $ kubectl rolling-update NAME \ - ([NEW_NAME] --image=IMAGE | -f FILE) - -## Passing a configuration file - -To initiate a rolling update using a configuration file, pass the new file to -`kubectl rolling-update`: - - $ kubectl rolling-update NAME -f FILE - -The configuration file must: - -* Specify a different `metadata.name` value. - -* Overwrite at least one common label in its `spec.selector` field. - -* Use the same `metadata.namespace`. - -Replication controller configuration files are described in -[Creating Replication Controllers](/docs/user-guide/replication-controller/operations/). - -### Examples - - // Update pods of frontend-v1 using new replication controller data in frontend-v2.json. - $ kubectl rolling-update frontend-v1 -f frontend-v2.json - - // Update pods of frontend-v1 using JSON data passed into stdin. - $ cat frontend-v2.json | kubectl rolling-update frontend-v1 -f - - -## Updating the container image - -To update only the container image, pass a new image name and tag with the -`--image` flag and (optionally) a new controller name: - - $ kubectl rolling-update NAME [NEW_NAME] --image=IMAGE:TAG - -The `--image` flag is only supported for single-container pods. Specifying -`--image` with multi-container pods returns an error. - -If no `NEW_NAME` is specified, a new replication controller is created with -a temporary name. Once the rollout is complete, the old controller is deleted, -and the new controller is updated to use the original name. - -The update will fail if `IMAGE:TAG` is identical to the -current value. For this reason, we recommend the use of versioned tags as -opposed to values such as `:latest`. Doing a rolling update from `image:latest` -to a new `image:latest` will fail, even if the image at that tag has changed. -Moreover, the use of `:latest` is not recommended, see -[Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information. - -### Examples - - // Update the pods of frontend-v1 to frontend-v2 - $ kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2 - - // Update the pods of frontend, keeping the replication controller name - $ kubectl rolling-update frontend --image=image:v2 - -## Required and optional fields - -Required fields are: - -* `NAME`: The name of the replication controller to update. - -as well as either: - -* `-f FILE`: A replication controller configuration file, in either JSON or - YAML format. The configuration file must specify a new top-level `id` value - and include at least one of the existing `spec.selector` key:value pairs. - See the - [Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/#replication-controller-configuration-file) - page for details. -
    -
    - or: -
    -
    -* `--image IMAGE:TAG`: The name and tag of the image to update to. Must be - different than the current image:tag currently specified. - -Optional fields are: - -* `NEW_NAME`: Only used in conjunction with `--image` (not with `-f FILE`). The - name to assign to the new replication controller. -* `--poll-interval DURATION`: The time between polling the controller status - after update. Valid units are `ns` (nanoseconds), `us` or `µs` (microseconds), - `ms` (milliseconds), `s` (seconds), `m` (minutes), or `h` (hours). Units can - be combined (e.g. `1m30s`). The default is `3s`. -* `--timeout DURATION`: The maximum time to wait for the controller to update a - pod before exiting. Default is `5m0s`. Valid units are as described for - `--poll-interval` above. -* `--update-period DURATION`: The time to wait between updating pods. Default - is `1m0s`. Valid units are as described for `--poll-interval` above. - -Additional information about the `kubectl rolling-update` command is available -from the [`kubectl` reference](/docs/user-guide/kubectl/kubectl_rolling-update/). - -## Walkthrough - -Let's say you were running version 1.7.9 of nginx: - -```yaml -apiVersion: v1 -kind: ReplicationController -metadata: - name: my-nginx -spec: - replicas: 5 - template: - metadata: - labels: - app: nginx - spec: - containers: - - name: nginx - image: nginx:1.7.9 - ports: - - containerPort: 80 -``` - -To update to version 1.9.1, you can use [`kubectl rolling-update --image`](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) to specify the new image: - -```shell -$ kubectl rolling-update my-nginx --image=nginx:1.9.1 -Created my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 -``` - -In another window, you can see that `kubectl` added a `deployment` label to the pods, whose value is a hash of the configuration, to distinguish the new pods from the old: - -```shell -$ kubectl get pods -l app=nginx -L deployment -NAME READY STATUS RESTARTS AGE DEPLOYMENT -my-nginx-ccba8fbd8cc8160970f63f9a2696fc46-k156z 1/1 Running 0 1m ccba8fbd8cc8160970f63f9a2696fc46 -my-nginx-ccba8fbd8cc8160970f63f9a2696fc46-v95yh 1/1 Running 0 35s ccba8fbd8cc8160970f63f9a2696fc46 -my-nginx-divi2 1/1 Running 0 2h 2d1d7a8f682934a254002b56404b813e -my-nginx-o0ef1 1/1 Running 0 2h 2d1d7a8f682934a254002b56404b813e -my-nginx-q6all 1/1 Running 0 8m 2d1d7a8f682934a254002b56404b813e -``` - -`kubectl rolling-update` reports progress as it progresses: - -``` -Scaling up my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 from 0 to 3, scaling down my-nginx from 3 to 0 (keep 3 pods available, don't exceed 4 pods) -Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 1 -Scaling my-nginx down to 2 -Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 2 -Scaling my-nginx down to 1 -Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 3 -Scaling my-nginx down to 0 -Update succeeded. Deleting old controller: my-nginx -Renaming my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 to my-nginx -replicationcontroller "my-nginx" rolling updated -``` - -If you encounter a problem, you can stop the rolling update midway and revert to the previous version using `--rollback`: - -```shell -$ kubectl rolling-update my-nginx --rollback -Setting "my-nginx" replicas to 1 -Continuing update with existing controller my-nginx. -Scaling up nginx from 1 to 1, scaling down my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 from 1 to 0 (keep 1 pods available, don't exceed 2 pods) -Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 down to 0 -Update succeeded. Deleting my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 -replicationcontroller "my-nginx" rolling updated -``` - -This is one example where the immutability of containers is a huge asset. - -If you need to update more than just the image (e.g., command arguments, environment variables), you can create a new replication controller, with a new name and distinguishing label value, such as: - -```yaml -apiVersion: v1 -kind: ReplicationController -metadata: - name: my-nginx-v4 -spec: - replicas: 5 - selector: - app: nginx - deployment: v4 - template: - metadata: - labels: - app: nginx - deployment: v4 - spec: - containers: - - name: nginx - image: nginx:1.9.2 - args: ["nginx", "-T"] - ports: - - containerPort: 80 -``` - -and roll it out: - -```shell -$ kubectl rolling-update my-nginx -f ./nginx-rc.yaml -Created my-nginx-v4 -Scaling up my-nginx-v4 from 0 to 5, scaling down my-nginx from 4 to 0 (keep 4 pods available, don't exceed 5 pods) -Scaling my-nginx-v4 up to 1 -Scaling my-nginx down to 3 -Scaling my-nginx-v4 up to 2 -Scaling my-nginx down to 2 -Scaling my-nginx-v4 up to 3 -Scaling my-nginx down to 1 -Scaling my-nginx-v4 up to 4 -Scaling my-nginx down to 0 -Scaling my-nginx-v4 up to 5 -Update succeeded. Deleting old controller: my-nginx -replicationcontroller "my-nginx-v4" rolling updated -``` - -You can also run the [update demo](/docs/user-guide/update-demo/) to see a visual representation of the rolling update process. - -## Troubleshooting - -If the `timeout` duration is reached during a rolling update, the operation will -fail with some pods belonging to the new replication controller, and some to the -original controller. - -To continue the update from where it failed, retry using the same command. - -To roll back to the original state before the attempted update, append the -`--rollback=true` flag to the original command. This will revert all changes. +{% include user-guide-content-moved.md %} +[Rolling Update Replication Controller](/docs/tasks/run-application/rolling-update-replication-controller/) From 3ab1fa20bcbb87e505437eaae45b2e664533b27e Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 17:07:34 -0700 Subject: [PATCH 403/407] Move Kubernetes API page. (#2849) --- _data/concepts.yml | 1 + docs/api.md | 102 +-------------------- docs/concepts/overview/kubernetes-api.md | 109 +++++++++++++++++++++++ 3 files changed, 112 insertions(+), 100 deletions(-) create mode 100644 docs/concepts/overview/kubernetes-api.md diff --git a/_data/concepts.yml b/_data/concepts.yml index b7d07f1910..14a9622998 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -11,6 +11,7 @@ toc: - docs/concepts/overview/working-with-objects/kubernetes-objects.md - docs/concepts/overview/working-with-objects/labels.md - docs/concepts/overview/working-with-objects/annotations.md + - docs/concepts/overview/kubernetes-api.md - title: Kubernetes Objects section: diff --git a/docs/api.md b/docs/api.md index 1c85283028..45d8fcccce 100644 --- a/docs/api.md +++ b/docs/api.md @@ -6,104 +6,6 @@ assignees: title: Kubernetes API Overview --- -Primary system and API concepts are documented in the [User guide](/docs/user-guide/). +{% include user-guide-content-moved.md %} -Overall API conventions are described in the [API conventions doc](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md). - -Remote access to the API is discussed in the [access doc](/docs/admin/accessing-the-api). - -The Kubernetes API also serves as the foundation for the declarative configuration schema for the system. The [Kubectl](/docs/user-guide/kubectl) command-line tool can be used to create, update, delete, and get API objects. - -Kubernetes also stores its serialized state (currently in [etcd](https://coreos.com/docs/distributed-configuration/getting-started-with-etcd/)) in terms of the API resources. - -Kubernetes itself is decomposed into multiple components, which interact through its API. - -## API changes - -In our experience, any system that is successful needs to grow and change as new use cases emerge or existing ones change. Therefore, we expect the Kubernetes API to continuously change and grow. However, we intend to not break compatibility with existing clients, for an extended period of time. In general, new API resources and new resource fields can be expected to be added frequently. Elimination of resources or fields will require following a deprecation process. The precise deprecation policy for eliminating features is TBD, but once we reach our 1.0 milestone, there will be a specific policy. - -What constitutes a compatible change and how to change the API are detailed by the [API change document](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api_changes.md). - -## OpenAPI and Swagger definitions - -Complete API details are documented using [Swagger v1.2](http://swagger.io/) and [OpenAPI](https://www.openapis.org/). The Kubernetes apiserver (aka "master") exposes an API that can be used to retrieve the Swagger v1.2 Kubernetes API spec located at `/swaggerapi`. You can also enable a UI to browse the API documentation at `/swagger-ui` by passing the `--enable-swagger-ui=true` flag to apiserver. - -We also host a version of the [latest v1.2 API documentation UI](http://kubernetes.io/kubernetes/third_party/swagger-ui/). This is updated with the latest release, so if you are using a different version of Kubernetes you will want to use the spec from your apiserver. - -Starting with kubernetes 1.4, OpenAPI spec is also available at `/swagger.json`. While we are transitioning from Swagger v1.2 to OpenAPI (aka Swagger v2.0), some of the tools such as kubectl and swagger-ui are still using v1.2 spec. OpenAPI spec is in Beta as of Kubernetes 1.5. - -Kubernetes implements an alternative Protobuf based serialization format for the API that is primarily intended for intra-cluster communication, documented in the [design proposal](https://github.com/kubernetes/kubernetes/blob/{{ page.githubbranch }}/docs/proposals/protobuf.md) and the IDL files for each schema are located in the Go packages that define the API objects. - -## API versioning - -To make it easier to eliminate fields or restructure resource representations, Kubernetes supports -multiple API versions, each at a different API path, such as `/api/v1` or -`/apis/extensions/v1beta1`. - -We chose to version at the API level rather than at the resource or field level to ensure that the API presents a clear, consistent view of system resources and behavior, and to enable controlling access to end-of-lifed and/or experimental APIs. The JSON and Protobuf serialization schemas follow the same guidelines for schema changes - all descriptions below cover both formats. - -Note that API versioning and Software versioning are only indirectly related. The [API and release -versioning proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/versioning.md) describes the relationship between API versioning and -software versioning. - - -Different API versions imply different levels of stability and support. The criteria for each level are described -in more detail in the [API Changes documentation](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api_changes.md#alpha-beta-and-stable-versions). They are summarized here: - -- Alpha level: - - The version names contain `alpha` (e.g. `v1alpha1`). - - May be buggy. Enabling the feature may expose bugs. Disabled by default. - - Support for feature may be dropped at any time without notice. - - The API may change in incompatible ways in a later software release without notice. - - Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support. -- Beta level: - - The version names contain `beta` (e.g. `v2beta3`). - - Code is well tested. Enabling the feature is considered safe. Enabled by default. - - Support for the overall feature will not be dropped, though details may change. - - The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, - we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating - API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature. - - Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have - multiple clusters which can be upgraded independently, you may be able to relax this restriction. - - **Please do try our beta features and give feedback on them! Once they exit beta, it may not be practical for us to make more changes.** -- Stable level: - - The version name is `vX` where `X` is an integer. - - Stable versions of features will appear in released software for many subsequent versions. - -## API groups - -To make it easier to extend the Kubernetes API, we implemented [*API groups*](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-group.md). -The API group is specified in a REST path and in the `apiVersion` field of a serialized object. - -Currently there are several API groups in use: - -1. the "core" (oftentimes called "legacy", due to not having explicit group name) group, which is at - REST path `/api/v1` and is not specified as part of the `apiVersion` field, e.g. `apiVersion: v1`. -1. the named groups are at REST path `/apis/$GROUP_NAME/$VERSION`, and use `apiVersion: $GROUP_NAME/$VERSION` - (e.g. `apiVersion: batch/v1`). Full list of supported API groups can be seen in [Kubernetes API reference](/docs/reference/). - - -There are two supported paths to extending the API. -1. [Third Party Resources](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md) - are for users with very basic CRUD needs. -1. Coming soon: users needing the full set of Kubernetes API semantics can implement their own apiserver - and use the [aggregator](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md) - to make it seamless for clients. - - -## Enabling API groups - -Certain resources and API groups are enabled by default. They can be enabled or disabled by setting `--runtime-config` -on apiserver. `--runtime-config` accepts comma separated values. For ex: to disable batch/v1, set -`--runtime-config=batch/v1=false`, to enable batch/v2alpha1, set `--runtime-config=batch/v2alpha1`. -The flag accepts comma separated set of key=value pairs describing runtime configuration of the apiserver. - -IMPORTANT: Enabling or disabling groups or resources requires restarting apiserver and controller-manager -to pick up the `--runtime-config` changes. - -## Enabling resources in the groups - -DaemonSets, Deployments, HorizontalPodAutoscalers, Ingress, Jobs and ReplicaSets are enabled by default. -Other extensions resources can be enabled by setting `--runtime-config` on -apiserver. `--runtime-config` accepts comma separated values. For ex: to disable deployments and jobs, set -`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/jobs=false` +[The Kubernetes API](/docs/concepts/overview/kubernetes-api/) diff --git a/docs/concepts/overview/kubernetes-api.md b/docs/concepts/overview/kubernetes-api.md new file mode 100644 index 0000000000..e3cd940ed0 --- /dev/null +++ b/docs/concepts/overview/kubernetes-api.md @@ -0,0 +1,109 @@ +--- +assignees: +- bgrant0607 +- erictune +- lavalamp +title: The Kubernetes API +--- + +Primary system and API concepts are documented in the [User guide](/docs/user-guide/). + +Overall API conventions are described in the [API conventions doc](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md). + +Remote access to the API is discussed in the [access doc](/docs/admin/accessing-the-api). + +The Kubernetes API also serves as the foundation for the declarative configuration schema for the system. The [Kubectl](/docs/user-guide/kubectl) command-line tool can be used to create, update, delete, and get API objects. + +Kubernetes also stores its serialized state (currently in [etcd](https://coreos.com/docs/distributed-configuration/getting-started-with-etcd/)) in terms of the API resources. + +Kubernetes itself is decomposed into multiple components, which interact through its API. + +## API changes + +In our experience, any system that is successful needs to grow and change as new use cases emerge or existing ones change. Therefore, we expect the Kubernetes API to continuously change and grow. However, we intend to not break compatibility with existing clients, for an extended period of time. In general, new API resources and new resource fields can be expected to be added frequently. Elimination of resources or fields will require following a deprecation process. The precise deprecation policy for eliminating features is TBD, but once we reach our 1.0 milestone, there will be a specific policy. + +What constitutes a compatible change and how to change the API are detailed by the [API change document](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api_changes.md). + +## OpenAPI and Swagger definitions + +Complete API details are documented using [Swagger v1.2](http://swagger.io/) and [OpenAPI](https://www.openapis.org/). The Kubernetes apiserver (aka "master") exposes an API that can be used to retrieve the Swagger v1.2 Kubernetes API spec located at `/swaggerapi`. You can also enable a UI to browse the API documentation at `/swagger-ui` by passing the `--enable-swagger-ui=true` flag to apiserver. + +We also host a version of the [latest v1.2 API documentation UI](http://kubernetes.io/kubernetes/third_party/swagger-ui/). This is updated with the latest release, so if you are using a different version of Kubernetes you will want to use the spec from your apiserver. + +Starting with kubernetes 1.4, OpenAPI spec is also available at `/swagger.json`. While we are transitioning from Swagger v1.2 to OpenAPI (aka Swagger v2.0), some of the tools such as kubectl and swagger-ui are still using v1.2 spec. OpenAPI spec is in Beta as of Kubernetes 1.5. + +Kubernetes implements an alternative Protobuf based serialization format for the API that is primarily intended for intra-cluster communication, documented in the [design proposal](https://github.com/kubernetes/kubernetes/blob/{{ page.githubbranch }}/docs/proposals/protobuf.md) and the IDL files for each schema are located in the Go packages that define the API objects. + +## API versioning + +To make it easier to eliminate fields or restructure resource representations, Kubernetes supports +multiple API versions, each at a different API path, such as `/api/v1` or +`/apis/extensions/v1beta1`. + +We chose to version at the API level rather than at the resource or field level to ensure that the API presents a clear, consistent view of system resources and behavior, and to enable controlling access to end-of-lifed and/or experimental APIs. The JSON and Protobuf serialization schemas follow the same guidelines for schema changes - all descriptions below cover both formats. + +Note that API versioning and Software versioning are only indirectly related. The [API and release +versioning proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/versioning.md) describes the relationship between API versioning and +software versioning. + + +Different API versions imply different levels of stability and support. The criteria for each level are described +in more detail in the [API Changes documentation](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api_changes.md#alpha-beta-and-stable-versions). They are summarized here: + +- Alpha level: + - The version names contain `alpha` (e.g. `v1alpha1`). + - May be buggy. Enabling the feature may expose bugs. Disabled by default. + - Support for feature may be dropped at any time without notice. + - The API may change in incompatible ways in a later software release without notice. + - Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support. +- Beta level: + - The version names contain `beta` (e.g. `v2beta3`). + - Code is well tested. Enabling the feature is considered safe. Enabled by default. + - Support for the overall feature will not be dropped, though details may change. + - The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, + we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating + API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature. + - Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have + multiple clusters which can be upgraded independently, you may be able to relax this restriction. + - **Please do try our beta features and give feedback on them! Once they exit beta, it may not be practical for us to make more changes.** +- Stable level: + - The version name is `vX` where `X` is an integer. + - Stable versions of features will appear in released software for many subsequent versions. + +## API groups + +To make it easier to extend the Kubernetes API, we implemented [*API groups*](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-group.md). +The API group is specified in a REST path and in the `apiVersion` field of a serialized object. + +Currently there are several API groups in use: + +1. the "core" (oftentimes called "legacy", due to not having explicit group name) group, which is at + REST path `/api/v1` and is not specified as part of the `apiVersion` field, e.g. `apiVersion: v1`. +1. the named groups are at REST path `/apis/$GROUP_NAME/$VERSION`, and use `apiVersion: $GROUP_NAME/$VERSION` + (e.g. `apiVersion: batch/v1`). Full list of supported API groups can be seen in [Kubernetes API reference](/docs/reference/). + + +There are two supported paths to extending the API. +1. [Third Party Resources](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md) + are for users with very basic CRUD needs. +1. Coming soon: users needing the full set of Kubernetes API semantics can implement their own apiserver + and use the [aggregator](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md) + to make it seamless for clients. + + +## Enabling API groups + +Certain resources and API groups are enabled by default. They can be enabled or disabled by setting `--runtime-config` +on apiserver. `--runtime-config` accepts comma separated values. For ex: to disable batch/v1, set +`--runtime-config=batch/v1=false`, to enable batch/v2alpha1, set `--runtime-config=batch/v2alpha1`. +The flag accepts comma separated set of key=value pairs describing runtime configuration of the apiserver. + +IMPORTANT: Enabling or disabling groups or resources requires restarting apiserver and controller-manager +to pick up the `--runtime-config` changes. + +## Enabling resources in the groups + +DaemonSets, Deployments, HorizontalPodAutoscalers, Ingress, Jobs and ReplicaSets are enabled by default. +Other extensions resources can be enabled by setting `--runtime-config` on +apiserver. `--runtime-config` accepts comma separated values. For ex: to disable deployments and jobs, set +`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/jobs=false` From d01672cd2ddf74a81a60a0dd87028e03c603cc14 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Wed, 15 Mar 2017 17:21:06 -0700 Subject: [PATCH 404/407] Move What is Kubernetes topic. (#2851) --- _data/concepts.yml | 1 + docs/concepts/overview/what-is-kubernetes.md | 117 +++++++++++++++++++ docs/whatisk8s.md | 111 +----------------- 3 files changed, 120 insertions(+), 109 deletions(-) create mode 100644 docs/concepts/overview/what-is-kubernetes.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 14a9622998..50b2a3e290 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -5,6 +5,7 @@ toc: - title: Overview section: + - docs/concepts/overview/what-is-kubernetes.md - docs/concepts/overview/components.md - title: Working with Kubernetes Objects section: diff --git a/docs/concepts/overview/what-is-kubernetes.md b/docs/concepts/overview/what-is-kubernetes.md new file mode 100644 index 0000000000..8a9d5b04db --- /dev/null +++ b/docs/concepts/overview/what-is-kubernetes.md @@ -0,0 +1,117 @@ +--- +assignees: +- bgrant0607 +- mikedanese +title: What is Kubernetes? +--- + +Kubernetes is an [open-source platform for automating deployment, scaling, and operations of application containers](http://www.slideshare.net/BrianGrant11/wso2con-us-2015-kubernetes-a-platform-for-automating-deployment-scaling-and-operations) across clusters of hosts, providing container-centric infrastructure. + +With Kubernetes, you are able to quickly and efficiently respond to customer demand: + + - Deploy your applications quickly and predictably. + - Scale your applications on the fly. + - Seamlessly roll out new features. + - Optimize use of your hardware by using only the resources you need. + +Our goal is to foster an ecosystem of components and tools that relieve the burden of running applications in public and private clouds. + +#### Kubernetes is: + +* **portable**: public, private, hybrid, multi-cloud +* **extensible**: modular, pluggable, hookable, composable +* **self-healing**: auto-placement, auto-restart, auto-replication, auto-scaling + +The Kubernetes project was started by Google in 2014. Kubernetes builds upon a [decade and a half of experience that Google has with running production workloads at scale](https://research.google.com/pubs/pub43438.html), combined with best-of-breed ideas and practices from the community. + +##### Ready to [Get Started](/docs/getting-started-guides/)? + +## Why containers? + +Looking for reasons why you should be using [containers](http://aucouranton.com/2014/06/13/linux-containers-parallels-lxc-openvz-docker-and-more/)? + +![Why Containers?](/images/docs/why_containers.svg) + +The *Old Way* to deploy applications was to install the applications on a host using the operating system package manager. This had the disadvantage of entangling the applications' executables, configuration, libraries, and lifecycles with each other and with the host OS. One could build immutable virtual-machine images in order to achieve predictable rollouts and rollbacks, but VMs are heavyweight and non-portable. + +The *New Way* is to deploy containers based on operating-system-level virtualization rather than hardware virtualization. These containers are isolated from each other and from the host: they have their own filesystems, they can't see each others' processes, and their computational resource usage can be bounded. They are easier to build than VMs, and because they are decoupled from the underlying infrastructure and from the host filesystem, they are portable across clouds and OS distributions. + +Because containers are small and fast, one application can be packed in each container image. This one-to-one application-to-image relationship unlocks the full benefits of containers. With containers, immutable container images can be created at build/release time rather than deployment time, since each application doesn't need to be composed with the rest of the application stack, nor married to the production infrastructure environment. Generating container images at build/release time enables a consistent environment to be carried from development into production. +Similarly, containers are vastly more transparent than VMs, which facilitates monitoring and management. This is especially true when the containers' process lifecycles are managed by the infrastructure rather than hidden by a process supervisor inside the container. Finally, with a single application per container, managing the containers becomes tantamount to managing deployment of the application. + +Summary of container benefits: + +* **Agile application creation and deployment**: + Increased ease and efficiency of container image creation compared to VM image use. +* **Continuous development, integration, and deployment**: + Provides for reliable and frequent container image build and deployment with quick and easy rollbacks (due to image immutability). +* **Dev and Ops separation of concerns**: + Create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure. +* **Environmental consistency across development, testing, and production**: + Runs the same on a laptop as it does in the cloud. +* **Cloud and OS distribution portability**: + Runs on Ubuntu, RHEL, CoreOS, on-prem, Google Container Engine, and anywhere else. +* **Application-centric management**: + Raises the level of abstraction from running an OS on virtual hardware to run an application on an OS using logical resources. +* **Loosely coupled, distributed, elastic, liberated [micro-services](http://martinfowler.com/articles/microservices.html)**: + Applications are broken into smaller, independent pieces and can be deployed and managed dynamically -- not a fat monolithic stack running on one big single-purpose machine. +* **Resource isolation**: + Predictable application performance. +* **Resource utilization**: + High efficiency and density. + +#### Why do I need Kubernetes and what can it do? + +At a minimum, Kubernetes can schedule and run application containers on clusters of physical or virtual machines. However, Kubernetes also allows developers to 'cut the cord' to physical and virtual machines, moving from a **host-centric** infrastructure to a **container-centric** infrastructure, which provides the full advantages and benefits inherent to containers. Kubernetes provides the infrastructure to build a truly **container-centric** development environment. + +Kubernetes satisfies a number of common needs of applications running in production, such as: + +* [co-locating helper processes](/docs/user-guide/pods/), facilitating composite applications and preserving the one-application-per-container model, +* [mounting storage systems](/docs/user-guide/volumes/), +* [distributing secrets](/docs/user-guide/secrets/), +* [application health checking](/docs/user-guide/production-pods/#liveness-and-readiness-probes-aka-health-checks), +* [replicating application instances](/docs/user-guide/replication-controller/), +* [horizontal auto-scaling](/docs/user-guide/horizontal-pod-autoscaling/), +* [naming and discovery](/docs/user-guide/connecting-applications/), +* [load balancing](/docs/user-guide/services/), +* [rolling updates](/docs/user-guide/update-demo/), +* [resource monitoring](/docs/user-guide/monitoring/), +* [log access and ingestion](/docs/user-guide/logging/overview/), +* [support for introspection and debugging](/docs/user-guide/introspection-and-debugging/), and +* [identity and authorization](/docs/admin/authorization/). + +This provides the simplicity of Platform as a Service (PaaS) with the flexibility of Infrastructure as a Service (IaaS), and facilitates portability across infrastructure providers. + +For more details, see the [user guide](/docs/user-guide/). + +#### Why and how is Kubernetes a platform? + +Even though Kubernetes provides a lot of functionality, there are always new scenarios that would benefit from new features. Application-specific workflows can be streamlined to accelerate developer velocity. Ad hoc orchestration that is acceptable initially often requires robust automation at scale. This is why Kubernetes was also designed to serve as a platform for building an ecosystem of components and tools to make it easier to deploy, scale, and manage applications. + +[Labels](/docs/user-guide/labels/) empower users to organize their resources however they please. [Annotations](/docs/user-guide/annotations/) enable users to decorate resources with custom information to facilitate their workflows and provide an easy way for management tools to checkpoint state. + +Additionally, the [Kubernetes control plane](/docs/admin/cluster-components) is built upon the same [APIs](/docs/api/) that are available to developers and users. Users can write their own controllers, [schedulers](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/scheduler.md), etc., if they choose, with [their own APIs](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/extending-api.md) that can be targeted by a general-purpose [command-line tool](/docs/user-guide/kubectl-overview/). + +This [design](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/principles.md) has enabled a number of other systems to build atop Kubernetes. + +#### Kubernetes is not: + +Kubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. We preserve user choice where it is important. + +* Kubernetes does not limit the types of applications supported. It does not dictate application frameworks (e.g., [Wildfly](http://wildfly.org/)), restrict the set of supported language runtimes (e.g., Java, Python, Ruby), cater to only [12-factor applications](http://12factor.net/), nor distinguish "apps" from "services". Kubernetes aims to support an extremely diverse variety of workloads, including stateless, stateful, and data-processing workloads. If an application can run in a container, it should run great on Kubernetes. +* Kubernetes does not provide middleware (e.g., message buses), data-processing frameworks (e.g., Spark), databases (e.g., mysql), nor cluster storage systems (e.g., Ceph) as built-in services. Such applications run on Kubernetes. +* Kubernetes does not have a click-to-deploy service marketplace. +* Kubernetes is unopinionated in the source-to-image space. It does not deploy source code and does not build your application. Continuous Integration (CI) workflow is an area where different users and projects have their own requirements and preferences, so we support layering CI workflows on Kubernetes but don't dictate how it should work. +* Kubernetes allows users to choose the logging, monitoring, and alerting systems of their choice. (Though we do provide some integrations as proof of concept.) +* Kubernetes does not provide nor mandate a comprehensive application configuration language/system (e.g., [jsonnet](https://github.com/google/jsonnet)). +* Kubernetes does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems. + +On the other hand, a number of PaaS systems run *on* Kubernetes, such as [Openshift](https://github.com/openshift/origin), [Deis](http://deis.io/), and [Eldarion](http://eldarion.cloud/). You could also roll your own custom PaaS, integrate with a CI system of your choice, or get along just fine with just Kubernetes: bring your container images and deploy them on Kubernetes. + +Since Kubernetes operates at the application level rather than at just the hardware level, it provides some generally applicable features common to PaaS offerings, such as deployment, scaling, load balancing, logging, monitoring, etc. However, Kubernetes is not monolithic, and these default solutions are optional and pluggable. + +Additionally, Kubernetes is not a mere "orchestration system"; it eliminates the need for orchestration. The technical definition of "orchestration" is execution of a defined workflow: do A, then B, then C. In contrast, Kubernetes is comprised of a set of independent, composable control processes that continuously drive current state towards the provided desired state. It shouldn't matter how you get from A to C: make it so. Centralized control is also not required; the approach is more akin to "choreography". This results in a system that is easier to use and more powerful, robust, resilient, and extensible. + +#### What does *Kubernetes* mean? K8s? + +The name **Kubernetes** originates from Greek, meaning "helmsman" or "pilot", and is the root of "governor" and ["cybernetic"](http://www.etymonline.com/index.php?term=cybernetics). **K8s** is an abbreviation derived by replacing the 8 letters "ubernete" with 8. diff --git a/docs/whatisk8s.md b/docs/whatisk8s.md index 8a9d5b04db..0b81cc80c5 100644 --- a/docs/whatisk8s.md +++ b/docs/whatisk8s.md @@ -5,113 +5,6 @@ assignees: title: What is Kubernetes? --- -Kubernetes is an [open-source platform for automating deployment, scaling, and operations of application containers](http://www.slideshare.net/BrianGrant11/wso2con-us-2015-kubernetes-a-platform-for-automating-deployment-scaling-and-operations) across clusters of hosts, providing container-centric infrastructure. +{% include user-guide-content-moved.md %} -With Kubernetes, you are able to quickly and efficiently respond to customer demand: - - - Deploy your applications quickly and predictably. - - Scale your applications on the fly. - - Seamlessly roll out new features. - - Optimize use of your hardware by using only the resources you need. - -Our goal is to foster an ecosystem of components and tools that relieve the burden of running applications in public and private clouds. - -#### Kubernetes is: - -* **portable**: public, private, hybrid, multi-cloud -* **extensible**: modular, pluggable, hookable, composable -* **self-healing**: auto-placement, auto-restart, auto-replication, auto-scaling - -The Kubernetes project was started by Google in 2014. Kubernetes builds upon a [decade and a half of experience that Google has with running production workloads at scale](https://research.google.com/pubs/pub43438.html), combined with best-of-breed ideas and practices from the community. - -##### Ready to [Get Started](/docs/getting-started-guides/)? - -## Why containers? - -Looking for reasons why you should be using [containers](http://aucouranton.com/2014/06/13/linux-containers-parallels-lxc-openvz-docker-and-more/)? - -![Why Containers?](/images/docs/why_containers.svg) - -The *Old Way* to deploy applications was to install the applications on a host using the operating system package manager. This had the disadvantage of entangling the applications' executables, configuration, libraries, and lifecycles with each other and with the host OS. One could build immutable virtual-machine images in order to achieve predictable rollouts and rollbacks, but VMs are heavyweight and non-portable. - -The *New Way* is to deploy containers based on operating-system-level virtualization rather than hardware virtualization. These containers are isolated from each other and from the host: they have their own filesystems, they can't see each others' processes, and their computational resource usage can be bounded. They are easier to build than VMs, and because they are decoupled from the underlying infrastructure and from the host filesystem, they are portable across clouds and OS distributions. - -Because containers are small and fast, one application can be packed in each container image. This one-to-one application-to-image relationship unlocks the full benefits of containers. With containers, immutable container images can be created at build/release time rather than deployment time, since each application doesn't need to be composed with the rest of the application stack, nor married to the production infrastructure environment. Generating container images at build/release time enables a consistent environment to be carried from development into production. -Similarly, containers are vastly more transparent than VMs, which facilitates monitoring and management. This is especially true when the containers' process lifecycles are managed by the infrastructure rather than hidden by a process supervisor inside the container. Finally, with a single application per container, managing the containers becomes tantamount to managing deployment of the application. - -Summary of container benefits: - -* **Agile application creation and deployment**: - Increased ease and efficiency of container image creation compared to VM image use. -* **Continuous development, integration, and deployment**: - Provides for reliable and frequent container image build and deployment with quick and easy rollbacks (due to image immutability). -* **Dev and Ops separation of concerns**: - Create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure. -* **Environmental consistency across development, testing, and production**: - Runs the same on a laptop as it does in the cloud. -* **Cloud and OS distribution portability**: - Runs on Ubuntu, RHEL, CoreOS, on-prem, Google Container Engine, and anywhere else. -* **Application-centric management**: - Raises the level of abstraction from running an OS on virtual hardware to run an application on an OS using logical resources. -* **Loosely coupled, distributed, elastic, liberated [micro-services](http://martinfowler.com/articles/microservices.html)**: - Applications are broken into smaller, independent pieces and can be deployed and managed dynamically -- not a fat monolithic stack running on one big single-purpose machine. -* **Resource isolation**: - Predictable application performance. -* **Resource utilization**: - High efficiency and density. - -#### Why do I need Kubernetes and what can it do? - -At a minimum, Kubernetes can schedule and run application containers on clusters of physical or virtual machines. However, Kubernetes also allows developers to 'cut the cord' to physical and virtual machines, moving from a **host-centric** infrastructure to a **container-centric** infrastructure, which provides the full advantages and benefits inherent to containers. Kubernetes provides the infrastructure to build a truly **container-centric** development environment. - -Kubernetes satisfies a number of common needs of applications running in production, such as: - -* [co-locating helper processes](/docs/user-guide/pods/), facilitating composite applications and preserving the one-application-per-container model, -* [mounting storage systems](/docs/user-guide/volumes/), -* [distributing secrets](/docs/user-guide/secrets/), -* [application health checking](/docs/user-guide/production-pods/#liveness-and-readiness-probes-aka-health-checks), -* [replicating application instances](/docs/user-guide/replication-controller/), -* [horizontal auto-scaling](/docs/user-guide/horizontal-pod-autoscaling/), -* [naming and discovery](/docs/user-guide/connecting-applications/), -* [load balancing](/docs/user-guide/services/), -* [rolling updates](/docs/user-guide/update-demo/), -* [resource monitoring](/docs/user-guide/monitoring/), -* [log access and ingestion](/docs/user-guide/logging/overview/), -* [support for introspection and debugging](/docs/user-guide/introspection-and-debugging/), and -* [identity and authorization](/docs/admin/authorization/). - -This provides the simplicity of Platform as a Service (PaaS) with the flexibility of Infrastructure as a Service (IaaS), and facilitates portability across infrastructure providers. - -For more details, see the [user guide](/docs/user-guide/). - -#### Why and how is Kubernetes a platform? - -Even though Kubernetes provides a lot of functionality, there are always new scenarios that would benefit from new features. Application-specific workflows can be streamlined to accelerate developer velocity. Ad hoc orchestration that is acceptable initially often requires robust automation at scale. This is why Kubernetes was also designed to serve as a platform for building an ecosystem of components and tools to make it easier to deploy, scale, and manage applications. - -[Labels](/docs/user-guide/labels/) empower users to organize their resources however they please. [Annotations](/docs/user-guide/annotations/) enable users to decorate resources with custom information to facilitate their workflows and provide an easy way for management tools to checkpoint state. - -Additionally, the [Kubernetes control plane](/docs/admin/cluster-components) is built upon the same [APIs](/docs/api/) that are available to developers and users. Users can write their own controllers, [schedulers](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/scheduler.md), etc., if they choose, with [their own APIs](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/extending-api.md) that can be targeted by a general-purpose [command-line tool](/docs/user-guide/kubectl-overview/). - -This [design](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/principles.md) has enabled a number of other systems to build atop Kubernetes. - -#### Kubernetes is not: - -Kubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. We preserve user choice where it is important. - -* Kubernetes does not limit the types of applications supported. It does not dictate application frameworks (e.g., [Wildfly](http://wildfly.org/)), restrict the set of supported language runtimes (e.g., Java, Python, Ruby), cater to only [12-factor applications](http://12factor.net/), nor distinguish "apps" from "services". Kubernetes aims to support an extremely diverse variety of workloads, including stateless, stateful, and data-processing workloads. If an application can run in a container, it should run great on Kubernetes. -* Kubernetes does not provide middleware (e.g., message buses), data-processing frameworks (e.g., Spark), databases (e.g., mysql), nor cluster storage systems (e.g., Ceph) as built-in services. Such applications run on Kubernetes. -* Kubernetes does not have a click-to-deploy service marketplace. -* Kubernetes is unopinionated in the source-to-image space. It does not deploy source code and does not build your application. Continuous Integration (CI) workflow is an area where different users and projects have their own requirements and preferences, so we support layering CI workflows on Kubernetes but don't dictate how it should work. -* Kubernetes allows users to choose the logging, monitoring, and alerting systems of their choice. (Though we do provide some integrations as proof of concept.) -* Kubernetes does not provide nor mandate a comprehensive application configuration language/system (e.g., [jsonnet](https://github.com/google/jsonnet)). -* Kubernetes does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems. - -On the other hand, a number of PaaS systems run *on* Kubernetes, such as [Openshift](https://github.com/openshift/origin), [Deis](http://deis.io/), and [Eldarion](http://eldarion.cloud/). You could also roll your own custom PaaS, integrate with a CI system of your choice, or get along just fine with just Kubernetes: bring your container images and deploy them on Kubernetes. - -Since Kubernetes operates at the application level rather than at just the hardware level, it provides some generally applicable features common to PaaS offerings, such as deployment, scaling, load balancing, logging, monitoring, etc. However, Kubernetes is not monolithic, and these default solutions are optional and pluggable. - -Additionally, Kubernetes is not a mere "orchestration system"; it eliminates the need for orchestration. The technical definition of "orchestration" is execution of a defined workflow: do A, then B, then C. In contrast, Kubernetes is comprised of a set of independent, composable control processes that continuously drive current state towards the provided desired state. It shouldn't matter how you get from A to C: make it so. Centralized control is also not required; the approach is more akin to "choreography". This results in a system that is easier to use and more powerful, robust, resilient, and extensible. - -#### What does *Kubernetes* mean? K8s? - -The name **Kubernetes** originates from Greek, meaning "helmsman" or "pilot", and is the root of "governor" and ["cybernetic"](http://www.etymonline.com/index.php?term=cybernetics). **K8s** is an abbreviation derived by replacing the 8 letters "ubernete" with 8. +[What is Kubernetes?](/docs/concepts/overview/what-is-kubernetes/) From b65d0a30c971d7cbd974dec9a0680d729dc69a16 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 17:31:38 -0700 Subject: [PATCH 405/407] Move Guide topic: Rolling Update Demo (#2850) * Move Guide topic: Rolling Update Demo * rename file --- docs/concepts/overview/what-is-kubernetes.md | 2 +- .../rolling-update-replication-controller.md | 4 +- docs/user-guide/deployments.md | 2 +- docs/user-guide/index.md | 2 +- docs/user-guide/update-demo/index.md | 105 +---------------- docs/user-guide/update-demo/index.md.orig | 107 ++++++++++++++++++ 6 files changed, 114 insertions(+), 108 deletions(-) create mode 100644 docs/user-guide/update-demo/index.md.orig diff --git a/docs/concepts/overview/what-is-kubernetes.md b/docs/concepts/overview/what-is-kubernetes.md index 8a9d5b04db..378a1f2c47 100644 --- a/docs/concepts/overview/what-is-kubernetes.md +++ b/docs/concepts/overview/what-is-kubernetes.md @@ -74,7 +74,7 @@ Kubernetes satisfies a number of common needs of applications running in product * [horizontal auto-scaling](/docs/user-guide/horizontal-pod-autoscaling/), * [naming and discovery](/docs/user-guide/connecting-applications/), * [load balancing](/docs/user-guide/services/), -* [rolling updates](/docs/user-guide/update-demo/), +* [rolling updates](/docs/tasks/run-application/rolling-update-replication-controller/), * [resource monitoring](/docs/user-guide/monitoring/), * [log access and ingestion](/docs/user-guide/logging/overview/), * [support for introspection and debugging](/docs/user-guide/introspection-and-debugging/), and diff --git a/docs/tasks/run-application/rolling-update-replication-controller.md b/docs/tasks/run-application/rolling-update-replication-controller.md index 19aadec01a..ae64d478a6 100644 --- a/docs/tasks/run-application/rolling-update-replication-controller.md +++ b/docs/tasks/run-application/rolling-update-replication-controller.md @@ -9,7 +9,7 @@ title: Rolling Update Replication Controller ## Overview -To update a service without an outage, `kubectl` supports what is called ['rolling update'](/docs/user-guide/kubectl/kubectl_rolling-update), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) and the [example of rolling update](/docs/user-guide/update-demo/) for more information. +To update a service without an outage, `kubectl` supports what is called ['rolling update'](/docs/user-guide/kubectl/kubectl_rolling-update), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) and the [example of rolling update](/docs/tasks/run-application/rolling-update-replication-controller/) for more information. Note that `kubectl rolling-update` only supports Replication Controllers. However, if you deploy applications with Replication Controllers, consider switching them to [Deployments](/docs/user-guide/deployments/). A Deployment is a higher-level controller that automates rolling updates @@ -242,7 +242,7 @@ Update succeeded. Deleting old controller: my-nginx replicationcontroller "my-nginx-v4" rolling updated ``` -You can also run the [update demo](/docs/user-guide/update-demo/) to see a visual representation of the rolling update process. +You can also run the [update demo](/docs/tasks/run-application/rolling-update-replication-controller/) to see a visual representation of the rolling update process. ## Troubleshooting diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index a55d1dd255..7610384eab 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -778,7 +778,7 @@ All existing Pods are killed before new ones are created when #### Rolling Update Deployment -The Deployment updates Pods in a [rolling update](/docs/user-guide/update-demo/) fashion +The Deployment updates Pods in a [rolling update](/docs/tasks/run-application/rolling-update-replication-controller/) fashion when `.spec.strategy.type==RollingUpdate`. You can specify `maxUnavailable` and `maxSurge` to control the rolling update process. diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index d7d7891142..c62ff4574c 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -87,6 +87,6 @@ Pods and containers * [Migrating from docker-cli to kubectl](/docs/user-guide/docker-cli-to-kubectl/) * [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/) * [Assign pods to selected nodes](/docs/user-guide/node-selection/) - * [Perform a rolling update on a running group of pods](/docs/user-guide/update-demo/) + * [Perform a rolling update on a running group of pods](/docs/tasks/run-application/rolling-update-replication-controller/) [Developer Guide]: https://github.com/kubernetes/community/blob/master/contributors/devel/README.md diff --git a/docs/user-guide/update-demo/index.md b/docs/user-guide/update-demo/index.md index 14bf74ee1f..610c884f24 100644 --- a/docs/user-guide/update-demo/index.md +++ b/docs/user-guide/update-demo/index.md @@ -1,107 +1,6 @@ --- -assignees: -- mikedanese title: Rolling Update Demo --- -This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/concepts/cluster-administration/manage-deployment/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information. - -The files for this example are viewable in [our docs repo -here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/update-demo). - -### Step Zero: Prerequisites - -This example assumes that you have forked the docs repository and [turned up a Kubernetes cluster](/docs/getting-started-guides/): - -```shell -$ git clone -b {{page.docsbranch}} https://github.com/kubernetes/kubernetes.github.io -$ cd kubernetes.github.io -``` - -### Step One: Turn up the UX for the demo - -You can use bash job control to run this in the background (note that you must use the default port -- 8001 -- for the following demonstration to work properly). -This can sometimes spew to the output so you could also run it in a different terminal. You have to run `kubectl proxy` in the root of the -Kubernetes repository. Otherwise you will get "404 page not found" errors as the paths will not match. You can find more information about `kubectl proxy` -[here](/docs/user-guide/kubectl/kubectl_proxy). - -```shell -$ kubectl proxy --www=docs/user-guide/update-demo/local/ & -I0218 15:18:31.623279 67480 proxy.go:36] Starting to serve on localhost:8001 -``` - -Now visit the [demo website](http://localhost:8001/static). You won't see anything much quite yet. - -### Step Two: Run the replication controller - -Now we will turn up two replicas of an [image](/docs/user-guide/images/). They all serve on internal port 80. - -```shell -$ kubectl create -f docs/user-guide/update-demo/nautilus-rc.yaml -``` - -After pulling the image from the Docker Hub to your worker nodes (which may take a minute or so) you'll see a couple of squares in the UI detailing the pods that are running along with the image that they are serving up. A cute little nautilus. - -### Step Three: Try scaling the replication controller - -Now we will increase the number of replicas from two to four: - -```shell -$ kubectl scale rc update-demo-nautilus --replicas=4 -``` - -If you go back to the [demo website](http://localhost:8001/static/index.html) you should eventually see four boxes, one for each pod. - -### Step Four: Update the docker image - -We will now update the docker image to serve a different image by doing a rolling update to a new Docker image. - -```shell -$ kubectl rolling-update update-demo-nautilus --update-period=10s -f docs/user-guide/update-demo/kitten-rc.yaml -``` - -The rolling-update command in kubectl will do 2 things: - -1. Create a new [replication controller](/docs/user-guide/replication-controller/) with a pod template that uses the new image (`gcr.io/google_containers/update-demo:kitten`) -2. Scale the old and new replication controllers until the new controller replaces the old. This will kill the current pods one at a time, spinning up new ones to replace them. - -Watch the [demo website](http://localhost:8001/static/index.html), it will update one pod every 10 seconds until all of the pods have the new image. -Note that the new replication controller definition does not include the replica count, so the current replica count of the old replication controller is preserved. -But if the replica count had been specified, the final replica count of the new replication controller will be equal to this number. - -### Step Five: Bring down the pods - -```shell -$ kubectl delete rc update-demo-kitten -``` - -This first stops the replication controller by turning the target number of replicas to 0 and then deletes the controller. - -### Step Six: Cleanup - -After you are done running this demo make sure to kill the proxy running in the background: - -```shell -$ jobs -[1]+ Running ./kubectl proxy --www=local/ & -$ kill %1 -[1]+ Terminated: 15 ./kubectl proxy --www=local/ -``` - -### Updating the Docker images - -If you want to build your own docker images, you can set `$DOCKER_HUB_USER` to your Docker user id and run the included shell script. It can take a few minutes to download/upload stuff. - -```shell -$ export DOCKER_HUB_USER=my-docker-id -$ ./docs/user-guide/update-demo/build-images.sh -``` - -To use your custom docker image in the above examples, you will need to change the image name in `docs/user-guide/update-demo/nautilus-rc.yaml` and `docs/user-guide/update-demo/kitten-rc.yaml`. - -### Image Copyright - -Note that the images included here are public domain. - -* [kitten](http://commons.wikimedia.org/wiki/File:Kitten-stare.jpg) -* [nautilus](http://commons.wikimedia.org/wiki/File:Nautilus_pompilius.jpg) +{% include user-guide-content-moved.md %} +[Rolling Update Replication Controller](/docs/tasks/run-application/rolling-update-replication-controller/) diff --git a/docs/user-guide/update-demo/index.md.orig b/docs/user-guide/update-demo/index.md.orig new file mode 100644 index 0000000000..14bf74ee1f --- /dev/null +++ b/docs/user-guide/update-demo/index.md.orig @@ -0,0 +1,107 @@ +--- +assignees: +- mikedanese +title: Rolling Update Demo +--- + +This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/concepts/cluster-administration/manage-deployment/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information. + +The files for this example are viewable in [our docs repo +here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/update-demo). + +### Step Zero: Prerequisites + +This example assumes that you have forked the docs repository and [turned up a Kubernetes cluster](/docs/getting-started-guides/): + +```shell +$ git clone -b {{page.docsbranch}} https://github.com/kubernetes/kubernetes.github.io +$ cd kubernetes.github.io +``` + +### Step One: Turn up the UX for the demo + +You can use bash job control to run this in the background (note that you must use the default port -- 8001 -- for the following demonstration to work properly). +This can sometimes spew to the output so you could also run it in a different terminal. You have to run `kubectl proxy` in the root of the +Kubernetes repository. Otherwise you will get "404 page not found" errors as the paths will not match. You can find more information about `kubectl proxy` +[here](/docs/user-guide/kubectl/kubectl_proxy). + +```shell +$ kubectl proxy --www=docs/user-guide/update-demo/local/ & +I0218 15:18:31.623279 67480 proxy.go:36] Starting to serve on localhost:8001 +``` + +Now visit the [demo website](http://localhost:8001/static). You won't see anything much quite yet. + +### Step Two: Run the replication controller + +Now we will turn up two replicas of an [image](/docs/user-guide/images/). They all serve on internal port 80. + +```shell +$ kubectl create -f docs/user-guide/update-demo/nautilus-rc.yaml +``` + +After pulling the image from the Docker Hub to your worker nodes (which may take a minute or so) you'll see a couple of squares in the UI detailing the pods that are running along with the image that they are serving up. A cute little nautilus. + +### Step Three: Try scaling the replication controller + +Now we will increase the number of replicas from two to four: + +```shell +$ kubectl scale rc update-demo-nautilus --replicas=4 +``` + +If you go back to the [demo website](http://localhost:8001/static/index.html) you should eventually see four boxes, one for each pod. + +### Step Four: Update the docker image + +We will now update the docker image to serve a different image by doing a rolling update to a new Docker image. + +```shell +$ kubectl rolling-update update-demo-nautilus --update-period=10s -f docs/user-guide/update-demo/kitten-rc.yaml +``` + +The rolling-update command in kubectl will do 2 things: + +1. Create a new [replication controller](/docs/user-guide/replication-controller/) with a pod template that uses the new image (`gcr.io/google_containers/update-demo:kitten`) +2. Scale the old and new replication controllers until the new controller replaces the old. This will kill the current pods one at a time, spinning up new ones to replace them. + +Watch the [demo website](http://localhost:8001/static/index.html), it will update one pod every 10 seconds until all of the pods have the new image. +Note that the new replication controller definition does not include the replica count, so the current replica count of the old replication controller is preserved. +But if the replica count had been specified, the final replica count of the new replication controller will be equal to this number. + +### Step Five: Bring down the pods + +```shell +$ kubectl delete rc update-demo-kitten +``` + +This first stops the replication controller by turning the target number of replicas to 0 and then deletes the controller. + +### Step Six: Cleanup + +After you are done running this demo make sure to kill the proxy running in the background: + +```shell +$ jobs +[1]+ Running ./kubectl proxy --www=local/ & +$ kill %1 +[1]+ Terminated: 15 ./kubectl proxy --www=local/ +``` + +### Updating the Docker images + +If you want to build your own docker images, you can set `$DOCKER_HUB_USER` to your Docker user id and run the included shell script. It can take a few minutes to download/upload stuff. + +```shell +$ export DOCKER_HUB_USER=my-docker-id +$ ./docs/user-guide/update-demo/build-images.sh +``` + +To use your custom docker image in the above examples, you will need to change the image name in `docs/user-guide/update-demo/nautilus-rc.yaml` and `docs/user-guide/update-demo/kitten-rc.yaml`. + +### Image Copyright + +Note that the images included here are public domain. + +* [kitten](http://commons.wikimedia.org/wiki/File:Kitten-stare.jpg) +* [nautilus](http://commons.wikimedia.org/wiki/File:Nautilus_pompilius.jpg) From ba34326fa0518c998dd3ad4305887b5917c1d7e5 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 17:41:59 -0700 Subject: [PATCH 406/407] Move Guide topic: Configuration Best Practices --- _data/concepts.yml | 1 + .../manage-deployment.md | 2 +- docs/concepts/configuration/overview.md | 119 ++++++++++++++++++ .../rolling-update-replication-controller.md | 2 +- docs/user-guide/config-best-practices.md | 116 +---------------- docs/user-guide/images.md | 2 +- docs/user-guide/index.md | 2 +- docs/user-guide/kubectl-conventions.md | 2 +- docs/user-guide/kubectl-overview.md | 2 +- 9 files changed, 128 insertions(+), 120 deletions(-) create mode 100644 docs/concepts/configuration/overview.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 50b2a3e290..f0059c0db7 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -52,6 +52,7 @@ toc: - title: Configuration section: + - docs/concepts/configuration/overview.md - docs/concepts/configuration/container-command-args.md - docs/concepts/configuration/manage-compute-resources-container.md diff --git a/docs/concepts/cluster-administration/manage-deployment.md b/docs/concepts/cluster-administration/manage-deployment.md index c1da7f3d5f..aef8bbfaf8 100644 --- a/docs/concepts/cluster-administration/manage-deployment.md +++ b/docs/concepts/cluster-administration/manage-deployment.md @@ -435,4 +435,4 @@ That's it! The Deployment will declaratively update the deployed nginx applicati ## What's next? - [Learn about how to use `kubectl` for application introspection and debugging.](/docs/user-guide/introspection-and-debugging/) -- [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/) +- [Configuration Best Practices and Tips](/docs/concepts/configuration/overview/) diff --git a/docs/concepts/configuration/overview.md b/docs/concepts/configuration/overview.md new file mode 100644 index 0000000000..0ef55e7b5e --- /dev/null +++ b/docs/concepts/configuration/overview.md @@ -0,0 +1,119 @@ +--- +assignees: +- mikedanese +title: Configuration Best Practices +--- + +This document is meant to highlight and consolidate in one place configuration best practices that are introduced throughout the user-guide and getting-started documentation and examples. This is a living document so if you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR. + +## General Config Tips + +- When defining configurations, specify the latest stable API version (currently v1). + +- Configuration files should be stored in version control before being pushed to the cluster. This allows a configuration to be quickly rolled back if needed, and will aid with cluster re-creation and restoration if necessary. + +- Write your configuration files using YAML rather than JSON. They can be used interchangeably in almost all scenarios, but YAML tends to be more user-friendly for config. + +- Group related objects together in a single file where this makes sense. This format is often easier to manage than separate files. See the [guestbook-all-in-one.yaml](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/all-in-one/guestbook-all-in-one.yaml) file as an example of this syntax. +(Note also that many `kubectl` commands can be called on a directory, and so you can also call +`kubectl create` on a directory of config files— see below for more detail). + +- Don't specify default values unnecessarily, in order to simplify and minimize configs, and to + reduce error. For example, omit the selector and labels in a `ReplicationController` if you want + them to be the same as the labels in its `podTemplate`, since those fields are populated from the + `podTemplate` labels by default. See the [guestbook app's](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/) .yaml files for some [examples](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/frontend-deployment.yaml) of this. + +- Put an object description in an annotation to allow better introspection. + + +## "Naked" Pods vs Replication Controllers and Jobs + +- If there is a viable alternative to naked pods (i.e., pods not bound to a [replication controller + ](/docs/user-guide/replication-controller)), go with the alternative. Naked pods will not be rescheduled in the + event of node failure. + + Replication controllers are almost always preferable to creating pods, except for some explicit + [`restartPolicy: Never`](/docs/user-guide/pod-states/#restartpolicy) scenarios. A + [Job](/docs/user-guide/jobs/) object (currently in Beta), may also be appropriate. + + +## Services + +- It's typically best to create a [service](/docs/user-guide/services/) before corresponding [replication + controllers](/docs/user-guide/replication-controller/), so that the scheduler can spread the pods comprising the + service. You can also create a replication controller without specifying replicas (this will set + replicas=1), create a service, then scale up the replication controller. This can be useful in + ensuring that one replica works before creating lots of them. + +- Don't use `hostPort` (which specifies the port number to expose on the host) unless absolutely + necessary, e.g., for a node daemon. When you bind a Pod to a `hostPort`, there are a limited + number of places that pod can be scheduled, due to port conflicts— you can only schedule as many + such Pods as there are nodes in your Kubernetes cluster. + + If you only need access to the port for debugging purposes, you can use the [kubectl proxy and apiserver proxy](/docs/user-guide/connecting-to-applications-proxy/) or [kubectl port-forward](/docs/user-guide/connecting-to-applications-port-forward/). + You can use a [Service](/docs/user-guide/services/) object for external service access. + If you do need to expose a pod's port on the host machine, consider using a [NodePort](/docs/user-guide/services/#type-nodeport) service before resorting to `hostPort`. + +- Avoid using `hostNetwork`, for the same reasons as `hostPort`. + +- Use _headless services_ for easy service discovery when you don't need kube-proxy load balancing. + See [headless services](/docs/user-guide/services/#headless-services). + +## Using Labels + +- Define and use [labels](/docs/user-guide/labels/) that identify __semantic attributes__ of your application or + deployment. For example, instead of attaching a label to a set of pods to explicitly represent + some service (e.g., `service: myservice`), or explicitly representing the replication + controller managing the pods (e.g., `controller: mycontroller`), attach labels that identify + semantic attributes, such as `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`. This + will let you select the object groups appropriate to the context— e.g., a service for all "tier: + frontend" pods, or all "test" phase components of app "myapp". See the + [guestbook](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/) app for an example of this approach. + + A service can be made to span multiple deployments, such as is done across [rolling updates](/docs/user-guide/kubectl/kubectl_rolling-update/), by simply omitting release-specific labels from its selector, rather than updating a service's selector to match the replication controller's selector fully. + +- To facilitate rolling updates, include version info in replication controller names, e.g. as a + suffix to the name. It is useful to set a 'version' label as well. The rolling update creates a + new controller as opposed to modifying the existing controller. So, there will be issues with + version-agnostic controller names. See the [documentation](/docs/user-guide/kubectl/kubectl_rolling-update/) on + the rolling-update command for more detail. + + Note that the [Deployment](/docs/user-guide/deployments/) object obviates the need to manage replication + controller 'version names'. A desired state of an object is described by a Deployment, and if + changes to that spec are _applied_, the deployment controller changes the actual state to the + desired state at a controlled rate. (Deployment objects are currently part of the [`extensions` + API Group](/docs/api/#api-groups).) + +- You can manipulate labels for debugging. Because Kubernetes replication controllers and services + match to pods using labels, this allows you to remove a pod from being considered by a + controller, or served traffic by a service, by removing the relevant selector labels. If you + remove the labels of an existing pod, its controller will create a new pod to take its place. + This is a useful way to debug a previously "live" pod in a quarantine environment. See the + [`kubectl label`](/docs/user-guide/kubectl/kubectl_label/) command. + +## Container Images + +- The [default container image pull policy](/docs/user-guide/images/) is `IfNotPresent`, which causes the + [Kubelet](/docs/admin/kubelet/) to not pull an image if it already exists. If you would like to + always force a pull, you must specify a pull image policy of `Always` in your .yaml file + (`imagePullPolicy: Always`) or specify a `:latest` tag on your image. + + That is, if you're specifying an image with other than the `:latest` tag, e.g. `myimage:v1`, and + there is an image update to that same tag, the Kubelet won't pull the updated image. You can + address this by ensuring that any updates to an image bump the image tag as well (e.g. + `myimage:v2`), and ensuring that your configs point to the correct version. + + **Note:** you should avoid using `:latest` tag when deploying containers in production, because this makes it hard + to track which version of the image is running and hard to roll back. + +## Using kubectl + +- Use `kubectl create -f ` where possible. This looks for config objects in all `.yaml`, `.yml`, and `.json` files in `` and passes them to `create`. + +- Use `kubectl delete` rather than `stop`. `Delete` has a superset of the functionality of `stop`, and `stop` is deprecated. + +- Use kubectl bulk operations (via files and/or labels) for get and delete. See [label selectors](/docs/user-guide/labels/#label-selectors) and [using labels effectively](/docs/concepts/cluster-administration/manage-deployment/#using-labels-effectively). + +- Use `kubectl run` and `expose` to quickly create and expose single container Deployments. See the [quick start guide](/docs/user-guide/quick-start/) for an example. + + diff --git a/docs/tasks/run-application/rolling-update-replication-controller.md b/docs/tasks/run-application/rolling-update-replication-controller.md index ae64d478a6..97b6f0bbd0 100644 --- a/docs/tasks/run-application/rolling-update-replication-controller.md +++ b/docs/tasks/run-application/rolling-update-replication-controller.md @@ -77,7 +77,7 @@ current value. For this reason, we recommend the use of versioned tags as opposed to values such as `:latest`. Doing a rolling update from `image:latest` to a new `image:latest` will fail, even if the image at that tag has changed. Moreover, the use of `:latest` is not recommended, see -[Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information. +[Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images) for more information. ### Examples diff --git a/docs/user-guide/config-best-practices.md b/docs/user-guide/config-best-practices.md index a7081a3519..85f65a9c5a 100644 --- a/docs/user-guide/config-best-practices.md +++ b/docs/user-guide/config-best-practices.md @@ -1,119 +1,7 @@ --- -assignees: -- mikedanese title: Best Practices for Configuration --- -This document is meant to highlight and consolidate in one place configuration best practices that are introduced throughout the user-guide and getting-started documentation and examples. This is a living document so if you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR. - -## General Config Tips - -- When defining configurations, specify the latest stable API version (currently v1). - -- Configuration files should be stored in version control before being pushed to the cluster. This allows a configuration to be quickly rolled back if needed, and will aid with cluster re-creation and restoration if necessary. - -- Write your configuration files using YAML rather than JSON. They can be used interchangeably in almost all scenarios, but YAML tends to be more user-friendly for config. - -- Group related objects together in a single file where this makes sense. This format is often easier to manage than separate files. See the [guestbook-all-in-one.yaml](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/all-in-one/guestbook-all-in-one.yaml) file as an example of this syntax. -(Note also that many `kubectl` commands can be called on a directory, and so you can also call -`kubectl create` on a directory of config files— see below for more detail). - -- Don't specify default values unnecessarily, in order to simplify and minimize configs, and to - reduce error. For example, omit the selector and labels in a `ReplicationController` if you want - them to be the same as the labels in its `podTemplate`, since those fields are populated from the - `podTemplate` labels by default. See the [guestbook app's](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/) .yaml files for some [examples](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/frontend-deployment.yaml) of this. - -- Put an object description in an annotation to allow better introspection. - - -## "Naked" Pods vs Replication Controllers and Jobs - -- If there is a viable alternative to naked pods (i.e., pods not bound to a [replication controller - ](/docs/user-guide/replication-controller)), go with the alternative. Naked pods will not be rescheduled in the - event of node failure. - - Replication controllers are almost always preferable to creating pods, except for some explicit - [`restartPolicy: Never`](/docs/user-guide/pod-states/#restartpolicy) scenarios. A - [Job](/docs/user-guide/jobs/) object (currently in Beta), may also be appropriate. - - -## Services - -- It's typically best to create a [service](/docs/user-guide/services/) before corresponding [replication - controllers](/docs/user-guide/replication-controller/), so that the scheduler can spread the pods comprising the - service. You can also create a replication controller without specifying replicas (this will set - replicas=1), create a service, then scale up the replication controller. This can be useful in - ensuring that one replica works before creating lots of them. - -- Don't use `hostPort` (which specifies the port number to expose on the host) unless absolutely - necessary, e.g., for a node daemon. When you bind a Pod to a `hostPort`, there are a limited - number of places that pod can be scheduled, due to port conflicts— you can only schedule as many - such Pods as there are nodes in your Kubernetes cluster. - - If you only need access to the port for debugging purposes, you can use the [kubectl proxy and apiserver proxy](/docs/user-guide/connecting-to-applications-proxy/) or [kubectl port-forward](/docs/user-guide/connecting-to-applications-port-forward/). - You can use a [Service](/docs/user-guide/services/) object for external service access. - If you do need to expose a pod's port on the host machine, consider using a [NodePort](/docs/user-guide/services/#type-nodeport) service before resorting to `hostPort`. - -- Avoid using `hostNetwork`, for the same reasons as `hostPort`. - -- Use _headless services_ for easy service discovery when you don't need kube-proxy load balancing. - See [headless services](/docs/user-guide/services/#headless-services). - -## Using Labels - -- Define and use [labels](/docs/user-guide/labels/) that identify __semantic attributes__ of your application or - deployment. For example, instead of attaching a label to a set of pods to explicitly represent - some service (e.g., `service: myservice`), or explicitly representing the replication - controller managing the pods (e.g., `controller: mycontroller`), attach labels that identify - semantic attributes, such as `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`. This - will let you select the object groups appropriate to the context— e.g., a service for all "tier: - frontend" pods, or all "test" phase components of app "myapp". See the - [guestbook](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/) app for an example of this approach. - - A service can be made to span multiple deployments, such as is done across [rolling updates](/docs/user-guide/kubectl/kubectl_rolling-update/), by simply omitting release-specific labels from its selector, rather than updating a service's selector to match the replication controller's selector fully. - -- To facilitate rolling updates, include version info in replication controller names, e.g. as a - suffix to the name. It is useful to set a 'version' label as well. The rolling update creates a - new controller as opposed to modifying the existing controller. So, there will be issues with - version-agnostic controller names. See the [documentation](/docs/user-guide/kubectl/kubectl_rolling-update/) on - the rolling-update command for more detail. - - Note that the [Deployment](/docs/user-guide/deployments/) object obviates the need to manage replication - controller 'version names'. A desired state of an object is described by a Deployment, and if - changes to that spec are _applied_, the deployment controller changes the actual state to the - desired state at a controlled rate. (Deployment objects are currently part of the [`extensions` - API Group](/docs/api/#api-groups).) - -- You can manipulate labels for debugging. Because Kubernetes replication controllers and services - match to pods using labels, this allows you to remove a pod from being considered by a - controller, or served traffic by a service, by removing the relevant selector labels. If you - remove the labels of an existing pod, its controller will create a new pod to take its place. - This is a useful way to debug a previously "live" pod in a quarantine environment. See the - [`kubectl label`](/docs/user-guide/kubectl/kubectl_label/) command. - -## Container Images - -- The [default container image pull policy](/docs/user-guide/images/) is `IfNotPresent`, which causes the - [Kubelet](/docs/admin/kubelet/) to not pull an image if it already exists. If you would like to - always force a pull, you must specify a pull image policy of `Always` in your .yaml file - (`imagePullPolicy: Always`) or specify a `:latest` tag on your image. - - That is, if you're specifying an image with other than the `:latest` tag, e.g. `myimage:v1`, and - there is an image update to that same tag, the Kubelet won't pull the updated image. You can - address this by ensuring that any updates to an image bump the image tag as well (e.g. - `myimage:v2`), and ensuring that your configs point to the correct version. - - **Note:** you should avoid using `:latest` tag when deploying containers in production, because this makes it hard - to track which version of the image is running and hard to roll back. - -## Using kubectl - -- Use `kubectl create -f ` where possible. This looks for config objects in all `.yaml`, `.yml`, and `.json` files in `` and passes them to `create`. - -- Use `kubectl delete` rather than `stop`. `Delete` has a superset of the functionality of `stop`, and `stop` is deprecated. - -- Use kubectl bulk operations (via files and/or labels) for get and delete. See [label selectors](/docs/user-guide/labels/#label-selectors) and [using labels effectively](/docs/concepts/cluster-administration/manage-deployment/#using-labels-effectively). - -- Use `kubectl run` and `expose` to quickly create and expose single container Deployments. See the [quick start guide](/docs/user-guide/quick-start/) for an example. - +{% include user-guide-content-moved.md %} +[Configuration Overview](/docs/concepts/configuration/overview/) diff --git a/docs/user-guide/images.md b/docs/user-guide/images.md index 5b523e5615..0d12db8904 100644 --- a/docs/user-guide/images.md +++ b/docs/user-guide/images.md @@ -25,7 +25,7 @@ your image. If you did not specify tag of your image, it will be assumed as `:latest`, with pull image policy of `Always` correspondingly. -Note that you should avoid using `:latest` tag, see [Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information. +Note that you should avoid using `:latest` tag, see [Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images) for more information. ## Using a Private Registry diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index c62ff4574c..94eaacf3d7 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -85,7 +85,7 @@ Pods and containers * [Downward API: accessing system configuration from a pod](/docs/user-guide/downward-api/) * [Images and registries](/docs/user-guide/images/) * [Migrating from docker-cli to kubectl](/docs/user-guide/docker-cli-to-kubectl/) - * [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/) + * [Configuration Best Practices and Tips](/docs/concepts/configuration/overview/) * [Assign pods to selected nodes](/docs/user-guide/node-selection/) * [Perform a rolling update on a running group of pods](/docs/tasks/run-application/rolling-update-replication-controller/) diff --git a/docs/user-guide/kubectl-conventions.md b/docs/user-guide/kubectl-conventions.md index 6f9be8d5bf..d3b3673dbf 100644 --- a/docs/user-guide/kubectl-conventions.md +++ b/docs/user-guide/kubectl-conventions.md @@ -23,7 +23,7 @@ If you need stable output in a script, you should: In order for `kubectl run` to satisfy infrastructure as code: -* Always tag your image with a version-specific tag and don't move that tag to a new version. For example, use `:v1234`, `v1.2.3`, `r03062016-1-4`, rather than `:latest` (see [Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information.) +* Always tag your image with a version-specific tag and don't move that tag to a new version. For example, use `:v1234`, `v1.2.3`, `r03062016-1-4`, rather than `:latest` (see [Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images) for more information.) * If the image is lightly parameterized, capture the parameters in a checked-in script, or at least use `--record`, to annotate the created objects with the command line. * If the image is heavily parameterized, definitely check in the script. * If features are needed that are not expressible via `kubectl run` flags, switch to configuration files checked into source control. diff --git a/docs/user-guide/kubectl-overview.md b/docs/user-guide/kubectl-overview.md index cc88f1b64d..8b16e6731e 100644 --- a/docs/user-guide/kubectl-overview.md +++ b/docs/user-guide/kubectl-overview.md @@ -33,7 +33,7 @@ where `command`, `TYPE`, `NAME`, and `flags` are: * To specify multiple resource types individually: `TYPE1/name1 TYPE1/name2 TYPE2/name3 TYPE<#>/name<#>`
    Example: `$ kubectl get pod/example-pod1 replicationcontroller/example-rc1` * To specify resources with one or more files: `-f file1 -f file2 -f file<#>` - [Use YAML rather than JSON](/docs/user-guide/config-best-practices/#general-config-tips) since YAML tends to be more user-friendly, especially for configuration files.
    + [Use YAML rather than JSON](/docs/concepts/configuration/overview/#general-config-tips) since YAML tends to be more user-friendly, especially for configuration files.
    Example: `$ kubectl get pod -f ./pod.yaml` * `flags`: Specifies optional flags. For example, you can use the `-s` or `--server` flags to specify the address and port of the Kubernetes API server.
    **Important**: Flags that you specify from the command line override default values and any corresponding environment variables. From f7648e7c274a72be254dd12f6f3033ba5bb9116a Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 15 Mar 2017 18:20:29 -0700 Subject: [PATCH 407/407] Move Guide topic: Jobs (#2853) * Move Guide topic: Jobs * add job.yaml file * change title --- _data/concepts.yml | 3 + docs/concepts/jobs/job.yaml | 15 + .../run-to-completion-finite-workloads.md | 385 ++++++++++++++++++ docs/user-guide/jobs.md | 382 +---------------- 4 files changed, 405 insertions(+), 380 deletions(-) create mode 100644 docs/concepts/jobs/job.yaml create mode 100644 docs/concepts/jobs/run-to-completion-finite-workloads.md diff --git a/_data/concepts.yml b/_data/concepts.yml index f0059c0db7..31bfd9396b 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -31,6 +31,9 @@ toc: - title: Pods section: - docs/concepts/workloads/pods/pod-lifecycle.md + - title: Jobs + section: + - docs/concepts/jobs/run-to-completion-finite-workloads.md - title: Cluster Administration section: diff --git a/docs/concepts/jobs/job.yaml b/docs/concepts/jobs/job.yaml new file mode 100644 index 0000000000..ece4512a8a --- /dev/null +++ b/docs/concepts/jobs/job.yaml @@ -0,0 +1,15 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: pi +spec: + template: + metadata: + name: pi + spec: + containers: + - name: pi + image: perl + command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] + restartPolicy: Never + diff --git a/docs/concepts/jobs/run-to-completion-finite-workloads.md b/docs/concepts/jobs/run-to-completion-finite-workloads.md new file mode 100644 index 0000000000..fe17bb32e6 --- /dev/null +++ b/docs/concepts/jobs/run-to-completion-finite-workloads.md @@ -0,0 +1,385 @@ +--- +assignees: +- erictune +- soltysh +title: Run to Completion Finite Workloads +--- + +* TOC +{:toc} + +## What is a Job? + +A _job_ creates one or more pods and ensures that a specified number of them successfully terminate. +As pods successfully complete, the _job_ tracks the successful completions. When a specified number +of successful completions is reached, the job itself is complete. Deleting a Job will cleanup the +pods it created. + +A simple case is to create one Job object in order to reliably run one Pod to completion. +The Job object will start a new Pod if the first pod fails or is deleted (for example +due to a node hardware failure or a node reboot). + +A Job can also be used to run multiple pods in parallel. + +### extensions/v1beta1.Job is deprecated + +Starting from version 1.5 `extensions/v1beta1.Job` is being deprecated, with a plan to be removed in +version 1.6 of Kubernetes (see this [issue](https://github.com/kubernetes/kubernetes/issues/32763)). +Please use `batch/v1.Job` instead. + +## Running an example Job + +Here is an example Job config. It computes π to 2000 places and prints it out. +It takes around 10s to complete. + +{% include code.html language="yaml" file="job.yaml" ghlink="/docs/user-guide/job.yaml" %} + +Run the example job by downloading the example file and then running this command: + +```shell +$ kubectl create -f ./job.yaml +job "pi" created +``` + +Check on the status of the job using this command: + +```shell +$ kubectl describe jobs/pi +Name: pi +Namespace: default +Image(s): perl +Selector: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495 +Parallelism: 1 +Completions: 1 +Start Time: Tue, 07 Jun 2016 10:56:16 +0200 +Labels: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495,job-name=pi +Pods Statuses: 0 Running / 1 Succeeded / 0 Failed +No volumes. +Events: + FirstSeen LastSeen Count From SubobjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: pi-dtn4q +``` + +To view completed pods of a job, use `kubectl get pods --show-all`. The `--show-all` will show completed pods too. + +To list all the pods that belong to a job in a machine readable form, you can use a command like this: + +```shell +$ pods=$(kubectl get pods --show-all --selector=job-name=pi --output=jsonpath={.items..metadata.name}) +echo $pods +pi-aiw0a +``` + +Here, the selector is the same as the selector for the job. The `--output=jsonpath` option specifies an expression +that just gets the name from each pod in the returned list. + +View the standard output of one of the pods: + +```shell +$ kubectl logs $pods +3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901 +``` + +## Writing a Job Spec + +As with all other Kubernetes config, a Job needs `apiVersion`, `kind`, and `metadata` fields. For +general information about working with config files, see [here](/docs/user-guide/simple-yaml), +[here](/docs/user-guide/configuring-containers), and [here](/docs/user-guide/working-with-resources). + +A Job also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). + +### Pod Template + +The `.spec.template` is the only required field of the `.spec`. + +The `.spec.template` is a [pod template](/docs/user-guide/replication-controller/#pod-template). It has exactly +the same schema as a [pod](/docs/user-guide/pods), except it is nested and does not have an `apiVersion` or +`kind`. + +In addition to required fields for a Pod, a pod template in a job must specify appropriate +labels (see [pod selector](#pod-selector)) and an appropriate restart policy. + +Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` is allowed. + +### Pod Selector + +The `.spec.selector` field is optional. In almost all cases you should not specify it. +See section [specifying your own pod selector](#specifying-your-own-pod-selector). + + +### Parallel Jobs + +There are three main types of jobs: + +1. Non-parallel Jobs + - normally only one pod is started, unless the pod fails. + - job is complete as soon as Pod terminates successfully. +1. Parallel Jobs with a *fixed completion count*: + - specify a non-zero positive value for `.spec.completions` + - the job is complete when there is one successful pod for each value in the range 1 to `.spec.completions`. + - **not implemented yet:** each pod passed a different index in the range 1 to `.spec.completions`. +1. Parallel Jobs with a *work queue*: + - do not specify `.spec.completions`, default to `.spec.Parallelism` + - the pods must coordinate with themselves or an external service to determine what each should work on + - each pod is independently capable of determining whether or not all its peers are done, thus the entire Job is done. + - when _any_ pod terminates with success, no new pods are created. + - once at least one pod has terminated with success and all pods are terminated, then the job is completed with success. + - once any pod has exited with success, no other pod should still be doing any work or writing any output. They should all be + in the process of exiting. + +For a Non-parallel job, you can leave both `.spec.completions` and `.spec.parallelism` unset. When both are +unset, both are defaulted to 1. + +For a Fixed Completion Count job, you should set `.spec.completions` to the number of completions needed. +You can set `.spec.parallelism`, or leave it unset and it will default to 1. + +For a Work Queue Job, you must leave `.spec.completions` unset, and set `.spec.parallelism` to +a non-negative integer. + +For more information about how to make use of the different types of job, see the [job patterns](#job-patterns) section. + + +#### Controlling Parallelism + +The requested parallelism (`.spec.parallelism`) can be set to any non-negative value. +If it is unspecified, it defaults to 1. +If it is specified as 0, then the Job is effectively paused until it is increased. + +A job can be scaled up using the `kubectl scale` command. For example, the following +command sets `.spec.parallelism` of a job called `myjob` to 10: + +```shell +$ kubectl scale --replicas=$N jobs/myjob +job "myjob" scaled +``` + +You can also use the `scale` subresource of the Job resource. + +Actual parallelism (number of pods running at any instant) may be more or less than requested +parallelism, for a variety or reasons: + +- For Fixed Completion Count jobs, the actual number of pods running in parallel will not exceed the number of + remaining completions. Higher values of `.spec.parallelism` are effectively ignored. +- For work queue jobs, no new pods are started after any pod has succeeded -- remaining pods are allowed to complete, however. +- If the controller has not had time to react. +- If the controller failed to create pods for any reason (lack of ResourceQuota, lack of permission, etc.), + then there may be fewer pods than requested. +- The controller may throttle new pod creation due to excessive previous pod failures in the same Job. +- When a pod is gracefully shutdown, it takes time to stop. + +## Handling Pod and Container Failures + +A Container in a Pod may fail for a number of reasons, such as because the process in it exited with +a non-zero exit code, or the Container was killed for exceeding a memory limit, etc. If this +happens, and the `.spec.template.spec.restartPolicy = "OnFailure"`, then the Pod stays +on the node, but the Container is re-run. Therefore, your program needs to handle the case when it is +restarted locally, or else specify `.spec.template.spec.restartPolicy = "Never"`. +See [pods-states](/docs/user-guide/pod-states) for more information on `restartPolicy`. + +An entire Pod can also fail, for a number of reasons, such as when the pod is kicked off the node +(node is upgraded, rebooted, deleted, etc.), or if a container of the Pod fails and the +`.spec.template.spec.restartPolicy = "Never"`. When a Pod fails, then the Job controller +starts a new Pod. Therefore, your program needs to handle the case when it is restarted in a new +pod. In particular, it needs to handle temporary files, locks, incomplete output and the like +caused by previous runs. + +Note that even if you specify `.spec.parallelism = 1` and `.spec.completions = 1` and +`.spec.template.spec.restartPolicy = "Never"`, the same program may +sometimes be started twice. + +If you do specify `.spec.parallelism` and `.spec.completions` both greater than 1, then there may be +multiple pods running at once. Therefore, your pods must also be tolerant of concurrency. + +## Job Termination and Cleanup + +When a Job completes, no more Pods are created, but the Pods are not deleted either. Since they are terminated, +they don't show up with `kubectl get pods`, but they will show up with `kubectl get pods -a`. Keeping them around +allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output. +The job object also remains after it is completed so that you can view its status. It is up to the user to delete +old jobs after noting their status. Delete the job with `kubectl` (e.g. `kubectl delete jobs/pi` or `kubectl delete -f ./job.yaml`). When you delete the job using `kubectl`, all the pods it created are deleted too. + +If a Job's pods are failing repeatedly, the Job will keep creating new pods forever, by default. +Retrying forever can be a useful pattern. If an external dependency of the Job's +pods is missing (for example an input file on a networked storage volume is not present), then the +Job will keep trying Pods, and when you later resolve the external dependency (for example, creating +the missing file) the Job will then complete without any further action. + +However, if you prefer not to retry forever, you can set a deadline on the job. Do this by setting the +`spec.activeDeadlineSeconds` field of the job to a number of seconds. The job will have status with +`reason: DeadlineExceeded`. No more pods will be created, and existing pods will be deleted. + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: pi-with-timeout +spec: + activeDeadlineSeconds: 100 + template: + metadata: + name: pi + spec: + containers: + - name: pi + image: perl + command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] + restartPolicy: Never +``` + +Note that both the Job Spec and the Pod Template Spec within the Job have a field with the same name. +Set the one on the Job. + +## Job Patterns + +The Job object can be used to support reliable parallel execution of Pods. The Job object is not +designed to support closely-communicating parallel processes, as commonly found in scientific +computing. It does support parallel processing of a set of independent but related *work items*. +These might be emails to be sent, frames to be rendered, files to be transcoded, ranges of keys in a +NoSQL database to scan, and so on. + +In a complex system, there may be multiple different sets of work items. Here we are just +considering one set of work items that the user wants to manage together — a *batch job*. + +There are several different patterns for parallel computation, each with strengths and weaknesses. +The tradeoffs are: + +- One Job object for each work item, vs. a single Job object for all work items. The latter is + better for large numbers of work items. The former creates some overhead for the user and for the + system to manage large numbers of Job objects. Also, with the latter, the resource usage of the job + (number of concurrently running pods) can be easily adjusted using the `kubectl scale` command. +- Number of pods created equals number of work items, vs. each pod can process multiple work items. + The former typically requires less modification to existing code and containers. The latter + is better for large numbers of work items, for similar reasons to the previous bullet. +- Several approaches use a work queue. This requires running a queue service, + and modifications to the existing program or container to make it use the work queue. + Other approaches are easier to adapt to an existing containerised application. + + +The tradeoffs are summarized here, with columns 2 to 4 corresponding to the above tradeoffs. +The pattern names are also links to examples and more detailed description. + +| Pattern | Single Job object | Fewer pods than work items? | Use app unmodified? | Works in Kube 1.1? | +| -------------------------------------------------------------------- |:-----------------:|:---------------------------:|:-------------------:|:-------------------:| +| [Job Template Expansion](/docs/user-guide/jobs/expansions) | | | ✓ | ✓ | +| [Queue with Pod Per Work Item](/docs/user-guide/jobs/work-queue-1/) | ✓ | | sometimes | ✓ | +| [Queue with Variable Pod Count](/docs/user-guide/jobs/work-queue-2/) | ✓ | ✓ | | ✓ | +| Single Job with Static Work Assignment | ✓ | | ✓ | | + +When you specify completions with `.spec.completions`, each Pod created by the Job controller +has an identical [`spec`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). This means that +all pods will have the same command line and the same +image, the same volumes, and (almost) the same environment variables. These patterns +are different ways to arrange for pods to work on different things. + +This table shows the required settings for `.spec.parallelism` and `.spec.completions` for each of the patterns. +Here, `W` is the number of work items. + +| Pattern | `.spec.completions` | `.spec.parallelism` | +| -------------------------------------------------------------------- |:-------------------:|:--------------------:| +| [Job Template Expansion](/docs/user-guide/jobs/expansions/) | 1 | should be 1 | +| [Queue with Pod Per Work Item](/docs/user-guide/jobs/work-queue-1/) | W | any | +| [Queue with Variable Pod Count](/docs/user-guide/jobs/work-queue-2/) | 1 | any | +| Single Job with Static Work Assignment | W | any | + + +## Advanced Usage + +### Specifying your own pod selector + +Normally, when you create a job object, you do not specify `spec.selector`. +The system defaulting logic adds this field when the job is created. +It picks a selector value that will not overlap with any other jobs. + +However, in some cases, you might need to override this automatically set selector. +To do this, you can specify the `spec.selector` of the job. + +Be very careful when doing this. If you specify a label selector which is not +unique to the pods of that job, and which matches unrelated pods, then pods of the unrelated +job may be deleted, or this job may count other pods as completing it, or one or both +of the jobs may refuse to create pods or run to completion. If a non-unique selector is +chosen, then other controllers (e.g. ReplicationController) and their pods may behave +in unpredicatable ways too. Kubernetes will not stop you from making a mistake when +specifying `spec.selector`. + +Here is an example of a case when you might want to use this feature. + +Say job `old` is already running. You want existing pods +to keep running, but you want the rest of the pods it creates +to use a different pod template and for the job to have a new name. +You cannot update the job because these fields are not updatable. +Therefore, you delete job `old` but leave its pods +running, using `kubectl delete jobs/old-one --cascade=false`. +Before deleting it, you make a note of what selector it uses: + +``` +kind: Job +metadata: + name: old + ... +spec: + selector: + matchLabels: + job-uid: a8f3d00d-c6d2-11e5-9f87-42010af00002 + ... +``` + +Then you create a new job with name `new` and you explicitly specify the same selector. +Since the existing pods have label `job-uid=a8f3d00d-c6d2-11e5-9f87-42010af00002`, +they are controlled by job `new` as well. + +You need to specify `manualSelector: true` in the new job since you are not using +the selector that the system normally generates for you automatically. + +``` +kind: Job +metadata: + name: new + ... +spec: + manualSelector: true + selector: + matchLabels: + job-uid: a8f3d00d-c6d2-11e5-9f87-42010af00002 + ... +``` + +The new Job itself will have a different uid from `a8f3d00d-c6d2-11e5-9f87-42010af00002`. Setting +`manualSelector: true` tells the system to that you know what you are doing and to allow this +mismatch. + +## Alternatives + +### Bare Pods + +When the node that a pod is running on reboots or fails, the pod is terminated +and will not be restarted. However, a Job will create new pods to replace terminated ones. +For this reason, we recommend that you use a job rather than a bare pod, even if your application +requires only a single pod. + +### Replication Controller + +Jobs are complementary to [Replication Controllers](/docs/user-guide/replication-controller). +A Replication Controller manages pods which are not expected to terminate (e.g. web servers), and a Job +manages pods that are expected to terminate (e.g. batch jobs). + +As discussed in [life of a pod](/docs/user-guide/pod-states), `Job` is *only* appropriate for pods with +`RestartPolicy` equal to `OnFailure` or `Never`. (Note: If `RestartPolicy` is not set, the default +value is `Always`.) + +### Single Job starts Controller Pod + +Another pattern is for a single Job to create a pod which then creates other pods, acting as a sort +of custom controller for those pods. This allows the most flexibility, but may be somewhat +complicated to get started with and offers less integration with Kubernetes. + +One example of this pattern would be a Job which starts a Pod which runs a script that in turn +starts a Spark master controller (see [spark example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/spark/README.md)), runs a spark +driver, and then cleans up. + +An advantage of this approach is that the overall process gets the completion guarantee of a Job +object, but complete control over what pods are created and how work is assigned to them. + +## Cron Jobs + +Support for creating Jobs at specified times/dates (i.e. cron) is available in Kubernetes [1.4](https://github.com/kubernetes/kubernetes/pull/11980). More information is available in the [cron job documents](http://kubernetes.io/docs/user-guide/cron-jobs/) diff --git a/docs/user-guide/jobs.md b/docs/user-guide/jobs.md index 438b63dff7..502de5a513 100644 --- a/docs/user-guide/jobs.md +++ b/docs/user-guide/jobs.md @@ -1,385 +1,7 @@ --- -assignees: -- erictune -- soltysh title: Jobs --- -* TOC -{:toc} +{% include user-guide-content-moved.md %} -## What is a Job? - -A _job_ creates one or more pods and ensures that a specified number of them successfully terminate. -As pods successfully complete, the _job_ tracks the successful completions. When a specified number -of successful completions is reached, the job itself is complete. Deleting a Job will cleanup the -pods it created. - -A simple case is to create one Job object in order to reliably run one Pod to completion. -The Job object will start a new Pod if the first pod fails or is deleted (for example -due to a node hardware failure or a node reboot). - -A Job can also be used to run multiple pods in parallel. - -### extensions/v1beta1.Job is deprecated - -Starting from version 1.5 `extensions/v1beta1.Job` is being deprecated, with a plan to be removed in -version 1.6 of Kubernetes (see this [issue](https://github.com/kubernetes/kubernetes/issues/32763)). -Please use `batch/v1.Job` instead. - -## Running an example Job - -Here is an example Job config. It computes π to 2000 places and prints it out. -It takes around 10s to complete. - -{% include code.html language="yaml" file="job.yaml" ghlink="/docs/user-guide/job.yaml" %} - -Run the example job by downloading the example file and then running this command: - -```shell -$ kubectl create -f ./job.yaml -job "pi" created -``` - -Check on the status of the job using this command: - -```shell -$ kubectl describe jobs/pi -Name: pi -Namespace: default -Image(s): perl -Selector: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495 -Parallelism: 1 -Completions: 1 -Start Time: Tue, 07 Jun 2016 10:56:16 +0200 -Labels: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495,job-name=pi -Pods Statuses: 0 Running / 1 Succeeded / 0 Failed -No volumes. -Events: - FirstSeen LastSeen Count From SubobjectPath Type Reason Message - --------- -------- ----- ---- ------------- -------- ------ ------- - 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: pi-dtn4q -``` - -To view completed pods of a job, use `kubectl get pods --show-all`. The `--show-all` will show completed pods too. - -To list all the pods that belong to a job in a machine readable form, you can use a command like this: - -```shell -$ pods=$(kubectl get pods --show-all --selector=job-name=pi --output=jsonpath={.items..metadata.name}) -echo $pods -pi-aiw0a -``` - -Here, the selector is the same as the selector for the job. The `--output=jsonpath` option specifies an expression -that just gets the name from each pod in the returned list. - -View the standard output of one of the pods: - -```shell -$ kubectl logs $pods -3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901 -``` - -## Writing a Job Spec - -As with all other Kubernetes config, a Job needs `apiVersion`, `kind`, and `metadata` fields. For -general information about working with config files, see [here](/docs/user-guide/simple-yaml), -[here](/docs/user-guide/configuring-containers), and [here](/docs/user-guide/working-with-resources). - -A Job also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). - -### Pod Template - -The `.spec.template` is the only required field of the `.spec`. - -The `.spec.template` is a [pod template](/docs/user-guide/replication-controller/#pod-template). It has exactly -the same schema as a [pod](/docs/user-guide/pods), except it is nested and does not have an `apiVersion` or -`kind`. - -In addition to required fields for a Pod, a pod template in a job must specify appropriate -labels (see [pod selector](#pod-selector)) and an appropriate restart policy. - -Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` is allowed. - -### Pod Selector - -The `.spec.selector` field is optional. In almost all cases you should not specify it. -See section [specifying your own pod selector](#specifying-your-own-pod-selector). - - -### Parallel Jobs - -There are three main types of jobs: - -1. Non-parallel Jobs - - normally only one pod is started, unless the pod fails. - - job is complete as soon as Pod terminates successfully. -1. Parallel Jobs with a *fixed completion count*: - - specify a non-zero positive value for `.spec.completions` - - the job is complete when there is one successful pod for each value in the range 1 to `.spec.completions`. - - **not implemented yet:** each pod passed a different index in the range 1 to `.spec.completions`. -1. Parallel Jobs with a *work queue*: - - do not specify `.spec.completions`, default to `.spec.Parallelism` - - the pods must coordinate with themselves or an external service to determine what each should work on - - each pod is independently capable of determining whether or not all its peers are done, thus the entire Job is done. - - when _any_ pod terminates with success, no new pods are created. - - once at least one pod has terminated with success and all pods are terminated, then the job is completed with success. - - once any pod has exited with success, no other pod should still be doing any work or writing any output. They should all be - in the process of exiting. - -For a Non-parallel job, you can leave both `.spec.completions` and `.spec.parallelism` unset. When both are -unset, both are defaulted to 1. - -For a Fixed Completion Count job, you should set `.spec.completions` to the number of completions needed. -You can set `.spec.parallelism`, or leave it unset and it will default to 1. - -For a Work Queue Job, you must leave `.spec.completions` unset, and set `.spec.parallelism` to -a non-negative integer. - -For more information about how to make use of the different types of job, see the [job patterns](#job-patterns) section. - - -#### Controlling Parallelism - -The requested parallelism (`.spec.parallelism`) can be set to any non-negative value. -If it is unspecified, it defaults to 1. -If it is specified as 0, then the Job is effectively paused until it is increased. - -A job can be scaled up using the `kubectl scale` command. For example, the following -command sets `.spec.parallelism` of a job called `myjob` to 10: - -```shell -$ kubectl scale --replicas=$N jobs/myjob -job "myjob" scaled -``` - -You can also use the `scale` subresource of the Job resource. - -Actual parallelism (number of pods running at any instant) may be more or less than requested -parallelism, for a variety or reasons: - -- For Fixed Completion Count jobs, the actual number of pods running in parallel will not exceed the number of - remaining completions. Higher values of `.spec.parallelism` are effectively ignored. -- For work queue jobs, no new pods are started after any pod has succeeded -- remaining pods are allowed to complete, however. -- If the controller has not had time to react. -- If the controller failed to create pods for any reason (lack of ResourceQuota, lack of permission, etc.), - then there may be fewer pods than requested. -- The controller may throttle new pod creation due to excessive previous pod failures in the same Job. -- When a pod is gracefully shutdown, it takes time to stop. - -## Handling Pod and Container Failures - -A Container in a Pod may fail for a number of reasons, such as because the process in it exited with -a non-zero exit code, or the Container was killed for exceeding a memory limit, etc. If this -happens, and the `.spec.template.spec.restartPolicy = "OnFailure"`, then the Pod stays -on the node, but the Container is re-run. Therefore, your program needs to handle the case when it is -restarted locally, or else specify `.spec.template.spec.restartPolicy = "Never"`. -See [pods-states](/docs/user-guide/pod-states) for more information on `restartPolicy`. - -An entire Pod can also fail, for a number of reasons, such as when the pod is kicked off the node -(node is upgraded, rebooted, deleted, etc.), or if a container of the Pod fails and the -`.spec.template.spec.restartPolicy = "Never"`. When a Pod fails, then the Job controller -starts a new Pod. Therefore, your program needs to handle the case when it is restarted in a new -pod. In particular, it needs to handle temporary files, locks, incomplete output and the like -caused by previous runs. - -Note that even if you specify `.spec.parallelism = 1` and `.spec.completions = 1` and -`.spec.template.spec.restartPolicy = "Never"`, the same program may -sometimes be started twice. - -If you do specify `.spec.parallelism` and `.spec.completions` both greater than 1, then there may be -multiple pods running at once. Therefore, your pods must also be tolerant of concurrency. - -## Job Termination and Cleanup - -When a Job completes, no more Pods are created, but the Pods are not deleted either. Since they are terminated, -they don't show up with `kubectl get pods`, but they will show up with `kubectl get pods -a`. Keeping them around -allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output. -The job object also remains after it is completed so that you can view its status. It is up to the user to delete -old jobs after noting their status. Delete the job with `kubectl` (e.g. `kubectl delete jobs/pi` or `kubectl delete -f ./job.yaml`). When you delete the job using `kubectl`, all the pods it created are deleted too. - -If a Job's pods are failing repeatedly, the Job will keep creating new pods forever, by default. -Retrying forever can be a useful pattern. If an external dependency of the Job's -pods is missing (for example an input file on a networked storage volume is not present), then the -Job will keep trying Pods, and when you later resolve the external dependency (for example, creating -the missing file) the Job will then complete without any further action. - -However, if you prefer not to retry forever, you can set a deadline on the job. Do this by setting the -`spec.activeDeadlineSeconds` field of the job to a number of seconds. The job will have status with -`reason: DeadlineExceeded`. No more pods will be created, and existing pods will be deleted. - -```yaml -apiVersion: batch/v1 -kind: Job -metadata: - name: pi-with-timeout -spec: - activeDeadlineSeconds: 100 - template: - metadata: - name: pi - spec: - containers: - - name: pi - image: perl - command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] - restartPolicy: Never -``` - -Note that both the Job Spec and the Pod Template Spec within the Job have a field with the same name. -Set the one on the Job. - -## Job Patterns - -The Job object can be used to support reliable parallel execution of Pods. The Job object is not -designed to support closely-communicating parallel processes, as commonly found in scientific -computing. It does support parallel processing of a set of independent but related *work items*. -These might be emails to be sent, frames to be rendered, files to be transcoded, ranges of keys in a -NoSQL database to scan, and so on. - -In a complex system, there may be multiple different sets of work items. Here we are just -considering one set of work items that the user wants to manage together — a *batch job*. - -There are several different patterns for parallel computation, each with strengths and weaknesses. -The tradeoffs are: - -- One Job object for each work item, vs. a single Job object for all work items. The latter is - better for large numbers of work items. The former creates some overhead for the user and for the - system to manage large numbers of Job objects. Also, with the latter, the resource usage of the job - (number of concurrently running pods) can be easily adjusted using the `kubectl scale` command. -- Number of pods created equals number of work items, vs. each pod can process multiple work items. - The former typically requires less modification to existing code and containers. The latter - is better for large numbers of work items, for similar reasons to the previous bullet. -- Several approaches use a work queue. This requires running a queue service, - and modifications to the existing program or container to make it use the work queue. - Other approaches are easier to adapt to an existing containerised application. - - -The tradeoffs are summarized here, with columns 2 to 4 corresponding to the above tradeoffs. -The pattern names are also links to examples and more detailed description. - -| Pattern | Single Job object | Fewer pods than work items? | Use app unmodified? | Works in Kube 1.1? | -| -------------------------------------------------------------------- |:-----------------:|:---------------------------:|:-------------------:|:-------------------:| -| [Job Template Expansion](/docs/user-guide/jobs/expansions) | | | ✓ | ✓ | -| [Queue with Pod Per Work Item](/docs/user-guide/jobs/work-queue-1/) | ✓ | | sometimes | ✓ | -| [Queue with Variable Pod Count](/docs/user-guide/jobs/work-queue-2/) | ✓ | ✓ | | ✓ | -| Single Job with Static Work Assignment | ✓ | | ✓ | | - -When you specify completions with `.spec.completions`, each Pod created by the Job controller -has an identical [`spec`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). This means that -all pods will have the same command line and the same -image, the same volumes, and (almost) the same environment variables. These patterns -are different ways to arrange for pods to work on different things. - -This table shows the required settings for `.spec.parallelism` and `.spec.completions` for each of the patterns. -Here, `W` is the number of work items. - -| Pattern | `.spec.completions` | `.spec.parallelism` | -| -------------------------------------------------------------------- |:-------------------:|:--------------------:| -| [Job Template Expansion](/docs/user-guide/jobs/expansions/) | 1 | should be 1 | -| [Queue with Pod Per Work Item](/docs/user-guide/jobs/work-queue-1/) | W | any | -| [Queue with Variable Pod Count](/docs/user-guide/jobs/work-queue-2/) | 1 | any | -| Single Job with Static Work Assignment | W | any | - - -## Advanced Usage - -### Specifying your own pod selector - -Normally, when you create a job object, you do not specify `spec.selector`. -The system defaulting logic adds this field when the job is created. -It picks a selector value that will not overlap with any other jobs. - -However, in some cases, you might need to override this automatically set selector. -To do this, you can specify the `spec.selector` of the job. - -Be very careful when doing this. If you specify a label selector which is not -unique to the pods of that job, and which matches unrelated pods, then pods of the unrelated -job may be deleted, or this job may count other pods as completing it, or one or both -of the jobs may refuse to create pods or run to completion. If a non-unique selector is -chosen, then other controllers (e.g. ReplicationController) and their pods may behave -in unpredicatable ways too. Kubernetes will not stop you from making a mistake when -specifying `spec.selector`. - -Here is an example of a case when you might want to use this feature. - -Say job `old` is already running. You want existing pods -to keep running, but you want the rest of the pods it creates -to use a different pod template and for the job to have a new name. -You cannot update the job because these fields are not updatable. -Therefore, you delete job `old` but leave its pods -running, using `kubectl delete jobs/old-one --cascade=false`. -Before deleting it, you make a note of what selector it uses: - -``` -kind: Job -metadata: - name: old - ... -spec: - selector: - matchLabels: - job-uid: a8f3d00d-c6d2-11e5-9f87-42010af00002 - ... -``` - -Then you create a new job with name `new` and you explicitly specify the same selector. -Since the existing pods have label `job-uid=a8f3d00d-c6d2-11e5-9f87-42010af00002`, -they are controlled by job `new` as well. - -You need to specify `manualSelector: true` in the new job since you are not using -the selector that the system normally generates for you automatically. - -``` -kind: Job -metadata: - name: new - ... -spec: - manualSelector: true - selector: - matchLabels: - job-uid: a8f3d00d-c6d2-11e5-9f87-42010af00002 - ... -``` - -The new Job itself will have a different uid from `a8f3d00d-c6d2-11e5-9f87-42010af00002`. Setting -`manualSelector: true` tells the system to that you know what you are doing and to allow this -mismatch. - -## Alternatives - -### Bare Pods - -When the node that a pod is running on reboots or fails, the pod is terminated -and will not be restarted. However, a Job will create new pods to replace terminated ones. -For this reason, we recommend that you use a job rather than a bare pod, even if your application -requires only a single pod. - -### Replication Controller - -Jobs are complementary to [Replication Controllers](/docs/user-guide/replication-controller). -A Replication Controller manages pods which are not expected to terminate (e.g. web servers), and a Job -manages pods that are expected to terminate (e.g. batch jobs). - -As discussed in [life of a pod](/docs/user-guide/pod-states), `Job` is *only* appropriate for pods with -`RestartPolicy` equal to `OnFailure` or `Never`. (Note: If `RestartPolicy` is not set, the default -value is `Always`.) - -### Single Job starts Controller Pod - -Another pattern is for a single Job to create a pod which then creates other pods, acting as a sort -of custom controller for those pods. This allows the most flexibility, but may be somewhat -complicated to get started with and offers less integration with Kubernetes. - -One example of this pattern would be a Job which starts a Pod which runs a script that in turn -starts a Spark master controller (see [spark example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/spark/README.md)), runs a spark -driver, and then cleans up. - -An advantage of this approach is that the overall process gets the completion guarantee of a Job -object, but complete control over what pods are created and how work is assigned to them. - -## Cron Jobs - -Support for creating Jobs at specified times/dates (i.e. cron) is available in Kubernetes [1.4](https://github.com/kubernetes/kubernetes/pull/11980). More information is available in the [cron job documents](http://kubernetes.io/docs/user-guide/cron-jobs/) +[Run to Completion Finite Workloads](/docs/concepts/jobs/run-to-completion-finite-workloads/)