From 04a783d3730888e544fd762bcfaa1a2c502ee720 Mon Sep 17 00:00:00 2001 From: Jeff Sloyer Date: Thu, 22 Dec 2016 10:35:51 -0500 Subject: [PATCH 01/68] added subject docs on allowing all users --- docs/admin/authorization.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/admin/authorization.md b/docs/admin/authorization.md index 523dd256d9..0387a101ed 100644 --- a/docs/admin/authorization.md +++ b/docs/admin/authorization.md @@ -441,6 +441,28 @@ subjects: name: system:serviceaccounts ``` +For all authenticated users: +```yaml +subjects: +- kind: Group + name: system:authenticated +``` + +For all unauthenticated users: +```yaml +subjects: +- kind: Group + name: system:unauthenticated +``` + +For all users: +```yaml +subjects: +- kind: Group + name: system:authenticated +- kind: Group + name: system:unauthenticated + ## Webhook Mode When specified, mode `Webhook` causes Kubernetes to query an outside REST From f3aef1dbf2d532a8075707afe111638598aff468 Mon Sep 17 00:00:00 2001 From: Eric Hole Date: Fri, 30 Dec 2016 13:22:58 -0700 Subject: [PATCH 02/68] Added kubeadm to list of tools under kubectl. Updated _data/tools.yml to reflect kubeadm. --- _data/tools.yml | 2 ++ docs/tools/index.md | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/_data/tools.yml b/_data/tools.yml index 31c355e9b0..92764f795a 100644 --- a/_data/tools.yml +++ b/_data/tools.yml @@ -7,6 +7,8 @@ toc: section: - title: Kubectl path: /docs/user-guide/kubectl/ + - title: Kubeadm + path: /docs/getting-started-guides/kubeadm - title: Kubefed path: /docs/admin/federation/kubefed/ - title: Kubernetes Dashboard diff --git a/docs/tools/index.md b/docs/tools/index.md index 351b5968c1..8ab734c289 100644 --- a/docs/tools/index.md +++ b/docs/tools/index.md @@ -14,6 +14,10 @@ Kubernetes contains the following built-in tools: [`kubectl`](/docs/user-guide/kubectl/) is the command line tool for Kubernetes. It controls the Kubernetes cluster manager. +##### Kubeadm + +[`kubeadm`](/docs/getting-started-guides/kubeadm/) is the command line tool for easily provisioning a secure Kubernetes cluster on top of physical or cloud servers or virtual machines (currently in alpha). + ##### Kubefed [`kubefed`](/docs/admin/federation/kubefed/) is the command line tool From 67ce3155b3c643fa180eecd4f19b0cfd74a69971 Mon Sep 17 00:00:00 2001 From: Jess Frazelle Date: Thu, 15 Dec 2016 12:05:37 -0800 Subject: [PATCH 03/68] move community.html to community/index.html to be more consistent with the rest of the repo Signed-off-by: Jess Frazelle --- community.html => community/index.html | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename community.html => community/index.html (100%) diff --git a/community.html b/community/index.html similarity index 100% rename from community.html rename to community/index.html From 65a8beae339a333a96efe0606be54c813dafb73b Mon Sep 17 00:00:00 2001 From: Jess Frazelle Date: Thu, 15 Dec 2016 12:13:46 -0800 Subject: [PATCH 04/68] move case-study-styles.html to an actual css file and make it a class Signed-off-by: Jess Frazelle --- _includes/case-study-styles.html | 313 ------------------------------- _includes/head-header.html | 1 + case-studies/index.html | 2 +- case-studies/pearson.html | 2 +- case-studies/wikimedia.html | 2 +- css/gridpage.css | 310 ++++++++++++++++++++++++++++++ partners/index.html | 2 +- 7 files changed, 315 insertions(+), 317 deletions(-) delete mode 100644 _includes/case-study-styles.html create mode 100644 css/gridpage.css diff --git a/_includes/case-study-styles.html b/_includes/case-study-styles.html deleted file mode 100644 index 48058bd27b..0000000000 --- a/_includes/case-study-styles.html +++ /dev/null @@ -1,313 +0,0 @@ - - diff --git a/_includes/head-header.html b/_includes/head-header.html index fa859604d2..43e56595b4 100644 --- a/_includes/head-header.html +++ b/_includes/head-header.html @@ -7,6 +7,7 @@ + {% if page.class == "gridPage" %}{% endif %} diff --git a/case-studies/index.html b/case-studies/index.html index f593d73fb9..35082bbf2c 100644 --- a/case-studies/index.html +++ b/case-studies/index.html @@ -1,5 +1,6 @@ --- title: Case Studies +class: gridPage --- @@ -99,7 +100,6 @@ title: Case Studies {% include footer.html %} -{% include case-study-styles.html %} diff --git a/case-studies/pearson.html b/case-studies/pearson.html index 50f16ce7ae..9d0943392b 100644 --- a/case-studies/pearson.html +++ b/case-studies/pearson.html @@ -1,5 +1,6 @@ --- title: Pearson Case Study +class: gridPage --- @@ -82,7 +83,6 @@ title: Pearson Case Study {% include footer.html %} -{% include case-study-styles.html %} diff --git a/case-studies/wikimedia.html b/case-studies/wikimedia.html index 2d3b686128..ff019a3fd7 100644 --- a/case-studies/wikimedia.html +++ b/case-studies/wikimedia.html @@ -1,5 +1,6 @@ --- title: Wikimedia Case Study +class: gridPage --- @@ -97,7 +98,6 @@ title: Wikimedia Case Study {% include footer.html %} -{% include case-study-styles.html %} diff --git a/css/gridpage.css b/css/gridpage.css new file mode 100644 index 0000000000..96a39af9c6 --- /dev/null +++ b/css/gridpage.css @@ -0,0 +1,310 @@ +#caseStudyTitle { + margin-top: 1em !important; +} + +.gridPage p { + color: rgb(26,26,26) !important; + margin-left: 0 !important; + padding-left: 0 !important; + font-weight: 300 !important; +} + +.gridPage #mainContent { + padding: 0; +} + +.gridPage #mainContent .content { + padding-top: 0; +} + +.gridPage main { + max-width: 1100px !important; +} + +.gridPage .content { + position: relative; + margin: 0 auto 50px; + max-width: 90%; +} + +.gridPage .content p { + line-height: 24px !important; +} + +.gridPage .content h3 { + padding: 0 !important; +} + +.gridPage #hero h5 { + padding-left: 20px; + margin: 0; +} + +.case-studies { + position: relative; + display: flex; + justify-content: space-between; + flex-wrap: wrap; + margin-top: 50px; +} + +.case-study { + position: relative; + width: 50%; + padding: 0 40px 0 242px; + margin-bottom: 60px; + min-height: 152px; +} + +.case-study:nth-child(3), .case-study:nth-child(4) { + margin-bottom: 0; +} + +.case-study img { + position: absolute; + top: 0; + left: 0; +} + +.gridPage #mainContent .content .case-study p { + font-family: "Roboto", sans-serif; + font-size: 16px; + padding: 0; +} + +p.attrib { + font-style: italic; +} + +.gridPage #video { + background: #f9f9f9; + height: auto; + /*height: 340px;*/ +} + +.gridPage #video main { + position: relative; + max-width: 900px !important; + height: 100%; + display: flex; + justify-content: center; + align-items: center; + padding: 50px 20px; +} + +.gridPage #video main > div { + width: 50%; +} + +.gridPage #video main #zulilyLogo { + width: 100px; +} + +.gridPage #video main img { + max-width: 100%; +} + +.gridPage #video h3 { + font-size: 32px; + font-weight: 300; + line-height: 38px; + max-width: 80%; + margin: 0 0 1em 0; +} + +.gridPage #video p { + margin: 0; +} + +.gridPage #video p.attrib { + margin-bottom: 20px; +} + +.gridPage #video button > h6 { + font-size: 18px; + font-weight: 500; + margin: 1em 0; + color: #326de6; +} + +.gridPage #users { + padding: 50px; +} + +.gridPage #users main { + max-width: 1150px !important; +} + +.gridPage #users main h3 { + padding-left: 20px; + margin-bottom: 20px; +} + +.gridPage #usersGrid { + position: relative; + display: flex; + flex-wrap: wrap; + justify-content: center; +} + +.gridPage #usersGrid a { + display: inline-block; + margin: 5px; +} + +.gridPage #usersGrid a img { + box-shadow: 1px 1px 2px transparent; + transition: box-shadow 0.25s; +} + +.gridPage #usersGrid a img:hover { + box-shadow: 1px 1px 2px #cccccc; +} + +.gridPage #usersGrid a:last-child img, +.gridPage #usersGrid a:last-child img:hover { + box-shadow: 1px 1px 2px transparent; +} + +.tell-your-story { + border: 1px solid #dddddd; + border-radius: 6px; + box-shadow: 1px 2px 2px #dddddd; +} + +.gridPage .feature { + position: relative; + padding: 20px 0 20px 242px; +} + +.gridPage .feature img { + position: absolute; + top: 20px; + left: 0; +} + +section.bullets { + background-color: #eeeeee; + margin-bottom: 50px; +} + +section.bullets main { + position: relative; + max-width: 1100px; + padding: 50px 0; +} + +section.bullets .content { + position: relative; + display: flex; + flex-wrap: wrap; + margin-bottom: 0 !important; +} + +.bullet { + position: relative; + width: 50%; + padding: 15px 30px; +} + +.bullet h4 { + margin-bottom: 0.5em; +} + +.bullet li { + margin-left: 1.25em; + list-style: disc; + font-weight: 300; + color: rgb(26,26,26); + line-height: 1.5em; + margin-bottom: 0.5em; +} + +.details h4, p { + margin-bottom: 0.5em; +} + +.gridPage .feature p.quote { + font-size: 20px; + line-height: 28px !important; +} + +@media screen and (max-width: 1024px){ + .case-study { + padding: 0 10%; + margin-bottom: 50px; + } + + .case-study img { + position: relative; + } + + .case-study p.quote { + margin-top: 20px !important; + } + + .case-study p.attrib { + font-style: italic; + } +} + +@media screen and (max-width: 900px){ + .gridPage #video main { + flex-direction: column; + align-items: center; + } + + .gridPage #video main > div { + width: 400px; + } + + .gridPage #video main > div + div { + margin-top: 30px; + } + + .gridPage #video h3 { + max-width: 100%; + } +} + +@media screen and (max-width: 640px){ + .case-study { + width: 100%; + } + + .case-study:nth-child(3) { + margin-bottom: 60px; + } + + .case-study img { + left: 50%; + transform: translateX(-50%); + } + + .gridPage .feature { + margin-top: 50px; + padding: 180px 0 0; + } + + .gridPage .feature img { + top: 0; + left: 50%; + transform: translateX(-50%); + } +} + +@media screen and (max-width: 480px){ + .gridPage #hero { + padding-right: 20px; + padding-left: 20px; + } + + .gridPage #video main > div { + width: 80%; + min-width: 280px; + } + + .bullet { + width: 100%; + } +} diff --git a/partners/index.html b/partners/index.html index 5a8da670a9..a65be512ff 100644 --- a/partners/index.html +++ b/partners/index.html @@ -1,5 +1,6 @@ --- title: Partners +class: gridPage --- @@ -23,7 +24,6 @@ title: Partners {% include footer.html %} -{% include case-study-styles.html %} + + + + diff --git a/_layouts/docwithnav.html b/_layouts/docwithnav.html index 65418b3ece..454ccd69ce 100755 --- a/_layouts/docwithnav.html +++ b/_layouts/docwithnav.html @@ -75,57 +75,7 @@ {% include_cached footer.html %} +{% include footer-scripts.html %} - - - - - From 1c0a25bd12984084d22bf44f761bb279297dc3c5 Mon Sep 17 00:00:00 2001 From: Jess Frazelle Date: Thu, 15 Dec 2016 12:29:15 -0800 Subject: [PATCH 08/68] make it clearer where the {{content}} starts Signed-off-by: Jess Frazelle --- _layouts/docwithnav.html | 1 + 1 file changed, 1 insertion(+) diff --git a/_layouts/docwithnav.html b/_layouts/docwithnav.html index 454ccd69ce..073e1c9344 100755 --- a/_layouts/docwithnav.html +++ b/_layouts/docwithnav.html @@ -47,6 +47,7 @@ +

Edit This Page

From 354394bf5262e09a3c5a7507404ad0eb8134e408 Mon Sep 17 00:00:00 2001 From: Jess Frazelle Date: Thu, 15 Dec 2016 12:35:33 -0800 Subject: [PATCH 09/68] add basic layout Signed-off-by: Jess Frazelle --- _layouts/basic.html | 15 +++++++++++++++ case-studies/index.html | 14 ++------------ case-studies/pearson.html | 15 ++------------- case-studies/wikimedia.html | 15 ++------------- community/index.html | 17 +++-------------- partners/index.html | 15 ++------------- 6 files changed, 26 insertions(+), 65 deletions(-) create mode 100644 _layouts/basic.html diff --git a/_layouts/basic.html b/_layouts/basic.html new file mode 100644 index 0000000000..d2b684df52 --- /dev/null +++ b/_layouts/basic.html @@ -0,0 +1,15 @@ + + + + {% include head.html %} + + + {% include header.html %} + + {{ content }} + + {% include footer.html %} + + {% include footer-scripts.html %} + + diff --git a/case-studies/index.html b/case-studies/index.html index 9a09623206..e4e15111c5 100644 --- a/case-studies/index.html +++ b/case-studies/index.html @@ -1,16 +1,10 @@ --- title: Case Studies +layout: basic class: gridPage +cid: caseStudies --- - - -{% include head.html %} - - - -{% include header.html %} -

Kubernetes User Case Studies

A collection of users running Kubernetes in production.
@@ -103,7 +97,3 @@ class: gridPage
-{% include footer.html %} - - - diff --git a/case-studies/pearson.html b/case-studies/pearson.html index 34afa89522..45428b0bf3 100644 --- a/case-studies/pearson.html +++ b/case-studies/pearson.html @@ -1,16 +1,10 @@ --- title: Pearson Case Study +layout: basic class: gridPage +cid: caseStudies --- - - -{% include head.html %} - - - -{% include header.html %} -

Pearson Case Study

@@ -85,8 +79,3 @@ class: gridPage - -{% include footer.html %} - - - diff --git a/case-studies/wikimedia.html b/case-studies/wikimedia.html index 50a3910b75..7431c13650 100644 --- a/case-studies/wikimedia.html +++ b/case-studies/wikimedia.html @@ -1,16 +1,10 @@ --- title: Wikimedia Case Study +layout: basic class: gridPage +cid: caseStudies --- - - -{% include head.html %} - - - -{% include header.html %} -

Wikimedia Case Study

@@ -100,8 +94,3 @@ class: gridPage - -{% include footer.html %} - - - diff --git a/community/index.html b/community/index.html index b142585874..4686ebb442 100644 --- a/community/index.html +++ b/community/index.html @@ -1,17 +1,11 @@ --- title: Community +layout: basic +cid: community --- - - -{% include head.html %} - - - -{% include header.html %} -
-

Community

+

Community

@@ -65,8 +59,3 @@ title: Community
- -{% include footer.html %} - - - diff --git a/partners/index.html b/partners/index.html index 65127b8b3b..13ee6e2396 100644 --- a/partners/index.html +++ b/partners/index.html @@ -1,16 +1,10 @@ --- title: Partners +layout: basic class: gridPage +cid: partners --- - - -{% include head.html %} - - - -{% include header.html %} -

Kubernetes Partners

Growing the Kubernetes ecosystem.
@@ -27,8 +21,6 @@ class: gridPage
-{% include footer.html %} - @@ -36,6 +28,3 @@ class: gridPage - - - From 1aaa7920fee7ad33b68fb8618f4d322b768b9925 Mon Sep 17 00:00:00 2001 From: Jess Frazelle Date: Thu, 15 Dec 2016 12:37:53 -0800 Subject: [PATCH 10/68] add index to the basic layout as well Signed-off-by: Jess Frazelle --- index.html | 31 ++----------------------------- 1 file changed, 2 insertions(+), 29 deletions(-) diff --git a/index.html b/index.html index 902f16d644..5e860832d7 100644 --- a/index.html +++ b/index.html @@ -1,15 +1,8 @@ --- +layout: basic +cid: home --- - - - -{% include head.html %} - - - -{% include header.html %} -
@@ -186,23 +179,3 @@ - - - - - - From 956dcf8cd6a2eee39c6637d6607a0c10ece7d971 Mon Sep 17 00:00:00 2001 From: Brandon Cole Date: Wed, 7 Dec 2016 20:53:56 -0500 Subject: [PATCH 11/68] Updating downward-api/index.md for spec.nodeName There are a couple awesome enhancements that would benefit from being documented here: 1. `spec.nodeName` for obtaining the host name 2. `spec.serviceAccountName` for obtaining the name of the service account the pod is running under Right now these enhancements are documented only in a couple of issues and pull requests: 1. https://github.com/kubernetes/kubernetes/pull/27880 2. https://github.com/kubernetes/kubernetes/issues/24657 3. https://github.com/kubernetes/kubernetes/issues/21317 I have verified the functionality in the latest codebase: 1. https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/conversion.go#L200 Also updating downward-api/dapi-pod.yaml to add a concrete example. --- docs/user-guide/downward-api/dapi-pod.yaml | 8 ++++++++ docs/user-guide/downward-api/index.md | 6 +++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/downward-api/dapi-pod.yaml b/docs/user-guide/downward-api/dapi-pod.yaml index 7d688aa0e9..5de0260bfc 100644 --- a/docs/user-guide/downward-api/dapi-pod.yaml +++ b/docs/user-guide/downward-api/dapi-pod.yaml @@ -8,6 +8,10 @@ spec: image: gcr.io/google_containers/busybox command: [ "/bin/sh", "-c", "env" ] env: + - name: MY_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName - name: MY_POD_NAME valueFrom: fieldRef: @@ -20,4 +24,8 @@ spec: valueFrom: fieldRef: fieldPath: status.podIP + - name: MY_POD_SERVICE_ACCOUNT + valueFrom: + fieldRef: + fieldPath: spec.serviceAccountName restartPolicy: Never diff --git a/docs/user-guide/downward-api/index.md b/docs/user-guide/downward-api/index.md index 00cef8f43e..33b2b06c49 100644 --- a/docs/user-guide/downward-api/index.md +++ b/docs/user-guide/downward-api/index.md @@ -22,9 +22,11 @@ the Pod's name, for example, and inject it into this well-known variable. The following information is available to a `Pod` through the downward API: -* The pod's name +* The node's name +*   The pod's name * The pod's namespace * The pod's IP +* The pod's service account name * A container's cpu limit * A container's cpu request * A container's memory limit @@ -101,10 +103,12 @@ In future, it will be possible to specify an output format option. Downward API volumes can expose: +* The node's name * The pod's name * The pod's namespace * The pod's labels * The pod's annotations +* The pod's service account name * A container's cpu limit * A container's cpu request * A container's memory limit From ecefdc9df22f20e83ab4caf5151bd1965a3c59c4 Mon Sep 17 00:00:00 2001 From: Jared Date: Wed, 18 Jan 2017 14:25:58 -0800 Subject: [PATCH 12/68] Add "Needs UX Review" tag I'd like to tag existing issues that are specifically UX related for triaging, since we have a graphic designer who can fix them. --- docs/contribute/review-issues.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/contribute/review-issues.md b/docs/contribute/review-issues.md index 8b16d450c6..38d4424c7b 100644 --- a/docs/contribute/review-issues.md +++ b/docs/contribute/review-issues.md @@ -4,7 +4,7 @@ title: Reviewing Documentation Issues {% capture overview %} -This page explains how you should review and prioritize documentation issues made for the [kubernetes/kubernetes.github.io](https://github.com/kubernetes/kubernetes.github.io){: target="_blank"} repository. The purpose is to provide a way to organize issues and make it easier to contribute to Kubernetes documentation. The following should be used as the standard way of prioritizing, labeling, and interacting with issues. +This page explains how documentation issues are reviewed and prioritized for the [kubernetes/kubernetes.github.io](https://github.com/kubernetes/kubernetes.github.io){: target="_blank"} repository. The purpose is to provide a way to organize issues and make it easier to contribute to Kubernetes documentation. The following should be used as the standard way of prioritizing, labeling, and interacting with issues. {% endcapture %} {% capture body %} @@ -26,6 +26,9 @@ Issues should be sorted into different buckets of work using the following label * Issues that are suggestions for better processes or site improvements that require community agreement to be implemented * Topics can be brought to SIG meetings as agenda items +#### Needs UX Review +* Issues that are suggestions for improving the user interface of the site or fixing a broken UX. + ### Prioritizing Issues The following labels and definitions should be used to prioritize issues. If you change the priority of an issues, please comment on the issue with your reasoning for the change. From a2deedd3f5ccf05f3cac627b24cd93a6b889fd33 Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Tue, 24 Jan 2017 11:20:59 +0100 Subject: [PATCH 13/68] Logging sidecar refactoring --- docs/user-guide/counter-pod.yaml | 12 -- docs/user-guide/logging/counter-pod.yaml | 12 -- .../logging/examples/counter-pod.yaml | 10 ++ .../examples/fluentd-sidecar-config.yaml | 25 ++++ .../two-files-counter-pod-agent-sidecar.yaml | 39 ++++++ ...o-files-counter-pod-streaming-sidecar.yaml | 38 +++++ .../examples/two-files-counter-pod.yaml | 26 ++++ docs/user-guide/logging/overview.md | 132 ++++++++++++++++-- docs/user-guide/logging/stackdriver.md | 12 +- ...car.png => logging-with-sidecar-agent.png} | Bin .../logging-with-streaming-sidecar.png | Bin 0 -> 37555 bytes 11 files changed, 261 insertions(+), 45 deletions(-) delete mode 100644 docs/user-guide/counter-pod.yaml delete mode 100644 docs/user-guide/logging/counter-pod.yaml create mode 100644 docs/user-guide/logging/examples/counter-pod.yaml create mode 100644 docs/user-guide/logging/examples/fluentd-sidecar-config.yaml create mode 100644 docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml create mode 100644 docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml create mode 100644 docs/user-guide/logging/examples/two-files-counter-pod.yaml rename images/docs/user-guide/logging/{logging-with-sidecar.png => logging-with-sidecar-agent.png} (100%) create mode 100644 images/docs/user-guide/logging/logging-with-streaming-sidecar.png diff --git a/docs/user-guide/counter-pod.yaml b/docs/user-guide/counter-pod.yaml deleted file mode 100644 index c624b4d339..0000000000 --- a/docs/user-guide/counter-pod.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: v1 -kind: Pod -metadata: - name: counter -spec: - containers: - - name: count - image: ubuntu:14.04 - args: [bash, -c, - 'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done'] - - diff --git a/docs/user-guide/logging/counter-pod.yaml b/docs/user-guide/logging/counter-pod.yaml deleted file mode 100644 index c624b4d339..0000000000 --- a/docs/user-guide/logging/counter-pod.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: v1 -kind: Pod -metadata: - name: counter -spec: - containers: - - name: count - image: ubuntu:14.04 - args: [bash, -c, - 'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done'] - - diff --git a/docs/user-guide/logging/examples/counter-pod.yaml b/docs/user-guide/logging/examples/counter-pod.yaml new file mode 100644 index 0000000000..f997886386 --- /dev/null +++ b/docs/user-guide/logging/examples/counter-pod.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Pod +metadata: + name: counter +spec: + containers: + - name: count + image: busybox + args: [/bin/sh, -c, + 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'] diff --git a/docs/user-guide/logging/examples/fluentd-sidecar-config.yaml b/docs/user-guide/logging/examples/fluentd-sidecar-config.yaml new file mode 100644 index 0000000000..6f7793c406 --- /dev/null +++ b/docs/user-guide/logging/examples/fluentd-sidecar-config.yaml @@ -0,0 +1,25 @@ +apiVersion: v1 +data: + fluentd.conf: | + + type tail + format none + path /var/log/1.log + pos_file /var/log/1.log.pos + tag count.format1 + + + + type tail + format none + path /var/log/2.log + pos_file /var/log/2.log.pos + tag count.format2 + + + + type google_cloud + +kind: ConfigMap +metadata: + name: fluentd-config \ No newline at end of file diff --git a/docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml b/docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml new file mode 100644 index 0000000000..9737f13493 --- /dev/null +++ b/docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml @@ -0,0 +1,39 @@ +apiVersion: v1 +kind: Pod +metadata: + name: counter +spec: + containers: + - name: count + image: busybox + args: + - /bin/sh + - -c + - > + i=0; + while true; + do + echo "$i: $(date)" >> /var/log/1.log; + echo "$(date) INFO $i" >> /var/log/2.log; + i=$((i+1)); + sleep 1; + done + volumeMounts: + - name: varlog + mountPath: /var/log + - name: count-agent + image: gcr.io/google_containers/fluentd-gcp:1.30 + env: + - name: FLUENTD_ARGS + value: -c /etc/fluentd-config/fluentd.conf + volumeMounts: + - name: varlog + mountPath: /var/log + - name: config-volume + mountPath: /etc/fluentd-config + volumes: + - name: varlog + emptyDir: {} + - name: config-volume + configMap: + name: fluentd-config diff --git a/docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml b/docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml new file mode 100644 index 0000000000..87bd198cfd --- /dev/null +++ b/docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Pod +metadata: + name: counter +spec: + containers: + - name: count + image: busybox + args: + - /bin/sh + - -c + - > + i=0; + while true; + do + echo "$i: $(date)" >> /var/log/1.log; + echo "$(date) INFO $i" >> /var/log/2.log; + i=$((i+1)); + sleep 1; + done + volumeMounts: + - name: varlog + mountPath: /var/log + - name: count-log-1 + image: busybox + args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log'] + volumeMounts: + - name: varlog + mountPath: /var/log + - name: count-log-2 + image: busybox + args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log'] + volumeMounts: + - name: varlog + mountPath: /var/log + volumes: + - name: varlog + emptyDir: {} diff --git a/docs/user-guide/logging/examples/two-files-counter-pod.yaml b/docs/user-guide/logging/examples/two-files-counter-pod.yaml new file mode 100644 index 0000000000..6ebeb717a1 --- /dev/null +++ b/docs/user-guide/logging/examples/two-files-counter-pod.yaml @@ -0,0 +1,26 @@ +apiVersion: v1 +kind: Pod +metadata: + name: counter +spec: + containers: + - name: count + image: busybox + args: + - /bin/sh + - -c + - > + i=0; + while true; + do + echo "$i: $(date)" >> /var/log/1.log; + echo "$(date) INFO $i" >> /var/log/2.log; + i=$((i+1)); + sleep 1; + done + volumeMounts: + - name: varlog + mountPath: /var/log + volumes: + - name: varlog + emptyDir: {} diff --git a/docs/user-guide/logging/overview.md b/docs/user-guide/logging/overview.md index 61c068a4d9..a9f00471cc 100644 --- a/docs/user-guide/logging/overview.md +++ b/docs/user-guide/logging/overview.md @@ -19,14 +19,17 @@ The guidance for cluster-level logging assumes that a logging backend is present ## Basic logging in Kubernetes -In this section, you can see an example of basic logging in Kubernetes that outputs data to the standard output stream. This demonstration uses a [pod specification](/docs/user-guide/logging/counter-pod.yaml) with a container that writes some text to standard output once per second. +In this section, you can see an example of basic logging in Kubernetes that +outputs data to the standard output stream. This demonstration uses +a [pod specification](/docs/user-guide/logging/examples/counter-pod.yaml) with +a container that writes some text to standard output once per second. -{% include code.html language="yaml" file="counter-pod.yaml" ghlink="/docs/user-guide/counter-pod.yaml" %} +{% include code.html language="yaml" file="examples/counter-pod.yaml" ghlink="/docs/user-guide/logging/examples/counter-pod.yaml" %} To run this pod, use the following command: ```shell -$ kubectl create -f http://k8s.io/docs/user-guide/counter-pod.yaml +$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml pod "counter" created ``` @@ -34,12 +37,9 @@ To fetch the logs, use the `kubectl logs` command, as follows ```shell $ kubectl logs counter -0: Tue Jun 2 21:37:31 UTC 2015 -1: Tue Jun 2 21:37:32 UTC 2015 -2: Tue Jun 2 21:37:33 UTC 2015 -3: Tue Jun 2 21:37:34 UTC 2015 -4: Tue Jun 2 21:37:35 UTC 2015 -5: Tue Jun 2 21:37:36 UTC 2015 +0: Mon Jan 1 00:00:00 UTC 2001 +1: Mon Jan 1 00:00:01 UTC 2001 +2: Mon Jan 1 00:00:02 UTC 2001 ... ``` @@ -105,17 +105,119 @@ Kubernetes doesn't specify a logging agent, but two optional logging agents are ### Using a sidecar container with the logging agent -![Using a sidecar container with the logging agent](/images/docs/user-guide/logging/logging-with-sidecar.png) +You can use a sidecar container in one of the following ways: -You can implement cluster-level logging by including a dedicated logging agent for each application on your cluster. You can include this logging agent as a _sidecar container_ in the pod spec for each application; the sidecar container should contain only the logging agent. +* The sidecar container streams application logs to its own `stdout`. +* The sidecar container runs a logging agent, which is configured to pick up logs from an application container. -The concrete implementation of the logging agent, the interface between agent and the application, and the interface between the logging agent and the logs backend are completely up to a you. For an example implementation, see the [fluentd sidecar container](https://github.com/kubernetes/contrib/tree/b70447aa59ea14468f4cd349760e45b6a0a9b15d/logging/fluentd-sidecar-gcp) for the Stackdriver logging backend. +#### Streaming sidecar container -**Note:** Using a sidecar container for logging may lead to significant resource consumption. +![Sidecar container with a streaming container](/images/docs/user-guide/logging/logging-with-streaming-sidecar.png) + +By having your sidecar containers stream to their own `stdout` and `stderr` +streams, you can take advantage of the kubelet and the logging agent that +already run on each node. The sidecar containers read logs from a file, a socket, +or the journald. Each individual sidecar container prints log to its own `stdout` +or `stderr` stream. + +This approach allows you to separate several log streams from different +parts of your application, some of which can lack support +for writing to `stdout` or `stderr`. The logic behind redirecting logs +is minimal, so it's hardly a significant overhead. Additionally, because +`stdout` and `stderr` are handled by the kubelet, you can use built-in tools +like `kubectl logs`. + +Consider the following example. A pod runs a single container, and the container +writes to two different log files, using two different formats. Here's a +configuration file for the Pod: + +{% include code.html language="yaml" file="examples/two-files-counter-pod.yaml" ghlink="/docs/user-guide/logging/examples/two-files-counter-pod.yaml" %} + +It would be a mess to have log entries of different formats in the same log +stream, even if you managed to redirect both components to the `stdout` stream of +the container. Instead, you could introduce two sidecar containers. Each sidecar +container could tail a particular log file from a shared volume and then redirect +the logs to its own `stdout` stream. + +Here's a configuration file for a pod that has two sidecar containers: + +{% include code.html language="yaml" file="examples/two-files-counter-pod-streaming-sidecar.yaml" ghlink="/docs/user-guide/logging/examples/two-files-counter-pod-streaming-sidecar.yaml" %} + +Now when you run this pod, you can access each log stream separately by +running the following commands: + +```shell +$ kubectl logs counter count-log-1 +0: Mon Jan 1 00:00:00 UTC 2001 +1: Mon Jan 1 00:00:01 UTC 2001 +2: Mon Jan 1 00:00:02 UTC 2001 +... +``` + +```shell +$ kubectl logs counter count-log-2 +Mon Jan 1 00:00:00 UTC 2001 INFO 0 +Mon Jan 1 00:00:01 UTC 2001 INFO 1 +Mon Jan 1 00:00:02 UTC 2001 INFO 2 +... +``` + +The node-level agent installed in your cluster picks up those log streams +automatically without any further configuration. If you like, you can configure +the agent to parse log lines depending on the source container. + +Note, that despite low CPU and memory usage (order of couple of millicores +for cpu and order of several megabytes for memory), writing logs to a file and +then streaming them to `stdout` can double disk usage. If you have +an application that writes to a single file, it's generally better to set +`/dev/stdout` as destination rather than implementing the streaming sidecar +container approach. + +Sidecar containers can also be used to rotate log files that cannot be +rotated by the application itself. [An example](https://github.com/samsung-cnct/logrotate) +of this approach is a small container running logrotate periodically. +However, it's recommended to use `stdout` and `stderr` directly and leave rotation +and retention policies to the kubelet. + +#### Sidecar container with a logging agent + +![Sidecar container with a logging agent](/images/docs/user-guide/logging/logging-with-sidecar-agent.png) + +If the node-level logging agent is not flexible enough for your situation, you +can create a sidecar container with a separate logging agent that you have +configured specifically to run with your application. + +**Note**: Using a logging agent in a sidecar container can lead +to significant resource consumption. Moreover, you won't be able to access +those logs using `kubectl logs` command, because they are not controlled +by the kubelet. + +As an example, you could use [Stackdriver](/docs/user-guide/logging/stackdriver/), +which uses fluentd as a logging agent. Here are two configuration files that +you can use to implement this approach. The first file contains +a [ConfigMap](/docs/user-guide/configmap/) to configure fluentd. + +{% include code.html language="yaml" file="examples/fluentd-sidecar-config.yaml" ghlink="/docs/user-guide/logging/examples/fluentd-sidecar-config.yaml" %} + +**Note**: The configuration of fluentd is beyond the scope of this article. For +information about configuring fluentd, see the +[official fluentd documentation](http://docs.fluentd.org/). + +The second file describes a pod that has a sidecar container running fluentd. +The pod mounts a volume where fluentd can pick up its configuration data. + +{% include code.html language="yaml" file="examples/two-files-counter-pod-agent-sidecar.yaml" ghlink="/docs/user-guide/logging/examples/two-files-counter-pod-agent-sidecar.yaml" %} + +After some time you can find log messages in the Stackdriver interface. + +Remember, that this is just an example and you can actually replace fluentd +with any logging agent, reading from any source inside an application +container. ### Exposing logs directly from the application ![Exposing logs directly from the application](/images/docs/user-guide/logging/logging-from-application.png) -You can implement cluster-level logging by exposing or pushing logs directly from every application itself; however, the implementation for such a logging mechanism is outside the scope of Kubernetes. - +You can implement cluster-level logging by exposing or pushing logs directly from +every application; however, the implementation for such a logging mechanism +is outside the scope of Kubernetes. diff --git a/docs/user-guide/logging/stackdriver.md b/docs/user-guide/logging/stackdriver.md index eb3380ca5c..b71947ee1f 100644 --- a/docs/user-guide/logging/stackdriver.md +++ b/docs/user-guide/logging/stackdriver.md @@ -27,16 +27,16 @@ fluentd-gcp-v1.30-f02l5 1/1 Running 0 5d ``` To understand how logging with Stackdriver works, consider the following -synthetic log generator pod specification [counter-pod.yaml](/docs/user-guide/logging/counter-pod.yaml): +synthetic log generator pod specification [counter-pod.yaml](/docs/user-guide/logging/examples/counter-pod.yaml): -{% include code.html language="yaml" file="counter-pod.yaml" ghlink="/docs/user-guide/counter-pod.yaml" %} +{% include code.html language="yaml" file="examples/counter-pod.yaml" ghlink="/docs/user-guide/logging/examples/counter-pod.yaml" %} This pod specification has one container that runs a bash script that writes out the value of a counter and the date once per second, and runs indefinitely. Let's create this pod in the default namespace. ```shell -$ kubectl create -f counter-pod.yaml +$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml pod "counter" created ``` @@ -68,14 +68,14 @@ by deleting the currently running counter container: ```shell $ kubectl delete pod counter -pods/counter +pod "counter" deleted ``` and then recreating it: ```shell -$ kubectl create -f counter-pod.yaml -pods/counter +$ kubectl create -f http://k8s.io/docs/user-guide/logging/examples/counter-pod.yaml +pod "counter" created ``` After some time, you can access logs from the counter pod again: diff --git a/images/docs/user-guide/logging/logging-with-sidecar.png b/images/docs/user-guide/logging/logging-with-sidecar-agent.png similarity index 100% rename from images/docs/user-guide/logging/logging-with-sidecar.png rename to images/docs/user-guide/logging/logging-with-sidecar-agent.png diff --git a/images/docs/user-guide/logging/logging-with-streaming-sidecar.png b/images/docs/user-guide/logging/logging-with-streaming-sidecar.png new file mode 100644 index 0000000000000000000000000000000000000000..75fbe10f73bbea621f9a51baf638412665e815d5 GIT binary patch literal 37555 zcmZ@=byStxw%-WS(t=2df^-SeB}jKkNQxlc2+}A>NJw`|DIgL`Nl8g5NF&k+NO!}V z-+kkb@&0*djC0QwVefCRwdS0^T7;>q%3Z^~iHky^t|`br)I_1s_TXP*Y)trytH$~j z_z#Awl!7)k{NscDG!*`f<0P-|3g179{EH^VoaPB%q;!+fbJKFPbn`HCu|RovcyL-f z*t(jVIazQzx;#tU5WR^)-9jlme4y=_zV_#ZVec^=*4AkCd+WNv3^$EDwh&T}w~9!#NI#6W#E_@~XPZ<2UZ@o}Ul!-%OC&&qxvf*=bHo zxSO$gc=|%XdTO_pCAHuI5yk_eAgef2Vbsf8*e`E2(kd*&S1{-?<)xV5&)7VYB*Mry zg`&mwrwq}<-@gh-g=I8w6%`d@ zCMG5b(gYk6H%H6p2I+}!mwfs})RV+jHd$ezBm?V9lv?nUmxlBKk-c1+$};SZrn-6! z5fM@L{^@3uD+A+W%jU~Qs)U)HfRZH9A~wypOX{C~nYU+HyIwnVj-NN1;E+B!WnlAeAfmT^w{_WuCXxXmU z6LJcrJ8bHPQ$Cwvk!X0F1+iNdu@9YvB1t`_=4Cs)Mg6wvXXElX8?91hot&H&FV0Wn z!o$P82Kml2SCXMB)a>u(XpvdmWn=R`?M`5GIXF1DQkJ0eOV6|FO4qnbS&Tq?-2k=T zzu{?JKl*jCN0}%6xb>U?(kH|1WYL|k($XF)7#dFI%+1Z=D0fN|1$o+#vgNCj<+Cy| zd6=z^lvdC&?Z(@2`i#VXvAKtZs&N=6ZXKTjRIeAE%p~ykJ0m6 zPBsO$j;<1xg-^)ZDm;cW9E3-~`Q2GC{XUt@z6G+=ZP8;)Zv1I9PaGT^zF79%d_U3o zm^B4m@@<@?#u(}2um`J(PIulpur)6m+g&tC+)vFU^6lv85cE5%iGcdn2+#eV4Q(Yg zL66BTJU3pSedL?{?=M|L#vu)iaY6xJX%n@grrYH%L2F!{ul!T_uVfNQY;0^exw|Kr zRgUwfLAxVWV3bsyFVLQ6lE)6e z8<*hGiz;Fh6ubc^{d-rJnQ*+NnVnq(@f}(D?(g5(p9+)1YJ`m4b(4{4H+j~V%J{yt zv`^vyme1S#>goG+kqV+^hTu1N3nBE-savk{#*FqgrmEyhWFo z*uGc!jTmR%KEm5mFTPzb$`{aPn7G`E_>Pi23>AvF&3B!ekQq_sd3+B$uF~zm8(zZ|2ifE`D<6mbJ zf=XAg$8~*wOzg|1KY;2q{|Ki1q z8mN3$Rs+A|Eoj!FFLv?Dm ziFyU6;Z`C}gHio+v^#Q9WZ(aOPWys(Yb+!3V4ayvx4-6eO?u8(OL^D!U(#lioFYGk zAuab*t?;a>li9qC*&ycU;I(4fVWb99xIE>1=bkv)UB@hCyFOOa-cn>;UnV3h{B2-> zSofo0U}K|rL_|dPyLaepY-|gQi}Y-4cq1bt$gl|xzPh)!=QR86DoRpE=QMVSk)FJUTPQyl0k{ zLgM4e@Ne>A|M>CaZ9+o(!TN6=j}61GUi-L#fq~Ft1K{svqiG{yBF^=u@LoCexx5e~ zpcTc2XV=r$zt0l&;1%Wr6_smCKeIZIcb9VV^Mh9g3+G`bpSji;ayB0SrLvoDq%rS^ zc%ZFKMnElyx^w3a1{Rh%jLR#B>D*>l;5|cTTSM*$3*UsM*b0ZzX}Le=Zj(fAP%iV) z$%!WP!Rh}T+gvs3!;Q6PmX?p+0~XLao#|AnFL>&@{Uu0NLPS8L zuG!qRCdEBC#tfIEKAEw7!TA>r)OURKJHH8Yym@D_2Mq;}uX^Y(4+$*S( zXGnW`m^L=tf1Ql%_0j1m8~@^DCzq$^0avs{_xaovp2dt zzWSaf`cF+Yc+<16garfy-0kFc{j2N;Bg)#^S~-;uXLG7ST2T=fb?fe3%-Go2WC{Pq z=FfKgRAy#oo^U)=GlZ|AYMkdWFB?Cnq@}eE4}b1*4!fzVs~aC5pR7Ta-+wd?X!9DJ z?;n-&1Ja&o``DEM5E!b& zct%fAwe{794bj=TL)QdBN|WGQOwThKx-Vb!%&Q4i@|d=CNPZ%Bc+XTrj8`1obX!45 zNm^F6U~1H6G?{8{b2IwwTcWk!)r4_(RldPAR5vz`DmU-k`C038e6K0dNE!-nrZzTK zGsC-beAvX&Mz)ZGoILkH81>m%aCjcmCUF5kc2Hg(gI-BpDfRlqXWP9c5({BrVK+-l z%X3Gv2Sg7^!;GZ3xVUr&2Z#-e-{su&ij3PH&RE>v4@q0CXkNT8rh#{AxC3LjWptE6 zE{eKkcj>3|(Uw`!`}a8qx4w;y={eQDG$+W*ze>^G)zdRCurhkIl;w{?@$>Vekop$& z+AEq8tf;I+Lw))31%(n35n;G{cMkf)(GgxsO$pH`k*}{W3u>K84B-HzZ6CT|oIlsm zNl;c9bdFR_&AK$l6e=&bVZdr}SQG9#lbZ0pkm7Sx!f0YY(o?$qJuFfm{ohAiCiad< zDy-Ck?M^!X8Th1Ihle5YLm>`OSXZI5;)#koT!-rN%s8bLa=}2E+S^~t_A+?n?92lt z^)J2NaatTjN8)w3@dphRLP$q0SeBbBWtyEO_-k-*ULKGn@T4|f;^H6z$;bNowduJS zwq3F&Z~M_R@=T{=+Oxi4Y9E%>e`Idy*1CQcZ1viKU1n@{M|jk4*W?L>EPxznWaxe; zD@@utIyijQ0>jYj%Hd7>`68i#vFm+8jh2~o&b7bEDKTtV`VRkxo{1@hMXLb$FsKiO z73T-P9mncDxK0k%@n8{A&47PRt*yiIm6GPx#w$?)a43(CkGE|v<`m`R{_V&w@}-c28NJenb9tO5qT3H!?<4-*n6n#xiP4na=HC=Vy^2*9A%&eN{>o+-#>M*3GrH_t| zZb?Yc(bCeYYiXV4>IKuhX}=<)F?UccPxO$@mY42!psMEnNG{c!HT}^Dp9NaVyvGOb zwnpLi{^oa%yKJ==B>cD1tE)X4I=v4!BB7Oz3IVz{cD#r6;q-&W)2EF^wGFQm5_IAo zB?7#+-=X$kVPUDMtHbRT8~ydmW(W5>qh_dr#U_*6<{fN@q+g0)U5?hBZOmubwe9IHp0T0*1USDW^M;~pJO z`#bOcRiPGiLWbQF3C4(9p=Ht<@qsBR1SeS z*vrApT(q;lf6gQza5!~-dUy$yRj)GW)>^N1U&_Q}+DJvkKD463#ajpdv9Pd6&?$bz zVr^^NgLDC_orSLU{QP_v8L$$tgTZQ9;$q2ZY2L8~*4vXuIbjpZQq@DxS`3pIKM^ZQ z*X*AqZB@N(8y`{=$RTtkIwLX7o^pz-^N${(;jn~kp z^bPd&&%Xk9sRt}?t|UxOK_OyP<7B3+tv$pmB2vGLgM(wnr;@~exZ0&Bw>rxtG*;?ZksYaj12FqC9l3pZIy-acwn3kLZ^|1HC1QPp=lo_uURR4MLt^Jk%zsri{XIwb^=$k%2GsViEwBrJVGToYK zQP*7gxlK0dy_i}0@s0JIT{ zi?id8f9l+SJHrBAyLP`zi!9&H+`O`FdU_hcjT!6f>k_ut?89?sHa0ltH#Qu$cNUf2 zgocLFL-(E2<&AA;+}46e&)M8O>3fl2P3#B-{)e6b@;^_g#B-ojYR zrDi|M4fev!C}1Ya`Xk%Tjk$3i4b0bBMGd6(Js`61IM21QL8-f#36o2C!XO{dSE7*M z;NTFDk&)3Sf5Ui8mlzZjv;a~CE&t+h6FfF^d*miHVtm)s$(UFz^S1EIQIJ7`kHuB)P8*y=Qz4o_eTYn?7 zTyfCtT@S~^_;@M6y@MKHZSW?1Hg@OYm7jc8<$^Oay z>VDd4kMuPK1%(p*%8r@uF?X6o1O@lP-@Hj1AwL=YZFr3aHIn{b(oJ!GaF~m;sQR)% z;h^KuFy*2{{?Bg{0R@}Bs#VwWUaR_)NQV$0`^DCHe(?CWS?g;SKhgHe_fSkBTU%RS zVJ=TMfQHfm=v!{oBlF$6`5a&)1pkJMb>H|(>O6Y%WT;4k@S|bvTw!4$1-IF^pstpd zmVc2Z$L^qjwPoD1=-yXK;=Gn`ed4Z^)S1VxuaiE{agFV~qlx(S8W&bl$Xzv$`rV*6 z6v2F&E;a74LP}SXSMPl7X(7RA5rr;_=izZ!DRFtWYcen}z}Xr?csci!00lD2(Z%WJ z?<@)bDc943b-9zh71!Wg=D>_Zpg+$zOd8M95|#f-?*CN@A)xtZV{7Ye0UV#cG^Q7= zd2B2h$P~+xaf264A+utMYo^|%Q&t6kz@U{TaGP)DIyeVykNhgssEbrj%tE?(Y+S7y z&exhwT(k2TCaJ24Fj}bD*ov)rEqg=5qM{CNh&;EePtCwiLzmIm+TME=o$uoC52wd3hKhUwj8( zE9QH$X9VTf70;rjh;%eyW`kqUgzWiLM#s@>fV@pRPE>UThds@Vi$nMIJ%7aZb7JBq ztZ?b0sX}H})^BZXc_K0z+Dc^-(-ZPim*3xRXx{i`ge!Ep<8*xC{gtJ_S@`|z-=oY7 zeePfJ0SUCl`X^l;>l4ABK5>8V?nWLTKs7fv_tC#o5)*)@SGKpeu?Y#~WMwh$s-}m+ z7|aP=WWgcM%gz1O*V}8i0esVziipT`vf9z)x%s=^#f1e^J3D-Hb8{TPY!*IxsY8OW zAB)iArms=nTTp;SZmFuO;^0%!h2HVQW}?b^dVl?QQp}|&VB(IW?fJ*0UlXY*wisVu zqq3wXB^{(_wkR;Se#EEprVl;fT}p~R8pq3>ogFJjM@LYqQ&kld%!H|Pi9zXdb8_PT zQ*JJI0?^jPz3lsUc>@)d^?Mu`{?hjQIYmWQYYPi*!lI&QI>5H)o2s~o=;%!DKWPi) z9!7sQ`FH-jtL0E4JKi9zton0nGl`5SA6{^W@hGqckk}3t*<+u z9@xWva?_9uV8cYRoBB#A@_bEFU7biZOT5obzKtTAS@Hd|pP8r7FZ~zKj(4ZD3Y0bG zoeV5*6_=KlF62mu^xnR6r%8a1uSykQO^i!w?q|$zIBd|$larJ6)X5khy-QS(qPPOh zC7`T~%kONvlm9a0u8pCgq0>w=n$O`MkEbQCLV38kmj&#`K2a5_Wzk+ICm%63sB_0m zp`xPFdi=$%Wq)EyIgug*#eYK1ffei1s8q>1_}x|V*VFCCoQkA zltX?MMcdMGA*7z;;7x?2BBziFsNoW2g+ii!%o@+VitKfk#-a8~vCb51_vrxGy$ zWn=#5`?^2%D=lSUtUiRFOqNTDygUz2%nEm?mpYgt zn^WftAT%`ydu*^9x)x!}BK6sxzkwOTPLAjLw|7AN`Pvx6vu8^;!gC*5w&29}Q^u#J z;(|0-(0VI1?DOaQ8&eHEZZ{r^v9q&}fO=3vdGlth%JtB@1H;4l(7q=|+uIkZWi)n_ zI|s=aa(IhMO7JTxO6yjL23{^KET|h#9laKNxO$JDpOzm+oCZjZCQU(i1`3Lbi|5D6 z%^UtqPW}K`_1#T0k{)s5%4Ia-Kg7iihJ%>+xqIq&&=GiFEdWEj1Vu#Xo(Ly)APhBV z?BBqWc%KLpNL4}SGaJa2@dcUnE-y__;eWnI?^l2Sr=7z?4*&%YXHQ>R1r-+; z8v>a73ywsG8~ZhS9HWT5r>^BpUr1O?K50=u2&KZqd*iNEs5<-xMAXZd%K0e zpfg_(gB!6<+*jNB)C*c~^J1ckrmsJ2F$NNy1gXuT+4JJy>=4WG(LopWeDPu&Fl(n9 z`#$|Hk4*n}m3+_rk6!0X6^b|St^G^Dy;QdHOdE~2bEvf2R3yr6?(Sx_`xbn#^3*(?Wnu1?nOz6Iej_Ig?<`TX46?ZV>XRrDF^!SW$lk~(bcag&CVF=K7t zVc)TZYLzN9psjy~hJv1364PUX?3@EGYUYZdxUBeGu%Y?8;nF>Q>@^O?egV)eORMi@ zrT1O0Z*d05zC2R;o9U+zsFE;ViErE(Q`8_<{sqU#=8XO2kuWfJR^`quo?juH3op~h zKH1ztE5n;HwJ!~H5V-e&v?gC1&lX+q#@|~yOQ~AEN+tCxTrslTwMVw*W@ z)HKW6i}qUFfp*3`oIQDE721lNi@4ZWlDdXp$f7Jz3}sLfFK9P7Yxgl9Ong3C04wft)x(6Uf9u&}pGSXK z9C0IFw){5>(38j^4f6;d@00uLqp^vJ+z%gcQ&Lj4_V*DF*7@?n7la8@SO-)p7Enh?L^XdUBg`uRhlo)_2c(PAGM@I#$|E_KYu@X^}W?Ehm6A__M z-+O!80Alf2_1~JTbH_vEL>PD)`ubs2ReXR!F=5XUwHDfWBp_Va@7emz#?}Z*UU0V# z)}y%Snwycwi2oVDQJ3xS{^wG0@^CduQgh@iLpYC7D<*R_~8sI=zkUYoC0z zIJ1Ls+GCH;@b+l0RjewWf%|8Ok)w&%ix)y5)B$0811B5gCVH?hP+N2Dc!mvL*Ny7j z0<*H{42_IS76jXU;p5G38Zx%mKC^?3R&WAKC0?fZRP_It4`yTh9Xi+ExaOdO%{`#evF8YE5G7ihX ztGO<}wzWMJ^*+EyfpKEI*?^t1`?viK#ZxV>+;)IBWV|%3o$_(@*jl`y+&E_nq7Qm}9L-_QfC2<$9w^ngD>>(k&BE6UCe@XvyW zK~zZapCFZS77`wg3+MRij|R6D5+HWXi!N75+#MV^p~s@3Dk@%Tg|9wNUjf7C#(f(y zuYB!cJ_CFLeup_+=ti_(I)Tt)zjKxN{b{eNL;Kt9 z+mdPn;e-D3#hNTO&ojt!~nG<+l|0c@1#)(je;7;dN>C;>A7Y-crTPvGFdxt(X)K05N;TOEzQ zy!x8Rk4#3xj{i5Sh{)sDU{r5E-=1s#1+L2ae?R)yl02dXZGrbp&s?{+gw#wG(^kVt z((6yn-u^DRuv>nwcnb(>y8r6kd}xg%l~9QPg70K%ZjKILhwmYS z^97!cY{3byF=*Pcv2U_)%m zo!iKuMtnN=M`6F8zbCQ1Obcmb`Vu=mjmSVKC50OI;)~&phAAhBvt?s<+sk6zvW}k; zm-rxLe$x9a`-6!|`q3kDSQu$&l>Umkksuk)f%XHq^X%zv6+H6ujma=*o$?C4%+qac zNK=%uuwWdsgwuZ<|MCkFw1G3-`)ZSbIPZJXS7SBuzWkaG0%|qzP&vP3+&qqzAqC}w)VZ6 zt%_M!@fy?wdWTf2_KyWqMzfhebousOOVALB;lD29yX<8ZmAaSICRs%go~sR@g7{)S zTB0tBNHU}qr zYj^h*G$|5NQp9Gz5Q2#l3K$i(55&aaii&$*eU3@sdpjWAg~-RsY5dn@Wo40KfPF!n z9{>fnMMS8eXD5q!e`&roj1)I0j7XLOPy;gV5#GzD`;n~2;RDd-$y!&e=g*&`k;&K> zSf8x&k6Bsf5gL7eVIa?6vu7I#p^~F>L&huoNsyD!dDS(kn6=-&TEoV|vR&xy+dDL* zZQ3sI(#ipRJc+ZN9LBQSzz*9H2nD?T zbG-5y;pF5b*gaQ3DU6_#poR7C1b_8-3CP4vbS#30urB}uXIEF>r0_nQgU|pfK)`Xz zbiO0*q_WdpWnRh{cJ_wm;KsO~}@*!qA9B4E6 zxuO!XpJAA3{Pt2XuSPx^+1lINpOX%5`$Pa|L5Kou^VofTOnyw03$|i?yVtiKh6M+= z7Xc{f-P+r;)56=))z&usm8XC-P!LIjz_q_6CPo9NDOucC7^Kauf5U9bnWB_ne6;~s zMcy2!?=C5?x)}P?mX}xcn*0zCJQ#{$9;(svU`_v(mWb`g-xU^kh|gd8xcUk3v(T`x z*`Mq<`1qi1U!@iG3;@6}2Y}^lqektIuJ&-mD5waMU=ip|O- z`~%hDcO(8Zj9Q!8{4a^*;E+banj2A)k>$0xv6phd1OLpDK)~QJx;7&N12+U8k-!7A z4tW6dGBUxEefn~8Sbyq0mL!^9?B4YhrA2ZLouIt~-a75t0n=u7Wu?pCxU%L~y~n1y zwssKAZpdTofd4RlgU_me^m0b32oN}u+NiP~JnAIpG%VVjnNjzbV33xVqA-W0uBnkE zSiK^+Jf0Hl{`tA(9c;5gKy*tR8;!OwN4%6fzXe4^;G>EvX1C~a z(PsL|P^Q=p4-E|ssu@Q5x&9&9eUL$u1srKDe`ZD!&hD)ax&>%Mf^N%15DCF}q;h5ma(hU1 zwLnB$8>m!tHZ%a*#v^Ox{m!E1G(bAJ0Qs+~Bvn zaXAUka)XNM4xlg43qhS1h@z9={FKxJu>v!5^HO&L8kTDBMk@3m+;9#6Y(^r!wKZM|39L*CV&=(dj5C~)^R3)=spH4pUmmIRO!!%WV7_3~vh!8;k?t^)wib41*hB|F>Od1zGMghxdch>3|w zoYwtc?eyCgMAV!kBO=(3*90oAf!uffeD|lsAaY>LMQNkIR8-_jOG#O!7OX&Z!Cw!% zJ!lDu?tsx_psSlS1ucsrsljHLRajiycxDVZpA}RWS7TujE)LEMe970AM~)$TIiwsm zz$cI-kBh4-kHe(kS3eFn5dsn^SrwJiXgHL1^Up}YFw?7l&H*bHf(%`NIWU;`5tE&c zj_yK8;^OGe%U7>rK-L6MiKK@>76xi^Iag6;So;!$$hoqn3n@#>yI&msgaZ)`WO1`^ zhYi{X#qX4t21{Z2;hW>dBon8d1%-?36p#X1e*PrfoNjVlP|W2pFW39<*<5>^e>uYTLnc$))8Z_^rI@kWw{{k&TM-ED}`veEG zF13Qd{~whZ_elcafHn`uUO9w*N03@|@XOjA42W~^35P0aYF5a|W>+shHGXw6)OP+z zibVJo=z(BGqnF#fjFU3yBLobYTGDaOj-e+{nh1&OySwwlQ-zd~SYzyOmIa zo96X*HXs=#A3nT{CmXK}AfoxEF3p6KVdm%Or&sY5$Dc}CL%_5-KzO$g8Ytpi+fCLG z1B}3z8L5C;!61JV42cYGn(*&Ey}hC?u;Rp=oSaW|wbbP0i+1zXfcF8OM^ZmIIcAlK z0c0orzg(Ty5@_wzm<-`3wgsx$C3;beJdxc_b%iKG&6LI_~P zGc8!bSvc?!@qHMY(oD^kmF3TV(jxH}U<;*{-o*5iP$b-zy*HH81!MNI1V~d0*nqNS zUEu}^H<1Kp7HEe%Suln%;V0Vv_NA5ko<9G3h<=QNJa7L65rHy+v@mV}cQ4 z-WKuLxW&MLG3~Q^?aGxa&%JTPcAvm>Hq-+)0m1A1e8!W*O+s)+VCDlYOIX=zR0MVgcmz~E zBY-!6ku?HAvqBPNur!FSg;3ziXKDq~q$1F7sJ?ys_IhKyvQPzIQPBTF6nSzGJGP@~ zAoK+S41jKFkZPPDJpxb*LX9}y2Wx0321dq~@o|vwZvr7~h5%4GwE3q`pGv|)W=BBt z28Cm56HA-$$^Q0sU}h%mYkbO=(1q>()Ln;ATp-v#s9U0<)PO&~!MU1+FfCwlbAYE~ zpKU4p%~QaHzr)Us59qDGzhCn5;+!6;1-KKr_fI5d3G(CTv2%$|5b<`P65Zg?G1qxr zb1RqBh(}B{Wdnm$lEHExMA@8$4-S>^K*M!8`K_})#yhW z{~w+(&0($)-Mkro-*#l{=l;RLYZ&KXYIIBRLxS$1tn3`%SriK11u`v2J`I>SDyb*9 zwpIv0{?>2DCcrqjkPu70XMu}|a)3AyCPF3?*g4+<;v_8e+0FYZgW$)rbDA_pfC>Dn z&r%vB4?|80sJz#($FS>mc6Qk?Dl4D;M8GgY=3$6-(`|V`Eh07b;K2hJ)E;lXCJ8u9 zHp9W>fpDtZzoGXs-Bzsr{+Cn$8Q1!(h~S{*EG#c$%gN1-eJvCc6c(O?(yf6MWp7_! z3PA`FF(A#!)tMP?ShewLOA2mh9-ATN8>W!Q1%wwoTpw>r{y^Fa!*N z*postI9mc<*#;DAW@aV;c5C-gp=`w*R#^@XPKI2cgFQ-IDEl*rO3Ykbov;vBqegwq ztdKbZ*?nH*U;=lpGN?rnUB8|H6K8d`)W8ZIuF@F<*F-uo8T^Ewc?TBg@9E#aMJ}@Z zFNm`qNYDv^42}YEfCQ!?@c$NYTW2>m2q1?FPw!{q0WBEGHo(CKSp;YD+1~QN5rCSb zpPxT}Ml8_F^G$#Fx!FTk4*(%x=C?(i{ns z$(Vuo4?WfFDVlCFe31(o4TSD~DJW z8mwZ(rrrMiN2LbsbnK7**AYNkd8K9FOSlpd@b4jV;|q|8q^2eb0Io%@Ax#4VT+m@T39#0q`#e%?#d@?`PV63S{deYF(F}K*kNAuJu3# z^mt9jKXvz;WblshAsR&#BoA`z*+}W*)&iBZFL(I(;vd)R>uZCRx_c<}{pU|P&=#Ph z&8)3OBq!e}F=@*5-vEpQyAnm`e@09t;1KlY4Ib#|2+N&5vx0L8x2+z+Mj?S+=yA6= zIin>0)dSPx=H;EA-i51lAx%vZP4^L3M9}wy51MW{5DFL>ra*y_z|-+4VoHIGenlb# z*pB}{Ds?e?dx*P%I@1my5NWSh@yNOFVJGG?vzs(h1IGAEc*_n4_}nuyxKpB;S`chG zOLWgZ(#jcHX^sD-xWm-fFfhbe8PCbV{8GPnl+)0o)Ma6STaavUPu z<`5G@G;w&w*B}YJ13~M@F&y-e_wV(ty=6+3hb|5Jf(%^Yz$Bm^D>=MaDa@*X$GV@d z?>K>vTr`zpa9bU*AA(N94Q)g%yI5UUmjZZV>ETUhn?MF}697xlbS5oTmYtoQ7h(MW zF@Xv9`)IuM4NwgX`aNL3&Tejoe>XR$t6~i3KH9#2i%mScwl-;J+hYPq$`hJ={rvnq z#r>XZX_UYViitY=^g~C?EiCT8070eqO++&Wix()#INE7Y=1;S;Q0dw%o z-F&+WIN*X>8|b}sa5>f6NK^CImyR-1xS=ccp{C{rjEg7bzBb@cfKOKbV%HMP3?7;v z6vgA>!;t*RJFebeUS6(wzVa*Bhl-(5QHzZX`ZZiT8m@$|Cj*x2&9|;wd{k&{nb&3X z9T*Kl`(c7CaplWnhk$zu4gorBXb6bFiCiW>rk%sBgzWg!+THMjmgAC=ZiuVtY>teALO>xtR!GujS!y1B8XE_v1rp~(AhUzodqLZo%Lz9>fH5Zv zyR+6kU++dZ0Ss$+AkI3E%_sn|ICeA&d|au@cbQTHV`7N!@$#;om5O`qVFQ@s%6<2a zP7GMSv4g|LG=A{WFo;E0KyLyX5d{29Q%j2;5=w6JeX5fu{gB`Z1>JBSMj7aHh`lzu zu+ZkF_auzLEmsI(?|`cQ{rd-rZ$EDkW&MGxX8m1UG=eLAf{KGx{TyVwwa6R5?Gz-7 z?&&Ewm_w9hz3@E-aw5$@3u8c_g!oZ`G>HhDkUj_j5db3e#Fzm%Jpp9>@w4dRM-NGa zTOmh@0Kkhcp}CwK9M^bwc&>B;{%W4z{^Yu(YGq}$6-`A+3700SKTF#^2izyXg${fclf`@*EaqVn2UCx2qPM8w)E8lvc2!tK9kP*Y~RXx$VbCN6np`AuHV2Jcw@sVT2K1 zs4#KRj+Db!Jh5jfh>5pZSy-&;;e9bRr77;fok!2@DE1FBq16yMq69_!O{2>qF6Adl zACjb=c97X);hK38A`%CJ7MTG4d!}9sn$#%ol=)DBX3a)9vTw*yxX^L8%uzZ$O6wm{%e8rrPN{49ig-y ze}^K@z!F{2wMhG;SEZlgU&q?@I7R`n6ngmZA$P?ULn|w007V83hA$}rP(3cr%X4ED z7xzH_sp-J+DEhDvb`qDNZ=$*=v@`2EC; z0>NDzx$0#4ep+1qEL@Zf*IjyFX#`-B2jEMG`zKnf>2FK_5u%oigd5HH`oGcQg& z2Cv^iPjBo!j4YW_1MXJnau3$-UW}+RHFU1z5sphd^?y;xxR02na2X1Yf;1p!pnnm| z%T_mT-h`whI@n3F0FhD8wa z3KUnMV&A5x>A+OzfG{wk#TYjFPy#$hULRR>&$^&Xc?-Dk1RS#` zA3)2EOiN1xI-Z%Eo=*K3v~ZB8W}E~)Pxe?q@l!W6#E0{WPa_<^_H6nkB5HlWIfsdO zL&EPY61ns9h)o;rb#Y$Ntf-a$G5f7un}o`o^=j&G30>FWepeTlYzX9l0sxnBZiBi8 zq8DO50}cjJ>D9u*Q!t9RbzggfeU6N5$nc;fAvDI<`2CY3kO1B!p^+L1+FE1~+hT012h=xLJ6*!ta z_7i-l;rdqMxp$DIy59U0mCf9Z9gqrLummn&=DvLSGPbJ9O~lCP&x@Ftm;qF9ppvf>!Lr!Om_?GPPh@z%SSnqJ9`~8y~CRY1d0j=*Io&gSg^HrQbjX zHhuaO0}cVQyGUyK$KbDDb6mSONl9P9V#DKanUob3HA5~H-N?uY*;Jl~jm!CnVbOsq| zZ*80u^)@*f8y5X%odEp==(N$0=dXvqaD4&=`rYt=oQ#f+P*-=9oa0r2HEbZ`OK4wP7@PjVlC_6g3}@ z>R7%avNZHuI90gt+z`@`Mv^d!ioh9(tvJg75(Pmg0~B`+!J7!(he1XQiUeGq2I~$Z zt*-;x0Akj_uVL}miP)zUc4PKCT^FDRjSO^q5eU_pEGc2rU%+&K1`(1)a((Y63jk%? z1};tEu=^jYs}D0&z>ii`R!*Nzj1$`OKiClCT-x=YIa(bwTFE0~8Bi!m#@%*j4Z(jS zlttNkqwmBlQnLR>Q zk3A;|_iH7xex5ff9iF-xWvzOd+{@9$&M?Hg``4jwZ)KxE&9K7MCX->EmnkJ5>;9jf zT{*JMAy+Z^7YDcbKr!k7nF((80w`$xV*eY7`hi^o+~2Y{Ik2|&MdQReYeG0c2|%(B zKoNxN*`{#q1g`Ldng*fzLKx5hm|ub8Yen&KduJyXT={?kaUYQQs+QbbTymkw9Upj~ zx$pW-1H*-~?t|0A!^>+9>=A+}ODX;Yn(Yih>U5yz^#in&4tm?3+>N*!keK(j{ zBQ*fpgy!bvXLM(~$?)Hj!zu60Zb)n;o*W;$*SIc48CKs=XC3snl+l>rv!8HNg9a3w z{0hAHCMURrivTpHjp{v9;WX91bkojGb(Xjz{W2_VcX376bdE85UW^qdOMJt2=XXof z^<;ahEGI4drOZGqs_AS858C>~12-Cx=TAWinqYDHX%3kIxQ7}54nzEM4ag?<{LYSA z%gf8Jj;1Aa$CsAc<2gERy2H=DQ3BfGMnixC*gI7ESa;M+v>FKgeZTH_ma?{vj!XzH zr$X$Nfr+Uw=Za)YE?@YbY|q^a~p(ffTK9{nmx>CxC&F! zT%8w!2@0BZ%x`j*~K}0LfS4;!JvK@_YF@P;JjrR*Z*g2Z-;R&k52=BJ&Gh;_^$<(4Nh1tTx5hO*orWN zLMQx6rQ2jOE0b`2U>%&*_5SP!=owStK1&%MdS5+x;D-h+Zv0$?j`AKRViVcHB+#^T zw9Cm&xZsBt1$cJ@Y?LSAD>wFXQM$t6@(FRMBf#2XX-zH&JRkF zFapi8-cRqw$_l@^rIZHh<<77-Gl|^U<1^GRjCS+fv5E5;JzlzRj)NmoIYl z@j9BB%?KTrf3fx=dBBUaK^AZyH1Sq@^#6|ldF$cOwP8gYz!_RogBv=Fms_d z(m3EZr}~FqaI-qznf#d{(pjXT22>iHL?`gvD2QUTb#UloRnrA|W}7b09hbl^+5}zS zNJ@W5#?ugPC0`!@^ml+0;64xP)PMtA!Z`(E%hsly2dXd_Ki)ggPDtHolyn$3b97Ow zQP-8b;R6<&Cw5o2My8x8-oM+V>D1jf-&69^QaCNmXvL%x&P4RYXS_$hBBG-Ui(-(n zhU+n=R#wa%1GM7C4J%Gbm(@>838|@yHa4tK?5tac@EZ|=uUCVU zUkFYP<3}kb9U5SSqX8R$4lWImu|Qvdx+c$4RNHQRWpL(E2DwoE4C&f(szMA#{GpFqL?*EA3b`5kZO2`KQabpk{&jiJ8W^PUj#ySMSsK7PI1DaN%SAHb`fKVKyGIzIT zrH#E-7%lr!gNuvVkRJou2e=ui+7B2+it9`CI;ya6Oo7DVxy$R*O-pkv$U1 z$ja7`kv$Sw*^-nk>;JevzjMxYopYV*jPLdB^SSC1-uHdK?&s_Ie5{wX>ufDgT#p1~ z8hYrHLW>`#@IRp~i{3kBT2oGEnaH@6^Zjm5*%!yxyQ*~Gr5Ie-N;~Y}?YZ_(1f-K}RZL7%``{GFdS9jNqy>iHfjB&2LUB_)#hfKYl<5q;`9;Db0@Irf7S#y9%6OL6F#npPz;AU*} zfG3uurCD@Uv|Llq=qN?dLMNWAloU7j8Ty+oB9YBUyo23X?SI~n`g)Q7D;{(GmsE}t zUAMqoZ?oPX0|Ul}UTaQrV$1^K;?Gf`cQP|q0imL3IJOV6hZEQ&(6IqGP6B*P0+DD7 z`j2u9E&~y8Gsf7P5K-}CdRo=e@`pkK2+&~gi?9T}|N8YQNJK`&#{fUZ9z=QUO(Gt^ z3|1Hk%U0j~WGOeZ&c)7|+Gs_YfW+q%X{R*a#(qcF`hiK|??h)-e#Ki7h1UiUp1Xl3&k5_B?qzzW-TuH+Vf0qkHTN6xJq~q6&Q{tonIEz4;E+CG6G((0 zAp-&O*t{<5v9nuz%jN=k*7`V6_?KZxCgJ#qiHS*^6i;piPR;+uBz7!%5gXJrlpAq^ zxMRu!4Gz!Rfo-cp|Kx0fea^r{R9Kc@j~rZjXE8G&^B%8NaFeWX9cnK9#gii|JEK1R zTcb%*pRzB^%S%RkO29qdP_4o$0bVN}%NPyUwr0mKQ(d{#s;W&qu}!G*uLUkden)k0 zgR&C{+ZWWTx@J(YMdtXUh`ZS>&|g*p~24t&;-`u`|f>`MYhA1&(@3(L8B z(%J1Y>r2-^2Xm_8ZbA5viaJsH?`H-w3leG^KnD^nK8&F?)ST{>7#MD&V>qXz#C7=a zecY4)NuFps4HJ9maghK1RyblRO`u0~(g$rmu_ z0tPH4t8_;B0cMKkmjc4tc*T~M$KtMczR`0zEG~Xq`P#L?2+=zt6J4}i{2g$i{8~pF zc}5`{8wiXU<2NKu@Jgwy&bYXhTUV^c2o<}>aO0QeK+#*c7Y-p2YX7xs*S6m5ukfmP zXFn3)tY>fkRvb$`v>%S!E5P0^({P<3lka{*x+CcQQU8bECDiJ(zfb9F=!L!eLP1Q|Js)TX$Q}z5r8_&)?LUr%^gWWYt zALQiN-GkEPX0*ulv!Ada6f6K9GRm&aX`Ba&u>ch6yGG}iygF&LVLySg z9|7RzVH44?;)jN30|^KLADe_9_3t#zPBkhnm6X?Sor;op>tX;jvlvu!DOG`XOQ9St zJ2z28!}XgY*ZWF9;y37L-5lTO=IeHU`_`UcP>^W_Ucd~$puaO!&o#S{1n{pO9KkT& z{C51xgSw&w7JV16v$hMZDJpsxS%oM6Nh=-FnAqh8>PZ%Bb(Za@vD`@6Lm@3T^a`J< zs#r-^iytebOZhUWRhAOXWy>Z@(s7?CsB$wRfx&FnpdRlN#Nk3kfduRkT|mBRSTm7g_J_%gh}Hmv(wXGNRIukHkI z4nHF~p0_GvCn;)+Db$D=6eV>+{7PZz0m_lhaA}>$)GPd*kU4aljY!9lmv>^nuc?fG z^VPl@LSO9Rw7v=THR$^EZGsJ^?e5TH7+!8u*eofSMtZH=r={dZ@Ba%7t;V|06`qrGZPh z*y<#p3i0N7G_jn}|EEn}O1=>u5>lyWQ&t7xmReeD`BDSgFoHOKt@~zuev2=Ul5Rj^ zJEiBAeco}*3OOy-XVfjy^Jv-S-e_l)*bSYnN40QdR}MwS#g!uaLztk4(XSWNEAl5m z)c}xC4)4|kOpqFjI@nCnz$;;1nx^3z9-y7@gXxm<8)a#9Zta$DiM-0odO7FZ4o65N z&~d$GWMu4*J$jX_?C6-6`H?RS3Q&v|i2!^B3`;LxG|ku#hFBh3$kAiR1Qx8HG|8+8 z%wqp$@nb0YA}*!m)}tK1cKy5Vrb$PJKus$C-a!FB>_Cy7Xc1i?fw9o#c-BM<~DMktMwP)lAsf4&(O zg>S3a!+1S|E-Ja5MWFzu%-vt?G?|J*GEw|kG2M+xA%`^M+2=w1Xp`rr!(+WV+JdjY z@z2M(IR_UHTc{@2>i6(TvuYZ#V ziS?AK>2R;m6W1!8lwA_u41?+@L0^t&zZS>7QWy>cnZW~YEi%mVJ0ZMRLN&@l|K@c# zU0vrnw~~a~2=PS+ghWJICV{(H_4f8Yf#B7VJxo&xOo13$`bTO>XL~OwD_aWtE@^a% zD1~Vr*R1uWk8+#F(zNM?xZop1RjEYhH-Y}T&FQlGmIA9zA*e$+0fud8f~);P_NOa; zjMqFpsqXNcJ~r+zcm=0mKkw8+gkGAF$Gn!dvT{J?$8=2qZT=6_O4~94bDT!!#zBvE zat=S@9{vlS=*-VcVaY7%bv%wFu1{r1R=B8q@#3u+Pl^muOW8f{YNRDBSPT|IK~A%_*1e$L+S0^`yvn1 z;X^32$Kl7{^S!$}-NfQ6Pheo6Jk5K+xJN#9Pftut072Nv!4d0>et!-XlMwAi0|nN_jB}?Vd3G9n41q$`q$6C!-msp?HD%a&ZKaK^89A)tz4fc zBskLMYIeKP(r^uWi^tFOZ1aRO=G3NlBLb{!@sqc?vC|yvN)N}sla=KTJ1T`?XV`2) ze)fO!2kE-QdHKr9|IdHv|I4?rmt7o@no~ae|LZ%|A*yBD)-E$dTOK;ZSe&eZWEj2xQAB_3U{**=;8j+P zPfO(8n9*wv}AqW08A6lLToBf83cHB@tWejSHpCR3WD46i8Wnp`cn{t@Q-q^_VA=1pJu-8+HdfTmjb5A)w?Mw`EG_!xv}ZiSo965X4Y@BANq);^O%=h6G^Aug?q4VfGTe{dGojghf;bS)*)3XtpFqW zd^1Y(M5w`U+)XW&-%@ioW6p9#NZ{jR!KLjAp*7%t7QnmFEpUUQ7~gMh51y&LqFVUB z$^ddK8+GaG-H)MIm<+?bE=pqn3Q46~^*4P@WS#2MR|{V-PyZA|Sx*|0s(wexqVU?v zm3btfbx-z|?!>e$Mw`sH-((`|d->Q9kCMlP&Y+b~K~2DCi}~oY3g5%7M(F5l2NdIF zqQbK8IXs@At*3ariOzmBu8_2w!o^|1C}P&>F!8ayZ{oQ;-L{8<(R<2#Yr`>04jL{3 zw4^HK_N_rEM`8V>@%~*6t8?d!X>8fPars`(UhX`&Z{y39oCCOG!-ZAFsB_mIL zE>N=Z1`KZgMg0;N$2xxhrRU1oV~b5~&H9^0W!POm5ye-TftF66&Q8Ky~F<10kej0kUH z?V>dQU;Y3Od#pUVK)bd^?f17E4hR2tEkGD|gnuIMeQ(K2JF*c{mJ93G(GI)I>LMtd zu4XrGB&DWyeyyn)FJW2k1{#|N$*P(hQxt#jX|jJf3#+{8VMZ(z)f8T3r+6LU)`~!3 zoB-XPzJKrDCTvwyP>-JFk5FD%&|*N$*(iFEyj#89Nl4@45T42RV*tB);5gKVL)fvP z_3TXsm{kq2Rp+J`z6^H4qOo=Vm8XDYUghKR@*fxB=#!nDnb`##OKiF;mj`h3XH{8U z{$NnagbNTD{}?=(o-M*zq{F9nVC?XYi!-W*qP76T*@g>j3;i(hdGP2>HJvroQVG-K zMSPMc!j=j#`Y*UbQ|^TRcNV+&RSS!>$>wC`M0})eoHEg95Tc-GuYHXVzk&Ip?7F$R z_{Gi~tuv?;Vd25(i__93qJibT?uAo-3UtSFq}WVK2&9GO0)fIuK8(FC>#CL23WU~O z$_i>G!uJ33prFg^fIfH)0_Oe_+aGt8{u}hdG^>tTeSHM&;`0KVeotF8dcphw*Agb5 z^KW&XKd2P;6c$2VD9@&G*?af0vHcWy{AZ=BT~zC+n0jn^t|oWBZV+&oa%*%}BuLHd zLF7z1BaaMEMa^+T3}lo9MCl?P7}t9+be5nxHMtSQdO_#_znovftO&+xR+&=xw6smi0rt%A*Le``DYjJ+0|+cRP+=W?=Zoi`1% zHS&qxVc^kNbaHeY#`sq{-oN$)ol+aXrD2F+-zc3scU!OXZ@3HYq3OlLb2F-H;X**9 zx||V#EoNmkxr>u?U&81w!Q)=BjR*MNj5mc#jBd>;o*bG=48G3Zz+N%2w@sY&-`r=N zTq^#=n~6N60xEa_M#d&pPo6*b$1wr+CMLt5paaP- zew-@xgm!I<8dDW*cAL{ai5anh&F3k}PWx1@DD5Igw>U90k=2-_9UNvSL0Oi_2Xja( zJN^R0@O9~iOxq;e&m+fjIu7m^@Dg>&%urnU{O=5pN$w(9+h@qrz1RQQ-dC{`$1~D_ z@XY=$nYJeoxh9adxi`B+(Z8hJs|gqbd35{E9a%`cGlf(0m@gcjV}Zl&Mn9-=!xpD~ z#ecr)*#J`74t*urlwEKQ6T$=Q{_@__opw@mSid=#iRpmytChX0j;3iCYzvx5VN*Q9 z!#sO`_pjg59`X&KUo3PDA$Qy#q|*5Ke|@%oH0<){sai+_#WmntV&zt#WAkcRma3(S9Ds#IWK%6GI7dt? z&4qincW1mpBnUAek|0I64)y{;U|ToiSNrxh6FYd?RxuF~)?-k=8xZcNa%`r!vNE`goj%b*`_?5V85&|ZI2i@?3Wh*R`nA49$#^S~+f zPP=d&tZsGYm%PdwaNNk)`CcIh7iqGoHZ!$Ukwq<8#P*BhXs)6*$%XJ5dEB*NvH}yB;gLE6I&@02zvu`gR;SfXFRl-PTCnIUpm$!L=FE^Hb=ZTW?K6|7%me z{8H#c__3StQ5p>gjlan0T3yvSTW+0Hwryf4t?3@Ok375dHBK{))MAc|wb|*hTUj4I zUeq$ab5*Fn_H)MxukV9`obpr~83Sv*rHfyw^U^ObdZn~pMP?l7+c@zd4m+l4c76Zu z07Q^-K8a$=k+nHIavyBdf!i+d*L-el%&ckhuAWv4p1xVrToA==dG&SLqI5^0Ze#hC zGd$-1MilR`iq@&*_5-_p`9^i9ry)@>MOZTW{o+B6{7)Mv8=v2IqI2e0x086{%hWLc zz`;QJ%2-(daA`3rWy8kot=MDL)1V~%2>%~S?SVz8g-r-=fS)JT ztK6$?wTfEZJyTlUCNeH}BxD@I{zg7|&7QZ=Vs-!B_l%?Mo!oy7`%?xWBRIYuS-*E? z_|`0{XWT7!$E?z96OKKZqZ-p0lY(Ui0T3UOcqxvbJ^{g3WF)hLcOPKt zy1CUjj|EmRAAB7SqpNWdy<&bu^X3FQ0dkp{qq1Y7qX~N?hFA&z^Zc3linA^eV_ywq zcczcZ{Yg3#Cu!c$dr|CCUx)UrjpgU0*wumWs&QhT2QKCaC4V|5>qrG`&ggP@|uv4n&1=i9aDWh?cBD_yM#?^&UUd<>fYGypcA7#g2xYb z@BP!zI2hr-eVfMYdRCfhPwJIT%(IG88*)WhW)8@^L*4X4aGvuD$epet2Oa?MDWH`Q z0&6aO9!ivYI6p2xa|3Y&B#pc7+_{5|2xW!nGSCKaA_W{{@}n0o_F_YCi7m+_{zbuZ(ltsr+IJj{OoE#@{Rp^+am|9 zAwp~10n?EnpCEuB{4l~63EK7s&_)u@O^5+8NBGw6)3et>9YwfI077irdjiK+5Jwh$ zfPrrbNj?n(m5H1oD9+kB`hn{sS+@bH5c$BwvO0kI2y&*j3}7Zl^A`n} z=q*Ba0P?&H5-xl_gS>|29}hSQ-hbf)ku<*Ns4)>xegkN4=|Z=ax0!)szU~E3$O(2t z9|jD1CnskU_QvN@uA=cp<91F?rpP}4wbhbmp#il7btNzh7(@vA0Wz0h;P$eM(<=Nj z=*sKDp$i8g3&A`U)`h`?gUs^hxZeEfp)k+SuzDG`<-WxNYl~sqytJu-Zu#ziZ{=A9 zB$Fo!n%KEn_N=?l)>W|ITAhoDC4&74_yTA(;M@<;A2+B7mwmbs=kfXO7CO8k+X>+d z3EVRk(R`hck%)j?qcv5P6EAb1K;158W(6d2LEnoD%SQYlkiswcp34;*E3yPLr~Bq} zU0r(f5O_ulc;#>n)nJeil5YO&*cy`OEjrR8PL#uIXAheci7?d=+zSc@#ckk3T@_xk zc+9}N|Dt#ktm(Du*K^*!g}5*Vq{S@M0KtHi`~5zLfjD|*tIpq<7DGtpB9A#8H$xFgZ-g%zc9`5JK$18BX#!kkm(9u zJ!$nVuzyh!6QE{D^=L*^E@GB8EOIQQ#qhoW>-9%D31svp*3rk`+;)*<&dP7rC%MbE zp5qD3&@a6m3g{Bd45b`+2J`k;JArz!KqCvCoMMU1?@qi@5IxC%_qb(j9FlcJItz@3 zu}N~#p-Uys){srY@-hMxygSn_LYYv6k_dYh)HR9dHcOXBUw~L_g=q-ju{Rt-S>^xu zE%QgoIF9C3^O9Vfv<%@-L#U>m-BsKWz+YEZ-0}8PTjVfghTKoJ|LK5g%z0$t&Fqa@nYvDLzE*iw1#O4_*}Kr-eet< ztE9~MXu|x}R6x_nt*q9ISH3UXJfR$0ihqz$;C$z|T=8JN7PYvG#s#|SF+J+uk00;B z7H(d_LbgQ@h+zp9KNycRS9J` ziuM;_{%XV$A$cSBgGJad897p`b8V{iFHQ#Itp;&<7uZ)ZZ*9s-wX zA#FqHBT3!<6W<*7MDqlJXQh3k{Mg=|IiIgccd=uuzz^k{dsTOP73=)^IA-)H#dUVn z`=tV4a3fD-ZDJ1wu*?ea97>esw8#eDw_~?@_B0DfO4#&C5mZ0nXob&T<=ERYuZ?y0 zm8n7^3w8Oi&A04?0u}ZF5IfH!BZ+t#9HOy|%cVdLNg^&C66iP-B#k$^$FYjl;yv=|s5@~Mm zcfpb>?KWo&auz8i-gw&qnr4xj;TnZnL3wtxftH450Ih!2I;&EXs&kP_*jx{qg?15P z*R-!$)K*kQtb1joPsylqv07PKS)W|Nb0t_{G=VU{g`sXzBN)}noau!5#t$vq4cA~zB{5Qb^l{>mqj7twi1yjwBdF8>Z1TU$p&Mhdo&?jeW) z&p|@7^-~uzS_vx+YS$Tj!ZfXn-4Hh%JaB*#W-6KGVa9bh1ancYu@$6CS0^hyG4Duc zCf+CPxsa%Jr1`xA0s4A7e-ITM>)a`JP5*Yw zsIN_FrTNjR%kAc=S*_}IF?jv_{eFxv$Rmm6c#fuW4Z7J{Gy?mQ7J#xV=J z@58!apg<5+!RQRhYt#wrIw+>phX0cNDWyr!li|MmK#W8Y6}VYJZ3&Xg*dYU!3u;V2 z?&%ymC%bXo%Y>MciCo|ABrp>6%l;WRUtG`I%&Pe%%>w;!M}b^W1xePBX}A;5oq}zSkv80+_c2|I3eO8z5VjLtlOJ93gGh zl5lLJQ?TeOEh?x{wtIxQG3>#zvw!1R6FhOgA<=FrrwoGgZ>BdsLWjEeq4`f$)0vRi z7y_|kHa1HR3D;izZkK<4wvabY`~2wN_YSJ*a@2o>$Hw;d8tS97?IsHaXBbt;V9C3V zIsorSr!{axhT|CXp1^>typfAODf}d^I_q2&(!?*Z|ym&a+8Cy>D&_* z!$wMnL``ZK@IpLcnbAf2@Dg1J5mhH#zH}Gw$(iqNpg!oay#0g{boC((fRNuFO2yW# zSNJJ*lnfU7E4s28FGYKJ{C4J-`Kjf2MaO_CAv8JQEb2g$32Lsmq@)ARG64$I1s^{g zFU9mK{D*!o@Izj^#j=>?aG{&Nq2N9(RsBuUHrtoxmPNC2>a6*DS6`ueT)eKBrvh(k7KEs&uDvVh`+G0Qj$=Jm~tU}q5ogBv6 z{qlD2xW?C)%&+#GwDI4WUmw5`u^v~gJA^5QwOl-<*`t5Pb$!bf?lk@o+Ol1eIJ**{ zQ#HKzXcI7DzA^a5g_2oD!l{$_5lPSV97h1%+Hq`6h$fA73epXkfXsw9(&%#Z+wBB>Zn@y-ZN zjWEr9Gr}oh#qhWSgo1SY5sIem+CL|v*+o6c%izXq9)u>T4aZ#KiatY5a0(4ns8hNF ztZ^;S?w}2l*VGIz@N`rI`_GD_wQywIFf`fwq;4}HYXSNbwO?mS5bOZ#unH;e&{zK# z)PjvC1(NJ;9|sJu#sK+b7^?r>;}D{k-`P;G+K6J{B$;zYWB7wE1o&%r zojN$d16b-9+gVYViBI-AU4u|wvaww=F-czgrv%+P5q{b4xgvZr!JX@^tSCN}fgSGgJ}g>i)58{|)eAzdxYP+Ce#YQ`E!Zfe2zHK4;trx?k| z??H$5D2P*wG6p+cWMpvh8G^XP&Zokp^7~&iR#H=8Z%vIqq@Jl9aUmgcFc`sj;8{L} zN0)+e9Frgfm1NMvKBtQDm`73Z4h0k*1%B-gAv+E_5Ch0KCS#;9cq?mgD)atozYhw|MZ;R(5e7CG{2)a#k|z3fOPWe5SBe z$0w~kr!a1GpSCWVo1dT6%0>*WQF+lokrn&1Gw$wEcu|Be{cP>%h`@4?o?q&D?^Z-- zI-IWkh7H?|e2O~enH@|5kyqYy&+d2yL{)pg49!gTqH5U2@v{ow991KB}QK!9#gkv0ui#tWk4)=F( zHUb{#D)*4Yth6%MKSOGc?T-T#zJQ&tzqaGdrh9CKK9hPVWn-o zw_M!s+nI*7zkqYHJfG;BH#qOO!%Us}qRvD}sp)h9o(Wj--GaKG2mvrQ(KNk6Ap|fG z6A6s9>?Cwsgi#vX_%c*(wQz`F%0XYrJRac1F72sVq*d*0wL042ykF*SbofU^431U^apVc7&rPe)IxPTQ$WWj)|NUhH56gY&$Af4|7kYS|;j)~m`Z+p!KQ1nAA+{LO zXaqlX@3uq#1P46{=0Rf2M7P-rG}e9W75ZktIOur^mo-9VE@vB4K=yoAHEStG4G6~= z!il-|@893Q0N-3&v6B@NtzPM5aUk9jM>j`cdth1ViFlaxfTEm(YF}Bp($7OzE)}7#9;IgZpHv8RHh1`@1vDNrqIOEw>O4{hq;N$ye1hd5v8T4H-q>K z)}FiAJ`tLu0{1T0ALYjDlnifpmA5g_`jRAlG|2upjP7PqX1_^Ged4Eyd6IxpabFH$ zf#GZ&7LI$GqKFuUt}zPm*DTi2ERi6F-QM7VRsvS6-ziB+Vk?B@s0tmod4IVy+J2F~ zzP@7msmS_R=@(!Mk-=1r6iwm?3=DwrXbqbj%Q-!ld{D;i(bFN-I=_vCt!}1a8sTB4 z*f4iu0$XDKLwfBaA*T_4s+sXNi+|md;Z5EKa^4JyOCBBS323n8u)_lK6mFSfR$7GJ zm%uuK>VPXk_(aNN2){sNbB4H>m>8&&3%DIl?8<4tOj-@QM zG|;CJqXlrCORyj#M(-icd3k~TY*He6s|ti8Zj`8N(4^11S2+d9(}a?KHMznm8o=Dl zahute@@6yla)CGHw#_os*X+aGH{0$J-|zcwzH~a}?Z43%HiU-EhayhBY6gJT(9fUB zpkIBqK2!fo(Ala)Y=`=(V1_}V>_cM|h?-oGQ#`iVH|1SyUOeeg!g`iu=UV*JCZXt@ zI3bgZsN{bK4bmU@+9o8nO`MvT1W7p606)1Y+3PI9(b7XP=6(Fs4yJ}rJ@5{bC%Keb z#h%a_gz@J@hyL*zmLfqVNHM;-lyHncMSJ816&(?r?~Q~nVxzSVb8&P`Mu9@FSG)Bl zZY4g5lGgdg=mjHaYH;SChbuPQ&)A2sS#!d`3{?6fUQdj5Lx?mX3FsAULnXjlm6M;J zg7*(NL?CwYVZHsSiGfN?zjo9m+NgK{u@R(6yB3O%mM)HH3UKQ{iaQ#|8td!1@&1G6 z2Nsg@=8D%y9rF~wBhe1v5G6#y5m$-%gcuJ#eoQ*^rQ@eE=rnIzr}FOEJCtHqhT>|| z53&N+Y*sRGQ*zTL)|RHXj}OKFCh_dH2CNV*edj5vYC^uy@9i1Y!Ti_!ZfC0}6tCz9T+kCbCqa2#*;!Aw3h zKK?3qanLL^^};xuTvMH{>?P@DpGS{T6^I&7wQOtbts7n0dDisp{Sxj7Dnc6MW_H}N zJsuAX4t@C&K^&NjXM&F6BkDlD;4-ZBBs5uy*rpK{_!2(}$!0%>hU8%`#2h+ADWklg@inD3Stvs9zq^)>rwGR;1F*N&TO|ktCg5IZ?1lm3BUgrzoxK5- zm2?^N4I1_q`eIn+2eEZjeffeBGjQxN-=b&HgP4Hh8z#HE4+v*I+goEI6ruXx_a226 zglH(xh-@P62Bv+&^^b||;iE^xxD{xz?#0LNClT-*wq&f+1bliRjL!jEwt%46RpEJS zm(j;rLFwtMS!6BG`;_O@Wr*)R_W@lf0bxZQv-m)Os^(|EQd3b`Dt<-l?Eo(k^anLS z1W3fe%{X%LklUM%byn4zp(m~46ZOQH(I|9V`&|7=)|Vy7Y{xC<=3-LD4iJlAN&4!8kdxm1uQS{J4b&pc^~`8y=krAXWRO;x7qji z6ADf5g98I`7IyKczcLN5B`dE*n_3|F2$p;es1fAw3!XGX@BkAYum?m0A#tm&B1O~~ zD-q2b_QNKpST*)-Te;^-0X`oqC~YeHN_Ob^@x`z3&qjHeU*VTFuXzMMg8mA^Ktd#S zVTw*N;JSs~9CA^Aa0TKGDjr#1!w{rzZI?j<=S19{NJj(e{mg>2$ zWe?IDT@;!tJ$XI%uh7}YY~r^20dFDOI}(*32N;PbMEk_BfNG7VnjW$XBDtGj4!Jg? zUmVKFDko-oELI}!5M{U;F5(*SPDJt#(c>crI062hNBd6gLPv#nY@#EBCC+_@7J4Zl z-<&5;7E;nm11W}h$P-aKY{$$iK?2p{1c*je*|kqN7?e<9;D%s2i9v0dX;I`P&L@X6 zy$U(>#IfU8dpN$25e~FF(`D7@k!z73LEK54HHsh%CX{Hbmb*|M(0W;^E7kh0-=*OS z^Z6jl3{@RtQPTI5`5p(FNl8oRkeDP&%;1e+h z5t{+NG2!+h=v?Hw0bM_hl|~pdNyJP|NZ%3aiF=Jl_5c~TXeu#w5y&@q2Udh0AV;@7 zLuUuxVw4PZ{#@o{9%9o>Rz9*{zkzMz?!oUyPQt56O}&N$_*?%i`g2Z3R3HV=h=Tk% z?^W6|@!EX@s9ECT;+JsYat?d(9esd~gGfx7y8jhcL{D~RX1@my9xUv`p}!dZnZ#ZC zJTOdv*I90)ZAP^sR8h#1gKw1-5A*_uOax=Fuv=ZS!&E{<$5`e^;p*_-Ck1uZ*3uq4 zc){jAz5oP5x1T$BphE0c0PnlQDgay@6q{W(V>*WfK;v^^+#@^Lo?KAo>ysmfKDa?4 zH=~T&j9Jwm0~nfCB8A4=j5L76Q;ORo5juI_6UKMDU7d*)o6bRWGW1T2oSebl10=d*=*y`HTUCq+J55KVq+vCiCqs+-3CKK70jli`(Hcs0*D z9ldSYL39cP8V3rt32SBr8ae@F)0VmWz;6T78L`Ey%Z7AHZ zd%U|mU4)kp(>swCi(!$7PKMo`FFOv6B=Huar-$_XKHhe~(nJbXB8|@kN4p8UumS$lfZ&e z#KUa*sv~&N4HG)FG2Eb-@jloO*X+cX!7Mn2cVf#Rv{0-7&ds2I13fmd@x5f@`Qf%r z^aK4^RLyoO=2dE0T9HWRx&yLMgQmdI0lDDiCKqf>)MXCtXtxdQ1E z5Aux-kngvgJJ5dKrRDq6bF_W!3HD#8@*f>+uPRtIw^;eL!}Al_kHfZf?eE%Zs>gSC zt|*(hF^|5n(>HGL;NTGw3H$>Y@ptuRVwm0^piMfRqJ9;(7KbLV0$T9<``6CV#pODh zf)H+6!uPQJ3obn(>yu==JY!0@y_4pcE6F^G*^9M`8KXTa8^6BLt~&qxoA+5!D6X?# zS|b0ZWoWc)+risjW~vIHBPrlDU05yw?D>9jo`^LvM_T@$KS#k?H+OXS&CJ;E*+xag z>%v)JZfQyB9a(j2S|)1;38sb3M~i5LaWXhKi$88o4H+KPc%*@fdBye}qalHovjsi5xE^c1oZbA8_t)Bd%Er!x zxg?9YpB50XetgiE$)%{{Y+7n$6Y>fQ z5*~gWqU)jTBm+}3#!*Qg3mrSoASW9Md6$fgcEntyzV|-cKkl(M8(ob5 zhi)AYj@2J@&-y!K=UlUgP97A0rE_&USkqr!wNJywefR^*#$0^A#A4raldR!VxobS4 zy>W!t#9}^9KtuGaz)y$3fB*!{`P>cz1>{GY_UJhPIOhbwH@m+)`RY3r{YwqDMO#-Dad z>{Q^M=dX8Nkdh2KXt&j1Ujk%6eiXYUcJ5T$gKPOe{YI@uoj*cG(O=yhv56tv7C-1} zh4DB2Uc(E61BS@I_vD>-b8`!M`t-2NqmU{xPk_E5mv4yVCv~`>DkmJTnVX;c&6~Fk zisZ5AS()$cvRy^h13T$rc<*l46L{(uyob(SB8DFMO7_+9rpBU?y9MUZxQdXzMkq1q zULHRa`sLb*7bA_?CP=1xm7Y%aS3Pm26eoWZCn&j@S)b z9gor1S>ch7JkI4VD=Q;**+*v&0=*e|EZ={(b5a4Bb$E4k_4LNdzT(orST|w}+Tq9W z@d%TMiLLKNrUYLGj-a!%&k71o6eS5<=mbAozWFbTlzF zl^VpA-k*PPz!M$tGt=Lk#BJ%iS)l=%qxVE-t zub9M#t2_LJNI@lX{tdH`Z7G7xKOE>>IIsc`Gv>sLso*0F$GPq5@J*G%m{MEJ1;C$1 zA?mwmNMLVLi$Acx#EcB$yJ#O_QJ~)BNA|%}9e0$#|CyMW_QPKtzldDc10o`J^|uOb zn8|aV)oSusv%DWH^1J)l0c*0%&(E_J*dke>sXcgLY*V13sT6UbgGN-8#llY`wik~Lww(3(W2bVk{62@car47 zl6PQCT~|Z4x;scG&n}*>{K7(_chi3G@$V+CPu@MQciA`EcH!^RT+RJHC5fYk)?FaL z2HLpB};K^Y@TX1{i?Y>mQKc*UIQVXP~@y?;Z~QI{Kmk z=e}n)n24t3>rVB*(YBKImbH^G|BJVb>8MgzJIOOqJ45!XW&XrgzXM&9^M~D}IP!bC zc$fH?0%%jzmxCBSAboDeCbeyurc#>g)2p7Y4-_b1{=D04W0Bm!yNxJX{vSyH^Jo8W dzp31yaG$xqPmN)`pM?J@DO^00NjARwzX10w8{7Z@ literal 0 HcmV?d00001 From 56d88e956615acc9f6545ab56497ce8244afb96f Mon Sep 17 00:00:00 2001 From: Yonathan Randolph Date: Thu, 12 Jan 2017 14:08:10 -0800 Subject: [PATCH 14/68] Fix code formatting --- docs/admin/static-pods.md | 56 +++++++++++++++++++-------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 4766002917..74aee8f926 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -22,45 +22,45 @@ For example, this is how to start a simple web server as a static pod: 1. Choose a node where we want to run the static pod. In this example, it's `my-node1`. - ```shell - [joe@host ~] $ ssh my-node1 - ``` +```shell +[joe@host ~] $ ssh my-node1 +``` 2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubernetes.d/static-web.yaml`: - ```shell - [root@my-node1 ~] $ mkdir /etc/kubernetes.d/ - [root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml - apiVersion: v1 - kind: Pod - metadata: - name: static-web - labels: - role: myrole - spec: - containers: +```shell +[root@my-node1 ~] $ mkdir /etc/kubernetes.d/ +[root@my-node1 ~] $ cat </etc/kubernetes.d/static-web.yaml +apiVersion: v1 +kind: Pod +metadata: + name: static-web + labels: + role: myrole +spec: + containers: + - name: web + image: nginx + ports: - name: web - image: nginx - ports: - - name: web - containerPort: 80 - protocol: tcp - EOF - ``` + containerPort: 80 + protocol: tcp +EOF +``` 2. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument. On Fedora edit `/etc/kubernetes/kubelet` to include this line: - ```conf - KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" - ``` +```conf +KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" +``` - Instructions for other distributions or Kubernetes installations may vary. +Instructions for other distributions or Kubernetes installations may vary. 3. Restart kubelet. On Fedora, this is: - ```shell - [root@my-node1 ~] $ systemctl restart kubelet - ``` +```shell +[root@my-node1 ~] $ systemctl restart kubelet +``` ## Pods created via HTTP From b9f53d7991169226ac0bf6353aadf9ca6a9f66c1 Mon Sep 17 00:00:00 2001 From: Yonathan Randolph Date: Thu, 12 Jan 2017 15:01:14 -0800 Subject: [PATCH 15/68] Fix typo tcp -> TCP --- docs/admin/static-pods.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/static-pods.md b/docs/admin/static-pods.md index 74aee8f926..8c9e482d0d 100644 --- a/docs/admin/static-pods.md +++ b/docs/admin/static-pods.md @@ -44,7 +44,7 @@ spec: ports: - name: web containerPort: 80 - protocol: tcp + protocol: TCP EOF ``` From 64f86459e0a80d40ebec31609b246961cd3111f8 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Wed, 11 Jan 2017 10:16:44 -0500 Subject: [PATCH 16/68] Fix broken link. --- docs/getting-started-guides/ubuntu/networking.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting-started-guides/ubuntu/networking.md b/docs/getting-started-guides/ubuntu/networking.md index 7b4d3bcb28..54ca15c6ad 100644 --- a/docs/getting-started-guides/ubuntu/networking.md +++ b/docs/getting-started-guides/ubuntu/networking.md @@ -9,7 +9,7 @@ This page shows how to the various network portions of a cluster work, and how t This page assumes you have a working Juju deployed cluster. {% endcapture %} -Kubernetes supports the [Container Network Interface (CNI)]](https://github.com/containernetworking/cni). +Kubernetes supports the [Container Network Interface (CNI)](https://github.com/containernetworking/cni). This is a network plugin architecture that allows you to use whatever Kubernetes-friendly SDN you want. Currently this means support for Flannel. @@ -50,4 +50,4 @@ how large your cluster will potentially scale. Class A IP ranges with /24 are a good option. {% endcapture %} -{% include templates/task.md %} \ No newline at end of file +{% include templates/task.md %} From 5ac4ac6eba08090abb191e8093582a79302d7215 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Fri, 13 Jan 2017 10:40:12 -0500 Subject: [PATCH 17/68] Add conjure-up instructions. --- docs/getting-started-guides/ubuntu/index.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/index.md b/docs/getting-started-guides/ubuntu/index.md index 7211cc0e92..5e706f0add 100644 --- a/docs/getting-started-guides/ubuntu/index.md +++ b/docs/getting-started-guides/ubuntu/index.md @@ -13,7 +13,23 @@ There are multiple ways to run a Kubernetes cluster with Ubuntu. These pages exp Supports AWS, GCE, Azure, Joyent, OpenStack, Bare Metal and local workstation deployment. -### Operational Guides +### Quick Start + +[conjure-up](http://conjure-up.io/) provides quick wasy to deploy Kubernetes on multiple clouds and bare metal. It provides a user-friendly UI that prompts you for cloud credentials and configuration options: + +Available for Ubuntu 16.04 and newer: + +``` +sudo apt-add-repository ppa:juju/stable +sudo apt-add-repository ppa:conjure-up/next +sudo apt update +sudo apt install conjure-up +conjure-up +``` + +### Operational Guides + +These are more in-depth guides for users choosing to run Kubernetes in production: - [Installation](/docs/getting-started-guides/ubuntu/installation) - [Validation](/docs/getting-started-guides/ubuntu/validation) From 3d39da4670a9b00bb206e8cafcbaa1717cdcc772 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Wed, 18 Jan 2017 15:12:37 -0500 Subject: [PATCH 18/68] Add github.com as a network dependency. --- docs/getting-started-guides/ubuntu/installation.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/getting-started-guides/ubuntu/installation.md b/docs/getting-started-guides/ubuntu/installation.md index b8f892dc46..cc93995144 100644 --- a/docs/getting-started-guides/ubuntu/installation.md +++ b/docs/getting-started-guides/ubuntu/installation.md @@ -38,7 +38,8 @@ so we can find them. - OpenStack deployments are currently only tested on Icehouse and newer. - Network access to the following domains - *.jujucharms.com - - gcr.io + - gcr.io + - github.com - Access to an Ubuntu mirror (public or private) From e2a732f4ebfdac2ff4406d48dd47ec70ab4c2a0c Mon Sep 17 00:00:00 2001 From: xilabao Date: Wed, 11 Jan 2017 23:14:36 -0600 Subject: [PATCH 19/68] Update groups to passwordfile ref to https://github.com/kubernetes/kubernetes/pull/39651 --- docs/admin/authentication.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/admin/authentication.md b/docs/admin/authentication.md index 1cfa8c5f13..a7cca01126 100644 --- a/docs/admin/authentication.md +++ b/docs/admin/authentication.md @@ -116,10 +116,11 @@ authentication is currently supported for convenience while we finish making the more secure modes described above easier to use. The basic auth file format is implemented in `plugin/pkg/auth/authenticator/password/passwordfile/...` -and is a csv file with 3 columns: password, user name, user id. +and is a csv file with a minimum of 3 columns: password, user name, user id, followed by +optional group names. Note, if you have more than one group the column must be double quoted e.g. ```conf -password,user,uid +password,user,uid,"group1,group2,group3" ``` When using basic authentication from an http client, the API server expects an `Authorization` header From d09498f7c1c9453b4c46c63b3c8c7d77f79d55d5 Mon Sep 17 00:00:00 2001 From: Bogdan Dobrelya Date: Tue, 10 Jan 2017 14:54:54 +0100 Subject: [PATCH 20/68] Add docs for Kargo installer for Kubernetes Add documentation for Getting Started guides. Kargo is an incubated Kubernetes project https://github.com/kubernetes-incubator/kargo. Signed-off-by: Bogdan Dobrelya --- _data/guides.yml | 1 + _data/tasks.yml | 2 +- docs/getting-started-guides/kargo.md | 104 +++++++++++++++++++++++++++ docs/index.md | 1 + 4 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 docs/getting-started-guides/kargo.md diff --git a/_data/guides.yml b/_data/guides.yml index fdade5b1ec..f66d0706ee 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -8,6 +8,7 @@ toc: - docs/whatisk8s.md - docs/getting-started-guides/kubeadm.md - docs/getting-started-guides/kops.md + - docs/getting-started-guides/kargo.md - docs/hellonode.md - docs/getting-started-guides/kubectl.md - docs/getting-started-guides/binary_release.md diff --git a/_data/tasks.yml b/_data/tasks.yml index 13315bf5ed..231a583524 100644 --- a/_data/tasks.yml +++ b/_data/tasks.yml @@ -1,5 +1,5 @@ bigheader: "Tasks" -abstract: "Step-by-step instructions for performing operations with Kuberentes." +abstract: "Step-by-step instructions for performing operations with Kubernetes." toc: - docs/tasks/index.md diff --git a/docs/getting-started-guides/kargo.md b/docs/getting-started-guides/kargo.md new file mode 100644 index 0000000000..93f1373c75 --- /dev/null +++ b/docs/getting-started-guides/kargo.md @@ -0,0 +1,104 @@ +--- +title: Installing Kubernetes On-premise/Cloud Providers with Kargo +--- + + + +## Overview + +This quickstart helps to install a Kubernetes cluster hosted +on GCE, Azure, OpenStack, AWS or Baremetal with +[`Kargo`](https://github.com/kubernetes-incubator/kargo) tool. + +Kargo is a composition of [Ansible](http://docs.ansible.com/) playbooks, +[inventory](https://github.com/kubernetes-incubator/kargo/blob/master/docs/ansible.md) +generation CLI tools and domain knowledge for generic OS/Kubernetes +clusters configuration management tasks. It provides: + +* [High available cluster](https://github.com/kubernetes-incubator/kargo/blob/master/docs/ha-mode.md) +* [Composable](https://github.com/kubernetes-incubator/kargo/blob/master/docs/vars.md) + (Choice of the network plugin, for instance) +* Support most popular Linux + [distributions](https://github.com/kubernetes-incubator/kargo#supported-linux-distributions) +* Continuous integration tests + +To choose a tool which fits your use case the best, you may want to read this +[comparison](https://github.com/kubernetes-incubator/kargo/blob/master/docs/comparisons.md) +to [kubeadm](../kubeadm) and [kops](../kops). + +## Creating a cluster + +### (1/4) Ensure the underlay [requirements](https://github.com/kubernetes-incubator/kargo#requirements) are met + +#### Checklist + +* You must have cloud instances or baremetal nodes running for your future Kubernetes cluster. + A way to achieve that is to use the + [kargo-cli tool](https://github.com/kubernetes-incubator/kargo/blob/master/docs/getting-started.md). +* Or provision baremetal hosts with a tool-of-your-choice or launch cloud instances, + then create an inventory file for Ansible with this [tool](https://github.com/kubernetes-incubator/kargo/blob/master/contrib/inventory_generator/inventory_generator.py). + +### (2/4) Compose the deployment + +#### Checklist + +* Customize your deployment by usual Ansible meanings, which is + [generating inventory](https://github.com/kubernetes-incubator/kargo/blob/master/docs/getting-started.md#building-your-own-inventory) + and overriding default data [variables](https://github.com/kubernetes-incubator/kargo/blob/master/docs/vars.md). + Or just stick with default values (Kargo will choose Flannel networking plugin for you + then). This includes steps like deciding on the: + * DNS [configuration options](https://github.com/kubernetes-incubator/kargo/blob/master/docs/dns-stack.md) + * [Networking plugin](https://github.com/kubernetes-incubator/kargo#network-plugins) to use + * [Versions](https://github.com/kubernetes-incubator/kargo#versions-of-supported-components) + of components. + * Additional node groups like [bastion hosts](https://github.com/kubernetes-incubator/kargo/blob/master/docs/ansible.md#bastion-host) or + [Calico BGP route reflectors](https://github.com/kubernetes-incubator/kargo/blob/master/docs/calico.md#optional--bgp-peering-with-border-routers). +* Plan custom deployment steps, if any, or use the default composition layer in the + [cluster definition file](https://github.com/kubernetes-incubator/kargo/blob/master/cluster.yml). + Taking the best from Ansible world, Kargo allows users to execute arbitrary steps via the + ``ansible-playbook`` with given inventory, playbooks, data overrides and tags, limits, batches + of nodes to deploy and so on. +* For large deployments (100+ nodes), you may want to + [tweak things](https://github.com/kubernetes-incubator/kargo/blob/master/docs/large-deployments.md) + for best results. + +### (3/4) Run the deployment + +#### Checklist + +* Apply deployment with + [kargo-cli tool](https://github.com/kubernetes-incubator/kargo/blob/master/docs/getting-started.md) + or ``ansible-playbook`` + [manual commands](https://github.com/kubernetes-incubator/kargo/blob/master/docs/getting-started.md#starting-custom-deployment). + +### (4/4) (Optional) verify inter-pods connectivity and DNS resolve with [Netchecker](https://github.com/kubernetes-incubator/kargo/blob/master/docs/netcheck.md) + +#### Checklist + +* Enusre the netchecker-agent's pods can resolve DNS requests and ping each over within the default namespace. + Those pods mimic similar behavior of the rest of the workloads and serve as cluster health indicators. + +## Explore contributed add-ons + +See the [list of contributed playbooks](https://github.com/kubernetes-incubator/kargo/tree/master/contrib) +to explore other deployment options. + +## What's next + +Kargo has quite a few [marks on the radar](https://github.com/kubernetes-incubator/kargo/blob/master/docs/roadmap.md). + +## Cleanup + +To delete your scratch cluster, you can apply the +[reset role](https://github.com/kubernetes-incubator/kargo/blob/master/roles/reset/tasks/main.yml) +with the manual ``ansible-playbook`` command. + +Note, that it is highly unrecommended to delete production clusters with the reset playbook! + +## Feedback + +* Slack Channel: [#kargo](https://kubernetes.slack.com/messages/kargo/) +* [GitHub Issues](https://github.com/kubernetes-incubator/kargo/issues) + diff --git a/docs/index.md b/docs/index.md index 24814e55ae..1201403936 100644 --- a/docs/index.md +++ b/docs/index.md @@ -20,6 +20,7 @@ title: Kubernetes Documentation
  • Minikube: Install a single-node Kubernetes cluster on your local machine for development and testing.
  • Installing Kubernetes on AWS with kops: Bring up a complete Kubernetes cluster on Amazon Web Services, using a tool called kops.
  • Installing Kubernetes on Linux with kubeadm (Alpha): Install a secure Kubernetes cluster on any pre-existing machines running Linux, using the built-in kubeadm tool.
    • +
  • Installing Kubernetes On-premise/Cloud Providers with Kargo: Deploy a Kubernetes cluster on-premise baremetal or hosted on cloud providers, with Ansible and kargo tools.

    • Guides, Tutorials, Tasks, and Concepts

    From b1dfb93e1b39b4719330cdf3162a9a6d4d0a3887 Mon Sep 17 00:00:00 2001 From: luks Date: Mon, 9 Jan 2017 23:18:17 -0200 Subject: [PATCH 21/68] minor fix and explain cluster-cidr/allocate-node-cidr in the controller-manager section --- docs/getting-started-guides/scratch.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/getting-started-guides/scratch.md b/docs/getting-started-guides/scratch.md index 9b53e28e78..f95f132be6 100644 --- a/docs/getting-started-guides/scratch.md +++ b/docs/getting-started-guides/scratch.md @@ -405,7 +405,7 @@ Arguments to consider: - `--cluster-domain=` to the dns domain prefix to use for cluster DNS addresses. - `--docker-root=` - `--root-dir=` - - `--configure-cbr0=` (described above) + - `--configure-cbr0=` (described below) - `--register-node` (described in [Node](/docs/admin/node) documentation.) ### kube-proxy @@ -774,10 +774,8 @@ Template for controller manager pod: Flags to consider using with controller manager: - `--cluster-name=$CLUSTER_NAME` - - `--cluster-cidr=` - - *TODO*: explain this flag. - - `--allocate-node-cidrs=` - - *TODO*: explain when you want controller to do this and when you want to do it another way. + - `--cluster-cidr=`, the CIDR range for pods in cluster. + - `--allocate-node-cidrs=`, if you are using `--cloud-provider=`, allocate and set the CIDRs for pods on the cloud provider. - `--cloud-provider=` and `--cloud-config` as described in apiserver section. - `--service-account-private-key-file=/srv/kubernetes/server.key`, used by the [service account](/docs/user-guide/service-accounts) feature. - `--master=127.0.0.1:8080` From 5e77cd1825c652c63fa5b3891cd1a7860ad49c03 Mon Sep 17 00:00:00 2001 From: Justin Nauman Date: Wed, 28 Dec 2016 14:54:52 -0600 Subject: [PATCH 22/68] Updates on the PV zone affinity documentation Updates docs around the PV distribution process per the linked issues --- docs/admin/multiple-zones.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/admin/multiple-zones.md b/docs/admin/multiple-zones.md index e215b31716..b6421e71c2 100644 --- a/docs/admin/multiple-zones.md +++ b/docs/admin/multiple-zones.md @@ -51,7 +51,7 @@ admission controller automatically adds zone labels to them. The scheduler (via `VolumeZonePredicate` predicate) will then ensure that pods that claim a given volume are only placed into the same zone as that volume, as volumes cannot be attached across zones. - + ## Limitations There are some important limitations of the multizone support: @@ -158,8 +158,7 @@ kubernetes-minion-wf8i Ready 2m beta.kubernetes.io ### Volume affinity -Create a volume (only PersistentVolumes are supported for zone -affinity), using the new dynamic volume creation: +Create a volume using the dynamic volume creation (only PersistentVolumes are supported for zone affinity): ```json kubectl create -f - < kubectl get pv --show-labels From d7ce58eacca7fccdebfd5c9f7d29dfa32d54c8f0 Mon Sep 17 00:00:00 2001 From: sallydeng Date: Mon, 23 Jan 2017 16:47:37 +0800 Subject: [PATCH 23/68] Update ui.md I installed gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0. When I deploy a containerized app through dashboad ui, the appname can be "dxltest1" . The "dxltest1" can be deployed successfully. so the appname can be end with a number. --- docs/user-guide/ui.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/ui.md b/docs/user-guide/ui.md index 818573e0da..da2b2e611c 100644 --- a/docs/user-guide/ui.md +++ b/docs/user-guide/ui.md @@ -66,7 +66,7 @@ The deploy wizard expects that you provide the following information: - **App name** (mandatory): Name for your application. A [label](/docs/user-guide/labels/) with the name will be added to the Deployment and Service, if any, that will be deployed. - The application name must be unique within the selected Kubernetes [namespace](/docs/admin/namespaces/). It must start and end with a lowercase character, and contain only lowercase letters, numbers and dashes (-). It is limited to 24 characters. Leading and trailing spaces are ignored. + The application name must be unique within the selected Kubernetes [namespace](/docs/admin/namespaces/). It must start with a lowercase character, and end with a lowercase character or a number, and contain only lowercase letters, numbers and dashes (-). It is limited to 24 characters. Leading and trailing spaces are ignored. - **Container image** (mandatory): The URL of a public Docker [container image](/docs/user-guide/images/) on any registry, or a private image (commonly hosted on the Google Container Registry or Docker Hub). The container image specification must end with a colon. From 4bbd9ba6ce31884d692abd54316b178f205f1abe Mon Sep 17 00:00:00 2001 From: scjane Date: Fri, 20 Jan 2017 16:22:22 +0800 Subject: [PATCH 24/68] Update index.md --- docs/user-guide/update-demo/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/user-guide/update-demo/index.md b/docs/user-guide/update-demo/index.md index 7f0952348c..f2e3f8a0f2 100644 --- a/docs/user-guide/update-demo/index.md +++ b/docs/user-guide/update-demo/index.md @@ -1,9 +1,9 @@ ---- -assignees: -- mikedanese -title: Rolling Update Demo ---- - +--- +assignees: +- mikedanese +title: Rolling Update Demo +--- + This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/user-guide/managing-deployments/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information. The files for this example are viewable in [our docs repo @@ -67,7 +67,7 @@ The rolling-update command in kubectl will do 2 things: Watch the [demo website](http://localhost:8001/static/index.html), it will update one pod every 10 seconds until all of the pods have the new image. Note that the new replication controller definition does not include the replica count, so the current replica count of the old replication controller is preserved. -But if the replica count had been specified, the final replica count of the new replication controller will be equal this number. +But if the replica count had been specified, the final replica count of the new replication controller will be equal to this number. ### Step Five: Bring down the pods From 0fd32489a85bcfd77639c54bddf18609dbbafe39 Mon Sep 17 00:00:00 2001 From: scjane Date: Tue, 24 Jan 2017 16:24:57 +0800 Subject: [PATCH 25/68] Update ha-master-gce.md --- docs/admin/ha-master-gce.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/ha-master-gce.md b/docs/admin/ha-master-gce.md index 91e0019272..3eb2918655 100644 --- a/docs/admin/ha-master-gce.md +++ b/docs/admin/ha-master-gce.md @@ -84,7 +84,7 @@ The following sample commands demonstrate this process: $ KUBE_DELETE_NODES=false KUBE_GCE_ZONE=replica_zone KUBE_REPLICA_NAME=replica_name ./cluster/kube-down.sh ``` -2. Add a new replica in place of the old one: +
    1. Add a new replica in place of the old one:
    ```shell $ KUBE_GCE_ZONE=replica-zone KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh @@ -102,7 +102,7 @@ A two-replica cluster is thus inferior, in terms of HA, to a single replica clus * When you add a master replica, cluster state (etcd) is copied to a new instance. If the cluster is large, it may take a long time to duplicate its state. -This operation may be speed up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration) here +This operation may be speed up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration) (we are considering adding support for etcd data dir migration in future). ## Implementation notes From f99d34ff1ab2afb9c6cb1f129ce07c7d7d11c329 Mon Sep 17 00:00:00 2001 From: Cemalettin Koc Date: Wed, 18 Jan 2017 19:32:34 +0200 Subject: [PATCH 26/68] Align kubeconfig resolution with the rest of documentation It seems that this part has a missing documentation. Here is the reference of `kubectl` configuration. https://kubernetes.io/docs/user-guide/kubectl/kubectl_config/ --- docs/user-guide/sharing-clusters.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/user-guide/sharing-clusters.md b/docs/user-guide/sharing-clusters.md index c21acca84a..4faf7d9826 100644 --- a/docs/user-guide/sharing-clusters.md +++ b/docs/user-guide/sharing-clusters.md @@ -105,8 +105,7 @@ and/or run `kubectl config -h`. 1. `--kubeconfig=/path/to/.kube/config` command line flag 2. `KUBECONFIG=/path/to/.kube/config` env variable -3. `$PWD/.kube/config` -4. `$HOME/.kube/config` +3. `$HOME/.kube/config` If you create clusters A, B on host1, and clusters C, D on host2, you can make all four clusters available on both hosts by running From d890d8f97a653562ea0094e9c4a4da3085d36668 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Tue, 24 Jan 2017 15:40:00 -0800 Subject: [PATCH 27/68] Fixed typo changed "speed" to "sped" for proper tense --- docs/admin/ha-master-gce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/ha-master-gce.md b/docs/admin/ha-master-gce.md index 3eb2918655..025035cca7 100644 --- a/docs/admin/ha-master-gce.md +++ b/docs/admin/ha-master-gce.md @@ -102,7 +102,7 @@ A two-replica cluster is thus inferior, in terms of HA, to a single replica clus * When you add a master replica, cluster state (etcd) is copied to a new instance. If the cluster is large, it may take a long time to duplicate its state. -This operation may be speed up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration) +This operation may be sped up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration) (we are considering adding support for etcd data dir migration in future). ## Implementation notes From 7abcc6c8540a7b0a4f6712e77e0f5cb134271bed Mon Sep 17 00:00:00 2001 From: Michael Fraenkel Date: Tue, 24 Jan 2017 06:31:10 -0700 Subject: [PATCH 28/68] Document new optional support for ConfigMap and Secret Corresponds to proposal https://github.com/kubernetes/community/pull/175 --- docs/user-guide/configmap/index.md | 70 ++++++++++++++++++++++++++-- docs/user-guide/secrets/index.md | 74 +++++++++++++++++++++++++++--- 2 files changed, 134 insertions(+), 10 deletions(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index d630ed2aa8..2e80ec8d9b 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -291,6 +291,37 @@ SPECIAL_LEVEL_KEY=very SPECIAL_TYPE_KEY=charm ``` +#### Optional ConfigMap in environment variables + +There might be situations where environment variables are not +always required. These environment variables can be marked as optional in a +pod like so: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: dapi-test-pod +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox + command: [ "/bin/sh", "-c", "env" ] + env: + - name: SPECIAL_LEVEL_KEY + valueFrom: + configMapKeyRef: + name: a-config + key: akey + optional: true + restartPolicy: Never +``` + +When this pod is run, its output will include the lines: + +```shell +``` + ### Use-Case: Set command-line arguments with ConfigMap ConfigMaps can also be used to set the value of the command or arguments in a container. This is @@ -422,6 +453,38 @@ very You can project keys to specific paths and specific permissions on a per-file basis. The [Secrets](/docs/user-guide/secrets/) user guide explains the syntax. +#### Optional ConfigMap via volume plugin + +Volumes and files provided by a ConfigMap can be also be marked as optional. +The ConfigMap or the key specified does not have to exist. The mount path for +such items will always be created. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: dapi-test-pod +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox + command: [ "/bin/sh", "-c", "ls /etc/config" ] + volumeMounts: + - name: config-volume + mountPath: /etc/config + volumes: + - name: config-volume + configMap: + name: no-config + optional: true + restartPolicy: Never +``` + +When this pod is run, the output will be: + +```shell +``` + ## Real World Example: Configuring Redis Let's take a look at a real-world example: configuring redis using ConfigMap. Say we want to inject @@ -517,9 +580,10 @@ $ kubectl exec -it redis redis-cli ## Restrictions -ConfigMaps must be created before they are consumed in pods. Controllers may be written to tolerate -missing configuration data; consult individual components configured via ConfigMap on a case-by-case -basis. +ConfigMaps must be created before they are consumed in pods unless they are +marked as optional. Controllers may be written to tolerate missing +configuration data; consult individual components configured via ConfigMap on +a case-by-case basis. ConfigMaps reside in a namespace. They can only be referenced by pods in the same namespace. diff --git a/docs/user-guide/secrets/index.md b/docs/user-guide/secrets/index.md index 6942b52ff3..7da0f446fd 100644 --- a/docs/user-guide/secrets/index.md +++ b/docs/user-guide/secrets/index.md @@ -372,6 +372,41 @@ files. When a secret being already consumed in a volume is updated, projected keys are eventually updated as well. The update time depends on the kubelet syncing period. +#### Optional Secrets as Files from a Pod + +Volumes and files provided by a Secret can be also be marked as optional. +The Secret or the key within a Secret does not have to exist. The mount path for +such items will always be created. + +```json +{ + "apiVersion": "v1", + "kind": "Pod", + "metadata": { + "name": "mypod", + "namespace": "myns" + }, + "spec": { + "containers": [{ + "name": "mypod", + "image": "redis", + "volumeMounts": [{ + "name": "foo", + "mountPath": "/etc/foo" + }] + }], + "volumes": [{ + "name": "foo", + "secret": { + "secretName": "mysecret", + "defaultMode": 256, + "optional": true + } + }] + } +} +``` + #### Using Secrets as Environment Variables To use a secret in an environment variable in a pod: @@ -418,6 +453,30 @@ $ echo $SECRET_PASSWORD 1f2d1e2e67df ``` +#### Optional Secrets from Environment Variables + +You may not want to require all your secrets to exist. They can be marked as +optional as shown in the pod: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: optional-secret-env-pod +spec: + containers: + - name: mycontainer + image: redis + env: + - name: OPTIONAL_SECRET + valueFrom: + secretKeyRef: + name: mysecret + key: username + optional: true + restartPolicy: Never +``` + #### Using imagePullSecrets An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry @@ -449,7 +508,8 @@ can be automatically attached to pods based on their service account. Secret volume sources are validated to ensure that the specified object reference actually points to an object of type `Secret`. Therefore, a secret -needs to be created before any pods that depend on it. +needs to be created before any pods that depend on it, unless it is marked as +optional. Secret API objects reside in a namespace. They can only be referenced by pods in that same namespace. @@ -469,12 +529,12 @@ not common ways to create pods.) When a pod is created via the API, there is no check whether a referenced secret exists. Once a pod is scheduled, the kubelet will try to fetch the -secret value. If the secret cannot be fetched because it does not exist or -because of a temporary lack of connection to the API server, kubelet will -periodically retry. It will report an event about the pod explaining the -reason it is not started yet. Once the secret is fetched, the kubelet will -create and mount a volume containing it. None of the pod's containers will -start until all the pod's volumes are mounted. +secret value. If a required secret cannot be fetched because it does not +exist or because of a temporary lack of connection to the API server, the +kubelet will periodically retry. It will report an event about the pod +explaining the reason it is not started yet. Once the secret is fetched, the +kubelet will create and mount a volume containing it. None of the pod's +containers will start until all the pod's volumes are mounted. ## Use cases From 324451c3378b4ba01d27e3db6117f86d0517a598 Mon Sep 17 00:00:00 2001 From: Michael Fraenkel Date: Tue, 24 Jan 2017 16:54:22 -0700 Subject: [PATCH 29/68] mend --- docs/user-guide/configmap/index.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/user-guide/configmap/index.md b/docs/user-guide/configmap/index.md index 2e80ec8d9b..3227e416c0 100644 --- a/docs/user-guide/configmap/index.md +++ b/docs/user-guide/configmap/index.md @@ -317,10 +317,7 @@ spec: restartPolicy: Never ``` -When this pod is run, its output will include the lines: - -```shell -``` +When this pod is run, the output will be empty. ### Use-Case: Set command-line arguments with ConfigMap From 220352d117c4bf774d7e2f2d766979fd077ba361 Mon Sep 17 00:00:00 2001 From: Scott Collier Date: Thu, 12 Jan 2017 17:11:24 -0600 Subject: [PATCH 30/68] removing word that doesn't flow correctly --- docs/admin/federation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/federation/index.md b/docs/admin/federation/index.md index 59a0199d7b..b9cd397514 100644 --- a/docs/admin/federation/index.md +++ b/docs/admin/federation/index.md @@ -249,7 +249,7 @@ http://kubernetes.io/docs/user-guide/configmap/. ### Kubernetes 1.4 and earlier: Setting federations flag on kube-dns-rc -If your cluster is running Kubernetes version 1.4 or earlier, you must to restart +If your cluster is running Kubernetes version 1.4 or earlier, you must restart KubeDNS and pass it a `--federations` flag, which tells it about valid federation DNS hostnames. The flag uses the following format: From f331ebe332f0be66dd6d3f63ab78c504c6e15b44 Mon Sep 17 00:00:00 2001 From: divyenpatel Date: Mon, 16 Jan 2017 12:14:57 -0800 Subject: [PATCH 31/68] Updated known issues in docs/getting-started-guides/vsphere.md --- docs/getting-started-guides/vsphere.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/vsphere.md b/docs/getting-started-guides/vsphere.md index 36372144c6..ba07d14890 100644 --- a/docs/getting-started-guides/vsphere.md +++ b/docs/getting-started-guides/vsphere.md @@ -79,7 +79,7 @@ Sample Config: #### Known issues * [Volumes are not removed from a VM configuration if the VM is down](https://github.com/kubernetes/kubernetes/issues/33061). The workaround is to manually remove the disk from VM settings before powering it up. -* [FS groups are not supported in 1.4.7](https://github.com/kubernetes/kubernetes/issues/34039) +* [FS groups are not supported in 1.4.7](https://github.com/kubernetes/kubernetes/issues/34039) - This issue is fixed in 1.4.8 ### Kube-up (Deprecated) From 6afde69458e2c6878250fdcb9d564dfb053188df Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Wed, 11 Jan 2017 16:35:00 -0800 Subject: [PATCH 32/68] fixed broken link re: issue #1687 Updated link to go to https://github.com/kubernetes/kubernetes/tree/master/examples/guestbook which is a redis example mentioned in https://kubernetes.io/docs/samples/ --- docs/user-guide/jobs/work-queue-2/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/jobs/work-queue-2/index.md b/docs/user-guide/jobs/work-queue-2/index.md index 98d327d9a9..621be7f795 100644 --- a/docs/user-guide/jobs/work-queue-2/index.md +++ b/docs/user-guide/jobs/work-queue-2/index.md @@ -31,7 +31,7 @@ Here is an overview of the steps in this example: ## Starting Redis For this example, for simplicitly, we will start a single instance of Redis. -See the [Redis Example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/redis/README.md) for an example +See the [Redis Example](https://github.com/kubernetes/kubernetes/tree/master/examples/guestbook) for an example of deploying Redis scalably and redundantly. Start a temporary Pod running Redis and a service so we can find it. From e93bb9dc275120dfbe0b565f742ca55eec020a7f Mon Sep 17 00:00:00 2001 From: Hugues Malphettes Date: Wed, 25 Jan 2017 07:05:09 +0800 Subject: [PATCH 33/68] Format JSON: missing comma --- docs/user-guide/pods/pod-spec-common.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/pods/pod-spec-common.json b/docs/user-guide/pods/pod-spec-common.json index c449c34de5..8621593d07 100644 --- a/docs/user-guide/pods/pod-spec-common.json +++ b/docs/user-guide/pods/pod-spec-common.json @@ -24,7 +24,7 @@ } ], "resources": { - "cpu": "" + "cpu": "", "memory": "" } } From 0e1670ccb827891e451c224a1f670fa0adb0cb8a Mon Sep 17 00:00:00 2001 From: Scott Collier Date: Thu, 12 Jan 2017 17:04:08 -0600 Subject: [PATCH 34/68] looks like a couple of words got swapped when describing the ConfigMap for kube-dns, just fixing that up --- docs/admin/federation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/federation/index.md b/docs/admin/federation/index.md index b9cd397514..bb15fe09ff 100644 --- a/docs/admin/federation/index.md +++ b/docs/admin/federation/index.md @@ -236,7 +236,7 @@ metadata: name: kube-dns namespace: kube-system data: - federations: = + federations: = ``` where `` should be replaced by the name you want to give to your From 27d0048498cbef12631e35d2608b0a25349a54e6 Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 25 Jan 2017 17:25:31 +0800 Subject: [PATCH 35/68] Update master-node-communication.md --- docs/admin/master-node-communication.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/master-node-communication.md b/docs/admin/master-node-communication.md index 91ecff7ef9..d654fd5173 100644 --- a/docs/admin/master-node-communication.md +++ b/docs/admin/master-node-communication.md @@ -91,7 +91,7 @@ HTTP connections and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will not validate the certificate provided by the HTTPS endpoint nor provide client credentials so while the -connection will by encrypted, it will not provide any guarantees of integrity. +connection will be encrypted, it will not provide any guarantees of integrity. These connections **are not currently safe** to run over untrusted and/or public networks. From b10426383e8ed88ce8173980fd2c011faf6f5de9 Mon Sep 17 00:00:00 2001 From: Bogdan Dobrelya Date: Wed, 25 Jan 2017 10:24:59 +0100 Subject: [PATCH 36/68] Fix the tool link for Kargo docs Signed-off-by: Bogdan Dobrelya --- docs/getting-started-guides/kargo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/kargo.md b/docs/getting-started-guides/kargo.md index 93f1373c75..0a2e1d1c44 100644 --- a/docs/getting-started-guides/kargo.md +++ b/docs/getting-started-guides/kargo.md @@ -38,7 +38,7 @@ to [kubeadm](../kubeadm) and [kops](../kops). A way to achieve that is to use the [kargo-cli tool](https://github.com/kubernetes-incubator/kargo/blob/master/docs/getting-started.md). * Or provision baremetal hosts with a tool-of-your-choice or launch cloud instances, - then create an inventory file for Ansible with this [tool](https://github.com/kubernetes-incubator/kargo/blob/master/contrib/inventory_generator/inventory_generator.py). + then create an inventory file for Ansible with this [tool](https://github.com/kubernetes-incubator/kargo/blob/master/contrib/inventory_builder/inventory.py). ### (2/4) Compose the deployment From 94e3c350f538224229c7b573426332415a42ad72 Mon Sep 17 00:00:00 2001 From: Bilgin Ibryam Date: Wed, 25 Jan 2017 09:42:02 +0100 Subject: [PATCH 37/68] Typo fixed --- docs/user-guide/node-selection/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/node-selection/index.md b/docs/user-guide/node-selection/index.md index b2f13bd5ae..0f45c1f809 100644 --- a/docs/user-guide/node-selection/index.md +++ b/docs/user-guide/node-selection/index.md @@ -91,7 +91,7 @@ rather than against labels on the node itself, which allows rules about which po The affinity feature consists of two types of affinity, "node affinity" and "inter-pod affinity/anti-affinity." Node affinity is like the existing `nodeSelector` (but with the first two benefits listed above), while inter-pod affinity/anti-affinity constrains against pod labels rather than node labels, as -described in the three item listed above, in addition to having the first and second properties listed above. +described in the third item listed above, in addition to having the first and second properties listed above. `nodeSelector` continues to work as usual, but will eventually be deprecated, as node affinity can express everything that `nodeSelector` can express. From 8521bdc2b831c00b2c97a65cc0b537f75a150f80 Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 25 Jan 2017 16:00:22 +0800 Subject: [PATCH 38/68] Update dns.md --- docs/admin/dns.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/dns.md b/docs/admin/dns.md index 4ac5bf0ddb..7f1c3c4641 100644 --- a/docs/admin/dns.md +++ b/docs/admin/dns.md @@ -69,7 +69,7 @@ is no longer supported. When enabled, pods are assigned a DNS A record in the form of `pod-ip-address.my-namespace.pod.cluster.local`. -For example, a pod with ip `1.2.3.4` in the namespace `default` with a DNS name of `cluster.local` would have an entry: `1-2-3-4.default.pod.cluster.local`. +For example, a pod with IP `1.2.3.4` in the namespace `default` with a DNS name of `cluster.local` would have an entry: `1-2-3-4.default.pod.cluster.local`. #### A Records and hostname based on Pod's hostname and subdomain fields @@ -308,7 +308,7 @@ If you have created the service or in the case it should be created by default b #### Are DNS endpoints exposed? -You can verify that dns endpoints are exposed by using the `kubectl get endpoints` command. +You can verify that DNS endpoints are exposed by using the `kubectl get endpoints` command. ``` kubectl get ep kube-dns --namespace=kube-system From 246a87b77cf472b7add6f4428d75f7cdb683b21d Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 25 Jan 2017 14:45:42 +0800 Subject: [PATCH 39/68] Update multiple-schedulers.md --- docs/admin/multiple-schedulers.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/admin/multiple-schedulers.md b/docs/admin/multiple-schedulers.md index 5548f32904..bf7386e492 100644 --- a/docs/admin/multiple-schedulers.md +++ b/docs/admin/multiple-schedulers.md @@ -107,6 +107,7 @@ scheduler as an annotation in that pod spec. Let's look at three examples. ```shell kubectl create -f pod1.yaml ``` + 2. Pod spec with `default-scheduler` annotation {% include code.html language="yaml" file="multiple-schedulers/pod2.yaml" ghlink="/docs/admin/multiple-schedulers/pod2.yaml" %} @@ -120,6 +121,7 @@ scheduler as an annotation in that pod spec. Let's look at three examples. ```shell kubectl create -f pod2.yaml ``` + 3. Pod spec with `my-scheduler` annotation {% include code.html language="yaml" file="multiple-schedulers/pod3.yaml" ghlink="/docs/admin/multiple-schedulers/pod3.yaml" %} From b556ee60d635725c7889f23205552d3a8af68e15 Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 25 Jan 2017 14:03:15 +0800 Subject: [PATCH 40/68] Update multi-cluster.md --- docs/admin/multi-cluster.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/multi-cluster.md b/docs/admin/multi-cluster.md index 67a2589d40..085a9afa9f 100644 --- a/docs/admin/multi-cluster.md +++ b/docs/admin/multi-cluster.md @@ -8,7 +8,7 @@ You may want to set up multiple Kubernetes clusters, both to have clusters in different regions to be nearer to your users, and to tolerate failures and/or invasive maintenance. This document describes some of the issues to consider when making a decision about doing so. -If you decide to have multiple clusters, Kubernetes provides a way to [federate them](/docs/admin/federation/) +If you decide to have multiple clusters, Kubernetes provides a way to [federate them](/docs/admin/federation/). ## Scope of a single cluster @@ -40,7 +40,7 @@ Reasons to have multiple clusters include: ## Selecting the right number of clusters The selection of the number of Kubernetes clusters may be a relatively static choice, only revisited occasionally. -By contrast, the number of nodes in a cluster and the number of pods in a service may be change frequently according to +By contrast, the number of nodes in a cluster and the number of pods in a service may change frequently according to load and growth. To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run From f5206300102ad248b954e1f02ab29ea47e88f759 Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 25 Jan 2017 11:56:55 +0800 Subject: [PATCH 41/68] Update cluster-components.md --- docs/admin/cluster-components.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/cluster-components.md b/docs/admin/cluster-components.md index 8519a199f4..a3d4f37779 100644 --- a/docs/admin/cluster-components.md +++ b/docs/admin/cluster-components.md @@ -107,7 +107,7 @@ the Kubernetes runtime environment. or via local configuration file) and: * Mounts the pod's required volumes * Downloads the pod's secrets -* Run the pod's containers via docker (or, experimentally, rkt). +* Runs the pod's containers via docker (or, experimentally, rkt). * Periodically executes any requested container liveness probes. * Reports the status of the pod back to the rest of the system, by creating a "mirror pod" if necessary. From 9ba738b6ca59f490d662c00c303cc94bc548ee44 Mon Sep 17 00:00:00 2001 From: scjane Date: Wed, 25 Jan 2017 11:24:25 +0800 Subject: [PATCH 42/68] Update rescheduler.md --- docs/admin/rescheduler.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/rescheduler.md b/docs/admin/rescheduler.md index 1d81ac63f2..9e3fc61c39 100644 --- a/docs/admin/rescheduler.md +++ b/docs/admin/rescheduler.md @@ -31,10 +31,10 @@ To avoid situation when another pod is scheduled into the space prepared for the the chosen node gets a temporary taint "CriticalAddonsOnly" before the eviction(s) (see [more details](https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md)). Each critical add-on has to tolerate it, -the other pods shouldn't tolerate the taint. The tain is removed once the add-on is successfully scheduled. +while the other pods shouldn't tolerate the taint. The taint is removed once the add-on is successfully scheduled. *Warning:* currently there is no guarantee which node is chosen and which pods are being killed -in order to schedule critical pods, so if rescheduler is enabled you pods might be occasionally +in order to schedule critical pods, so if rescheduler is enabled your pods might be occasionally killed for this purpose. ## Config From 9ec7921a3e2cf5d1a10bb8b93d8de886fe563935 Mon Sep 17 00:00:00 2001 From: Viet Hung Nguyen Date: Tue, 17 Jan 2017 12:12:48 +0700 Subject: [PATCH 43/68] Update dead link --- docs/tutorials/stateful-application/basic-stateful-set.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/tutorials/stateful-application/basic-stateful-set.md b/docs/tutorials/stateful-application/basic-stateful-set.md index 25621354af..ef15bf3607 100644 --- a/docs/tutorials/stateful-application/basic-stateful-set.md +++ b/docs/tutorials/stateful-application/basic-stateful-set.md @@ -25,6 +25,7 @@ following Kubernetes concepts. * [Cluster DNS](/docs/admin/dns/) * [Headless Services](/docs/user-guide/services/#headless-services) * [PersistentVolumes](/docs/user-guide/volumes/) +* [PersistentVolume Provisioning](http://releases.k8s.io/{{page.githubbranch}}/examples/persistent-volume-provisioning/) * [StatefulSets](/docs/concepts/abstractions/controllers/statefulsets/) * [kubectl CLI](/docs/user-guide/kubectl) From e7e7515d7388f047254796d9e85438154e1f7e0d Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Thu, 12 Jan 2017 13:08:41 +0100 Subject: [PATCH 44/68] Update API groups description --- docs/api.md | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/docs/api.md b/docs/api.md index 7c6ae95e73..b52e55a298 100644 --- a/docs/api.md +++ b/docs/api.md @@ -72,28 +72,38 @@ in more detail in the [API Changes documentation](https://github.com/kubernetes/ ## API groups -To make it easier to extend the Kubernetes API, we are in the process of implementing [*API -groups*](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/api-group.md). These are simply different interfaces to read and/or modify the -same underlying resources. The API group is specified in a REST path and in the `apiVersion` field -of a serialized object. +To make it easier to extend the Kubernetes API, we implemented [*API groups*](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-group.md). +The API group is specified in a REST path and in the `apiVersion` field of a serialized object. Currently there are several API groups in use: -1. the "core" group, which is at REST path `/api/v1` and is not specified as part of the `apiVersion` field, e.g. - `apiVersion: v1`. -1. the "extensions" group, which is at REST path `/apis/extensions/$VERSION`, and which uses - `apiVersion: extensions/$VERSION` (e.g. currently `apiVersion: extensions/v1beta1`). - This holds types which will probably move to another API group eventually. -1. the "componentconfig" and "metrics" API groups. +1. the "core" (oftentimes called "legacy", due to not having explicit group name) group, which is at + REST path `/api/v1` and is not specified as part of the `apiVersion` field, e.g. `apiVersion: v1`. +1. the named groups are at REST path `/apis/$GROUP_NAME/$VERSION`, and use `apiVersion: $GROUP_NAME/$VERSION` + (e.g. `apiVersion: batch/v1`). Full list of supported API groups can be seen in [Kubernetes API reference](/docs/reference/). -In the future we expect that there will be more API groups, all at REST path `/apis/$API_GROUP` and -using `apiVersion: $API_GROUP/$VERSION`. We expect that there will be a way for [third parties to -create their own API groups](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/extending-api.md), and to avoid naming collisions. +There are two supported paths to extending the API. +1. [Third Party Resources](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md) + are for users with very basic CRUD needs. +1. Coming soon: users needing the full set of Kubernetes API semantics can implement their own apiserver + and use the [aggregator](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md) + to make it seamless for clients. -## Enabling resources in the extensions group + +## Enabling API groups + +Certain resources and API groups are enabled by default. They can be enabled or disabled by setting `--runtime-config` +on apiserver. `--runtime-config` accepts comma separated values. For ex: to disable batch/v1, set +`--runtime-config=batch/v1=false`, to enable batch/v2alpha1, set `--runtime-config=batch/v2alpha1`. +The flag accepts comma separated set of key=value pairs describing runtime configuration of the apiserver. + +IMPORTANT: Enabling or disabling groups or resources requires restarting apiserver and controller-manager +to pick up the `--runtime-config` changes. + +## Enabling resources in the groups DaemonSets, Deployments, HorizontalPodAutoscalers, Ingress, Jobs and ReplicaSets are enabled by default. -Other extensions resources can be enabled by setting runtime-config on -apiserver. runtime-config accepts comma separated values. For ex: to disable deployments and jobs, set +Other extensions resources can be enabled by setting `--runtime-config` on +apiserver. `--runtime-config` accepts comma separated values. For ex: to disable deployments and jobs, set `--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/jobs=false` From b26190b58b148f75e9f7db91680c60ac1a8747cf Mon Sep 17 00:00:00 2001 From: Michael Nelson Date: Tue, 17 Jan 2017 18:05:32 +1100 Subject: [PATCH 45/68] Note about the need to fix permissions for a hostPath volume mount #2260 This is a work-around for #2260 until the underlying problem is addressed (https://github.com/kubernetes/kubernetes/issues/2630). --- .../stateful-application/basic-stateful-set.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/tutorials/stateful-application/basic-stateful-set.md b/docs/tutorials/stateful-application/basic-stateful-set.md index ef15bf3607..2a72070ac8 100644 --- a/docs/tutorials/stateful-application/basic-stateful-set.md +++ b/docs/tutorials/stateful-application/basic-stateful-set.md @@ -285,6 +285,16 @@ web-0 web-1 ``` +Note, if you instead see 403 Forbidden responses for the above curl command, +you will need to fix the permissions of the directory mounted by the `volumeMounts` +(due to a [bug when using hostPath volumes](https://github.com/kubernetes/kubernetes/issues/2630)) with: + +```shell +for i in 0 1; do kubectl exec web-$i -- chmod 755 /usr/share/nginx/html; done +``` + +before retrying the curl command above. + In one terminal, watch the StatefulSet's Pods. ```shell From d166363fe34bd72c2ebb6f16da353f277e4a244e Mon Sep 17 00:00:00 2001 From: Jordan Liggitt Date: Mon, 19 Dec 2016 10:02:32 -0500 Subject: [PATCH 46/68] Update ABAC examples to use authenticated/unauthenticated groups --- docs/admin/authorization.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/admin/authorization.md b/docs/admin/authorization.md index 8f2b60472c..11b7cabeb2 100644 --- a/docs/admin/authorization.md +++ b/docs/admin/authorization.md @@ -85,8 +85,8 @@ properties: - `kind`, type string: valid values are "Policy". Allows versioning and conversion of the policy format. - `spec` property set to a map with the following properties: - Subject-matching properties: - - `user`, type string; the user-string from `--token-auth-file`. If you specify `user`, it must match the username of the authenticated user. `*` matches all requests. - - `group`, type string; if you specify `group`, it must match one of the groups of the authenticated user. `*` matches all requests. + - `user`, type string; the user-string from `--token-auth-file`. If you specify `user`, it must match the username of the authenticated user. + - `group`, type string; if you specify `group`, it must match one of the groups of the authenticated user. `system:authenticated` matches all authenticated requests. `system:unauthenticated` matches all unauthenticated requests. - `readonly`, type boolean, when true, means that the policy only applies to get, list, and watch operations. - Resource-matching properties: - `apiGroup`, type string; an API group, such as `extensions`. `*` matches all API groups. @@ -115,8 +115,11 @@ The tuple of attributes is checked for a match against every policy in the policy file. If at least one line matches the request attributes, then the request is authorized (but may fail later validation). -To permit any user to do something, write a policy with the user property set to -`"*"`. +To permit any authenticated user to do something, write a policy with the +group property set to `"system:authenticated"`. + +To permit any unauthenticated user to do something, write a policy with the +group property set to `"system:unauthenticated"`. To permit a user to do anything, write a policy with the apiGroup, namespace, resource, and nonResourcePath properties set to `"*"`. @@ -165,7 +168,8 @@ up the verbosity: 5. Anyone can make read-only requests to all non-resource paths: ```json - {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user": "*", "readonly": true, "nonResourcePath": "*"}} + {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group": "system:authenticated", "readonly": true, "nonResourcePath": "*"}} + {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group": "system:unauthenticated", "readonly": true, "nonResourcePath": "*"}} ``` [Complete file example](http://releases.k8s.io/{{page.githubbranch}}/pkg/auth/authorizer/abac/example_policy_file.jsonl) @@ -504,7 +508,7 @@ An example request body: "resourceAttributes": { "namespace": "kittensandponies", "verb": "GET", - "group": "*", + "group": "unicorn.example.org", "resource": "pods" }, "user": "jane", @@ -627,7 +631,7 @@ __EOF__ --- snip lots of output --- -I0913 08:12:31.362873 27425 request.go:908] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"resourceAttributes":{"namespace":"kittensandponies","verb":"GET","group":"*","resource":"pods"},"user":"jane","group":["group1","group2"]},"status":{"allowed":true}} +I0913 08:12:31.362873 27425 request.go:908] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"resourceAttributes":{"namespace":"kittensandponies","verb":"GET","group":"unicorn.example.org","resource":"pods"},"user":"jane","group":["group1","group2"]},"status":{"allowed":true}} subjectaccessreview "" created ``` From e1a55e327e7b198e7937599ddbb6330b51096acb Mon Sep 17 00:00:00 2001 From: s0rc3r3r01 Date: Thu, 5 Jan 2017 15:29:13 +0000 Subject: [PATCH 47/68] corrected container name the container name is "kubedns" and not "kube-dns" --- docs/admin/dns.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/dns.md b/docs/admin/dns.md index 7f1c3c4641..7a845c9f3f 100644 --- a/docs/admin/dns.md +++ b/docs/admin/dns.md @@ -280,7 +280,7 @@ If you see that no pod is running or that the pod has failed/completed, the DNS Use `kubectl logs` command to see logs for the DNS daemons. ``` -kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kube-dns +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz ``` From bb802690a5e23689ae7ceba17b6e48fc81c3e7cc Mon Sep 17 00:00:00 2001 From: Drinky Pool Date: Fri, 6 Jan 2017 10:00:48 +0800 Subject: [PATCH 48/68] modify get pods to be more accurate modify the "kubectl get pods" to "kubectl get pods -l purpose=demonstrate-envars" --- .../define-environment-variable-container.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/configure-pod-container/define-environment-variable-container.md b/docs/tasks/configure-pod-container/define-environment-variable-container.md index 7faba40522..214142fa80 100644 --- a/docs/tasks/configure-pod-container/define-environment-variable-container.md +++ b/docs/tasks/configure-pod-container/define-environment-variable-container.md @@ -38,7 +38,7 @@ Pod: 1. List the running Pods: - kubectl get pods + kubectl get pods -l purpose=demonstrate-envars The output is similar to this: From 018f51074edbb5fc91cc24ca087017735efd1779 Mon Sep 17 00:00:00 2001 From: sandeepveldi Date: Wed, 21 Dec 2016 00:35:46 -0500 Subject: [PATCH 49/68] Add Windows version details Adding windows version related download path. --- docs/user-guide/prereqs.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/user-guide/prereqs.md b/docs/user-guide/prereqs.md index 19e6d8612b..6e630ec0ea 100644 --- a/docs/user-guide/prereqs.md +++ b/docs/user-guide/prereqs.md @@ -17,6 +17,9 @@ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s htt # Linux curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl + +# Windows +curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/windows/amd64/kubectl.exe ``` If you want to download a specific version of kubectl you can replace the nested curl command from above with the version you want. (e.g. v1.4.6, v1.5.0-beta.2) From 2f814b83adb05b4d2a2094e5f1500b654389e6dd Mon Sep 17 00:00:00 2001 From: Michal Gebauer Date: Thu, 26 Jan 2017 10:51:23 +0100 Subject: [PATCH 50/68] Typo in pod.md --- docs/concepts/abstractions/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/abstractions/pod.md b/docs/concepts/abstractions/pod.md index 33637168db..07f0d18934 100644 --- a/docs/concepts/abstractions/pod.md +++ b/docs/concepts/abstractions/pod.md @@ -31,7 +31,7 @@ Each Pod is meant to run a single instance of a given application. If you want t ### How Pods Manage Multiple Containers -Pods are designed to support multiple cooperating processes (as containers) that form a cohesive unit of service. The containers in a Pod are automatically co-located and co-scheduled on the same phyiscal or virtual machine in the cluster. The containers can share resources and dependencies, communicate with one another, and coordinate when and how they are terminated. +Pods are designed to support multiple cooperating processes (as containers) that form a cohesive unit of service. The containers in a Pod are automatically co-located and co-scheduled on the same physical or virtual machine in the cluster. The containers can share resources and dependencies, communicate with one another, and coordinate when and how they are terminated. Note that grouping multiple co-located and co-managed containers in a single Pod is a relatively advanced use case. You should use this pattern only in specific instances in which your containers are tightly coupled. For example, you might have a container that acts as a web server for files in a shared volume, and a separate "sidecar" container that updates those files from a remote source, as in the following diagram: From e1369ecc4ffac45179aee3ce17118e9ef416d7e3 Mon Sep 17 00:00:00 2001 From: scjane Date: Thu, 26 Jan 2017 11:04:36 +0800 Subject: [PATCH 51/68] Update node-problem.md --- docs/admin/node-problem.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/node-problem.md b/docs/admin/node-problem.md index b4f3e6ee31..08ca3208dd 100644 --- a/docs/admin/node-problem.md +++ b/docs/admin/node-problem.md @@ -31,7 +31,7 @@ See more information kernel log now. It doesn't support log tools like journald. * The kernel issue detection of node problem detector has assumption on kernel -log format, now it only works on Ubuntu and Debian. However, it is easy to extend +log format, and now it only works on Ubuntu and Debian. However, it is easy to extend it to [support other log format](/docs/admin/node-problem/#support-other-log-format). ## Enable/Disable in GCE cluster @@ -194,7 +194,7 @@ and detects known kernel issues following predefined rules. The Kernel Monitor matches kernel issues according to a set of predefined rule list in [`config/kernel-monitor.json`](https://github.com/kubernetes/node-problem-detector/blob/v0.1/config/kernel-monitor.json). -The rule list is extensible, you can always extend it by [overwriting the +The rule list is extensible, and you can always extend it by [overwriting the configuration](/docs/admin/node-problem/#overwrite-the-configuration). ### Add New NodeConditions From 9b59945b0ce504e79da9aa63dcf8013907ca009d Mon Sep 17 00:00:00 2001 From: scjane Date: Thu, 26 Jan 2017 10:45:44 +0800 Subject: [PATCH 52/68] Update salt.md It seems that the ordered list in the "Best Practices" section is unnecessary as there is only one item "1". So can we delete the list? --- docs/admin/salt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/salt.md b/docs/admin/salt.md index ab495521bd..05d0ae9250 100644 --- a/docs/admin/salt.md +++ b/docs/admin/salt.md @@ -92,7 +92,7 @@ In addition, a cluster may be running a Debian based operating system or Red Hat ## Best Practices -1. When configuring default arguments for processes, it's best to avoid the use of EnvironmentFiles (Systemd in Red Hat environments) or init.d files (Debian distributions) to hold default values that should be common across operating system environments. This helps keep our Salt template files easy to understand for editors who may not be familiar with the particulars of each distribution. +When configuring default arguments for processes, it's best to avoid the use of EnvironmentFiles (Systemd in Red Hat environments) or init.d files (Debian distributions) to hold default values that should be common across operating system environments. This helps keep our Salt template files easy to understand for editors who may not be familiar with the particulars of each distribution. ## Future enhancements (Networking) From 8a299b94cfb4000d6e5eec30f9a5d973f0728cd9 Mon Sep 17 00:00:00 2001 From: scjane Date: Thu, 26 Jan 2017 10:18:24 +0800 Subject: [PATCH 53/68] Update garbage-collection.md --- docs/admin/garbage-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/garbage-collection.md b/docs/admin/garbage-collection.md index 082480ebac..a960172426 100644 --- a/docs/admin/garbage-collection.md +++ b/docs/admin/garbage-collection.md @@ -24,7 +24,7 @@ threshold has been met. ### Container Collection The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers any single -pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting 'Min Age' to zero and setting 'MaxPerPodContainer' and 'MaxContainers' respectively to less than zero. +pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting 'MinAge' to zero and setting 'MaxPerPodContainer' and 'MaxContainers' respectively to less than zero. Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. 'MaxPerPodContainer' and 'MaxContainer' may potentially conflict with each other in situations where retaining the maximum number of containers per pod ('MaxPerPodContainer') would go outside the allowable range of global dead containers ('MaxContainers'). 'MaxPerPodContainer' would be adjusted in this situation: A worst case scenario would be to downgrade 'MaxPerPodContainer' to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`. From ae656cc45237eb19c72a1890bf01c6796c12d90c Mon Sep 17 00:00:00 2001 From: leonliao Date: Mon, 16 Jan 2017 18:05:54 +0800 Subject: [PATCH 54/68] Instruct beginners to start minikube cluster with proxy Beginners don't know the details for troubleshooting, better instruction will save time for beginners to trouble shoot logs in minikube complaining gcr.io not reachable and etc. --- .../stateless-application/hello-minikube.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/stateless-application/hello-minikube.md b/docs/tutorials/stateless-application/hello-minikube.md index 9e6a93e5c3..772561444a 100644 --- a/docs/tutorials/stateless-application/hello-minikube.md +++ b/docs/tutorials/stateless-application/hello-minikube.md @@ -70,12 +70,23 @@ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s htt chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl ``` +Determine whether you can access sites like https://cloud.google.com/container-registry/ directly without a proxy, by opening a new terminal and using +```shell +export http_proxy="" +export https_proxy="" +curl https://cloud.google.com/container-registry/ +``` -Start the Minikube cluster: +If NO proxy is required, start the Minikube cluster: ```shell minikube start --vm-driver=xhyve ``` +If a proxy server is required, use below way to start Minikube cluster with proxy setting: + +```shell +minikube start --vm-driver=xhyve --docker-env HTTP_PROXY=http://your-http-proxy-host:your-http-proxy-port --docker-env HTTPS_PROXY=http(s)://your-https-proxy-host:your-https-proxy-port +``` The `--vm-driver=xyhve` flag specifies that you are using Docker for Mac. The default VM driver is VirtualBox. From 48af43f282f60318ad6627d24bc7c523cff23a22 Mon Sep 17 00:00:00 2001 From: leonliao Date: Thu, 19 Jan 2017 12:10:17 +0800 Subject: [PATCH 55/68] Rephrase the sentence Rephrase the `use the below way` to use `the following method` --- docs/tutorials/stateless-application/hello-minikube.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/stateless-application/hello-minikube.md b/docs/tutorials/stateless-application/hello-minikube.md index 772561444a..31348aa586 100644 --- a/docs/tutorials/stateless-application/hello-minikube.md +++ b/docs/tutorials/stateless-application/hello-minikube.md @@ -82,7 +82,7 @@ If NO proxy is required, start the Minikube cluster: ```shell minikube start --vm-driver=xhyve ``` -If a proxy server is required, use below way to start Minikube cluster with proxy setting: +If a proxy server is required, use the following method to start Minikube cluster with proxy setting: ```shell minikube start --vm-driver=xhyve --docker-env HTTP_PROXY=http://your-http-proxy-host:your-http-proxy-port --docker-env HTTPS_PROXY=http(s)://your-https-proxy-host:your-https-proxy-port From ef79cdc518af9d1f9cfdff02f8dfb520857af214 Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 26 Jan 2017 10:39:39 -0800 Subject: [PATCH 56/68] Update hello-minikube.md --- docs/tutorials/stateless-application/hello-minikube.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/stateless-application/hello-minikube.md b/docs/tutorials/stateless-application/hello-minikube.md index 31348aa586..36b92dd97a 100644 --- a/docs/tutorials/stateless-application/hello-minikube.md +++ b/docs/tutorials/stateless-application/hello-minikube.md @@ -70,7 +70,7 @@ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s htt chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl ``` -Determine whether you can access sites like https://cloud.google.com/container-registry/ directly without a proxy, by opening a new terminal and using +Determine whether you can access sites like [https://cloud.google.com/container-registry/](https://cloud.google.com/container-registry/) directly without a proxy, by opening a new terminal and using ```shell export http_proxy="" export https_proxy="" From f59094324eee014cd548d477e3721df3d71fd5ed Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 26 Jan 2017 12:08:42 -0800 Subject: [PATCH 57/68] grammar issue Fixed https://github.com/kubernetes/kubernetes.github.io/pull/2348 --- docs/getting-started-guides/centos/centos_manual_config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started-guides/centos/centos_manual_config.md b/docs/getting-started-guides/centos/centos_manual_config.md index f51aacf477..7f8f5ea7f2 100644 --- a/docs/getting-started-guides/centos/centos_manual_config.md +++ b/docs/getting-started-guides/centos/centos_manual_config.md @@ -17,7 +17,7 @@ This is a getting started guide for CentOS. It is a manual configuration so you The Kubernetes package provides a few services: kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy. These services are managed by systemd and the configuration resides in a central location: /etc/kubernetes. We will break the services up between the hosts. The first host, centos-master, will be the Kubernetes master. This host will run the kube-apiserver, kube-controller-manager and kube-scheduler. In addition, the master will also run _etcd_. The remaining hosts, centos-minion-n will be the nodes and run kubelet, proxy, cadvisor and docker. -All of then run flanneld as networking overlay. +All of them run flanneld as networking overlay. **System Information:** From 2fc014494fd32e91f6879db13a6e80521a48e7d3 Mon Sep 17 00:00:00 2001 From: Steve Lasker Date: Wed, 25 Jan 2017 16:31:23 -0800 Subject: [PATCH 58/68] Update user-guide/quick-start.md to include Azure Added Azure as a target --- docs/user-guide/quick-start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/quick-start.md b/docs/user-guide/quick-start.md index 6ad7940526..e1b0829843 100644 --- a/docs/user-guide/quick-start.md +++ b/docs/user-guide/quick-start.md @@ -13,7 +13,7 @@ This guide will help you get oriented to Kubernetes and running your first conta ## Launching a simple application, and exposing it to the Internet Once your application is packaged into a container and pushed to an image registry, you're ready to deploy it to Kubernetes. -Through integration with some cloud providers (for example Google Compute Engine and AWS EC2), Kubernetes also enables you to request it to provision a public IP address for your application. +Through integration with some cloud providers (for example Google Compute Engine, AWS EC2, Azure ACS), Kubernetes also enables you to request it to provision a public IP address for your application. For example, [nginx](http://wiki.nginx.org/Main) is a popular HTTP server, with a [pre-built container on Docker hub](https://registry.hub.docker.com/_/nginx/). The [`kubectl run`](/docs/user-guide/kubectl/kubectl_run) commands below will create two nginx replicas, listening on port 80, and a public IP address for your application. From 4fb74447e003debc13e819f89559b28087a563c9 Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 26 Jan 2017 14:09:44 -0800 Subject: [PATCH 59/68] Update quick-start.md --- docs/user-guide/quick-start.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/quick-start.md b/docs/user-guide/quick-start.md index e1b0829843..6cef04810d 100644 --- a/docs/user-guide/quick-start.md +++ b/docs/user-guide/quick-start.md @@ -13,7 +13,7 @@ This guide will help you get oriented to Kubernetes and running your first conta ## Launching a simple application, and exposing it to the Internet Once your application is packaged into a container and pushed to an image registry, you're ready to deploy it to Kubernetes. -Through integration with some cloud providers (for example Google Compute Engine, AWS EC2, Azure ACS), Kubernetes also enables you to request it to provision a public IP address for your application. +Through integration with some cloud providers (for example Google Compute Engine, AWS EC2, and Azure ACS), Kubernetes also enables you to request it to provision a public IP address for your application. For example, [nginx](http://wiki.nginx.org/Main) is a popular HTTP server, with a [pre-built container on Docker hub](https://registry.hub.docker.com/_/nginx/). The [`kubectl run`](/docs/user-guide/kubectl/kubectl_run) commands below will create two nginx replicas, listening on port 80, and a public IP address for your application. @@ -70,4 +70,4 @@ service "my-nginx" deleted ## What's next? -[Learn about how to configure common container parameters, such as commands and environment variables.](/docs/user-guide/configuring-containers) +* [Learn about how to configure common container parameters, such as commands and environment variables.](/docs/user-guide/configuring-containers) From ed5fe607f3edc45a0c4db432385bc1ef1c761765 Mon Sep 17 00:00:00 2001 From: Rudi C Date: Thu, 26 Jan 2017 06:12:33 -0500 Subject: [PATCH 60/68] Document cross-region ECR support Fetching images from a registry hosted in a different region has been supported since 1.3 with https://github.com/kubernetes/kubernetes/pull/24369 --- docs/user-guide/images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/images.md b/docs/user-guide/images.md index 37cfd53b9d..8daaeaeb7c 100644 --- a/docs/user-guide/images.md +++ b/docs/user-guide/images.md @@ -89,7 +89,7 @@ The kubelet will fetch and periodically refresh ECR credentials. It needs the f Requirements: - You must be using kubelet version `v1.2.0` or newer. (e.g. run `/usr/bin/kubelet --version=true`). -- Your nodes must be in the same region as the registry you are using +- If your nodes are in region A and your registry in a different region B, you need version `v1.3.0`. - ECR must be offered in your region Troubleshooting: From 3a474df151b79b761f1a55227e824b60124e6e21 Mon Sep 17 00:00:00 2001 From: Rudi C Date: Thu, 26 Jan 2017 08:38:39 -0500 Subject: [PATCH 61/68] Update images.md --- docs/user-guide/images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/images.md b/docs/user-guide/images.md index 8daaeaeb7c..4ce0043a7b 100644 --- a/docs/user-guide/images.md +++ b/docs/user-guide/images.md @@ -89,7 +89,7 @@ The kubelet will fetch and periodically refresh ECR credentials. It needs the f Requirements: - You must be using kubelet version `v1.2.0` or newer. (e.g. run `/usr/bin/kubelet --version=true`). -- If your nodes are in region A and your registry in a different region B, you need version `v1.3.0`. +- If your nodes are in region A and your registry in a different region B, you need version `v1.3.0` or newer. - ECR must be offered in your region Troubleshooting: From cef61c8ba9d6d33a9cdf9a3a060784c2f5bbae51 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Thu, 26 Jan 2017 14:16:28 -0800 Subject: [PATCH 62/68] Fix small grammar issue add 'is' --- docs/user-guide/images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/images.md b/docs/user-guide/images.md index 4ce0043a7b..5b523e5615 100644 --- a/docs/user-guide/images.md +++ b/docs/user-guide/images.md @@ -89,7 +89,7 @@ The kubelet will fetch and periodically refresh ECR credentials. It needs the f Requirements: - You must be using kubelet version `v1.2.0` or newer. (e.g. run `/usr/bin/kubelet --version=true`). -- If your nodes are in region A and your registry in a different region B, you need version `v1.3.0` or newer. +- If your nodes are in region A and your registry is in a different region B, you need version `v1.3.0` or newer. - ECR must be offered in your region Troubleshooting: From f7254169cc9e939bef9988132906e5a156048cb4 Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 26 Jan 2017 14:40:34 -0800 Subject: [PATCH 63/68] Add meetup resources to community pages --- community/index.html | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/community/index.html b/community/index.html index 4686ebb442..dea7b0ccb1 100644 --- a/community/index.html +++ b/community/index.html @@ -17,9 +17,12 @@ cid: community community meeting takes place via video conference to discuss the state of affairs, get a calendar invite to participate.

    +

    You can also join Kubernauts all around the world through our + [Kubernetes Meetup Community](https://www.meetup.com/topics/kubernetes/) and the + [Kubernetes Cloud Native Meetup Community](https://www.meetup.com/Kubernetes-Cloud-Native-Online-Meetup/).

    -

    SIGs

    +

    Community Special Interest Groups (SIGs)

    Have a special interest in how Kubernetes works with another technology? See our ever growing lists of SIGs, from AWS and Openstack to Big Data and Scalability, there's a place for you to contribute and instructions From d6aa14202601d807693a06b6dbb4fdd260e5291a Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 26 Jan 2017 14:48:49 -0800 Subject: [PATCH 64/68] Update index.html --- community/index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/community/index.html b/community/index.html index dea7b0ccb1..5815fa61a9 100644 --- a/community/index.html +++ b/community/index.html @@ -18,8 +18,8 @@ cid: community get a calendar invite to participate.

    You can also join Kubernauts all around the world through our - [Kubernetes Meetup Community](https://www.meetup.com/topics/kubernetes/) and the - [Kubernetes Cloud Native Meetup Community](https://www.meetup.com/Kubernetes-Cloud-Native-Online-Meetup/).

    + Kubernetes Meetup Community and the + Kubernetes Cloud Native Meetup Community.

    Community Special Interest Groups (SIGs)

    From 4c3e7d69514450ff30830cbbe8c8202955d17727 Mon Sep 17 00:00:00 2001 From: Jared Date: Thu, 26 Jan 2017 14:50:26 -0800 Subject: [PATCH 65/68] Update index.html --- community/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/community/index.html b/community/index.html index 5815fa61a9..201c70a776 100644 --- a/community/index.html +++ b/community/index.html @@ -22,7 +22,7 @@ cid: community Kubernetes Cloud Native Meetup Community.

    -

    Community Special Interest Groups (SIGs)

    +

    Special Interest Groups (SIGs)

    Have a special interest in how Kubernetes works with another technology? See our ever growing lists of SIGs, from AWS and Openstack to Big Data and Scalability, there's a place for you to contribute and instructions From ef4ff9b6fb019ff5516c26477dc87530f1716475 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Thu, 26 Jan 2017 15:02:50 -0800 Subject: [PATCH 66/68] reimplementation of PR #2165 PR #2165 was LGTM'ed, but the CLA check failed even after contributor signed it, so I'm reimplementing those changes here. --- docs/getting-started-guides/centos/centos_manual_config.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/centos/centos_manual_config.md b/docs/getting-started-guides/centos/centos_manual_config.md index 7f8f5ea7f2..031a5bb3f5 100644 --- a/docs/getting-started-guides/centos/centos_manual_config.md +++ b/docs/getting-started-guides/centos/centos_manual_config.md @@ -122,9 +122,9 @@ KUBE_API_ARGS="" **Warning** This network must be unused in your network infrastructure! `172.30.0.0/16` is free in our network. ```shell -$ systemctl start etcd -$ etcdctl mkdir /kube-centos/network -$ etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }" +systemctl start etcd +etcdctl mkdir /kube-centos/network +etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }" ``` * Configure flannel to overlay Docker network in /etc/sysconfig/flanneld on the master (also in the nodes as we'll see): From dbc615e862335cbaed8a82582e755c0b28a2fb50 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Thu, 26 Jan 2017 15:13:51 -0800 Subject: [PATCH 67/68] Reimplementing #1708 PR #1708 was Tech LGTM'ed, but the CLA needed to be re-signed (the new CLA needed to be signed), so I'm reimplementing the changes here. --- docs/getting-started-guides/kubeadm.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting-started-guides/kubeadm.md b/docs/getting-started-guides/kubeadm.md index 1765e0a1f4..40827c8825 100644 --- a/docs/getting-started-guides/kubeadm.md +++ b/docs/getting-started-guides/kubeadm.md @@ -113,9 +113,9 @@ To initialize the master, pick one of the machines you previously installed `kub # kubeadm init **Note:** this will autodetect the network interface to advertise the master on as the interface with the default gateway. -If you want to use a different interface, specify `--api-advertise-addresses=` argument to `kubeadm init`. +If you want to use a different interface, specify `--api-advertise-addresses ` argument to `kubeadm init`. -If you want to use [flannel](https://github.com/coreos/flannel) as the pod network, specify `--pod-network-cidr=10.244.0.0/16` if you're using the daemonset manifest below. _However, please note that this is not required for any other networks besides Flannel._ +If you want to use [flannel](https://github.com/coreos/flannel) as the pod network, specify `--pod-network-cidr 10.244.0.0/16` if you're using the daemonset manifest below. _However, please note that this is not required for any other networks besides Flannel._ Please refer to the [kubeadm reference doc](/docs/admin/kubeadm/) if you want to read more about the flags `kubeadm init` provides. From e3494fd1430f77545cf819bb457039d0173e6de2 Mon Sep 17 00:00:00 2001 From: Slava Semushin Date: Tue, 3 Jan 2017 16:36:53 +0100 Subject: [PATCH 68/68] docs/user-guide/pod-security-policy: misc improvements. - update link - improve wording - update list of volumes --- docs/user-guide/pod-security-policy/index.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/pod-security-policy/index.md b/docs/user-guide/pod-security-policy/index.md index 6a756c4766..da1c6514d6 100644 --- a/docs/user-guide/pod-security-policy/index.md +++ b/docs/user-guide/pod-security-policy/index.md @@ -8,7 +8,7 @@ Objects of type `podsecuritypolicy` govern the ability to make requests on a pod that affect the `SecurityContext` that will be applied to a pod and container. -See [PodSecurityPolicy proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/security-context-constraints.md) for more information. +See [PodSecurityPolicy proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/security-context-constraints.md) for more information. * TOC {:toc} @@ -39,7 +39,7 @@ into three categories: restrictive value. - *Controlled by an allowable set*: Fields of this type are checked against the set to ensure their value is allowed. -- *Controlled by a strategy*: Items that have a strategy to generate a value provide +- *Controlled by a strategy*: Items that have a strategy to provide a mechanism to generate the value and a mechanism to ensure that a specified value falls into the set of allowable values. @@ -102,6 +102,10 @@ to the volume sources that are defined when creating a volume: 1. downwardAPI 1. fc 1. configMap +1. vsphereVolume +1. quobyte +1. azureDisk +1. photonPersistentDisk 1. \* (allow all volumes) The recommended minimum set of allowed volumes for new PSPs are