From c7e57cb67cde0349063d34872aba6766a14d56a6 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 5 Aug 2019 17:39:49 -0500 Subject: [PATCH 01/49] TLS is Transport Layer Security; not level (#14998) See https://en.wikipedia.org/wiki/Transport_Layer_Security --- content/en/docs/tasks/administer-cluster/securing-a-cluster.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/securing-a-cluster.md b/content/en/docs/tasks/administer-cluster/securing-a-cluster.md index 36de25de82..e588ecc369 100644 --- a/content/en/docs/tasks/administer-cluster/securing-a-cluster.md +++ b/content/en/docs/tasks/administer-cluster/securing-a-cluster.md @@ -28,7 +28,7 @@ and provides recommendations on overall security. As Kubernetes is entirely API driven, controlling and limiting who can access the cluster and what actions they are allowed to perform is the first line of defense. -### Use Transport Level Security (TLS) for all API traffic +### Use Transport Layer Security (TLS) for all API traffic Kubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the majority of installation methods will allow the necessary certificates to be created and distributed to From 5a4088af4200b4a4fc833267251dbd637c25b730 Mon Sep 17 00:00:00 2001 From: janitha09 Date: Mon, 5 Aug 2019 15:41:49 -0700 Subject: [PATCH 02/49] Update install-minikube.md (#15087) kubernetes-cli is now included as a dependency https://chocolatey.org/packages/Minikube --- content/en/docs/tasks/tools/install-minikube.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/tools/install-minikube.md b/content/en/docs/tasks/tools/install-minikube.md index 32b51751e0..032c3fdba4 100644 --- a/content/en/docs/tasks/tools/install-minikube.md +++ b/content/en/docs/tasks/tools/install-minikube.md @@ -158,7 +158,7 @@ Hyper-V can run on three versions of Windows 10: Windows 10 Enterprise, Windows The easiest way to install Minikube on Windows is using [Chocolatey](https://chocolatey.org/) (run as an administrator): ```shell -choco install minikube kubernetes-cli +choco install minikube ``` After Minikube has finished installing, close the current CLI session and restart. Minikube should have been added to your path automatically. From d9279c72f14cb67d11afff3424fe742499aa3877 Mon Sep 17 00:00:00 2001 From: Kaitlyn Barnard Date: Mon, 5 Aug 2019 16:09:50 -0700 Subject: [PATCH 03/49] Update OWNERS (#15681) --- content/en/blog/OWNERS | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/en/blog/OWNERS b/content/en/blog/OWNERS index dd05474d19..f234f081dc 100644 --- a/content/en/blog/OWNERS +++ b/content/en/blog/OWNERS @@ -2,9 +2,6 @@ # Owned by Kubernetes Blog reviewers -options: - no_parent_owners: false - approvers: - sig-docs-blog-owners # Defined in OWNERS_ALIASES From cb1272defef22865e0e4b0be9e880883c53d805b Mon Sep 17 00:00:00 2001 From: Alex Punnen Date: Tue, 6 Aug 2019 04:41:50 +0530 Subject: [PATCH 04/49] Update pod.md (#15301) fixed broken link --- content/en/docs/concepts/workloads/pods/pod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/workloads/pods/pod.md b/content/en/docs/concepts/workloads/pods/pod.md index 4fed2c5e5a..7dff25cbb5 100644 --- a/content/en/docs/concepts/workloads/pods/pod.md +++ b/content/en/docs/concepts/workloads/pods/pod.md @@ -154,7 +154,7 @@ and rollout management. Controllers like [StatefulSet](/docs/concepts/workloads/controllers/statefulset.md) can also provide support to stateful Pods. -The use of collective APIs as the primary user-facing primitive is relatively common among cluster scheduling systems, including [Borg](https://research.google.com/pubs/pub43438.html), [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html), [Aurora](http://aurora.apache.org/documentation/latest/reference/configuration/#job-schema), and [Tupperware](http://www.slideshare.net/Docker/aravindnarayanan-facebook140613153626phpapp02-37588997). +The use of collective APIs as the primary user-facing primitive is relatively common among cluster scheduling systems, including [Borg](https://research.google.com/pubs/pub43438.html), [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html), [Aurora](http://aurora.apache.org/documentation/latest/reference/configuration/#job-schema), and [Tupperware](https://www.slideshare.net/Docker/aravindnarayanan-facebook140613153626phpapp02-37588997). Pod is exposed as a primitive in order to facilitate: From d4a3eaa2f51be96de571ed710ab79b1b99859497 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michael=20Vorburger=20=E2=9B=91=EF=B8=8F?= Date: Tue, 6 Aug 2019 01:55:20 +0200 Subject: [PATCH 05/49] use log_path busybox.log instead of busybox/0.log in crictl.md (#15415) because with busybox/0.log it fails as shown below (unless you mkdir busybox, with the correct permission, which is an extra step we can avoid for a tutorial like this) FATA[0000] Starting the container "3c71f8c3abfcac0f8357fa25be896062f05f88e1d9acbd7a033afeee9a54f3c4" failed: rpc error: code = Unknown desc = failed to create containerd task: failed to create container loggers: failed to create and open log file: open busybox/0.log: no such file or directory --- content/en/docs/tasks/debug-application-cluster/crictl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/debug-application-cluster/crictl.md b/content/en/docs/tasks/debug-application-cluster/crictl.md index f1f8694c2e..f7bfec87ff 100644 --- a/content/en/docs/tasks/debug-application-cluster/crictl.md +++ b/content/en/docs/tasks/debug-application-cluster/crictl.md @@ -295,7 +295,7 @@ deleted by the Kubelet. "command": [ "top" ], - "log_path":"busybox/0.log", + "log_path":"busybox.log", "linux": { } } From da9db0b0c4ff9ea52c3cde967231f5880b415a86 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michael=20Vorburger=20=E2=9B=91=EF=B8=8F?= Date: Tue, 6 Aug 2019 02:09:54 +0200 Subject: [PATCH 06/49] porting some information from kubeadm-init.md to kubelet-integration.md (#15423) * porting some information from kubeadm-init.md to kubelet-integration.md doc from reference/setup-tools/kubeadm/kubeadm-init.md to setup/production-environment/tools/kubeadm/kubelet-integration.md includes edits I just proposed in #15421 and #80136 * improve wording re. systemd in kubelet-integration.md * removing quotes and adding the in kubelet-integration.md * Update content/en/docs/setup/production-environment/tools/kubeadm/kubelet-integration.md Co-Authored-By: Lubomir I. Ivanov --- .../tools/kubeadm/kubelet-integration.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/kubelet-integration.md b/content/en/docs/setup/production-environment/tools/kubeadm/kubelet-integration.md index f3ca59d898..ef9b958a7e 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/kubelet-integration.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/kubelet-integration.md @@ -157,15 +157,19 @@ has finished performing the TLS Bootstrap. ## The kubelet drop-in file for systemd -The configuration file installed by the kubeadm DEB or RPM package is written to +kubeadm ships with configuration for how systemd should run the kubelet. +Note that the kubeadm CLI command never touches this drop-in file. + +This configuration file installed by the `kubeadm` [DEB](https://github.com/kubernetes/kubernetes/blob/master/build/debs/10-kubeadm.conf) or [RPM package](https://github.com/kubernetes/kubernetes/blob/master/build/rpms/10-kubeadm.conf) is written to `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` and is used by systemd. +It augments the basic [`kubelet.service` for RPM](https://github.com/kubernetes/kubernetes/blob/master/build/rpms/kubelet.service) (resp. [`kubelet.service` for DEB](https://github.com/kubernetes/kubernetes/blob/master/build/debs/kubelet.service))): ```none [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" -# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating +# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, From 97297af5efe1a7a04a98aadf6bdf206afa71b9bf Mon Sep 17 00:00:00 2001 From: Kobayashi Daisuke Date: Tue, 6 Aug 2019 09:25:54 +0900 Subject: [PATCH 07/49] Only one word typo fix (#15617) * Fix typo * Fix typo * Cange back to "will" --- content/en/docs/concepts/containers/images.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/containers/images.md b/content/en/docs/concepts/containers/images.md index b2f374ea06..c408cc1e6b 100644 --- a/content/en/docs/concepts/containers/images.md +++ b/content/en/docs/concepts/containers/images.md @@ -70,7 +70,7 @@ Credentials can be provided in several ways: - Configuring Nodes to Authenticate to a Private Registry - all pods can read any configured private registries - requires node configuration by cluster administrator - - Pre-pulling Images + - Pre-pulled Images - all pods can use any images cached on a node - requires root access to all nodes to setup - Specifying ImagePullSecrets on a Pod @@ -243,7 +243,7 @@ template needs to include the `.docker/config.json` or mount a drive that contai All pods will have read access to images in any private registry once private registry keys are added to the `.docker/config.json`. -### Pre-pulling Images +### Pre-pulled Images {{< note >}} If you are running on Google Kubernetes Engine, there will already be a `.dockercfg` on each node with credentials for Google Container Registry. You cannot use this approach. From a17f259b8d292bfa75e895b9a82348dca3ae645b Mon Sep 17 00:00:00 2001 From: Melony QIN Date: Tue, 6 Aug 2019 04:11:53 +0100 Subject: [PATCH 08/49] update cheat sheet (#15677) add sort by capacity --- content/en/docs/reference/kubectl/cheatsheet.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/en/docs/reference/kubectl/cheatsheet.md b/content/en/docs/reference/kubectl/cheatsheet.md index 19a0b78f4f..10e5b140ef 100644 --- a/content/en/docs/reference/kubectl/cheatsheet.md +++ b/content/en/docs/reference/kubectl/cheatsheet.md @@ -158,6 +158,10 @@ kubectl get services --sort-by=.metadata.name # List Services Sorted by Name # List pods Sorted by Restart Count kubectl get pods --sort-by='.status.containerStatuses[0].restartCount' +# List pods in test namespace sorted by capacity + +kubectl get pods -n test --sort-by=.spec.capacity.storage + # Get the version label of all pods with label app=cassandra kubectl get pods --selector=app=cassandra -o \ jsonpath='{.items[*].metadata.labels.version}' From cf3c352881d4b567c93c5ee217115662aee7f86d Mon Sep 17 00:00:00 2001 From: RA489 Date: Tue, 6 Aug 2019 08:43:55 +0530 Subject: [PATCH 09/49] None driver doesn't work with snap docker. (#15667) --- content/en/docs/tasks/tools/install-minikube.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/tools/install-minikube.md b/content/en/docs/tasks/tools/install-minikube.md index 032c3fdba4..f7c66a5dc8 100644 --- a/content/en/docs/tasks/tools/install-minikube.md +++ b/content/en/docs/tasks/tools/install-minikube.md @@ -73,7 +73,7 @@ If you do not already have a hypervisor installed, install one of these now: • [VirtualBox](https://www.virtualbox.org/wiki/Downloads) {{< note >}} -Minikube also supports a `--vm-driver=none` option that runs the Kubernetes components on the host and not in a VM. Using this driver requires [Docker](https://www.docker.com/products/docker-desktop) and a Linux environment but not a hypervisor. +Minikube also supports a `--vm-driver=none` option that runs the Kubernetes components on the host and not in a VM. Using this driver requires [Docker](https://www.docker.com/products/docker-desktop) and a Linux environment but not a hypervisor. It is recommended to use the apt installation of docker from ([Docker](https://www.docker.com/products/docker-desktop), when using the none driver. The snap installation of docker does not work with minikube. {{< /note >}} ### Install Minikube using a package From 608693830e317bf861bb1ddccde78a9411da72ec Mon Sep 17 00:00:00 2001 From: Jesse Hu Date: Tue, 6 Aug 2019 11:15:54 +0800 Subject: [PATCH 10/49] Correct 'print default' command in kubeadm-config.md (#15662) * Correct 'print default' command in kubeadm-config.md `kubeadm config print-default` command doesn't exist in v1.15.0. It's now `kubeadm config print init-defaults` and `kubeadm config print join-defaults`. * Remove the unnecessary v1.11.0 statement --- .../docs/reference/setup-tools/kubeadm/kubeadm-config.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md index b740d18e9c..d5c10082d7 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md @@ -16,10 +16,9 @@ You can execute `kubeadm config view` to view the ConfigMap. If you initialized kubeadm v1.7.x or lower, you must use `kubeadm config upload` to create the ConfigMap before you may use `kubeadm upgrade`. -In Kubernetes v1.11.0, some new commands were added. You can use `kubeadm config print-default` -to print the default configuration and `kubeadm config migrate` to convert your old configuration -files to a newer version. `kubeadm config images list` and `kubeadm config images pull` can be used -to list and pull the images that kubeadm requires. +You can use `kubeadm config print` to print the default configuration and `kubeadm config migrate` to +convert your old configuration files to a newer version. `kubeadm config images list` and +`kubeadm config images pull` can be used to list and pull the images that kubeadm requires. In Kubernetes v1.13.0 and later to list/pull kube-dns images instead of the CoreDNS image the `--config` method described [here](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon) From 4fc5bf2c617ad83364a792b10563e238e7e9bb67 Mon Sep 17 00:00:00 2001 From: Naoki Oketani Date: Tue, 6 Aug 2019 12:17:53 +0900 Subject: [PATCH 11/49] fix broken links and changes added by editorconfig (#15628) --- content/en/docs/concepts/storage/volumes.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index 4ee3e00355..b92d4ff12e 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -149,7 +149,7 @@ spec: fsType: ext4 ``` -#### CSI Migration +#### CSI Migration {{< feature-state for_k8s_version="v1.14" state="alpha" >}} @@ -241,7 +241,7 @@ spec: fsType: ext4 ``` -#### CSI Migration +#### CSI Migration {{< feature-state for_k8s_version="v1.14" state="alpha" >}} @@ -1359,16 +1359,16 @@ documentation](https://kubernetes-csi.github.io/docs/) {{< feature-state for_k8s_version="v1.14" state="alpha" >}} The CSI Migration feature, when enabled, directs operations against existing in-tree -plugins to corresponding CSI plugins (which are expected to be installed and configured). -The feature implements the necessary translation logic and shims to re-route the -operations in a seamless fashion. As a result, operators do not have to make any -configuration changes to existing Storage Classes, PVs or PVCs (referring to +plugins to corresponding CSI plugins (which are expected to be installed and configured). +The feature implements the necessary translation logic and shims to re-route the +operations in a seamless fashion. As a result, operators do not have to make any +configuration changes to existing Storage Classes, PVs or PVCs (referring to in-tree plugins) when transitioning to a CSI driver that supersedes an in-tree plugin. -In the alpha state, the operations and features that are supported include +In the alpha state, the operations and features that are supported include provisioning/delete, attach/detach, mount/unmount and resizing of volumes. -In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented +In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the "Types of Volumes" section above. ### Flexvolume {#flexVolume} From 7459a83f62ea27bd34e344f2df2b567bf5e554f9 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 6 Aug 2019 13:19:53 +1000 Subject: [PATCH 12/49] Add Canonical On-Prem and Managed, Correct URL (#15675) Ubuntu / Canonical also provides deployment options for on-premesis and managed Kubernetes, as linked on our Kubernetes landing page. --- content/en/docs/setup/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/_index.md b/content/en/docs/setup/_index.md index 1fe527caac..b61c5286f5 100644 --- a/content/en/docs/setup/_index.md +++ b/content/en/docs/setup/_index.md @@ -75,7 +75,7 @@ The following production environment solutions table lists the providers and the | [Cisco Container Platform](https://cisco.com/go/containers) | | | ✔ | | | | [Cloud Foundry Container Runtime (CFCR)](https://docs-cfcr.cfapps.io/) | | | | ✔ |✔ | | [CloudStack](https://cloudstack.apache.org/) | | | | | ✔| -| [Canonical](https://www.ubuntu.com/kubernetes/docs/quickstart) | | ✔ | | ✔ |✔ | ✔ +| [Canonical](https://ubuntu.com/kubernetes) | ✔ | ✔ | ✔ | ✔ |✔ | ✔ | [Containership](https://containership.io/containership-platform) | ✔ |✔ | | | | | [Digital Rebar](https://provision.readthedocs.io/en/tip/README.html) | | | | | | ✔ | [DigitalOcean](https://www.digitalocean.com/products/kubernetes/) | ✔ | | | | | From 7126c8e9b526cf9c83e19f76c432357ed6c7a5bc Mon Sep 17 00:00:00 2001 From: Max Kaminskiy Date: Mon, 5 Aug 2019 20:21:53 -0700 Subject: [PATCH 13/49] Fixed typo. From utf=8 to utf-8. (#15684) --- content/en/examples/application/job/redis/worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/examples/application/job/redis/worker.py b/content/en/examples/application/job/redis/worker.py index 49e5dae798..b8abbee917 100644 --- a/content/en/examples/application/job/redis/worker.py +++ b/content/en/examples/application/job/redis/worker.py @@ -14,7 +14,7 @@ print("Initial queue state: empty=" + str(q.empty())) while not q.empty(): item = q.lease(lease_secs=10, block=True, timeout=2) if item is not None: - itemstr = item.decode("utf=8") + itemstr = item.decode("utf-8") print("Working on " + itemstr) time.sleep(10) # Put your actual work here instead of sleep. q.complete(item) From cdc25736d74197460d86f5e19a9ed6ee68b1ec2f Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 6 Aug 2019 10:45:56 +0300 Subject: [PATCH 14/49] kubeadm: fix links in kubeadm-alpha page (#15674) --- .../en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md index 35e38af3f7..9233ea85fa 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md @@ -14,7 +14,7 @@ weight: 90 ## kubeadm alpha certs renew {#cmd-certs-renew} You can renew all Kubernetes certificates using the `all` subcommand or renew them selectively. -For more details about certificate expiration and renewal see the [certificate management documentation](docs/tasks/administer-cluster/kubeadm/kubeadm-certs). +For more details about certificate expiration and renewal see the [certificate management documentation](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/). {{< tabs name="tab-certs-renew" >}} {{< tab name="renew" include="generated/kubeadm_alpha_certs_renew.md" />}} @@ -44,7 +44,7 @@ to enable the automatic copy of certificates when joining additional control-pla ## kubeadm alpha certs check-expiration {#cmd-certs-check-expiration} This command checks expiration for the certificates in the local PKI managed by kubeadm. -For more details about certificate expiration and renewal see the [certificate management documentation](docs/tasks/administer-cluster/kubeadm/kubeadm-certs). +For more details about certificate expiration and renewal see the [certificate management documentation](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/). {{< tabs name="tab-certs-check-expiration" >}} {{< tab name="check-expiration" include="generated/kubeadm_alpha_certs_check-expiration.md" />}} @@ -75,7 +75,7 @@ to enable the DynamicKubeletConfiguration feature. The subcommand `pivot` can be used to convert a static Pod-hosted control plane into a self-hosted one. -[Documentation](/docs/setup/independent/self-hosting) +[Documentation](/docs/setup/production-environment/tools/kubeadm/self-hosting/) {{< tabs name="selfhosting" >}} {{< tab name="selfhosting" include="generated/kubeadm_alpha_selfhosting.md" />}} From cc64c39bac8af9655fe12cd3cbe81ce62ab8b58a Mon Sep 17 00:00:00 2001 From: Eduardo Roldan Date: Tue, 6 Aug 2019 04:49:55 -0300 Subject: [PATCH 15/49] Document the enableServiceLinks flag (#15611) * Document the enableServiceLinks flag There are many cases of service discovery environment variables clashing with env vars expected by programs, causing hard to debug problems. The enableServiceLinks flag added in 1.13 should be mentioned here and not only on the API reference. * Update connect-applications-service.md * Update content/en/docs/concepts/services-networking/connect-applications-service.md Co-Authored-By: Qiming * rephrased the action to use active voice style --- .../services-networking/connect-applications-service.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/en/docs/concepts/services-networking/connect-applications-service.md b/content/en/docs/concepts/services-networking/connect-applications-service.md index 073e83abdd..96d7cc195b 100644 --- a/content/en/docs/concepts/services-networking/connect-applications-service.md +++ b/content/en/docs/concepts/services-networking/connect-applications-service.md @@ -134,6 +134,12 @@ about the [service proxy](/docs/concepts/services-networking/service/#virtual-ip Kubernetes supports 2 primary modes of finding a Service - environment variables and DNS. The former works out of the box while the latter requires the [CoreDNS cluster addon](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/coredns). +{{< note >}} +If the service environment variables are not desired (because possible clashing with expected program ones, +too many variables to process, only using DNS, etc) you can disable this mode by setting the `enableServiceLinks` +flag to `false` on the [pod spec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core). +{{< /note >}} + ### Environment Variables From 779b2be65cbb0e2dc34dde8058ba5deb356bdfcc Mon Sep 17 00:00:00 2001 From: Ben Hall Date: Tue, 6 Aug 2019 19:03:55 +0100 Subject: [PATCH 16/49] Update Interactive Terminal button. Add button directly to port 30000 for dashboard. Use new Start script to launch minikube. (#15689) --- layouts/shortcodes/kat-button | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/kat-button b/layouts/shortcodes/kat-button index 55ebc70b93..3165e30150 100644 --- a/layouts/shortcodes/kat-button +++ b/layouts/shortcodes/kat-button @@ -1,3 +1,3 @@ -
+
From 6f25d24cab6939c7c2cf1df8b82766290b5eaeb9 Mon Sep 17 00:00:00 2001 From: Rita Zhang Date: Tue, 6 Aug 2019 11:27:57 -0700 Subject: [PATCH 17/49] Add OPA Gatekeeper blog post (#14487) * Add OPA Gatekeeper blog post * address comments * Apply suggestions from code review Co-Authored-By: Tim Bannister * address more comments * Update based on beta release * Update publish date; reword intro --- ...er-Policy-and-Governance-for-Kubernetes.md | 194 ++++++++++++++++++ .../blog/2019-08-06-opa-gatekeeper/v3.png | Bin 0 -> 93540 bytes 2 files changed, 194 insertions(+) create mode 100644 content/en/blog/_posts/2019-08-06-OPA-Gatekeeper-Policy-and-Governance-for-Kubernetes.md create mode 100644 static/images/blog/2019-08-06-opa-gatekeeper/v3.png diff --git a/content/en/blog/_posts/2019-08-06-OPA-Gatekeeper-Policy-and-Governance-for-Kubernetes.md b/content/en/blog/_posts/2019-08-06-OPA-Gatekeeper-Policy-and-Governance-for-Kubernetes.md new file mode 100644 index 0000000000..b4aa017d59 --- /dev/null +++ b/content/en/blog/_posts/2019-08-06-OPA-Gatekeeper-Policy-and-Governance-for-Kubernetes.md @@ -0,0 +1,194 @@ + +--- +layout: blog +title: "OPA Gatekeeper: Policy and Governance for Kubernetes" +date: 2019-08-06 +slug: OPA-Gatekeeper-Policy-and-Governance-for-Kubernetes +--- + +**Authors:** Rita Zhang (Microsoft), Max Smythe (Google), Craig Hooper (Commonwealth Bank AU), Tim Hinrichs (Styra), Lachie Evenson (Microsoft), Torin Sandall (Styra) + +The [Open Policy Agent Gatekeeper](https://github.com/open-policy-agent/gatekeeper) project can be leveraged to help enforce policies and strengthen governance in your Kubernetes environment. In this post, we will walk through the goals, history, and current state of the project. + +The following recordings from the Kubecon EU 2019 sessions are a great starting place in working with Gatekeeper: + +* [Intro: Open Policy Agent Gatekeeper](https://youtu.be/Yup1FUc2Qn0) +* [Deep Dive: Open Policy Agent](https://youtu.be/n94_FNhuzy4) + +## Motivations + +If your organization has been operating Kubernetes, you probably have been looking for ways to control what end-users can do on the cluster and ways to ensure that clusters are in compliance with company policies. These policies may be there to meet governance and legal requirements or to enforce best practices and organizational conventions. With Kubernetes, how do you ensure compliance without sacrificing development agility and operational independence? + +For example, you can enforce policies like: + +* All images must be from approved repositories +* All ingress hostnames must be globally unique +* All pods must have resource limits +* All namespaces must have a label that lists a point-of-contact + +Kubernetes allows decoupling policy decisions from the API server by means of [admission controller webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to intercept admission requests before they are persisted as objects in Kubernetes. [Gatekeeper](https://github.com/open-policy-agent/gatekeeper) was created to enable users to customize admission control via configuration, not code and to bring awareness of the cluster’s state, not just the single object under evaluation at admission time. Gatekeeper is a customizable admission webhook for Kubernetes that enforces policies executed by the [Open Policy Agent (OPA)](https://www.openpolicyagent.org), a policy engine for Cloud Native environments hosted by CNCF. + +## Evolution + +Before we dive into the current state of Gatekeeper, let’s take a look at how the Gatekeeper project has evolved. + +* Gatekeeper v1.0 - Uses OPA as the admission controller with the kube-mgmt sidecar enforcing configmap-based policies. It provides validating and mutating admission control. Donated by Styra. +* Gatekeeper v2.0 - Uses Kubernetes policy controller as the admission controller with OPA and kube-mgmt sidecars enforcing configmap-based policies. It provides validating and mutating admission control and audit functionality. Donated by Microsoft. + * Gatekeeper v3.0 - The admission controller is integrated with the [OPA Constraint Framework](https://github.com/open-policy-agent/frameworks/tree/master/constraint) to enforce CRD-based policies and allow declaratively configured policies to be reliably shareable. Built with kubebuilder, it provides validating and, eventually, mutating (to be implemented) admission control and audit functionality. This enables the creation of policy templates for [Rego](https://www.openpolicyagent.org/docs/latest/how-do-i-write-policies/) policies, creation of policies as CRDs, and storage of audit results on policy CRDs. This project is a collaboration between Google, Microsoft, Red Hat, and Styra. + +![](/images/blog/2019-08-06-opa-gatekeeper/v3.png) + +## Gatekeeper v3.0 Features + +Now let’s take a closer look at the current state of Gatekeeper and how you can leverage all the latest features. Consider an organization that wants to ensure all objects in a cluster have departmental information provided as part of the object’s labels. How can you do this with Gatekeeper? + +### Validating Admission Control + +Once all the Gatekeeper components have been [installed](https://github.com/open-policy-agent/gatekeeper) in your cluster, the API server will trigger the Gatekeeper admission webhook to process the admission request whenever a resource in the cluster is created, updated, or deleted. + +During the validation process, Gatekeeper acts as a bridge between the API server and OPA. The API server will enforce all policies executed by OPA. + +### Policies and Constraints + +With the integration of the OPA Constraint Framework, a Constraint is a declaration that its author wants a system to meet a given set of requirements. Each Constraint is written with Rego, a declarative query language used by OPA to enumerate instances of data that violate the expected state of the system. All Constraints are evaluated as a logical AND. If one Constraint is not satisfied, then the whole request is rejected. + +Before defining a Constraint, you need to create a Constraint Template that allows people to declare new Constraints. Each template describes both the Rego logic that enforces the Constraint and the schema for the Constraint, which includes the schema of the CRD and the parameters that can be passed into a Constraint, much like arguments to a function. + +For example, here is a Constraint template CRD that requires certain labels to be present on an arbitrary object. + +```yaml +apiVersion: templates.gatekeeper.sh/v1beta1 +kind: ConstraintTemplate +metadata: + name: k8srequiredlabels +spec: + crd: + spec: + names: + kind: K8sRequiredLabels + listKind: K8sRequiredLabelsList + plural: k8srequiredlabels + singular: k8srequiredlabels + validation: + # Schema for the `parameters` field + openAPIV3Schema: + properties: + labels: + type: array + items: string + targets: + - target: admission.k8s.gatekeeper.sh + rego: | + package k8srequiredlabels + + deny[{"msg": msg, "details": {"missing_labels": missing}}] { + provided := {label | input.review.object.metadata.labels[label]} + required := {label | label := input.parameters.labels[_]} + missing := required - provided + count(missing) > 0 + msg := sprintf("you must provide labels: %v", [missing]) + } +``` + +Once a Constraint template has been deployed in the cluster, an admin can now create individual Constraint CRDs as defined by the Constraint template. For example, here is a Constraint CRD that requires the label `hr` to be present on all namespaces. + +```yaml +apiVersion: constraints.gatekeeper.sh/v1beta1 +kind: K8sRequiredLabels +metadata: + name: ns-must-have-hr +spec: + match: + kinds: + - apiGroups: [""] + kinds: ["Namespace"] + parameters: + labels: ["hr"] +``` + +Similarly, another Constraint CRD that requires the label `finance` to be present on all namespaces can easily be created from the same Constraint template. + +```yaml +apiVersion: constraints.gatekeeper.sh/v1beta1 +kind: K8sRequiredLabels +metadata: + name: ns-must-have-finance +spec: + match: + kinds: + - apiGroups: [""] + kinds: ["Namespace"] + parameters: + labels: ["finance"] +``` + +As you can see, with the Constraint framework, we can reliably share Regos via the Constraint templates, define the scope of enforcement with the match field, and provide user-defined parameters to the Constraints to create customized behavior for each Constraint. + +### Audit + +The audit functionality enables periodic evaluations of replicated resources against the Constraints enforced in the cluster to detect pre-existing misconfigurations. Gatekeeper stores audit results as `violations` listed in the `status` field of the relevant Constraint. + +```yaml +apiVersion: constraints.gatekeeper.sh/v1beta1 +kind: K8sRequiredLabels +metadata: + name: ns-must-have-hr +spec: + match: + kinds: + - apiGroups: [""] + kinds: ["Namespace"] + parameters: + labels: ["hr"] +status: + auditTimestamp: "2019-08-06T01:46:13Z" + byPod: + - enforced: true + id: gatekeeper-controller-manager-0 + violations: + - enforcementAction: deny + kind: Namespace + message: 'you must provide labels: {"hr"}' + name: default + - enforcementAction: deny + kind: Namespace + message: 'you must provide labels: {"hr"}' + name: gatekeeper-system + - enforcementAction: deny + kind: Namespace + message: 'you must provide labels: {"hr"}' + name: kube-public + - enforcementAction: deny + kind: Namespace + message: 'you must provide labels: {"hr"}' + name: kube-system +``` + +### Data Replication + +Audit requires replication of Kubernetes resources into OPA before they can be evaluated against the enforced Constraints. Data replication is also required by Constraints that need access to objects in the cluster other than the object under evaluation. For example, a Constraint that enforces uniqueness of ingress hostname must have access to all other ingresses in the cluster. + +To configure Kubernetes data to be replicated, create a sync config resource with the resources to be replicated into OPA. For example, the below configuration replicates all namespace and pod resources to OPA. + +```yaml +apiVersion: config.gatekeeper.sh/v1alpha1 +kind: Config +metadata: + name: config + namespace: "gatekeeper-system" +spec: + sync: + syncOnly: + - group: "" + version: "v1" + kind: "Namespace" + - group: "" + version: "v1" + kind: "Pod" +``` + +## Planned for Future + +The community behind the Gatekeeper project will be focusing on providing mutating admission control to support mutation scenarios (for example: annotate objects automatically with departmental information when creating a new resource), support external data to inject context external to the cluster into the admission decisions, support dry run to see impact of a policy on existing resources in the cluster before enforcing it, and more audit functionalities. + +If you are interested in learning more about the project, check out the [Gatekeeper](https://github.com/open-policy-agent/gatekeeper) repo. If you are interested in helping define the direction of Gatekeeper, join the [#kubernetes-policy](https://openpolicyagent.slack.com/messages/CDTN970AX) channel on OPA Slack, and join our [weekly meetings](https://docs.google.com/document/d/1A1-Q-1OMw3QODs1wT6eqfLTagcGmgzAJAjJihiO3T48/edit) to discuss development, issues, use cases, etc. diff --git a/static/images/blog/2019-08-06-opa-gatekeeper/v3.png b/static/images/blog/2019-08-06-opa-gatekeeper/v3.png new file mode 100644 index 0000000000000000000000000000000000000000..8752726703b6a3f600ec248542b4c80ddef41d8b GIT binary patch literal 93540 zcmZ_0Wk6h8(lw0JxVyW%yF+k-2X}W18r&^G8;20wT>`<~-K}wlpg})!?_}ngx$pP0 zyU#vn@2c8suUb{7BUO}Skl^v*!N9ecl9JF8^+2;2^1yAr2>G*2GKT(BIwvjMX?;a+cKOnH|c(yJDaZX z(0c4>)b(LmWB#bS+^`T`F>8ZBulEI8QX=*zrC2b~NELl1#4KS{Vei-0lVJyktHrn= z6~_?e&X$is73=;}w*R>+kS}NB{PFRHw_{@>7@1rYDgq3R6hblpfC~QSLn1K14!tQh zUHbp=o1`eIC@7g2>d&?RXCUwz>Rp@CgRUeJ<$vGruWQh&t`Yz9h<^|3{6-@Ra-ZHH zw;}!CX}n!S3lKYlf(Qi5R4TgIzw~rr32^z}d$2MAiq+|HW2Lv0^QRP|g%#(OK6@sw zrir}%ny6l~V>2FxI+~nR>E=SX@@`MibCdhWqQH{m%oPUl5yKW+XdU_{$|_so+41nY z_8vx^INWozs6};kb*HAM$)9Rmb0%OLbh0kVIMNj8ocXhuwf2&-|BvB*uPV@5Jir<- zMbGhTX3ObYZ_AoP-o8R7vtBdmiR*fs*W~fdOvvE9yJ>&K!uq;_Sox=)Lp04!TM&yh z6IZlMOj7RdT-c1-q)%h%T((bEYD53F)W6&`Y6W4pv^7BD&-hdm$>}lfHuUuC%2d3u zwidmwua6^Lt#w@4_R{<5-Z@jqM-#P_h={0rR?r7^aWITl8v!1E+BuK)AI|G+#<+NV z?%u)AFfyIbRTyp6V;;>C(3)ggFz{hYVKEdwI|agHDF|FH&&&kht^1&y|9E5u8kzHC zL~7h>X=qRZpOr!hB<%m!@kV=SfbgCmpt*Po-16OqKM55TRr6LDfz$KNLXS_ueR;ST+0BaGUSk!UBT7LiM!RKimY2f@0Qh4e9Tf-ipuS03QiC0rf=}nSAV@ z-J@eNfQ5lEsVlcr4_b=km>PB{T(Tp^z#Ufj=Uo_~7F)HcP0~US4`+&?PbtRxv9Yn0 zQY_+Z1q#1;3Vl4&vr9?#I!1>+PN0(S$HK-^`o}$*SD_YrvuMqNyNv9(Z;#5Veyl1I zEIbq?d79g58~U7bZAp)13kg`@3tEn6I)_SzV*Gn+{qZI>0eB-Iha_U$+Ur{jty_zbZ9PzWsXe@q704W*RDPjyhB zEb^?5T1~)hhw7o5)~zpR{qPN^3Z!VHw#~EQTSuyASsPxFB=8RW&lLW%@4-ex?&q`& zu_RBDwG8MY77OH4l?b>KHQQ>L9D$J+x{mG8Xo6SeT&r6*-o{NUY`na*Pm%i|n(t|D z|GOJSp{Sz*#PdYO!~!=}G#}`)>#|gT`c+4Y6c#K>ZkB44L7@o{QG?MGcnW!5yM_8i34!d1d58HRv*t-5ST`$@;rV+uOjii z2qe`>#s7N5J2tLcf_e*8#w+Pj zYM83_+vOT;pASEk@h=~Kj<9UMnH#NeCQYJ{%XXd;%s9IqnM62-LWY3V=zB=D2Xat9 zGyr>+yqrn?ojlGz+!k0ODY|zYy!s1`v(j}Fc0!_vC+{W9{QksJB-;0+lKh6M7wTl~ z^mIoAV_(cBgcF~ULnB?_drtPJlWa}r)7F{zq*)~Lo zo0bya;;hj>QQ{qxB$=%fF>ITkc9TaesR;rCg3K_#UE9H=_4#^x^I>7e#iRrZQpTrR zDFctohpU`dLL?-l`)-^c0pHJB8T!Km%{eMyQJ&E}!eJ|e2vBKDVXHX%rYGV40sThH z=$kz%%P&y*+eF2DBEn5)bwQy=+GCNg7vUmCcV}zb%}!)5XXh`--tC{{NZaf{$z#PY z`~EL9>;A8PzNfZ^7;Q$8y+9#uj7lLt4_-w|9VzB(+5g!+oud?;p?)g2?U0FAM3`xu z+I4$%&8Lkk{ZL{*odeJkjH=%S8SM_hOtRuvbPZX0>VIHmtU#lK^M^h?01 zA6nSBp%?|qrf$4YP*hY@FBG4F-|OQB>`E=?63+X$z8Kq|a2QqRZCA*j9%gbtZ}A}5 zXt#uZoJ4CCTv)bMaZlelWa8}hPf#O;zyn;69q7Ceu}>5&Hrf+ZcDy_w!qp*QXC1O9 z?!c1xvVK><0NRN69-sTa`mlZ+eLGzIySqWbuqdE~ju$Qnsf8Y`A6^jO6*bWQ0ltN> z;HRNEV#2dQV;MZAJz+?uN3+LO9dS4Yp5R*a#-;UWQ!=L)a^=p6f&)Uo!W+z>`YMm8Brpvq^WKmXM~Cw+IaR2PIn-FR`AZd)12_0oeP(#zuiOCD-$ zq1(T2|NrL^Ef^Qc>w%r+$@fFbpMu86$5#IxGtt|-| zkq*!TqH3n2Y63$}6BgYESzl`#!s)618;=vS0vyM)HiINQL6PSs197j=i9?)6)C&s> zJr;FaZ?GdHD+`Csq{Des`!m^DZ@$f2kl2_Y)9Y$)t7d>$-s<@FbY*JIHr#cqscU2w z&bdX;DNu|SRtere^A4T20nViX&!qv5T@l>O#+145{j#hy=?RhKpry>Bt<0b;EQwR# z!dT#j3kL*rLx2L;kdDF_+5b#1IN-M1-$bkgKK}JeB(_ED`^WQsl9vlu-@ES{&@&%X zuW)fLA0}R(CRAuK?-%@^7eanK-w%lNm#7tORy^SYKWI3fY3himJnH-PJXi-j4gw9nOT(F-Kc2QmjoN8Ko&E+EA^ zquTGs96tp^!XkHSdEh{#5cF+k3S-@(+q!s7O~S_&eCi?k63%QDv`1- zBImdhLDXp~JS)9N`R|=9N(?tBn8o_)g(KW_%duE*Jp+_O4)3{7_5XzvB2HEvDVL?h zQ}|OlgRcrX1Bg)=FY^t3g1rg66+{<3)`+pVH;kx8`{>r`2`6sEwSa zKy4-XeJ#%L34~)xXQSYCMQxDAV7?c7bZq#_Pft&ej^AhZjf>A-s_n<~?aKYy`6t*$s_~z@BN#esv$MY$=%OM+*GL1L z^O7Fx`);pR*5i5q(9&k=48|l27_>=^b(oRi)6J*gNxvr%OO=U|vj0B3jm#kYmc7N| z+4Q;RO}m1i?~glPYzc+DFjy|A6T&=$hI{A z*o<9WJU{CUk7Ie(Bfg*iAb7kx$C13$UQV%HXN%UdkPz~J@m+NuAfj`x!_z9fAlh3U z3r+SL$i?Ml0*10Xi^HPf#s%o_KM?i7|4#aI5kBESd26gw7KNx z<&DrX@RG8+*3i;Vo?d^;Sd=epcIgTG%57hLRIt?p_;&Jw0XyQNBK}0YPfdm- z83%FUBEs)=zvTxUzYo~@yRxz}gw$36GUHe3G8=jg&q_=}#~o{#S>z?fGT z7E7<7`Xcp*mJw1pR;n0JVo@%sMz><+|} z2#C~bS&o<~Qr>TvusdbA;va=N@g(MpTu4bt0VYzH&864q4=8hW6)C%elGr3T#j$ZA zint|=Y4Paj`Y-~G;b;|HLp;-}92ea%{|A*V(7|(3DtyMy=MxkN*ui}+HpMapK02&A z?NJ&ml^-`O7(~C(bi|chR}yYAGd)(jbhj^hlW;Blnaok$5u!>^X6X=QLNXvO8iyp{ z|Fw&eHwGZRGHh%<{Py;?%_GTL()oO$4V#gU(f0D;cExdfG>xMt9ih%eJcuBhRIu0*z-}7jr2U8N4_Av$uER!Dg)ClZidP3;Bmi%bVzYv6e0*8rU9` zYs(h#H&B`6S#|1(us1^OmYazKl`^#w-S9H34K_8+`bivz1-4^-rj8|dPs$nZ1AXWx*~1SQk$zmWyw zHCk$VR*~*FV+K> zKtH1QZMS!K2v~K88+LKRD`4GnwF|>xSOtIcg+M{{YA>gH0Vsdg7-iv~phQ7zMX5g_ z2vmXs8r0pyPmD5bQ>G(xFr`gMu&Ej=u#K3p=Ydlcm|sE$dt5{@)XK_P|7|!U7&Kaz zDI;TNw^2^aewhjbxBEKC+}dL2d=2De!V4b=-evrEiEI%$f(yYp-y+e1LtNx`Rp7vl zeP_?4!qsBNE=p95J|x43x?oS_7mY+y6XcLN+pr$rwUHAmjtL`vlinv(71jz4-2=#U z(1d){@f$>^=zL(;fKxTyavW$b&gLd3v8DU!wgk`#N6YBstJO6 zw7%oFgMa~`RmUg5(y25+>Zx(Iva?emqmKG_lqt|dG>jlz#A5Ml&y|cFS(9`pb~xus zplkeC4a#b&={0Vfe60Y`v7=Bw|HoRJ@`@$1!mt?@Amttt7kf|v$i5A zl+om_IiZjaKcg|1GN5ioQKJ}W>l0pNu;vxS@+$p6eYL4po6G? zSqTN)z*?M_Flgp5WD4kC_Mwm{Ex=rdJqP5jUAKinqXc+-HM3;6TzLxXPMuO>|p8I zOW5I-sl1_r5xl!;b}|=lYv-Y_pR@5bmwm`F)JSF{*8{%~4`K`S8F^S*Y%w6$>k)kF zvRY%uOPRyjYGIKVAhl0L{J72`z^u3A*q}xKP+)zS`S-dlAi&>O#iq_s_kXs2*OjOFTAb*|d-H*Koo$fH@SE5B9FYZ_bGg9$B9Y@)hoJ1uU>a`s$jkYr zf+b|@3RUw#x(-~qtU5+Ab&wlg?zAR9%2YSUC$KjIrRYfsrIqc-Rl`p2aEuJDai-@^ zyJk8FL@}bZ=q{247#H>WEKJSshp`dXmyG3sd?}4<=kS}8bST0Py|MYMn+1?FE4g{e z>+B2WPq1^bsuQb{?A{3Rt}d>jXf6=ekT{BKVB@s^?2iGaD%H2Kv7L0YcC%18sz{v(uX7+ca}S3M+Cu8cyYV@o=GHjAdr;a>(JR#3#{My@<5Eer9n??sSe2JBn+JiPd@-{& z^>__2Qxl-XNU{}BD;eVk>S+9?B-!n>)9zPPe9DA{C002x(H^Cd_GD;G({NMZ7d-8W zeP7$HXy)Wr%_0XPEJ@mDl)E)T#B%;lB3O-~fS%^*?1n3QW>0-0u6u}!Fm*0orTI~C z$ETE)v|Y3em|k9v2G!R&lgCV{6HuUcF3CF0_-U{m?n!4K|`&@}XX$ zc79eapmrJkdlzU|J*eu2Qk2!aE=U4TL|%!5``KU|6#gm9M{mts&Fct6z8{`iXb!T> zsG_8Ud4dlJrPI=FI0I2ehg2I=(^6!+*~eI#h8baojT&4;qj+}i9TVz>-2RC3MuKpxOhcShbq8y0URS^} zbpKOy_*S<=bbbK==exC!G$Aum{1Ae-r-yxCptul3^Ey)+q>5SM+E|n3fGYm2D6vC_ z(^zSy+B48B4&1i6I}*IV_MMAK2X0kde#c3GsAxgYxKB)V6t6E?X7grGfI^(PhD}oH zp0yJv)Vu!M-epkEn$qek#D$|Ft5NoWy>@@zWVvYGQp7SoWI8_VMOJZ>L2@s(wL!8p zYX4>TbS4)4`xM4(4v(?$4S&pT3N9DX!rr}Icupb|aSc<<5MG8g_o+~cDI3ITL+c83 z9_FMN%|dHG^r; zOf7Zw-tGfR4sarZ@%Rdn*g^!Dntf3bMqS2YY=s|Y)!*b{@k$vqss|~^7uYaGT|YWN z)bF=JDrd&?-VUzHb?SVTE+Bi*#D>pn8H@koiljyUH!Ww!05&P$U{qFClC5^EkS)cl z2ZoS^t z!Lf_vZp5D5*q^Xs&(u2;ZNhn6+Q?a)%t9oX*S(nhI%sEcD$eCJ6jgfsE>sTISoeV*{MPUL43jdZo4nv**IgB zr}aVJRf0;I1jFOPwpTq=fpw~)_I2y*%Xp?jVl z8+#bSP$o+yDxnL53PXP&^sRt)x` zPppem_QU7*;WR(&q`l%OUU+!AYU(|ES=pZgAFWa{}ITBz*|lWr{us;w@Q7fHxlhi_6E#6q%%V5Upk-JoJWPALjmG?#* zP4;n@X-C7Bip=Urc*Ar^g$7*{ZA(hbLLoHtqB>rV!|9>ZR@#O^ch7`=_qa3<;S`at z0ydT47_(c{g1OG@H@XJ966CMn>WN4G^a5bW>wZTVvz}J@i*<`qUKGXmvB=<^wSA{` zJ}rJ9eZEG5lWUT_qimZD$0Dc_g5-*&Qo@Uwa&Q5)LQQkKhTk0x%i);VGyK>u1g))^ zH!vP!p9#$UGuIEd*b;c0{2^O?8|(KT>avUC>(Li zQUuKzQ^c;a!=ux}iv>-(H`G`u}~o5IsEbB@+Y}nStPJ#j*+8lnLmQK zCbL`wtJ%@ox9%}zoBPBL+Y%!naUhTb8+V_u-pC+Q`C)L$&e!c zolpz}BC}<3harwW4XA7Lms$?{4ErBGS8p;f*Mo&}Wb=jRpgZ}sj# z4%pv+D8WKuiec4l9*o(?PTTq!nx{pB>{J?Z)8e(PD_Dc)CiO2>%4ik}n^})`$KKky z%Y8wg=g0-!{|3yMtU{*(hR=35g)Y2x*J8D4IEINqVX-zUivS~(fuvi$N&spjUb771 zE1T1VmycevT&XP|Gv19>S#^xPRnBpqT0WyqAr7t^4PZh0y#tR2F|MRM=jIL^O7Bou z6m(c`Lo=24tTX}fTVmWx#1Q&-DF_b$b&>UI-y_ie^_lhRf*~HT;Djp^|9uK07XN2& zg%T(<^zv!s_MXv75CRZZ{n|<8yRQs7spM4ERwY2VUvgc$pK^9wjo~{?otQiBi_bpo zi_}I2Ya|oA?=c{3rL|5Kr`PMT;3d+gNApyBf>Ba} zJK0zW4&QLfrxAupBh}OQAj$SSWAS__ALOrzS?zN-`SkC?@tXi@QHN)4Zf*o}XO!u_ zXJd#+L#%^taN7wFs8|RVf+b`N)E1HOGbiQkOK*paiODOd!1xp^p-G%@#e9{n(QZZL2AE36{jV=IIczw-nN zy9cvt9c|d(IbcL5FKTgVskp7}F;Gqkt!jTFXW*>?rC72V-@DgPF3U#J_tf zTSjx|@+3wmlyPmIt^QNefW2nf*_qt-Tk9WHDkiHl2pYlZWU+<~D~Z%#Wo2dS?y(Dd zcKVADxNa&*)`#>dm~Mr8DMFij4(;zX>5BO8cYj)0PdpdM+63a@zRl7AW;tw9wMx1H zelq1q1(t`ODlVO0S3h6dKSK}U_YB{N-+!c9)Z=B@X|mbtMJ?Ji0Fk1Ol;D_$Z=Q zK|(?rji@}c{FYGd-I7>CYX2_G*WSJ!b{ zY!$`KO%Qj}vD3GdZlH|FYRvTL!A2_kF&@);&?ubkz=NMSJx%pn@gd!S3dN&L|Bt0rCf(IAf6F&e^dZtj*l?ufM_P~A7K8*IT=x%k@Ru!>4{(4`{v zY?4*Ay*#+WG<*;-XYVqt^o5n+mje5GV5sn4`F1`jtCI&R%8LvS@v2Zn-20s(#=jGy zHz#%OQ0C4YZo0*-rx|s-Wv-l3U-Cyi5~ijvnG3sAlgxemph7WfHLTLn26i)|cD1}r zxHy;oX(c#hmJLII61d1Wj-c1$Ck#KuzgGI^>2DZE`C02yj-v{XFWyuVcdP=VE~UP% zCwiRG$3)*o^n+WK_8X&N`mmR8_P34xGZHn>4XR#;slrPUHmuu)CYN!_*pzSbRLi7? z+?k9iJ`7`h6(OJSAEUtGz`3uE?jA!$lIgW3#6*L7s4ckd*s1QblGUY(X0ier)T5KR z;Nqd|PL&ALfRS5=aj6e7|91n6LRoNvQyVOG2_o0|8M+FYAZG-1<>m%hmu2dqDdk*a zLDvj`1K2E;lmy*5>t>1%XIzi|h!OwBf^9`ndIK3XExYvCijU^7kqc8snZ#cB$ai0j zz1>2+|F>R6FBTnw$fWMc!{!zlMj8<96!X}6>UBxD0TP8fZ4l z;FPG?egI6Fr*nvuXJSOYMg1K)xbT5k^D$OHV(>z0kCq!oP^dfx#uV0{3CT(t!9;4| zaaXW(&*2kysd#)8CEMRTaW)>PDQRCQDH#rNWYz@t#SJ$&moYWd`gTI9LQl@x2XHbL zQP-~uf+O{CazWa8FmWm!#E+(%&yro!6W)<5K3BXDtC}1)>N2?>K7(M1w-e2_-Ck-} z``8BP+kBkV@6(|A8Ff(SV6aQ7J5D%j{9)0K?}U6+eo-VB+j5) z@VoXKN<`acXMA0edNfG2()5@Y%An1tKV8Rr+R~VJ66%kh~w) zVRd3tV2jmzfpD}iN@A{;3zJEVftTrHo0#crsj|>!GNP3XVU-|FE4gDMksHI#KEs}T zddYW&Y5L$`^*JX&3a0(#x<4}9iRFNX5D&kOJ71WIo_va#iNgGE?SEAS$o+v6R%r;1 zcWp2=K8B9)mC3dirfPJs0RsdqH=uW{RB+IgXc1}pu=gQ>0pHBtk~ZoXFdP?&nHm|O zH5b#1BZel6M$$XcLiVrKFUGtgPe@K%mQMPQW}0yUvu`nQb(g=wBV@2YD=`K_X-3m-7x>YDl z9>R*<*JAVNT=Yonq?ppK87Ros>{$?8^Y518$Vk$^Q>k@MyTR=fmX6F-7gK<~*NsGv zzpu2^NXIH>(j+(0$TZ*Bi6~UIQtu9(E*H125lmpll-3PL5#kj6+!?{#uOzzmi=CG` zGsTg>$5_1^2!~tsi`^IUw0qgK5yqoA7=b_ULG1ca2hZmkbr&q+ zc9Gib>RmqqkL)j^thP0$b^Z0bw{FeVR~IVt_AzrZG?Ju9;AtHsQC%FY`ge1jHJENK z0NkqOfQ!Rd@!WgtkhwZ`Zi&5L(Wpgq`YCKkQVN3LbLG$%9z?KL(_!QPgHn|@RWZDYY3Ax!@$g+g;qv8^>=&J9A)QMrDrr|f$zUmz5`1^C(2&^+R5PqL)p#a2deE0^}KE!^?9_8rMGLD z3F>s^&VCTe6A)sUSo@CE7(Q&AbUqjC?AoXb+O2b?7YPT`TUn8l1Ta|TM_V|$OQ zKPrX6I>aMr#g9c>qO44HE4X68+i7bAaP2D04Obi6lcgKriYkZwrT z8fDct*i6~H{7={-SZoVDf;#Ep7t_{FM$;p`yM7Evv>409oex;}nSy&gFHJy7aCM5*omE)1nyY_mT#1 z)i)#T5_D$SC3FU9?~M`jsgP@p?uR{^kk z0NB-wPY=v(_OH;3H%VD_@nin&ofuNTqG%QRyUHu@fed?&J{tu$E=M6!iQxJnvYYVzMl z5|xqf(PfMAWH0W=@Cn=S&c_cYEERKr4b~SXp%`!c5GIVQnAn~2^aZ1N9KAQL*Ha56 zb{`PrrK{OdJhSQ95}RGEl)2gh=X}RW$#Lwr*Cd$TwgBM9F=6Kj9M3OcA`ffermz7f zmZ{(>+DC%CZ!8hTi8Zx+uE0D9;|WY*PYg~TvQTt?sK8Hi0o?Y>hhKDrte z9;6?2gq9Y2$55%>9=;1mWat_i3rcmJ7qPJKyiL9J$OAi}+g^wVeI~FJLCeb2N?Rn}waEN%c za|cjIBNH4eF4hJ;=1}N5WO%$Y$-oKkC;vz zpIPPb>8Xir(>QaxapOZMJXN#!{kDfvWqJb0^5>Ut*yi%0Grtl@^^g!c05e~GWScDz z;-=xCc66ev^^=UZvQ%z!0$x^MaWU|?gyKy>JTn~fIWkhj*#n2&iAfUu0wtpEdCj;Q zadS;rxy*{oMiIeW^q{2l2(2}!E;`u7Oz|K~tu`-?BqL2=r5gmhn$!5=k>zrB(&&s36z+PKzXw84LCpNN&t~b8M3xqC?KSIv?bnR`qUMr-=dK8H2d*%=e^b}Y4o%>o z@A5tOKvvuHlreYo9WWkAg@12~soR_1-V^RX5R;<`dX#q(c6RI(9i2o$ihNYy(5U~)9dNV3GS*aYf56#)5OTw6Le2r2M2aFq?X-Si zowp0B`$ZL!9BM=yDZ{_W@9#cApz;B`Q>ESb+YY8OR4sL9+s};DPeiwB9bhaoorXVK z%CD4)LMRP5*2hS7U_57&STFnXPxHwL6L`6Xf%fF&;^-P#>KkKNUpIHArqh-P7j*Js z#u6HN1kghm7;4`HoufVMd~e{Xb3!T1>Zj{_aEy^{guKhqNpqYlI{R<`az9WLYEhH7 z6omgqNq*KJ;kaA4piZV#1ynWR`M&1VqE`(EVtl4cVx3>*;^P;sc)NVIZvhb3H|YTt z_<^B|0T{lJ<{neu>S8hV?#N!giuQ%q?!Jw0Pl|^Q?+BYfC0`zysKd5GBYe&}lZ|df zv01|n7{b~behE7MqTordO2|^vXnAQRNu<%|gD?4Ah`v03<)(N0n8O?Cc(pp76!8mpxf*OVWrE;^Y@7Y&q8{lYpBJhYSTO6rR# z8L96^D5MG^cWeQWE`tXll->N1doS!`wo5M6kuweqQueA|L^74LnOrmsid|;xW95wL z@xEeB+9;crpelGKA;EcOY{iIs^${;E7W*s{Q|D{5B7tpi{u6i4o;c937uEDxWdOn*`!cyCqMHRfQ+Sjo>?UodHjBGte@dnAoR1N%#10ifXZL?mU z{FR58S^cO#t+Jovz*fQ}D|3+E8fs^x)Ns)T5x{SKYxWh`JZ%<#VnP;?CJyB?eRuw= zsjXjJ^l4m;e>~>*vQkIG!rR@&PM-Xg1b<8c#8#{|-gqHyhPa^MX09TzW(TUO8!kJH zBbiC()*>o3#8PQEudtu?Nj5=7$?ww%KmCVidOAuA=o63aSY}A>+Gia;KJHavf+jC< zr$?}?56BhjBZcauFp`q>PP<>WVz~t%92(;*;8%_*#?ux0bnL$CMhE1;0wrjyTQaZ3 z8592&bKX*vw=ClDo|Ig#<-XI!gO3giZ7>jw1|z#p<@k7W&lj#uu-!+YV4HEVoIy9^ z-CN_SGKAgj+{%qBDIu6y0?zYgxm!Y#;>In-8sIyD^0OfU{&y1-KpuLUQ?Ep z#KxXTY{nt%vf(}nJ>g@gP}44utmrl7?$%7o;URu^wnYa}aBF7%w137223?TYM-PPm zf!HImRoJB%=W#`9#!9p-xknWht@6~~p&n7{Cv=<1%R3Dj{BYD2CCzdn%q8>){o72E z-kvHx(_bHes#Iu|FR1tguop?{M$x;Ie{d?05zgro0Z@&IKcdqUTwbcdd(|qYDDW#DZN3k0RfWj@6s?PqY#9dBVv^w;hX}@GXW5!9yR>tE)VS0}ggn zT7!9)wr~VbsBzfrTHc6(w^CBo5MN+ePrKDSq_}X}!2WGWt0$Gf2Nx`eGt_#1Ja7Is zD`y&b#gw5b{)lkh?WJu?L1dzatx0F-BRP#PX*(*oOs59dfb@AhWqKhDsCFE84I58i zL4_z%{1p(n_B$I_)BO<#+A;dsWZKQ@a22zZ{Eagi*N>_?)=U2`zmpZg464YB?{XAYR#mp!JPQ+X1N7A8Ulxt}WIz}KPV6a}wXr6}Sl(cRu z&10aX<9Ci6Xsy(#qZUDQr+c|$Z}x*H%aIEw3Nbe!ZO`q;NMey=?LiDZj*^Je1av;O zJBQu#)=rBAM-`P_is2J+rfhpr+Cqt2Q$F(7gF?lE3~Bo;Ybc=8@(;m1e5R5|K|6io z6aJmiN8&+~cA*LnK|h6B98FaOi|GPY38gKI>F2^9f6IuS*EFIWRcd?0;l{@(UEG-6 zMG_@S*@X%`#d(rO(iodYbaTd!DdI`77_g0vng92NT?H8k&PX2fGaFnVuKg;u#9~U@7HqCo z*0{{Oz|*Pr7ks308{~`d$?HIjSUToeaPvs)k4!2(UqJyX0LyXSDI+bD8x43)|C2cW z#%&d~0;w+((P43{rDcfCUX>6;Ovr>s5h$Fs6=O&8m_ep(0W1-TmYKZP%Fya{odFMg zj6~r9A|ZRqt5yf^TFdykxEZIuiVtKn5p!pd$SAd331{Ri^TYqu>-2VHI(fl;_H1u^ zD)D$DB85zW2byYK#jqDfsE^1uVm_%@H!!)y5q{78ePs=*;SI_3z#RqJ(926H9AtB+ zZ-w<%&I=6+>fpqNlImOqD=D_iwPWq34bgAFJ8HbGN^E4J?lP-x_1?7{XQHvc&*B*# zp;4XcMN%`>w%Qv|jmuk}1!*VJJKsE1b4yJ@&r6|+aw$&8u*^>z9ibHpgW|}ZPhNZD zUayH|L+{@FtBptiUkbw!tI?iW(mkOHXTA}M8^xn-54G2N2_!D)t^f6dw|fcxO*EpF z;C#FgUN)ZW$KNdhe=&p3B&yH%BZB7N#gP8Odj%wjEM#IyMUXeT%X1ivTl-2~<1RnhuceKuu;DcB8RaEnW+pm#n0@V05}g>5Y8g1s^jMmjC7stM zrzVg|70AZN95X@P%}zHQRc8jC_4KnErABO$D`hboRbZ~gm2nX|+=(ko`vX6yJ;JdT zJ(}>a6l{MQbNX#|r-A40*$5p)nHOst^*08RdSs(~^=^6VHS{7L+Yg(Z9Luj&4Ngv` zw_?JdJs>=AE~_?omTdO66K=aCvrUC?M{qCZcG1==dDAG^;xXp_^`2L1Bv|Ca2vl|{ z)mzOY9RpB+k%+%)7aKu*Ir4G4pL`ZavYW77xNVJCRJH}%oyvta)<*Dw3WNAjUtL`T zt2fkEU7fE%ZdIiuOqKvgZ6@;s*t&nwz;z?`p z3DbfXDD$}-vt!QMrf(U&dgpsXW<-z!f7rJ|c-)kr`WDbsT) zdC6NuOa9ItCu|aT$4C_nL`^^wk7RuM1fe}gziJ1wd}5rdjci{6r|dABTe23;k!4W7 zD7eT{8QTE631$Qyu4qWt4?<>9GPurb>51=G`$|fp<*Uw%Mhfv4td7V9^d6h{DJTD~ z@!7-Ss%t0KbH6{aRnCC?zS3RI9On&ldta@FSve({WeMY4Yr}nnD4M1LT_0)`RT)02 zs20%ho`6 z7?d*sOY-=i3ZDWC1f@aI!t=NF-+uZ4sBVb2<5tARkr6Wl(2Vm>P7*mIs;i;@g1=pS zp86nSz3o+fQ{xW93|bv37fqur^>)F~m{vXXg@RxvwtG>>r2)iz_E)hz(GkG=XQ01$&n3 zhjKOO=yAzM!bcM-aOle2c=`%mEP|&8#*!6c{GFfyJ%0b`!gpOcqPLf%jo48yuD@T7 zc1dzf?|68%DfF^S(PMEXs83}zLt9D&MtrbM;LwpZ;U7Z?%q4mL1X$T-EYsLhF`Z2k z&}ml9IlVnFu3z$+wc+!80re|&aa;~E*;%^>KG#TC&zr35D;BG`7XMHMU;G+vvGyrX zKRiU8@aPz!#yk*xm^``K-)VnvP9;kAU9N=}*;zURtgA}p#aqJEU{-f?=~(|;wXfyc z_*=E#zA!YNf*Go|_Cu?jzlh+ICa5g2Lmk-dse{duwfNOz{Wf{?qCPP~Hc=gQWh z>vmdGZR>}~zIRn`Wz5G+VyR8%BjxACjV%6m^oqiTuAvYNRjOSx=Y2y5MrXj`5q6Fh zYJ2lE#4IL<`WQ5=6b;o{=eNJEa3{hnh#<5ce?2hRH0WGAV&ux z6%QE=JqSp_*nS~QF;+NITXvW=Nn64*oXAQDfC!^9Vly%?5wC7VoyK!7eoYkE!*p0| zOCwKaPr};`nj4xca5*_o5yVaV8AGoZR*}J2jHbtnt8%>}Kj>oTsNT@We(2`;)5Em4 zv`{*>$PhQTD-k!d#697;>SFe2fzqY|kzfJK*j@I)(Y-(}tX7q&G0CQ;&=jCd6c^K6 zhm4CPr~NhGCXL^Wjb25+vS*q5a0F34t1mwCeOxU%mnI8G-eC;UM0VskUs{PU{M(UA z>JR0;?B~R&)zwRP(Ro_9XpnvKdR}+~t*`*k0wghrXp^j8@lwZeE!I|#h;8>@X6yty@X zd5v{5iLhYm7sV&>lWC(rH8~b9R$aQiF(7*;6ryYAEPYjYGbYyC$R8G2bW(YY#WQ<8>17l} z`LJrQbJ(ZW)G=$88#wsVVsu)_|tPpsOFEoK#Qis%Pb?ok%ZApcu0ZERf`H2%7uAarjuFD;F^L z_RXMDJtu9}Fnn+JVE*qBNDo}d_e;q49LRSZNC;804J1dM!0eNP_d`CBw+9Y1U`9Ci zVkm+T%+CH_x)#FXTo9f0`qahKmz}n3Ps~g=MsZ}vWL@EBQ$I8e=9(+;3k;kjPykA@B54+_%cY$NJ zEE{NKoKgSkJRDl(V&PztZ`1xrtewd|{|G&sRYX?R%YzkZ#9Yc>fM)<*HjrEZrm_xS zF8}xC%#csLCTsb53>q3#(FF=G%)d8O zEd@lcq;mL+-q#(YuGXJHIRH2?aW0u7qYo1Flk8-0RX=jAY8BA^oh z%Y)v|J)kP9Fl#~V!racq@PwhoSzjrgK&Pt$5u1vrMNsSS$N-D( zrFp8P8s7WU!+tn(cuKjv{=_=rqL>U!6^wKlAJUE#{W41rzFZeOq8~$)DE-Da4I7q% zS%f!KecH(g+nu|$-j|rCk&T2C@*RBth6`|aH40?A=O3l4pNqgOFB6zNy4b?e1WB1W z&})V|GN0H1*n2jrb|p5x27;hN2SeXqIz)d@@df7mr$%xFWdSo@l)ZRh-g`4Nry7{M zRvM0m3^#dy&U-mlr<52C$+cOG;zq6#uHThb)8+we7@c1TqPAW#!=>wNol&RXGZW&* z-pSA|7N;|qC_*5Css;N;TE@w5s`6HMet#&^rbaXRu3q-(WGDSwzUY@lzT2o}IG=3LXT7E6|S zRhwF7AOHzejDwejkW;P`z9dJMl#_p<2+)z8?(_38%g#;@2nlPm=jUURf|Rq>wA(mN|I*cc;qq;XDv;25W{9>4|kLlxOo} zKq&Dqg@b>CR>eQ9beU;sdz&u^0o~)kxXm>71}CrKY>R`<$Iq0fA28f`tk8uiO6LV_ zhEUQZQu*}Bhb@GZHk*(yURisu4?5(UzR~u3E`i~2nbmkLZzY0#XTb~NSGE>-gHNXL zCP$R!fc=tUqE8sZ{Ed)OU3)0fZ&jLI``}q*m&5ycdO8rIrhxqe(Bu?TYrW!ts(}z4 zJ2(-&Ok;hJnEmd~yn3)v4qSsqXJH6J>j7hDh`G&FZpDnu?$QnP90N=a8%ta03~A7F zG%KQuo7HRQG;BItwcSwYuA4jSj7lNQ`6*{Oi!mM*-xve`=+B12a5SemZW1K|*2^&6X_#2-CN3ScHEetNsuo9{>d~YS zU|;L5yGHF~>y`MeWC)t+%v?epPSat2N5LUdD-KT514_3dO5@t=JXMgEnW&P~acYO~ z2fIO7aS}QQB4GtBVVX{U-AiGW^JBnUu-|{$kPx^d9Iw zC@_^NXw{@v-%91sp-k6ObnP|3fu;OL=k3?D6CF%LMQDg#aJw-Sdxp6znFUUctR?Ud zxn0|{E6>*H#{uQG@hy<=PU5Y$$xyz*!+|{lV8V7lvGF!ztFptXc~`fI@kZAie_Zvg z!3#jltDS=LQt)ua!8wao^DEmNljaXuavIv<#k=<11>3~Q41MUwn~A5Y>y}L2c2II| z;!n_+2I@1&L85K+##K``OjUsiTA(km~o@e)vZiu zHLbq`fD7dN!Rkgy>sN1}%;(vBndmVLInp%uBh=l2a)cp_clO&gDdjpB+?)40t}!a} zZtS8ISO*RqKo)}y@3HXvrl4}|HEjxV)RegMe*B1N$Hva!?5od@5(n??)#_fI@wZDG zWbK-x&?9&m^vS^OUX*<6#$IL%-0nUdOK5_|#Zl^!fbp zMbMm=EqKdJ(X7p}Drg=BNc1O;!r$t_?2SjTgKW6t-Zp;({A&+SFDu9W8JlK1nd@A~ z-tr@Zaz=w6g{!){3Y*27nk>tRsNSZGowd}c<19vo*=c^tDq^-~(&~|c1s9WeL@KC( zWt{@4#=7u!m*vX5?*5~`t8-`rS|VbH{GBzp{MGZ&j6$ZvX#J{jQJv)NUOK5{v!3vu)nEvisA*tz7{eBxYCZ^fJh5%;Qll& z6X7IN*P8EL`hyS9+x&>UcaC7_7la$e8AxB3BWTM|D5^hDU@JnfzV5XE4`1tOqhs6Y zJsyI_TCQx=?=!)IR0-V#&6X|z!Zx2pA9^oAqMN~l(OhC%fxmoO#&Zz{n8?Z-y3Z() zm5OeqX+KW+M09zGmhA-g%|1I(AV|k>C`$+Yx{Ublx_8if`+g7t&C&U>=*XuxN{eGG zj~)}^XR#unrfM;|dGYhNdPCz@+2I%WJ36%Z(-aJ8ERkRO6H$dS^i0%aNQ#1nlGHwB zzF4bGaHv_ER-8m*^Q@EyYfxY`sT;Y*z7DO-U5OB(Oji8)DLy>EGQa)!{_$3{$6m=H zd$IEN10fa$Qo}Sj4JC;WP-9(rvcfG<~eAKC*+?MbKZnltFj`hujRej$| zP3!Mb+{f$YO34)ZsGheF%L*<|WRYiaR9r8s*|m5v9=YI#y`Pq(GEktheQAPIug5g? zAh~rpaR>Pw&)7YQf|*=MaS{fOQM$yp0qX_Ut}v%Z5_ID{~i|st9#t!G`)SL*m{9%pD%X_nU$OO85{N*Zxk6#j<>C%;= z|KQ=LcSSmYMu?@P!CjVM<%{g<6uCa7#rQKQ&fL__B83^MK=*!^i~IJz2-B7~u*Y9u zO`2QK59GZy?9S#|a=Z1W-lY(9G~1c_=0%#5(icEi4uk13iYR9h!gW7J6SppNKV6sY zW(pSH$zpUNso40zUmT0+cRl9y-n4z%QPw;+ilG8U{u0UO2WUra2Zz4T`K^GVU)R6&X!|v6jq8jXL#_`mLREK5BMdOnw(Ohdk#-;v{>M zE=_qQaYHU*uW}9-uYEj5)M2aV0ew?`RF5ad$#ZJXSll=nOv#KYavffgPm!u?L-F~y zTXpUr;~a$7ban`v&FVCU-vyedxRr<4zhauVD|5L%}R=&5kG|xAG#<%W?cw{ zn!g}3JArVCT7GegRUxzBpIf@yd=|P5V9zg$Za#r+LQDjS$k-!J!4Gy7uVy=ENzo^mWp1&XP3wVm4%$#zq7aimKv!WhBnV`0o zeK^1(i#^JULacqMC&UIrmF5-oEc>pztE^4-_T<3}8X&5sRMOWbV*Y(!Rg|YeD@<%5 z9@!mZ13faj%R6aktz-T)|BC3bPiFI|^s^Ir8Mr{^xBfZ_7R=OVBw&6>5KKL-d9g%( zVqYZ+4xcjsoZ>KZ;=&=PpfU2o_xibzK$US}B8z91N%j4EB!`Z#>MoUc0q$(4y}V<> zoD-TF2oZK;lN@5dA6(B6)wTMm#;`SjOb6EY>M~O`+Grv%@bd;TR)j3{ENKuza_wrg za?g5wm&4lVs{8AX5DHo*EJ&e`BIZ-rfdAp{4CJ(Y;r{*cth+2J9q(XD_WGGf{)F<3 zpwi|&T1$h`av)!%i9Kp_=@F1apolza#J^5sR(n*!dZCSh)iu%K*qYh*)d6%PSf%GjIDV==rIO$R5KL#QisGnJdDZY!YBqsag?~>q`+6Mo9~vfYt0Af5 zS%_5ilr;p`K_eQbV|(GJwnIcwcBF=D2yCU?vZM!4!C*>J4|aurO)Q)eVcXyY1?fHa z=6Oei_=@!>rR|3W2y}@dHHyH-ST(nx8Pd%-@&0I7?LMKGr><&QuBL<}@Kq>*t8I3M z9#U-6o#6+s--0BsNUzX;z6(DRIR+4gd{VtR_-y@DNGh0Q3!=?>Ow$$HdUp$DMthty zX+Z9ZdGHcgXjA^kI={>>iH#I0s_U5*$7!*DK4K)eY78B3|xT)qpPVS!D$=( z)BWl1q4aG6c<&+}m<`r`?QaXk6%+RE=99`R^)+!+u^+-8SRX$uTc=xH>?^x*U0s=Q zv9WsDw+gI*LIC&V2-=lI9o9GLj{Vk}O@LlhcIPX`=}VK4WWi%I6K;(67a}FXy8||B z$bcVF8IszoqDC+w#}LAQ?GW|da53M}2^o;&6iY-iqShg&OVvHf6w{Xs&I6@l!Mqj6 zt*=>uW99xUtfG2%RhBZ_@LN=02CAm$T3dz6Q0N=X+Z+{qg*)x^cc@vyln9`e{#ZPe z#%%UW{r8(3T|C>8)yVA*+g{gm$z0l|f~1v>3qDYwK{g)afq0U&Vo=a9!c)3{eU#&0Mu zqTq;?G-a{~|0$VmV00`XwtLLrHM%rqe44+6{JzFGE4it*;b{2oS|G@&G2-gyf7o08 z7*s*ogt&kCER zoYU)zS=Qw!ls_^4CKmH@VmLwjII?xY?TZbW%OOMlK`H7rCg*Oei{gC_@TGLYh|CpP zpp>*wwe)-M?@1PV(xjJm%Mu*re0dVTnvADLck5iqXdi-4Ec7?JOWyP=U8;98rXm3y zaHnk;vLf#(T5|cTftY34tlg2_mdPPds?q?k3$$y#=4K%2@b` zBF4;OIuB3de`=3|)wW&mg-^N?&uRhMA!^7?SC5dK-C5Gj)~tJH)t1>`OT=$XZqj4y zUN$dt1YyZIZMVJPm#xUcZkmgIuSo{G^UUk49!EFZq$sifn)XKS1>MLdphhtTrh5j0 zh!BXI27|Q^9o`bLe0_W;LUuAbKJj|#>Frik-3cT$5+F})SZX&P5s{hjleOJ{|IPYy z=Y&{@GP_|JH5@m&a3fX|XyB~q0q|m?cDtf($J5j|07Z6M*af)tLVzF|G3{3333l4z z$L}Kc+%rseU%{vq79bW11oQd+-dv1VPCble^=J{aySFSmwJ1l7XZ0xI74be-8WhzH zFgOOOmHBR0UaHCR{0RaOO&v_hd#vU%{349JvO4Py+Ot4y4UzZ-WvumC55nGY*q5;T z*qv|e>JG+oSlw09eu{$_%`KUG*)0T5hC!#bbuR7rKoZf!-m75hJ#n_LBt7+~?Y7Iz z=slUDx9fhZBFvbX#O%x&e5A$?e*efD{;gW+`no|3_}p*NkWqWdNqAXz7jj2~8S zR(9qO9<&Iw^NQ0a z`sw`~fJ1MmONhW)Nf#ECm^02QfRic2w_#BAEmIud$&f3&p<%uNn6xqu0a|Lc^RG-4 zU5Sx{TvhsknttUi;6%21Y6RZMlXn?+Q3+@p*1`)Yd`{a2_xEL-P$8(+3Tg9>mC!4f zOuck>oG!Neu5?sp_w4hm0Vl7co7ZY^i1zbB_M9p1HS^xREm>g_B?xd@5D?Xw{5dm0 z%|U*J-nE9lL}4aWl)xevlEBJoW3}{>ybBDViDA`RfFF#{AsVQd(^$N?`gP)YWQ?DD zqu%3#mtUCsO$n-W^qX;-wK#M|_~)ffFyC7U@h_5Wnt2a~eyX|V>Fu|s2W*9{rxRuj zZGkenn94>L>({DHrU3|M=$9U+!_=Z~ z8S-=#)3B@8Y=7uS0naFSz7%1^yIbZX5!c6GRZHv4<^batH;>!w!2tic3nvW4BVzLj zV##_Rg&*mvfD%(rbhwzM2Eg#7Du98T;* z?M~6SM56Seaw^m8RT8U_7+I!RZrxo|J0vGUY`*cXkYg0O3>uVs9S}}ewSohyL`G!_ zT0cug$=_0Wx(QE_V$W^#zKa=)>RBE4f9tn+`q}CnJ<qBxc6$K^u$g32McYhRFm+UM}#TXH<{cZawJ5?GDY z!9lpICXmSRMHLj?j;txO(4Qe@NzI%-x=;GT6Nx_km#eBqZFu@v(}!eYFrnlI@Nl6d zM@m%4w0iN2VDQCDTaUt&%Q*O2zjt^LQdx#GhV>e;9aA)d?7>J{A*5+vj?uws8*|pU z$+*QZ?{~I1KjugoX=yp5vE{D?P}BOW1#@?f`}H(;3wm(X49W3lVhv1<((XkR>EdL{ zRNXy^x5Y*(of+ZVOSw{d#7SGIUXlIy^^O!F$Em+`RSb>3%^_YpXBG)%c)N^zY;Fn{71 zIaqgBTaru>&tl5e0>gKmaeP(?PjSa?LzK&-IJ4eA*dRYACT@7&`~>VMZ2WTzF$xB) zpaW(f^9aU=5xd^~C^__+qP??}&AdSBoyH1Yh3%W_}v z*zg}>Q(G?XmQ49oeZ(23cMf<#Lbu6?i3l;Nq*U78z@kj_?o_Js8QzaoVP<-{-}l*0 zIvH_CHyw&Q`XYcB@i0Qt>p}W-A#<%91{3)m{nVGN=t%pF0EK&xbK&^CHUX&C7M2{ z>iR>Y&Z4z$_&~Owe2E+x_uY0?>I=noqz<>pyOm6Up(+TeZd6&G-a8Bo#mlz^ zt$#e>uX!)vYq#L~+L|6>X0h&x5Rc~k055Q-z46Qb z?0zT@7A6Groc+~%`9g%EVv$gh%G960Ms1_l3&2hDBP7E5SI{-TK;L=7Z9ar{8}uZB zc+l2k=m=;hRF_`VVGq_m6w|7Pmde?YUr(%&NZAwdj}2(afVbMp2}E$p7k8N)Fogwx&=0SsU}=$DB6uVYX>{Ldw-!Nho)sui;&FIJ2dJn%!FiQYgX_Pu_jkPbKGh8USa zt)2K5fXoU6$jF7s8Rd+_U`iP~|2^nRqpAcg4l zdm}aFYx@ILJ4hyJ<|<@X9=Ls;5C}$La!U3eJo(?B@Bev1K4dZ*unAcwax2O)T=>o- z#4am0^uVIZP0~vK`<)TJwm9^q^0ts7=4KLP@?uRb<8b0r;7`l{pU2dqA(Bh&sQ}3? zZuBlSO8JfRgvR;r4d}mQLlvMZzO_X|s&~H}F)qtimi%Y%{$H)jgDztOo6vNcVpH#9 zOMRWzwL^w`-k&q_YnA^0Z>59=(h+$Kbq)RB9r*A5P$CCJE_A9MlF!M~U;?Q%!0>Gv z0MA5eTri$u);4fgZ$%tgt^SJ=Yx!kB+>@9&* z*QndSsR~mbtZRmpZb;9+eu3l4W|w}gf%!{F{O~|Pk-g(Y`VIRpY*+S)sVDm{SxyC) z*s-?Qv3^_NWFVl`GW!oz!m7GlcvYeuY7mgX(>mGxy@Yz-R>vh%kw!a>04!iny*$C4$;>_L82bKvn^G8D8Pk_b*k0l_WRYM~ zFj#nz+jAAPO{_gvmeo;~9j7flI6RB2Ap-uX(5QhZ#WX5l3p-n+K5<|I)lVZ9XpPF1 z$Wr{YR7$Dqsy}LdWQ*W`(T-NY=rPXJ?9cuzVEy+CF%lp`WXtk&9@Z=es%kyCp0}va ziJRkiUbe0z=0K7*ZrLPX++oZejti*v+L#C}UoiJtQRn-I4xZ6VdN*%ciF6%*WpcBe z%hT=I32t;L?-b$^sU@NCy)D3u(98N?T6`n`#mk9+Kp2 z!O_>YB8ly6Du)xQ&Itct?=dj<1(-aP)i8_$)0}nb#tA`K8gTIiJ?!6`09<2mPp!8_>QJ8GF61P#mnoC;2fAwyL_aPh3 zcSk*nM^{I7QO!ThUHFAU=I*#7yO4B6&{<%4<#Z?jS9VcqAy#0-7vas1v&r(Fw&}H2 zGmEtC*sJ=T1e-xqq5Nk(S(PIdKc=t%gXB4!Zj#f{duL7dHZ5-BXnxq@tXTw5-HWt{ z3Ga%6ZiR5-u3fUfCWEMkpY|{ZEC`+PvD!|<>e4f>qu9%!kC|>|+ z{;2>!Qn^^KYXQ%J%TTNY1`bNClI3oh3qSE=NctQRAdq=CmM4?wKL;LYS&Av$pytrK z+>T-8UNGT*uG!aj%P1OBK0t%eRx3T}+3efhRZH>iARHbkpHjj4n6Ni+#-1gNp#ZVx z1~&3u78zn>mZ>cI*|Sq&0x~$Fhp)^D7=yOMRq5y}<|=G=vYB^KMQWkIRe9ul5<0z_ z`YSPM%H^Sf!5M)$o0?P#oGL$LkKf#@J!p#sl0HME= zI;9zl%qYWaxo4fQxH>h|&hd@~E({7C}l%T7gmmeX5l@yoJ7b zm3d^oH%>>G!LR=5K!|F6Sx>uX-fn|Q+d=uZML+*d($ibO6}DUPME!%}oI>)L_eKTK zHLXr+Y*xi_%)Tvr{i*dV54ZywPK3(5MnhDQ{iU6pc@p`*xs=9WVZd=N?g~XTA`v;e zpyd2!$Wr84m$=}q`OR=aSTTmiTEK8VLCDUNOlW0e53DKW)o!`1)Tw@qNUx3ee8?l= zB(uSQelRak$aaM_D0>WojHlex%MSlV1&( z5ZL|;L}y(A8?m^thh5^yH2#-jXeLx->z&~Pjlf$&g1et{?bM!Nc|R?dtw;_nEL+sumMvH!Mr#URY|cU z3_+FP#<4x)Y@f{$j{WEVOTI^J5(~-Bmh*tBsVLGPAu-)=44}1(-Z2o?v!9cI!Iw&V zE|BUB&?EXA6+}(Q0_|5u{Bf7-#tlH0Y!sPr!@woo{v`s=yq!BDjoIQ$SrZo?_oggu zok7_8Tla1I}n znH8z36SGuSt{APJ3L+TqdB?^-1qg4$Vqneb%*puZh5R5DLo|}n>ktZN-F7m+gD>yI7piARQWbBP#=j zbbCmnt!kRq0ns^*rPpF5bhQH>PnXguE|I1xcoo;y={l*Hsf|ZI)_Z8r;1CWrxixz% zj9ULG_MQOYu^Z2MaNgk+0kB*aL>1dVb5PSW*K>RaF^|V{YVaUJ%@5iS{WKBZJfs~R zHbu>vV(UX!oU&!eYQpMo)?lWUj*G7z4FPYx879smv{tH|5kHboW8+I=bGy{~NU3J$e`q-n} ze{g6*0H-l$#o_}FFrnBaMIW-0X9BUH2iG~sGP{MssQI6C;GJ;q%3<$Z$w#N&b>8qV z4~>mzaodWZtXNA>HJR5sdQVa6Q_DYGh2t<>BU!pi)Srv~G{n&O9Kv{|EP<7hBL@$! z<%=Y2r~>&@)EV^}R#oOqlZAXse=%_%$8a?hMnaN%b0}ryB!l@!gvO`+fPK`loz zAX!o1`Su09a&*VLO0DtI%GQ7@CG;O+f91inx;F6Jhz78z5o0n8ysr_5!i`-QO2jSK zuoK^V)z>Z6FU6oBI@I$uJ8qF_wx(FcG$(@u%vNCpc1$HcC4(*~HCA+sI*l8%@Ow*N zBYtzg>K>TF&mvY6H68R^7()nsg{Q&OiZU+c9H}a@wU3-ifw2CU3J*^Tl;bo%r@z!5Qr6sGmjhD`2q&G!>um{G}l>Z9NZyL;b*n@Bu zu|&X!J+02B4HZiFf6}-l`FImYp3t4&ZXU3XJ&v7`5`xG1Kj8lQD?s)!=qVt+`!=gR zeG3PlaI1XrQ_dE70#-*k=fyqU7$=C)k%Tssl)**n1rt#_AZ9dee+Tl-Wn4J_DA{6|L zC7N*5ek|H}gg?;NVF}3Es~l8yFfZ0f#=V>UlwBCr21e`D>863qTV@2tr#C!LP=%npn=k7hn zypl24xp?zFGB4-rWI8tdfEs17wVvHV+2hW(99 zTl2qZ!oG?nf(u=elVWxH9M;b$i-@Hk@Cdua~bS+5!gQ zrmpB@_A!CsSzTA&_NVE<7!k{iA>M>r$3?YKl?ie)+K{u1;u%7NM$uDsb7R|wB>=7&B8MwkeWRg$*iV-v1!W0me$nEYZ7a=1wECc?ABT z;3rm+^+%^xYv!UOuGqSqOcYk1P-3<)xtk_o4rw5L7$A1@JTWyb;!+bO?yz*G!j;u& z(r{yyIU|Sih*@~95^ayJ>4B%JAKTWdINwC*#22D6GoLZ4jXRKguV06vF@1ptVOyBa zq%T?4?uWy$y6OffnsAj=-dH2$EB!x607doGzDx!XK@JSK+Q^<)cjl#*Bt5buGnNMV zN#{PZ*u>+f2y$5M91QK1JmlFjx~;dY&E8l3@eJahP|J24r&|ubsF5xW)@*|Z@vYqn z5e0{Acjw;7>)|QZd4n=nI&QRP1tJ3eCugf$mvnnbS`gj`1yp6cXUk&P0p$LyE5enP zOITVgi!1i+gZGgyfrQCo4&$%J@O$Y@-tN~h6B`br?}TD8OF&7q?b-)7jWdD)Qabu& z9Cln#2+o8+xfKO=8%-7}xJaiP&x*IyZa-q&)mXByWXHOk(T~rm=V(Ry3Su8il&HK` ziQ!BPCsbJDEp=G~_)BGCD#@!O+`U34wzGcTA!=4Ptzsqv`9Q_y=*whw31(b_o@o;y zE$+-3w#s$tAy-Bdf+hBY^%-PY?v7MV)DNsBhXGYj9dM_Q?Nxu%E7_Ig6!Jz-A#%$n zOXL&C+bjhi=9^jFQl%VyE}!xuHU2Ib)yy`VD+S$S$f)fJfaHd^mIEL9b8JMc56qJa z1B3)5Aw_WFdg<98mb7!X4`Ji_!bC_DfP6o7*hv2$T0jpU{e^Xv+k>DYp5xt0e~jX0iu+F*1^YC?Ab z<-~c-LFZpvD#BQ?OjXV-UB-i|B@VPdcWDO-+u|+XC-s&xkUz0)bD~7E!myGwi67)u zW(5y&b|GYU5KV2Ie z9Q_gISH}>c3=%e@Hq{bu^!y0233%xw4yqEjuY!DOMhTRzr5+fMB`C$S{t|o6;9J; zZ3zr^fULREe01cPC;MuPJmh{O^CXrN&2q6vHE-cHZbtaGYQJx6*6Ssc?>_KtO1vr| zz=|-m{+Jg}fj4rW`d^4PGs}e0*fyA)>q(p!hb7097xvx0uP@p4>A_xm5oH+vyl*~<>2mA!AmhYG|2fbJIISh9q`!5l6uHJL1Hgcl=U!+UD!Kopx+e(=n~^bOm7twarLX5cI}lM zH|X$2)?kz7qoL1CBoGnjoNR&ld~Lve1{^>S#>aorXFi(y^adrpC!Qm>FksGU_O`jH zW57w@$w3eu&n89VY53LDTZ%JcoY;#ZX^&MU(1VkBqzk+YQKV3Ex)KlMhOQ|G&rZ_X zI!@dZG2_TWRYS{sA6axJGt+D<(=34TMdat1?MmK`>+d>(9u#GnqTp$i<9dXQu{^Y? zM>o!y7XQ1Up7tiDTaEyio%T2s>tJ0!L^Ugt zXU|;+B}@yLQ30}X3>;dcGI%(AUV$DK}*uF{q#kL=gWTs4-qB22W~?h ze;#5ZTBUkj1gZunljXG?%xIC&&j!{0p+dPu6K6qI)j^(4*8Lfvvb&H*o*)t{lZ}G_ zEsP*usN+z@TCl(cyExOGBy<(gtRQ&ZI{`wz9=7Uioc67@Xm)DYNfa&*>~~v#2_qMs zo6()x?KFW;Z0~ODn0hX9hhe6o z1d8s)nU}|bdANkUC(z>*SifI;#c{HCei^|liSLvgZjK?DC^PDjVXLtcol98eU^ifF znF#){r?ENUu-Zp+?L-;8Q89bkrn;x&*wrRy39Kfu^Zch$$Oi16Kk+YMAj_7B%UA1n zJLKaFI|N$2iQEJ&!n$n*8vOct_|x)C!zr#pQ}ekKxDA4agM)6P4tN;85@7*IK|CdS zrAZXKlxgJ|bKavb`Z`}H)3sZ@Zz-7ti{QTG(5)+)jNZ*Mb9+sK z@8s^HC`hbi76k12vnRvb^I9EMVc3xXCsR}|k``kJkORxjek)Nhx?>TP;{6@g?WQS{ zY&UmEFDSxdLfcrrPaFigZiXF?lX(xT3;wGQ#a_YzZdy&qf0jJ@ch2Mr%G+O5NMkLc zltNui2$6h4L=60FK{*{b&LodhtgIaNxy$SoK3eXue{866WgcD}@+429hEeq#V0%i3wrowm(!N77*-+MYUAqKi}>{g*7F!QusCj`qusf95%L;Kt_7c?|n1Xm)kIL_mjl zEnSmrZ@o*^LZnB9b_`Wqf^B1XB@ajg?9AM$e( zsUT4Fkm>X(&Ekh>E2ZNTzb1(ns>_lMyM+*QeyCXi23CrHN5#y8iUHZGArw0J;wBS6oaoEp9%QW4EF7Sm$qM#%KR4TPPO7e-*P@uG~y>M`&)Ku1k++WFd;4cw3;@ajQ zesI9~qDy}9ksV!jJ=8ZL@StkLA6A6B2)NL~gSAlXMx8Avlt9kzq#!?6wnDcd8TS@8 z^w5j^w3F`pE@Jz=QCPPYf{tZ<+B0Uj>*i1^%A<+(*W8YEpmJ>iulsijk%}H1LG2P% zM~M2Adr2i0k6q|28)@(P&q-`VYb~c~9{;@gqjm9X`lNl^EDGD1>TMbept+=;HjN0* z(>OVmywxteHYWX^aG`N*K&+C8sywnZhFg{jSz%!(PcA+Z@z)x(bHB!Rqd#{$7!^t_ zS}RQzLmFrybSJQC4NwpJOMJ}|yT_roh@MA} z+AwJ!eKFkWA_rt-)u%{twvrxPNImjMu2K~1z4=6xQU*H_bNr`$o}pcg;R9-6bnE>n zY$%~yQ*9c=QO?v1sizJ8WDtr?m|qV`F#A+uZAb_kh=w&cQ#SHY2Oj;Zl<2T3e_0`w zGPCAw>oj%7r3Xl9dE9Z|Y@9JW9Z?g5ZE22++Frj(G;*JX=s}Z~31QDMUuW)6M}XN= zV&{%MP^}~L)W{TmZB!t-e&h=sz*HAgPqop%7b_<)UNc(#mccty4b3%A)K2mms9Twtb$yLGKZ8O zCbtHVM9xbt~J_p;T)^pN40ZS(TS%-vO`KcNXc4SV&ELHkj_VU<)(ZtrYzmD%V z-F3BU_kGW0RqB`y&+Qc$sSKwwHV_(TqpgF! z-~30`LtxRmqGKm58Xy%{#T-9RSdY=txoanA8vOMg8z*Dpi#K98(J4&8D-0dLBS36o z9=|B%C<&wQuHL|f9#hD78sDh&Me0$GiABR4_ppx7B(Y#pD;T~V;4l=dh6F;yy`I?D z(0YaeUN0!}VT5A7Mjl7yMV317kT??6^MZWUQE6Ysk@1{jpuZ#z-p7Des)tbg+EMbC zT^NX`OX00- zb8*pLQYg^>i)w+rs_0(vpB3-P1(S#KbMP&oG&^xAqG>snNT$KvBEygevH=B|4GKVA zWT~DcE7Ao=&&l6tFyR-@?sW^lZCiT)(PJhCAo2kSS>(kGIy_CCEp~{u#buD^ep18{ z{ru1(lZ25MJ>o}hN%3G}-CA^vD?B-gAFup)Xza-7-Jt%IeEdb?UC>WDnRbG( zrq6^u^|S=x)Igd=5g^mxo!n1mtwr~D!XKWk6Jv0i#0}xBDrao}qe)NLS$&AqiadBr z1=F(X<^b#ilO%!aBRt@eAt03m#_I!|6o|Lv2^6rm`q5cU%yv;_fzwm0y3HlX$}MxV zf9kysGw79#wySrN{u0@I$EA9@(-QC^Y-|cgr zbH48%f0lb@lF3|2t|WdRbll~Or4ui8cPW<5I3qDd2HPOPdAEwG@bER6k2ktgUF+5X zGH9KLx8LI!7SaBAGS)5<#%wJLwwjhZ#q)$T(Ull8k1}|qvZ@B8R#(oT6QuPaQr7s7 zLcB|2mVVreMG;O2pRDEsn9jt6KMM9;PFZ2P51N9sg5b54{U-~>4!$Ag7Uhl8A z@?XcKeDEI2ty@ImrtUV`m=yUC_Z;ZNJ}ZAIxL+zT?^94>*pYxZTQs9pDrcao8oQGb zS72#XJ}VY7U&l?6NE!*s1>(sbcrQe6xI>`|8Oy;!K|zs@j&Vpa*M9!2#t0>-d-RAA zBmiztno*!E^_?}ob|;Y5(A^z}MR)3?H!j3si;LWZ;OsWtSWji^8l3&Ue^NFGEbFC@ zpeGc|*r8$CT7qlB{j&7sD%(kqqIMu^j2c>>0x`?DV_R@cH&wP^?a<>?25_&OUo%K6 z@zqvAc81K*oi^}H@mUlb@%{4zA~@B$Z@dYA3!T+NZmDDL`4UDEHYh&U@wjOyXuG^niCK#b?a27k54q)Yc*=_#QD+GczX85J>C+H}|w2Hg+;3S}og?sU?{*Qtm#38F#DfQTp# zes+=OQc=C+A21u4m3n=KoP6v-XV(;(1hV=ObIYlzaXu*7jnx{0ex9o7R=VLE(MY0t z#`9x?f`kfLE17@5*Glhv?v7(@8eu!d?sH&>$+nnPhQ^T2A_a*1Q}{T8+x=Us&tGQ! z*4VPX)I0v27wS$y0V*9-l!X6?1UpSrvxvGiwuw$GLZyk2cjDu1w11_f!Z|vYI9f;X zyQgsGs-CS7bX}kiDQQ$e;4an_<7DC=ft~A+e$gS8v`{%R*1Ytty$U%osV`GboyxqD zl9ZQ0JBYo)UQURkT!wE_+=NT_{0QNqS#2?njTu#IA;DIN;W&W^mAkBlyMy^ZcO6%d z*RDZpC?7_+HcCmp*H$LCxQg19jmpT8D*k=J)8h1RHcDNY3*8q}a=Wb-QOQp`JtaVq zvYArb^|}tZuZ6&0Dgf0hxclk>OMBY1rLYb3Mgm=Qz1P`J;WLYnvziq2w~LN0Fh5lj z8OJ;WI~qd35&tscV=uQ~LyoTFENTz03-)H0+NQ^WA5$?BRUQy45@fw4HnR4_wbgD1 z;%r7_QdMyN>^>EFgmsGaKi=9L0n{ci42?lwb{ncqQ1D1z%8)Z&oe5kpr3zfdKKa_I zRu@?2gLGmplGkI$jQX>7&Lsi@9u|j0Fz7)Cu1ifw9Ss)#ShCmq869;E^ufM9hdnWz zLLLYep$_Cwd*^sqa_v2C*Z344`BP&sW+M{t%{B6WOt28L2|o|ZkSi9Niz$}&UYQV9 z6>`q%m^d-qAD|scR3sEqS+0APPvho1s)&#nqg)I3=j2OA=<8JtwA^4KcaE79`s>XP z{nF;Jv4Oma{N6>hPoq*Y$a~zv@iE4$EBWd3E*vwp`Tyg_6>9?)`p<(Z*CGxBl)~T6 z6n9+{@2nTm!GAJo*mLo-EjxNEoQ~SOaX--Wlj6dVFeQfivy*<@+n(!?^{OXfDc*At zise7$_}}r-qXjgDo_CsC0fgUR9wvalZf#gqSG5AbF>g&2z)#u=qjMSIkU%HI1+Q4< zlC-AL^?=E*``h0^+5;hV*KC_Bfd?4-Mg<`0konK}(o3lSvL63Z0Xsc9U=_>)c;Bo> zR((+m)61)7PJ_EvA%c8@unH2o#nI)_&=#bD@oaR4T+$*?S<+Xz)P)BHya(QlFpEJ**BN_L~OF2tv1FlYKkB#J*LY#>TY)%1oAwYqz=lim7n7Usrz zf3M?-MoT(N0MYUX!NBfI`sc2(Xmn|AW*BEjB_lJeSf1^LH&H)w$m#{LIuT#`d1z3Z zI!|bn<~F!H;_AZ5^#W+x;O3p^QvkRUfykm*JNHiAx7fXe&18-3PzrVohm3!CNN?vX z4@QpNgb`|tCel{*k>e?oaX2hNCX2k7=exdGAb{`cF)~&PvXpb_#*oR4PvDw=Y z6xo$4`i^kLV3;tILcz;0H6G@g48x~|hk;A5pjSqK4{75doQNA*BFJGgX>v+AbPPbq zOT3_)Z9M~^nT@d!#UYbrc!zF8h!8RQ3 zRzeub%_itrwdGgY#AJSr&p_u(vebp2FN0yV#>Yo1XivcAgYX%%F>hSJ@}A4AUqMaY zC_?pIRB*(L3N(NImHDFZp|@;U@}{YN^tvbo$;WPOEqn3(8%_FcORhbmCJL#Lc}L_I6i?FnN)%u_EMdJGM<-Dn4>*e_kuZxN)>luTYKiuPNYnbVbvIm+^`>ih zA}#AX$fWc!@@DX07dfRh?gaJX%N2;r*MQI$=9((R<0(5hRG4h=S)$ zft@WW*gy^XR|YAhk@kAJe_jA%4#XnbhM8~J!_9f8&l^gwLs+6}uz6wfGnpuYtS-=8 z?g?FLw-Q&x(R>{cZBT(JS{Knt@C~cM&S^uQ2|AoexEa0YU`agEF$o%SQW@FcN~kLT9m&e)(3jS1%@4hr zMZ~}DF{6s>3?^UiP3MJ3M?0l~lE;tEqSa`7USayux(jb)wLNeG?j7Pf)TY={044wV zEO1z{yN_Id&0gfsT7II~b-tee;|XbJmzxJY~qT_DyKWc|1!(z6YZb?uZn!BTHBR1U=AYJg_ zZBnStdXhQQgm&pu*m+4dz%N_fvd4)+iBVREEDvfKc?f1&JG|mZINMxtg_r1bt?bUUKbhZJ zgm&>*W^agl$soIN@`!ds;+Q*|VXHoI`w8mxTy_-kYG%tV->L{kcQJ`uh2}0Ck2t%p zq`?$D5OSco)ts#?E|+KX@ci*7VZb`Ol*|lW+R>R|0W=3EedX3&|2fE2NP}f)ZkmV6F*t}}v>vwPKm6@XN zRpLI{h@m!%@DXI z>}PP~O3d}7kt5HKx2*E1=%7N#K@>Vldh^{?dCH6Ef7Id4X!6|XpmswcChc__F(lW+ z{f}T7VozN2bf?mfs`C5%b){v7a<0oF=~a3`@0_~L=qb&X&KZ|Gwzx}{E*p7JqGD59-m0>KY0u>wT{@LIh$vx~Ct9Tw zT*4TA*SNRtXAT)3YWY@;moOpaQB(51#EcXG)B*{4FnbQ2Fo7pkL9h0zf$XdwGEOvu zzCQ3^h%5mrP)7~4p`F(Wr!Pco2Q~orR6{%|lr)Xsl`))~e8aSG@}#k=d9y`?3M7SN`yKXP0FrKMUE^b0 zeLp*wFI~cBgz4A?c|&k3{&e#e1)Ckhr4UaZ!G3=_-fW*Sh3|s z40sGm{pJL6^k~acdYuas3H3PA!P7=A@wzd-MW4X80pk|*8oQ6b-Yx_~qZ2v;ZpV)G zkvOr(1npHXLpg2dEK5Cfy4d7vD|@}wC)*XTH?fKQUFH&*`4M_ELTARvKL5*GR*VU&dXLSd|%5cPaQ??vl~?>#VMe3Dd;?$7+*S#t2T?)LljW}Im0M!IJz zgv?JB3iD+oGJYyU<$1*-OONNZ8DK?;Bj|mX?xv^$`62;JWG8$Cd_n0j!g*hhX*GZR z?g7Hy^AOmx9S)aGDPQ2{CtwuxXlu#S+}8Z~^i~0bu9Sx@0kSI#W#*2r`x2TFx~7

M`Brp#EgT(+y< zoZv*IRUKL~OLqd(rS_xHeu3w2~wl+TX4sA+Mw^S8|8FN<8-M z(fRO%?ud)~C*%V0w8Iq|h|4242=?udb_Z+OZCc7EXIuR>-E+A)$N40(alY~R*^~ll ztZYnJL?GKbvY=%n0hxJyNwf&-JKlgHH+zNX*@Ovdwl>xfc=0Ecj%3gK)N6ZM4Z=h^VILC;54 zoQ0ejJwdsgT}f4IejWtn8CH{dfCx=RKX*W(_`@+?r*SW(OJ`b$Y*@ZS^Z+?rNhJJ1 z2hQM@Udx9$TmY|pA!R^A%nKjivXCdwpGk#|WqvXkwqx-1rPssxb4&wQUatsFnd z&Td#1&$$!~A(z2cLv7vlF>{<9(5dbwHw>2HU*<(!3k1LH4kycf=ifKlXnIz`y-fH z?zpHt>eK8WK*-JY>61?&>{#2Rjw7@6=%4O4+|3swV1_*BZ!ez&XgR;-D6!?thfR{G z!kU}Je_dsIV#qun6_l;#&!~}|;0_;O%qftp_U09j3wGyOpoWgTUnxLdy7)kIZE0*F zql#Dy8X=k5BF|26W13@sd>wJNv_n|_(L5fRJ7UGE>j-2=j$bI?L}*vYGK%weqN-?; zFp_ouwqO3{NL5X@Ri<+pQC+R~bstmQGoMuOM2?(AzbA~s5-n}_<$CaSNO)J6Yv(Jp zPS|xvaPz4rl+8sN$!T=;NqmCLsug5kifWae4ExvTmWKq=E`fu(O0|`o?n*F7C-g1> zmO5(i4+wV2mbwhy{tQiCnfb(*&qp2l4+?zE90appgxCaPp=wGhV}-P+AR zj%2aM-#EkHB9I98KImzGh@BRBXJ^v9dWf+g@?*-}3i(Fx1u0%5x`rW$ck7Ly#HAFjw=P-bC$;%{U(sYq^8 zBlBU<8V8C*$PO?mrZ~WfziWSfOqF2<}y}LY!HTKe40_l?N5g zE!3FChZMd02cIxH?|GIMT!5r`j07A77P}s^@nH_=52bL+DBa-vcNd(0cQM)1Vwjas zR{^c)^}EvQIkN5xA-F!G_(F4WTgFR`6$&0{ZOD=^Gd~m@X55{WmGnSbQ3NazsU$ve zTU<*z*f?@H+>G~{e8q`W)wcAt@cCD|Rn+J?@nQ2jRf*V2%Ey|jJu0driZz>m-vh>f zzyF_V*IfzFWcznjto8X_oA^0hY2d-aGOO&MrrM-3%n^@{bbiZ|Pj34BEGhI5(8EZw+SJcpPZOgDzf)5bLH!^*#_VI@)S(R!(cMy|j z(^vGr9tQsf)j>UXchy{2GeBjUo)#*nn;{GE&x&)9(7-S4Wzc#}{$#HBoMIYEtc1gn zH?L4?%2Q*n84?O{aaTKkpS_g$`L0IoNo|)qP2~Q1YM*Gpz~>S9MK!gNg{K#AJrskQ ztP6VAGUHdBnGu)_VwxY0pE8KA{W1Pm(P6qLRKPRG_t)%5lIZSZ=KeWTDz}RFSGdYU zmscf~!2JpHuZ7k(K{H=I?c?wwF5RFHSFB!>*C@arQ{v$ zYUzvhEJJDcMD#{bx~U-pd={ujY9$=hE*AA6FXupNt-~Sm=ngZo;jDBb|5naJP(ByE z=c8kn-_iJH>7%n#-lj}oL@!Nj z1bEbTAC1|J%5D;~Qu`vD(D9I^zqCrn1ubC&68@J9&?0oOPz)za?d{iqKdu`~x8vWL z`)rngH#gDX+ZPa|$JNsZ!KAgy3yDti%w;k&bz_}H0jjG!cHY2Uw9z2iC~dmcC1t* zv2W7YFRGmlvmtlKmrM2Xn(Yh*6Y=Z{7H}MzqRm*`}h6Hy9`FQ=? zg?BC*0(=^jK5_NYNc+!6JO*Ah*hTMeC{L$2M2j|ZT$bfExn(GL+I8|Sq z|DRSo8rsfNeACQodmI*MsW(5a&Qeg++{B`v`q@$^hQz*3`M4QN&|+}~DF0)AWnqHm zpWCXhg5FrE%2GG}rG^)iF))drCR-JtrqFo{_RD-(!*W1J`c*zDxH`zC--f7^UNO43 zIEkgs?I;gAn9!jojMnEFJ4c9^Q3lXu>8}@9!#dT8#0=0?KhC{bz!cv3l>TiU;t)T> z*BvxH9LTWQo2pEYJ6RJ+u5+_M5B$KU#=I+sNOC#2tvNI1%o)i;{hdHNil1fbxKQmF zAM)tAIa{ius(l*R6S65@k{ki6( zijmhPeUy=oia8s!O~-zB*}I6>J`}D?(%buZCo9q)6Eq1$%27g7x{9H#B?)>KbHW9Q z>L{k;H}>~8&6;wTMP_lW;omwVu3C9IDZigAT+!*pS`193QzMci8^mQKu!zojSUcZc z#_rkI@k5_i_W3Gt#~!oKn*{fLBp-y=_cQk&OC%TS8IBmFIy%Br+A+B_I5+l>;kN5# zUeR#66V0LPy&cfGYVz!t<*|L4r?(4&0=2~|s?y{6fe~UMU#1YN2- z(o*A>wFY(A2JApBowC8Bqk6<0lfd}h*F>JUW7u3&a3DPZzMiBtIrUDs5#a3Io zBvux7xp)1NqDCAL5NyusrwKIG!47d+paMh()pvi$Bz7 zT0u4pmb|dBuY5KwvuQQm2dUmRHxaRGQ)bq2)o+;*y0lDr*(l!+)Kz&!V_LIT&57WiPZ3muH>yL8+|A%`_8dz0`>hi3Ht%R4Bg(K$UBa`U8ZwfKEVHI zl?dbo;?<-Tj6L_o&*XGk;y(E>!5JNfE^o#9t_FP(l?xfJ%wJ_;v+uyv!BEzN!>1j= z+E6uVpw1@^ov<@HZ8>D=hn-F!yWsYdb|w#i8`RSW(6&5yg_LUZWzzLzEen39-b@=nrA% zx*v>@yB{<|0&6VgQr4t44%Q@rPo)SEH582Pc0-FOO`>5mb}^l^lfM3g;g%IirK~Q2 zk{U7lVFR%6kv3)UVV-3>&IhBeA5ZZoo6F!kxh9E^gEchV1?Dc^zJo=I5!LZHagoh( z!%JV`(d~kcSMh7!Mi`C376o0;wRUrRYfa0hg*cGj9U1a>yXUClk_m;e%U`wnnF2Ve zXoZDE_AgaINSctbDCS4Er-Kw(aaF=PYZs;DZQ1-?n_oHm!J>f zW`s=qd5Gua`j(<=_glCO@rD@Apu)-JN{ z7DWd+gCY?3x^~;RK^+JNmSkiLsAAo(?nDc{7W~Gm`P(S1>f`$;^NP{)V`xfEvLFi` zrEHf(2Kjz7Anp8*v1ZS;-!e~2{*ZgD?%Aw}yGq}5c8Jk1VwaNnTW%z>&>+n)8a-7% zRHzCn&u80pGmy@^F60U9McTi(rsIhy<#)AR2VP!CCC`C7(Xsr=Rv&ee-#o%cZ?vg3 zcb$USZlEN*SjU{xVq8|Jx)hB3<%Ux>n|fPjKSE0P)-VeoyRH8;Q&AbH5-xZBE%DkN zO2khsCgx*&AjZ*oTrYn7rxz80^ysSSK}+i{2D@VgME(8!x=)QVVViICA4bA^f0o#o zTPTq;eyW|T%eWXbbM+ov==yHiecS?$JVt{$^Q0>n z9^~q1dGjT%|Z5@0R(p_Q#*BIa`#R8ql$Tak3 zaXS*m+v|~CP^&f?+RsuF4a`r-L-^kz8a zG?~+^8GT3@5(*>lB^x*3#u-^{Le5Yj^25CID_x=azS#3f)>>tGxMY`RpvsLjvMeg((SgiRmx=CU}z@H~e=*T$i_}STO+{{BOUteD!a@`92!5PQj zCw?D`cqj&YC$kF1P-;5p4a4GR)SNwGZUB@ zg{@GJFFOsQD6nkG#N$oSHGNC_per>xYL&?kw29xYB#GDahT6+#xu|PFF=5F-JG0Uc z1@omMdMziTOW6YM=%~)kvw>~3Rhq;7us=IYr5IW&nNUXOsRLr?<|`-I%|l0=kKxJL*PhtcVpO7=@Mv&}R_H zq6#=G)^RIHcAz}VW_|GAN1dKB@BTQb<$|iKUOSU&KLK-&Sp>c~tNE`ERkg4l&q&?^ z95VzfLV97FFC8F;SjASgC>CS8`MEqc9W+Q9%&Hr~7Z|$AuY2c1n}2$$7!H^1tUHqV zgFP0K4YU;IjabwN4x+e{-rTitI;$Ra=e@Rrswc0tb(jH|4-5LKH`ID(znTuG}d8_AHQZ83w z$a)+12@oMUq&AIWoMe5=6#)qeN>W7PJjvw!*5~pg`_!W$A)`=)*XojfV4um4U9)Vx z0qvvlo<*QANCl6V7cD}~9DT5x?P?}+3);uevr=d^u+V~lyf-9_x7&i_0P+jo((7Sn z#SPQN*tlz}l6(Ya$ArO}IdXR`{YM8IJ1Hd}MAx(O9-KkXlPm~xRq;+5;J4dZn&d^p z@5JNH>3W|vm46~O&}`V1O6+v1H>6B!g1}lZVx*E$4Ou!T=?SVUJQbTiJ(92G&X)3$ zp;U&<{#m@RpK|eSGFP3g`4~n-{nz=A-I9Vht=JZv;as=nbgPN4s%l~Eu0><_-bDl1 z=PS|_-$TGa$_s<+xJ<~R*-tNU_Lh*DcG*D6OlQ?^ls`e<#1hm}0y%PUU@=%DBg6i>=|C8yu)))uObCg-Mr@6GwP>|Zc?umPIRO8TV&$VRj1YDnu*Y|9s8 z!qQ$aq0!R@str;QGY_Y&0iTHHF(u}~E{SFGZUhl=?BnVki&{gx!uDB)SR)6mZ_S$j zd>q}c7H-W(vG`m=7EKqmfUK~)+2&P0i2KnE70kQu zgSA}qxjMvqR5h1b7GRxknNJ6M8zZV}+xowoJ+C6KnR?2R4_cNBFR7S(D=2uY6`Wit* zL=g11Z)gF(G*$(ETJq%%BwtUd8ep4_PNpTv4Uau0m2^pxi|q4597a*HQ7Jnz&c^0d zTJG44*K0Mk*FLTYryh-U;_d0E&U24~Y8yB^m7f!i$rAwVkat(Lqw+EJWXoDJ8pVZk z{hxDmmN3Y93ZfC=Ya8nF>Mo%)gTh+m5eS+q!SY#Bb39xB>M>UrLRVqz;_@zWEGqi` zWCTQk7t_|(#;seqS2dH{e@%AZO4?S?QG~4#DqGi4gd_RXS)2hiTjG|Bk2UPK;)Y=J za31F3eo`+l?${%AJ{H`NN$vh)pX-t$uXg&1>44bX(`&*$=nvP88O^(xB{ zz_zsR)q9hR%fI~|f(4wiU}YdDLS4m*z@85ypA>bl>7!YuSGY)M@||Bmb~kBwbQ$~R zu$el}rE#oEAB~3|Tn3)TY2b`TtxFJ2M+wb@t(Fh3l{&B$bmli#xB0+*R$St-0CgPU z`Qnm64Rsm-ZSc_4``WQiBG6(jY^pY2~Td&~d3nV+OdOiF&9MBJ;#Lfa+DEZfb*o~W?3e_+69 zB$W+V^0<^612`H4o%`cw?yq@_rcT^Rcw%*uM>C$oK4E-{5{nk>N2S48K3aQ0k z?OBh<&$$V+kD}ZAwe9Cgsr~KDSp(6qP+dcQeFcnw8yKyG(t8`i#q~L^qYUAJnHZZ3 zs(6xxxm>*cAM+clw2}z=ix*jEe|(%SVbP%lA~T`uo1Gw=eV*_h{9=B8+{;Ix)j(b| z>0O8yI!&dh^yEw)fb$B^nf=gu?BbN`&FMT2{4}SgnjwgKADt#Je`O^c=TBcSn>5zk zeC>HF&i%!OjhU2-V7f@|T-v4K*LIank99rInmeVVqazyHZ$c*Mq9!T`j3Icy3)F=3 z;%aBLU+&q27m(O53KRBp9S!b&*8eqF!qOfFQQF8_BC4PCW4R5lrfnx5A*@_ss_;b6 zcH23Y@)j}S7WQ9E7UL+#Jaz61KiZ?HkC3r3IDMdammHw9Iq!~vOyfqTzd;KK)|)f& zX+GqID*rM&;g=DV{70`FYwcFc#A4ObZvkR-JJ|_$|~EtHxu;o=?xKVxa=3%TYNX>AXD-^4k`nbNFi*PfLT?4ZbffFflKt1kuB_9 zcR8qfzoxVwp)Kz!#H`dAHDPtYO}5j+))7=Zh?#LZ{2}xADcpN+hYL{z{*-g>OZ6JP1^u}-nr1FjZ9HqN!=}d}C&Gf`fOc1b&W4JI} z&mKngcl&p!W3mO56V@fU1;T*5WIZKr~FM>FpHr;Iuvp&tWOm~&g(c>I+-cpm(Oy!70q;4cqJSD>P}(M(bhe%Onm_km@NFtBx>cw4WY^)jX@D zI*-z*+=YP5@f>cfkgZXX%kibV^5o&)WjpV<5>He&C_9F@VzpLlt~j)=$^~Fz zU^MQ|a$zjkDTgss$|}3w)ipNaK@+;>(!WSPOUntl$mJA}l<(hn`(5mu9rKsKF8fa5 zIq`6sHRPDCTgitZhk>8i2}!w7Ju;Xgw<-1oo7Cj=1lG*Opu*|(942Q*qEmrXTZf+iEzEO$nY(7_TyMtS4yqRow-Bdlz%xh(@?77|>?U5q&3HINF zp1?S4BB@xhy!9lL1tZmT)IDEB&YNvlPNMfzhN`1) z{@iM%I&}Q$ngD4t%4NhMYd;6F1j9ZBEP!(~mAQNx)t@ zEbM(r-S_KeM5V~(dc=hiJZ~66LvPiL$&*rAs*EAa-RAuQ6b1%JIaM*`cHQJ+|}JKPMVj(7&O3UNyHjc=#yb)UQ_R5X07=63PY?i2waMc%*U;B6tC(P_cw-7{4m?oCZ_l2q>4?We1Jm zgR^{zQ;>z+zlblBB|t)(IkAM^U!USoTS z*?%S!gSXIm3J-pJsEe`gXHP@!K0>}ms{o5J$}8TwSWzg-%8GoEq2%_(%31mnr044UKjWe9NvFiVDnF9vJP|(3 zle(Pn&ItTHuK!@{dO+%}%ID8=^@{jD&Ds8rI*nllB1Lv%&O!i|Z)6v;kWkDgy|4G{ zahpA1UR}}rU%RM=7%+|YM$#I$V+C?b7z69QeWV79f4&2jWc_2%hzlci%=wz?-e2!? z!jc0zf(y5d04cA?>5=6Z9GS--8A|_kjKJS_M+Ez>&G(-j%Z_0}W>qeKcePQ`93}z+ zW81_1uk{Q!yF*g>T}rQ#)pm6XTiGL>_C~t4<3vzgMty^cd`L=@2`U4Rj|SDXbds*{ zP&*=2U(rqCd#?Hq7WKbEaPA9BTx^=5IiFAgv>ymcRfm~IQ84}assG(M2%!BLeB+t> z)^|6E$C{sDoHc&fHseByh{3=Tc@n(E6jd4aA&mSGS#JZ>)^IYb=q_qY#t?*<{qVt2 zAg2x{c@(f=tJ{WhD`@>~@Nj2fn&o0|iHr}e)QjbT^&W}|6nrIteYxPRrO^um1HMNR zlm8CTMuR-31ZN`dwUxS#p0IIaC49+Hd(N7^W=R*cH^M>3w);y^uj&e?7PHj=&2ToDXaq=zTfP+PGr9r;OKlkfA68qTR{Ntds~ z**sd|*>AvCLGQgG@79+p;Qv2~Z~isVzKEYx)$X+tvOivSb#$u{J}WmrGOedg6h{4ZU10<60?b*Mf^x4c#>B!fCxukr&6} zFuikmg=NloR+KcD(Iz?T*dD{?-_Uy}MPIz&WziE1ruuTfa7>u`(0>3cxi8=){cBIh z8(w1q%Cn1%5IQ9I0xjPxYgV zm*IPwd>ZeTtg4+4r`B9ICvR6JJ(6hGJodBj!0)f{`-{^IsUJQVh6TJD8g*RI-o}FD z64mC6uDb+D-e*}xs_m>(Z@D(M8sza=T=~#NdNj<(e*cJ@AmEQ>sAg*B^&iuB9*nWY z2*}3JU+zu)P5(j!r-^J%a37YLXw%>1r5~*h+hxat!(MoE#XaPoE2y zKo6knzZpR>dk?x5OQH~l5}r4~&Fh`3ZzpVl9rY2<(k*-!CsIPc_vDd z(hGK&m<_}zi#U0<_9-tY*++>78)Wi>lr$9TpMeg4tO0K~s&NCCl>$gA*D$G~1G!n? zP>xf6q&P?#WMU~#TQ;yU@2%@WIGVx?wM_gm;vetX94heEg?$G1{`>9hWL3+-`qI&o zC`o8b&hC;{X}#&01n2^QhGVfo{XRP;p*M!w@clna#7aBDp*7=oq+XgOF<{ZjKV<#Gom;Hq3=;)i; zA6Kk78&q_uzG{{yw|JX+H7}}evAFrhTn~tZ{%|E@m@fYFUoJp;5Q~(RIisurrnyWB zK=yL+wB8_z^abuq@OzH*MO-GZV}n-0YQQAzmaAB}ND3Oy@Ck(CCAs;cO-^e$RK@p7?9`ud4$^n* z9f7rsujUS*e17Km1q?g(_jOu+iStDG$6-z=TW_QK`b1=`2zo;v*tEp?1(-jHcS`)V z1QZuwQ`zwi&Mfxn;M_mDhYmlf1f5=qNtvnc!m4D~V=p*@ILU}5+lZMo%wqn7{>ei1 z%fR?JQ(zBo$QLFcZf@?aFjC*n`*puKBN39|w>*+B$Xw=j zcKdC`sg8y4L6NV;f{x9r69+GM60hjBReFja4S+;BLEOAhx3ePoCGC-5`sfL%yj39o zd1t)alC1e@;d|$`1kjQfSp3pbBHa+-Z-;5FR44WjGw*~_+6fF+wt-tfr2;EaxIZGW z7A82atpH^tLVF?Jx|nm5`CIousPVX-i=nz`5@{eo97S7@|vw^OEnJb zTwdE~Z;NV)GLQf*_Ts`mI)X4jI>A=l+L}>RRP=(xieLaq6 z8wnQ|*Gt0g7wXO7Etj%|!C(|pl+J<_U4;HmpP%ne#T=@0T2Z3RrMzE~rkINSl9MbH zR8{8;wC{eTwR!6=X9q2_=nt_Jf{LbHiltz{dISVmS~5#0C@91^AQAC?d&WF)fG+*? z7u?uGK(_hv*4yFg=D<918LCo*j5;l9h97mlgaObprN5b%)nQ@vl&JzX<||! zNTF1((fG5qRWMV)4aLR9<>BeL(qY}_-e4$^&QWLxd{6bI{X_7)u19&85Cxh%Hv-`7 z7fqa#TQLQdWxDj@)?{-C0g8U2DS1bvfYMU>o&Ejmv7o@fD$7}V)^%{f zo<9wG&X<|&T20q)F5SWKS6bY6GSM1^I%$fEiaVQWA*pmF+XHa}lyQQA3>A%m>l_3z z?w5VIm*6oV!!C4xbM;h^ds#kzzL6(~a91o0lPhZ5_~%l9RX|%*|7@k{s!^-KHt9OC zS_G0xuRjY*?U|#Hgjmi$QEh<3wFP~_*OM+OG4Yc|iuYJ58?(0UBy92PA*r)Xp@3WHesD z;dIe8`S=f!I60BmnY5ng5%m#$a(u|tDSoptq917jUuIf>->!~yju-P6X_M4&#O7xs zH1tv!H3Fp*s9n%Z4yOv#t2&8>g4(!V_C?+Y8-(17WnsiS;o!Y5y5QW8N{ZxlnRz$# z&H8Pq>WR-QTF!pEL=F9eD440hU}9jqp}zUIP0!z>#UR_A#Z)W*!7i6Am?q$8xSE)p zoL5&T(9!WqRF$!JLUe}Avoc}9RJihX=GKvLe!kX52wpZPJw5#nJ6Otkl=gQg8iLm& z9PayBZaXD4<##9Vuu)MqS*cq`#{r||X%k%Ft)G1a!>3rCu%jbO#Imw0lK>;B!nFwh zSNLyNmVSk{>y!doLt{ywEx`X?an44hvT2W?JtozL#`Lf^0`rCZwKD+PmVpCa5#S(^rfw5!tSN%YMl^B1;+@*S(jFyP~ z$*XU(v-v4;!3`9kp)zbVy(VS)%D%on5;hxC6~ov}=8hLfW){S-oBNJ;Vgc8a3*e*l zCyEI@s1t$rO21dc%J!#HqNgZ6J;Qev^%W~R7+BaLt>8sI$0cUQ_*}WaUrX|zdK$_@ zn^;tP{)JW3{Fw!MK|^g<2I@H))P@GR|M#SuT31au>s{0lsCK0N$WnP!OdS;aZ*>}M ziMj38PPnb5(I_^6P$NwQ&u4z%i{p4dve};?Ck$VIyT)J5u>*D_wl@D@FOt=fK|6Gz4z z!5nMTgqHpCrEUSG$!Y)V%c-4TkZxr|#pikPPD*$pLHBQ3T1mQH8t!y^IuE!V40|E-o(e`aE%V^tEp@I?uG-ugRd2%~c(w zgJDl4IruU>Gw$|9;gZ|nQualLdAXH&<^e4%7X?(Eqh5O@zGgnsc zIN-jev*w)ovFWtvkamB+(eTTdbU$~6|7388A>+*-68a)cSC(=JcmjY+y>dp~ueCCpbNEIlc z)ZJQ6NZ3Bq;9Diy{{$~>`=vDRY^u+Uz7-4H;Nqpsr^++6Gb3p~N zd(H!65aPcY5L>n-o~FKlZ&AiXI5%BhiP5b0{xq*%Z=L2mxZgL{^?A&0hcJ;GikV_! zWd#W*{~<~3|G4_b=sLS@?Iumq*tTspPQ!+6Y}>YNHMZ5*R%6?2+}Orm)Xuf5iqyyi9M()yyWPm<&pSmkz*6KwNz(%AZy8awfC;L8UVAC{CK(;V$^N(Hq& zyS_disQ)~8c$j%4EO(RffvVEec%O|9C|Xt7$$H(CYG2q_(UOx(jv~R!>)f5J6qSj( z5^p*HR$?$c;$*Q}2;UvZc2}up6qPl}l4OJat}y~=%W~_SvYO^cP|(4{Wo{3J6`^+v zPXx33l_}-&=bYICBH%jll50t<#Ygil0apfkJ0vG0^fA->)z#zmVSk-jBsDk~NoT)m z$vMWg&xneY6zp!znP7T)x(lBjxa+?%kOF`_W3(+JfQ)eq0u4{^YrvH*PH+kf3lq}Q zN8Q=IKD#o4&!-{RwlRY{JzoF0tkZHoVQYK2UAbg44Drc>s#@$xNl8J+uPTzs7%nd# zJ}b=I&3+Ai$4oMX0V(EHr`xL)URIM8_+|Bzz(k*0}Un@|QRllan#&)f5%clyy|g)Fp(o`N0m4 zjvkz@fgr1{uI{{IF+At`+95wb|GLQBO!6;`rV2Xxm>Lt)wLgKoF1xDyyEMD51rVo8 z#^c&IArIrN{I+|_-ZfDE4ZblNj^}ucx zPI#+9V34l6J|YOXzMf||uXh85g2T`mq7==qn~@yrUHq>X2pVlJv69`mOiaO2qf97& zqjumP`F(+F`EjG@8+-C2jt92Yf3E(y7LF2imuAchq~HcxS2KkT(mN>)&S^Fz;fe_a z?YRXD9{USt>cqad>&-)@4^|ghl7^q{cK-#awyajc5(_v98=GQ_u&gYSb6rn^j~p?m zCK4qL&G!8KH@*kqihjWjMspgNKtLuD|K^`!fF>9vO4$M~a~GVxu|+WYSJ8ZOpb(K= z!}avEsT^ygK06mZocY4emTxosGDyKZ5uGueo%T1eviT@Kd1dzZxCbTt3|(S=0jLf6 zzue17NJ#t&L!;R|JDVsP!#&}7xt{s#dYR9lo85ftC*bA9S5s4SKJ$(LLdl|3K}`)? zMTHz%ei9}|A>sWuO$q^UCYk_MdBt&aAg7s~DOU#?%^efebWB=pSoja;JEJM$rDlUi z7_{QgnjpNjXvN%9&!`gRUsScH$`ybaA9*M89Ar=<%K1dL@rrNcAMfwc&?b+#8{{1j zdcb*AzO`Qbuzfe!5BeI=&<_mys!-d4D9@A7?B`@{rneBo@nG2bQ$VX zLvu5h`9ejYoq92bOj_Zr5a_Y{`9WUv)?|*5U3ef2#^t%BP!MeFe=SZF(dN$8O)>3lJr0qWF85ZtIF5=@l{{jbB{438a7SPu*S6bM!4fNWG2` zjiIYV6PfJoCE3>O4wf1oR{WYqGuUV}hZ{h*4mng|Ho0()^Y90znlVE*RA{(diat@I z`ZzUBmQ)d4((}PTURhhq6UTnxk3&pZWrx$LVyM5ZO1m3d1&}B-Cv)yBHQD9?Vax_` z*I#3C>`quP!qABu&Hws&Zz78UZHfXg&48Er?CT?^8<&pGsjJe6`RlQ)w6wh9F(XYR z+!PyKYRXoW#k)~Sr~b5(vL(CMsH;EtN1&hhpG9p=zg1&9*=ZLrbL)gdDEP#Dnm;U0 zc_fUd^a8%s=xIt4!+0@?^2G6H?0N&(M5LO3ID%2mkufkgn2Lc*u&R!4Oiv()H(eOd&W*F&%vi4KmAZnVu;5 z`o(K0cDoG@y*DG`L zSM9Bg+!GN>UOK9nDHX?DFm9)lfhXtGr)|YfI^n4f=!Y`%ErRudC{CF$EecU9@rGni?BnA2>holF>LCdhK1_484ZutX-;}kcoVxwMRaz8Q-#~TlPS>7skAc*rP0x!pbQ}usN`wpy2n?Oi z%%l@D$2`|SgdHLhj_$Y~bNbipKMDwRQUnj=8H{f*^VRdXF9e^CN*|6?Ur6_HwyyUU z#2-OlaHm*gywa|l+%Q_9cO!9X*L5(LmR^HqaGKiH&!J&B{dSeb;#HT0C`3f}R13&G zahDy%`wa3ha`m0Q{c0gggyoGnpU5g*wPaglY9^M(s=WTSBVWVv50#7Xu|dvoISJ9C zTcG4pik!bA!gM`4i#&L+T)~kNu~)uLg~O6Q=c416;3lf-5OnT{cD2vV;hwHvGtqk{ zTF4nDy81KFLNZW;k>o^5%r9naY7|zHGyHm?lm30+>(XNJ&WF=5l~tTeg=8c5uQku!0Bd+NE?5;)dNg{FO4nFS8vR4ui+~$W#&B)B zk~1_}KA@sn7DUc@6cJIWs=XwsU>aVRm6dfg(E>`jlK4?Kj_!MEJD9!EM0dKYJ)^Nx z4ek}1ivO597);S`jb@e1U=;RoTL`#=owPiM0I&uoWvO}SY=!kcV+T8S39Sj#iRPNH zg2553P|lftVkwm9=N`SY9yuGc1-E?4*By+CkzYStFG8Sa2!#OfEusjHK<`vtZW-+^ z=d_o4HMp6w^(KLbsG|wgEBBxlQnVDHk zMn)tcN7FDyV)EaSU4V?^;<$FguJOLiqss@R3CK&E!82dpvj0{itGGctnJ(w4+}0YA z)eC}6L~N~VgC6bu?3k8sKh~CkIm}nH!#7jN{Em;z8fFnA%(U1kmq(XwXpgL^?Kt^< zDmkwVFiytQnOPS>^a)yvyMkD!PNt`&$|lsKSJO2fj6K)Y@Q z+0X^%PfB%vYpNg@etG7n&{X%vPJj9_;O6qJlYDn>MtSnfrDUd)DlSt^M=BAEb&Zs7 z2D@dd#(mb?^8m3yn1??zzSt&!=iHBIzxpBYKYvg31i-@8WiI`G*yaf4_ap$s9ffiu zkwPUu(TK#k{@kZ+CsOl;pr4Sd)AYMa>5djjw&4n@v>U#8ZSF5oSji7lJ-L>zm=YWU z4QLE&0XOr;;JdcCNf&5GrQB+oa8J-gO?;p=Z$F}Dx#eSl;qkBH?e?6^Szj^G-`2<| z^|U;M91yiY_H#_K68#lBqq0zU@yFh~e=srk(>+&;;n?8GCqD zZZ`8tmy0RbV}e#lKkr##bZ3Nb?2eFyOay33}L|M%QDZ=#vB)4*Yfv z{e3RlJBosP^C9K)O*Co}Z^)D1O@gXs3IRmoYR}=%@4xo;^d1RUwBrwO&_lv-&oBGs z1n|OHFKC+W9?eiC#agA&2G#k**8MD6IiBJ(*T&2G0#(}C3EcV`*2DC38RL+d{O}aZ zDR(aoc+&lu1L1%2d&;anrUzW~aAleXx`Ca>2!cy@Vu%!ty?q;jW{#n?QvRwPuagD! zpuzwION)MY4lMw}`^jFa2bt5dM%waXt50V*GJf$^x8WTmEZpG)qo$#`05F>E^2YTa zf%qI&xQ$)onY`-%i6pyZ!C-?M_1K)PP=aTVa&z#Kz`6^h`0V&<`AW?M#wHrbn9|p2 zxWYh`NpBRxD6G~=;aS8~Pv=I}61oW{LN~mzpzJd3R@$!9Z|OX*Zsu|YsfeCo?sbw!dPzKu+}TSFY%sO z$(Uqw;bOC_7pX+@ULx^%WZMGxJw~rb1odNO*xV_2HA5SO%DfW}|aGiD5 z0Wy++ogMQlfGm@cllQ++2p}T^^jb79i49k+c2&$IMpz`hMiH8ZVv9_|TU*QB;e?UVTfWaM6wFn(=sLUc1u}jBPj2=2mSX zJqjfbWBMo#b?rvW6(e0o!|DUg4A_CYu=xR0+KviyG7Gmu|J)|m#lD86!>#W&KfzM$ zG?afksVUJe8(RldN`LU5N~uDl+pzgU3Jb-HEo zs?HjE2o8qTT-I#v)kB*3y^M?@EB4Bg!^rS(OhQ87f|mQ3v(Yz}lOF>}*W4T5!f2E*ni95#1B~XCflSp{xc!T(A@J;TDJ#FVoW@dm9TAs_(EYW>C4w@! zJENWVj)7&K=H~%?sfoOwlX0nHbim`0cq~S+^v%NeB%=FTt;Ph%$bRnx9v0*AcpB@~ zrdS#YI}k$+iOsjyv$raz6SY@4U#MUBph{&bAh$u8sLDz7aBwnU9`QaP>h@$sVgmDP zLxK@f=oI$dW-M{_Sd=w<9cCRX(W@c?zpH{ANm~=Eqc~l8HjcAR<=ACj`jCbl_zOY1 zR>pwvAb%B`E&)h$&Vg>zw#VhLJG%~`Q`0h>7XZQ-)SrxQ>grfmomMO_claN6 zP%0SNI5QOtPb}lLD-?u( z>$xDdSY2OADWb=&IWf_w2wY_n!eGmAIvR{6ae2NN=pKg?ry?Tq4erV*UqeQEd?#=G z0i$-hIO()lt&7kon_w9j934}QzAzl-SVI(CmMaWvb|>_cf}XxZ>dvZA90@`-ZOj)Y zM>c2Zf0R2P(eBh}`XCL5QRH)P&o(w{qjD+qs}Fu#CDkv8$%8i-!{DE}7^`z+k5Z#& zzcx8&QR6`=8K<_O2zXmDR1$+BpH(>R-|e0lXDHh);@#%@1k$^mwcqV9RkU47SXmZa ztuQ0Mo1aZ;&_ZYnmGLcUP;ql~2eYAcFXakKU37MplU&u?cupCz6*_wc`#OUR(!jg5 zcFNYtr;Xu)i8X)2>G8Qa@MP{@pRgIUjMr-~FmvB0?{bCfcxatBcPA}`d8LJrkSKhY z3<|hNVX)O;00}*eKj?nAfjPmq42wsoyV8 zk@1U|T~i@+xAzB`w3ej*G&U+>&^KuL-{U*0n}H)r5kA6Yyy{ly=`_Ag#< z*RHJXiFVd@73Hy}O0H!08m|ev`)8)cIxsji{)zcwN9dQ(XwWxs^ODIV{8n~oNAy)j zRZ3<>oo#)X94L69*-1?N0ahau)L(vVDz|d6_l!M)j3$gsbkU7N+V~R_b9;D*1pX=n z-@-k6ArTm4KeB!GvLG2QpdfQ)c3n(Q3%uWC{7pn5iff*M*e_6FGyv51;O;Klyk6Yu z>gz8u#Ue1f(Nr4}(*7~bnvn5HVGK+GD)X*7+C&^N(GBo--bgAI4Ns@72pnq;ajQ$82t0gN zHB7maH7sJMA|+A%O|1KKV5aFlN?d{OGBXjOSIY<9w-%(VjP2)K8!nC$82yOB92jps zjCGcTO*DW}pWU*O7J3|Jy^x}dX*7dpZh$gLnCOrNKs5@ zbuI;o=N@vccall}om`IM0Q5~n3Z^a^t1LHtLbT$qb|~}MmTt!zMrpH6U{H(2}?Mf<@-LV%>>H%UB!V@%m`)1+Rorz1PL^|h2T4IcGdhL&~Vo;+v&csz8S?-->cVJr0q=vPJ503amgb{%9YnyW;UkHWjrc<`DQ^o8W1DI z?(S|6O!WRhPCLCUyg2x{xWE&w6=!JZtXG@Z>|~&#s<+g@=twIkY#krS9rzu3cO~$u zf@)|ckmE77dT>la%x#`MUq-HNTlokPgIf^u)<*i%NXfMB%=*bM^v0$LOBKEuT+%Nb zhD_|`Qi9Xd4zRHeC7~#)olKhFGneYQuuY>4uDA=daSs_27xR?~MnB%>Me~JC@V{wT zIgha|*c3^{QG~A)>~aG&(7!3QHc+6RUnndjN4R37#bx{P>vrieA}cY%FnTxpzR9k& zMZY63s7EiPyARLVkJCu_a1IRs@g2b(5peCddP8ew4LlQMuQ^#vr%3^iNlqSg*G*d~ z<-1mIt{lv#j{;UC9f;#>8(3*kibx#Hc^~7@H^c(&jPLqj5m)hr?<_SQk>;9%65iB% zPYd|?-_TuURQ2J>w36Qx&MW8FEMKK<~H8NAej1QXpuO zAU&v+1Xb2_Oz_BxSH=?QWPWG)K$lGm-UC(4?H&aFx&viyGp?o+M{TO%Bbq82`R~MW z73gU4AExAupp7L|DFecoQ7^74Oq2O!$8&oExKwu=(ug=WXyfW|#cl3E?Fk|~SP_YC zh{nel{a@%g=WAnvO#ubsA!rlqiWWlxc7>aspRG*E=dIU+ew;;T*Gy4nSu0-8Kb3WW z8U~NFLZ;0;(OcH*SQoG`Gur^Xi3fJ>g<(wQ%JKgc6MZ06P|m@QN~aIG+;1&}MMc%V z^^{lE6LATY2C@rZ8KoFk~kmrZ8G`_F2e|Xt=K=pvyd!Urt-k`g29XcVtv?RnQR< z`vMJR;*sT4&i&=e+B~87&z6atXdJ%oFzHHGyP-vY2v@Wp)AC0^3*T2%wiXhX;gZ{@ zAhueCJeb|QP}}t_XZi74Nj5WX!`A$&f0rrQWlm+(OY*aQX|Yw(F;JB`{{_RK(*vP9 z!g>_nH*T02kbpqugcZT{A0@0L-aMUGZct37s+7BI=rkwULhdcm4ER#gHlIXf>?sDuhr`NcIOHg`Tft;?v8y;zTT(`1X$}Wyj0Xr`_Y74UKI1bj zgm}kWX+}4r!>1)56?ahFl0WjlJs`ZkoO%0afa{qaOy;=U{|>eV_%%1}`ex$)5D)mo z1WY{*xd-{4j#(OYHtCcW?$!stthey=A=twpP@y4)$7TGm{F!zFF@8A#2Q7@ur-n{&)P41lu&_2xNDh;7sZtc& zt}t|hQgUHip&YTa$-tSep&TiGj;)bDO70^f#}&Hns^iL-#9^}@r2EohIZwO#iPc5mB?!@1*Bf7vh6U6m_OTIql+^*|Qo_K|Y7Es8FQ99p4{h;cd*BM^ z7-0qGZ84xYFfSm?ULLV1#EGPFzCX-Z-#5QZn9@P{z^E{9$FZBF`goGC_8JW188DV&g2X{C4e7OY=o^qihFr{L6JTFV{-7TDXMDA~nvh zm=ld;Fuag&W_|p$wsF7852?*T3JMBVYaW5*WSB!GglI4R55U-n|8K=rIJ*dJb?Q9^ zorYOftu=C8RaFl4^tzRxRqv3x=)o1*gG&7V;ZJ1}A-C!(oguYQl*!cSdO~P`E--`X zdUSU&mPlB2aD!S!q-f(dn{jj*x55M(18tGkYIjV7ePNF!@b|fs!TGTG;)N7-?yhd# zSB(qknr3iz0+CPGU4|?~JY?mu3A2~hbkXANcjbsRlQwOb<|>_WHB4&&@JERAa`Y8+ znj2ay>Wim45{Qu}uj?gg7LvTq5`&rq03px(dFQQ!s^cv+s3zV%4d}l-!sZtiIx%OX zm*7Mk99ymCN?}?{2Wu^Uf-N7%o%M>FGFd9(!Y*aUtOlOvoR=?e!}}WCi`L9YuL(#J^QrUx_^FO5JNsB$06$Cm%WvMYarP`Ey3dBf zKQ1G=d>~~z=4O(z^$Y`!&t)+{%T#G_jj5k=VT1U4zi?a8tHj^W4G!!9)vi`1?4XZ7 z^iqSsl~>&ID#5;on7Pe;=^4Gwv!v@77pv@BtR0 z?UL4t>guz~_wQX|)U>o<;ZW`Xt!v0w-<`})4Q7q-VNU-}I%Qu#y}nNEppN})@#k6| z^WK+xfbQ}ECXl4y{@$gcDJVEt@}1E1ANI`zShC+>ymI0qFyWE1gm|g5}ral^fRFSeRdz1tYYRo}VC}eil=Tztl z+Zq}goYs6!Z_~Qy8dv=aB$I|;;^X7b?W+d zxMf3^hb5|vv0v>)hZk~hM_3=g-Hy{0m@;47DS0)61Z`I?{2Nn9XjHsMax0}!!}cor zVu#ffBs?XvEJhF2j#bF7b24Hey%^2Rk-{3zkWr5lpfGd7MTqYnsxd@>SyeIW7Bfut z@t)dS>LcG48s4cpX)p;+f`r7xpmF3Ez^o4MS04VfMR7&y1`=QM%64?`=WT-ShRGU- z=X=|VE_^PB!Kl2?ou#$L6I&_9ncduHZ6pX@eQ3%yBJ&B>J%+mfa>um91e$pl6f=WU z=IX1fV`~k?s(6+TmzOX|ANv&-N!#7E(<0d1%i6&65LXbe?wQD)G{W9sr*k|;uvcph zkd!*|tI`h(w

vJ$j?lDL>1l#z!`9yJZVljrBl%TyZtR7b^PLz~{+u+N>YS>F39V zPDQh1aT&w&&E`6uFYbY0ir5Ot{+u*`*1&0@DS~wALBaT{$Zh};@PbSl?Jdeio(agc z9ddT?z5t=jMoi|ooJkUBZt#9u*~oS<+rbq-W`Pbj9Qn2O3~>MKf=7qj+-}TZaYXSE z4JiLjM_Q3>>Tl+j58uh|t>Y7(TrmmQ`pm{P7IN)Pp?s7{#R? zd2x`k5(9sg2X@&oU4nEL^^I;+#Z~Ub-%k3`Po$d-h|45PFzqj>QnDcUPdk=Y8(8u| zH|}l!h~0}To!OV1*JGE#MJ|L4b`+CBYex$2DO#It?pvTG+Qofm`igzOx#S}op-CgZ zf|eX|1&D3c#+}+1c3yQ2W3kQZQ+Kt+KcG0tqMQ zl5_!i^*^fGl?K@aFu=>&f6Uzg@pUgy7jK6u-_;P@ifL61bE<-ntN}MS+ai$YE$<0b z`OFD@5uT*GG&lG<(?j zgX+e4`$hWgm|m5vSKz2DFvaxnX+Q5soC|c4=sY8bO7qSVcBFq!nvz7&}*e&62PN$#IE-cD^`kC*$*9Cz($3@RZm8+>sW4G=5J6RrA{|aQEs=0rr81(x@ql zJ-+Y|8!*Q(<<~^FmfaAUh7Zv2UC~uT*HR$^Aj9}IwY9d7*QWFmKvyTilRfY+;?JKK z=eDNq|8o@EVDVx$$)9w@$mhKQo-%=->+IcOZABEAxaSiqkdJw~Jv{L}Oiq6Psuc_e z@iF>g`EI?~&>O0XW$+!{t!O!}1&_F{&?--4$ zP3Se(RPp~eacuesVyvpIuBJx&N=}nt%0||`xx-` zIPjMc0&hSHi}CQ)T9vMUUy8~@&_@=anMFUEHweh;NAhUZ>L3CBD`9SK?%zI^e_#Wf zPoNSa)3Wo*00sGl*C9B80*fd1oweI!W<^|k*+xuNX;nk{0bpI*9F^ym@_dj=@c>$l zy0?ZonH80luWaMt^F^ca$1-$B(>Nw#h2%H>IW7UPN2#uFMoQ{v6dOc*@zq2eayUFsiaIkXx=bM&SPhaLKVs1}n0(ya8 zdg-c;bN(rOy%Q#!$v3|q0ldHYg@vm>zz1*YV1_a}AxYC#-gB3eq``LZbRO3lTEL$g zqFTl3;(N}_%rMz(3eZms#ZfeaJ^}sOE7E*MV}=)o_V)IM6sS{EQyx#3Cyg6cjkd)k z{~QbqGGKC*_oCJqykS;Xu1Z_vd3|59^E@0lLCoVZK6wCYdDvJr| z@LT`re)R;E{s4S)Oh%)jObY3M?7wwSz=Yb^gG%(tmDt>h zSpdf$P9X`fZGAg{UDTf)7n6M`#Erye3#-*Z=e-Sz{o#(JQ*oa{u@w*irdkCMq&dH~ zXe@@0s#~`Q?q_-915rksB7k}T7yzHwIy)g@HU_V(;=g-2{nEAmei^{fh99WwB7hq> zQ~Q=QDq7RQ9Lx)-BAMjmNj7x?G*I{%cz(FrZ8__FP6qO=>J3Yf7eP1RDoqB3=ZNT zum5Q-5Tg?TbvT`~?|=rldc8l$Bn^C1^2pGm6WU!4=q;COzySg>sn_#`m{tN+?|ZHI zwDZjipz}K{q@SXI{6}itK{J483Gh8Ml{R7l^WH2=Y*%kx&r19=1 zfOr!MY&MgSbm(9mEP2-vNF;SKEu7ER$I!6v5*Hf_T7L)OFq%vfwmn~tm{?fsT&QGu zJ(_l$D0AVIk~*BOa=s?IzP-NKUEE)6KX4%s@M-*)s*4C>P1sfT-csd3P2hD;6otnf z1T;-aWTw*XCaf7ue0c*bTeCm;tp=&8CJG8lcRJR4`k(yKk{;jn?{Sp1@O=ZjKB?rIn;dpFgA3BW1Wr9Y_>^RQa*2QKQ(J-nz_5-+Q|=ci zsiMBpywH`Pg<(?$not$lTExIY1TX*q=>!yh?GO)SnQN6>>r}~oz0FmJfAkB=lWy3N z{5M|k-IkWLQBs8WlKfvP7otw}aAoUh3%+xKf=c;Bo6~%#Oi>Sjqed()E{^~F&Ix>S z4Tvr4aReM%_ungCWg_C@;uAP>(v3u2ZIF_oROLiOMAKxR00RY3EdbUv1MPe{vF?r< zx+WEeF74LY^dfFCM?pjDLt6m&zHO~P9kPJ?rH7nMS)kvXEMEs&?=1j|Ml|?vI^Ngy zN4JeQjCvDGh*a|&=6TTLBGqRLQ-@aZpE8Pr%q8^C9~76=-}_+xP6z=;PH_9S7C{hJ zd7<*HI<}3JD7`h|{AB(}uT^|jgWt;OVMB@Evr+L_p^=V)1s?2@kXL{GxT7|^-(~J6w^^D15M@6tEU{Udc2=m4J(1=EL^e~rYs2dzdt+=Mm% zOWRQ`$enYgb9*=xBM5mH7&eG+s6=*3B@S|JYQoCO^C~Xu(713nn_Get1BbSA^i%Sa zFg;Jj>4%EMwDPHq$s63jPisLR>A2b4&|$OnK#$oqmZzP(OE@bVUfK`$|py+_YC3Y>Y_k|prN3>WkO+6D|pgoGlj$S7IZgGBkaVR2J!aGk zE=5|fF1QkvBVXzJXEiAws6_$7(IQL+-opCStA0({y1kA1SL&BVp#m5C zNzFPN!{26TI-H?3lKi`COwfFVl8&OSTqL}6@^R-L4A58aDtGLAINGcjrNa;io2)4l;vaklJ?w(E z`ti?MC;QkVV(xunJ#s_JI{j>B^j)?D*10uW?6q0y1IE}XIB#z6#sI%=Vn2o(DC$!l zB|%Frj!&JsFA2@(k4wIH7hPuzlQ&^iKm6mcX+Y2`BX5Sdt2TZdohQ&J3v4+q*O|gw zl%JL+W=IIejlR`dh5W*)v6>LvV;O$*q>xNVnD!pebrA{fh`9-MV1`-3{r6t#B{x&} z1X#|r-#f2sgRZz?`yxLnkMl)#g(`cg``pU z&n63++z=Xz4A827$UiLYW=&o1ajtqhMT%2ru`7YjHfoY86&5PaZcewTPa7EL~Y9SA+Qxn6Ca(u-n{|4z-yVz@s3=KY8Qvw+Q~ zuR0(`ke*Hz!#%-PpCNxF@4#oA%!0H)H$-Xf;uj+dAVuifGR)d^H&PuPw(6I0Vy!Io}N z_R8z-gyQx&K9+N(6i-`_)$RGHOdzqiH%oLd})xAa^VyvLlO^R?2Gl??i}T!XjYe_#I5W-2~7XF6CKd;Z$7<8Q+a!m z2cJ9VZYBiHKY~d|ReU^MEA!6K4L6ta=`)W+vI9p$!sKzOvPlEQj`1msA!eXe8yZayn9b;wLIds4s^OZt}8QA;z`UPtwX?g%X^RQzJ z&I{N}eFSixIVdR`q=Rk|tdiR>|7yqI%|9W>IJ(KXoH3WrUAB*2<7aVbfYNH=oYp#q zWxBe%zcY3H=ujO2n!({VdFv&N0XE8tp)}#qql4!$^vdXeEfS-z8$kGSjY%21))GB4 zZeybdzUJt_)wnE$kk;vBZ=T?3Hj0gyTQ_KOKH%QMQ%KE&b4dE@ii@22bQAj1>lN8} z8hOfU`Dw`xm(=c2=}}dId05$qP86-i8UZBz!2eq4C$h;Eqk-1PQ3=(KtdzB#a0qeW z05y5$r?2kyDp0H0KN64`&32$U_mbBaJQhmMscg+-lW0AIVsontRrF+VIU<{tM{ms? zCh;Oyw*Aj70p~G`4|4X~$THy~f~9o?11=WO3oe(RsT&=?pyQn6_bWsiNQtGV@tmo} z&JLtinw?NY`G_&nsOAL7(%Jm^`mDxUTUrEDpWJ}aO^+OpYRTr~x#b8Zr{LEA6|{GE zQ7W!$59$@s&Z;N_0 zTp`7#_>=s%gV1CwUZgeB0=1ugyq@emCKgM2b23bhcoojRX{UQBH|q0+X-jPXS5#)< zK~i#CU4@)0Q?#!X{mB94XdtNE!=R1Az`^6RMJ8`26)n}Qp+++-Ox0Sa9}ma7F-&ov zPFLOLQTkATZK2HyVpYx!eN^gnvW#SvN3PQ!jx%y?x558}v)}gfuDjgQJ0H5aAu2b$R-)REt~kN=ZFqzMB!>RLHOa9s)z%n(W~AUVe1FQk zfz9)BJ&Q;zGdlCg=|RUPp)uR(s-`-^)8n;IEj-IbSCZZ+R?P&hy*+-D=HpLxZweTu_l zYt$D=cPhb@PW$7;j;a}6-tdOrCQcL5hV0*y{nuhC1OxhX_`OsH%aVPbK4CAVP|t1= zGGg*9Z(rh1S6JYjs?wLc=WWi4qv%7 zsQ%Zr{rz&daGwMb1#G2H83i1NSsgx@5<>)M6`pVaUpCeteHCx{N<3MpJ29#wG=sue zPmdt(;#>Gbxg68BY?P0&C@8l)(;iiLePu*S`d);nwjwp!<<(VSmo$lJgbnt}BI5rN z*S{t>Jr3B2nk&6peuWxw!}XrrXm|rUtSRK|nboJ=)aM62spsz+f?|{o1BCX#^V_mE z{i`<3H7PMwFZ@s0p@(!3{?o=o_h%S;DKu@DA8?=Lzg%f?Sg*rIijp+J*SeIS`<_ex z|7(7zTCuTw#w@~s&oO#M|+1i#oY5it{2v!cstH&|sgX7@f&^_aFi4vjN)o7vIq*)#A+R3i4F@Ghm z!K2g>5)_0DdM85%+4JfCS0(=y92JtTtvPJ8)Rz{Idc-pA2n+Vk!H0?@6Wka|FqB3+E)XS_JHh62)q%|fdU1&Z{rf+Sm zDWWuMcQHpOn(@h|4dfQ^b?~T_9UUDp#0YUNKn_}FfK2eeRPsM@z9FOnX~D3;Ucez@ zjQ%i3A7X)#;qGqoy$BO|s?im8*tcj@{Q}XC=%KQ91)9ZVO5Ps$=!iijkRfU*VrK^q+06rW@Ohvn!z}OLc3P~C)!~I;YQK9Ff-29 z(42#EIGF5ae0&6;etiJDQLGG*{Qo{nKA8=$ZY?LH8($rw_BteRqB|r=?+5?K=KT%L ze1v`aN`?KR9VN1jCo+NQdBr+C!1RzXV01va%*pTm|6c%X=t1en4dK;ilA$J77d)SC zp&(7JZ-C;1_2}oOGbi%wv2{@AjbdmsIDthNUq~FB$OG~280DB5t;QPmZ-E;-F2ZFa z!dtA`*{m?~nlFgakOt^vzZP;tO>ED@Q7H#RyH;{e7Q>a$4x6=MS7n8q#o$nX`fmNA zM%^oe3-SraHCe-|L6`8kEwq9wG+#w}KXd_oGT@}E`CQ|?j44LD^5og|Mwj&$vDa2? z_McxFVrL9tgKmp6I933R9~1-x#NUr1y~h);BMbT>YrYf->^9K-C(+f44-#9{Ly zvtd4LNAe};&4PaDYPC2{9h8dtv85E~gs^`^ken*o;7eO$;nw;HvRltT+MF5TVgHim zMghUYnki!gLNed+k3{a2T`8mb>i_6&HZ<6xWfOE2w*0*3+i|p355;zy&_?zMGF!&e zBLK)Ij&)Hy#CLzvbIKD~xVub9VVyRCC?%;s)HX(3qe_ityK)O>8N zM*029@_ylm@7dhVWi{ur30+0zkc|V0(h@GWs5>CC2?P=g^|66asqAGiH8%mG zB%$?b59M3<{%31ur1Hx9Z0aHllma!AS+io(POYdWI&cAQ%b;g$lLFQkx}#6vf?8!= zmd)Qu8p@TFRkG$uvqbgBWc1ZsolGl^o=O_#oQ`6~8>YL+*Dy@GGB$Nh8LSi#Sy6q@eP{>_KgX~AVIceqm*2wJM z94duLj{^qh(Lu0>bT3d=H5AhXN-I6phTpK~WjQ8@>dqiA<;x5J2rlNImPZme6iG%v zd{e1-B*yIEK|pBy*%1A3d~5W(9=Z3RKtoY2zTA*6a$frT`>iLolLQ+oodj(AHT|Ms z?c=NKi;Q5I%~$?xCM@bs+lH7{)(t54Yht|m?js-h{~N;hfnOXA$(dn14+Jg*Z6N2A z_J|YDb7Iwvv%+1h)Uow5+}VVIl?TwVJ`VqN~dZpbzSKpq5l$@9i8gO%;jxO z`@Ab8-otw!Xd34E7duSB_Zj_LX8lCODG*(+@w&51=W>J+e7^V)jD#1QolP9U@emKE zk{~4WxMS?|?OZ#c*mj#=zaWGp!RmzzB(E^7A3|c5Nj06^Cz@1)n4_CaLQ)1B*CA&u z_j`7TM&y;rmsspAqQmAC1idXdF1e!se%jy z{RI4JGU#ujd~r#MVa+w2$~8RvN8`Tcy3LKJ&_4m3$}%4v3?SXnwm23MJ8*t(^=F+V zc#=ijG&M>*8z$M@j)`@}v56lp`rnmsv;6&3NY0EDf9q-P`P!^JW4k;Uhaw0WlLn{m zdp|RlN6XaH#@?8f6bzpAC@Le@>F0a(MRg)|-8FIX5sU8wizk?Nm}{{sqg8C6eq*21 zSiL2}q6;8Rx*aOK zWez3et18NEC29Zgm41ygJ+k?rn=PEFKY%&1%uDHX2?gawu!Y?(YwAyq~8j$EU*OG(2eqOB z`VnfJ>FjtiZ-~#p@Mp{Z(_fL{HlsiH3b7=II+A}mF$r9RJ>B~v89Hj}?=d+JR|=<= zdi6(xro|#&E+6MPiF&ulLd1B_^}wC3bnA8;b~aNEa}1Va>fwB{D*y{85M{A+uDj)@ zkS9nPnb+q_VcJQZzApG)r9cHrjC}ggXq07mB^zj5tjrc7g zyY~JFt%C*<#Rutc5o~HBG6Hmhieg67piX8<0XJ8*NP7zt{dyli4s0rF0QMWx*9nAR z-kpn#ddTbj@qzUwvK*$%5~Z8lpr+1OsIqJQ{$D!`bUX^dTisU{)JTORC|A;a+kxjM zQj#N$Q@2--CLu0sOD5ptUgjl}-}WN&RMX@^3f9uuKqey9m_A|1N=vn2q^+f83s3>6 z>FOuc(U!WKi1D7QSZ)vG!`?opwETwqzlWg$lGSreAFd3xmc5}I#iM7C_$&A|>?E*B zOnqxW&B*iyB9Aw(9nY-t4H6G6OZ?9`V{Fb(>gYf4Cbxzg>xk5%FG1XO!{+Mj94_j3cN&sMrV<0#NL2?mswhsEt2YuPkwdFD+xo8u~tDYuSrH0i`>>VE-E$f8UowjDPWdivs7-{Xjy5n4gu^* zx%L~K4vr-c))BG~T$9(H2jeG9iqpD05Sm*U*``#0D*wN$BlsGn_~8v|e1WQ{(PH#l z?yssT$(M-BDelzgN6%}lzaz9a&IaG&Ts0#gV-UM_AQE}ZZG1;b2G#`SekT@(VdLON z+F^h^SQ^#WSh<5`(-Fukg~XrnZ(^{t1`;`o47Ts%V1Km#Qu0u5f{SnpfI;d3D2QQz z6ac3bLi2b5R1MKNT~eSGlR{=8e^p?Ko)nXh5TCXt!EO?PF8?fXGUyzNM;oOIrCI7B z`ay~|8E2ecVKnbH__wxT%q&}0Zby-WC3<+Fdqw&$4`~%OaW6{fzurAa&c+q9YKF@4 z$;}?ov6k9=!^f7D^PB5RhCb1)mbPJNPFQh+PtlmJG)tCi0c z2$9h36@5Xz6b@f6z|z2+B9KanRDFU8byn>aSd8#rfuv48V0B9li(rb< z+tcHmC&}1NA4-z96~H`j)_OXf))^L=$6>G6V#mY|w1&W}l<8h4JLF2yhFG(;{9)ZM z_x%wj97p^Q09%fm`+>R2(Bt=W_v5e9X6UEYIo-#ha#CE!Bu^wJnZFT7XvAA-DJdbe z>R*)>N>$5_)Xj?G*_Mya&Z6-+?KY3*%GiIseOcOBMrAoSoKA9>-6iB`+xtI3)uA!1AAvZOYcGzf{7G8A{ZYF>GxTixqacw2 z`E$G<8woeWP|IPeAU!IH0;pTN1VWmfTA#%$bi`dr(~@9$7PMoryZ533L31ufW+j3? ze^Mq=Pb12B;^8!5w3Of$vsmL)c{HBEo88rbo?imsYhqX22^y^+DVufy$QD9wlDGj) zXo7BIGA7EE{R;C@wl3n1^#8_>?s#CiCPUirgT52HX#A;vIO5Wi?&q`mGq=;HTirR4 z>y;F2f}P;kh+9bdqzhjsa?^BRMyPgVOgS&|odi?U@ zy%4z+`XYj!)!Ew@r)~h*vo7r~j_O%pW#dVY9HIOzHog>G#M~2Spyl0XLyqH*XBm5Q z%hf?^2ZC`c8->%@;KkDa+C>%Z!T=j4zD)6#aq)N5$c?kBG4k|^oW4-dsthG*6Vt~L z(x{OW*lTYZ^Ahr*nH-)R%*x#kQ6~Ljv7pJgh*(qQC@W{WTs3kM3U;{6byu#9q!ZfC zg7Y$u*oL}u9k04!yd`MQMj-f1lr1DzAJV)ERJxvt>b1C-uY?QV^pcaL9cjLg3%yMNhyh{dNK6FX8O z{=3Fm^O8UGt3Vv7VZEPgQU5W%9IKyIomkXKpN=o4fi;VxoY`7tK^12ycJFHUtjHp= z#KXBDETjJB1g8W4hYhuj4^>lvElo^Ummq+q4wOj?5j*PzG<*qU*whv+#!c}3;F4tC z27xIyPzl&yoE#YtVx5g#!ZkUErgSVuYt$LW{*{VjImzF#lMpsb0Aq{gbEGzDuv=P} z7ZTL)j4zCmrHk#SOuWyIIcem9ZjM+-cv!@}-G|#ctD$mJ7iy2fM8x>t0{Da&pK9U$ z5?jv^)cnDyk4-3gYunm}WKK_nSAxupZIKtlObq3(6=oAOhJ5BT!Io(JIEc)pDR}N~ z$3p70Ox+$ueBvbz)$0PN-qLyE`!{zzcv1hqKbdsEDJQ*Ri3j{P(2vMCTBXCyDseU5#J!mrkcXe~wywUhIz zcC^Ou(2C(9TO#ya8p^X1BRDaOQEzaRt8MbAxjyJg^|h?w5l4@d2C^IV^P)&f@`g0; z7sQBDx^^Mc?bF?flBFZ%PlUi2m2j$D zC717vMmPV+<4zwN_o`MRa(9#CQsqiFXTHmA0;fl_$7Qae0L_b!{e1b)nHwowPl^D^ z`spifawAXU|AZ>PKVrakiC)lNKP|wl=3!zv{2NV{h`0VIG89KxnR4Xs!nLAmHl4TH z5dGe?=VXczI1DD*jeJ-<31A;6Q6w_6Atj? z$Uf+MUj~KkdJltah$mS^JPzkcFrULvdXhQx7sppCU z#K~}JRFv#DXX0}lf3BJDSMZ7|@zFsuB42jQdeJuNRInuca==N#wwWX~VGVaHJXMGSL~W2juO0X5E@3!9QBG~3B1Sd&7;N;(706e+3gQY!sV zBXM`+Q4}0VD~35fF~P~2)|bl~rp^)#RWLPxIi#EV>YIDeo58ECnKE3?>8*qZst>~b z>lKy`wW8@6Qi3Gfj08wlxNMMwp02e6v|m{;a(}8_N$6L`z$~w~Id4tI zI>^USR=^p5uffQxR5L0pC6%A0^COoZOO6*yx|)^ z%+aa}ul0uXEc=@4Rl;-4mJtN1=%F;d8V#(#h-Vdin}>%qCxnD&vCxsC&!3Xk)Jcjf zemL?1lL{9VpregH=q#2x!A24Z68iaVe@_Gy)czFxADZ&=@@7!3qC%BKssA>WLRvi` zR!he~lFub`RvKu&b3e}?H~l^>ZyW9G=}Neb6WyQgoO7i*;%}U+M$L9u-e0DOay>lJ z>|j}duuLsMb7}#WE%+l*HiE84=v9z*aD{4>?4eo20UH<8eG#k)5TI_RB~miDZxYZw z{5D9l&ptV7nXsHaAlwg-Zg_Tx3N3#B;@&qpNf&cI$MY5bsRf~Z?aY3yRKyZ}JOnPs zN9(AKU^o?VaG<`OUFC121-32#^m|t@+>&6S7d9WiBTtwgMF2?;rG?*+??v5`Iqm9?b+E70G!Zbt+l}v zrPGbr5AZfHx&B%d>aItU^k9VVe5B~DKcXYA2p5};M<;gW;M=$ej{eZeePQV^C!2kV zTz%?W^rtt#OY1Z2&FKnO0;OC}GQBp)+VUo;--o5Y-2&;*x&OFfHT=UI^}rKp|E%RFoKw$J5jI}~)3KIpmD5@2m%hJ|#*f!& z00r=;+mIISs1_wpi>B1|^_i!w#KmDOK*^l{TOd?NK}1C>@3>hT#ZFi1cj;AD7sSn9 zr#RO*&G=SdhkR&`^_nYnXy`|E)g$a&>-cC-uj4;^ew&-$*Xm0H2h51dnH}VMo@M+{ zADg6PZ0Y-O5w@(X#JXv%k&E<+4UY=hdMgqapn^VQTvGLQCrl&R-eSH`KXVn!+GCg* zj@<|~dwF!%MQ~NkujvLa{3;9-&p*)ww5cMD^zFB0tb7=aet?|)Cn&w^uJ18#qvvgtx8peu>w0bY1 zPXzoVp<~0t>Pa7>DodlYx_RHONqz|Wh+CwbkmC*i`enTw+;PfW@oPd3HW8Cp*B1xX z#3#c#Vf6wrc0PpxIc1Tx!?M?NIoyIA?!m}m(+DMgp`XOgn#XBIEuug@QdC5IiDahw z4mHlJ4dUm=&zmm`g=DgW9k4@%c^w5-6mjf^veg}w_CIKgOh2$(C_s8yHsgb=ioFH z2S~mfc680VD~6`mFQ09a%?zV{Byye&ih$fX=Ckp65I(M>QCLoA(nWCOp>Q%n4JX6j ze}WCJ^?@)O!>E&8``#U^?nR0d5m@#Bd);JyJza$^0B*|oKK=ApZ(Uuh=M=Wq|W@Il!}#O9ME{3C0SX%tFZ`EF2lz=Gds zrStQ-aq*Am`_IMWmlm$xTg5R(;^OT5A*NmJg~TDOmEvNQbplpq|75iVZD+61ZjGn? zdYsqQ8og;8*n+)~DLUaiy(NI7XmF|vRWa5AnD!>nnA)>a{&gm$|HY0wEDb=P0<95m&*$)x8Dpv{ix|L2G|HBl;QdRTQ1)>;U)EUm#_KG44Z8 z(-Bi1VYS+>?Rm3R+Mwro7C){0d~<9Mi+9DQ)nKD(2Z<|uHklKE>{7HU>)rbUq{Amv zR*^t!pZn!woKOE|*e-i)LV-KW?dQOvt%|z#Rm{WvI%IoE>7H7QV1#S55&QX|y{1Fy zb$9$dV-8FbLMa@_)VHq;^K>ZtQ{pk~Qc(*3o*KgFV#IBat@;zOd_{lP`KhL7@_Q zr!S$+oy)|NHO3OMSpsX5=wd|cfEK=$rtqH-u#3|C`~v@MJEW%Mam4EC#ezSx-|qKt zz21G(6`1#{N*a}hmB45LY6N}_~kRu#^qM4zvB$vpOC#l0DOJA2D0>TY9XFGvXN%yQ3 zU9ch&!!V8H$h1hMA}qu94rQf9mC$%JSK!y^-jf(Po8nDKPlMl z-;p1S&iIL6h1%RsBVQJ>_&u^kZ~M!rD-HVUWJx|!tomL+lqP7ZEoArrl@=i8zuASh zrOA4IKqnXq9xWWtukg8ZS$;Cd%@52P%ENd$v2yw!lF~?DzZ=ixb1-hE6N)BZwGJQp=t0}jDc^d(a79==y42+!CxZn~?)=O!-UcM`>UaFq{ zmDZN1M7)Q{;6prdCS*R-sW$=TT2@oW&Ux-4cUz8*>$eC zpKV@D8TYi-DpP7@t&fb9jqkO-0yI&LW(Sh@8pqv{6oeWq@)g`(=KFsja@mtTtde? zX|NY2KKP*(8d+%&^ZDoAAdL<@W7%M2I|WW;*t6Lz(}5BNGnP|hCV($uH6G;Sra9;3 zFj!+@Eox2Z1OX^94PVuiRMoY58lB_#H0>T{x<3N90x%(8qO9O7TS9gT)w`r zt(sXJS~)NMcaeD&0=lxe6TFe9(+8#AT#xP$=n9`gw(AfL^(yz;OwxZ$5ebMR=^FAj zuI6|BVte}stP`5mb^P&~&Sot%@nhyIE>e`prpEK4CZDrkUuqm4;CD)^)Y{g!w^4PJ z^n-F;tquDcblV!)G5Z>;Tby>oiIt^__jXc$m{b?I^iKTcCq{Loi3_Ej70R0vLEzqT zf<}Kt8PjAGHyrthfZXsELBE%$!YQ!57P9qcSQ6lk!*18A=Dc-T$3d~eqRLW&K&djrXjoiMzTUufh@9YT=YxEX;o(6BQ-4Ho9FlY_=a6 z`jGJn%W1C1g4Sm-+YP@#>RJQ%QC!>awn4(6r8cZ7nw34#O(mJ(H5GlO=R|fgO zO7fQSh{^~Z1`Pp|+-C4?x=| z)GgV6FWc48)|MYIdB7PC9;S0rDrUh&67Z_QF4kKwFs!yXOE?he)mxzKsJ;?a34AN= zJl*MeShlWG?Sf($0+l0YZ-{9(0VbP(Th%f(!;8(|*;9+Z8!;P6HZr_>$@x4yRd?Uo zF%KlCLV~8IP<6)J5L~JqxH_G1QrO+6a`Lg!cT4g?4h4VL_|o}mKjXdCZIJg=3kLI% zCUr%c1Qe?hdso*NmeXe9^5bWjpJ9G?+YxMYbazlGup8$vU@{?LUMx87UG!<*J6oAg zZluXK*YCv!Eg-;e9v>rU{aQy^@zYl@(0}BdbNl!M=qe-lQc5tjFRoN{#vuR07kis3 z3jzeH57<8(^Eho7i^_dkm7?5~1p(OW$A8D_wh5!Jj{tyQyZ8Y3B;N6%Wx6z2CQ#!d z;MEYCZKkN!Og>LkAh3IWU?@3Jh&|m49phH| zN=@i0>AXUnKe4|K`CS@nnsANZ-~H8WJ6T6KY3sVMSLWQ!`|kb+wpsmSlwX(Q1?t#K z@Nc5F{-=kbMR!N2p0>p)cWm&lUuBw#`S7C*cbH>8`JJ~=uK*-?t}3k%j82#12S9Vt zwM{WBVGj3S9UUhkuB#O2eq}9K^t~0hI=72WE1katF2HkN27;{oz#$(avx*6c_f5tG zr)T}=RzYYMiXgkyWmS0$&va(hkEL4t8mg?pzV-Z*&%}FP7#Snh`XyDow zr0C=N?2hsH4G?-E@^LQm;K^5LL)9nm*IUk1yS3w2eR3 zM|a$Ld+F}@7=hs1b^anmyPYRRv{C$>;D2a49zczy2awmLJ;UEHg`x!AzsaIu12XCt z5}^Pobehb<5*2q=JkJ7YLeCwP$w69s+3XzTgPEewK%yBjR~5AyeKM@67PRj17in@F z56uAtsYN<4&#nh3^^DE%Ggs>I=92-oaBeJR+}_{4Aou$`i~4ugwV|nWhUx7pFO)TK zYLf1L2M!8bd@`12?!x0+b`vUM(7QMy4pZ|{ZBj65BY9+Y*qK>Cc_?|!OcEW+so?VD zg!L(Bv#0Q?po#s`rm>AXrh1H9>(-A*=g zD7KWiy~cP00wx2nXj`G88w7Fq!iyzNDj{_ximqjz=NEFBDex46o=Vj2eBBL~;zY>e}_@eN4{I-x-wjkxvSc!8iyKTv6=ZdgQv{&aVa z)ki1xXYcAG}42DT~_#)MP_!rnpu?AW{+&Q6l^uvHU8soGbTt{ z(19SURK2v_t6J?6`_OTiu)_5WQEn=Pd;@X-#5Cj2nf}42?w= zP;%Az_Ua|vH~mp@{l5`Hw%ZMA$5m{kMlOwQ=vG9{DH){-z)E1{-q(0Eh(shD@k3%( z|9eGo`iP6{2b96#g36Rj%3m^R9*srr*d>w8h7%3sR@H7axpG8EJnm#!Fn!D;mHE7H zzPZ)3?X3PdnXl4^18p%H1{l~1saupwpx=5oN#e84pZz&wMPvG1`MdCBx&TIVh;@SR z!}1}8EN)am=;HsySA_aO0N(&QN;APgZq{G3n^M}r)fHn8^HO9LLlB}<+ZRp}O6|yR zM}$uIIu5v_4xkV>Z30LtV<3}z4>&)Vfr7r-Fb9m*gYXOPy2#3o{l>rcHOCIE_!E;O zf|LeFE0oT5Z4xAco*a6y9sult$K3w#UtySkt4hn5ZQ-r3TgX@)TEqjk&+ z*KFsw$jTO9@3;b$SNNIWatoXZpcQlYRcJRE0VQ<75VCWXgL=@sh&|gsh5sy9o(%A^ zlGYs5nczza`e)53Z13clcq8M5EiB-YL4NSLIMdz|-?A>xIWngxx}3Y4wy31-`S`D- zKKj*g>kr?dcAQf5CWl$+FIJ@{4xxWm{t0SU9zZzz0;rH&Ax~2HpZ~Wz5yEmL4aPe6 zu{>PeiQpBe2+5w3eEXTkF}5$8ZHQb=X+Tqq)&^#e+#l&MfNDg&z~#e=Fi-tx&GVoO zrPf!`oz^U-QO^R26AU2-vGbbx)^)vQoF%Nj^KwPshf@x*9GvG{G)!-Q#tADvRPlSh zekKX4GyuPq$D)qmSNUP6i0<;Z?SBSVe6j56YDJB)KZe^$^Xnw)L}_=!cxV=bAaY1r z=+GQ$3vz>m1UU%}To%YPJm4e#n!Un0Q)Aj2l1hlFa+a>XApaVgR)_uedV+LQLRX!z z>Q-4a9n~}Q2CbzxkUWMk@v_7v;`KTxwc1dCz5oAw_=-T=pXvSU%Tu-n7W*2ob;^ae zWC+>eIXOSqpW$fcLvyf#A~$&V+aTd_5VNwP0g3tdZpR?kSF_Pnze0Iiemo$*1zv+? zyX%SV8tsqmf^cU~QHuTSL9Ly=`G>%x{#xXESf#+4h;F^MV0aHP?YawFWmg$IaqzIn z(uxb+0{i~Hyq{%SS|@35{{NPATmeX2 zM+!O3nSZ(z%DdqUR9W82sn?G{5?j*qVeow&<3#XdH%UEE*NF@$O!lCCAnX+M>Jq-o z;QcS6h770ZK(UWl{dMMNUcWZ zYJZMIBnYRJ8}@N6vAk>H``-t&_!GiX3bity{-${9_P}LZ|6B+mRTA3&`WCPye|3U=Cn2^)4`4m#$maXe>tEEcYfn=!1PMm~L z_|m|Dg!CR6ngu$DoO3&~>2~v)jH0<@ZtF@e19|kl85f(^?Y&|e7aHO0{YZ!kfrxTy zv|ECj_;eKW8uPgIysPu<#)6kpwI^KU1(6XzpjhmKIt3YX+q(&=(9@LiIgQgknsS^C zebwU!e)E6sQa>v+Vk9Z8c(#I$M)-bOc9Gt8Ac2p~Zk31}w5W&V%rBeWiR@7sg8;?D z+3y>Zyty)}{M#m6T9!9#F*FpG@AdxgzXDzxs`n4Hn-Yo~6Qe?Vt457}YN3diWny=8 zeYtUr;KcQbAWVhZvm)CcR;IOfXi_g%EREDF&%)2(;&Ozv`{EM^OmXt=Py_1Of;wtgwtoPJ&!hufy+aB@UbVny_dn|aUEB91tdr}sl}aJn z^%l`eWWj$QPYX1QeJi#FgEeHxxto>OTes86BZ+z zt%L@PfY1w=q>wQ%pn-7EhlniSSDbfb2w*vji;V@-s#jv4UBbOThinpLVn?a74zjFI ze^L!aCs%d#D(sZ_^`t3`%*W6JX?=3cx3uUJbwdC9XH`IwtnqH-NS5bgl$QW@MuG3~ zAg>3i?v?|5Yz*};NvKS%;sW5`B9OWP<=Y2~HF-I-+}zyn6B7vK31|b~2F?uj)xT)( z@3)XBv5?F>-*g47Ubu&RsI-pi)>5k~t*jR_e5ZhbzqO{QL66V>4kt+1w77aQOGz(# zjxcqrL=pU%i}R9iMJKi&76w2^?2zGo{6H;G-@pJFfRs_-y9~?`1m=Zaz}a-U_siZ0 z=QLUg$geBYYLgF)6GKSPTLAqLCL6mqy1$sNyo*zO_TobC4JRd9U!8~J`D zLhe%uX~nVS@uv(hpSl(}>lKUwfF(@-SpfL_fti_EndLpmG>Mj7fG-}6Pjf;gpZ;Gv zJrK<#6BN`R6h79OlK<7z)U-Lad5$ob@#X({0nQnlRoJZOW%Ut={<{GEi-3SzJkxDQ zW;TLtJb!?^3#ZNeaC~&+FNI<}Stj9*S^4i+sjlI_V_F(6R_$b^iBXc5$EZ%TUHLBn zbLPWj)B_3$ZVeWz4Wp|=j!mGD{2^YZr(c1FUU;bFKpS!K;2irMHCbM5BEecJsOa|HP{39 z8U#3vxM28~^kK(M*uU9Qtaqw37GMt7TBa*=f~46MUs8kX`9H=c#|5wIOd%>{2?Zh| zhz?F7gKkt7H#oFVs-*1))MSC&%gFLq^zILaGo)L(9hO^aU-rl;H&;QDWYa=p?^T{5 zI9dhz3QjTk|MaAkW=3uSK+_IWZ?h5QBrK^ zyD0TkN8SPH6dKoTAF1?Yvck zPzj7Bo-qm!!64ld#75ezrWUwQ-JA@r`{*}H>h8o*1O8F^Di~I$8~o=X5Wrf1=Wp9xkCP3%e4ZMYxL^2Wo`5?x(!O#U`Bz(u z(p9>659p}&TWU_cjbc-%9aKKMMMd(H#h?-Os-F$(t*SE5o}(5;$d&$v?v-gvA<3D0 z3*M?!plJa_i32;yipz+#R+R{cS4CI1+h5|2!Rc zhnhc~va64Sj#7mHEj>uz?hxx4I6wiz0hvz=%NiSQltT<| z$v2cHDtv^HxeId{yJru>WxdmHo0va=wVA{Hh2b-BSqqJhhVdleaVo3zpfIK$90lM# z5$}v4uP-xFYr_fC*7#?6|efpq%(qd5W9a00Ps!RuC-(%Qe;aDqn_ zOsRXzaZnisc|)#I%b*}r>k$;w;<_Lk1^mbdJtnzbf|)1oU11)d9* zFPqv=cl=hFkHp7(lc|5qU8TXGle~`ao3)Q$T*R7?Z*7WazGBYnstUV%C*Tli?hAu+ z+W?ltECb|Nm6Qf0JGZ-iOONWX|2T?dCQM*e5jwk*45WQR({j3gqrsGyW2k~EPPio1 z)H#LGRy!rI8L|NDH8nybd zLeirtQTdC2nNI`64-^{M3w7ZnlmKT6NAWgF=-vj;(<;1K43>n0_#2N1|MF7;I@zT{KK* zePISp>m_Q1zC9qn@y|~KjZGe(rbt{*BWjht?&f)H^l!v>NpL-Jhwpp4LIuXLDKMhU z{WyLAP#@Jm(Z^_(pnmFzIxVI%nQV449NQW*O0&JrV~$jxuV0~STRjpS&Ntzw0U}=w zjQXUU+=qHXLc*SQEuN-!jt&VK$V;;fh?dPX7@YWV`ZYT_=n`_p0=7q|!-FVsZi*bK z+=M;(UuW~v`A=+*hiohVWclv5fIj&YlV79ufS-Xh45=IKr*Xv)=mGl#bt zyLIH_Na{4X3(jPt`cD`6Vtyz~duUP@IJiLw^8G@xQB1~75hON6Aw!aSmE!>tht?p7 z`jg>75PHE^kxEsPNq*#b7j`oQq3=pOP&-27nPLY5-hP*AKkJxII#MHJ#~<+uG*gd` zKEl&BYoE zSgYy$tM6_}AF@b-Z_3BMZELhATc)t?o%J*QoN(TJlpO`LwB`KxDGIpw%Ub^8hZ|M{o+HZ< zZvI}FnufaoItNM@3AHp=iIfLuAn8DSodKcAtDpVTcz-RGeH=>-uDO{F`qZ^GX*K9< zS?M6!2L`=#B$X8J0tR7$Ma7C&7~vm|l)(4imDVN)%{;9y!!TCU`6SE4I%KgP_g=5?vncar3>s7_UAjupLVS1n3~TWHRA z`6@^RAqIGRjjyArEo&d>n0UYcoL5Esrqkl5K7eRV%L zWu)Xpv@u@UrI)Y9Qv%?al(Ivi=ZEn<jQ%cJ|t8aflK_W*br-AVGDW#WYLtg->MRC~NfnN1#_${&_&71Q1FH7ac> za8i@@I#rW~QyGPy-ErI%&l6>oOB$ivgyn2PWq~egIUoF|!B&0*3xwfjU(z>p7eU6r zP374?c(slEkaxG^uX)|K-2OtAC@v30;O7m$kMnPYgFO6jnkm1dEi#VFQ6cG_6pj%h z5&eqT&li=EW{DMQY|lH`P=L2W4wCXBP)Vw6gv`8xHiHrRjK(@GMv1H{+t+r% zY2tPgQa$WYAFohRY_80uu8sNf+YHO56zfm-iG>aJe+09hg)YMw zh*2xAPr{LHvYSHm5noHDH$P93^Lh+ugsKxnsP|6D*k;DIF;yJ$IPb}lh>O`a=o(*2 zD=AnW#sE4F?3pZ?r8TrA3k zj{v-(24&=4Vb%~dU7z)AnUYZHP7l7{jsI6r>alkIxzy|eHaO^8WRI$q_&O(I*^A#2 z4*1C6Q$1|3c#;|V;kj56!BFCp(fmil&?3_imQTd$cx2J!GD)Pk%n5W0+lge`6?5Gj zj`^H-t(X>7CCmZUu2byVoC^kZuYDrz{{3YXBq8sqCFGSxTT!5O7`XLsUfjy!71wwa z48wdTj0D^R-mee){$L7s@poaKB-?tOkY#3EgTcoBdsSwp%vo1v=AQznFGuiU@vXN`Tt*aQm6hIHMn4}`zfju;&WmJR7?flZy zbs#YPV*_a`ddNes{nT`F<#Y>GVQnx%!7ODQN)A*VBqiLFiBQm1d8ReOXOb6;#y@Ki zonp%%z2&<4yO50#(M=D?h|l?vQNT><|28jmzTVBwrwW4&RaWwI($cj;)d@W-;6_VR z3Q}nSj5A!O=N^u1DpSr8#%y(uRi$xCTLBYwf|CK=W012$w`}B1a@Lq z)B9x$Nn350F;uMRMJ!M&>TP90VB(nO>U!w0jK^B)`(g>O?;kV}*=0CXt}R+dF)Y4g z**A#1w|f)$J~I>gK0Ce>P8+5RP08HQs#Go$X}*CVWiqr{T}qg;#%hH-&7l58fyeWx zHkVfql0ELys%1k)y*zx5;a5i9WzC=FzF;@4uDW|nMIGuh%BdbaHi)xyZa+!}p*jw* zwoG>gh6w=?d)I`z{8yo9aLrUV2mFuxp5$4&KuKq23ff(Gk2^Lj;o+Xzb&(^Vh|xgn zZUp8~LQL56%+8D3e?dh&kc5!CZA3C-a1yv%iKx%&WX=-ULjL1M32mXdiU_m}6R!wy zo%D!s0)9QpLNrw{MkCdvUvD{3;)5TlHD;|m;EgAOzitICXDjSM7o7EkmK+9#&V6J> zDHK{#CjYuCq?Ch6xJI|^MN(Z}#@vuUhxp4P|OW{SkIubs@W|Rn`itOi}3mWAIA=d z$#OL}3?%>Vg#dPl>7$U6Qs)SA?k{`ne~EaqH-Y8~_GQu zv{!@c$Z}K^U?u<5rqN)9VPW|P$U7jr;&9c5k$rDA_zk7g<|cQLPIk1<^HazcZPJw^ zM!cjg4RR-=*p{`K|9|Q)0KZHSdP2s(Czjpd#1P>hAUTy?Qj#yZK8i$-ye@!k-?e$2 zf~`ZY1jAg~FEJUBBb_$ilbTd^zS=5PtPi_hY+H#DGHq=Wq6T=KC z>uP`xVhUPnoe;eR#IyJbYpe@g5^!f}E>1X+b4WgH-FfFS^&Igp9*FS&i}KK@K}6tx zLB>T5@BT5Rm?Yc+d-+6{)D9D7FJrd|c|pl(%*(=bsMH`2?Y?#V)6A59V`9|qDrgPX zX1yH;;3A*5Uw`6pD_wScbE62J$*L~%& z$_M{B0m;bM|uMCSL+qx!5Ah-kw z5M0y265O5O?(Q`15F8qp1Z^zAT^rXR32wpN9YTQM4sT^xMuB*aVjees!}^s z49f`J@_=8C5((JqD;26VYNSs(=*PWPO$oZ%BTFgS!TV+K*?751fanFUN_xWSN-8wx z@~xN9#&8uP(2jQutO`m1!UT{R4_i~tN#@tg%mhiDHIxp}nwLU;FP)s6epk_yEM<3# z*eW^KC@3yFMvEeL?arzhCNiU8T+}(- z&dUrJKOtDNA6n>8QTbvhZ1S|~rV5>>zjozDb6yO`D4fG!zw*>q};pxIu_%{qsZ8w;rwP+_ZU>9gDxSyk)8Xd zf_RCl>WGT*4+bGbWX10d__zbenN&79NJyM+v3|;?g~*SbQv1)w94PWoJz%<-Y`A)V zoEk<(j;-k{emQ(35HoifRaU{0eQAON-mB<7dnZnJn<`y^IWujPWgq0SSG@(tDY<;8 zWqYs}*XWOU+rIp9)A81KXZk}U)8M040F{r^F>M35E6|LBza_EepoX+Em9vGT>Y?xn z|LKoVnpLjy^75jhq7I4-gUQaBd-r^M;0Tv>sJc6`{Gt4^s`Q(({Ql( zerP=82S2`=;-tcX6Z(e0J*0Nn8SBcJuNtA=^ax`D5(VJYe=|PwFt9^@bq&_ve;C<>4j@|oa&D1KNhB&Ijd93rC z-e1XtOSgFOY}duZy8@M@JZ;f z@vx|n{QlvF2}P2z_c`YPMAiQix8kI%v^S|jS!csKmqz$CSh}$wKVP1BO{N0bZCkyv z7zTR>$Z%%>HWWTPeBU{4);(EA#@BSj8NbK3^cC$2>nnWG^q|p8CL&He8E?Pnh~38L z*Ygxy?^Ysowxr7;7TyZG198m2kpkgf_DX7VNKbynasJK zWJ2(U$MN{fLax?n$R7~G&$jjyzA z5pf#BhnKtHV2u-Jqb8UV$E4tcPlCcHW1O~Nn=G6`=sR|wXZYW>jWl?b#Yna+=dxJ& zXujIIOF0Q`s9PA!Om}EBJc(sw%`8ad&b4HxL{D_W=+HTelT{;)@L$m#2uwrFrTF|Q zEv?dSf%CV8GXr3ID2ipu#WU`IU#07A)dZNVfa`R4vYC*a2Zn}RJ-s|x1VU`C!+iT% zT%NbnFvAuSYA!4L_cSP4tMv&HY}O^^Zu1t0%iMNMQ|fbYQ8G6)zH|Li9{C-mJGfE$ za;(wuP_qLh`I6kNi%4Z{eLc%Rj)RmTz5-ZM+Ko?jgaogCQmbJc$S1QdyIf52TGu_D z|Ky|dvffejh0;+WpRwI_YBKqBVrHe|c@AzPglqXd0=e%Q&SBfFJ^y)PwH$>bY4pT+ zaJc4))82T#vug>Vp$`tdVurY%-vhTkxxb6znod(8)|>~TR%7I%;n{TT1!SD0(~%r@ z!XHN6Zsq!AKG$$D*QbV_=SJBk)U$Kf+9R<|YDaJReW;C3TSVd#VL}{rxYinV9nrL+ zCRsdU6tyzvef;@3-DI6tCCtv{DCEGFOH^=WZ5mqlhLBR$FEqC4FfL*Vbd2%W!unJ8 z%n-g_%;DR`p00sgS-;Rag}i|pyymY`!+X^9^mMHzXWHp9Z9MMIC$-Nf1z;I|zh&tq zbCogc?iO7}*3R}OGSumQE~_@$XK%F%2HVW&A_1aeO2w$FD5yKzx%sUG#eNyGy}C;2 z$&ul^VDXU%$+L4DrTj9n^FwCa31j6ryxBmWvVw3 zDaE4Q_H2nee9cDoux^lMOEpUqn48Akhr2cF^&S{4QnLixskQTxXDmEi$#`nSW{fH4 zT3I6M84{>B-ne-sBgFg_sOZER%HXfI*xEv{uLkOVw=UvGpjp>AmD}v9g;|&N-WQ~X zgGn%U@t~@?opXUV^6T)UB%W(EvlnjtE)8tiw!A&GeB<%UK-2FC|DQZ3IX&PE+8Gi! zcsSsQIVd9@-i@ zUf=qp=@PIYV+)v6;);HSGrB_rE%}b-{Ym|k&SweNtzRQo!DOYV>9_7%GI7>Br*#o2 z$?-_ARo{att>l*So9`p8$0&XCh%?nIrV@3s9*H7!{;i^-_|SNVry$^&F7>giA>)iK}xZ8@Px(lB`Xe ztvRnS(ZXXWZoW0<`$9Us{RrR6(RD3&8RqkEq257p3g=^De$)3)S<2h&oV)3Wt_U~V zGaqh*Dt*U(O7s%A2$jjUBFK5FZ$f10epe!XciJ~J5CYY;7#73O6lE{LwRPxSWOQ^F zU_TH5kcJ@zg+f=WLCVd1ZPurqw6)Wl%}AD@xzi_a*Cnf>lam^6ei)DzC*Jrl3vpyM z)nEk?-i#erxWLTZ+^LB5>9BQWOqAtvVm@-@gZU)wo{o8B5zT#E@4*#__7m5wcS5UI zdk!y6tChalia4({9i72)JL*w}0A)5lHK?$2JA><344*}ggx-!fER1>>V+Y*}fSFbK z1O5C`U*;>4CVyzlz)WQEVAnQ?{0JO6(P&^{v1}gPCf>J=K6$V&@)#m==06wcj6(ry zg?lXnnXuVFM8M(TY3^+B+9DjDR5Z(DN7V_K?~5`d<=m|MAgj&_uB6Eo<5 zQGT@DanT^(Q5@9K#1iQ5zoDNj`%f-H$NB`ZR+e*hYNahQr!YxD*uk$kt37P@C`T39 z*yfh0?1wwO7 zq;v<4LFe;VBRD(MijtCVKrtJ_D;)vxBUCL|^$fIkER<$uhUR{W)(~ z^@m3TvYQLIG1&ueKdb(qkvXobu4t&i%(?~PjP7{Wb%*6zx1u|^5exj=^ z7^ZAjN6vbj=2(oBc!0{sRGDaS|5D%?!y78NYa#LN^p7ZG*~GhK>{(4&a9$=;3m*4i=$3I?$GNP^HB!#z^QYyZV18QzoWl^bR+%qj{}u8 zsw98Gjey>KD0C{Ir`yFpOR&?yX;XECs>jO=50GHKO_%H9kdwz0X8I8u8?R$sJEdZH zZ4BO73T-lIuvrXs0F43WPh{gF92(a`1F#mokBeK;RIhoTuN;7Wz%RI6XarH|g_sYM z?klGy#z5S0KhST@iA7m8m|nk8()qG1n<7dY!@q&5M#u?&;BAzI?ijO{82Ckf*(yLh zQ~2rheuyJ}J~7Rx!OsJ@Z!c5(LWjVNIJE=5hX^129R)P9P#Q8X;NZ{;rNu?m9m_6Q{Yzi|37sPp zJ4Fz${!`Ld;z!@rwEc>B5yeH;u+Y$7J3^1z3Z`Cw{URk7m&ZCUmat;N`p6?toOvnQ z;dirF_}fS{C~0KIJa5)gXmGm6#A=7)EdgRcQCNg)dZAGsDsi4=1>8JI zU;)A|*#M$~JBSG_NzzC+-jdEX!I!*Puau;ffMS`6$_t*|uYK4Id@n#~;?xIUNPitJ zRUDd3e;~RDzqHf}7fLuho?0`|C>K&JReoM8d25<}F~nXOqLa5p055+*v=AE4$euVm zT-Nuh>xQE=u3p?@&v+AG5#ox&i#MI8w>k|?nXVzB?Daq(R6r_b;{-Sce`mt4!vdJr zEoSYTk?WfVx$zZZ+oplejg54V<*z_v)Kke@B#!A@Q+Lp!4n=-x4Du+Gi9Y}{JhHbA~?~>L;@8d=~ZHhiKftSnMY7CYm z_qOJcVXiQBMkVF=>}_e~S^@d)T}zjuAR{%?{Igm=Vb$<+MXC^_(d`Tu)kAS!ejvKC zfjpqN7HynmBFxbJ;OoR=W5A6&8)|!D$i-J&*GZe$y5^4Z#~u*%uGqV^1yY7%zJ;HI z8_yWfK4I8=^V05$55V9#<_eAtCvEf@tD`rXTtxa;sv@aKVMdHy5W?hB&p#pa4H~nG z%?&W7`4{HL$02}E(%o61P&Z%z1DVi{E%%a&qPXOQctWHyx3?{{yX@y{r3os}ECMe2 zK3?_}eGmO^9%AG5RpEKk?4=L9WLHw2@vr2)jf0+Y-y6LZzfX^^s5s^<#q}=>lKf`_ z@rk`pz+9$@N9q_-J2YVXi^|s)3QNR|i_vF#YNYVu*lsk?++1TMVIBQw&ye&g}CI0`|5#j4Ao=+BzHa z;xw)Rl!L;yEx6czma}JExpb0s7e|7H!7afUG3TJyelE*3!u0oictXdTk5U9f{$?Lw zC;6P!g@D!WCehg+#48JdCOtF!@*anT(>A~WmKoWpeXV_y}gxA1a`di+!~cy;sh_XWGpB}eOo3y7vZ~j z4(4pMD+{M_d_9_cCDl;z@8N#@deL>2QVt!`pPiVB>eGG}emq*y=1neU#lULF-okn~u;AM2lfb{)iLGXG>vn zyFY*Q{Pbqaj+&qh5{w~I^86m%Gf)Vv6;AIR>VRj7s$iDO*dHazT1CjhBL6$i{;^Id zq!7VYtV}os`QHG3MM9k#>pCh@|Gn|BL6RCzr`N_w!0PtqCWVj?d3JX8`}^~mDuC$- zScr}1YrruRn&sr`)iW$unB6`<_;@?M#~mUeo0d6 zlyfCr^YoGpVb~jw!dDv3Xr)`v*<}KKj6<;hpJ{w(0h=gV|Sr_lKCfAB`6@FcHETlXl1fO$;C1s9uRwJKjCB=u^ zh0(4ql64g@c9zgr{|W*nGqjq~>qX&Xl`B1ENXH^T$wvdU6#nZOs6yb7%t~OeUZ7Jo z8j!sk(1~Q9f>P-1p{3NWtKYdU`rRJ92Z{{G0G-ZcrOh`hzYB1{%iup}`VoEk@=@y* zNaCYPN|t{kdZeFQD$kH%T1=0ag_IcX&;b*Jlp>V+@aCYs^~GsMk#I6K)%%%4RPmME zt`&*B1lga%4syK&k$1B1R0VTplE0eOqp|I|5`s>;PAr5q@7fJ;qDh$$oMl!TC8`qI z*@?~l-}K))TS?a1Xd-Qw6BIkRGPvb!D-(})U)U8bPeoSb5>Ti`*=}tKBP2<{#7tX) zD$FDY_nSsv7dyySdSw%j(!5>?D>fsFc1SgH0;}x>KQm`hV_2eXc~`D7@GoTUk4K~@ zj|fhJirc8F;=O(Q)(k^J=Z6fh=u>25J5_Vx-Wc-a!uD6$_O7n%NsQW5{`OyvdyK;< zNGl||Om3@Jqa!gVx6tv3E?v=7c6Onw!n~$z^BLLqFakALw50t+r9L;ASyp-bcVnX4Yu9GE0NJ&d6oqQ7nNxV`n(+y0X}7`r+Ij1e%IA6cs~^swu^JfW zoI#sv-nOmn+{6Z*Ucc1crJ;Dl+aP(`8SXEW6g4K;b`Hs2`AgZT&7W0jNdX`TG_hzU zJkmzW+BoNu5`-BtMPBKM&wceTTM{6ZP{+vZ_Uk;;pxuVaQE{tP;yduHyU2Trbk5F`4@;H2atOmx8%Z>` zZ%ymw7j`H+oafOaUcf6E7KZewM=wMzJwO==v}pJ|&N+y|Ig%qw!`LhTY&d@}@W59x z9TqkHY-M@A;PsESSbJos)+9uG8SXBTm`>sig|+Y9NcFwFTNq8Ghg5V(tMaaZbpv%7Z!Ig&%6>Bl@XbOSme}{zG|;a*LQ@mv zdHNM#6$`KsS%gL10r#d9A^$c;v)?{v)XdD7#KR0pznY1+Z2q&e{yUNh2cV#1Vh&J2 zgF(-x0Y8G(yX(^wE{BXVUGQ6GT3TA+R(U%%P2isItrgOFX&+f*WAn+;c_*5W0Sdb? ziZ*VIva)i;@KxHsw(WnO9Gev$0Rf>}m^SWa&N8!qCQlZOWdr0zj9OOT>;Smq$^NO# z8`r6D%@G#4cpB}YaV&J-U8Y{H6`!EJi9%C=d@^9!6HPQwXTzgE!8UI=i2UEH%?MtT zJ`ms>{H9^qlj9a;n6GEBBa_moZES4de0Nl zuRim6C^Qk6)A(HfPfQ0!-iVtO!5Ud5mI){rZ|NjPusze9AMfkU2Nd_btiEi{pSeq* zXnU1@`+tn)2~;LOT+IMVw3rE~HGSP0oc acqbI>AODj1VI2_;_()4Ah*yXjLH-Z78Y1!l literal 0 HcmV?d00001 From efa8bf1c3729396e445938e965a338d1c91a6738 Mon Sep 17 00:00:00 2001 From: aimeeu Date: Tue, 6 Aug 2019 13:43:56 -0500 Subject: [PATCH 18/49] Update link to Docs SIG agenda document (#15703) The basics about our docs/Style guidelines: "add it to the agenda" updated the link to point to 2019 agenda document --- content/en/docs/contribute/start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/contribute/start.md b/content/en/docs/contribute/start.md index 4a60234318..ace1dae1eb 100644 --- a/content/en/docs/contribute/start.md +++ b/content/en/docs/contribute/start.md @@ -52,7 +52,7 @@ formatting, and typographic conventions. Look over the style guide before you make your first contribution, and use it when you have questions. Changes to the style guide are made by SIG Docs as a group. To propose a change -or addition, [add it to the agenda](https://docs.google.com/document/d/1Ds87eRiNZeXwRBEbFr6Z7ukjbTow5RQcNZLaSvWWQsE/edit#) for an upcoming SIG Docs meeting, and attend the meeting to participate in the +or addition, [add it to the agenda](https://docs.google.com/document/d/1zg6By77SGg90EVUrhDIhopjZlSDg2jCebU-Ks9cYx0w/edit#) for an upcoming SIG Docs meeting, and attend the meeting to participate in the discussion. See the [advanced contribution](/docs/contribute/advanced/) topic for more information. From b9f6598c9ea4ec36a12c56a1dcce860f062e77cf Mon Sep 17 00:00:00 2001 From: Patrick East Date: Tue, 6 Aug 2019 15:08:50 -0700 Subject: [PATCH 19/49] Add "Admission Controller" to the glossary (#15196) This gives a brief description for the admission controller and links to the main docs for them. --- .../glossary/admission-controller.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 content/en/docs/reference/glossary/admission-controller.md diff --git a/content/en/docs/reference/glossary/admission-controller.md b/content/en/docs/reference/glossary/admission-controller.md new file mode 100644 index 0000000000..352ccb4f52 --- /dev/null +++ b/content/en/docs/reference/glossary/admission-controller.md @@ -0,0 +1,22 @@ +--- +title: Admission Controller +id: admission-controller +date: 2019-06-28 +full_link: /docs/reference/access-authn-authz/admission-controllers/ +short_description: > + A piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object. + +aka: +tags: +- extension +- security +--- +A piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object. + + + +Admission controllers are configurable for the Kubernetes API server and may be “validating”, “mutating”, or +both. Any admission controller may reject the request. Mutating controllers may modify the objects they admit; +validating controllers may not. + +* [Admission controllers in the Kubernetes documentation](/docs/reference/access-authn-authz/admission-controllers/) From 492f6eb6c31456385ec204d9f7950bc9670e48bc Mon Sep 17 00:00:00 2001 From: Ori Tzoran Date: Wed, 7 Aug 2019 01:10:50 +0300 Subject: [PATCH 20/49] streamline `grep` usage in minikube_before_you_begin (#15360) - tested on macOS with BSD's grep and GNU's. same flags, same output. - there are some 55 CPU flags on mac, so highlighting VMX makes it easier to grasp - using `egrep` on linux and `grep` on mac, not necessary - precision: the VT-x feature is NOT an OS feature... --- content/en/docs/tasks/tools/install-minikube.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/content/en/docs/tasks/tools/install-minikube.md b/content/en/docs/tasks/tools/install-minikube.md index f7c66a5dc8..278cf47b3b 100644 --- a/content/en/docs/tasks/tools/install-minikube.md +++ b/content/en/docs/tasks/tools/install-minikube.md @@ -18,17 +18,19 @@ This page shows you how to install [Minikube](/docs/tutorials/hello-minikube), a {{< tabs name="minikube_before_you_begin" >}} {{% tab name="Linux" %}} To check if virtualization is supported on Linux, run the following command and verify that the output is non-empty: -```shell -egrep --color 'vmx|svm' /proc/cpuinfo +``` +grep -E --color 'vmx|svm' /proc/cpuinfo ``` {{% /tab %}} + {{% tab name="macOS" %}} To check if virtualization is supported on macOS, run the following command on your terminal. ``` -sysctl -a | grep machdep.cpu.features +sysctl -a | grep -E --color 'machdep.cpu.features|VMX' ``` -If you see `VMX` in the output, the VT-x feature is supported on your OS. +If you see `VMX` in the output (should be colored), the VT-x feature is enabled in your machine. {{% /tab %}} + {{% tab name="Windows" %}} To check if virtualization is supported on Windows 8 and above, run the following command on your Windows terminal or command prompt. ``` From f2c7919fc9b87fc9c7df058d64b91da7e34bfd8f Mon Sep 17 00:00:00 2001 From: mhamdi semah Date: Wed, 7 Aug 2019 00:56:53 +0200 Subject: [PATCH 21/49] Issue with k8s.io/docs/tasks/administer-cluster/dns-debugging-resolution/ (#14816) From 322e88de9abd498e24559d95bbd683b3983d1976 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Wed, 7 Aug 2019 00:00:52 +0100 Subject: [PATCH 22/49] Update device plugin documentation + related pages (#14331) * Update device plugin docs for Kubernetes 1.14 & related pages This change is mainly about feature-state shortcodes & similar. (if there's a way to get the short version string, eg "1.14" rather than "v1.14", then that could go in place of the hard coded value in this commit). * Fix code block formatting eg code blocks incorrectly marked as: ```shell * Use glossary shortcodes where appropriate * Hyperlink to Prometheus * Tidy Markdown formatting * Change example vendor domain name Use a name inside ".example" to highlight that this is a DNS domain name. * Reword device plugins documentation * Tweak headings for device plugins * Add "what's next" to device plugins docs * Tweak wording for device plugins docs * Add KubeVirt device plugins --- .../compute-storage-net/device-plugins.md | 126 +++++++++++------- .../docs/reference/glossary/device-plugin.md | 4 +- .../extended-resource-node.md | 10 +- .../extended-resource.md | 10 +- .../docs/tasks/manage-gpus/scheduling-gpus.md | 96 ++++++------- 5 files changed, 133 insertions(+), 113 deletions(-) diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 26f48b79e0..d59abf97d8 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -6,31 +6,33 @@ content_template: templates/concept weight: 20 --- -{{< feature-state state="beta" >}} - {{% capture overview %}} -Starting in version 1.8, Kubernetes provides a -[device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md) -for vendors to advertise their resources to the kubelet without changing Kubernetes core code. -Instead of writing custom Kubernetes code, vendors can implement a device plugin that can -be deployed manually or as a DaemonSet. The targeted devices include GPUs, -High-performance NICs, FPGAs, InfiniBand, and other similar computing resources -that may require vendor specific initialization and setup. +{{< feature-state for_k8s_version="v1.10" state="beta" >}} + +Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md) +that you can use to advertise system hardware resources to the +{{< glossary_tooltip term_id="kubelet" >}}. + +Instead of customising the code for Kubernetes itself, vendors can implement a +device plugin that you deploy either manually or as a {{< glossary_tooltip term_id="daemonset" >}}. +The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters, +and other similar computing resources that may require vendor specific initialization +and setup. + {{% /capture %}} {{% capture body %}} ## Device plugin registration -The device plugins feature is gated by the `DevicePlugins` feature gate which -is disabled by default before 1.10. When the device plugins feature is enabled, -the kubelet exports a `Registration` gRPC service: +The kubelet exports a `Registration` gRPC service: ```gRPC service Registration { rpc Register(RegisterRequest) returns (Empty) {} } ``` + A device plugin can register itself with the kubelet through this gRPC service. During the registration, the device plugin needs to send: @@ -38,15 +40,15 @@ During the registration, the device plugin needs to send: * The Device Plugin API version against which it was built. * The `ResourceName` it wants to advertise. Here `ResourceName` needs to follow the [extended resource naming scheme](/docs/concepts/configuration/manage-compute-resources-container/#extended-resources) - as `vendor-domain/resource`. - For example, an Nvidia GPU is advertised as `nvidia.com/gpu`. + as `vendor-domain/resourcetype`. + (For example, an NVIDIA GPU is advertised as `nvidia.com/gpu`.) Following a successful registration, the device plugin sends the kubelet the list of devices it manages, and the kubelet is then in charge of advertising those resources to the API server as part of the kubelet node status update. -For example, after a device plugin registers `vendor-domain/foo` with the kubelet +For example, after a device plugin registers `hardware-vendor.example/foo` with the kubelet and reports two healthy devices on a node, the node status is updated -to advertise 2 `vendor-domain/foo`. +to advertise that the node has 2 “Foo” devices installed and available. Then, users can request devices in a [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) @@ -55,10 +57,11 @@ specification as they request other types of resources, with the following limit * Extended resources are only supported as integer resources and cannot be overcommitted. * Devices cannot be shared among Containers. -Suppose a Kubernetes cluster is running a device plugin that advertises resource `vendor-domain/resource` -on certain nodes, here is an example user pod requesting this resource: +Suppose a Kubernetes cluster is running a device plugin that advertises resource `hardware-vendor.example/foo` +on certain nodes. Here is an example of a pod requesting this resource to run a demo workload: ```yaml +--- apiVersion: v1 kind: Pod metadata: @@ -69,7 +72,14 @@ spec: image: k8s.gcr.io/pause:2.0 resources: limits: - vendor-domain/resource: 2 # requesting 2 vendor-domain/resource + hardware-vendor.example/foo: 2 +# +# This Pod needs 2 of the hardware-vendor.example/foo devices +# and can only schedule onto a Node that's able to satisfy +# that need. +# +# If the Node has more than 2 of those devices available, the +# remainder would be available for other Pods to use. ``` ## Device plugin implementation @@ -107,6 +117,8 @@ If the operations succeed, the device plugin returns an `AllocateResponse` that runtime configurations for accessing the allocated devices. The kubelet passes this information to the container runtime. +### Handling kubelet restarts + A device plugin is expected to detect kubelet restarts and re-register itself with the new kubelet instance. In the current implementation, a new kubelet instance deletes all the existing Unix sockets under `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion @@ -114,37 +126,44 @@ of its Unix socket and re-register itself upon such an event. ## Device plugin deployment -A device plugin can be deployed manually or as a DaemonSet. Being deployed as a DaemonSet has -the benefit that Kubernetes can restart the device plugin if it fails. -Otherwise, an extra mechanism is needed to recover from device plugin failures. +You can deploy a device plugin as a DaemonSet, as a package for your node's operating system, +or manually. + The canonical directory `/var/lib/kubelet/device-plugins` requires privileged access, so a device plugin must run in a privileged security context. -If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins` -must be mounted as a -[Volume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core) +If you're deploying a device plugin as a DaemonSet, `/var/lib/kubelet/device-plugins` +must be mounted as a {{< glossary_tooltip term_id="volume" >}} in the plugin's [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core). -Kubernetes device plugin support is in beta. As development continues, its API version can -change. We recommend that device plugin developers do the following: +If you choose the DaemonSet approach you can rely on Kubernetes to: place the device plugin's +Pod onto Nodes, to restart the daemon Pod after failure, and to help automate upgrades. + +## API compatibility + +Kubernetes device plugin support is in beta. The API may change before stabilization, +in incompatible ways. As a project, Kubernetes recommends that device plugin developers: * Watch for changes in future releases. * Support multiple versions of the device plugin API for backward/forward compatibility. If you enable the DevicePlugins feature and run device plugins on nodes that need to be upgraded to a Kubernetes release with a newer device plugin API version, upgrade your device plugins -to support both versions before upgrading these nodes to +to support both versions before upgrading these nodes. Taking that approach will ensure the continuous functioning of the device allocations during the upgrade. ## Monitoring Device Plugin Resources -In order to monitor resources provided by device plugins, monitoring agents need to be able to -discover the set of devices that are in-use on the node and obtain metadata to describe which -container the metric should be associated with. Prometheus metrics exposed by device monitoring -agents should follow the -[Kubernetes Instrumentation Guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/instrumentation.md), -which requires identifying containers using `pod`, `namespace`, and `container` prometheus labels. -The kubelet provides a gRPC service to enable discovery of in-use devices, and to provide metadata +{{< feature-state for_k8s_version="v1.13" state="alpha" >}} + +In order to monitor resources provided by device plugins, monitoring agents need to be able to +discover the set of devices that are in-use on the node and obtain metadata to describe which +container the metric should be associated with. [Prometheus](https://prometheus.io/) metrics +exposed by device monitoring agents should follow the +[Kubernetes Instrumentation Guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/instrumentation.md), +identifying containers using `pod`, `namespace`, and `container` prometheus labels. + +The kubelet provides a gRPC service to enable discovery of in-use devices, and to provide metadata for these devices: ```gRPC @@ -155,31 +174,36 @@ service PodResourcesLister { } ``` -The gRPC service is served over a unix socket at `/var/lib/kubelet/pod-resources/kubelet.sock`. -Monitoring agents for device plugin resources can be deployed as a daemon, or as a DaemonSet. -The canonical directory `/var/lib/kubelet/pod-resources` requires privileged access, so monitoring -agents must run in a privileged security context. If a device monitoring agent is running as a -DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a -[Volume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core) -in the plugin's +The gRPC service is served over a unix socket at `/var/lib/kubelet/pod-resources/kubelet.sock`. +Monitoring agents for device plugin resources can be deployed as a daemon, or as a DaemonSet. +The canonical directory `/var/lib/kubelet/pod-resources` requires privileged access, so monitoring +agents must run in a privileged security context. If a device monitoring agent is running as a +DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a +{{< glossary_tooltip term_id="volume" >}} in the plugin's [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core). Support for the "PodResources service" is in beta, and is enabled by default. -## Examples +## Device plugin examples {#examples} -For examples of device plugin implementations, see: +Here are some examples of device plugin implementations: -* The official [NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin) - * Requires [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker) which allows you to run GPU enabled docker containers. - * A detailed guide on how to [schedule NVIDIA GPUs](/docs/tasks/manage-gpus/scheduling-gpus) on k8s. -* The [NVIDIA GPU device plugin for COS base OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu) +* The [AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin) +* The [Intel device plugins](https://github.com/intel/intel-device-plugins-for-kubernetes) for Intel GPU, FPGA and QuickAssist devices +* The [KubeVirt device plugins](https://github.com/kubevirt/kubernetes-device-plugins) for hardware-assisted virtualization +* The [NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin) + * Requires [nvidia-docker](https://github.com/NVIDIA/nvidia-docker) 2.0, which allows you to run GPU-enabled Docker containers. +* The [NVIDIA GPU device plugin for Container-Optimized OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu) * The [RDMA device plugin](https://github.com/hustcat/k8s-rdma-device-plugin) * The [Solarflare device plugin](https://github.com/vikaschoudhary16/sfc-device-plugin) -* The [AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin) -* The [SRIOV Network device plugin](https://github.com/intel/sriov-network-device-plugin) -* The [Intel device plugins](https://github.com/intel/intel-device-plugins-for-kubernetes) for GPU, FPGA and QuickAssist devices +* The [SR-IOV Network device plugin](https://github.com/intel/sriov-network-device-plugin) * The [Xilinx FPGA device plugins](https://github.com/Xilinx/FPGA_as_a_Service/tree/master/k8s-fpga-device-plugin/trunk) for Xilinx FPGA devices {{% /capture %}} +{{% capture whatsnext %}} +* Learn about [scheduling GPU resources](/docs/tasks/manage-gpus/scheduling-gpus/) using device plugins +* Learn about [advertising extended resources](/docs/tasks/administer-cluster/extended-resource-node/) on a node +* Read about using [hardware acceleration for TLS ingress](https://kubernetes.io/blog/2019/04/24/hardware-accelerated-ssl-tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes + +{{% /capture %}} diff --git a/content/en/docs/reference/glossary/device-plugin.md b/content/en/docs/reference/glossary/device-plugin.md index be653a7580..d29b495953 100644 --- a/content/en/docs/reference/glossary/device-plugin.md +++ b/content/en/docs/reference/glossary/device-plugin.md @@ -4,7 +4,7 @@ id: device-plugin date: 2019-02-02 full_link: /docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/ short_description: > - Device Plugins are containers running in Kubernetes that provide access to a vendor specific resource. + Containers running in Kubernetes that provide access to a vendor specific resource. aka: tags: - fundamental @@ -14,4 +14,4 @@ tags: -[Device Plugin](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) are containers running in Kubernetes that provide access to a vendor specific resource. Device Plugins advertise these resources to kubelet and can be deployed manually or as a DeamonSet, rather than writing custom Kubernetes code. +[Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) are containers running in Kubernetes that provide access to a vendor-specific resource. Device Plugins advertise these resources to {{< glossary_tooltip term_id="kubelet" >}}. They can be deployed manually or as a {{< glossary_tooltip term_id="daemonset" >}}, rather than writing custom Kubernetes code. diff --git a/content/en/docs/tasks/administer-cluster/extended-resource-node.md b/content/en/docs/tasks/administer-cluster/extended-resource-node.md index 1cddabf8e8..49e491d251 100644 --- a/content/en/docs/tasks/administer-cluster/extended-resource-node.md +++ b/content/en/docs/tasks/administer-cluster/extended-resource-node.md @@ -10,8 +10,6 @@ This page shows how to specify extended resources for a Node. Extended resources allow cluster administrators to advertise node-level resources that would otherwise be unknown to Kubernetes. -{{< feature-state state="stable" >}} - {{% /capture %}} @@ -60,7 +58,7 @@ you call dongles. Start a proxy, so that you can easily send requests to the Kubernetes API server: -``` +```shell kubectl proxy ``` @@ -153,7 +151,7 @@ Then a Container could request any number of bytes of special storage, up to 800 Here is a PATCH request that removes the dongle advertisement from a Node. -```shell +``` PATCH /api/v1/nodes//status HTTP/1.1 Accept: application/json Content-Type: application/json-patch+json @@ -169,7 +167,7 @@ Host: k8s-master:8080 Start a proxy, so that you can easily send requests to the Kubernetes API server: -``` +```shell kubectl proxy ``` @@ -189,6 +187,8 @@ Verify that the dongle advertisement has been removed: kubectl describe node | grep dongle ``` +(you should not see any output) + {{% /capture %}} diff --git a/content/en/docs/tasks/configure-pod-container/extended-resource.md b/content/en/docs/tasks/configure-pod-container/extended-resource.md index e71f09ea76..36d957ca01 100644 --- a/content/en/docs/tasks/configure-pod-container/extended-resource.md +++ b/content/en/docs/tasks/configure-pod-container/extended-resource.md @@ -6,10 +6,10 @@ weight: 40 {{% capture overview %}} -This page shows how to assign extended resources to a Container. - {{< feature-state state="stable" >}} +This page shows how to assign extended resources to a Container. + {{% /capture %}} @@ -141,9 +141,3 @@ kubectl delete pod extended-resource-demo-2 * [Advertise Extended Resources for a Node](/docs/tasks/administer-cluster/extended-resource-node/) {{% /capture %}} - - - - - - diff --git a/content/en/docs/tasks/manage-gpus/scheduling-gpus.md b/content/en/docs/tasks/manage-gpus/scheduling-gpus.md index 79cf7e2832..6d1de803cf 100644 --- a/content/en/docs/tasks/manage-gpus/scheduling-gpus.md +++ b/content/en/docs/tasks/manage-gpus/scheduling-gpus.md @@ -7,10 +7,10 @@ title: Schedule GPUs {{% capture overview %}} -Kubernetes includes **experimental** support for managing AMD and NVIDIA GPUs spread -across nodes. The support for NVIDIA GPUs was added in v1.6 and has gone through -multiple backwards incompatible iterations. The support for AMD GPUs was added in -v1.9 via [device plugin](#deploying-amd-gpu-device-plugin). +{{< feature-state state="beta" for_k8s_version="1.10" >}} + +Kubernetes includes **experimental** support for managing AMD and NVIDIA GPUs +(graphical processing units) across several nodes. This page describes how users can consume GPUs across different Kubernetes versions and the current limitations. @@ -20,22 +20,20 @@ and the current limitations. {{% capture body %}} -## v1.8 onwards +## Using device plugins -**From 1.8 onwards, the recommended way to consume GPUs is to use [device -plugins](/docs/concepts/cluster-administration/device-plugins).** +Kubernetes implements {{< glossary_tooltip text="Device Plugins" term_id="device-plugin" >}} +to let Pods access specialized hardware features such as GPUs. -To enable GPU support through device plugins before 1.10, the `DevicePlugins` -feature gate has to be explicitly set to true across the system: -`--feature-gates="DevicePlugins=true"`. This is no longer required starting -from 1.10. +As an administrator, you have to install GPU drivers from the corresponding +hardware vendor on the nodes and run the corresponding device plugin from the +GPU vendor: -Then you have to install GPU drivers from the corresponding vendor on the nodes -and run the corresponding device plugin from the GPU vendor -([AMD](#deploying-amd-gpu-device-plugin), [NVIDIA](#deploying-nvidia-gpu-device-plugin)). +* [AMD](#deploying-amd-gpu-device-plugin) +* [NVIDIA](#deploying-nvidia-gpu-device-plugin) -When the above conditions are true, Kubernetes will expose `nvidia.com/gpu` or -`amd.com/gpu` as a schedulable resource. +When the above conditions are true, Kubernetes will expose `amd.com/gpu` or +`nvidia.com/gpu` as a schedulable resource. You can consume these GPUs from your containers by requesting `.com/gpu` just like you request `cpu` or `memory`. @@ -48,7 +46,7 @@ when using GPUs: * You can specify GPU in both `limits` and `requests` but these two values must be equal. * You cannot specify GPU `requests` without specifying `limits`. -- Containers (and pods) do not share GPUs. There's no overcommitting of GPUs. +- Containers (and Pods) do not share GPUs. There's no overcommitting of GPUs. - Each container can request one or more GPUs. It is not possible to request a fraction of a GPU. @@ -79,14 +77,12 @@ has the following requirements: To deploy the AMD device plugin once your cluster is running and the above requirements are satisfied: +```shell +kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.10/k8s-ds-amdgpu-dp.yaml ``` -# For Kubernetes v1.9 -kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/r1.9/k8s-ds-amdgpu-dp.yaml -# For Kubernetes v1.10 -kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/r1.10/k8s-ds-amdgpu-dp.yaml -``` -Report issues with this device plugin to [RadeonOpenCompute/k8s-device-plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin). +You can report issues with this third-party device plugin by logging an issue in +[RadeonOpenCompute/k8s-device-plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin). ### Deploying NVIDIA GPU device plugin @@ -99,22 +95,20 @@ has the following requirements: - Kubernetes nodes have to be pre-installed with NVIDIA drivers. - Kubernetes nodes have to be pre-installed with [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker) -- nvidia-container-runtime must be configured as the [default runtime](https://github.com/NVIDIA/k8s-device-plugin#preparing-your-gpu-nodes) - for docker instead of runc. -- NVIDIA drivers ~= 361.93 +- Kubelet must use Docker as its container runtime +- `nvidia-container-runtime` must be configured as the [default runtime](https://github.com/NVIDIA/k8s-device-plugin#preparing-your-gpu-nodes) + for Docker, instead of runc. +- The version of the NVIDIA drivers must match the constraint ~= 361.93 To deploy the NVIDIA device plugin once your cluster is running and the above requirements are satisfied: -``` -# For Kubernetes v1.8 -kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.8/nvidia-device-plugin.yml - -# For Kubernetes v1.9 -kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml +```shell +kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta/nvidia-device-plugin.yml ``` -Report issues with this device plugin to [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin). +You can report issues with this third-party device plugin by logging an issue in +[NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin). #### NVIDIA GPU device plugin used by GCE @@ -124,9 +118,9 @@ that is compatible with the Kubernetes Container Runtime Interface (CRI). It's t on [Container-Optimized OS](https://cloud.google.com/container-optimized-os/) and has experimental code for Ubuntu from 1.9 onwards. -On your 1.12 cluster, you can use the following commands to install the NVIDIA drivers and device plugin: +You can use the following commands to install the NVIDIA drivers and device plugin: -``` +```shell # Install NVIDIA drivers on Container-Optimized OS: kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/daemonset.yaml @@ -134,13 +128,13 @@ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/containe kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/nvidia-driver-installer/ubuntu/daemonset.yaml # Install the device plugin: -kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.12/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml +kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.14/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml ``` -Report issues with this device plugin and installation method to [GoogleCloudPlatform/container-engine-accelerators](https://github.com/GoogleCloudPlatform/container-engine-accelerators). +You can report issues with using or deploying this third-party device plugin by logging an issue in +[GoogleCloudPlatform/container-engine-accelerators](https://github.com/GoogleCloudPlatform/container-engine-accelerators). -Instructions for using NVIDIA GPUs on GKE are -[here](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus) +Google publishes its own [instructions](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus) for using NVIDIA GPUs on GKE . ## Clusters containing different types of GPUs @@ -156,7 +150,14 @@ kubectl label nodes accelerator=nvidia-tesla-k80 kubectl label nodes accelerator=nvidia-tesla-p100 ``` -For AMD GPUs, you can deploy [Node Labeller](https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller), which automatically labels your nodes with GPU properties. Currently supported properties: +## Automatic node labelling {#node-labeller} + +If you're using AMD GPU devices, you can deploy +[Node Labeller](https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller). +Node Labeller is a {{< glossary_tooltip text="controller" term_id="controller" >}} that automatically +labels your nodes with GPU device properties. + +At the moment, that controller can add labels for: * Device ID (-device-id) * VRAM Size (-vram) @@ -172,13 +173,11 @@ For AMD GPUs, you can deploy [Node Labeller](https://github.com/RadeonOpenComput * AI - Arctic Islands * RV - Raven -Example result: - -```console +```shell kubectl describe node cluster-node-23 ``` -The output is similar to: +``` Name: cluster-node-23 Roles: Labels: beta.amd.com/gpu.cu-count.64=1 @@ -191,9 +190,10 @@ The output is similar to: kubernetes.io/hostname=cluster-node-23 Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 - ...... + … +``` -Specify the GPU type in the pod spec: +With the Node Labeller in use, you can specify the GPU type in the Pod spec: ```yaml apiVersion: v1 @@ -213,5 +213,7 @@ spec: accelerator: nvidia-tesla-p100 # or nvidia-tesla-k80 etc. ``` -This will ensure that the pod will be scheduled to a node that has the GPU type +This will ensure that the Pod will be scheduled to a node that has the GPU type you specified. + +{{% /capture %}} From dd6f88dfe78e4fb61c1c7cc81bccee20bb96481a Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Wed, 7 Aug 2019 00:22:49 +0100 Subject: [PATCH 23/49] Reword & migrate static pod task page (#14831) - Shape this to be more of a Task page - Move static pod task to pod & container section --- .../tasks/administer-cluster/static-pod.md | 143 ----------- .../configure-pod-container/static-pod.md | 243 ++++++++++++++++++ static/_redirects | 1 + 3 files changed, 244 insertions(+), 143 deletions(-) delete mode 100644 content/en/docs/tasks/administer-cluster/static-pod.md create mode 100644 content/en/docs/tasks/configure-pod-container/static-pod.md diff --git a/content/en/docs/tasks/administer-cluster/static-pod.md b/content/en/docs/tasks/administer-cluster/static-pod.md deleted file mode 100644 index d75cd2f4a9..0000000000 --- a/content/en/docs/tasks/administer-cluster/static-pod.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -reviewers: -- jsafrane -title: Static Pods -content_template: templates/concept ---- - -{{% capture overview %}} - -**If you are running clustered Kubernetes and are using static pods to run a pod on every node, you should probably be using a [DaemonSet](/docs/concepts/workloads/controllers/daemonset/)!** - -*Static pods* are managed directly by kubelet daemon on a specific node, without the API server observing it. It does not have an associated replication controller, and kubelet daemon itself watches it and restarts it when it crashes. There is no health check. Static pods are always bound to one kubelet daemon and always run on the same node with it. - -Kubelet automatically tries to create a *mirror pod* on the Kubernetes API server for each static pod. -This means that the pods are visible on the API server but cannot be controlled from there. - -{{% /capture %}} - - -{{% capture body %}} - -## Static pod creation - -Static pod can be created in two ways: either by using configuration file(s) or by HTTP. - -### Configuration files - -The configuration files are just standard pod definitions in json or yaml format in a specific directory. Use `kubelet --pod-manifest-path=` to start kubelet daemon or add the `staticPodPath: ` field in the [KubeletConfiguration file](/docs/tasks/administer-cluster/kubelet-config-file), which periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there. -Note that kubelet will ignore files starting with dots when scanning the specified directory. - -For example, this is how to start a simple web server as a static pod: - -1. Choose a node where we want to run the static pod. In this example, it's `my-node1`. - - ``` - [joe@host ~] $ ssh my-node1 - ``` - -2. Choose a directory, say `/etc/kubelet.d` and place a web server pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`: - -```shell -[root@my-node1 ~] $ mkdir /etc/kubelet.d/ -[root@my-node1 ~] $ cat </etc/kubelet.d/static-web.yaml -apiVersion: v1 -kind: Pod -metadata: - name: static-web - labels: - role: myrole -spec: - containers: - - name: web - image: nginx - ports: - - name: web - containerPort: 80 - protocol: TCP -EOF -``` - -3. Configure your kubelet daemon on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument or add the `staticPodPath: ` field in the [KubeletConfiguration file](/docs/tasks/administer-cluster/kubelet-config-file). - On Fedora edit `/etc/kubernetes/kubelet` to include this line: - - ``` - KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" - ``` - - Instructions for other distributions or Kubernetes installations may vary. - -4. Restart kubelet. On Fedora, this is: - - ``` - [root@my-node1 ~] $ systemctl restart kubelet - ``` - -### Pods created via HTTP - -Kubelet periodically downloads a file specified by `--manifest-url=` argument and interprets it as a json/yaml file with a pod definition. It works the same as `--pod-manifest-path=`, i.e. it's reloaded every now and then and changes are applied to running static pods (see below). - -## Behavior of static pods - -When kubelet starts, it automatically starts all pods defined in directory specified in `--pod-manifest-path=` or `--manifest-url=` arguments or add the `staticPodPath: ` field in the [KubeletConfiguration file](/docs/tasks/administer-cluster/kubelet-config-file), i.e. our static-web. (It may take some time to pull nginx image, be patient…): - -```shell -[joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c -``` - -If we look at our Kubernetes API server (running on host `my-master`), we see that a new mirror-pod was created there too: - -```shell -[joe@host ~] $ ssh my-master -[joe@my-master ~] $ kubectl get pods -NAME READY STATUS RESTARTS AGE -static-web-my-node1 1/1 Running 0 2m -``` - -Labels from the static pod are propagated into the mirror-pod and can be used as usual for filtering. - -Notice we cannot delete the pod with the API server (e.g. via [`kubectl`](/docs/user-guide/kubectl/) command), kubelet simply won't remove it. - -{{< note >}} -Make sure the kubelet has permission to create the mirror pod in the API server. If not, the creation request is rejected by the API server. See -[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/). -{{< /note >}} - -```shell -[joe@my-master ~] $ kubectl delete pod static-web-my-node1 -pod "static-web-my-node1" deleted -[joe@my-master ~] $ kubectl get pods -NAME READY STATUS RESTARTS AGE -static-web-my-node1 1/1 Running 0 12s -``` - -Back to our `my-node1` host, we can try to stop the container manually and see, that kubelet automatically restarts it in a while: - -```none -[joe@host ~] $ ssh my-node1 -[joe@my-node1 ~] $ docker stop f6d05272b57e -[joe@my-node1 ~] $ sleep 20 -[joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED ... -5b920cbaf8b1 nginx:latest "nginx -g 'daemon of 2 seconds ago ... -``` - -## Dynamic addition and removal of static pods - -Running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes pods as files appear/disappear in this directory. - -```shell -[joe@my-node1 ~] $ mv /etc/kubelet.d/static-web.yaml /tmp -[joe@my-node1 ~] $ sleep 20 -[joe@my-node1 ~] $ docker ps -// no nginx container is running -[joe@my-node1 ~] $ mv /tmp/static-web.yaml /etc/kubelet.d/ -[joe@my-node1 ~] $ sleep 20 -[joe@my-node1 ~] $ docker ps -CONTAINER ID IMAGE COMMAND CREATED ... -e7a62e3427f1 nginx:latest "nginx -g 'daemon of 27 seconds ago -``` - -{{% /capture %}} diff --git a/content/en/docs/tasks/configure-pod-container/static-pod.md b/content/en/docs/tasks/configure-pod-container/static-pod.md new file mode 100644 index 0000000000..4225a7a586 --- /dev/null +++ b/content/en/docs/tasks/configure-pod-container/static-pod.md @@ -0,0 +1,243 @@ +--- +reviewers: +- jsafrane +title: Create static Pods +content_template: templates/task +--- + +{{% capture overview %}} + + +*Static Pods* are managed directly by the kubelet daemon on a specific node, +without the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} +observing them. +Unlike Pods that are managed by the control plane (eg, a +{{< glossary_tooltip text="Deployment" term_id="deployment" >}}; +instead, the kubelet watches each static Pod (and restarts it if it crashes). +There are no health checks for the containers in a static Pod. + +Static Pods are always bound to one {{< glossary_tooltip term_id="kubelet" >}} on a specific node. + +Kubelet automatically tries to create a {{< glossary_tooltip text="mirror Pod" term_id="mirror-pod" >}} +on the Kubernetes API server for each static Pod. +This means that the Pods running on a node are visible on the API server, +but cannot be controlled from there. + +{{< note >}} +If you are running clustered Kubernetes and are using static +Pods to run a Pod on every node, you should probably be using a +{{< glossary_tooltip text="DaemonSet" term_id="daemonset" >}} +instead. +{{< /note >}} + +{{% /capture %}} + +{{% capture prerequisites %}} + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + +This page assumes you're using {{< glossary_tooltip term_id="docker" >}} to run Pods, +and that your nodes are running the Fedora operating system. +Instructions for other distributions or Kubernetes installations may vary. + + +{{% /capture %}} + + +{{% capture steps %}} + +## Create a static pod {#static-pod-creation} + +You can configure a static Pod two different ways: either by using configuration file(s) or by HTTP. + +### Filesystem-hosted static Pod manifest {#configuration-files} + +The configuration files are just standard Pod definitions in JSON or YAML format in a specific directory. Use `kubelet --pod-manifest-path=` to start the kubelet or add the `staticPodPath: ` field in the [KubeletConfiguration file](/docs/tasks/administer-cluster/kubelet-config-file), which periodically scans the directory and creates/deletes static Pods as YAML/JSON files appear/disappear there. +Note that the kubelet will ignore files starting with dots when scanning the specified directory. + +For example, this is how to start a simple web server as a static Pod: + +1. Choose a node where you want to run the static Pod. In this example, it's `my-node1`. + + ```shell + ssh my-node1 + ``` + +2. Choose a directory, say `/etc/kubelet.d` and place a web server Pod definition there, e.g. `/etc/kubelet.d/static-web.yaml`: + +```shell +# Run this command on the node where kubelet is running +mkdir /etc/kubelet.d/ +cat </etc/kubelet.d/static-web.yaml +apiVersion: v1 +kind: Pod +metadata: + name: static-web + labels: + role: myrole +spec: + containers: + - name: web + image: nginx + ports: + - name: web + containerPort: 80 + protocol: TCP +EOF +``` + +3. Configure your kubelet on the node to use this directory by running it with `--pod-manifest-path=/etc/kubelet.d/` argument or add the `staticPodPath: ` field in the [KubeletConfiguration file](/docs/tasks/administer-cluster/kubelet-config-file). + On Fedora edit `/etc/kubernetes/kubelet` to include this line: + + ``` + KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubelet.d/" + ``` + +4. Restart the kubelet. On Fedora, you would run: + + ```shell + # Run this command on the node where the kubelet is running + systemctl restart kubelet + ``` + +### Web-hosted static pod manifest {#pods-created-via-http} + +Kubelet periodically downloads a file specified by `--manifest-url=` argument +and interprets it as a JSON/YAML file that contains Pod definitions. +Similar to how [filesystem-hosted manifests](#configuration-files) work, the kubelet +refetches the manifest on a schedule. If there are changes to the list of static +Pods, the kubelet applies them. + +If you want to use this approach, create a YAML file: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: static-web + labels: + role: myrole +spec: + containers: + - name: web + image: nginx + ports: + - name: web + containerPort: 80 + protocol: TCP +``` + +and store it on a web server so that you can pass the URL of that file to the kubelet. + +Configure the kubelet on your selected node to use this web manifest by running it with `--manifest-url=` + + On Fedora, edit `/etc/kubernetes/kubelet` to include this line: + + ``` + KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --manifest-url=` + ``` + +Now, restart the kubelet. On Fedora, you would run: + + ```shell + # Run this command on the node where the kubelet is running + systemctl restart kubelet + ``` + + + +## Observe static pod behavior {#behavior-of-static-pods} + +When the kubelet starts, it automatically starts all defined static Pods. As you have +defined a static Pod and restarted the kubelet, the new static Pod should +already be running. + +You can view running containers (including static Pods) by running (on the node): +```shell +# Run this command on the node where kubelet is running +docker ps +``` + +The output might be something like: + +``` +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f6d05272b57e nginx:latest "nginx" 8 minutes ago Up 8 minutes k8s_web.6f802af4_static-web-fk-node1_default_67e24ed9466ba55986d120c867395f3c_378e5f3c +``` + +You can see the mirror Pod on the API server: + +```shell +kubectl get pods +``` +``` +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 2m +``` + +{{< note >}} +Make sure the kubelet has permission to create the mirror Pod in the API server. If not, the creation request is rejected by the API server. See +[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/). +{{< /note >}} + + +{{< glossary_tooltip term_id="label" text="Labels" >}} from the static Pod are +propagated into the mirror Pod. You can use those labels as normal via +{{< glossary_tooltip term_id="selector" text="selectors" >}}, etc. + +If you try to use `kubectl` to delete the mirror Pod from the API server, +the kubelet _doesn't_ remove the static Pod: + +```shell +kubectl delete pod static-web-my-node1 +``` +``` +pod "static-web-my-node1" deleted +``` +You can see that the Pod is still running: +```shell +kubectl get pods +``` +``` +NAME READY STATUS RESTARTS AGE +static-web-my-node1 1/1 Running 0 12s +``` + +Back on your node where the kubelet is running, you can try to stop the Docker +container manually. +You'll see that, after a time, the kubelet will notice and will restart the Pod +automatically: + +```shell +# Run these commands on the node where the kubelet is running +docker stop f6d05272b57e # replace with the ID of your container +sleep 20 +docker ps +``` +``` +CONTAINER ID IMAGE COMMAND CREATED ... +5b920cbaf8b1 nginx:latest "nginx -g 'daemon of 2 seconds ago ... +``` + +## Dynamic addition and removal of static pods + +The running kubelet periodically scans the configured directory (`/etc/kubelet.d` in our example) for changes and adds/removes Pods as files appear/disappear in this directory. + +```shell +# This assumes you are using filesystem-hosted static Pod configuration +# Run these commands on the node where the kubelet is running +# +mv /etc/kubelet.d/static-web.yaml /tmp +sleep 20 +docker ps +# You see that no nginx container is running +mv /tmp/static-web.yaml /etc/kubelet.d/ +sleep 20 +docker ps +``` +``` +CONTAINER ID IMAGE COMMAND CREATED ... +e7a62e3427f1 nginx:latest "nginx -g 'daemon of 27 seconds ago +``` + +{{% /capture %}} diff --git a/static/_redirects b/static/_redirects index b07146cea0..718f9c64af 100644 --- a/static/_redirects +++ b/static/_redirects @@ -296,6 +296,7 @@ /docs/tasks/administer-cluster/romana-network-policy/ /docs/tasks/administer-cluster/network-policy-provider/romana-network-policy/ 301 /docs/tasks/administer-cluster/running-cloud-controller.md /docs/tasks/administer-cluster/running-cloud-controller/ 301 /docs/tasks/administer-cluster/share-configuration/ /docs/tasks/access-application-cluster/configure-access-multiple-clusters/ 301 +/docs/tasks/administer-cluster/static-pod/ /docs/tasks/configure-pod-container/static-pod/ 301 /docs/tasks/administer-cluster/upgrade-1-6/ /docs/tasks/administer-cluster/upgrade-downgrade/upgrade-1-6/ 301 /docs/tasks/administer-cluster/weave-network-policy/ /docs/tasks/administer-cluster/network-policy-provider/weave-network-policy/ 301 /docs/tasks/configure-pod-container/apply-resource-quota-limit/ /docs/tasks/administer-cluster/apply-resource-quota-limit/ 301 From 2928d6ac9ceba3ecfc8880bfe7ac8ed6a73bd4bb Mon Sep 17 00:00:00 2001 From: Tahmid Shakil Date: Wed, 7 Aug 2019 06:34:51 +0700 Subject: [PATCH 24/49] Fix troubleshooting instruction at troubleshooting-kubeadm.md (#15234) ## Issue Section: *Another workaround is to overwrite the existing `kubeconfig` for the "admin" user* Line: 173 `sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config` fails, since the directory `$HOME/.kube` doesn't exist. ## Fix `mkdir $HOME/.kube` added prior to `sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config`. --- .../tools/kubeadm/troubleshooting-kubeadm.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md index 5cb4a43570..d38bdd7e9b 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md @@ -170,6 +170,7 @@ Unable to connect to the server: x509: certificate signed by unknown authority ( ```sh mv $HOME/.kube $HOME/.kube.bak + mkdir $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` From 7bcc824779251861c974db63ee33904eb528fdbe Mon Sep 17 00:00:00 2001 From: Manish Bansal Date: Wed, 7 Aug 2019 05:06:50 +0530 Subject: [PATCH 25/49] Powershell note for kubectl patch command (#15308) * Improvement for kubectl patch command This command does not work in Powershell. A specific note would help the developers to take care of the same. * Fixed review comments Added different tabs for bash and PowerShell. * Removing note for PowerShell Hugo is not able to add note along with shortcode. Hence removing note. --- .../run-application/update-api-object-kubectl-patch.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/content/en/docs/tasks/run-application/update-api-object-kubectl-patch.md b/content/en/docs/tasks/run-application/update-api-object-kubectl-patch.md index d8b426a7f9..3c59c00e06 100644 --- a/content/en/docs/tasks/run-application/update-api-object-kubectl-patch.md +++ b/content/en/docs/tasks/run-application/update-api-object-kubectl-patch.md @@ -69,9 +69,14 @@ spec: Patch your Deployment: -```shell +{{< tabs name="kubectl_patch_example" >}} +{{{< tab name="Bash" codelang="bash" >}} kubectl patch deployment patch-demo --patch "$(cat patch-file-containers.yaml)" -``` +{{< /tab >}} +{{< tab name="PowerShell" codelang="posh" >}} +kubectl patch deployment patch-demo --patch $(cat patch-file-containers.yaml) +{{< /tab >}}} +{{< /tabs >}} View the patched Deployment: From 2fdc40944bc3d7d4bca7bb6ab85a2173d6233d9a Mon Sep 17 00:00:00 2001 From: Manish Bansal Date: Wed, 7 Aug 2019 05:08:50 +0530 Subject: [PATCH 26/49] Sentence correction for Info section on concepts/architecture/nodes (#15343) * Sentence correction * Fixed review comments --- content/en/docs/concepts/architecture/nodes.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/docs/concepts/architecture/nodes.md b/content/en/docs/concepts/architecture/nodes.md index d53ca69c91..58f9d014f1 100644 --- a/content/en/docs/concepts/architecture/nodes.md +++ b/content/en/docs/concepts/architecture/nodes.md @@ -111,9 +111,8 @@ on a Node. ### Info -General information about the node, such as kernel version, Kubernetes version -(kubelet and kube-proxy version), Docker version (if used), OS name. -The information is gathered by Kubelet from the node. +Describes general information about the node, such as kernel version, Kubernetes version (kubelet and kube-proxy version), Docker version (if used), and OS name. +This information is gathered by Kubelet from the node. ## Management From 306a6201b88baed2c673f650f6086f3fc5d1f8ce Mon Sep 17 00:00:00 2001 From: lyveng <1996412+lyveng@users.noreply.github.com> Date: Wed, 7 Aug 2019 05:42:54 +0530 Subject: [PATCH 27/49] Fixed broken link to logging conventions in kubectl cheatsheet (#15699) --- content/de/docs/reference/kubectl/cheatsheet.md | 2 +- content/en/docs/reference/kubectl/cheatsheet.md | 2 +- content/fr/docs/reference/kubectl/cheatsheet.md | 2 +- content/ko/docs/reference/kubectl/cheatsheet.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/de/docs/reference/kubectl/cheatsheet.md b/content/de/docs/reference/kubectl/cheatsheet.md index d30084c90f..bedd682cbd 100644 --- a/content/de/docs/reference/kubectl/cheatsheet.md +++ b/content/de/docs/reference/kubectl/cheatsheet.md @@ -322,7 +322,7 @@ Ausgabeformat | Beschreibung ### Kubectl Ausgabe Ausführlichkeit und Debugging -Die Ausführlichkeit von Kubectl wird mit den Flags `-v` oder `--v ` gesteuert, gefolgt von einer Ganzzahl, die die Protokollebene darstellt. Allgemeine Protokollierungskonventionen für Kubernetes und die zugehörigen Protokollebenen werden [hier](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md) beschrieben. +Die Ausführlichkeit von Kubectl wird mit den Flags `-v` oder `--v ` gesteuert, gefolgt von einer Ganzzahl, die die Protokollebene darstellt. Allgemeine Protokollierungskonventionen für Kubernetes und die zugehörigen Protokollebenen werden [hier](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) beschrieben. Ausführlichkeit | Beschreibung --------------| ----------- diff --git a/content/en/docs/reference/kubectl/cheatsheet.md b/content/en/docs/reference/kubectl/cheatsheet.md index 10e5b140ef..ea846c4254 100644 --- a/content/en/docs/reference/kubectl/cheatsheet.md +++ b/content/en/docs/reference/kubectl/cheatsheet.md @@ -348,7 +348,7 @@ Output format | Description ### Kubectl output verbosity and debugging -Kubectl verbosity is controlled with the `-v` or `--v` flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described [here](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md). +Kubectl verbosity is controlled with the `-v` or `--v` flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described [here](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md). Verbosity | Description --------------| ----------- diff --git a/content/fr/docs/reference/kubectl/cheatsheet.md b/content/fr/docs/reference/kubectl/cheatsheet.md index a1f8939278..3b830901f8 100644 --- a/content/fr/docs/reference/kubectl/cheatsheet.md +++ b/content/fr/docs/reference/kubectl/cheatsheet.md @@ -315,7 +315,7 @@ Pour afficher les détails sur votre terminal dans un format spécifique, vous p | `-o=yaml` | Affiche un objet de l'API formaté en YAML | ### Verbosité de l'affichage de Kubectl et débogage -La verbosité de Kubectl est contrôlée par une des options `-v` ou `--v` suivie d'un entier représentant le niveau de log. Les conventions générales de logging de Kubernetes et les niveaux de log associés sont décrits [ici](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md). +La verbosité de Kubectl est contrôlée par une des options `-v` ou `--v` suivie d'un entier représentant le niveau de log. Les conventions générales de logging de Kubernetes et les niveaux de log associés sont décrits [ici](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md). | Verbosité | Description | |-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| diff --git a/content/ko/docs/reference/kubectl/cheatsheet.md b/content/ko/docs/reference/kubectl/cheatsheet.md index cfc244dc1f..3dd586cbde 100644 --- a/content/ko/docs/reference/kubectl/cheatsheet.md +++ b/content/ko/docs/reference/kubectl/cheatsheet.md @@ -340,7 +340,7 @@ kubectl api-resources --api-group=extensions # "extensions" API 그룹의 모든 ### Kubectl 출력 로그 상세 레벨(verbosity)과 디버깅 -Kubectl 로그 상세 레벨(verbosity)은 `-v` 또는`--v` 플래그와 로그 레벨을 나타내는 정수로 제어된다. 일반적인 쿠버네티스 로깅 규칙과 관련 로그 레벨이 [여기](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md)에 설명되어 있다. +Kubectl 로그 상세 레벨(verbosity)은 `-v` 또는`--v` 플래그와 로그 레벨을 나타내는 정수로 제어된다. 일반적인 쿠버네티스 로깅 규칙과 관련 로그 레벨이 [여기](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md)에 설명되어 있다. 로그 레벨 | 세부 사항 --------------| ----------- From 088bc912d36a8c8e3530be2c7eda6901e7e43ae0 Mon Sep 17 00:00:00 2001 From: Victor Ude Date: Tue, 6 Aug 2019 19:18:53 -0500 Subject: [PATCH 28/49] Explain how to cleanup the created pod (#15698) * Explain how to cleanup the created pod I'm still learning. Feel free to provide a better example. * Update configure-redis-using-configmap.md --- .../configuration/configure-redis-using-configmap.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/en/docs/tutorials/configuration/configure-redis-using-configmap.md b/content/en/docs/tutorials/configuration/configure-redis-using-configmap.md index 3603f2e6d7..8e2c9a924c 100644 --- a/content/en/docs/tutorials/configuration/configure-redis-using-configmap.md +++ b/content/en/docs/tutorials/configuration/configure-redis-using-configmap.md @@ -99,6 +99,11 @@ kubectl exec -it redis redis-cli 2) "allkeys-lru" ``` +Delete the created pod: +```shell +kubectl delete pod redis +``` + {{% /capture %}} {{% capture whatsnext %}} From 64bfa2a77039aa9db1cc0781dd68fe5d1c8800c3 Mon Sep 17 00:00:00 2001 From: houjun Date: Wed, 7 Aug 2019 10:50:51 +0800 Subject: [PATCH 29/49] Fix release stage for two features (#15688) --- content/en/docs/setup/release/notes.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/docs/setup/release/notes.md b/content/en/docs/setup/release/notes.md index 5bd3b00edc..59896b6778 100644 --- a/content/en/docs/setup/release/notes.md +++ b/content/en/docs/setup/release/notes.md @@ -279,14 +279,12 @@ Configure each cluster with a different cluster name using `kube-controller-mana ### Stable -- You can now create a non-preempting Pod priority. If set on a class, the pod will continue to be prioritized above queued pods of a lesser class, but will not preempt running pods. ([#74614](https://github.com/kubernetes/kubernetes/pull/74614), [@denkensk](https://github.com/denkensk)) - - Third party device monitoring is now enabled by default (KubeletPodResources). ([#77274](https://github.com/kubernetes/kubernetes/pull/77274), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) -- The kube-apiserver’s `watch` can now be enabled for events using the `--watch-cache-sizes` flag. ([#74321](https://github.com/kubernetes/kubernetes/pull/74321), [@yastij](https://github.com/yastij)) ### Beta - Admission webhooks can now register for a single version of a resource (for example, `apps/v1 deployments`) and be called when any other version of that resource is modified (for example `extensions/v1beta1 deployments`). This allows new versions of a resource to be handled by admission webhooks without needing to update every webhook to understand the new version. See the API documentation for the `matchPolicy: Equivalent` option in MutatingWebhookConfiguration and ValidatingWebhookConfiguration types. ([#78135](https://github.com/kubernetes/kubernetes/pull/78135), [@liggitt](https://github.com/liggitt)) +- Third party device monitoring is now enabled by default (KubeletPodResources). ([#77274](https://github.com/kubernetes/kubernetes/pull/77274), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) - The CustomResourcePublishOpenAPI feature is now beta and enabled by default. CustomResourceDefinitions with [structural schemas](https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190425-structural-openapi.md) now publish schemas in the OpenAPI document served at `/openapi/v2`. CustomResourceDefinitions with non-structural schemas have a `NonStructuralSchema` condition added with details about what needs to be corrected in the validation schema. ([#77825](https://github.com/kubernetes/kubernetes/pull/77825), [@roycaihw](https://github.com/roycaihw)) - Online volume expansion (ExpandInUsePersistentVolumes) is now a beta feature. As such, it is enabled by default. ([#77755](https://github.com/kubernetes/kubernetes/pull/77755), [@gnufied](https://github.com/gnufied)) - The `SupportNodePidsLimit` feature is now beta, and enabled by default. It is no longer necessary to set the feature gate `SupportNodePidsLimit=true`. ([#76221](https://github.com/kubernetes/kubernetes/pull/76221), [@RobertKrawitz](https://github.com/RobertKrawitz)) @@ -301,6 +299,7 @@ Configure each cluster with a different cluster name using `kube-controller-mana ### Alpha +- You can now create a non-preempting Pod priority. If set on a class, the pod will continue to be prioritized above queued pods of a lesser class, but will not preempt running pods. ([#74614](https://github.com/kubernetes/kubernetes/pull/74614), [@denkensk](https://github.com/denkensk)) - kubelet now allows the use of XFS quotas (on XFS and suitably configured ext4fs filesystems) to monitor storage consumption for ephemeral storage. This method of monitoring consumption, which is currently available only for `emptyDir` volumes, is faster and more accurate than the old method of walking the filesystem tree. Note that it does not enforce limits, it only monitors consumption. To utilize this functionality, set the feature gate `LocalStorageCapacityIsolationFSQuotaMonitoring=true`. For ext4fs filesystems, create the filesystem with `mkfs.ext4 -O project ` and run `tune2fs -Q prjquota `block device`; XFS filesystems need no additional preparation. The filesystem must be mounted with option `project` in `/etc/fstab`. If the primary partition is the root filesystem, add `rootflags=pquota` to the GRUB config file. ([#66928](https://github.com/kubernetes/kubernetes/pull/66928), [@RobertKrawitz](https://github.com/RobertKrawitz)) - Finalizer Protection for Service LoadBalancers (ServiceLoadBalancerFinalizer) has been added as an Alpha feature, which is disabled by default. This feature ensures the Service resource is not fully deleted until the correlating load balancer resources are deleted. ([#78262](https://github.com/kubernetes/kubernetes/pull/78262), [@MrHohn](https://github.com/MrHohn)) - Inline CSI ephemeral volumes can now be controlled with PodSecurityPolicy when the CSIInlineVolume alpha feature is enabled. ([#76915](https://github.com/kubernetes/kubernetes/pull/76915), [@vladimirvivien](https://github.com/vladimirvivien)) From 7e100a2c958b9d005fbeb84fb049f673a5532e86 Mon Sep 17 00:00:00 2001 From: RA489 Date: Wed, 7 Aug 2019 14:38:50 +0530 Subject: [PATCH 30/49] Fix broken link to kubectl cheatsheet (#15713) From 5702f54c8d808c83ba1682efa788257865d456c4 Mon Sep 17 00:00:00 2001 From: June Yi Date: Thu, 8 Aug 2019 06:44:43 +0900 Subject: [PATCH 31/49] Add gochist to sig-docs-en-owners (#15692) --- OWNERS_ALIASES | 1 + 1 file changed, 1 insertion(+) diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES index 51732cb349..3e8f3b1532 100644 --- a/OWNERS_ALIASES +++ b/OWNERS_ALIASES @@ -41,6 +41,7 @@ aliases: - bradtopol - chenopis - cody-clark + - gochist - jaredbhatti - jimangel - kbarnard10 From 8eba965dff767145c260dc259d62666a05dc8ac9 Mon Sep 17 00:00:00 2001 From: aimeeu Date: Wed, 7 Aug 2019 16:46:37 -0500 Subject: [PATCH 32/49] Update Participating in SIG Docs and Submit a PR (#15697) Add hyperlink to 'communicate' in Participating in SIG Docs section. Hyperlink points to Participating in SIG Docs discussions section further down the page. Add info to Submit a pull request bullet 5 about putting fixes # or closes # to the PR description so GitHub will automatically close the issue --- content/en/docs/contribute/start.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/content/en/docs/contribute/start.md b/content/en/docs/contribute/start.md index ace1dae1eb..67fbe8d38a 100644 --- a/content/en/docs/contribute/start.md +++ b/content/en/docs/contribute/start.md @@ -40,7 +40,7 @@ Not all tasks can be done in the GitHub UI, but these are discussed in the The Kubernetes documentation is maintained by a {{< glossary_tooltip text="Special Interest Group" term_id="sig" >}} (SIG) -called SIG Docs. We communicate using a Slack channel, a mailing list, and +called SIG Docs. We [communicate](#participate-in-sig-docs-discussions) using a Slack channel, a mailing list, and weekly video meetings. New participants are welcome. For more information, see [Participating in SIG Docs](/docs/contribute/participating/). @@ -247,8 +247,10 @@ pull request if it detects that you pushed a new branch to your fork. is the same as the commit summary, but you can change it if needed. The body is populated by your extended commit message (if present) and some template text. Read the template text and fill out the details it asks for, - then delete the extra template text. Leave the - **Allow edits from maintainers** checkbox selected. Click + then delete the extra template text. If you add to the description `fixes #<000000>` + or `closes #<000000>`, where `#<000000>` is the number of an associated issue, + GitHub will automatically close the issue when the PR merges. + Leave the **Allow edits from maintainers** checkbox selected. Click **Create pull request**. Congratulations! Your pull request is available in From c2b0321dbfee7b9edd3a2ecd20ab928939fb9dda Mon Sep 17 00:00:00 2001 From: codyc <37271623+thecrudge@users.noreply.github.com> Date: Wed, 7 Aug 2019 16:49:19 -0500 Subject: [PATCH 33/49] added limits to walkthrough (#15707) --- .../run-application/horizontal-pod-autoscale-walkthrough.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md b/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md index 444a180b17..02a7ef6e7d 100644 --- a/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md +++ b/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md @@ -65,7 +65,7 @@ It defines an index.php page which performs some CPU intensive computations: First, we will start a deployment running the image and expose it as a service: ```shell -kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80 +kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --limits=cpu=500m --expose --port=80 ``` ``` service/php-apache created From 0fc8aa746a9fe63202a306285c43302db6ffd2bc Mon Sep 17 00:00:00 2001 From: codyc <37271623+thecrudge@users.noreply.github.com> Date: Wed, 7 Aug 2019 16:51:19 -0500 Subject: [PATCH 34/49] CentOS Repo for 1.15 (#15708) Corrects centos repo for kubernetes 1.15 --- .../en/docs/setup/production-environment/container-runtimes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index e6362d9264..b63330438d 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -189,7 +189,7 @@ apt-get install cri-o-1.13 {{< tab name="CentOS/RHEL 7.4+" codelang="bash" >}} # Install prerequisites -yum-config-manager --add-repo=https://cbs.centos.org/repos/paas7-crio-311-candidate/x86_64/os/ +yum-config-manager --add-repo=https://cbs.centos.org/repos/paas7-crio-115-release/x86_64/os/ # Install CRI-O yum install --nogpgcheck cri-o From d3c216fedc4ad6a186d38e057f6fcbba8dfd1553 Mon Sep 17 00:00:00 2001 From: Johannes Liebermann Date: Wed, 7 Aug 2019 23:53:20 +0200 Subject: [PATCH 35/49] Fix typo in authentication.md (#15721) Change "certificates authorities" to "certificate authorities". --- content/en/docs/reference/access-authn-authz/authentication.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/access-authn-authz/authentication.md b/content/en/docs/reference/access-authn-authz/authentication.md index 54330b0a79..62cc906e23 100644 --- a/content/en/docs/reference/access-authn-authz/authentication.md +++ b/content/en/docs/reference/access-authn-authz/authentication.md @@ -71,7 +71,7 @@ can be accomplished using an [authenticating proxy](#authenticating-proxy) or th ### X509 Client Certs Client certificate authentication is enabled by passing the `--client-ca-file=SOMEFILE` -option to API server. The referenced file must contain one or more certificates authorities +option to API server. The referenced file must contain one or more certificate authorities to use to validate client certificates presented to the API server. If a client certificate is presented and verified, the common name of the subject is used as the user name for the request. As of Kubernetes 1.4, client certificates can also indicate a user's group memberships From c02cf959b7d3a455f4bf85a71fe0b1561e18ec20 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andreas=20Lindh=C3=A9?= Date: Thu, 8 Aug 2019 00:33:18 +0200 Subject: [PATCH 36/49] More consistent naming of services in Ingress docs (#15718) It switched from a naming scheme of service1/service2 to s1/s2. That can be confusing for new people, and I propose we stick to only one naming scheme. --- .../docs/concepts/services-networking/ingress.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/content/en/docs/concepts/services-networking/ingress.md b/content/en/docs/concepts/services-networking/ingress.md index b049ce4d37..c81be65379 100644 --- a/content/en/docs/concepts/services-networking/ingress.md +++ b/content/en/docs/concepts/services-networking/ingress.md @@ -215,7 +215,7 @@ Events: ``` The Ingress controller provisions an implementation-specific load balancer -that satisfies the Ingress, as long as the Services (`s1`, `s2`) exist. +that satisfies the Ingress, as long as the Services (`service1`, `service2`) exist. When it has done so, you can see the address of the load balancer at the Address field. @@ -230,9 +230,9 @@ you are using, you may need to create a default-http-backend Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address. ```none -foo.bar.com --| |-> foo.bar.com s1:80 +foo.bar.com --| |-> foo.bar.com service1:80 | 178.91.123.132 | -bar.foo.com --| |-> bar.foo.com s2:80 +bar.foo.com --| |-> bar.foo.com service2:80 ``` The following Ingress tells the backing load balancer to route requests based on @@ -384,7 +384,7 @@ Rules: Host Path Backends ---- ---- -------- foo.bar.com - /foo s1:80 (10.8.0.90:80) + /foo service1:80 (10.8.0.90:80) Annotations: nginx.ingress.kubernetes.io/rewrite-target: / Events: @@ -407,14 +407,14 @@ spec: http: paths: - backend: - serviceName: s1 + serviceName: service1 servicePort: 80 path: /foo - host: bar.baz.com http: paths: - backend: - serviceName: s2 + serviceName: service2 servicePort: 80 path: /foo .. @@ -438,9 +438,9 @@ Rules: Host Path Backends ---- ---- -------- foo.bar.com - /foo s1:80 (10.8.0.90:80) + /foo service1:80 (10.8.0.90:80) bar.baz.com - /foo s2:80 (10.8.0.91:80) + /foo service2:80 (10.8.0.91:80) Annotations: nginx.ingress.kubernetes.io/rewrite-target: / Events: From 05690331ce3d1fdfda9151c34a6ce101471631c3 Mon Sep 17 00:00:00 2001 From: Giri Kuncoro Date: Thu, 8 Aug 2019 08:53:18 +0700 Subject: [PATCH 37/49] Add dynamic provisioning Indonesian translation (#15583) * Add dynamic provisioning Indonesian translation * Rephrasing few sentences in translation --- .../concepts/storage/dynamic-provisioning.md | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 content/id/docs/concepts/storage/dynamic-provisioning.md diff --git a/content/id/docs/concepts/storage/dynamic-provisioning.md b/content/id/docs/concepts/storage/dynamic-provisioning.md new file mode 100644 index 0000000000..8f636373af --- /dev/null +++ b/content/id/docs/concepts/storage/dynamic-provisioning.md @@ -0,0 +1,128 @@ +--- +title: Penyediaan Volume Dinamis +content_template: templates/concept +weight: 40 +--- + +{{% capture overview %}} + +Penyediaan volume dinamis memungkinkan volume penyimpanan untuk dibuat sesuai permintaan (_on-demand_). +Tanpa adanya penyediaan dinamis (_dynamic provisioning_), untuk membuat volume penyimpanan baru, admin kluster secara manual harus +memanggil penyedia layanan cloud atau layanan penyimpanan, dan kemudian membuat [objek PersistentVolume](/docs/concepts/storage/persistent-volumes/) +sebagai representasi di Kubernetes. Fitur penyediaan dinamis menghilangkan kebutuhan admin kluster untuk menyediakan +penyimpanan sebelumnya (_pre-provision_). Dengan demikian, penyimpanan akan tersedia secara otomatis +ketika diminta oleh pengguna. + +{{% /capture %}} + + +{{% capture body %}} + +## Latar Belakang + +Penyediaan volume dinamis diimplementasi berdasarkan objek API StorageClass dari +grup API `storage.k8s.io`. Seorang admin kluster dapat mendefinisikan berbagai macam +objek StorageClass sesuai kebutuhan, masing-masing menentukan *plugin volume* (disebut +juga *provisioner*) yang menyediakan sebuah volume beserta kumpulan parameter untuk +diteruskan oleh _provisioner_ ketika proses penyediaan. + +Seorang kluster admin dapat mendefinisikan dan mengekspos berbagai templat penyimpanan +(dari sistem penyimpanan yang sama maupun berbeda) di dalam kluster, masing-masing dengan +kumpulan parameter tertentu. Desain ini memastikan bahwa pengguna tidak perlu khawatir betapa +rumitnya mekanisme penyediaan penyimpanan, tapi tetap memiliki kemampuan untuk +memilih berbagai macam pilihan penyimpanan. + +Info lebih lanjut mengenai _storage class_ dapat dilihat [di sini](/docs/concepts/storage/storage-classes/). + +## Mengaktifkan Penyediaan Dinamis (_Dynamic Provisioning_) + +Untuk mengaktifkan penyediaan dinamis, seorang admin kluster perlu untuk +terlebih dahulu membuat (_pre-create_) satu atau beberapa objek StorageClass +untuk pengguna. +Objek StorageClass mendefinisikan _provisioner_ mana yang seharusnya digunakan +dan parameter apa yang seharusnya diberikan pada _provisioner_ tersebut saat +penyediaan dinamis dipanggil. +Manifestasi berikut ini membuat sebuah StorageClass "slow" yang +menyediakan _persistent_ disk standar. + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: slow +provisioner: kubernetes.io/gce-pd +parameters: + type: pd-standard +``` + +Manifestasi berikut ini membuat sebuah StorageClass "fast" yang menyediakan +SSD _persistent_ disk. + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: fast +provisioner: kubernetes.io/gce-pd +parameters: + type: pd-ssd +``` + +## Menggunakan Penyediaan Dinamis + +Pengguna dapat melakukan permintaan untuk penyediaan penyimpanan dinamis dengan +memasukkan StorageClass di dalam PersistentVolumeClaim. Sebelum Kubernetes v1.6, +ini dapat dilakukan melalui anotasi `volume.beta.kubernetes.io/storage-class`. +Hanya saja, anotasi ini sudah usang sejak v1.6. Pengguna sekarang dapat dan seharusnya +menggunakan _field_ `storageClassName` dari objek PersistentVolumeClaim. Nilai +dari _field_ ini haruslah sesuai dengan nama StorageClass yang dikonfigurasi oleh +admin (lihat bagian [di bawah](#enabling-dynamic-provisioning)). + +Untuk memilih StorageClass "fast", sebagai contoh, pengguna dapat membuat +PersistentVolumeClaim seperti ini: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: claim1 +spec: + accessModes: + - ReadWriteOnce + storageClassName: fast + resources: + requests: + storage: 30Gi +``` + +Klaim ini menghasilkan _persistent_ disk SSD yang disediakan secara otomatis. +Ketika klaim dihilangkan, volume akan musnah. + +## Perilaku _Default_ + +Penyediaan dinamis dapat diaktifkan pada setiap kluster supaya semua klaim +dapat disediakan secara dinamis jika tidak ada StorageClass yang dispesifikasikan. +Seorang kluster admin dapat mengaktifkan perilaku ini dengan cara: + +- Menandai satu objek StorageClass sebagai _default_; +- Memastikan bahwa [_admission controller_ `DefaultStorageClass`](/docs/reference/access-authn-authz/admission-controllers/#defaultstorageclass) + telah aktif pada API server. + +Seorang admin dapat menandai StorageClass yang spesifik sebagai _default_ dengan menambahkan +anotasi `storageclass.kubernetes.io/is-default-class`. +Ketika StorageClass default tersebut ada pada kluster dan pengguna membuat PersistentVolumeClaim +tanpa menspesifikasikan `storageClassName`, _admission controller_ `DefaultStorageClass` secara +otomatis menambahkan _field_ `storageClassName` dengan StorageClass _default_. + +Perhatikan bahwa hanya bisa ada satu _default_ StorageClass pada sebuah kluster, +atau PersistentVolumeClaim tanpa menspesifikasikan `storageClassName` secara eksplisit +tidak bisa terbuat. + +## Kesadaran (_Awareness_) Topologi + +Pada kluster [Multi-Zona](/docs/setup/multiple-zones), Pod dapat tersebar di banyak Zona +pada sebuah Region. Penyimpanan dengan *backend* Zona-Tunggal seharusnya disediakan pada +Zona-Zona dimana Pod dijalankan. Hal ini dapat dicapai dengan mengatur +[Mode Volume Binding](/docs/concepts/storage/storage-classes/#volume-binding-mode). + +{{% /capture %}} From 4518c983d224f9e63f86c6d347ca9abd52080af1 Mon Sep 17 00:00:00 2001 From: June Yi Date: Thu, 8 Aug 2019 23:33:19 +0900 Subject: [PATCH 38/49] Third Korean l10n work for release-1.15 (#15744) * Add Link /docs/reference/ in Korean (#15610) * Translate tasks/debug-application-cluster/resource-usage-monitoring in Korean (#15593) * Translate tasks/inject-data-application/define-environment-variable-container in Korean (#15606) * ko: Update outdated files in dev-1.15-ko.3 (#15605) * Translate home/supported-doc-version in Korean (#15621) * ko: Update some files in reference/glossary (#15631) * Translate reference/glossary/cloud-provider.md in Korean (#15685) * Translate docs/reference/using-api/api-overview in Korean (#15644) * Translate tasks/debug-application-cluster/resource-metrics-pipeline in Korean (#15665) Co-Authored-By: June Yi Co-Authored-By: Seokho Co-Authored-By: Tim Bannister Co-authored-by: Yoon Co-authored-by: JiMyung Lee Co-authored-by: lapee79 Co-authored-by: Sunghoon Kang Co-authored-by: Yuk, Yongsu Co-authored-by: Lawrence Kay --- content/ko/docs/concepts/_index.md | 2 +- .../ko/docs/concepts/overview/components.md | 10 +- .../controllers/replicationcontroller.md | 4 +- .../workloads/pods/init-containers.md | 252 ++++++++------- .../docs/concepts/workloads/pods/podpreset.md | 3 +- content/ko/docs/contribute/participating.md | 28 +- .../ko/docs/home/supported-doc-versions.md | 28 ++ content/ko/docs/reference/_index.md | 4 +- .../docs/reference/glossary/cloud-provider.md | 17 + content/ko/docs/reference/glossary/cluster.md | 12 +- .../reference/glossary/container-runtime.md | 2 +- .../ko/docs/reference/glossary/daemonset.md | 12 +- .../ko/docs/reference/glossary/deployment.md | 10 +- .../docs/reference/glossary/init-container.md | 10 +- .../ko/docs/reference/glossary/limitrange.md | 8 +- .../ko/docs/reference/glossary/static-pod.md | 4 +- .../ko/docs/reference/kubectl/cheatsheet.md | 2 +- content/ko/docs/reference/using-api/_index.md | 5 + .../docs/reference/using-api/api-overview.md | 111 +++++++ content/ko/docs/setup/_index.md | 3 + .../tasks/debug-application-cluster/_index.md | 5 + .../resource-metrics-pipeline.md | 53 +++ .../resource-usage-monitoring.md | 116 +++++++ .../tasks/inject-data-application/_index.md | 5 + .../define-environment-variable-container.md | 120 +++++++ .../tutorials/kubernetes-basics/_index.html | 2 + .../basic-stateful-set.md | 306 +++++++++--------- content/ko/examples/pods/inject/envars.yaml | 15 + 28 files changed, 819 insertions(+), 330 deletions(-) create mode 100644 content/ko/docs/home/supported-doc-versions.md create mode 100644 content/ko/docs/reference/glossary/cloud-provider.md create mode 100644 content/ko/docs/reference/using-api/_index.md create mode 100644 content/ko/docs/reference/using-api/api-overview.md create mode 100755 content/ko/docs/tasks/debug-application-cluster/_index.md create mode 100644 content/ko/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md create mode 100644 content/ko/docs/tasks/debug-application-cluster/resource-usage-monitoring.md create mode 100644 content/ko/docs/tasks/inject-data-application/_index.md create mode 100644 content/ko/docs/tasks/inject-data-application/define-environment-variable-container.md create mode 100644 content/ko/examples/pods/inject/envars.yaml diff --git a/content/ko/docs/concepts/_index.md b/content/ko/docs/concepts/_index.md index c2720da086..d78e60b5b8 100644 --- a/content/ko/docs/concepts/_index.md +++ b/content/ko/docs/concepts/_index.md @@ -7,7 +7,7 @@ weight: 40 {{% capture overview %}} -개념 섹션을 통해 쿠버네티스 시스템을 구성하는 요소와 클러스터를 표현하는데 사용되는 추상 개념에 대해 배우고 쿠버네티스가 작동하는 방식에 대해 보다 깊이 이해할 수 있다. +개념 섹션을 통해 쿠버네티스 시스템을 구성하는 요소와 {{< glossary_tooltip text="클러스터" term_id="cluster" length="all" >}}를 표현하는데 사용되는 추상 개념에 대해 배우고 쿠버네티스가 작동하는 방식에 대해 보다 깊이 이해할 수 있다. {{% /capture %}} diff --git a/content/ko/docs/concepts/overview/components.md b/content/ko/docs/concepts/overview/components.md index 40b6fe95b9..575d5e2e5c 100644 --- a/content/ko/docs/concepts/overview/components.md +++ b/content/ko/docs/concepts/overview/components.md @@ -80,11 +80,13 @@ cloud-controller-manager는 클라우드 밴더 코드와 쿠버네티스 코드 ## 애드온 -애드온은 클러스터 기능을 이행하는 파드와 서비스다. -이 파드는 디플로이먼트, 레플리케이션 컨트롤러, 기타 등등에 의해 관리될 수도 있다. -네임스페이스를 갖는 애드온 오브젝트는 `kube-system` 네임스페이스 내에서 생성되어 진다. +애드온은 쿠버네티스 리소스({{< glossary_tooltip text="데몬셋" term_id="daemonset" >}}, +{{< glossary_tooltip text="디플로이먼트" term_id="deployment" >}} 등)를 +이용하여 클러스터 기능을 구현한다. 이들은 클러스터 단위의 기능을 제공하기 때문에 +애드온에 대한 네임스페이스 리소스는 `kube-system` 네임스페이스에 속한다. -선택된 일부 애드온이 아래에 설명되었으며, 사용가능한 전체 확장 애드온 리스트는 [애드온](/docs/concepts/cluster-administration/addons/)을 참조한다. +선택된 일부 애드온은 아래에 설명하였고, 사용가능한 전체 확장 애드온 리스트는 +[애드온](/docs/concepts/cluster-administration/addons/)을 참조한다. ### DNS diff --git a/content/ko/docs/concepts/workloads/controllers/replicationcontroller.md b/content/ko/docs/concepts/workloads/controllers/replicationcontroller.md index 38d1c99937..6dcfd6908b 100644 --- a/content/ko/docs/concepts/workloads/controllers/replicationcontroller.md +++ b/content/ko/docs/concepts/workloads/controllers/replicationcontroller.md @@ -108,8 +108,8 @@ echo $pods nginx-3ntk0 nginx-4ok8v nginx-qrm3m ``` -여기서 셀렉터는 레플리케이션 컨트롤러의 셀렉터와 같다 ( -`kubectl describe` 의 출력에서 볼 수 있는 것과, 다른 형식의 파일인 `replication.yaml` 의 것). `--output=jsonpath` 옵션은 +여기서 셀렉터는 레플리케이션 컨트롤러(`kubectl describe` 의 출력에서 보인)의 셀렉터와 같고, +다른 형식의 파일인 `replication.yaml` 의 것과 동일하다. `--output=jsonpath` 옵션은 반환된 목록의 각 파드에서 이름을 가져오는 표현식을 지정한다. diff --git a/content/ko/docs/concepts/workloads/pods/init-containers.md b/content/ko/docs/concepts/workloads/pods/init-containers.md index c05c27de61..19fe8e18b5 100644 --- a/content/ko/docs/concepts/workloads/pods/init-containers.md +++ b/content/ko/docs/concepts/workloads/pods/init-containers.md @@ -5,126 +5,104 @@ weight: 40 --- {{% capture overview %}} -이 페이지는 초기화 컨테이너에 대한 개요를 제공한다. 초기화 컨테이너는 -앱 컨테이너들이 실행되기 전에 실행되는 특수한 컨테이너이며, 앱 이미지에는 없는 +이 페이지는 초기화 컨테이너에 대한 개요를 제공한다. 초기화 컨테이너는 +{{< glossary_tooltip text="파드" term_id="pod" >}}의 앱 컨테이너들이 실행되기 전에 실행되는 특수한 컨테이너이며, 앱 이미지에는 없는 유틸리티 또는 설정 스크립트 등을 포함할 수 있다. + +초기화 컨테이너는 `containers` 배열(앱 컨테이너를 기술하는)과 나란히 +파드 스펙에 명시할 수 있다. {{% /capture %}} -이 특징은 1.6에서 베타를 빠져나왔다. 초기화 컨테이너는 앱 `containers` 배열과 나란히 -파드 스펙에 명시될 수 있다. 베타 어노테이션의 값은 여전히 존중되며 파드 스펙 필드 값을 덮어쓴다. -하지만, 베타 어노테이션은 1.6과 1.7에서 사용 중단(deprecated)되었다. -1.8에서 어노테이션은 더는 지원되지 않으므로 파드 스펙 필드로 변환되어야 한다. - {{% capture body %}} + ## 초기화 컨테이너 이해하기 -[파드](/ko/docs/concepts/workloads/pods/pod-overview/)는 앱들을 실행하는 다수의 컨테이너를 -포함할 수 있다. 또한, 파드는 앱 컨테이너 실행 전에 동작되는 하나 이상의 -초기화 컨테이너도 포함할 수 있다. +{{< glossary_tooltip text="파드" term_id="pod" >}}는 앱들을 실행하는 다수의 컨테이너를 +포함할 수 있고, 또한 앱 컨테이너 실행 전에 동작되는 하나 이상의 +초기화 컨테이너도 포함할 수 있다. -다음의 경우를 제외하면, 초기화 컨테이너는 일반적인 컨테이너와 매우 유사하다. +다음의 경우를 제외하면, 초기화 컨테이너는 일반적인 컨테이너와 매우 유사하다. * 초기화 컨테이너는 항상 완료를 목표로 실행된다. -* 각 초기화 컨테이너는 다음 초기화 컨테이너가 시작되기 전에 성공적으로 완료되어야 한다. +* 각 초기화 컨테이너는 다음 초기화 컨테이너가 시작되기 전에 성공적으로 완료되어야 한다. -만약 파드를 위한 초기화 컨테이너가 실패한다면, 쿠버네티스는 초기화 컨테이너가 성공할 때까지 파드를 -반복적으로 재시작한다. 그러나, 만약 파드가 `restartPolicy`을 절대 하지 않음(Never)으로 설정한다면, 파드는 재시작되지 않는다. +만약 파드를 위한 초기화 컨테이너가 실패한다면, 쿠버네티스는 초기화 컨테이너가 성공할 때까지 파드를 +반복적으로 재시작한다. 그러나, 만약 파드의 `restartPolicy`을 절대 하지 않음(Never)으로 설정했다면, 파드는 재시작되지 않는다. -컨테이너를 초기화 컨테이너로 지정하기 위해서는, 파드 스펙에 앱 `containers` 배열과 나란히 -`initContainers` 필드를 +컨테이너를 초기화 컨테이너로 지정하기 위해서는, +파드 스펙에 앱 `containers` 배열과 나란히 `initContainers` 필드를 [컨테이너](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) 타입 오브젝트들의 JSON 배열로서 추가한다. -초기화 컨테이너의 상태는 `.status.initContainerStatuses` 필드를 -통해서 컨테이너 상태 배열로 반환된다 (`.status.containerStatuses`와 -유사하게). +초기화 컨테이너의 상태는 `.status.initContainerStatuses` 필드를 +통해서 컨테이너 상태 배열로 반환된다 +(`.status.containerStatuses` 필드와 유사하게). ### 일반적인 컨테이너와의 차이점 -초기화 컨테이너는 앱 컨테이너의 리소스 상한, 볼륨, 보안 세팅을 포함한 -모든 필드와 특징을 지원한다. 그러나, 초기화 컨테이너를 위한 리소스 요청량과 상한은 -약간 다르게 처리된다. 이것에 대해서는 아래 [리소스](#리소스)에 문서화되어 있다. 또한, 초기화 컨테이너는 -준비성 프로브(readiness probe)를 지원하지 않는다. 왜냐하면 초기화 컨테이너는 -파드가 준비 상태가 되기 전에 완료를 목표로 실행되어야 하기 -때문이다. +초기화 컨테이너는 앱 컨테이너의 리소스 상한(limit), 볼륨, 보안 세팅을 포함한 +모든 필드와 기능을 지원한다. +그러나, 초기화 컨테이너를 위한 리소스 요청량과 상한은 +[리소스](#리소스)에 문서화된 것처럼 다르게 처리된다. -만약 다수의 초기화 컨테이너가 파드에 지정되어 있다면, 해당 초기화 컨테이너들은 순차적으로 -한 번에 하나씩 실행된다. 각 초기화 컨테이너들은 다음 초기화 컨테이너가 실행되기 전에 성공되어야 한다. -모든 초기화 컨테이너들이 실행 완료되었을 때, 쿠버네티스는 파드를 초기화하고 -애플리케이션 컨테이너를 평소와 같이 실행한다. +또한, 초기화 컨테이너는 준비성 프로브(readiness probe)를 지원하지 않는다. 왜냐하면 초기화 컨테이너는 +파드가 준비 상태가 되기 전에 완료를 목표로 실행되어야 하기 때문이다. -## 초기화 컨테이너는 무엇을 위해서 사용될 수 있는가? +만약 다수의 초기화 컨테이너가 파드에 지정되어 있다면, Kubelet은 해당 초기화 컨테이너들을 +한 번에 하나씩 실행한다. 각 초기화 컨테이너는 다음 컨테이너를 실행하기 전에 꼭 성공해야 한다. +모든 초기화 컨테이너들이 실행 완료되었을 때, Kubelet은 파드의 애플리케이션 컨테이너들을 +초기화하고 평소와 같이 실행한다. -초기화 컨테이너는 앱 컨테이너와는 별도의 이미지를 가지고 있기 때문에, 시동(start-up)에 -관련된 코드에 몇 가지 이점을 가진다. +## 초기화 컨테이너 사용하기 -* 보안 상 앱 컨테이너 이미지에서는 바람직하지 않은 유틸리티를 포함하고 - 실행시킬 수 있다. -* 앱 이미지에는 없는 셋업을 위한 유틸리티 또는 맞춤 코드를 포함한다. +초기화 컨테이너는 앱 컨테이너와는 별도의 이미지를 가지고 있기 때문에, 시동(start-up)에 +관련된 코드로서 몇 가지 이점을 가진다. + +* 앱 이미지에는 없는 셋업을 위한 유틸리티 또는 맞춤 코드를 포함할 수 있다. 예를 들어, 셋업 중에 단지 `sed`, `awk`, `python`, 또는 `dig`와 같은 도구를 사용하기 위해서 다른 이미지로부터(`FROM`) 새로운 이미지를 만들 필요가 없다. +* 앱 컨테이너 이미지의 보안성을 떨어뜨릴 수도 있는 유틸리티를 안전하게 실행할 수 있다. * 애플리케이션 이미지 빌더와 디플로이어 역할은 독립적으로 동작될 수 있어서 공동의 단일 앱 이미지 형태로 빌드될 필요가 없다. * 초기화 컨테이너는 앱 컨테이너와 다른 파일 시스템 뷰를 가지도록 Linux 네임스페이스를 사용한다. - 결과적으로, 초기화 컨테이너에는 앱 컨테이너가 가질 수 없는 시크릿에 접근 권한이 주어질 수 있다. -* 앱 컨테이너들은 병렬로 실행되는 반면, 초기화 컨테이너들은 어떠한 앱 - 컨테이너라도 시작되기 전에 실행 완료되어야 하므로, 초기화 컨테이너는 사전 조건들이 + 결과적으로, 초기화 컨테이너에는 앱 컨테이너가 가질 수 없는 + {{< glossary_tooltip text="시크릿" term_id="secret" >}}에 접근 권한이 주어질 수 있다. +* 앱 컨테이너들은 병렬로 실행되는 반면, 초기화 컨테이너들은 어떠한 앱 + 컨테이너라도 시작되기 전에 실행 완료되어야 하므로, 초기화 컨테이너는 사전 조건들이 충족될 때까지 앱 컨테이너가 시동되는 것을 막거나 지연시키는 간편한 방법을 제공한다. + ### 예제 초기화 컨테이너를 사용하는 방법에 대한 몇 가지 아이디어는 다음과 같다. -* 다음과 같은 셀 커맨드로, 서비스가 생성될 때까지 기다리기. - - for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; done; exit 1 +* 다음과 같은 셸 커맨드로, + {{< glossary_tooltip text="서비스" term_id="service">}}가 생성될 때까지 기다리기. + ```shell + for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; done; exit 1 + ``` * 다음과 같은 커맨드로, 다운워드 API(Downward API)를 통한 원격 서버에 해당 파드를 등록하기. + ```shell + curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$()&ip=$()' + ``` - `curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$()&ip=$()'` +* 다음과 같은 커맨드로 앱 컨테이너가 시작되기 전에 일정 시간 기다리기. + ```shell + sleep 60 + ``` -* `sleep 60`와 같은 커맨드로 앱 컨테이너가 시작되기 전에 일정 시간 기다리기. -* git 저장소를 볼륨 안에 클론하기. -* 설정 파일에 값을 지정하고 메인 앱 컨테이너를 위한 설정 파일을 동적으로 생성하기 위한 템플릿 도구를 실행하기. - 예를 들어, 설정에 POD_IP 값을 지정하고 메인 앱 설정 파일을 Jinja를 통해서 생성. +* Git 저장소를 {{< glossary_tooltip text="볼륨" term_id="volume" >}} 안에 클론하기. -더 자세한 사용 예제는 [스테이트풀 셋 문서](/docs/concepts/workloads/controllers/statefulset/) -과 [프로덕션 파드 가이드](/docs/tasks/configure-pod-container/configure-pod-initialization/)에서 확인한다. +* 설정 파일에 값을 지정하고 + 메인 앱 컨테이너를 위한 설정 파일을 동적으로 생성하기 위한 템플릿 도구를 실행하기. + 예를 들어, 설정에 `POD_IP` 값을 지정하고 + 메인 앱 설정 파일을 Jinja를 통해서 생성. -### 사용되고 있는 초기화 컨테이너 +### 사용 중인 초기화 컨테이너 -쿠버네티스 1.5에 대한 다음의 yaml 파일은 두 개의 초기화 컨테이너를 포함한 간단한 파드에 대한 개요를 보여준다. -첫 번째는 `myservice`를 기다리고 두 번째는 `mydb`를 기다린다. 두 컨테이너들이 +쿠버네티스 1.5에 대한 다음의 yaml 파일은 두 개의 초기화 컨테이너를 포함한 간단한 파드에 대한 개요를 보여준다. +첫 번째는 `myservice`를 기다리고 두 번째는 `mydb`를 기다린다. 두 컨테이너들이 완료되면, 파드가 시작될 것이다. -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: myapp-pod - labels: - app: myapp - annotations: - pod.beta.kubernetes.io/init-containers: '[ - { - "name": "init-myservice", - "image": "busybox:1.28", - "command": ["sh", "-c", "until nslookup myservice; do echo waiting for myservice; sleep 2; done;"] - }, - { - "name": "init-mydb", - "image": "busybox:1.28", - "command": ["sh", "-c", "until nslookup mydb; do echo waiting for mydb; sleep 2; done;"] - } - ]' -spec: - containers: - - name: myapp-container - image: busybox:1.28 - command: ['sh', '-c', 'echo The app is running! && sleep 3600'] -``` - -쿠버네티스 1.6에는 새로운 구문이 있다. 다만, 예전 어노테이션 구문도 1.6과 1.7에서는 여전히 동작한다. 새로운 구문은 -1.8 또는 더 높은 버전에서 사용되어야 한다. 초기화에 대한 선언은 `spec`으로 옮겨졌다. - ```yaml apiVersion: v1 kind: Pod @@ -146,8 +124,6 @@ spec: command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;'] ``` -1.5 구문도 1.6에서 여전히 동작하지만, 1.6 구문 사용을 추천한다. 쿠버네티스 1.6에서는, 초기화 컨테이너가 API에서 필드로 -만들어졌었다. 베타 어노테이션은 1.6과 1.7에서 여전히 지원되지만, 1.8이나 더 높은 버전에서는 지원되지 않는다. 아래의 yaml file은 `mydb`와 `myservice` 서비스의 개요를 보여준다. @@ -182,6 +158,7 @@ kubectl apply -f myapp.yaml pod/myapp-pod created ``` +그리고 파드의 상태를 확인한다. ```shell kubectl get -f myapp.yaml ``` @@ -190,6 +167,7 @@ NAME READY STATUS RESTARTS AGE myapp-pod 0/1 Init:0/2 0 6m ``` +혹은 좀 더 자세히 살펴본다. ```shell kubectl describe -f myapp.yaml ``` @@ -227,14 +205,43 @@ Events: 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container with docker id 5ced34a04634; Security:[seccomp=unconfined] 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container with docker id 5ced34a04634 ``` + +파드의 초기화 컨테이너의 상태를 보기 위해, 다음을 실행한다. ```shell kubectl logs myapp-pod -c init-myservice # Inspect the first init container kubectl logs myapp-pod -c init-mydb # Inspect the second init container ``` -`mydb` 및 `myservice` 서비스를 시작하고 나면, 초기화 컨테이너가 완료되고 +`mydb` 및 `myservice` 서비스를 시작하고 나면, 초기화 컨테이너가 완료되고 `myapp-pod`가 생성된 것을 볼 수 있다. +여기에 이 서비스를 보이기 위해 사용할 수 있는 구성이 있다. + +```yaml +--- +apiVersion: v1 +kind: Service +metadata: + name: myservice +spec: + ports: + - protocol: TCP + port: 80 + targetPort: 9376 +--- +apiVersion: v1 +kind: Service +metadata: + name: mydb +spec: + ports: + - protocol: TCP + port: 80 + targetPort: 9377 +``` + +`mydb`와 `myservice` 서비스 생성하기. + ```shell kubectl apply -f services.yaml ``` @@ -243,26 +250,31 @@ service/myservice created service/mydb created ``` +초기화 컨테이너들이 완료되는 것과 `myapp-pod` 파드가 Runnning 상태로 +변경되는 것을 볼 것이다. + ```shell kubectl get -f myapp.yaml +``` +``` NAME READY STATUS RESTARTS AGE myapp-pod 1/1 Running 0 9m ``` -이 예제는 매우 단순하지만 사용자만의 초기화 컨테이너를 생성하는데 -영감을 줄 것이다. +이 간단한 예제는 사용자만의 초기화 컨테이너를 생성하는데 +영감을 줄 것이다. [다음 순서](#what-s-next)에는 더 자세한 예제의 링크가 있다. ## 자세한 동작 -파드 시동 시, 네트워크와 볼륨이 초기화되고 나면, 초기화 컨테이너가 -순서대로 시작된다. 각 초기화 컨테이너는 다음 컨테이너가 시작되기 전에 성공적으로 -종료되어야 한다. 만약 런타임 문제나 실패 상태로 종료되는 문제로인하여 초기화 컨테이너의 시작이 -실패된다면, 초기화 컨테이너는 파드의 `restartPolicy`에 따라서 재시도 된다. 다만, -파드의 `restartPolicy`이 항상(Always)으로 설정된 경우, 해당 초기화 컨테이너는 +파드 시동 시, 네트워크와 볼륨이 초기화되고 나면, 초기화 컨테이너가 +순서대로 시작된다. 각 초기화 컨테이너는 다음 컨테이너가 시작되기 전에 성공적으로 +종료되어야 한다. 만약 런타임 문제나 실패 상태로 종료되는 문제로인하여 초기화 컨테이너의 시작이 +실패된다면, 초기화 컨테이너는 파드의 `restartPolicy`에 따라서 재시도 된다. 다만, +파드의 `restartPolicy`이 항상(Always)으로 설정된 경우, 해당 초기화 컨테이너는 `restartPolicy`을 실패 시(OnFailure)로 사용한다. 파드는 모든 초기화 컨테이너가 성공되기 전까지 `Ready`될 수 없다. 초기화 컨테이너의 포트는 -서비스 하에 합쳐지지 않는다. 초기화 중인 파드는 `Pending` 상태이지만 +서비스 하에 합쳐지지 않는다. 초기화 중인 파드는 `Pending` 상태이지만 `Initializing`이 참이 되는 조건을 가져야 한다. 만약 파드가 [재시작](#파드-재시작-이유)되었다면, 모든 초기화 컨테이너는 @@ -275,69 +287,59 @@ myapp-pod 1/1 Running 0 9m 코드는 멱등성(indempotent)을 유지해야 한다. 특히, `EmptyDirs`에 있는 파일에 쓰기를 수행하는 코드는 출력 파일이 이미 존재할 가능성에 대비해야 한다. -초기화 컨테이너는 앱 컨테이너의 필드를 모두 가지고 있다. 그러나, 쿠버네티스는 -`readinessProbe`가 사용되는 것을 금지한다. 초기화 컨테이너가 완료 상태와 준비성을 +초기화 컨테이너는 앱 컨테이너의 필드를 모두 가지고 있다. 그러나, 쿠버네티스는 +`readinessProbe`가 사용되는 것을 금지한다. 초기화 컨테이너가 완료 상태와 준비성을 구분해서 정의할 수 없기 때문이다. 이것은 유효성 검사 중에 시행된다. -초기화 컨테이너들이 실패를 영원히 지속하는 상황을 방지하기 위해서 -파드의 `activeDeadlineSeconds`와 컨테이너의 `livenessProbe`를 -사용한다. +초기화 컨테이너들이 실패를 영원히 지속하는 상황을 방지하기 위해서 +파드의 `activeDeadlineSeconds`와 컨테이너의 `livenessProbe`를 사용한다. -파드 내의 각 앱과 초기화 컨테이너의 이름은 유일해야 한다. 어떤 +파드 내의 각 앱과 초기화 컨테이너의 이름은 유일해야 한다. 어떤 컨테이너가 다른 컨테이너와 같은 이름을 공유하는 경우 유효성 오류가 발생한다. ### 리소스 -초기화 컨테이너에게 명령과 실행이 주어진 경우, 리소스 사용에 대한 +초기화 컨테이너에게 명령과 실행이 주어진 경우, 리소스 사용에 대한 다음의 규칙이 적용된다. -* 모든 컨테이너에 정의된 특정 리소스 요청량 또는 상한 중 가장 +* 모든 컨테이너에 정의된 특정 리소스 요청량 또는 상한 중 가장 높은 것은 *유효한 초기화 요청량/상한* 이다. * 리소스를 위한 파드의 *유효한 초기화 요청량/상한* 은 다음 보다 더 높다. * 모든 앱 컨테이너의 리소스에 대한 요청량/상한의 합계 * 리소스에 대한 유효한 초기화 요청량/상한 -* 스케줄링은 유효한 요청/상한에 따라 이루어진다. 즉, - 초기화 컨테이너는 파드의 삶에서는 사용되지 않는 초기화를 위한 리소스를 - 예약할 수 있다. -* 파드의 *유효한 QoS 계층* 에서 QoS 계층은 초기화 컨테이너들과 +* 스케줄링은 유효한 요청/상한에 따라 이루어진다. 즉, + 초기화 컨테이너는 파드의 삶에서는 사용되지 않는 초기화를 위한 리소스를 + 예약할 수 있다. +* 파드의 *유효한 QoS 계층* 에서 QoS(서비스의 품질) 계층은 초기화 컨테이너들과 앱 컨테이너들의 QoS 계층과 같다. -쿼터 및 상한은 유효한 파드의 요청량 및 상한에 따라 +쿼터 및 상한은 유효한 파드의 요청량 및 상한에 따라 적용된다. -파드 레벨 cgroup은 유효한 파드 요청량 및 상한을 기반으로 한다. 이는 스케줄러와 같다. +파드 레벨 cgroup은 유효한 파드 요청량 및 상한을 기반으로 한다. +이는 스케줄러와 같다. + ### 파드 재시작 이유 -파드는 다음과 같은 사유로, 초기화 컨테이너들의 재-실행을 일으키는, 재시작을 수행할 수 +파드는 다음과 같은 사유로, 초기화 컨테이너들의 재-실행을 일으키는, 재시작을 수행할 수 있다. -* 사용자가 초기화 컨테이너 이미지의 변경을 일으키는 파드 스펙 업데이트를 수행했다. - Init Container 이미지를 변경하면 파드가 다시 시작된다. 앱 컨테이너 - 이미지의 변경은 앱 컨테이너만 재시작시킨다. -* 파드 인프라스트럭처 컨테이너가 재시작되었다. 이는 일반적인 상황이 아니며 노드에 +* 사용자가 초기화 컨테이너 이미지의 변경을 일으키는 파드 스펙 업데이트를 수행했다. + Init Container 이미지를 변경하면 파드가 다시 시작된다. 앱 컨테이너 + 이미지의 변경은 앱 컨테이너만 재시작시킨다. +* 파드 인프라스트럭처 컨테이너가 재시작되었다. 이는 일반적인 상황이 아니며 노드에 대해서 root 접근 권한을 가진 누군가에 의해서 수행됐을 것이다. -* 파드 내의 모든 컨테이너들이, 재시작을 강제하는 `restartPolicy`이 항상으로 설정되어 있는, - 동안 종료되었다. 그리고 초기화 컨테이너의 완료 기록이 가비지 수집 +* 파드 내의 모든 컨테이너들이, 재시작을 강제하는 `restartPolicy`이 항상으로 설정되어 있는, + 동안 종료되었다. 그리고 초기화 컨테이너의 완료 기록이 가비지 수집 때문에 유실되었다. -## 지원 및 호환성 - -Api서버 버전 1.6.0 또는 더 높은 버전으로 구성된 클러스터는 `.spec.initContainers` -필드를 사용하여 초기화 컨테이너를 지원한다. 이전 버전들은 초기화 컨테이너를 알파 또는 -베타 어노테이션을 사용하여 지원한다. `.spec.initContainers` 필드는 알파 또는 베타 -어노테이션에도 반영되어 있어서 버전 1.3.0 이상의 Kubelet이 초기화 컨테이너를 실행할 수 -있도록 한다. 따라서, 버전 1.6 api서버가 기존에 생성된 파드들의 초기화 컨테이너 기능 손실 없이 -안전하게 버전 1.5.x로 롤백할 수 있게 한다. - -Api서버 및 Kubelet 버전 1.8.0 이상에서는, 사용 중단된 어노테이션을 -`.spec.initContainers` 필드로 변환하는 것이 필요한, 알파 및 베타 어노테이션의 지원이 중단되었다. - {{% /capture %}} {{% capture whatsnext %}} * [초기화 컨테이너를 가진 파드 생성하기](/docs/tasks/configure-pod-container/configure-pod-initialization/#creating-a-pod-that-has-an-init-container) +* [초기화 컨테이너 디버깅](/docs/tasks/debug-application-cluster/debug-init-containers/) 알아보기 {{% /capture %}} diff --git a/content/ko/docs/concepts/workloads/pods/podpreset.md b/content/ko/docs/concepts/workloads/pods/podpreset.md index 1a4f036794..204f7d9ec1 100644 --- a/content/ko/docs/concepts/workloads/pods/podpreset.md +++ b/content/ko/docs/concepts/workloads/pods/podpreset.md @@ -69,8 +69,7 @@ weight: 50 minikube에서는 클러스터가 시작할 때 `--extra-config=apiserver.runtime-config=settings.k8s.io/v1alpha1=true` 플래그를 추가한다. -1. 어드미션 컨트롤러 `PodPreset`을 활성화하였다. - 이것을 이루는 방법 중 하나는 +1. 어드미션 컨트롤러 `PodPreset`을 활성화하였다. 이것을 이루는 방법 중 하나는 API 서버를 위해서 명시된 `--enable-admission-plugins` 옵션에 `PodPreset`을 포함하는 것이다. minikube에서는 클러스터가 시작할 때 diff --git a/content/ko/docs/contribute/participating.md b/content/ko/docs/contribute/participating.md index 5162b44177..93e5335c22 100644 --- a/content/ko/docs/contribute/participating.md +++ b/content/ko/docs/contribute/participating.md @@ -201,22 +201,30 @@ SIG Docs 승인자가 되는 방법과 GitHub 그룹에 당신을 추가하기를 요청한다. `kubernetes-website-admins` GitHub 그룹의 멤버만이 신규 멤버를 GitHub 그룹에 추가할 수 있다. -#### 웹사이트 관리자 되기 +#### 승인자의 책임 -`kubernetes-website-admins` GitHub 그룹의 멤버는 GitHub 그룹의 멤버십을 관리할 수 있고 -리포지터리를 세팅하거나 웹훅(webhook)을 추가, 삭제하고 트러블슈팅하는 것을 포함한 -모든 관리 권한을 가질 수 있다. -모든 SIG Docs 승인자가 이 수준의 액세스를 할 필요는 없다. +승인자는 리뷰와 풀리퀘스트를 웹사이트 리포지터리에 머지하여 문서를 개선한다. 이 역할에는 추가적인 권한이 필요하므로, 승인자에게는 별도의 책임이 부여된다. -만약 이 수준의 접근 권한이 필요하다면, 현재 웹사이트 관리자나 -[쿠버네티스 Slack](https://kubernetes.slack.com) #sig-docs 채널에서 말한다. +- 승인자는 PR들을 리포에 머지하는 `/approve` 명령을 사용할 수 있다. + + 부주의한 머지로 인해 사이트를 파괴할 수 있으므로, 머지할 때에 그 의미를 확인해야 한다. + +- 제안된 변경이 컨트리뷰션 가이드 라인에 적합한지 확인한다. + + 질문이 생기거나 확실하지 않다면 자유롭게 추가 리뷰를 요청한다. + +- PR을 `/approve` 하기 전에 Netlify 테스트 결과를 검토한다. + + 승인 전에 반드시 Netlify 테스트를 통과해야 한다 + +- 승인 전에 PR에 대한 Netlify 프리뷰 페이지를 방문하여, 제대로 보이는지 확인한다. #### PR Wrangler SIG Docs 승인자는 -[PR Wrangler 로테이션 스케줄러](https://github.com/kubernetes/website/wiki/PR-Wranglers)에 -올라서 주 단위로 돌아가며 역할을 수행한다. -모든 SIG Docs 승인자는 이 로테이션에 참여하게 된다. 보다 자세한 내용은 +[PR Wrangler 회람 스케줄러](https://github.com/kubernetes/website/wiki/PR-Wranglers)에 +참여하여 주 단위로 돌아가며 역할을 수행한다. +SIG Docs는 모든 승인자들이 이 회람에 참여하기를 기대한다. 보다 자세한 내용은 [일주일 간 PR Wrangler 되기](/docs/contribute/advanced#be-the-pr-wrangler-for-a-week) 문서를 참고한다. diff --git a/content/ko/docs/home/supported-doc-versions.md b/content/ko/docs/home/supported-doc-versions.md new file mode 100644 index 0000000000..69245a2f41 --- /dev/null +++ b/content/ko/docs/home/supported-doc-versions.md @@ -0,0 +1,28 @@ +--- +title: 쿠버네티스 문서의 버전 지원 +content_template: templates/concept +card: + name: about + weight: 10 + title: 문서의 버전 지원 +--- + +{{% capture overview %}} + +이 웹 사이트에는 현재 버전의 쿠버네티스와 이전 4개 버전의 +쿠버네티스에 대한 문서가 포함되어 있습니다. + +{{% /capture %}} + +{{% capture body %}} + +## 현재 버전 + +현재 버전은 +[{{< param "version" >}}](/). + +## 이전 버전 + +{{< versions-other >}} + +{{% /capture %}} diff --git a/content/ko/docs/reference/_index.md b/content/ko/docs/reference/_index.md index 4ad5e4a590..8ca5921f15 100644 --- a/content/ko/docs/reference/_index.md +++ b/content/ko/docs/reference/_index.md @@ -16,13 +16,13 @@ content_template: templates/concept ## API 레퍼런스 -* [쿠버네티스 API 개요](/docs/reference/using-api/api-overview/) - 쿠버네티스 API에 대한 개요 +* [쿠버네티스 API 개요](/ko/docs/reference/using-api/api-overview/) - 쿠버네티스 API에 대한 개요 * 쿠버네티스 API 버전 + * [1.15](/docs/reference/generated/kubernetes-api/v1.15/) * [1.14](/docs/reference/generated/kubernetes-api/v1.14/) * [1.13](/docs/reference/generated/kubernetes-api/v1.13/) * [1.12](/docs/reference/generated/kubernetes-api/v1.12/) * [1.11](/docs/reference/generated/kubernetes-api/v1.11/) - * [1.10](/docs/reference/generated/kubernetes-api/v1.10/) ## API 클라이언트 라이브러리 diff --git a/content/ko/docs/reference/glossary/cloud-provider.md b/content/ko/docs/reference/glossary/cloud-provider.md new file mode 100644 index 0000000000..a0fcc998e9 --- /dev/null +++ b/content/ko/docs/reference/glossary/cloud-provider.md @@ -0,0 +1,17 @@ +--- +title: 클라우드 공급자 +id: cloud-provider +date: 2018-04-12 +full_link: /docs/concepts/cluster-administration/cloud-providers +short_description: > + 클라우드 공급자는 쿠버네티스 클러스터를 실행할 수 있는 클라우드 컴퓨팅 플랫폼을 제공하는 회사. + +aka: +tags: +- community +--- + 클라우드 공급자는 쿠버네티스 클러스터를 실행할 수 있는 클라우드 컴퓨팅 플랫폼을 제공하는 회사. + + + +클라우드 컴퓨팅 플랫폼을 제공하는 클라우드 공급자 또는 클라우드 서비스 공급자(CSP)라고 한다. 이들은 인프라 서비스(IaaS) 또는 플랫폼 서비스(PaaS)를 제공할 수 있다. 클라우드 공급자는 쿠버네티스 클러스터를 호스트 하며 클러스터와 상호작용하는 로드밸런서, 스토리지 클래스 등의 서비스도 제공한다. diff --git a/content/ko/docs/reference/glossary/cluster.md b/content/ko/docs/reference/glossary/cluster.md index 50277df48e..92f8eabd37 100755 --- a/content/ko/docs/reference/glossary/cluster.md +++ b/content/ko/docs/reference/glossary/cluster.md @@ -1,19 +1,17 @@ --- title: 클러스터(Cluster) id: cluster -date: 2018-04-12 +date: 2019-06-15 full_link: short_description: > - 쿠버네티스를 통해 관리되는 컨테이너화 된 애플리케이션을 실행하는, 노드라고 불리는 기계의 집합. + 쿠버네티스에서 관리하는 컨테이너화된 애플리케이션을 실행하는 노드라고 하는 기계의 집합. 클러스터는 최소 1개의 워커 노드와 최소 1개의 마스터 노드를 가진다. -aka: +aka: tags: - fundamental - operation --- - 쿠버네티스를 통해 관리되는 컨테이너화 된 애플리케이션을 실행하는, 노드라고 불리는 기계의 집합. +쿠버네티스에서 관리하는 컨테이너화된 애플리케이션을 실행하는 노드라고 하는 기계의 집합. 클러스터는 최소 1개의 워커 노드와 최소 1개의 마스터 노드를 가진다. - -클러스터는 여러 개의 워커 노드와 적어도 하나의 마스터를 가진다. - +워커 노드는 애플리케이션의 구성요소인 파드를 호스트한다. 마스터 노드는 워커 노드와 클러스터 내 파드를 관리한다. 다수의 마스터 노드는 장애극복(failover)과 고가용성의 클러스터에서 사용한다. diff --git a/content/ko/docs/reference/glossary/container-runtime.md b/content/ko/docs/reference/glossary/container-runtime.md index 8a26ee8147..112282e178 100644 --- a/content/ko/docs/reference/glossary/container-runtime.md +++ b/content/ko/docs/reference/glossary/container-runtime.md @@ -15,7 +15,7 @@ tags: -쿠버네티스느 여러 컨테이너 런타임을 지원한다. [Docker](http://www.docker.com), +쿠버네티스는 여러 컨테이너 런타임을 지원한다. [Docker](http://www.docker.com), [containerd](https://containerd.io), [cri-o](https://cri-o.io/), [rktlet](https://github.com/kubernetes-incubator/rktlet)과 [Kubernetes CRI (컨테이너 런타임 인터페이스)](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md)를 구현한 모든 소프트웨어. diff --git a/content/ko/docs/reference/glossary/daemonset.md b/content/ko/docs/reference/glossary/daemonset.md index 580f6bf0be..81405e5437 100755 --- a/content/ko/docs/reference/glossary/daemonset.md +++ b/content/ko/docs/reference/glossary/daemonset.md @@ -1,20 +1,20 @@ --- -title: 데몬 셋(DaemonSet) +title: 데몬셋(DaemonSet) id: daemonset date: 2018-04-12 full_link: /docs/concepts/workloads/controllers/daemonset short_description: > - 파드의 복제품이 클러스터의 노드 집합에 걸쳐 동작하는 것을 확실히 한다. + 파드의 복제본을 클러스터 노드 집합에서 동작하게 한다. -aka: +aka: tags: - fundamental - core-object - workload --- - {{< glossary_tooltip text="파드" term_id="pod" >}}의 복제품이 {{< glossary_tooltip text="클러스터" term_id="cluster" >}}의 노드 집합에 걸쳐 동작하는 것을 확실히 한다. + {{< glossary_tooltip text="파드" term_id="pod" >}} 복제본을 {{< glossary_tooltip text="클러스터" term_id="cluster" >}} 노드 집합에서 동작하게 한다. - + -일반적으로 모든 {{< glossary_tooltip text="노드" term_id="node" >}}에서 실행돼야 하는 로그 수집기 및 모니터링 에이전트 등의 시스템 데몬을 디플로이하기 위해서 사용된다. +일반적으로 모든 {{< glossary_tooltip text="노드" term_id="node" >}}에서 실행돼야 하는 로그 수집기 및 모니터링 에이전트 등의 시스템 데몬을 배포하기 위해서 사용된다. diff --git a/content/ko/docs/reference/glossary/deployment.md b/content/ko/docs/reference/glossary/deployment.md index c6ef06e304..39c5e4d7f6 100755 --- a/content/ko/docs/reference/glossary/deployment.md +++ b/content/ko/docs/reference/glossary/deployment.md @@ -4,17 +4,17 @@ id: deployment date: 2018-04-12 full_link: /docs/concepts/workloads/controllers/deployment/ short_description: > - 레플리케이션 된 애플리케이션을 관리하는 API 오브젝트. + 복제된(replicated) 애플리케이션을 관리하는 API 오브젝트. -aka: +aka: tags: - fundamental - core-object - workload --- - 레플리케이션 된 애플리케이션을 관리하는 API 오브젝트. + 복제된 애플리케이션을 관리하는 API 오브젝트. - + -각 레플리카는 {{< glossary_tooltip text="파드" term_id="pod" >}}로 표현되며, 각 파드는 클러스터의 노드에 분산된다. +각 레플리카는 {{< glossary_tooltip text="파드" term_id="pod" >}}로 표현되며, 파드는 클러스터의 노드에 분산된다. diff --git a/content/ko/docs/reference/glossary/init-container.md b/content/ko/docs/reference/glossary/init-container.md index 890b34505e..fdb4d2c82c 100755 --- a/content/ko/docs/reference/glossary/init-container.md +++ b/content/ko/docs/reference/glossary/init-container.md @@ -2,17 +2,17 @@ title: 초기화 컨테이너(Init Container) id: init-container date: 2018-04-12 -full_link: +full_link: short_description: > - 앱 컨테이너가 동작하기 전에 완료되기 위해 실행되는 하나 이상의 초기화 컨테이너. + 앱 컨테이너가 동작하기 전에 완료되기 위해 실행되는 하나 이상의 초기화 컨테이너. -aka: +aka: tags: - fundamental --- - 앱 컨테이너가 동작하기 전에 완료되기 위해 실행되는 하나 이상의 초기화 컨테이너. + 앱 컨테이너가 동작하기 전에 완료되기 위해 실행되는 하나 이상의 초기화 컨테이너. - + 한 가지 차이점을 제외하면, 초기화 컨테이너는 일반적인 앱 컨테이너와 동일하다. 초기화 컨테이너는 앱 컨테이너가 시작되기 전에 완료되는 것을 목표로 실행되어야 한다. 초기화 컨테이너는 연달아 실행된다. 다시말해, 각 초기화 컨테이너의 실행은 다음 초기화 컨테이너가 시작되기 전에 완료되어야 한다. diff --git a/content/ko/docs/reference/glossary/limitrange.md b/content/ko/docs/reference/glossary/limitrange.md index 5588e45449..26802da637 100755 --- a/content/ko/docs/reference/glossary/limitrange.md +++ b/content/ko/docs/reference/glossary/limitrange.md @@ -4,7 +4,7 @@ id: limitrange date: 2019-04-15 full_link: /docs/concepts/policy/limit-range/ short_description: > - 네임스페이스 안의 컨테이너나 파드의 리소스 사용량을 제한하는 제약을 제공한다. + 네임스페이스 내에 컨테이너나 파드당 리소스 소비를 한정하는 제약 조건을 제공한다. aka: tags: @@ -16,8 +16,8 @@ related: - container --- - 네임스페이스 안의 {{< glossary_tooltip text="컨테이너" term_id="container" >}}나 {{< glossary_tooltip text="파드" term_id="pod" >}}의 리소스 사용량을 제한하는 제약을 제공한다. + 네임스페이스 내에 {{< glossary_tooltip text="컨테이너" term_id="container" >}}나 {{< glossary_tooltip text="파드" term_id="pod" >}}당 리소스 소비를 한정하는 제약 조건을 제공한다. -범위 제한은 타입별로 만들 수 있는 객체의 수와 -네임스페이스 안의 개별 {{< glossary_tooltip text="컨테이너" term_id="container" >}}나 {{< glossary_tooltip text="파드" term_id="pod" >}}가 요청하거나 소비한 컴퓨팅 리소스의 양을 제한한다. +범위 제한은 타입별로 만들 수 있는 오브젝트의 개수와 +네임스페이스 안에 개별 {{< glossary_tooltip text="컨테이너" term_id="container" >}}나 {{< glossary_tooltip text="파드" term_id="pod" >}}가 요청하거나 소비할 컴퓨팅 리소스의 양을 제한한다. diff --git a/content/ko/docs/reference/glossary/static-pod.md b/content/ko/docs/reference/glossary/static-pod.md index 91d5d2794c..ec5ea8c8e8 100755 --- a/content/ko/docs/reference/glossary/static-pod.md +++ b/content/ko/docs/reference/glossary/static-pod.md @@ -4,11 +4,11 @@ id: static-pod date: 2091-02-12 full_link: /docs/tasks/administer-cluster/static-pod/ short_description: > - 특정 노드의 kubelet 데몬이 직접 관리하는 파드 + 특정 노드의 Kubelet 데몬이 직접 관리하는 파드 aka: tags: - fundamental --- - API 서버가 관찰하지 않고, 특정 노드의 kubelet 데몬이 + API 서버가 관찰하지 않고, 특정 노드의 Kubelet 데몬이 직접 관리하는 {{< glossary_tooltip text="파드" term_id="pod" >}}. diff --git a/content/ko/docs/reference/kubectl/cheatsheet.md b/content/ko/docs/reference/kubectl/cheatsheet.md index 3dd586cbde..76e05c79e6 100644 --- a/content/ko/docs/reference/kubectl/cheatsheet.md +++ b/content/ko/docs/reference/kubectl/cheatsheet.md @@ -344,7 +344,7 @@ Kubectl 로그 상세 레벨(verbosity)은 `-v` 또는`--v` 플래그와 로그 로그 레벨 | 세부 사항 --------------| ----------- -`--v=0` | 일반적으로 운영자에게 유용함. +`--v=0` | 일반적으로 클러스터 운영자(operator)에게 *항상* 보여지게 하기에는 유용함. `--v=1` | 자세한 정보를 원하지 않는 경우, 적절한 기본 로그 수준. `--v=2` | 서비스와 시스템의 중요한 변화와 관련이있는 중요한 로그 메시지에 대한 유용한 정상 상태 정보. 이는 대부분의 시스템에서 권장되는 기본 로그 수준이다. `--v=3` | 변경 사항에 대한 확장 정보. diff --git a/content/ko/docs/reference/using-api/_index.md b/content/ko/docs/reference/using-api/_index.md new file mode 100644 index 0000000000..a224824aa1 --- /dev/null +++ b/content/ko/docs/reference/using-api/_index.md @@ -0,0 +1,5 @@ +--- +title: 쿠버네티스 API 사용하기 +weight: 10 +toc-hide: true +--- diff --git a/content/ko/docs/reference/using-api/api-overview.md b/content/ko/docs/reference/using-api/api-overview.md new file mode 100644 index 0000000000..3c77ed24b8 --- /dev/null +++ b/content/ko/docs/reference/using-api/api-overview.md @@ -0,0 +1,111 @@ +--- +title: 쿠버네티스 API 개요 +content_template: templates/concept +weight: 10 +card: + name: 레퍼런스 + weight: 50 + title: API 개요 +--- + +{{% capture overview %}} +이 페이지는 쿠버네티스 API에 대한 개요를 제공한다. +{{% /capture %}} + +{{% capture body %}} +REST API는 쿠버네티스의 근본적인 구조이다. 모든 조작, 컴포넌트 간의 통신과 외부 사용자의 명령은 API 서버에서 처리할 수 있는 REST API 호출이다. 따라서, 쿠버네티스 플랫폼 안의 모든 것은 +API 오브젝트로 취급되고, +[API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/)에 상응하는 항목이 있다. + +대부분의 작업은 API에 의존하고 있는 +[kubectl](/docs/reference/kubectl/overview/) 커맨드라인 인터페이스 또는 +[kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/)과 같은 다른 커맨드라인 툴을 통해 수행할 수 있다. +그러나, REST 호출 사용을 통해서 API에 직접 접근할 수도 있다. + +쿠버네티스 API를 사용하는 애플리케이션을 작성하는 경우 +[클라이언트 라이브러리](/docs/reference/using-api/client-libraries/)중 하나의 사용을 고려한다. + +## API 버전 규칙 + +필드를 없애거나 리소스 표현을 재구성하기 쉽도록, +쿠버네티스는 `/api/v1`이나 `/apis/extensions/v1beta1`과 같이 +각각 다른 API 경로에서 복수의 API 버전을 지원한다. + +아래를 위해 버전은 리소스나 필드 수준보다는 API 수준에서 설정된다. + +- API가 시스템 리소스와 동작에 대해 명확하고 일관성 있게 표현하는 것을 보장 +- 수명 종료(end-of-life) 또는 실험적인 API 접근 제어 활성화 + +JSON과 Protobuf 직렬화 스키마 모두 스키마 변경에 대해서 동일한 가이드라인을 따른다. 이후 설명에서는 이 형식 모두를 다룬다. + +{{< note >}} +API 버전 규칙과 소프트웨어 버전 규칙은 간접적으로 연관된다. +[API와 릴리스 버전 부여에 관한 제안](https://git.k8s.io/community/contributors/design-proposals/release/versioning.md)에는 API 버전 규칙과 소프트웨어 버전 규칙 간의 관계가 기술되어 있다. +{{< /note >}} + +API 버전의 차이는 수준의 안정성과 지원의 차이를 나타낸다. [API 변경 문서](https://git.k8s.io/community/contributors/devel/api_changes.md#alpha-beta-and-stable-versions)에서 각 수준의 기준에 대한 더 많은 정보를 찾을 수 있다. + +아래는 각 수준의 기준에 대한 요약이다. + +- 알파(Alpha) 수준: + - 버전 이름에 `alpha`가 포함된다. (예: `v1alpha1`) + - 버그가 있을 수도 있다. 이 기능을 활성화하면 버그가 노출될 수 있다. 기본적으로 비활성화되어 있다. + - 기능에 대한 기술 지원이 언제든 공지 없이 중단될 수 있다. + - 다음 소프트웨어를 릴리스할 때 공지 없이 API의 호환성이 깨지는 방식으로 변경될 수 있다. + - 버그의 위험이 높고 장기간 지원되지 않으므로 단기간 테스트 용도의 클러스터에서만 사용하기를 권장한다. + +- 베타(Beta) 수준: + - 버전 이름에 `beta`가 포함된다. (예: `v2beta3`). + - 코드가 잘 테스트되었다. 이 기능을 활성화 시켜도 안전하다. 기본적으로 활성화되어 있다. + - 구체적인 내용이 바뀔 수는 있지만, 전반적인 기능에 대한 기술 지원이 중단되지 않는다. + - 오브젝트에 대한 스키마나 문법이 다음 베타 또는 안정화 릴리스에서 호환되지 않는 방식으로 바뀔 수도 있다. 이런 경우, 다음 버전으로 이관할 수 있는 가이드가 제공된다. 이때 API 오브젝트의 삭제, 편집 또는 재생성이 + 필요할 수도 있다. 편집 절차는 좀 생각해볼 필요가 있다. 이 기능에 의존하고 있는 애플리케이션은 다운타임이 필요할 수도 있다. + - 이후 여러 버전에서 잠재적으로 호환되지 않을 수도 있으므로 사업적으로 중요하지 않은 용도로만 사용하기를 권장한다. 복수의 클러스터를 가지고 있어서 독립적으로 업그레이드할 수 있다면, 이런 제약에서 안심이 될 수도 있겠다. + + {{< note >}} +베타 기능을 사용해보고 피드백을 제공하자. 일단 베타가 끝나면, 실질적으로 더 많은 변경이 어렵다. + {{< /note >}} + +- 안정화(stable) 수준: + - 버전 이름이 `vX`이고 `X` 는 정수다. + - 안정화 버전의 기능은 이후 여러 버전에 걸쳐서 소프트웨어 릴리스에 포함된다. + +## API 그룹 + +[*API 그룹*](https://git.k8s.io/community/contributors/design-proposals/api-machinery/api-group.md)은 쿠버네티스 API를 더 쉽게 확장하게 해준다. API 그룹은 REST 경로와 직렬화된 객체의 `apiVersion` 필드에 명시된다. + +현재 다음과 같은 다양한 API 그룹이 사용되고 있다: + +* *핵심* (또는 *레거시*라고 불리는) 그룹은 `apiVersion: v1`와 같이 `apiVersion` 필드에 명시되지 않고 REST 경로 `/api/v1`에 있다. +* 이름이 있는 그룹은 REST 경로 `/apis/$GROUP_NAME/$VERSION`에 있으며 `apiVersion: $GROUP_NAME/$VERSION`을 사용한다 + (예를 들어 `apiVersion: batch/v1`). 지원되는 API 그룹 전체의 목록은 [쿠버네티스 API 참조 문서](/docs/reference/)에서 확인할 수 있다. + +[사용자 정의 리소스](/docs/concepts/api-extension/custom-resources/)로 API를 확장하는 경우에는 다음 두 종류의 경로가 지원된다. + + - 기본적인 CRUD 요구에는 + [CustomResourceDefinition](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/) + - 쿠버네티스 API의 의미론적 전체 집합으로 사용자만의 Apiserver를 구현하려는 경우에는 [aggregator](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/aggregated-api-servers.md) + + +## API 그룹 활성화 시키기 + +특정 리소스와 API 그룹은 기본적으로 활성화되어 있다. 이들은 apiserver에서 `--runtime-config`를 설정해서 활성화하거나 +비활성화 시킬 수 있다. `--runtime-config`는 쉼표로 분리된 값을 허용한다. 예를 들어: + - batch/v1을 비활성화하려면 `--runtime-config=batch/v1=false`로 설정 + - batch/v2alpha1을 활성화하려면 `--runtime-config=batch/v2alpha1`로 설정 +이 플래그는 apiserver의 런타임 구성을 설명하는 쉼표로 분리된 키=값 쌍의 집합을 허용한다. + +{{< note >}} +그룹이나 리소스를 활성화 또는 비활성화하려면, apiserver와 controller-manager를 재시작하여 +`--runtime-config` 변경을 반영해야 한다. +{{< /note >}} + +## 그룹 내 리소스 활성화 시키기 + +데몬셋, 디플로이먼트, HorizontalPodAutoscaler, 인그레스, 잡 및 레플리카셋이 기본적으로 활성화되어 있다. +다른 확장 리소스는 apiserver의 `--runtime-config`를 설정해서 +활성화할 수 있다. `--runtime-config`는 쉼표로 분리된 값을 허용한다. 예를 들어 디플로이먼트와 잡을 비활성화하려면, +`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/ingresses=false`와 같이 설정한다. +{{% /capture %}} + + diff --git a/content/ko/docs/setup/_index.md b/content/ko/docs/setup/_index.md index a8693e3b58..29789d5dbd 100644 --- a/content/ko/docs/setup/_index.md +++ b/content/ko/docs/setup/_index.md @@ -66,6 +66,7 @@ card: | [Amazon](https://aws.amazon.com) | [Amazon EKS](https://aws.amazon.com/eks/) |[Amazon EC2](https://aws.amazon.com/ec2/) | | | | | [AppsCode](https://appscode.com/products/pharmer/) | ✔ | | | | | | [APPUiO](https://appuio.ch/)  | ✔ | ✔ | ✔ | | | | +| [Banzai Cloud Pipeline Kubernetes Engine (PKE)](https://banzaicloud.com/products/pke/) | | ✔ | | ✔ | ✔ | ✔ | | [CenturyLink Cloud](https://www.ctl.io/) | | ✔ | | | | | [Cisco Container Platform](https://cisco.com/go/containers) | | | ✔ | | | | [Cloud Foundry Container Runtime (CFCR)](https://docs-cfcr.cfapps.io/) | | | | ✔ |✔ | @@ -81,6 +82,7 @@ card: | [Giant Swarm](https://giantswarm.io/) | ✔ | ✔ | ✔ | | | [Google](https://cloud.google.com/) | [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/) | [Google Compute Engine (GCE)](https://cloud.google.com/compute/)|[GKE On-Prem](https://cloud.google.com/gke-on-prem/) | | | | | | | | | [IBM](https://www.ibm.com/in-en/cloud) | [IBM Cloud Kubernetes Service](https://cloud.ibm.com/kubernetes/catalog/cluster)| |[IBM Cloud Private](https://www.ibm.com/in-en/cloud/private) | | +| [Ionos](https://www.ionos.com/enterprise-cloud) | [Ionos Managed Kubernetes](https://www.ionos.com/enterprise-cloud/managed-kubernetes) | [Ionos Enterprise Cloud](https://www.ionos.com/enterprise-cloud) | | | [Kontena Pharos](https://www.kontena.io/pharos/) | |✔| ✔ | | | | [Kubermatic](https://www.loodse.com/) | ✔ | ✔ | ✔ | | | | [KubeSail](https://kubesail.com/) | ✔ | | | | | @@ -100,6 +102,7 @@ card: | [Supergiant](https://supergiant.io/) | |✔ | | | | | [SUSE](https://www.suse.com/) | | ✔ | | | | | [SysEleven](https://www.syseleven.io/) | ✔ | | | | | +| [Tencent Cloud](https://intl.cloud.tencent.com/) | [Tencent Kubernetes Engine](https://intl.cloud.tencent.com/product/tke) | ✔ | ✔ | | | ✔ | | [VEXXHOST](https://vexxhost.com/) | ✔ | ✔ | | | | | [VMware](https://cloud.vmware.com/) | [VMware Cloud PKS](https://cloud.vmware.com/vmware-cloud-pks) |[VMware Enterprise PKS](https://cloud.vmware.com/vmware-enterprise-pks) | [VMware Enterprise PKS](https://cloud.vmware.com/vmware-enterprise-pks) | [VMware Essential PKS](https://cloud.vmware.com/vmware-essential-pks) | |[VMware Essential PKS](https://cloud.vmware.com/vmware-essential-pks) diff --git a/content/ko/docs/tasks/debug-application-cluster/_index.md b/content/ko/docs/tasks/debug-application-cluster/_index.md new file mode 100755 index 0000000000..0613fed1ef --- /dev/null +++ b/content/ko/docs/tasks/debug-application-cluster/_index.md @@ -0,0 +1,5 @@ +--- +title: "모니터링, 로깅, 그리고 디버깅" +weight: 80 +--- + diff --git a/content/ko/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md b/content/ko/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md new file mode 100644 index 0000000000..ca5003a98d --- /dev/null +++ b/content/ko/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md @@ -0,0 +1,53 @@ +--- +title: 리소스 메트릭 파이프라인 +content_template: templates/concept +--- + +{{% capture overview %}} + +쿠버네티스 1.8 부터 컨테이너 CPU 및 메모리 사용량과 같은 리소스 사용량 메트릭은 +쿠버네티스의 Metrics API를 통해 사용할 수 있다. 이 메트릭은 +`kubectl top` 커맨드 사용과 같이 사용자가 직접적으로 액세스하거나, +Horizontal Pod Autoscaler 같은 클러스터의 컨트롤러에서 결정을 내릴 때 사용될 수 있다. + +{{% /capture %}} + + +{{% capture body %}} + +## Metrics API + +Metrics API를 통해 주어진 노드나 파드에서 현재 사용중인 +리소스의 양을 알 수 있다. 이 API는 메트릭 값을 저장하지 +않으므로 지정된 노드에서 10분 전에 사용된 리소스의 양을 +가져오는 것과 같은 일을 할 수는 없다. + +이 API와 다른 API는 차이가 없다. + +- 다른 쿠버네티스 API의 엔드포인트와 같이 `/apis/metrics.k8s.io/` 하위 경로에서 발견될 수 있다 +- 동일한 보안, 확장성 및 신뢰성 보장을 제공한다 + +[k8s.io/metrics](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1beta1/types.go) +리포지터리에서 이 API를 정의하고 있다. 여기에서 이 API에 대한 더 상세한 정보를 찾을 수 있다. + +{{< note >}} +이 API를 사용하려면 Metrics server를 클러스터에 배포해야 한다. 그렇지 않으면 사용할 수 없다. +{{< /note >}} + +## Metrics Server + +[Metrics server](https://github.com/kubernetes-incubator/metrics-server)는 클러스터 전역에서 리소스 사용량 데이터를 집계한다. +쿠버네티스 1.8 부터 `kube-up.sh` 스크립트에 의해 생성된 클러스터에는 기본적으로 Metrics server가 +디플로이먼트 오브젝트로 배포된다. 만약 다른 쿠버네티스 설치 메커니즘을 사용한다면, 제공된 +[배포 yaml들](https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy)을 사용하여 Metrics server를 배포할 수 있다. +이 방식은 쿠버네티스 1.7 이상에서 지원된다. (상세 사항은 아래를 참조) + +Metric server는 각 노드에서 [Kubelet](/docs/admin/kubelet/)에 의해 노출된 Summary API에서 메트릭을 수집한다. + +Metrics server는 쿠버네티스 1.7에서 도입된 +[쿠버네티스 aggregator](/docs/concepts/api-extension/apiserver-aggregation/)를 +통해 메인 API 서버에 등록된다. + +[설계 문서](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md)에서 Metrics server에 대해 자세하게 배울 수 있다. + +{{% /capture %}} diff --git a/content/ko/docs/tasks/debug-application-cluster/resource-usage-monitoring.md b/content/ko/docs/tasks/debug-application-cluster/resource-usage-monitoring.md new file mode 100644 index 0000000000..41e33f80d2 --- /dev/null +++ b/content/ko/docs/tasks/debug-application-cluster/resource-usage-monitoring.md @@ -0,0 +1,116 @@ +--- +content_template: templates/concept +title: 리소스 모니터링 도구 +--- + +{{% capture overview %}} + +애플리케이션을 스케일하여 신뢰할 수 있는 서비스를 제공하려면, +애플리케이션이 배포되었을 때 애플리케이션이 어떻게 동작하는지를 이해해야 한다. +컨테이너, [파드](/ko/docs/concepts/workloads/pods/pod), +[서비스](/docs/concepts/services-networking/service), 그리고 전체 클러스터의 특성을 +검사하여 쿠버네티스 클러스터 내의 애플리케이션 성능을 검사할 수 있다. 쿠버네티스는 각 레벨에서 +애플리케이션의 리소스 사용량에 대한 상세 정보를 제공한다. +이 정보는 애플리케이션의 성능을 평가하고 +병목 현상을 제거하여 전체 성능을 향상할 수 있게 해준다. + +{{% /capture %}} + +{{% capture body %}} + +쿠버네티스에서 애플리케이션 모니터링은 단일 모니터링 솔루션에 의존하지 않는다. +신규 클러스터에서는 기본적으로 두 개의 개별 파이프라인을 사용하여 모니터링 통계를 +수집할 수 있다. + +- [**리소스 메트릭 파이프라인**](#리소스-메트릭-파이프라인)은 HorizontalPodAutoscaler + 컨트롤러와 같은 클러스터 구성요소나 `kubectl top` 유틸리티에 관련되어 있는 메트릭들로 + 제한된 집합을 제공한다. 이 메트릭은 + [metrics-server](https://github.com/kubernetes-incubator/metrics-server) + 에 의해서 수집되며 `metrics.k8s.io` API를 통해 노출된다. `metrics-server`는 클러스터 + 상의 모든 노드를 발견하고 각 노드의 + [Kubelet](/docs/reference/command-line-tools-reference/kubelet)에 CPU와 메모리 + 사용량을 질의한다. Kubelet은 [cAdvisor](https://github.com/google/cadvisor)에서 + 데이터를 가져온다. `metrics-server`는 경량의 단기 인메모리 저장소이다. + +- 프로메테우스 같이 [**완전한 메트릭 파이프라인**](#완전한-메트릭-파이프라인)은 보다 풍부한 + 메트릭에 액세스할 수 있게 해준다. 추가적으로 쿠버네티스는 Horizontal Pod Autoscaler와 + 같은 메커니즘을 사용하여 현재 상태를 기반으로 클러스터를 자동으로 확장 또는 + 조정함으로써 이런 메트릭에 응답할 수 있다. 모니터링 파이프라인은 Kubelet에서 + 메트릭을 가져온 다음 `custom.metrics.k8s.io` 이나 + `external.metrics.k8s.io` API로 구현된 어댑터를 통해 + 이들을 쿠버네티스에 노출한다. + +## 리소스 메트릭 파이프라인 + +### Kubelet + +Kubelet은 쿠버네티스 마스터와 노드들 사이의 다리 역할을 한다. 이는 머신 상에서 실행되는 파드들과 컨테이너들을 관리한다. Kubelet은 각 파드를 이를 구성하는 컨테이너들로 변환하며 컨테이너 런타임 인터페이스를 통해 컨테이너 런타임에서 개별 컨테이너의 사용량 통계를 가져온다. 레거시 도커 통합에서는 cAdvisor에서 이 정보를 가져온다. 그런 다음 Kubelet 리소스 메트릭 API를 통해 집계된 파드 리소스 사용량 통계를 노출한다. 이 API는 Kubelet의 인증되고 읽기 전용의 포트들 상에서 `/metrics/resource/v1alpha1`으로 제공된다. + +### cAdvisor + +cAdvisor는 오픈 소스 컨테이너 자원 사용률/성능 분석 에이전트이다. 이는 컨테이너 전용으로 설계되었으며 도커 컨테이너를 기본적으로 지원한다. 쿠버네티스에서 cAdvisor는 Kubelet 바이너리와 통합된다. cAdvisor는 머신 내 모든 컨테이너를 자동으로 발견하며 CPU, 메모리, 파일시스템, 네트워크 사용량 통계를 수집한다. cAdvisor는 또한 machine 상의 'root' 컨테이너 분석에 의한 전체 머신 사용량도 제공한다. + +Kubelet은 기본 포트 4194를 통해 머신의 컨테이너에 대한 단순한 cAdvisor UI를 노출한다. +아래 그림은 전체 머신의 사용량을 예제로 보여준다. 하지만, 이 기능은 v1.10에서는 사용 중단(deprecated)으로 +표시되었으며, v1.12에서는 완전히 제거되었다. + +![cAdvisor](/images/docs/cadvisor.png) + +v1.13부터, [cAdvisor를 데몬셋으로 배포](https://github.com/google/cadvisor/tree/master/deploy/kubernetes)하여 cAdvisor UI에 액세스할 수 있다. + +## 완전한 메트릭 파이프라인 + +쿠버네티스를 위한 많은 완전한 메트릭 솔루션들이 존재한다. + +### 프로메테우스 + +[프로메테우스](https://prometheus.io)는 기본적으로 쿠버네티스, 노드, 프로메테우스 자체를 모니터링할 수 있다. +[Prometheus Operator](https://coreos.com/operators/prometheus/docs/latest/)는 +쿠버네티스에서 프로메테우스 설정을 단순화하고, +[Prometheus adapter](https://github.com/directxman12/k8s-prometheus-adapter)를 +사용하여 커스텀 메트릭 API를 제공할 수 있게 해준다. +프로메테우스는 강력한 쿼리 언어와 데이터 쿼리와 시각화를 위한 내장 대시보드를 제공한다. +또한 [Grafana](https://prometheus.io/docs/visualization/grafana/)에서는 +데이터 소스로 프로메테우스가 지원된다. + +### Sysdig +[Sysdig](http://sysdig.com)는 완전한 스펙트럼 컨테이너와 플랫폼 인텔리전스를 제공하며, +진정한 컨테이너 네이티브 솔루션이다. Sysdig는 시스템 호출, 쿠버네티스 이벤트, 프로메테우스 메트릭, +statsD, JMX 등의 데이터를 하나의 창으로 통합하여 환경에 대한 포괄적인 그림을 제공한다. +또한 Sysdig는 강력하고 사용자 정의가 가능한 솔루션을 제공하기 위해 쿼리를 실행할 수 있는 API를 제공한다. +Sysdig는 오픈 소스로 만들어졌다. [Sysdig와 Sysdig Inspect](https://sysdig.com/opensource/inspect/)는 +자유롭게 트러블슈팅, 분석, 포렌식을 수행할 수 있는 기능을 제공한다. + +### 구글 클라우드 모니터링 + +구글 클라우드 모니터링은 호스팅 모니터링 서비스로 애플리케이션의 +중요한 메트릭을 시각화하고 경고하는데 사용할 수 있으며, +쿠버네티스에서 메트릭을 수집하고 +[Cloud Monitoring Console](https://app.google.stackdriver.com/)을 +통해 이 메트릭들에 접근할 수 있다. 대시보드를 만들고 사용자 정의하여 쿠버네티스 클러스터에서 +수집한 데이터를 시각화할 수 있다. + +이 동영상은 힙스터(Heapster)를 기반으로 구글 클라우드 모니터링을 구성하고 실행하는 방법을 보여준다. + +[![힙스터를 기반으로 구글 클라우드 모니터링을 구성하고 실행하는 방법](https://img.youtube.com/vi/xSMNR2fcoLs/0.jpg)](https://www.youtube.com/watch?v=xSMNR2fcoLs) + + +{{< figure src="/images/docs/gcm.png" alt="구글 클라우드 모니터링 대시보드 예제" title="구글 클라우드 모니터링 대시보드 예제" caption="대시보드는 클러스터 전역의 리소스 사용량을 보여준다." >}} + +## 크론잡 모니터링 + +### Kubernetes Job Monitor + +[Kubernetes Job Monitor](https://github.com/pietervogelaar/kubernetes-job-monitor) 대시보드를 사용하여 클러스터 관리자는 실행되고 있는 잡들과 완료된 잡의 상태를 볼 수 있다. + +### New Relic 쿠버네티스 모니터링 통합 + +[New Relic 쿠버네티스](https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration) 통합은 쿠버네티스 환경의 성능에 대한 가시성을 향상시킨다. New Relic의 쿠버네티스 통합은 쿠버네티스 오브젝트의 메트릭을 리포팅하는 것으로 컨테이너 오케스트레이션 계층을 측정한다. 통합을 통해 쿠버네티스 노드, 네임스페이스, 디플로이먼트, 레플리카 셋, 파드, 컨테이너에 대한 인사이트를 얻을 수 있다. + +중요 기능: +사전 구축된 대시보드에서 데이터를 확인하여 쿠버네티스 환경에 대한 즉각적인 인사이트를 확인한다. +자동으로 보고되는 데이터의 인사이트로 커스텀 쿼리와 차트를 생성한다. +쿠버네티스 데이터에 대해 경고 조건을 생성한다. +이 [페이지](https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration)에서 더 알아볼 수 있다. + +{{% /capture %}} diff --git a/content/ko/docs/tasks/inject-data-application/_index.md b/content/ko/docs/tasks/inject-data-application/_index.md new file mode 100644 index 0000000000..e7ae5375f4 --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/_index.md @@ -0,0 +1,5 @@ +--- +title: "애플리케이션에 데이터 주입하기" +weight: 30 + +--- \ No newline at end of file diff --git a/content/ko/docs/tasks/inject-data-application/define-environment-variable-container.md b/content/ko/docs/tasks/inject-data-application/define-environment-variable-container.md new file mode 100644 index 0000000000..5cdf78075c --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/define-environment-variable-container.md @@ -0,0 +1,120 @@ +--- +title: 컨테이너를 위한 환경 변수 정의하기 +content_template: templates/task +weight: 20 +--- + +{{% capture overview %}} + +본 페이지는 쿠버네티스 파드의 컨테이너를 위한 환경 변수를 +정의하는 방법에 대해 설명한다. + +{{% /capture %}} + + +{{% capture prerequisites %}} + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + +{{% /capture %}} + + +{{% capture steps %}} + +## 컨테이너를 위한 환경 변수 정의하기 + +파드를 생성할 때, 파드 안에서 동작하는 컨테이너를 위한 환경 변수를 설정할 +수 있다. 환경 변수를 설정하려면, 구성 파일에 `env`나 `envFrom` 필드를 +포함시켜야 한다. + +이 예제에서, 한 개의 컨테이너를 실행하는 파드를 생성한다. 파드를 위한 구성 +파일은 `DEMO_GREETING` 이라는 이름과 `"Hello from the environment"`이라는 +값을 가지는 환경 변수를 정의한다. 다음은 파드를 위한 구성 파일 +예시이다. + +{{< codenew file="pods/inject/envars.yaml" >}} + +1. YAML 구성 파일을 활용해 파드를 생성한다. + + ```shell + kubectl apply -f https://k8s.io/examples/pods/inject/envars.yaml + ``` + +1. 실행 중인 파드들의 목록을 조회한다. + + ```shell + kubectl get pods -l purpose=demonstrate-envars + ``` + + 출력은 아래와 비슷할 것이다. + + ``` + NAME READY STATUS RESTARTS AGE + envar-demo 1/1 Running 0 9s + ``` + +1. 파드 안에 실행되고 있는 컨테이너의 셸에 접근한다. + + ```shell + kubectl exec -it envar-demo -- /bin/bash + ``` + +1. 셸 안에서, 환경 변수를 나열하기 위해 `printenv` 커맨드를 실행한다. + + ```shell + root@envar-demo:/# printenv + ``` + + 출력은 아래와 비슷할 것이다. + + ``` + NODE_VERSION=4.4.2 + EXAMPLE_SERVICE_PORT_8080_TCP_ADDR=10.3.245.237 + HOSTNAME=envar-demo + ... + DEMO_GREETING=Hello from the environment + DEMO_FAREWELL=Such a sweet sorrow + ``` + +1. 셸에서 빠져나오기 위해, `exit`을 입력한다. + +{{< note >}} +`env` 나 `envFrom` 필드를 이용해 설정된 환경 변수들은 컨테이너 이미지 +안에서 명시된 어떠한 환경 변수들보다 더 우선시된다. +{{< /note >}} + +## 설정 안에서 환경 변수 사용하기 + +파드의 구성 파일 안에서 정의한 환경 변수는 파드의 컨테이너를 위해 설정하는 커맨드들과 인자들과 같이, 구성 파일 안의 다른 곳에서 사용할 수 있다. 아래의 구성 파일 예시에서, `GREETING`, `HONORIFIC`, 그리고 `NAME` 환경 변수들이 각각 `Warm greetings to`, `The Most honorable`, 그리고 `Kubernetes`로 설정되어 있다. 이들 환경 변수들은 이후 `env-print-demo` 컨테이너에 전달되어 CLI 인자에서 사용된다. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: print-greeting +spec: + containers: + - name: env-print-demo + image: bash + env: + - name: GREETING + value: "Warm greetings to" + - name: HONORIFIC + value: "The Most Honorable" + - name: NAME + value: "Kubernetes" + command: ["echo"] + args: ["$(GREETING) $(HONORIFIC) $(NAME)"] +``` + +컨테이너가 생성되면, `echo Warm greetings to The Most Honorable Kubernetes` 커맨드가 컨테이너에서 실행된다. + +{{% /capture %}} + +{{% capture whatsnext %}} + +* [환경 변수](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/)에 대해 알아본다. +* [시크릿을 환경 변수로 사용하기](/docs/user-guide/secrets/#using-secrets-as-environment-variables)에 대해 알아본다. +* [EnvVarSource](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvarsource-v1-core)를 확인한다. + +{{% /capture %}} \ No newline at end of file diff --git a/content/ko/docs/tutorials/kubernetes-basics/_index.html b/content/ko/docs/tutorials/kubernetes-basics/_index.html index 405572ef30..9fea785eee 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/_index.html +++ b/content/ko/docs/tutorials/kubernetes-basics/_index.html @@ -42,6 +42,8 @@ card: +
+

쿠버네티스 기초 모듈

diff --git a/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md b/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md index bfe22aa5d6..8adc83eafb 100644 --- a/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md +++ b/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md @@ -6,13 +6,13 @@ weight: 10 --- {{% capture overview %}} -이 튜토리얼은 스테이트풀셋([StatefulSets](/docs/concepts/workloads/controllers/statefulset/))을 이용하여 -애플리케이션을 관리하는 방법을 소개한다. 어떻게 스테이트풀셋의 파드(Pod)을 생성하고 삭제하며 +이 튜토리얼은 스테이트풀셋([StatefulSets](/docs/concepts/workloads/controllers/statefulset/))을 이용하여 +애플리케이션을 관리하는 방법을 소개한다. 어떻게 스테이트풀셋의 파드(Pod)을 생성하고 삭제하며 스케일링하고 업데이트하는지 시연한다. {{% /capture %}} {{% capture prerequisites %}} -튜토리얼을 시작하기 전에 다음의 쿠버네티스 컨셉에 대해 +튜토리얼을 시작하기 전에 다음의 쿠버네티스 컨셉에 대해 익숙해야 한다. * [파드](/docs/user-guide/pods/single-container/) @@ -23,17 +23,17 @@ weight: 10 * [스테이트풀셋](/docs/concepts/workloads/controllers/statefulset/) * [kubectl CLI](/docs/user-guide/kubectl/) -이 튜토리얼은 클러스터가 퍼시스턴스볼륨을 동적으로 프로비저닝 하도록 -설정되었다고 가정한다. 만약 클러스터가 이렇게 설정되어 있지 않다면, -튜토리얼 시작 전에 수동으로 2개의 1 GiB 볼륨을 +이 튜토리얼은 클러스터가 퍼시스턴스볼륨을 동적으로 프로비저닝 하도록 +설정되었다고 가정한다. 만약 클러스터가 이렇게 설정되어 있지 않다면, +튜토리얼 시작 전에 수동으로 2개의 1 GiB 볼륨을 프로비저닝해야 한다. {{% /capture %}} {{% capture objectives %}} -스테이트풀셋은 상태 유지가 필요한(stateful) 애플리케이션과 분산시스템에서 -이용하도록 의도했다. 그러나 쿠버네티스 상에 스테이트풀 애플리케이션과 -분산시스템을 관리하는 것은 광범위하고 복잡한 주제이다. 스테이트풀셋의 기본 기능을 보여주기 위해 -이 둘을 결합하지 않고, 스테이트풀셋을 사용한 +스테이트풀셋은 상태 유지가 필요한(stateful) 애플리케이션과 분산시스템에서 +이용하도록 의도했다. 그러나 쿠버네티스 상에 스테이트풀 애플리케이션과 +분산시스템을 관리하는 것은 광범위하고 복잡한 주제이다. 스테이트풀셋의 기본 기능을 보여주기 위해 +이 둘을 결합하지 않고, 스테이트풀셋을 사용한 단순 웹 애플리케이션을 배포할 것이다. 이 튜토리얼을 마치면 다음 항목에 대해 익숙해질 것이다. @@ -48,26 +48,26 @@ weight: 10 {{% capture lessoncontent %}} ## 스테이트풀셋 생성하기 -아래 예제를 이용해서 스테이트풀셋을 생성하자. 이는 -[스테이트풀셋](/docs/concepts/workloads/controllers/statefulset/) 개념에서 보인 -예제와 유사하다. 이것은 `web`과 이 스테이트풀셋 파드의 IP 주소를 게시하는 -[헤드리스 서비스](/docs/concepts/services-networking/service/#headless-services)인 +아래 예제를 이용해서 스테이트풀셋을 생성하자. 이는 +[스테이트풀셋](/docs/concepts/workloads/controllers/statefulset/) 개념에서 보인 +예제와 유사하다. 이것은 `web`과 이 스테이트풀셋 파드의 IP 주소를 게시하는 +[헤드리스 서비스](/docs/concepts/services-networking/service/#headless-services)인 `nginx` 를 생성한다. {{< codenew file="application/web/web.yaml" >}} 위에 예제를 다운로드 받아서 파일이름을 `web.yaml`으로 저장하자. -2개의 터미널창을 사용한다. 첫째 터미널에서 -[`kubectl get`](/docs/reference/generated/kubectl/kubectl-commands/#get)을 이용해서 +2개의 터미널창을 사용한다. 첫째 터미널에서 +[`kubectl get`](/docs/reference/generated/kubectl/kubectl-commands/#get)을 이용해서 스테이트풀셋의 파드가 생성되는지 감시하자. ```shell kubectl get pods -w -l app=nginx ``` -두번째 터미널에서 -[`kubectl apply`](/docs/reference/generated/kubectl/kubectl-commands/#apply)로 +두번째 터미널에서 +[`kubectl apply`](/docs/reference/generated/kubectl/kubectl-commands/#apply)로 `web.yaml`에 정의된 헤드리스 서비스와 스테이트풀셋을 생성한다. ```shell @@ -76,8 +76,8 @@ service/nginx created statefulset.apps/web created ``` -상기 명령어는 [NGINX](https://www.nginx.com) 웹 서버를 -실행하는 2개의 파드를 생성한다. `nginx` 서비스와 +상기 명령어는 [NGINX](https://www.nginx.com) 웹 서버를 +실행하는 2개의 파드를 생성한다. `nginx` 서비스와 `web` 스테이트풀셋이 성공적으로 생성되었는지 알아보자. ```shell @@ -92,9 +92,9 @@ web 2 1 20s ### 차례대로 파드 생성하기 -N개의 레플리카를 가진 스테이트풀셋은 배포시에 -순차적으로 {0..N-1} 순으로 생성된다. -첫째 터미널에서 `kubectl get` 명령의 출력 내용을 살펴보자. +N개의 레플리카를 가진 스테이트풀셋은 배포시에 +순차적으로 {0..N-1} 순으로 생성된다. +첫째 터미널에서 `kubectl get` 명령의 출력 내용을 살펴보자. 결국 그 내용은 아래 예와 비슷할 것이다. ```shell @@ -110,7 +110,7 @@ web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 18s ``` -`web-1` 파드는 `web-0` 파드가 [Running과 Ready](/docs/user-guide/pod-states) 상태가 되기 전에 +`web-1` 파드는 `web-0` 파드가 [Running과 Ready](/docs/user-guide/pod-states) 상태가 되기 전에 시작하지 않음을 주의하자. ## 스테이트풀셋 안에 파드 @@ -129,17 +129,17 @@ web-1 1/1 Running 0 1m ``` -[스테이트풀셋](/docs/concepts/workloads/controllers/statefulset/) 개념에서 -언급했듯 스테이트풀셋의 파드는 끈끈하고 고유한 정체성을 가진다. -이 정체성은 스테이트풀 컨트롤러에서 각 파드에 주어지는 -고유한 순번에 기인한다. 파드의 이름의 형식은 -`<스테이트풀셋 이름>-<순번>` 이다. 앞서 `web` 스테이트풀셋은 +[스테이트풀셋](/docs/concepts/workloads/controllers/statefulset/) 개념에서 +언급했듯 스테이트풀셋의 파드는 끈끈하고 고유한 정체성을 가진다. +이 정체성은 스테이트풀 컨트롤러에서 각 파드에 주어지는 +고유한 순번에 기인한다. 파드의 이름의 형식은 +`<스테이트풀셋 이름>-<순번>` 이다. 앞서 `web` 스테이트풀셋은 2개의 레플리카를 가졌으므로 `web-0` 과 `web-1` 2개 파드를 생성한다. ### 안정적인 네트워크 신원 사용하기 -각 파드는 각 순번에 따른 안정적인 호스트네임을 갖는다. 각 파드에서 -`hostname` 명령어를 실행하도록 +각 파드는 각 순번에 따른 안정적인 호스트네임을 갖는다. 각 파드에서 +`hostname` 명령어를 실행하도록 [`kubectl exec`](/docs/reference/generated/kubectl/kubectl-commands/#exec)를 이용하자. ```shell @@ -148,13 +148,13 @@ web-0 web-1 ``` -`dnsutils` 패키지에서 `nslookup` 명령을 제공하는 컨테이너를 -실행하도록 [`kubectl run`](/docs/reference/generated/kubectl/kubectl-commands/#run)을 이용하자. -파드의 호스트네임에 `nslookup`을 이용하면 클러스터 내부 DNS 주소를 +`dnsutils` 패키지에서 `nslookup` 명령을 제공하는 컨테이너를 +실행하도록 [`kubectl run`](/docs/reference/generated/kubectl/kubectl-commands/#run)을 이용하자. +파드의 호스트네임에 `nslookup`을 이용하면 클러스터 내부 DNS 주소를 확인할 수 있다. ```shell -kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm +kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm nslookup web-0.nginx Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local @@ -171,7 +171,7 @@ Address 1: 10.244.2.6 ``` 헤드리스 서비스의 CNAME은 SRV 레코드를 지칭한다 -(Running과 Ready 상태의 각 파드마다 1개). +(Running과 Ready 상태의 각 파드마다 1개). SRV 레코드는 파드의 IP 주소를 포함한 A 레코드 엔트리를 지칭한다. 첫째 터미널에서 스테이트풀셋의 파드를 가져오자. @@ -179,8 +179,8 @@ SRV 레코드는 파드의 IP 주소를 포함한 A 레코드 엔트리를 지 ```shell kubectl get pod -w -l app=nginx ``` -두번째 터미널에서 스테이트풀셋 내에 파드를 모두 삭제하기위해 -[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands/#delete)를 +두번째 터미널에서 스테이트풀셋 내에 파드를 모두 삭제하기위해 +[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands/#delete)를 이용하자. ```shell @@ -189,7 +189,7 @@ pod "web-0" deleted pod "web-1" deleted ``` -스테이트풀셋이 재시작되고 두 파드가 Running과 Ready 상태로 +스테이트풀셋이 재시작되고 두 파드가 Running과 Ready 상태로 전환되도록 기다리자. ```shell @@ -204,7 +204,7 @@ web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 34s ``` -파드의 호스트네임과 클러스터 내부 DNS 엔트리를 보기 위해 +파드의 호스트네임과 클러스터 내부 DNS 엔트리를 보기 위해 `kubectl exec`과 `kubectl run`을 이용하자. ```shell @@ -212,7 +212,7 @@ for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname'; done web-0 web-1 -kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm /bin/sh +kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm /bin/sh nslookup web-0.nginx Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local @@ -228,23 +228,23 @@ Name: web-1.nginx Address 1: 10.244.2.8 ``` -파드의 순번, 호스트네임, SRV 레코드와 A 레코드이름은 변경되지 않지만 -파드의 IP 주소는 변경될 수 있다. 이는 튜토리얼에서 사용하는 클러스터나 -다른 클러스터에도 동일하다. 따라서 다른 애플리케이션이 IP 주소로 +파드의 순번, 호스트네임, SRV 레코드와 A 레코드이름은 변경되지 않지만 +파드의 IP 주소는 변경될 수 있다. 이는 튜토리얼에서 사용하는 클러스터나 +다른 클러스터에도 동일하다. 따라서 다른 애플리케이션이 IP 주소로 스테이트풀셋의 파드에 접속하지 않도록 하는 것이 중요하다. -스테이트풀셋의 활성 맴버를 찾아 연결할 경우 -헤드리스 서비스(`nginx.default.svc.cluster.local`)의 CNAME을 쿼리해야 한다. -CNAME과 연관된 SRV 레코드는 스테이트풀셋의 -Running과 Ready 상태의 모든 파드들을 +스테이트풀셋의 활성 맴버를 찾아 연결할 경우 +헤드리스 서비스(`nginx.default.svc.cluster.local`)의 CNAME을 쿼리해야 한다. +CNAME과 연관된 SRV 레코드는 스테이트풀셋의 +Running과 Ready 상태의 모든 파드들을 담고 있다. -애플리케이션에서 이미 활성상태(liveness)와 준비성(readiness) 테스트하는 -연결 로직을 구현되어 있다면 +애플리케이션에서 이미 활성상태(liveness)와 준비성(readiness) 테스트하는 +연결 로직을 구현되어 있다면 파드`web-0.nginx.default.svc.cluster.local`, -`web-1.nginx.default.svc.cluster.local`)의 SRV레코드를 안정적으로 사용할 수 있어 -애플리케이션은 파드가 Running과 Ready 상태로 전환할 때 +`web-1.nginx.default.svc.cluster.local`)의 SRV레코드를 안정적으로 사용할 수 있어 +애플리케이션은 파드가 Running과 Ready 상태로 전환할 때 파드의 주소를 검색할 수 있다. ### 안정적인 스토리지에 쓰기 {#writing-to-stable-storage} @@ -257,16 +257,16 @@ NAME STATUS VOLUME CAPACITY ACCE www-web-0 Bound pvc-15c268c7-b507-11e6-932f-42010a800002 1Gi RWO 48s www-web-1 Bound pvc-15c79307-b507-11e6-932f-42010a800002 1Gi RWO 48s ``` -스테이트풀셋 컨트롤러는 2개의 [퍼시스턴트볼륨](/docs/concepts/storage/persistent-volumes/)에 -묶인 2개의 퍼시스턴트볼륨클레임을 생성했다. 본 튜토리얼에서 사용되는 클러스터는 퍼시스턴트볼륨을 동적으로 +스테이트풀셋 컨트롤러는 2개의 [퍼시스턴트볼륨](/docs/concepts/storage/persistent-volumes/)에 +묶인 2개의 퍼시스턴트볼륨클레임을 생성했다. 본 튜토리얼에서 사용되는 클러스터는 퍼시스턴트볼륨을 동적으로 프로비저닝하도록 설정되었으므로 생성된 퍼시스턴트볼륨도 자동으로 묶인다. -NGINX 웹서버는 기본 색인 파일로 -`/usr/share/nginx/html/index.html`을 이용합니다. -스테이트풀셋 `spec`내의 `volumeMounts` 필드는 `/usr/share/nginx/html` 디렉터리가 +NGINX 웹서버는 기본 색인 파일로 +`/usr/share/nginx/html/index.html`을 이용합니다. +스테이트풀셋 `spec`내의 `volumeMounts` 필드는 `/usr/share/nginx/html` 디렉터리가 퍼시스턴트볼륨으로 제공되는지 보증합니다. -파드의 호스트네임을 `index.html` 파일에 작성하고 +파드의 호스트네임을 `index.html` 파일에 작성하고 NGINX 웹서버가 해당 호스트네임을 제공하는지 확인해보자. ```shell @@ -278,7 +278,7 @@ web-1 ``` {{< note >}} -위에 curl 명령어로 403 Forbidden 아닌 응답을 보려면 +위에 curl 명령어로 403 Forbidden 아닌 응답을 보려면 `volumeMounts`로 마운트된 디렉터리의 퍼미션을 수정해야 한다 ([hostPath 볼륨을 사용할 때에 버그](https://github.com/kubernetes/kubernetes/issues/2630)로 인함). @@ -302,7 +302,7 @@ kubectl delete pod -l app=nginx pod "web-0" deleted pod "web-1" deleted ``` -첫번째 터미널에서 실행 중인 `kubectl get`명령어의 출력을 확인하고, +첫번째 터미널에서 실행 중인 `kubectl get`명령어의 출력을 확인하고, 모든 파드가 Running과 Ready 상태로 전환될때까지 기다리자. ```shell @@ -325,16 +325,16 @@ web-0 web-1 ``` -비록 `web-0`과 `web-1`이 재스케줄링되어도 계속해서 -자신의 호스트네임을 제공하는데 이는 각 퍼시스턴트볼륨클레임에 -연관된 퍼시스턴트볼륨이 해당 `volumeMounts`로 재마운트되기 때문이다. -`web-0`과 `web-1`의 스케줄링에 관계없이 +비록 `web-0`과 `web-1`이 재스케줄링되어도 계속해서 +자신의 호스트네임을 제공하는데 이는 각 퍼시스턴트볼륨클레임에 +연관된 퍼시스턴트볼륨이 해당 `volumeMounts`로 재마운트되기 때문이다. +`web-0`과 `web-1`의 스케줄링에 관계없이 각각의 퍼시스턴트볼륨은 적절하게 마운트된다. ## 스테이트풀셋 스케일링 -스테이트풀셋을 스케일링하는 것은 레플리카 개수를 늘리거나 줄이는 것을 의미한다. 이것은 `replicas` 필드를 갱신하여 이뤄진다. -[`kubectl scale`](/docs/reference/generated/kubectl/kubectl-commands/#scale)이나 -[`kubectl patch`](/docs/reference/generated/kubectl/kubectl-commands/#patch)을 +스테이트풀셋을 스케일링하는 것은 레플리카 개수를 늘리거나 줄이는 것을 의미한다. 이것은 `replicas` 필드를 갱신하여 이뤄진다. +[`kubectl scale`](/docs/reference/generated/kubectl/kubectl-commands/#scale)이나 +[`kubectl patch`](/docs/reference/generated/kubectl/kubectl-commands/#patch)을 이용해서 스테이트풀셋을 스케일링할 수 있다. ### 스케일 업 @@ -345,7 +345,7 @@ web-1 kubectl get pods -w -l app=nginx ``` -다른 터미널창에서 `kubectl scale`을 이용하여 레플리카 개수를 +다른 터미널창에서 `kubectl scale`을 이용하여 레플리카 개수를 5로 스케일링하자. ```shell @@ -353,7 +353,7 @@ kubectl scale sts web --replicas=5 statefulset.apps/web scaled ``` -첫번째 터미널에서 실행 중인 `kubectl get`명령어의 출력을 확인하고, +첫번째 터미널에서 실행 중인 `kubectl get`명령어의 출력을 확인하고, 3개의 추가 파드가 Running과 Ready 상태로 전환될때까지 기다리자. ```shell @@ -376,10 +376,10 @@ web-4 0/1 ContainerCreating 0 0s web-4 1/1 Running 0 19s ``` -스테이트풀셋 컨트롤러는 레플리카개수를 스케일링한다. -[스테이트풀셋 생성](#ordered-pod-creation)으로 스테이트풀셋 컨트롤러는 -각 파드을 순차적으로 각 순번에 따라 생성하고 후속 파드 시작 전에 -이전 파드가 Running과 Ready 상태가 될때까지 +스테이트풀셋 컨트롤러는 레플리카개수를 스케일링한다. +[스테이트풀셋 생성](#ordered-pod-creation)으로 스테이트풀셋 컨트롤러는 +각 파드을 순차적으로 각 순번에 따라 생성하고 후속 파드 시작 전에 +이전 파드가 Running과 Ready 상태가 될때까지 기다린다. ### 스케일 다운 {#scaling-down} @@ -418,8 +418,8 @@ web-3 1/1 Terminating 0 42s ### 순차 파드 종료 -컨트롤러는 순번의 역순으로 한번에 1개 파드를 삭제하고 -다음 파드를 삭제하기 전에 +컨트롤러는 순번의 역순으로 한번에 1개 파드를 삭제하고 +다음 파드를 삭제하기 전에 각각이 완전하게 종료되기까지 기다린다. 스테이트풀셋의 퍼시스턴트볼륨클레임을 가져오자. @@ -435,23 +435,23 @@ www-web-4 Bound pvc-e11bb5f8-b508-11e6-932f-42010a800002 1Gi RWO ``` -여전히 5개의 퍼시스턴트볼륨클레임과 5개의 퍼시스턴트볼륨이 있다. +여전히 5개의 퍼시스턴트볼륨클레임과 5개의 퍼시스턴트볼륨이 있다. 파드의 [안전한 스토리지](#writing-to-stable-storage)를 탐색하면서 스테이트풀셋의 파드가 삭제될 때에 파드에 마운트된 스테이트풀셋의 퍼시스턴트볼륨이 삭제되지 않은 것을 보았다. 스테이트풀셋 스케일 다운으로 파드 삭제할 때에도 여전히 사실이다. ## 스테이트풀셋 업데이트하기 -쿠버네티스 1.7 이상에서 스테이트풀셋 컨트롤러는 자동 업데이트를 지원한다. -전략은 스테이트풀셋 API 오브젝트의 `spec.updateStrategy` 필드로 결정된다. -이 기능은 컨테이너 이미지, 스테이트풀셋의 리소스 요청이나 -혹은 한계와 레이블과 파드의 어노테이션을 업그레이드하기 위해 사용될 수 있다. -`RollingUpdate`과 `OnDelete`의 2개의 +쿠버네티스 1.7 이상에서 스테이트풀셋 컨트롤러는 자동 업데이트를 지원한다. +전략은 스테이트풀셋 API 오브젝트의 `spec.updateStrategy` 필드로 결정된다. +이 기능은 컨테이너 이미지, 스테이트풀셋의 리소스 요청이나 +혹은 한계와 레이블과 파드의 어노테이션을 업그레이드하기 위해 사용될 수 있다. +`RollingUpdate`과 `OnDelete`의 2개의 유효한 업데이트 전략이 있다. `RollingUpdate` 업데이트 전략은 스테이트풀셋에서 기본 값이다. ### 롤링 업데이트 -`RollingUpdate` 업데이트 전략은 스테이트풀셋을 보장하면서 스테이트풀셋 내에 파드를 역순으로 업데이트합니다. +`RollingUpdate` 업데이트 전략은 스테이트풀셋을 보장하면서 스테이트풀셋 내에 파드를 역순으로 업데이트합니다. 스테이트풀셋 `web`의 업데이트 전략을 `RollingUpdate`으로 패치하자. @@ -460,7 +460,7 @@ kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"RollingUpda statefulset.apps/web patched ``` -터미널 창에서 스테이트풀셋 `web`의 컨테이너 이미지를 바꾸도록 +터미널 창에서 스테이트풀셋 `web`의 컨테이너 이미지를 바꾸도록 또 패치하자. ```shell @@ -506,15 +506,15 @@ web-0 0/1 ContainerCreating 0 0s web-0 1/1 Running 0 10s ``` -스테이트풀셋 내에 파드는 순번의 역순으로 업데이트된다. -이 스테이트풀셋 컨트롤러는 각 파드를 종료시키고 다음 파드를 업데이트하기 전에 -그것이 Running과 Ready 상태로 전환될때까지 기다린다. -알아둘 것은 비록 스테이트풀셋 컨트롤러에서 이전 파드가 Running과 Ready 상태가 되기까지 -다음 파드를 업데이트하지 않아도 현재 버전으로 파드를 업데이트하다 실패하면 복원한다는 것이다. -업데이트를 이미 받은 파드는 업데이트된 버전으로 복원되고 아직 업데이트를 받지 못한 파드는 -이전 버전으로 복원한다. -이런 식으로 컨트롤러는 간헐적인 오류가 발생해도 -애플리케이션을 계속 건강하게 유지하고 +스테이트풀셋 내에 파드는 순번의 역순으로 업데이트된다. +이 스테이트풀셋 컨트롤러는 각 파드를 종료시키고 다음 파드를 업데이트하기 전에 +그것이 Running과 Ready 상태로 전환될때까지 기다린다. +알아둘 것은 비록 스테이트풀셋 컨트롤러에서 이전 파드가 Running과 Ready 상태가 되기까지 +다음 파드를 업데이트하지 않아도 현재 버전으로 파드를 업데이트하다 실패하면 복원한다는 것이다. +업데이트를 이미 받은 파드는 업데이트된 버전으로 복원되고 아직 업데이트를 받지 못한 파드는 +이전 버전으로 복원한다. +이런 식으로 컨트롤러는 간헐적인 오류가 발생해도 +애플리케이션을 계속 건강하게 유지하고 업데이트도 일관되게 유지하려 한다. 컨테이너 이미지를 살펴보기 위해 파드를 가져오자. @@ -529,13 +529,13 @@ k8s.gcr.io/nginx-slim:0.8 스테이트풀셋의 모든 파드가 지금은 이전 컨테이너 이미지를 실행 중이이다. -**팁** 롤링 업데이트 상황을 살펴보기 위해 `kubectl rollout status sts/` +**팁** 롤링 업데이트 상황을 살펴보기 위해 `kubectl rollout status sts/` 명령어도 사용할 수 있다. #### 단계적으로 업데이트 하기 {#staging-an-update} `RollingUpdate` 업데이트 전략의 파라미터인 `partition`를 이용하여 스테이트풀셋의 단계적으로 업데이트할 수 있다. -단계적 업데이트는 스테이트풀셋의 모든 파드를 현재 버전으로 유지하면서 +단계적 업데이트는 스테이트풀셋의 모든 파드를 현재 버전으로 유지하면서 스테이트풀셋의 `.spec.template`에 변경을 허용한다. 스테이트풀셋 `web`의 `updateStrategy` 필드에 partition을 추가하자. @@ -579,12 +579,12 @@ k8s.gcr.io/nginx-slim:0.8 ``` 비록 업데이트 전략이 `RollingUpdate`이지만 스테이트풀셋은 -파드를 그것의 원래 컨테이너로 복원한다. -파드의 순번이 `updateStrategy`에서 지정된 +파드를 그것의 원래 컨테이너로 복원한다. +파드의 순번이 `updateStrategy`에서 지정된 `파티션`보다 작기 때문이다. #### 카나리(Canary) 롤링 아웃 -[위에서](#staging-an-update) 지정한 `partition`값을 차감시키면 +[위에서](#staging-an-update) 지정한 `partition`값을 차감시키면 변경사항을 테스트하기 위해 카나리 롤아웃을 할 수 있다. 스테이트풀셋에 partition을 차감하도록 패치하자. @@ -614,7 +614,7 @@ k8s.gcr.io/nginx-slim:0.7 ``` `partition`을 바꾸면 스테이트풀셋 컨트롤러는 자동으로 -`web-2` 파드를 업데이트하는데 +`web-2` 파드를 업데이트하는데 이는 해당 파드의 순번이 `partition` 이상이기 때문이다. `web-1` 파드를 삭제하자. @@ -649,17 +649,17 @@ k8s.gcr.io/nginx-slim:0.8 ``` -`web-1` 는 원래 환경설정으로 복원되었는데 -이는 파드의 순번이 partition보다 작기 때문이다. -스테이트풀셋의 `.spec.template`이 갱신되면, 지정된 partition 이상의 순번을 -가진 모든 파드는 업데이트된다. 미만의 순번을 가진 파드라면 삭제되거나 +`web-1` 는 원래 환경설정으로 복원되었는데 +이는 파드의 순번이 partition보다 작기 때문이다. +스테이트풀셋의 `.spec.template`이 갱신되면, 지정된 partition 이상의 순번을 +가진 모든 파드는 업데이트된다. 미만의 순번을 가진 파드라면 삭제되거나 종료되어 원래 환경설정으로 복원된다. #### 단계적 롤아웃 -[카나리 롤아웃](#rolling-out-a-canary)에서 했던 방법과 비슷하게 -분할된 롤링 업데이트를 이용하여 단계적 롤아웃(e.g. 선형, 기하 또는 지수적 롤아웃)을 -수행할 수 있다. 단계적 롤아웃을 수행하려면 -컨트롤러가 업데이트를 일시 중지할 순번으로 +[카나리 롤아웃](#rolling-out-a-canary)에서 했던 방법과 비슷하게 +분할된 롤링 업데이트를 이용하여 단계적 롤아웃(e.g. 선형, 기하 또는 지수적 롤아웃)을 +수행할 수 있다. 단계적 롤아웃을 수행하려면 +컨트롤러가 업데이트를 일시 중지할 순번으로 `partition`를 정하자. partition은 현재 `2`이다. partition을 `0`으로 바꾸자. @@ -700,20 +700,20 @@ k8s.gcr.io/nginx-slim:0.7 ``` -`partition`을 `0`으로 이동하여 스테이트풀셋 컨트롤러에서 계속해서 +`partition`을 `0`으로 이동하여 스테이트풀셋 컨트롤러에서 계속해서 업데이트 처리를 하도록 허용하였다. ### 삭제시 동작 `OnDelete` 업데이트 전략은 예전 동작(1.6 이하)으로, 이 업데이트 전략을 선택하면 스테이트풀셋 컨트롤러는 스테이트풀셋의 -`.spec.template` 필드에 수정 사항이 발생해도 자동으로 파드를 업데이트하지 않는다. +`.spec.template` 필드에 수정 사항이 발생해도 자동으로 파드를 업데이트하지 않는다. 이 전략은 `.spec.template.updateStrategy.type`을 `OnDelete`로 설정하여 선택할 수 있다. ## 스테이트풀셋 삭제하기 -스테이트풀셋은 비종속적(non-cascading), 종속적(cascading) 삭제를 둘 다 지원한다. -비종속적 삭제에서는 스테이트풀셋이 지워질 때에 스테이트풀셋의 파드는 지워지지 않는다. +스테이트풀셋은 비종속적(non-cascading), 종속적(cascading) 삭제를 둘 다 지원한다. +비종속적 삭제에서는 스테이트풀셋이 지워질 때에 스테이트풀셋의 파드는 지워지지 않는다. 종속적 삭제에서는 스테이트풀셋과 그에 속한 파드가 모두 지워진다. ### 비종속적 삭제 @@ -724,9 +724,9 @@ k8s.gcr.io/nginx-slim:0.7 kubectl get pods -w -l app=nginx ``` -다른 터미널에서는 스테이트풀셋을 지우기 위해 -[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands/#delete) 명령어를 이용하자. -이 명령어에 `--cascade=false` 파라미터가 추가되었다. +다른 터미널에서는 스테이트풀셋을 지우기 위해 +[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands/#delete) 명령어를 이용하자. +이 명령어에 `--cascade=false` 파라미터가 추가되었다. 이 파라미터는 쿠버네티스에 스테이트풀셋만 삭제하고 그에 속한 파드는 지우지 않도록 요청한다. ```shell @@ -744,7 +744,7 @@ web-1 1/1 Running 0 7m web-2 1/1 Running 0 5m ``` -비록 `web`이 삭제되고 있어도, 모든 파드는 여전히 Running과 Ready 상태이다. +비록 `web`이 삭제되고 있어도, 모든 파드는 여전히 Running과 Ready 상태이다. `web-0`을 삭제하자. ```shell @@ -769,17 +769,17 @@ web-2 1/1 Running 0 7m kubectl get pods -w -l app=nginx ``` -두번째 터미널에서 스테이트풀셋을 다시 생성하자. -`nginx` 서비스(가지지 말았어야 하는)를 삭제하기 전까지는 그 서비스가 이미 존재한다는 에러를 +두번째 터미널에서 스테이트풀셋을 다시 생성하자. +`nginx` 서비스(가지지 말았어야 하는)를 삭제하기 전까지는 그 서비스가 이미 존재한다는 에러를 볼 것이라는 것을 명심하자. ```shell kubectl apply -f web.yaml statefulset.apps/web created -Error from server (AlreadyExists): error when creating "web.yaml": services "nginx" already exists +service/nginx unchanged ``` -이 에러는 무시하자. 이것은 다만 해당 서비스가 있더라도 +이 에러는 무시하자. 이것은 다만 해당 서비스가 있더라도 nginx 헤드리스 서비스를 생성하려고 했음을 뜻한다. 첫째 터미널에서 실행 중인 `kubectl get` 명령어의 출력을 살펴보자. @@ -800,14 +800,14 @@ web-2 0/1 Terminating 0 3m web-2 0/1 Terminating 0 3m ``` -`web` 스테이트풀셋이 다시 생성될때 먼저 `web-0` 시작한다. -`web-1`은 이미 Running과 Ready 상태이므로 `web-0`이 Running과 Ready 상태로 -전환될 때는 단순히 이 파드에 적용됬다. 스테이트풀셋에`replicas`를 2로 하고 -`web-0`을 재생성했다면 `web-1`이 -이미 Running과 Ready 상태이고, +`web` 스테이트풀셋이 다시 생성될때 먼저 `web-0` 시작한다. +`web-1`은 이미 Running과 Ready 상태이므로 `web-0`이 Running과 Ready 상태로 +전환될 때는 단순히 이 파드에 적용됬다. 스테이트풀셋에`replicas`를 2로 하고 +`web-0`을 재생성했다면 `web-1`이 +이미 Running과 Ready 상태이고, `web-2`은 종료되었을 것이다. -파드의 웹서버에서 제공한 `index.html` 파일 내용을 +파드의 웹서버에서 제공한 `index.html` 파일 내용을 다른 관점으로 살펴보자. ```shell @@ -816,10 +816,10 @@ web-0 web-1 ``` -스테이트풀셋과 `web-0` 파드를 둘다 삭제했으나 여전히 `index.html` 파일에 입력했던 -원래 호스트네임을 제공한다. 스테이트풀셋은 -파드에 할당된 퍼시스턴트볼륨을 결코 삭제하지 않기때문이다. -다시 스테이트풀셋을 생성하면 `web-0`을 시작하며 +스테이트풀셋과 `web-0` 파드를 둘다 삭제했으나 여전히 `index.html` 파일에 입력했던 +원래 호스트네임을 제공한다. 스테이트풀셋은 +파드에 할당된 퍼시스턴트볼륨을 결코 삭제하지 않기때문이다. +다시 스테이트풀셋을 생성하면 `web-0`을 시작하며 원래 퍼시스턴트볼륨을 다시 마운트한다. ### 단계식 삭제 @@ -830,14 +830,14 @@ web-1 kubectl get pods -w -l app=nginx ``` -다른 터미널창에서 스테이트풀셋을 다시 지우자. 이번에는 +다른 터미널창에서 스테이트풀셋을 다시 지우자. 이번에는 `--cascade=false` 파라미터를 생략하자. ```shell kubectl delete statefulset web statefulset.apps "web" deleted ``` -첫째 터미널에서 실행 중인 `kubectl get` 명령어의 출력을 살펴보고 +첫째 터미널에서 실행 중인 `kubectl get` 명령어의 출력을 살펴보고 모든 파드가 Terminating 상태로 전환될때까지 기다리자. ```shell @@ -857,13 +857,13 @@ web-1 0/1 Terminating 0 29m ``` -[스케일 다운](#scaling-down) 섹션에서 보았듯 파드는 -각 순번의 역순으로 하나씩 종료된다. 파드가 종료될 때 -스테이트풀 컨트롤러는 이전 파드가 +[스케일 다운](#scaling-down) 섹션에서 보았듯 파드는 +각 순번의 역순으로 하나씩 종료된다. 파드가 종료될 때 +스테이트풀 컨트롤러는 이전 파드가 완전히 종료되기까지 기다린다. -스테이트풀셋과 그 파드를 종속적으로 삭제하는 중에 연관된 헤드리스 서비스를 -삭제하지 않음을 주의하자. +스테이트풀셋과 그 파드를 종속적으로 삭제하는 중에 연관된 헤드리스 서비스를 +삭제하지 않음을 주의하자. 꼭 `nginx` 서비스를 수동으로 삭제해라. ```shell @@ -879,7 +879,7 @@ service/nginx created statefulset.apps/web created ``` -스테이트풀셋의 모든 파드가 Running과 Ready 상태로 전환될 때 +스테이트풀셋의 모든 파드가 Running과 Ready 상태로 전환될 때 `index.html` 파일 내용을 검색하자. ```shell @@ -888,8 +888,8 @@ web-0 web-1 ``` -스테이트풀셋과 그 내부의 모든 파드를 삭제했지만 퍼시스턴트볼륨이 마운트된 채로 -다시 생성되고 `web-0`과 `web-1`은 여전히 +스테이트풀셋과 그 내부의 모든 파드를 삭제했지만 퍼시스턴트볼륨이 마운트된 채로 +다시 생성되고 `web-0`과 `web-1`은 여전히 각 호스트네임을 제공한다. 최종적으로 `web` 스테이트풀셋과`nginx` 서비스를 삭제한다. @@ -904,29 +904,29 @@ statefulset "web" deleted ## 파드 관리 정책 -일부 분산 시스템의 경우 스테이트풀셋의 순서 보증은 -불필요하거나 바람직하지 않다. 이러한 시스템은 고유성과 신원만 필요하다. -이를 해결하기 위해 쿠버네티스 1.7에서 `.spec.podManagementPolicy`를 +일부 분산 시스템의 경우 스테이트풀셋의 순서 보증은 +불필요하거나 바람직하지 않다. 이러한 시스템은 고유성과 신원만 필요하다. +이를 해결하기 위해 쿠버네티스 1.7에서 `.spec.podManagementPolicy`를 스테이트풀셋 API 오브젝트에 도입했다. ### OrderedReady 파드 관리 -`OrderedReady` 파드 관리는 스테이트풀셋에서는 기본이다. -이는 스테이트풀셋 컨트롤러가 지금까지 위에서 설명했던 순서를 +`OrderedReady` 파드 관리는 스테이트풀셋에서는 기본이다. +이는 스테이트풀셋 컨트롤러가 지금까지 위에서 설명했던 순서를 보증함을 뜻한다. ### Parallel 파드 관리 -`Parallel` 파드 관리는 스테이트풀셋 컨트롤러가 모든 파드를 -병렬로 시작하고 종료하는 것으로 다른 파드를 시작/종료하기 전에 -파드가 Running과 Ready 상태로 전환되거나 완전히 종료되기까지 +`Parallel` 파드 관리는 스테이트풀셋 컨트롤러가 모든 파드를 +병렬로 시작하고 종료하는 것으로 다른 파드를 시작/종료하기 전에 +파드가 Running과 Ready 상태로 전환되거나 완전히 종료되기까지 기다리지 않음을 뜻한다. {{< codenew file="application/web/web-parallel.yaml" >}} 상기 예제를 다운로드받아 파일 이름을 `web-parallel.yaml`로 저장하자. -이 매니페스트는 `web` 스테이트풀셋의 `.spec.podManagementPolicy`이 +이 매니페스트는 `web` 스테이트풀셋의 `.spec.podManagementPolicy`이 `Parallel`인 것 말고는 이전에 다운로드 받았던 것과 동일하다. 터미널에서 스테이트풀셋의 파드를 감시하자. @@ -960,7 +960,7 @@ web-1 1/1 Running 0 10s 스테이트풀셋 컨트롤러는 `web-0`와 `web-1`를 둘다 동시에 시작했다. -두번째 터미널을 열어 놓고 다른 터미널창에서 스테이트풀셋을 +두번째 터미널을 열어 놓고 다른 터미널창에서 스테이트풀셋을 스케일링 하자. ```shell @@ -980,7 +980,7 @@ web-3 1/1 Running 0 26s ``` -스테이트풀 컨트롤러는 두개의 새 파드를 시작하였다. +스테이트풀 컨트롤러는 두개의 새 파드를 시작하였다. 두번째 것을 런칭하기 위해 먼저 런칭한 것이 Running과 Ready 상태가 될 떄까지 기다리지 않는다. 이 터미널을 열어 놓고 다른 터미널에서 `web` 스테이트풀셋을 삭제하자. @@ -1017,10 +1017,10 @@ web-3 0/1 Terminating 0 9m web-3 0/1 Terminating 0 9m ``` -스테이트풀 컨트롤러는 모든 파드를 동시에 삭제한다. 파드를 삭제하기 전에 +스테이트풀 컨트롤러는 모든 파드를 동시에 삭제한다. 파드를 삭제하기 전에 그 파드의 순서상 후계자를 기다리지 않는다. -`kubectl get` 명령어가 실행된 터미널을 닫고 +`kubectl get` 명령어가 실행된 터미널을 닫고 `nginx` 서비스를 삭제하자. ```shell @@ -1029,9 +1029,9 @@ kubectl delete svc nginx {{% /capture %}} {{% capture cleanup %}} -이 튜토리얼에서 사용된 퍼시턴트볼륨을 위한 +이 튜토리얼에서 사용된 퍼시턴트볼륨을 위한 퍼시스턴트 스토리지 미디어를 삭제해야 한다. -모든 스토리지를 반환하도록 환경, 스토리지 설정과 +모든 스토리지를 반환하도록 환경, 스토리지 설정과 프로비저닝 방법에 따른 단계를 따르자. {{% /capture %}} diff --git a/content/ko/examples/pods/inject/envars.yaml b/content/ko/examples/pods/inject/envars.yaml new file mode 100644 index 0000000000..ebf5214376 --- /dev/null +++ b/content/ko/examples/pods/inject/envars.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Pod +metadata: + name: envar-demo + labels: + purpose: demonstrate-envars +spec: + containers: + - name: envar-demo-container + image: gcr.io/google-samples/node-hello:1.0 + env: + - name: DEMO_GREETING + value: "Hello from the environment" + - name: DEMO_FAREWELL + value: "Such a sweet sorrow" From 1830e56fa22e152fc863109e8d1062680e025a66 Mon Sep 17 00:00:00 2001 From: Jordan Liggitt Date: Thu, 8 Aug 2019 20:19:27 -0400 Subject: [PATCH 39/49] Clarify PSP/PV interaction (#15756) --- content/en/docs/concepts/policy/pod-security-policy.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/en/docs/concepts/policy/pod-security-policy.md b/content/en/docs/concepts/policy/pod-security-policy.md index 086d5c9d5d..8c22d902f2 100644 --- a/content/en/docs/concepts/policy/pod-security-policy.md +++ b/content/en/docs/concepts/policy/pod-security-policy.md @@ -420,6 +420,11 @@ The **recommended minimum set** of allowed volumes for new PSPs are: - secret - projected +{{< warning >}} +PodSecurityPolicy does not limit the types of `PersistentVolume` objects that may be referenced by a `PersistentVolumeClaim`. +Only trusted users should be granted permission to create `PersistentVolume` objects. +{{< /warning >}} + **FSGroup** - Controls the supplemental group applied to some volumes. - *MustRunAs* - Requires at least one `range` to be specified. Uses the From ad4bc585260cd01c3bd836055f1df10c0b96d4a5 Mon Sep 17 00:00:00 2001 From: Karen Bradshaw Date: Thu, 8 Aug 2019 20:21:25 -0400 Subject: [PATCH 40/49] add weight to static-pod, fix date mirror-pod (#15752) --- content/en/docs/reference/glossary/mirror-pod.md | 3 +-- content/en/docs/reference/glossary/static-pod.md | 2 +- content/en/docs/tasks/configure-pod-container/static-pod.md | 1 + .../configure-pod-container/translate-compose-kubernetes.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/reference/glossary/mirror-pod.md b/content/en/docs/reference/glossary/mirror-pod.md index 9e8f81479e..c925e4d672 100755 --- a/content/en/docs/reference/glossary/mirror-pod.md +++ b/content/en/docs/reference/glossary/mirror-pod.md @@ -1,8 +1,7 @@ --- title: Mirror Pod id: mirror-pod -date: 2091-02-12 -full_link: +date: 2019-08-06 short_description: > An object in the API server that tracks a static pod on a kubelet. diff --git a/content/en/docs/reference/glossary/static-pod.md b/content/en/docs/reference/glossary/static-pod.md index 89ccdded1f..b633aee0f2 100755 --- a/content/en/docs/reference/glossary/static-pod.md +++ b/content/en/docs/reference/glossary/static-pod.md @@ -1,7 +1,7 @@ --- title: Static Pod id: static-pod -date: 2091-02-12 +date: 2019-02-12 full_link: /docs/tasks/administer-cluster/static-pod/ short_description: > A pod managed directly by kubelet daemon on a specific node diff --git a/content/en/docs/tasks/configure-pod-container/static-pod.md b/content/en/docs/tasks/configure-pod-container/static-pod.md index 4225a7a586..567c6f199a 100644 --- a/content/en/docs/tasks/configure-pod-container/static-pod.md +++ b/content/en/docs/tasks/configure-pod-container/static-pod.md @@ -2,6 +2,7 @@ reviewers: - jsafrane title: Create static Pods +weight: 170 content_template: templates/task --- diff --git a/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md b/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md index 5fb479db8a..c5b714d770 100644 --- a/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md +++ b/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md @@ -3,7 +3,7 @@ reviewers: - cdrage title: Translate a Docker Compose File to Kubernetes Resources content_template: templates/task -weight: 170 +weight: 200 --- {{% capture overview %}} From a306ab2e3506bd15b27f729f5d45f6d6d87996d7 Mon Sep 17 00:00:00 2001 From: Ismail Alidzhikov Date: Fri, 9 Aug 2019 08:21:24 +0300 Subject: [PATCH 41/49] Remove kubelet --allow-privileged flag for etcd HA setup with kubeadm (#15332) Signed-off-by: ialidzhikov --- content/fr/docs/setup/independent/setup-ha-etcd-with-kubeadm.md | 2 +- content/ja/docs/setup/independent/setup-ha-etcd-with-kubeadm.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/fr/docs/setup/independent/setup-ha-etcd-with-kubeadm.md b/content/fr/docs/setup/independent/setup-ha-etcd-with-kubeadm.md index 047f9e53d9..684cce95f6 100644 --- a/content/fr/docs/setup/independent/setup-ha-etcd-with-kubeadm.md +++ b/content/fr/docs/setup/independent/setup-ha-etcd-with-kubeadm.md @@ -49,7 +49,7 @@ kubeadm contient tout ce qui est nécessaire pour générer les certificats déc cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf [Service] ExecStart= - ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true + ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests Restart=always EOF diff --git a/content/ja/docs/setup/independent/setup-ha-etcd-with-kubeadm.md b/content/ja/docs/setup/independent/setup-ha-etcd-with-kubeadm.md index 30bc58ff23..a8abb62930 100644 --- a/content/ja/docs/setup/independent/setup-ha-etcd-with-kubeadm.md +++ b/content/ja/docs/setup/independent/setup-ha-etcd-with-kubeadm.md @@ -51,7 +51,7 @@ this example. cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf [Service] ExecStart= - ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true + ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests Restart=always EOF From d4741f1bc41c6cdc7d80cd7e6a5ad12f739c1dea Mon Sep 17 00:00:00 2001 From: Carlos Date: Fri, 9 Aug 2019 21:59:13 +0200 Subject: [PATCH 42/49] Add content/es/docs/reference/glossary/pod-priority (#15325) * ammend signature * Update content/es/docs/reference/glossary/pod-priority.md Co-Authored-By: Rael Garcia * Update content/es/docs/reference/glossary/pod-priority.md Co-Authored-By: Rael Garcia * Update content/es/docs/reference/glossary/pod-priority.md Co-Authored-By: Rael Garcia --- .../es/docs/reference/glossary/pod-priority.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 content/es/docs/reference/glossary/pod-priority.md diff --git a/content/es/docs/reference/glossary/pod-priority.md b/content/es/docs/reference/glossary/pod-priority.md new file mode 100644 index 0000000000..f8126ae2e8 --- /dev/null +++ b/content/es/docs/reference/glossary/pod-priority.md @@ -0,0 +1,17 @@ +--- +title: Pod Priority +id: pod-priority +date: 2019-01-31 +full_link: /docs/concepts/configuration/pod-priority-preemption/#pod-priority +short_description: > + Pod Priority indica la importancia de un {{< glossary_tooltip text="Pod" term_id="pod" >}} con relación a otros {{< glossary_tooltip text="Pods" term_id="pod" >}}. + +aka: +tags: +- operation +--- + Pod Priority indica la importancia de un {{< glossary_tooltip text="Pod" term_id="pod" >}} con relación a otros {{< glossary_tooltip text="Pods" term_id="pod" >}}. + + + +[Pod Priority](/docs/concepts/configuration/pod-priority-preemption/#pod-priority) da la habilidad de configurar prioridades del programador de un {{< glossary_tooltip text="Pod" term_id="pod" >}} más altas o bajas que otros {{< glossary_tooltip text="Pods" term_id="pod" >}} - lo cual es una característica importante para cargas de trabajo en producción. From d6bcc7b8d990c4114a7c9b09a4c0ecd5b16e0947 Mon Sep 17 00:00:00 2001 From: Kohei Toyoda Date: Sat, 10 Aug 2019 06:33:15 +0900 Subject: [PATCH 43/49] Fix orders of yaml for examples. (#15763) --- content/en/examples/admin/resource/pvc-limit-greater.yaml | 2 +- content/en/examples/admin/resource/pvc-limit-lower.yaml | 2 +- content/en/examples/application/mysql/mysql-pv.yaml | 2 +- content/en/examples/debug/fluentd-gcp-configmap.yaml | 2 +- content/en/examples/federation/policy-engine-service.yaml | 4 ++-- content/en/examples/pods/pod-projected-svc-token.yaml | 2 +- content/en/examples/pods/storage/pv-claim.yaml | 2 +- content/en/examples/pods/storage/pv-pod.yaml | 2 +- content/en/examples/pods/storage/pv-volume.yaml | 2 +- content/en/examples/service/access/hello-service.yaml | 2 +- 10 files changed, 11 insertions(+), 11 deletions(-) diff --git a/content/en/examples/admin/resource/pvc-limit-greater.yaml b/content/en/examples/admin/resource/pvc-limit-greater.yaml index 4728bb6834..2d92bf92b3 100644 --- a/content/en/examples/admin/resource/pvc-limit-greater.yaml +++ b/content/en/examples/admin/resource/pvc-limit-greater.yaml @@ -1,5 +1,5 @@ -kind: PersistentVolumeClaim apiVersion: v1 +kind: PersistentVolumeClaim metadata: name: pvc-limit-greater spec: diff --git a/content/en/examples/admin/resource/pvc-limit-lower.yaml b/content/en/examples/admin/resource/pvc-limit-lower.yaml index edb1a9c66b..ef819b6292 100644 --- a/content/en/examples/admin/resource/pvc-limit-lower.yaml +++ b/content/en/examples/admin/resource/pvc-limit-lower.yaml @@ -1,5 +1,5 @@ -kind: PersistentVolumeClaim apiVersion: v1 +kind: PersistentVolumeClaim metadata: name: pvc-limit-lower spec: diff --git a/content/en/examples/application/mysql/mysql-pv.yaml b/content/en/examples/application/mysql/mysql-pv.yaml index 6f4e692f3b..c89779a83f 100644 --- a/content/en/examples/application/mysql/mysql-pv.yaml +++ b/content/en/examples/application/mysql/mysql-pv.yaml @@ -1,5 +1,5 @@ -kind: PersistentVolume apiVersion: v1 +kind: PersistentVolume metadata: name: mysql-pv-volume labels: diff --git a/content/en/examples/debug/fluentd-gcp-configmap.yaml b/content/en/examples/debug/fluentd-gcp-configmap.yaml index 71e0ac5d82..4abdbc8b91 100644 --- a/content/en/examples/debug/fluentd-gcp-configmap.yaml +++ b/content/en/examples/debug/fluentd-gcp-configmap.yaml @@ -1,5 +1,5 @@ -kind: ConfigMap apiVersion: v1 +kind: ConfigMap data: containers.input.conf: |- # This configuration file for Fluentd is used diff --git a/content/en/examples/federation/policy-engine-service.yaml b/content/en/examples/federation/policy-engine-service.yaml index 287a972d64..982870b06b 100644 --- a/content/en/examples/federation/policy-engine-service.yaml +++ b/content/en/examples/federation/policy-engine-service.yaml @@ -1,5 +1,5 @@ -kind: Service apiVersion: v1 +kind: Service metadata: name: opa namespace: federation-system @@ -10,4 +10,4 @@ spec: - name: http protocol: TCP port: 8181 - targetPort: 8181 \ No newline at end of file + targetPort: 8181 diff --git a/content/en/examples/pods/pod-projected-svc-token.yaml b/content/en/examples/pods/pod-projected-svc-token.yaml index df8ebc1925..985073c8d3 100644 --- a/content/en/examples/pods/pod-projected-svc-token.yaml +++ b/content/en/examples/pods/pod-projected-svc-token.yaml @@ -1,5 +1,5 @@ -kind: Pod apiVersion: v1 +kind: Pod metadata: name: nginx spec: diff --git a/content/en/examples/pods/storage/pv-claim.yaml b/content/en/examples/pods/storage/pv-claim.yaml index 197db917c8..b33f6faa4c 100644 --- a/content/en/examples/pods/storage/pv-claim.yaml +++ b/content/en/examples/pods/storage/pv-claim.yaml @@ -1,5 +1,5 @@ -kind: PersistentVolumeClaim apiVersion: v1 +kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: diff --git a/content/en/examples/pods/storage/pv-pod.yaml b/content/en/examples/pods/storage/pv-pod.yaml index 7bae89e339..a2eddcf088 100644 --- a/content/en/examples/pods/storage/pv-pod.yaml +++ b/content/en/examples/pods/storage/pv-pod.yaml @@ -1,5 +1,5 @@ -kind: Pod apiVersion: v1 +kind: Pod metadata: name: task-pv-pod spec: diff --git a/content/en/examples/pods/storage/pv-volume.yaml b/content/en/examples/pods/storage/pv-volume.yaml index 33ca6843f4..36fe3c5424 100644 --- a/content/en/examples/pods/storage/pv-volume.yaml +++ b/content/en/examples/pods/storage/pv-volume.yaml @@ -1,5 +1,5 @@ -kind: PersistentVolume apiVersion: v1 +kind: PersistentVolume metadata: name: task-pv-volume labels: diff --git a/content/en/examples/service/access/hello-service.yaml b/content/en/examples/service/access/hello-service.yaml index 1e4c7a6c32..71344ecb8b 100644 --- a/content/en/examples/service/access/hello-service.yaml +++ b/content/en/examples/service/access/hello-service.yaml @@ -1,5 +1,5 @@ -kind: Service apiVersion: v1 +kind: Service metadata: name: hello spec: From 5c41bb04825c13020564483034173e30ecf859c9 Mon Sep 17 00:00:00 2001 From: Gabriele Cimato Date: Fri, 9 Aug 2019 14:47:13 -0700 Subject: [PATCH 44/49] docs: update deploy-intro, fix node js sentence (#15758) --- .../tutorials/kubernetes-basics/deploy-app/deploy-intro.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html b/content/en/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html index 8b3604c844..9c7a36916b 100644 --- a/content/en/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html +++ b/content/en/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html @@ -92,7 +92,7 @@ weight: 10

For your first Deployment, you'll use a Node.js application packaged in a Docker container. (If you didn't already try creating a - Node.js application and deploying a Node.js application and deploying it using a container, you can do that first by following the + Node.js application and deploying it using a container, you can do that first by following the instructions from the Hello Minikube tutorial).

From 8344dda2f6a7f09bdd1f47302c5e8ac0ec851f71 Mon Sep 17 00:00:00 2001 From: Zane Bitter Date: Sun, 11 Aug 2019 20:53:10 -0400 Subject: [PATCH 45/49] Use literals in jsonpath examples (#15775) It's very difficult to figure out e.g. the correct quotes to use from regular text that is subject to being converted for typographers quotes. Use code literals instead to ensure that nothing is modified, and to distinguish literals from accompanying text. Signed-off-by: Zane Bitter --- content/en/docs/reference/kubectl/jsonpath.md | 26 +++++++++---------- content/fr/docs/reference/kubectl/jsonpath.md | 26 +++++++++---------- content/zh/docs/reference/kubectl/jsonpath.md | 24 ++++++++--------- 3 files changed, 38 insertions(+), 38 deletions(-) diff --git a/content/en/docs/reference/kubectl/jsonpath.md b/content/en/docs/reference/kubectl/jsonpath.md index 1eed8c22be..457129e315 100644 --- a/content/en/docs/reference/kubectl/jsonpath.md +++ b/content/en/docs/reference/kubectl/jsonpath.md @@ -65,18 +65,18 @@ Given the JSON input: } ``` -Function | Description | Example | Result -------------------|---------------------------|---------------------------------------------------------------|------------------ -text | the plain text | kind is {.kind} | kind is List -@ | the current object | {@} | the same as input -. or [] | child operator | {.kind} or {['kind']} | List -.. | recursive descent | {..name} | 127.0.0.1 127.0.0.2 myself e2e -\* | wildcard. Get all objects | {.items[*].metadata.name} | [127.0.0.1 127.0.0.2] -[start:end :step] | subscript operator | {.users[0].name} | myself -[,] | union operator | {.items[*]['metadata.name', 'status.capacity']} | 127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8] -?() | filter | {.users[?(@.name=="e2e")].user.password} | secret -range, end | iterate list | {range .items[*]}[{.metadata.name}, {.status.capacity}] {end} | [127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]] -'' | quote interpreted string | {range .items[*]}{.metadata.name}{'\t'}{end} | 127.0.0.1 127.0.0.2 +Function | Description | Example | Result +--------------------|---------------------------|-----------------------------------------------------------------|------------------ +`text` | the plain text | `kind is {.kind}` | `kind is List` +`@` | the current object | `{@}` | the same as input +`.` or `[]` | child operator | `{.kind}` or `{['kind']}` | `List` +`..` | recursive descent | `{..name}` | `127.0.0.1 127.0.0.2 myself e2e` +`*` | wildcard. Get all objects | `{.items[*].metadata.name}` | `[127.0.0.1 127.0.0.2]` +`[start:end :step]` | subscript operator | `{.users[0].name}` | `myself` +`[,]` | union operator | `{.items[*]['metadata.name', 'status.capacity']}` | `127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8]` +`?()` | filter | `{.users[?(@.name=="e2e")].user.password}` | `secret` +`range`, `end` | iterate list | `{range .items[*]}[{.metadata.name}, {.status.capacity}] {end}` | `[127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]]` +`''` | quote interpreted string | `{range .items[*]}{.metadata.name}{'\t'}{end}` | `127.0.0.1 127.0.0.2` Examples using `kubectl` and JSONPath expressions: @@ -95,4 +95,4 @@ C:\> kubectl get pods -o=jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.stat C:\> kubectl get pods -o=jsonpath="{range .items[*]}{.metadata.name}{\"\t\"}{.status.startTime}{\"\n\"}{end}" ``` -{{% /capture %}} \ No newline at end of file +{{% /capture %}} diff --git a/content/fr/docs/reference/kubectl/jsonpath.md b/content/fr/docs/reference/kubectl/jsonpath.md index 5ec93be569..c77167e62e 100644 --- a/content/fr/docs/reference/kubectl/jsonpath.md +++ b/content/fr/docs/reference/kubectl/jsonpath.md @@ -67,18 +67,18 @@ En plus de la syntaxe de modèle JSONPath originale, les fonctions et syntaxes s } ``` -Fonction | Description | Exemple | Résultat -------------------|----------------------------|---------------------------------------------------------------|------------------ -text | le texte en clair | le type est {.kind} | le type est List -@ | l'objet courant | {@} | identique à l'entrée -. or [] | opérateur fils | {.kind} ou {['kind']} | List -.. | descente récursive | {..name} | 127.0.0.1 127.0.0.2 myself e2e -\* | joker. Tous les objets | {.items[*].metadata.name} | [127.0.0.1 127.0.0.2] -[start:end :step] | opérateur d'indice | {.users[0].name} | myself -[,] | opérateur d'union | {.items[*]['metadata.name', 'status.capacity']} | 127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8] -?() | filtre | {.users[?(@.name=="e2e")].user.password} | secret -range, end | itération de liste | {range .items[*]}[{.metadata.name}, {.status.capacity}] {end} | [127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]] -'' | protège chaîne interprétée | {range .items[*]}{.metadata.name}{'\t'}{end} | 127.0.0.1 127.0.0.2 +Fonction | Description | Exemple | Résultat +--------------------|----------------------------|-----------------------------------------------------------------|------------------ +`text` | le texte en clair | `le type est {.kind}` | `le type est List` +`@` | l'objet courant | `{@}` | identique à l'entrée +`.` ou `[]` | opérateur fils | `{.kind}` ou `{['kind']}` | `List` +`..` | descente récursive | `{..name}` | `127.0.0.1 127.0.0.2 myself e2e` +`*` | joker. Tous les objets | `{.items[*].metadata.name}` | `[127.0.0.1 127.0.0.2]` +`[start:end :step]` | opérateur d'indice | `{.users[0].name}` | `myself` +`[,]` | opérateur d'union | `{.items[*]['metadata.name', 'status.capacity']}` | `127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8]` +`?()` | filtre | `{.users[?(@.name=="e2e")].user.password}` | `secret` +`range`, `end` | itération de liste | `{range .items[*]}[{.metadata.name}, {.status.capacity}] {end}` | `[127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]]` +`''` | protège chaîne interprétée | `{range .items[*]}{.metadata.name}{'\t'}{end}` | `127.0.0.1 127.0.0.2` Exemples utilisant `kubectl` et des expressions JSONPath : @@ -97,4 +97,4 @@ C:\> kubectl get pods -o=jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.stat C:\> kubectl get pods -o=jsonpath="{range .items[*]}{.metadata.name}{\"\t\"}{.status.startTime}{\"\n\"}{end}" ``` -{{% /capture %}} \ No newline at end of file +{{% /capture %}} diff --git a/content/zh/docs/reference/kubectl/jsonpath.md b/content/zh/docs/reference/kubectl/jsonpath.md index e85041951c..3d202ca567 100644 --- a/content/zh/docs/reference/kubectl/jsonpath.md +++ b/content/zh/docs/reference/kubectl/jsonpath.md @@ -49,15 +49,15 @@ JSONPath 模板由 {} 包起来的 JSONPath 表达式组成。 ] } ``` -| 函数 | 描述 | 示例 | 结果 | -| ----------------- | ---------- | ---------------------------------------- | ---------------------------------------- | -| text | 纯文本 | kind is {.kind} | kind is List | -| @ | 当前对象 | {@} | 与输入相同 | -| . or [] | 子运算符 | {.kind} 或者 {['kind']} | List | -| .. | 递归下降 | {..name} | 127.0.0.1 127.0.0.2 myself e2e | -| * | 通配符，获取所有对象 | {.items[*].metadata.name} | [127.0.0.1 127.0.0.2] | -| [start:end :step] | 下标运算符 | {.users[0].name} | myself | -| [,] | 并集运算符 | {.items[*]['metadata.name', 'status.capacity']} | 127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8] | -| ?() | 过滤 | {.users[?(@.name=="e2e")].user.password} | secret | -| range, end | 迭代列表 | {range .items[*]}[{.metadata.name}, {.status.capacity}] {end} | [127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]] | -| "" | 引用解释执行字符串 | {range .items[*]}{.metadata.name}{"\t"}{end} | 127.0.0.1 127.0.0.2 | \ No newline at end of file +| 函数 | 描述 | 示例 | 结果 | +| ------------------- | ---------- | ------------------------------------------ | ------------------------------------------ | +| `text` | 纯文本 | `kind is {.kind}` | `kind is List` | +| `@` | 当前对象 | `{@}` | 与输入相同 | +| `.` 或者 `[]` | 子运算符 | `{.kind}` 或者 `{['kind']}` | `List` | +| `..` | 递归下降 | `{..name}` | `127.0.0.1 127.0.0.2 myself e2e` | +| `*` | 通配符，获取所有对象 | `{.items[*].metadata.name}` | `[127.0.0.1 127.0.0.2]` | +| `[start:end :step]` | 下标运算符 | `{.users[0].name}` | `myself` | +| `[,]` | 并集运算符 | `{.items[*]['metadata.name', 'status.capacity']}` | `127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8]` | +| `?()` | 过滤 | `{.users[?(@.name=="e2e")].user.password}` | `secret` | +| `range`, `end` | 迭代列表 | `{range .items[*]}[{.metadata.name}, {.status.capacity}] {end}` | `[127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]]` | +| `''` | 引用解释执行字符串 | `{range .items[*]}{.metadata.name}{'\t'}{end}` | `127.0.0.1 127.0.0.2` | From c5add88844bbfae97df84c877f6240e1772b1d6a Mon Sep 17 00:00:00 2001 From: Joe Date: Mon, 12 Aug 2019 08:57:11 +0800 Subject: [PATCH 46/49] Update overview toc in chinese. (#15737) --- content/zh/docs/concepts/overview/_index.md | 4 ++++ content/zh/docs/concepts/overview/components.md | 1 + content/zh/docs/concepts/overview/kubernetes-api.md | 5 +++++ content/zh/docs/concepts/overview/what-is-kubernetes.md | 1 + .../zh/docs/concepts/overview/working-with-objects/_index.md | 4 ++++ .../overview/working-with-objects/kubernetes-objects.md | 1 + 6 files changed, 16 insertions(+) create mode 100755 content/zh/docs/concepts/overview/_index.md create mode 100755 content/zh/docs/concepts/overview/working-with-objects/_index.md diff --git a/content/zh/docs/concepts/overview/_index.md b/content/zh/docs/concepts/overview/_index.md new file mode 100755 index 0000000000..9ab3e94ae4 --- /dev/null +++ b/content/zh/docs/concepts/overview/_index.md @@ -0,0 +1,4 @@ +--- +title: "概述" +weight: 20 +--- \ No newline at end of file diff --git a/content/zh/docs/concepts/overview/components.md b/content/zh/docs/concepts/overview/components.md index e01069d409..124f5a2d78 100644 --- a/content/zh/docs/concepts/overview/components.md +++ b/content/zh/docs/concepts/overview/components.md @@ -6,6 +6,7 @@ redirect_from: - "/docs/admin/cluster-components/" - "/docs/admin/cluster-components.html" content_template: templates/concept +weight: 20 --- {{% capture overview %}} 本文档概述了 Kubernetes 所需的各种二进制组件, 用于提供齐全的功能。 diff --git a/content/zh/docs/concepts/overview/kubernetes-api.md b/content/zh/docs/concepts/overview/kubernetes-api.md index 9f79f25aa4..9a920f043f 100644 --- a/content/zh/docs/concepts/overview/kubernetes-api.md +++ b/content/zh/docs/concepts/overview/kubernetes-api.md @@ -1,3 +1,8 @@ +--- +title: Kubernetes API +weight: 30 +--- + # Kubernetes API 概述 [API协议文档](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md)描述了主系统和API概念。 diff --git a/content/zh/docs/concepts/overview/what-is-kubernetes.md b/content/zh/docs/concepts/overview/what-is-kubernetes.md index 58c4aad000..53cef26eca 100644 --- a/content/zh/docs/concepts/overview/what-is-kubernetes.md +++ b/content/zh/docs/concepts/overview/what-is-kubernetes.md @@ -3,6 +3,7 @@ approvers: - k8s-merge-robot title: 认识 Kubernetes? +weight: 10 --- Kubernetes 是一个跨主机集群的 [开源的容器调度平台，它可以自动化应用容器的部署、扩展和操作](http://www.slideshare.net/BrianGrant11/wso2con-us-2015-kubernetes-a-platform-for-automating-deployment-scaling-and-operations) , 提供以容器为中心的基础架构。 diff --git a/content/zh/docs/concepts/overview/working-with-objects/_index.md b/content/zh/docs/concepts/overview/working-with-objects/_index.md new file mode 100755 index 0000000000..5dcf6e4ed5 --- /dev/null +++ b/content/zh/docs/concepts/overview/working-with-objects/_index.md @@ -0,0 +1,4 @@ +--- +title: "使用 Kubernetes 对象" +weight: 40 +--- \ No newline at end of file diff --git a/content/zh/docs/concepts/overview/working-with-objects/kubernetes-objects.md b/content/zh/docs/concepts/overview/working-with-objects/kubernetes-objects.md index f75a2bf352..14c85b0862 100644 --- a/content/zh/docs/concepts/overview/working-with-objects/kubernetes-objects.md +++ b/content/zh/docs/concepts/overview/working-with-objects/kubernetes-objects.md @@ -5,6 +5,7 @@ redirect_from: - "/docs/concepts/abstractions/overview/" - "/docs/concepts/abstractions/overview.html" content_template: templates/concept +weight: 10 --- {{% capture overview %}} From aea0fefe9698d47addf36bc2c36764b3efc1ca99 Mon Sep 17 00:00:00 2001 From: Kohei Ota Date: Mon, 12 Aug 2019 09:59:10 +0900 Subject: [PATCH 47/49] remove study (#15805) --- content/en/case-studies/spotify/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/case-studies/spotify/index.html b/content/en/case-studies/spotify/index.html index f408d217b6..67f2c848ee 100644 --- a/content/en/case-studies/spotify/index.html +++ b/content/en/case-studies/spotify/index.html @@ -1,5 +1,5 @@ --- -title: Spotify study Case Study +title: Spotify Case Study linkTitle: Spotify case_study_styles: true cid: caseStudies From 16ae99f229a5c1d0104b3757db2ac7a3e8356386 Mon Sep 17 00:00:00 2001 From: Max <16919345+mkorbi@users.noreply.github.com> Date: Mon, 12 Aug 2019 10:11:13 +0200 Subject: [PATCH 48/49] German translation for proxies and controller metrics (#15746) * i18n cloud controller * add controller metrics * 2nd check on controller metrics * add proxies * check proxies * add controller metrics * 2nd check on controller metrics * add proxies * check proxies * Revert "Merge branch 'i18n-003' of github.com:mkorbi/website into i18n-003" This reverts commit 76bf403bd12bdff8f9a02f19cb125fcce333b4f8. --- .../controller-metrics.md | 41 ++++++++++++ .../cluster-administration/proxies.md | 64 +++++++++++++++++++ 2 files changed, 105 insertions(+) create mode 100644 content/de/docs/concepts/cluster-administration/controller-metrics.md create mode 100644 content/de/docs/concepts/cluster-administration/proxies.md diff --git a/content/de/docs/concepts/cluster-administration/controller-metrics.md b/content/de/docs/concepts/cluster-administration/controller-metrics.md new file mode 100644 index 0000000000..2444c5d37a --- /dev/null +++ b/content/de/docs/concepts/cluster-administration/controller-metrics.md @@ -0,0 +1,41 @@ +--- +title: Controller Manager Metriken +content_template: templates/concept +weight: 100 +--- + +{{% capture overview %}} +Controller Manager Metriken liefern wichtige Erkenntnisse über die Leistung und den Zustand von den Controller Managern. + +{{% /capture %}} + +{{% capture body %}} +## Was sind Controller Manager Metriken + +Die Kennzahlen des Controller Managers liefert wichtige Erkenntnisse über die Leistung und den Zustand des Controller Managers. +Diese Metriken beinhalten gängige Go Language Laufzeitmetriken wie go_routine count und controller-spezifische Metriken wie z.B. +etcd Request Latenzen oder Cloud Provider (AWS, GCE, OpenStack) API Latenzen, die verwendet werden können um den Zustand eines Clusters zu messen. + +Ab Kubernetes 1.7 stehen detaillierte Cloud Provider Metriken für den Speicherbetrieb für GCE, AWS, Vsphere und OpenStack zur Verfügung. +Diese Metriken können verwendet werden, um den Zustand persistenter Datenträgeroperationen zu überwachen. + +Für GCE werden diese Metriken beispielsweise wie folgt aufgerufen: + +``` +cloudprovider_gce_api_request_duration_seconds { request = "instance_list"} +cloudprovider_gce_api_request_duration_seconds { request = "disk_insert"} +cloudprovider_gce_api_request_duration_seconds { request = "disk_delete"} +cloudprovider_gce_api_request_duration_seconds { request = "attach_disk"} +cloudprovider_gce_api_request_duration_seconds { request = "detach_disk"} +cloudprovider_gce_api_request_duration_seconds { request = "list_disk"} +``` + +## Konfiguration + +In einem Cluster sind die Controller Manager Metriken unter `http://localhost:10252/metrics` auf dem Host verfügbar, auf dem der Controller Manager läuft. + +Die Metriken werden im [Prometheus Format](https://prometheus.io/docs/instrumenting/exposition_formats/) ausgegeben. + +In einer Produktionsumgebung können Sie Prometheus oder einen anderen Metrik Scraper konfigurieren, um diese Metriken regelmäßig zu sammeln und in einer Art Zeitreihen Datenbank verfügbar zu machen. + +{{% /capture %}} \ No newline at end of file diff --git a/content/de/docs/concepts/cluster-administration/proxies.md b/content/de/docs/concepts/cluster-administration/proxies.md new file mode 100644 index 0000000000..7422d55135 --- /dev/null +++ b/content/de/docs/concepts/cluster-administration/proxies.md @@ -0,0 +1,64 @@ +--- +title: Proxies in Kubernetes +content_template: templates/concept +weight: 90 +--- + +{{% capture overview %}} +Auf dieser Seite werden die im Kubernetes verwendeten Proxies erläutert. +{{% /capture %}} + +{{% capture body %}} + +## Proxies + +Es gibt mehrere verschiedene Proxies, die die bei der Verwendung von Kubernetes begegnen können: + +1. Der [kubectl Proxy](/docs/tasks/access-application-cluster/access-cluster/#directly-accessing-the-rest-api): + + - läuft auf dem Desktop eines Benutzers oder in einem Pod + - Proxy von einer lokalen Host-Adresse zum Kubernetes API Server + - Client zu Proxy verwendet HTTP + - Proxy zu API Server verwendet HTTPS + - lokalisiert den API Server + - fügt Authentifizierungs-Header hinzu + +1. Der [API Server Proxy](/docs/tasks/access-application-cluster/access-cluster/#discovering-builtin-services): + + - ist eine Bastion, die in den API Server eingebaut ist + - verbindet einen Benutzer außerhalb des Clusters mit Cluster IPs, die sonst möglicherweise nicht erreichbar wären + - läuft im API Server Prozess + - Client zu Proxy verwendet HTTPS (oder http, wenn API Server so konfiguriert ist) + - Proxy zum Ziel kann HTTP oder HTTPS verwenden, der Proxy wählt dies unter Verwendung der verfügbaren Informationen aus + - kann verwendet werden, um einen Knoten, Pod oder Service zu erreichen + - führt einen Lastausgleich durch um einen Service zu erreichen, wenn dieser verwendet wird + +1. Der [kube Proxy](/docs/concepts/services-networking/service/#ips-and-vips): + + - läuft auf jedem Knoten + - Proxy unterstüzt UDP, TCP und SCTP + - versteht kein HTTP + - stellt Lastausgleich zur Verfügung + - wird nur zum erreichen von Services verwendet + +1. Ein Proxy/Load-balancer vor dem API Server: + + - Existenz und Implementierung variieren von Cluster zu Cluster (z.B. nginx) + - sitzt zwischen allen Clients und einem oder mehreren API Servern + - fungiert als Load Balancer, wenn es mehrere API Server gibt + +1. Cloud Load Balancer für externe Services: + + - wird von einigen Cloud Anbietern angeboten (z.B. AWS ELB, Google Cloud Load Balancer) + - werden automatisch erstellt, wenn der Kubernetes Service den Typ `LoadBalancer` hat + - unterstützt normalerweiße nur UDP/TCP + - Die SCTP-Unterstützung hängt von der Load Balancer Implementierung des Cloud Providers ab + - die Implementierung variiert je nach Cloud Anbieter + +Kubernetes Benutzer müssen sich in der Regel um nichts anderes als die ersten beiden Typen kümmern. Der Cluster Administrator stellt in der Regel sicher, dass die letztgenannten Typen korrekt eingerichtet sind. + +## Anforderung an Umleitungen + +Proxies haben die Möglichkeit der Umleitung (redirect) ersetzt. Umleitungen sind veraltet. + +{{% /capture %}} \ No newline at end of file From 4aa772302740f50c0301493ddf3ab8e88df5f629 Mon Sep 17 00:00:00 2001 From: Xiang Dai <764524258@qq.com> Date: Mon, 12 Aug 2019 20:19:12 +0800 Subject: [PATCH 49/49] Update disruptions.md (#15811) The number of StatefulSet always start from 0, change to make docs exact. Signed-off-by: Xiang Dai <764524258@qq.com> --- content/en/docs/concepts/workloads/pods/disruptions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/workloads/pods/disruptions.md b/content/en/docs/concepts/workloads/pods/disruptions.md index 6ef7ac8f8b..00265e0433 100644 --- a/content/en/docs/concepts/workloads/pods/disruptions.md +++ b/content/en/docs/concepts/workloads/pods/disruptions.md @@ -169,8 +169,8 @@ The deployment notices that one of the pods is terminating, so it creates a repl called `pod-d`. Since `node-1` is cordoned, it lands on another node. Something has also created `pod-y` as a replacement for `pod-x`. -(Note: for a StatefulSet, `pod-a`, which would be called something like `pod-1`, would need -to terminate completely before its replacement, which is also called `pod-1` but has a +(Note: for a StatefulSet, `pod-a`, which would be called something like `pod-0`, would need +to terminate completely before its replacement, which is also called `pod-0` but has a different UID, could be created. Otherwise, the example applies to a StatefulSet as well.) Now the cluster is in this state: